Finding Winning Tickets with Limited (or No) Supervision
Authors: Mathilde Caron
Ari Morcos
Piotr Bojanowski
Julien Mairal
Armand Joulin
Venue:  (2020)
Keywords: Lottery Tickets Hypothesis
Self-Supervised Learning
Deep Learning
Image Recognition

Abstract: The lottery ticket hypothesis argues that neural networks contain sparse subnetworks, which, if appropriately initialized (the winning tickets), are capable of matching the accuracy of the full network when trained in isolation. Empirically made in different contexts, such an observation opens interesting questions about the dynamics of neural network optimization and the importance of their initializations. However, the properties of winning tickets are not well understood, especially the importance of supervision in the generating process. In this paper, we aim to answer the following open questions: can we find winning tickets with few data samples or few labels? can we even obtain good tickets without supervision? Perhaps surprisingly, we provide a positive answer to both, by generating winning tickets with limited access to data, or with self-supervision---thus without using manual annotations---and then demonstrating the transferability of the tickets to challenging classification tasks such as ImageNet.


Under review as a conference paper at ICLR 2020
FINDING WINNING TICKETS WITH LIMITED (OR NO) SUPERVISION
Anonymous authors Paper under double-blind review
ABSTRACT
The lottery ticket hypothesis argues that neural networks contain sparse subnet- works, which, if appropriately initialized (the winning tickets), capable of matching the accuracy of the full network when trained in isolation. Empirically made in different contexts, such an observation opens interesting questions about the dynamics of neural network optimization and the importance of their initial- izations. However, the properties of winning tickets are well understood, es- pecially the importance of supervision in the generating process. In this paper, we aim answer the following open questions: can we find winning tickets with few data samples or few labels? can we even obtain "good" tickets without su- pervision? We find that winning tickets found in these scenarios are, perhaps sur- prisingly, competitive with winning tickets generated on the full ImageNet dataset when evaluated on ImageNet classification task.
INTRODUCTION
Recently, Frankle & Carbin (0119) have observed that sparse subnetworks of over-parameterized neural networks could achieve good predictions when trained in isolation, as long as they are appro- priately initialized; these "lucky" starting points have been termed winning tickets. Building these tickets typically achieved by pruming the weights with lowest magnitude of an over wer-parametrized network that has been trained to convergence, before resetting the remaining weights to their initial values, or at some point early in training, and repeating the procedure. The properties of these ef- fective subnetworks and their initialization are however not well understood yet. For example, the original lottery ticket hypothesis argued that these winning tickets emerge from the network initial- ization. Yet, original experiments were performed on classification tasks with shallow architectures and small datasets, and further experiments on deeper models and more challenging problems sug- gest that they may rather appear later, though still early, during training (Frankle et al. (019). Recent works (Morcos et al. 2019; Yu et al. 2019 Zhou et al. 2019) have studied other properties of the winning tickets generation process: Morcos al. (2019) have shown that winning tickets can be transferred across datasets with similar natural image statistics, and Yu et al. (2019) have exposed the existence of lottery tickets to other domains, such as text and reinforcement learning.
In this paper, we expand on this line of work and empirically answer other important open questions on winning tickets and the data-labels distribution on which they are generated. In particular, we want to know if "good" tickets can be obtained when few data samples, or few labels, or even no labels, are available. Answering these questions is important since it may speed-up the winning ticket generation process, which is computationnally expensive, and also open new perspectives about generating sparse subnetworks that may be trained efficiently on new tasks.
To this effect, we design several experiments that isolate the impact of the data and label distributions on the quality of the resulting winning tickets. First, we analyze the extent to which winning tickets are label-dependent by generating "label-agnostic" winning tickets with self-supervised tasks (Gi- daris et al. 2018; Doersch & Zisserman 2017). Then, we evaluate the impact of reducing the num- bers of samples per class the number of classes used for winning ticket generation. Finally, evaluate the performance of semi sumivisuereieed approaches to winning ticket generation. Most of our experiments are conducted on ImageNet; we remark indeed that deep networks trained smaller

---

Under review as a conference paper at ICLR 2020
datasets such CIFAR-10 are already sparse at convergence, making pruning less challenging, and conclusions drawn about lottery tickets potentially misleading if this effect is not accounted for.
Overall, our experiments show that winning tickets are surprssnggy robust to many of these data/label distribution changes. Indeed, using only 10% of the datasct or removing entirely the labels still leads to effective winning tickets. As result, we manage to extract winning tickets 5x faster by training on a subset of data, with no modifications to the core algorithm. This is of partic- ular interest because finding winning tickets is highly data, labels and resource-demanding since requires training a network to convergence at least once (and generally dozens of times in practice).
Our paper makes the following contributions: (i) We combine self-supervised learning with winning tickets generation, showing that good winning tickets can be found without labels. (ii) We show that finding winning tickets can be accelerated by factor 5x on ImageNet by using only a subset of the data. (iii) We also show that using large datasets is important to study lottery tickets, since deep networks trained on CIFAR-10 10 are natually sparse, making conclusions potentially misleading.
RELATED WORK
Pruning. Pruning is an approach to model compression (Han et al., (013)) and regularization (Le-
Cun et al. 1990) in which weights or nodes/filters are removed, typically by clamping them to zero
an active research which has primarily focused on pruning an already trained network (Han)
ett al. 2015 Li et al. 2017) pruning while training (Prakash al. 2019). In particular, the it- crative pruning during training Han et al.] (2015) has been extended to continuous pruming (Guo ct al. 2016), layer-wise pruning (Dong et al. (217)) and with weight sharing (Ullrich et al. 2017).
Pruning during training has been considered with Eg regularization (Louizos et al.) 2018). 8). binary
convolution (Rastegari al. 2016) using the hashing trick for weight sharing (Chen al.. 2015
There are several ways to prune a network; among them are unstructured pruning (LeCun et al., 1990 [Han et al.) 2015) which corresponds discarding individual connections structured prun- ing that operates by removing entire channels or layers (see Liu et al. (2019) for a review of the different methods). The lottery ticket hypothesis focuses on unstructured pruning early in training, which has the advantage of significantly reducing the training cost of the resulting model.
The lottery tickets hypothesis. The lottery tickets hypothesis (Frankle & Carbin 2019) explores the possibility of pruning early in training by revealing that some sparse subnetworks inside neural networks can reach accuracy matching that of the full network when trained in isolation. Setting the weights of the sparse architecture appropriately is critical reach good performance. |Frankle & Carbin (2019) provide a proof of concept of the lottery ticket hypothesis on small vision benchmarks while Frankle al. (2019) conduct further experiments with deeper networks, which result in the introduction of rewinding and a consequent revision to the original hypothesis. Liu et al. (2019) question the importance of weights resetting and observe for moderate pruning rates and without rewinding that the pruned architecture alone is responsible for successful training. Zhou et al. (2019) conduct ablation studies on the lottery tickets hypothesis and show, among others, the importance of the signs of the reset weights. Similar to our paper, these works aim at better understanding winning tickets properties; however none of them investigate their relationship with the data and labels on which they trained, though Yu et al. (2019) investigate lottery tickets in reinforcement learning problems. Closer to our work. Morcos ct al. (2019) show that winning tickets initializations transfer across different image classification datasets, thus suggesting that winning tickets do not entirely overfit to the particular data distribution on which they are found. Here, we follow up on these observations by measuring the importance of labels and data size to the quality of winning tickets.
Learning without supervision. As we study situations with limited access to labels, our work is also related to self-supervised learning where a network is trained on pretext task that does not re- quire any manual annotations. Two main broad types of self-supervised learning approaches appear in the litterature. The first one consists of methods where the pretext task is created by manipu- lating the input data. This includes predicting relative spatial location, colorizing grayscale images or predicting the rotation applied to an image (Doersch et al., 2015: Wang & Guptal 2015] Zhang et al.) 2016: Noroozi & Favaro| 2016 Pathak et al.) [017] Gidaris et al. 2018 Tian et al. 2019 Hénaff et al. 2019). The second one is composed of methods (Dosovitskiy et al. 2016 Bojanowski
& Joulin 2017 Wu et al. 2018 Caron et al. 2018) where images are treated different instances
2

---

Under review as a conference paper at ICLR 2020
that should be discriminated from another. Representations leart using self-supervision most often evaluated via transfer learning to supervised task. The better the pre-training with self- supervised learning, the better the performance the transfer task. In this work, we propose new way of exploiting signal learned with self-supervision by findind winning tickets.
BACKGROUND ON LOTTERY TICKETS AND IMPLEMENTATION
In this section, we provide some background on the lottery tickets hypothesis and more particularly about the procedure to find winning tickets. We also give more details about our implementation.
3.1 EXTRACTING AND TRAINING SUBNETWORKS
We detail in this section the approach of Frankle et al. 2019) to extract sparse and trainable subnet- works from an over-pare porr-arameterized network. These subnetworks, or winning tickets, are generated with a magnitude-bant based unstructured iterative pruning process (Han et al. 2015) and a weight reset- ting scheme (Frankle & Carbin 2019 Frankle et al. 2019).
Subnetwork and dataset. A subnetwork or winning ticket (W, m) represented by the associa- tion of mask m in {0, 13d and weights W in IRd The convolutional network, convnet, function associated with a subnetwork ()))) is denoted by fmow, where @ is element wise product. We consider a training set { .......; (xN, yN)} of N pairs of images and their labels.
Magnitude-based unstructured iterative pruning, Han et al. (2015) propose an algorithm to prune networks by estimating which weights are important. This approach consists of compressing networks by alternatively minimizing a loss Ly and pruning the network parameters with the small- est magnitude, hence progressively reducing the network size. We refer the training objective Lg as the winning ticket generation task. More specifically, at each pruning iteration, the network is first trained to minimize the following problem:
min 1 N
Cy(yr fmow (æu));
thus arriving weights W*, The mask is updated by setting to zero the elements already masked plus the smallest elements of [[W*[]]] míj] + 0}. This method operates as post-processing step to prune an already- trained network while preserving good accuracy at high compression rates.
Weight resetting. In addition to weight pruning, Frankle & Carbin (219)) propose to also find good initialization for each subnetwork, such that may be retrained from scratch. On scall-scale computer vision datasets and with shallow architectures, they indeed show that sub-architectures found with iterative magnitude pruning can be retrained from the start, as long as their weights reset to their initial values. Further experiments, however, have shown that this observation does not exactly hold for more challenging benchmarks such as ImageNet (Frankle et al.) 2019 Liu al. 2019). Specifically, Frankle et al. (2019) found that resetting weights to their value "early in training" can still lead to good trainable subnetworks. Formally, at each pruning iteration, the subnetwork is reset weights Wi obtained after & weight updates from the first pruning iteration.
Winning ticket generation task. Most studies about the lottery tickets hypothesis evaluate win- ning tickets in the same setting (i.e. task and dataset) used find them. other words, the resulting winning tickets performance is measured by re-training them on the same objective Lg that was used the generation process (Eq. D. In this paper, however, we vary the generation loss Cg while the evaluation remains constant. For example, to generate "label-agnostic" winning tickets (i.e. that do not depend on labels), Eg might correspond to a self-supervised loss. We evaluate winning tickets by reporting their performance after training on label classification the full dataset.
3.2 IMPLEMENTATION
Lottery tickets. We follow the lottery tickets setup of Morcos et al. (2019). At each pruning iteration, we globally pruned 20% of the remaining weights. The last fully-connected and batch-
3

---

Under review as a conference paper at ICLR 2020
norm layers parameters are left unpruned. We apply up to 30 pruning iterations to reach extreme pruning rates where only 0.1% of the weights remain. Late resetting is set to 3 x 1, 3. samples, which corresponds to 3 epochs on full ImageNet. More details about the rewinding parameter are in the Appendix
B
We use the "random" baseline of Morcos et al. (2019) (random mask and weights).
Datasets and models. We use CIFAR- 10 (Krizhevsky 2009) or ImageNet (Deng et al. 2009) datasets. On ImageNet, consider AlexNet (Krizhevsky al. 2012) and ResNet-50 (He al. 2016) architectures. On CIFAR-10 we use ResNet-18 and the modified VGG-19 (Simonyan & Zisserman 2014) of Morcos al. (2019) (multilayer perceptron (MLP) replaced by a fully con- nected layer). Models are trained with an i2 penalization of the weights. Details about optimization schemes are in Appendix A We run each experiment with (CIFAR-10) or 3 (ImageNet) random seeds, and report the mean and standard error of the accuracy (tables of results are in Appendix E
EXPERIMENTAL STUDY
In this section, we look at how "label-agnostic" winning tickets transfer to label-dependent task. Then, we investigate if we can find good performing winning tickets with reduced number of samples and/or classes. Finally, we also question if the "label-agnostic" winning tickets are comple- mentary to the tickets found on subset only of data through semi-supervised generation task.
4.1 LABEL--AGOSTEE WINNING TICKETS
In this section, we interested in the properties of "label-agnosttcc winning tickets, i.e. found with self-supervised tasks, w.r.t. to a target supervised task. We compare their performance with winning tickets found with supervision to measure the importance of labels in the winning tickets generation.
Experimental setting. We consider 2 self- self-supervised methods: RotNet (Gidaris al. 2018) and
the Exemplar approach of Dosovitskiy et al. (2016) following Doersch & Zisserman (017)) imple- mentation. RotNet consists predicting the rotation which was applied the input image among set of 4 possible large rotations: {0*900; 180", 270%). Exemplar is a classification problem where each image and its transformations form a class, leading to as many classes as there are training ex- amples. We choose these 2 self-supervised tasks because they have opposite characteristics: RotNet encourages discriminative features to data transformations and has small number of classes, while Exemplar encourages invariance to data transformations and its output space dimension is large.
90",
Transferring label-agnostic winning tickets to supervised task. In Figure we show the performance on label classification of the lahel-agnostic winning tickets at different pruning ratios. These tickets yield good accuracy label classification on both CIFAR-I0 and ImageNet datasets. Indeed, on CIFAR-10 10 (Figure Laj, when only 6.9% of the weights remain, the RotNet winning ticket performs as well as the labels winning ticket. The label-agnostic tickets only fail for AlexNet where most of the weights are in the MLP used in the classifier. An explanation is that, since the MLP corresponds to the last layers of the network, it is greatly influenced by the labels. We investigate this hypothesis in the set of experiments below. Note that, however, on all the other models, the gap of performance between tabel-agnosti tickets and random masks increases when the network is severely pruned. In the extreme pruning regime, the label-agnostic winning tickets perform as well as the ones generated with the supervised task. Results in Appendix G suggest that the quality of the pruned mask itself is similar for labels and self- sepersueee subnetworks but the weights of self- supervised winning tickets are not as good starting points as the ones from labels. Overall, winning tickets are rather robust to the absence of labels.
Layerwise lottery tickets. We further investigate the difference between label-agnostic and super- vised lottery tickets in AlexNet by looking at their performance as we prune up to a given depth. In Figurez] we verify that for the convolutional layers, the label-agnostic tickets match the performance of supervised tickets, and the performance drops only with the MLP. Note that, even this effect is particularly visible with AlexNet, it is happening with most self-superviseell trained network. Indeed, it has been observed that most self-supervised approaches (Jing Tian 2019) produce good shallow and mid-level features but poor high level features. This means that we should expect the
4

---

Under review as a conference paper at ICLR 2020
ResNet-18
VGG-19
0.9
0.9
CIFAR-10 ARR10.0. ac0
0.8
0.8
0.7
Labels
0.7
RotNet
0.6
Exemplar
0.6
Random
0.5
Random adjusted
0.5
100
50
12.56.2 3.1 1.6 0.8
0.2
100
25 12.56.2 3.1
0.4 0.2
% of remaining weights
remaining weights
(a) CIFAR-10
ResNet-5()
AlexNet
0.7
0.7
ImageNet val acc
0.5
0.5
0.3
0.3
0.1
0.1
100 50
25
12.56.2 3.1 1.6 0.8 0.4 0.2
100
50
25 12.56.2 3.1 1.6 0.8 0.4 0.2 % of remaining weights
of remaining weights
(b) ImageNet
Figure 1: We report CIFAR-10 10 test (b) ImageNet val (a) accuracy for winning tickets found with label-agnostic tasks: RotNet Exemplar. The x-axis corresponds different pruning ratios. We compare the performance with supervised winning tickets and random subnetworks (randomly drawn weights from the initialization distribution and randomly permuted masks). On CIFAR-IO, deep models are highly sparse with only ~ 15% of non-zero weights. Thus, we adjust the random baseline to start with the correct mask the natural level of network sparsity 445 for details).
AlexNet: conv
0.6 AlexNet: convs 1-3
0.6 AlexNet: convs 1-5
AlexNet: all
IngNet val. 1
0.6
0.6
Labels
0.5
0.5
0.5
0.5
Romet
Random mask
0.4
100 50 25 12.5 6.2 3.1 of remaining weights
100 50 25 12.5 6.2 3.1 % remaining weights
100 50 25
12.5 6.2
3.1
100
50 25
12.5 6.2 3.1
of remainnin weights
% remaining weights
ImageNet top-1 validation accuracy of AlexNet winning tickets generated by pruning partly or entirely (all) network with 2 generation tasks: labels classification or RotNet. We also show for reference results when the network layers are randomly pruned.
Figure 2:
quality of label-agnostic winning tickets depend on the depth of the pruning. In Appendix C we confirm this intuition with a ResNet-50, and show the difference between mabel-agnosti and supervised lottery tickets increases with depth.
4.2 LOW-SHOT WINNING TICKETS
In this section, interesting in the quality of winning tickets generated with only few anno- tated images. We explore several ways to extract subset by varying its size or its distribution and study the impact of these choices the subsequent winning tickets quality.
5

---

Under review as a conference paper at ICLR 2020
0.7
Net val acc @
ImageNet val acc
0.5
0.5
100% ImNet
100% ImNet
30% ImNet
30% ImNet classes
0.3
0.3
10% ImNet
10% of ImN et classes
ImNet
ImNet classes
0.1
Random mask
0.1
Random musk
100 50
25 12.56.2 3,1 1.6 0.8 0.4 0.2 % of remaining weights
100 50 25 12.5 6.2 3.1 1.6 0.8 0.4 0.2 % of remaining weights
(a) Varying dataset size
(h) Varying number classes
Figure 3: Top-1 accuracy ImageNet validation set of winning tickets generated with different training data. The x-axis represents different levels of winning ticket sparsity.
Experimental setting. We consider either 1%, 10% or 30% of ImageNet, with 1% corresponding to approximately 13k images. When varying the dataset size, we sample the subset per class, i.e., we preserve all the classes. When varying the number of classes, we sample a subset of classes and keep all the images for these classes, i... 10% of ImageNet means that we only have 100 classes.
Varying dataset size. First, we generate winning tickets using a subset of the training data while preserving the same number of classes. Results for CIFAR-10 are in Appendix F Note that generat- ing winning tickets with 10% of ImageNet takes 5x less time than with the full dataset. In Figure Ba show the ImageNet validation accuracy for the winning tickets of ResNet-50 generated with these different subsets. less extreme pruning scenarii, there is a gap of 1 4% between win- ning tickets generated with the full training set and 10% of it but the winning tickets behave in the same ballpark for settings for extreme pruning, i.e., for more than 90% of weights pruned. This experiment shows that winning tickets are not data hungry, especially for extreme pruning.
Varying the number of classes. In Figure 3b] we analyze the impact of changing the joint data- label distribution while keeping the subsets size constant with that of Figure (3b Specifically, we are interested in the impact of the number of classes used during the generation of the tickets. We observe that reducing the number of labels has little effect on the quality the winning tickets. This observation reinforces the findings on label-agnostic generation, that winning tickets are robust to the absence or deterioration of the labels. Even more surprisingly. we observe good performance with only 1% of the classes, i.e., 10 classes and 13k images, leading to 10x faster generation.
0.9
CIFAR-10 aco
Figure 4: CIFAR-10 top-1 test accuracy of win- ning tickets generated by labels classification on randomly permuted labels. Deep models are highly sparse on CIFAR-10 with only 15% of non- weights (see 4.5 for details). For this reason we adjust the random baseline to start with the correct mask for 85% sparsity.
0.8
Labels
0.7
Permated Labels
Randam
0.6
Random adjusted
100
50
25 6..56.2 3.1 1.6 0.8 0.4 0.2 % of remaining weights
Decorrelating label and image distributions. A possible explanation for these positive results about aggosticc and low-shot winning tickets is that winning tickets generated with any task related the right target distribution lead to good tickets. As control experiment, propose

---

Under review as a conference paper at ICLR 2020
ImageNet val acc
0.7
ImageNet val acc
0.7
ImNet
0.5
Supervised: all imgs/ labols
0.5
ImNet: 10% classes
Semi Sup: all imgs 10% labels
ImNet: no labels
Shot: 10% imgs 10% labels
Places-365
Lanel-Agnosti:: imgs 0% labels
CIFAR-10
0.3
Random mask
0.3
Random mask
100
50
25
12.5 6.2 3.1
1.6
100
50
3.1 % of remaining weights
25
12.5
6.2
1.6
of remaining weights
(a) Semi-supervised
(b) Transfer between datasets
Figure 5: (a) ImageNet top-1 validation accuracy of winning tickets generated with different access to both data and labels. We compare using the full dataset, all images but only 10% of labels (semi- supervised), 10% of labels only and all images but no labels (self-supervised), (b) ImageNet top-1 validation accuracy of winning tickets generated from other datasets.
to learn winning tickets on a classification tasks where the labels have been shuffled, i.e., where the image and label distributions have been decorrelated. In Figure we show on CIFAR-I0 the performance of winning tickets generated on randomly permuted labels. As expected, decorrelating the label and the image distributions destroys the quality of the subsequent winning tickets, since they are not better than the adjusted random baseline (see Sec. 4.5 for details on this baseline).
4.3 SEMI-SUPERVISED WINNING TICKETS
In this experiment, we investigate if label-agnostic and low-shot winning tickets are complementary.
Experimental setting. We find winning tickets by training networks in the semi-supervise setting Zhai et al. (2019) with 10% of labeled images from ImageNet and the remaining 90% unlabeled. In particular, the generation task L, corresponds the sum of label classification loss applied to labeled examples only, and RotNet loss applied to all data samples.
Results. In Figure a, we compare these winning tickets with the habel-agnostic, the low-shot and the supervised ones. Surprisingly, the low-shot tickets perform better than the label-agnostic ones. More importantly, we observe that winning tickets generated in the supervisd setting perform better in general than both the lahol-agnostic and low-shot ones. This shows that properties of both appraoches capture different statistics that add up to generate better winning tickets.
4.4 COMPARAISON WITH WINNING TICKETS TRANSFERRED FROM OTHER DATASETS
In this experiment, we compare the winning tickets transferred from other datasets to those obtained without labels or limited number of data.
Experimental setting. We follow the setting of Morcos et al. (019) and use Places365 Zhouu al. 2017) and CIFAR-10 which count respectively 1.8M and 50k training data samples and 365 and 10 categories. We consider these two datasets because they have different statistics, in particular Places365 is very large while CIFAR-10 of comparable size with the subsets used for our low-shot experiments. For both datasets, we generate winning tickets with supervision.
Results. In Figure 5-b, we compare these lottery tickets with those transferred from other data distributions even if trained with more data. Interestingly, winning tickets generated with only 10% of the images perform better than the tickets generated on Places365 with 14x more data and 3x more classes. This suggests that lottery tickets are more sensitive to the data distribution than the number of samples or the use of labels to generate them.

---

Under review as a conference paper at ICLR 2020
4.5
CAVEAT ABOUT PRUNING DEEP NETWORKS ON CIFAR-10
10-
weight magnitude
Figure 6: Magnitude of the weights of trained network on two different datasets: CIFAR-10 (green) and ImageNet (red). We perform thresh- olding at machine precision value (bottom of y- axis). On CIFAR-10, a trained VGG-19 84.5% sparse while trained ResNet-1 18 is 80.3% sparse.
10"3
CIFAR-I0: VGG-19
10
CIFAR-10: ResNet-18
ImageNet: VGG-19
ImageNet: ResNet-18
10-
0%
25%
50% sorted weights
75%
100%
Somewhat surprisingly, we find that prior any pruning, a large proportion of the weights of a deep architecture trained on CIFAR-IO has converged naturally to zero during training. For example, we observe in Figure 6 that 85% of the weights of VGG-19 and ~ 80% of that of a ResNet-18 at convergence are zeroed (Appendix [Disowss results on more architectures). As a result, it is trivial to prune these networks without any loss in accuracy. Unstructured magnitude-basee pruning acts here as training since are freezed to zero weights that were going to zero anyway (Zhou et al.) 2019). Overall, while pruning their natural level of sparsity (~ 80%) is still meaningful, analyzing pruning below this rate may not be conclusive. We find that in the random global pruning baseline (which can remove non zero weights) in Figure Ta pruning at rates below the natural sparsity of the network degrades accuracy, while pruning of weights that are already zeroed has no effect. Inconveniently, this performance gap carries over to higher pruning rates (in which we are interested in) and can lead to misleading interpretations. For fair comparison, we adjust the random mask baseline Figure [aa we remove this effect by first pruning the weights that naturally converge to zero after training. Then, we randomly mask the remaining non-zeroed weights to get different final desired pruning rates. The remaining non-masked weights are randomly initialized. This baseline therefore corrects for the natural sparsity present in CIFAR-10 10 networks.
CIFAR-10 large networks originally tuned for ImageNet rates above
5 DISCUSSION
Overall, our study empirically shows that we can find winning tickets with good accuracy on label classification with no supervision all. This observation suggests that winning tickets are, some extent, surprisingly labels and task-independent. We also find that they are not very data-demanding since dataset subset is sufficient to generate them. Finding winning tickets with subset of Ima- geNet is 5x faster than using the entire dataset, without modifying the original algorithm. Such a speedup is especially crucial for extreme pruning rates that are computationally demanding.
Nonetheless, we find several critical limitations to winning tickets in our study. First, none of the tickets found with limited access labels and or data matches the accuracy of tickets found with all the labeled data when considering moderate pruning rates (more than ~ 10% of unpruned weights) on ImageNet. Indeed, we consistently observe decrease in performance compared to the full over- parametrized network as soon we prune the network. Therefore, for these rates, winning tickets are certainly label and data dependant. Second, we find that pruning large modem architectures on CIFAR-10 10 should be done with caution as these networks tend to be sparse at convergence, making unstructured pruning at rates below 80% particularly simple. Third, we observe that winning tickets particularly sensitive to the late resetting parameter (see Appendix B for discussion about our choice of rewind parameter). The definition of "early in training" is somehow ill-defined: network weights change much more for the first epochs than for the last ones. Thus, by resetting weights early in their optimization, they still contain a vast amount of information.
Finally, we have observed winning tickets through the scope of label classification for images with natural-looking statistics only. Considering more self-supervised tasks and even different types of tasks (generative modeling for example) might complement furthermore this study.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Piotr Bojanowski and Armand Joulin. Unsupervised learning by predicting noise. In Proceedings of the International Conference on Machine Learning (ICML). 2017.
Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsu- pervised learning of visual features. In Proceedings of the European Conference on Computer Vision (ECCV), 2018.
Wenlin Chen, James Wilson, Stephen Tyree, Kilian Weinberger, and Yixin Chen. Compressing neu- ral networks with the hashing trick. In Proceedings of the International Conference Machine Learning (ICML), 2015.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR), 2009.
Carl Doersch and Andrew Zisserman. Multi-task self-supervvsed visual learning. In Proceedings of the International Conference Computer Vision (ICCV), 2017.
Carl Doersch, Abhinav Gupta, and Alexei A Efros. Unsupervised visual representation learning by context prediction. In Proceedings of the International Conference on Computer Vision (ICCV), 2015.
Xin Dong, Shangyu Chen, and Sinno Pan. Learning to prune deep neural networks via layer-wise optimal brain surgeon. In Advances in Neural Information Processing Systems (NIPS), 2017.
Alexey Dosovitskiy, Philipp Fischer, Jost Tobias Springenberg, Martin Riedmiller, and Thomas Brox. Diseriminative unsupervised feature learning with exemplar convolutional neural networks. IEEE Transactions Pullern Analysis and Machine Intelligence, 3809) 2016.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representations (ICLR), 2019.
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M Roy, and Michael Carbin. The lottery ticket hypothesis at scale. arXiv preprint arXiv:1903.01611 2019.
Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. In International Conference on Learning Representations (ICLR), 2018.
Yiwen Guo, Anbang Yao, and Yurong Chen. Dynamic network surgery for efficient dnns. In Advances in Neural Information Processing Systems (NIPS), 2016.
Song Han, Jeff Pool, John Tran, and William Dally, Learning both weights and connections for efficient neural network. In Advances in Neural Information Processing Systems (NIPS), 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
Olivier J Hénaff, Ali Razavi, Carl Doersch, SM Eslami, and Aaron van den Oord. Data-efficient image recognition with contrastive predictive coding. arXiv preprint arXiv: 2019.
Longlong Jing and Yingli Tian. Self-supervised visual feature learning with deep neural networks: A survey, arXiv preprint N.Kiv:9902.000 2019.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convo- lutional neural networks. In Advances in Neural Information Processing Systems (NIPS), 2012.
Yann LeCun, John S Denker, and Sara A Solla. Optimal brain damage. In Advances in Neural Information Processing Systems (NIPS), 1990.

---

Under review as a conference paper at ICLR 2020
Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. In International Conference on Learning Representations (ICLR), 2017.
Zhuang Liu, Mingiee Sun, Tinghui Zhou, Gao Huang, and Trevor Darrell. Rethinking the value of network pruning. In International Conference on Learning Representations (ICLR), 2019.
Christos Louizos, Max Welling, and Diederik P Kingma. Learning sparse neural networks through 1.0 regularization. In International Conference on Learning Representations (ICLR), 2018.
Ari S Morcos, Haonan Yu, Michela Paganini, and Yuandong Tian. One ticket to win them all: generalizing lottery ticket initializations across datasets and optimizers. arXiv preprint arXiv: 02773, 2019.
Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. In Proceedings of the European Conference on Computer Vision (ECCV), 2016.
Deepak Pathak, Ross Girshick, Piotr Dollár, Trevor Darrell, and Bharath Hariharan. Learning fea- tures by watching objects move. In Proceedings of the Conference on Computer Vision and Puttern Recognition (CVPR), 2017.
Aaditya Prakash, James Storer, Dinei Florencio, and Cha Zhang. Repr: Improved training of con- volutional filters. In Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. Xnor-net: Imagenet classification using binary convolutional neural networks. In Proceedings of the European Con- ference on Computer Vision (ECCV), 2016.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint
2014.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. arXiv preprint arXiv: 2019.
Karen Ullrich. Edward Meeds, and Max Welling, Soft weight-sharing for neural network compres- sion. In International Conference on Learning Representations (ICLR), 2017.
Xiaolong Wang and Abhinav Gupta. Unsupervised learning of visual representations using videos. In Proceedings of the International Conference on Computer Vision (ICCV), 2015.
Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non- parametric instance discrimination. In Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
Haonan Yu, Sergey Edunov, Yuandong Tian, and Ari S Morcos. Playing the lottery with rewards and multiple languages: lottery tickets in rl and nlp. arXiv preprint 1xXiv:1906.0 02768, 2019.
Xiaohua Zhai, Avital Oliver, Alexander Kolesnikov, and Lucas Beyer. S41: Self-supervised semi- supervised learning. Proceedings of the International Conference on Computer Vision (ICCV), 2019.
Richard Zhang, Phillip Isola, and Alexei A Efros. Colorful image colorization. In Proceedings of the European Conference on Computer Vision (ECCV), 2016.
Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 mil- lion image database for scene recognition. IEEE Transactions on Pattern Analysis and Machine
Intelligence, 40(6): 145--1444 2017.
Hattie Zhou, Janice Lan, Rosanne Liu, and Jason Yosinski. Deconstructing lottery tickets: Ze- ros, signs, and the supermask. In "Workshop on Identifying and Understanding Deep Learning Phenomena (ICML)", 2019.
10

---

Under review as a conference paper at ICLR 2020
6 APPENDICES
HYPERPARAMETER, AND MODEL DETAILS
We detail in this appendix the different hyperparameters used in our experiments. For all trainings on ImageNet, we use stochastic gradient descent with a momentum of 0.9. We use Adam optimizer on CIFAR-10. On ImageNet, unless specified otherwise, we perform standard data augmentation consisting in croppings of random sizes and aspect ratios and horizontal flips Krizhevsky et al.) (2012). On CIFAR-10, we use horizontal flips and croppings of fixed size on a 2-padded input image. We use Py Porcch version 1.0 for all our experiments. On ImageNet, we use the "Dali" library for fast data augmentation implementation
ImageNet Labels full dataset- AlexNet: we train for 90 epochs with a total batch-size of 4096 distributed over 8 GPUs (512 samples per GPU), learning rate of 0.4, weight decay 0.0001. We decay the learning rate by a factor 10 at epochs 30 and 60.
ImageNet Labels- full dataset ResNet-55:: we train for 90 epochs with total batch-size of 1536 distributed over 16 GPUs (96 samples per GPU), learning rate of 0.1, weight decay 0.0001. We decay the learning rate by a factor 10 at epochs 50, 65 and 80.
ImageNet Labels 30% dataset: we train for 200 epochs with total batch-size of 1536 distributed over 16 GPUs (96 samples per GPU), learning rate of 0.1 warmed up during the first epochs, weight decay of 0.0003. We decay the learning rate by factor 10 epochs 140, 160 and 180.
ImageNet Labels 30% of classes: we train for 200 epochs with a total batch-size of 1536 distributed over 16 GPUs (96 samples per GPU), learning rate of 0.03 warmed up during the first 5 epochs, weight decay of 0.001. We decay the learning rate by a factor 10 at epochs 140, 160 and 180.
ImageNet Labels 10% dataset: we train for 200 epochs with total batch-size of 768 distributed over 8 GPUs (96 samples per GPU), learning rate of 0.1 warmed up during the first 5 epochs, weight decay of 0.001. We decay the learning rate by factor 10 epochs 140, 160 and 180.
ImageNet Labels- 10% of classes: we train for 200 epochs with a total batch-size of 768 distributed over 8 GPUs (96 samples per GPU), learning rate of 0.3 warmed up during the first 5 epochs, weight decay of 0.0003. We decay the learning rate by a factor 10 at epochs 140, 160 and 180.
Image.Net Labels 1% dataset: we train for 1000 epochs with total batch-size of 768 distributed over GPUs (96 samples per GPU), learning rate of 0.1, weight decay of 0.001. We decay the learning rate by factor 10 at epochs 700, 800 and 900. We follow Zhai et al. (019) and use random color data augmentation during training.
ImageNet Labels 1% of classes: we train for 1000 epochs with a total batch-size of 768 distributed over 8 GPUs (96 samples per GPU), learning rate of 0.01, weight decay of 0.003. We decay the learning rate by factor 10 at epochs 700, 800 and 900. We follow Zhai et al. (2019) and use random color data augmentation during training.
We reproduce the semi-supervised method of Zhai et al. (2019) and follow precisely their hyperparameter. We train for 200 epochs with total batch-size of 2048 distributed over 32 GPUs (64 samples per GPU), learning rate of 0.1 warmed up during the first 5 epochs, weight decay of 0.0003. We decay the learning rate by factor 10 at epochs 140, 160 and 180.
ImageNet Semi-supervised RotNet:
ImageNet RoiNet - ResNet-50: we train for 90 epochs with a total batch-size of 1536 distributed over 16 GPUs (96 samples per GPU), learning rate of 1 warmed up during the first 5 epochs, weight decay of 0.00001. We decay the learning rate by a factor 10 at epochs 50, 65 and 80
train for 30 epochs with a total batch-size of 1536 distributed over 32 GPUs (48 samples per GPU), learning rate of 1, weight decay of 0.00001. We decay the learning rate by factor 10 epochs 20 and 25. We follow the
ImageNet Exemplar ResNet-50:
gf wtt.tb.coooooooooo /NUTTTA/DDAAATTT
11

---

Under review as a conference paper at ICLR 2020
Exemplar implementation based on triplet margin loss from Doersch & Zisserman (017 As the goal of the task is to learn invariance to data transformation, we use data color augmentation and random small rotations on top of standard data augmentation scheme.
ImageNet RotNet AlexNet: we train for 90 epochs with a total batch-size of 8192 dis- tributed over 16 GPUs (512 samples per GPU), learning rate of 0.5 warmed up during the first 2 epochs, weight decay of 0.00001. We decay the learning rate by a factor 10 at epochs 30 and 60.
ImageNet Exemplar- AlexNet: we train for 30 epochs with total batch-size of 4096 distributed over 16 GPUs (256 samples per GPU), learning rate of 0.1, weight decay of 0.0001. We decay the learning rate by factor 10 at epoch 20. We follow the Exemplar implementation based on triplet margin loss from Doersch & Zisserman (2017). We use data color augmentation and random small rotations on top of standard data augmentation scheme.
Places Labels ResNet-50: we train for 90 epochs with a total batch-size of 2048 dis- tributed over 16 GPUs (128 samples per GPU), learning rate of 0.1,, weight decay of 0.0001. We decay the learning rate by a factor 10 at epochs 50, 65 and 80.
CIFAR-10 Labels & RotNet VGG-19 & ResNet-18: we train for 160 epochs with a total batch-size of 512 on 1 GPU, learning rate of 0.001, weight decay of 0.0001. We decay the learning rate by a factor 10 at epochs 80 and 120.
CIFAR-10 Exemplar- VGG-19 ResNet-18: we train for 180 epochs with total batch- size of 512 on 1 GPU, learning rate of 0.0003, weight decay of 0.0001. We decay the learning rate by a factor 10 at epochs 120. As the goal of the task is to learn invariance to data transformation, we use data color augmentation and random small rotations on top of standard data augmentation scheme.
CIFAR-10 Randomly permuted labels VGG-19 & ResNet-18: we train for 160 epochs with total batch-size of 512 on GPU, learning rate of 0.001, weight decay of 0.0001. We decay the learning rate by a factor 10 at epochs 80 and 120. We do not apply data augmentation.
B LATE RESETTING PARAMETER
We follow Frankle et al. 2019) and use late resetting (or rewind) for the winning tickets generation process. Indeed, before re- re-training winning ticket, reset its weights their value "carly in training" of the full over-parameterized network. In our work, we set the late resetting parameter to 1 epoch on CIFAR-10. However, when dataset size, total number of epochs, mini-batch sizes or learning rate vary, it becomes more complicated to choose a rewind criterion that guarantees a fair comparison between all settings. A choice can be rewind point where "the same amount of information" " has been processed. Thus, in our work, we choose to set the rewind parameter to 3 x 280, 000 samples for all our experiments on ImageNet, which corresponds to epochs on full ImageNet. We describe in Table what this rewind parameter corresponds to in terms of number of epoch, number of data samples seen, number of gradient update and percentage of total training for our different experiments. Moreover, we show in Figure 7 the performance of winning tickets generated using 10% of ImageNet with different values of rewind. Each of the considered value corresponds to keeping one of the criteria (number of epoch, number of data samples seen, number of gradient update or percentage of total training) fixed compared to the rewind parameter on full ImageNet (first row of Table I
# epochs
samples seen
# gradient updates % of training
ImageNet 100%
3
3 1, 280, 000
2500
3.3%
ImageNet 30%
10
~ 3 x 1, 280, 000
2500
5.0%
ImageNet 10%
30
~ 3 x 1, 280, 000
5000
15%
ImageNet 1 1%
300
~ 3 x 1, 280, 000
5000
30%
Table
12

---

Under review as a conference paper at ICLR 2020
ImageNet val acc @)
0.7
IniNe 100%
0.5
epochs
Im rrewind: 3.3% of full training
IniN:105 rewwndd: 2500 gradient steps samples socn
NeNe
0.3
Random mask
100
6.2 of remaining weights
12.5
3.1
Figure 7: ImageNet top validation accuracy winning tickets generated with a subset of 10% of
ImageNet dataset. We show the influence of different values for the late resetting parameter.
AlexNet: conv
NletNet: convs 1-3
AlexNet: convs 1-5
AlexNet: all
Labels
IngNet val
0.5
0.5
0.5
0.5
Random
0.44
0.4
0.4
0.4
100 25 6.2 3.1
25
31
25
31
25 6.2 31
remaining
weights
remmining
remaining
0.8
ResNet-50: block
ResNet-50: blocks
1-2
0.
RcsNect50: blocks 1-3
ResNct-50: all
Inter acc
0.7
0.7
0.7
0.7
0.6
0.6 12.5 remaining weights
0.6
0.6
6.2 romaining weights
25
12.5 6.2
12.5 remmining weights
remmining weights
Figure 8: ImageNet top validation accuracy of winning tickets generated by pruning partly or entirely (all) a network with 2 generation tasks: labels classification or RotNet. AlexNet is used in top row and ResNet-50 the bottom row. We also show for reference results when the network layers are randomly pruned.
C LAYERWISE WINNING TICKETS
In this appendix we show more results the layerwise winning tickets experiment (see 4.1p. We generate winning tickets by pruning only the n first convolutional layers network, and leaving the remaining of the network unpruned. On AlexNet (first row of Figure 8) we consider 4 situations. From left to right in the first row of Figure 8). we prune the first convolutiona layer; up to the third convolutiona layer, up the fifth convolutional layer; the whole network. For ResNet-50, from left to right in the bottom row of Figure 8) we prume the first convolutiona layers; the first 24 convolutiona layers; the first 43 convolutional layers; the whole network. Consistently with observations from the self-supervised literature, self-supervised winning tickets are competitive with labels ones when only the shallow and mid-level layers are pruned. Indeed, these layers are not specialized yet for the labels classification tasks and many works have shown that representations learned are similar these depths.
D SPARSE TRAINED NETWORKS ON CIFAR-10
provide results more architectures about the proportion of weights zeroed during training on CIFAR-10 compared to ImageNet. Note that we do not consider the parameters of the last fully-comnected layer (since do
In this appendix, in Figure
the batch-norm layers parameters, not prune it in our setting). For all the considered VGGs we use the modified version of Morcos al. 2019), replacing the final MIP by fully connected layer.
13

---

Under review as a conference paper at ICLR 2020
ResNet-18
ResNet-50
10
CIFAR-10
10
CIFAR-10
weight magnitude
ImageNet
weight magnitude
ImageNet
10
10 -3
10
10
10
10
0%
25%
50%
75%
100%
0%
25%
50%
75%
100%
sorted weights
sorted weights
VGG-11
VGG-13
10
CIFAR-10
10
CIFAR-10
weight magniude
ImugeNet
weight magnituee
ImageNet
10
10-3
10
10
10-
10
0%
25%
50%
75%
100%
0%
25%
50%
75%
100%
sorted weights
sorted weights
VGG-16
VGG-19
10
CIFAR-10
10
CIFAR-10
weight magnitude
ImageNet
weight magnitude
ImageNet
10
10-3
10"
10 -
10 -7
10-7
0%
25%
50%
75%
100%
0%
25%
50%
75%
100%
sorted weights
sorted weights
Figure Magnitude of the weights of a trained network on two different datasets: CIFAR-10 (green) and ImageNet (red) with different datasets. We perform thresholding at machine precision value (bottom of y-axis).
E EXACT NUMBERS FROM FIGURES
In Tables 6 report accuracies for each setting presented in the main paper. We report mean and standard errors for our experiments which we run with 3 different seeds for ImageNet and Places-365 and different seeds for CIFAR-10.
F VARYING DATASET SIZE ON CIFAR-10 10
We choose to present results with ImageNet only in the main paper because varying the dataset size of an already small dataset such as CIFAR- 10 is less conclusive. Moreover, noted before 4.5) studying sparsity on CIFAR-10 10 should be done precautiously since trained deep networks are already sparse. In this Appendix, present results CIFAR-10 in Table I0 With VGG-19 architecture, we observe that decreasing the dataset size still leads to reasonable winning tickets. However, with ResNet- 18 observe important performance gap of winning tickets generated with fraction of
14

---

Under review as a conference paper at ICLR 2020
Remaining weights
Labels
RotNet
Exemplar
Random
Random adjusted
86.54 = 0.08
0.8
86.45 + 0.09
86.66 0.11
86.34 0.14
86.05 + 0.10
0.64
86.45 + 0.06
86.72 0.07
86.36 + 0.06
85.63 0.05
0.512
86.39 = 0.11
86.61 0.06
86.47 0.10
85.22 0.11
0,410
86.55 + 0.06
86.64 0.13
86.64 0.06
84.56 0.05
0.328
86.45 = 0.07
86.55
86.38 = 0.07
83.73
0.262
86.48 = 0.13
86.48
86.41 = 0.100
83.05 = 0.09
134
86.47 + 0.05
86.53 0.06
85.34 + 0.05
80.97 0.13
85.65 + 0.12
0.069
86.96 + 0.06
86.44
84.86 + 0.03
78.68 + 0.10
83.73 + 0.08
0.035
86.91 + 0.08
+
84.40 + 0.08
76.03 +.19
81.97 0.14
0.018
36.80
85.54
84.07 = 0.14
72.11
19.66
0.009
86.41 0.06
$4.59 0.09
8
66.90=0.00
77.04 0.27
0.005
85.76 = 0.13
0.14
79.69 = 0.11
59.71 #
72.34 + 0.28
0.002
84.50 = 0.05
0.099
75.59 = 0.16
52.08 0.35
66.67 = 0.38
0.001
81.71 + 0.12
75.41
70.60 + 0.24
34.29 2.59
60.31 + 0.29
Table 2: Mcans and standard errors for accuracies from Figure lb (Label-agnostic winning tickets: CIFAR-10 ResNet-18).
Remaining weights
Labels
RotNet
Exemplar
Random
Random adjusted
92.62 = 0.05
0.8
92.45 = 0.05
92.48 + 0.08
92.28 + 0.06
92.26 = 0.06
0.64
92.58 + 0.08
92.31 + 0.07
92.29 0.06
91.93 0.05
0.512
92.46 + 0.05
92.38 + 0.04
92.17 0.07
91.82 + 0.05
0.410
92.44 = 0.07
92.30
92.33 = 0.09
91.58 = 0.04
0.328
92.51 + 0.06
0.007
92.39 + 0.05
91.24 = 0.08
0.262
92.56 = 0.04
92.76
0.05
90.76 0.09
0.134
92.61 = 0.06
92.67
92.57 0.06
89.61 0.05
92.41 #
0.069
92.68 = 0.08
92.68 0.07
92.31 0.07
$7.86=0.11
91.76 + 0.06
0.035
92.64 0.10
9..17
91.83 + 0.05
85.62 1.62+00111
90.54 0.09
0.018
92.40 + 0.06
+
90.90 + 0.04
$
88.67 0.04
0.009
92.30 0.07
89.53 9.04
78.26
86.36 0.11
0.005
91.89 0.08
89.22
0.00
70.68 - 0.47
83.62 - 0.12
0.002
90.76 + 0.08
8.12
85.69 0.08
59.21 1.19
79.30 + 0.30
0.001
76.81 + 4.61
81.63 + 0.58
77.04 = 0.30
27.18 - 7.04
71.29 + 0.44
Table 3: Means and standard errors for accuracies from Figure (Label-agnostic winning lickets: CIFAR-10 VGG-19).
the dataset compared to using the entire dataset. Indeed, when only only 10% of CIFAR-10 (5000 images) the generated winning tickets are at random level.
G RANDOM RE-INITIALIZATION OF WINNING TICKETS
In this Appendix, we present results with random re-initialization for winning tickets found with labels or with RotNet self-supervised task on both ImageNet and CIFAR-10 in Figure II On Im- ageNet [lb] consistently with the experiments of Frankle et al. (2019). we observe that resetting the weights accordingly is crucial to get high accuracy. Indeed, on both ResNet-50 and AlexNet, for labels, the subnetworks that are reset to their weights early in training (dark blue plain line) perform significantly better than subnetworks randomly re-initialized (dark blue dashed line). Inter- estingly, this is not the case for RotNet winning tickets: starting from original weights (pink plain line) gives only a very slight boost (or even no boost at all) of performance compared to randomly re- re-initialization (pink dashed line). Overall, labels RotNet subnetworks perform in the same ball park when randomly re- in-iiiialized, but using the original weights gives large boost of performance for labels but not for RotNet. Thus, it suggests that the information carried by the pruned mask itself
15

---

Under review as a conference paper at ICLR 2020
Remaining weights
Labels
RotNet
Exemplar
Random
75.26 + 0,04
0.8
75.57 + 0.05
74.92 + 0.02
73.53 0.07
+
0.64
75.58 + 0.05
74.33 + 0.06
1 0.04
73.52 0.44
0.512
75.86 = 0.01
7.04
72.71 + 0.54
0,410
76.06 + 0.02
7
71.68 + 0.59
0.328
75.79 = 0.01
72.87
70.46 7.02
0.262
75.96 0.01
72.08 0.09
70.68 0.00
1.02
0.134
75.24 0.07
69.88 + 0.05
68.15
64.56
0.069
69.38 + 0.10
66.94 + 0.03
+0.30+0000
+
0.035
65.16+0.14
63.77
61.96 0.10
51.09 4.441
0.018
0,24
59.68
57.61 0.199
5.70
0.009
57.19 0.27
+.22
51.75 =
33.10
0.005
52.14
47.89
45.44 = 0.05
23.17
0.002
43.77 + 0.15
39.07 0.37
38.06 = 0.13
9.75 = 3.52
0.001
32.17 + 0.48
28.71 + 0.38
29.70 + 0.11
0.62 + 0.42
Table 4: Mcans and standard errors for accuracies from Figure lb (Label-agnostic winning tickets: ImageNet ResNet-50).
Remaining weights
Labels
RotNet
Exemplar
Random
1
58.01 + 0.07
0.8
58.54 = 0.25
57.80 = 0.02
56.21 0.10
57.61 0.06
0.64
58.62 + 0.17
0..04
55.51 = 0.08
56.86 + 0.11
0.512
59.06 = 0.21
5.04
54.48 0.03
55.82 0.14
0.410
58.83 + 0.21
5..37+
53.51 0.11
54.83 + 0.02
0.328
59.39 + 0.13
54.41
+2.36+0.12
53.20 + 0.08
0.262
58.93 + 0.20
0.02
50.99 0.08
51.75
0.134
58.87 0.09
19.58
16.63+0.00
46.36 1 0.07
0.069
57.46 + 0.08
45.62
41.53
40.18 0.06
0,035
54.36 0.22
+
36.61#0.11
33.47 + 0.05
0.018
50.43 0.19
3.15
31.23 +.23+0016
26.45 + 0.09 0.09
0.009
45.22 + 0.07
31.46
27.03 + 0.06
20.91 + 0.34
0.005
37.42 0.14
26.20 0.11
23.38 + 0.14
16.28 0.21
0.002
28.66 + 0.31
20.66 0.188
19.34 0.20
11.29 + 0.22
0.001
21.18 + 0.24
15.46
15.53 - 0.03
1.73 + 1.33
Table 5: Means and standard errors for accuracies from Figure 1b (Label-agnostic winning tickets: ImageNet AlexNet).
similar for labels and RotNet subnetworks but the weights the RotNet winning tickets are not as good starting points as the weights from labels winning tickets.
up to a certain level of sparsity (that roughly corresponds to the natural level of sparsity of the network), using random re-initialization or weights "early in training" gives similar performance. However, for more extreme pruning rates, using a particular set of weights gives significantly better performance than random re re-initilliaation. Our results consistent with the findings of Liu et al. (2019) (Figure 7.a of their paper), Franklle et al. (2019) (Appendix A of their paper) Morcos al. 2019) (Appendix A2 of their paper).
CIFAR- 10 lIa
16

---

Under review as a conference paper at ICLR 2020
Remaining weights
100% ImNet
30% ImNet
10% ImNet
1% ImNet
Random
1
75.26 + 0.04
75.57 = 0.05
74.58 + 0.07
74.86 0.03
73.86 + 0.07
74.22 = 0.27
0.64
75.58 + 0.05
1.04
74.52 0.04
73.28 0.04
73.52 = 0.44
0.512
75.86 0.01
74.38
7
0.06
72.71 = 0.54
0.410
76.06 0.02
7.02
7.09
71.68 = 0.59
0.328
75.79 0.01
+
73.37 + 0.04
7.02
70.46 = 0.87
0.262
75.96 + 0.01
+.11
73.01 0.09
70.47
69.34 = 1.02
0. 131
75.24 +: 0.07
70.21
71.15 + 0.06
6.070
64.56 = 1.82
0.069
69.38 0.10
6.010
69.48 0.04
6.24
56.71 = 3.73
0,035
#
0.00
67.28 - 0.04
60.61 0.37
51.09 = 4.41
0.018
61.45
60.77
#.00
55.48
42.40 5.70
0,009
#7.99
56.76
4.74
33.10 = 6.45
0.005
52.14 + 0.21
0.10
55.56 0.30
25.61 + 10.52
23.17 = 5.86
0.002
43.77 + 0.15
4.446
46.89 0.27
18.53 + 5.37
9.75 + 3.52
0.001
32.17 = 0.48
32.76 0.35
36.62 = 0.10
12.06 3.27
0.62 + 0.42
Table 6: Means and standard errors for accuracies from Figure 3a (Varying dataset size).
Remaining weights
Full ImNet
30% ImNet classes 10% ImNet classes 1% ImNet classes
Random
75.26 = 0.04
0.8
75.57 + 0.05
74.53 0.03
74.63 + 0.05
74.44 0.01
74.22 + 0.27 0.27
0.64
75.58 + 0.05
74.20 + 0.02
74.08 + 0.10
73.79
+
0.512
75.86 + 0.01
73.70 0.05
73.75 + 0.05
+
+
0.410
76.06 1.066:00 0.02
72.95 0.04
73.30
0.10
71.68
0.328
75.79 +
72.07 + 0.06
72.86 0.02
4.11
70.46
0,262
7
71.22 + 0.05
12.2300.0
69.34 0.007
0.134
0.077
68.27 + 0.08
70.28 + 0.09
64.56 0.11
0.069
0.100
65.10 0.08
67.99 + 0.10
0.004
56.71
0.035
0.14
+
0.11
51.09
0.018
61.45
58.39
61.78
0..6
4,.40
0.009
57.19
54.45
0.06
33.10 49.18
0.005
0.221
19.38
51.66 0.26
23.17 5.86 42.65
0.002
43.77 0.15
42.57 + 0.09
45.12 + 0.30
3.19
9.75 3.52
0.001
92.17 0.48
30.10 0.32
37.36 + 0.03
+7.54+0.08
0.62 + 0.42
Table 7: Means and standard errors for accuracies from Figure 3b
(Varying number of classes).
Remaining weights
Semi-Sup
Places-365
CIFAR-10
0.8
74.89 0.06
74.27 + 0.07
72.57 + 0.22
0,64
74.76 - 0.04
73.90 0.02
72.13 = 0.16
0.512
74.60 0.07
71.58 0.26
0,410
74.48 - 0.08
7.01
70.78 + 0.28
0,328
74.22 0.03
72.61
69.77 0.21
0.262
74.03 + 0.02
71.91 0.08
68.76 + 0.27
0.134
72.59 = 0.08
0.11
65.65 = 0.29
0.069
70.09 = 0.05
65.93
61.61 = 0.35
0.035
67.15 0.08
0.10
56.03 + 0.46
0.018
62.90 0.15
4.124
47.53 + 0.47
0.009
58.40 + 0.11
$2.95
36.95 = 0.71
Table 8: Means and standard errors for accuracies from Figure ((a) Semi-supervised & (b) Transfer betveen datasets).
17

---

Under review as a conference paper at ICLR 2020
0.9
0.9
CIFAR- 1--10 test acc
CIFAR- 1-000 tostt acc
0.8
0.8
100% CIFAR-I0
100% CIFAR-10
0.7
30% CIFAR-10
30% CIFAR-10
0.6
10% CIFAR-10
0.6
10% CIFAR-10
Random mask
Random mask
0.5
Random adjusted
0.5
Random adjusted
100 50
25 12.56.2 3.1 1.6 0.8 0.4 0.2 of remaining weights
100 50 25 12.56.2 3.1 1.6 0.8 0.4 0.2 of remaining weights
VGG-19
(b) ResNet-18
Top-1 accuracy CIFAR-10 test set of winning tickets generated with different training data. The x-axis represents different levels winning ticket sparsity.
Figure 10:
ResNet-18
VGG-19
0.9
0.9
CIFAR 10 acc
0.8
0.8
Labels
0.7
RotNet
0.7
F
Labels Random Reinit
0.6
RotNet Random Reinit
0.6
Random
0.5
Random ajusted
0.5
100
50
25 12.56.2 3.1 1.6 0.8 0.4 0.2 % of remaining weights
100
50
25 12.56.2 3.1 1.6 0.8 0.4 0.2 % of remaining weights
(a) CIFAR-10
ResNet-50
AlexNet
0.7
0.7
ImageNet val @
0.5
0.5
0.3
0.3
0.1
0.1
100 50
25
12.56.2 3.1 1.6 0.8 0.4 0.2
100 50
25 12.56.2 3.1 1.6 0.8 0.4 0.2 % of remaining weights
% of remaining weights
(b) ImageNet
We report CIFAR-10 test (b) and ImageNet val (a) top-1 accuracy for winning tickets found with RotNet label-agnostic task. The x-axis corresponds to different pruning ratios. We com- pare the performance with supervised winning tickets and random subnetworks (randomly drawn weights from the initialization distribution and randomly permuted masks). On CIFAR-10. deep models are highly sparse with only ~ 15% of non-zero weights. Thus, we adjust the random base- line start with the correct mask the natural level of network sparsity (4.5 details). The dashed lines correspondong to applying the pruning masks found with labels or RotNet, but with randomly re-initialized weights.
Figure 11:
18


Reviews:
[{"review_id": "ryxRdy869S", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper empirically studies the lottery ticket hypothesis with limited or no supervision. First, the authors use self-supervised learning to generate winning tickets, showing that \"good\" (reasonable) winning tickets can be found without labels. Second, the authors show that finding \"good\" (reasonable) winning tickets can be accelerated by a factor 5 on ImageNet by using only a subset of the data. The authors also argue that using large datasets is important to study lottery tickets, since deep networks trained on CIFAR-10 are natually sparse, making conclusions potentially misleading.\n\nThe experimental results are rich and provide more understanding of winning ticket generation with limited or no supervision. The results on self-supervised learning task (including the layer-wise pruning results) and a subset of training dataset are reasonable and kind of expected, but it is still good that this paper provide solid experimental results to verify this. As the paper observed, \"none of the tickets found with limited access to labels and or data matches the accuracy of tickets found with all the labeled data when considering moderate pruning rates (more than 10% of unpruned weights)\non ImageNet. Indeed, we consistently observe a decrease in performance compared to the full overparametrized network as soon as we prune the network.\" In this sense, winning tickets are certainly label and data dependant. This undermines the *bold* claim in the abstract that \"we provide a positive answer to both questions, by generating winning tickets with limited access to data, or with self-supervision\". From my perspective, the ability to exactly perserve the accuracy while pruning the weights (see the flat regions of \"Lables\" curves in Figure 1,2,3,4,5) is the interesting part of the lottery ticket hypothesis. We have several different ways to achieve a descreased accuracy with a smaller network, the dynamics there may be a mixture of the lottery ticket hypothesis and standard model pruning, which needs more careful experiment design to separate different dynamics.\n\n\"using large datasets is important to study lottery tickets, since deep networks trained on CIFAR-10 are natually sparse, making conclusions potentially misleading.\" \"The definition of “early in training” is somehow ill-defined: network\nweights change much more for the first epochs than for the last ones.\" These two messages are important to future study of the lottery ticket hypothesis. This paper raises the issue of ill-definedness of “early in training”, but did not provide a solution. \n\nOverall, I found that the experimental results in this paper are solid and provide more understandings of the lottery ticket hypothesis. However, I feel that the novelty of this paper is limited, and do not provide much new insights. Therefore, it does not reach the bar of being published at ICLR, from my perspective. Therefore, I say \"Weak Reject\".", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "SkxRhOSTFB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies the problem of finding sparse networks in a limited supervision setup. The authors build on the lottery ticket work of Frankle & Carbin and investigate the validity of their idea when one has few or no labels. This work is an immediate followup on Morcos et al. who investigated the transferability of lottery tickets.\n\nThis work is more observational rather than algorithmic or theoretical. Authors study various small sample/label setups where network sparsification works well. \n\nMain contribution is Section 4.1 where self-supervision is investigated. However given that lottery tickets are transferable (Morcos paper) it is really not that surprising that semisupervised learning algorithms will do a decent job as well. I also don't see a practical benefit beyond transfer learning setup.\n\nSection 4.2 essentially sweeps through supervised problem parameters such as reducing sample size, adding noise etc and . The main application seems to be extracting lottery tickets faster by downsampling the data however this aspect is again fairly obvious. \n\nIn short, unfortunately, this paper doesn't cut it for ICLR. As improvements, I would recommend adding standard semi-supervised training techniques to their comparison. I was surprised to not see pseudo-labeling or consistency training (e.g. virtual adversarial training).", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "SJeuh5z19B", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In the original lottery ticket paper, it points out that training the pruned architecture from scratch with initial weight can achieve the same performance compared to fine-tuning it. This work further discuss this phenomenon when data or label is not enough. It is good to see the few-data/label can still provide a comparable results. But some experiment’s results and its setting are confusing, while also makes me concerned about the conclusion solidness.\n\n1) Usually in classification task (especially in cifar10 dataset), 0.5% to 1% accuracy could be a huge gap between two models. For example, in the original “Lottery Ticket Hypothesis” paper, using initial weight only has roughly 0.5% improvement compared to random initialization. But the figures in this paper do not contain a zoom-in details for each line, make me hard to distinguish the performance between each setting. If the author does not provide a detailed version, it will look like theses model have the same performance, which is actually wrong. The author should either plot a zoom-in figure especially when the pruning ratio larger than 50% or give a Table with accuracy of each setting. And it is better to complete the figure with several random seed and plot the error bar to avoid randomness.\n\n2) Does the “Random - adjusted” item in Figure. 1 mean the correctly pruning architecture with random initialization? In \"rethinking the value of network pruning\", Liu et al. points that in the large learning rate setting (lr=0.1, which is also your setting), random initialization can achieve the same performance compared to the lottery ticket. In my perspective, I want to see whether few-data/label also works on random initialization instead of lottery tickets. I expect the author to explain the “Random -adjusted” experiment setting clearly in the response and I suggest the author to discuss the\nrandom initialization part specifically.\n\n3) Figure.3 only shows the “varying dataset size” experiments on ImageNet. The experiments on cifar10 is lacked. The author should complete this part in the response.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "B1gdk_E5cH", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors try to provide empirical answers to several important open questions on winning tickets. They conduct most of experiments on ImageNet and results show that winning ticket is robust, and few data samples can also obtain good winning tickets.\n\nGenerally, the paper has conducted extensive experiments on three open questions and results prove their assumptions.\n\nAs describe in page 7, lottery tickets are sensitive to data distributions. I’m wondering, whether there will be winning ticket for multi-task learning with limited data each task? Will this be helpful in distilling the model?", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper studies finding winning tickets with limited supervision. The authors consider a variety of different settings. An interesting contribution is to show that findings on small datasets may be misleading. That said, all three reviewers agree that novelty is limited, and some found inconsistencies and passages that were hard to read: Based on this, it seems the paper doesn't quite meet the ICLR bar in its current form. 
Decision: Reject