Sparse Winning Tickets are Data-Efficient Image Recognizers
Authors: Mukund Varma T
Xuxi Chen
Zhenyu Zhang
Tianlong Chen
Subhashini Venugopalan
Zhangyang Wang
Venue: NeurIPS 2022 Accept (2022)

Abstract: Improving the performance of deep networks in data-limited regimes has warranted much attention. In this work, we empirically show that “winning tickets” (small sub-networks) obtained via magnitude pruning based on the lottery ticket hypothesis, apart from being sparse are also effective recognizers in data-limited regimes. Based on extensive experiments, we find that in low data regimes (datasets of 50-100 examples per class), sparse winning tickets substantially outperform the original dense networks. This approach, when combined with augmentations or fine-tuning from a self-supervised backbone network, shows further improvements in performance by as much as 16% (absolute) on low-sample datasets and long-tailed classification. Further, sparse winning tickets are more robust to synthetic noise and distribution shifts compared to their dense counterparts. Our analysis of winning tickets on small datasets indicates that, though sparse, the networks retain density in the initial layers and their representations are more generalizable. Code is available at https://github.com/VITA-Group/DataEfficientLTH.

Sparse Winning Tickets are Data-Efficient Image Recognizers
Mukund Varma T Xuxi Chen", Zhenyu Zhang', Tianlong Chen', Subhashini Venugopalan', Zhangyang Wang
Indian Institute Technology Madras, UUniversity of Texas Austin, 'Google
mukundvarmat@@mail. com, mutbhashini@googlee {xxchen, zhenyu. zeeccy,,,zeeyu...
zhang, tianlong- chen, Stttlassann/utteeeeet
Abstract
Improving the performance of deep networks in data-limited regimes has warranted much attention. In this work, empirically show that "winning tickets" (small sub- networks) obtained via magnitude pruning based on the lottery ticket hypothesis [. apart from being sparse are also effective recognizers in data-limited regimes. Based on extensive experiments, we find that in low data regimes (datasets of 50-100 examples per class), sparse winning tickets substantially outperform the original dense networks. This approach, when combined with augmentations or fine-tuning from self-supervised backbone network, shows further improvements in performance by as much as 16% (absolute) on low sample datasets and long- tailed classification. Further, sparse winning tickets are more robust to synthetic noise and distribution shifts compared to their dense counterparts. Our analysis of winning tickets on small datasets indicates that, though sparse, the networks retain density in the initial layers and their representations are more generalizable. Code is available at /tttpp////ggggg...o/////ooorrrrr
Introduction
Deep convolutional networks (2 have achieved wide success on a variety of tasks, but they do demand large amounts of data. However, there exist various domains and tasks where training data- labeled or unlabeled are limited. Often, in such data-limited regimes, transfer learning has emerged as dominant approach [3] as it achieves superior performance compared to training from scratch with random initialization [4] some domains, the highly specialized training performant models can still be a challenge, e.g. in scientific or medical images [5], or just images from different distribution differing vastly in size, color, or channels than seen in typical image datasets. such cases, training large-capacity data-hungry networks might be less performant than training models of similar architecture and lower capacity (6). While much of the literature on sparser and lower capacity networks has focused more efficient models and mobile deployment 77]. it is also known [8] that sparsity as a regularization could reduce overfitting. Thus it is interesting to study sparse networks in the context of low data or data efficient regimes (Fig. D
One of the popular ways of obtaining a sparse network with similar architecture but lower capacity is pruning [9] Frankle et. al. [1] formulated the lottery ticket hypothesis (LTH) and proposed to use iterative magnitude-ee
pruning (IMP). They demonstrated that dense networks can be magnitude pruned to obtain highly sparse sub-networks ("winning tickets") which can match or potentially outperform the original dense network. The caveat that the winning ticket needs to use the same random initialization as the original dense network. This approach also induces an inductive bias specific to the task to be learned, which leads to a better network when compared to dense network of similar size trained from scratch [10). Further, such pruning was recently found to reduce sample complexity theory and practice [II]. It has also been suggested that the number of samples required to achieve zero generalization error is proportional to the number of the non-pruned weights in the
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

hidden layer [.]. However, despite implications of sparse networks being suitable for low data regimes, only limited recent works [ [3 have validated them practically, and no prior work has evaluated their effectiveness for image recognition tasks in the low data setting.
In this work, we provide empirical evidence to endorse the suitability of sparse networks, iden- tified by IMP, for data-limited image recogni- tion. We conduct extensive experiments to com- pare the relationship between network capacity, training data size, and image classification accu- racy. We first show that sparse winning tickets have substantially better performance on low 50-100 samples per class) data regimes com- pared to their original dense counterparts. Next, we show that sparse winning tickets for data-
100
Classification Accuracy (in %
90
80
70
60
efficient learning compliments and can be com-
50
bined with existing methods such as augmen- tations or fine-tuning (from self-supervised backbone), to further boost their attainable per- formance by much as 16%. We further sess the generalization capacity of the networks through detailed robustness tests synthetic and domain-shifted datasets. We also eva waae special data-deficient cases like long-tailed clas- sification to compare performance when some classes have far few samples than others. Finally, we try to understand the properties of sparse net- works that lead to their superior performance. We find that both capacity and connectivity to- gether are important for generalization. Our key contributions can be summarized as:
40
ResNes-AutoAug (denze)
ResNes AAttong (winning ticket)
30
100
50
20
10
2
1
Train Data Size (in %)
Figure 1: Sparse winning tickets are data efficient rec- ognizers. Winning tickets (green) obtained by iterative magnitude pruning not only sparser but also show considerably improved performance, by as much as 16% (absolute), compared to the original network (blue) when trained on smaller subsets (as little as 1% size) of CI- FARIO dataset.
1. We find that the sparsity of the winning ticket increases with reduction in training data. 2. We demonstrate that winning tickets can easily combine with existing data-efficient training strategies such as augmentations to outperform similarly trained dense networks on low data regimes by as much 16%.
3. We further show that winning tickets, on low data regimes, are robust to synthetic noise and domain shifts, and show improved performance long-tal long-tailed classification.
4. Our analysis indicates that on data-limited regimes, sparse winning tickets continue to preserve density in initial layers, and both capacity and connectivity are important for generalization.
2 Related Works
Sample-Efficien Efficient Image Recognition. Learning from few samples and less data has been long- standing topic of interest in machine learning. While transfer learning, particularly from ImageNet pretrained models 3 is the most popular approach, semi-supervised techniques [15] and self-
supervision [16] have been successful when there is an abundance of unlabeled data in the same domain. All of these typically involve pretraining the model on data from a similar domain, or the same domain, often with variety of augmentations and then tuning on few labeled examples. These are complimentary to our approach, and we explore combining augmentations and transferring in our experiments. Few-shot methods are also related and they usually look at tuning just the final classification layer [5] [7]. A different approach is meta-learning [18). which aims to identify network from the support classes, seen during pre- traininng and which can easily adapt to the target set of classes in few steps. Random pruning was mentioned for few-shot learning [19] but much was left for future investigation. Our work is different from typical few-shot methods two ways: !I) We do not limit ourselves to transfer learning and also focus on training from scratch directly from the small dataset; ② We cover data sets with approximately 50-100 samples per class which lies between most few shot settings and typical deep learning settings with more data.

---

There are also works that directly use small datasets without external dependence on pre- -reiiriin often relying on training strategies to prevent overfittnng Some researchers propose use the cosine (t-vMF) loss [2]] along with the standard cross entropy for training. These regularizing loss functions reduce intra-class variance. Some other methods rely on extracting useful geometric priors to help avoid over- wer-fitting, such as preset cosine filters [22] and full convolutions (23) which exploits the absolute spatial location of objects in the image and improves translation invariance to strengthen the visual inductive prior. In this work, we cover datasets containing roughly 50-100 samples per class and the above methods have shown strong performance in this regime [24] and offer easy implementation. Therefore, in this work compare against these methods and study if they supplement pruning techniques.
loss 20 while some use the t-Distributed von
Pruning, Among pruning based methods, the iterative magnitude pruning (IMP) (described in Sec. 3.1 25 is most relevant to our work. IMP is a process that helps identify sparse sub-networks ("winning tickets") within a larger dense network that is capable of matching or potentially out- performing the dense network. This has been successful on a variety of tasks including image classification [26), natural language processing 27 28), and generative modelling [29]. While IMP has been popular for obtaining sparse networks, we are not aware of works that explore its relationship to data efficiency. In this paper, carry out detailed experiments to validate if sparsity, particularly "winning tickets" obtained via IMP, improves data-efficiency of deep neural networks.
3 Methodology and Experiments
Our approach primarily uses iterative magnitude pruning (IMP) and study the performance of the "winning tickets" in low data regimes. In this section, we describe IMP, and provide an overview of experimental studies including the basic experimental setup, augmentation strategies we compare against, and the datasets used for different studies.
3.1 Preliminary: Iterative Magnitude Pruning (IMP)
IMP [25] is introduced as way to identify sparse sub- sub-networks within a dense network that match or exceed the performance of the full dense network. They view these sub-networks as having won the initialization "lottery" and refer to them as "winning tickets".. We use the IMP procedure with and describe it briefly below.
rewinding 30 31
Let f(x, 0) denote the neural network, where f denotes its parameters and x denotes its input. Then the sub-network can be characterized by a binary mask m and can be applied to the original network via a Hadamard product (0) to obtain the sub-network f(x, 0 C m). For a network initialized with Oo the IMP process aims find a final mask m of specified sparsity ratio (s) as follows:
1. initialize m to be all ones mask.
2. train f(x, 0o 3 m) for steps to get 0r. This will be used as the initialization for the later pruning iterations.
3. continue to train f(x, 0, m) to convergence.
4. remove portion of the weights (determined by the pruning rate p) with the smallest magnitudes from 0 m. Update m by setting the removed weights to O.
5. repeat steps 3, 4 until the specified sparsity ratio (s) of pruned vs. original weights is achieved.
Note that when r the above algorithm reduces to IMP without rewinding. Rewinding is found to be essential for the successful and stable identification of winning tickets in large networks [30] 31]. Since this an iterative process, we obtain sub-networks of different sparsity at each stage. In all our experiments, the sub-network that has the best performance, across all iterations, on the validation set termed as the "sparse winning ticket".
3.2 Overview of Experimental Studies
We provide a broad overview of our experiments before presenting the details and datasets used.
(1) We first study how the network capacity of models pruned via IMP affects classification per- formance when training on datasets of different sizes. Since augmentations are often employed during the training of models in low-data regimes, we also perform this study with models trained
3

---

using different augmentation strategies. We compare the performance of dense models with just augmentations against those with both augmentation and pruning.
(2) Next, to study if sparsity prevents over-fitting by avoiding memorization, we evaluate the perfor- mance of the pruned models synthetically transformed dataset and domain-shifted dataset.
(3) Aside from augmentations, we also compare IMP against other training techniques employed in low-data regimes discussed in Sec. 2 as well fine-tuning from ImageNet and SimCLR backbones. All our main experiments are on ResNet-18, and we also show these results hold on ResNet-50, non-residual networks like VGG, and parameter-effiete MobileNet-v2.
(4) To verify the generalizability of the approach and measure performance of sparse winning tickets in data-limited regimes, we perform studies on (a) 3 real datasets with few (50-80) images per class, and also (b) simulated datasets with imbalanced (long-tail of) classes. We also present results on more complex data subsets of ImageNet and CIFAR100 100 with only 5-50 images per class.
(5) Finally, we try to understand why sparse winning tickets are able perform well in data- limited regimes. We do this by analyzing network capacity and connectivity, and also how pruning affects the layers which layers are getting pruned, the layer-wise representation similarity, and the generalizability of the representations.
3.3 Experimental Setup
We use the ResNet model (specifically ResNet-18 unless otherwise specified) in all our experiments. Since we are working with smaller-sized images (often 32x32 or 64x64), we modify the initial convolution layer to a 3x3 kernel, with padding with max-pooling. For images of size 224 we do not make any modifications. We follow the IMP lottery findi--inding procedure with 16 pruning iterations where 20% of the weights are pruned after each iteration. each pruning iteration, the model is trained for 200 epochs to minimize the cross entropy loss using the SGD optimizer with weight decay 0.0005. The initial learning rate is set to O. and then cosine decayed for 200 epochs. For fine-tuning experiments, we use a lower learning rate of 0.001. During the first iteration, we use the rewinding technique and set r = epochs.
3.4 Augmentation Strategies
Augmentations are one of the default strategies applied when working with limited data. In this work, we look augmentations in two scenarios: (1) as a comparison to the performance of the winning ticket, i.e., comparing the dense model with augmentations to that of a pruned model without augmentations, and (2) we combine augmentations with IMP to see the benefit of augmentation in addition to pruning. We carry out experiments with four different augmentation strategies each of different strengths. The strategy Basic consists of just RandomCrop and RandomFlip. All the other strategies are composed on top of the Basic augmentation. Contrast strategy additionally consists of ColorJitter and CandomGray [66.. Auto uses Basic augmentations and also the best policies identified for each dataset by AutoAugment (32). Rand is based on RandAugment [33], and composes four randomly selected augmentations in addition to the Basic augmentations. We can potentially order these 4 augmentation strategies by increasing strength Basic Contrast Auto < Rand.
3.5 Datasets
CIFARIO (34 is our primary dataset for all analysis experiments. consists of 60,000 colour images (train: 50,000, val: 10,000) of size 32x32 split into 10 classes. For experimental comparisons to the data-limited regime, we sub-sample at various sizes while maintaining the original class-size ratios. E.g.. at 1% data size, the training set contains 50 images per class. We aways evaluate the full validation set in all our experiments. For the robustness and extended experiments, we use additional datasets described in the corresponding sections.
CIFARIO-C [35). We use the CIFARIO-C (C for corrupted) dataset to evaluate the robustness of the models to (unseen) synthetic transforms. It contains 19 different transforms each of 5 different strength levels. These transforms can be broadly classified into Noise (e.g. impulse noise), Blur (c.g. motion blur), Weather (c.g. snow), Digital (c.g. JPEG), and Extra (c.g. brightness).

---

CIFARIO.2 [36]. To evaluate the network's performance domain shifts. we use the CIFAR10.2 dataset derived from Tinylmages [37]. It contains the exact same classes the original CIFARIO dataset but has a different and harder collection of images ideal for evaluating distributional shifts.
CLaMM 38 ISIC (39] and EuroSAT 40 Aside from experiments subsampled versions
of CIFAR, we use three diverse low-data regime datasets covering different domains with varying numbers of classes and images per class. The EuroSAT dataset [40] contains 64 x 64 sized satellite
images from 10 classes with 50 images per class captured from Sentinel-2 satellite. It consists of
multiple channels. We follow one standard procedure [40] to use the RGB channels only. ISIC 2018 dataset [39] contains 224 x 224 sized dermoscopic skin lesion images from 7 classes with 80 samples per class. The CLaMM dataset [38] contains 224 x 224 sized gray-scale latin scripts from hand-written books from 12 classes with 50 images per class.
CIFARIO R100 34 For studies on the performance of the sparse networks on imabalanced and long- tailed classification tasks we derive a dataset based on CIFAR 100. We selectively sample the CIFARI00 dataset such that the class-wise image count follows long-tailed distribution (defined by exponential curve given by: x (i/(n-1) where N is total number of samples in the dataset, is the class index and m==100) is the number of classes in the dataset. The degree of
is defined by the imbalance factor (A), i. lower value indicates more classes have lower sample count. The derived long-tailed class-wise distribution of CIFAR 100 is visualized in the Appendix C
4 Results
4.1 Sparse winning tickets show superior performance in low-data regimes.
100% train data
50% train data
20% train data
85.0
90.0
Classification Accuzzy (in %6
95.0
87.5
92.5
90.0
85.0
100
75
50
25
100
75
50
25
100
75
50
25
10%
train data
2% train data
1% train data
85
50
80
50
75
100
75
50
25
100
75
50
23
100
75
50
25
Network Density (in %))
Figure Combining pruning with augmentation strategies on CIFARIO. Performance on data subsets of different sizes (subplots), at different sparsity levels (x-axis) for augmentation strategies (lines). * indicates winning tickets. Winning tickets are sparser and substantially better performing at low-data regimes (row 2). Augmentations further improve the performance of the winning tickets.
We first verify our main claim and show the suitability of sparse winning tickets for data-limited regimes. Fig. Ba(ad Fig. presents the results training a ResNet-18 network on varying CIFARIO subsets of sizes 100%, 50%, 20%, 10%, 2%, and 1%. At the lower data sizes, the sparse winning ticket (denoted as WT) shows considerably improved performance compared to the dense model.
We also investigate how pruning compares with augmentation strategies and if both of these can be combined. Fig. Ba shows the results of the perforrenn dense and sparse networks with augmentation. The best is chosen across all augmentations for each data size and are indicated by Dense+Aug" (or D+Aug") for the dense model and WT++ for the winning ticket. We can see that across all data sizes, models pruned with augmentation outperform unpruned ones and as the data size decreases the gap between the two widens indicating that sparse networks are suitable for Rata-limited regimes.
5

---

Fig. 2 presents detailed comparisons of the performance of the model on the different data subsets (each subplot) at different sparsity levels (subplot x x-axis) for the different augmentation strategies (lines of different styles). In almost all cases, the best performing model is sparse i.e density < 100%. While in the higher data regimes (top row), there decrease performance with increased sparsity, the trend is reversed in the low data regimes (bottom row). More importantly at the lower data sizes i. 2%, 1%, it's clear that the sparse winning tickets (indicated by stars in Fig. 2 significantly outperform the dense (100% density) models with augmentations. Further, among the augmentation strategies, Auto-augment seems to consistently yield the best performance, except in the 1% data size where Rand-augment does much better. We further discuss the relevance of applying augmentations during the pruning stage in Appendix.
4.2 Winning tickets are robust to distributional shifts.
100
103
100
Clusdiration Accuracy (in %)
Closilfiat fon Accurasy (in SG)
Arge WYsakg"
80
Clorittration Accurny
60
40-
60
200
29% 20
150
10055
20%
1056
Traiz. Data Size (in 56)
Train Data Size (in %0)
Erain Data Stae (It 361
(a) Evaluated on CIFARIO 10.
(b) On CIFARIO-C synthetic transforms
(c) On CIFAR10.2 domain shifts.
Figure 3: Comparison of the dense model, and sparse winning ticket (WT), and best performing dense and sparse models with augmentation (+Aug*) trained on varying CIFARIO subsets and evaluated on CIFAR10 10, and CIFARIO-C IO-C and CIFAR10.2 for robustness.
Syuthotic Transforms (Cifar1U 21%
100156
MI56
10%6
2%
1%
Clocsiiction Accuracy (In ())
76
70
80
70
60
100
Domsin shißed [Cifar10
10016
6456
20%
1056
25%
1%
50.0
77.6
75.0
72.5
100
30
200
30
100
100
100
100
50
Network: Dessity (in 56)
Basc
Conrrrat
Stard
Anto
Figure 4: Robustness of models trained on CIFARIO subsets synthetic transforms (row-1) and domain-shifted (row-2) datasets. Shows performance of models trained on CIFARIO subsets of different sizes (columns) different sparsity levels (x-axis) for 4 augmentation strategics (lines). Sparse winning tickets identificd on the CIFAR10 val. set are indicated by and A indicates the most robust models.
As the data size gets smaller, the chance of overfitting and memorization the training samples increases. Thus, to study robustness, we evaluate the performance of the identified sparse winning tickets on two types of distributional shifts: (a) synthetic transforms via the CIFAR10-C dataset [35] and (b) domain-shifti via the CIFARIO. dataset [36]. Note that only perform evaluation (i.e. inference) on these datasets to study the robustness of the winning tickets to these transformations. We evaluate the network's performance on:
(a) synthetic transforms on the CIFAR10-C dataset. Row-1 in Fig. and Fig. 3b visualizes the average accuracy of the models across all 19 unseen transforms.

---

CIFARIO (25)
CIFARIO (1S))
BUROSAT
D+AUG"
WT+AS0*
D+ALO"
WT+ADG*
D+AS0"
D+AUG"
RANDOM (R18)
70.05%
$0.299
IMKGINET ISIT (R18)
75.50%
77.92%
47.469
59.72%
90.755
IMAGENET INIT. (R50)
51.06%
57,03%
61.73%
64.88%
92.89%
SIMCLR IsI. (R18)
0900
19%
COSINE Loss
64.82%
12.63%
45.8716
64.67%
49.60%
60.15
$9.0316
61.00%
$2.28%
$5.5815
TMM Loss
62.23%
72.81%
41.7015
64.54%
24.50%
59.87%
36.22%
59.309 30.30%%
71.14%
85. 2615
FULL CoNy.
62.16%
73.24%
49:30%
64. 20.
46.77%
57.91%
36.4214
58.20%
76.05%
77.066%
HARMONIC NeTs
61.36%
66.48%
22.977
49.85%
42.58%
44.14%
30.5646
52.51%
79.40S
83.2844
VGG
64.64%
71 11.0
$1.89 888
61. 11.
MORILONETVV2
64.64/
71.01%
$1.8%
61.63%
Table I: Comparing the dense model (D+Aug") and winning ticket (WT+Aug") using IMP combination
with other data-efficient training techniques on (columns 2. 3) 2% and 1% data subsets of CIFARIO, (columns 4, 5, 6) diverse datasets with few examples. R18 and R50 denote ResNet-18 18 and ResNet-50. The sparse winning ticket substantially outperforms dense model with all approaches.
(b) domain shifted CIFAR10.2 dataset consisting of wider variety of harder images from the CIFARIO classes. Row-2 in Fig. 4 and Fig. [3b pbepeentts the results.
In both cases, similar to performance on the CIFARIO data (Fig. 2 on 10% and lower training data sizes, sparser winning tickets generalize better these unseen transforms and distribution shifted data, much more so with augmentations. The difference in classification accuracy between the sparse winning ticket and dense model is also more clearly noted from Figures. 3b and 3c where we look at the just the best performing models (as determined CIFARIO val. set) dense model and winning tickets with and without augmentations. These results clearly indicate that sparse winning tickets, when trained as typical with augmentations, are capable of avoiding memorization of training samples at the smaller data sizes, and much more so than the dense models. This reinforces its suitability for data-limites regimes.
4.3
IMP complements existing data-efficient training.
Table. | (columms CIFARIO 2%, 1 ()) summarizes results using IMP to supplement and complement existing data-efficient methods. In particular, we apply IMP in combination with methods that have been shown to work well on small datasets: fine-tuning from ImageNet and SimCLR [16] backbones, self-regularizing cosine loss [20), use of full convolutions
(23) (Full Conv.), as well emergent methods, t-VMF loss [21] and harmonic nets [22]. Networks are trained using the best-identified augmentation strategy at each data size i.e Auto-augment at 2% and Rand-augment at 1% data sizes respectively. Despite the simplicity of our approach, the winning ticket identified from randomly initialized network outperforms almost all specialized data-efficient methods. When combined with the above data-efficient techniques, IMP further improves performance on the CIFARIO val. set on an average by 8% and 15% at 2% and 1% data sizes respectively. IMP shows more significant magnitude-based pruning strategy does not work
gains with non-fine-tuning methods, as (absolute) well with pre-trained initializations [1I, though we do see substantial improvements even on fine- tuning, As seen in Table I our results ResNet-18 extend beyond residual networks to VGG, and MobileNet-v2 based architectures.
4.4 Generalization to low sample datasets.
To evaluate generalization to datasets other than subsets of CIFARIO, we experiment on three di- verse datasets. The EuroSAT [40]. ISIC [39] and CLaMM (38) datasets cover different domains satel- lite imagery, medical images, and books respectively with 7-12 classes but few images (50-80) per class. Table. I (columns CLaMM, ISIC, EuroSAT) com- pares the performance of the dense model and IMP, using the Basic augmentation strategy and in com- bination with existing data efficient techniques for low sample data. The winning tickets outperform the dense model on an average by 12%, 2.3%, and 5.5%
DATASET
D+AUG
AT+
IMAGENET (5%)
28.82%
31.04%
CIFAR 100 (2%)
17.21%
25.06%
CIFAR 100 (1%)
11.21%
16.44%
Table 2: Comparison of the dense (D+Aug), and sparse winning ticket (WT+Aug) trained complex data subsets with many classes.

---

on the CLaMM, ISIC and EuroSAT datasets respectively. Table. discusses results on 2% (or 10
samples/class) and 1% (or samples/class) data subsets of CIFARTOK and ImageNet 5% 42 (or 50 samples/clas)), and our results indicate the effectiveness of winning tickets to handle much larger number of unique classes even in the absence of sufficient data.
4.5 Long-tailed classification.
Long-tailed classification a specific data-limited setting where some classes of the dataset contain very limited sam- ples compared to the rest. The presence of the dominant class can lead to over-fitting Therefore, we evaluate the effectiveness of sparse networks in such a dataset derived from CIFAR 100 (see. Sec. (.5). Table. B summarizes these results and we can see that the winning ticket has slightly better performance than the original dense net- work. An important note is that the total dataset size after imbalance is between 40% and 20% of the actual size (100-200 examples per class) therefore, the performance improvements are smaller. Nevertheless, the gains indicate that sparse networks are able to avoid over-fitting due to class imbalance and not just uniformly reduced data size.
IMBALANCE FACTOR
D+AUG.
WT+AUG.
0.10
64.77
64.92
0.05
58,43
59.43
0.02
50.91
51.72
0.01
45.13
46.73
Table 3: Comparison of the dense and sparse winning ticket with basic augmentations for long-tailed classification on data that's 's imbal- anced to different extents.
5
Discussion
Here we empirically investigate the factors that contribute the winning ticket network's performance gains in data-limited regimes.
5.1 Network capacity and connectivity are important.
We first investigate if it is the winning network's capacity or the connectivity be- tween neurons (that are retained after pruning) that contribute to the performance. During pruning even as the network capacity reduced, the connectivity between layers is also changed. We study the original dense net- work, and the sparse winning ticket with 3 ad- ditional networks: (1) A small dense network which is dense network but has overall param- eters reduced to match the winning ticket. (2)
Denmo+Auu
Classification Accuracy (in %)
100
Small D. Deane Augg
Randim Prennt Aug"
Rome Loayrr Promue Arg"
WT L'ugg
80
randomly pruned network where we take the original dense model, but randomly set some of the weights to zero to match the sparsity of the winning ticket. (3) A lame-laye--pparee network that follows the exact same layer-wise densities as the winning ticket but the connectivity be- tween the layers is random i.e. the binary mask each layer is randomly initialized but must match the total number of active parameters in that specific layer the winning ticket. We train these networks on the different CIFARIO 10 sub- set sizes using the identified best augmentation strategy and compare their performance in Fig.
60
40
20
100%
50%
20%
10%
2%
1%
Train Data Size (in %)
Figure 5: Comparison of the base dense, small dense. randomly pruned, same-layer-sparee and winning ticket models trained on varying CIFARIO data sizes.
[5] The performance of the networks with lower capacities (small dense, random pruned, same-layer-sparse and winning ticket) are all improved compared to the base dense network particularly in the lowest data size setting, indicating that a smaller network size improves data-efficient performance. However, the winning tickets significantly outperform a network of similar capacity i.e. the "Small Dense", indicating that beyond capacity perhaps the network connections also play an important role. To probe the importance of connectivity, we compare the winning ticket (WT+Aug*) against randomly pruned (Random Prune + Aug*) and the same-layer-sparrs network and observe that it outperforms both, emphasizing the importance of
8

---

connectivity. These lead us to conclude that both network capacity and connectivity play vital role improving the data efficiency of sparse networks.
5.2
Which layers are getting pruned?
Laye--wiaa Deneity (In %%
100%
50%
20%
10%
2%
1%
80
20
10 Layer Number
Figure 6: Layer-wise density of the sparsest winning tickets (i.e pruning iteration 16) for different training subsets. On smaller training data sizes, more weights from the initial layers retained, and peaks appear at the residual connection layers of cach block.
Fig. 6 presents the layer-wise density plots of the sparsest winning ticket at various training data sizes. Here density is defined as the ratio of the number of active parameters to the total number of parameters in that specific layer. We use the sparsest model identified (i.e 16th pruning iteration) using the best augmeniation strategy at that particular data size. We can see striking difference between the sparse models identified Particularly, the initial layers are denser as the training data size reduces. Retaining more weights in the initial layers appears to allow the sparse models to keep the filters for detecting primitive features such edges, and corners perhaps helping it generalize better. Another interesting thing to note is that the identified sparse networks at lower data sizes have three peaks that correspond to the residual connection layers of each block. As discussed in Sec. 4.2 the winning tickets exhibit higher robustness to several corruptions, especially in the least data sizes (2%, 1%). We hypothesize that the retention of denser residual layers enables minimal change to the output even upon input change (or identity connections), hence enabling increased robustness 43
higher dataset sizes when compared to the lower ones.
5.3 Generalizability of the learned representations.
To empirically verify that the representations of the sparse winning ticket are more transferable we take 4 models: the dense network train on 100% of the data, along with the dense, dense+aug., and WT+aug. networks trained on % of CIFARIO. In Table. present the results of freezing the backbone and fine-tuning only the final linear classifier on the CI- FARI00 dataset. Surprisingly the winning ticket not only outperforms the dense (1%) and dense+aug. net- work, but it also outperforms the dense model trained on 100% of CIFAR1O. This indicates that the sparse sub-networks identified the least data size learns generalizable representations of the input image.
SUB-NETWORK
ACCURACY
REFERENCE (DENSE, 100%)
20.79%
DENSE (1%)
6.42%
DENSE+AUG. (1%)
4.94%
WT+AUG. (1%)
23.67%
Table 4: Performance of linear classifiers on CI- FAR 100 for different sub-networks (backhones) trained CIFARIO data (% of data indicated in brackets) to evaluate generalizability representa- tions.
5.4 Layer-wise representation similarity
We use Centered Kernel Alignment (CKA) 44 to study the similarity of internal representation structure across different models in Fig. 7 CKA computes the normalized Hilbert-Schmidt inde- pendence criterion between two representations either within or across networks, averaged across samples within a mini-batch. It is better suited to measure similarity between high dimensional repre- sentations where the dimensions far exceed the number data points. We visualize the intra-layer wise similarity across three models: dense network trained on 100% data, along with dense+ Aug", and WT+ networks trained on 1 1% CIFARIO. We see that the sparse winning ticket exhibits larger span of similar initial layers unlike the others which exhibit more block similarity.

---

Desee-ta (PN)
W (N8)
(100%)
Dumor (130
Dasse
DaserAag
Figure 7: CKA Layer wise representation similarity across various networks. Comparing the similarities of the layers in the Dense, Dense+Aug', and WT+Aug" with themselves, and then the Dense+Aug" and WT+Aug" with the Dense models.
is possible that the increased layer-wise similarity directly related the denser residual connections ob- served in Sec. 5.2 To probe this, we plot the ratio of norms of the output from the residual branch and the main branch in Fig. g A higher value would indicate stronger propagation of information from the resid- ual branch while a lower value would indicate the vice-versa. Surprisingly, these ratios are lower the sparse network when compared to the dense network, which potentially suggests that the sparse network is in fact learning globally generalizable features. It expected that a network trained on the complete 100% data is sufficiently more generalizable than one trained on fewer samples. Therefore, we compare the WT, Dense (+ Aug*) networks trained only on 1 %% data to a network trained on the complete 100% data, and notice higher layer-wise similarity in the case of the sparse network. These observations go in hand to further support our empirical analysis regarding the generalizability of the Iearned representations.
2.5
Roven Darznn Donow Ang"
Ratio of norms of residual and main branch
2.0
1.5
1.0
0.5
0.0
3
Residual Layer Number
Figure 8: Ratio of norms of the output from the residual branch and the main branch for net- work trained on 1% data subset.
Conclusion
In this work, we hypothesized that winning tickets identified via magnitude pruning would be more effective in low-data regimes than their dense counterparts. Based on extensive experiments on sub-sampled versions of the CIFARIO dataset, we find that with decreased training data, the winning ticket gets sparser and when combined with augmentations considerably outperforms the original dense network. We also verify that the sparsity of the winning tickets helps avoid memorization and prevents over-fitting, by evaluating performance that IMP continues to show improvements in performance when combined with other data-cfficient training strategies for low-data regimes. We further evaluate the generalizability of the approach diverse datasets and simulated imbalanced datasets which have 50-100 examples per class. Finally, analyze why sparse winning tickets show superior performance low-data regimes. Our findings indicate that their performance can be attributed to combined properties of lower network capacity and connectivity, and help the network learn more generalizable representations.
distributionally shifted datasets. We show
References
[1] Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference Learning Representations, 2018.
[2] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770-778, 2016.
[3] Simon Kornblith, Jonathon Shlens, and Quoc V Le. Do better imagenet models transfer better? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2019.
10

---

[4] Kaiming He, Ross Girshick, and Piotr Dollár. Rethinking imagenet pre-training. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 1918-4927, 2019.
[5] Wei-Yu Chen, Yen-Cheng Liu, 7solt Kira, Yu-Chiang Wang, and Jia-Bin Huang. A closer look at few-shot classification. In International Conference on Learning Representations, 2019.
[6] Maithra Raghu, Chiyuan Zhang. Jon Kleinberg, and Samy Bengio. Transfusion: Understanding transfer learning for medical imaging. Advances in neural information processing systems, 32, 2019.
[7] Andrew Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andrectto, and Hartwig Adam. Mobilencts: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv: 1eeei70440000 04861, 2017.
[8] Shai Shalev-Shwartz and Shai Ben-David. Understanding machine learning: From theory to algorithms. Cambridge university press, 2014.
[9] Michael 7hu and Suyog Gupta. To prune, or not prune: exploring the efficacy of pruning for model compression. arXiv preprint arXiv:1710.011
2017.
[10] Franco Pellegrini and Giulio Biroli. Sifting out the features by pruning: Are convolutional networks the winning lottery ticket of fully connected ones? arXiv preprint arXiv:2 204. 104 13343, 2021.
[11] Shuai Zhang, Meng Wang, Sijia Liu, Pin- Yu Chen, and Jinjun Xiong. Why lottery ticket wins? a thcoretical perspective of sample complexity sparse neural networks. Advances in Neural Information Processing Systems, 34:2707- 2720, 2021.
[12] Orevaoghene Ahia, Julia Kreutzer, and Sara Hooker. The low-resource double bind: An empirical study of pruning for low-resource machine translation. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 3313-3333, 2021.
[13] Tianlong Chen, Zhenyu Zhang, Santosh Balachandra, Haoyu Ma. Zehao Wang. Zhangyang Wang, et al. Sparsity winning twice: Better robust generalization from more efficient training. In International Conference on Learning Representations. 2022.
[14] Tianlong Chen, Yu Cheng, Zhe Gan, Jingjing Liu, and Zhangyang Wang. Data-efficient gan training beyond (just) augmentations: A lottery ticket perspective. Advances in Neural Information Processing Systems, 14:20941-200955 2021.
[15] David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin A Raffel. Mixmatch: A holistic approach to semi-supervised learning. Advances in neural information processing systems, 32, 2019.
leaming
[16] Ting Chen, Simon Komblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning visual representations. International conference on machine learning, pages 1597-1607, PMLR, 2020.
[17] Yonglong Tian, Yue Wang, Dilip Krishnan, Joshua B Tenenbaum, and Phillip Isola. Rethinking few-shot image classification: a good embedding is all you need? In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XIV 16, pages 266-282. Springer, 2020.
[18] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. International Conference on Machine Learning, pages 1 1116-1135 PMLR, 2017.
[19] Chen Liu, Chengming Xu, Yikai Wang, Li Zhang, and Yanwei Fu. An embarrassingly simple baseline to one-shot learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition Workshops, pages 922 923, 2020.
[20] Bjom Barz and Joachim Denzles. Deep learning on small datasets without pre-training using cosine loss. In Proceedings of the IEEF/CVF Winter Conference on Applications of Computer Vision, pages 1371-1380. 2020.
[21] Takumi Kobayashi. t-vmf similarity for regularizing in-class feature distribution. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
[22] Matej Ulieny, Vladimir A Krylov, and Rozenn Dahyot. Harmonic networks with limited training samples. In 2019 27th European Signal Processing Conference (EUSIPCO), pages 1-5. IEFE, 2019.
[23] Osman Semih Kayhan and Jan C van Gemert, On translation invariance in cnns: Convolutional layers can exploit absolute spatial location. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14274 14285, 2020.

---

Towards an [24] Lorenzo Brigato, Bjöm Barz, Luca locchi, and Joachim Denzler. Tune it or don't t use it: Benchmarking data-efficient image classification. In Proceedings the IEEE/CVF International Conference on Computer Vision, pages 107 1071-1080, 2021.
[25] Song Han, Huizi Mao, and William Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint 1rxiv:1510.00144 2015.
[26] Zhuang Liu, Mingjie Sun, Tinghui Zhou, Gao Huang, and Trevor Darrell. Rethinking the value of network pruning. In International Conference on Learning Representations, 2018.
[27] Trevor Gale, Erich Elsen, and Sara Hooker. The state of sparsity in deep neural networks. arXiv preprint arXiv:1902.099 09974, 2019.
[28] Tianlong Chen, Jonathan Frankle, Shiyu Chang, Sijia Liu, Yang Zhang. Zhangyang Wang, and Michael Carbin. The lottery ticket hypothesis for pre-trained BERT networks. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6- 12, 2020, virtual, 2020.
[29] Xuxi Chen, Zhenyu Zhang, Yongduo Sui, and Tianlong Chen. Gans can play lottery tickets too. In International Conference on Learning Representations, 2020.
[30] Jonathan Frankle, Gintare Karolina Dziugaite, and M Daniel. Roy, and michael carbin. the lottery ticket 2(3), 2019.
hypothesis at scale. arXiv preprint arXiv:1903.cmmmm
[31] Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M. Roy, and Michael Carbin. Linear mode connec- tivity and the lottery ticket hypothesis. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119, pages 3259-3269. PMLR, 2020.
[32] Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quae V Le. Autoaugment: Learning augmentation strategies from data. In Proceedings of the IEFF/CVF Conference on Computer Vision and Puttern Recognition, pages 113-123, 2019.
[33] Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with reduced search space. In Proceedings of the IEEÉ/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 702-703, 2020.
[34] Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.
[35] Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. Proceedings of the International Conference on Learning Representations, 2019.
[36] Shangyun Lu, Bradley Nott, Aaron Olson, Alherto Todeschini, Hossein Vahabi, Yair Carmon, and Ludwig Schmidt. Harder or different? a closer look at distribution shift in dataset reproduction. In ICML Workshop Uncertainty and Robustness in Deep Learning, 2020.
[37] Antonio Torralba, Rob Fergus, and William T. Freeman. 80 million tiny images: A large data set for nomparametric object and scene recognition. IEEE Trunsactions on Pattern Analysis and Machine Intelligence, 30: 1958-1970, 2008
[38] Dominique Stutzmann. Clustering of medieval scripts through computer image analysis: evaluation protocol. Digital Medievalist, 10, 06 2016.
[39] Noel Codella, Veronica Rotemberg, Philipp Tschandl, M Emre Celebi, Stephen Dusza, David Gutman, Brian Helba, Aadi Kalloo, Konstantinos Liopyris, Michael Marchetti, al. Skin lesion analysis toward melanoma detection 2018: A challenge hosted by the international skin imaging collaboration (isic). arXiv preprint 1xXX::1992.033000 2019.
[40] Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth, Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEFF Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12:2217-2226 2019.
[41] Victor Sanh, Thomas Wolf, and Alexander Rush. Movement pruning: Adaptive sparsity by fine-tuning. Advances in Neural Information Processing Systems, 33, 2020.
[42] Osman Semih Kayhan and Jan C. van Gemert. On translation invariance in cnns: Convolutional layers can exploit absolute spatial location. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). June 2020.
12

---

[43] Shuzhi Yu and Carlo Tomasi. Identity connections in residual nets improve noise stability. arXiv preprint arXiv:1903 10944, 2019.
[44] Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In International Conference Machine Learning, pages 3519-3529 PMLR. 2019.
[45] Ian Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. stat, 1050:20, 2015.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [N/A]
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed reproduce the main exper- imental results (either in the supplemental material or as URL)? [Yes] Please find code Supplemental Material, Relevant datasets are cited in Sec. 3.5
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [N/A]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [No]
4. If you using existing assets (e.g., code, data, models) curating/releasing assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [No]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
13

---

A
Augmentation is important to find better Winning Tickets
We can see from 3a that 1% data size, the performance gap between WT+aug.. Dense+aug. is significantly higher than their corresponding non-augmented versions. To verify the importance of augmentation during lottery ticket finding, we select the WT identified via Base- augeent strategy, reinitialize the network to 0, (refer Sec. (31) and train using the best augmentation strategy at each data size. From Table. [ we can see that the winning ticket identified via Base-augment cannot match the performance of the WT+aug* even when trained later with the same best augmentation strategy (indicated by WT((ase)+Auug)) Therefore, augmentation plays a significant role during ticket finding.
DATA SIZE
WT
WT(BASE)+AUG*
WT+AUG*
DATA SIZE
RATIO
100%
95.45%
95.85%
96.12%
100%
1.07
50%
93.42%
94.1% %
94.01% %
50%
1.09
20%
88.54%
90.01 %
89.99%
20%
1.22
10%
78.93%
83.23%
85.01%
10%
1.33
2%
45.25% %
57.74%
70.05% %
2%
1.39
1%
38.31 %
47.4%
59.66%
1%
1.23
Table 5: Comparison the winning tickets trained with Basc- augment (WT), best augment (WT+aug") and WT(Base) trained with best augment (WT(Bas))aaug" to evaluate the importance of augmentation during lottery ticket finding.
Table 6: Average ratio of norms of the remaining weights from the winning ticket and corresponding dense network trained on CIFARIO subsets.
B Adversarial Robustness of Winning Tickets
To evaluate the network's performance on adversarial attacks, we perform the FGSM [45] one-step attack with an epsilon value of 8/255. Similar to Sec. 4.2) Fig. 9 presents adversarial robustness of models trained on CIFARI0 10 subsets across different augmentation strategies and sparsity levels. Unlike the other cases (i.e synthetic and domain shifts), the sparse networks perform worse than their corresponding dense counterparts across all data sizes. We hypothesize that since the IMP procedure only retains the maximum magnitude weights, they would directly affect the gradients, hence increasing the effects of the adversarial attack. To verify the same, compute the average ratio of norms of the remaining weights between WT+aug. and Dense+aug across layers. Table. 6 clearly indicates that the winning tickets though sparse, have higher magnitude of weights across all data sizes (ratio > 1). At the least data sizes (2%, 1%), the absolute value of the remaining weights is roughly 1.3 times its dense counterpart thereby increasing the magnitude of the adversarial attack for a given epsilon value.
Clositiation Sovurary (to 50
Adversorrial (FCSMM
1000
609
20%
10%
2X
10
100
Natwock Emeelyy Goo 55)
Tramd
Figure 9: Robustness of models trained on CIFARIO subsets on adversarial attacks. Shows perfor- mance of models trained on CIFARIO subsets of different sizes (columns) at different sparsity levels (x-axis) for augmentation strategies (lines). A indicates most robust models.
C Long-tailed CIFAR100 Data Distribution
We visualize the long-tailed data distribution of the sampled CIFAR100 dataset for varying À values in Fig. T0 The number of samples per class varies from 5-500 samples/class when X = up to 50-500 samples/class when À = 0.1.
14

---

500
0.1
0.05
0.02
Number of Images
400
300
200
100
0
20
40
60
80
100
Class
Number
Figure 10: CIFARI00 long-tailed class-wise sample distribution across different imbalance factors A.
15


Reviews:
[{"review_id": "om3VTfruO_", "reviewer": "Reviewer_KGu3", "summary": "The paper presents a thorough study of sparse wining tickets in the context of limited data settings. Using iterative magnitude pruning and rewinding, the authors train deep models with limited data, and show that sparse winning tickets outperform standard dense networks. The authors also show that augmentation helps improve the performance for the sparse tickets. In addition, the authors study a long-tailed classification setup and observe that sparse WT outperform dense models even when trained with auto-augment.", "questions": "1. Can the authors share the parameters for their augmentation setup? ", "limitations": "The authors have mostly addressed the limitations by including experiments for a variety of setups.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**:\n\n1. The paper makes an interesting observation on the few-shot learning capabilities of sparse models. In addition, the experiments mostly support their claims. \n2. The experiments in Sec. 5 reveal some interesting properties of sparse wining tickets that aid in data-efficient learning.\n3. Technically, the paper appears to be sound, with principled experiments and observations.\n\n**Weaknesses**:\n\n1. While not a weakness per se, it appears that the combination of IMP + augmentation is actually the winning combination for data-efficient learning. Especially, fig. 2 and fig. 4 both show that augmentation appears to play a larger role. Perhaps the authors could add some points to their discussion.\n2. Another point of improvement could be training other architectures (transformers, VGG) to see if these properties are dependent on the architecture.\n3. Liu et al [1] ,have also studied pruning for few shot learning. The authors should possibly include a citation and discussion.\n\n\nOverall, the paper is a strong contribution by itself and reveals interesting properties.\n\n[1] Liu et al., An Embarrassingly Simple Baseline to One-shot Learning, 2022\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "cc7dDYoKkgd", "reviewer": "Reviewer_7i7B", "summary": "The paper investigates whether sparse winning tickets (obtained using Iterative Magnitude Pruning) improve image classification performance in low-data tasks (defined as 50-100 samples per class). The paper shows that sparse winning tickets outperform the full network by 10-15%, *but* this requires training using data augmentation. Without data augmentation, the winning tickets perform similarly to the dense network.\n\nThe paper’s contribution is empirical: it shows that sparse winning improves performance on tiny datasets.", "questions": "All my questions are related to missing explanations/analyses for the empirical results.\n\n1. Why do winning tickets improve performance? Is there any relation between the number of model parameters and the size of the dataset (including data augmentation)?\n2. Why is data augmentation required for sparse winning tickets to improve performance on very small datasets (last two subfigures from Figure 2)?\n3. Why do sparse winning tickets preserve the filters? (Figure 7)\n4. Why do sparse winning tickets preserve the residual connections (Figure 7)?\n5. What’s the connection between the output norms and the generalization of the representations? (Figure 6)\n6. Figures 3a/b/c show that training using data augmentation provides a 3x improvement on the winning ticket compared to the dense network (in the 1% data scenario). Why does this happen? In contrast, for over 10% of the training data, the improvement is similar for the WT and the dense network.", "limitations": "I don't see any negative social impact on their work.\n\nThe paper should clearly mention that the findings only apply to obtaining sparse winning tickets using Iterative Magnitude Pruning (IMP). Thus, all limitations of IMP also apply to the current work (e.g., IMP doesn't work well for pre-trained networks, and the presented experiments also show that the identified tickets on pre-trained networks provide small performance improvement).", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n- The paper provides a **good empirical contribution** to the literature on sparse networks and learning from small datasets. The results raise further questions that the community might continue investigating. I find the paper to be **relevant to the conference**.\n- The paper contains **extensive experiments**, investigating winning tickets in different low-data scenarios (standard classification, synthetic noise, synthetic transforms, domain shifts, highly imbalanced datasets). They show that winning tickets consistently improve performance when the network is trained from scratch.\n\n# Weaknesses\n- The paper’s core weakness is having a **very shallow analysis of the experimental results**. The paper has extensive experiments with useful results, but I found the provided analysis of the results to be weak. The authors provide little reasoning/explanations for their findings. Contribution 4 (line 60) is not well supported by the provided experiments and written analysis. The discussion section shows additional experiments and only briefly discusses the results without providing answers to any *why does this happen?* questions.\n\t- Section 5.1, and Figure 5, don’t answer why the network capacity and connectivity are important.\n\t- Section 5.2\n\t\t- The paper mentions: “Retaining more weights in the initial layers appears to allow the sparse models to keep the filters for detecting primitive features such as edges”. What does this result imply? Why do sparse winning tickets preserve the filters? Why don’t denser winning tickets preserve the filters? One hypothesis is that sparse tickets (on low-data) are simple functions (because they keep few parameters on the upper layers) of very well-extracted low-level features (because they keep most parameters on the first layers). Please discuss the findings if you want to keep Figure 7 in the discussion section.\n\t\t- “the identified sparse networks at lower data sizes have three peaks which correspond to the residual connection layers of each block”. What does this imply?\n\t\t- “the residual connections enable stronger gradients, and hence the maximum magnitude weights are expected to reside on the residual branch thereby increasing its density”. Why do you believe that large gradients imply large weight magnitude? There can be large gradients in both negative and positive directions (thus, the sum of the weights updates will be zero, and the weights don’t change).\n\t\t- “The ratio of norms are lower for the winning ticket suggesting perhaps the model learns more generalizable representations”. What’s the connection between the output norms and the generalization of the representations?\n\t\t- “In fact, the WT+Aug network showcases higher similarity to the network trained on 100% data when compared to the dense+Aug network. The above observations go in hand to further support our empirical analysis regarding the generalizability of the learned representations.” How does this observation support your empirical analysis on generalization? It’s not self-evident what you are implying.\n\t- **Potential solutions**: The paper would be significantly improved if the authors could explain their results more deeply. If not, then I suggest that the paper should claim *only* to identify an interesting phenomenon (i.e., sparse winning tickets outperform dense networks on small datasets). Also, the abstract/introduction/Contribution should clearly specify that the paper is purely empirical.\n\n# Small changes\n- In the title, replace “recognizers” with “classifiers” because the paper investigates only classification tasks\n- line 33: what “inductive bias” does pruning provide?\n- Improve citations. Many cited papers were published in conferences, but you’ve referenced the arxiv article. Please reference the published versions of the papers.\n- In section 3.2, consider reordering the experiments: 1, 3, 2, 4, 5. I think 1 and 3 are related: 1 investigates data augmentation, and 3 investigates other training methods for small datasets (outside data augmentation). 2 and 4 also seem related because 2 analyses memorization and 4 investigates generalization.\n- Consider moving section 3.5 (datasets) before section 3.4 (Augmentation strategies)\n- Line 220: clarify what type of “memorization” are you referring to. As I understand, you test the “memorization” of the domains of the images because the experiments from Section 4.2 investigates synthetic transforms and domain shifts.\n- Lines 235-237 provide an overstatement: “these results clearly demonstrate that winning tickets… are capable of avoiding memorization of training samples at the smaller data sizes, and much more so than the dense models”. There are many types of memorization, and the presented results don’t “clearly demonstrate” that winning tickets “avoid” it. Rephrase and reduce the overstatement.\n- line 196 can be removed\n- line 248 - add a short one-line about the argument [37] makes on why IMP doesn’t work well on pre-trained networks (i.e., because the magnitude of the weight doesn’t change much during fine-tuning, and the pre-training task largely determines the winning tickets)\n- align tables 3 and 4\n- Put figures in the order they are referenced in the paper. Figure 2 is referenced after Figure 3 .\n- Move the title of Section 4.2 before the text starts (line 220)\n- In line 316, add a short one-line description of how the CKA works\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "LwUL9RpB7b", "reviewer": "Reviewer_Skos", "summary": "This paper shows that winning tickets can be successfully used in data-limited regimes. The authors conduct several experiments to empirically validate their assumptions. The authors first show that sparse winning tickets have superior performance in data-limited regimes, also compared to a dense network of similar size or a randomly pruned network. Then, they also show that they are robust to distributional shifts, such as domain shifts and synthetic transforms applied to the data. Most of the results are based on the CIFAR10 dataset, but they consider also other low sample datasets from various application fields, including cases with unbalanced classes.", "questions": "- Fig. 4 is not very clear, how do you identify the sparse winning tickets and the most robust models (indicated using stars and triangles in the plots)?\n- In the paper the authors consider also the case where IMP is applied in combination with well-known pre-training methods for low data regimes, such as SimCLR. However, it is not clear to me how the fine tuning is performed, IMP usually works when the weights are rewinded to initialization (or to an early epoch), therefore I'm not sure if (and how) it can be combined with pre-training techniques.", "limitations": " The authors adequately addressed the limitations and potential negative societal impact of their work", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The authors show an interesting application of lottery tickets. \n- The experiments presented to validate their assumption are accurate and quite convincing, considering also synthetic noise and domain shifts. The authors consider also the case where lottery tickets are used in combination of other techniques for limited-data regimes, such as data augmentation.\n- The paper is well written and easy to follow.\n- I think that the Discussion section is very interesting, since it tries to give some insights on why lottery tickets can be a valid option in low data regimes. Since the paper is an empirical study, it is important to provide also an analysis that tries to explain the performance shown in the experiments and identifies the properties of the lottery tickets that are useful in this context.\n\nWeaknesses:\n- Since this paper is based on empirical evidence, I would have appreciated to see some experiments on different architectures. In the paper the authors perform all the experiments using ResNet-18, which is a quite simple architecture. I am curious to see if the same results could be obtained using also other architectures.\n- I found the results not very surprising. Even though there are no prior works on sparse networks for data-limited regimes in the context of image classification, there are some results in NLP.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "KrPrJ-pxfB", "reviewer": "Reviewer_ED1y", "summary": "This paper provides empirical evidence that sparse networks obtained using the lottery ticket hypothesis are also effective recognisers in settings where small datasets are available. Extensive experiments are conducted on multiple datasets and several interesting findings are reported, e.g., sparse networks in low data regimes are robust to synthetic noise, density is preserved in initial layers etc.", "questions": "\nSee comments/questions in weakness section.", "limitations": "Does not apply to this paper.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\nThe paper is well written.\n\nInteresting findings are reported.\n\nExtensive studies are conducted.\n\n\nWeaknesses\n\nMost of the experiments and findings are based on the CIFAR datasets. Why not using a more challenging dataset, like ImageNet? It would make the results more convincing.\n\nNovelty is limited, existing methods are used, and extensive experiments are run (and the findings are definitely interesting) but it would be desirable if some novelty was included.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept