Does Dataset Lottery Ticket Hypothesis Exist?
Authors: Zhiqiang Shen
Eric Xing
Venue: Submitted to ICLR 2023 (2023)
Keywords: Dataset Lottery Ticket Hypothesis
Self-supervised Learning

Abstract: Tuning hyperparameters and exploring the suitable training schemes for the self-supervised models are usually expensive and resource-consuming, especially on large-scale datasets like ImageNet-1K. Critically, this means only a few establishments (e.g., Google, Meta, etc.) have the ability to afford the heavy experiments on this task, which seriously hinders more engagement and better development of this area. An ideal situation is that there exists a subset from the full large-scale dataset, the subset can correctly reflect the performance distinction when performing different training frameworks, hyper-parameters, etc. This new training manner will substantially decrease resource requirements and improve the computational performance of ablations without compromising accuracy on the full dataset. We formulate this interesting problem as the dataset lottery ticket hypothesis and the target subsets as the winning tickets. 
  
In this work, we analyze this problem through finding out partial empirical data on the class dimension that has a consistent {\em Empirical Risk Trend} as the full observed dataset. We also examine multiple solutions, including (i) a uniform selection scheme that has been widely used in literature; (ii) subsets by involving prior knowledge, for instance, using the sorted per-class performance of the strong supervised model to identify the desired subset, WordNet Tree on hierarchical semantic classes, etc., for generating the target winning tickets.
 
We verify this hypothesis on the self-supervised learning task across a variety of recent mainstream methods, such as MAE, DINO, MoCo-V1/V2, etc., with different backbones like ResNet and Vision Transformers. The supervised classification task is also examined as an extension. We conduct extensive experiments for training more than 2K self-supervised models on the large-scale ImageNet-1K and its subsets by 1.5M GPU hours, to scrupulously deliver our discoveries and demonstrate our conclusions. According to our experimental results, the winning tickets (subsets) that we find behave consistently to the original dataset, which generally can benefit many experimental studies and ablations, saving 10x of training time and resources for the hyperparameter tuning and other ablation studies.

Under review as a conference paper at ICLR 2023
DOES DATASET LOTTERY TICKET HYPOTHESIS Ex- IST?
Anonymous authors Paper under double-blind review
ABSTRACT
Tuning hyperparameters and exploring the suitable training schemes for the self- supervised models are usually expensive and assurce-conssuming especially on large-scale datasets like ImageNet-IK. Critically, this means only few establish- ments (e.g., Google, Meta, etc.) have the ability to afford the heavy experiments this task, which seriously hinders more engagement and better development in this area. An ideal situation is that there exists a subset from the full large-scale dataset, the subset can correctly reflect the performance distinction' when per- forming different training frameworks, hyper-parameters etc. This new training manner can substantially decrease resource requirements and improve the com- putational performance of ablations without compromising accuracy using subset discovered configuration to the full dataset. We formulate this interesting problem as the dataset lottery ticket hypothesis and the target subsets as the winning tickets. In this work, we analyze this problem through finding out partial empirical data on the class dimension that has a consistent Empirical Risk Trend as the full observed dataset. We also examine multiple solutions, including (i) uniform selection scheme that has been widely used in literature; (ii) subsets by involving prior knowledge, for instance, using the sorted per-class performance of the strong su- pervised model to identify the desired subset, WordNet Tree on hierarchical se- mantic classes, etc., for generating the target winning tickets.
We verify this hypothesis the self-supervised learning task across variety of recent mainstream methods, such as MAE, DINO, MoCo-VI/V2. etc., with differ- ent backbones like ResNet and Vision Transformers. The supervised classification task is also examined as an extension. We conduct extensive experiments for train- ing more than 2K self-supervised models on the large-scale ImageNet-1K 1K and its subsets by 1.5M GPU hours, to scrupulously deliver our discoveries and demon- strate our conclusions. According to our experimental results, the winning tickets (subsets) that we find behave consistently to the original dataset, which generally can benefit many experimental studies and ablations, saving 10x of training time and resources for the hyperparameter tuning and other ablation studies.
INTRODUCTION
In the recent years, large deep neural networks, such as Convolutional Neural Networks (CNNs) (Le- cun & Bengio, 1995; He et al., 2016) and Transformers (Vaswani et al., 2017) have achieved break- throughs in the fields of supervised learning (Tan & Le, 2019; Dosovitskiy et al., 2020) and self- supervised learning (Kenton & Toutanova, 2019; Brown et al., 2020; Caron et al., 2021; 2020; He et al., 2022) empowered by the large-scale datasets. Naturally, the computational resources required for training these models the large data increasing accordingly. A dilemma is that many researchers do not have such resources to conduct experiments on the large datasets directly, espe- cially on the expensive self-supervised learning by tuning the hyperparameters and exploring the proper training settings and frameworks. A commonly-used practice in the vision domain is to con- duct ablations on relatively smaller datasets like CIFAR (Krizhevsky, 2009) and MNIST (Lecun et al., 1998), and then transfer the tuned optimal configurations to the large datasets like ImageNet- 1K (Deng et al., 2009). While in many cases, is observed that the models' behaviors and properties
In this work, we focus on the performance trend relative accuracy trend trained on the subset and full data across different train/cval configurations. The absolute accuracy on the individual subset not necessary.

---

Under review as a conference paper at ICLR 2023
10
SV1__ineer Jiner
5/25, linar DINO 8/16 Insar
DINO IV16 Inear
Loss
DINO 5/10, Innar
DINO 5/10, Jineer
GIND I/16 Inew
Esoch
Pre-train LoGs
DINO 8/16 linca
mre
RS-0
RS-1
DINO, Jnner
--. DINO 5/10 Jiner
Fre train Lass
DINO_ Bero__006__000 Inneer
Pre train Loss
DINO_B/11 0016
200
600
2000
Full (oracle)
Winning ticket
200
400
000
200
400
100
500
Epoch
Epoch
RS-2
RS-3
Figure I:
Illustration of the loss curves from self-supervised pre-training on full data, random subsets and winning ticket subset. ViT-Small and ViT-Base models are used as the backbone networks.
on the small datasets are quite different from the ones learned on the large dataset, making it im- proper to directly transfer hyperparameters found on small data the large one. Considering that self-supervised learning does not use human-annotater labels for training models, another popu- lar solution emerged by randomly choosing a subset from the full dataset, for example, randomly selecting 100 classes among 1,000 in ImageNet-IK IK for self-supervised ablations and exploring ex- periments (Tian et al., 2020; Kalantidis et al., 2020; Ermolov et al., 2021). Such a scheme has shown great advantages in lower resource demand of costly learning frameworks for fast hyperparameter tuning and model exploration with large backbone architectures.
However, according to the learning rule of Empirical Risk Minimization (ERM) (Vapnik, 1991; 1999) principle, the training convergence of ERM is guaranteed when the number of parameters of the neural networks scales linearly with the number of training examples. Under this principle, it is challenging to only leverage a subset of data to model the properties of the full larger number of training data, since in both of the settings, models are trained minimize their average error over the current training samples. Moreover, ERM is unable to provide generalization on unseen distributions from the subset to the whole, making this strategy full of uncertainty. Consequently, natural concern has been raised in this work: What kind of subset is qualified for evaluating self- superviseelsspprviseel methods on full data?
Goal of Dataset Lottery Ticket Hypothesis (DLTH): The goal of DLTH is to find out subset as the winning ticket from a large-scale dataset, this subset has the same or similar empirical behaviors and performance trends the original full dataset when performing different training approaches and hyper-parameters on it, meaning that can truly reflect the performance changes according to different training settings. Different from (i) data pruning (Zhang et al., 2021; Sorscher et al., 2022) that removes low-contribution and forgetting data to replace the full dataset, (ii) dataset dis- tillation (Wang et al., 2018; Cazenavette al., 2022) and condensation (Zhao al., 2020) that will generate a new compressed data, the proposed DLTH will not predict the accuracy of full data but select proper subset from the original data and the models trained on the subset under various con- figurations have consistent performance trend to the models trained on the full data. Thus, we can further use this subset for fast typer-parameter tuning, frameworks exploration or time-consuming tasks. More detailed discussions with the related tasks are provided in Appendix G.
Overfitting Issue on the Subset: The key observed issue of using subset data is that the self- supervised pre-training on the smaller training data (subset) will frailly suffer from overfitting. To reveal this, we visualize the pre-training losses in Fig. 1. Following DINO protocol (Caron et al., 2021), we train with 400 epochs for ViT-Base and 800 epochs for ViT-Small. On the full data, the evolution of training loss first drops rapidly, then slowly rises a little bit, and finally continues to descend on both small and base models. While the randomly selected subsets (RS-ID), the losses of base and small models are dropping constantly with a plateau. It is interesting to see on our identified winning ticket, the rebounding phenomenon on loss has emerged again at around 400 epoch of blue curve and the magnitude of loss value is generally larger than those on the random subsets, which is also more aligned to the trend of the full data.
2

---

Under review as a conference paper at ICLR 2023
Identifying Winning Tickets (Subsets). The most common practice is to randomly select sub classes among the full data's 's categories as the subset (Tian et al., 2020), however, it is natural that randomly selected subset is not guaranteed for reflecting the true accuracy change the full data when imposing different training settings. To this end, we propose to fix the learned hypothesis or models the full observed dataset, and identify the winning ticket which has the approximating distribution as the full observed data empirical distribution. We argue that the subset categories should have the consistency on performance similar that of the full dataset. For instance, the overall linear probing performance of DINO (Caron et al., 2021) is better than MAE (He et al., 2022) with the same ViT-B/16 backbone, while on the specific class such as toy poodle, MAE has the better accuracy than DINO, meaning that this class has an inconsistency on the two self-supervised models and will not be selected. The consistency indicates that the individual per-class accuracy should match the global accuracy across different frameworks". We also empirically examine other policies on determining the proper winning ticket subsets, including: (i) randomly generated lists as the baselines; (ii) incorporating prior knowledge (c.g.. performance-driven scheme according to the pre-trained supervised models, semantic hierarchy on WordNet Tree, etc.).
The practice of randomly selecting subset to evaluate different methods is more popular in self- supervised learning task. Thus, it is more crucial to clarify and understand this manner in self- supervised domain. While, supervised learning is also eligible explore the dataset lottery ticket hypothesis and we provide the results on supervised learning in Appendix due to the limited space. All of our models are trained large-scale ImageNet- IK and its subsets instead of MNIST, CIFAR, etc., with 3 runs for each, to avoid potential mis-observations and deliver more reliable conclusions.
Contributions:
Our experimental results indicate that a randomly selected subset without any prior knowledge unstable and generally not qualified for reflecting the properties of self supervueee models on the full data, and might further deliver misleading observations and conclusions. This is fairly risky if the studies rely heavily on such kind of subset, instead of the original full dataset.
We propose the Dataset Lottery Ticket Hypothesis (DLTH), a novel problem that studies the pos- sibility of identifying the subset which can reflect the performance consistency with the full data. Moreover, we propose the policies of Empirical Risk Trend and incorporating prior knowledge to generate the winning tickets. To our best knowledge, this is the first work to study the feasibility of dataset lottery ticket hypothesis, and it can be starting point for exploration on this problem.
We provide comprehensive experiments on ImageNet-Ie 1K and its subsets with variety of ad- vanced self-supervised frameworks, such DINO (Caron et al., 2021), MAE (He et al., 2022), MoCo series (He et al., 2020; Chen et al., 2020b). to verify the effectiveness and superiority of our proposed dataset winning ticket policies. We will make our found winning ticket subset publicly available to benefit other research in this field.
FROM EMPIRICAL RISK MINIMIZATION TO DATASET LOTTERY TICKET HYPOTHESIS
In the learning problem, a model is used to learn a conditional probability distribution or decision function. The hypothetical or mapping space of the model contains all the conditional probability distributions or decision functions. The goal of learning is to find an ideal hypothesis or mapping function h H with its parameter by observing the properties of the hypothesis space. Assume the input X and target Y of the model are random variables, which follow the joint probability distribution P(X, Y). we define a loss function L that measures the differences between predictions h(x)) and corresponding targets y. i.e., y) ~ P. Usually, the smaller the value of the loss function L on validation set, the better the model learned by the optimization. The expectation of the loss function is formulated as:
Rexp hh) Ep Ep[C(h(z; 0), v)) - -f
/xxx C()(z;0).0 v) P(x, y)dady
(1)
The learning perspective usually treats the optimization problem as simply minimizing the expected loss or called risk function. The distribution P is unknown in most practical situations. Instead, we
2 As shown in Fig. 3, on the linear probing evaluation, it is intriguing to see that DINO has the better overall accuracy, while MAE is more stable individual accuracy across different categorics.
3

---

Under review as a conference paper at ICLR 2023
Observed model performance
on full dataset
PuI Sevar-a-- d4 Ree-1
Model
Optimization method
Full Data
Can we find winning ticket on dataset?
Trend Consistency
Prior knowledge
ixix
X
X
x
Model
Optimization method
Sub Data
Observed model performance winning ticket
Suggested hyperparameters, optimizers, etc.
Tuning hyperparameters on Transfer tuned hyperparameters from winning ticket to the full dataset
winning ticket
Figure 2: Illustration of the motivation on the proposed DLTH. We aim to identify a subset from the original data through leveraging prior knowledge the dataset winning ticket, meanwhile, the subset should have the property that the observed model performance on is consistent with the performance on full data across different training frameworks, settings and hyperparameter choices.
have access to set of training data (=, y.)} (i 1,2...., m), we can leverage the empirical distribution P(x, Y) on the available training data to approximate the true distribution P. Thus, we further can approximate the expected risk according to the empirical risk:
m Σ m
Remp (h)
L (h (æ:;; 0), y:)
Through a proper optimizer, set of parameters is found on the training set so that the function can approximate the real mapping relationship. The next step is to learn such function h" (i.e., finding the learnable parameters 0*) that the empirical risk (ER) is minimal:
h arg min Remp (h)
(3)
h
According to the learning process of empirical risk, in this paper. we study the problem of whether there exist smaller available subsets as the winning tickets for training sulf-supervised models with similar performance trends to the full data. Training on the subsets is faster since the data size significantly smaller than of the original data. Fig. 2 shows the motivation that aim to identify winning ticket on the dataset dimension. Here, we formally state the dataset lottery ticket hypothesis.
dataset con- tains subset that has the same similar empirical behaviors and performance trends the original full dataset when performing different training approaches and hyper-parameters on
Definition 1 (Dataset Lottery Ticket Hypothesis). large-scale, naturally-colleaa
Formally, in our Dataset Lottery Ticket Hypothesis setting, we have the learned risk function h* (such as MAE (He et al., 2022), DINO (Caron et al., 2021), SwAV (Caron et al., 2020), etc.) together with their parameters 0* (trained models) from the full training data, our goal is identify a subset of training data D. = {( y+))eT as a winning ticket that the ER is minimal and consistent to different risk functions. We can leverage the subset data's empirical distribution P,(X, Y) as the available empirical risk:
1 Σ ch" T
RDr (h)
(4)
FE
Where T is the sample index of dataset winning ticket. Different from the usual learning goal that is to find an optimal function h" identifying the winning ticket the dataset is ill-formed problem
4

---

Under review as a conference paper at ICLR 2023
100
Prebing
Winning
DIVO B/26 avg: 78.2% avg:
50
MAE 8/16 aug: 67 .8%.
400
1000
Class
Winning Ticket Class ID
Linear Probing
DINO B/8 avg: 80.1%
DINO B/ avg: 78.2%
McCo-V3 8/16 avg: 76.7%
MAE B/16 avg: 67.
200
400
600
800
1000
Class ID
Linaar Prabing Acc.
100
Winning Ticket Class
avg: 80.1%
DINO 8/1 avg: 78.2%
50
McCo-V3 B/16 76.7%
SwAy ResNett50 aeg: 75.
MAE B/26 avg: 67.8%6
0 200 400
600
BDD 1000
Class ID
Figure 3: Illustration of per-class Top-1 accuracy of different trained models on full ImageNet-Ik K validation set. In each subfigure, the black lines above the bubbles are the winning classes. From top to bottom subfigures, there are 367, 155 and 44 classes with particular distribution density that meet the conditions of accuracy trend on different frameworks. The specific names of targeted class IDs are provided in Appendix. More details are in Sec. 3.1.
with uncertainty and randomness, our idea is to make inevitable by considering that the individual selected class (ticket) should have the consistent performance behaviors as it on the full data. Note that the pre-trained parameters are always fixed during this procedure. Thus, we identify the targeting class according to the consistency the individual class empirical risk, we can find out the winning classes by maximizing the consistency metric which is defined as follows:
D = arg max Ceons([R (hppmmp(h)))ee (hī))SPP)
(5)
D,ED
where Crons is the consistency policies for discovering dataset winning tickets. P is the set of hypothesis functions, i.e., the learning methods. Eq. 5 indicates that when given the learned models (hypothesis functions and their parameters) hypothesis is to find subset D, as the winning ticket that meets the consistency relationship among different hypothesis functions. So that tuning hyperparameters. We also examined the generalizability of the winning ticket for assessing new learning functions.
the observed full dataset D, the dataset lottery ticket
use this subset for different usages, such as fast
Algorithm 1 Per-class Empirical Consistency
After acquiring the desired dataset winning ticket (subset), we can tune the hyperpa- rameters and re-train the model following:
Input: {h₂ P}} is the pre-trained model pool with linear probing classifiers to identify the desired classes based on the consistency of per-class empiri- cal accuracy, Cs the number of classes in the dataset, S. is the validation samples in class c. T''s is the thresh- old if the identified size of winning ticket is too large. Marget is the desired size of winning ticket after a ran- dom selection operation.
min L(O, D2) min Σ (E.D)ED
(h(2,,0)2) y)
0
(6)
We also provide a brief explanation from the theory perspective in Appendix F.I.
Output: Selected winning ticket μr"
1: for c 1: do
WINNING TICKETS POLICIES
2: if C: (Eies, Remp (hy(pha 0))ljep) is True then
3.1 PER-CLASS
3:
μ.append(cc 4: end if
EMPIRICAL RISK CONSISTENCY
5: end for
In this section, introduce the policy of identifying the winning ticket by leverag- ing the per-class empirical risk consistency.
6: if len(u)> T then
7: μr Random Selection(j,, Murges
8: end if
9: return μ²
5

---

Under review as a conference paper at ICLR 2023
The procedure is follows:
(i) Train standard self-supe backbone models P on the original full data.
(ii) Train supervised linear probing classifiers finetune above trained backbones the full dataset. (iii) Calculate per-class prediction using the above classifiers and test the models on the validation set to match the result the full data across frameworks. This is to identify that the particular class has the same accuracy trend as that on the full data, i.c., consistency condition between per-class accuracy and global accuracy on different models.
Specifically, there exists a well learned model/hypothesis pool P, the criterion of consistency follows their empirical risk on the full dataset. For instance, we consider three self-supervised methods: DINO, MAE and MoCo V3 with ViT-B/16 as the backbone network under linear probing evaluation, the consistency metric Crous will be True only if meets the following condition (the higher of the model accuracy, the lower Remp):
Coons RRmmp (PADINONTEES) Remp (IMMCo V3vren)) <
Remp (HAAAFSTTSS))
For class C, if Ceous is True, will select it as one class of the winning ticket. We formulate it as:
Couns
Σ (ADINOOTsss (2:)) < Σ Remp(tosoooo ViNaaous (x:)) iES.
Σ SES.
(hMAARAAA (8))
sESr
The results on linear probing eval of the per-class empirical risk consistency shown in Fig. 3. In practice, pre-trained model pools: (1) PDINOOEE - RMoCo V3wmae RMAEETRSS as in Fig. 3 (upper sub- figure). The black lines above the bubbles the targeted winning classes, is observed that 367 classes meet the requirement in ImageNet- IK; (2) RDINOVERS RDINO vrane RMoCo V3
per-class validation accuracy as the consistency indicator. We examine three
RMA Rsw AV ReNNee RMAESTRO as in Fig. (bottom subfigure). The full lists of classes are provided in the Appendix.
Eamann F in Fig. (middle subfigure); (3) RDINO RDINO RMaCo V3yman.
3.2 UNIFORM SAMPLING
Acc. (Top-1)
90
DINO Base/16 Linear
Uniform sampling, a.k.a. random sam-
DINO Small/16 Linear
pling, is an ordinary baseline to gener- ate the subsets. Previous literature has been following this scheme generate the subset with a uniform probability density function, e.g., randomly sample N classes from the 1000 classes on ImageNet- IK. Fig. 4, exhibit the results of RS-ID from different self-supervised methods,
80
MoCoV2, Reweeeete_Lninneee Liner
MAE Base/16, Linear
MAE Base/16, Finetuning
70
10
RS ID
Figure 4: Illustration of accuracy comparison on differ- ent RS subsets. Each model trained by three trials on both pre-training and linear probing/finetuning stages.
and derive the following discoveries: (i) It can be observed that the discriminability of weak and strong backbones is absent on DINO, as illustrated in Fig. 4 of black curves. (ii) Some trends are inconsistent, for instance, from subset RS-3 to RS-4, the accuracy increases on DINO ViT-Base/ while decreases on MAE and MoCo. These unreliabilities are expected and inevitable since these subsets are randomly selected without any guidance or restriction.
3.3
PRIOR KNOWLEDGE
There are multiple prior knowledge that can utilized as the policies to find out the winning tickets:
Performance-Drives The performance of each class has significant influence on reflecting the difficulties classification, also showing the similarity measure on the latent feature space, so that it is crucial for leveraging the performance criterion of selecting the potential tickets. As shown in Fig. 5 (1), we use the pre-trained EfficientNet-L2 (Tan & Le, 2019) to sort the accuracy of per-class and select with four patterns: PD-Top (easiest) PD-Mi d, PD-Botton Bottom (hardest), and PD-Uni form. Some selected categories of images are shown in the ellipses of Fig. 5.
WordNet Hierarchy: In WordNet, the concepts are linked together in hierarchy. This makes it easy to navigate between concepts. For instance, given a concept like motorcar, we can find out the concepts that more specific, i.e., the immediate hyponyms. As shown in Fig. 5 (2), since the

---

Under review as a conference paper at ICLR 2023
100
Unifom Distribution
Acc. (Top-1)
80
root
PD-Top
PDD-Mid
PD Bottoott
60
animal
boat
fruit
40
250
400
600
860
1000
catamaran
canoe
wreck yawl
liner
Sorted Class
Performance Driven
WordNet Hierarchy
Figure 5:
Illustration of the policies for Performance Driven (left) and WordNet Hierarchy (right). We also visualize images in the selected classes to reflect the difficulty level of the selected subsets.
classes of ImageNet- KK are the leaf nodes the WordNet tree, we can merge leaf nodes that have common parent and repeat this operation until obtaining enough coarse classes the winning ticket.
Word-embedding Clustering: this strategy. we aim to obtain the word-level embeddings of ImageNet- 1K human-readable labels and perform unsupervised clustering method them. Firstly, a pre-trained CLIP model (Radford et al., 2021) with ViT-L/14 (Dosovitskiy et al., 2020) backbone" with the property of context-dependency is employed to represent semantic labels through word embedding vectors. Next, both k-medoids an kmeans clustering algorithms are tested group semantically similar labels in order reduce the overlapping. Finally, different quality label features are utilized as the candidates in different clusters, and then the high-ranked semantic labels picked out from all clusters form the winning ticket.
Semantic-embedding Clustering: Instead of using the word-level embeddings of ImageNet- K human-readable labels, we use the embedding by taking the average feature from pre-trained DINO (Caron et al., 2021) model of VIT-B/8 for all images of that class in the validation set. Other procedures are the same the above word-embedding clustering. This strategy also applicable the label is not available in the dataset.
Policies
Subset Pool Abbreviation
Random Sampling
[RS-ID} (IDE {0, 1,2,
Rys }
Empirical Risk Consistency
|ERC-DD] (IDE {0, 1, 2,.. kipne)
Performance Driven
PD- TD-Top
PD PD-Mid
7DD-Bttomm
PD Uni Uniifnif
WordNet Hierarchy
{WNH-ID)
ID} (IDE {0, 2........ 1
Word Embedding Clustering
WEC
Semantic Embedding Clustering
SEC
Number Image Reduction
(NiR-ID) (IDE 10,1,2..... k2ER))
Table 1:
Left column is different policies for identifying the desired dataset winning ticket. Right column is the abbreviations of the corresponding policies in the main text.
EXAMINING WINNING TICKETS ON VARIOUS ARCHITECTURES
We employ both Pearson Correlation Coefficient (PCC) (Pearson, 1895) (pp) and Spearman's is Rank Correlation (Spearman, 1961) (px) to examine the correlation of the performance between the full set and the selected winning ticket. The former metric focuses on the linear correlation of two input sets of variables, the latter reflects the monotonic correlation of them. Here, study the dataset lottery ticket hypothesis on different architectures and strategies used in practice. Specifically, we
Ptttps: / / thub. com ani shatha ye/ magenet- a mp1 e 1abel s/bl ob/master/ imagenet - 1immplll1. aaee a. json. *BERT (Kenton & Toutanova, 2019) model also tested in ablation experiments.

---

Under review as a conference paper at ICLR 2023
Training Data
Accuracy Trend
Ppr values (indicators)
70
Full Set (frome of reference)
Diso Bace/16 linear
60
DINO 5 5mallie ltnear
MoCoVZ MoCoV2 McCoV VoccoV--Restttee Resterto--aee Restttoo-2000p 2000p
MoCoV2-Reshhn ResRatto- dRatSo-200pp 2000p
MoCoVZ-Reshh Res Notso s00ep
MAE Basu Linear
MAE Basa/16 Finetuning
ImageNet-1100 (Tian et al., 2020)
178p
P4 0.821
P/
Ncc (10033
100
PD-Top
Ps
P4 0.821
80
PD-Uniform
M6 0.947
P5
(Cur winning ticket-1)
21 = 0.964
50
ERC
0.944
(Cur winning 2icket-2)
Px 0.964
Table Overview of accuracy trends and correlation values (higher better) across different frame- works. The first row represents the accuracy on full data, in other rows, each is corresponding to one subset. As limited by the pages, more subsets' results will be provided the Appendix.
consider the typical deep neural architectures of residual networks (He et al., 2016) and vision transformers (Dosovitskiy et al., 2020).
ResNet. We examine the dataset lottery ticket hypothesis applied to the typical ConvNet of ResNet-50 (He et al., 2016). We use MoCo VI, V2 as the training methods with 200 and 800 epochs. All of our training settings follow their default design, well as the following vision transformer models.
Vision Transformer. We assess the dataset lottery ticket hypothesis on VIT models. We use the ViT- Base and ViT-Small architectures with MAE and DINO frameworks. The MAE model is trained with 800 epochs and DINO ViT-Base, ViT-Small trained with 400, 800 epochs, respectively.
Results. As shown in Table 2, PD-Top indicates that we choose the best dest-performed classes accord- ing pre-trained model, i.e., the easiest categories. Our results show that this subset has the worst consistency on linear correlation. We can also see the first two subsets have the same Pa. which means that they enjoy the same property of ranked performance. Moreover, choosing the hardest classes (PD-Botton) om) has good correlations to the full data and is one of our winning tickets. The correlation among the framework configurations in Table uation to reflect the quality of the selected subset, as our goal of this paper is to identify the proper subset that has the same correlation or performance trend as on the full data. Thus, for some heavy and costly tasks like ablation study, architecture search, etc., it is more practical and efficient for us to employ the selected subset instead of the full data to explore the best configuration that we de- sired. The results of WEC, SEC and WNH similar to PD-Uni form so they omitted in Table and given in Appendix.
the most crucial indicator for eval-
4.1 CONSISTENCY
Consistency Data Augmentation. To assess the consistency of performance different data augmentations, we use MoCo V1 and V2 which only contain the difference on data augmentation. The results shown in Fig. (1), it is surprising to see that nearly all the subsets not sensitive
8

---

Under review as a conference paper at ICLR 2023
to the data augmentation factor, obtaining similar accuracy on MoCo V1 and V2, but generally still consistent on performance.
Consistency on Training Budgets. On the self-supervised task, it is known that more training budget can substantially achieve better performance. In this subsection, we examine whether this property still exists the dataset winning ticket. The results shown in Fig. 6 (2).
Consistency on Evaluation Strategies. To assess the consistency of performance on different eval- uation strategies, employ MAE with finetuning and linear probing evaluations. The results shown in Fig. 6 (3), can be observed that the performance on all subsets is consistent.
80
Acc. (Top-1)
Acc. (Top-1)
Acc. (Top-1) 60
80
80
60
MoCo V2
60
800 epochs
Fineturing Linear Probing
MoCo V1
0
7
Model ID
Model ID
Model ID
(1) Consistency on Data Augmentations
(2) Consistency on Training Budgets
(3) Consistency Evalustion Strategies
Figure 6: Illustration of the consistency conditions for Data Augmentation (left), Training Budgets (middle) and Evaluation Strategies (right). In each subfigure, the abscissa axis represents the used subsets: CMC subset; PD-Top; PD-Mid; PD-Bottom; PD-Uniform; RS-1; RS-2; RS-3; RS-4.
4.2
GENERALIZABILITY
NoCoVI-RessNett--Oo 20000
MoCovI-RssNetSD
MoCoVZ
DINo_Smalli2 llncar
DINO,
DINO Small/16__nnooh
MoCoV2 Ree--BeeNNe 2000
MoCoV2-ResMeesOoo
Lottory linear
MocoVV2-ReaNNettuc
MoCoV2-ResHeetS--5
MAE 26 Finetuning
MAE Beca/26 Finetuning
MAE Beca/18 Finetuning
MAE Base/26 Linear
MAE Base/16 Linear
MAE Base/16 Linear
Figure 7: Illustration of the polygon radar chart for random sampling, PD-Uniform and our lottery ticket on several indicator approaches. Our lottery ticket has the best ability across these indicators.
It is interesting to see the generalizability of the identified winning tickets on those training methods outside the selection policies. Our conducted experiments above actually involved this verification: The per-class empirical risk consistency does not employ MoCo VI and V2 but from our results in Table it seems the winning tickets can still handle them well. Since MoCo V2 performs heavier data augmentation and maintains the other training settings the same as VI, this result demonstrates that the winning ticket is still a good indicator and can reflect the performance trend on input dimen- sion when the transformations are changed. Besides the input transformations, i.e., MoCo V1 to V2, we also include two more factors: (i) batch size effect, and (ii) stochastic regularization enabled by DropPath (Huang et al., 2016). We study how these subtle changes could affect the generalizability on winning ticket of the self-supervised models, the results and discussions are our Appendix. To assess the generalizability beyond the visual datasets, further provide the preliminary results the AG's News Topic Classification Dataset (Zhang et al., 2015) with the comparison between the random subset and lottery ticker subset in Appendix I.
DISCUSSIONS AND FUTURE WORK
We have introduced dataset lottery ticket hypothesis (DLTH), an initial study to investigate whether there exists winning tickets (subsets) that can reflect the consistency on performance when perform- ing different self-supervised approaches and hyperparameters. Through extensive empirical evalu- ations, we conclude that the randomly generated subset is not qualified for indicating the training settings and configurations of full data. We proposed several policies by incorporating prior knowl- edge to find out the desired winning tickets. In our experiments, the following trends are consistent: (i) Random subsets are not guaranteed obtain consistent results. in most cases they unreliable. (ii) Prior knowledge (c.g.. the difficulties of classes) can substantially improve the quality of the selected winning tickets which makes the selected ticket more aligned the full data.

---

Under review as a conference paper at ICLR 2023
(iii) Two drawbacks of subsets noticed, even the winning tickets: (1) subsets sensitive to different backbones if the training frameworks are strong, such as DINO; (2) subsets are not much sensitive to data augmentations, such as the case from MoCo V1 to V2.
Limitations and Future Work. Currently, we only verified our hypothesis on the vision domain of ImageNet-II IK modality, do not investigate on more modalities such as video, text, etc. Considering that different disciplines may have different characteristics or properties, for the future work, we hope our hypothesis can be proven in other domains such as NLP and speech datasets.
POTENTIAL ETHICAL IMPACT
As machine learning algorithm has been well studied in data-intensive applications, such as the large-scale ImageNet-IK classification, but is often hampered when the training data is small, this work leveraging subset of data may have the following potential positive and negative impacts in the society. For the positive impacts, the proposed dataset lottery ticket scheme can significantly help save training time, computational resources and energy for tuning hyperparameters of heavy models, which is highly eco-friendly. However, for the negative impacts, as this work tackles this problem through involving prior knowledge for identifying dataset winning tickets, the model may be biased on the training subset if the full data is unbalanced. Thus, should be cautious the result of failure from the system which could cause unreliable conclusions, such as the unbalanced dataset like medical images and further was misleading in some particular domains.
REFERENCES
Jordan T Ash, Chicheng Zhang, Akshay Krishnamurthy, John Langford, and Alekh Agarwal. Deep batch active learning by diverse, uncertain gradient lower bounds. arXiv preprint arXiv:1906.03671, 2019. 17
James Bergstra, Rémi Bardenet, Yoshua Bengio, and Balázs Kégl. Algorithms for hyper- hyper-paramett parameter optimization. Advances in neural information processing systems, 24, 2011. 17
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In Advances in neural information processing systems, volume 33, pp. 1877 (901,2020) 17
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Advances in Neural Information Processing Systems, 33:9912-9924, 2020. 1, 4, 17
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In ICCV, 2021. 1, 3, 4, 7, 17, 18
George Cazenavette, Tongzhou Wang, Antonio Torralba, Alexei Efros, and Jun-Yan Zhu. Dataset distillation by matching training trajectories. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 4750-4759, 2022. 2
Tianlong Chen, Jonathan Frankle, Shiyu Chang, Sijia Liu, Yang Zhang, Zhangyang Wang, and Michael Carbin. The lottery ticket hypothesis for pre- rreitrained bert networks. Advances in neural information processing systems, 33: 5834-15846, 2020a. 17
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.042: 04297, 2020b. 3, 18
Zhengyang Chen, Sanyuan Chen, Yu Wu, Yao Qian, Chengyi Wang, Shujie Liu, Yanmin Qian, and Michael Zeng. Large-scale self-supervised speech representation learning for automatic speaker verification. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6147-6151. IEEE, 2022. 17
Gui Citovsky, Giulia DeSalvo, Claudio Gentile, Lazaros Karydas, Anand Rajagopalan, Afshin Ros- tamizadeh, and Sanjiv Kumar. Batch active learning at scale. Advances in Neural Information Processing Systems, 34:1 1933-11944, 2021. 17
10

---

Under review as a conference paper at ICLR 2023
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale scale hi- erarchical image database. In 2009 JEEE conference on computer vision and pattern recognition, pp. 248-255, Icee, 2009.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An im- age is worth 16x16 words: Transformers for image recognition at scale. In International Confer- ence on Learning Representations, 2020. 7.8
Alcksandr Ermolov, Aliaksandr Siarohin, Enver Sangineto, and Nicu Sebe. Whitening for self- supervised representation learning. In International Conference on Machine Learning, pp. 3015- 3024. PMLR, 2021. 2
Utku Evei, Trevor Gale, Jacob Menick, Pablo Samuel Castro, and Erich Elsen. Rigging the lottery: Making all tickets winners. In International Conference Machine Learning, pp. 2943-2952. PMLR, 2020, 17
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In ICLR, 2019. 17
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016. 8,8
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and paitern recognition, pp. 9729-97299 99339778, 2020. 3, 18, 19
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. CVPR, 2022. 1, 4, 17, 19, 20
Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:3451-3460, 2021. 17
Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q Weinberger. Deep networks with stochastic depth. In European conference on computer vision, pp. 646-661. Springer, 2016.
Frank Hutter, Lars Kotthoff, and Joaquin Vanschoren. Automated machine learning: methods, sys- tems, challenges. Springer Nature, 2019. 17
Yannis Kalantidis, Mert Bulent Sariyildiz, Noe Pion, Philippe Weinzaepfel, and Diane Larlus. Hard negative mixing for contrastive learning. Advances in Neural Information Processing Systems, 33:21798-21888 2020. 2
Jacob Devlin Ming-We Wei Chang Kenton and Lee Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings NAACL-ILLT, 4186, 2019. 1, 7, 17
pp. 4171-
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.
Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner. Gradient- learning applied to document recog- nition. Proceedings of the IEEE, 96(11):2278-2) 2324. 1998. doi: 10. 109/5.726791. 1
Yann Lecun and Yoshua Bengio. Convolutional networks for images, speech, and time-series. In The handbook of brain theory and neural networks. MIT Press, 1995.
Andy T Liu, Shang-Weee Wen Li, and Hung-yi Lee. Tera: Self-supervised learning of transformer encoder representation for speech. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:2351-2366, 2021. 17
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: /907. 1692, 2019. 17

---

Under review as a conference paper at ICLR 2023
Baharan Mirzasoleiman, Jeff Bilmes, and Jure Leskovec. Coresets for data-efficient training of machine learning models. In International Conference on Machine Learning, pp. 6950-6960. PMLR. 2020. 17
Ari Morcos, Haonan Yu, Michela Paganini, and Yuandong Tian. One ticket to win them all: general- izing lottery ticket initializations across datasets and optimizers. Advances in neural information processing systems, 32, 2019. 17
Karl Pearson. Vii. on regression and inheritance in the case of two parents. proceedings of the royal society of London, 58(347-352);240-244 1895. 7
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning.
pp. 8748-8763. PMLR, 2021. 17
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alee Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International Conference Machine Learning, pp. 8821-8831. PMLR, 2021. 17
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06/15, 2022. 17
Pedro Savarese, Hugo Silva, and Michael Maire. Winning the lottery with continuous sparsification. Advances in Neural Information Processing Systems, 33:1 1380-11390, 2020.
17
Burr Settles, Active learning literature survey. 2009. 17
Jasper Snoek, Hugo Larochelle, and Ryan P Adams. Practical bayesian optimization of machine learning algorithms. Advances in neural information processing systems, 25, 2012.
Ben Sorscher, Robert Geirhos, Shashank Shekhar, Surya Ganguli, and Ari S Morcos. Beyond neural scaling laws: beating power law scaling via data pruning. arXiv preprint arXiv:2206. /4486, 2022.
2. 16, 17
Charles Spearman. The proof and measurement of association between two things. 1961.
Mingxing Tan and Quoe Le. Efficientnet: Rethinking model scaling for convolutional neural net- 6114. 14. PMLR, 2019. 1,6
works. In International conference on machine learning, pp. 6105-6114
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. In European
conference on computer vision, pp. 776-794. Springer, 2020. 2, 3,
Vladimir Vapnik. Principles of risk minimization for learning theory. Advances in neural informa- tion processing systems, 4, 1991. 2
Vladimir Vapnik. The nature of statistical learning theory. Springer science & business media, 1999. 2
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- lion processing systems, 30, 2017.
Chaoqi Wang, Guodong Zhang, and Roger Grosse. Picking winning tickets before training by preserving gradient flow. arXiv preprint arXiv:2002.07376 2020. 17
Tongzhou Wang, Jun-Yan Zhu, Antonio Torralba, and Alexei A Efros. Dataset distillation. arXiv
preprint arXiv: 2018. 2
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le.
Xlnet: Generalized autoregressive pretraining for language understanding. Advances in neural
information processing systems, 32, 2019. 17
12

---

Under review as a conference paper at ICLR 2023
Haoran You, Chaojian Li, Pengfei Xu, Yonggan Fu, Yue Wang, Xiaohan Chen, Richard G Baraniuk, Zhangyang Wang, and Yingyan Lin. Drawing early-bird tickets: Towards more efficient training of deep networks. arXiv preprint arXiv:1909.1199 2019. 17
Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text clas- sification. Advances in neural information processing systems, 28, 2015. 9, 19
Zhenyu Zhang, Xuxi Chen, Tianlong Chen, and Zhangyang Wang. Efficient lottery ticket finding: Less data more. International Conference on Machine Learning, pp. 12380-12390. PMLR, 2021. 2, 17
Bo Zhao, Konda Reddy Mopuri, and Hakan Bilen. Dataset condensation with gradient matching. arXiv preprint arXiv:2006.05922 059229 2020. 2
Hattie Zhou, Janice Lan, Rosanne Liu, and Jason Yosinski. Deconstructing lottery tickets: Zeros, signs, and the supermask. Advances in neural information processing systems, 32, 2019. 17
13

---

Under review as a conference paper at ICLR 2023
APPENDIX
In the appendix, we provide details omitted in the main text, including:
Section A: Exploration experiments of more classes or more samples in per-class. (Section "Examining Winning Tickets Various Architectures" of the main paper.)
Section B: Results of dataset lottery ticket hypothesis on supervised learning. (Section "Exam- ining Winning Tickets Various Architectures" of the main paper.)
Section C: An introduction of experimental settings. (Section "Examining Winning Tickets on Various Architectures" of the main paper.)
Section D: Full lists of winning tickets on ImageNet- IK. (Section 3 "Winning Tickets Policies" of the main paper.)
Section E: More results of different subsets. (Table 2 "Overview of accuracy trends and correlation values." of the main paper.)
Section F: More results and discussions on the study of generalizability. (Section 4.2 "Generaliz- ability" of the main paper.)
Section G: Background of our work. (Section 3 "Winning Tickets Policies" of the main paper.)
Section H: Implementation details and experimental settings. (Section "Examining Winning Tickets on Various Architectures" of the main paper.)
Section I: Results AG's news topic classification dataset. (Section 4.2 "Generalizability" of the main paper.)
A MORE CLASSES OR MORE SAMPLES IN PER-CLASS?
Instead of selecting subset along with the class, also keep all categories and reduce the number images in each class (NiR). This strategy will make the sub- set more diverse but the same category has fewer training samples. In this section, we examine the influence of number of im- ages in each class for the self-supervised learning. To study this, we keep the set of original classes and randomly select 1/10 of the images in each class. We illustrate the loss evolution of self-supervised pre-training and linear probing in Fig. 8. It can be observed that the strategy of Nip has low accuracy of 38% on MoCo V2 and 46% on DINO (we use Full Set and the low absolute performance subset PD Bottom as comparisons.). Though it is not necessary the perfor- mance is low since we focus more on the trend across the frameworks, the model is concerned to be "dull" and may be biased to some particular classes that contain few samples since samples in each class are insufficient.
ACC. mmp 12
[Full Seti DINO, D Small/16 jinee/ Jinear
60
(PD-8cttom) DINO, Small/1 linear
(PD-8ottom) McCoV2, ResNet5S
(NIR) DINO Small/16, linear
(NER) MaCoV2_FesNet50 Jinear
100
Epoch
Figure 8: Comparison of linear probing accuracy on Full Set, PD Bot om and NiR subsets.
B DATASET LOTTERY TICKET HYPOTHESIS ON SUPERVISED LEARNING
We examine the dataset lottery ticket hypothesis on supervised learning with the following net- works: ResNet-50, ResNeXt50 (32x (d), RegNet_ Y_ Y_a__33__000 3_F.. DenseNet121, MobileNet. V3 ConvNext_Small on the subsets NiR, PD-Top, PD PD-Mid, DD-Bottom, PD-Uniforms ERC and RS. We did not use ViT here as it is hungry for data in supervised learning. The results are shown in Table 3, which are aligned with those on self-supervised learning task. The slight dif- ference observe is that supervised learning, PD-Uni form performs much better than it self-supervised learning. The common conclusion is that the subset from the dimension of sample's number N R and random sampling subset RS are mediocre with weak alignment to the full data.
14

---

Under review as a conference paper at ICLR 2023
Training
ResNet-90
ResNeXit0,,7X44
RegNes_Y_ .__2GF
Accuracy (Top-1)
DemeNNe121
MochlleNet.11J
Con/Nest_S
Pr-
318
51.12
$2.34
53.12
$2.15
15.71
29.17
0.87.
0.856
81.04
85.36
85.52
84.84
79,94
81.27
0 0.922,
0.886
P
20-Mid
83.53
K5.12
86.97
86.14
83.73
83.11
A -0885, A. 0.943
PD-Bett.on
62.29
61.84
64.71
65.01
60.21
00.00
0.911, 0.829
P3-Dn1
84.51
86.53
83.19
0.958, 0.943
BRC-100
84.79
85.81
89.31
88.32
83.75
84.65
A P.- 1000
FRC-155
82.37
85.19
87.53
86.21
80.75
82.40
0.971. J. 0.943
Table 3: Accuracy trends and correlation values (higher better) on supervised learning across different backbone networks.
C MORE DETAILS OF SUBSET GENERATION EXPERIMENTS
Considering that if the size of the subset is too large, the subset will lose its role and advantage of fast hyperparameter searching.
this end, in the policy of per-class empirical consistency. T is set to 200 and uaget is set to 100. It means that if the size of selected subset is large than 200, we will randomly choose 100 classes among them as the new subset, so that we can guarantee the final scales of subsets will not be larger than T. Otherwise, the subset will keep the originally identified classes
D FULL NAMES OF SUBSET LISTS ON IMAGENET-IK
D.1 PD-BOTTOM (WINNING TICKET 1)
n01 667778
n01693334
n01729977
n01740131
n01744401
n01753488
n01756291
n01773549
nC1773797
n01775062
n02088466
n02089973
n02096294
n02106030
n02107908
n02109961
n02110185
n02114712
n02119022
n02123045
n02123159
n02124075
n02167151
n02229544
n02395406
n02403003
nC2412080
n02415577
n02441942
n02443114
n02443484
n02493793
nC2497673
n02669723
n02749479
n02769748
n02776631
n02808440
nC2895154
n02974003
n02979186
n02988304
n02999410
n03016953
n03D45698
n03125729
n03146219
n03179701
n03249569
n03461385
nC3485407
n03642806
n03657121
no3658185
n03710637
n03710721
nC3770679
n03773504
n03782006
n03787032
n03793489
n03832673
nC3865082
n03871628
n03895866
R03950228
n03976657
n04008634
no4081281
n04152593
n04239074
nO4264628
n04285008
n04286575
nC4355933
n04356056
n04357314
nO4380533
n04392985
n04428191
nC4443257
n04493381
n04525038
n04557648
n04560804
no4589890
no4591157
nO4592741
n04599235
n07579787
n07584110
n07734744
nC7860988
n07892512
n07930864
n09332890
n09399592
n09428293
m1214458C
n13133613
D.2 ERC (WTNNTNG TICKET 2))
n01484850
n01498041
nC1580077
n01601694
n01608432
n01622779
n01669191
n01685808
nC1688243
n01694178
n01698640
n01728920
n01755581
n01770393
nC1824575
n01829413
n01855032 n01873310
n01914609
n01978287
nC1981276
n01983481
n01985128
n01990800
n02002556
n02007558
nC2012849
n02056570
n02074367
n02077923
102088238
n02090721
nC2091032
n02091467
n02093859
n02096437
n02097209
n02098105
nC2099849
n02100583
n02101006
n02108089
n02110806
n02111277
nC2111500
n02111889
n02114548
n02128385
n02177972
n02190166
n02346627
n02356798
n02363005
n02444B19
n02445715
n02481823
n02484975
n02486261
n02641379
n02690373
n02786058
n02797295
nC2799071
n02804414
n02807133
n02823428
n02843684
n02865351
nC2892201
n02948072
n02977058
n02992211
103000684
n03018349
n03062245
n03110669
n03131574
n03207941
103388183
n03394916
no3445924
n03447721
n03450230
n03459775
n03482405
n03538406
nC3584829
n03595614
n03598930
no3649909
15

---

Under review as a conference paper at ICLR 2023
n03697007
n03764736
n03786901
n03873416
n03876231
n03877472
103877845
n03888605
n03891251
n03903868
n03929855
n03935335
n03938244
n03967562
n03977966
n03998194
n04005630
n04033901
n04039381
n04041544
n04099969
n04153751
n04209133
n04238763
n04251144
n04273569
nC4277352
n04311174
n04336792
n04355338
n04367480
n04371774
n04417672
n04418357
n04442312
n04465501
n04467665
n04482393
nC4485082
n04487394
n04509417
n04515003
n04517823
n04523525
n04532106
n04542943
n04553703
n04554684
n04579432
n04589890
nC4597913
n04604644
n07565083
407615774
n07693725
n07697537
n07717556
n07718472
n07753275
n07873807
n07875152
n09229709
m12620546
m12768682
m13052670
E MORE RESULTS
More results of accuracy trends and correlation values are shown in Fig. 4.
Training Data
Accuracy Trend
Ppr Ps values
Acc. (Top-1)
90
Pp 0.912
PD-Mid
80
P. 0.821
70
Acc. [Top- 11
Pp 0.916
WNH
80
P. 0.857
70
Acc. (Tpp-1)
90
Pp 0.877
WEC
80
P. 0.857
70
Acc. (Top-1) 1)
90
Pp 0.914
SEC
80
P. 0.929
-70
Table 4: More accuracy trends and correlation values (higher is better) across different frameworks.
F MORE ANALYSES ON GENERALIZABILITY
Settings: (i) Batch size effect. this factor, we employ MAE MeFF_Bas//16__neeeee as the baseline
and examine the consistency between the winning ticket and full data. The default batch size in MAE framework is 4,096, we test 2,048 and 1,024 respectively for studying the generalizability of our selected winning ticket. (ii) Stochastic regularization with DropPath. On this factor, employ DINO Small/16 linear as the baseline, the default DropPath ratio DINO framework is 0.1, we test additional 0.3, 0.5, respectively. Our results indicate that the selected subsets have good generalizability on slightly different training settings.
F.1 A BRIEF THEORY ANALYSIS DATASET LOTTERY TICKET HYPOTHESIS
Inspired by (Sorscher et al., 2022), we also use a teacher-student perceptron scheme to explain our DLTH theoretically. We consider select the subset training dataset (the subset has P examples with {x"" {x".,") y²) |==l....PP P., where x" ~ / (0, Ix)) by keeping only the examples with the smallest margin |2-|| Speobe xx| along probe student Spobe which matches one of proposed selection policies. The selected subset will follow the distribution p(z) along the direction of S Sprohe Similarly, obtain the generalized theory for an arbitrary data distribution p(z) for the small-margin
16

---

Under review as a conference paper at ICLR 2023
selection strategy.
Spobe contains overlap with the teacher that is determined by the angle 0
cos-1 Synte T S 2lla /Tla
After the subset has been selected, we train a new student S from scratch on the selected subset. The typical learning algorithm is to train a classifier for the training data with the maximal margin A min, S. (yy"x") to find the optimized S. Different from data pruning of computing the generalization error Eg of this student, which is governed by the overlap between the student and teacher, Eg
cos /O)) where R S T/| | 2- we focus on the consistency of test error es across different configurations for the final perceptron as a function.
G BACKGROUND
Self-supervised Learning. Self-supervised pre-training has been recognized as a promising tech- nique in various fields, such as Computer Vision (Caron et al., 2021; 2020; He et al., 2022), NLP (Kenton & Toutanova, 2019; Yang et al., 2019; Liu et al., 2019; Brown et al., 2020), Speech (Hsu et al., 2021; Liu et al., 2021; Chen et al., 2022) and Multimodality (Radford et al., 2021: Ramesh et al., 2021: 2022). The major drawback of this technique is the costly training overhead, which dramatically impedes more investigation and exploration. For instance, GPT-3 model (Brown et al., 2020) contains 175 billion parameters and is trained roughly requiring 1,024 A100 GPUs for more than month, the estimated training cost of it is at least $4.6 million. In the vision domain, DINO ViT-B/8 (Caron et al., 2021) is also trained on 176 GPUs which is inaccessible for most research institutions.
Efficient Hyperparameter Tuning, The most common hyperparameter tuning method is the grid search. While it is always in low-efficiency when the number of hyperparameters is large, as well as the big training data and model size. Random search with human experience and Bayesian optimiza- tion (Bergstra et al., 201 I; Snoek et al., 2012; Hutter et al., 2019) are two popular fast hyperparameter tuning methods and have been successfully applied to many applications in practice. In this study, focus finding proper subset (winning ticket) from the full data, which is orthogonal to these black-box strategies and can be utilized with them simultaneously. Moreover, a suitable subset is more efficient if the model's training cost is high.
Model-level Lottery Ticket Hypothesis. The original lottery ticket hypothesis (Frankle & Carbin, 2019) and its follow-ups (Zhou et al., 2019; Evci et al., 2020; Wang et al., 2020; Chen et al., 2020a; Morcos et al., 2019; You et al., 2019; Savarese et al., 2020) are proposed on the neural network dimension as the model pruning technique. The core of this hypothesis is that randomly-initialzze neural network contains subnetwork that can match the test accuracy of the original model after training for the same number of iterations. PrAC (Zhang et al., 2021) observed that a sparse model can be explored with training and pruning the dense network on the compact PrAC set, which has a different goal from the proposed DLTH. In this work, we study the lottery ticket hypothesis on the datatset dimension to reflect the consistency of performance trend, and propose several practice ways to identify the dataset winning tickets.
Active Learning and Dataset Pruning. Active learning (Settles, 2009) is similar the semi- supervised learning strategy by involving the human participation, and the process of selecting suit- able candidate sets for manual labeling through machine learning algorithms. The idea of active learning is to obtain the sample that is more difficult to classify through ML approach, then, let human reconfirm and review them. After that, we can re-use the manually labeled data with a su- pervised learning or semi-supervised learning model. There are many recent literature following this interesting paradigm (Ash et al., 2019; Mirzasoleiman et al., 2020; Citovsky et al., 2021). In contrast, DLTH focuses on selecting a subset with consistent performance trends as the full data. For the dataset pruning, Sorscher et al. (Sorscher et al., 2022) focused on the scaling of testing error with dataset size and proposed a self-supervised pruning metric to demonstrate comparable results to the supervised metrics. Different from the dataset pruning setting, in our scenario, we do not care about the absolute accuracy of the individual model trained the subset but the consistency to the models trained on full data. The two tasks essentially have different goals, and our pruning ratio on the original dataset is also much higher than dataset pruning.
17

---

Under review as a conference paper at ICLR 2023
H IMPLEMENTATION DETAILS AND EXPERIMENTAL SETTINGS
H.1 DINO (CARON ET AL., 2021) PRETRAINING AND LINEAR PROBING
We conduct experiments two configurations of DINO framework (Caron et al., 2021): (I)
/inerr and (II) DINO-Smai Small/mllaaia / 1a1.11111iinaaa liniir. The pre-training settings are provided in Table 5. The linear probing settings are in Table 6.
arch. optimizer patch size out dim form_ast_layee
vit_smal
base
AdamW
Adam
16
16
65,536
65,536
false
true
cormup_teacher.cc
0.04
0.04
teacher_ temp
0.07
0.07
30
50
momentum _eaccher
0.996
0.996
use use_bn in head drop-path_rate usc_fpl6 weight_decay
false
false
0.1
0.1
false
false
0.04
0.04
weight decay end
0.4
0.4
clip grad batch_size epochs freeze_last.layer Ir warmup-epode -epochs min Ir global crops scale local local_crops _cropscopooooe ssops_scale lecaa_crops_uummie seed
0.0
0.3
1024
1024
800
400
1
3
0.0005
0.00075
10
10
2c-05
2c-06
[0.25, 1.0]
[0.25, 1.0]
[0.05, 0.25]
[0.05, 0.25]
10
10
0
0
Table 5: DINO (Caron et al., 2021) pre-training setting subsets.
arch. patch_size n last blocks
vit small
vit base
16
16
4
avgpool patchtokens
false teacher
true
checkpoint.key
teacher
epochs Ir batch_size optimizer weight decay optimizer momentum
100
100
0,001
0.001
1,024
1,024
SGD
SGD
0.0
0.0
0.9
0.9
Barninggrate_scheede
cosine decay
cosine decay
Table 6: DINO (Caron et al., 2021) linear probing setting on subsets.
H.2 MoCo (HE ET AL., 2020) PRETRAINING AND LINEAR PROBING
We conduct experiments with three configrations on MoCo V1&V2 (He et al., 2020; Chen et al., 2020b) frameworks: (I) -ow.ooo-eeeeeee5-222000000 (III) 5022-eeeeett--0000 provided in Table and Table 8, respectively.
50-20eee and The details of pre-training and linear probing settings are
(II)
18

---

Under review as a conference paper at ICLR 2023
config. arch. bate
C1
C3
ResNet50
ResNet50
ResNet50
256
256
256
epochs
200
200
800
Ir
0.03
0.03
0.03
learning rate schedule
cosine decay SGD
cosine decay
cosine decay
optimizer
SGD
SGD
momentum
0.9
0.9
0.9
weight.decos
le-4
le-4
le-4
moco-dim
128
128
128
moco-k
65,536
65,536
65,536
moco-m
0.999
0.999
0.999
moco-t
0.07
0.2
0.2
mip_head
false
true
true
aug-plus
false
true
true
Table 7: MoCo (He al., 2020) pre-training setting on subsets.
config.
C1
C2
C3
arch.
ResNer50
ResNet50
ResNet50
batch_size
256
256
256
cpochs Ir
100
100
100
30
30
30
learning rate schedule
[60, 80]
[60, 80]
[60, 80]
optimizer
SGD
SGD
SGD
momentum
0.9
0.9
0.9
weight_decay augmentation
0
RandomResizedCrop
RandomResizedCrrpp
RandomRessizedCrop
Table 8: MoCo (He et al., 2020) linear probing setting subsets.
H.3 MAE (HE AL., 2022) PRETRAINING, FINETUNING AND LINEAR PROBING
We conduct experiments on MAE (He et al., 2022) with one pretraining configuration and two evaluation protocols: (I) MAE Base 6 Linear and (II) MAE Base/16 F Pinetuning
The setting used for our experiments is in Table 9. Following the default protocal, we do not use color jittering, drop path, or gradient clip. We also use the linear scaling rule: Ir base Iree_rxbba batchssse 256. End-to-end fine-tuning: The setting used in experiments is provided Table 10. Linear probing: Our linear classifier training setting is provided Table 11.
Pre-training:
arch. optimizer base learning rate weight decay optimizer momentum batch_ size learning rrteinnngrrtt...e sccheulle warmup_epoore augmentation
vit_ base
AdamW
1.5e-4
0.05
B1, 92=0.9 0.95 4,096
cosine decay
RandomResizedCrop
Table MAE (He et al., 2022) pre-training setting on subsets.
I RESULTS ON AG'S NEWS TOPIC CLASSIFICATION DATASET
Training and Evaluation Settings. To further examine the generalizability of the proposed DLTH, we extend our method to the NLP domain to see whether this hypothesis still holds using AG's news topic classification dataset (Zhang et al., 2015). This dataset is widely used as text classification benchmark. consists 4 coarse classes from the original corpus. Each class contains 30K training samples and 1. 1.9K testing samples. The total number of training samples is 120K and testing 7.6K.
19

---

Under review as a conference paper at ICLR 2023
arch.
vit base
optimizer
Adam W
base Jearninggratee
le-3
0.05
optimizer momentum layer-wise_Ir _r_cc batch_size
coamiizer_momentum
31, -099, 0.999
0.75
1,024
learning rate schedule
cosine decay
warmup epochs
training_epoche
100
augmentation
RandAug (9, 0.5)
labol_smoothing mixup cutmix drop path
0.1
0.8
1.0
0.1
Table 10: MAE (He et al., 2022) end-to-end fine-tuning setting subset.
arch. optimizer base learningg rate weight_decar
vit base
LARS
0.1
pppimizer_momentun
batch_size
16,384 cosine decay
learning rate schedule warmup epochs training_epocte augmentation
10
90
RandomResizeeCCrp
Table 11: MAE (He et al., 2022) linear probing setting on subsets.
Configurations and Results. We explore four configurations as shown in Table 12. The results are provided in Table 13. For the lottery ticket subset, simply select consistent samples across four configurations until reaching 10% of the full data. We do this twice to construct different lottery ticket subsets mitigate variance in different runs. We observe that the models trained on lottery ticket subsets are much more robust with higher correlation values to the full data than the modles trained on the random subset baselines.
config. batch size epochs Ir learning_ratee scheuue optimizer
C1
C2
C3
C4
64
64
128
64
10
10
10
10
5
10
StepLR SGD
CosincAmealingLLR
StepLR SGD
StepLR SGD
SGD
Table 12: Training configurations on AG's news topic classification dataset.
Accuracy Trend
Training Data
Pp value
C1
C2
C3
C4
Full
0.910
0.905
0.896
0.909
RS-0
0.864
0.862
0.862
0.863
0.764
RS-1
0.863
0.867
0.863
0.871
0.437
RS-2
0.863
0.870
0.859
0.868
0.589
RS-3
0.870
0.868
0.863
0.864
0.617
RS-4
0.861
0.862
0.860
0.867
0.555
Lottery Ticket-0
0.871
0.864
0.863
0.867
0.814
Lottery Ticket-1
0.870
0.865
0.864
0.871
0.863
Table 13: Accuracy trends and correlation values on AG's news classification dataset.
20


Reviews:
[{"review_id": "oJvJ0SRvcBH", "reviewer": "Reviewer_kkgk", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper generalizes Lottery Tickets Hypothesis (LTH) to the subset selection domain, by defining a Dataset Lottery Ticket as a subset that has the same or similar empirical behaviors and performance trends as the original full dataset, which can be identified by some specific approaches (e.g., WordNet Hierarchy). It provides a novel problem that studies the possibility of identifying the subset which can reflect the performance consistency with the full data.", "summary_of_the_review": "Considering the pros and cons of this paper, I think this paper is good but still has a large space to improve, e.g., more datasets for evaluation.", "strength_and_weaknesses": "Pros:\n1. The paper is well-written and easy to follow. \n2. This paper provides a novel perspective on sub-dataset selection.\n3. This paper empirically reveals that a randomly selected subset without any prior knowledge is unstable and generally not qualified for reflecting the properties of self-supervised models on the full data. This helps us to use a more reasonable way to screen the important samples of the dataset without misconceptions.\n\nCons:\n1. My major concern is why should we guarantee that the subset categories have consistency in performance similar to that of the full dataset. This is the key contribution/target of this manuscript, and the authors provide an example to explain this issue. However, I am still not sure why we should not select these inconsistent classes in the classification task. For example, can we select some samples which have similar empirical behaviors and performance trends in these classes rather than removing these classes? Dataset Lottery Ticket Hypothesis (DLTH) that recklessly retains all the samples in consistent classes seems to contradict the target of accelerated training.\n2. It seems that only a few establishments (e.g., Google, Meta, etc.) can afford the heavy experiments on large-scale datasets training. This issue is caused by large-scale datasets and the over-parameterized networks that go with them. This paper follows a train-select-retrain process, which is hard to reduce the loss of training resources caused by hyper-parameter adjustments (the full dataset has to be fed to the network for training and testing at the beginning). If it is a misunderstanding, please feel free to correct it.\n3. This manuscript utilizes empirical risk minimization for evaluating model performances. However, in the case of such a dataset with a large network, serious overfitting often occurs after the dataset is screened. It seems more reasonable to use structure risk minimization to overcome the overfitting problem (e.g., through weight decay) and adjustment of the penalty factor should be a consideration.\n4. This manuscript finds the LTH property on datasets, but only verifies this hypothesis on the vision domain, i.e., ImageNet-1K. This is not very convincing although the paper adopts a variety of recent mainstream backbones and baselines. More datasets are appreciated for conducting experiments.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "lxhLqTUnl_6", "reviewer": "Reviewer_gNgq", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces the data lottery ticket hypothesis, which intends to select a subset that matches the performance of the original dataset. This hypothesis, if it works, can reduce training costs and democratize self-supervised pretraining. The authors first suggest that a good data lottery ticket subset should match the original performance in training, generalization, and model consistency. Based on these principles, the authors question the random sampling approach and propose the policies of Empirical Risk Trend and incorporate prior knowledge to generate the winning tickets. The analysis is demonstrated with a suite of self-supervised experiments with various structures (DINO, MAE, MOCOV). \n\n", "summary_of_the_review": "I lean to reject this paper mainly because (1) the consistency principle is not well justified and could be much influenced by subset size and model structures, (2) the algorithm is heavily dependent on class labels and semantics, which may not be available in many datasets, including most NLP tasks where self-training is most heavily used (3) the overall presentation of the paper can be more organized and polished, (4) the reproducing details and the codes for the experiments are missing.", "strength_and_weaknesses": "Strengths: \n1. Data lottery ticket is an important issue and investigating the lottery ticket problem from the view of training data is quite novel. Although data pruning often achieves the same effect as the data lottery ticket, data pruning has a different motivation that aims at removing biases and noises. \n\nWeaknesses:\n1. The definition of the data lottery ticket problem is ambiguous. The author stated in Definition 1 that \"a subset that has the same or similar empirical behaviors and performance trends as the original full dataset when performing different training approaches and hyperparameters on it\". \n-  It lacks an elaboration on \"performance trends\" which can be pre-trained performance, training error, generalization error, downstream performance, etc.\n-  The performance trends involve the selection of models. However, different pools of models may have different performance result trends. The author didn't give a clear definition as to how these training structures and training approaches should be chosen. \n- A formal definition could be given instead of a text description for better clarity. \n\n2. The motivation for choosing a lottery ticket with a consistency principle is ill-justified. For example, the authors stated in 3.2 that \" Some trends are inconsistent, for instance, from subset RS-3 to RS-4, the accuracy increases on DINO ViT-Base/16 while decreases on MAE and MoCo\" and use this to support their claim that uniform sampling lacks consistency. However, the models are tested via linear probing and the increase in model hyperparameters may result in overfitting or instability during fine-tuning and lead to these performance drops. Overall, this inconsistency may be the result of the difficult balancing of model size and subset size. It does not necessarily reflect the trend of the expressiveness of the models.\n\n3. I'm doubtful about the training settings. The authors use full-dataset trained backbones to calculate test accuracies and only test if the performance trends are consistent. In most applications, users are more concerned about the performance achieved by directly tuning their model on the subsets. The paper lacks a discussion on this more practical setting. It seems that section 4.2 discusses this subject but only approximate numbers are provided in the radar char and lack precise results. \n\n4. The class semantic-based methods for choosing lottery tickets are hard to apply beyond labeled image datasets. For example, many pre-trained models are using crawled images without labels. In these cases, the authors would not have the label semantics that is crucial for constructing lottery ticket subsets. \n\n5. The experiment is overall imprecise and missing important settings. \n- Most results are provided as visualization and charts, which are not precise enough. Eg., in Figure 4, the trend of DINO_base and DINO_small is plotted as line chart (performance vs set ID ), yet the scale for set ID is set to 2, making analyzing the trends between RS-3 and RS-4 difficult. Also, in Figure 7, no precise number is given and it is confusing what the scale \"2,4,6,8,10\" means. \n- I'm not convinced by the results that inconsistency is an issue. As shown in Table 2, the Pearson correlation reaches 0.893 using random sampling subsets. After using some simple subset selection procedures, this consistency rises to 0.944. \n- The training accuracy for direct learning on the subsets is missing, as stated in 3. \n- The reproducing details are missing. I can't find the code or the hyperparameters for these experiments. \n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "_T05JvcB6g", "reviewer": "Reviewer_PcqN", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work proposes the Dataset Lottery Ticket Hypothesis(DLTH), a novel problem that studies the possibility of identifying the subset which can reflect the performance consistency with the full data. By Empirical Risk Trend, this work demonstrates the existence of dataset-winning tickets. And extensive experiments are conducted across various self-supervised frameworks, which verify the effectiveness and superiority of the proposed dataset-winning ticket policies.", "summary_of_the_review": "None", "strength_and_weaknesses": "### Strength:\n\n1. This paper is well-written and easy to follow.\n2. The idea of studying the dataset lottery ticket hypothesis is novel and interesting.\n3. Comprehensive experiments on ImageNet-1K across several self-supervised frameworks are conducted, which is a laudable effort.\n\n### Weakness:\n\nI have one concern about this work: for the randomly sampling strategy, what's the performance of randomly sampling partial data points across all classes rather than selecting partial classes, which seems less biased from the full set distribution and serves as a more competitive baseline?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "RT0VskS5Ha", "reviewer": "Reviewer_drZn", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper inspects the dataset lottery ticket hypothesis, where training on just subsets have similar empirical behaviors and performance trends as training on the full set. So that analysis and hyper-parameter tuning can be conducted efficiently. Various sampling strategy with different models are compared, the proposed ERC strategy achieves the best consistency. ", "summary_of_the_review": "Interesting paper that tries to build a performance-consistent sub-training method. However, there are some issues mentioned in the weakness part. I would like to raise my score if these questions are properly addressed.", "strength_and_weaknesses": "Strength:\n1. The paper is well motivated, building a performance-consistent sub-training method which can be used as an efficient proxy for hyper-parameter tuning is interesting. \n\n2. The method is simple and easy to implement. \n\nWeakness:\n\n1. Talking about performance consistency, more insightful discussion or theoretical analysis may be required to fully justify the proposed method. Utilizing only 7 models is not enough for this claim. \n2. The name “dataset lottery ticket hypothesis” is a little misleading to me. At the first glance, the “dataset lottery ticket hypothesis” may refer to: there exists a subset that can fully retain the model performance.  This paper, however, concentrates on maintaining the relative performance of different models and a few hyper-parameters.\n\n3. Though the proposed ERC strategy is reported can well retain the “performance trends”. Picking samples at class level is quite coarse. There are many other fine-grained data selection methods, ranging from active learning to efficient model training. Though not primarily designed for maintaining “performance trends” consistency, comparing and discussing these methods will strengthen the paper. \n\n4. The rightmost 2 figures of Figure 2 seem to be exactly the same. There are also some typos like subsection 4.1 should refer Figure 6 instead of Figure 7. \n\n\nminor:\n1. It seems that “empirical behaviors and performance trends” refers only to the relative accuracy throughout the paper. I recommend outlining this earlier in the narration. \n\nReferences:\n[1] J T. Ash et.al. Deep Batch Active Learning by Diverse, Uncertain Gradient Lower Bounds. ICLR ‘20 \n\n[2] B. Mirzasoleiman et.al. Coresets for Data-efficient Training ofMachine Learning Models. ICML ‘20\n\n[3] G.Citovsky et.al. Batch Active Learning at Scale. NeurIPS ‘21 \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "1pqkOj-nNn", "reviewer": "Reviewer_xj8W", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper investigates the dataset lottery tickets hypothesis in the context of self-supervised learning. The main hypothesis is that one can identify a subset of the data which has empirical trend similar to the full dataset, i.e. hyperparameters that have better performance on the subset also show better performance on the full dataset.\n\nAuthors propose several methods to find dataset subsets using a SSL pretrain model and potential label information. Authors empirically investigate which approach can identify winning tickets on the ImageNet dataset.\n", "summary_of_the_review": "The paper investigates the question of dataset lottery tickets for self-supervised learning approach. While the topic is interesting and important, the paper clarity needs to be improved.\nIn particular, it is unclear if the models are re-trained on the dataset subset in the empirical evaluation. \nComparisons with dataset pruning approaches should also be added to better demonstrate the significance of the work.", "strength_and_weaknesses": "Strength:\n- The paper has sound motivation. SSL research involves a non-trivial computational cost, and it is important to reduce this compute burden to stimulate more research in this area.\n\nWeaknesses:\n- It is unclear if the SSL models are retrained on the ‘winning’ subset. If the models are not retrained, this would be a major limitation as model trained on the subset might not have empirical trends similar to the models trained on the full dataset.\n- Paper clarity could be improved. I found that the paper was hard to follow. For instance, some important details are missing in the ticket policies description (how do you select the number of classes in PD-{Top,Bottom,…), how do you select the merging order and stopping criteria in WNH-ID. Same question for WEC/SEC. The NIR-ID scheme is not explain in the main text. Figures are also hard to understand. For instance, the x-axis in Figure 6 is not explained. It would be nice to provide more details in the caption and explain the main take-away from the figures. \n- If the SSL model are retrained on the smaller subset, what are the computational gain provided by the approach.\n-Authors should compare with dataset pruning approaches as they might already find winning lottery tickets.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject