Efficiency for Free: Ideal Data Are Transportable Representations
Authors: Peng Sun
Yi Jiang
Tao Lin
Venue: NeurIPS 2024 poster (2024)
Keywords: Efficient Learning
Efficient Data
Ideal Data
Primary area: other

Abstract: Data, the seminal opportunity and challenge in modern machine learning, currently constrains the scalability of representation learning and impedes the pace of model evolution.
In this work, we investigate the efficiency properties of data from both optimization and generalization perspectives.
Our theoretical and empirical analysis reveals an unexpected finding: for a given task, utilizing a publicly available, task- and architecture-agnostic model (referred to as the `prior model' in this paper) can effectively produce efficient data.
Building on this insight, we propose the Representation Learning Accelerator (ReLA), which promotes the formation and utilization of efficient data, thereby accelerating representation learning.
Utilizing a ResNet-18 pre-trained on CIFAR-10 as a prior model to inform ResNet-50 training on ImageNet-1K reduces computational costs by $50\%$ while maintaining the same accuracy as the model trained with the original BYOL, which requires $100\%$ cost.
Our code is available at: \url{https://github.com/LINs-lab/ReLA}.

Efficiency for Free: Ideal Data Are
Transportable Representations
Peng Sun Yi Jian' Tao Lin
'Zhejiang University Westlake University
sunpengdwestlake edu. en, yi_jiangozju.. edu. en, lintaoOwestlake edu.
Abstract
Data, the seminal opportunity and challenge in modern machine learning, currently constrains the scalability of representation learning and impedes the pace of model evolution. In this work, we investigate the efficiency properties of data from both optimization and generalization perspectives. Our theoretical and empirical analysis reveals an unexpected finding: for a given task, utilizing publicly available, task- and anchitecture-agnosiic model (referred as the 'prior model' in this paper) can effectively produce efficient data. Building on this insight, we propose the Representation Learning Accelerator (RELA), which promotes the formation and utilization of efficient data, thereby accelerating representation learning. Utilizing a ResNet-18 18 pre-trained CIFAR-I0 as a prior model inform ResNet-50 training on ImageNet-11 K reduces computational costs by 50% while maintaining the same accuracy the model trained with the original BYOL, which requires 100% cost. Our code is available at: https: htpps://gtt thub. com/ Seto/LINs-lleee lablRallblRRaae
Introduction
Prior (up)
Representation Space
Download
Generate
Originall Ale.
Slow cirning
Original
Feel
Alg.
RELA+AIg
aast. Cearnin
Init. cp
0OO
Accelerated
RELA+AIg
Loss
Figure 1: Framework Intuition RELA: (1) Framework: RELA serves as both a data optimizer and an auxiliary accelerator. Initially, it operates as a data optimizer by leveraging an dataset and a pre-trained model (e.g.. one sourced from online repositories) to generate an efficient dataset. Subsequently. RELA functions as an auxiliary accelerator, enhancing existing (self-) self-vsuuee learning algorithms through the effective utilization of the efficient dataset, thereby promoting efficient representation learning. (2) Intuition: The central concept of RELA is to create an sificiett-dtt-diie model & to rapidly converge towards a 'proximal representation up of the target model Φ= during the early stages of training. This approach significantly accelerates the overall learning process.
shortcut pathway within the learning process, cnabling the initial
The available of massive datasets [20, 49] and recent advances in parallel data processing [28, 42] have facilitated the rapid evolution of large deep models, such as GPT-4 [1] and LVM [2]. These
"Corresooding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

models excel in numerous learning tasks, attributable to their impressive representation capabilities. However, the emergence of vast amounts of data within the modern deep learning paradigm raises two fundamental challenges: (i) the demand for human annotations of huge datasets consumes significant social resources [45, 20, 43]; (ii) training large models with increasing data and model capacity suffers from intensive computational burden [6, 16, 55).
The community has made considerable efforts to enhance learning efficiency. Self-supervised learning methods [12. 69, 30, 10, 25, 14, 9, 3], with their superior representation learning devoid of human annotations via the self-learning paradigm, attempt to tackle the challenge (i). Concurrently, research has been conducted to mitigate data efficiency issues in challenge (ii): dataset distillation approaches [65, 56, 11, 72. 53, 54] have successfully synthesized a small distilled dataset, on which models trained on this compact dataset can akin to one trained on the full dataset.
However, challenges (i) and (ii) persist and yet are far from being solved [43, 45, 6, 16, 55], partic- ularly the intervention of these two learning paradigms. In this paper, we identify two issues: (a) inefficiency in the self-supervised learning procedure compared to conventional supervised learning arises due to sub-optimal self-generating targets [30, 62]; (b) although training on the distilled dataset is efficient and effective, the distillation process of optimization-basee approaches [11, 72, 37] is computationally demanding [18, 56]. often surpassing the computational load of training on the full dataset. This limitation restricts its potential accelerate representation learning. To tackle these challenges, we propose a novel open problem in the domain of representation learning:
Problem 1 (Accelerating Representation Learning through Free Models) According to the No Free Lunch theorem [66], it is evident that accelerating the learning process without incorporating prior knowledge is inherently challenging. Fortunately, numerous publicly available pre-trained models can be accessed online, offering form of free prior knowledge. Despite this, effectively utilizing these models poses several implicit challenges, as these models may not be directly relevant to our target learning task, they may not be sufficiently well-trained. This leads to a question:
How can we leverage task- and architecture-agnosti publicly available models to accelerate representation learning for a specific task?
To address Problem 1. we propose RELA to utilize a freely available model downloaded from the internet generate efficient data for training. This approach aims to accelerate training during the initial stages by effectively leveraging these generated data, thereby establishing a rapid pathway for representation learning (see Figure ). Specifically, we list our five key contributions below the first step toward bridging representation learning with data-efficient learning:
(a) Revealing beneficial/detrimentts data properties for efficien//infffiieett jeeff-Jsuerriised learning (see Section 3.2) ). We present a comprehensive analysis of linear models, demonstrating that data properties significantly influence the learning process by impacting the optimization of model training. Our findings reveal that modifications to the data can markedly enhance or impair this optimization. Additionally, we indicate that optimal training necessitates specific data properties -perfect bijective mappings between the samples and targets within a dataset.
(b) Identifying the inefficiency problems of (self- jsupevised learning from a data-centric perspective (see Section 3.3). ). Specifically, we identify several factors contributing to the inefficiencies in (self- jsupervised learning over real-world data. For instance, prevalent data augmentation techniques in modern deep learning can introduce a 'noisy mapping" issue, which may exacerbate the negative effects associated with inefficient data.
(c) Generalization bound for models trained on optimized efficient data (see Section 3.4). Although the efficiency properties of data do not inherently ensure the generalization of the trained model, i.c.. efficient data alone cannot guarantee generalization ability, we present generalization bound to analyze models trained on such data.
(d) A novel method RELA generate and exploit efficient data (see Section 4 J. Leveraging our theoretical insights regarding the bounds of generalization and convergence rate, we introduce RELA, a novel optimization-free method tailored efficiently generate and effectively exploit efficient data for accelerating representation learning.
(e) An application of our RELA: accelerating (self-Jsuperviied learning (see Section 5 and Appendix l). Extensive experiments across four widely-used datasets, seven neural network ar-

---

chitectures, eight self-supervised learning algorithms demonstrate the effectiveness and efficiency of RELA. Training models with RELA significantly outperforms training on the original dataset with the same budget, and even exceeds the performance of training on higher budget.
2
Related Work
This section integrates two distinct deep learning areas: (a) techniques to condense datasets while pre- serving efficacy; (b) self-supervised learning methods that enable training models unlabeled data.
2.1 Dataset Distillation: Efficient yet Effective Learning Using Fewer Data
The objective of dataset distillation is to create a significantly smaller dataset that retains competitive performance relative to the original dataset.
Refining proxy metrics between original and distilled datasets. Traditional approaches involve replicating the behaviors of the original dataset within the distilled onc. These methods aim to minimize discrepancies between surrogate neural network models trained on both synthetic and original datasets. Key metrics for this process include matching gradients [72, 32, 70, 44], features [63], distributions [71, 73], and training trajectories [11, 17, 22, 18, 68, 24]. However, these methods suffer from substantial computational overhead due the incessant calculation of discrepancies between the distilled and original datasets. The optimization of the distilled dataset involves minimizing these discrepancies, necessitating multiple iterations until convergence. As a result, scaling to large datasets, such as ImageNet [20], becomes challenging.
Extracting key information from original into distilled datasets. A promising strategy involves identifying metrics that capture essential dataset information. These methods efficiently scale to large datasets like ImageNet-IK using robust backbones without necessitating multiple comparisons between original and distilled datasets. For instance, SR [67] condenses the entire dataset into a model, such pre-trained neural networks like ResNet- 18 [26], and then extracts the knowledge from these models into images and targets, forming a distilled dataset. Recently, RDED [56] posits that images accurately recognized by strong observers, such as humans and pre-traiee pre-trained models, more critical for learning.
Summary. We make the following observations regarding scalable dataset distillation methods utilizing various metrics: (a) a few of these metrics have proven effective for data distillation at the scale of ImageNet. (b) all these metrics require human-labeled data; (c) there is currently no established theory elucidating the conditions under which data distillation is feasible; (d) despite their success, the theory behind training neural networks with reduced data is underexplored.
2.2 Self-supervised Learning: Representation Learning Using Unlabeled Data
The primary objective of self-supervised learning is to extract robust representations without relying on human-labeled data. These representations should be competitive with those derived from supervised learning and deliver superior performance across multiple tasks.
Contrasting self-generated positive and negative Samples. Contrastive learning- based methods implicitly assign one-hot label each sample and its augmented versions to facilitate discrimination. Since InfoNCE [47], various works [25, 12, 13, 8, 31, 15, 74, 40, 9, 27J have advanced contrastive learning. MoCo [25, 13, 15] uses momentum encoder for consistent negatives, effective for both CNNs and Vision Transformers. SimCLR [12] employs strong augmentations and a nonlinear projection head. Other methods integrate instance classification [8]. data augmentation [31, 74], clustering [40, 9]. and adversarial training [27]. These enhance alignment and uniformity of represen- tations on the hypersphere [64].
Asymmetric model-generating representations as targets. Asymmetric network methods achieve self-supervised learning with only positive pairs [30, 50, 14], avoiding representational collapse through asymmetric architectures. BYOL [30] uses a predictor network and momentum encoder. Richemond al. [50] show BYOL performs well without batch statistics. SimSiam [14] halts the gradient to the target branch, mimicking the momentum encoder's s effect. DINO [10] employs self-distillation loss. UniGrad [59] integrates asymmetric networks with contrastive learning methods within a theoretically unified framework.
3

---

Revealing Critical Properties of Efficient Learning over Data
We begin by presenting formal definitions of supervised learning over a (efficient) dataset.
Definition (Supervised learning over data). For dataset D (Dx, Dy) {(x, y: | drawn from the data distribution (X.Y Y) in space (X. y), the goal of a model learning algorithm is identify optimal model o that minimizes the expected ervor defined by:
((xx))((X,))
(1)
where l indicates the loss function and e denotes a predetermined deviation. This is rypically achieved through parameterized model Φo. where denotes the model parameter within the parameter space @ The optimal parameter 6 is determined by training the model to minimize the empirical loss over the dataset:
0 == arg min ece (C( D; == arg min e { ((oo))))))))
(2)
The training process leverages an optimization algorithm such as stochastic gradient descent [51, 33).
Definition 2 (Data-elficient Learning) Data-efficient learning seeks to derive opti-
mized/efficient dataset, denoted as S Sy) {x,, [)) from the original dataset D. The objective to enable models Φgs trained to achieve the desired generalization performance, as defined in (1), with fewer training steps and reduced computational budget compared to training on the original dataset D.
3.1 Unifying (Self-)Supervised Learning from a Data-Centric Perspective
To ease the understanding and our methodology design in Section 3.4, we unify both conventional supervised learning and self-supervised learning as learning to map samples in Dx to targets in Dy: this view forms 'supervised learning' from data-centric perspective. Specifically, these two learning paradigms involve generating targets Dy = (y the empirical loss (2). The only difference lies in the target generation models (or simply labelers) v::
(xx s.t. x Dx and minimizing
(a) Conventional supervised learning, referred to as human-supereen via human annotation. Note that the targets stored and used statically throughout the training. (b) Self repervupeee learning (also see Footnote 2), e.g., BYOL [30] utilizes an Exponential Moving Average (EMA) version of the learning model o to generate targets y EMA o (x). Note that the targets are dynamically changing during training as the model Φe keeps evolving.
learning, generates targets
This unified perspective allows us jointly examine and address the inefficient training issue of (self- jsppervied learning from data-centric perspective, in which in Scction 3.2 we first study the impact of samples Dx and targets Dy the model training process and then investigate whether and how a distilled dataset (Sx, Sy) can facilitate this process.
3.2 Empirical and Theoretical Investigation of Data-Centric Efficient Learning
To elucidate the ideal data properties of training on a dataset D, we examine the simple task over bimodal Gaussian mixture distribution as case study. We begin by defining the problem.
Definition 3 (Bimodal Gaussian mixture distribution). Given two Gaussian distributions No 1) and NG (u2, I), where μn and μ2 are the means and "² the variance (here we set μl P22 = 2 and Σ 0.5). We define bimodal mixture data G (Gx, Gy) as:
G = [xx v) = (1 - X + x s.t. Bermoullipp = 0.5). No. ~ N1. (3)
Moreover, we define a corresponding binary classification neural network model as:
fe(x) == ((
(9 ReLU (0 ( + 01a) + g
(4)

---

where a(2) = I+e- function for the hidden layer, which provides non-linearity to the model; 0 and 0 are the weights and biases of the hidden layer; (l1) and @(4) are the weights and biases of the output layer.
I+E- is the sigmoid activation function: ReLU(z) = m0, 2) is the activation
Modern representation learning fundamentally relies on optimization (see Definition 1). We show that modifications to the data influence the convergence rate of the optimization process, thereby impacting the overall representation learning procedure. Furthermore, we try to uncover several key properties of data efficiency through our theoretical analysis of the case study. In the following, we denote the modified distribution by G' = (G'x, Gyy) and examine the altered samples G'x and corresponding targets Gly independently.
Investigating the properties of modified samples. The modification process here only rescales the variance of the original sample distribution Gx defined in Definition with new Σ (rather than the default 0.5), while let Gy == Gy: see explanations in Appendix E. Therefore, we examine the distilled samples G'x by setting the variable E within the interval (0, 1).
Results in Figure demonstrate that the distilled samples G'x with smaller variance achieve faster convergence and better performance compared that of G. To elucidate the underlying mechanism, we provide a rigorous theoretical analysis in Appendix B culminating in Theorem 1.
Theorem 1 (Convergence rate of learning efficient samples). For the classification task stated in Definition 3, the convergence rate for the model fe trained after steps over distilled data G' is:
Es, [C(fo.;G': [
L(fo- Gl:
Ö(E3)
(5)
where E denotes the MSE loss, i.e., ((ū. 3) := | y11 and for indicates the optimal model, Ô signifies the asymptotic complexity. Modified samples characterized by smaller value of E facilitate faster convergence.
Investigating the properties of modified targets. On top of the property understanding for modified samples, we further investigate the potential of modified targets via G'. In detail, for modified samples, consider the most challenging (c.f. Figure 2b) yet the common case, namely G'x with
= (see explanations Appendix M) ). For the corresponding modified targets Gyy, similar to the prior data-efficient methods [56, 67], for any sample x drawn from G'x, we refine its label by assigning ý p fe- (x) (1 p) y- Here, p denotes the relabeling intensity coefficient, and so- represents strong pre-trained model (simply, utilize the model trained on the data in Figure 2c ).
Theorem (Convergence rate of learning on re-labeled data). For the classification task as in Definition 3, we have the convergence rate for the model fo trained after t steps over modified data G
Eg, -
0(1
(6)
Note that p controls the upper bound of the convergence rate, indicating that using modified targets with a higher value of enables faster convergence.
Results in Figure 2
illustrate that the modified targets Gyy with higher values of lead to faster train- ing convergence and better performance. See theoretical analysis in Theorem 2 and Appendix B.
3.3 Extended Understanding of Data-Centric Efficient Learning
The empirical and theoretical investigations regarding the properties of modified samples G'x and targets Gyy- in Section 3,2 are limited to simplified case (as described in Definition 3) and may not extend to all practical scenarios, such as training a ResNet [26] on the ImageNet dataset [20].
Interestingly. we observe that the advantageous modifications of both samples G'y and targets Gyy converge towards a unified principle: minimizing or preventing any sample x from being labeled with multiple inaccurate targets y. This principle emphasizes the importance of providing accurate and informative targets y for each sample x, as analyzed Remark 1, and suggests extending this insight to any complex dataset like S.
5

---

-.
I Values
-14
0.59
Σ Values
0.1
Intial Farameter Min
0225
02
0.1
0.7
0.8
25
0.20
0.3
-.7
0.75
04
0.5
0.9
1.0
-1.9
-11
015
0.75
-17
Training Steps
-0.5
0.5
1.5
2.5
3.5
(a) Loss landscape for 2 (b) Evaluate G'x with different E
(c) Optimal G'x (C 0.1)
Figure 2: Investigating modified samples with varied E values. Following [39], Figure 2a visualizes the validation loss landscape within two-dimensional parameter space. along with three training trajectories corresponding different Σ settings. Figure 2b illustrates the performance of models trained using samples with varied Σ. The optimal case in our task, utilizing samples with > 0.1 (which achieves the lowest validation loss in Figure 2b), is visualized in Figure 2c, where the color bar represents the values of targets y.
35
Values 0.1
Values
Iriial Parameter
0.1
0.6
05
Gobel miniumm
-12
0.2
0.7
02311
0.3
0.8
-14
0.775
044
0.5
1.0
-19
-1.
-11
0.100
0,775
-1.
-16
200
400
600
800
1000
-3
-2 -1
Training Steps
-.5
6.5
1.5
2.5
3.5
(a) Loss landscape for p
(b) Evaluate Gy with different
(c) Optimal Gy (p 1.0)
Figure 3: Investigating modified targets with varied p values. We present a visualization of the validation loss
landscape in Figure 3a, including three training trajectories that correspond to different p settings. Figure 3b illustrates the performance of models trained using targets with varying P values. The optimal scenario for our task, which uses targets with p 1.0, is depicted Figure 3c.
Remark 1 (Ideal data properties avoid implicitly introduced gradient noise from data)
Intuitively, the semantic information within each sample x should be unique and not identical to another sample. Consequently, the exact target y, which represents the semantic information of x, should also be unique and informative. This implies the necessity of establishing bijective (or one-to-one) mappings between samples and their respective targets.
In contrast, when a sample x (or several similar samples) is labeled with multiple different targets y. it may implicitly introduce noise into the gradient V((x,)) y), thereby hindering the optimization.
However, real-world datasets often deviate from the ideal properties described above, as discussed in Remark 2 below and further analyzed in Appendix M.
Remark 2 (Imperfect mappings and inaccurate targets in real-world datasets). practice we observe that 'noisy mappings' between input samples and targets are prevalent in real-world datasets. As illustrated in Figure 6, several common phenomena contribute to this issue:
Similar or identical input samples may be assigned different targets due to using data augmen- tations, which is common in both reffesupee and human sumaaaseee learning settings.
Inaccurate targets may be generated, particularly in supervised learning scenarios. In cumman-supervised learning, all samples within class are mapped one-hot target.
These imperfect mappings and inaccurate targets pose challenges to achieving optimal training efficiency and effectiveness for real-world datasets.
3.4 Generalization-buundee Efficient Data Synthesis
Given insights from Remark an effective approach to generate efficient data from original data D involves employing high-quality labeler 2/b to relabel each sample within D. This process

---

results in the formation of an optimized dataset S. However, the generalization ability of models trained on these optimized datasets S is not inherently assured. For given labeler v. we derive the generalization bound, which quantifies the representation distance between models &e and Φ= trained on the optimized dataset S. This is presented in Theorem 3 (see Appendix D for the proof).
Definition (Representation distance"). We introduce our proposed metric as DRep( + ④r: D) ==
inf {Ex~D (e(W@ + b, (x(x))
(7)
WeRmxx bekm
which quantifies the distance between a source model Φg and a target model Φy with respect to a dataset D, and the loss function is defined as E(g.y) 1 # y)-
"Intuitively, smaller DRp( Φr; D) indicates that the model 4 can be transformed into Φr over data D via a linear model with relative ease. This also implies that @g can achieve the same linear evaluation performance as Qy on data D.
Theorem 3 (Generalization bound with labeler 2b) Assuming the model Φg X' - y belongs hypothesis space 4 Then, for any (0, 1), with probability least - 6, we have
Deep (0o Φ x) S Bsample + Brarget + B
88)
where BTareet = Deep (2b + Dx): B Model = DRep( (gg + ub: Sx) + 2D (+) + xx )
and Bsample = (Sx. Dx) Drv represents the total variation divergence [4], and RDx is the empirical Rademacher complexity.
Drawing insights from Theorem 1, 2 and 3, we define the properties of ideal data below:
Definition 5 (Properties of ideal efficient data, including samples Sx and targets Sy). To meet the objectives of Definition 2. an ideal efficient data requires: ( Targets (Sy):
generating targets (x) from the labeler 1b that are accurate (i.e., align with human-annotating
targets and optimal model Φ'), aiming to minimize DRep(V- - x"; Dx);
the generated target (xxx should be informative corresponding sample x (i.e., forming
perfect bijective mappings with the original samples x), according to Remark 5;
Samples (Sx): low distribution disparity represented by Drv (Sx, and high sample diversity denoted as [S aiming at minimizing Drv Dx).
" "Alignment with auman--annotated targets can occur via direct prediction or linear transportabiility.
To meet the requirement @ specified in Remark 5, we utilize weak data augmentation on Dx to generate the set Sx, aimed at enhancing the diversity |Sxl while ensuring the term Drv Dx) remains minimal. However, satisfying requirement 4 is non-trivial, as capturing labeler &b that matches o² is intractable (i.e., achieving (1e + x" Dx) = 0 is challenging).
Fortunately, our experiments in real-world scenarios, as discussed in Section A, demonstrate that employing prior model as the labeler v/ generally approximates o² within the representation space. Therefore, we posit that introducing prior model vi to generate an efficient dataset S can typically accelerate the early stages of learning. Subsequently, the original dataset D should be employed.
In (8), the last term, ( tb; Sx) 29D, (+) + O(D 11) includes 29 (4) and
x(IDx I-),, which depend the neural architecture and the size of Dx. respectively, and can be considered constants. Thus, optimizing the model involves minimizing (Re - 8b; Sx) through
training Φg. A detailed technical solution is provided in Section 4.
4 Methodology
Building upon the theoretical insights from Section 3.2, we propose our RELA (see Figure 1): (a) RELA-D (c) is used to generate efficient data (c.f. Section 4.1 ); (b) RELA-F (4) guides the models to train over the efficient data from RELA-D (c.f. Section 4.2).
7

---

4.1 RELA-D (9): Synthesis of Efficient Dataset
Motivated by two property requirements in Definition 5, here we introduce our optimization-free synthesis process of both samples and targets in RELA-D (see technical details in Appendix F).
Generating transportable representations as the targets. We argue that well-trained models (called prior models) on diverse real-world datasets using various neural network architectures and algo- rithms converge towards the same linear representation space. In other words, the generated pseudo representations Ry for samples Dx using these prior models are linearly transportable to each other and to the human-annokaling targets. The empirical verifications refer to Appendix A. We further justify Appendix F.1 that the requirement ① in Definition can be achieved by employing a prior model as the ideal labeler &b I + IR, i.c., generating Ry = { {x) x ~ } as the targets. The generation process of targets is conducted only once (refer to Appendix H for details), and the generated targets Ry are stored and combined with the samples Dx to form the data D = Ry).
Efficient and distribution-alige -lligned sample generation. To satisfy requirement @ in Definition 5 efficiently, we employ basic data augmentations into data Dx such RandomResizeCrop with minimum scale of 0.5 (as opposed to the default of 0.08) and RandomHorizontall1F with p = 0.5.
4.2 RELA-F (4) Assist Learning with Generated Efficient Dataset
In this section, showcase the significance of understanding ideal data properties and generated efficient dataset in assisting self-supervised learning, given this self-supervised paradigm on unlabeled data suffers from significant inefficiency issues compared to human- supervise learning [62].
Here we propose plug-and-play method that can be seamlessly integrated into any existing self- supervised learning algorithm, significantly enhancing its training efficiency by introducing an additional loss term. Formally, the loss function is defined as follows:
A LRELL (1 X) Last, where CRELA Ex,y-(Dx, Ry) [e (Wo) - b,y)],
where yl := 1 y /|z/|/||lly) be the loss function, Last denotes the loss specified by any self-supervised learning method, respectively. Furthermore, the data = (Dx: Ryy are collected using the strategy outlined in Section 4.1, with updates occurring at each k-th epoch.
The dynamic coefficient À E {0, 1} divides the training process into two distinct stages. Initially. À is set to 1 to emphasize CRELA. assuming its crucial role in the early learning phase. As the model Φa improves and self-generated targets become more reliable in Last, an adaptive attenuation algorithm adjusts À to (note that the initial À tuning-free for all cases and see Appendix J for details). a result, only single loss term in (9) is calculated, ensuring no extra computational cost with RELAA
To enhance the recognition of ReLA-aided algorithms, we re-denote those that are used in their names. For example, the YOL algorithm [30], when enhanced with RELA, is re-denoted as BYOL ( ).). Furthermore, the prior models downloaded from the internet are not consistently robust, the aforementioned dynamic setting of À also prevents the model Øg from overfitting to potentially weak generated targets. The efficacy of our proposed RELA is empirically validated in Section 5.
5 Experiments
This section describes the experimental setup and procedures undertaken to test our hypotheses and evaluate the effectiveness of our proposed methodologies.
Experimental setting. We list the settings below (see more details in Appendix K
Datasets: For low-resolution data (32 x 32), we evaluate our method on two datasets, i.e., CIFAR-10 [35] and CIFAR- 100 [34]. For high-resolution data,
conduct experiments two large-scale datasets including Tiny-Iman ImageNet (64 x 64) [36] and full ImageNet- 1K (224 x 224) [20], to assess the scalability and effectiveness of our method on more complex and varied datasets.
Neural network architectures: Similar to prior works/henchmarks of dataset distillation [56] and self- supervised learning [57, 19], several backbone architectures to evaluate the generalizability
8

---

Table 1: Benchmark our RELA with various prior models against BYOL. We compare evaluation results of the models trained using BYOL with 10%, 20% and 50% training budget/steps: BYOL ( with different prior models; BYOL with full budget, denoted as BYOL* in this table. Regarding the prior models used for our RELA, we respectively utilize six models with increasing representation capabilities, including * randomly initialized network (Rand.); four "trained models (CFI0-T, CF100-T, TIN-I, --I corresponding four datasets (listed below); CLIP-RN50, The evaluations are performed across four datasets, ie.. CIFAR- 10 (CF-10), CIFAR-100 (0F-100), Viny-ImageNer (T-IN), and ImageNet- IK (IN-1K). We underline the results that outperform the full training, and bold the results that achieve the highest performance using a specific ratio of budget. All the networks used for training are ResNet--1. 18, except the ResNet-50 used for IN-IK.
BYOL ( 4) w/
Dataset
BYOL
Rand.
CF10-T
CF100-T
TIN-T
INIK-T
CLIP-RN50|
BYOL
10 58.3 = 0.1
71.4 0.0
8..1
78.2
79.6 0.1
82.0 81,6 0.1 83.9
CF-10
20 70.1 = 0.2
77.1
0.2
8.1
0.0
8.1
84,4 0.1
82.7 + 0.2
50 77.9 0.0
82.7 0.1
86.5
86.2
0.1
87.3 0.2
86.7 + 0.0
10 26.9 = 0.2
41.8
0.1
=1.4 0.1
53.5
0.1
56.4 1: 0.2
55,4 + 0.1
CF-100
34.8- = 0.3
4..1
0.1
55.7
56.7
+ 0.0
59.5
0.1
57.9 0.0
52.5 +: 0.3
50 41.4 = 0.3
54.6 0.2
59.7
59.8
60.0
1:0::00 0.1
61.6 0.1
61.0 0.0
10 25.1 = 0.3
34.5 + 0.3
3,1
38.4
0.0
41.2
+ 0.1
41,6 + 0.1
39.6 + 0.4
T-IN
20 30.7 - 0.1
38.2 + 0.0
4.09
42.3 + 0.0
43.2
+
0.1
44.1 + 0.1
42.6 + 0.1
43.6 + 0.3
50 37.7 = 0.2
43.9 + 0.1
4.1
45.9 : 0.1
45.8
+
0.1
46.4 0.1
46.3 + 0.1
10 44.5 = 0.1
51.7 +1.7
53.7
53.3
0.1
53.6
+
0.1
54,9 0.1
56.2 + 0.1
IN-IK
20
55.3 = 0.0
56.9 +
0.0
57.6 5..6
0.1
57.8
4
0.1
58.0
59.5 1 0.1
61.9 + 0.1
50 60.8 0.2
61.1 0.1 62.1 0.1 61.8 = 0.1
61.7
0.0
61.9
1 0.0
62.9 0.1
of our method, including ResNet- (18, 50, 101) [26], EfficientNet-BO [58]. MobileNet- V2 [52], ViT [21], and a series of CLIP-based models [49]. These architectures represent a range of model complexities and capacities, enabling a comprehensive assessment of our approach.
Referring to prior widely-used benchmark [57, 19], we consider several state-of-the-art methods as baselines for broader practical impact, including: SimCLR [12], Barlow Twins [69], BYOL [30], DINO [10], MoCo [25], SimSiam [14], SWAV [9]. and Viereg [3].
Baselines:
Evaluation: Following previous benchmarks and research [57, 19, 12, 3], we evaluate all the trained models using offline linear probing strategy to reflect the representation ability of the trained models, and ensure a fair and comprehensive comparison with baseline approaches.
Implementation details: We implement our method by extending a popular self-supervised learning open-source benchmark [57] and use their configurations therein. This includes using Adam W as the optimizer, with mini-batch size of 128 (except for ImageNet-1K, where we use mini-batch size of 512). We implement our method through Py Torch [48], and all experiments are conducted on NVIDIA RTX 4090 GPUs See more detailed configurations and hyper-parameters in Appendix K.
5.1 Primary Experimental Results and Analysis
Recall that our RELA-D ()), as illustrated in Figure 1 and Section 4.1. requires an unlabeled dataset and any pre-trained model freely available online to generate the efficient dataset. To justify the superior performance and generality of our RELA across various unlabeled datasets using prior models with different representation abilities, our comparisons in this subsection start with BYOL 2 and then extend to other self-supervised learning methods.
Table 1 demonstrates the efficacy and efficiency of our RELA in facilitating the learning of robust representations. Overall, BYOL 4 consistently outperforms the original BYOL when trained with a reduced budget. In certain cases, such as CIFAR-100 BYOL (4) employing only 10% of the budget can surpass the performance of B YOO-ttriined models using the entire budget Specifically:
(a) A stronger prior model (e.g., CLIP) enhances the performance of RELA more effectively than a weaker model (e.g., Rand.):
(b) Our RELA is not sensitive to the prior knowledge. For instance, using CF10-T as the prior model can achieve competitive performance compared to that trained on extensive datasets (e.g., CLIP);
NNote that (1) BYOL is competitive across various datasets [30, 3, 57, 12], and (2) various sepf-suue learning methods can be unified in the same framework [59] (see detailed analysis in Appendix G).

---

Table 2: Evaluating our RELA cross-architecture settings. Our RELA-D ( distills datasets with prior RN18 (Rand.) and CLIP-IRNIO1, RN50x4, ViT B/32, ViT B/16, ViT L/14), then versus transfer to ResNet-18; MobileNet-V2; EfficientNet-BO; VIT T/16. We train models using 10% budget through (original) BYOL (
Dataset
Arch.
Original
(4) w/ RNI8
RNI01
RN50x4
VIT B/32
VIT B/16
VIT 1/14
R8
58.3
71.4 =1.4
81.9
82.1 + 0.3
83.2
0.2
83.1
0.1
82.4 + 0.1
MobilcNet-V2
0.1
69.4 + 0.0
8
80.8
+ 0.0
81.6
0.1
82.9
+ 0.2
81.2 + 0.2
CF-10
EfficientNet-B0
23.9 4.2
68.8 0.6
8322
0.1
87.4
0.1
86,4 0.1
83.1 0.1
T/16
43.4 0.1
57.1
65.9 0.0
66.4 0.1
69.9
= 0.3
0.1
63.7 + 0.1
ResNet- 18
25.1 + 0.3
0.3 0.3
0.1
39.1
0.4
35.8 =01
28.4 1 0.2 0.1
MobileNet-Ve V2
8.8 +: 0.1
28.3
0.1
36.8
1.2
36.0 = 0.0
$7.9
20.6 1 0.5
T-IN
EfficiemtNet-BO0
4.1 1 0.0
33.2
0.3
43.5
0.
41.7
0.2
44.0 = 0.1
44.2
4
0.0
37.9
1 0.1
ViT T/16
12.5 4: 0.1
24.6
0.0
26.1
2 0.1 27.6
1 0.1 26.9 = 0.2
24.5 + 0.1
21.6
+ 0.0
Table 3: Evaluating our RELA across different self-supervised learning methods. We extend our analysis
beyond BYOL by training and evaluating models using seven additional self-supervised learning methods, along with their AaLgm-ngeeeteed counterparts 4 )) utilizing randomly initialized ResNet- 18 (Rand.) and
CLIP-RN50 prior models for the RELA-D (. All methods are trained using 10% budget.
Dataset
Method
SimCLR
Barlow
DINO
MoCo
SimSiam
SWAV
Vicreg
Original
70.7 0.2
63.7 + 0.3 66.2 +: 0.2
67.4 +: 0.4
45.8 + 0.4
66.2 + 0.3
71.3 + 0.2
CF-10
(4) w/ Rand.
70,9 +
68.8
0.2
7.11
70,9
0.1
66.7
0.1
69.5 0.2
71.3 + 0.1
CLIP-RN50
0.1
76.5
82.4 0.2
79.8 0.1
79.3 0.1
77.3
80.1 + 0.1
Original
30.4 +..4+00.1
28.9
+.3
2721+0.2
17.8
+ 0.3
34.0 2.12+0.1
T-IN
(4) w/ Rand.
30.7 + 0.2
31.9 0.1
0.2
0.1
25.4
+ 0.1
0.2
34.1 0.1
CLIP-RN50
33.0 + 0.3
33.5 = 0.2
35.1
+
0.0
37.1
32.6
+ 0.1
32.6
39.1 + 0.2
(c) A randomly initialized model can effectively aid in accelerating learning through our RELA. This can be considered an effective scenario of
supervision" [7] using pseudo targets.
Cross-architecture generalization. RELA-D generates efficient datasets using specific neural architecture. evaluate the generalization ability of these datasets, it is essential to test their performance various architectures not used in the distillation process. Table 2 presents the performance of our RELA in conjunction with various prior models and trained model architectures, demonstrating its robust generalization ability. Specifically:
(a) The integration of RELA always enhances the performance of original BYOL;
(b) Our RELA method exhibits minimal sensitivity to the architecture of the prior model, as evidenced by the comparable performance of BYL 4 using both ViT- and ResNet-based models.
Combining RELA across various self-supervised learning methods. To demonstrate the ef- fectiveness and versatility of RELA in enhancing various self-supervised learning methods, conduct experiments with widely-used techniques. Table 3 presents the results, highlighting the robust generalization capability of RELA. Our findings consistently show that RELA improves the performance of these methods while maintaining the same budget ratio, emphasizing its potential on learning using unlabeled data, Additionally, we provide the results when combining RELA with human- supervsee learning in Appendix I.
6 Conclusion and Limitation
In this paper, to address the Problem 1, we investigate the optimal properties of data, including samples and targets, to identify the properties that improve generalization and optimization in deep learning models. Our theoretical insights indicate that targets which are informative and linearly transportable to strong representations (e.g., human annotations) enable trained models to exhibit robust representation abilities. Furthermore, we empirically find that well-trained models (called prior models) across various tasks and architectures serve as effective labelers for generating such targets. Consequently, we propose the Representation Learning Accelerator (RELA), which leverages any freely available prior model generate high-quality targets for samples. Additionally, RELA can enhance existing (self-)suuerviised learning approaches by utilizing these generated data to accelerate training. However, theoretical analysis is restricted to the simplified scenario described in Definition 3, which has limited applicability in real-world contexts.
10

---

7 Acknowledgement
We thank Xinyi Shang, Bowen Ding and and the anonymous reviewers for their invaluable comments and feedback. We also thank Bei Shi for assisting with the partial code implementation. This work was supported part by the National Science and Technology Major Project (No. (022D0015101) the Research Center for Industries of the Future (RCIF) at Westlake University, and the Westlake Education Foundation.
References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774,220 2023.
[2] Yutong Bai, Xinyang Geng, Karttikeya Mangalam, Amir Bar, Alan Yuille, Trevor Darrell, Jitendra Malik, and Alexei A Efros. Sequential modeling enables scalable learning for large vision models. arXiv preprint 22.000882.. 2023.
[3] Adrien Bardes, Jean Ponce, and Yann LeCun. Vicreg: Variance-invariance--ovv ization for self-supervised learning. arXiv preprint arXiv:2105.04906 2021.
regular-
[4] Andrew R Barron, Lhszl Gyorfi, and Edward C van der Meulen. Distribution estimation consistent in total variation and in two types of information divergence. IEEE transactions on Information Theory, 38(5): 1434-1444, 1992.
[5] Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Vaughan. A theory of learning from different domains. Machine Learning, 79:151-1755 2010.
[6] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D) Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1 1877779999., 2020.
[7] Collin Burns, Pavel Izmailov, Jan Hendrik Kirchner, Bowen Baker, Leo Gao, Leopold Aschen- brenner, Yining Chen, Adrien Ecoffet, Manas Joglekar. Jan Leike, et al. Weak tt-oon gener- alization: Eliciting strong capabilities with weak supervision. arXiv preprint arXiv:2312.099390. 2023.
[8] Yue Cao, Zhenda Xie, Bin Liu, Yutong Lin, Zheng Zhang, and Han Hu, Parametric instance classification for unsupervised visual feature learning. In NeurlPS, 2020.
[9] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. In NeurIPS, 2020.
[10] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In ICCV, 2021.
[11] George Cazenavette, Tongzhou Wang, Antonio Torralba, Alexei A Efros, and Jun-Yan Yan Zhu. Dataset distillation by matching training trajectories. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Patlern Recognition, pages 4750-4755 2022.
[12] Ting Chen, Simon Korbblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In ICML, 2020.
[13] Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint 20V:20003.04297 2020.
[14] Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. CVPR, 2021.
[15] Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. arXiv preprint arXiv:2104.0 02057, 2021.
[16] Yu Cheng, Duo Wang, Pan Zhou, and Tao Zhang. A survey of model compression and acceleration for deep neural networks. arXiv preprint arXiv:1710.09282. 2017.

---

[17] Justin Cui, Ruochen Wang, Si Si, and Cho-Jui Hsich. De-bench: Dataset condensation bench- mark. Advances in Neural Information Processing Systems. 35:810-82 2022.
[18] Justin Cui, Ruochen Wang, Si Si, and Cho-Jui Hsich. Scaling up dataset distillation to imagenet- Ik with constant memory. In International Conference on Machine Learning, pages 6565-6590. PMLR, 2023.
[19] Victor Guilherme Turrisi Da Costa, Enrico Fini, Moin Nabi, Nicu Sebe, and Elisa Ricci. solo- learn: A library of self-supervieed methods for visual representation learning. Journal of Machine Learning Research, 23(56):1 2022.
[20] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. leee, 2009.
[21] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.1 1929, 2020.
[22] Jiawei Du, Yidi Jiang, Vincent YF Tan, Joey Tianyi Zhou, and Haizhou Li. Minimizing the accumulated trajectory error to improve dataset distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3759-3758, 2023.
[23] Noah Golowich, Alexander Rakhlin, and Ohad Shamir. Size-independent sample complexity of neural networks, 2019.
[24] Ziyao Guo, Kai Wang, George Cazenavette, Hui Li, Kaipeng Zhang, and Yang You. To- wards lossless dataset distillation via difficult--lligned trajectory matching. arXiv preprint arXiv:2310005773 2023.
[25] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020.
[26] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016
[27] Qianjiang Hu, Xiao Wang, Wei Hu, and Guo-Jun Qi. Adco: Adversarial contrast for efficient learning of unsupervised representations from self-trained negative adversaries. In CVPR, 2021.
[28] Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Chen, Hy- oukJoong Lee, Jiquan Ngiam, Quoc V Le, Yonghui Wu, al. Gpipe: Efficient training of giant neural networks using pipeline parallelism. Advances in neural information processing systems, 32, 2019.
[29] Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: generalization in neural networks. Advances in neural information processing systems, 31, 2018.
Convergence and
[30] Grill Bastien, Strub Florian. Altché Florent, Tallec Corentin, Pierre Richemond H., Buchatskaya Elena, Doersch Carl, Bernardo Pires Avila, Zhaohan Guo Daniel, Mohammad Azar Gheshlaghi, Piot Bilal, Kavukcuoglu Koray, Munos Rémi, and Valko Michal. Bootstrap your own latent new approach to self-supervised learning. In NeurlPS, 2020.
[31] Yannis Kalantidis, Mert Bulent Sariyildiz, Noe Pion, Philippe Weinzaepfel, and Diane Larlus. Hard negative mixing for contrastive learning. In NeurIPS, 2020.
[32] Jang-Hyun Kim, Jinuk Kim, Seong Joon Oh, Sangdoo Yun, Hwanjun Song, Joonhyun Jeong, Jung-Woo Ha, and Hyun Oh Song. Dataset condensation via efficient synthetic-data param- eterization. In International Conference on Machine Learning, pages 11102-11118. PMLR, 2022.
12

---

[33] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
[34] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[35] Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. Ci and cifar-100 datasets. URI: https:///ww. cs. toronto. edu/rriz/ifar. liml, 6(1):1, 2009.
[36] Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7(7):3, 2015.
[37] Dong Bok Lee, Seanie Lee, Joonho Ko, Kenji Kawaguchi, Juho Lee, and Sung Ju Hwang. Self-supervised dataset distillation for transfer learning. In Proceedings of the International Conference on Learning Representations (ICLR), 2024.
[38] Jachoon Lee, Lechao Xiao, Samuel S Schoenholz, Yasaman Bahri, Roman Novak, Jascha Sohl-Dickstein, and Jeffrey Pennington. Wide neural networks of any depth evolve as linear models under gradient descent * Journal of Statistical Mechanics: Theory and Experiment, 2020( 140022, December 2020.
[39] Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss landscape of neural nets. Advances in neural information processing systems, 31, 2018.
[40] Junnan Li, Pan Zhou, Caiming Xiong, and Steven CH Hoi. Prototypical contrastive learning of unsupervised representations. In ICLR, 2021.
[41] Yuanzhi Li and Yingyu Liang. Learning overparameterized neural networks via stochastic gradient descent structured data. Advances in neural information processing systems, 31, 2018.
[42] Tao Lin, Sebastian U Stich, Kumar Kshitij Patel, and Martin Jaggi. Don't 't use large mini-batches, use local sgd. arXiv preprint arXiv:1808.072/11 2018.
[43] Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. Gpt understands, too. A/ Open, 2023.
[44] Yanqing Liu, Jianyang Gu, Kai Wang, Zheng Zhu, Wei Jiang, and Yang You. Dream: Efficient dataset distillation by representative matching. arXiv preprint arXiv:2302.14416. 2023.
[45] Sachin Mehta, Maxwell Horton, Fartash Faghri, Mohammad Hossein Sekhavat, Mahyar Na- jibi, Mehrdad Farajtabar, Oncel Tuzel, and Mohammad Rastegari. Catlip: Clip-level visual recognition accuracy with 2.7 x faster pre- mre-trainng on web-scale image-text data. arXiv preprins arXiv:2404. /5653, 2024.
[46] Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar. Foundations of Machine Learning. 2018. Second Edition.
[47] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding, arXiv preprint 1rXiv:1807.03748, 2018
[48] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
[49] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pages 8748-8763 8763-876....1 PMLR, 2021.
[50] Pierre H Richemond, lean-Bastien Grill, Florent Altché, Corentin Tallec, Florian Strub, Andrew Brock, Samuel Smith, Soham De, Razvan Pascanu, Bilal Piot, et al. Byol works even without batch statistics, arXiv preprint arXiv:2010.10244 2020.
13

---

[51] Herbert E. Robbins. A stochastic approximation method. Annals of Mathematical Statistics, 22:400-407, 1951.
[52] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4510-4520, 2018.
[53] Shitong Shao, Zeyuan Yin, Muxin Zhou, Xindong Zhang, and Zhiqiang Shen. Generalized large-scale data condensation via various backbone and statistical matching. arXiv preprint arXiv:2311.1995 2023.
[54] Shitong Shao, Zikai Zhou, Huanran Chen, and Zhiqiang Shen. Elucidating the design space of dataset condensation. arXiv preprint arXiv:2404. 137333, 2024.
[55] Emma Strubell, Ananya Ganesh, and Andrew McCallum, Energy and policy considerations for deep learning in nip. arXiv preprint arXiv: 1wxivv:906.02244 2019.
[56] Peng Sun, Bei Shi, Daiwii Yu, and Tao Lin. On the diversity and realism of distilled dataset: An efficient dataset distillation paradigm. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), 2024.
[57] Igor Susmelj, Matthias Heller, Philipp Wirth, Jeremy Prescott, and Malte Ebner et al. Lightly. GitHub. Note:
2020.
[58] Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International conference on machine learning, pages 6105-61 6116 14. PMLR, 2019.
[59] Chenxin Tao, Honghui Wang, Xizhou Zhu, Jiahua Dong, Shiji Song, Gao Huang, and Jifeng Dai. Exploring the equivalence of siamese self-supervised learning via a unified gradient framework. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 14431-14440, 2022.
[60] Yuandong Tian, Xinlei Chen, and Surya Ganguli. Understanding self-supervised learning dynamics without contrastive pairs. In ICML, 2021.
[61] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 911)). 2008
[62] Guangrun Wang, Keze Wang, Guangcong Wang. Philip HS Tomr, and Liang Lin. Solving inefficiency of self-supervised representation learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9505-9515 2021.
[63] Kai Wang. Bo Zhao, Xiangyu Peng, Zheng Zhu, Shuo Yang. Shuo Wang. Guan Huang, Hakan Bilen, Xinchao Wang, and Yang You. Cafe: Learning to condense dataset by aligning features. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pages 1216611200, 2022.
[64] Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In ICML, 2020.
[65] Tongzhou Wang, Jun-Yan Yan Zhu, Antonio Torralba, and Alexei A Efros. Dataset distillation. arXiv preprint arXiv: 10959, 2018
[66] David Wolpert and William G Macready. No free lunch theorems for optimization. IEEE Iransactions evolutionary computation, 1(1):67-82, 1997.
[67] Zeyuan Yin, Eric Xing, and Zhiqiang Shen. Squeeze, recover and relabel: Dataset condensation at imagenet scale from a new perspective. arXiv preprint arXiv:2306.14 13092, 2023.
[68] Ruonan Yu, Songhua Liu, and Xinchao Wang. Dataset distillation: A comprehensive review. arXiv preprint arXiv:2301.00 000014, 2023.
[69] Jure Zhontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self- supervised learning via redundancy reduction. In ICML, 2021.
14

---

[70] Lei Zhang, Jie Zhang, Bowen Lei, Subhabrata Mukherjee, Xiang Pan, Bo Zhao, Caiwen Ding. Yao Li, and Dongkuan Xu. Accelerating dataset distillation via model augmentation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 11950-11959, 2023.
[71] Bo Zhao and Hakan Bilen. Dataset condensation with distribution matching. In Proceedings of the IEEE/CVF Winter Conference Applications of Computer Vision, pages 6514-6515555 6523, 2023.
[72] Bo Zhao, Konda Reddy Mopuri, and Hakan Bilen. Dataset condensation with gradient matching. arXiv preprini arXiv:2006.05999, 2020.
[73] Ganlong Zhao, Guanbin Li, Yipeng Qin, and Yizhou Yu. Improved distribution matching for dataset condensation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7856-7866 2023.
[74] Rui Zhu, Bingchen Zhao, Jingen Liu, Zhenglong Sun, and Chang Wen Chen. Improving contrastive learning by visualizing feature transformation. ICCV, 2021.
15

---

Contents
Introduction
2
Related Work
2.1
Dataset Distillation: Efficient yet Effective Learning Using Fewer Data
2.2
Self-supervised Learning: Representation Learning Using Unlabeled Data
3
Revealing Critical Properties of Efficient Learning over Data
3.1
Unifying Selff-SSuprriiee Learning from Data-Centric Perspective
3.2
Empirical and Theoretical Investigation of Data-Centric Efficient Learning
3.3
Extended Understanding of Data-Centric Efficient Learning..
3.4
Generalization-bouuee Efficient Data Synthesis
4
Methodology
4.1
RELA-D (): Synthesis of Efficient Dataset
4.2
RELA-F (4)
Assist Learning with Generated Efficient Dataset
8
5
Experiments
5.1
Primary Experimental Results and Analysis
6
Conclusion and Limitation
10
7 Acknowledgement
11
A Ablation Study
17
B Proof of Theorem
18
B.1
Setup
18
B.2
Algorithm
18
B.3
Bounds with Variance
18
B.4 Nonlinear case
19
B.5 From the Perspective of Feature Learning
19
Proof of Theorem
20
C.1 Setting
................................
20
C.2 Bounds with
20
C,3 Nonlinear case
20
D Proof of Theorem 3
21
D.I Relation between data distribution and Rademacher complexity
25
Explanation of Rescaling Samples
25
Detailed Methodology of RELA-D
F.1 Proof for Ideal Properties of Prior Models
27
G Analysis of Different Seff-Supervised Learning Methods
28
G.I A Unified Framework for SSL
28
G.2 Contrastive Learning Methods
28
................................
G.3 Asymmetric Network Methods
29
G.4 Feature Decorrelation Methods
30
H Budget of RELA for Data Synthesis
31
I RELA in Labeled Dataset Distillation and Human-Supervised Learning
31
LI Experimental Setup
I.2
Main Results
31
J RELA Algorithm
32
16

---

K Experimental Details
32
K.1
Detailed Setup for Experiments Section 3.2
32
K.2
Detailed Sctup for Experiments in Section 5
33
L Batch PCA Reduction
33
L,1
Principal Component Analysis
33
M Analyze the Practical Data Augmentations
35
M.1
Assumptions and Definitions
M.2
Augmented Distributions
................................
M.3
Increased Variance Leads Increased Overlap
M.4
Conclusion
37
M.5
Empirical Analysis for Real-world Datasets
A
Ablation Study
We conduct ablation studies to understand the impact of each component of RELA on performance.
cuP
04
0.774
Computational Cost (%)
Prior Models
INI6-
CFI0TT
TINT
0.
Top-1 Arcuue
60
CY188-1
0.4
(Rand])
Dynamic (Rand))
Band.
0.2
Statie (TIN-T)
20
Nave
Note
C78F
CF1S8-
TB:
INIKT
CLIF
Value constant
Dataset
(a) Representation similarity
(b) Dynamic vs. static strategies (c) Computational cost analysis
Figure 4: Ablation study on BYOL components and parameters. (a) analyze the representation
similarity between various source models (indicated on the x-axis) and target models (indicated on the y-axis). (b) We compare the static RELA weight setting strategy with our adaptive strategy. Dotted lines (*- represent adaptive strategy, while solid lines ('-') denote the static À setting strategy. Specifically, in the static weight setting (e.g.. 0.4), the first 40% of training leverages RELA, with remaining 60% employing the original algorithm. (c) We present the computational cost, quantified as training time/steps, of our RELA across various prior models.
Empirical representation similarity. Our foundational assumption of our RELA is that the representations of rell-trained models, developed using various neural network architectures and algorithms on diverse real-world datasets, exhibit linear transportability to one another. To test this hy- pothesis, assess representation similarity, metric that quantifies the linear transferability between pre-trained models. The results, depicted Figure 4a, demonstrate that representations from robust models (e.g., CLIP) can be effectively transferred to less robust models (c.g., CF10-T). This finding aligns with our results in Table 1, showing that leveraging powerful models (c.g., CLIP and INIK-T) consistently enhances learning in models trained on datasets with limited knowledge, such as CF-10.
Combining RELA and BYOL with static setting strategies. The coefficient X, as introduced in Section 4.2, is pivotal in controlling the weight of the RELA phase during training. To assess the robustness of our adaptive strategy, which dynamically adjusts X, we compare it to a static À setting strategy. The results in Figure 4b indicate that larger (smaller) RELA weights are advantageous when using a strong (weak) prior model. Nonetheless, static settings lack generalizability across various scenarios, whereas adaptive strategy demonstrates superior generalization capabilities.
Analysis of the computational cost when using RELA with different prior models. To validate that our RELA, in conjunction with various prior models, effectively reduce computational costs in self-supervised learning, we conducted experiments comparing the computational expense required for BYOL (4) achieve equivalent performance the original BYOL trained with full budget. The results, illustrated in Figure 4c, consistently demonstrate that RELA assists BYOL lowering training costs while maintaining equivalent performance levels. Furthermore, evident that employing robust prior models consistently leads to greater reductions in training budgets.
17

---

B Proof of Theorem 1
In this section, we prove a slightly modified version of Theoreml, extending the distribution to Generalized Gaussian Mixture(GGM) and making some assumptions for technical reasons. Yet this proof could still reflect the essential of the theorem.
B.1 Setup
Notation N(Ja, a, 8) denotes the generalized Gaussian distribution with pdf 2a1(1705 e - (Ix-a//a)) for Bernoulli distribution.
We focus on the 1-dim situation. Assume that P1 < P2. Define the original data distribution(NN
N( Cu. Su) and Ni N(J2, Cuu 5b))
G = ((x,g) v) 2. B(1, / 1, 1-
No 1+ -N1]
2
and the modified one (Ng = N(u1, B) and N2 N(p2, a,
G' = ((x,)) y ~ 2 B(1, "
1- No6 +
1 + -M).
1, - 2
Our task is predicting y given Note that {=1}, which is bit different from the definition in Section 3.2. In 1-dim situation, we just need one parameter for this classification task, so define fo(x) sign(x 0) to fit the distribution. We could compute the generalization loss original
distribution:
0+p1
C(fo)) = f dF_ +
J dF, dF+)/2 (1
dFF/2
6+F2
a0
Obviously 8* H12 we have:
12-11
((So) - (((o-) = 12-21
dF
dF)/22
0+p2
20g
S1 . (0 - 0²)² (or C₁ 10 - 0*l)
where C1 Cj are constants, Fo F1 F denote the CDF of No Ni and N(O, 1, 1,) respectively. The inequality above is due the fact that function h(x) (f², dF - d/F)/z² has limits 0 and so is bounded.
B.2 Algorithm
For dataset {xx / set the loss function L(O) Σn e [v:(x; + +0), {(v) _ - u
We apply the stocastic gradient descent algorithm and assume the online setting (n = 1): at step draw one sample (I,, y1) from G" then use the gradient V (O,), to update (1 E (0, 1), N):
0++1 0, L(,))
(.(),) = f + (x+ - yr)-
It can be observed that randomness of x leads to noies on gradient.
B.3 Bounds with Variance
We prove the proposition that lower variance of GG can make convergence faster, i.c.
E [(C(fe,) C(fo-) bounded by an increasing function of variance (t fixed).
Proof. From above, we could get
0, (1 - n [ v1-1) + (1 () 91-2) + ... + (1 - -20]]
18

---

and so:
(C((o))) - -(fo CE [0, - 0+)
(00 0¹) - Σ (1 7) (zt ÿt- + 0-) ]2 2
C {
,=1
C
(1 - 17) - 000² 2
(1 7)22 1) (21-) yc-s + 0r)2
Ci
(1- 8²) +
"1
(1 (1 - )))
Q2T(3/3)
T(1/B) + (1 11 122 11l
(2 11)
The last two equalities is due to the fact that for (xr v) ~ G¹
E x y + 07 = 0 a² 2²1(3/5)
E [xx
+ T(1/3
112 M4
T(1/3)
2
B.4 Nonlinear case
In this subsection, we conduct some qualitative analysis on the nonlinear case. The setting is the same as that in Section 3.2. We point out the differences compared with the linear case above: x Rd, E {0, 1} and
fo(x) = (0
ReLU (0 (0l
0
+ 014)
where (2) = I+e- is the sigmoid function; ReLU(z) = max(0, =) is the activation function for the hidden layer, which provides non-linearity the model; Ø| and Ø are the weights and biases of the hidden layer; 9!'l and gl4) are the weights and biases of the output layer.
To make things explicit, we still assume the online setting and set the loss function L(O) = (Jo
2) Assume after some iterations, g 141 + Ø 0 and g 12 0 (coordinate-wiss In
this situation, could see that if x is close to its mean( or μ2), the sign of ReLU (g!²lx (
will be the same as y. So So will become an optimal classifier if 0!') + too and o - - xo. We focus on 0 [1], using SGD:
aL 0 "00''
0."¹
aL. ag!l) (fo(x) - () ReLU (01 + g
Note that we drop the variable value in (-) to make the expression more compact.
Then we can analyze the phenomenon qualitatively: larger 2 will make convergence slower. The reason is that the larger Σ is, when x is drawn from M₁ (y 1), Øl + Ø < 0 is more likely to happen(i.e. straying far away from the mean), causing O to stop updating; what's worse, when is drawn from No (y 0), with larger probability 0(3 x + の 0 which will make 9! to go in the opposite direction. In summary, it is that makes the gradient noisy thus impacts the convergence rate.
B.5 From the Perspective of Feature Learning
In essence, the theoretical results [41] could also be interpreted as a proof of the theorem. [41] study the learning of two-layer ReLU neural network for k-class classification via stochastic gradient descent (SGD), assuming that each class corresponds / (atterns((ssriiuttionn))) with every two of the k x distributions of the input data are separated by a distance 8. Below is the main theorem in [41]:
19

---

Proposition Suppose some assumptions are satisfied, then for every > 0, there is M poly(k, 1.66,,,1111////. such that for every m Z M, after doing minibatch SGD with batch size B poly(k, L, 1/6, 1/e, log m) and learning rale y mopobytk11/6///.looo m] for = poly(k. 1, 1, 1/c, log m) iterations, with high probability, the generalization error(the probability that the model misclassifies) of the learned model is at most c.
Theoretical results above show that a larger 8 helps network to learn more efficiently. (In our case, ó can be roughly viewed the Mahalanobis distance between the two Gaussians, which is inverse proportion the variance.) Also, Appendix D.2 of [41] demonstrates an example very similar to ours in which the input data is drawn from Gaussian distributions with different means, indicating increasing variance of the Gaussian causes the test accuracy to decrease and takes longer time get good solution.
C Proof of Theorem 2
C.1 Setting
Use the same setting in Section B(linear case) except that
G' == (<=,(>,)))>>> 2. B(1, -,, 1- 1, No +
1+
N1, = p (x) + +1 p)u)
2
In other words, we modify the distribution of y instead of x this time.
C.2 Bounds with p
We're going to prove that higher p can make convergence faster, i.e. E [C(fo.. - [(fe)) )] is bounded by an decreasing function of p fixed).
Proof. The crucial part x y 0" p(x - (x) + 0*) (1 p) y 0*) and in fact
E|r + 0-1 - E2| - so- (x) 0- 60 0.
Similarly, we can get bounds with p (see Cj in Section B):
E - L(fo-)
n) 11
o' +0 $ C0(1-1)++((1-( n)) 1p Bl - sor (x) 0*
+(1-p).E]x -
< C2(1 1)) + (1 (1-1))))(C - -C4p)
where C22
C4 10, 0*1, C3 = C₁ E|r + 0%1, C4 C/ 60 0.
C.3 Nonlinear case
To see the impact of modifying y more clearly, we directly set p = and conduct a similar analysis as in Section B.4. Let's still focus on g and use the same assumptions in Section B.4, then we have:
@!! =0!" "0!1)
OL
0L 00 (Jo(x) for (1))ee (0 + 0
Note is replaced by fe- (x). For instance x is drawn from No 0) but strays far away from the μ1, causing g 0 (3] 0. In this situation for is likely regard as sample from Mr (i.e. So (x) close to thus making dL" to go in the right direction instead of the opposite. This explains why larger p make convergence faster.
20

---

D Proof of Theorem 3
We follow some proof steps in [5]. Let's begin by introducing some notations used in this section. Notation and Setup X' is the imput space, Ds and Dr are two distributions over X. Let y = {0. 1}. H. denotes a hypothesis class from X' to y. To simplify notations, Yh, E let cs f)
Ex (1((x) * f (x))), and (h, f) be empirical enror (cr f), (r f) similar).
Then introduce some concepts and lemmas, most of which are from [5].
Definition 6 (H--dvergence). The H-divergence between two distributions 'D and DD' is defined
as:
d7 (D, = 2 sup [P)p - Prp (1 (h)]l hE7
where I(h) (x: h(x) = 1}.
Definition 7 (Total Variation Distance). For two distributions and D', the total variation distance of them is defined as:
sup Drv(D,D) D')
(Prp(A) (A)
ACF
where F denotes the collection of all events in the probability space.
Lemma For two distributions and D', by definition it's easy to see that:
2 (D,(DDD)) s DTv(D,D)
2
Definition (Symmetric Difference Hypothesis Space).
HAH {g : {0,1)l h(x) (xxx) Yh, H}
@ denotes the XOR operation.
Lemma 2.
1 Wh, E Hiles - hh, - dHaP DT) 2
- er
Proof. only need note that h(x) + hx() = [1(x) N'(x)), so
sup ge HAH PrDs [(g)] - Prp, [T()]] = sup A.EEH
les (h, eT (h, h!)l
this is done by definition.
With above notations we can derive general proposition related with Theorem
Proposition Assumimg H a hypothesis class from to y and d &' E H. we have:
ExwDr [[(((x), Φ' (x)) s Ex-Ds (((0(x),'(xx)) + Div where loss function is ((j, y) = 10 # v).
Proof. using the lemmas above.
( Φ¹) < Es (1), x') + eT (0. )¹) es (Ø. Φ')
1
<s (0) Φ') + dHan (Ds, Dr)
2
$ cs (p. Φ') + (ry(Ds,)) Dr)
21

---

To obtain some useful inequalities of generalization bound, we need to introduce the Rademacher complexity.
Definition 9 (Rademacher complexity of function class) Given sample of points S {z1, 22,,.. 2m C Z, and considering a function class of real-valued functions funciioo over Z, the empirical Rademacher complexity of F given S is defined as:
1 - Es RF(F)
(z) JEF
sup
m
where are independent and identically distributed Rademacher random variables. In other words, for i 1,2..... m, the probability that a; =1 equal the probability that a; = -1, and both are z. Further, let P be probability distribution over Z. The Rademacher complexity of the function class with respect to P for sample size m is:
Rpm (F) Pm [Rs(F)
Lemma (Generalization bound with Rademacher complexity). Let F be family of loss functions F = {(x, v) + E((x, y), h) h H) with {((z, h) E [0, 1] all (I, y) and h. Then, with probability - 6, the generalization gap is
log log(2/5) (2/5)
- (1)) - 2 (F) + 3//
2n
for all h E H and samples U of size n.
The proof of this classical result could be found in most machine learning textbooks, like [46]. Since e(-) is 1-Lipschitz, we can derive that Rys (E H) < Ryg (H).
Theorem 3 involves the representation distance, below prove the triangle inequality for representa- tion distance.
Lemma 4 (Triangle inequality for Representation distance). for any functions Øs, th,, dU and distribution D, we have:
DRepl0s + Φr; D) DRop (Øs - -u; D) - DRop (ou + +r; D)
Proof. Let's denote:
dj (os - ØU; D)
d2 DRop (&u - 0T; D)
By definition:
dj inf W. bR
E Ee-De(Wigs(()) b1, (z()))
inf d2 Wg bye
B (x) + b2 (x()))
We need show:
(os + ΦT; D) s d1 + d2
Consider the composition of the transformations:
(
Wi.b.
(ar)
Wa,ba (T(())
The combined transformation can be written as:
(w)) bi) b2 W2W1ds(x) + Wab1 + b2
Using the properties of the loss function f, we can write:
((W2W10((2 W2b1 + (r()) < ((Wids(z) + b1, (z)) ((W2du(2) + b2, or(x))
22

---

This inequality holds for the reason that it can only break when the two items at right-hand side are both O, in which the left side is also due to rule of composition. Thus the inequality holds for all and W1, b1, W2, b2-
Taking expectations over D:
W (1 th2, or & Be (uw. dr(2z))
Taking the infimum over Wi, bi and W2, b2:
inf Wi.bi.Wa..
Ex~ ( + W2b1 + b2, xr(2)) I d1 + d2
Since the left-hand side an upper bound for DRep (Øs - Φr; D), we have:
DRep(ds + ΦT; D) S di + d2
Therefore, the triangle inequality holds for the metric DRep-
Now we are ready to prove Theorem
Proof. For arbitrary W, b, with probability 1 1 6:
(2g(2/))
Deep (0o - xX) S ex (Wos + b,) (W oo bb,"" 2D (0) + +3/
2 2
Taking the infimum over W, b:
log (2g(2/))
DRep + xx; < DRep + ;x" + 29 (8)
(0) + +3/
2 Dxl
Using the triangle inequality for representation distance:
DRep(0g - x Dx) S Do + th: Dx) DRep(1) -00 (x)
For arbitrary W,b. b, using the proposition above,
Deep (Do - vi; Dx)= I <Dx (W Φo + b, S 65x (W + +b, + Drv(Sx, Dx)
Taking the infimum over W, b:
Deep Φ t/; Dx) & DRep( (80 v; + Div(Sx, Dx)
Combining the above results, we have:
log(2/8)
DRop( (20 -> x6;X)) S DRop(1 Dxx (Rop - 1x, + 22 3
+ Dx
+ DDrv(xx,Dr
Impact of initialization In this part we leverage the Neural Tangent Kernel (NTK) framework [29] to deduce that neural network initialized closer target function f* converges faster during training. Under the NTK regime, neural networks exhibit linearized training dynamics, allowing us to predict how the network's output evolves during training.
In the context of supervised learning, consider a neural network with parameters and output function f(; 0). where is the input. The goal is to approximate target function f by minimizing loss function, typically the mean squared error (MSE):
n
1(O) = Σ (sx 0) y.)²
23

---

where {(24, yo) is the training data and y: = (x,).
Under gradient descent with learning 7/, the parameter updates
0.+1 0, VoL(0)
In the NTK regime, where the network width tends to infinity, the network's 's output evolves linearly with respect to the parameters around initialization 0g [38]. We can approximate:
f(z; 0) = f(;; Bb) + (of(2; (00 0o)
This linearization allows to express the evolution of the network's output
n
- =())= -1 ΣΣ K(o,,:)(((aa)) uz)
where f+(x) = fx; 0.) the network output at time and <(x,x')= Vos(x; Vol the Neural Tangent Kernel.
In continuous time (gradient flow), the training dynamics can be described by differential equation:
df((z)
a
dt Σ
(((ra))((((a))))> -v))
Vectorizing over all training inputs, we have:
df,
((ff-- y)
dt
where f, [fe( fe(22)...., (zx.) y = [v1, ....... yn and K is the NTK matrix with entries Kij K 2z), This differential equation has the solution:
f, y Kr (fo y)
This equation shows that the error y decays exponentially over time, with the rate governed by the NTK matrix K. Since K is symmetric, we can decompose K into its eigenvalues {A₂) } and corresponding eigenvectors {v,}:
K= ΣnNe
the solution becomes:
f. y e -e-sttyyy
where c, v, y)). The rate of convergence for each component of the error is proportional to the corresponding eigenvalue X, of the NTK. Larger eigenvalues lead to faster decay. Initial error matters: the initial error fo y scales amplitude of the exponential terms. smaller initial error means the network starts closer to the target function, reducing the time required for the error to decay to a specific threshold.
Suppose we want the error to be less than e:
||f, - -yyl S
Using the solution:
24

---

ff.
yll S l|e
Amin [|]|5o yll
where Amia is the smallest (positive) eigenvalue of K. Solving for time t:
[ - yll
In
Amin
By leveraging the NTK framework, we' ve shown that better initialization network's s initial output function is closer to the target function f*) leads to faster convergence during training. This is because the training dynamics under NTK are linear, and the exponential error decay directly influenced by the magnitude of the initial error. Analysis above implies that we can train network that is closer to the target function, will converge faster to the target function.
the neural
D.1 Relation between data distribution and Rademacher complexity
Now let's look the Rademacher complexity appeared above more earefully. Let I-L-ipschitz positive homogeneous activation o; be given, and
H = (x (x+ 0r (WLoT-11 (1...) (Wix).... [Will B,
Then using Theorem in [23], for samples of size m we have bound for the empirical Rademacher complexity:
22 B² 1XI|F
RH) - m
where X E _dxm is the input data matrix, and || ||r denotes the Frobenius norm.
further assume that the data are drawn from the distribution G (stated in Section 3.2) with covariance E (for simplicity let μl = M. f2 μo), then we can bound the Rademacher complexity of with respect to G:
Rg.m (H) ES [Rs(H))
< 1 V2L 2
BL Esnam ||1||F
22 In2 EL
Er 1 (1 dm
dm
dmp2 2Σ
dm
M
/r
m
2
The right part of the last inequality is an increasing function with respect to E, and r(-) denotes gamma function, .(....) is the Kummer's confluent hypergeometric function, given by:
(n)) n
M(a, b,,
b(m)m! = Fi;
n=
where:
a (0) 1 (n) ala + 1) .... ..
(a
1),
the rising factorial.
Remark. Usually, the Rademacher complexity Rg(E o F) could be bounded by of RsFF). For example, if é is L-lipschitz, then Rs F) L- R--ss(F That's why directly compute the Rademacher complexity of H instead of & H.
E
Explanation of Rescaling Samples
Dataset distillation seeks to create condensed dataset that allows models to achieve performance comparable to those trained on the full dataset, but with fewer training steps. In this section, we will demonstrate that rescaling the variance of Gaussian distributions, as defined in Definition 3, does not affect the optimal performance of models trained on these rescaled data.
25

---

Proof. Consider two Gaussian distributions No(m1 and Mi (1 I²I) with means μr = 1 and H2 = 2, and variance o². We define the bimodal mixture distribution G (Gx, Gy) such that (x, y) is sampled according to:
x (1 v) Xn X1, with y Bernoull((0.5,, Xg No. x1 ~ Ni.
The decision rule for optimal classification is determined by the likelihood ratio test. For given sample x, the log-likelihood ratio (x) is given by:
P(x 1) P(x = 0)
( log
Since Xy and X1 are drawn from Gaussian distributions, their probability density functions are:
1
1 111
P(x y 0) (2To2)d/2 2d/2
exp
2n²
x
1
P(x y (2To2)a/3 exp
20 ||x
112112
Substituting these into the log-likelihood ratio, we have:
(2F02)a73 exp
20 x - M2ll3
A(x) log
(2ro exp
20
1
x
111 122)
Simplifying, we obtain:
1
A(x)
|x 112|12
+
||x - 1/1112
20
20
Further simplification gives:
A(x)
20 (1|x 11112
||x 12213)
Since the optimal decision threshold for balanced classes (i.e., P(u = 1) P(y = 0) 0.5) is (x) = 0, we set:
xx
N1 11 - ||x H2ll 20211=000 0.
=
Expanding and rearranging, we derive:
|x| - 2p1x + 2r ||x|1 2 + +2uzx 11 = 0.
This simplifies to:
2( 11)x+ + (Nî (22)
2( 41)x x = 2/ n
x 12 μ²
2(122 - 11)
Solving yields;
x U2 + #1
2
Therefore, the optimal decision boundary is 1.5, which is independent of the variance o². This completes the proof.
Regarding efficiency, utilizing scaled data to train similar models with fewer training steps is proven in Section B. Additionally, rescaling each Gaussian distribution preserves their means, which aligns with the objectives of conventional distribution matching- dataset distillation methods. These methods aim to distill data while maintaining the distributional properties, specifically their means.
26

---

F Detailed Methodology of RELA-D
Recall that Φg Rd - R". We then introduce a transport matrix W E R and define the combined model as ( R During the training phase, the parameters W and 0 are jointly optimized. However, as the dimension increases, the computational complexity of the optimization grows rapidly. To address this issue, we propose reducing the dimensionality of the target matrix Ry from m7 to n, where n < m:
My -y=Vi Vn
Ry
-
S.L. Ry Ryl
(
VAV", V, V(a n),
(10)
where j and a denote the mean and standard deviation, respectively, of each column in the Ry. Practically, we use batch PCA to perform the computation shown in (10), as illustrated in L.
F1 Proof for Ideal Properties of Prior Models
We aim demonstrate that modern deep learning methods effectively train models to as robust prior models by extracting sufficient information from samples as representations.
Therefore, poist the existence of prior model & capable of losslessly extracting the information of samples Dx when trained using the InfoNCE loss [47], a method prevalently employed in contemporary deep learning algorithms [12].
Proof. To demonstrate that an encoder & trained with the InfoNCE loss preserves all information from the input data Dx, proceed follows.
Definitions and Setup
Let X' denote the input data space with data distribution Pdam (z) Let Ppos (T, x+ denote the dis- tribution of positive pairs, typically generated via data augmentation. The encoder & X - Rd maps inputs d-dimensional representations. The similarity function q IRd I2d - R (e.g., dot product) measures similarity between representations. Given positive pair (æ,x*) let {z, fi=l be N negative samples drawn i.i.d. from Pána(()), The InfoNCE loss defined as:
((((()) 11/T
LIGONNE (E. 9) - -B log
(E((())E(())) 1//T
where 7 > 0 is a temperature parameter.
2. InfoNCE Mutual Information Lower Bound
The InfoNCE objective serves lower bound to the mutual information between representations of positive pairs:
T(((X):E(xt)) 2 J(E.q)
where
J(E.9) (e poppe ((E()),E(xt)) - log
Σ ( 11/T
As the number of negative samples N oo, this bound becomes tight, approaching the true mutual information ((E(X); ((X(X))((())))) 3. Optimal Encoder and Similarity Function Let (g*, "=""") denote the optimal encoder and similarity function that maximize J[E, 4). Under the assumption of infinite negative samples and an expressive similarity function, the optimal similarity satisfies:
Ppas (a, xx+
( (x), = llog
Paata (2 Pasta (x+ +C
where C' is a constant independent of (x. xt)) This aligns 9 with the pointwise mutual information (PMI) between and xt 4. Injectivity through Mutual Information Maximization Maximizing I(E(X);E(X*) encourages the encoder & to capture as much information about X as possible. To ensure injectivity:
Sufficient Dimensionality: The representation dimension d must be at least as large as the intrinsic dimensionality of X.
27

---

Table 4: Notations used in this section.
Notation
Meaning
U1, M2
current concerned samples
unspecified samples
u.. uo
samples from online branch
samples from unspecified target branch
ug, t
samples from weight-sharing target branch
ug. bod
samples from stop-gradient target branch
U2 2m
samples from momentum encoder target branch
V
unspecified sample set sample set current batch sample set memory bank sample set of all previous samples
banch
bank
Voc
Expressive Architecture:
The encoder & should be sufficiently expressive, potentially utilizing architectural constraints (e.g.. invertible networks) to promote injectivity.
Under these conditions, maximizing mutual information implies that 5 approximates an injective mapping on the support of (z), i.e., (1(x) {'(x') = x = x' almost surely. 5. Existence of the Inverse Mapping Given that 5* is injective, there exists deterministic inverse mapping IRd X such that:
(x))
for all E X
This mapping g be constructed the inverse of E- on its image:
g(2) 5- (2) where (x)
G Analysis of Different Self-Supervised Learning Methods
We refer to the primary thcoretical results for existing self-supervised learning methods as presented in [59], where all methods unified under simple and cohesive framework, detailed below. Specifically, UniGrad [59] demonstrates that most self-supervised learning methods can unified by analyzing their gradients.
G.1 A Unified Framework for SSL
typical self-supervised learning framework employs a siamese network with two branches: an online branch and target branch. The target branch serves the training target for the online branch.
Given an input image x, two augmented views I and 12 are generated, serving as inputs for the two branches. The encoder f(.) extracts representations u, _ fxx;) for = 1,2 from these views.
Table 4 details the notations used. #1 and u2 denote the current training samples, while v denotes unspecified samples. uf and v" are representations from the online branch. Three types of target branches are widely used: I) weight-sharing with the online branch (uz and v²); 2) weight-sharing but detached from gradient back-proppiee propagation (uğ and ) 3) a momentum encoder updated from the online branch (u)r and )² If unspecified, u/ and o' are used. A symmetric loss is applied to the two augmented views, as described in [14].
/ represents the sample set in the current training step. Methods vary in constructing this set: Vhatch includes all samples from the current batch, Vbank uses memory bank storing previous samples, and V.o includes all previous samples, potentially larger than memory bank.
G.2 Contrastive Learning Methods
Contrastive learning relies on negative samples to prevent representational collapse and enhance performance. Positive samples are derived from different views of the same image, while negative
28

---

samples come from other images. The goal is to attract positive pairs and repel negative pairs, typically using the InfoNCE loss [47]:
exp (cos(19, ub)/T) Ev' Ev exp (cos(uf, of)/
L E 01:
log
(11)
where cos(-) denotes cosine similarity, and is the temperature hyper- paramme.r.eeter.. This formulation can be adapted for various methods, discussed below.
MoCo [25, 13J. MoCo uses a momentum encoder for the target branch and a memory bank for storing previous representations. Negative samples drawn from this memory bank. The gradient for sample uf is:
ØL dus
1 TN
U2
8y2 m2
(12)
1 EVaank
exp (cos(uf, uf ""')"" //r)
where 8y Eeank exp (cos(uf. (2. )/T) and N is the number of samples in the batch.
SimCLR [12]. SimCLR shares weights between the target and online branches and does not stop back-pre pre-ppropagation. It uses all representations from other images in the batch as negative samples. The gradient is:
aL
1
us
+
Sper
Oug
TN
1 E Vbatch iag
Σ
l00
(13)
+
Un +
TN
t E Vhatch Juj
to
exp (cos(u" (cos()".w()/z)
If the gradient through the target branch is stopped, the where 4, = Σx Vuntch loe exp (cos( 1/5) second term vanishes.
Unified Gradient. The gradient for these methods can be unified as:
ØL = TN
ui
Σ sou V'EV
(14)
Oug
comprising a weighted sum of positive and negative samples. The term -už pulls positive samples together, while Evey 8. pushes negative samples apart. The main difference between methods lies in the target branch used and the construction of the contrastive sample set V.
G.3 Asymmetric Network Methods
Asymmetric network methods learn representations by maximizing the similarity of positive pairs without using negative samples. These methods require symmetry-breaking network designs to avoid representational collapse. A predictor h(-) is appended after the online branch, and the gradient to the target branch is stopped. The objective function is:
L
E 01.42
cos(h(uf)) ub)
(15)
Relation BYOL [30]. BYOL uses a momentum encoder for the target branch, i.e., w/ uz!" in (q5()
Relation to Simsiam [14]. Simsiam shows that momentum encoder is unnecessary and only applies the stop-gradient operation to the target branch, i.e., n/z ug in (q5).
Unified Gradient. Despite the performance of asymmetrie network methods, the avoidance of collapse solutions is not well understood. DirectPred [60] explores this by studying training dynamics and proposes analytical solution for the predictor lt()).
DirectPred formulates the predictor as h(v)) = Wav, with W, calculated based on the correlation matrix (ve"). ). The correlation matrix, F, is computed as the moving average for each batch:
29

---

F E po where Pv is the moving average weight. Decomposing F into eigenvalues Ap and eigenvectors U, W is:
W = UAAU*, A, = x + exmaz I,
(16)
where Amaz is the eigenvalue of F and is cyyper-paammen to boost small eigenvalues.
DirectPred also derives the gradient:
aL
Σ (pous (oougTV")0") v2)o")
W u2 +
(17)
Oug
Whuul 2N
V"EVo
where - W. u. and Evevv (pouf vn)e"" act as positive and negative gradients respectively, and ui WTW is balance factor.
"f (F
Though negative samples are absent in the loss function, they emerge from the predictor network's optimization. The eigenspace of the predictor W₁, aligns with the feature correlation matrix F. encoding its information. During propagappa,,,onn this encoded information functions negative gradient, influencing the optimization direction.
G.4 Feature Decorrelation Methods
Feature decorrelation methods have recently emerged as a novel approach in self-supervised learning. These methods aim to reduce redundancy among different feature dimensions prevent collapse. Various loss functions have been proposed for this purpose. We examine their relations below.
Relation to Barlow Twins [69]. Barlow Twins employs the following loss function:
C Σ (W - - + Eww, 1=1 Jipé
C
L
(18)
where W= = +/4 Eng pff wew.eeeee...... Evatec ufu5" a cross-correlation matrix, denotes the number of feature dimensions, and a balancing hyper-paran arper-paramete.. The diagonal elements of W are encouraged to be close to while the off-diagonal elements are forced towards 0.
Despite appearing different, Eq. (18) operates similarly to previous methods from gradient perspec- tive, calculated as:
aL 2 Oup
Aug +
Σ
US
of
(19)
N
where A - (1- A) and (Waiag)is = 6ey Wij is the diagonal matrix of W. Barlow Twins applies batch normalization instead of &2 normalization to the representation
Relation VICReg [3]. VICReg modifies Barlow Twins with the following loss function:
A
L
Σ
W
N
oq uug Mantch
(20)
X2
ma std(of))),
where W" ='= Nー E (vf (f T is the covariance matrix of the same view, std(
denotes the standard deviation the i-th channel v, 7 is constant target value, and λ X2 are balancing weights.
The gradient derived as follows:
aL
2
ug
+X
Σ
Toll
To N
J0
Oup
N
Vintel
(21)
2X
+
Bug
N
30

---

where 7 ū is the de-centered sample, A Az diag(1(7 std( > 0) Ø std( Here, diag(r) is matrix with the vector 2 on its di- agonal, 1(+) is the indicator function, and denotes element-wiss division.
2 N <N-11 and B
N cX(N-1)
(2 W diag
VICReg does not normalize vt instead, it uses de-centering and a standard deviation term in the loss function.
Unified Gradient. Given the equivalence of v" and vo, the gradient for feature decorrelation methods can be unified as:
aL 2 N
u 2
u2 +
Σ
Vi
(22)
Oug
N
1" Vbatch
where -us is the positive gradient, and Σ E
4o g0 Ui is the negative gradient. is
N=
balancing factor. The difference between methods lies the subscript for the negative coefficient. Feature decorrelation methods function similarly to other self-supervised methods, with the positive and negative gradients derived from the diagonal and off-diagonal elements of the correlation matrix.
H Budget of RELA for Data Synthesis
While training on the distilled dataset is both efficient and effective, the distillation process in optimization-based approaches [11, 72, 65] is computationally intensive [18, 56), often exceeding the computational load of training on the full dataset.
In contrast, the synthetic data generated by our RELA framework requires a budget less than that of training for single epoch (c.f. Section 4.1). This because the synthesis budget of our RELA is equivalent to performing inference over the entire dataset Dx Consequently, this computational
expense negligible given that training epochs typically number around 100.
I RELA in Labeled Dataset Distillation and Human-Supervised Learning
We apply our RELA in labeled dataset distillation and human-supervised learning.
I.1 Experimental Setup
Datasets and Neural Network Architectures. We conduct experiments on datasets of varying scales and resolutions.
Small-scale: We evaluate on CIFAR-10 10 (32 x 32) [35] and CIFAR-100 (32 x 32) [34].
Large-scale: We utilize Imag- (64 64) [36] and ImageNet- (224 224) [20].
Consistent with previous works on dataset distillation [67, 73, 24], we use ConvNet [24] and ResNet- [18,50} [26] as backbone networks across all datasets. Specifically, Conv-3 is employed for 1IFAR-10/100 and Conv-4 for Tiny-ImageNet and ImageNet-IK.
We compare our method with several SOTA distillation methods capable of scaling to large high-resolution datasets, including G- VBBB [53], L [67], and RDED [56]. To the best of our knowledge, L, G-VBSM, and RDED are the only published works that efficiently scale to datasets of any size, making them our closest baselines. All distilled datasets synthesized from these baselines undergo the same post-training process. Results are reported in Table 5. For our RELA, the prior models used for distillation are identical to the pre-trained models employed the baseline methods.
Baselines.
1.2 Main Results
Table demonstrates the superiority of our REL.A, despite incorporating zero-cost sample synthesis process.
31

---

Table 5: Comparison with SOTA Baseline Dataset Distillation Methods. In the table, bold indicates the hest result. IPC refers the numher of Images Per Class for distilled datasets.
ConvNet
Reshiet- 18
ResNiet-S0
Damser IPC
G-VBSM
SRe2I.
ROED
REl (Qurs)
G-YBSM
SReZI
RDED
RELA (Chrra
G-VBSM
SReZ1
RDED
RELA (Ours)
21:2+002
22.2
40.3
170
35.0
202
31.7
71.
61.3
13441003
129
10
218
32.5
11.5
04
0.1
31.7
12. 0.61881
4.
778
CF-100
10
R8.7
942
0.3
473
51.3+ 0.1
470
42.7
+-0
0.3
648
47.5 0.5
+00
66.2
0.1
50
53.8
0.4
52.2
1
55.
02330222
56.3
1 0.3
600 1
0.1
57.4
0.02
6..0
68.8
168:10
62.2
12.200.33
60.6
1
08.8
1.3
69.6
0.2
8.6
0.
11.
+
1700004
27.0
+ 0.3
9. 0.1
135++02
154+
24.2 1.4
3.2 0.2
123
148 +5
212 1.1
TIN
3391022
34,5
42,9 199100
0.3
43.6
4,4
5588
10.1
8&.
226
127
0.3
204
32.
02
31.1
4:.1
48.5
383
4.2
37.8
04
96.0
0.3
88.8
0.6
52,9 +
51.6+ 0.2
49.5
02
55.3
0.2
61.3+0.1
60.3
5844
62.3 +.1+
61.3 0.1
J RELA Algorithm
Algorithm Adaptive Loss Weighting Algorithm for RELA and Self-Supervised Learning
Require: Number of training steps T, Initial f, 2.0, Initial Ef 1.0, Initial À
1: for each training step / to T do
2: if X then
3:
le < Value of CRILA [Retrieve the current loss value from the optimization process)
4:
L + 0.999 f 0.001 x & (Calculate the short-term loss]
5:
L. f 0.99 x E, + 0.01 x & [Calculate the long-term loss]
6:
end if
7:
if exp( ma f- 0}) 2 0.995 then
8:
0 [RELA learning is converged and over]
9:
end if
t- X CRELA + (1 X) LS... (Control the RELA and the SSL states using A)
11: end for
In essence, this algorithm is designed detect the convergence of RELA. Upon convergence, it transitions to the original algorithm for self-supervised learning. This implicitly segments the learning procedure into two phases: the fast stage (ReLA) and the slow stage (SSL). Moreover, Figure depicts the dynamics the training process.
Accuracy (%)
4astrre Camter (00
wwwe A
Top-1
Rel.A Tralados
Tralalng
Fae
14
Trilaing
2008
axe
2009
4008
Training Steps
Training Steps
(b) RELA with Rand. as prior model Figure Comparison of training dynamics between RELA and the original (Orig.) BYOL algorithm.
(a) RELA with CF10-T as prior model
K Experimental Details
K.1 Detailed Setup for Experiments in Section 3.2
We show the detailed setup for the experiments in Section 3.2 in Tables 6 and 7.
32

---

Table 6: Architecture of the simple neural network model.
Layer
Type
Input Units
Output Units
Input Layer
2
Hidden Layer
Fully Connected (Linear)
2
Activation (ReLU)
Output Layer
Fully Connected (liinerr)
50
Activation (Sigmoid)
Loss Function
Mean Squared Error (MSE)
Table 7: Optimization parameters for the simple neural network model.
Parameter
Value
Training Steps
1000
Batch Size
Optimizer
Stochastic Gradient Descent (SGD)
Learning Rate
0.002
Momentum
0.98
K.2 Detailed Setup for Experiments in Section
Training details for self-supervised learning methods.
We show the details of training in Table 8.
Table Training parameters and optimizer settings for self-supervised learning methods.
Parameter
Value
Epochs 100
Optimizer
AdamW
Learning Rate
0.001
Weight Decay 0.01
Linear evaluation details.
We show the details of training the linear model in Table 9.
L Batch PCA Reduction
To begin, we provide full and rigorous mathematical framework for Principal Component Analysis (PCA).
L.1 Principal Component Analysis
Given data matrix Y E Raxd and the desired number of components k, where k: S d, PCA aims to extract reduced data matrix Yretuce E IRa*k that retains the maximum variance from the original dataset.
First, the data needs to be centered by subtracting the mean each feature. This can be mathematically represented as:
Yenttered = Y 1
where 1, is column vector of ones with length Next, the covariance matrix of the centered data computed:
1
C
Y
centered Y Centered
33

---

Table 9: Linear evaluation parameters for various datascts.
Dataset
Batch Size
Linear Model
Loss Function
CIFAR-10
128
nn mm.LLineaffeaure_dii,, 10)
CrossEntrrpyl.osss))
CIFAR-100
128
nn. LaLLeaaffeatureddiii 100)
CrossEntropLoossO)
Tiny-Image ImageNet
128
nn Lineanafecauure_diii 200)
CrossEntropyLoss))
ImageNct-IK
1024
nn.1 mn.Linear(feature_dnr 1000)
CrossEntropyl.com Loss()
Optimizer
Adam (learning rate: 3c-4)
To identify the principal components, eigendecomposition is performed on the covariance matrix:
C VAV*
where V is the matrix of eigenvectors and A is a diagonal matrix of eigenvalues. The eigenvectors are then sorted in descending order based on their corresponding eigenvalues. The top k eigenvectors selected to form the projection matrix:
= [V1, V2..... Va]
where v, represents the i-th eigenvector. Finally, the centered data projected onto the new subspace:
Y
reduced
Y centered W
The resulting reduced data matrix Y reducee contains the principal components of the original data. Each principal component is a linear combination of the original features, designed to capture as much variance as possible in the reduced space.
Algorithm 2 Batch PCA Reduction
Require: Data matrix E Radd Number of components k
Ensure: Reduced data matrix reduced g
1: n + number of rows in Y
2: <- BATCHSIZE (Pre-set Batch }
3: μ + Σ Y,, (Compute the mean of Y)
4: Y Ceentered <- Y Iu. (Center the data)
5: Σ + -dxx { /nitialize the covariance matrix)
6: for 0 n step do
7: j +- -minni + m. 11)
8:
B centered J.
9: Σ E+ B B {Update the covariance matrix
10: end for
11: E + n-1 (Normalize the covariance matrix)
12: V eigenvectors of Σ corresponding to the largest & eigenvalues
13: Yreduced < 0дх* [Iiitialize the reduced data matrix)
14: for = to n step do
15: j + min(i + m. n)
16: B< Y centered 18 j.
17: Y Yreduced [:: :. BV
end for
19:
return Y reduceed
Then we prove that the batch PCA we utilized here is exactly same to the standard PCA.
Proof. To prove that the Batch PCA algorithm achieves the same result the standard PCA, need to show that the covariance matrix and the reduced data matrix computed by the Batch PCA are equivalent to those computed by the standard PCA.
First, let's show that the covariance matrix Σ computed in the Batch PCA is equivalent to the covariance matrix C in the standard PCA.
34

---

In the standard PCA, the covariance matrix is computed as:
C
centered
Y centered
In the Batch PCA, the covariance matrix computed as:
Σ
BTB
B B B
+... +B Bb-1)
Y
centered
Y
centered
where B; is the i-th batch of the centered data matrix Y oemtered, and b is the number of batches.
Therefore, the covariance matrix computed by the Batch PCA is equivalent to the covariance matrix computed by the standard PCA.
Next, let's show that the reduced data matrix Y reduced computed in the Batch PCA is equivalent to the one computed the standard PCA.
In the standard PCA, the reduced data matrix is computed as:
Y reduced
Y centered W
where W is the matrix of the top eigenvectors the covariance matrix. In the Batch PCA, the reduced data matrix is computed as:
Bo B V
roduced
BB-1
Y Ceettered (0 m, |]
Y Ventered m 2m, .]
Y ceenereall 1) 1)m n.
Y centered
where V is the matrix of the top eigenvectors of the covariance matrix E, and is the batch size.
Since we have shown that the covariance matrices C and Σ are equivalent, their eigenvectors W and V are also equivalent. Therefore, the reduced data matrix computed by the Batch PCA is equivalent the one computed by the standard PCA.
In conclusion, we have proven that the Batch PCA algorithm achieves the same result as the standard PCA by showing that the covariance matrix and the reduced data matrix computed by the Batch PCA are equivalent to those computed by the standard PCA.
M Analyze the Practical Data Augmentations
Here we presents a rigorous mathematical proof demonstrating that higher data augmentation in- tensity leads to increased overlap between samples from different classes. By modeling the sample distributions and augmentation process, show that the variance of the augmented distributions increases with augmentation strength, resulting in wider and more overlapping distributions. We provide an exact calculation of the intersection point and approximate the overlap area using the cumulative distribution function of the standard normal distribution. Our theoretical analysis confirms the positive correlation between data augmentation intensity and inter-class overlap rate.
M.1 Assumptions and Definitions
35

---

Targeting 'Dug'
Human-L.abeled
Self-Generated
Similar Views. Different Targets. What should Iearn's
Dafa-Augmented Views
Or
Using Other View
Machine Learner
Targeting "Cat'
Figure 6: A scenario of similar views targeting different targets. In some cases, we may randomly crop two similar views from the original images augmentations and input them into the machine learning model. This lead confusion in the model due the differing targets assigned by humans generated autonomously.
There are two classes C1 and Cz. with samples drawn from probability distributions P1 and Pz, respectively. The data augmentation intensity is denoted by a.
Assumption 1
Definition 10 (Data Distribution and Augmentation). The samples are satisfied X1 Pi. X2 P2- The augmented samples are represented as Xi X1 + and X; X2 + where с denotes the augmentation perturbation following the distribution Qa. The variance of Qa is proportional a.
M.2 Augmented Distributions
Let Pi,a and P2.a denote the augmented sample distributions:
(a,)(=')= = (Pi * Qa)(z') P2)((x)) = (P2 * Qx)
(23)
(24)
where represents the convolution operation.
Variance of Augmented Distributions Let o² be the variance of Qa, with o = ka, where is constant, The variances of the augmented distributions are:
2. = =0p, + 0- = =o², + ka
(25)
2 p., + 0² 0z ka
(26)
Overlap Rate Definition
Let R denote the overlap region. The overlap rate O(a) is defined as:
O(a) = ((o)=/[Paoa
Pract'd + +I P2o(x') dx'
(27)
M.3 Inereased Variance Leads to Increased Overlap
As increases, 03 increases, resulting in larger variances of Pı.a and Pa.a- This makes the distributions wider and more dispersed, increasing the overlap region.
Specific Derivation for One-Dimensional Gaussian Distributions Gaussian distributions:
Assume Pi and Pa are two
Pi (x) = NN ) Pa(x)) NN22 oz2)
(28)
(29)
36

---

The augmented distributions are:
(1,a(x)) N(1, of ka)
(30)
P2 (x²) (u2, opp ka)
(31)
Exact Calculation of Intersection Point and solving for x:
Equating the two Gaussian probability density functions
(- (x: 1212 2 + ka)
(- (x 2(077, ko))
p22
exp
exp
(32)
2 + ka)
2
(o%, + kox)
Simplifying the equation yields an analytical solution for the intersection point. To simplify the analysis, we assume oi, = in which case the intersection point is (M + #2)/2.
Area of Overlap Region Let Aμ
|M μ2). The overlap region be represented using the cumulative distribution function (CDF) of the standard normal distribution, denoted as ((2):
1 + erf
z V2
]
((2)
(33)
2
The variance of the augmented distributions is o oî, ka. Therefore, the approximate area of the overlap region is:
Au
(aa) 24h
-1
(34)
12(0% + kk))
M.4 Conclusion
Theorem As the data augmentation intensity a increases, the overlap rate (ax) between samples from different classes increases.
Proof. Increasing a leads to an increase in the variance of the sample distributions, making them wider and more likely to overlap.
Specifically, increasing increases o + ka in the denominator, thereby decreasing the argument of ((). Since (2) increases decreases for 0, ((a) increases increases.
In summary, we have rigorously proven the positive correlation between data augmentation intensity and inter-class overlap rate from statistical and probabilistic perspective.
M.5 Empirical Analysis for Real-world Datasets
Beyond the theoretical analysis for simple case, we also provide empirical analysis for four real- world datasets under different intensities of data augmentation. We utilize TSNE [61] to visualize the (augmented) samples in Figure 7. All the results demonstrate that:
1. minscale increases from 0.02 to 0.5, the data points gradually change from being tightly clustered to more dispersed. This indicates that higher data augmentation intensity expands the range of sample distribution;
2. when minscale=0.02, the data points of the two classes exhibit significant overlap, and the boundary becomes blurred. In contrast, larger minscale values such and 0.5 allow for better separation between classes;
37

---

(a) Setting Minscale 0.02
(b) Setting Minscale 0.04
(c) Setting Minscale 0.08
(d) Setting Minscale 0.2
(e) Setting Minscale
Without using Augmentation
Figure 7: Visualization of samples with varying levels of data augmentation intensity. We visualize the sam- ples of classes from CIFAR-IO 10 in 2 2-dimensional space, respectively setting minscale in RandomResizeCrop f0.02, 0.04, 0.08, 0.2, 0.5) and without data augmentation.
3. when data augmentation is not used (the last figure), there is a clear gap between the two classes, and the data points within each class are very compact. This suggests that the feature distribution of the original samples is relatively concentrated.
38

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We show them in Section and our Abstract.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We show them Section 6.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [No]
39

---

Justification:
Most of them are proven in Appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We show the details of techniques in Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
40

---

Answer:
[Yes]
Justification:
We use the public datasets.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. cc/ Stapcc/guiiess/Coeeeee
wwwsssssssooooooooooo for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
/ //iips.cooo wttpccccppppcpeeeecececccccccccc hut.a//gggiideeseeSuuuuiiiii /Codeeeooiissseeeeee wwtisubmmssiooooo sswssssoiooooooooo onPolicy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We show the experimental details in Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We show them in Section 5.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
41

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
show in tables or
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [No]
Justification:
We provide most of them in Appendix.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: http://eurrrs.ccccc http:p/cccc...hhccscsiiieeeeeeee
Answer: [Yes]
Justification: The research presented in the paper adheres to all aspects the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative socictal impacts of the work performed?
Answer: [No]
Justification: The paper does not comprehensively discuss the potential societal impacts, both positive and negative, of the work performed.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
42

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [No]
Justification: The paper does not describe any safeguards for the responsible release of data or models that carry high risk for misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All assets utilized in this paper, including code, data, and models, have been properly attributed to their respective creators or original owners.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version the asset used and, possible, include URI
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswi twwode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
43

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not introduce any new assets; therefore, the question of documentation is not applicable.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve any crowdsourcing experiments research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to NeurlPS Code of Ethics, workers involved data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: The paper does not involve research with human subjects. Therefore, the discussion of potential risks, disclosure to subjects, and Institutional Review Board (IRB) approvals are not applicable.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
44

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
45


Reviews:
[{"review_id": "fuIZP1ITFd", "reviewer": "Reviewer_fJin", "summary": "The authors propose a theoretically motivated dynamic distilled datasets. With use of these transportable representation the authors show that they can outperform the performance of the original dataset and in some cases even the complete dataset.", "questions": "See weakness", "limitations": "The authors did not present a detailed study of where the approach can fail and limitation of their analysis.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "- The methods have a sound theoratical basis, the proofs presented although for a narrow setup our quite interetsing. \n\n- The setup for experiments is well defined. \n\n- The results show extensive performance improvement.", "weaknesses": "- The writing style can improve and make the paper approachable. The notations are often terse and lack details about symbols until later. Especially for proofs in appendix.\n\n- It might be more prudent to add experiments of larger scale.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> [W1] The writing style can improve and make the paper approachable. The notations are often terse and lack details about symbols until later. Especially for proofs in appendix.\n> \n\n[R1] Thank you for your feedback. We will revise the paper to improve its clarity and make it more accessible. Specifically, we will provide more detailed explanations of symbols and notations earlier in the text, particularly in the proofs section of the appendix.\n\n> [W2] It might be more prudent to add experiments of larger scale.\n> \n\n[R2] Thank you for your feedback. The ImageNet-1K dataset utilized in this study is already a substantial and challenging benchmark in the field of dataset distillation [a,b,c,d,e,f,g]. We have conducted additional large-scale experiments, as detailed in attached PDF and our response to Reviewer zbwg.\n\nOur additional experiments detailed in the attached PDF include the following: (1) Utilization of the higher resolution dataset CelebA-HQ (1024 × 1024). (2) Implementation of a continual learning task. (3) Comparisons with more state-of-the-art (SOTA) baselines. (4) Reporting of computation time and peak GPU memory usage. (5) Segmentation task performance on the VOC 2012 dataset. (6) Evaluation on the larger ImageNet-21K dataset.\n\n[a] Sun, Peng, et al. \"On the diversity and realism of distilled dataset: An efficient dataset distillation paradigm.\" CVPR 2024.\n\n[b] Lee, Yongmin, and Hye Won Chung. \"SelMatch: Effectively scaling up dataset distillation via selection-based initialization and partial updates by trajectory matching.\" ICML 2024.\n\n[c] Lee, Dong Bok, et al. \"Self-supervised dataset distillation for transfer learning.\" ICLR 2024.\n\n[d] Chen, Xuxi, et al. \"Data distillation can be like vodka: Distilling more times for better quality.\" ICLR 2024\n\n[e] Yin, Zeyuan, Eric Xing, and Zhiqiang Shen. \"Squeeze, recover and relabel: Dataset condensation at imagenet scale from a new perspective.\" NeurlPS 2023 Spotlight.\n\n[f] Shao, Shitong, et al. \"Generalized large-scale data condensation via various backbone and statistical matching.\" CVPR 2024 Highlight.\n\n[g] Shao, Shitong, et al. \"Elucidating the Design Space of Dataset Condensation.” Preprint 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kCQ2vTWjb8", "reviewer": "Reviewer_zbwg", "summary": "The paper addresses the scalability constraints in representation learning by proposing a novel Representation Learning Accelerator (RELA). Current paradigms, focusing separately on self-supervised learning and dataset distillation, overlook the potential of intermediate acceleration. The authors define ideal data properties for optimization and generalization, enabling effective transport of model-generated representations. RELA leverages a task- and architecture-agnostic public model to form a dynamic data subset, enhancing (self-)supervised learning. Empirical results show that using CLIP ViT B/16 as a prior model, RELA-aided BYOL can train a ResNet-50 from scratch with 50% of ImageNet-1K, surpassing full dataset performance. This approach improves representation learning efficiency, offering impactful implications for reducing data requirements and computational resources in model training.", "questions": "1. Can RELA be extended to ImageNet-21K? If so, please provide some experimental results.\n\n2. How does the proposed framework perform on other downstream tasks, such as object detection or segmentation?", "limitations": "The limitations are briefly discussed in the appendix. For more limitations, refer to the Weaknesses and Questions sections.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "[S1] The paper tackles critical issues in efficient training and dataset distillation (DD).\n\n[S2] RELA introduces a novel approach in dataset distillation, effectively bridging representation learning with data-efficient methods.\n\n[S3] The paper is clearly and effectively written.\n\n[S4] Comprehensive theoretical analysis supports the proposed claims. The paper explores analytical concepts in dataset distillation, employing information theory principles.\n\n[S5] Extensive experimental and ablation studies are conducted in both the main paper and appendix.", "weaknesses": "[W1] The paper lacks generalization ability to high-resolution datasets, such as 1Kx1K, which are common in practical datasets like clinical and aerial images.\n\n[W2] The practical applications of dataset distillation are not discussed. Demonstrating applicability in neural architecture search (NAS), continual learning, federated learning, and privacy preservation would be valuable for both the community and real-world scenarios.\n\n[W3] Comparisons with some state-of-the-art methods, including DREAM [a], DataDAM [b], and SeqMatch [c], are missing from Table 6. Including these comparisons would strengthen the evaluation of the proposed method.\n\n[W4] An analysis of the computational costs is absent. It is crucial to examine the training costs, including both memory and training time, for the proposed process and state-of-the-art methods to ensure data-efficient training algorithms.\n\n[W5] The theoretical analysis does not generalize to complicated data distributions, such as two arbitrary mixtures of Gaussian distributions or a Mixture of Generalized Gaussian distributions (MGG).\n\nUpon addressing these weaknesses, I will consider changing my initial rating to strong accept.\n\n-------------\n\nReferences:\n\n[a] Liu, Yanqing, et al. \"Dream: Efficient dataset distillation by representative matching.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.\n\n[b] Sajedi, Ahmad, et al. \"Datadam: Efficient dataset distillation with attention matching.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.\n\n[c] Du, Jiawei,  et al. \"Sequential subset matching for dataset distillation.\" Advances in Neural Information Processing Systems 36 (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> [W1] The paper lacks generalization ability to high-resolution datasets, such as 1Kx1K, which are common in practical datasets like clinical and aerial images.\n> \n\n[R1] Thank you for your feedback. We have conducted additional experiments using the CelebA-HQ dataset (1024 $\\times$ 1024). The results, presented in Table 1 of the attached PDF, indicate that our ReLA effectively accelerates self-supervised learning on high-resolution datasets.\n\n> [W2] The practical applications of dataset distillation are not discussed. Demonstrating applicability in neural architecture search (NAS), continual learning, federated learning, and privacy preservation would be valuable for both the community and real-world scenarios.\n> \n\n[R2] We would like to emphasize that the primary application of our ReLA method is to accelerate representation learning. This extends the scope of conventional dataset distillation, as traditional methods require more computational resources in distillation process than training on the full dataset, making them unsuitable for accelerating learning processes (refer to our detailed analysis in [R4]).\n\nFurthermore, we have applied ReLA to continual learning, with the experimental results presented in Table 2 of the attached PDF. These results demonstrate that ReLA outperforms the baseline in this context.\n\n> [W3] Comparisons with some state-of-the-art methods, including DREAM [a], DataDAM [b], and SeqMatch [c], are missing from Table 6. Including these comparisons would strengthen the evaluation of the proposed method.\n> \n\n[R3] We excluded comparisons with methods [a,b,c] because they are inefficient when dealing with large-scale datasets, such as ImageNet-1K, particularly with large backbone architectures like ResNet-18, as demonstrated in previous studies [d,e,f].\nHence, we evaluate these methods [a,b,c] against our proposed ReLA on four datasets with smaller backbone networks. The results of these comparisons are presented in Table 3 of the attached PDF, demonstrating that ReLA consistently outperforms these methods.\n\nWe will add these baselines into our manuscript.\n\n> [W4] An analysis of the computational costs is absent. It is crucial to examine the training costs, including both memory and training time, for the proposed process and state-of-the-art methods to ensure data-efficient training algorithms.\n> \n\n[R4] As detailed in Appendix K, the SOTA dataset distillation methods generally incur higher computational costs than training on the entire dataset. For example, the SOTA efficient method RDED [d] necessitates a fully trained model on ImageNet-1K for distillation. Additionally, several other SOTA methods [a,b,c] also demand higher computational budgets compared to training on the full dataset, as documented in their respective studies.\n\nIn contrast, our proposed ReLA method can distill a dataset using less than the computational budget required for a single epoch of full dataset training (see Appendix K for a detailed analysis).\n\nThe computational costs of ReLA, along with comparisons to baseline methods, are presented in Table 4 of the attached PDF. The experimental results indicate that while ReLA slightly increases computational time and peak GPU memory usage when using a partial dataset for training, it significantly enhances accuracy, surpassing the performance achieved by training on the entire dataset.\n\nWe will add these comparisons into our manuscript.\n\n> [W5] The theoretical analysis does not generalize to complicated data distributions, such as two arbitrary mixtures of Gaussian distributions or a Mixture of Generalized Gaussian distributions (MGG).\n> \n\n[R5] The theoretical analysis presented in this paper can indeed be generalized to arbitrary mixtures of Gaussian distributions. Although Section 3 exemplifies this with a specific mixture of Gaussian distributions, the proofs and theoretical framework detailed in Appendices B and C provide a generalized approach applicable to any mixture of two Gaussian distributions, regardless of their means and variances.\n\n> [Q1] Can RELA be extended to ImageNet-21K? If so, please provide some experimental results.\n> \n\n[R6] The ImageNet-1K dataset used in this study is already a substantial and challenging benchmark in the field of dataset distillation [a,b,c,d,e,f]. We extended our experiments to the ImageNet-21K dataset. As shown in Table 6 in the attached PDF, our ReLA approach also accelerates training for BYOL on ImageNet-21K, achieving higher performance with the same data usage as the original BYOL.\n\n> [Q2] How does the proposed framework perform on other downstream tasks, such as object detection or segmentation?\n> \n\n[R7] We evaluate the pre-trained models on a downstream segmentation task, as detailed in Table 5 of the attached PDF. The results demonstrate that models trained with our ReLA framework outperform baseline methods in both classification and segmentation tasks.\n\n[a] Liu, Yanqing, et al. \"Dream: Efficient dataset distillation by representative matching.\" ICCV 2023.\n\n[b] Sajedi, Ahmad, et al. \"Datadam: Efficient dataset distillation with attention matching.\" ICCV 2023.\n\n[c] Du, Jiawei, Qin Shi, and Joey Tianyi Zhou. \"Sequential subset matching for dataset distillation.\" NeurIPS 2023.\n\n[d] Sun, Peng, et al. \"On the diversity and realism of distilled dataset: An efficient dataset distillation paradigm.\" CVPR 2024.\n\n[e] Yin, Zeyuan, Eric Xing, and Zhiqiang Shen. \"Squeeze, recover and relabel: Dataset condensation at imagenet scale from a new perspective.\" NeurlPS 2023 Spotlight.\n\n[f] Shao, Shitong, et al. \"Generalized large-scale data condensation via various backbone and statistical matching.\" CVPR 2024 Highlight.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ICd7tD4kuK", "reviewer": "Reviewer_K3hg", "summary": "This paper proposes to accelerate the training of self-supervised learning with a pre-trained teacher model and use the prediction of this teacher as a target. The method is motivated from a toy example that the training convergence is dominated by the variance of the random variable. Then, the authors conjure that maximizing the mutual information between samples and targets can accelerate the convergence, based on which the authors give their solution that adds a cosine similarity term between the outputs from the student model and the teacher to the base self-supervised learning loss. Experiments demonstrate that the proposed technique enhances the performance when there are only part of data.", "questions": "Please refer to the weakness part above.", "limitations": "The authors have mentioned that Conjecture 1 remains speculative. However, I do think this is an important limitation that serves as the core of the whole method and article.", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The experiments are sufficient and satisfactory to demonstrate the effectiveness of the proposed method.\n2. Most of the claims are backed up with theoretical analysis.", "weaknesses": "1. I do not think the writing style of the article is proper for a technical paper. Overall speaking, it is over packaged from the motivation throughout to the final solution. They try to introduce a big picture in the abstract and introduction parts. However, readers can hardly know what they actually do until reading the method part. I recommend introducing the technical solution first and then discussing the underlying motivations.\n2. Technically, my major concern lies in the motivation of the proposed method. In Conjecture 1, the authors speculate that maximizing the mutual information between samples and targets can speed up training. This conjecture is based on an over simplified example that conducts classification for a Gaussian Mixture Model with 2 Gaussian distributions. It's true that diminishing the variance can result in faster training in this simple case, but it is far from the real-world cases, which would be much more complex. In this way, although the article involves extensive theoretical analysis, the core part, which connects the motivation and the final solution, is not well supported.\n3. In fact, this conjecture, as well as the final solution that minimizes the cosine loss with sample predictions and targets generated by a well-trained teacher, is contradictory to many recent studies. For example, in RDED [a], hard patches are selected to form the synthetic samples. After all, hard patches should have larger classification loss, corresponding to samples closer to the decision boundary in the toy case. Recent studies like [b] and [c] also demonstrate that weak trajectory or teacher models may benefit learning more. Given these studies, the conjecture of this article seems not so plausible without substantial evidences.\n4. Based on above analysis, I would like to believe the acceleration comes from additional supervision signals, which transforms a self-supervised learning problem to an almost fully supervised one, since a well-trained teacher is involved. If that is the case, the conclusion and technique proposed in the article would appear trivial, since similar conclusion and technique have been well studied even several years ago [d,e,f,g].\n5. The authors analyze *the optimal properties of distilled data*. Here an assumption that specifies the distillation method is missing. After all, different methods would result in different distill data. Not all of them satisfy the proposed properties.\n6. I do not think the introduction of dynamic dataset distillation is necessary. The authors randomly sample a portion of real data to form the training dataset and use different data in each epoch, which can be viewed as mini-batch training on a subset and is not closely related to dataset distillation. In fact, I think the settings in DiM [h] and [i] are more adhere to this concept. In other words, the setting in this article is actually not coupled with dataset distillation. Only using a subset of data can achieve satisfactory performance is the result of introduction of pre-trained teacher. The method is not well aligned with dataset distillation serving as the motivation. In this case, the results may not appear surprising since the method transforms a self-supervised learning problem to an almost fully supervised one.\n\n[a] On the Diversity and Realism of Distilled Dataset: An Efficient Dataset Distillation Paradigm (Peng Sun et al., CVPR 2024)\n\n[b] Dataset Quantization (Daquan Zhou & Kai Wang & Jianyang Gu et al., ICCV 2023)\n\n[c] SelMatch: Effectively Scaling Up Dataset Distillation via Selection-Based Initialization and Partial Updates by Trajectory Matching (Yongmin Lee et al., ICML 2024)\n\n[d] Self-Supervised Dataset Distillation for Transfer Learning (Dong Bok Lee & Seanie Lee et al., ICLR 2024)\n\n[e] Boosting Self-Supervised Learning via Knowledge Transfer (Noroozi et al., CVPR 2018)\n\n[f] Knowledge Distillation Meets Self-Supervision (Xu et al., ECCV 2020)\n\n[g] SEED: Self-supervised Distillation For Visual Representation (Fang et al., ICLR 2021)\n\n[h] DiM: Distilling Dataset into Generative Model (Kai Wang & Jianyang Gu et al., 2023) \n\n[I] Data Distillation Can Be Like Vodka: Distilling More Times For Better Quality (Xuxi Chen & Yu Yang et al., ICLR 2024)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> [W1] […] I recommend introducing the technical solution first and then discussing the underlying motivations.\n> \n\n[R1] Thank you for your feedback. We will revise the paper to present the technical solution earlier, enhancing clarity and focus.\n\n> [W2] Technically, my major concern lies in the motivation of the proposed method. In Conjecture 1, […] This conjecture is based on an over simplified example that conducts classification for a Gaussian Mixture Model with 2 Gaussian distributions. […]\n> \n\n[R2] We would like to clarify the following points:\n\n1. Prior research efforts [n,o], including the Outstanding Paper at NeurIPS 2022, have utilized simplified mixtures of Gaussian distributions to study data efficiency and other properties, acknowledging the complexity of analyzing real-world data distributions. These studies typically extend theoretical insights to more complex datasets through empirical validation.\n2. Our work provides theoretical insights into data properties relevant to efficient training. These insights do not suggest a reduction in the variance of real-world data but rather support our Conjecture 1, which forms the basis of our proposed technical solution. The validity of our solution is demonstrated in Section 5, where we present extensive experimental results on real-world datasets.\n\n> [W3] In fact, this conjecture, as well as the final solution that minimizes the cosine loss with sample predictions and targets generated by a well-trained teacher, is contradictory to many recent studies. For example, in RDED [a], hard patches are selected to form the synthetic samples. After all, hard patches should have larger classification loss, corresponding to samples closer to the decision boundary in the toy case. Recent studies like [b] and [c] also demonstrate that weak trajectory or teacher models may benefit learning more. […]\n> \n\n[R3] We would like to clarify the following points:\n\n1. In fact, RDED [a] promotes the selection of easy patches (those with smaller classification loss, as indicated in their Equation (8)) and utilizes well-trained teacher models for target generation, as specified in their Algorithm 1. This methodology is consistent with our theoretical and empirical analyses and aligns with the recommendations found in several recent studies [e,f,g,j,k,l].\n2. The papers reviewer referenced [b, c] utilize weak teachers for trajectory matching or patch selection in image generation. However, none of these studies advocate for the use of weak teachers in target generation.\n\n> [W4] Based on above analysis, I would like to believe the acceleration comes from additional supervision signals, which transforms a self-supervised learning problem to an almost fully supervised one, since a well-trained teacher is involved. […]\n> \n\n[R4] We would like to clarify the following points:\n\n1. Unlike prior studies [a,b,e,f,g,j,k,l] that directly utilize a well-trained model specific to the dataset, our approach, ReLA, leverages any publicly available model to generate supervision signals. For instance, ReLA can use a model pre-trained on CIFAR-10 to generate supervision signals that accelerate training for BYOL on a 10% subset of ImageNet-1K, resulting in a 7.7% increase in accuracy compared to the original BYOL (see Sections 4 and 5).\n2. Our aim is to propose a simple, effective, plug-and-play, and theoretically supported method to accelerate representation learning. In addition to empirical validation in our paper and similar evidence in prior studies [d,e,f,g,j,k,l], we also provide theoretical insights into how these generated supervision signals enhance the convergence rate of learning (see Section 3.2).\n3. We have not transformed a self-supervised learning problem into an almost fully supervised one. The additional supervision loss is integrated with the original loss term and modulated by a dynamic coefficient (see Section 4.2). Furthermore, our ablation study in Section 5.2 illustrates that using only the supervision loss can result in catastrophic performance.\n4. Moreover, our method can also be used to accelerate supervised learning, as demonstrated in our experiments and analysis in Appendix L.\n\n> [W5] The authors analyze *the optimal properties of distilled data*. […], different methods would result in different distill data. Not all of them satisfy the proposed properties.\n> \n\n[R5] We would like to clarify the following points:\n\n1. Our analysis try to identify the ideal properties of efficient data to offer insights that support our proposed framework. Most existing dataset distillation methods fail to fully satisfy these properties.  The relationship between our analyzed properties and previous studies is discussed in Section 3.2 and Appendix H.\n2. However, our theoretical results are consistent with the empirical results in many previous studies [e,f,g,j,k,l,m], which indicate that: (i) selecting/synthesizing samples that are easy or close to the class data mean, and (ii) using well-trained models to relabel samples, can achieve superior performance with fewer training steps.\n\n> [W6] I do not think the introduction of dynamic dataset distillation is necessary. […], the results may not appear surprising since the method transforms a self-supervised learning problem to an almost fully supervised one.\n> \n\n[R6] We would like to clarify the following points:\n\n1. Our framework ReLA is defined under the concept of dynamic dataset distillation because ReLA involves dynamically editing samples and labels during training.\n2. Our approach does not rely solely on the introduction of a pre-trained teacher, as detailed in our analysis in [R4].  Furthermore, our experiments in Section 5 show that introducing weak teachers also contributes positively to our framework.\n3. In addition to self-supervised learning tasks, we demonstrate that our ReLA framework outperforms state-of-the-art dataset distillation methods in conventional supervised learning tasks (see Appendix L).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3jiRmEOrbe", "reviewer": "Reviewer_7uZz", "summary": "This paper considers data set distillation to a more concise form for purposes of representational efficiency, partly as an attempt to consider what forms of deployment might benefit from such forms and more importantly seeking to offer a universal form of translator to extract such compact representations from diverse data sets and their resulting influence towards training efficiency. It includes what appears to be an attempt at a nearly comprehensive survey of related literature which is itself a nice part of the paper.", "questions": "I am partly fascinated by scope not addressed by the paper as to where these forms of distilled data sets might be of added benefit due to their less noisy representations.\n\nMore of a hypothetical question, no need to answer: Are there any known applications where data augmentation is known to interfere with implementations? Perhaps that might be a starting point of where valuable sources of impact of this form of distillation could be considered.\n\nCan you talk further about the extent of novelty and related work for the ReLA-D tool?", "limitations": "I don't know if their is any strong interest in dataset distillations in practice where they may be used in production systems. If there are perhaps you could highlight furhter as it may make the importance of the paper more clear.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "- Originality\nI didn't get a strong sense of significant originality, although that was sort of a vague sense and welcome further input from the authors. The five points were obviously clearly articulated in section one, I wondeer if you could color these with their specific relations to prior work?\n\n    - Quality\nThe paper was not standout or anything, but for those with interest in this less common application it might have usefulness.\n\n    - Clarity\nThe Table 1 was kind of awkwardly placed, it distracted from flow of paper and would possibly be better served as paragraphs instead of a table.\n\n    - Significance\nI had trouble interpreting the significance of the ReLA_D model for generic data set distillation, asking the authors for further input below. Otherwise a contribution of the work appeared to be associated with the demonstrations of impact of distillations towards training efficiency, which is sort of intuitive and probably more useful as a validation of their tool than any significant result.", "weaknesses": "The way the paper is written it almost appears that authors are approaching the concepts of dataset distillation considerations (eg in Table 1) as if such matters are unexplored in literature, which I would sort of find surprising. I think the paper would have been better structured as leading with the ReLA-D model convention for data set distillation as a more prominent part of the writeup, including more detail as to how that relates to prior work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> [W1] The way the paper is written it almost appears that authors are approaching the concepts of dataset distillation considerations (eg in Table 1) as if such matters are unexplored in literature, which I would sort of find surprising. I think the paper would have been better structured as leading with the ReLA-D model convention for data set distillation as a more prominent part of the writeup, including more detail as to how that relates to prior work.\n> \n\n[R1] Thank you for your valuable feedback. We will revise the manuscript to explicitly situate the ReLA-D model within the context of existing literature on dataset distillation. Additionally, we will enhance the discussion to clarify how our work builds upon and differentiates from prior research, as outlined in our [R3] below.\n\n> [Q1] I am partly fascinated by scope not addressed by the paper as to where these forms of distilled data sets might be of added benefit due to their less noisy representations.\n> \n\n[R2] Thank you for your feedback. We have conducted additional experiments and further explorations, as detailed in attached PDF and our response to Reviewer zbwg.\n\nIn the attached PDF, we have included additional experiments covering: (a) the higher-resolution CelebA-HQ dataset (1024 × 1024); (b) a continual learning task; (c) comparisons with more state-of-the-art baselines; (d) reports on computation time and peak GPU memory usage; (e) a segmentation task on the VOC 2012 dataset; and (f) the larger ImageNet-21K dataset.\n\n> [Q2] More of a hypothetical question, no need to answer: Are there any known applications where data augmentation is known to interfere with implementations? Perhaps that might be a starting point of where valuable sources of impact of this form of distillation could be considered.\n> \n\n[R3] To the best of our knowledge, no such applications currently exist.\n\n> [Q3] Can you talk further about the extent of novelty and related work for the ReLA-D tool?\n> \n\n[R4] Additional discussions include the following:\n\n1. As detailed in Appendix K, the SOTA dataset distillation methods generally demand a higher computational budget compared to training on the full dataset. For instance, to distill ImageNet-1K, the SOTA efficient methods [a,b,c] require a model that is well-trained on ImageNet-1K. In contrast, our ReLA can distill a dataset using less than the budget required for training a single epoch (refer to our analysis in Appendix K). The technical solution is detailed in Section 4.\n2. Our ReLA is capable of distilling large datasets for both self-supervised and supervised learning tasks, whereas conventional methods are typically limited to distilling small datasets or are exclusively applicable to supervised learning.\n\n[a] Sun, Peng, et al. \"On the diversity and realism of distilled dataset: An efficient dataset distillation paradigm.\" CVPR 2024.\n\n[b] Yin, Zeyuan, Eric Xing, and Zhiqiang Shen. \"Squeeze, recover and relabel: Dataset condensation at imagenet scale from a new perspective.\" NeurlPS 2023 Spotlight.\n\n[c] Shao, Shitong, et al. \"Generalized large-scale data condensation via various backbone and statistical matching.\" CVPR 2024 Highlight.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear Reviewers,

Thank you again for your constructive comments, which have been very helpful in improving our paper.

During the rebuttal period, we have addressed your concerns in detail in our responses. We have also provided additional experimental results in the attached PDF.

Thank you very much for your precious time and attention.

Best wishes,

Authors of Paper 19675

Meta-review: Some reviewers observe that the theoretical work is on a simplified case. However, as the authors argue effectively in rebuttal, it is valid to perform theoretical analysis on a simplified case, and then use empirical evaluation to determine if the theory does indeed transfer to practice. The rebuttal includes a nice improvement to the theory (generalizing to MMG), but of course that case still remains simple compared to practice.  But, to emphasize: this is not a weakness of the paper per se, the theory remains interesting in its own right, and as a motivation for the empirical method.

The reviewers all agree that the empirical work does indeed show improvements, and the rebuttal's additional experiments show results on additional datasets as requested by the reviewers, including a scale-up to ImageNet-21K.

Reviewer K expresses concern that "the acceleration comes from additional supervision signals, which transforms a self-supervised learning problem to an almost fully supervised one, since a well-trained teacher is involved".  If this were the case, it would reduce the significance of the paper, but the rebuttal argues well that the supervision signal on its own is insufficient, and indeed the observation that "Small-scale CIFAR models improve ImageNet" also rebuts the concern.
Decision: Accept (poster)