Title: Efficiency for Free: Ideal Data Are Transportable Representations
Venue: NeurIPS 2024
Accepted: yes
Keywords: Efficient Learning

Efficient Data

Ideal Data
Abstract: Data, the seminal opportunity and challenge in modern machine learning, currently constrains the scalability of representation learning and impedes the pace of model evolution.
In this work, we investigate the efficiency properties of data from both optimization and generalization perspectives.
Our theoretical and empirical analysis reveals an unexpected finding: for a given task, utilizing a publicly available, task- and architecture-agnostic model (referred to as the `prior model' in this paper) can effectively produce efficient data.
Building on this insight, we propose the Representation Learning Accelerator (ReLA), which promotes the formation and utilization of efficient data, thereby accelerating representation learning.
Utilizing a ResNet-18 pre-trained on CIFAR-10 as a prior model to inform ResNet-50 training on ImageNet-1K reduces computational costs by $50\%$ while maintaining the same accuracy as the model trained with the original BYOL, which requires $100\%$ cost.
Our code is available at: \url{https://github.com/LINs-lab/ReLA}.

Efficiency for Free: Ideal Data Are Transportable Representations

Abstract

Data, the seminal opportunity and challenge in modern machine learning, currently constrains the scalability of representation learning and impedes the pace of model evolution. In this work, we investigate the efficiency properties of data from both optimization and generalization perspectives. Our theoretical and empirical analysis reveals an unexpected finding: for a given task, utilizing a publicly available, taskand architecture-agnostic model (referred to as the 'prior model' in this paper) can effectively produce efficient data. Building on this insight, we propose the Representation Learning Accelerator (RELA), which promotes the formation and utilization of efficient data, thereby accelerating representation learning. Utilizing a ResNet-18 pre-trained on CIFAR-10 as a prior model to inform ResNet-50 training on ImageNet-1K reduces computational costs by 50% while maintaining the same accuracy as the model trained with the original BYOL, which requires 100% cost. Our code is available at: https://github.com/LINs-lab/ReLA .

1 Introduction

Original

Download

Feed

1

Prior (

�

)

Generate

Alg.

Accelerated

RELA+Alg.

Loss

Representation

Space

Init.

�

�

2

� ⋆

RELA+Alg.

Fast Learning

Original Alg.

The available of massive datasets [20, 49] and recent advances in parallel data processing [28, 42] have facilitated the rapid evolution of large deep models, such as GPT-4 [1] and LVM [2]. These

models excel in numerous learning tasks, attributable to their impressive representation capabilities. However, the emergence of vast amounts of data within the modern deep learning paradigm raises two fundamental challenges: (i) the demand for human annotations of huge datasets consumes significant social resources [45, 20, 43]; (ii) training large models with increasing data and model capacity suffers from intensive computational burden [6, 16, 55].

The community has made considerable efforts to enhance learning efficiency. Self-supervised learning methods [12, 69, 30, 10, 25, 14, 9, 3], with their superior representation learning devoid of human annotations via the self-learning paradigm, attempt to tackle the challenge (i). Concurrently, research has been conducted to mitigate data efficiency issues in challenge (ii): dataset distillation approaches [65, 56, 11, 72, 53, 54] have successfully synthesized a small distilled dataset, on which models trained on this compact dataset can akin to one trained on the full dataset.

However, challenges (i) and (ii) persist and yet are far from being solved [43, 45, 6, 16, 55], particularly the intervention of these two learning paradigms. In this paper, we identify two issues: (a) inefficiency in the self-supervised learning procedure compared to conventional supervised learning arises due to sub-optimal self-generating targets [30, 62]; (b) although training on the distilled dataset is efficient and effective, the distillation process of optimization-based approaches [11, 72, 37] is computationally demanding [18, 56], often surpassing the computational load of training on the full dataset. This limitation restricts its potential to accelerate representation learning. To tackle these challenges, we propose a novel open problem in the domain of representation learning:

Problem 1 (Accelerating Representation Learning through Free Models) . According to the No Free Lunch theorem [66], it is evident that accelerating the learning process without incorporating prior knowledge is inherently challenging. Fortunately, numerous publicly available pre-trained models can be accessed online, offering a form of free prior knowledge. Despite this, effectively utilizing these models poses several implicit challenges, as these models may not be directly relevant to our target learning task, or they may not be sufficiently well-trained. This leads to a question:

How can we leverage task- and architecture-agnostic publicly available models to accelerate representation learning for a specific task?

To address Problem 1 , we propose RELA to utilize a freely available model downloaded from the internet to generate efficient data for training. This approach aims to accelerate training during the initial stages by effectively leveraging these generated data, thereby establishing a rapid pathway for representation learning (see Figure 1 ). Specifically, we list our five key contributions below as the first step toward bridging representation learning with data-efficient learning:

(a) Revealing beneficial/detrimental data properties for efficient/inefficient (self-)supervised learning (see Section 3.2 ). We present a comprehensive analysis of linear models, demonstrating that data properties significantly influence the learning process by impacting the optimization of model training. Our findings reveal that modifications to the data can markedly enhance or impair this optimization. Additionally, we indicate that optimal training necessitates specific data properties-perfect bijective mappings between the samples and targets within a dataset.

(b) Identifying the inefficiency problems of (self-)supervised learning from a data-centric perspective (see Section 3.3 ). Specifically, we identify several factors contributing to the inefficiencies in (self)supervised learning over real-world data. For instance, prevalent data augmentation techniques in modern deep learning can introduce a 'noisy mapping' issue, which may exacerbate the negative effects associated with inefficient data.

(c) Generalization bound for models trained on optimized efficient data (see Section 3.4 ). Although the efficiency properties of data do not inherently ensure the generalization of the trained model, i.e., efficient data alone cannot guarantee generalization ability, we present a generalization bound to analyze models trained on such data.

(d) A novel method RELA to generate and exploit efficient data (see Section 4 ). Leveraging our theoretical insights regarding the bounds of generalization and convergence rate, we introduce RELA, a novel optimization-free method tailored to efficiently generate and effectively exploit efficient data for accelerating representation learning.

(e) An application of our RELA : accelerating (self-)supervised learning (see Section 5 and Appendix I ). Extensive experiments across four widely-used datasets, seven neural network ar-

chitectures, eight self-supervised learning algorithms demonstrate the effectiveness and efficiency of RELA. Training models with RELA significantly outperforms training on the original dataset with the same budget, and even exceeds the performance of training on higher budget.

2 Related Work

This section integrates two distinct deep learning areas: (a) techniques to condense datasets while preserving efficacy; (b) self-supervised learning methods that enable training models on unlabeled data.

2.1 Dataset Distillation: Efficient yet Effective Learning Using Fewer Data

The objective of dataset distillation is to create a significantly smaller dataset that retains competitive performance relative to the original dataset.

Refining proxy metrics between original and distilled datasets. Traditional approaches involve replicating the behaviors of the original dataset within the distilled one. These methods aim to minimize discrepancies between surrogate neural network models trained on both synthetic and original datasets. Key metrics for this process include matching gradients [72, 32, 70, 44], features [63], distributions [71, 73], and training trajectories [11, 17, 22, 18, 68, 24]. However, these methods suffer from substantial computational overhead due to the incessant calculation of discrepancies between the distilled and original datasets. The optimization of the distilled dataset involves minimizing these discrepancies, necessitating multiple iterations until convergence. As a result, scaling to large datasets, such as ImageNet [20], becomes challenging.

Extracting key information from original into distilled datasets. A promising strategy involves identifying metrics that capture essential dataset information. These methods efficiently scale to large datasets like ImageNet-1K using robust backbones without necessitating multiple comparisons between original and distilled datasets. For instance, SRe 2 L [67] condenses the entire dataset into a model, such as pre-trained neural networks like ResNet-18 [26], and then extracts the knowledge from these models into images and targets, forming a distilled dataset. Recently, RDED [56] posits that images accurately recognized by strong observers, such as humans and pre-trained models, are more critical for learning.

Summary. We make the following observations regarding scalable dataset distillation methods utilizing various metrics: (a) a few of these metrics have proven effective for data distillation at the scale of ImageNet. (b) all these metrics require human-labeled data; (c) there is currently no established theory elucidating the conditions under which data distillation is feasible; (d) despite their success, the theory behind training neural networks with reduced data is underexplored.

2.2 Self-supervised Learning: Representation Learning Using Unlabeled Data

The primary objective of self-supervised learning is to extract robust representations without relying on human-labeled data. These representations should be competitive with those derived from supervised learning and deliver superior performance across multiple tasks.

Contrasting self-generated positive and negative Samples. Contrastive learning-based methods implicitly assign a one-hot label to each sample and its augmented versions to facilitate discrimination. Since InfoNCE [47], various works [25, 12, 13, 8, 31, 15, 74, 40, 9, 27] have advanced contrastive learning. MoCo [25, 13, 15] uses a momentum encoder for consistent negatives, effective for both CNNs and Vision Transformers. SimCLR [12] employs strong augmentations and a nonlinear projection head. Other methods integrate instance classification [8], data augmentation [31, 74], clustering [40, 9], and adversarial training [27]. These enhance alignment and uniformity of representations on the hypersphere [64].

Asymmetric model-generating representations as targets. Asymmetric network methods achieve self-supervised learning with only positive pairs [30, 50, 14], avoiding representational collapse through asymmetric architectures. BYOL [30] uses a predictor network and a momentum encoder. Richemond et al. [50] show BYOL performs well without batch statistics. SimSiam [14] halts the gradient to the target branch, mimicking the momentum encoder's effect. DINO [10] employs a self-distillation loss. UniGrad [59] integrates asymmetric networks with contrastive learning methods within a theoretically unified framework.

3 Revealing Critical Properties of Efficient Learning over Data

We begin by presenting formal definitions of supervised learning over a (efficient) dataset.

Definition 1 (Supervised learning over data) . For a dataset D = ( D X , D Y ) = { ( x i , y i ) } | D | i =1 , drawn from the data distribution ( X,Y ) in space ( X , Y ) , the goal of a model learning algorithm is to identify an optimal model ϕ ⋆ that minimizes the expected error defined by:

where ℓ indicates the loss function and ϵ denotes a predetermined deviation. This is typically achieved through a parameterized model ϕ θ , where θ denotes the model parameter within the parameter space Θ . The optimal parameter θ D is determined by training the model to minimize the empirical loss over the dataset:

The training process leverages an optimization algorithm such as stochastic gradient descent [51, 33].

Definition 2 (Data-efficient Learning) . Data-efficient learning seeks to derive an optimized/efficient dataset, denoted as S = ( S X , S Y ) = { ( x j , y j ) } | S | j =1 , from the original dataset D . The objective is to enable models ϕ θ S trained on S to achieve the desired generalization performance, as defined in (1) , with fewer training steps and a reduced computational budget compared to training on the original dataset D .

3.1 Unifying (Self-)Supervised Learning from a Data-Centric Perspective

To ease the understanding and our methodology design in Section 3.4 , we unify both conventional supervised learning and self-supervised learning as learning to map samples in D X to targets in D Y : this view forms 'supervised learning' from a data-centric perspective. Specifically, these two learning paradigms involve generating targets D Y = { y | y = ψ ( x ) s.t. x ∼ D X } and minimizing the empirical loss (2). The only difference lies in the target generation models (or simply labelers) ψ :

(a) Conventional supervised learning , referred to as human-supervised learning, generates targets via human annotation. Note that the targets are stored and used statically throughout the training.

(b) Self-supervised learning (also see Footnote 2), e.g., BYOL [30] utilizes an Exponential Moving Average (EMA) version of the learning model ϕ θ to generate targets y = EMA [ ϕ θ ]( x ) . Note that the targets are dynamically changing during training as the model ϕ θ keeps evolving.

This unified perspective allows us to jointly examine and address the inefficient training issue of (self-)supervised learning from a data-centric perspective, in which in Section 3.2 we first study the impact of samples D X and targets D Y on the model training process and then investigate whether and how a distilled dataset S = ( S X , S Y ) can facilitate this process.

3.2 Empirical and Theoretical Investigation of Data-Centric Efficient Learning

To elucidate the ideal data properties of training on a dataset D , we examine the simple task over a bimodal Gaussian mixture distribution as a case study. We begin by defining the problem.

Definition 3 (Bimodal Gaussian mixture distribution) . Given two Gaussian distributions N 0 ( µ 1 , Σ 2 I ) and N 1 ( µ 2 , Σ 2 I ) , where µ 1 and µ 2 are the means and Σ 2 is the variance (here we set µ 1 = 1 , µ 2 = 2 and Σ = 0 . 5 ). We define a bimodal mixture data G = ( G X , G Y ) as:

Moreover, we define a corresponding binary classification neural network model as:

where σ ( z ) = 1 1+ e -z is the sigmoid activation function; ReLU ( z ) = max(0 , z ) is the activation function for the hidden layer, which provides non-linearity to the model; θ [2] and θ [3] are the weights and biases of the hidden layer; θ [1] and θ [4] are the weights and biases of the output layer.

Modern representation learning fundamentally relies on optimization (see Definition 1 ). We show that modifications to the data can influence the convergence rate of the optimization process, thereby impacting the overall representation learning procedure. Furthermore, we try to uncover several key properties of data efficiency through our theoretical analysis of the case study. In the following, we denote the modified distribution by G ′ = ( G ′ X , G ′ Y ) and examine the altered samples G ′ X and corresponding targets G ′ Y independently.

Investigating the properties of modified samples. The modification process here only rescales the variance of the original sample distribution G X defined in Definition 3 with new Σ (rather than the default 0 . 5 ), while let G ′ Y := G Y ; see explanations in Appendix E . Therefore, we examine the distilled samples G ′ X by setting the variable Σ within the interval (0 , 1) .

Results in Figure 2 demonstrate that the distilled samples G ′ X with smaller variance Σ achieve faster convergence and better performance compared to that of G . To elucidate the underlying mechanism, we provide a rigorous theoretical analysis in Appendix B , culminating in Theorem 1 .

Theorem 1 (Convergence rate of learning on efficient samples) . For the classification task stated in Definition 3 , the convergence rate for the model f θ trained t after steps over distilled data G ′ is:

where ℓ denotes the MSE loss, i.e., ℓ (ˆ y, y ) := ∥ ˆ y -y ∥ 2 , and f θ ⋆ indicates the optimal model, ˜ O signifies the asymptotic complexity. Modified samples characterized by a smaller value of Σ facilitate faster convergence.

Investigating the properties of modified targets. On top of the property understanding for modified samples, we further investigate the potential of modified targets via G ′ . In detail, for modified samples, we consider the most challenging (c.f. Figure 2b ) yet the common case, namely G ′ X with Σ = 1 (see explanations in Appendix M ). For the corresponding modified targets G ′ Y , similar to the prior data-efficient methods [56, 67], for any sample x drawn from G ′ X , we refine its label by assigning ˆ y = ρ · f θ ⋆ ( x ) + (1 -ρ ) · y . Here, ρ denotes the relabeling intensity coefficient, and f θ ⋆ represents a strong pre-trained model (simply, we utilize the model trained on the data in Figure 2c ).

Theorem 2 (Convergence rate of learning on re-labeled data) . For the classification task as in Definition 3 , we have the convergence rate for the model f θ trained after t steps over modified data G ′ :

Note that ρ controls the upper bound of the convergence rate, indicating that using modified targets with a higher value of ρ enables faster convergence.

Results in Figure 2 illustrate that the modified targets G ′ Y with higher values of ρ lead to faster training convergence and better performance. See theoretical analysis in Theorem 2 and Appendix B .

3.3 Extended Understanding of Data-Centric Efficient Learning

The empirical and theoretical investigations regarding the properties of modified samples G ′ X and targets G ′ Y in Section 3.2 are limited to a simplified case (as described in Definition 3 ) and may not extend to all practical scenarios, such as training a ResNet [26] on the ImageNet dataset [20].

Interestingly, we observe that the advantageous modifications of both samples G ′ X and targets G ′ Y converge towards a unified principle: minimizing or preventing any sample x from being labeled with multiple or inaccurate targets y . This principle emphasizes the importance of providing accurate and informative targets y for each sample x , as analyzed in Remark 1 , and suggests extending this insight to any complex dataset like S .

4

3

2

1

0

1

2

2

1

0

1

2

3

4

Values

0.1

0.5

1.0

Initial Parameter

Global Minimum

-2.7

-2.5

-2.3

-2.1

-1.9

-1.7

-1.5

-1.3

-1.1

-0.9

(a) Loss landscape for

Σ

0

200

400

600

800

1000

Training Steps

0.075

0.100

0.125

0.150

0.175

0.200

0.225

0.250

Validation Loss

Values

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

(b) Evaluate

G

′

X

with different

Σ

0.5

0.5

1.5

2.5

3.5

0.5

0.5

1.5

2.5

3.5

0.0

0.2

0.4

0.6

0.8

1.0

(c) Optimal

G

′

X

(

Σ = 0

.

1

)

3

2

1

0

1

2

2

1

0

1

2

3

Values

0.1

0.5

1.0

Initial Parameter

Global Minimum

-2.6

-2.4

-2.1

-1.9

-1.6

-1.4

-1.2

-0.9

-0.7

(a) Loss landscape for

ρ

0

200

400

600

800

1000

Training Steps

0.075

0.100

0.125

0.150

0.175

0.200

0.225

0.250

Validation Loss

Values

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

(b) Evaluate

G

′

Y

with different

ρ

0.5

0.5

1.5

2.5

3.5

0.5

0.5

1.5

2.5

3.5

0.0

0.2

0.4

0.6

0.8

1.0

(c) Optimal

G

′

Y

(

ρ

= 1

.

0

)

Remark 1 (Ideal data properties avoid implicitly introduced gradient noise from data) . Intuitively, the semantic information within each sample x should be unique and not identical to another sample. Consequently, the exact target y , which represents the semantic information of x , should also be unique and informative. This implies the necessity of establishing bijective (or one-to-one) mappings between samples and their respective targets.

In contrast, when a sample x (or several similar samples) is labeled with multiple different targets y , it may implicitly introduce noise into the gradient ∇ ℓ ( x , y ) , thereby hindering the optimization.

However, real-world datasets often deviate from the ideal properties described above, as discussed in Remark 2 below and further analyzed in Appendix M .

Remark 2 (Imperfect mappings and inaccurate targets in real-world datasets) . In practice, we observe that 'noisy mappings' between input samples and targets are prevalent in real-world datasets. As illustrated in Figure 6 , several common phenomena contribute to this issue:

Similar or identical input samples may be assigned different targets due to using data augmentations, which is common in both self-supervised and human-supervised learning settings.

Inaccurate targets may be generated, particularly in self-supervised learning scenarios.

In human-supervised learning, all samples within a class are mapped to a one-hot target.

These imperfect mappings and inaccurate targets pose challenges to achieving optimal training efficiency and effectiveness for real-world datasets.

3.4 Generalization-bounded Efficient Data Synthesis

Given insights from Remark 1 , an effective approach to generate efficient data S from original data D involves employing a high-quality labeler ψ to relabel each sample x within D . This process

results in the formation of an optimized dataset S . However, the generalization ability of models trained on these optimized datasets S is not inherently assured. For a given labeler ψ , we derive the generalization bound, which quantifies the representation distance between models ϕ θ and ϕ ⋆ , trained on the optimized dataset S . This is presented in Theorem 3 (see Appendix D for the proof).

Definition 4 (Representation distance a ) . We introduce our proposed metric as

̸

which quantifies the distance between a source model ϕ S and a target model ϕ T with respect to a dataset D , and the loss function is defined as ℓ (ˆ y, y ) := 1 (ˆ y = y ) .

Theorem 3 (Generalization bound with labeler ψ ) . Assumimg the model ϕ θ : X → Y belongs to a hypothesis space Φ . Then, for any δ ∈ (0 , 1) , with probability at least 1 -δ , we have

where BTarget = DRep ( ψ → ϕ ⋆ ; D X ) , BModel = DRep ( ϕ θ → ψ ; S X ) + 2 R D X (Φ) + ˜ O ( | D X | -1 ) , and BSample = DTV ( S X , D X ) . DTV represents the total variation divergence [4], and R D X is the empirical Rademacher complexity.

Drawing insights from Theorem 1 , 2 and 3, we define the properties of ideal data below:

Definition 5 (Properties of ideal efficient data, including samples S X and targets S Y ) . To meet the objectives of Definition 2 , an ideal efficient data requires: ➀ Targets ( S Y ):

generating targets ψ ( x ) from the labeler ψ that are accurate (i.e., align with human-annotating targets a and optimal model ϕ ⋆ ), aiming to minimize DRep ( ψ → ϕ ⋆ ; D X ) ;

the generated target ψ ( x ) should be informative to corresponding sample x (i.e., forming perfect bijective mappings with the original samples x ), according to Remark 5 ;

➁ Samples ( S X ): low distribution disparity represented by DTV ( S X , D X ) and high sample diversity denoted as | S X | , aiming at minimizing DTV ( S X , D X ) .

To meet the requirement ➁ specified in Remark 5 , we utilize weak data augmentation on D X to generate the set S X , aimed at enhancing the diversity | S X | while ensuring the term DTV ( S X , D X ) remains minimal. However, satisfying requirement ➀ is non-trivial, as capturing a labeler ψ that matches ϕ ⋆ is intractable (i.e., achieving DRep ( ψ → ϕ ⋆ ; D X ) = 0 is challenging).

Fortunately, our experiments in real-world scenarios, as discussed in Section A , demonstrate that employing a prior model as the labeler ψ generally approximates ϕ ⋆ within the representation space . Therefore, we posit that introducing a prior model ψ to generate an efficient dataset S can typically accelerate the early stages of learning. Subsequently, the original dataset D should be employed.

In (8), the last term, DRep ( ϕ θ → ψ ; S X ) + 2 R D X (Φ) + ˜ O ( | D X | -1 ) , includes 2 R D X (Φ) and ˜ O ( | D X | -1 ) , which depend on the neural architecture and the size of D X , respectively, and can be considered constants. Thus, optimizing the model involves minimizing DRep ( ϕ θ → ψ ; S X ) through training ϕ θ . A detailed technical solution is provided in Section 4 .

4 Methodology

Building upon the theoretical insights from Section 3.2 , we propose our RELA (see Figure 1 ): (a) RELA-D ( ) is used to generate efficient data (c.f. Section 4.1 ); (b) RELA-F ( ) guides the models to train over the efficient data from RELA-D ( ) (c.f. Section 4.2 ).

4.1 RELA-D ( ): Synthesis of Efficient Dataset

Motivated by two property requirements in Definition 5 , here we introduce our optimization-free synthesis process of both samples and targets in our RELA-D (see technical details in Appendix F ).

Generating transportable representations as the targets. Weargue that well-trained models (called prior models ) on diverse real-world datasets using various neural network architectures and algorithms converge towards the same linear representation space. In other words, the generated pseudo representations R Y for samples D X using these prior models are linearly transportable to each other and to the human-annotating targets. The empirical verifications refer to Appendix A . We further justify in Appendix F.1 that the requirement ➀ in Definition 5 can be achieved by employing a prior model as the ideal labeler ψ : R d → R m , i.e., generating R Y = { ψ ( x ) | x ∼ D X } as the targets. The generation process of targets is conducted only once (refer to Appendix H for details), and the generated targets R Y are stored and combined with the samples D X to form the data D = ( D X , R Y ) .

Efficient and distribution-aligned sample generation. To satisfy requirement ➁ in Definition 5 efficiently, we employ basic data augmentations into data D X such as RandomResizeCrop with a minimum scale of 0.5 (as opposed to the default of 0.08) and RandomHorizontalFlip with p = 0 . 5 .

4.2 RELA-F ( ): Assist Learning with Generated Efficient Dataset

In this section, we showcase the significance of understanding ideal data properties and generated efficient dataset in assisting self-supervised learning, given this self-supervised paradigm on unlabeled data suffers from significant inefficiency issues compared to human-supervised learning [62].

Here we propose a plug-and-play method that can be seamlessly integrated into any existing selfsupervised learning algorithm, significantly enhancing its training efficiency by introducing an additional loss term. Formally, the loss function is defined as follows:

where ℓ ( z , y ) := 1 -z · y / ( ∥ z ∥∥ y ∥ ) be the loss function, L SSL denotes the loss specified by any self-supervised learning method, respectively. Furthermore, the data D = ( D X , R Y ) are collected using the strategy outlined in Section 4.1 , with updates occurring at each k -th epoch.

The dynamic coefficient λ ∈ { 0 , 1 } divides the training process into two distinct stages. Initially, λ is set to 1 to emphasize L RELA, assuming its crucial role in the early learning phase. As the model ϕ θ improves and self-generated targets become more reliable in L SSL , an adaptive attenuation algorithm adjusts λ to 0 (note that the initial λ is tuning-free for all cases and see Appendix J for details). As a result, only a single loss term in (9) is calculated, ensuring no extra computational cost with RELA.

To enhance the recognition of RELA-aided algorithms, we re-denote those that are used in their names. For example, the BYOL algorithm [30], when enhanced with RELA, is re-denoted as BYOL ( ). Furthermore, as the prior models downloaded from the internet are not consistently robust, the aforementioned dynamic setting of λ also prevents the model ϕ θ from overfitting to potentially weak generated targets. The efficacy of our proposed RELA is empirically validated in Section 5 .

5 Experiments

This section describes the experimental setup and procedures undertaken to test our hypotheses and evaluate the effectiveness of our proposed methodologies.

Experimental setting. We list the settings below (see more details in Appendix K ).

Datasets: For low-resolution data ( 32 × 32 ), we evaluate our method on two datasets, i.e., CIFAR-10 [35] and CIFAR-100 [34]. For high-resolution data, we conduct experiments on two large-scale datasets including Tiny-ImageNet ( 64 × 64 ) [36] and full ImageNet-1K ( 224 × 224 ) [20], to assess the scalability and effectiveness of our method on more complex and varied datasets.

Neural network architectures: Similar to prior works/benchmarks of dataset distillation [56] and selfsupervised learning [57, 19], we use several backbone architectures to evaluate the generalizability

of our method, including ResNet-{18, 50, 101} [26], EfficientNet-B0 [58], MobileNet-V2 [52], ViT [21], and a series of CLIP-based models [49]. These architectures represent a range of model complexities and capacities, enabling a comprehensive assessment of our approach.

Baselines: Referring to a prior widely-used benchmark [57, 19], we consider several state-of-the-art methods as baselines for a broader practical impact, including: SimCLR [12], Barlow Twins [69], BYOL [30], DINO [10], MoCo [25], SimSiam [14], SwAV [9], and Vicreg [3].

Evaluation: Following previous benchmarks and research [57, 19, 12, 3], we evaluate all the trained models using offline linear probing strategy to reflect the representation ability of the trained models, and ensure a fair and comprehensive comparison with baseline approaches.

Implementation details: We implement our method by extending a popular self-supervised learning open-source benchmark [57] and use their configurations therein. This includes using AdamW as the optimizer, with a mini-batch size of 128 (except for ImageNet-1K, where we use a mini-batch size of 512). We implement our method through PyTorch [48], and all experiments are conducted on NVIDIA RTX 4090 GPUs. See more detailed configurations and hyper-parameters in Appendix K .

5.1 Primary Experimental Results and Analysis

Recall that our RELA-D ( ), as illustrated in Figure 1 and Section 4.1 , requires an unlabeled dataset and any pre-trained model freely available online to generate the efficient dataset. To justify the superior performance and generality of our RELA across various unlabeled datasets using prior models with different representation abilities, our comparisons in this subsection start with BYOL [30] 2 and then extend to other self-supervised learning methods.

Table 1 demonstrates the efficacy and efficiency of our RELA in facilitating the learning of robust representations. Overall, BYOL ( ) consistently outperforms the original BYOL when trained with a reduced budget. In certain cases, such as on CIFAR-100, BYOL ( ) employing only 10% of the budget can surpass the performance of BYOL-trained models using the entire budget Specifically:

(a) A stronger prior model (e.g., CLIP) enhances the performance of RELA more effectively than a weaker model (e.g., Rand.);

(b) Our RELA is not sensitive to the prior knowledge. For instance, using CF10-T as the prior model can achieve competitive performance compared to that trained on extensive datasets (e.g., CLIP);

(c) Arandomly initialized model can effectively aid in accelerating learning through our RELA. This can be considered an effective scenario of 'weak-to-strong supervision' [7] using pseudo targets.

Cross-architecture generalization. RELA-D ( ) generates efficient datasets using a specific neural architecture. To evaluate the generalization ability of these datasets, it is essential to test their performance on various architectures not used in the distillation process. Table 2 presents the performance of our RELA in conjunction with various prior models and trained model architectures, demonstrating its robust generalization ability. Specifically:

(a) The integration of RELA always enhances the performance of original BYOL;

(b) Our RELA method exhibits minimal sensitivity to the architecture of the prior model, as evidenced by the comparable performance of BYOL ( ) using both ViT-based and ResNet-based models.

Combining RELA across various self-supervised learning methods. To demonstrate the effectiveness and versatility of RELA in enhancing various self-supervised learning methods, we conduct experiments with widely-used techniques. Table 3 presents the results, highlighting the robust generalization capability of RELA. Our findings consistently show that RELA improves the performance of these methods while maintaining the same budget ratio, emphasizing its potential on learning using unlabeled data. Additionally, we provide the results when combining RELA with human-supervised learning in Appendix I .

6 Conclusion and Limitation

In this paper, to address the Problem 1 , we investigate the optimal properties of data, including samples and targets, to identify the properties that improve generalization and optimization in deep learning models. Our theoretical insights indicate that targets which are informative and linearly transportable to strong representations (e.g., human annotations) enable trained models to exhibit robust representation abilities. Furthermore, we empirically find that well-trained models (called prior models) across various tasks and architectures serve as effective labelers for generating such targets. Consequently, we propose the Representation Learning Accelerator (RELA), which leverages any freely available prior model to generate high-quality targets for samples. Additionally, RELA can enhance existing (self-)supervised learning approaches by utilizing these generated data to accelerate training. However, our theoretical analysis is restricted to the simplified scenario described in Definition 3 , which has limited applicability in real-world contexts.

References

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 , 2023.

Yutong Bai, Xinyang Geng, Karttikeya Mangalam, Amir Bar, Alan Yuille, Trevor Darrell, Jitendra Malik, and Alexei A Efros. Sequential modeling enables scalable learning for large vision models. arXiv preprint arXiv:2312.00785 , 2023.

Adrien Bardes, Jean Ponce, and Yann LeCun. Vicreg: Variance-invariance-covariance regularization for self-supervised learning. arXiv preprint arXiv:2105.04906 , 2021.

Andrew R Barron, Lhszl Gyorfi, and Edward C van der Meulen. Distribution estimation consistent in total variation and in two types of information divergence. IEEE transactions on Information Theory , 38(5):1437-1454, 1992.

Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Vaughan. A theory of learning from different domains. Machine Learning , 79:151-175, 2010.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems , 33:1877-1901, 2020.

Collin Burns, Pavel Izmailov, Jan Hendrik Kirchner, Bowen Baker, Leo Gao, Leopold Aschenbrenner, Yining Chen, Adrien Ecoffet, Manas Joglekar, Jan Leike, et al. Weak-to-strong generalization: Eliciting strong capabilities with weak supervision. arXiv preprint arXiv:2312.09390 , 2023.

Yue Cao, Zhenda Xie, Bin Liu, Yutong Lin, Zheng Zhang, and Han Hu. Parametric instance classification for unsupervised visual feature learning. In NeurIPS , 2020.

Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. In NeurIPS , 2020.

Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In ICCV , 2021.

George Cazenavette, Tongzhou Wang, Antonio Torralba, Alexei A Efros, and Jun-Yan Zhu. Dataset distillation by matching training trajectories. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 4750-4759, 2022.

Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In ICML , 2020.

Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.04297 , 2020.

Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In CVPR , 2021.

Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. arXiv preprint arXiv:2104.02057 , 2021.

Yu Cheng, Duo Wang, Pan Zhou, and Tao Zhang. A survey of model compression and acceleration for deep neural networks. arXiv preprint arXiv:1710.09282 , 2017.

Justin Cui, Ruochen Wang, Si Si, and Cho-Jui Hsieh. Dc-bench: Dataset condensation benchmark. Advances in Neural Information Processing Systems , 35:810-822, 2022.

Justin Cui, Ruochen Wang, Si Si, and Cho-Jui Hsieh. Scaling up dataset distillation to imagenet1k with constant memory. In International Conference on Machine Learning , pages 6565-6590. PMLR, 2023.

Victor Guilherme Turrisi Da Costa, Enrico Fini, Moin Nabi, Nicu Sebe, and Elisa Ricci. sololearn: A library of self-supervised methods for visual representation learning. Journal of Machine Learning Research , 23(56):1-6, 2022.

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A largescale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition , pages 248-255. Ieee, 2009.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 , 2020.

Jiawei Du, Yidi Jiang, Vincent YF Tan, Joey Tianyi Zhou, and Haizhou Li. Minimizing the accumulated trajectory error to improve dataset distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 3749-3758, 2023.

Noah Golowich, Alexander Rakhlin, and Ohad Shamir. Size-independent sample complexity of neural networks, 2019.

Ziyao Guo, Kai Wang, George Cazenavette, Hui Li, Kaipeng Zhang, and Yang You. Towards lossless dataset distillation via difficulty-aligned trajectory matching. arXiv preprint arXiv:2310.05773 , 2023.

Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR , 2020.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778, 2016.

Qianjiang Hu, Xiao Wang, Wei Hu, and Guo-Jun Qi. Adco: Adversarial contrast for efficient learning of unsupervised representations from self-trained negative adversaries. In CVPR , 2021.

Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V Le, Yonghui Wu, et al. Gpipe: Efficient training of giant neural networks using pipeline parallelism. Advances in neural information processing systems , 32, 2019.

Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. Advances in neural information processing systems , 31, 2018.

Grill Jean-Bastien, Strub Florian, Altché Florent, Tallec Corentin, Pierre Richemond H., Buchatskaya Elena, Doersch Carl, Bernardo Pires Avila, Zhaohan Guo Daniel, Mohammad Azar Gheshlaghi, Piot Bilal, Kavukcuoglu Koray, Munos Rémi, and Valko Michal. Bootstrap your own latent - a new approach to self-supervised learning. In NeurIPS , 2020.

Yannis Kalantidis, Mert Bulent Sariyildiz, Noe Pion, Philippe Weinzaepfel, and Diane Larlus. Hard negative mixing for contrastive learning. In NeurIPS , 2020.

Jang-Hyun Kim, Jinuk Kim, Seong Joon Oh, Sangdoo Yun, Hwanjun Song, Joonhyun Jeong, Jung-Woo Ha, and Hyun Oh Song. Dataset condensation via efficient synthetic-data parameterization. In International Conference on Machine Learning , pages 11102-11118. PMLR, 2022.

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.

Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. Cifar-10 and cifar-100 datasets. URl: https://www. cs. toronto. edu/kriz/cifar. html , 6(1):1, 2009.

Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N , 7(7):3, 2015.

Dong Bok Lee, Seanie Lee, Joonho Ko, Kenji Kawaguchi, Juho Lee, and Sung Ju Hwang. Self-supervised dataset distillation for transfer learning. In Proceedings of the International Conference on Learning Representations (ICLR) , 2024.

Jaehoon Lee, Lechao Xiao, Samuel S Schoenholz, Yasaman Bahri, Roman Novak, Jascha Sohl-Dickstein, and Jeffrey Pennington. Wide neural networks of any depth evolve as linear models under gradient descent *. Journal of Statistical Mechanics: Theory and Experiment , 2020(12):124002, December 2020.

Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss landscape of neural nets. Advances in neural information processing systems , 31, 2018.

Junnan Li, Pan Zhou, Caiming Xiong, and Steven CH Hoi. Prototypical contrastive learning of unsupervised representations. In ICLR , 2021.

Yuanzhi Li and Yingyu Liang. Learning overparameterized neural networks via stochastic gradient descent on structured data. Advances in neural information processing systems , 31, 2018.

Tao Lin, Sebastian U Stich, Kumar Kshitij Patel, and Martin Jaggi. Don't use large mini-batches, use local sgd. arXiv preprint arXiv:1808.07217 , 2018.

Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. Gpt understands, too. AI Open , 2023.

Yanqing Liu, Jianyang Gu, Kai Wang, Zheng Zhu, Wei Jiang, and Yang You. Dream: Efficient dataset distillation by representative matching. arXiv preprint arXiv:2302.14416 , 2023.

Sachin Mehta, Maxwell Horton, Fartash Faghri, Mohammad Hossein Sekhavat, Mahyar Najibi, Mehrdad Farajtabar, Oncel Tuzel, and Mohammad Rastegari. Catlip: Clip-level visual recognition accuracy with 2.7 x faster pre-training on web-scale image-text data. arXiv preprint arXiv:2404.15653 , 2024.

Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar. Foundations of Machine Learning . 2018. Second Edition.

Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748 , 2018.

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems , 32, 2019.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning , pages 8748-8763. PMLR, 2021.

Pierre H Richemond, Jean-Bastien Grill, Florent Altché, Corentin Tallec, Florian Strub, Andrew Brock, Samuel Smith, Soham De, Razvan Pascanu, Bilal Piot, et al. Byol works even without batch statistics. arXiv preprint arXiv:2010.10241 , 2020.

Herbert E. Robbins. A stochastic approximation method. Annals of Mathematical Statistics , 22:400-407, 1951.

Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 4510-4520, 2018.

Shitong Shao, Zeyuan Yin, Muxin Zhou, Xindong Zhang, and Zhiqiang Shen. Generalized large-scale data condensation via various backbone and statistical matching. arXiv preprint arXiv:2311.17950 , 2023.

Shitong Shao, Zikai Zhou, Huanran Chen, and Zhiqiang Shen. Elucidating the design space of dataset condensation. arXiv preprint arXiv:2404.13733 , 2024.

Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considerations for deep learning in nlp. arXiv preprint arXiv:1906.02243 , 2019.

Peng Sun, Bei Shi, Daiwei Yu, and Tao Lin. On the diversity and realism of distilled dataset: An efficient dataset distillation paradigm. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2024.

Igor Susmelj, Matthias Heller, Philipp Wirth, Jeremy Prescott, and Malte Ebner et al. Lightly. GitHub. Note: https://github.com/lightly-ai/lightly , 2020.

Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International conference on machine learning , pages 6105-6114. PMLR, 2019.

Chenxin Tao, Honghui Wang, Xizhou Zhu, Jiahua Dong, Shiji Song, Gao Huang, and Jifeng Dai. Exploring the equivalence of siamese self-supervised learning via a unified gradient framework. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 14431-14440, 2022.

Yuandong Tian, Xinlei Chen, and Surya Ganguli. Understanding self-supervised learning dynamics without contrastive pairs. In ICML , 2021.

Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research , 9(11), 2008.

Guangrun Wang, Keze Wang, Guangcong Wang, Philip HS Torr, and Liang Lin. Solving inefficiency of self-supervised representation learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 9505-9515, 2021.

Kai Wang, Bo Zhao, Xiangyu Peng, Zheng Zhu, Shuo Yang, Shuo Wang, Guan Huang, Hakan Bilen, Xinchao Wang, and Yang You. Cafe: Learning to condense dataset by aligning features. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 12196-12205, 2022.

Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In ICML , 2020.

Tongzhou Wang, Jun-Yan Zhu, Antonio Torralba, and Alexei A Efros. Dataset distillation. arXiv preprint arXiv:1811.10959 , 2018.

David H Wolpert and William G Macready. No free lunch theorems for optimization. IEEE transactions on evolutionary computation , 1(1):67-82, 1997.

Zeyuan Yin, Eric Xing, and Zhiqiang Shen. Squeeze, recover and relabel: Dataset condensation at imagenet scale from a new perspective. arXiv preprint arXiv:2306.13092 , 2023.

Ruonan Yu, Songhua Liu, and Xinchao Wang. Dataset distillation: A comprehensive review. arXiv preprint arXiv:2301.07014 , 2023.

Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Selfsupervised learning via redundancy reduction. In ICML , 2021.

Lei Zhang, Jie Zhang, Bowen Lei, Subhabrata Mukherjee, Xiang Pan, Bo Zhao, Caiwen Ding, Yao Li, and Dongkuan Xu. Accelerating dataset distillation via model augmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11950-11959, 2023.

Bo Zhao and Hakan Bilen. Dataset condensation with distribution matching. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pages 6514-6523, 2023.

Bo Zhao, Konda Reddy Mopuri, and Hakan Bilen. Dataset condensation with gradient matching. arXiv preprint arXiv:2006.05929 , 2020.

Ganlong Zhao, Guanbin Li, Yipeng Qin, and Yizhou Yu. Improved distribution matching for dataset condensation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 7856-7865, 2023.

Rui Zhu, Bingchen Zhao, Jingen Liu, Zhenglong Sun, and Chang Wen Chen. Improving contrastive learning by visualizing feature transformation. In ICCV , 2021.

Contents

A Ablation Study

We conduct ablation studies to understand the impact of each component of RELA on performance.

Noise

Rand.

CF10-T

CF100-T

TIN-T

IN1K-T

CLIP

Noise

Rand.

CF10-T

CF100-T

TIN-T

IN1K-T

CLIP

0.16

0.16

0.16

0.16

0.16

0.31

0.22

0.05

1.00

0.70

0.72

0.78

0.81

0.79

0.17

0.62

1.00

0.90

0.88

0.91

0.84

0.17

0.61

0.92

1.00

0.90

0.91

0.85

0.12

0.54

0.80

0.81

1.00

0.87

0.78

0.04

0.26

0.48

0.48

0.56

1.00

0.66

0.04

0.28

0.44

0.45

0.51

0.74

1.00

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

Value of constant

65

70

75

80

Top-1 Accuracy (%)

Static (Rand.)

Dynamic (Rand.)

Static (TIN-T)

Dynamic (TIN-T)

CF-10

CF-100

T-IN

IN-1K

Dataset

0

20

40

60

80

100

Computational Cost (%)

Prior Models

Rand.

CF10-T

CLIP

(a) Representation similarity

(b) Dynamic vs. static strategies

(c) Computational cost analysis

Empirical representation similarity. Our foundational assumption of our RELA is that the representations of well-trained models, developed using various neural network architectures and algorithms on diverse real-world datasets, exhibit linear transportability to one another. To test this hypothesis, we assess representation similarity, a metric that quantifies the linear transferability between pre-trained models. The results, depicted in Figure 4a , demonstrate that representations from robust models (e.g., CLIP) can be effectively transferred to less robust models (e.g., CF10-T). This finding aligns with our results in Table 1 , showing that leveraging powerful models (e.g., CLIP and IN1K-T) consistently enhances learning in models trained on datasets with limited knowledge, such as CF-10.

Combining RELA and BYOL with static λ setting strategies. The coefficient λ , as introduced in Section 4.2 , is pivotal in controlling the weight of the RELA phase during training. To assess the robustness of our adaptive strategy, which dynamically adjusts λ , we compare it to a static λ setting strategy. The results in Figure 4b indicate that larger (smaller) RELA weights are advantageous when using a strong (weak) prior model. Nonetheless, static settings lack generalizability across various scenarios, whereas our adaptive strategy demonstrates superior generalization capabilities.

Analysis of the computational cost when using RELA with different prior models. To validate that our RELA, in conjunction with various prior models, can effectively reduce computational costs in self-supervised learning, we conducted experiments comparing the computational expense required for BYOL ( ) to achieve equivalent performance to the original BYOL trained with a full budget. The results, illustrated in Figure 4c , consistently demonstrate that RELA assists BYOL in lowering training costs while maintaining equivalent performance levels. Furthermore, it is evident that employing robust prior models consistently leads to greater reductions in training budgets.

B Proof of Theorem 1

In this section, we prove a slightly modified version of Theorem1, extending the distribution to Generalized Gaussian Mixture(GGM) and making some assumptions for technical reasons. Yet this proof could still reflect the essential of the theorem.

B.1 Setup

Notation N ( µ, α, β ) denotes the generalized Gaussian distribution with pdf β 2 α Γ(1 /β ) e -( | x -µ | /α ) β , B for Bernoulli distribution.

We focus on the 1-dim situation. Assume that µ 1 < µ 2 . Define the original data distribution( N 0 = N ( µ 1 , α 0 , β 0 ) and N 1 = N ( µ 2 , α 0 , β 0 ) )

and the modified one ( N ′ 0 = N ( µ 1 , α, β ) and N ′ 1 = N ( µ 2 , α, β ) ):

Our task is predicting y given x . Note that y ∈ {± 1 } , which is a bit different from the definition in Section 3.2. In 1-dim situation, we just need one parameter for this classification task, so define f θ ( x ) := sign ( x + θ ) to fit the distribution. We could compute the generalization loss on original distribution:

Obviously θ ⋆ = -µ 1 + µ 2 2 , we have:

where C 1 , C ′ 1 are constants, F 0 F 1 F denote the CDF of N 0 N 1 and N (0 , 1 , β 0 ) respectively. The inequality above is due to the fact that function h ( x ) = ( ∫ 1 -1 dF -∫ x +1 x -1 dF ) /x 2 has limits at 0 and so is bounded.

B.2 Algorithm

For a dataset { ( x i , y i ) } n i =1 , set the loss function L ( θ ) = 1 n ∑ n i =1 ℓ [ y i ( x i + θ )] , ℓ ( v ) = 1 2 (1 -v ) 2 . We apply the stocastic gradient descent algorithm and assume the online setting ( n = 1 ): at step t draw one sample ( x t , y t ) from G ′ then use the gradient ∇ L ( θ t ) to update θ ( η ∈ (0 , 1) , t ∈ N ):

It can be observed that randomness of x leads to noies on gradient.

B.3 Bounds with Variance

We prove the proposition that lower variance of GG can make convergence faster, i.e. E [ L ( f θ t ) -L ( f θ ⋆ )] is bounded by an increasing function of variance ( t fixed).

Proof. From above, we could get

and so :

The last two equalities is due to the fact that for ( x, y ) ∼ G ′

B.4 Nonlinear case

In this subsection, we conduct some qualitative analysis on the nonlinear case. The setting is the same as that in Section 3.2. We point out the differences compared with the linear case above: x ∈ R d , y ∈ { 0 , 1 } and

where σ ( z ) = 1 1+ e -z is the sigmoid function; ReLU ( z ) = max(0 , z ) is the activation function for the hidden layer, which provides non-linearity to the model; θ [2] and θ [3] are the weights and biases of the hidden layer; θ [1] and θ [4] are the weights and biases of the output layer.

To make things explicit, we still assume the online setting and set the loss function L ( θ ) = 1 2 ( f θ ( x ) -y ) 2 . Assume after some iterations, θ [2] · µ 1 + θ [3] < 0 and θ [2] · µ 2 + θ [3] > 0 (coordinate-wise). In this situation, we could see that if x is close to its mean( µ 1 or µ 2 ), the sign of ReLU ( θ [2] x + θ [3] ) will be the same as y . So f θ will become an optimal classifier if θ [1] → + ∞ and θ [4] →-∞ . We focus on θ [1] , using SGD:

Note that we drop the variable value in σ ( · ) to make the expression more compact.

Then we can analyze the phenomenon qualitatively: larger Σ will make convergence slower. The reason is that the larger Σ is, when x is drawn from N 1 ( y = 1 ), θ [2] x + θ [3] < 0 is more likely to happen(i.e. straying far away from the mean), causing θ [1] to stop updating; what's worse, when x is drawn from N 0 ( y = 0 ), with larger probability θ [2] x + θ [3] > 0 which will make θ [1] to go in the opposite direction. In summary, it is Σ that makes the gradient noisy thus impacts the convergence rate.

B.5 From the Perspective of Feature Learning

In essence, the theoretical results in [41] could also be interpreted as a proof of the theorem. [41] study the learning of a two-layer ReLU neural network for k -class classification via stochastic gradient descent (SGD), assuming that each class corresponds to l patterns(distributions), with every two of the k × l distributions of the input data are separated by a distance δ . Below is the main theorem in [41]:

Proposition 1 . Suppose some assumptions are satisfied, then for every ϵ > 0 , there is M = poly ( k, l, 1 /δ, 1 /ϵ ) such that for every m ≥ M , after doing a minibatch SGD with batch size B = poly ( k, l, 1 /δ, 1 /ϵ, log m ) and learning rate η = 1 m · poly ( k,l, 1 /δ, 1 /ϵ, log m ) for T = poly ( k, l, 1 /δ, 1 /ϵ, log m ) iterations, with high probability, the generalization error(the probability that the model misclassifies) of the learned model is at most ϵ .

Theoretical results above show that a larger δ helps network to learn more efficiently. (In our case, δ can be roughly viewed as the Mahalanobis distance between the two Gaussians, which is inverse proportion to the variance.) Also, Appendix D.2 of [41] demonstrates an example very similar to ours in which the input data is drawn from Gaussian distributions with different means, indicating increasing variance of the Gaussian causes the test accuracy to decrease and takes longer time to get a good solution.

C Proof of Theorem 2

C.1 Setting

Use the same setting in Section B(linear case) except that

In other words, we modify the distribution of y instead of x this time.

C.2 Bounds with ρ

We're going to prove that higher ρ can make convergence faster, i.e. E [ L ( f θ t ) -L ( f θ ⋆ )] is bounded by an decreasing function of ρ ( t fixed).

Similarly, we can get bounds with ρ (see C ′ 1 in Section B):

C.3 Nonlinear case

To see the impact of modifying y more clearly, we directly set ρ = 1 and conduct a similar analysis as in Section B.4. Let's still focus on θ [1] and use the same assumptions in Section B.4, then we have:

Note y is replaced by f θ ∗ ( x ) . For instance x is drawn from N 0 ( y = 0 ) but strays far away from the µ 1 , causing θ [2] x + θ [3] > 0 . In this situation f θ ∗ is likely to regard x as a sample from N 1 (i.e. f θ ∗ ( x ) close to 1) thus making θ [1] to go in the right direction instead of the opposite. This explains why larger ρ can make convergence faster.

D Proof of Theorem 3

̸

We follow some proof steps in [5]. Let's begin by introducing some notations used in this section. Notation and Setup X is the input space, D S and D T are two distributions over X . Let Y = { 0 , 1 } . H denotes a hypothesis class from X to Y . To simplify notations, ∀ h, f ∈ H let ϵ S ( h, f ) = E x ∼D S [ 1 ( h ( x ) = f ( x ))] , and ˆ ϵ S ( h, f ) be empirical error ( ϵ T ( h, f ) , ˆ ϵ T ( h, f ) similar).

Then we introduce some concepts and lemmas, most of which are from [5].

Definition 6 ( H -divergence) . The H -divergence between two distributions D and D ′ is defined as:

Definition 7 (Total Variation Distance) . For two distributions D and D ′ , the total variation distance of them is defined as:

where F denotes the collection of all events in the probability space.

Lemma 1 . For two distributions D and D ′ , by definition it's easy to see that:

Definition 8 (Symmetric Difference Hypothesis Space) .

⊕ denotes the XOR operation.

Lemma 2 .

Proof. only need note that h ( x ) ⊕ h ′ ( x ) = | h ( x ) -h ′ ( x ) | , so

this is done by definition.

With above notations we can derive a general proposition related with Theorem 3.

Proposition 2 . Assumimg H is a hypothesis class from X to Y and ϕ , ϕ ′ ∈ H , we have:

where loss function is ℓ (ˆ y, y ) := 1 (ˆ y = y )

Proof. using the lemmas above,

̸

To obtain some useful inequalities of generalization bound, we need to introduce the Rademacher complexity.

Definition 9 (Rademacher complexity of a function class) . Given a sample of points S = { z 1 , z 2 , . . . , z m } ⊂ Z , and considering a function class F of real-valued functions over Z , the empirical Rademacher complexity of F given S is defined as:

where σ i are independent and identically distributed Rademacher random variables. In other words, for i = 1 , 2 , . . . , m , the probability that σ i = +1 is equal to the probability that σ i = -1 , and both are 1 2 . Further, let P be a probability distribution over Z . The Rademacher complexity of the function class F with respect to P for sample size m is:

Lemma 3 (Generalization bound with Rademacher complexity) . Let F be a family of loss functions F = { ( x, y ) ↦→ ℓ (( x, y ) , h ) : h ∈ H} with ℓ (( x, y ) , h ) ∈ [0 , 1] for all ℓ, ( x, y ) and h . Then, with probability 1 -δ , the generalization gap is

for all h ∈ H and samples U of size n .

The proof of this classical result could be found in most machine learning textbooks, like [46]. Since ℓ ( · ) is 1 -Lipschitz, we can derive that R U S ( ℓ ◦ H ) ≤ R U S ( H ) .

Theorem 3 involves the representation distance, below we prove the triangle inequality for representation distance.

Lemma 4 (Triangle inequality for Representation distance) . for any functions ϕ S , ϕ T , ϕ U and distribution D , we have:

Proof. Let's denote:

By definition:

We need to show:

Consider the composition of the transformations:

The combined transformation can be written as:

Using the properties of the loss function ℓ , we can write:

This inequality holds for the reason that it can only break when the two items at right-hand side are both 0, in which the left side is also 0 due to rule of composition. Thus the inequality holds for all x and W 1 , b 1 , W 2 , b 2 .

Taking expectations over x ∼ D :

Taking the infimum over W 1 , b 1 and W 2 , b 2 :

Since the left-hand side is an upper bound for DRep ( ϕ S → ϕ T ; D ) , we have:

Therefore, the triangle inequality holds for the metric DRep.

Now we are ready to prove Theorem 3.

Proof. For arbitrary W,b , with probability 1 -δ :

Taking the infimum over W,b :

Using the triangle inequality for representation distance:

For arbitrary W,b , using the proposition above,

Taking the infimum over W,b :

Combining the above results, we have:

Impact of initialization In this part we leverage the Neural Tangent Kernel (NTK) framework [29] to deduce that a neural network initialized closer to a target function f ⋆ converges faster during training. Under the NTK regime, neural networks exhibit linearized training dynamics, allowing us to predict how the network's output evolves during training.

In the context of supervised learning, consider a neural network with parameters θ and output function f ( x ; θ ) , where x is the input. The goal is to approximate a target function f ⋆ ( x ) by minimizing a loss function, typically the mean squared error (MSE):

where { ( x i , y i ) } n i =1 is the training data and y i = f ⋆ ( x i ) .

Under gradient descent with learning rate η , the parameter updates are:

In the NTK regime, where the network width tends to infinity, the network's output evolves linearly with respect to the parameters around initialization θ 0 [38]. We can approximate:

This linearization allows us to express the evolution of the network's output as:

where f t ( x ) = f ( x ; θ t ) is the network output at time t and K ( x, x ′ ) = ∇ θ f ( x ; θ 0 ) ⊤ ∇ θ f ( x ′ ; θ 0 ) is the Neural Tangent Kernel.

In continuous time (gradient flow), the training dynamics can be described by a differential equation:

Vectorizing over all training inputs, we have:

where f t = [ f t ( x 1 ) , f t ( x 2 ) , . . . , f t ( x n )] ⊤ , y = [ y 1 , y 2 , . . . , y n ] ⊤ and K is the NTK matrix with entries K ij = K ( x i , x j ) . This differential equation has the solution:

This equation shows that the error f t -y decays exponentially over time, with the rate governed by the NTK matrix K . Since K is symmetric, we can decompose K into its eigenvalues { λ j } and corresponding eigenvectors { v j } :

where c j = v ⊤ j ( f 0 -y ) . The rate of convergence for each component of the error is proportional to the corresponding eigenvalue λ j of the NTK. Larger eigenvalues lead to faster decay. Initial error matters: the initial error f 0 -y scales the amplitude of the exponential terms. A smaller initial error means the network starts closer to the target function, reducing the time required for the error to decay to a specific threshold.

Suppose we want the error to be less than ϵ :

the solution becomes:

Using the solution:

where:

is the rising factorial.

Remark. Usually, the Rademacher complexity R S ( ℓ ◦ F ) could be bounded by of R S ( F ) . For example, if ℓ is L -lipschitz, then R S ( ℓ ◦ F ) ≤ L · R S ( F ) . That's why we directly compute the Rademacher complexity of H instead of ℓ ◦ H .

E Explanation of Rescaling Samples

Dataset distillation seeks to create a condensed dataset that allows models to achieve performance comparable to those trained on the full dataset, but with fewer training steps. In this section, we will demonstrate that rescaling the variance of Gaussian distributions, as defined in Definition 3, does not affect the optimal performance of models trained on these rescaled data.

where λ min is the smallest (positive) eigenvalue of K . Solving for time t :

By leveraging the NTK framework, we've shown that a better initialization(meaning the neural network's initial output function is closer to the target function f ⋆ ) leads to faster convergence during training. This is because the training dynamics under NTK are linear, and the exponential error decay is directly influenced by the magnitude of the initial error. Analysis above implies that if we can train a network that is closer to the target function, it will converge faster to the target function.

D.1 Relation between data distribution and Rademacher complexity

Now let's look at the Rademacher complexity appeared above more carefully. Let 1-Lipschitz positive homogeneous activation σ i be given, and

Then using Theorem 1 in [23], for samples S of size m we have bound for the empirical Rademacher complexity: √

where X ∈ R d × m is the input data matrix, and ∥ · ∥ F denotes the Frobenius norm.

If we further assume that the data are drawn from the distribution G (stated in Section 3.2) with covariance Σ (for simplicity let µ 1 = -µ, µ 2 = µ ), then we can bound the Rademacher complexity of H with respect to G :

The right part of the last inequality is an increasing function with respect to Σ , and Γ( · ) denotes the gamma function, M ( · , · , · ) is the Kummer's confluent hypergeometric function, given by:

Proof. Consider two Gaussian distributions N 0 ( µ 1 , σ 2 I ) and N 1 ( µ 2 , σ 2 I ) with means µ 1 = 1 and µ 2 = 2 , and variance σ 2 . We define the bimodal mixture distribution G = ( G X , G Y ) such that ( x , y ) is sampled according to:

The decision rule for optimal classification is determined by the likelihood ratio test. For a given sample x , the log-likelihood ratio Λ( x ) is given by:

Since x 0 and x 1 are drawn from Gaussian distributions, their probability density functions are:

Substituting these into the log-likelihood ratio, we have:

Further simplification gives:

Simplifying, we obtain:

This simplifies to:

Solving yields:

Since the optimal decision threshold for balanced classes (i.e., P ( y = 1) = P ( y = 0) = 0 . 5 ) is Λ( x ) = 0 , we set:

Expanding and rearranging, we derive:

Therefore, the optimal decision boundary is x = 1 . 5 , which is independent of the variance σ 2 . This completes the proof.

Regarding efficiency, utilizing scaled data to train similar models with fewer training steps is proven in Section B. Additionally, rescaling each Gaussian distribution preserves their means, which aligns with the objectives of conventional distribution matching-based dataset distillation methods. These methods aim to distill data while maintaining the distributional properties, specifically their means.

F Detailed Methodology of RELA-D

Recall that ϕ θ : R d → R n . We then introduce a transport matrix W ∈ R m × n and define the combined model as W ϕ θ ( · ) ∈ R m . During the training phase, the parameters W and θ are jointly optimized. However, as the dimension m increases, the computational complexity of the optimization grows rapidly. To address this issue, we propose reducing the dimensionality of the target matrix R Y from m to n , where n < m :

where µ and σ denote the mean and standard deviation, respectively, of each column in the R Y . Practically, we use batch PCA to perform the computation shown in (10), as illustrated in L.

F.1 Proof for Ideal Properties of Prior Models

We aim to demonstrate that modern deep learning methods can effectively train models to serve as robust prior models by extracting sufficient information from samples as representations.

Therefore, we poist the existence of a prior model ξ capable of losslessly extracting the information of samples D X when trained using the InfoNCE loss [47], a method prevalently employed in contemporary deep learning algorithms [12].

Proof. To demonstrate that an encoder ξ trained with the InfoNCE loss preserves all information from the input data D X , we proceed as follows.

1. Definitions and Setup

Let X denote the input data space with data distribution p data ( x ) . Let p pos ( x, x + ) denote the distribution of positive pairs, typically generated via data augmentation. The encoder ξ : X → R d maps inputs to d -dimensional representations. The similarity function q : R d × R d → R (e.g., dot product) measures similarity between representations. Given a positive pair ( x, x + ) , let { x -i } N i =1 be N negative samples drawn i.i.d. from p data ( x ) . The InfoNCE loss is defined as:

where τ > 0 is a temperature parameter.

2. InfoNCE as a Mutual Information Lower Bound

The InfoNCE objective serves as a lower bound to the mutual information between representations of positive pairs:

where

As the number of negative samples N →∞ , this bound becomes tight, approaching the true mutual information I ( ξ ( X ); ξ ( X + )) . 3. Optimal Encoder and Similarity Function Let ( ξ ∗ , q ∗ ) denote the optimal encoder and similarity function that maximize J ( ξ, q ) . Under the assumption of infinite negative samples and an expressive similarity function, the optimal similarity satisfies:

where C is a constant independent of ( x, x + ) . This aligns q ∗ with the pointwise mutual information (PMI) between x and x + . 4. Injectivity through Mutual Information Maximization Maximizing I ( ξ ( X ); ξ ( X + )) encourages the encoder ξ to capture as much information about X as possible. To ensure injectivity:

Sufficient Dimensionality: The representation dimension d must be at least as large as the intrinsic dimensionality of X .

Expressive Architecture: The encoder ξ should be sufficiently expressive, potentially utilizing architectural constraints (e.g., invertible networks) to promote injectivity.

Under these conditions, maximizing mutual information implies that ξ ∗ approximates an injective mapping on the support of p data ( x ) , i.e., ξ ∗ ( x ) = ξ ∗ ( x ′ ) ⇒ x = x ′ almost surely. 5. Existence of the Inverse Mapping Given that ξ ∗ is injective, there exists a deterministic inverse mapping g : R d →X such that:

This mapping g can be constructed as the inverse of ξ ∗ on its image:

G Analysis of Different Self-Supervised Learning Methods

We refer to the primary theoretical results for existing self-supervised learning methods as presented in [59], where all methods are unified under a simple and cohesive framework, detailed below. Specifically, UniGrad [59] demonstrates that most self-supervised learning methods can be unified by analyzing their gradients.

G.1 A Unified Framework for SSL

A typical self-supervised learning framework employs a siamese network with two branches: an online branch and a target branch. The target branch serves as the training target for the online branch.

Given an input image x , two augmented views x 1 and x 2 are generated, serving as inputs for the two branches. The encoder f ( · ) extracts representations u i ≜ f ( x i ) for i = 1 , 2 from these views.

Table 4 details the notations used. u 1 and u 2 denote the current training samples, while v denotes unspecified samples. u o 1 and v o are representations from the online branch. Three types of target branches are widely used: 1) weight-sharing with the online branch ( u s 2 and v s ); 2) weight-sharing but detached from gradient back-propagation ( u d 2 and v d ); 3) a momentum encoder updated from the online branch ( u m 2 and v m ). If unspecified, u t 2 and v t are used. A symmetric loss is applied to the two augmented views, as described in [14].

V represents the sample set in the current training step. Methods vary in constructing this set: V batch includes all samples from the current batch, V bank uses a memory bank storing previous samples, and V ∞ includes all previous samples, potentially larger than a memory bank.

G.2 Contrastive Learning Methods

Contrastive learning relies on negative samples to prevent representational collapse and enhance performance. Positive samples are derived from different views of the same image, while negative

samples come from other images. The goal is to attract positive pairs and repel negative pairs, typically using the InfoNCE loss [47]:

where cos( · ) denotes cosine similarity, and τ is the temperature hyper-parameter. This formulation can be adapted for various methods, discussed below.

MoCo [25, 13]. MoCo uses a momentum encoder for the target branch and a memory bank for storing previous representations. Negative samples are drawn from this memory bank. The gradient for sample u o 1 is:

where s v = exp (cos( u o 1 ,v m ) /τ ) ∑ y m ∈V bank exp (cos( u o 1 ,y m ) /τ ) and N is the number of samples in the batch.

SimCLR [12]. SimCLR shares weights between the target and online branches and does not stop back-propagation. It uses all representations from other images in the batch as negative samples. The gradient is:

where t v = exp (cos( v s ,u o 1 ) /τ ) ∑ y s ∈V batch \ v s exp (cos( v s ,y s ) /τ ) . If the gradient through the target branch is stopped, the second term vanishes.

Unified Gradient. The gradient for these methods can be unified as:

comprising a weighted sum of positive and negative samples. The term -u t 2 pulls positive samples together, while ∑ v t ∈V s v v t pushes negative samples apart. The main difference between methods lies in the target branch used and the construction of the contrastive sample set V .

G.3 Asymmetric Network Methods

Asymmetric network methods learn representations by maximizing the similarity of positive pairs without using negative samples. These methods require symmetry-breaking network designs to avoid representational collapse. A predictor h ( · ) is appended after the online branch, and the gradient to the target branch is stopped. The objective function is:

Relation to BYOL [30]. BYOL uses a momentum encoder for the target branch, i.e., u t 2 = u m 2 in Eq.(15).

Relation to Simsiam [14]. Simsiam shows that a momentum encoder is unnecessary and only applies the stop-gradient operation to the target branch, i.e., u t 2 = u d 2 in Eq.(15).

Unified Gradient. Despite the performance of asymmetric network methods, the avoidance of collapse solutions is not well understood. DirectPred [60] explores this by studying training dynamics and proposes an analytical solution for the predictor h ( · ) .

DirectPred formulates the predictor as h ( v ) = W h v , with W h calculated based on the correlation matrix E v ( vv T ) . The correlation matrix, F , is computed as the moving average for each batch:

F ≜ ∑ v o ∈V ∞ ρ v v o v oT , where ρ v is the moving average weight. Decomposing F into eigenvalues Λ F and eigenvectors U , W h is:

where λ max is the max eigenvalue of F and ϵ is a hyper-parameter to boost small eigenvalues.

DirectPred also derives the gradient:

where -W T h u t 2 and ∑ v o ∈V ∞ ( ρ v u o 1 T v o ) v o act as positive and negative gradients respectively, and λ = u o 1 T W T h u t 2 u o 1 T ( F + ϵ 2 I ) u o 1 is a balance factor.

Though negative samples are absent in the loss function, they emerge from the predictor network's optimization. The eigenspace of the predictor W h aligns with the feature correlation matrix F , encoding its information. During back-propagation, this encoded information functions as a negative gradient, influencing the optimization direction.

G.4 Feature Decorrelation Methods

Feature decorrelation methods have recently emerged as a novel approach in self-supervised learning. These methods aim to reduce redundancy among different feature dimensions to prevent collapse. Various loss functions have been proposed for this purpose. We examine their relations below.

Relation to Barlow Twins [69] . Barlow Twins employs the following loss function:

̸

where W = 1 N ∑ v o 1 ,v s 2 ∈V batch v o 1 v sT 2 is a cross-correlation matrix, C denotes the number of feature dimensions, and λ is a balancing hyper-parameter. The diagonal elements of W are encouraged to be close to 1 , while the off-diagonal elements are forced towards 0 .

Despite appearing different, Eq. (18) operates similarly to previous methods from a gradient perspective, calculated as:

where A = I -(1 -λ ) W diag and ( W diag ) ij = δ ij W ij is the diagonal matrix of W . Barlow Twins applies batch normalization instead of ℓ 2 normalization to the representation v .

Relation to VICReg [3] . VICReg modifies Barlow Twins with the following loss function:

̸

where W ′ = 1 N -1 ∑ v o 1 ∈V batch ( v o 1 -¯ v o 1 )( v o 1 -¯ v o 1 ) T is the covariance matrix of the same view, std( v ) i denotes the standard deviation of the i -th channel of v , γ is a constant target value, and λ 1 , λ 2 are balancing weights.

The gradient is derived as follows:

where ˜ v = v -¯ v is the de-centered sample, λ = 2 λ 1 N 2 c ( N -1) 2 , and B = N cλ ( N -1) (2 λ 1 W ′ diag + λ 2 2 diag( 1 ( γ -std( v o 1 ) > 0) ⊘ std( v o 1 ))) . Here, diag( x ) is a matrix with the vector x on its diagonal, 1 ( · ) is the indicator function, and ⊘ denotes element-wise division.

VICReg does not normalize v ; instead, it uses de-centering and a standard deviation term in the loss function.

Unified Gradient. Given the equivalence of v s and v o , the gradient for feature decorrelation methods can be unified as:

where -u t 2 is the positive gradient, and ∑ v o ∈V batch ( u oT v o N ) v o 1 is the negative gradient. λ is a balancing factor. The difference between methods lies in the subscript for the negative coefficient. Feature decorrelation methods function similarly to other self-supervised methods, with the positive and negative gradients derived from the diagonal and off-diagonal elements of the correlation matrix.

H Budget of RELA for Data Synthesis

While training on the distilled dataset is both efficient and effective, the distillation process in optimization-based approaches [11, 72, 65] is computationally intensive [18, 56], often exceeding the computational load of training on the full dataset.

In contrast, the synthetic data generated by our RELA framework requires a budget less than that of training for a single epoch (c.f. Section 4.1). This is because the synthesis budget of our RELA is equivalent to performing inference over the entire dataset D X . Consequently, this computational expense is negligible given that training epochs typically number around 100 .

I RELA in Labeled Dataset Distillation and Human-Supervised Learning

We apply our RELA in labeled dataset distillation and human-supervised learning.

I.1 Experimental Setup

Datasets and Neural Network Architectures. We conduct experiments on datasets of varying scales and resolutions.

Small-scale: We evaluate on CIFAR-10 ( 32 × 32 ) [35] and CIFAR-100 ( 32 × 32 ) [34].

Large-scale: We utilize Tiny-ImageNet ( 64 × 64 ) [36] and ImageNet-1K ( 224 × 224 ) [20].

Consistent with previous works on dataset distillation [67, 73, 24], we use ConvNet [24] and ResNet{18,50} [26] as our backbone networks across all datasets. Specifically, Conv-3 is employed for CIFAR-10/100, and Conv-4 for Tiny-ImageNet and ImageNet-1K.

Baselines. We compare our method with several SOTA distillation methods capable of scaling to large high-resolution datasets, including G-VBSM [53], SRe 2 L [67], and RDED [56]. To the best of our knowledge, SRe 2 L, G-VBSM, and RDED are the only published works that efficiently scale to datasets of any size, making them our closest baselines. All distilled datasets synthesized from these baselines undergo the same post-training process. Results are reported in Table 5. For our RELA, the prior models used for distillation are identical to the pre-trained models employed in the baseline methods.

I.2 Main Results

Table 5 demonstrates the superiority of our RELA, despite incorporating a zero-cost sample synthesis process.

J RELA Algorithm

In essence, this algorithm is designed to detect the convergence of RELA. Upon convergence, it transitions to the original algorithm for self-supervised learning. This implicitly segments the learning procedure into two phases: the fast stage (RELA) and the slow stage (SSL). Moreover, Figure 5 depicts the dynamics of the training process.

0

1000

2000

3000

4000

Training Steps

20

40

60

80

Top-1 Accuracy (%)

ReLA Training

ReLA Validation

Orig. Training

Orig. Validation

Fast Stage

Slow Stage

0

1000

2000

3000

4000

Training Steps

10

20

30

40

50

60

70

Top-1 Accuracy (%)

ReLA Training

ReLA Validation

Orig. Training

Orig. Validation

Fast Stage

Slow Stage

(a) RELA with CF10-T as the prior model

(b) RELA with Rand. as the prior model

K Experimental Details

K.1 Detailed Setup for Experiments in Section 3.2

We show the detailed setup for the experiments in Section 3.2 in Tables 6 and 7.

K.2 Detailed Setup for Experiments in Section 5

Training details for self-supervised learning methods. We show the details of training in Table 8.

Linear evaluation details. We show the details of training the linear model in Table 9.

L Batch PCA Reduction

To begin, we provide a full and rigorous mathematical framework for Principal Component Analysis (PCA).

L.1 Principal Component Analysis

Given a data matrix Y ∈ R n × d and the desired number of components k , where k ≤ d , PCA aims to extract a reduced data matrix Y reduced ∈ R n × k that retains the maximum variance from the original dataset.

First, the data needs to be centered by subtracting the mean of each feature. This can be mathematically represented as:

where 1 n is a column vector of ones with length n . Next, the covariance matrix of the centered data is computed:

To identify the principal components, eigendecomposition is performed on the covariance matrix:

where V is the matrix of eigenvectors and Λ is a diagonal matrix of eigenvalues. The eigenvectors are then sorted in descending order based on their corresponding eigenvalues. The top k eigenvectors are selected to form the projection matrix:

where v i represents the i -th eigenvector. Finally, the centered data is projected onto the new subspace:

The resulting reduced data matrix Y reduced contains the principal components of the original data. Each principal component is a linear combination of the original features, designed to capture as much variance as possible in the reduced space.

Algorithm 2 Batch PCA Reduction

Require: Data matrix Y ∈ R n × d , Number of components k

Ensure: Reduced data matrix Y reduced ∈ R n × k

1: n ← number of rows in Y

2: m ← BATCHSIZE {Pre-set Batch size}

4: Y centered ← Y -µ {Center the data}

3: µ ← 1 n ∑ n i =1 Y i, · {Compute the mean of Y }

5: Σ ← 0 d × d {Initialize the covariance matrix}

6: for i = 0 to n step m do

7: j ← min( i + m,n )

8: B ← Y centered [ i : j, · ]

9: Σ ← Σ + B ⊤ B {Update the covariance matrix}

10: end for

11: Σ ← Σ n -1 {Normalize the covariance matrix}

12: V ← eigenvectors of Σ corresponding to the largest k eigenvalues

13: Y reduced ← 0 n × k {Initialize the reduced data matrix}

14: for i = 0 to n step m do

15: j ← min( i + m,n )

16: B ← Y centered [ i : j, · ]

17: Y reduced [ i : j, · ] ← BV

18: end for

19: return Y reduced

Then we prove that the batch PCA we utilized here is exactly same to the standard PCA.

Proof. To prove that the Batch PCA algorithm achieves the same result as the standard PCA, we need to show that the covariance matrix and the reduced data matrix computed by the Batch PCA are equivalent to those computed by the standard PCA.

First, let's show that the covariance matrix Σ computed in the Batch PCA is equivalent to the covariance matrix C in the standard PCA.

In the standard PCA, the covariance matrix is computed as:

In the Batch PCA, the covariance matrix is computed as:

where B i is the i -th batch of the centered data matrix Y centered, and b is the number of batches.

Therefore, the covariance matrix computed by the Batch PCA is equivalent to the covariance matrix computed by the standard PCA.

Next, let's show that the reduced data matrix Y reduced computed in the Batch PCA is equivalent to the one computed in the standard PCA.

In the standard PCA, the reduced data matrix is computed as:

where W is the matrix of the top k eigenvectors of the covariance matrix.

In the Batch PCA, the reduced data matrix is computed as:

where V is the matrix of the top k eigenvectors of the covariance matrix Σ , and m is the batch size.

Since we have shown that the covariance matrices C and Σ are equivalent, their eigenvectors W and V are also equivalent. Therefore, the reduced data matrix computed by the Batch PCA is equivalent to the one computed by the standard PCA.

In conclusion, we have proven that the Batch PCA algorithm achieves the same result as the standard PCA by showing that the covariance matrix and the reduced data matrix computed by the Batch PCA are equivalent to those computed by the standard PCA.

M Analyze the Practical Data Augmentations

Here we presents a rigorous mathematical proof demonstrating that higher data augmentation intensity leads to increased overlap between samples from different classes. By modeling the sample distributions and augmentation process, we show that the variance of the augmented distributions increases with augmentation strength, resulting in wider and more overlapping distributions. We provide an exact calculation of the intersection point and approximate the overlap area using the cumulative distribution function of the standard normal distribution. Our theoretical analysis confirms the positive correlation between data augmentation intensity and inter-class overlap rate.

M.1 Assumptions and Definitions

Data-Augmented Views

Targeting 'Dog'

Targeting 'Cat'

Similar Views.

Different Targets.

Machine What should I learn?

Or

1

2

Human-Labeled

Self-Generated

Using Other Views

Assumption 1 . There are two classes C 1 and C 2 , with samples drawn from probability distributions P 1 and P 2 , respectively. The data augmentation intensity is denoted by α .

Definition 10 (Data Distribution and Augmentation) . The samples are satisfied X 1 ∼ P 1 , X 2 ∼ P 2 . The augmented samples are represented as X ′ 1 = X 1 + ϵ and X ′ 2 = X 2 + ϵ , where ϵ denotes the augmentation perturbation following the distribution Q α . The variance of Q α is proportional to α .

M.2 Augmented Distributions

Let P 1 ,α and P 2 ,α denote the augmented sample distributions:

where ∗ represents the convolution operation.

Variance of Augmented Distributions Let σ 2 α be the variance of Q α , with σ 2 α = kα , where k is a constant. The variances of the augmented distributions are:

Overlap Rate Definition Let R denote the overlap region. The overlap rate O ( α ) is defined as:

M.3 Increased Variance Leads to Increased Overlap

As α increases, σ 2 α increases, resulting in larger variances of P 1 ,α and P 2 ,α . This makes the distributions wider and more dispersed, increasing the overlap region.

Specific Derivation for One-Dimensional Gaussian Distributions Assume P 1 and P 2 are two Gaussian distributions:

The augmented distributions are:

M.4 Conclusion

Theorem 4 . As the data augmentation intensity α increases, the overlap rate O ( α ) between samples from different classes increases.

Proof. Increasing α leads to an increase in the variance of the sample distributions, making them wider and more likely to overlap.

Specifically, increasing α increases σ 2 P 1 + kα in the denominator, thereby decreasing the argument of Φ( · ) . Since Φ( z ) increases as z decreases for z > 0 , O ( α ) increases as α increases.

In summary, we have rigorously proven the positive correlation between data augmentation intensity and inter-class overlap rate from a statistical and probabilistic perspective.

M.5 Empirical Analysis for Real-world Datasets

Beyond the theoretical analysis for a simple case, we also provide empirical analysis for four realworld datasets under different intensities of data augmentation. We utilize TSNE [61] to visualize the (augmented) samples in Figure 7. All the results demonstrate that:

as minscale increases from 0.02 to 0.5, the data points gradually change from being tightly clustered to more dispersed. This indicates that higher data augmentation intensity expands the range of sample distribution;

when minscale=0.02, the data points of the two classes exhibit significant overlap, and the boundary becomes blurred. In contrast, larger minscale values such as 0.2 and 0.5 allow for better separation between classes;

Exact Calculation of Intersection Point Equating the two Gaussian probability density functions and solving for x :

Simplifying the equation yields an analytical solution for the intersection point. To simplify the analysis, we assume σ 2 P 1 = σ 2 P 2 , in which case the intersection point is ( µ 1 + µ 2 ) / 2 .

Area of Overlap Region Let ∆ µ = | µ 1 -µ 2 | . The overlap region can be represented using the cumulative distribution function (CDF) of the standard normal distribution, denoted as Φ( z ) :

The variance of the augmented distributions is σ 2 α = σ 2 P 1 + kα . Therefore, the approximate area of the overlap region is:

60

40

20

0

20

40

60

40

20

0

20

40

(a) Setting Minscale =

0

.

02

60

40

20

0

20

40

60

40

20

0

20

40

(b) Setting Minscale =

0

.

04

60

40

20

0

20

40

60

40

20

0

20

40

(c) Setting Minscale =

0

.

08

80

60

40

20

0

20

40

60

40

20

0

20

40

60

(d) Setting Minscale =

0

.

2

80

60

40

20

0

20

40

60

40

20

0

20

40

60

(e) Setting Minscale =

0

.

5

40

20

0

20

40

20

10

0

10

20

(f) Without using Augmentation

when data augmentation is not used (the last figure), there is a clear gap between the two classes, and the data points within each class are very compact. This suggests that the feature distribution of the original samples is relatively concentrated.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We show them in Section 1 and our Abstract.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We show them in Section 6.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [No]

Justification: Most of them are proven in Appendix.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We show the details of our techniques in Appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We use the public datasets.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We show the experimental details in Appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We show them in Section 5.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [No]

Justification: We provide most of them in Appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research presented in the paper adheres to all aspects of the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [No]

Justification: The paper does not comprehensively discuss the potential societal impacts, both positive and negative, of the work performed.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [No]

Justification: The paper does not describe any safeguards for the responsible release of data or models that carry a high risk for misuse.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: All assets utilized in this paper, including code, data, and models, have been properly attributed to their respective creators or original owners.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not introduce any new assets; therefore, the question of documentation is not applicable.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve any crowdsourcing experiments or research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve research with human subjects. Therefore, the discussion of potential risks, disclosure to subjects, and Institutional Review Board (IRB) approvals are not applicable.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers data set distillation to a more concise form for purposes of representational efficiency, partly as an attempt to consider what forms of deployment might benefit from such forms and more importantly seeking to offer a universal form of translator to extract such compact representations from diverse data sets and their resulting influence towards training efficiency. It includes what appears to be an attempt at a nearly comprehensive survey of related literature which is itself a nice part of the paper.\n\nSTRENGTHS:\n- Originality\nI didn't get a strong sense of significant originality, although that was sort of a vague sense and welcome further input from the authors. The five points were obviously clearly articulated in section one, I wondeer if you could color these with their specific relations to prior work?\n\n    - Quality\nThe paper was not standout or anything, but for those with interest in this less common application it might have usefulness.\n\n    - Clarity\nThe Table 1 was kind of awkwardly placed, it distracted from flow of paper and would possibly be better served as paragraphs instead of a table.\n\n    - Significance\nI had trouble interpreting the significance of the ReLA_D model for generic data set distillation, asking the authors for further input below. Otherwise a contribution of the work appeared to be associated with the demonstrations of impact of distillations towards training efficiency, which is sort of intuitive and probably more useful as a validation of their tool than any significant result.\n\nWEAKNESSES:\nThe way the paper is written it almost appears that authors are approaching the concepts of dataset distillation considerations (eg in Table 1) as if such matters are unexplored in literature, which I would sort of find surprising. I think the paper would have been better structured as leading with the ReLA-D model convention for data set distillation as a more prominent part of the writeup, including more detail as to how that relates to prior work.\n\nQUESTIONS:\nI am partly fascinated by scope not addressed by the paper as to where these forms of distilled data sets might be of added benefit due to their less noisy representations.\n\nMore of a hypothetical question, no need to answer: Are there any known applications where data augmentation is known to interfere with implementations? Perhaps that might be a starting point of where valuable sources of impact of this form of distillation could be considered.\n\nCan you talk further about the extent of novelty and related work for the ReLA-D tool?\n\nLIMITATIONS:\nI don't know if their is any strong interest in dataset distillations in practice where they may be used in production systems. If there are perhaps you could highlight furhter as it may make the importance of the paper more clear.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes to accelerate the training of self-supervised learning with a pre-trained teacher model and use the prediction of this teacher as a target. The method is motivated from a toy example that the training convergence is dominated by the variance of the random variable. Then, the authors conjure that maximizing the mutual information between samples and targets can accelerate the convergence, based on which the authors give their solution that adds a cosine similarity term between the outputs from the student model and the teacher to the base self-supervised learning loss. Experiments demonstrate that the proposed technique enhances the performance when there are only part of data.\n\nSTRENGTHS:\n1. The experiments are sufficient and satisfactory to demonstrate the effectiveness of the proposed method.\n2. Most of the claims are backed up with theoretical analysis.\n\nWEAKNESSES:\n1. I do not think the writing style of the article is proper for a technical paper. Overall speaking, it is over packaged from the motivation throughout to the final solution. They try to introduce a big picture in the abstract and introduction parts. However, readers can hardly know what they actually do until reading the method part. I recommend introducing the technical solution first and then discussing the underlying motivations.\n2. Technically, my major concern lies in the motivation of the proposed method. In Conjecture 1, the authors speculate that maximizing the mutual information between samples and targets can speed up training. This conjecture is based on an over simplified example that conducts classification for a Gaussian Mixture Model with 2 Gaussian distributions. It's true that diminishing the variance can result in faster training in this simple case, but it is far from the real-world cases, which would be much more complex. In this way, although the article involves extensive theoretical analysis, the core part, which connects the motivation and the final solution, is not well supported.\n3. In fact, this conjecture, as well as the final solution that minimizes the cosine loss with sample predictions and targets generated by a well-trained teacher, is contradictory to many recent studies. For example, in RDED [a], hard patches are selected to form the synthetic samples. After all, hard patches should have larger classification loss, corresponding to samples closer to the decision boundary in the toy case. Recent studies like [b] and [c] also demonstrate that weak trajectory or teacher models may benefit learning more. Given these studies, the conjecture of this article seems not so plausible without substantial evidences.\n4. Based on above analysis, I would like to believe the acceleration comes from additional supervision signals, which transforms a self-supervised learning problem to an almost fully supervised one, since a well-trained teacher is involved. If that is the case, the conclusion and technique proposed in the article would appear trivial, since similar conclusion and technique have been well studied even several years ago [d,e,f,g].\n5. The authors analyze *the optimal properties of distilled data*. Here an assumption that specifies the distillation method is missing. After all, different methods would result in different distill data. Not all of them satisfy the proposed properties.\n6. I do not think the introduction of dynamic dataset distillation is necessary. The authors randomly sample a portion of real data to form the training dataset and use different data in each epoch, which can be viewed as mini-batch training on a subset and is not closely related to dataset distillation. In fact, I think the settings in DiM [h] and [i] are more adhere to this concept. In other words, the setting in this article is actually not coupled with dataset distillation. Only using a subset of data can achieve satisfactory performance is the result of introduction of pre-trained teacher. The method is not well aligned with dataset distillation serving as the motivation. In this case, the results may not appear surprising since the method transforms a self-supervised learning problem to an almost fully supervised one.\n\n[a] On the Diversity and Realism of Distilled Dataset: An Efficient Dataset Distillation Paradigm (Peng Sun et al., CVPR 2024)\n\n[b] Dataset Quantization (Daquan Zhou & Kai Wang & Jianyang Gu et al., ICCV 2023)\n\n[c] SelMatch: Effectively Scaling Up Dataset Distillation via Selection-Based Initialization and Partial Updates by Trajectory Matching (Yongmin Lee et al., ICML 2024)\n\n[d] Self-Supervised Dataset Distillation for Transfer Learning (Dong Bok Lee & Seanie Lee et al., ICLR 2024)\n\n[e] Boosting Self-Supervised Learning via Knowledge Transfer (Noroozi et al., CVPR 2018)\n\n[f] Knowledge Distillation Meets Self-Supervision (Xu et al., ECCV 2020)\n\n[g] SEED: Self-supervised Distillation For Visual Representation (Fang et al., ICLR 2021)\n\n[h] DiM: Distilling Dataset into Generative Model (Kai Wang & Jianyang Gu et al., 2023) \n\n[I] Data Distillation Can Be Like Vodka: Distilling More Times For Better Quality (Xuxi Chen & Yu Yang et al., ICLR 2024)\n\nQUESTIONS:\nPlease refer to the weakness part above.\n\nLIMITATIONS:\nThe authors have mentioned that Conjecture 1 remains speculative. However, I do think this is an important limitation that serves as the core of the whole method and article.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses the scalability constraints in representation learning by proposing a novel Representation Learning Accelerator (RELA). Current paradigms, focusing separately on self-supervised learning and dataset distillation, overlook the potential of intermediate acceleration. The authors define ideal data properties for optimization and generalization, enabling effective transport of model-generated representations. RELA leverages a task- and architecture-agnostic public model to form a dynamic data subset, enhancing (self-)supervised learning. Empirical results show that using CLIP ViT B/16 as a prior model, RELA-aided BYOL can train a ResNet-50 from scratch with 50% of ImageNet-1K, surpassing full dataset performance. This approach improves representation learning efficiency, offering impactful implications for reducing data requirements and computational resources in model training.\n\nSTRENGTHS:\n[S1] The paper tackles critical issues in efficient training and dataset distillation (DD).\n\n[S2] RELA introduces a novel approach in dataset distillation, effectively bridging representation learning with data-efficient methods.\n\n[S3] The paper is clearly and effectively written.\n\n[S4] Comprehensive theoretical analysis supports the proposed claims. The paper explores analytical concepts in dataset distillation, employing information theory principles.\n\n[S5] Extensive experimental and ablation studies are conducted in both the main paper and appendix.\n\nWEAKNESSES:\n[W1] The paper lacks generalization ability to high-resolution datasets, such as 1Kx1K, which are common in practical datasets like clinical and aerial images.\n\n[W2] The practical applications of dataset distillation are not discussed. Demonstrating applicability in neural architecture search (NAS), continual learning, federated learning, and privacy preservation would be valuable for both the community and real-world scenarios.\n\n[W3] Comparisons with some state-of-the-art methods, including DREAM [a], DataDAM [b], and SeqMatch [c], are missing from Table 6. Including these comparisons would strengthen the evaluation of the proposed method.\n\n[W4] An analysis of the computational costs is absent. It is crucial to examine the training costs, including both memory and training time, for the proposed process and state-of-the-art methods to ensure data-efficient training algorithms.\n\n[W5] The theoretical analysis does not generalize to complicated data distributions, such as two arbitrary mixtures of Gaussian distributions or a Mixture of Generalized Gaussian distributions (MGG).\n\nUpon addressing these weaknesses, I will consider changing my initial rating to strong accept.\n\n-------------\n\nReferences:\n\n[a] Liu, Yanqing, et al. \"Dream: Efficient dataset distillation by representative matching.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.\n\n[b] Sajedi, Ahmad, et al. \"Datadam: Efficient dataset distillation with attention matching.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.\n\n[c] Du, Jiawei,  et al. \"Sequential subset matching for dataset distillation.\" Advances in Neural Information Processing Systems 36 (2024).\n\nQUESTIONS:\n1. Can RELA be extended to ImageNet-21K? If so, please provide some experimental results.\n\n2. How does the proposed framework perform on other downstream tasks, such as object detection or segmentation?\n\nLIMITATIONS:\nThe limitations are briefly discussed in the appendix. For more limitations, refer to the Weaknesses and Questions sections.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-09"}

comments: SUMMARY:
The authors propose a theoretically motivated dynamic distilled datasets. With use of these transportable representation the authors show that they can outperform the performance of the original dataset and in some cases even the complete dataset.

STRENGTHS:
- The methods have a sound theoratical basis, the proofs presented although for a narrow setup our quite interetsing. 

- The setup for experiments is well defined. 

- The results show extensive performance improvement.

WEAKNESSES:
- The writing style can improve and make the paper approachable. The notations are often terse and lack details about symbols until later. Especially for proofs in appendix.

- It might be more prudent to add experiments of larger scale.

QUESTIONS:
See weakness

LIMITATIONS:
The authors did not present a detailed study of where the approach can fail and limitation of their analysis.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-13