Efficiency for Free: Ideal Data Are Transportable Representations

Authors: peng  sun
yi  jiang
tao  lin
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 18:57:44

Abstract: Data, the seminal opportunity and challenge in modern machine learning, currently constrains the scalability of representation learning and impedes the pace of model evolution.
In this work, we investigate the efficiency properties of data from both optimization and generalization perspectives.
Our theoretical and empirical analysis reveals an unexpected finding: for a given task, utilizing a publicly available, task- and architecture-agnostic model (referred to as the `prior model' in this paper) can effectively produce efficient data.
Building on this insight, we propose the Representation Learning Accelerator (ReLA), which promotes the formation and utilization of efficient data, thereby accelerating representation learning.
Utilizing a ResNet-18 pre-trained on CIFAR-10 as a prior model to inform ResNet-50 training on ImageNet-1K reduces computational costs by $50\%$ while maintaining the same accuracy as the model trained with the original BYOL, which requires $100\%$ cost.
Our code is available at: \url{https://github.com/LINs-lab/ReLA}.

Main paper: 
 definitions and setup: 
 Let X denote the input data space with data distribution p data (x). Let p pos (x, x + ) denote the distribution of positive pairs, typically generated via data augmentation. The encoder ξ : X → R d maps inputs to d-dimensional representations. The similarity function q : R d × R d → R (e.g., dot product) measures similarity between representations. Given a positive pair (x, x + ), let {x - i } N i=1 be N negative samples drawn i.i.d. from p data (x). The InfoNCE loss is defined as: L InfoNCE (ξ, q) = -E (x,x + )∼ppos log e q(ξ(x),ξ(x + ))/τ N i=1 e q(ξ(x),ξ(x - i ))/τwhere τ > 0 is a temperature parameter. 
related work: 
 This section integrates two distinct deep learning areas: (a) techniques to condense datasets while preserving efficacy; (b) self-supervised learning methods that enable training models on unlabeled data.The objective of dataset distillation is to create a significantly smaller dataset that retains competitive performance relative to the original dataset.Refining proxy metrics between original and distilled datasets. Traditional approaches involve replicating the behaviors of the original dataset within the distilled one. These methods aim to minimize discrepancies between surrogate neural network models trained on both synthetic and original datasets. Key metrics for this process include matching gradients [72, 32, 70, 44] , features [63] , distributions [71, 73] , and training trajectories [11, 17, 22, 18, 68, 24] . However, these methods suffer from substantial computational overhead due to the incessant calculation of discrepancies between the distilled and original datasets. The optimization of the distilled dataset involves minimizing these discrepancies, necessitating multiple iterations until convergence. As a result, scaling to large datasets, such as ImageNet [20] , becomes challenging.Extracting key information from original into distilled datasets. A promising strategy involves identifying metrics that capture essential dataset information. These methods efficiently scale to large datasets like ImageNet-1K using robust backbones without necessitating multiple comparisons between original and distilled datasets. For instance, SRe 2 L [67] condenses the entire dataset into a model, such as pre-trained neural networks like ResNet-18 [26] , and then extracts the knowledge from these models into images and targets, forming a distilled dataset. Recently, RDED [56] posits that images accurately recognized by strong observers, such as humans and pre-trained models, are more critical for learning.Summary. We make the following observations regarding scalable dataset distillation methods utilizing various metrics: (a) a few of these metrics have proven effective for data distillation at the scale of ImageNet. (b) all these metrics require human-labeled data; (c) there is currently no established theory elucidating the conditions under which data distillation is feasible; (d) despite their success, the theory behind training neural networks with reduced data is underexplored.The primary objective of self-supervised learning is to extract robust representations without relying on human-labeled data. These representations should be competitive with those derived from supervised learning and deliver superior performance across multiple tasks.Contrasting self-generated positive and negative Samples. Contrastive learning-based methods implicitly assign a one-hot label to each sample and its augmented versions to facilitate discrimination.Since InfoNCE [47] , various works [25, 12, 13, 8, 31, 15, 74, 40, 9, 27] have advanced contrastive learning. MoCo [25, 13, 15] uses a momentum encoder for consistent negatives, effective for both CNNs and Vision Transformers. SimCLR [12] employs strong augmentations and a nonlinear projection head. Other methods integrate instance classification [8] , data augmentation [31, 74] , clustering [40, 9] , and adversarial training [27] . These enhance alignment and uniformity of representations on the hypersphere [64] .Asymmetric model-generating representations as targets. Asymmetric network methods achieve self-supervised learning with only positive pairs [30, 50, 14] , avoiding representational collapse through asymmetric architectures. BYOL [30] uses a predictor network and a momentum encoder. Richemond et al. [50] show BYOL performs well without batch statistics. SimSiam [14] halts the gradient to the target branch, mimicking the momentum encoder's effect. DINO [10] employs a self-distillation loss. UniGrad [59] integrates asymmetric networks with contrastive learning methods within a theoretically unified framework.The InfoNCE objective serves as a lower bound to the mutual information between representations of positive pairs: I(ξ(X); ξ(X + )) ≥ J (ξ, q) whereJ (ξ, q) = E (x,x + )∼ppos q(ξ(x), ξ(x + )) -τ • log N i=1 e q(ξ(x),ξ(x - i ))/τAs the number of negative samples N → ∞, this bound becomes tight, approaching the true mutual information I(ξ(X); ξ(X + )). 3. Optimal Encoder and Similarity Function Let (ξ * , q * ) denote the optimal encoder and similarity function that maximize J (ξ, q). Under the assumption of infinite negative samples and an expressive similarity function, the optimal similarity satisfies:q * (ξ * (x), ξ * (x + )) = log p pos (x, x + ) p data (x)p data (x + ) + Cwhere C is a constant independent of (x, x + ). This aligns q * with the pointwise mutual information (PMI) between x and x + . 4. Injectivity through Mutual Information Maximization Maximizing I(ξ(X); ξ(X + )) encourages the encoder ξ to capture as much information about X as possible. To ensure injectivity:• Sufficient Dimensionality: The representation dimension d must be at least as large as the intrinsic dimensionality of X .samples come from other images. The goal is to attract positive pairs and repel negative pairs, typically using the InfoNCE loss [47] :EQUATIONwhere cos(•) denotes cosine similarity, and τ is the temperature hyper-parameter. This formulation can be adapted for various methods, discussed below.MoCo [25, 13] . MoCo uses a momentum encoder for the target branch and a memory bank for storing previous representations. Negative samples are drawn from this memory bank. The gradient for sample u o 1 is:EQUATIONwheres v = exp (cos(u o 1 ,v m )/τ ) y m ∈V bank exp (cos(u o 1 ,y m )/τ )and N is the number of samples in the batch. SimCLR [12] . SimCLR shares weights between the target and online branches and does not stop back-propagation. It uses all representations from other images in the batch as negative samples. The gradient is:EQUATIONwheret v = exp (cos(v s ,u o 1 )/τ )y s ∈V batch \v s exp (cos(v s ,y s )/τ ) . If the gradient through the target branch is stopped, the second term vanishes.Unified Gradient. The gradient for these methods can be unified as:EQUATIONcomprising a weighted sum of positive and negative samples. The term -u t 2 pulls positive samples together, while v t ∈V s v v t pushes negative samples apart. The main difference between methods lies in the target branch used and the construction of the contrastive sample set V. 
revealing critical properties of efficient learning over data: 
 We begin by presenting formal definitions of supervised learning over a (efficient) dataset.Definition 1 (Supervised learning over data) .For a dataset D = (D X , D Y ) = {(x i , y i )} |D| i=1, drawn from the data distribution (X, Y ) in space (X , Y), the goal of a model learning algorithm is to identify an optimal model ϕ ⋆ that minimizes the expected error defined by:EQUATIONwhere ℓ indicates the loss function and ϵ denotes a predetermined deviation. This is typically achieved through a parameterized model ϕ θ , where θ denotes the model parameter within the parameter space Θ. The optimal parameter θ D is determined by training the model to minimize the empirical loss over the dataset:EQUATIONThe training process leverages an optimization algorithm such as stochastic gradient descent [51, 33] .Definition 2 (Data-efficient Learning) . Data-efficient learning seeks to derive an optimized/efficient dataset, denoted as S = (S X , S Y ) = {(x j , y j )} |S| j=1 , from the original dataset D. The objective is to enable models ϕ θ S trained on S to achieve the desired generalization performance, as defined in (1), with fewer training steps and a reduced computational budget compared to training on the original dataset D.To ease the understanding and our methodology design in Section 3.4 , we unify both conventional supervised learning and self-supervised learning as learning to map samples in D X to targets in D Y : this view forms 'supervised learning' from a data-centric perspective. Specifically, these two learning paradigms involve generating targets D Y = {y | y = ψ(x) s.t. x ∼ D X } and minimizing the empirical loss (2) . The only difference lies in the target generation models (or simply labelers) ψ:(a) Conventional supervised learning, referred to as human-supervised learning, generates targets via human annotation. Note that the targets are stored and used statically throughout the training. (b) Self-supervised learning (also see Footnote 2), e.g., BYOL [30] utilizes an Exponential Moving Average (EMA) version of the learning model ϕ θ to generate targets y = EMA[ϕ θ ](x). Note that the targets are dynamically changing during training as the model ϕ θ keeps evolving.This unified perspective allows us to jointly examine and address the inefficient training issue of (self-)supervised learning from a data-centric perspective, in which in Section 3.2 we first study the impact of samples D X and targets D Y on the model training process and then investigate whether and how a distilled dataset S = (S X , S Y ) can facilitate this process.To elucidate the ideal data properties of training on a dataset D, we examine the simple task over a bimodal Gaussian mixture distribution as a case study. We begin by defining the problem.Definition 3 (Bimodal Gaussian mixture distribution) . Given two Gaussian distributions N 0 (µ 1 , Σ 2 I) and N 1 (µ 2 , Σ 2 I), where µ 1 and µ 2 are the means and Σ 2 is the variance (here we set µ 1 = 1, µ 2 = 2 and Σ = 0.5). We define a bimodal mixture data G = (G X , G Y ) as:EQUATIONMoreover, we define a corresponding binary classification neural network model as: [1] • ReLU θ [2] x + θ [3] + θ [4] ,f θ (x) := σ θwhere σ(z) = 1 1+e -z is the sigmoid activation function; ReLU(z) = max(0, z) is the activation function for the hidden layer, which provides non-linearity to the model; θ [2] and θ [3] are the weights and biases of the hidden layer; θ [1] and θ [4] are the weights and biases of the output layer.Modern representation learning fundamentally relies on optimization (see Definition 1 ) . We show that modifications to the data can influence the convergence rate of the optimization process, thereby impacting the overall representation learning procedure. Furthermore, we try to uncover several key properties of data efficiency through our theoretical analysis of the case study. In the following, we denote the modified distribution byG ′ = (G ′ X , G ′ Y )and examine the altered samples G ′ X and corresponding targets G ′ Y independently.Investigating the properties of modified samples. The modification process here only rescales the variance of the original sample distribution G X defined in Definition 3 with new Σ (rather than the default 0.5), while let G ′ Y := G Y ; see explanations in Appendix E . Therefore, we examine the distilled samples G ′ X by setting the variable Σ within the interval (0, 1). Results in Figure 2 demonstrate that the distilled samples G ′ X with smaller variance Σ achieve faster convergence and better performance compared to that of G. To elucidate the underlying mechanism, we provide a rigorous theoretical analysis in Appendix B , culminating in Theorem 1 .Theorem 1 (Convergence rate of learning on efficient samples) . For the classification task stated in Definition 3 , the convergence rate for the model f θ trained t after steps over distilled data G ′ is:EQUATIONwhere ℓ denotes the MSE loss, i.e., ℓ(ŷ, y) := ∥ŷ -y∥ 2 , and f θ ⋆ indicates the optimal model, Õ signifies the asymptotic complexity. Modified samples characterized by a smaller value of Σ facilitate faster convergence.Investigating the properties of modified targets. On top of the property understanding for modified samples, we further investigate the potential of modified targets via G ′ . In detail, for modified samples, we consider the most challenging (c.f. Figure 2b ) yet the common case, namely G ′ X with Σ = 1 (see explanations in Appendix M ). For the corresponding modified targets G ′ Y , similar to the prior data-efficient methods [56, 67] , for any sample x drawn from G ′ X , we refine its label by assigning ŷ = ρ • f θ ⋆ (x) + (1 -ρ) • y. Here, ρ denotes the relabeling intensity coefficient, and f θ ⋆ represents a strong pre-trained model (simply, we utilize the model trained on the data in Figure 2c ).Theorem 2 (Convergence rate of learning on re-labeled data) . For the classification task as in Definition 3 , we have the convergence rate for the model f θ trained after t steps over modified data G ′ :EQUATION) Note that ρ controls the upper bound of the convergence rate, indicating that using modified targets with a higher value of ρ enables faster convergence.Results in Figure 2 illustrate that the modified targets G ′ Y with higher values of ρ lead to faster training convergence and better performance. See theoretical analysis in Theorem 2 and Appendix B .The empirical and theoretical investigations regarding the properties of modified samples G ′ X and targets G ′ Y in Section 3.2 are limited to a simplified case (as described in Definition 3 ) and may not extend to all practical scenarios, such as training a ResNet [26] on the ImageNet dataset [20] .Interestingly, we observe that the advantageous modifications of both samples G ′ X and targets G ′ Y converge towards a unified principle: minimizing or preventing any sample x from being labeled with multiple or inaccurate targets y. This principle emphasizes the importance of providing accurate and informative targets y for each sample x, as analyzed in Remark 1 , and suggests extending this insight to any complex dataset like S. X (Σ = 0.1) Figure 2 : Investigating modified samples with varied Σ values. Following [39] , Figure 2a visualizes the validation loss landscape within a two-dimensional parameter space, along with three training trajectories corresponding to different Σ settings. Figure 2b illustrates the performance of models trained using samples with varied Σ. The optimal case in our task, utilizing samples with Σ = 0.1 (which achieves the lowest validation loss in Figure 2b ), is visualized in Figure 2c , where the color bar represents the values of targets y. Y (ρ = 1.0) Figure 3 : Investigating modified targets with varied ρ values. We present a visualization of the validation loss landscape in Figure 3a , including three training trajectories that correspond to different ρ settings. Figure 3b illustrates the performance of models trained using targets with varying ρ values. The optimal scenario for our task, which uses targets with ρ = 1.0, is depicted in Figure 3c .Remark 1 (Ideal data properties avoid implicitly introduced gradient noise from data) . Intuitively, the semantic information within each sample x should be unique and not identical to another sample. Consequently, the exact target y, which represents the semantic information of x, should also be unique and informative. This implies the necessity of establishing bijective (or one-to-one) mappings between samples and their respective targets.In contrast, when a sample x (or several similar samples) is labeled with multiple different targets y, it may implicitly introduce noise into the gradient ∇ℓ(x, y), thereby hindering the optimization.However, real-world datasets often deviate from the ideal properties described above, as discussed in Remark 2 below and further analyzed in Appendix M .Remark 2 (Imperfect mappings and inaccurate targets in real-world datasets) . In practice, we observe that 'noisy mappings' between input samples and targets are prevalent in real-world datasets. As illustrated in Figure 6 , several common phenomena contribute to this issue:• Similar or identical input samples may be assigned different targets due to using data augmentations, which is common in both self-supervised and human-supervised learning settings. • Inaccurate targets may be generated, particularly in self-supervised learning scenarios.• In human-supervised learning, all samples within a class are mapped to a one-hot target.These imperfect mappings and inaccurate targets pose challenges to achieving optimal training efficiency and effectiveness for real-world datasets.Given insights from Remark 1 , an effective approach to generate efficient data S from original data D involves employing a high-quality labeler ψ to relabel each sample x within D. This process results in the formation of an optimized dataset S. However, the generalization ability of models trained on these optimized datasets S is not inherently assured. For a given labeler ψ, we derive the generalization bound, which quantifies the representation distance between models ϕ θ and ϕ ⋆ , trained on the optimized dataset S. This is presented in Theorem 3 (see Appendix D for the proof).Definition 4 (Representation distance a ) . We introduce our proposed metric asEQUATIONwhich quantifies the distance between a source model ϕ S and a target model ϕ T with respect to a dataset D, and the loss function is defined as ℓ(ŷ, y) := 1(ŷ ̸ = y).a Intuitively, a smaller DRep(ϕ S → ϕ T ; D) indicates that the model ϕ S can be transformed into ϕ T over data D via a linear model with relative ease. This also implies that ϕ S can achieve the same linear evaluation performance as ϕ T on data D.Theorem 3 (Generalization bound with labeler ψ) . Assumimg the model ϕ θ : X → Y belongs to a hypothesis space Φ. Then, for any δ ∈ (0, 1), with probability at least 1 -δ, we haveEQUATIONwhereB Target = D Rep (ψ → ϕ ⋆ ; D X ) , B Model = D Rep (ϕ θ → ψ; S X ) + 2R D X (Φ) + Õ(|D X | -1) , and B Sample = D TV (S X , D X ) . D TV represents the total variation divergence [4] , and R D X is the empirical Rademacher complexity.Drawing insights from Theorem 1 , 2 and 3, we define the properties of ideal data below: Definition 5 (Properties of ideal efficient data, including samples S X and targets S Y ) . To meet the objectives of Definition 2 , an ideal efficient data requires: ➀ Targets (S Y ):• generating targets ψ(x) from the labeler ψ that are accurate (i.e., align with human-annotating targets a and optimal model ϕ ⋆ ), aiming to minimize D Rep (ψ → ϕ ⋆ ; D X ) ; • the generated target ψ(x) should be informative to corresponding sample x (i.e., forming perfect bijective mappings with the original samples x), according to Remark 5 ;➁ Samples (S X ): low distribution disparity represented by D TV (S X , D X ) and high sample diversity denoted as |S X |, aiming at minimizing D TV (S X , D X ) .a Alignment with human-annotated targets can occur via direct prediction or linear transportability.To meet the requirement ➁ specified in Remark 5 , we utilize weak data augmentation on D X to generate the set S X , aimed at enhancing the diversity |S X | while ensuring the term D TV (S X , D X ) remains minimal. However, satisfying requirement ➀ is non-trivial, as capturing a labeler ψ that matches ϕ ⋆ is intractable (i.e., achieving D Rep (ψ → ϕ ⋆ ; D X ) = 0 is challenging).Fortunately, our experiments in real-world scenarios, as discussed in Section A , demonstrate that employing a prior model as the labeler ψ generally approximates ϕ ⋆ within the representation space. Therefore, we posit that introducing a prior model ψ to generate an efficient dataset S can typically accelerate the early stages of learning. Subsequently, the original dataset D should be employed.In (8) , the last term,D Rep (ϕ θ → ψ; S X ) + 2R D X (Φ) + Õ(|D X | -1 ) , includes 2R D X (Φ) and Õ(|D X | -1), which depend on the neural architecture and the size of D X , respectively, and can be considered constants. Thus, optimizing the model involves minimizing D Rep (ϕ θ → ψ; S X ) through training ϕ θ . A detailed technical solution is provided in Section 4 . 
methodology: 
 Building upon the theoretical insights from Section 3.2 , we propose our RELA (see Figure 1 ): (a) RELA-D ( ) is used to generate efficient data (c.f. Section 4.1 ); (b) RELA-F ( ) guides the models to train over the efficient data from RELA-D ( ) (c.f. Section 4.2 ).Motivated by two property requirements in Definition 5 , here we introduce our optimization-free synthesis process of both samples and targets in our RELA-D (see technical details in Appendix F ).Generating transportable representations as the targets. We argue that well-trained models (called prior models) on diverse real-world datasets using various neural network architectures and algorithms converge towards the same linear representation space. In other words, the generated pseudo representations R Y for samples D X using these prior models are linearly transportable to each other and to the human-annotating targets. Efficient and distribution-aligned sample generation. To satisfy requirement ➁ in Definition 5 efficiently, we employ basic data augmentations into data D X such as RandomResizeCrop with a minimum scale of 0.5 (as opposed to the default of 0.08) and RandomHorizontalFlip with p = 0.5.In this section, we showcase the significance of understanding ideal data properties and generated efficient dataset in assisting self-supervised learning, given this self-supervised paradigm on unlabeled data suffers from significant inefficiency issues compared to human-supervised learning [62] .Here we propose a plug-and-play method that can be seamlessly integrated into any existing selfsupervised learning algorithm, significantly enhancing its training efficiency by introducing an additional loss term. Formally, the loss function is defined as follows:EQUATIONwhere ℓ(z, y) := 1 -z • y/(∥z∥∥y∥) be the loss function, L SSL denotes the loss specified by any self-supervised learning method, respectively. Furthermore, the data D = (D X , R Y ) are collected using the strategy outlined in Section 4.1 , with updates occurring at each k-th epoch.The dynamic coefficient λ ∈ {0, 1} divides the training process into two distinct stages. Initially, λ is set to 1 to emphasize L RELA , assuming its crucial role in the early learning phase. As the model ϕ θ improves and self-generated targets become more reliable in L SSL , an adaptive attenuation algorithm adjusts λ to 0 (note that the initial λ is tuning-free for all cases and see Appendix J for details). As a result, only a single loss term in ( 9) is calculated, ensuring no extra computational cost with RELA.To enhance the recognition of RELA-aided algorithms, we re-denote those that are used in their names. For example, the BYOL algorithm [30] , when enhanced with RELA, is re-denoted as BYOL ( ). Furthermore, as the prior models downloaded from the internet are not consistently robust, the aforementioned dynamic setting of λ also prevents the model ϕ θ from overfitting to potentially weak generated targets. The efficacy of our proposed RELA is empirically validated in Section 5 . 
experiments: 
 This section describes the experimental setup and procedures undertaken to test our hypotheses and evaluate the effectiveness of our proposed methodologies.Experimental setting. We list the settings below (see more details in Appendix K ).• Datasets: For low-resolution data (32×32), we evaluate our method on two datasets, i.e., CIFAR-10 [35] and CIFAR-100 [34] . For high-resolution data, we conduct experiments on two large-scale datasets including Tiny-ImageNet (64 × 64) [36] and full ImageNet-1K (224 × 224) [20] , to assess the scalability and effectiveness of our method on more complex and varied datasets.• Neural network architectures: Similar to prior works/benchmarks of dataset distillation [56] and selfsupervised learning [57, 19] , we use several backbone architectures to evaluate the generalizability of our method, including ResNet-{18, 50, 101} [26] , EfficientNet-B0 [58] , MobileNet-V2 [52] ,ViT [21] , and a series of CLIP-based models [49] . These architectures represent a range of model complexities and capacities, enabling a comprehensive assessment of our approach.• Baselines: Referring to a prior widely-used benchmark [57, 19] , we consider several state-of-the-art methods as baselines for a broader practical impact, including: SimCLR [12] , Barlow Twins [69] , BYOL [30] , DINO [10] , MoCo [25] , SimSiam [14] , SwAV [9] , and Vicreg [3] .• Evaluation: Following previous benchmarks and research [57, 19, 12, 3] , we evaluate all the trained models using offline linear probing strategy to reflect the representation ability of the trained models, and ensure a fair and comprehensive comparison with baseline approaches.• Implementation details: We implement our method by extending a popular self-supervised learning open-source benchmark [57] and use their configurations therein. This includes using AdamW as the optimizer, with a mini-batch size of 128 (except for ImageNet-1K, where we use a mini-batch size of 512). We implement our method through PyTorch [48] , and all experiments are conducted on NVIDIA RTX 4090 GPUs. See more detailed configurations and hyper-parameters in Appendix K .Recall that our RELA-D ( ), as illustrated in Figure 1 and Section 4.1 , requires an unlabeled dataset and any pre-trained model freely available online to generate the efficient dataset. To justify the superior performance and generality of our RELA across various unlabeled datasets using prior models with different representation abilities, our comparisons in this subsection start with BYOL [30] 2 and then extend to other self-supervised learning methods.Table 1 demonstrates the efficacy and efficiency of our RELA in facilitating the learning of robust representations. Overall, BYOL ( ) consistently outperforms the original BYOL when trained with a reduced budget. In certain cases, such as on CIFAR-100, BYOL ( ) employing only 10% of the budget can surpass the performance of BYOL-trained models using the entire budget Specifically:(a) A stronger prior model (e.g., CLIP) enhances the performance of RELA more effectively than a weaker model (e.g., Rand.); (b) Our RELA is not sensitive to the prior knowledge. For instance, using CF10-T as the prior model can achieve competitive performance compared to that trained on extensive datasets (e.g., CLIP); Cross-architecture generalization. RELA-D ( ) generates efficient datasets using a specific neural architecture. To evaluate the generalization ability of these datasets, it is essential to test their performance on various architectures not used in the distillation process. Table 2 presents the performance of our RELA in conjunction with various prior models and trained model architectures, demonstrating its robust generalization ability. Specifically:(a) The integration of RELA always enhances the performance of original BYOL; (b) Our RELA method exhibits minimal sensitivity to the architecture of the prior model, as evidenced by the comparable performance of BYOL ( ) using both ViT-based and ResNet-based models.Combining RELA across various self-supervised learning methods. To demonstrate the effectiveness and versatility of RELA in enhancing various self-supervised learning methods, we conduct experiments with widely-used techniques. Table 3 presents the results, highlighting the robust generalization capability of RELA. Our findings consistently show that RELA improves the performance of these methods while maintaining the same budget ratio, emphasizing its potential on learning using unlabeled data. Additionally, we provide the results when combining RELA with human-supervised learning in Appendix I . 
conclusion and limitation: 
 In this paper, to address the Problem 1 , we investigate the optimal properties of data, including samples and targets, to identify the properties that improve generalization and optimization in deep learning models. Our theoretical insights indicate that targets which are informative and linearly transportable to strong representations (e.g., human annotations) enable trained models to exhibit robust representation abilities. Furthermore, we empirically find that well-trained models (called prior models) across various tasks and architectures serve as effective labelers for generating such targets. Consequently, we propose the Representation Learning Accelerator (RELA), which leverages any freely available prior model to generate high-quality targets for samples. Additionally, RELA can enhance existing (self-)supervised learning approaches by utilizing these generated data to accelerate training. However, our theoretical analysis is restricted to the simplified scenario described in Definition 3 , which has limited applicability in real-world contexts.  
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [No] Justification: The paper does not describe any safeguards for the responsible release of data or models that carry a high risk for misuse.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: All assets utilized in this paper, including code, data, and models, have been properly attributed to their respective creators or original owners.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.• For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA]Justification: The paper does not introduce any new assets; therefore, the question of documentation is not applicable.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA]Justification: The paper does not involve any crowdsourcing experiments or research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA]Justification: The paper does not involve research with human subjects. Therefore, the discussion of potential risks, disclosure to subjects, and Institutional Review Board (IRB) approvals are not applicable.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.• We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
: 
 The available of massive datasets [20, 49] and recent advances in parallel data processing [28, 42] have facilitated the rapid evolution of large deep models, such as GPT-4 [1] and LVM [2] . These models excel in numerous learning tasks, attributable to their impressive representation capabilities. However, the emergence of vast amounts of data within the modern deep learning paradigm raises two fundamental challenges: (i) the demand for human annotations of huge datasets consumes significant social resources [45, 20, 43] ; (ii) training large models with increasing data and model capacity suffers from intensive computational burden [6, 16, 55] .The community has made considerable efforts to enhance learning efficiency. Self-supervised learning methods [12, 69, 30, 10, 25, 14, 9, 3] , with their superior representation learning devoid of human annotations via the self-learning paradigm, attempt to tackle the challenge (i). Concurrently, research has been conducted to mitigate data efficiency issues in challenge (ii): dataset distillation approaches [65, 56, 11, 72, 53, 54] have successfully synthesized a small distilled dataset, on which models trained on this compact dataset can akin to one trained on the full dataset.However, challenges (i) and (ii) persist and yet are far from being solved [43, 45, 6, 16, 55] , particularly the intervention of these two learning paradigms. In this paper, we identify two issues: (a) inefficiency in the self-supervised learning procedure compared to conventional supervised learning arises due to sub-optimal self-generating targets [30, 62] ; (b) although training on the distilled dataset is efficient and effective, the distillation process of optimization-based approaches [11, 72, 37] is computationally demanding [18, 56] , often surpassing the computational load of training on the full dataset. This limitation restricts its potential to accelerate representation learning. To tackle these challenges, we propose a novel open problem in the domain of representation learning:Problem 1 (Accelerating Representation Learning through Free Models) . According to the No Free Lunch theorem [66] , it is evident that accelerating the learning process without incorporating prior knowledge is inherently challenging. Fortunately, numerous publicly available pre-trained models can be accessed online, offering a form of free prior knowledge. Despite this, effectively utilizing these models poses several implicit challenges, as these models may not be directly relevant to our target learning task, or they may not be sufficiently well-trained. This leads to a question: How can we leverage task-and architecture-agnostic publicly available models to accelerate representation learning for a specific task?To address Problem 1 , we propose RELA to utilize a freely available model downloaded from the internet to generate efficient data for training. This approach aims to accelerate training during the initial stages by effectively leveraging these generated data, thereby establishing a rapid pathway for representation learning (see Figure 1 ). Specifically, we list our five key contributions below as the first step toward bridging representation learning with data-efficient learning:(a) Revealing beneficial/detrimental data properties for efficient/inefficient (self-)supervised learning (see Section 3.2 ). We present a comprehensive analysis of linear models, demonstrating that data properties significantly influence the learning process by impacting the optimization of model training.Our findings reveal that modifications to the data can markedly enhance or impair this optimization. Additionally, we indicate that optimal training necessitates specific data properties-perfect bijective mappings between the samples and targets within a dataset.(b) Identifying the inefficiency problems of (self-)supervised learning from a data-centric perspective (see Section 3.3 ) . Specifically, we identify several factors contributing to the inefficiencies in (self-)supervised learning over real-world data. For instance, prevalent data augmentation techniques in modern deep learning can introduce a 'noisy mapping' issue, which may exacerbate the negative effects associated with inefficient data.(c) Generalization bound for models trained on optimized efficient data (see Section 3.4 ) . Although the efficiency properties of data do not inherently ensure the generalization of the trained model, i.e., efficient data alone cannot guarantee generalization ability, we present a generalization bound to analyze models trained on such data.(d) A novel method RELA to generate and exploit efficient data (see Section 4 ) . Leveraging our theoretical insights regarding the bounds of generalization and convergence rate, we introduce RELA, a novel optimization-free method tailored to efficiently generate and effectively exploit efficient data for accelerating representation learning.(e) An application of our RELA: accelerating (self-)supervised learning (see Section 5 and Appendix I ). Extensive experiments across four widely-used datasets, seven neural network ar-chitectures, eight self-supervised learning algorithms demonstrate the effectiveness and efficiency of RELA. Training models with RELA significantly outperforms training on the original dataset with the same budget, and even exceeds the performance of training on higher budget. 
 
 Appendix: We conduct ablation studies to understand the impact of each component of RELA on performance. Empirical representation similarity. Our foundational assumption of our RELA is that the representations of well-trained models, developed using various neural network architectures and algorithms on diverse real-world datasets, exhibit linear transportability to one another. To test this hypothesis, we assess representation similarity, a metric that quantifies the linear transferability between pre-trained models. The results, depicted in Figure 4a , demonstrate that representations from robust models (e.g., CLIP) can be effectively transferred to less robust models (e.g., CF10-T). This finding aligns with our results in Table 1 , showing that leveraging powerful models (e.g., CLIP and IN1K-T) consistently enhances learning in models trained on datasets with limited knowledge, such as CF-10.Combining RELA and BYOL with static λ setting strategies. The coefficient λ, as introduced in Section 4.2 , is pivotal in controlling the weight of the RELA phase during training. To assess the robustness of our adaptive strategy, which dynamically adjusts λ, we compare it to a static λ setting strategy. The results in Figure 4b indicate that larger (smaller) RELA weights are advantageous when using a strong (weak) prior model. Nonetheless, static settings lack generalizability across various scenarios, whereas our adaptive strategy demonstrates superior generalization capabilities.Analysis of the computational cost when using RELA with different prior models. To validate that our RELA, in conjunction with various prior models, can effectively reduce computational costs in self-supervised learning, we conducted experiments comparing the computational expense required for BYOL ( ) to achieve equivalent performance to the original BYOL trained with a full budget. The results, illustrated in Figure 4c , consistently demonstrate that RELA assists BYOL in lowering training costs while maintaining equivalent performance levels. Furthermore, it is evident that employing robust prior models consistently leads to greater reductions in training budgets.In this section, we prove a slightly modified version of Theorem1, extending the distribution to Generalized Gaussian Mixture(GGM) and making some assumptions for technical reasons. Yet this proof could still reflect the essential of the theorem.Notation N(µ, α, β) denotes the generalized Gaussian distribution with pdf β 2αΓ(1/β) e -(|x-µ|/α) β , B for Bernoulli distribution. We focus on the 1-dim situation. Assume that µ 1 < µ 2 . Define the original data distribution(N 0 = N(µ 1 , α 0 , β 0 ) andN 1 = N(µ 2 , α 0 , β 0 )) G := {(x, y) | y ∼ 2 • B(1, 1 2 ) -1, x ∼ 1 -y 2 • N 0 + 1 + y 2 • N 1 }and the modified one(N ′ 0 = N(µ 1 , α, β) and N ′ 1 = N(µ 2 , α, β)): G ′ := {(x, y) | y ∼ 2 • B(1, 1 2 ) -1, x ∼ 1 -y 2 • N ′ 0 + 1 + y 2 • N ′ 1 } .Our task is predicting y given x. Note that y ∈ {±1}, which is a bit different from the definition in Section 3.2. In 1-dim situation, we just need one parameter for this classification task, so define f θ (x) := sign(x + θ) to fit the distribution. We could compute the generalization loss on original distribution:EQUATION, we have:L(f θ ) -L(f θ ⋆ ) = ( µ 2 -µ 1 2α 0 - µ 2 -µ 1 2α 0 dF - - θ+µ 1 α 0 - θ+µ 2 α 0 dF )/2 ≤ C 1 • (θ -θ ⋆ ) 2 (or C ′ 1 |θ -θ ⋆ |) where C 1 , C ′1 are constants, F 0 F 1 F denote the CDF of N 0 N 1 and N(0, 1, β 0 ) respectively. The inequality above is due to the fact that function h(x) = (1 -1 dF - x+1x-1 dF )/x 2 has limits at 0 and so is bounded.For a dataset {(x i , y i )} n i=1 , set the loss functionL(θ) = 1 n n i=1 ℓ [y i (x i + θ)], ℓ(v) = 1 2 (1 -v) 2 .We apply the stocastic gradient descent algorithm and assume the online setting (n = 1): at step t draw one sample (x t , y t ) from G ′ then use the gradient ∇L(θ t ) to update θ (η ∈ (0, 1), t ∈ N):θ t+1 = θ t -η∇L(θ t ), ∇L(θ t ) = θ + (x t -y t ).It can be observed that randomness of x leads to noies on gradient.We prove the proposition that lower variance of GG can make convergence faster, i.e. E [L(f θt ) -L(f θ ⋆ )] is bounded by an increasing function of variance (t fixed).Proof. From above, we could getθ t = (1 -η) t θ 0 -η (x t-1 -y t-1 ) + (1 -η)(x t-2 -y t-2 ) + • • • + (1 -η) t-1 (x 0 -y 0 )and so :E [L(f θt ) -L(f θ ⋆ )] ≤ C 1 E (θ t -θ ⋆ ) 2 = C 1 E {   (1 -η) t (θ 0 -θ ⋆ ) -η t j=1 (1 -η) j-1 (x t-j -y t-j + θ ⋆ )   2 } = C 1 E   (1 -η) 2t (θ 0 -θ ⋆ ) 2 + η 2 t j=1 (1 -η) 2(j-1) (x t-j -y t-j + θ ⋆ ) 2   = C 1 (1 -η) 2t (θ 0 -θ ⋆ ) 2 + η (2 -η) (1 -(1 -η) 2t ) α 2 Γ(3/β) Γ(1/β) + 1 - µ 2 -µ 1 2 2The last two equalities is due to the fact that for (x, y)∼ G ′ E [x -y + θ ⋆ ] = 0 , E (x -y + θ ⋆ ) 2 = α 2 Γ(3/β) Γ(1/β) + 1 - µ 2 -µ 1 2 2 .In this subsection, we conduct some qualitative analysis on the nonlinear case. The setting is the same as that in Section 3.2. We point out the differences compared with the linear case above:x ∈ R d , y ∈ {0, 1} and f θ (x) := σ θ [1] • ReLU θ [2] x + θ [3] + θ [4] where σ(z) = 1 1+e -z is the sigmoid function; ReLU(z) = max(0, z) is the activation function for the hidden layer, which provides non-linearity to the model; θ [2] and θ [3] are the weights and biases of the hidden layer; θ [1] and θ [4] are the weights and biases of the output layer.To make things explicit, we still assume the online setting and set the loss function L(θ) = 1 2 (f θ (x)y) 2 . Assume after some iterations, θ [2] • µ 1 + θ [3] < 0 and θ [2] • µ 2 + θ [3] > 0 (coordinate-wise). In this situation, we could see that if x is close to its mean(µ 1 or µ 2 ), the sign of ReLU θ [2] x + θ [3] will be the same as y. So f θ will become an optimal classifier if θ [1] → +∞ and θ [4] → -∞. We focus on θ [1] , using SGD:θ [1] t+1 = θ [1]t -η ∂L ∂θ [1] , ∂L ∂θ [1] = (f θ (x) -y)σ(1 -σ)ReLU θ [2] x + θ [3] Note that we drop the variable value in σ(•) to make the expression more compact.Then we can analyze the phenomenon qualitatively: larger Σ will make convergence slower. The reason is that the larger Σ is, when x is drawn from N 1 (y = 1), θ [2] x + θ [3] < 0 is more likely to happen(i.e. straying far away from the mean), causing θ [1] to stop updating; what's worse, when x is drawn from N 0 (y = 0), with larger probability θ [2] x + θ [3] > 0 which will make θ [1] to go in the opposite direction. In summary, it is Σ that makes the gradient noisy thus impacts the convergence rate.In essence, the theoretical results in [41] could also be interpreted as a proof of the theorem. [41] study the learning of a two-layer ReLU neural network for k-class classification via stochastic gradient descent (SGD), assuming that each class corresponds to l patterns(distributions), with every two of the k × l distributions of the input data are separated by a distance δ. Below is the main theorem in [41] : Proposition 1 . Suppose some assumptions are satisfied, then for every ϵ > 0, there is M = poly(k, l, 1/δ, 1/ϵ) such that for every m ≥ M , after doing a minibatch SGD with batch size B = poly(k, l, 1/δ, 1/ϵ, log m) and learning rate η = 1 m•poly(k,l,1/δ,1/ϵ,log m) for T = poly(k, l, 1/δ, 1/ϵ, log m) iterations, with high probability, the generalization error(the probability that the model misclassifies) of the learned model is at most ϵ.Theoretical results above show that a larger δ helps network to learn more efficiently. (In our case, δ can be roughly viewed as the Mahalanobis distance between the two Gaussians, which is inverse proportion to the variance.) Also, Appendix D.2 of [41] demonstrates an example very similar to ours in which the input data is drawn from Gaussian distributions with different means, indicating increasing variance of the Gaussian causes the test accuracy to decrease and takes longer time to get a good solution.Use the same setting in Section B(linear case) except thatG ′ := {(x, y ′ ) | y ∼ 2 • B(1, 1 2 ) -1, x = 1 -y 2 • N 0 + 1 + y 2 • N 1 , y ′ = ρ • f θ * (x) + (1 -ρ)y} .In other words, we modify the distribution of y instead of x this time.We're going to prove that higher ρ can make convergence faster, i.e. E [L(f θt ) -L(f θ ⋆ )] is bounded by an decreasing function of ρ (t fixed).Proof. The crucial part x -y ′ + θ ⋆ = ρ(x -f θ * (x) + θ ⋆ ) + (1 -ρ)(x -y + θ ⋆ ), and in factE|x -y + θ ⋆ | -E|x -f θ * (x) + θ ⋆ | := ϵ 0 > 0 .Similarly, we can get bounds with ρ (see C ′ 1 in Section B):E [L(f θt ) -L(f θ ⋆ )] ≤ C ′ 1 E (1 -η) t |θ 0 -θ ⋆ | + (1 -(1 -η) t )|x -y ′ + θ ⋆ | ≤ C 2 (1 -η) t + C ′ 1 (1 -(1 -η) t ) [ρ • E|x -f θ * (x) + θ ⋆ | +(1 -ρ) • E|x -y + θ ⋆ |] ≤ C 2 (1 -η) t + (1 -(1 -η) t )(C 3 -C 4 ρ) where C 2 = C ′ 1 |θ 0 -θ ⋆ |, C 3 = C ′ 1 E|x -y + θ ⋆ |, C 4 = C ′ 1 ϵ 0 > 0.To see the impact of modifying y more clearly, we directly set ρ = 1 and conduct a similar analysis as in Section B.4. Let's still focus on θ [1] and use the same assumptions in Section B.4, then we have:θ [1] t+1 = θ [1]t -η ∂L ∂θ [1] ,∂L ∂θ [1] = (f θ (x) -f θ * (x))σ(1 -σ)ReLU θ [2] x + θ [3] Note y is replaced by f θ * (x). For instance x is drawn from N 0 (y = 0) but strays far away from the µ 1 , causing θ [2] x + θ [3] > 0. In this situation f θ * is likely to regard x as a sample from N 1 (i.e. f θ * (x) close to 1) thus making θ [1] to go in the right direction instead of the opposite. This explains why larger ρ can make convergence faster.We follow some proof steps in [5] . Let's begin by introducing some notations used in this section. Notation and Setup X is the input space, D S and D T are two distributions over X . Let Y = {0, 1}. H denotes a hypothesis class from X to Y. To simplify notations, ∀h, f ∈ H let ϵ S (h, f ) = E x∼D S [1(h(x) ̸ = f (x))], and εS (h, f ) be empirical error (ϵ T (h, f ), εT (h, f ) similar).Then we introduce some concepts and lemmas, most of which are from [5] .Definition 6 (H-divergence) . The H-divergence between two distributions D and D ′ is defined as:d H (D, D ′ ) = 2 sup h∈H |Pr D [I(h)] -Pr D ′ [I(h)]|whereI(h) = {x : h(x) = 1}.Definition 7 (Total Variation Distance) . For two distributions D and D ′ , the total variation distance of them is defined as:D TV (D, D ′ ) = sup A⊆F |Pr D (A) -Pr D ′ (A)|where F denotes the collection of all events in the probability space.Lemma 1 . For two distributions D and D ′ , by definition it's easy to see that:1 2 d H (D, D ′ ) ≤ D TV (D, D ′ )Definition 8 (Symmetric Difference Hypothesis Space) .H∆H := {g : X → {0, 1}|g(x) = h(x) ⊕ h ′ (x) ∀h, h ′ ∈ H} ⊕ denotes the XOR operation. Lemma 2 . ∀h, h ′ ∈ H, |ϵ S (h, h ′ ) -ϵ T (h, h ′ )| ≤ 1 2 d H∆H (D S , D T ) Proof. only need note that h(x) ⊕ h ′ (x) = |h(x) -h ′ (x)|, so sup g∈H∆H |Pr D S [I(g)] -Pr D T [I(g)]| = sup h,h ′ ∈H |ϵ S (h, h ′ ) -ϵ T (h, h ′ ) |this is done by definition.With above notations we can derive a general proposition related with Theorem 3.Proposition 2 . Assumimg H is a hypothesis class from X to Y and ϕ, ϕ ′ ∈ H, we have:E x∼D T ℓ(ϕ(x), ϕ ′ (x)) ≤ E x∼D S ℓ(ϕ(x), ϕ ′ (x)) + D TV (D S , D T )where loss function is ℓ(ŷ, y) := 1(ŷ ̸ = y).Proof. using the lemmas above,ϵ T (ϕ, ϕ ′ ) ≤ ϵ S ϕ, ϕ ′ + ϵ T ϕ, ϕ ′ -ϵ S ϕ, ϕ ′ ≤ ϵ S ϕ, ϕ ′ + 1 2 d H∆H (D S , D T ) ≤ ϵ S ϕ, ϕ ′ + D TV (D S , D T )To obtain some useful inequalities of generalization bound, we need to introduce the Rademacher complexity.Definition 9 (Rademacher complexity of a function class) . Given a sample of points S = {z 1 , z 2 , . . . , z m } ⊂ Z, and considering a function class F of real-valued functions over Z, the empirical Rademacher complexity of F given S is defined as:R S (F) = 1 m E σ sup f ∈F m i=1 σ i f (z i )where σ i are independent and identically distributed Rademacher random variables. In other words, for i = 1, 2, . . . , m, the probability that σ i = +1 is equal to the probability that σ i = -1, and both are 1 2 . Further, let P be a probability distribution over Z. The Rademacher complexity of the function class F with respect to P for sample size m is:R P,m (F) := E S∼P m [R S (F)]Lemma 3 (Generalization bound with Rademacher complexity) . Let F be a family of loss functions F = {(x, y) → ℓ((x, y), h) : h ∈ H} with ℓ((x, y), h) ∈ [0, 1] for all ℓ, (x, y) and h. Then, with probability 1 -δ, the generalization gap isL(h) -L(h) ≤ 2R U (F) + 3 log(2/δ) 2n ,for all h ∈ H and samples U of size n.The proof of this classical result could be found in most machine learning textbooks, like [46] . Sinceℓ(•) is 1-Lipschitz, we can derive that R U S (ℓ • H) ≤ R U S (H).Theorem 3 involves the representation distance, below we prove the triangle inequality for representation distance. Lemma 4 (Triangle inequality for Representation distance) . for any functions ϕ S , ϕ T , ϕ U and distribution D, we have:D Rep (ϕ S → ϕ T ; D) ≤ D Rep (ϕ S → ϕ U ; D) + D Rep (ϕ U → ϕ T ; D)Proof. Let's denote:d 1 = D Rep (ϕ S → ϕ U ; D) d 2 = D Rep (ϕ U → ϕ T ; D)By definition:d 1 = inf W1∈R m×n ,b1∈R m E x∼D ℓ(W 1 ϕ S (x) + b 1 , ϕ U (x)) d 2 = inf W2∈R m×n ,b2∈R m E x∼D ℓ(W 2 ϕ U (x) + b 2 , ϕ T (x))We need to show:D Rep (ϕ S → ϕ T ; D) ≤ d 1 + d 2Consider the composition of the transformations:ϕ S (x) W1,b1 ----→ ϕ U (x) W2,b2 ----→ ϕ T (x)The combined transformation can be written as:W 2 (W 1 ϕ S (x) + b 1 ) + b 2 = W 2 W 1 ϕ S (x) + W 2 b 1 + b 2Using the properties of the loss function ℓ, we can write:ℓ(W 2 W 1 ϕ S (x) + W 2 b 1 + b 2 , ϕ T (x)) ≤ ℓ(W 1 ϕ S (x) + b 1 , ϕ U (x)) + ℓ(W 2 ϕ U (x) + b 2 , ϕ T (x))This inequality holds for the reason that it can only break when the two items at right-hand side are both 0, in which the left side is also 0 due to rule of composition. Thus the inequality holds for all x and W 1 , b 1 , W 2 , b 2 .Taking expectations over x ∼ D:E x∼D ℓ(W 2 W 1 ϕ S (x)+W 2 b 1 +b 2 , ϕ T (x)) ≤ E x∼D ℓ(W 1 ϕ S (x)+b 1 , ϕ U (x))+E x∼D ℓ(W 2 ϕ U (x)+b 2 , ϕ T (x))Taking the infimum overW 1 , b 1 and W 2 , b 2 : inf W1,b1,W2,b2 E x∼D ℓ(W 2 W 1 ϕ S (x) + W 2 b 1 + b 2 , ϕ T (x)) ≤ d 1 + d 2Since the left-hand side is an upper bound for D Rep (ϕ S → ϕ T ; D), we have:D Rep (ϕ S → ϕ T ; D) ≤ d 1 + d 2Therefore, the triangle inequality holds for the metric D Rep .Now we are ready to prove Theorem 3.Proof. For arbitrary W, b, with probability 1 -δ:D Rep (ϕ θ → ϕ ⋆ ; X) ≤ ϵ X (W ϕ θ + b, ϕ ⋆ ) ≤ ϵ D X (W ϕ θ + b, ϕ ⋆ ) + 2R D X (Φ) + 3 log(2/δ) 2 |D X |Taking the infimum over W, b:D Rep (ϕ θ → ϕ ⋆ ; X) ≤ D Rep (ϕ θ → ϕ ⋆ ; D X ) + 2R D X (Φ) + 3 log(2/δ) 2 |D X |Using the triangle inequality for representation distance:D Rep (ϕ θ → ϕ ⋆ ; D X ) ≤ D Rep (ϕ θ → ψ; D X ) + D Rep (ψ → ϕ ⋆ ; D X )For arbitrary W, b, using the proposition above,D Rep (ϕ θ → ψ; D X ) ≤ ϵ D X (W ϕ θ + b, ψ) ≤ ϵ S X (W ϕ θ + b, ψ) + D TV (S X , D X )Taking the infimum over W, b:D Rep (ϕ θ → ψ; D X ) ≤ D Rep (ϕ θ → ψ; S X ) + D TV (S X , D X )Combining the above results, we have:D Rep (ϕ θ → ϕ ⋆ ; X) ≤ D Rep (ψ → ϕ ⋆ ; D X ) + D Rep (ϕ θ → ψ; S X ) + 2R D X (Φ) + 3 log(2/δ) 2 |D X | + D TV (S X , D X )Impact of initialization In this part we leverage the Neural Tangent Kernel (NTK) framework [29] to deduce that a neural network initialized closer to a target function f ⋆ converges faster during training. Under the NTK regime, neural networks exhibit linearized training dynamics, allowing us to predict how the network's output evolves during training.In the context of supervised learning, consider a neural network with parameters θ and output function f (x; θ), where x is the input. The goal is to approximate a target function f ⋆ (x) by minimizing a loss function, typically the mean squared error (MSE):L(θ) = 1 2 n i=1 (f (x i ; θ) -y i ) 2where {(x i , y i )} n i=1 is the training data and y i = f ⋆ (x i ). Under gradient descent with learning rate η, the parameter updates are:θ t+1 = θ t -η∇ θ L(θ t )In the NTK regime, where the network width tends to infinity, the network's output evolves linearly with respect to the parameters around initialization θ 0 [38] . We can approximate:f (x; θ) ≈ f (x; θ 0 ) + ∇ θ f (x; θ 0 ) ⊤ (θ -θ 0 )This linearization allows us to express the evolution of the network's output as:f t (x) = f 0 (x) -η t-1 s=0 n i=1 K(x, x i )(f s (x i ) -y i ) where f t (x) = f (x; θ t ) is the network output at time t and K(x, x ′ ) = ∇ θ f (x; θ 0 ) ⊤ ∇ θ f (x ′ ; θ 0 ) is the Neural Tangent Kernel.In continuous time (gradient flow), the training dynamics can be described by a differential equation:df t (x) dt = - n i=1 K(x, x i )(f t (x i ) -y i )Vectorizing over all training inputs, we have:df t dt = -K(f t -y)where f t = [f t (x 1 ), f t (x 2 ), . . . , f t (x n )] ⊤ ,y = [y 1 , y 2 , . . . , y n ] ⊤ and K is the NTK matrix with entries K ij = K(x i , x j ). This differential equation has the solution:f t -y = e -Kt (f 0 -y)This equation shows that the error f t -y decays exponentially over time, with the rate governed by the NTK matrix K. Since K is symmetric, we can decompose K into its eigenvalues {λ j } and corresponding eigenvectors {v j }:K = j λ j v j v ⊤ jthe solution becomes:f t -y = j e -λj t c j v j where c j = v ⊤ j (f 0 -y).The rate of convergence for each component of the error is proportional to the corresponding eigenvalue λ j of the NTK. Larger eigenvalues lead to faster decay. Initial error matters: the initial error f 0 -y scales the amplitude of the exponential terms. A smaller initial error means the network starts closer to the target function, reducing the time required for the error to decay to a specific threshold.Suppose we want the error to be less than ϵ:∥f t -y∥ ≤ ϵUsing the solution: ∥f t -y∥ ≤ ∥e -λmint ∥∥f 0 -y∥ where λ min is the smallest (positive) eigenvalue of K. Solving for time t:t ≥ 1 λ min ln ∥f 0 -y∥ ϵBy leveraging the NTK framework, we've shown that a better initialization(meaning the neural network's initial output function is closer to the target function f ⋆ ) leads to faster convergence during training. This is because the training dynamics under NTK are linear, and the exponential error decay is directly influenced by the magnitude of the initial error. Analysis above implies that if we can train a network that is closer to the target function, it will converge faster to the target function.Now let's look at the Rademacher complexity appeared above more carefully. Let 1-Lipschitz positive homogeneous activation σ i be given, andH := x → σ L (W L σ L-1 (• • • σ 1 (W 1 x) • • • )) : ∥W i ∥ F ≤ B, x ∈ R d .Then using Theorem 1 in [23] , for samples S of size m we have bound for the empirical Rademacher complexity:R S (H) ≤ 1 + √ 2L ln 2 m B L ∥X∥ Fwhere X ∈ R d×m is the input data matrix, and ∥ • ∥ F denotes the Frobenius norm.If we further assume that the data are drawn from the distribution G (stated in Section 3.2) with covariance Σ (for simplicity let µ 1 = -µ, µ 2 = µ), then we can bound the Rademacher complexity of H with respect to G:EQUATIONThe right part of the last inequality is an increasing function with respect to Σ, and Γ(•) denotes the gamma function, M (•, •, •) is the Kummer's confluent hypergeometric function, given by:M (a, b, z) = ∞ n=0 a (n) z n b (n) n! = 1 F 1 (a; b; z),where:a (0) = 1, a (n) = a(a + 1)(a + 2) • • • (a + n -1),is the rising factorial.Remark. Usually, the Rademacher complexity R S (ℓ • F) could be bounded by of R S (F). For example, ifℓ is L-lipschitz, then R S (ℓ • F) ≤ L • R S (F).That's why we directly compute the Rademacher complexity of H instead of ℓ • H.Dataset distillation seeks to create a condensed dataset that allows models to achieve performance comparable to those trained on the full dataset, but with fewer training steps. In this section, we will demonstrate that rescaling the variance of Gaussian distributions, as defined in Definition 3, does not affect the optimal performance of models trained on these rescaled data.Proof. Consider two Gaussian distributions N 0 (µ 1 , σ 2 I) and N 1 (µ 2 , σ 2 I) with means µ 1 = 1 and µ 2 = 2, and variance σ 2 . We define the bimodal mixture distribution G = (G X , G Y ) such that (x, y) is sampled according to:x = (1 -y) • x 0 + y • x 1 , with y ∼ Bernoulli(0.5), x 0 ∼ N 0 , x 1 ∼ N 1 .The decision rule for optimal classification is determined by the likelihood ratio test. For a given sample x, the log-likelihood ratio Λ(x) is given by:Λ(x) = log P (x | y = 1) P (x | y = 0) .Since x 0 and x 1 are drawn from Gaussian distributions, their probability density functions are:P (x | y = 0) = 1 (2πσ 2 ) d/2 exp - 1 2σ 2 ∥x -µ 1 ∥ 2 , P (x | y = 1) = 1 (2πσ 2 ) d/2 exp - 1 2σ 2 ∥x -µ 2 ∥ 2 .Substituting these into the log-likelihood ratio, we have:Λ(x) = log 1 (2πσ 2 ) d/2 exp -1 2σ 2 ∥x -µ 2 ∥ 2 1 (2πσ 2 ) d/2 exp -1 2σ 2 ∥x -µ 1 ∥ 2 .Simplifying, we obtain:Λ(x) = - 1 2σ 2 ∥x -µ 2 ∥ 2 + 1 2σ 2 ∥x -µ 1 ∥ 2 .Further simplification gives:Λ(x) = 1 2σ 2 ∥x -µ 1 ∥ 2 -∥x -µ 2 ∥ 2 .Since the optimal decision threshold for balanced classes (i.e., P (y = 1) = P (y = 0) = 0.5) is Λ(x) = 0, we set:∥x -µ 1 ∥ 2 -∥x -µ 2 ∥ 2 = 0.Expanding and rearranging, we derive:∥x∥ 2 -2µ 1 x + µ 2 1 -∥x∥ 2 + 2µ 2 x -µ 2 2 = 0.This simplifies to:2(µ 2 -µ 1 )x + (µ 2 1 -µ 2 2 ) = 0, 2(µ 2 -µ 1 )x = µ 2 2 -µ 2 1 , x = µ 2 2 -µ 2 1 2(µ 2 -µ 1 ).Solving yields:x = µ 2 + µ 1 2 .Therefore, the optimal decision boundary is x = 1.5, which is independent of the variance σ 2 . This completes the proof.Regarding efficiency, utilizing scaled data to train similar models with fewer training steps is proven in Section B. Additionally, rescaling each Gaussian distribution preserves their means, which aligns with the objectives of conventional distribution matching-based dataset distillation methods. These methods aim to distill data while maintaining the distributional properties, specifically their means.Recall that ϕ θ : R d → R n . We then introduce a transport matrix W ∈ R m×n and define the combined model as Wϕ θ (•) ∈ R m . During the training phase, the parameters W and θ are jointly optimized. However, as the dimension m increases, the computational complexity of the optimization grows rapidly. To address this issue, we propose reducing the dimensionality of the target matrix R Y from m to n, where n < m:EQUATIONwhere µ and σ denote the mean and standard deviation, respectively, of each column in the R Y . Practically, we use batch PCA to perform the computation shown in (10) , as illustrated in L.We aim to demonstrate that modern deep learning methods can effectively train models to serve as robust prior models by extracting sufficient information from samples as representations.Therefore, we poist the existence of a prior model ξ capable of losslessly extracting the information of samples D X when trained using the InfoNCE loss [47] , a method prevalently employed in contemporary deep learning algorithms [12] .Proof. To demonstrate that an encoder ξ trained with the InfoNCE loss preserves all information from the input data D X , we proceed as follows.Asymmetric network methods learn representations by maximizing the similarity of positive pairs without using negative samples. These methods require symmetry-breaking network designs to avoid representational collapse. A predictor h(•) is appended after the online branch, and the gradient to the target branch is stopped. The objective function is:EQUATIONRelation to BYOL [30] . BYOL uses a momentum encoder for the target branch, i.e., u t 2 = u m 2 in Eq. (15) . [14] . Simsiam shows that a momentum encoder is unnecessary and only applies the stop-gradient operation to the target branch, i.e., u t 2 = u d 2 in Eq.( 15). Unified Gradient. Despite the performance of asymmetric network methods, the avoidance of collapse solutions is not well understood. DirectPred [60] explores this by studying training dynamics and proposes an analytical solution for the predictor h(•).DirectPred formulates the predictor as h(v) = W h v, with W h calculated based on the correlation matrix E v (vv T ). The correlation matrix, F , is computed as the moving average for each batch:F ≜ v o ∈V∞ ρ v v o v oT, where ρ v is the moving average weight. Decomposing F into eigenvalues Λ F and eigenvectors U , W h is:EQUATIONwhere λ max is the max eigenvalue of F and ϵ is a hyper-parameter to boost small eigenvalues.DirectPred also derives the gradient:EQUATIONwhere -W T h u t 2 and v o ∈V∞ (ρ v u o 1 T v o )v o act as positive and negative gradients respectively, andλ = u o 1 T W T h u t 2 u o 1 T (F +ϵ 2 I)u o 1is a balance factor.Though negative samples are absent in the loss function, they emerge from the predictor network's optimization. The eigenspace of the predictor W h aligns with the feature correlation matrix F , encoding its information. During back-propagation, this encoded information functions as a negative gradient, influencing the optimization direction.Feature decorrelation methods have recently emerged as a novel approach in self-supervised learning. These methods aim to reduce redundancy among different feature dimensions to prevent collapse. Various loss functions have been proposed for this purpose. We examine their relations below.Relation to Barlow Twins [69] . Barlow Twins employs the following loss function:EQUATIONwhereW = 1 N v o 1 ,v s 2 ∈V batch v o 1 v sT2 is a cross-correlation matrix, C denotes the number of feature dimensions, and λ is a balancing hyper-parameter. The diagonal elements of W are encouraged to be close to 1, while the off-diagonal elements are forced towards 0.Despite appearing different, Eq. ( 18) operates similarly to previous methods from a gradient perspective, calculated as:EQUATIONwhereA = I -(1 -λ)W diag and (W diag ) ij = δ ij W ijis the diagonal matrix of W . Barlow Twins applies batch normalization instead of ℓ 2 normalization to the representation v.Relation to VICReg [3] . VICReg modifies Barlow Twins with the following loss function:EQUATIONwhereW ′ = 1 N -1 v o 1 ∈V batch (v o 1 -vo 1 )(v o 1 -vo 1 )T is the covariance matrix of the same view, std(v) i denotes the standard deviation of the i-th channel of v, γ is a constant target value, and λ 1 , λ 2 are balancing weights.The gradient is derived as follows:EQUATIONwhere ṽ = v -v is the de-centered sample, λ = 2λ1N 2 c(N -1) 2 , andB = N cλ(N -1) (2λ 1 W ′ diag + λ2 2 diag(1(γ -std(v o 1 ) > 0) ⊘ std(v o 1 ))).Here, diag(x) is a matrix with the vector x on its diagonal, 1(•) is the indicator function, and ⊘ denotes element-wise division.VICReg does not normalize v; instead, it uses de-centering and a standard deviation term in the loss function.Unified Gradient. Given the equivalence of v s and v o , the gradient for feature decorrelation methods can be unified as:EQUATIONwhere -u t 2 is the positive gradient, andv o ∈V batch u oT v o Nv o 1 is the negative gradient. λ is a balancing factor. The difference between methods lies in the subscript for the negative coefficient. Feature decorrelation methods function similarly to other self-supervised methods, with the positive and negative gradients derived from the diagonal and off-diagonal elements of the correlation matrix.While training on the distilled dataset is both efficient and effective, the distillation process in optimization-based approaches [11, 72, 65] is computationally intensive [18, 56] , often exceeding the computational load of training on the full dataset.In contrast, the synthetic data generated by our RELA framework requires a budget less than that of training for a single epoch (c.f. Section 4.1). This is because the synthesis budget of our RELA is equivalent to performing inference over the entire dataset D X . Consequently, this computational expense is negligible given that training epochs typically number around 100.We apply our RELA in labeled dataset distillation and human-supervised learning.Datasets and Neural Network Architectures. We conduct experiments on datasets of varying scales and resolutions.• Small-scale: We evaluate on CIFAR-10 (32 × 32) [35] and CIFAR-100 (32 × 32) [34] .• Large-scale: We utilize Tiny-ImageNet (64 × 64) [36] and ImageNet-1K (224 × 224) [20] .Consistent with previous works on dataset distillation [67, 73, 24] , we use ConvNet [24] and ResNet-{18,50} [26] as our backbone networks across all datasets. Specifically, Conv-3 is employed for CIFAR-10/100, and Conv-4 for Tiny-ImageNet and ImageNet-1K.Baselines. We compare our method with several SOTA distillation methods capable of scaling to large high-resolution datasets, including G-VBSM [53] , SRe 2 L [67], and RDED [56] . To the best of our knowledge, SRe 2 L, G-VBSM, and RDED are the only published works that efficiently scale to datasets of any size, making them our closest baselines. All distilled datasets synthesized from these baselines undergo the same post-training process. Results are reported in Table 5 . For our RELA, the prior models used for distillation are identical to the pre-trained models employed in the baseline methods.Table 5 demonstrates the superiority of our RELA, despite incorporating a zero-cost sample synthesis process. L ← λ • L RELA + (1 -λ) • L SSL {Control the RELA and the SSL states using λ} 11: end for In essence, this algorithm is designed to detect the convergence of RELA. Upon convergence, it transitions to the original algorithm for self-supervised learning. This implicitly segments the learning procedure into two phases: the fast stage (RELA) and the slow stage (SSL). Moreover, Figure 5 depicts the dynamics of the training process. We show the detailed setup for the experiments in Section 3.2 in Tables 6 and 7 . Training details for self-supervised learning methods. We show the details of training in Table 8 . Linear evaluation details. We show the details of training the linear model in Table 9 .To begin, we provide a full and rigorous mathematical framework for Principal Component Analysis (PCA).Given a data matrix Y ∈ R n×d and the desired number of components k, where k ≤ d, PCA aims to extract a reduced data matrix Y reduced ∈ R n×k that retains the maximum variance from the original dataset.First, the data needs to be centered by subtracting the mean of each feature. This can be mathematically represented as:Y centered = Y - 1 n 1 n 1 T n Ywhere 1 n is a column vector of ones with length n. Next, the covariance matrix of the centered data is computed: To identify the principal components, eigendecomposition is performed on the covariance matrix:C = 1 n -1 Y T centered Y centeredC = VΛV Twhere V is the matrix of eigenvectors and Λ is a diagonal matrix of eigenvalues. The eigenvectors are then sorted in descending order based on their corresponding eigenvalues. The top k eigenvectors are selected to form the projection matrix:W = [v 1 , v 2 , . . . , v k ]where v i represents the i-th eigenvector. Finally, the centered data is projected onto the new subspace:Y reduced = Y centered WThe resulting reduced data matrix Y reduced contains the principal components of the original data. Each principal component is a linear combination of the original features, designed to capture as much variance as possible in the reduced space. j ← min(i + m, n) 16: B ← Y centered [i : j, •] 17:Y reduced [i : j, •] ← BV 18: end for 19: return Y reduced Then we prove that the batch PCA we utilized here is exactly same to the standard PCA.Proof. To prove that the Batch PCA algorithm achieves the same result as the standard PCA, we need to show that the covariance matrix and the reduced data matrix computed by the Batch PCA are equivalent to those computed by the standard PCA.First, let's show that the covariance matrix Σ computed in the Batch PCA is equivalent to the covariance matrix C in the standard PCA.In the standard PCA, the covariance matrix is computed as:C = 1 n -1 Y T centered Y centeredIn the Batch PCA, the covariance matrix is computed as:Σ = 1 n -1 n-1 i=0 B T i B i = 1 n -1 B T 0 B 0 + B T 1 B 1 + . . . + B T b-1 B b-1 = 1 n -1 Y T centered Y centeredwhere B i is the i-th batch of the centered data matrix Y centered , and b is the number of batches.Therefore, the covariance matrix computed by the Batch PCA is equivalent to the covariance matrix computed by the standard PCA.Next, let's show that the reduced data matrix Y reduced computed in the Batch PCA is equivalent to the one computed in the standard PCA.In the standard PCA, the reduced data matrix is computed as:Y reduced = Y centered Wwhere W is the matrix of the top k eigenvectors of the covariance matrix.In the Batch PCA, the reduced data matrix is computed as:Y reduced =     B 0 V B 1 V . . . B b-1 V     =     Y centered [0 : m, •] Y centered [m : 2m, •] . . . Y centered [(b -1)m : n, •]     V = Y centered Vwhere V is the matrix of the top k eigenvectors of the covariance matrix Σ, and m is the batch size.Since we have shown that the covariance matrices C and Σ are equivalent, their eigenvectors W and V are also equivalent. Therefore, the reduced data matrix computed by the Batch PCA is equivalent to the one computed by the standard PCA.In conclusion, we have proven that the Batch PCA algorithm achieves the same result as the standard PCA by showing that the covariance matrix and the reduced data matrix computed by the Batch PCA are equivalent to those computed by the standard PCA.Here we presents a rigorous mathematical proof demonstrating that higher data augmentation intensity leads to increased overlap between samples from different classes. By modeling the sample distributions and augmentation process, we show that the variance of the augmented distributions increases with augmentation strength, resulting in wider and more overlapping distributions. We provide an exact calculation of the intersection point and approximate the overlap area using the cumulative distribution function of the standard normal distribution. Our theoretical analysis confirms the positive correlation between data augmentation intensity and inter-class overlap rate. 3. when data augmentation is not used (the last figure), there is a clear gap between the two classes, and the data points within each class are very compact. This suggests that the feature distribution of the original samples is relatively concentrated.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).Note that (1) BYOL is competitive across various datasets[30,3,57,12], and (2) various self-supervised learning methods can be unified in the same framework[59] (see our detailed analysis in Appendix G ). 


Decision: Some reviewers observe that the theoretical work is on a simplified case. However, as the authors argue effectively in rebuttal, it is valid to perform theoretical analysis on a simplified case, and then use empirical evaluation to determine if the theory does indeed transfer to practice. The rebuttal includes a nice improvement to the theory (generalizing to MMG), but of course that case still remains simple compared to practice.  But, to emphasize: this is not a weakness of the paper per se, the theory remains interesting in its own right, and as a motivation for the empirical method.

The reviewers all agree that the empirical work does indeed show improvements, and the rebuttal's additional experiments show results on additional datasets as requested by the reviewers, including a scale-up to ImageNet-21K.

Reviewer K expresses concern that "the acceleration comes from additional supervision signals, which transforms a self-supervised learning problem to an almost fully supervised one, since a well-trained teacher is involved".  If this were the case, it would reduce the significance of the paper, but the rebuttal argues well that the supervision signal on its own is insufficient, and indeed the observation that "Small-scale CIFAR models improve ImageNet" also rebuts the concern.

Reviews: review: paper_summary: paper_summary: The authors propose a theoretically motivated dynamic distilled datasets. With use of these transportable representation the authors show that they can outperform the performance of the original dataset and in some cases even the complete dataset.
strength_weakness: strength_weakness: - The writing style can improve and make the paper approachable. The notations are often terse and lack details about symbols until later. Especially for proofs in appendix.

- It might be more prudent to add experiments of larger scale.
questions: questions: See weakness
limitations: limitations: The authors did not present a detailed study of where the approach can fail and limitation of their analysis.
paper_summary: paper_summary: The paper addresses the scalability constraints in representation learning by proposing a novel Representation Learning Accelerator (RELA). Current paradigms, focusing separately on self-supervised learning and dataset distillation, overlook the potential of intermediate acceleration. The authors define ideal data properties for optimization and generalization, enabling effective transport of model-generated representations. RELA leverages a task- and architecture-agnostic public model to form a dynamic data subset, enhancing (self-)supervised learning. Empirical results show that using CLIP ViT B/16 as a prior model, RELA-aided BYOL can train a ResNet-50 from scratch with 50% of ImageNet-1K, surpassing full dataset performance. This approach improves representation learning efficiency, offering impactful implications for reducing data requirements and computational resources in model training.
strength_weakness: strength_weakness: [W1] The paper lacks generalization ability to high-resolution datasets, such as 1Kx1K, which are common in practical datasets like clinical and aerial images.

[W2] The practical applications of dataset distillation are not discussed. Demonstrating applicability in neural architecture search (NAS), continual learning, federated learning, and privacy preservation would be valuable for both the community and real-world scenarios.

[W3] Comparisons with some state-of-the-art methods, including DREAM [a], DataDAM [b], and SeqMatch [c], are missing from Table 6. Including these comparisons would strengthen the evaluation of the proposed method.

[W4] An analysis of the computational costs is absent. It is crucial to examine the training costs, including both memory and training time, for the proposed process and state-of-the-art methods to ensure data-efficient training algorithms.

[W5] The theoretical analysis does not generalize to complicated data distributions, such as two arbitrary mixtures of Gaussian distributions or a Mixture of Generalized Gaussian distributions (MGG).

Upon addressing these weaknesses, I will consider changing my initial rating to strong accept.

-------------

References:

[a] Liu, Yanqing, et al. "Dream: Efficient dataset distillation by representative matching." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

[b] Sajedi, Ahmad, et al. "Datadam: Efficient dataset distillation with attention matching." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

[c] Du, Jiawei,  et al. "Sequential subset matching for dataset distillation." Advances in Neural Information Processing Systems 36 (2024).
questions: questions: 1. Can RELA be extended to ImageNet-21K? If so, please provide some experimental results.

2. How does the proposed framework perform on other downstream tasks, such as object detection or segmentation?
limitations: limitations: The limitations are briefly discussed in the appendix. For more limitations, refer to the Weaknesses and Questions sections.
paper_summary: paper_summary: This paper proposes to accelerate the training of self-supervised learning with a pre-trained teacher model and use the prediction of this teacher as a target. The method is motivated from a toy example that the training convergence is dominated by the variance of the random variable. Then, the authors conjure that maximizing the mutual information between samples and targets can accelerate the convergence, based on which the authors give their solution that adds a cosine similarity term between the outputs from the student model and the teacher to the base self-supervised learning loss. Experiments demonstrate that the proposed technique enhances the performance when there are only part of data.
strength_weakness: strength_weakness: 1. I do not think the writing style of the article is proper for a technical paper. Overall speaking, it is over packaged from the motivation throughout to the final solution. They try to introduce a big picture in the abstract and introduction parts. However, readers can hardly know what they actually do until reading the method part. I recommend introducing the technical solution first and then discussing the underlying motivations.
2. Technically, my major concern lies in the motivation of the proposed method. In Conjecture 1, the authors speculate that maximizing the mutual information between samples and targets can speed up training. This conjecture is based on an over simplified example that conducts classification for a Gaussian Mixture Model with 2 Gaussian distributions. It's true that diminishing the variance can result in faster training in this simple case, but it is far from the real-world cases, which would be much more complex. In this way, although the article involves extensive theoretical analysis, the core part, which connects the motivation and the final solution, is not well supported.
3. In fact, this conjecture, as well as the final solution that minimizes the cosine loss with sample predictions and targets generated by a well-trained teacher, is contradictory to many recent studies. For example, in RDED [a], hard patches are selected to form the synthetic samples. After all, hard patches should have larger classification loss, corresponding to samples closer to the decision boundary in the toy case. Recent studies like [b] and [c] also demonstrate that weak trajectory or teacher models may benefit learning more. Given these studies, the conjecture of this article seems not so plausible without substantial evidences.
4. Based on above analysis, I would like to believe the acceleration comes from additional supervision signals, which transforms a self-supervised learning problem to an almost fully supervised one, since a well-trained teacher is involved. If that is the case, the conclusion and technique proposed in the article would appear trivial, since similar conclusion and technique have been well studied even several years ago [d,e,f,g].
5. The authors analyze *the optimal properties of distilled data*. Here an assumption that specifies the distillation method is missing. After all, different methods would result in different distill data. Not all of them satisfy the proposed properties.
6. I do not think the introduction of dynamic dataset distillation is necessary. The authors randomly sample a portion of real data to form the training dataset and use different data in each epoch, which can be viewed as mini-batch training on a subset and is not closely related to dataset distillation. In fact, I think the settings in DiM [h] and [i] are more adhere to this concept. In other words, the setting in this article is actually not coupled with dataset distillation. Only using a subset of data can achieve satisfactory performance is the result of introduction of pre-trained teacher. The method is not well aligned with dataset distillation serving as the motivation. In this case, the results may not appear surprising since the method transforms a self-supervised learning problem to an almost fully supervised one.

[a] On the Diversity and Realism of Distilled Dataset: An Efficient Dataset Distillation Paradigm (Peng Sun et al., CVPR 2024)

[b] Dataset Quantization (Daquan Zhou & Kai Wang & Jianyang Gu et al., ICCV 2023)

[c] SelMatch: Effectively Scaling Up Dataset Distillation via Selection-Based Initialization and Partial Updates by Trajectory Matching (Yongmin Lee et al., ICML 2024)

[d] Self-Supervised Dataset Distillation for Transfer Learning (Dong Bok Lee & Seanie Lee et al., ICLR 2024)

[e] Boosting Self-Supervised Learning via Knowledge Transfer (Noroozi et al., CVPR 2018)

[f] Knowledge Distillation Meets Self-Supervision (Xu et al., ECCV 2020)

[g] SEED: Self-supervised Distillation For Visual Representation (Fang et al., ICLR 2021)

[h] DiM: Distilling Dataset into Generative Model (Kai Wang & Jianyang Gu et al., 2023) 

[I] Data Distillation Can Be Like Vodka: Distilling More Times For Better Quality (Xuxi Chen & Yu Yang et al., ICLR 2024)
questions: questions: Please refer to the weakness part above.
limitations: limitations: The authors have mentioned that Conjecture 1 remains speculative. However, I do think this is an important limitation that serves as the core of the whole method and article.
paper_summary: paper_summary: This paper considers data set distillation to a more concise form for purposes of representational efficiency, partly as an attempt to consider what forms of deployment might benefit from such forms and more importantly seeking to offer a universal form of translator to extract such compact representations from diverse data sets and their resulting influence towards training efficiency. It includes what appears to be an attempt at a nearly comprehensive survey of related literature which is itself a nice part of the paper.
strength_weakness: strength_weakness: The way the paper is written it almost appears that authors are approaching the concepts of dataset distillation considerations (eg in Table 1) as if such matters are unexplored in literature, which I would sort of find surprising. I think the paper would have been better structured as leading with the ReLA-D model convention for data set distillation as a more prominent part of the writeup, including more detail as to how that relates to prior work.
questions: questions: I am partly fascinated by scope not addressed by the paper as to where these forms of distilled data sets might be of added benefit due to their less noisy representations.

More of a hypothetical question, no need to answer: Are there any known applications where data augmentation is known to interfere with implementations? Perhaps that might be a starting point of where valuable sources of impact of this form of distillation could be considered.

Can you talk further about the extent of novelty and related work for the ReLA-D tool?
limitations: limitations: I don't know if their is any strong interest in dataset distillations in practice where they may be used in production systems. If there are perhaps you could highlight furhter as it may make the importance of the paper more clear.

Comments: title: Thanks
Update to score
Reference
comment: Thanks for the detailed response and I really appreciate it. However, I still think the increasing of accuracy largely comes from the extra guidance of the teacher model. No matter whether it is trained on a large dataset, it is a well pre-trained teacher and bringing it to unsupervised tasks would be helpful. Thus, it is hard to say that the increasing of accuracy is due to the theorems in this article.

In other words, after reading the article, I cannot gain additional insights. After all, if we want to boost an unsupervised training task, bringing a trained teacher model is an easy and straightforward solution for us to try. I understand that the operations in this article may not be exactly the same as previous ones, but I do not think there is substantial difference in the overall framework.

Therefore, I still choose to maintain my score. I understand that there may be some divergence with other reviewers. I would respect the final decision made by the area chairs.
We appreciate your insightful feedback and thorough review of our paper. Your comments have been pivotal in enhancing our work and refining the proposed framework. We are committed to addressing any remaining concerns and welcome further discussions or clarifications.
We express our sincere appreciation for your insightful feedback and thorough review of our paper. Your comments have been instrumental in enhancing our work and refining the proposed ReLA method. In response to the specific questions ([Q1] and [Q2]) you raised, we have provided detailed explanations to address each concern comprehensively. Below, we summarize our key responses:

- **[R1] We drew insights from the theoretical analysis, designed an effective and efficient ReLA framework, and empirically validated it on extensive real-world datasets and downstream tasks. Our theoretical analysis and observed empirical results are well-aligned, motivating our ReLA design, the key contribution of the manuscript:**
    - Our theoretical insights, specifically detailed in Conjecture 1 and Definition 5, which indicate the importance of ***informative*** targets, are consistent with the conclusions drawn from the experimental analyses in Sections 4 and 6 of the referenced paper [1].
    - Furthermore, Theorem 4 suggests that using a combination of randomly selected samples and high-quality targets enhances performance, a finding corroborated by the empirical results in Table 1 of paper [1].
    - The experimental results shown in Figure 2 of paper [1] lack crucial experimental parameters and an introduction of the image distillation method. Hence, it is difficult to dismiss our theoretical results based solely on the empirical outcomes in Figure 2. Additionally, their analysis is confined to a single illustrative case, without further experiments to validate the observed phenomena.
    - Our substantial empirical gains align well with our theoretical predictions and insights. We provided extensive empirical results (Section 5) across various datasets, neural architectures, and learning algorithms, demonstrating that both weak and strong prior models can generate efficient data, significantly accelerating training through our ReLA.
    - Note that the empirical understanding preprint [1] was appeared after the NeurIPS 2024 submission deadline. Though we may share some overlapping observations (namely our theoretical analysis and [1]’s empirical study), such an overlap only strengthens our core ReLA contribution. We will incorporate discussions related to the findings presented in paper [1] into our manuscript.
- **[R2] Our method is independent of a pre-trained teacher model on large-scale data and fundamentally differs from knowledge distillation:**
    - Knowledge distillation [2] typically uses a well-trained, large model as the teacher to transfer knowledge to a smaller model, aiming to achieve comparable performance on the same dataset.
    - In contrast, our ReLA approach offers 'rapid guidance' by leveraging efficient data during the initial stages of model training (see Section 4.2). ReLA allows various prior models, including weak models or even randomly initialized and untrained models, to generate efficient data that significantly enhance training on large datasets like ImageNet-1K.

In response to the positive feedback from reviewers 7uZz and zbwg, who have acknowledged the merits of ReLA and subsequently increased their scores, we respectfully seek your final assessment. If our rebuttal has adequately addressed your concerns and highlighted the significance of our proposed method, we kindly request that you consider revising your score accordingly. An increased score is critically important to our work at this stage.

We remain committed to addressing any remaining concerns and are open to further discussions or clarifications. Your valuable feedback has been instrumental in refining our research, and we appreciate the opportunity to enhance our work based on your input. Thank you for your time and efforts throughout the review process. We look forward to your further feedback.

[1] Qin, Tian, Zhiwei Deng, and David Alvarez-Melis. "A Label is Worth a Thousand Images in Dataset Distillation." 2024.

[2] Hinton, Geoffrey, Oriol Vinyals, and Jeff Dean. "Distilling the knowledge in a neural network." *2015*.
Thank you for the detailed proof. I would like to increase my rating to 7.
**Proof of MGG version of Theorem 1：** 
## **Setup**

**Notation:** $\textup{N}(\mu,\alpha,\beta)$ denotes the generalized Gaussian distribution with pdf $\frac{\beta}{2 \alpha \Gamma(1 / \beta)} e^{-(|x-\mu| / \alpha)^\beta}$, $\textup{B}$ for Bernoulli distribution. 
We focus on the 1-dim situation. Assume that $\mu_1 < \mu_2$. Define the original data distribution ($\mathcal{N}_0 = \textup{N}(\mu_1, \alpha_0,\beta_0)$ and $\mathcal{N}_1 = \textup{N}(\mu_2, \alpha_0,\beta_0)$):

$$ G := \{(x, y) \mid y \sim 2 \cdot \textup{B}(1,\frac{1}{2}) - 1, x \sim \frac{1-y}{2} \cdot \mathcal{N}_0 + \frac{1+y}{2} \cdot \mathcal{N}_1 \} $$

and the modified one ($\mathcal{N}_0' = \textup{N}(\mu_1, \alpha,\beta)$ and $\mathcal{N}_1' = \textup{N}(\mu_2, \alpha,\beta)$):

$$ G' := \{(x, y) \mid y \sim 2 \cdot \textup{B}(1,\frac{1}{2}) - 1, x \sim \frac{1-y}{2} \cdot \mathcal{N}_0' + \frac{1+y}{2} \cdot \mathcal{N}_1'  \} $$

Our task is predicting $y$ given $x$. Note that $y \in \{\pm1\}$, which is a bit different from the definition in Section 3.2 . In the 1-dim situation, we just need one parameter for this classification task, so define $f_{\theta}(x) := \textup{sign}(x + \theta)$ to fit the distribution.

We could compute the generalization loss on the original distribution:

$$ \mathcal{L}(f _{\theta}) = \left( \int _{-\theta}^{+\infty} \, dF _- + \int^{-\theta} _{-\infty} \, dF _{+} \right) / 2 = \left( 1 - \int _{-\frac{\theta + \mu _2}{\alpha_0}}^{-\frac{\theta + \mu_1}{\alpha_0}} \, dF \right) / 2 $$

Obviously $\theta^\star = -\frac{\mu_1 + \mu_2}{2}$, we have:

$$ \mathcal{L}(f_{\theta}) - \mathcal{L}(f_{\theta^\star}) = \left( \int_{-\frac{\mu_2 - \mu_1}{2 \alpha_0}}^{\frac{\mu_2 - \mu_1}{2 \alpha_0}} \, dF - \int_{-\frac{\theta + \mu_2}{\alpha_0}}^{-\frac{\theta + \mu_1}{\alpha_0}} \, dF \right) / 2 \leq C_1 \cdot (\theta - \theta^\star)^2 \quad (\text{or } C_1' \, |\theta - \theta^\star|) $$

where $C_1$, $C_1'$ are constants, $F_0$, $F_1$, and $F$ denote the CDF of $\mathcal{N} _0$, $\mathcal{N} _1$, and $\textup{N}(0,1,\beta_0)$ respectively.

The inequality above is due to the fact that the function $h(x) = \left( \int^{1}_ {-1} \, dF - \int _{x-1}^{x+1} \, dF \right) / x^2$ has limits at 0 and so is bounded.

## **Algorithm**

For a dataset $\{(x_i, y_i)\}_{i=1}^{n}$,

set the loss function $L(\theta) = \frac{1}{n} \sum_{i=1}^{n} \ell \left[ y_i (x_i + \theta) \right]$, $\ell(v) = \tfrac{1}{2} (1-v)^2$.

We apply the stochastic gradient descent algorithm and assume the online setting ($n=1$): at step $t$, draw one sample $(x_t, y_t)$ from $G'$ then use the gradient $\nabla L(\theta_t)$ to update $\theta$ ($\eta \in (0,1), t \in \mathbb{N}$):

$$ \theta_{t+1} = \theta_t - \eta \nabla L(\theta_t) $$

$$ \nabla L(\theta_t) = \theta + (x_t - y_t) $$

It can be observed that the randomness of $x$ leads to noise on the gradient.

## **Bounds with Variance**

We prove the proposition that lower variance of GG can make convergence faster, i.e., $\mathbb{E}\left[ \mathcal{L}(f_{\theta_t}) - \mathcal{L}(f_{\theta^\star}) \right]$ is bounded by an increasing function of the variance ($t$ fixed).

## **Proof**

From above, we get:

$$ \theta_t = (1 - \eta)^{t} \theta_0 - \eta \left[ (x_{t-1} - y_{t-1}) + (1 - \eta) (x_{t-2} - y_{t-2}) + \dots + (1 - \eta)^{t-1} (x_0 - y_0) \right] $$

and so:

$$ \mathbb{E}\left[ \mathcal{L}(f_{\theta_t}) - \mathcal{L}(f_{\theta^\star}) \right] \leq C_1 \mathbb{E}\left[ (\theta_t - \theta^\star)^2 \right] $$

$$ = C_1 \mathbb{E} \left\( \left[ (1 - \eta)^{t} (\theta_0 - \theta^\star) - \eta \sum_{j=1}^{t} (1 - \eta)^{j-1} (x_{t-j} - y_{t-j} + \theta^\star) \right]^2 \right\) $$

$$ = C_1 \mathbb{E} \left[ (1 - \eta)^{2t} (\theta_0 - \theta^\star)^2 + \eta^2 \sum_{j=1}^{t} (1 - \eta)^{2(j-1)} (x_{t-j} - y_{t-j} + \theta^\star)^2 \right] $$

$$ = C_1 \left\( (1 - \eta)^{2t} (\theta_0 - \theta^\star)^2 + \frac{\eta}{(2 - \eta)} (1-(1 - \eta)^{2t})\left[\frac{\alpha^2 \Gamma(3 / \beta)}{\Gamma(1 / \beta)} + \left( 1 - \frac{\mu_2 - \mu_1}{2} \right)^2\right] \right\) $$
The last two equalities are due to the fact that for $(x, y) \sim G'$:

$$ \mathbb{E}\left[ x - y + \theta^\star \right] = 0 $$

$$ \mathbb{E}\left[ (x - y + \theta^\star)^2 \right] = \frac{\alpha^2 \Gamma(3 / \beta)}{\Gamma(1 / \beta)} + \left( 1 - \frac{\mu_2 - \mu_1}{2} \right)^2 $$
Thank you for your response. We apologize for the oversight regarding MGG. Upon re-evaluating the proofs in our paper, we have determined that our argument does not rely on any specific properties unique to MG compared to MGG. Consequently, transitioning from MG to MGG will mainly involve modifying certain constants and substituting $\Sigma^2$ with $\frac{\alpha^2 \Gamma(3 / \beta)}{\Gamma(1 / \beta)}$ (assuming the probability density function is $\frac{\beta}{2 \alpha \Gamma(1 / \beta)} e^{-(|x-\mu_i| / \alpha)^\beta}$).

Below, we present the revised proof of Theorem 1, now based on MGG. For Theorem 2, the only modification is in the setup, which now aligns with the setup used in the proof of Theorem 1 as outlined below. The core reasoning remains consistent with the original proof based on MG, indicating that the proof process detailed in Appendix A and B is generalizable. As for the other theorems in this paper, no updates are required since their proofs do not depend on assumptions regarding the data distribution.

Should you have any further questions or require additional clarification, please feel free to reach out.
Thank you for your thoughtful feedback and for reconsidering the scores based on our responses. We will ensure that the points addressed are more clearly and prominently articulated in the revised manuscript to enhance the clarity and impact of our contribution. Your input is invaluable in helping us improve the quality of our work.
Thank you for your prompt response and for taking the time to review our submission. We greatly appreciate your feedback and respect your decision to maintain your original score. Should you have any further comments or require additional clarification, we remain available to address any concerns.
> [Q2] I also perfectly understand that merely using supervised loss is insufficient. However, the experimental enhancement is not surprising to me at all. Compared with the baseline, the method in this article introduces a pre-trained teacher model on large-scale data, whose representation is definitely useful especially in self-supervised problems. The thing this article actually does is to introduce a knowledge-distillation-like loss term to enhance the performance, which is a popular technique in practice.
> 

[R2] We would like to clarify the following points:

1. Our method does not depend on a teacher model pre-trained on large-scale data. As demonstrated in Table 2, even a weak prior model can generate data that significantly enhance training on large datasets, such as ImageNet-1K. Specifically, 
    1. ReLA illustrates the ability to leverage a model pre-trained on CIFAR-10—a relatively small dataset—to produce data that accelerates BYOL training on a 10% subset of ImageNet-1K, yielding a 7.7% improvement in accuracy over the original BYOL.
    2. Even a randomly initialized and untrained model can provide sufficient acceleration. For example, when employing a randomly initialized model as the prior model in our ReLA framework to generate efficient data, it enhances BYOL training on a 10% subset of ImageNet-1K, resulting in a 6.1% increase in accuracy compared to the original BYOL. We believe this finding presents a novel contribution to the field.
2. Our proposed method ReLA is fundamentally different from knowledge distillation:
    1. Knowledge distillation [3] typically employs a well-trained large model as the teacher, with the aim to transfer its knowledge to a smaller model to retain similar performance on the corresponding dataset. In contrast, our proposed method, ReLA, is a simple, effective, plug-and-play approach that leverages publicly available models on the internet as prior models to generate efficient data, which, combined with the ReLA loss, accelerates downstream learning tasks and achieves much better performance (see Sections 4.1 and 4.2). 
        - ReLA imposes no restrictions on the architecture, scale, or pre-trained datasets of the prior models used for generating efficient data. To validate the feasibility of this approach, we tested the effectiveness of various prior models with different representational capacities, knowledge, and architectures, as shown in Tables 1 and 2.
        - To respond to the reviewers' feedback, we conducted additional evaluations on various downstream tasks (refer to the supplementary PDF).
    2. In ReLA, the dynamic coefficient used in the loss function during downstream task model training is critical. This coefficient dynamically decays over the course of training, particularly for the ReLA loss term related to efficient data. The rationale behind this decay is that the primary function of the efficient data is to provide 'rapid guidance' during the early stages of model training (see Section 4.2). Our ablation study, shown in Figure 4(b), further demonstrates the necessity of this dynamic decay, especially when the prior model used to generate the efficient data is weak. This approach is fundamentally different from the primary objective of knowledge distillation, which focuses on transferring the teacher model’s capabilities to the student model.
    3. In the knowledge distillation process [3], each augmented view of the data typically necessitates corresponding soft labels from the teacher model. This enables the student model to emulate the teacher's behavior across various transformations of the input data. However, this method results in a computational cost that increases linearly with the number of training epochs. Specifically, if the number of training epochs is \(N\), the computational overhead for the teacher model scales to \(N\) times the cost of performing inference on the entire dataset. In contrast, as detailed in Appendix K, ReLA requires only a single inference pass of the prior model over the entire dataset to generate efficient data. Further supporting evidence is provided in the attached PDF.

[1] Qin, Tian, Zhiwei Deng, and David Alvarez-Melis. "A Label is Worth a Thousand Images in Dataset Distillation." 2024.

[2] Yin, Zeyuan, Eric Xing, and Zhiqiang Shen. "Squeeze, recover and relabel: Dataset condensation at imagenet scale from a new perspective." NeurlPS 2023 Spotlight.

[3] Hinton, Geoffrey, Oriol Vinyals, and Jeff Dean. "Distilling the knowledge in a neural network." *2015*.
> [Q1] In R3, the authors have clarified the points in RDED, DQ, and SelMatch. However, there are indeed some works advocating for the use of weak teachers in target generation, such as Fig. 2 in [Qin et al. 2024]. I perfectly understand that it showed on arXiv after the NeurIPS submission deadline. Nevertheless, their conclusions seem more convincing because it is tested on real-world cases instead of merely analyzed through simple GMMs. Especially in the case when there are only a small number of training samples, which is the main focus of this article for efficient training, weak teachers can be more useful. Taking this into consideration, I do think there exists a significant gap between the theoretical analysis and the practical cases.
> 

[R1] We respectfully disagree with the reviewer and would like to clarify that there is no significant gap between the theoretical analysis and the empirical cases in our paper:

1. We drew insights from the theoretical analysis, then designed an effective and efficient technical ReLA framework, and empirically validated our proposed framework ReLA on extensive real-world datasets and downstream tasks.
    1. Our insights align with the conclusions presented in Sections 4 and 6 of the referenced paper [1], which are derived from their experimental analysis. Specifically, we establish the insights articulated in Conjecture 1 and Definition 5 through our Theorems 1, 2, and 4. Furthermore, our insights emphasize the importance of employing labelers to provide ***informative*** targets for efficient learning, which corroborates the conclusion of paper [1], asserting the significance of incorporating “structured information.”
    2. Moreover, our Theorem 4 indicates that using data consisting of both randomly selected samples and high-quality targets can lead to superior performance. This insight aligns with the empirical results presented in Table 1 of paper [1], where it is demonstrated that randomly selected samples, when labeled by a well-trained teacher, surpass the state-of-the-art SRe$^2$L method [2]. However, paper [1] does not provide a theoretical explanation for this observation.
    3. The experimental results presented in Figure 2 of paper [1] do not sufficiently illustrate the inapplicability of our theoretical findings and insights. Although the paper [1] investigates the teacher model's role in generating distilled soft labels, as depicted in Figure 2, it fails to provide essential details regarding the methodology for producing the corresponding distilled samples or images. The omission of critical experimental parameters—such as the optimizer, learning rate, and the number of training epochs for the student model—compromises the ability to accurately compare their experimental setup with the conditions assumed in our theoretical analysis.
    4. Moreover, the study presented in paper [1] lacks additional experiments to validate the phenomena observed in Figure 2, such as evaluations with alternative optimization algorithms or different distillation methods. Their current analysis is limited to a single illustrative case. It is also crucial to note that this paper [1] is a preprint submitted to ArXiv after the NeurIPS deadline, and its experimental details and methodologies have not yet undergone peer review.
2. The significant empirical gains we provided align well with our theoretical findings and insights. In fact, the primary objective of our theoretical analysis is to try to identify the ideal properties of data-efficient training in order to offer insights that support our Conjecture 1, Definition 5, and thereby our proposed technical framework ReLA. We included extensive empirical results (in Section 5) across various datasets, neural architectures, and learning algorithms, to support our findings: either a weak or strong prior model can create efficient data to significantly accelerate the training. 
    1. Small-scale CIFAR models improves ImageNet: ReLA can use a model pre-trained on CIFAR-10 to generate efficient data that accelerate training for BYOL on a 10% subset of ImageNet-1K, resulting in a 7.7% increase in accuracy compared to the original BYOL (see Table 2).
    2. Using CLIP as a prior model, ReLA-aided BYOL enables training a ResNet-50 from scratch on 50% of the ImageNet-1K dataset, achieving performance that exceeds models trained on the full dataset.

To the best of our knowledge, the presented empirical results and the acceleration provided to the community are novel and significant.
Thank you reviewers for your comments. 

I will keep my original score.
I would like to thank the authors for the detailed response. However, my main concerns are still not addressed. I understand that some papers have conducted theoretical analysis under the setting of GMM. There is still unaddressed controversy with some works. And, the technical contribution of the article still appears trivial to me.

1. In R3, the authors have clarified the points in RDED, DQ, and SelMatch. However, there are indeed some works advocating for the use of weak teachers in target generation, such as Fig. 2 in [Qin et al. 2024]. I perfectly understand that it showed on arXiv after the NeurIPS submission deadline. Nevertheless, their conclusions seem more convincing because it is tested on real-world cases instead of merely analyzed through simple GMMs. Especially in the case when there are only a small number of training samples, which is the main focus of this article for efficient training, weak teachers can be more useful. Taking this into consideration, I do think there exists a significant gap between the theoretical analysis and the practical cases.
2. I also perfectly understand that merely using supervised loss is insufficient. However, the experimental enhancement is not surprising to me at all. Compared with the baseline, the method in this article introduces a pre-trained teacher model on large-scale data, whose representation is definitely useful especially in self-supervised problems. The thing this article actually does is to introduce a knowledge-distillation-like loss term to enhance the performance, which is a popular technique in practice.

Taking these factors into consideration, I tend to maintain my original score. 

A Label is Worth a Thousand Images in Dataset Distillation, Qin et al., 2024
Thank you for addressing some of my comments in the rebuttal. After reviewing the appendices again, I still do not see how the theoretical analyses can be generalized to two Mixture of Generalized Gaussian distributions (MGG). This requires further discussion.

However, given that you have extended the work to high-resolution images, various tasks, and complex datasets, I am inclined to increase the contribution rating to 4. If you provide more discussion on the theoretical analysis and address all comments in the final draft, I will consider raising my score to a strong acceptance.
Perhaps the points addressed in your response [R4] should be more strongly articulated in the manuscript?
Thank you for your responses author(s). Based on your comments, and particularly towards [R4], I have increased the contribution score from 2 to 3 and the overall score from 4 to 6.
[a] Sun, Peng, et al. "On the diversity and realism of distilled dataset: An efficient dataset distillation paradigm." CVPR 2024.

[b] Zhou, Daquan, et al. "Dataset quantization." ICCV 2023.

[c] Lee, Yongmin, and Hye Won Chung. "SelMatch: Effectively scaling up dataset distillation via selection-based initialization and partial updates by trajectory matching." ICML 2024.

[d] Lee, Dong Bok, et al. "Self-supervised dataset distillation for transfer learning." ICLR 2024.

[e] Noroozi, Mehdi, et al. "Boosting self-supervised learning via knowledge transfer." CVPR 2018.

[f] Xu, Guodong, et al. "Knowledge distillation meets self-supervision.” ECCV 2020.

[g] Fang, Zhiyuan, et al. "Seed: Self-supervised distillation for visual representation.” ICLR 2021.

[h] Wang, Kai, et al. "Dim: Distilling dataset into generative model." Preprint 2023.

[i] Chen, Xuxi, et al. "Data distillation can be like vodka: Distilling more times for better quality." ICLR 2024

[j] Yin, Zeyuan, Eric Xing, and Zhiqiang Shen. "Squeeze, recover and relabel: Dataset condensation at imagenet scale from a new perspective." NeurlPS 2023 Spotlight.

[k] Shao, Shitong, et al. "Generalized large-scale data condensation via various backbone and statistical matching." CVPR 2024 Highlight.

[l] Shao, Shitong, et al. "Elucidating the Design Space of Dataset Condensation.” Preprint 2024.

[m] Zhao, Bo, and Hakan Bilen. "Dataset condensation with distribution matching." WACV 2023.

[n] Sorscher, Ben, et al. "Beyond neural scaling laws: beating power law scaling via data pruning." NeurlPS 2022 Outstanding Paper.

[o] Loureiro, Bruno, et al. "Learning gaussian mixtures with generalized linear models: Precise asymptotics in high-dimensions.” NeurlPS 2021.