Rectifying Soft-Label Entangled Bias in Long-Tailed Dataset Distillation
Authors: Chenyang Jiang
Hang Zhao
Xinyu Zhang
Zhengcen Li
Qiben Shan
Shaocong Wu
Jingyong Su
Venue: NeurIPS 2025 poster ()
Keywords: dataset distillation; long-tailed recognition
Primary area: deep_learning
TL;DR: A soft label technique tailored to long-tailed dataset distillation scenarios.

Abstract: Dataset distillation compresses large-scale datasets into compact, highly informative synthetic data, significantly reducing storage and training costs. However, existing research primarily focuses on balanced datasets and struggles to perform under real-world long-tailed distributions. In this work, we emphasize the critical role of soft labels in long-tailed dataset distillation and uncover the underlying mechanisms contributing to performance degradation.
Specifically, we derive an imbalance-aware generalization bound for model trained on distilled dataset. We then identify two primary sources of soft-label bias, which originate from the distillation model and the distilled images, through systematic perturbation of the data imbalance levels.
To address this, we propose ADSA, an Adaptive Soft-label Alignment module that calibrates the entangled biases. This lightweight module integrates seamlessly into existing distillation pipelines and consistently improves performance. On ImageNet-1k-LT with EDC and IPC=50, ADSA improves tail-class accuracy by up to 11.8\% and raises overall accuracy to 41.4\%. 
Extensive experiments demonstrate that ADSA provides a robust and generalizable solution under limited label budgets and across a range of distillation techniques.

Rectifying Soft-Label Entangled Bias in Long-Taile Dataset Distillation
Chenyang Jiang"?
Hang Zhao'
Xinyu Zhang'
Zhengcen Li
Qiben Shan?
Shaocong Wu?*
Jingyong Su
'Harbin Institute of Technology, Shenzhen PPegghheng Laboratory
{238936035, 200110431 10431, zhangxinyu, lizhengcen//sstu.hit edu, suj sujingyongühit.. edu. cn, {shanqb, sushc)Opcl. an
Abstract
Dataset distillation compresses large-scale datasets into compact, highly informa- tive synthetic data, significantly reducing storage and training costs. However, existing research primarily focuses on balanced datasets and struggles perform under real-world long-tailed distributions. In this work, we emphasize the critical role of soft labels long-tailed dataset distillation and uncover the underlying mechanisms contributing to performance degradation. Specifically, we derive an imbalance-aware generalization bound for model trained on distilled dataset. We then identify two primary sources of soft-label bias, which originate from the distillation model and the distilled images, through systematic perturbation of the data imbalance levels. To address this, we propose ADSA, an Adaptive Soft-label Alignment module that calibrates the entangled biases. This lightweight module integrates seamlessly into existing distillation pipelines and consistently improves performance. On ImageNet- Lk-LT with EDC and IPC=50, ADSA improves tail- class accuracy by up 11.8% and raises overall accuracy to 41.4%. Extensive experiments demonstrate that ADSA provides a robust and generalizable solution under limited label budgets and across range of distillation techniques.
Introduction
Pretrained models built on large-scale datasets exhibit superior predictive capabilities. However, training such large-scale models demands massive data volumes, resulting in significant costs in data storage, transmission, and computational resources. Moreover, the effectiveness of model training is highly dependent on the availability of high-quality datasets 2 With the increasing importance of data, the paradigm of data-centric artificial intelligence (DCAI) has gained increasing attention, encompassing data acquisition, optimization, and maintenance (3) and has become central contemporary model development in both research and industry. A key research challenge is to efficiently extract critical information from datasets, eliminate redundancy, and construct high- quality data, which can accelerate model development, reduce carbon emissions. and promote the democratization of large-scale model training.
Dataset distillation 4 aims to synthesize smaller and high-qualit synthetic dataset that serves as a surrogate for the original dataset. The core motivation is to treat the dataset as learnable parameters by representing images trainable tensors and optimizing them match the performance of models trained on the full dataset. Recent advances in dataset distillation have focused on improved optimization objectives ( parameterization strategies (6) training dynamics (7). and more effective
utilization of soft labels g
which has significantly enhanced its practicality and applicability.
*Corresponding authors.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Despite recent progress in dataset distillation, limited attention has been paid to the challenge of distilling long-tailed datasets, a pervasive issue in real-world scenarios. Real-world data typically follow a long-tailed distribution [9 10], where few head classes account for the majority of samples, while numerous tail classes are sparsely represented. Models trained on such distributions often suffer significant performance degradation on tail classes. Applying dataset distillation directly to these datasets tends to produce biased synthetic data, leading to performance degradation and optimization instability. This hinders the real-world applicability of dataset distillation. Such scenarios often involve a distributional mismatch between the original and synthetic datasets. LTDD [ attempls to address this issue by aligning the parameter distributions derived from models trained on both datasets during the distillation process. While LTDD provides a promising solution by aligning parameter distributions, it primarily operates the parameter-leve' matching and does not explicitly address the bias encoded in soft labels, an increasingly critical component in modern distillation pipelines.
Our work focuses on how soft labels, a widely used technique in recent dataset distillation methods [12 14 15 16] that significantly boosts performance, are affected by dataset imbalance. We first derive imbalance-aware generalization bound for dataset distillation, which theoretically reveals that imbalanced soft labels generated during the standard distillation process limit the performance of models trained on the distilled data. We then conduct an experiment that perturbs the distribution of the original dataset and observe its effect on the distilled images and labels. Through this analysis, we identify two primary sources of soft-label bias: the distillation model and the distilled images. These biases jointly contribute to significant performance degradation on tail classes.
13
To address the entangled bias inherent in soft labels, we introduce ADSA, an ADaptive Soft-label Aligmment module that calibrates the distillation model's predictions on distilled images by jointly removing both sources of bias. We observe that distilled images can serve as form of hold-out data to estimate and eliminate this bias. ADSA preserves the class-level relational information encoded in soft labels while eliminating the bias induced by the imbalanced distillation model and the image-specific distribution shift. It functions as a post-hoc module that does not participate in model training or image distillation, and can therefore be seamlessly integrated into various dataset distillation baselines, without requiring the design of complex training objectives, sampling strategies, distillation architectures to handle long- long-tailed distributions. Moreover, it can benefit from advanced methods for distilling imbalanced datasets at the ImageNet scale. This lightweight module is simple yet effective, and consistently improves accuracy across multiple state-of-the-art dataset distillation methods and diverse datasets, providing a robust and generalizable solution for imbalanced data scenarios across various distillation settings.
Our main contributions are summarized as follows:
We thcoretically and empirically reveal the importance of soft labels in dataset distillation by deriving an imbalance-aware generalization bound and designing experiments investigate the influence of imbalanced data. The theoretical analysis and corresponding experiment design provide an effective tool for future research on dataset distillation under distribution shift.
We reveal two distinct sources of soft-label bias originating from the imbalanced distillation model and the image-induns aagge-ndduedd distribution shift, both of which jointly degrade tail-class performance.
We propose ADSA, adaptive soft-label alignment module that calibrates soft-label distributions in a post-hoc manner. ADSA is lightweight, glug-and-play and can be seamlessly integrated into various dataset distillation methods without the need for complex training design.
2 Related Work
Dataset distillation reduces dataset size while preserving essential information, making it valuable for neural network training. It has been applied in real-world scenarios, such as continual learning. However, most real-world datasets exhibit long-tail distribution, and research on dataset distillation in such contexts remains limited. This section first reviews key studies on dataset distillation, followed by an overview of long-tail recognition and relevant research.
2.1 Dataset Distillation
Dataset distillation 4 aims to condense the knowledge of an original dataset into significantly smaller synthesized dataset. This process involves initializing the synthetic dataset trainable

---

parameters and optimizing it using gradients derived from a specified objective function. The objective function is a carefully designed matching criterion between the core information of the original and synthesized datasets, where an information function extracts essential features, and distance function computes the loss. This information extraction process typically involves model training or inference, where the model is referred to as distillation model. Finally, we evaluate performance by training an evaluation model on the distilled dataset and testing it on the test set. Methods are categorized into performance matching, gradient matching, parameter matching, and distribution matching based on the matching policy.
Performance matching 4 17 18 optimizes the performance of model trained on the distilled dataset directly. Gradient matching 19 20 22] and parameter matching [23 [2 7 both train distillation
model on the distilled dataset to mimic the behavior of model trained on the original dataset, with the
former focusing on gradient similarity and the latter on training parameter trajectory similarity.
The matching methods described above are challenging to deploy in large-scale dataset scenarios due the high computational and memory costs associated with distillation model training and loading in the inner loop of the distillation process. Consequently, distribution-base methods were initially proposed by Bo Zhao 221. which optimize the synthesized dataset by matching the feature distributions of neural networks. DataDAM (26] and DREAM (27) improved the matching
and sampling strategies to enhance performance. More recently, SRe2L [4 further boosted the performance of distribution-basee
methods by utilizing DeepInversion like (28) image distilling
approach and soft labels, establishing a three-stage framework of squeeze, recover, and relabel. Subsequently, D3S (29], EDC [16], and GVBSM [30] enhanced performance both theoretically and empirically. In addition to optimization- based methods, generative- based approaches have also been explored. These methods conduct the matching process in the latent space [31 [5] or modify the generative models' objective function to encourage the generation of more informative images (32
Soft labels for distilled images have been widely adopted by many methods [12 13 14 15 16 to enhance performance, and have recently received increasing attention. Qin et al. (33 highlight the crucial role of soft labels in dataset distillation and systematically evaluate their impact on model performance. The DD-Ranking benchmark (34] further demonstrates that dataset distillation techniques can lead to substantial performance improvements. GIFT (8) introduces a tailored loss function for training on distilled datasets to better exploit soft labels. Xiao et al. 35 find that high within-class diversity demands large-scale soft labels and propose the LPLD method to effectively prune them. The optimization of soft labels has also been explored in prior work 36 37,, while DRUPI (88 and LADD (39] design more effective soft-label formats to improve training efficiency.
Recent advancements have brought data distillation closer real-world applications, where data typically follows a long-tailed distribution. Directly applying distillation techniques to long-tailed datasets may result in a biased distilled dataset. LTDD pioneers long-tailed dataset distillation by identifying the limitations of expert trajectory matching on imbalanced data and proposing Weight Mismatch Avoidance and Adaptive Decoupled Matching to improve tail-class supervision and soft- label quality. However, its focus remains on parameter-level bias propagation, while our work explicitly targets soft-label bias and introduces a post-hoe calibration strategy to correct it.
2.2 Long-Tailed Recognition
Long-tailed recognition aims train perl-performing deep models, particularly on tail classes, from datasets following a long-tailed class distribution [ Traditional designs of sampling methods, network architecture, and loss functions cause the trained network to assign much higher confidence
to the head classes than to the tail classes, leading to poor predictions on the tail classes [40). Balanced
data acquisition 4T multi-branch networks 42 feature transferring 4K 44
45]
adaptive loss
function design 46
47
decoupled training [48] and logit post-hoc calibration 49 have been
well-explored. Decoupled training methods first train the model's 's backbone under standard settings and then apply balanced sampling to retrain the classifier, achieving significant performance gains. This reveals that the backbone trained on long-tailed dataset retains sufficient capacity to extract tail-class information. Our approach takes this further by utilizing fully trained distillation model without modification to recover tail- information within the distilled dataset, thereby enhancing the evaluation model's training.
3

---

3 Method
3.1
Imbalance-aware Upper Bound
Dataset distillation aims to distill the original dataset Drr into smaller synthetic dataset Daa- where the size of the latter is significantly smaller than the former, i... [Daal < D We define the loss function for model fo (hereafter referred to 0 for brevity) and dataset D as 1(0, D) [D] E(EyiED [(ff(())))) y), where denotes the cross-entropy function. The goal of dataset distillation is to minimize the performance gap on the test dataset Die between models 01r and Oad, trained on Der and Daa respectively:
Dad argmin [((eua, Die) 1(Ber, Die)l Daa
Oad argmin Daa).
0
For theoretical analysis, it is common to model the finite datasets Drr and Dad samples drawn from underlying distributions, Ptr Ptr(x,, y) and Paa(z, y), respectively. The empirical loss 1(0, D) is thus Monte Carlo approximation of the expected loss, (e)) Following this convention, we adopt the notation for expected losses our theoretical discussion.
The theoretical framework of D3S 29 views dataset distillation as a domain shift problem. For a classifier p(y]z) trained on data from Pad. its corresponding expected losses are defined as lad log ((y[z)]. Assuming the classifier's negative log- likeliihood bounded by a positive constant C, D3S provides the following generalization bound:
Epazl- - log [x]] and fer = Ep., [-
C
4r S lad +
2/2 V Raa,
(3)
where Rad = (Ptr
+ DEL ( (plaa))a()(())))
This result suggests that the upper bound on the original training loss 4r can be tightened by reducing the training loss laa on the distilled dataset and minimizing the distribution discrepancy Raa between the original training and distilled datasets. Under Per y) = Pto (Ir, 3) assumption
and with sufficiently large dataset, we have 4t & fer according to VC theory (50). Consequently, the test loss La can be considered approximately upper-bounded by Lad + 2v2 v Raa However, this
assumption does not hold in the long-tailed setting, where only the class-conditional distributions (xll)) Pte (a (y) are preserved in long-tailed recognition studies [51 46).
Following the setting in long-tailed recognition study [51 46), the training dataset Dad used for distillation follows long- taigtt distribution, while the test dataset is balanced. Let n& denote the number of samples in the k-th class in Dtr, where EK-1 n = |Drll and no >> n1 >... --
Theorem 3.1. In the long-tailed setting, where training and test distributions share the same class- conditional distributions (i.e., (xly) = (te(xlg) y)) but differ in their class priors (i.e., Per(y) + (t)) the discrepancy term Raa in the D3S bound (4) can be expressed in the following two equivalent forms:
Rad Dw.. + DKr. DKI. + const,
(5)
and
Rda =
DKL (pte (xL(Pre(V)lPPud()) + Σ Pte DK) (Pll))
(6)
The proof is provided in Appendix A.2 Under the relaxed assumption, the two resulting bounds introduce additional terms compared Eq. 4 leading several key insights regarding the desired properties of the distilled dataset: (i) The first term in Eq. 5 suggests that the learned posterior distribution (2(())))))))))))))) from a model trained on Dad should align with the label distribution in the test set. (ii) The second term in Eq. highhights the importance of aligning the feature distribution of the synthetic dataset with that of the original training data to ensure optimal performance. Additionally, the second term in Eq. 6 refines this insight by emphasizing a more fine-grained, per-class alignment due to the long-tailed assumption ptr (aly) = (xll)) (iii) The first term Eq. 6 further suggests natural experimental setup by adopting a balanced class distribution in the distilled dataset, where

---

(a)
Belasced
Lorg-ellll Dalanct
IWDD
mw2
(c)
Geneare by
68
Originad
Dotssrt
88
Ga utreeted by of veylng lovds
Toui
Moall Clee
Trais
Truas
Distitation Nodek(DN)
Relihee
Bistilaation Model $
Distilation Model @
(Dats for pencrating Imagges
(totty for auncratiny, baach)
Gcacuae
Perterbution Analysa
Digitaiiz
Ratanoed Safte
Systhetic
Relabd
Iepur
Syntheri Image
Syst-1 Label
Dostiee Untnent
Balanced Distilled
Distllel
Class Deerttattion Pricer
tnlalbaeeattLonn-ttilllll Dattaet
Synthetie Images
Perturiatisn
Figure I: Overview of the experimental framework and modules. (a) The conventional dataset distillation pipeline utilizes single model for both image and label generation. (b) The perturbation analysis framework employs separate models: indicate different combination of balanced/imbalanced dataset. then perturb the imbalance levels to observe the resulting performance. (c) The proposed adaptive soft-label alignment module(ADSA) Symbols y, Ay, and denote the average confidence, confidence adjustment by different levels, and the calibrated average confidence, respectively. We optimize Ay get the most uniform across classes, and use to calibrate the soft labels. The operator indicates addition logit space; soft labels are shown here for visualization clarity.
for image synthesis and another for soft label generation. Three configurations
each class contains the same number of samples as in the test set. While most existing works aim to improve distillation performance by focusing on insight (ii). such as EDC which enforces both global and class-aware feature matching corresponding to the second terms in Eq. and Eq. 6 the widely used and empirically effective soft-labeling technique remains relatively underexplored. Notably, soft labels are highly generalizable and can be seamlessly integrated into various distillation pipelines. Therefore, this work primarily focuses on insight (i). In Section 3.2 we demonstrate that conventional soft-labeling strategies introduce bias in long-tailed dataset distillation, resulting in mismatch in the first term of Eq. and subsequent performance degradation. We further identify two main sources of this bias that are inherent in existing soft-labeling methods.
3.2 Perturbation Analysis and Dual Bias in Soft Labels
In this section, we first empirically validate that mismatches in soft labels (first term in Eq. 5 and distilled images (second term in Eq. 5) under long-tailed distributions lead to performance degradation. Motivated by Theorem 3.1 which underscores the importance of soft labels under imbalanced distributions, we further investigate the influence pathway from the original long-tailed data distribution to the resulting soft labels by identifying two key distortion factors. To investigate how imbalance affects predictions of the final evaluation model, we propose a perturbation analysis of distilled images and their corresponding soft labels, as illustrated in Figure [ Unlike conventional dataset distillation pipelines where synthetic images and soft labels are jointly derived from the same distillation model, we decompose the process into two separate pipelines. This design enables controlled perturbation of either modality (distilled images or soft labels) by leveraging original datasets with varying degrees of class imbalance. By doing so, we can analyze the individual contribution of each modality to the downstream model's performance under long-tailed settings.
We employ an inversion-like objective
28
argmin
Σ
L(fer (ī y) + Rreg(Tad))
(7)
Daa
(Baa.u)EDea
to distill images, where (Fdd. y) E Dad. denotes a synthetic image-labe pair, with the predefined class label for each Zdd. and Rreg(Taa) as the distribution matching regularization term such as feature distribution constraints [[g(āas) g( ||2 where y denotes the network backbone of Soft labels are generated using pretrained model. This follows the mainstream pipeline recent
large-scale dataset distillation frameworks []4 16 29 30).
5

---

The original dataset, illustrated in the corresponding region of Figure [ can be either balanced or imbalanced. The model trained on it is accordingly referred to as a balanced or imbalanced distillation model. We analyze four configuration (referred to as configs for brevity) that differ in the composition of distillation models used to generate distilled images and assign soft labels: config (1) images distilled from an imbalanced model and labeled by a balanced model; config (2) images distilled from a balanced model and labeled by an imbalanced model; config (3) images both distilled and labeled by imbalanced model; and config (4) images both distilled and labeled by balanced model. Using the CIFAR-100 100 dataset [52], we designate 20 classes as tail classes and the remaining 80 as head classes. The number of images in tail classes is varied to simulate different levels of imbalance. The experimental details are provided in Appendix AA.3 Figure 2 preports the confidence value (the predicted probabilities for the ground truth class) for head and tail classes, the classification accuracy on tail classes, and the entropy of the softmax outputs. The x-axis represents the average number of samples across all tail classes or head classes. These metrics reflect the informativeness and reliability of the soft labels, both of which influence the evaluation model's performance, as discussed in 33
As shown in Figure 2 (a), config (1) (using only imbalanced images), config (2) (using only an imbalanced distillation model for labeling), and config (3) (both imbalanced) all experienced performance drop compared to config (4) (both balanced). This validates the theory presented in Section 3. which states that both biased distilled images and soft labels lead perfor- mance degradation. Furthermore, we observed that config (2), which used an imbalanced model for labeling, showed a greater performance de- crease than config (1). Figure (b) illustrates the entropy of the soft labels, showing that more imbalanced dataset leads to higher entropy, which indicates lack of muus-discriminative in-
Acouracy Tall Classes
[b] Entropy of Label Tall Classes
0.6
Accuracy
Entropy
4.0
0.2
0.0
2.0
50
100
100
Number
Number
(<) Confidence Tall Classes
(0 Confidence Head
2
2.0
2.0
Corfs
Cerls
Confidence
1.5
Confidence
1.5
1.0
1.0
0.5
0.5
formation in the soft labels 33
0.0
50
100
50
100
Number
Number
To further analyze the respective contributions of imbalanced distilled images and imbalanced distillation models to the final soft labels, we visualize the per-class confidence. We use the term labeling model to refer to the distillation model used to generate soft labels for the dis- tilled images. Panels (c) and (d) show that both imbalanced synthetic images and imbalanced labeling models lead to soft labels that are overconfident for head classes and underconfident for tail classes. Notably, even in config (I) where a balanced distillation model is used, its predictions are biased due to the imbalanced distilled images, and we refer to this as bias from distilled images. In contrast, the bias observed in config (2) stems from the prediction bias within the labeling model itself, which we term bias from the labeling model. As a result, the evaluation model trained on such biased soft labels from these two sources learns incorrect class probabilities, thereby inheriting this bias.
Figure 2: Effect of the number of images across head and tail classes on confidence, accuracy, and entropy. (a): Accuracy trend on tail classes. (b): Entropy of soft labels for tail classes. (C) and (d): Confidence scores for tail and head classes with increasing tail samples.
We can approximate the total bias in the soft labels using the following decomposition:
PDD x) = uarget (v x)) + er(y | (c) + e x),
(8) (yla:) represents the desired
where pohe (slær) denotes the observed soft label posterior, and target ( (unbiased) target posterior. The terms <r(y(=) and (y](zz) correspond to the hias introduced by the labeling model and the synthetic images, respectively.
The above observation reveals the entangled sources of bias in soft labels and underscores the necessity of their calibration. In practice, however, the additive decomposition assumption may not hold, as evidenced in panel (a) of Figure 2 where the presence of both biases does not result in the worst performance. To address the entangled bias, we propose an effective lightweight calibration module that adaptively debiases the loft- by utilizing the distilled images in next section.
2 All networks f involved in distillation will be trained separately for each contiguration.

---

3.3 Adaptive Soft Label Alignment Module
The calibrated soft labels must satisfy three key properties. First, they should eliminate entangled bi- ases to approximate the true posterior distribution on the test set, thereby tightening the generalization upper bound. Second, they should preserve the semantic relationships among classes maintain the informative structure of the original soft labels, which is a key factor in enhancing the informational richness of the distilled dataset [33 Third, the module should be adaptive across different datasets, maage-per-class (IPC) regimes, and imbalance factor (IF) conditions to ensure robust generalization across varying scenarios.
The proposed module is illustrated in Figure I c). The main idea to address entangled biases is direct and unified: we adaptively calibrate the model's predictive outputs for each distilled image obtain refined soft labels. To better illustrate the concept of ADSA, we present soft labels in Figure Tc), although the calibration is actually applied the logits.
First, to preserve semantic relationships among classes while smoothly adjusting each logit com- ponent, adopt the logit calibration method proposed in [4)]. The adjustment is defined as jrgmax
- 7 log Ty. where Ty denotes the empirical frequency of class in the training set, T is a calibration hyperparameter, [K] is the label set. f denotes the neural network, and fy (x) the y-th logit output. While such methods are typically applied during inference select the class with the highest logit and thus make a confident prediction, our focus lies in the relationship preserving property of soft labels. We obtain preliminary prediction probability for given image:
exp(fu(#) T log my))
plyle;)) E exp(Sy (x) - T log TTy)
We refer to p(y)a) (r) as the soft label without loss of generality.
We observe that the distilled images exhibit a distribution shift relative the original training dataset. Consequently, the distilled set can serve as a hold-out validation set to diagnose class-wise output imbalance from the model trained on the original data. To quantify this, we compute the class-wise average soft label across all distilled images p(ū i|,;; r) [p(y = r)}, where DD denotes the set of distilled images labeled as class i. Figure [c), the symbols y and ý correspond to p( [[] zz; 0) and p(ū]x;; iz; T) respectively. To achieve balanced confidence across all classes, we optimize the calibration strength such that the dlss confidence variance minimized. The optimal is defined as
K-1
1 K
T" argmin
1
Σ
plū = i]
Σ j=0
p(ū = x;-)
(10)
K
K
==
The soft labels are then calibrated as plulæ; r*) and integrated into the final distilled dataset.
4 Experiment
In this section, we conduct a comprehensive evaluation of our dataset distillation method under long-tailed distribution settings. We begin by detailing the experimental setup, including datasets, evaluation metrics, and baseline methods for comparison. Specifically, we reproduce dataset distilla- tion methods long-tailed datasets and benchmark our approach against ffate-ofthhear baselines, demonstrating its superior performance under imbalanced conditions. We then perform ablation studies to assess the adaptiveness of our method, particularly in extreme long-tail scenarios, and evaluate its robustness under varying soft-label budgets. Finally, we provide explanatory analyses and visualizations illustrate the effectiveness of the ADSA module.
Experimental Setup. We evaluate our method on (224x222 224) (54). The CIFAR-LT datasets constructed using exponential long-tail mail distributions as in 42 with imbalance factor (IF) r = no controlling class imbalance. Class sizes n, follow
52 and ImageNet-Ik-
ng-1
n4 1og ,े., and we test with r E 10, 50, 100. The ImageNet-L.T follows the setup in 43 We evaluate performance across varying IPC (images per class) settings and compare our method with state-of-the--e art basclines, including LTDD [II] SRe2L [14). GVBSM [30], and EDC [16]
7

---

Table 1: Comparison with baseline methods. Long-tailed datasets constructed using an exponential decay in class frequency. Dataset distillation is then applied to generate compact synthetic datasets, on which evaluation models are evaluated. The table reports Top-1 validation accuracy on the distilled datasets. We use IF (imbalance factor) and IPC (images per class) to denote imbalance severity and distilled dataset size, respectively. LTDD [I]] uses depth-3 ConvNet as the backbone, SRe2I. uses Res-Net-18, and GVBSM/EDC adopt multiple architectures distill and ResNet- to evaluate, as detailed in Appendix A.3
CIFAR- 10-LT
IPC=1
IPC=10
IPC=50
IF
10
50
100
10
50
100
10
50
100
LTDD
28.0#1.0
24.4+0.3
23.8+0.5
58.1
54.2+1.0
53.4+0.1
70.5+0.4
65.8e0.2
64.0=0.9
SRe2L
16.9+2.2
17.8+2.3
14.3+1.7
24.1
23.3+1.9
22.6#0.4
40.4=0.6
36.6=1.9
34.6#1.4
+ours
20.2+0.9
18.1+1.2
19.3+1.3
27.1+1.6
25.0+0.2
25.940.6
47.1
38.8+0.6
45.3+0.7
GVBSM
34.240.4
28.9+0.4
25.1
49.5+0.1
33.840.3
29.4+0.1
58.2+0.2
37.2+0.0
30.9+0.0
+ours
39.1+0.4
36.0:0.6
31.5+0.5
54.4+0.5
45.8+0.1
40.4=0.4
64.7=0.0
51.4=0.2
46,9:0.2
EDC
32.3+1.4
29.5+0.8
30.0#0.6
69.920.7
58 58..5
50.9+0.5
77.8=0.1
65.620.3
56.020.2
+OUES
35.2+0.8
3..4
39.3+0.6
73.2+0.4
69.8+0.2
68.7.0.3
80.8+0.2
76,420.4
74.8
CLEAR-100-LL
IPC=1
IPC=10
IPC=50
IF
100
10
100
10
100
LTDD
12.3+0.2
11.100.1
10.6#0.1
31.5#0.2
26.8+0.3
24.9=0.1
40.0=0.1
34.5=0.1
31.620.0
SRe2L
7.4+0.3
8,5+0.6
7.5+0.1
27.0=1.4
25.0+2.5
22.6+1.1
41.4=1.8
32.8+2.0
28.9+0.4
+ours
6.4+0.0
7.120.4
7.0=0.8
25.9+0.5
26.340.2
23.240.1
47.2+0.3
42.5a0.9
37.622.8
GVBSM
20.3+0.3
19.7+0.1
19.1
35.1
31.2+0.2
27.9+0.1
39.2=0.3
34.0=0.1
31.120.1
HOUrS
20.0#0.3
20.4+0.3
17.0#0.6
35,5+0.1
32,8+0.1
29.6+0.3
40.1=0.0
35,820,3
33.2=0.4
EDC
42.3+0.0
34.3+0.3
32.0+0.1
54.340.6
43.4+116
39.0&1.8
S7.0=0.8
45.7+1.6
40.9&1.8
+ours
43,420.4
37.9:0.5
34.120.3
55.5+0.1
46.0:0.1
41.620.1
58.0:0.2
48.120.1
43.5+0.2
Table 2: Evaluated ImageNet-LT. Classes are categorized into head, medium, and tail using thresholds of 100 and 20 samples per class, respectively.
Table 3: Integration with MTT and DREAM on CIFAR-10 10- LT. indicates results reported from LTDD M.
Table 4: Results under varying soft-label budgets. EP-k de- notes that soft labels are gener- ated only for the first & cpochs then repeatedly used.
Method
ImageNet- TTTnpp-
Method
IPC=10
TPC=50
Head
Mid
Tail
Overall
Randow
33.2
51.6
EP
SRe21.
+Ours
SRe2L+IPC10
+ours
35.1
38.0
20.6
24.3
14,2
7.6
25.6
27.2
IDM 53
[9]
51.9
56.1
1
17.3
17.7(+0.4)
SR<2L=IPC50
51.5
39.4
28.4
32.7
16.1
9.6
34.9
37.0
LTDD*
54.2
65.8
50.3
10
100
34.8
32.8
36.8(+2.0)
35.1(+2.3)
SRe2L+IPC100
53.8
29.7
9.5
36.4
MTT 24
33.4
53.0
300
34.9
37.0(+2.1)
EDC+IPC10
+ours
51.1
34.4
28.7
16.1
10.7
35.0
38.7
soft label
ours
40.4
37.9
56.6
51.4
EP
EDC
+Ours
+ours
47.0
33.8
21.3
37.3
DREAM 27]
56.0
1
28.0
31.0(+3.0)
EDC+IPC50
32.3
12.4
38.6
soft label
41.0
46.7
10
37.3
40.0(+2.7)
+ours
51.3
38.1
24.2
41.4
59.9
65.7
100
38.8
41.6(+2.8)
EDC+PCC100
55.8
32.9
12.7
39.6
EDC 16
77.9
64.4
300
38.7
41.9(+3.2)
38.0
22.8
42.4
82.3
77.0
ResNet- 18 is used as the default evaluation backbone for SRe2L, GVBSM, and EDC, while ConvNet adopted for LTDD, For a fair comparison, we follow the original training hyperparameters for all methods, and report the top-1 accuracy.
4.1 Main Results
As shown in Table our method consistently improves the performance of existing dataset distillation approaches across various IPC and imbalance factor (IF) settings. As IPC and IF increase, our method achieves further performance gains. Notably, integrating our method with SRe2I leads to improvements up to -10% in overall accuracy on both CIFAR- 10-LT and CIFAR-100-LT, Results on ImageNet-Ik-a (Table further validate its effectiveness at scale. For example, with SRe2L at IPC=10, our method improves tail-class accuracy from 7.6% to 14.2% and overall accuracy from 25.6% to 27.2% Similar trends are observed across all IPC settings and baselines. These gains highlight the robustness of our method in improving both average and tail-class performance for large-scale long tailed dataset distillation. We further include in Appendix A.5]a comparison with enhanced baselines, where the distillation models are trained using resampling techniques.
Figure 3 (left) shows the class-wise accuracy. Without calibration, the model over-predicts head classes, causing frequent tail-class misclassification and reduced accuracy. contrast, our method boosts both tail-class and overall performance. We also compare method with classic long-tailes tailed
8

---

F=10, IPC-10
1.60
IFuTO PC
MMEC, acomt 27
3RCA, accm0 23
0.80
Fears acon0.2
9.50
Sho3L RECnE.7
40
196)
0.80
0.50
0.40
0.40
0.50
10
0:19
20-14
80-54 60-79
0:16
60-59
60-26
80.59
103
200
303
10'
10a
102
Das
IPC
Inbalance Factor
[F=100
PC-10
IPC+50
S162, axomo
sPicE sccm
SRell-ourt see 6.38
Confiteee Difleerpe
0.60
3.00
3.33
0.6
3.43
0.4
0.40
9:10
125
0.66
02
0.20
6.55
0.20
0:39
20:39
49-59
60.79
10:99
3.00
0:19
20:39
4.-59
60-79
2.5
50
100
25
50
100
Dlss
Cas5
Class Index
Class Index
Figure 3: Left: Class-wiss accuracy under different imbalance factors (IF) and images per class (IPC). Our
method consistently improves overall and tail-class performance. Right: (a) Accuracy under varying IPC; (b) Accuracy under varying IF; (c) Class-wise confidence difference between original and distilled images; (d) Per-class soft-label confidence distributions. In panel (c) and (d)"imb"/"bba denote soft labels from models
trained on imbalanced/halanced original data; "cal" indicates calibrated soft labels 'imb' from the imbalanced model. Dashed lines represent outputs; solid lines EMA-smoothed results.
recognition methods in Table T6 17 Appendix [A.6), which shows that our approach provides a novel data-centric solution by directly transforming imbalanced data into a balanced and compact form, instead of modifying model architectures or loss functions.
4.2 Ablation Study
We first evaluate the effectiveness of our method across alternative dataset distillation frameworks and different soft-label budget settings. We then demonstrate the method's effectiveness in mitigat- ing two types of bias and further assess the robust- ness of our adaptive strategy under extreme IPC more accurately reflect the performance of ADSA under varying imbalance factors (IF), we adopt different pretraining epochs for models corresponding to each IF, and report the optimal performance achieved by ADSA as used in the bias mitigation and extreme-IF evaluation experiments. More detailed results and analyses provided in Appendix A.4
Table 5: Handling bias from soft labels. Y/N indi- presence/absence of bias.
biased img
biased label
method
IPC=1
IPC=10
IPC=50
10
Y
SRe21.
16.6
34.3
50.6
and conditions.
100
Y
Y
SRe21.
tours
19.0
18.3
41.4
24.4
62.6
31.5
tours
21.8
36.9
$7.6
Table 6: Handling bias from distilled images.
blased img
biased label
method
IPC=10
TPC=50
SRe2L
55.3
Y
Y
+ours
23.7
40.2
63.1
SRe2L
To demonstrate the versatility of ADSA, we ap- ply it to two representative dataset distillation ap- proaches: MTT 23 and DREAM (27), which are based on trajectory matching and gradient matching, respectively. Unlike in SRe21., where ADSA is applied for multiple epochs, here we integrate the ADSA module to generate calibrated soft labels using only a single distillation epoch. Table B summarizes the results under IPC 10 and 50. Our method consistently improves performance across both MTT and DREAM, outperforming the original methods as well as their soft-label-eniee label-enhanced variants. The improvements are especially notable on DREAM, where applying ADSA significantly boosts accuracy while preserving the simplicity of the original distillation framework.
Y
+ours
21.4
33.2
$8.4
Table 4 presents results under varying soft-label budgets. Conventional methods generate soft labels for each the N distilled images across all student training epochs, resulting a total budget of N x K labels. Our method demonstrates its efficiency by operating a smaller budget: we generate soft labels only for the first K' epochs (K' < K) and then reuse this static set of labels for the full K-epoch training schedule. Though reducing the number of soft-labeling epochs in SRe2L

---

and EDC on ImageNet-Ik-LL Ik-LT with IPC=50, our method consistently outperforms baselines, showing robustness under limited supervision.
We then verify the effectiveness of ADSA in mitigating the two types of biases discussed in Section 3.2
including the bias from distilled images and the bias from the distillation model. Table shows that when the distilled images are obtained from an unbiased distillation model (training on balanced dataset) while the soft labels are derived from a biased one (training on imbalanced dataset), our method improves performance. Table 6 iidicates that even when both the distilled images and soft labels are biased, our method still outperforms the baseline that uses soft labels from an unbiased distillation model. These two experiments verify that our method can effectively mitigate the two types of biases present the soft labels.
To further assess the adaptiveness of the ADSA module, we examine its performance under varying IPC and imbalance factor (IF) settings, shown in panels (a) and (b) in the right part of Figure Results show that no single fixed calibration strength (7) consistently yields optimal performance across all settings, while ADSA achieves the best accuracy in most cases. The advantage is especially notable under extreme IPC and IF values. Being hyperparameter-free ADSA dynamically adjusts calibration without tuning, ensuring robust and stable performance across scenarios.
4.3 Interpretability
Panel (c) in Figure 3 jillustrates class-wisc confidence differences, computed the model's confidence on original training images minus that on distilled images. The resulting pattern reveals a domain shift between the two sources, particularly pronounced in tail classes, suggesting that distilled images can effectively act as a proxy hold-out set for calibration. Panel (d) in Figure 3 shows the per-class confidence distribution from different sources of soft labels. It shows that the soft labels from the ADSA module align well with the soft labels from the distillation model trained on a balanced dataset. More analysis and visualization are provided in Appendix A.7
Conclusion
This paper is the first examine the crucial role of soft labels in long-tailed dataset distillation. We propose an imbalance-aware generalization bound and conduct perturbation analysis of existing distillation techniques. Our analysis reveals two sources of bias in soft labels, and we introduce simple yet effective module that consistently improves performance over baselines across various settings, without requiring additional architectural complexity or computational overhead.
Limitations and Future Work Our method assumes shared conditional distribution between training and test sets, i.e., pxr(xly) Pte (xly), which is common in long-tailed learning literature. However, this assumption may not hold in the presence of domain shift. Additionally, the current design relies on access to long-tail prior statistics and is not directly suited for online or streaming learning settings where such information is unavailable. Future work could focus on developing requeency-gnnoste methods for online scenarios and exploring more efficient techniques for soft-label representation, storage, and utilization. Additionally, tailored distrir-distiibutionn matching strategies, such as transfer learning from head-to-tail classes and feature augmentation methods common in long-tailed recognition, may further enhance tail-class feature alignment.
Acknowledgments
This work was supported by National Natural Science Foundation China (grant No. 02350710797) We would like thank the reviewers for their invaluable feedback and constructive comments to improve the paper.
References
[1] Anas Awadalla, Le Xue, Oscar Lo, Manli Shu, Hannah Lee, Etash Kumar Guha, Sheng Shen, Mohamed Awadalla, Silvio Savarese, Caiming Xiong, Ran Xu, Yejin Choi, and Ludwig Schmidt. MINT-It: Scaling open-source multimodal data by 10x: A multimodal dataset with one trillion
10

---

tokens. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024.
[2] Keshigeyan Chandrasegaran, Agrim Gupta, Lea M. Hadzic, Taran Kota, Jimming He, Cristobal Eyzaguirre. Zane Durante, Manling Li, Jiajun Wu, and Li Fei-Fei. Hourvideo: 1-hour video- language understanding. In The Thirty-eighi Conference on Neural Information Processing Systems Datasets and Benchmarky Track, 2024.
[3] Daochen Zha, Zaid Pervaiz Bhat, Kwei-Herng Lai, Fan Yang, Zhimeng Jiang, Shaochen Zhong, and Xia Hu. Data-centric artificial intelligence: A survey. ACM Comput. Surv., 57(5), January 2025.
[4] Tongzhou Wang, Jun- Yan Zhu, Antonio Torralba, and Alexei A. Efros. Dataset distillation. CoRR, abs/1811..19901 2018.
[5] Shaobo Wang, Yicun Yang, Zhiyuan Liu, Chenghao Sun, Xuming Hu, Conghui He, and Linfeng Zhang. Dataset distillation with neural characteristic function: A minmax perspective, 2025.
[6] Donghyeok Shin, HeeSun Bae, Gyuwon Sim, Wanmo Kang, and Il-chul Moon. Distilling dataset into neural field. In The Thirleenth International Conference on Learning Representations, 2025.
[7] Dai Liu, Jindong Gu, Hu Cao, Carsten Trinitis, and Martin Schulz. Dataset distillation by automatic training trajectories. In Computer Vision ECCV 2024, pages 334-351, Cham, 2025.
[8] Xinyi Shang, Peng Sun, and Tao Lin, Gift: Unlocking full potential of labels distilled dataset at near-zero cost. In The Thirteenth International Conference Learning Representations, 2025.
[9] Yifan Zhang, Bingyi Kang, Bryan Hooi, Shuicheng Yan, and Jiashi Feng. Deep long-tailed learn- ing: survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(9):00795- 10816, 2023.
[10] Lu Yang, He Jiang, Qing Song, and Jun Guo. A survey on long-tailed visual recognition. International Journal of Computer Vision, 130(7):1877-177, July 2022.
[11] Zhenghao Zhao, Haoxuan Wang, Yuzhang Shang, Kai Wang, and Yan Yan. Distilling long-taile tailed datasets, 2025.
[12] Yongchao Zhou, Ehsan Nezhadarya, and Jimmy Ba. Dataset distillation using neural feature regression. In Advances in Neural Information Processing Systems, volume 35, pages 9813- 9827, 2022.
[13] Justin Cui, Ruochen Wang, Si Si, and Cho-Jui Hsich. Scaling up dataset distillation to imagenet- 1k with constant memory. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 6565-6590, July 2023.
[14] Zeyuan Yin, Eric Xing, and Zhiqiang Shen. Squeeze, recover and relabel: Dataset condensation at imagenet scale from a new perspective. In Advances in Neural Information Processing Systems, volume 36, pages 73582-73603, 2023.
[15] Duo Su, Junjie Hou, Weizhi Gao, Yingjie Tian, and Bowen Tang. DM4: Dataset distillation via disentangled diffusion model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5809-5818, June 2024.
[16] Shitong Shao, Zikai Zhou, Huanran Chen. and Zhiqiang Shen. Elucidating the design space of dataset condensation. In The Thirty-Fight Annual Conference Neural Information Processing Systems, 2024.
[17] Timothy Nguyen, Roman Novak, Lechao Xiao, and Jaehoon Lee. Dataset distillation with infinitely wide convolutional networks. Advances in Neural Information Processing Systems, volume 34, pages 5186-5198, 2021.

---

[18] Yunzhen Feng, Shanmukha Ramakrishna Vedantam, and Julia Kempe. Embarrassingly simple dataset distillation. The Twelfth International Conference on Learning Representations, 2024.
[19] Bo Zhao, Konda Reddy Mopuri, and Hakan Bilen. Dataset condensation with gradient matching. In International Conference on Learning Representations, 2021.
[20] Sachyung Lee, Sanghyuk Chun, Sangwon Jung, Sangdoo Yun, and Sungroh Yoon. Dataset condensation with contrastive signals. In Proceedings of the 39th International Conference on Machine Learning. volume 162 of Proceedings of Machine Learning Research, pages 22352-12364, July 2022.
[21] Lei Zhang, Jie Zhang, Bowen Lei, Subhabrata Mukherjee, Xiang Pan, Bo Zhao, Caiwen Ding, Yao Li, and Dongkuan Xu. Accelerating dataset distillation via model augmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 11950-11959, June 2023.
[22] JAWHI DU, Qin Shi, and Joey Tianyi Zhou. Sequential subset matching for dataset distillation. In Advances in Neural Information Processing Systems, volume 36, pages 67487-67504, 2023.
[23] George Cazenavette, Tongzhou Wang, Antonio Torralba, Alexei A. Efros, and Jun-Yan Zhu. Dataset distillation by matching training trajectories. In Proceedings of the IEEF/CVF Confer- ence on Computer Vision and Puttern Recognition (CVPR), pages 10718-10727, June 2022.
[24] Ziyao Guo, Kai Wang, George Cazenavette, HUT LI, Kaipeng Zhang. and Yang You. To- wards lossless dataset distillation via difficulty-aligned trajectory matching. In The Twelfth International Conference on Learning Representations, 2024.
[25] Bo Zhao and Hakan Bilen. Dataset condensation with distribution matching. In Proceedings of the IEEE/CVE Winter Conference on Applications of Computer Vision (WACV), pages 6514-6523, January 2023.
[26] Ahmad Sajedi, Samir Khaki, Ehsan Amjadian, Lucy Z. Liu, Yuri A. Lawryshyn, and Kon- stantinos N. Plataniotis. Datadam: Efficient dataset distillation with attention matching. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 17097-17107, October 2023.
[27] Yanqing Liu, Jianyang Gu, Kai Wang. Zheng Zhu, Wei Jiang, and Yang You. Dream: Efficient dataset distillation by representative matching. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), pages 77314-17324 October 2023.
[28] Hongxu Yin, Pavlo Molchanov, Jose M. Alvarez, Zhizhong Li, Arun Mallya, Derek Hoiem, Niraj K. Jha, and Jan Kautz. Dreaming to distill: Data-free knowledge transfer via deepinversion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
[29] Noel Loo, Alaa Maalouf, Ramin Hasani, Mathias Lechner, Alexander Amini, and Daniela Rus. Large scale dataset distillation with domain shift. In Forty-First International Conference Machine Learning, 2024.
[30] Shitong Shao, Zeyuan Yin, Muxin Zhou, Xindong Zhang, and Zhiqiang Shen. Generalized large-scale data condensation via various backbone and statistical matching. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 16709-16718, June 2024.
[31] George Cazenavette, Tongzhou Wang, Antonio Torralba, Alexei A. Efros, and Jun- Yan Zhu. Generalizing dataset distillation via deep generative prior. Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1739-3748 June 2023.
[32] Jianyang Gu, Saeed Vahidian, Vyacheslav Kungurtsev, Haonan Wang, Wei Jiang, Yang You, and Yiran Chen. Efficient dataset distillation via minimax diffusion. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 15793-15803, June 2024.
12

---

[33] Tian Qin, Zhiwei Deng, and David Alvarez-Melis. A label is worth a thousand images in dataset distillation. In The Eightth Annual Conference on Neural Information Processing Systems, 2024.
[34] Zekai Li, Xinhao Zhong, Zhiyuan Liang, Yuhao Zhou, Mingjia Shi, Ziqiao Wang, Wangbo Zhao, Xuanlei Zhao, Haonan Wang, Ziheng Qin, Dai Liu, Kaipeng Zhang, Tianyi Zhou, Zheng Zhu, Kun Wang, Guang Li, Junhao Zhang, Jiawei Liu, Yiran Huang. Lingjuan Lyu, Jiancheng Lv, Yaochu Jin, Zeynep Akata, Jindong Gu, Rama Vedantam, Mike Shou, Zhiwei Deng, Yan Yan, Yuzhang Shang, George Cazenavette, Xindi Wu, Justin Cui, Tianlong Chen, Angela Yao, Manolis Kellis, Konstantinos N. Plataniotis, Bo Zhao, Zhangyang Wang. Yang You, and Kai Wang. Dd-ranking: Rethinking the evaluation of dataset distillation. GitHub repository, 2024.
[35] Lingao Xiao and Yang He. Are large-scale soft labels necessary for large-scale dataset distil- lation? In The Thirty-Eightl Annual Conference Neural Information Processing Systems, 2024.
[36] Ondrej Bohdal, Yongxin Yang, and Timothy Hospedales. Flexible dataset distillation: Learn labels instead of images, 2020.
[37] Ilia Sucholutsky and Matthias Schonlau. Soft-label dataset distillation and text dataset dis- tillation. In 2021 International Joint Conference on Neural Networks (IJCN), pages 1-8, 2021.
[38] Shaobo Wang, Yantai Yang, Shuaiyu Zhang, Chenghao Sun, Weiya Li, Xuming Hu, and Linfeng Zhang. Drupi: Dataset reduction using privileged information, 2024.
[39] Seoungyoon Kang, Youngsun Lim, and Hyunjung Shim. Label-augmented dataset distillation. In 2025 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 1457-1466, 2025.
[40] Chongsheng Zhang, George Almpanidis, Gaojuan Fan, Binquan Deng, Yanbo Zhang, Ji Liu, Aouaidjia Kamel, Paolo Soda, and João Gama. systematic review on long-tailed learning. 2024.
[41] Bang-Xin Shi, Tong Wei, Yuke Xiang, and Yu-Feng Li. How re-sampling helps for long-tail learning? In Thirty-Seventh Conference on Neural Information Processing Systems, 2023.
[42] Boyan Zhou, Quan Cui, Xiu-Shen Wei, and Zhao-Min Chen. Bbn: Bilateral- branch network with cumulative learning for long-tailen tailed visual recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
[43] Ziwei Liu, Zhongqi Miao, Xiaohang Zhan, Jiayun Wang. Boqing Gong, and Stella X. Yu. Large- scale long-tailed recognition an open world. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
[44] Peng Chu, Xiao Bian, Shaopeng Liu, and Haibin Ling. Feature space augmentation for long- tailed data. In Computer Vision ECCV 2020, pages 694-710, Cham, 2020.
[45] Jaehyung Kim, Jongheon Jeong, and Jinwoo Shin. M2m: Imbalanced classification via major- to-minor translation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
[46] Yin Cui, Menglin Jia, Tsung- Yi Lin, Yang Song, and Serge Belongie. Class-balanced loss based on effective number of samples. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
[47] Jiawei Ren, Cunjun Yu, shunan sheng, Xiao Ma, Haiyu Zhao, Shuai Yi, and hongsheng Li. Balanced meta-softmax for long-tailed visual recognition. In Advances in Neural Information Processing Systems, volume 33, pages 4186-4186, 2020.
[48] Bingyi Kang, Saining Xie, Marcus Rohrbach, Zhicheng Yan, Albert Gordo, Jiashi Feng, and Yannis Kalantidis. Decoupling representation and classifier for long-tailed recognition. In International Conference on Learning Representations, 2020.
13

---

[49] Aditya Krishna Menon, Sadeep Jayasumana, Ankit Singh Rawat, Himanshu Jain, Andreas Veit, and Sanjiv Kumar. Long-tail learning via logit adjustment. International Conference on Learning Representations, 2021.
[50] Vladimir Vapnik. The nature of statistical learning theory. Springer science & business media, 2013.
[51] Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning imbal- anced datasets with aabel-distrbuttinn- Processing Systems, volume 32, 2019.
margin loss. In Advances in Neural Information
[52] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[53] Ganlong Zhao, Guanbin Li, Yipeng Qin, and Yizhou Yu. Improved distribution matching for dataset condensation. In Proceedings of the IEEF/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pages 7856-7865, June 2023.
[54] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hicrarchical image database. In 2009 IEEE Conference Computer Vision and Pattern Recognition, pages 248-255, 2009.
[55] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and paltern recognition, pages 770-778, 2016.
[56] Tsung- Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollar. Focal loss for dense object detection. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), October 2017.
[57] Tianhong Li, Peng Cao, Yuan Yuan, Lijie Fan, Yuzhe Yang, Rogerio S Feris, Piotr Indyk, and Dina Katabi. Targeted supervised contrastive learning for long-tailed recognition. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 6918 0928,,698228888 2022.
[58] Xiao Zhang, Zhiyuan Fang. Yandong Wen, Zhifeng Li, and Yu Qiao. Range loss for deep face recognition with long-tailed training data. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), Oct 2017.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction clearly state the main contributions of the paper, including (1) the derivation of an imbalance-awant generalization bound, (2) the identification of entangled soft-label biases from teacher models and distilled images, and (3) the proposal of the ADSA module for post-hoc bias calibration. These claims are theoretically grounded in Section 3.1 and Appendix A, and experimentally validated in Section 4.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
The paper explicitly discusses and hilight limitations the "Conclusions" section, highlighting key assumptions such the shared class-conditional distribution (Pr(x = Pre (z y)). which may not hold under domain shift. It also notes that the proposed method relies access long-tail prior statistics, limiting its applicability in online or streaming scenarios.
Justification:
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best
15

---

judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: The paper provides a complete theoretical derivation Theorem 3.1 (Section 3.1) and the assumption. The full proof, including two equivalent forms of the bound and their implications, is presented in Appendix A.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: We have described the details of dataset, backbone and training in the main context and Appendix A.3 including the perturbation experiment details in Appendix A.3
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
16

---

(c) If the contribution is a new model (e.g.. a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We will submit the core code for SRe2L/GVBSM/EDC training on ImageNet- LT as supplementary files. The complete codebase will be released on GitHub upon paper acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips) cc/
guuiees/ooeeSumisssee
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
ssionooiccy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We have described the details of dataset, backbone and training in the main context and Appendix AA.3 including the perturbation experiment details in Appendix A.3 We also described the details of ablation study.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
17

---

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Due to the large experimental space involving combinations of IPC, imbalance factor (IF), and datasets (CIFAR/ImageNet, would be computationally expensive repeat all settings multiple times. Therefore, we report deterministic results under fixed seeds, which reflect consistent trends across extensive baselines and datasets (see Tables 1-4). While formal statistical variance is not reported, the performance gains are substantial and consistent, supporting the reliability of our claims.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula,
call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report I-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (c.g. negative error rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
All experiments are conducted on an 8 RTX 4090 server.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't it make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: The research fully complies with the NeurIPS Code of Ethics. All experiments are conducted on publicly available datasets (CIFAR-10/100, ImageNet-LT)), and no per- sonally identifiable sensitive data is used. The proposed method poses no foreseeable risk of harm, misuse, or unfair bias, and care taken to preserve anonymity in all submitted materials.
18

---

Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This paper focuses on foundational research in dataset distillation under long-tailed distributions, with no direct application to downstream systems or real-world deployments. While the method could enhance training efficiency and reduce carbon footprint, potential societal impacts such as fairness or misuse are minimal due to the abstract nature of the contribution. Therefore, we did not explicitly discuss broader impacts in the paper.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymeen of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
The paper does not release any pretrained large-scale models or internet- scraped datasets, and the distilled datasets used are based well-established public bench- marks (CIFAR-10/100, ImageNet-LT). Therefore, the work poses no foreseeable risks that require special safeguards.
Justification:
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
19

---

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The paper uses publicly available datasets (CIFAR- 10/100 and ImageNet-LT) and reproduces baseline methods such as SRe2L. GVBSM, and EDC. We cite the original papers corresponding to these assets and adhere to their respective licenses (e.g., CIFAR under MIT license, ImageNet under research-only terms). No scraped or proprietary data used.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi phcore thcode. com/ datasete
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does introduce new assets. All experiments are conducted on publicly available datasets using existing distillation baselines, and no new dataset, model, code repository is released submission time.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Question:
Answer: [NA]
20

---

The paper does not involve crowdsourcing research with human subjects. All experiments conducted using publicly available machine learning datasets without human annotation or interaction.
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve any research with human subjects or crowdsourcing. All experiments are conducted on publicly available datasets without human participation or interaction.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: The research does not involve any of large language models (LLMs) part the core methodology. LLMs were not used in algorithm design, theoretical development, experimental implementation.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMSs any important, original, non-standard components.
Please refer to our LLM policy for what should or should not be described.
21

---

A Technical Appendices and Supplementary Material
A.1 List of Notations
Table Summary of frequently used notations.
Symbol
Description
Dvr
Training original dataset
Daa
Distilled dataset
Die
Test dataset
D
Dataset
[D]
Size of dataset
fo
Model with parameters 0
Input data (e-g.. image)
Label (e.g., one-hot label or soft label)
(),
Loss function on dataset D with model parameters Cross-entropy loss on (xr
L(Jo(x),))
Per (2,
Joint distribution training data and labels
Pad(z,
Joint distribution of distilled data and labels
Pie (a.
Joint distribution test data and labels
blylz)
Predicted probability of label y given input æ. Could refer to model trained pad Upper bound log p0y(2:
Rad
Regularization term in generalization bound in Eq
nk
Number of samples class in Drr
K
Number of classes
(xl(pl
Kullback-Leible divergence between distributions y and Constant term independent of paa(2; y) Distilled images in the distillation process Original images from the training dataset Regularization term for distribution of distilled images and original images Feature extractor function, the backbone of the model
const
Dort
Rreg
g(x)
pphe (1)2z)
The observed soft label posterior distribution
PDD (ylz)
sarget
The ideal or desired soft label posterior distributtonnunbiisedd)
(1(2))
The bias introduced by the distillation model
er(ylz)
The bias introduced by the distilled images The label set, i.c., {0, 1,..., K 1} The y-th output of the model f(x) Calibration hyperparameter Empirical frequency of class y in Der
[K]
fo(
Ty
A.2 Derivation of Theorem 3.1
In conventional dataset distillation, assuming the cross-entropy loss function:
4, Ep.r 1- log p [x]]
(11)
lad Ep.al (- log [
(12)
where Der represents the training dataset distribution, and Dad represents the distilled dataset distribution. Then, following (2)]. we have:
C CVVD DKL (por Vlaaa (x) y))
ler < las +
(13)
2/2
which further leads to:
22

---

C 2
Le = Fer I lad +
VDR (per (P)((a)ppaa(x) + DKL (Per x) Paa x)).
(14)
The approximation 4y- z le is justified by VC theory [50] for sufficiently large training set, provided that the training and test distributions are identical (ptr (2, y) Pte (z, y)). Consequently, the upper bound on brr can be applied to les establishing the minimization of both the distilled loss lad and the distributional discrepancy as a viable strategy for constraining the test loss.
However, in long-tailed scenarios, the assumption (er y) pte(a, v) does not hold. Therefore, we introduce Pte while considering the conventional long-tail assumption piy (zly) = Pre (zly) [5]]
Pie (xll) Ptr (zly)
(15)
(x, y) (te(()
Ptr (2, v)
(16)
(er
The conditional distribution assumption in long-tailed recognition, ptr(x)l) = Pte Pte (xly), remains valid in
dataset distillation. In this setting, the original training set is long-tailed dataset Drr, from which distilled dataset Daa is derived. The model is trained on Daa and evaluated balanced test set Dic. Since Dtr and Drc. are identical to those used in standard long-tailed learning, the long-tailed assumption naturally extends to the distillation scenario.
Substituting Eq. into Eq. 13 we obtain:
C
lee S lad +
/
DDL (T,
(17)
2/2
C
(te
Lad +
DKL.
Per (2. 8) v)lapa(zz
(18)
2v² V
Per (t)
Expanding the KL divergence:
(y) Per (2, v) Ilpaa(2 (.)
Pte (v)
Pte Per v) dxdy (19)
DEL
Per u) log
Per(y)
Per (y)
per (y)Pad y)
There are two possible simplifications for Eq.
18 Form 1:
Prely) Per(y) (v) Per (I, a)llpaa(xx
DKL.
(20)
(re(y)
Pie (e)) v) dxdy
Per (1) Per (x. v) log
(21)
Per (u)Paaaa v)
(r)(y)
Per (v(=)) dxdy + /
Pto(U) (e)
per (x., v) log Per (x) drdy
Per (x. v) log
(22)
Per (1)
Paa (1)())
pea(z)
Pie (y)
Pte (v) dody
Per (y) Per (x, w) log
Per
(y)
= - Pealz, v) log
Per (w((x) dody / Pio v) log
Per (x) dxdy (=d(
(23)
(y)(y(
+ ff Pro(u) log
Pie (v)
dxdy
(24)
Per (y)
-/ prolu, log
Per (1)(z) (te()) drdy / (ro(I,e)) log
(x) Pee (re
drdy
(25)
Pto (t)( pa)(ylz)
(re(x)) Paa
+ DKL (KL(())())) IPPr(y))
=Dri. (Kr(ppre (2/ (paa(()))) Dwr.
(26)
DrL (pte (p)) (xz(Pic(=)llpertet + D (Pre (v) (ppr
=Dx (pro (plaa(((l) DxL (pte (x))aaa())) + const
(27)
= Dxx (ptc (pep + DKL (f
(Pre(alu)Ppa / Pealall pealaal)paaal)dd + const (28)
23

---

Here, the const term groups all expressions that are independent of our optimization target (r, y).
Form 2:
Ptaly) (a, y) (llapa((1)))
DEL
(29)
(v)
Ptc
Pee (y) (z. 5) drdy
Per (V) Per (1s, y) log
(30)
Per (xr(3)) (V)Paal
Pro(y)
per (3r
(1ly)) dndy +
Pte (v)
Pto Proly)per(s) (1)
Per (2., y) log
Per (v) Per (r. y) log Per (V)paa)))))
drdy
(31)
Per Per(y)
Paa
Ptr (x:ly))) drdy
Pro (ro log Pto (u (v)
Pie Per log
dy
(32)
pad
Per (U))(a))))))
Pte (ve(v)Dre DEr. (1w dy + DKr. (vte(()
(33)
Thus, two possible upper bounds exist: Form 1:
C
Le Lad +
v (KrssProoull:)lllaaaaaa + DE (pre + const
(34)
2v2
C
= =lad + / Rdd
(35)
2/2
Rad = (KL(Pie(ll=)llaaallll + DEL (ʃ (molewwirooosaalll / (aalah)naaaaa + const
(36)
Form 2:
C
he = lad +
2/2 proly) DE. (per (x)l)) (ppaa())))))) dy + D (pee(())))) (paa(y))
(37)
In Eq. 36 the first term suggests that the classifier Paa(y]a) trained on the distilled dataset should align with the classifier Pte de-yyla ()) obtained from long-tal tail calibration. This means that any long-tail calibration model trained on Ptr (z. y) can serve as a teacher model in dataset distillation. The second term indicates that the overall distribution of the distilled dataset should align with the test distribution, which can be estimated using methods like Gaussian mixture models to promote better alignment.
In Eq. 37 the first ferm suggests ensuring that the statistical information of each class remains consistent, meaning that the conditional distribution (xl()))) in the training set should align with the corresponding distribution p in the distilled dataset. The second term indicates that the class distribution of the distilled dataset should match that of the set, which implies maintaining fixed number of instances per class (IPC).
Since assume Pie K. obtain:
DxL (Per (1)))))aaa((())))) = -
(38)
which satisfies both:
I pio De (1l dy
(39)
and
DKL (eeelolu)peessdd)) pullala)paa()dd
(40)
A.3 Experiment Details
Distillation Pipeline The baseline methods we adopt (e.g. SRe2L 14, GVBSM 30) and
EDC [16) follow the same dataset distillation pipeline, as illustrated in Fig. [(), which consists of three stages. The first stage (squeeze stage) involves training distillation model on the original dataset. The second stage (recover stage) generates distilled images by optimizing the objective in Eq. 7 The third stage (relabel stage) uses the distillation model to predict soft labels for the distilled images. The resulting distilled images and their corresponding soft labels are then used together to evaluate the quality of the distilled dataset by measuring the performance of student models trained
on it.
24

---

Hyperparameters For all baselines, we adopt their default hyperparameter settings, optimizer and augmentation strategy except for special demonstration. The optimization problem in Equation 10 is solved by performing a search over the range E (0, (0. Specifically, DREAM splits each distilled image into four individual clips and resizes them to train the evaluation model. We generate a soft label for each clip once per epoch. In the soft label budget experiment, we generate soft labels for k epochs and use only these k-epoch soft labels to train the evaluation model for 300 epochs. Since part of the data in LTDD [II] is missing, we manually reproduced the results for CIFAR-10 with IPC=1, for CIFAR- 100 with IPC=1 across all imbalance factors (IF), and for IPC=10 and 50 under IF=50 and 100. For all other settings in LTDD, we used the data reported in their paper.
Architectures We use depth-3 ConyNet as both the evaluation and distillation backhone for MTT and DREAM, and adopt ResNet-1 18 as the evaluation backbone for SRe2I., GVBSM, and EDC. All backbones are manually trained on long-tailed CIFAR and ImageNet- 1k datasets without using any pretrained models from the PyTorch model zoo. The backbone used is shown in Table
For CIFAR dataset distillation, we use the default model architectures provided in the official codebases. For ImageNet-Ik, the original GVBSM and EDC papers employed EfficientNet-Bi and ShuffleNet- V2-x0.5, respectively. Due to the lack of publicly available training code for these models on ImageNet- Ik, we replace them with EfficientNet-V22 V2-S and ShuffleNet- V2-x1.5. We use the mean of calibrated soft label from trained distillation models as the final soft label.
Compute Resources All experiments are conducted on an RTX 4090 GPUs server, and the on computational cost depends on the underlying baseline methods. When IPC 50, dataset distillation typically takes less than 10 GPU hours. Among the methods, SRe2L is the fastest (around 2 GPU hours), while GVBSM is the slowest (around 10 GPU hours). On ImageNet, the SRe2L is the fastest (around 2 GPU days) and GVBSM the slowest (around 15 GPU days). The exact runtime may fluctuate depending on the GPU model, GPU/CPU utilization, and inter-GPU communication efficiency. The additional computational and memory overhead introduced by ADSA negligible.
Perturbation Analysis Details The perturbation study was performed on CIFARI00 with 100 classes, we limit the total number of images to be 10000 images, but the number of images for a head class could exceed 100, and then we distribute a images for the first 20 classes, then we distribute (10000 20 a) /1000 20). and then round integer. We use 10000 images to avoid the number of head class to exceed the maximum number(500) for each class.
The perturbation study is conducted CIFAR-100, which contains 100 classes. We limit the total number of images to 10,000 to ensure that the number of images per head class does not exceed the dataset's upper limit of 500 images per class. Specifically, we assign images to each of the first 20 classes (head classes), and distribute the remaining images uniformly across the remaining 80 classes (tail classes) using 10,000- 20-a followed by rounding to the nearest integer. Fixing the total 100-20
number of images, rather than using an exponentially decaying class frequency distribution, allows us to control the dataset size and eliminate its influence on performance. This enables a more controlled evaluation of the impact of tail-class sample sizes. Furthermore, by averaging the statistics over the first 20 classes, we can quantitatively assess performance across varying degrees of class imbalance while minimizing the influence of individual class identities.
A.4 Analysis of Pretraining Epoch Selection
In Tables 9 and TO we present the performance of distilled datasets generated by distillation models trained with different pretraining epochs on CIFAR-IO and CIFAR-100. Tables [1 and 12 summarize the best accuracy achieved under each imbalance factor (IF) by selecting the optimal epoch, while Tables 13 and 14 report the corresponding optimal epochs. In these tables, each column corresponds to different IPC setting: IPC = 1, 10, or 50. The results clearly indicate that the choice of training epochs has a substantial impact on the quality of the distilled dataset, with performance variations exceeding 10% in some cases. Moreover, we observe that the optimal number of epochs tends to increase with higher IF and IPC values. This suggests that for easier cases (e.g., low or IPC), early- stage models are sufficient to yield high-quality distilled data, whereas more challenging scenarios require prolonged training to achieve optimal performance. Notably, Tables TT and 12 can be viewed as enhanced versions of the results in Table I where SRe2L benefits from epoch tuning. Under
25

---

Table 8: Backbone architectures used by SRe2L, GVBSM, and EDC on CIFAR-IO, CIFAR-100, The models are used to distill images and assign soft label.
Method
CIFARIO
CIFARIO0
ImageNet-Ik
SRe2L GVBSM
ResNet-18
ResNet-18 ResNet-118
ResNet-18
ResNet-18, 18,
Nenv
ConvNet- MobileNetV2, WRN-16-2, ShuffleNet-
ResNet-18, MobileNetV2, EfficientNetV2-S, ShuffteNd-V2-x1.5
W128. MobileNetV2, WRN-16-2, ShuffleNet-
W128,
V2-x0.5
V2-x0.5
EDC
ResNet-18,
ConvNet-
ResNet-18, ConvNet-
ResNet-18, MobileNetV2,
W128. MobileNetV2.
W128, MobileNetV2.
EfficientNeeV2-S,
WRN-16-2. ShuffleNet-
WRN-16-2. ShuffleNet-
ShuffleNet-V2-x0.5.
V2-x0.5. ConvNet-D1.
V2-x0.5, ConvNet-W33:
AlexNet
ConvNet-D2, ConvNet- W32
ConvNet-D1, ConvNet-
D2
this enhanced setting, the performance gap between our method and the baselines further widens, particularly under high IPC, highlighting the robustness and scalability of our approach.
We adopt this flexible epoch selection strategy in Figure B)) Table Table and Table 15 to more accurately evaluate and reflect the effectiveness of our method. In particular, for Figure 3 (), using fixed number of training epochs (e.g.. 200) to match the baseline setting may result in anomalous accuracy trends, specifically, performance may even increase with higher imbalance factors (IF), which contradicts expected behavior. This is because fixed epoch value is not necessarily optimal, and the optimal epoch often shifts later as the imbalance increases.
Table 9: CIFAR-I0 accuracy under different pre- Table 10: CIFAR-100 accuracy under different pretraining epochs
training epochs
Epoch
IF
Method
10
50
Epoch
IF
Method
10
50
baseline
18.0
39.6
55.0
baseline
17.3
31.7
32.6
10
ours
23.1
41.9
63.2
ours
17.3
37.2
39.3
baseline
20.4
22.8
24.6
baseline
17.5
21.5
21.9
10
50
ours
26.2
34.9
38.7
10
50
ours
18.1
25.5
25.6
bascline
16.3
21.6
23.6
baseline
15.7
17.5
17.6
100
23.0
33.0
37.9
100
7.7
18.0
17.8
baseline
14.4
27.2
51.5
haseline
8.4
35.4
43.7
10
16.7
26.2
50.0
10
7.0
35.1
53.3
haseline
14.1
26.4
50.3
baseline
9.6
28.5
33.0
30
50
ours
18.2
30.7
63.7
50
50
ours
8.9
34.2
43.3
baseline
15.2
28.1
38.2
baseline
10.2
29.0
31.4
100
ours
22.0
32.0
59.1
100
ours
9.3
33.0
39.2
baseline
14.2
24.6
46.8
baseline
34.9
10
ours
15.7
25.2
53.6
ours
7.6
33.2
53.7
baseline
15.4
26.5
39.9
baseline
8.2
30.1
35.0
50
50
ours
18.7
32.1
52.1
100
ours
7.7
31.5
45.3
baseline
17.9
23.9
40.3
baseline
8.9
28.1
31.8
100
100
ours
19.4
29.0
53.8
ours
31.6
41.7
baseline
14.0
23.5
41.4
baseline
7.4
27.0
41.4
10
17.0
25.8
50.8
10
6.4
25.9
47.2
baseline
13.0
22.3
39.3
baseline
25.0
70
50
ours
14.3
29.0
56.2
200
50
ours
7.1
26.3
42,5
baseline
13.2
21.2
40.3
baseline
7.5
22.6
28.9
100
ours
15.3
28.1
54.7
100
ours
7.0
23.2
37.6
A.5
Comparison with Enhanced Resampling Baseline
Table 15 compares our method with enhanced baseline, where distillation models are trained using
a class-balanced resampling strategy that ensures each class appears with equal frequency during pre-training. The experimental results show that our method outperforms both the baseline and the
26

---

Table 11: CIFAR-10 10 max accuracy for different epochs
Table 12: CIFAR-100 max accuracy for different
epochs
IF
Method
10
IF
Method
1
10
50
baseline
18.0
39.6
55.0
haseline
9.1
35.4
46.1
23.1
41.9
63.2
7.6
37.2
53.7
baseline
20.4
26.5
50.3
baseline
9.6
30.1
35.0
ours
26.2
34.9
63.7
ours
8.9
34.2
45.3
baseline
17.9
28.1
40.3
baseline
8.9
29.0
31.8
100
ours
23.0
33.0
59.1
100
ours
9.3
33.0
41.7
Table 13: CIFAR-10 10 best epoch
Table 14: CIFAR 100 best epoch
IF
Method
1
10
50
IF
Method
1
10
50
baseline
10
10
baseline
100
50
100
10
ours
10
10
10
10
ours
100
10
100
baseline
10
50
30
baseline
.0
100
100
50
ours
10
30
50
ours
.0
50
100
baseline
50
30
50
baseline
100
50
100
100
ours
10
10
30
100
ours
50
50
100
resample setting in most cases, while resampling indeed improves over the baseline. The results show that our method could be seamlessly integrated a plug-and-play module to further boost the performance of resampling- based pipelines in most cases. It is important to note that, to enable fair comparison with the resampling-baat based methods, we also adopted the best-epoch selection strategy for all configurations (SRe2L, tresample, +ours, tresample+ours) For each imbalance factor (IF), the number of training epochs was selected to ensure optimal performance. Specifically, epochs were selected from (10, 30, 50) for CIFAR-10 10, and from (10, 50, 100, 200} for CIFAR-100. The results are also slightly better than those in Table
A.6 Comparison with Classic Long-tailed Recognition Methods
We also compare the results of our method with classic long-tailed recognition approaches in Table I6 and Table T7[as aas reported in (T)) to demonstrate that our method provides a data-centric solution for transforming imbalanced data into balanced form, enabling models to be trained directly without requiring specialized designs for long-tain tailtd scenarios.
A.7 Further Ablation and Visualization
orghelef-31 Ef
originl [1-221
orgenl 2001
Bost Tau Distribttinn
0.0
0.5
1.5
2.0
2.5
3.5
Bot Thu
Min Std Distribution
10-
Dualit
InageNe:
0.0
0.1
0.2
0.3
0.4
Xin Std
Clacs
Figure 4: Left: Distribution of selected 7 across different datasets. The value of varies within the same dataset due to differences in IPC, IF, and the random seed used in training. Right: t-SNE visualization of long-tailed and distilled datasets under different imbalance factors (IF). The top row shows the original long-tail tailed dataset distributions, which remain well-separated across different IF values. The bottom row presents the corresponding distilled datasets, where class clusters become increasingly compressed and less distinguishable as the imbalance factor increases.
27

---

Table 15: Performance comparison with a resampling baseline on CIFAR-10/100-1E
CIFAR-10-LT
IPC=1
IPC=10
TPC=50
Imbalance Factor
10
50
100
10
50
100
10
50
100
SRe2L.
19.9
19.9
18.7
39.0
27.2
21.3
55.2
51.1
24.0
+resample
18.1
21.7
25,5
38.1
41.3
36.4
54.8
54.6
49.5
+ours
23.7
26.9
21.4
40.2
35.4
33.2
63.1
63.7
58.4
+resampletours
19.5
22.3
24.1
38.0
44.1
37.1
58.8
56.6
56.1
CIFAR- 100-LL
IPC=1
IPC=10
IPC=50
Imbalance Factor
10
100
50
100
50
SRe21.
16.3
17.8
15.9
35.8
28.8
29.1
45.2
33.5
31.8
+resample
20.3
18.9
16.3
35.8
29.2
28.4
45.1
36.1
32.8
+ours
16.9
9.6
38.2
34.2
33.0
53,9
45.6
+rsample+ours
18.5
17.6
15.1
37.6
32.2
29.9
50.6
39.6
34.3
Table 17:
Comparison with Long- Taig-Tailed
Recognition Methods on ImageNet
Table 16: Comparison with Long-Tailed Recog-
Method
Backhone
ImageNat-LTiop-1)
nition Methods on CIFAR
Head
Mid
Tail
Overall
CIFAR-I0-ETE (toppp)))))
CIFAR-IIA
Softmax loss
55
ResNet-10
40,9
10.7
0.4
20.9
Method
Focal loss 5g
ReaNet-10
364
16.0
100
50
10
11K]
50
10
Range loss 53
ResNet-10
35.8
30.3
17.6
30.7
Softmax Loss 55
70.3
74.8
86.3
38.2
13.8
55.7
OI.TR
ResNet-10
43.2
35.1
18.5
35.6
FSA 44
ResNet-10
47.3
31.6
14.7
35.2
Loss 46
74.5
79.2
$7.4
39.6
45.3
57.9
cRT 48
ResNeXt-50
39.5
RBN 42
79.8
82.1
88.3
43.3
48.5
55.6
SiRe244ours
ResNet-18
39.4
32.7
16.1
37.0
EDC 10
68.8
72.1
76.3
41.8
46.0
SRc2l.tom
ResNet-18
51.8
34.4
16.1
38.7
EDC50+ours
74.0
77.0
82.3
43.8
47.9
58.1
EDC+ours
ReaNet-18
47.0
21.3
EDC-ours
ResNet-18
51.3
38.1
24.2
41.4
EDC+ours
ResNet-18
53.1
38.0
22.8
42.4
We visualize the distribution of the optimal calibration parameter т across different datasets in Figure Figure/Alett. All subfigures use the same feature extractor, which is the backbone of ResNet- model trained for 200 epochs on the balanced original dataset. We pass all images including both the original and the distilled images under different imbalance factors (IFs) through this feature extractor and jointly project them into oow-dimesiional space using t-SNE. This ensures that their spatial positions are directly comparable in the visualization. Figure demonstrates that the optimal calibration strength varies significantly with the dataset. Additionally. Figure right presents I-SNE visualization of the original and distilled image distributions. While the original datasets maintain a relatively uniform distribution in the feature space, regardless of the imbalance factor, the distribution of the distilled images becomes increasingly compressed and distorted as the imbalance factor increases. At lower imbalance factors, the distilled samples are more evenly dispersed across smaller
feature space. However, as the imbalance factor grows, these samples concentrate subregion of the space, deviating substantially from the distribution of the original dataset. This distortion may explain why a model trained on the distilled dataset, despite heing balanced, can still produce biased soft labels, as the underlying feature representation remains inherently skewed.
28


Reviews:
[{"review_id": "cntL9rYiwn", "reviewer": "Reviewer_bfAk", "summary": "The paper addresses the problem of distilling long-tailed dataset information. In particular, the authors derive a generalization error bound by adapting the formulation used in D3S [1] for the long-tailed setting under label shift. Based on that bound, it is argued that the distilled data can introduce bias through two sources: (i) the distilled images and (ii) their soft-labels, while their effect is evaluated via a perturbation analysis controlling different conditions. To account for these biases, post-hoc logit-adjustment (LA) [2] is used to correct soft-labels of the distilled dataset. Having observed a distribution shift between real and synthetic data distributions, the authors propose to minimize class-wise confidence variance to calibrate the strength factor of LA, leveraging distilled data. This adaptive soft label alignment leads to improvement in the overall and tail classification performance across different imbalance settings and SOTA data-distillation methods.\n\nExtra references:\n\n[1] Loo, Noel, et al. \"Large scale dataset distillation with domain shift.\" Forty-first International Conference on Machine Learning. 2024.\n\n[2] Menon, Aditya Krishna, et al. \"Long-tail learning via logit adjustment.\" arXiv preprint arXiv:2007.07314 (2020).", "questions": "- Q1: Can you include the intermediate steps when deriving the forms 1 and 2 when proving Th 3.1. to ensure correctness? \n\n- Q2: I can follow point (iii) in L175 in relation to the second term in Eq 5.but I do not follow why this has to hold for each class when looking at the second term of Eq 6. e.g., miss-labelled but identical samples could still minimize the second term in Eq 6.\n\n- Q3: I found the connection between the analysis in Sec 3.2. and the insights drawn from Th 3.1. to be unclear. \n   - Q3.1: Could you provide a connection between forms 1 and 2 with the three configs considered? e.g., clarify which term each configuration is trying to optimize. Also, including a config (4) with balanced/balanced would make for a more complete perturbation analysis.\n   - Q3.2: The generalization bound is derived on the full test loss, in that sense it would be interesting to see what happens to the overall performance (overall tail+head performance) \n   - Q3.3:  L242-244. This finding contradicts the theory developed which calls for a deeper analysis on this matter.\n   - Q3.4:  L212-214. Shouldn’t we draw that conclusion when comparing config 1 with config 3 instead where they only differ by the DM (2)?\n   - Q3.5: When solving Eq. 7. across the three configs, are you using the same g network and only ablate the f_{tr}? \n\nQ4: Fig3. (b) The proposed method does not display a decreasing performance trend as the IF increases in contrast to the baselines. Could you explain why this is the case?\n\nTypos/Minor:\n(I am not expecting an answer, please reflect upon these and refine if necessary)\n\n- T1. Define what the ADSA abbreviation means the first time you use it in the main text.\n- T2. L146. is \\theta_{syn}== \\theta_{dd}?\n- T3. Eq.3 the C is not defined.\n- T4. Eq.5 define the connection between the densities $p$ and the datasets. E.g. how is the p_tr related to D_tr?\n- T5. L150. did you mean training loss?\n- T6. Definition of Th 3.1. lacks rigor, what is C? Under what assumption Th 3.1. holds? The Th 3.1. as provided in [1] is a good reference point.\n- T7. Tab 1. SRe2L + ours (IP=1 and IF=50) should not be in bold.\n- T8. Fig 1. (b) please define the DM (1) and DM (2) abbreviations. Also it appears that (only for relabelling and generating images have been mixed).\n- T9. Fig3. Left panel, are these the correct titles? Right panel, what is the IF used for (a) and the IPC for (b)? \n- T10. L573.The first paragraph and second paragraph overlap significantly.\n- T11. L596. Please provide some more details on how these t-SNE plots were constructed. e.g., are you using the same feature extractor for all settings?\n- T12. L317. define what the soft-label budget means to make the paper self-contained.\n- T13. Eq.10 you could use a different index (e.g., j) on the inner summation to avoid confusion.\n\nFinal comments:\nAlthough the empirical results are convincing, the proposed method appears to be a trivial extension of logit-adjustment in the dataset condensation and is disconnected from the theory developed, limiting the depth of the study. Although the developed theory is interesting, the empirical results do not always support it, calling for further investigation. Concluding, I do not see how the theoretical insights connect with the proposed method, which nevertheless appears to perform well by using existing methodology (i.e., logit-adjustment).", "limitations": "yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n- Partial theoretical connection between the generalization bound and imbalance levels.\n- The experimental setting convincingly demonstrates that the proposed adaptive logit-adjustment applies to long-tailed dataset-distillation and boosts the overall performance across different baselines and imbalance settings.\n- The proposed method is simple and can be easily applied on top of methods originally developed for balanced dataset-distillation.\n\nWeaknesses:\n- W1 - The connection between the forms in Eqs. 5 and 6 and the perturbation analysis is unclear.\n- W2 - The developed theory attributes tail performance collapse to two different sources, yet the proposed method is disconnected from this finding.\n- W3 - Some details are missing, hindering the readability of the paper.", "was_revised": true, "final_justification": "Despite my initial concerns, the authors have made a substantial effort in their rebuttal, providing complementary theoretical derivations and additional experiments. While I still have some minor doubts regarding the connections between theory-perturbation analysis-method, I am now persuaded of the following:\n\n* The theory is sound in both of its alternative formulations.\n* Although the perturbation analysis remains somewhat tangential to the main theory, it effectively disentangles bias sources in practical terms (i.e., distinguishing how the long-tailed bias affects the generative and discriminative components).\n* The method—while not entirely novel—flexibly extends the original logit-adjusted approach and, for the first time, applies it to dataset distillation; it performs well under both types of bias, as demonstrated by the new experiments.\n* It can be used as an add-on to previous baselines that focus primarily on feature alignment, with consistent improvements.\n* The authors have established a partial theoretical connection between the method and the underlying theory, and have empirically demonstrated that refining the pseudo-labels corrects both the marginal and posterior terms.\n\nOn balance, these strengths outweigh my remaining reservations, and I am therefore raising my score.", "rebuttal": "We sincerely thank the reviewer for the thoughtful and detailed feedback. We have carefully addressed all the comments by providing additional explanations and clarification.\n\n**W1: The connection between the forms in Eqs. 5 and 6 and the perturbation analysis is unclear.**\n\n**A**: The perturbation analysis in Sec. 3.2 is deeper analysis of the first item in Eqs.6 in Sec. 3.1.\n\nSec. 3.1 (Eqs.6 and 6) highlights the need to match both soft label and image distributions. To further investigate soft label bias, Sec. 3.2 conducts a perturbation analysis showing that biases from both the distillation model and distilled images directly cause soft label mismatches. Importantly, these biases affect the soft labels directly and are not simply a consequence of image misalignment as described in Theorem 3.1 (see implication (iii), L174–L177). Then Sec. 3.3 introduces the ADSA module to jointly mitigate both types of soft label bias found in Sec. 3.2 in a unified manner.\n\nWe will revise L166–L178 and the introduction to better highlight our focus on soft label alignment.\n\n**W2: The proposed method is disconnected from two different bias sources.**\n\n**A:** Our proposed method could mitigate the two bias sources (distilled images and distillation model) in soft label, which correspond to the soft label part in Th. 3.1.\n\nWe have added two new experiments that explicitly evaluate our method's effectiveness against each individual bias.\n\nExp1: Addressing bias from the distillation model (biased soft labels)\n\n|IF| biased img | biased label | method | IPC=1 | IPC=10 | IPC=50 |\n|-|-|-|-| - | - | - |\n| 10|N|Y|SRe2L| 16.6| 34.3| 50.6|\n||N| Y|+ours| **19.0** | **41.4** | **62.6** |\n| 100|N| Y|SRe2L| 18.3| 24.4| 31.5|\n|| N| Y|+ours| **21.8** | **36.9** | **57.6** |\n\nExp2: Addressing bias in distilled images\n\n| IF| biased img | biased label | method | IPC=1 | IPC=10 | IPC=50 |\n| - | - | - | - | - | - | - |\n| 10| Y | N| SRe2L| 16.0 | 34.7 | 55.3|\n|| Y  | Y|+ours| **23.7** | **40.2**  | **63.1** |\n| 100 | Y | N| SRe2L|15.6 | 22.9 | 54.8 |\n||Y|Y|+ours| **21.4** | **33.2** | **58.4** |\n\n>  Note: \n>\n> 1. \"Y\" indicates the presence of bias; \"N\" indicates no bias. \"IF\" means Imbalance Factor.\n> 2. Biased img/label refers to one generated by a model trained on a long-tailed dataset.\n\nResults in Exp2 indicate that our method still outperforms the baseline even both the distilled images and soft labels are biased . This highlights the effectiveness of our method in mitigating the impact of image-induced bias. \n\nWe do not adopt the baseline setting (biased images with unbiased labels) in Exp2 because our method relies on class-frequency information from a long-tailed distribution, which aligns with practical scenarios. However, in Exp2, this information is inconsistent with the model providing unbiased labels.\n\nA key contribution of our method is its ability to implicitly rectify both sources of bias on soft labels including distilled image bias and soft label bias in a unified manner. \n\n**W3: Some details are missing, hindering the readability of the paper.**\n\nA complete definition of all symbols and abbreviations will be provided, and the experimental setups from Sec. 3.2 and Sec. 4 will be further detailed in the appendix.\n\n**Q1: Can you include the intermediate steps when deriving the forms 1 and 2?**\n\n**A:** We will include a more detailed derivation for Forms (1) and (2) in Appendix A.1 of the revised version. Below we briefly outline the derivation of Form (1) for clarity.\n\nWe first replace $p_{tr}$ with $p_{te}$ in the generalization bound from D3S [1]. Then, under the assumption of conditional feature consistency, $p_{tr}(x|y)=p_{te}(x|y)$, the test distribution can be rewritten as:\n$$\np_{te}(x,y)=\\frac{p_{te}(y)}{p_{tr}(y)}p_{tr}(x,y).\n$$\nSubstituting this expression into the bound from D3S yields:\n$$\nl_{te} \\leq l_{dd}+\\frac{C}{2 \\sqrt{2}} \\sqrt{D_{K L}\\left(\\frac{p_{te}(y)}{p_{tr}(y)}p_{tr}(x, y) \\| p_{dd}(x, y)\\right)}\n$$\nExpanding the KL divergence term:\n$$\n\\begin{aligned}\n& D_{K L}\\left(\\frac{p_{te}(y)}{p_{tr}(y)}p_{tr}(x, y) \\| p_{dd}(x, y)\\right)\\\\\\\\\n=& \\int\\frac{p_{te}(y)}{p_{tr}(y)}p_{tr}(x, y)\\log \\frac{p_{te}(y)p_{tr}(x,y)}{p_{tr}(y)p_{dd}(x,y)} dxdy\\\\\\\\\n=& \\int\\frac{p_{te}(y)}{p_{tr}(y)}p_{tr}(x, y)\\log \\frac{p_{tr}(x|y)}{p_{dd}(x|y)} dxdy + \\int\\frac{p_{te}(y)}{p_{tr}(y)}p_{tr}(x, y)\\log \\frac{p_{te}(y)p_{tr}(y)}{p_{tr}(y)p_{dd}(y)} dxdy\\\\\\\\\n=& \\int p_{te}(y)p_{tr}(x|y)\\log \\frac{p_{tr}(x|y)}{p_{dd}(x|y)} dxdy+\\int p_{te}(y)\\log \\frac{p_{te}(y)p_{tr}(y)}{p_{tr}(y)p_{dd}(y)} d\\\\\\\\\n=& \\int p_{te}(y)D_{KL} \\left(p_{tr}(x|y)\\| p_{dd}(x|y) \\right) dy+D_{KL}(p_{te}(y)\\| p_{dd}(y))\n\\end{aligned}\n$$\nFinally, we convert the integral form into an empirical summation form for practical implementation. This gives the decomposition into Form (1). Form (2) could be derived by applying the alternative decomposition $p(x,y)=p(y|x)p(x)$.\n\n**Q2: Identical samples could still minimize the second term in Eq 6.**\n\n**A:** Although identical samples can help reduce the second term, mismatched labels can lead to a significant increase in the first term. \n\n**Q3: The connection between the analysis in Sec. 3.2. and the insights drawn from Th 3.1. to be unclear.**\n\n**A:** We believe Q3 relates closely to W1, as both concern the connection between Theorem 3.1 and the perturbation analysis in Section 3.2. As noted in W1, Theorem 3.1 provides the theoretical foundation for both distilled images and soft label, while Section 3.2 offers empirical insights into soft label mismatch, while mainstream work focuses on distilled images.\n\n**Q3.1: Connection between configs in 3.2 and two forms in 3.1. Add config (4).**\n\n**A:** The three configss in Sec. 3.2 are designed to investigate the bias sources in the soft label distribution rather than distilled image distribution, and thus directly relate to the soft label part (first item) in form (2): $D_{KL}(p_{te}(y|x)\\Vert p_{dd}(y|x))$.\n\nIn fact, config 4 corresponds to the rightmost endpoint of all curves in configs 1-3, where the number of tail samples reaches 100. We will explicitly include config 4 in the revised version.\n\n**Q3.2：It would be interesting to see what happens to the overall performance (overall tail+head performance) given the derived generalization bound.**\n\n**A:** All experiments in Sec. 4 report the overall performance across all classes. Additionally, we provide a breakdown of performance across head, mid, and tail classes on ImageNet in Table 2, and a more fine-grained class-wise analysis on CIFAR-100 in Figure 3. These results show that our method improves both overall and tail-class performance.\n\n**Q3.3: L242-244. This finding contradicts the theory developed.**\n\n**A:** The finding in L242–244 provides a deeper investigation into the soft-label component of Th. 3.1. We identify two sources of soft-label mismatch - originating from both the distillation model and the distilled images.\n\nMost existing work in dataset distillation has primarily focused on improving image matching insteal of soft label. And different performance of baseline methods in Sec.4.1 also supports the image-matching component of Th. 3.1. However, this paper mainly focuses on the less-explored role of soft labels in dataset distillation.\n\n**Q3.4:  L212-214. We should draw that conclusion by comparing config 1 with config 3.**\n\n**A:** We will revise the statement to ensure greater clarity since config 1 shows better performance with biased images.\n\n**Q3.5: When solving Eq. 7. across the three configs, are you using the same $g$ network and only ablate the $f_{tr}$?**\n\n**A:** We do **not** use a shared or fixed $g$ across different configurations, and both $f_{tr}$ and its $g$ component are re-initialized and trained independently for each configuration. The $g$ network is introduced to describe the feature distribution matching in L192 more clearly. \n\n**Q4: Fig3. (b) The proposed method does not display a decreasing performance trend as the IF increases.**\n\n**A:** The issue stems from our default setting (Epoch=200) for SRe2L leading to suboptimal performance. We compared distilled data using models trained with different epochs on CIFAR10.\n\nExp: Effect of Model Epoch on Performance with IF=10/100\n\n| Epoch | IF| Method | IPC=10 | IPC=50|\n| - | - | - | - |-|\n| 10 |10| SRe2L  | 39.6 | 55.0 |\n| | | +ours| 41.9 | 63.2 |\n| 30  | 10 | SRe2L  | 27.2  | 51.5 |\n|  | | +ours  | 26.2   | 50.0 |\n| 50 | 10 | SRe2L  | 24.6 | 46.8 |\n| |  | +ours  | 25.2 | 53.6|\n\n| Epoch | IF   | Method | IPC=10   | IPC=50   |\n| - | - | - | - | - |\n| 10    | 100  | SRe2L  | 21.6 | 23.6     |\n| | | +ours  | 33.0 | 37.9 |\n| 30    | 100  | SRe2L  | 28.1 | 38.2 |\n|  || +ours  | 32.0 | 59.1 |\n| 50 | 100  | SRe2L  | 23.9 | 40.3 |\n| | | +ours  | 29.1  | 53.8 |\n\nWe observe that distilled performance first increases then drops with increasing distillation model epochs. The optimal epoch becomes larger as IF or IPC increases. This is because early-stage models yield richer soft labels[1], while [2–3] highlight the importance of proper epoch selection in trajectory matching.\n\nIn the original setting, we use fixed epochs (200) for the model. As the IF increases, the optimal epoch shifts later, which explains the seemingly anomalous improvement in performance with higher imbalance.\n\nWe revised our experimental setup by selecting the best-performing model from $\\text{Epoch}\\in\\set{10,30,50}$.\n\n| tau\\IF|10|50|100|500|1000|\n|-|-|-|-|-|-|\n|0| 32.6|28.2|29.2|23.9|21.9|\n|1|39.7|33.5|34.0|30.1|25.3|\n|1.5|40.6|33.3|34.4|28.8|28.2|\n|ours|**41.6**|**42.7**|**35.8**|**32.5**|**29.3**|\n\nWe can observed that the previous anomalous performance trend disappears.\n\n**Typos：**\n\n**A:** We sincerely thank the reviewer for the careful feedback. We have corrected all identified issues in our internal revision.\n\nRefs:\n\n[1] Large Scale Dataset Distillation with Domain Shift, ICML, 2024"}, {"review_id": "1RahvmZYsX", "reviewer": "Reviewer_GTsB", "summary": "In this paper, authors study dataset distillation problem in the setting where the dataset follows a long-tailed distribution. First, they conduct a theoretical and empirical analysis, showing that the bias comes from both distilled images as well the distillation model. Based on this observation, the authors then propose adaptive soft label alignment module. In this proposed method, they adopt a logit calibration method. They show that the logit calibration method can be applied to existing dataset distillation methods in a post-hoc manner. Their experiment results show that their method effectively mitigate the poor performance issued caused by long tailed datasets.", "questions": "1. I am a bit confused about the experiment setup. My current understanding is that they authors frame their method as a simple add on of existing dataset distillation pipeline. And in their main experiments (Section 4.1 Table 1, 2), they show existing methods performance (such as SRe2L, EDC) and their add on. Just to clarify, authors trained all the experts, distilled all the images from the long-tailed distribution, the \"ours\" part is to apply Eq. 9 and 10 to soft labels. In other words, during the entire dataset distillation pipeline, there is no access to balanced datasets. \n\n\n2. I am also quite confused about Figure 1(a), why does original dataset equal imbalanced dataset plus balanced dataset? Overall, I think this paper could benefit from more clear writing about the set up.", "limitations": "Yes, authors have addressed the limitations and potential negative societal impact.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n   1. Well-motivated problem:\n\nThe paper tackles a practically important issue—real-world datasets often exhibit class imbalances rather than a clean uniform distribution. Addressing dataset distillation specifically in long-tailed settings is clearly relevant and should interest the broader research community.\n\n   2. Insightful empirical analysis:\n\nThe paper provides a careful and insightful empirical analysis, clearly illustrating when and why existing dataset distillation methods fail under long-tailed distributions. The use of three distinct configurations to identify and isolate the sources of bias is particularly effective and enhances understanding of the underlying problem.\n\n**Weaknesses**\n\n   1. Incomplete handling of identified biases:\n\nMy understanding is that the central methodological contribution is the introduction of Eq. 9, which essentially applies a frequency-based penalization to mitigate bias in soft labels. While this approach logically addresses bias introduced by class imbalance in the training set, the authors neglect the bias arising specifically from the distilled images, even though it was explicitly highlighted as a key issue in the empirical analysis (Section 3.2). The paper would benefit from either a justification of why the image-induced bias is less critical or a discussion of possible solutions for addressing this image-specific bias.\n\n   2. Lack of essential baseline:\n\nTraining directly on class-imbalanced datasets is known to lead to suboptimal performance even beyond the dataset distillation setting. An intuitive baseline would be to apply well-established data balancing techniques (such as oversampling or weighted sampling) to the original long-tailed dataset prior to distillation. Such a baseline would clarify whether simple and well-known rebalancing methods could already sufficiently mitigate the issue without requiring specialized adjustments in dataset distillation methods.", "was_revised": true, "final_justification": "During rebuttal, authors have addressed my concerns regarding baselines. They also clarified a mis-understanding I have regarding the methodology and after the clarification, I think their method makes sense.", "rebuttal": "We sincerely thank the reviewer for the insightful suggestions on both the content and experimental aspects of our paper. In response, we have conducted additional experiments and carefully addressed each point in the replies below.\n\n**W1: Incomplete handling of identified biases. Central methodological contribution only mitigate the bias introduced by labels from distillation model, neglecting the bias from the distilled images.**\n\n**A:** Our proposed method could mitigate the two bias sources (distilled images and distillation model) in soft label. A key contribution of our method is its ability to implicitly rectify both sources of bias on soft labels including distilled image bias and soft-label bias in a unified manner. We conducted two additional experiments to prove that our method could mitigate both the bias succesfully.\n\nExp1: Addressing bias from the distillation model (biased soft labels)\n\nWe use soft labels produced by a model trained on an long-tailed dataset, while distilled images are obtained from a model trained on a balanced dataset.\n\n| IF   | biased img | biased label | method | IPC=1    | IPC=10   | IPC=50   |\n| ---- | ---------- | ------------ | ------ | -------- | -------- | -------- |\n| 10   | N          | Y            | SRe2L  | 16.6     | 34.3     | 50.6     |\n|      | N          | Y            | +ours  | **19.0** | **41.4** | **62.6** |\n| 100  | N          | Y            | SRe2L  | 18.3     | 24.4     | 31.5     |\n|      | N          | Y            | +ours  | **21.8** | **36.9** | **57.6** |\n\n> Note: \"Y\" indicates the presence of bias; \"N\" indicates no bias in that source. \"IF\" means Imbalance Factor.\n\nThese results clearly show that our method can effectively mitigate soft-label bias introduced by a biased distillation model.\n\nExp2: Addressing bias in distilled images\n\nThis experiment evaluates the bias introduced by distilled images. The baseline method SRe2L uses images from a long-tailed model and soft labels from a balanced one, while our method is applied when both images and labels are from long-tailed models.\n\n| IF   | biased img | biased label | method | IPC=1    | IPC=10   | IPC=50   |\n| ---- | ---------- | ------------ | ------ | -------- | -------- | -------- |\n| 10   | Y          | N            | SRe2L  | 16.0     | 34.7     | 55.3     |\n|      | Y          | Y            | +ours  | **23.7** | **40.2** | **63.1** |\n| 100  | Y          | N            | SRe2L  | 15.6     | 22.9     | 54.8     |\n|      | Y          | Y            | +ours  | **21.4** | **33.2** | **58.4** |\n\nThese results indicate that even when both the distilled images and soft labels are biased (i.e., derived from long-tailed models), our method still outperforms a baseline that uses soft labels from unbiased distillation model. This highlights the effectiveness of our method in mitigating the impact of image-induced bias.\n\nOurs method do not follow the baseline setting (biased images with unbiased labels) because our method requires class-frequency information, determined by the imbalance factor (IF), from the distillation model's training distribution. Therefore, it is incompatible to apply our method on a model trained with balanced dataset. This design also reflects realistic scenarios, where both the distillation model and synthetic images originate from the same long-tailed model.\n\nWe will revise the main text to better highlight this point and include this experiment in Sec. 4.2 of the revised version.\n\n**W2: Lack of essential baseline. An intuitive baseline would be to apply well-established data balancing techniques (such as oversampling or weighted sampling) to the original long-tailed dataset prior to distillation.**\n\n**A**:  We have conducted the additional experiments about enhance baseline and will include the results in the revised version of the paper, along with releasing the corresponding baseline code.\n\nExperiment Setup: We applied a resampling technique (random oversampling) to the long-tailed dataset prior to training the distillation model. The rest of the distillation pipeline remains unchanged. Below are the results for CIFAR-10 and CIFAR-100 under various imbalance factors (IF), image per class (IPC), and whether our method is used.\n\nExp1: Experiment on CIFAR-10, we use SRe2L as our baseline method.\n\n| IF      | method          | IPC=1    | IPC=10   | IPC=50   |\n| ------- | :--------------- | -------- | -------- | -------- |\n| **10**  | SRe2L           | 19.9     | 39.0     | 55.2     |\n|         | +resample | 18.1     | 38.1     | 54.8     |\n|         | +ours         | **23.7** | **40.2** | **63.1** |\n|         | +resample+ours | 19.5     | 38.0     | 58.8     |\n| **50**  | SRe2L           | 19.9     | 27.2     | 51.1     |\n|  | +resample     | 21.7     | 41.3     | 54.6     |\n|         | +ours | **26.9** | 35.4     | **63.7** |\n|         | +resample+ours | 22.3     | **44.1** | 56.6     |\n| **100** | SRe2L   | 18.7     | 21.3     | 24.0     |\n|         | +resample     | **25.5** | 36.4     | 49.5     |\n|  | +ours | 21.4     | 33.2     | **58.4** |\n|         | +resample+ours | 24.1     | **37.1** | 56.1     |\n\nExp2: Experiment on CIFAR-100, we use SRe2L as our baseline method.\n\n| IF   | method        | IPC=1    | IPC=10   | IPC=50   |\n| ---- | :------------- | -------- | -------- | -------- |\n| 10   | SRe2L  | 16.3     | 35.8     | 45.2     |\n|      | +resample | **20.3** | 35.8     | 45.1     |\n|      | +ours | 16.9     | **38.2** | **53.9** |\n|      | +resample+ours | 18.5     | 37.6     | 50.6     |\n| 50   | SRe2L  | 17.8     | 28.8     | 33.5     |\n|      | +resample     | **18.9** | 29.2     | 36.1     |\n|      | +ours | 18.2     | **34.2** | **45.6** |\n|      | +resample+ours | 17.6     | 32.2     | 39.6     |\n| 100  | SRe2L  | 15.9     | 29.1     | 31.8     |\n|      | +resample     | **16.3** | 28.4     | 32.8     |\n|      | +ours | 9.6      | **33.0** | **41.7** |\n|      | +resample+ours | 15.1     | 29.9     | 34.3     |\n\n>  Note: The original paper used SRe2L with its default distillation model training epoch (EP = 200). During the rebuttal process, we found this choice does not always yield optimal performance. Therefore, for fairness, we re-evaluated SRe2L with resampling or our methods, using the best-performing distillation model, selected from $\\text{EP}\\in\\set{10,30,50}$ for CIFAR-10, and $\\text{EP}\\in\\set{10,50,100,200}$ for CIFAR-100. We will update the reported SRe2L results in the main paper and provide the full version in the appendix. If needed, we are happy to share the full results during the discussion phase.\n>\n\nThe experimental results show that our method outperforms both the baseline and the resample setting in most cases, while resampling indeed improves over the baseline. The results show that our method could be seamlessly integrated as a plug-and-play module to further boost the performance of resampling-based pipelines. This highlights the generality and robustness of our approach across various settings.\n\nThe relatively lower performance under IPC = 1 may be attributed to the limited number of samples, which makes it difficult to reliably estimate the class-level soft-label imbalance. This in turn affects the accuracy of our calibration.\n\n**Q1: Confusion about the experiment setup. The \"ours\" part is to apply Eq. 9 and 10 to soft labels. In other words, during the entire dataset distillation pipeline, there is no access to balanced datasets.**\n\n**A**: Your understanding is correct. Yes, there is no access to any balanced dataset during the entire distillation pipeline. All baselines, including our add-on method, are trained only on the long-tailed dataset and evaluated on a balanced test set.\n\nIn Sec. 3.2, we use a balanced dataset only for analysis purposes, to isolate and identify the two sources of soft-label bias. However, in Sec. 3.3 and all experimental evaluations, the method is trained strictly on long-tailed data, aligning with realistic settings.\n\n**Q2: Clarification on Figure 1(a), why does original dataset equal imbalanced dataset plus balanced dataset?**\n\n**A:** Thank you for your thoughtful observation. The original dataset is *not* the sum of the imbalanced and balanced datasets. In Figure 1(a), our intention is to illustrate that the original dataset in dataset distillation can be either *imbalanced* or *balanced*. While many prior works in dataset distillation assume a balanced original dataset, real-world data often follows a long-tailed (imbalanced) distribution."}, {"review_id": "xtIehNJLsu", "reviewer": "Reviewer_Prxx", "summary": "This paper proposes a lightweight module, ADSA, to calibrate soft-label biases in long-tailed dataset distillation, which stem from both the teacher model and the synthetic images. By deriving an imbalance-aware generalization bound, the authors reveal the underlying causes of performance degradation. Extensive experiments demonstrate that ADSA consistently improves performance across various distillation methods and label budgets, achieving up to an 11.8% improvement in tail-class accuracy on ImageNet-1k-LT.", "questions": "1.Equation (3): The constant C is undefined. Could the authors clarify its meaning and boundary conditions in this context?\n\n2.Equation (15): It assumes ptr​ (x∣y)=pte(x∣y), a common setting in long-tailed learning. Does this assumption still hold in dataset distillation scenarios?\n\n3. Are the reported results from a single run? The absence of variance or confidence intervals makes it difficult to assess stability. It is recommended to report results over multiple random seeds.\n\nIf the authors are able to address the above issues, I would be willing to raise my score.", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n\nThe proposed lightweight module effectively mitigates these biases and delivers consistent performance improvements across various settings. Experimental results show that the method outperforms existing strong baselines.\n\nWeaknesses\n\n1.In Equations (1)–(2), the distilled dataset is denoted as 𝐷syn​ , but in Equations (4)–(6), it suddenly switches to 𝐷dd  (e.g., Rdd). The notation should be unified or clearly explained.\n\n2.LAD is presented as a comparative long-tailed distillation method, but its results are missing under some IPC/IF settings (e.g., IPC=1 is absent in Table 1). If feasible, these should be included for completeness.\n\n3.The experiments only report the hardware used, but lack details such as per-experiment runtime and memory usage (e.g., GPU hours required to distill ImageNet-LT with IPC=50), which limits reproducibility and practical cost assessment.", "was_revised": true, "final_justification": "The author has addressed most of my questions, including the setting of variance, etc.", "rebuttal": "We sincerely thank the reviewer for the careful and detailed comments on our paper, including the experimental design and completeness. In response, we have conducted extensive additional experiments and provided detailed clarifications and revisions as outlined below.\n\n**W1: The notation about $D_{dd}$ or $D_{syn}$ should be unified or clearly explained.**\n\n**A:** Thank you for pointing out the issue with the notation. We will revise the manuscript to unify the notation by consistently using the subscript $dd$, such as $D_{dd}$ and $\\theta_{dd}$, throughout the paper for clarity and consistency.\n\n**W2: Completion of LAD[1].**\n\n**A:** We have reproduced the results of LAD and reported the corresponding variances, as summarized in the table below:\n\nReproduction of LAD under Varying IPC and Imbalance Factors on CIFAR-10\n\n| IPC\\IF | 10       | 50       | 100      |\n| ------ | -------- | -------- | -------- |\n| 1      | 28.0±1.0 | 24.4±0.3 | 23.8±0.5 |\n| 10     | 58.1±0.3 | 54.2±1.0 | 53.4±0.1 |\n| 50     | 70.5±0.4 | 65.8±0.2 | 64.0±0.9 |\n\nReproduction of LAD under Varying IPC and Imbalance Factors on CIFAR-100\n\n| IPC\\IF | 10       | 50       | 100      |\n| ------ | -------- | -------- | -------- |\n| 1      | 12.3±0.2 | 11.1±0.1 | 10.6±0.1 |\n| 10     | 31.5±0.2 | 26.8±0.3 | 24.9±0.1 |\n| 50     | 40.0±0.1 | 34.5±0.1 | 31.6±0.0 |\n\n**W3: Details about per-experiment runtime and memory usage.**\n\n**A:**  We conducted our experiments on a server equipped with 8×RTX 4090 GPUs. Due to the relatively small scale of CIFAR10 and CIFAR100, the GPU utilization may not be fully saturated. As a result, their memory usage is approximately similar, and CIFAR10 runs slightly faster than CIFAR100. The approximate computational cost and GPU memory footprint for both CIFAR10/CIFAR100 are summarized below:\n\n|            | SRe2L | GVBSM | EDC  |\n| --------------- | ----- | ----- | ---- |\n| GPU Memory (GB) | 2     | 2     | 2    |\n| Time (GPU·h)    | 2     | 10    | 5    |\n\nThe conservative runtime estimates on ImageNet are as follows:\n\n|                 | SRe2L     | GVBSM      | EDC        |\n| --------------- | --------- | ---------- | ---------- |\n| GPU Memory (GB) | 10        | 20         | 20         |\n| Time (GPU·h)    | <40 (+16) | <360 (+80) | <180 (+80) |\n\nThe values in parentheses represent the extra GPU hours needed to re-train the distillation models on long-tailed datasets, since the public checkpoints from Torch are pre-trained on balanced ImageNet and thus unsuitable for a fair evaluation. We adopt the default batch sizes used in SRe2L, EDC, and GVBSM. The backbone architectures have been detailed in Table 5 of Appendix A.2. Moreover, we observe that during the soft-label generation phase, the GPU utilization of SRe2L, GVBSM, and EDC tends to be relatively low, indicating potential for further acceleration.\n\n**Q1: Clarify the meaning and boundary conditions about the constant $C$.**\n\n**A:** The constant $C$ represents an upper bound on the negative log-likelihood of the model $\\hat{p}(y|x)$, which is typically finite in practical training scenarios.\n\nLet $\\hat{p}(y|x)$ denote the predictive distribution of any classifier, typically referring to the output of a trained network on the distribution $p_{dd}(x, y)$. If the negative log-likelihood $-\\log \\hat{p}(y|x)$ is upper-bounded by a positive constant $C$, then we have the following inequality (as shown in Eq. 3) according to D3S[2]:\n$$\nl_{tr} \\le l_{dd} + \\frac{C}{2\\sqrt{2}} \\sqrt{R_{dd}}, \\\\\\\\\nR_{dd} = D_{KL}(p_{tr}(x)\\|p_{dd}(x)) + D_{KL}(p_{tr}(y | x)\\|p_{dd}(y | x)).\n$$\n\nHere, $l_{tr} = E_{p_{tr}}[-\\log \\hat{p}(y|x)]$ and $l_{dd} = E_{p_{dd}}[-\\log \\hat{p}(y|x)]$. The distributions $p_{tr}$ and $p_{dd}$ correspond to the data sampling distributions from the original training set $D_{tr}$ and the distilled dataset $D_{dd}$, respectively.\n\n**Q2: Equation (15): It assumes $p_{tr}(x|y) = p_{te}(x|y)$, a common setting in long-tailed learning. Does this assumption still hold in dataset distillation scenarios?**\n\n**A:** Yes, this assumption still holds. In long-tailed learning, the training set is a long-tailed dataset $D_{tr}$, and the test set is a balanced dataset $D_{te}$, satisfying $p_{tr}(x|y) = p_{te}(x|y)$.  \n\nIn long-tailed dataset distillation, the original training set is also the long-tailed dataset $D_{tr}$, from which a distilled dataset $D_{dd}$ is derived. The model is trained on $D_{dd}$ and evaluated on the same balanced test set $D_{te}$.  \n\nSince the original training and test sets used in our distillation setting are identical to those in long-tailed learning, the assumption $p_{tr}(x|y) = p_{te}(x|y)$ remains valid in the dataset distillation scenario.\n\n**Q3: The absence of variance or confidence intervals makes it difficult to assess stability.** \n\n**A:** We conducted three runs of experiments on CIFAR10 and CIFAR100, and report the mean and standard deviation in the tables below.\n\nExperiment results under Varying IPC and Imbalance Factors on CIFAR-10\n\n| IPC        | 1            | 1            | 1            | 10           | 10           | 10           | 50           | 50           | 50           |\n| ---------- | ------------ | ------------ | ------------ | ------------ | ------------ | ------------ | ------------ | ------------ | ------------ |\n| methods\\IF | 10           | 50           | 100          | 10           | 50           | 100          | 10           | 50           | 100          |\n| LAD        | 28.0±1.0     | 24.4±0.3     | 23.8±0.5     | 58.1±0.3     | 54.2±1.0     | 53.4±0.1     | 70.5±0.4     | 65.8±0.2     | 64.0±0.9     |\n| SRe2L      | 18.0±0.5     | 20.1±0.6     | 17.2±0.6     | 39.1±0.7     | 27.0±0.5     | 28.1±0.1     | 55.1±0.2     | 51.4±1.0     | 40.1±0.4     |\n| SRe2L+ours | **22.1±1.6** | **26.6±1.0** | **23.8±0.6** | **41.5±0.6** | **35.5±0.4** | **33.0±0.1** | **63.3±0.1** | **63.8±0.7** | **58.8±0.5** |\n| GVBSM      | 34.2±0.4     | 28.9±0.4     | 25.1±0.3     | 49.5±0.1     | 33.8±0.3     | 29.4±0.1     | 58.2±0.2     | 37.2±0.0     | 30.9±0.0     |\n| GVBSM+ours | **39.1±0.4** | **36.0±0.6** | **31.5±0.5** | **54.4±0.5** | **45.8±0.1** | **40.4±0.4** | **64.7±0.0** | **51.4±0.2** | **46.9±0.2** |\n| EDC        | 32.3±1.4     | 29.5±0.8     | 30.0±0.6     | 69.9±0.7     | 58.5±0.5     | 50.9±0.5     | 77.8±0.1     | 65.6±0.3     | 56.0±0.2     |\n| EDC+ours   | **35.2±0.8** | **36.4±1.0** | **39.3±0.6** | **73.2±0.4** | **69.8±0.2** | **68.7±0.3** | **80.8±0.2** | **76.4±0.4** | **74.8±0.1** |\n\nExperiment results under Varying IPC and Imbalance Factors on CIFAR-100\n\n| IPC        | 1            | 1            | 1            | 10           | 10           | 10           | 50           | 50           | 50           |\n| ---------- | ------------ | ------------ | ------------ | ------------ | ------------ | ------------ | ------------ | ------------ | ------------ |\n| methods\\IF | 10           | 50           | 100          | 10           | 50           | 100          | 10           | 50           | 100          |\n| LAD        | 12.3±0.2     | 11.1±0.1     | 10.6±0.1     | 31.5±0.2     | 26.8±0.3     | 24.9±0.1     | 40.0±0.1     | 34.5±0.1     | 31.6±0.0     |\n| SRe2L      | 17.3±0.9     | 17.5±0.2     | 15.7±0.3     | 35.4±0.2     | 30.1±0.1     | 15.7±0.3     | 46.1±1.1     | 35.0±0.2     | 31.8±0.7     |\n| SRe2L+ours | 17.2±0.3     | **18.1±0.1** | 9.3±0.2      | **38.2±0.0** | **34.2±0.1** | **33.0±0.0** | **53.7±0.1** | **45.3±0.2** | **41.7±0.1** |\n| GVBSM      | 20.3±0.3     | 19.7±0.1     | 19.1±0.2     | 35.1±0.0     | 31.2±0.2     | 27.9±0.1     | 39.2±0.3     | 34.0±0.1     | 31.1±0.1     |\n| GVBSM+ours | 20.0±0.3     | **20.4±0.3** | 17.0±0.6     | **35.5±0.1** | **32.8±0.1** | **29.6±0.3** | **40.1±0.0** | **35.8±0.3** | **33.2±0.4** |\n| EDC        | 42.3±0.0     | 34.3±0.3     | 32.0±0.1     | 54.3±0.6     | 43.4±1.6     | 39.0±1.8     | 57.0±0.8     | 45.7±1.6     | 40.9±1.8     |\n| EDC+ours   | **43.4±0.4** | **37.9±0.5** | **34.1±0.3** | **55.5±0.1** | **46.0±0.1** | **41.6±0.1** | **58.0±0.2** | **48.1±0.1** | **43.5±0.2** |\n\nWe corrected the number of training epochs used for the distillation model in SRe2L, which leads to slightly better performance compared to the original paper. During the rebuttal process, we found that using an early-stage distillation model can significantly improve the quality of the distilled dataset. We will update the revised version with the corrected results accordingly. Nevertheless, the original results remain meaningful, as our method is an add-on approach that focuses on relative improvements.\n\nRefs:\n\n[1] Distilling long-tailed datasets, CVPR, 2025\n\n[2] Large Scale Dataset Distillation with Domain Shift, ICML, 2024"}, {"review_id": "7cyJXvWuT8", "reviewer": "Reviewer_cicZ", "summary": "This paper addresses the important problem of distilling or compressing real-world long-tail data, which is highly relevant in many practical scenarios. A particularly notable contribution is the derivation of an imbalance-aware generalization bound for models trained on distilled datasets, providing valuable theoretical insight. Additionally, the proposed ADSA module is a key strength of the work, offering seamless integration into existing distillation pipelines, which enhances its practical applicability.", "questions": "**Major**\n- ADSA performs soft label calibration to address the bias introduced by the distillation model. However, it is unclear whether this also mitigates the bias inherent in the distilled images themselves. I believe the paper would benefit from an explicit explanation of how ADSA handles this aspect.\n- In Figure 3(b), why is there a sudden performance improvement at the end, even as the imbalance factor increases? What explains this behavior?\n- In scenarios with small IPC, the performance of ADSA sometimes drops. Do the authors have any insight or explanation for this?\n- In Figure 1(b), the balanced datasets trained model is stated to be used only for relabeling, yet it appears to be generating images. Similarly, the model used only for generating images also seems to produce labels. Could the authors clarify whether this is a typo or an inconsistency in the figure?\n\n**Minor**\n- When applied to a balanced training dataset, does ADSA lead to any performance degradation?\n- It seems that additional computation is required for ADSA. As the number of data points and classes increases, does the distillation speed or memory usage become a bottleneck?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 4, "originality": 4, "strengths_and_weaknesses": "**Strengths**\n- While I am not an expert in dataset distillation, the paper clearly communicates its core ideas and the importance of the problem it addresses. It is a well-written and accessible paper even for readers less familiar with the domain.\n- The experimental evaluation is thorough, and the inclusion of diverse visualization results is particularly impressive.\n- The method demonstrates consistent performance improvements across a wide range of baselines, which highlights its robustness.\n- The proposed hyperparameter-free approach is a significant advantage, as it enhances the method’s practicality and ease of adoption in real-world scenarios.", "was_revised": true, "final_justification": "After carefully revisiting both the authors’ rebuttal and the main paper, I acknowledge that the authors have addressed most of my concerns—particularly those related to bias in distilled images and certain empirical inconsistencies in the main paper. I also appreciate the clarification regarding the applicability to balanced datasets and the potential computational or memory bottlenecks as the number of data points or classes increases.", "rebuttal": "We sincerely thank the reviewer for the positive evaluation and insightful suggestions. We have carefully addressed all comments through additional experiments and clarification below.\n\n**Q1: It is unclear whether ADSA module also mitigates the bias inherent in the distilled images themselves.**\n\n**A:** Our proposed method could mitigate the two bias sources (distilled images and distillation model) in soft label.  A key contribution of our method is its ability to implicitly rectify both sources of bias on soft labels including distilled image bias and soft-label bias in a unified manner. We conducted two additional experiments to prove that our method could mitigate both the bias succesfully.\n\nExp1: Addressing bias from the distillation model (biased soft labels)\n\nWe use soft labels produced by a model trained on an long-tailed dataset, while distilled images are obtained from a model trained on a balanced dataset.\n\n| IF | biased img | biased label | method | IPC=1 | IPC=10 | IPC=50 |\n| - | - | - | - | - | - | - |\n| 10   | N  | Y | SRe2L  | 16.6     | 34.3     | 50.6     |\n|    | N | Y | +ours  | **19.0** | **41.4** | **62.6** |\n| 100  | N | Y  | SRe2L  | 18.3     | 24.4     | 31.5     |\n|   | N | Y | +ours  | **21.8** | **36.9** | **57.6** |\n\n> Note: \"Y\" indicates the presence of bias; \"N\" indicates no bias in that source. \"IF\" means Imbalance Factor.\n\nThese results clearly show that our method can effectively mitigate soft-label bias introduced by a biased distillation model.\n\nExp2: Addressing bias in distilled images\n\nThis experiment evaluates the bias introduced by distilled images. The baseline uses images from a long-tailed model and soft labels from a balanced one, while our method is applied when both images and labels are from long-tailed models.\n\n| IF   | biased img | biased label | method | IPC=1 | IPC=10 | IPC=50 |\n| - | - | - | - | - | - | - |\n| 10   | Y  | N | SRe2L  | 16.0 | 34.7 | 55.3  |\n|      | Y | Y  | +ours  | **23.7** | **40.2** | **63.1** |\n| 100  | Y | N | SRe2L  | 15.6 | 22.9 | 54.8 |\n|      | Y  | Y  | +ours  | **21.4** | **33.2** | **58.4** |\n\nThese results indicate that even when both the distilled images and soft labels are biased (i.e., derived from long-tailed models), our method still outperforms a baseline that uses soft labels from unbiased distillation model. This highlights the effectiveness of our method in mitigating the impact of image-induced bias.\n\nWe do not follow the baseline setting (biased images with unbiased labels) because our method requires class-frequency information, determined by the imbalance factor (IF), from the distillation model's training distribution. Therefore, a model trained on balanced data is incompatible. This design also reflects realistic scenarios, where both the distillation model and synthetic images originate from the same long-tailed dataset.\n\nWe will revise the main text to better highlight this point and include this experiment in Sec. 4.2 of the revised version.\n\n**Q2: In Figure 3(b), why is there a sudden performance improvement at the end, even as the imbalance factor increases?**\n\n**A:** The direct cause lies in our use of SRe2L’s default hyperparameters - specifically, the number of training epochs (EP=200) for the distillation model, which resulted in suboptimal performance. To investigate this, we compared the performance of synthetic datasets distilled using **models trained with different numbers of epochs** on CIFAR10:\n\nExp1: Effect of Model Epoch on Performance with IF=10\n\n| Epoch | IF   | Method | IPC=10    | IPC=50    |\n| - | - | - | - | - |\n| 10    | 10   | SRe2L  | 39.6 | 55.0 |\n|       |      | +ours  | 41.9  | 63.2 |\n| 30    | 10   | SRe2L  | 27.2    | 51.5    |\n|       |      | +ours  | 26.2    | 50.0  |\n| 50    | 10   | SRe2L  | 24.6      | 46.8     |\n|       |      | +ours  | 25.2     | 53.6    |\n\nExp2: Effect of Model Epoch on Performance with IF=100\n\n| Epoch | IF   | Method | IPC=10    | IPC=50    |\n| - | - | - | - | - |\n| 10    | 100  | SRe2L  | 21.6    | 23.6     |\n|       |      | +ours  | 33.0 | 37.9    |\n| 30    | 100  | SRe2L  | 28.1  | 38.2    |\n|       |      | +ours  | 32.0     | 59.1 |\n| 50    | 100  | SRe2L  | 23.9    | 40.3 |\n|       |      | +ours  | 29.1     | 53.8    |\n\nWe observe that as the epoch increases, the performance of the distilled dataset first improves and then declines, indicating the existence of an optimal epoch. Moreover, this optimal epoch tends to increase as the imbalance factor (IF) or images per class (IPC) increases. This means that performance of distilled dataset does not monotonically align with accuracy of distillation model. As shown in [1], early-stage models provide richer soft labels, while late-stage models yield overconfident outputs. Similarly, [2–3] show that proper model epoch selection is crucial for effective trajectory matching.\n\nIn the original setting, we use fixed training epochs (200) for the distillation model. As the imbalance factor (IF) increases, the optimal epoch shifts later, which explains the seemingly anomalous improvement in performance with higher imbalance.\n\nWe revised our experimental setup by selecting the best-performing model from $\\text{Epoch}\\in\\set{10,30,50}$, and replotted Figure 3(b) accordingly. As shown, the previously observed anomalous performance trend disappears.\nThe updated results are presented below.\n\n| tau\\IF |        10 |        50 |       100 |       500 |      1000 |      2000 |\n| - | - | - | - | - | - | - |\n|      0 |     32.6 |     28.2 |    29.2 |     23.9 |     21.9 |    14.5 |\n|      1 |    39.7 |     33.5 |     34.0 |     30.1 |     25.3 |     18.9 |\n|    1.5 |     40.6 |    33.3 |     34.4 |    28.8 |    28.2 |    20.5 |\n| ours | **41.6** | **42.7** | **35.8** | **32.5** | **29.3** | **27.4** |\n\nWe will provide all experimental results in the appendix and update the corresponding results in the paper. Additional results are available upon request during the discussion, given the rebuttal’s word limit.\n\n**Q3: ADSA performance sometimes  drops with small IPC. Why?**\n\n**A:** As noted in Section 3.1, low IPC makes it difficult to capture the original distribution, weakening alignment between labels and images. In addition, our method depends on stable soft-label statistics for calibration, which become unreliable with very limited data.\n\n**Q4: In Fig. 1(b), the balanced model seems to generate images, not just relabel.**\n\n**A:** Thank you for pointing out the error in the figure. The labels below the models in Fig. 1(b) were mistakenly swapped and will be corrected in the revised version.\n\n**Q5: When applied to a balanced training dataset, does ADSA lead to any performance degradation?**\n\n**A:** ADSA does not cause any performance degradation when applied to a balanced training dataset, which follows directly from its formulation. Specifically, the calibration mechanism defined in Eq. (9) is:\n$$\np(y|x; \\tau) = \\frac{\\exp(f_y(x) - \\tau \\log \\pi_y)}{\\sum_{y' \\in [C]} \\exp(f_{y'}(x) - \\tau \\log \\pi_{y'})},\n$$\nwhere $f_y(x)$ denotes the predicted logit for class $y$, and $\\pi_y$ represents the class prior (i.e., the frequency of class $y$ in the training data).  \n\nIn the case of a balanced dataset with $K$ classes, we have $\\pi_1 = \\pi_2 = \\cdots = \\pi_K = \\frac{1}{K}$. Substituting into the equation yields:\n$$\np(y|x; \\tau) = \\frac{\\exp(f_y(x) + \\tau \\log K)}{\\sum_{y' \\in [C]} \\exp(f_{y'}(x) + \\tau \\log K)}\n= \\frac{\\exp(f_y(x))}{\\sum_{y' \\in [C]} \\exp(f_{y'}(x))}\n$$\n\nThe calibration term cancels out as a constant, so our method reduces to standard distillation and causes no change in balanced settings.\n\n**Q6: Does ADSA introduce computational or memory bottlenecks as data and class counts grow?**\n\n**A:** We fully understand your concern. However, the computational and memory overhead introduced by our method is minimal and negligible compared to baseline methods, and it imposes no practical burden in real-world applications. We simulate runtime under varying numbers of classes and IPCs using ResNet-18, with time measured in **seconds**. \n\n| Resolution | CLASS_NUM \\ IPC | IPC=1 | IPC=10 | IPC=50 | IPC=100 |\n| - | - | - | - | - | - |\n| 32         | 10              | 1.7  | 1.7  | 2.0     | 2.0 |\n| 32         | 100             | 1.7  | 2.2  | 3.3   | 4.7   |\n| 224        | 1000            | 7.8  | 29.5 | 107.2 | 205.3 |\n\nWe only store logits of shape $\\text{IPC} \\times \\text{CLASS} \\times \\text{CLASS}$, requiring minimal GPU memory. Compared to standard dataset distillation baselines, our overhead remains insignificant, as the table shown below.\n\n| IPC  | CLASS_NUM | FP32 (4 Bytes) |\n| - | - | - |\n| 1    | 10    | 400 Bytes      |\n| 1    | 100   | 39.1 KB      |\n| 1    | 1000  | 3.8 MB        |\n| 10   | 10    | 3.9 KB        |\n| 10   | 100   | 390.6 KB      |\n| 10   | 1000  | 38.2 MB      |\n| 50   | 10    | 19.5 KB       |\n| 50   | 100   | 1.9 MB        |\n| 50   | 1000  | 190.7 MB      |\n\nRefs:\n\n[1] A Label Is Worth a Thousand Images in Dataset Distillation, NIPS, 2024\n\n[2] Towards Lossless Dataset Distillation via Difficulty-Aligned Trajectory Matching, ICLR, 2024\n\n[3] Dataset Distillation by Automatic Training Trajectories, ECCV, 2024"}]

Author rebuttal:

Meta-review: This paper presents a new method for long-tailed dataset distillation, which has not been well attempted in previous works. Specifically, the paper generates soft labels for compressed data points using post-hoc logit adjustment. Experiments demonstrate the effectiveness of the method. 

After carefully reviewing the paper, the AC find that the paper does not propose new dataset compression approaches, and the generation of soft labels using logit adjustment is not new in long-tail learning. These can be potential weaknesses of the paper.

After rebuttal, the paper received four positive ratings. All reviewers acknowledged the paper's contributions and expressed satisfaction with its technical merits. 

The AC slightly tends to concur with the reviewers' assessments and recommends acceptance. not new.
Decision: Accept (poster)