Title: Rectifying Soft-Label Entangled Bias in Long-Tailed Dataset Distillation
Venue: NeurIPS 2025
Accepted: yes
Keywords: dataset distillation; long-tailed recognition
Abstract: Dataset distillation compresses large-scale datasets into compact, highly informative synthetic data, significantly reducing storage and training costs. However, existing research primarily focuses on balanced datasets and struggles to perform under real-world long-tailed distributions. In this work, we emphasize the critical role of soft labels in long-tailed dataset distillation and uncover the underlying mechanisms contributing to performance degradation.
Specifically, we derive an imbalance-aware generalization bound for model trained on distilled dataset. We then identify two primary sources of soft-label bias, which originate from the distillation model and the distilled images, through systematic perturbation of the data imbalance levels.
To address this, we propose ADSA, an Adaptive Soft-label Alignment module that calibrates the entangled biases. This lightweight module integrates seamlessly into existing distillation pipelines and consistently improves performance. On ImageNet-1k-LT with EDC and IPC=50, ADSA improves tail-class accuracy by up to 11.8\% and raises overall accuracy to 41.4\%. 
Extensive experiments demonstrate that ADSA provides a robust and generalizable solution under limited label budgets and across a range of distillation techniques.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the important problem of distilling or compressing real-world long-tail data, which is highly relevant in many practical scenarios. A particularly notable contribution is the derivation of an imbalance-aware generalization bound for models trained on distilled datasets, providing valuable theoretical insight. Additionally, the proposed ADSA module is a key strength of the work, offering seamless integration into existing distillation pipelines, which enhances its practical applicability.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n- While I am not an expert in dataset distillation, the paper clearly communicates its core ideas and the importance of the problem it addresses. It is a well-written and accessible paper even for readers less familiar with the domain.\n- The experimental evaluation is thorough, and the inclusion of diverse visualization results is particularly impressive.\n- The method demonstrates consistent performance improvements across a wide range of baselines, which highlights its robustness.\n- The proposed hyperparameter-free approach is a significant advantage, as it enhances the method’s practicality and ease of adoption in real-world scenarios.\n\nQUESTIONS:\n**Major**\n- ADSA performs soft label calibration to address the bias introduced by the distillation model. However, it is unclear whether this also mitigates the bias inherent in the distilled images themselves. I believe the paper would benefit from an explicit explanation of how ADSA handles this aspect.\n- In Figure 3(b), why is there a sudden performance improvement at the end, even as the imbalance factor increases? What explains this behavior?\n- In scenarios with small IPC, the performance of ADSA sometimes drops. Do the authors have any insight or explanation for this?\n- In Figure 1(b), the balanced datasets trained model is stated to be used only for relabeling, yet it appears to be generating images. Similarly, the model used only for generating images also seems to produce labels. Could the authors clarify whether this is a typo or an inconsistency in the figure?\n\n**Minor**\n- When applied to a balanced training dataset, does ADSA lead to any performance degradation?\n- It seems that additional computation is required for ADSA. As the number of data points and classes increases, does the distillation speed or memory usage become a bottleneck?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-06-21"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a lightweight module, ADSA, to calibrate soft-label biases in long-tailed dataset distillation, which stem from both the teacher model and the synthetic images. By deriving an imbalance-aware generalization bound, the authors reveal the underlying causes of performance degradation. Extensive experiments demonstrate that ADSA consistently improves performance across various distillation methods and label budgets, achieving up to an 11.8% improvement in tail-class accuracy on ImageNet-1k-LT.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\nThe proposed lightweight module effectively mitigates these biases and delivers consistent performance improvements across various settings. Experimental results show that the method outperforms existing strong baselines.\n\nWeaknesses\n\n1.In Equations (1)–(2), the distilled dataset is denoted as 𝐷syn​ , but in Equations (4)–(6), it suddenly switches to 𝐷dd  (e.g., Rdd). The notation should be unified or clearly explained.\n\n2.LAD is presented as a comparative long-tailed distillation method, but its results are missing under some IPC/IF settings (e.g., IPC=1 is absent in Table 1). If feasible, these should be included for completeness.\n\n3.The experiments only report the hardware used, but lack details such as per-experiment runtime and memory usage (e.g., GPU hours required to distill ImageNet-LT with IPC=50), which limits reproducibility and practical cost assessment.\n\nQUESTIONS:\n1.Equation (3): The constant C is undefined. Could the authors clarify its meaning and boundary conditions in this context?\n\n2.Equation (15): It assumes ptr​ (x∣y)=pte(x∣y), a common setting in long-tailed learning. Does this assumption still hold in dataset distillation scenarios?\n\n3. Are the reported results from a single run? The absence of variance or confidence intervals makes it difficult to assess stability. It is recommended to report results over multiple random seeds.\n\nIf the authors are able to address the above issues, I would be willing to raise my score.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, authors study dataset distillation problem in the setting where the dataset follows a long-tailed distribution. First, they conduct a theoretical and empirical analysis, showing that the bias comes from both distilled images as well the distillation model. Based on this observation, the authors then propose adaptive soft label alignment module. In this proposed method, they adopt a logit calibration method. They show that the logit calibration method can be applied to existing dataset distillation methods in a post-hoc manner. Their experiment results show that their method effectively mitigate the poor performance issued caused by long tailed datasets.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n   1. Well-motivated problem:\n\nThe paper tackles a practically important issue—real-world datasets often exhibit class imbalances rather than a clean uniform distribution. Addressing dataset distillation specifically in long-tailed settings is clearly relevant and should interest the broader research community.\n\n   2. Insightful empirical analysis:\n\nThe paper provides a careful and insightful empirical analysis, clearly illustrating when and why existing dataset distillation methods fail under long-tailed distributions. The use of three distinct configurations to identify and isolate the sources of bias is particularly effective and enhances understanding of the underlying problem.\n\n**Weaknesses**\n\n   1. Incomplete handling of identified biases:\n\nMy understanding is that the central methodological contribution is the introduction of Eq. 9, which essentially applies a frequency-based penalization to mitigate bias in soft labels. While this approach logically addresses bias introduced by class imbalance in the training set, the authors neglect the bias arising specifically from the distilled images, even though it was explicitly highlighted as a key issue in the empirical analysis (Section 3.2). The paper would benefit from either a justification of why the image-induced bias is less critical or a discussion of possible solutions for addressing this image-specific bias.\n\n   2. Lack of essential baseline:\n\nTraining directly on class-imbalanced datasets is known to lead to suboptimal performance even beyond the dataset distillation setting. An intuitive baseline would be to apply well-established data balancing techniques (such as oversampling or weighted sampling) to the original long-tailed dataset prior to distillation. Such a baseline would clarify whether simple and well-known rebalancing methods could already sufficiently mitigate the issue without requiring specialized adjustments in dataset distillation methods.\n\nQUESTIONS:\n1. I am a bit confused about the experiment setup. My current understanding is that they authors frame their method as a simple add on of existing dataset distillation pipeline. And in their main experiments (Section 4.1 Table 1, 2), they show existing methods performance (such as SRe2L, EDC) and their add on. Just to clarify, authors trained all the experts, distilled all the images from the long-tailed distribution, the \"ours\" part is to apply Eq. 9 and 10 to soft labels. In other words, during the entire dataset distillation pipeline, there is no access to balanced datasets. \n\n\n2. I am also quite confused about Figure 1(a), why does original dataset equal imbalanced dataset plus balanced dataset? Overall, I think this paper could benefit from more clear writing about the set up.\n\nLIMITATIONS:\nYes, authors have addressed the limitations and potential negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses the problem of distilling long-tailed dataset information. In particular, the authors derive a generalization error bound by adapting the formulation used in D3S [1] for the long-tailed setting under label shift. Based on that bound, it is argued that the distilled data can introduce bias through two sources: (i) the distilled images and (ii) their soft-labels, while their effect is evaluated via a perturbation analysis controlling different conditions. To account for these biases, post-hoc logit-adjustment (LA) [2] is used to correct soft-labels of the distilled dataset. Having observed a distribution shift between real and synthetic data distributions, the authors propose to minimize class-wise confidence variance to calibrate the strength factor of LA, leveraging distilled data. This adaptive soft label alignment leads to improvement in the overall and tail classification performance across different imbalance settings and SOTA data-distillation methods.\n\nExtra references:\n\n[1] Loo, Noel, et al. \"Large scale dataset distillation with domain shift.\" Forty-first International Conference on Machine Learning. 2024.\n\n[2] Menon, Aditya Krishna, et al. \"Long-tail learning via logit adjustment.\" arXiv preprint arXiv:2007.07314 (2020).\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- Partial theoretical connection between the generalization bound and imbalance levels.\n- The experimental setting convincingly demonstrates that the proposed adaptive logit-adjustment applies to long-tailed dataset-distillation and boosts the overall performance across different baselines and imbalance settings.\n- The proposed method is simple and can be easily applied on top of methods originally developed for balanced dataset-distillation.\n\nWeaknesses:\n- W1 - The connection between the forms in Eqs. 5 and 6 and the perturbation analysis is unclear.\n- W2 - The developed theory attributes tail performance collapse to two different sources, yet the proposed method is disconnected from this finding.\n- W3 - Some details are missing, hindering the readability of the paper.\n\nQUESTIONS:\n- Q1: Can you include the intermediate steps when deriving the forms 1 and 2 when proving Th 3.1. to ensure correctness? \n\n- Q2: I can follow point (iii) in L175 in relation to the second term in Eq 5.but I do not follow why this has to hold for each class when looking at the second term of Eq 6. e.g., miss-labelled but identical samples could still minimize the second term in Eq 6.\n\n- Q3: I found the connection between the analysis in Sec 3.2. and the insights drawn from Th 3.1. to be unclear. \n   - Q3.1: Could you provide a connection between forms 1 and 2 with the three configs considered? e.g., clarify which term each configuration is trying to optimize. Also, including a config (4) with balanced/balanced would make for a more complete perturbation analysis.\n   - Q3.2: The generalization bound is derived on the full test loss, in that sense it would be interesting to see what happens to the overall performance (overall tail+head performance) \n   - Q3.3:  L242-244. This finding contradicts the theory developed which calls for a deeper analysis on this matter.\n   - Q3.4:  L212-214. Shouldn’t we draw that conclusion when comparing config 1 with config 3 instead where they only differ by the DM (2)?\n   - Q3.5: When solving Eq. 7. across the three configs, are you using the same g network and only ablate the f_{tr}? \n\nQ4: Fig3. (b) The proposed method does not display a decreasing performance trend as the IF increases in contrast to the baselines. Could you explain why this is the case?\n\nTypos/Minor:\n(I am not expecting an answer, please reflect upon these and refine if necessary)\n\n- T1. Define what the ADSA abbreviation means the first time you use it in the main text.\n- T2. L146. is \\theta_{syn}== \\theta_{dd}?\n- T3. Eq.3 the C is not defined.\n- T4. Eq.5 define the connection between the densities $p$ and the datasets. E.g. how is the p_tr related to D_tr?\n- T5. L150. did you mean training loss?\n- T6. Definition of Th 3.1. lacks rigor, what is C? Under what assumption Th 3.1. holds? The Th 3.1. as provided in [1] is a good reference point.\n- T7. Tab 1. SRe2L + ours (IP=1 and IF=50) should not be in bold.\n- T8. Fig 1. (b) please define the DM (1) and DM (2) abbreviations. Also it appears that (only for relabelling and generating images have been mixed).\n- T9. Fig3. Left panel, are these the correct titles? Right panel, what is the IF used for (a) and the IPC for (b)? \n- T10. L573.The first paragraph and second paragraph overlap significantly.\n- T11. L596. Please provide some more details on how these t-SNE plots were constructed. e.g., are you using the same feature extractor for all settings?\n- T12. L317. define what the soft-label budget means to make the paper self-contained.\n- T13. Eq.10 you could use a different index (e.g., j) on the inner summation to avoid confusion.\n\nFinal comments:\nAlthough the empirical results are convincing, the proposed method appears to be a trivial extension of logit-adjustment in the dataset condensation and is disconnected from the theory developed, limiting the depth of the study. Although the developed theory is interesting, the empirical results do not always support it, calling for further investigation. Concluding, I do not see how the theoretical insights connect with the proposed method, which nevertheless appears to perform well by using existing methodology (i.e., logit-adjustment).\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-02"}