Title: Hyperbolic Dataset Distillation
Venue: NeurIPS 2025
Accepted: yes
Keywords: Dataset Distillation

Hyperbolic Space

Distribution Matching
Abstract: To address the computational and storage challenges posed by large-scale datasets in deep learning, dataset distillation has been proposed to synthesize a compact dataset that replaces the original while maintaining comparable model performance. Unlike optimization-based approaches that require costly bi-level optimization, distribution matching (DM) methods improve efficiency by aligning the distributions of synthetic and original data, thereby eliminating nested optimization. DM achieves high computational efficiency and has emerged as a promising solution. However, existing DM methods, constrained to Euclidean space, treat data as independent and identically distributed points, overlooking complex geometric and hierarchical relationships. To overcome this limitation, we propose a novel hyperbolic dataset distillation method, termed HDD. Hyperbolic space, characterized by negative curvature and exponential volume growth with distance, naturally models hierarchical and tree-like structures. HDD embeds features extracted by a shallow network into the Lorentz hyperbolic space, where the discrepancy between synthetic and original data is measured by the hyperbolic (geodesic) distance between their centroids. By optimizing this distance, the hierarchical structure is explicitly integrated into the distillation process, guiding synthetic samples to gravitate towards the root-centric regions of the original data distribution while preserving their underlying geometric characteristics. Furthermore, we find that pruning in hyperbolic space requires only 20\% of the distilled core set to retain model performance, while significantly improving training stability. Notably, HDD is seamlessly compatible with most existing DM methods, and extensive experiments on different datasets validate its effectiveness. To the best of our knowledge, this is the first work to incorporate the hyperbolic space into the dataset distillation process. The code is available at https://github.com/Guang000/HDD.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates an issue with dataset distillation, where synthesized data in distribution matching method are trained under the setting of independent and identical distribution and fail to capture the complex geometric and hierarchical relationship. To address this problem, the authors propose a new dataset distillation strategy called Hyperbolic Dataset Distillation method (HDD). HDD embeds data into hyperbolic space and optimizes geodesic distance between the centroids of synthetic and original data, naturally capturing tree-like and hierarchical relationships. This approach allows hierarchical pruning and it is compatible with most existing distribution matching methods.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\nThis paper is well-motivated, and well-organized. The authors provide comprehensive experiments on various datasets such as CIFAR-10, CIFAR-100, TinyImageNet. They provide insightful analysis of the experimental results and a detailed method description.\n\n**Weakness**\n1. The encoder used to extract features before hyperbolic embedding. This may lead to the loss of important hierarchical information or cause feature collapse, where synthetic samples concentrate around the origin without preserving meaningful structure.\n2. The effectiveness depend on tuning of curvature $K$ and scaling factor $\\lambda$. \n3. Some definitions are not clear: (a) In line 265, the definition of $K_i$ is ambiguous; (b) The correct presentation of Eq. (1) should be: $ \\mathcal{S}^* = \\arg \\min_{\\mathcal{S}} E_{(x, t) \\sim P_T} || \\ell(\\theta_{\\text{syn}}(x), t) - \\ell(\\theta_{\\text{real}}(x), t) || $. Eq. (2) contains a similar mistake.\n4. Some notations are abused. For example, the authors define $x$ as data instance, $t$ as label and $s$ as synthetic data with super script $syn$ in Sec 3.1 Problem Definition. However, in Sec 3.1 Hyperbolic Geometry, $x_t$ and $x_s$ are redefined as time component and spatial component. It is recommended to revise the notation to ensure consistency and clarity.\n\nQUESTIONS:\n1. How is the curvature associated with sample $i$ computed?\n2. The authors mention that distribution matching methods improve efficiency, but optimization-based methods typically achieve higher performance. Is it feasible to apply HDD to optimization-based methods? If so, could the authors provide experimental results?\n3. Could the authors provide more concrete evidence that the performance gains from HDD are indeed due to the presence of hierarchical structure in the original data? For the standard benchmark datasets used in the paper, the assumption of hierarchical relationships is not verified.\n\nLIMITATIONS:\nIn Sec. 5, the authors mention a limitation that more advanced distribution alignment objectives, such as KL divergence, are not explored in hyperbolic space, which could potentially improve performance further.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-22"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis study proposes Hyperbolic Dataset Distillation (HDD), a dataset distillation based on hyperbolic space, instead of Euclidean space-based dataset distillation. Since hyperbolic space naturally implies a hierarchical structure, we expect it to model the complex geometry and hierarchical relationship of datasets better than existing methods. As a specific methodology, this study proposes a distribution matching objective over hyperbolic space using Lorentz hyperbolic space. Experimental results show the effectiveness of HDD.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n1. This manuscript is well written and easy to read.\n\n2. To my knowledge, this is the first study in the field of dataset distillation to discuss optimization in hyperbolic space rather than Euclidean space, which is interesting and significant because it brings new perspectives and possibilities to the field of dataset distillation.\n\n3. This study proposes a reasonable and logical methodology using hyperbolic geometry. It is also easy to apply because HDD can be implemented with the materials of prior research without additional materials.\n\n**Weaknesses**\n1. My major concern is the originality and novelty of the work. As the authors note in Related works, the direction of introducing hyperbolic spaces into machine learning has been well studied. The proposed HDD is a methodology that replaces the Euclidean nature of vanilla DM with a hyperbolic nature. This work appears to be a combination of existing techniques, applying hyperbolic geometry to dataset distillation. Therefore, I think that this study lacks originality and novelty because HDD has not been modified for dataset distillation.\n\n2. Lack of a clear rationale for applying hyperbolic space to dataset distillation. I was expecting to read about what problems Euclidean space DM has and how introducing hyperbolic space can solve them, but the manuscript only has statements to support this, and no theoretical or experimental evidence (only performance comparison).\n\n3. Although hyperbolic space has the advantage of an inherent hierarchical structure, there are no experimental results for benchmark datasets that contain a hierarchical structure. This makes it difficult to verify that synthetic datasets created with HDDs actually contain a hierarchical structure well.\n\nQUESTIONS:\n1. The experimental results (Table 1) show that adding HDDs improves performance, but I'm curious about the rationale for this.\n\n2. In this manuscript, it is mentioned that the closer to the centroid, the better the representation of category prototypes because it encodes the overall geometric structure of the dataset. However, it is difficult to accept the claim because it is difficult to judge which image is higher-level or lower-level when looking at the visualization of distilled images in the Appendix. I would like to know what hyperbolic space the distilled images should be visualized in and the interpretation of it.\n\n3. I'm also curious about the results of HDD on a benchmark dataset with a hierarchy structure.\n\n4. If a hierarchy structure exists in the optimized synthetic dataset, it is able to prune along the hierarchy after training. If this direction is correct, I would be interested to see a comparison with MDC [1], as well.\n\n[1] Multisize Dataset Condensation\n\nLIMITATIONS:\nSee weaknesses and questions", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Hyperbolic Dataset Distillation (HDD), a drop-in modification to existing distribution-matching methods (DM, IDM, Dance) that embeds real and synthetic features into Lorentz hyperbolic space and minimizes the geodesic distance between their centroids. The key insight is that hyperbolic geometry inherently emphasizes low-level “prototype” samples, which helps retain class semantics and hierarchical structure during distillation. The authors demonstrate that HDD consistently improves performance across five datasets (Fashion-MNIST, CIFAR-10, CIFAR-100, SVHN, Tiny-ImageNet) for a variety of IPC (images per class) settings and enables “hierarchical pruning,” where only 20% of the real data is used to distill without hurting final accuracy.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\n1. The main idea of doing centroid matching in Lorentz space is simple, intuitive, and reasonably novel in the distillation setting. The use of hyperbolic geometry biases the synthetic data toward prototypical examples and aligns well with the distribution-matching objective.\n\n2. The improvements are consistent across multiple datasets and baselines. For instance, adding HDD to IDM yields a 4% absolute gain on CIFAR-10 at IPC=10 (from 57.3% to 61.3%). Similar trends hold across other datasets, including CIFAR-100 and Tiny-ImageNet. Standard deviation are given and in most cases the proposed method yields a statistically significant result.\n\n3. The hierarchical pruning results are strong. The fact that the authors can discard 80% of the real dataset and still get the same final accuracy is surprising and interesting. The pruning also appears to stabilize training.\n\n## Weaknesses\n\n1. The paper lacks any ablation over curvature ($K$) or the weight $\\lambda$ on the centroid loss. These hyperparameters likely matter quite a bit, especially given the numerical instability of hyperbolic operations, and should be studied.\n\n2. No runtime or memory comparison is given. Log/exp maps and acosh are not cheap, and it’s unclear whether this adds significant overhead over the Euclidean baseline.\n\n3. Although the idea is new in the context of dataset distillation, it’s not fundamentally new in general. Centroid matching is well-known, and using a different geometry for it is not that far from existing work.\n\n4. The scope is fairly limited: experiments top out at Tiny-ImageNet (64×64 resolution), and it’s unclear whether the method scales to larger datasets like ImageNet-1k or non-image domains.\n\n## Verdict\n\nThis is a clean and reasonably well-executed paper that introduces a simple yet effective modification to DM-style dataset distillation. While the originality is limited and runtime/memory comparisons are missing, the empirical results are solid and the implementation is relatively straightforward. I recommend a weak accept rating.\n\nQUESTIONS:\n1. How sensitive is HDD to the curvature K? Are there stability issues for very negative values?\n\n2. Have you tried using more sophisticated centroids as opposed to just tangent-space approximation of the Fréchet mean? In fact, due to some recent work on differentiating through the Fréchet mean [a], you may actually be able to perform the operation with the true mean (although this will likely be considerably slower). \n\n### References\n\n[a] Differentiating through the Fréchet Mean. https://arxiv.org/pdf/2003.00335\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nDataset distillation compresses a large dataset into a small set of synthetic examples that achieve performance comparable to training on the full original data. In dataset distillation methods, distribution matching algorithms align feature distributions between real and synthetic dataset. This paper aims to leverage the hierarchical structure inherent in dataset samples, and introduce a novel hyperbolic data distillation (HDD) method. The objective in distribution matching is reformulated as minimizing the Lorentzian hyperbolic distance between the Riemannian means of the original and synthetic datasets. Experiments show that the proposed HDD method is easy to add on top of two existing method across 5 different datasets leading to better performance improvements.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\n**Novelty:** To my knowledge, this is the first work to explore dataset distillation through the lens of hyperbolic geometry. The formulation of distribution matching loss in hyperbolic space is simple but sound. \n\n**Interesting observations:** The observation that samples closer to origin contribute more strongly to the Fréchet mean of the dataset is interesting. The authors equate this to reducing outlier contribution (line 213-214). This is also validated in Table 2 where removing 80% samples exhibiting highest curvature does not impact the performance on CIFAR-10 dataset. \n\n**Empirical Effectiveness:** The authors mention that HDD can be *plugged* into existing distribution matching methods and achieves meaningful gains over over using the methods alone. The experiments show results across distribution matching methods like DM[62], IDM[64] and hybrid methods like DANCE[58] (alternates between cross entropy and distribution matching).\n\n## Weaknesses\n\n**How can HDD augment existing distribution matching methods?** \n\n- Firstly, It is straightforward to understand how HDD loss can replace maximum mean discrepancy loss (MMD) the DM[62] paper. However, there are some missing implementation details with IDM[64] and DANCE[58]. Since both methods have an additional cross entropy loss, is the cross entropy loss still in euclidean space or in hyperbolic space? What is reason for choosing one or the other?\n- The paper only shows two distribution matching methods, the original DM[62] and IDM[64]. Is this generalizable to other distribution matching methods like CAFE [50], M3D[59], IID[12]? Is it as simple as replacing the MMD loss with HDD loss or does it require additional tuning?\n\n**Visualizations**: Which samples contribute the most/least in synthetic dataset? I suggest the authors to also visualize such samples. Do samples closer to origin look more prototypical than samples farther away suggested in Figure 1.\n\n**Class wise contributions:** Figure 4 shows that samples clustering is quite disperse. This will affect class-wise separability. How does class-aware regularization (similar to IDM or IID) look like in hyperbolic space? While this doesn't affect the novelty introduced in the paper, I'm curious how a hyperbolic equivalent of class centralization constraint (compact clustering in IID) look like and how would it impact how samples look like in the space?\n\nQUESTIONS:\nMain questions are listed in weaknesses. To summarize, I suggest the authors to clarify how the cross entropy loss of IDM and DANCE is added to HDD, whether as hyperbolic or euclidean? I also suggest the authors to visualize the samples of synthetic and real datasets that are closer to the origin and therefore are more _important_ for distribution matching. Finally, I want to know the authors' perspective on how a class-wise constraint can be added to HDD? \n\n**Minor**:\n- What is IPC? IPC of synthetic dataset is referred to multiple times in the paper but is not explained.\n- duplicate citations ([23],[24])\n\nLIMITATIONS:\nThe authors discuss limitations adequately.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-07"}