Are Large-scale Soft Labels Necessary for Large-scale Dataset Distillation?

Authors: lingao  xiao
yang  he
Venue: NeurIPS_cc_2024_Conference
Published: 2024-04-27 14:51:43

Abstract: In ImageNet-condensation, the storage for auxiliary soft labels exceeds that of the condensed dataset by over 30 times.
However, are large-scale soft labels necessary for large-scale dataset distillation?
In this paper, we first discover that the high within-class similarity in condensed datasets necessitates the use of large-scale soft labels.
This high within-class similarity can be attributed to the fact that previous methods use samples from different classes to construct a single batch for batch normalization (BN) matching.
To reduce the within-class similarity, we introduce class-wise supervision during the image synthesizing process by batching the samples within classes, instead of across classes.
As a result, we can increase within-class diversity and reduce the size of required soft labels.
A key benefit of improved image diversity is that soft label compression can be achieved through simple random pruning, eliminating the need for complex rule-based strategies. Experiments validate our discoveries.
For example, when condensing ImageNet-1K to 200 images per class, our approach compresses the required soft labels from 113 GB to 2.8 GB (40$\times$ compression) with a 2.6\% performance gain.
Code is available at: https://github.com/he-y/soft-label-pruning-for-dataset-distillation

Main paper: 
 related works: 
 Dataset Distillation. DD [1] first introduces dataset distillation, which aims to learn a synthetic dataset that is equally effective but much smaller in size. The matching objectives include performance matching [1, 11, 12, 13, 14] , gradient matching [4, 15, 16, 17] , distribution or feature matching [5, 2, 18] , trajectory matching [3, 19, 20] , representative matching [21, 22] , loss-curvature matching [23] , and Batch-Norm matching [6, 7, 9, 10] .Dataset Distillation of Large-Scale Datasets. Large-scale datasets scale up in terms of image size and the number of total images, incurring affordable memory consumption for most of the welldesigned matching objectives targeted for small datasets. MTT [3] is able to condense Tiny-ImageNet (ImageNet-1K subsets with images downsampled to 64 × 64 and 200 classes). IDC [24] conducts experiments on ImageNet-10, which contains an image size of 224 × 224 but has only 10 classes. TESLA [20] manages to condense the full ImageNet-1K dataset by exactly computing the unrolled gradient with constant memory or complexity. SRe 2 L [6] decouples the bilevel optimization into three phases: 1) squeezing, 2) recovering, and 3) relabeling. The proposed framework surpasses TESLA [20] by a noticeable margin. CDA [7] improves the recovering phase by introducing curriculum learning. RDED [8] replaces the recovering phase with an optimization-free approach by concatenating selected image patches. SC-DD [10] uses self-supervised models as recovery models. Existing methods [7, 8, 10] place high emphasis on improving the recovering phase; however, the problem of the relabeling phase is overlooked: a large amount of storage is required for the relabeling phase.Label Compression. The problem of excessive storage seems to be fixed if the teacher model generates soft labels immediately used by the student model on the fly. However, when considering the actual use case of distilled datasets (i.e., Neural Architecture Search), using pre-generated labels enjoys speeding up training and reduced memory cost. More importantly, the generated labels can be repeatedly used. FKD [25] employs label quantization to store only the top-k logits. In contrast, our method retains full logits, offering an orthogonal approach to quantization. A comparison to FKD is provided in Appendix D.3. Unlike FerKD [26] , which removes some unreliable soft labels, our strategy targets higher pruning ratios.Comparison with G-VBSM [9] . In one recent work, G-VBSM also mentioned re-batching the images within classes; however, the motivation is that having a single image in a class is insufficient [9] . It re-designed the loss by introducing a model pool, matching additional statistics from convolutional layers, and updating the statistics of synthetic images using exponential moving averages (EMA). Additionally, an ensemble of models is involved in both the data synthesis and relabel phase, requiring a total of N forward propagation from N different models, where N = 4 is used for ImageNet-1K experiments. On the other hand, we aim to improve the within-class data diversity for reducing soft label storage. Furthermore, to account for the re-batching operation, we introduce class-wise supervision while all G-VBSM statistics remain global. 
preliminaries: 
 The conventional Batch Normalization (BN) transformation is defined as follows:EQUATIONwhere γ and β are parameters learned during training, µ and σ 2 are the mean and variance of the input features, and ϵ is a small constant to prevent division by zero. Additionally, the running mean and running variance are maintained during network training and subsequently utilized as µ (mean) and σ 2 (variance) during the inference phase, given that the true mean and variance of the test data are not available.The matching object of SRe2L [7] follows DeepInversion [27] , which optimizes synthetic datasets by matching the models' layer-wise BN statistics:EQUATIONwhere the BN's running mean BN RM l and running variance BN RV l are used to approximate the expected mean E (µ l | T ) and expected variance E σ 2 l | T of the original dataset T , repsectively. The BN loss matches BN for layers l, and µ l ( x) and σ 2 l ( x) are the mean and variance of the synthetic images x.The BN loss term is used as a regularization term applied to the classification loss L CE . Therefore, the matching objective is:EQUATIONwhere θ T is the model pretrained on the original dataset T . The symbol α is a small factor controlling the regularization strength of BN loss. Feature Cosine similarity can be formally put as:EQUATIONwhere x c and x ′ c are two images from the same class c, f (•) are the features extracted from a pretrained model, and n is the feature dimension.The similarity between images is not the only determinant of diversity since images can be dissimilar to each other yet not representative of the original dataset. Therefore, to further validate the diversity of our synthetic dataset, we consider an additional metric: the Maximum Mean Discrepancy (MMD) between synthetic datasets and original datasets. This measure helps evaluate how well the synthetic data represents the original data distribution. The following proposition clarifies the relationship between MMD and dataset diversity: Proposition 2. A lower MMD suggests that the synthetic dataset captures a broader range of features similar to the original dataset, indicating greater diversity.The empirical approximation of MMD can be formally defined as [28, 29] ,EQUATIONwhere KX,Y =1 |X|•|Y | |X| i=1 |Y | j=1 K (f (x i ) , f (y j )) with {x i } |X| i-1 ∼ X, {y i } |Y | i=1 ∼ Y .T and S denote real and synthetic datasets, respectively; K is the reproducing kernel (e.g., Gaussian kernel); P is the feature (embedding) distribution, and f (•) is the feature representation extracted by model θ, where f (T ) ∼ P T , f (S) ∼ P S .Pruning for Large-scale Distillation (LPLD)The previous objective function follows Eq. 3; it uses a subset of classes B c to match the BN statistics of the entire dataset, and images in the same class are independently generated, causing an low image diversity within classes. However, inspired by He et al. [30] , images in the same class should work collaboratively, and images that are optimized individually (see Baseline B in work [30] ) do not lead to the optimal performance when IPC (Images Per Class) gets larger. Step 1: Re-batching Images within Class. Subsequently, to obtain a collaborative effect among different images of the same class, we sample images from the same class and provide the images with class-wise supervision [4, 24] . Fig. 4 illustrates the changes.Step 2: Introducing Class-wise Supervision. However, the running mean and variance approximate the original dataset's expected mean and variance in a global aspect. The matching objective becomes sub-optimal in class-wise matching situation. To this end, we propose to track BN statistics for each class separately. Since we only track the running mean and variance, the extra storage is marginal even when up to 1K classes in ImageNet-1K (see Appendix E.2 and E.4).Step 3: Class-wise Objective Function. EQUATIONEQUATIONBatch Norm Loss with Class-wise BN Statistics (6) We want to emphasize that even though we are adjusting the BN loss with class-wise statistics, the global statistics of the dataset are still taken into account. The output logits for calculating CE loss are produced using global statistics. This is because altering µ and σ without fine-tuning γ and β could lead to a decline in model performance, resulting in less effective supervision.Theoretical Number of Updates for Stable Class-wise BN Statistics. Traditional BN layers do not compute class-wise statistics; therefore, we need to either keep track of the class-wise statistics while training a model from scratch or compute these statistics using a pretrained model. We prefer the latter as the former requires extensive computing resources. To understand how many BN statistics updates are needed, we can first look at the update rules of BN running statistics for a class c: where ϵ is the momentum. Since the momentum factor for the current batch statistics is usually set to a small value (i.e., ϵ = 0.1), we can theoretically compute existing running statistics that can be statistically significant after how many updates, assuming all other factors are fixed.EQUATIONSince the running statistics are computed per class, we provide the theoretical number of updates required to stabilize all class statistics (see Appendix A for the proof):EQUATION)where n is the number updates needed, q c is the probability that class c appears in a batch, T is a probability threshold, ε is the momentum parameter in Batch Normalization, δ is the acceptable relative deviation (where 0 ≤ δ ≤ 1), C is some constant, and τ is the desired convergence tolerance for the BN statistics. How Eq. 8 guides our experiment design is detailed in Appendix E.3.Excelling in Both Similarity Measures. By adopting the changes provided in Sec. 3.3.1, our synthetic dataset is more diverse and representative than the existing methods. First, our dataset exhibits smaller feature cosine similarity within classes compared to datasets produced by existing methods, as shown in Table 1 . This indicates that our synthetic images are less similar to each other and, thus, more diverse. Second, our dataset exhibits a significantly lower MMD shown in Fig 3 compared to datasets produced by existing methods. This suggests that our synthetic dataset better captures the feature distribution of the original dataset. After obtaining a diverse dataset, the next move is to address superfluous soft labels.Random Label Pruning. Different from dataset pruning metrics, which many wield training dynamics [31, 32] , label pruning is inherently different since the labels in different epochs are independently generated or evaluated. Subsequently, these methods do not directly apply, and we modify these metrics to determine which epochs contain the most useful augmentations and soft labels. Through empirical study, we find that using soft labels carefully pruned from different metrics is no better than simple random pruning. As a result, we can discard complex rule-based pruning metrics, attaining both simplicity and efficiency. After obtaining the soft label pool, we have to decide which labels will be used. Following the previous random pruning scheme, we randomly sample the labels for model training in order to ensure diversity and avoid any prior knowledge.Improved Label Pool. Considering that random selection may be the most efficient choice, we rethink the diversity of the label pool, as labels at the epoch-level are not the finest elements. The augmentations such as CutMix and Mixup are performed at the batch level, where the same augmentations are applied to images within the same batch and are different across batches. Therefore, we improve the label pool by allowing batches in different epochs to form a new epoch. The improved label pool breaks the fixed batch orders and the fixed combination of augmentations within an epoch, allowing a more diverse training process while reusing the labels. Our label pruning method is illustrated in Fig. 5 .Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [Yes]Justification: [TODO]Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced. • All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 
experiment settings: 
 Dataset details can be found in Appendix B and detailed settings are provided in Appendix C. Computing resources used for experiments can be found in Appendix E.5.Our experiment results are evaluated on Tiny-ImageNet [33] , ImageNet-1K [34] , and ImageNet-21K-P [35] . We follow the data pre-processing procedure of SRe 2 L [6] and CDA [7] .Squeeze. We modify the pretrained model by adding class-wise BN running mean and running variance; since they are not involved in computing the BN statistics, they do not affect performance.As mentioned in Sec. Recover. We perform data synthesis following Eq. 6. The batch size for the recovery phase is the same as the IPC. Besides, we adhere to the original setting in SRe 2 L.Relabel. We use pretrained ResNet18 [36] for all experiments as the relabel model except otherwise stated. For Tiny-ImageNet and ImageNet-1K, we use Pytorch pretrained model. For ImageNet-21K-P, we use Timm pretrained model.For validation, we adhere to the hyperparameter settings of CDA [7] .Pruning Setting. For label pruning, we exclude the last batch (usually with an incomplete batch size) of each epoch from the label pool. There are two random processes: (1) Random candidate selection from all batches. (2) Random reuse of candidate labels. Tiny-ImageNet. Table 2a presents a comparison between the label pruning outcomes on Tiny-ImageNet for our approach, SRe 2 L [6] , and the subsequent work, CDA [7] . Our method not only consistently surpasses SRe 2 L across identical pruning ratios but also achieves comparable results to SRe 2 L while using 40× fewer labels. When compared to CDA, our method exhibits closely matched performance, yet it demonstrates superior accuracy preservation. For instance, at a 40× label reduction, our method secures a notable 7.5% increase in accuracy over CDA, even though the improvement stands at a mere 0.1% at the 1× benchmark. Table 2b provides the pruning results on ResNet50 and ResNet101. Although there are consistent improvements observed when compared to ResNet18, scaling to large networks does not necessarily bring improvements.ImageNet-1K. Table 3 compares the ImageNet-1K pruning results with SOTA methods on ResNet18.Our method outperforms other SOTA methods at various pruning ratios and different IPCs. More importantly, our method consistently exceeds the unpruned version of SRe 2 L with 30× less storage. Such a result is not impressive at first glance; however, when considering the actual storage, the storage is reduced from 29G to 0.87G. In addition, we notice the performance at 10× (or 90%) pruning ratio degrades slightly, especially for large IPCs. For example, merely 0.2% performance degradation on IPC200 using ResNet18. Pruning results of larger IPCs can be found in Appendix D.2. Label Pruning Metrics. From Table 5a , we empirically find that using different metrics explained in Appendix E.1 is no better than random pruning. In addition, as mentioned in FerKD [25] , calibrating the searching space by discarding a portion of easy or hard images can be beneficial. We conduct a similar experiment to perform random pruning on a calibrated label pool, and the metric for determining easy or hard images is "confidence". However, as shown in Table 5b , no such range can consistently outperform the non-calibrated ones (last row). An interesting observation is that the label pruning law at large pruning ratio seems to coincide partially with data pruning, where removing hard labels becomes beneficial [37] .Generalization. Table 6a shows the performance under large compression rates. Smaller IPC datasets suffer more from label pruning since it requires more augmentation and soft label pairs to boost data diversity. Furthermore, label pruning results on ResNet50 are provided in Table 6b . Not only scaling to large networks of the same family (i.e., ResNet) but Table 6c also demonstrates the generalization capability of the proposed method across different network architectures. An analogous trend is evident in the context of label pruning: comparable performance is achieved with 10× fewer labels. This reinforces the statement that the necessity for extensive augmentations and labels can be significantly reduced if the dataset exhibits sufficient diversity.Large Dataset. ImageNet-21K-P has 10,450 classes, significantly increasing the disk storage as each soft label stores a probability of 10,450 classes. The IPC20 dataset leads to a 1.2 TB (i.e., 1285 GB) label storage, making the existing framework less practical. However, with the help of our method, it can surpass SRe 2 L [6] by a large margin despite using 40× less storage. For example, we attain an 8.9% accuracy improvement on IPC20 with label storage reduced from 1285 GB to 32 GB.Pruning for Optimization-Free Approach. RDED [8] is an optimization-free approach during the "recover" phase. However, extensive labels are still required for post-evaluation. To prune labels, consistent improvements are observed using the improved label pool, as shown in Table 8 . Comparison with G-VBSM [9] . Compared to G-VBSM [9] , which uses an ensemble of 4 models to recover and relabel, our method outperforms it at various pruning ratios with only a single model (see Table 9 ). Furthermore, the techniques used for G-VBSM apply to our method. By adopting label generation with ensemble and a loss function of "MSE+0.1 × GT" [9] , our method can be further improved by a large margin on IPC10 of ImageNet-1K, using ResNet18. Implementation details can be found in Appendix C.4.Visualization. Fig. 2b visualizes our method on three classes. More visualizations are provided in Appendix F.Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)? 
conclusion: 
 To answer the question "whether large-scale soft labels are necessary for large-scale dataset distillation?", we conduct diversity analysis on synthetic datasets. The high within-class similarity is observed and necessitates large-scale soft labels. Our LPLD method re-batches images within classes and introduces class-wise BN supervision during the image synthesis phase to address this issue. These changes improve data diversity, so that simple random label pruning can perform on par with complex rule-based pruning metrics. Additionally, we randomly conduct pruning on an improved label pool. Finally, LPLD is validated by extensive experiments, serving a strong baseline that takes into account actual storage. Limitations and future works are provided in Appendix E.6. The ethics statement and broader impacts can be found in Appendix E.7.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes]Justification: [TODO]Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: [TODO]Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes]Justification: [TODO]Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes]Justification: [TODO]Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: [TODO]Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [Yes]Justification: [TODO]Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA]Justification: [TODO]Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
: 
 Figure 1 : The relationship between performance and total storage of auxiliary information needed. Our method achieves SOTA performance with fewer soft labels than images.We are pacing into the era of ImageNet-level condensation, and the previous works [1, 2, 3, 4, 5] fail in scaling up to large-scale datasets due to extensive memory constraint. Until recently, Yin et al. [6] decouple the traditional distillation scheme into three phases. First, a teacher model is pretrained with full datasets (squeeze phase). Second, images are synthesized by matching the Batch Normalization (BN) statistics from the teacher and student models (recover phase). Third, auxiliary data such as soft labels are pregenerated from different image augmentations to create abundant supervision for posttraining (relabel phase).However, the auxiliary data are 30× larger than the distilled data in ImageNet-1K. To attain correct and effective supervision, the exact augmentations and soft labels of every training epoch are stored [6, 7, 8, 9, 10] . The required soft label storage is the colored circles in Fig. 1 .In this paper, we consider whether large-scale soft labels are necessary, and what causes the excessive requirement of these labels? To answer these questions, we provide an analysis of the distilled images using SRe 2 L [7] , and we find that within-class diversity is at stake as shown in Fig. 2 . To be more precise, we analyze the similarity using Feature Cosine Similarity and Maximum Mean Discrepancy in Sec. 3.2. The high similarity of images within the same class requires extensive data augmentation to provide different supervision.To address this issue, we propose Label Pruning for Large-scale Distillation (LPLD). Specifically, we modified the algorithms by batching images within the same class, leveraging the fact that different classes are naturally independent. Furthermore, we introduce class-wise supervision to align our changes. In addition, we have explored different label pruning metrics and found that simple random pruning was performed on par with carefully selected labels. To further increase diversity, we improve the label pool by introducing randomness in a finer granularity (i.e., batch-level). Our method effectively distills the images while requiring less label storage compared to image storage, as shown in Fig. 1 .The key contributions of this work are: (1) To the best of our knowledge, it is the first work to introduce label pruning to large-scale dataset distillation. (2) We discover that high within-class diversity necessitates large-scale soft labels. (3) We re-batch images and introduce class-wise supervision to improve data diversity, allowing random label pruning to be effective with an improved label pool. (4) Our LPLD method achieves SOTA performance using a lot less label storage, and it is validated with extensive experiments on various networks (e.g., ResNet, EfficientNet, MobileNet, and Swin-V2) and datasets (e.g., Tiny-ImageNet, ImageNet-1K, and ImageNet-21K). 
 
 Appendix: We aim to determine a lower bound on the number of batches (updates) n required to ensure that the Batch Normalization (BN) statistics for each class in the ImageNet dataset converge within a specified tolerance τ , with high probability. The dataset has a varying number of images per class, affecting the probability of each class appearing in a batch during sampling.A Probability of Class Appearance in a Batch: When sampling a batch of size B, the probability that class c does not appear in the batch is (1 -p c ) B . Therefore, the probability that class c appears in the batch is:q c = 1 -(1 -p c ) B .This represents the likelihood that at least one image from class c is included in a given batch.Let n be the total number of batches sampled during training.We assume that batches are sampled independently with replacement from the dataset. Under this assumption, each batch is an independent trial where class c appears with probability q c . Therefore, the number of batches M where class c appears follows a binomial distribution:M ∼ Binomial(n, q c ).Remark: In practice, batches are often sampled without replacement within an epoch, introducing dependency between batches. However, for large datasets where the total number of images N is significantly larger than the batch size B and the number of batches n, the dependence becomes negligible. In such cases, the binomial distribution serves as a reasonable approximation.The expected value of M is:E[M ] = nq c .To ensure that M is not significantly less than its expected value E[M ], we apply the Chernoff bound:Pr (M ≤ (1 -δ)E[M ]) ≤ exp - δ 2 E[M ] 2 ,where δ ∈ (0, 1) represents the acceptable relative deviation from the expected value. This bound provides a way to quantify the probability that a random variable deviates from its expected value, which is crucial for making high-confidence guarantees.To ensure the probability that M is less than(1 -δ)E[M ] is at most T 1 , we set: exp - δ 2 nq c 2 ≤ T 1 .Solving for n:n ≥ -2 ln(T 1 ) δ 2 q c .To ensure this condition holds for all classes, we use the minimum value of q c : n ≥ -2 ln(T 1 ) δ 2 min(q c ) .BN Statistics Update: Batch Normalization updates its running statistics using an exponential moving average. The update rule for the BN statistics of class c at iteration t + 1 is:BN t+1 c = (1 -ε)BN t c + ε BN t+1 c ,where ε is the momentum parameter, andBN t+1 cis the BN statistics estimated from the current batch for class c.Since BN statistics for class c are updated only when class c appears in the batch, we consider only the updates corresponding to those batches. After M such updates:BN t+M c = (1 -ε) M BN t c + ε M k=1 (1 -ε) M -k BN k c ,where BN k c is the BN statistics estimated in the k-th batch containing class c.Assuming BN k c is an unbiased estimator of the true BN statistics µ c when class c appears, the expected value is:E[BN t+M c ] = (1 -ε) M BN t c + µ c 1 -(1 -ε) M .Convergence Within Tolerance: To ensure that the BN statistics converge within a tolerance τ of the true statistics µ c :E[BN t+M c ] -µ c ≤ τ.Since:E[BN t+M c ] -µ c = (1 -ε) M BN t c -µ c ,and assuming BN t c -µ c ≤ C for some constant C, we have:(1 -ε) M C ≤ τ.Taking natural logarithms:M ln(1 -ε) + ln(C) ≤ ln(τ ).Using the approximation ln(1 -ε) ≈ -ε for small ε:-M ε + ln(C) ≤ ln(τ ).Solving for M :M ≥ M 0 = ln C τ ε .Origin of M 0 : Here, M 0 is derived from the BN convergence requirement that ensures:(1 -ε) M0 BN t c -µ c ≤ τ.It represents the minimum number of updates required for the BN statistics of class c to converge within the desired tolerance τ .Event Definitions:• Let E 1 be the event that class c appears in sufficient batches (as guaranteed by the Chernoff bound).• Let E 2 be the event that the BN statistics for class c converge within the desired tolerance τ .Target Probability: We aim to ensure that both events occur simultaneously with high probability:P (E 1 ∩ E 2 ) ≥ 1 -T.Union Bound Application: For any two events, the probability of their intersection satisfies:P (E 1 ∩ E 2 ) = 1 -P (E 1 ∩ E 2 ) = 1 -P (E 1 ∪ E 2 ) ≥ 1 -P (E 1 ) -P (E 2 ).Error Probability Allocation: For simplicity, we allocate the total acceptable failure probability T equally between the two events:P (E 1 ) ≤ T 2(allocated to Chernoff bound),P (E 2 ) ≤ T 2 (allocated to BN convergence).Chernoff Bound Analysis: For event E 1 , we require that the probability of class c appearing in fewer than the expected number of batches is at most T 2 :P (M ≤ (1 -δ)nq c ) ≤ T 2 .Applying the Chernoff bound:exp - δ 2 nq c 2 ≤ T 2 .Solving for n:- δ 2 nq c 2 ≤ ln T 2 , n ≥ -2 ln T 2 δ 2 q c .BN Convergence Requirement: For event E 2 , we require that the number of batches M where class c appears is sufficient for BN convergence:M ≥ M 0 = ln C τ ε .To ensure that this condition holds when event E 1 occurs, we use the fact that, with probability at least 1 -T 2 , we have:M ≥ (1 -δ)nq c .Therefore, to guarantee M ≥ M 0 , we require:(1 -δ)nq c ≥ M 0 = ln C τ ε .Solving for n:n ≥ ln C τ (1 -δ)εq c .Final Combined Bound: To ensure that both conditions hold for all classes, we use min(q c ):n ≥ max      -2 ln T 2 δ 2 min(q c ) Chernoff Bound , ln C τ (1 -δ)ε min(q c ) BN Convergence     , where δ represents the acceptable relative deviation from the expected number of batches, ε is the momentum parameter in BN updates, T denotes the acceptable total failure probability (T = T 1 +T 2 ), τ is the convergence threshold for BN statistics, C represents an upper bound on BN t c -µ c at initialization, and min(q c ) represents the minimum probability that a class appears in a batch.• With probability at least 1 -T 2 , each class c appears in at least (1 -δ)nq c batches (event E 1 occurs).• With probability at least 1 -T 2 , the BN statistics for each class c converge within tolerance τ (event E 2 occurs).• By the union bound, both events E 1 and E 2 occur simultaneously with probability at least 1 -T .We perform experiments on the following three datasets:• Tiny-ImageNet [33] is the subset of ImageNet-1K containing 500 images per class of a total of 200 classes, and spatial sizes of images are downsampled to 64 × 64. • ImageNet-1K [34] We use Pytorch pretrained ResNet-18 [36] , with a Top-1 accuracy of 69.76%, as both the recovery and relabeling model. Class-wise BN statistics are computed using a modified version of the training script of the source provided in Table 10 . The recovery, or data synthesis, phase is provided in Table 11 , which follows CDA [7] except by changing the batch size to an IPC-dependent size. Relabel and validation processes share the same setting as provided in Table 12 . 18 , respectively. Note that CutMix used in ImageNet-1K is replaced with CutOut [42] , and a relatively large label smooth of 0.2 is used during the ImageNet-21K-P pretraining phase. We incorporate the same changes to the relabel/validation phase of the synthetic dataset.RDED. RDED [8] has several different changes to the SRe 2 L settings. (1) The batch size is adjusted according to the IPC size (i.e., 100 for IPC10 and 200 for IPC50). ( 2) It uses additional augmentation (i.e., ShufflePatch to shuffle the position of patches). Such augmentations are considered additional storage since the exact order of patch shuffling needs to be stored. (3) Weaker augmentation (i.e., a larger lower bound for the random area of the resized crop). ( 4) A smoothed learning rate scheduler.We adhere to all the changes for experiments regarding RDED.G-VBSM. In Table 9 , we adopt the several techniques used for G-VBSM [9] . (1) Soft labels are generated with an ensemble of models. Specifically, we use ResNet18 [36] , MobileNetV2 [39] , EfficientNet-B0 [43] , ShuffleNetV2-0.5 [44] . 20 demonstrates that our method exhibits commendable scalability across large IPCs. We observe non-marginal enhancements when deploying even larger IPCs, such as IPC300 and IPC400. Moreover, our approach achieves nearly identical accuracy levels -specifically, 65.3% vs. 65.2% -when comparing the use of IPC300 at 1× with IPC400 at 30× less labels. Compared to the full ImageNet-1K dataset, our method preserves a large portion of the accuracy with 10× less storage. This performance is achieved despite the vastly different storage requirements of 178G and 13G, respectively, indicating a higher flexibility of IPC choice with a fixed storage budget.The label quantization technique mentioned in Fast Knowledge Distillation (FKD) [25] is orthogonal to the proposed method for several reasons. Firstly, as demonstrated in Table 21 , there are six components related to soft labels. FKD only compresses the prediction logits (component 6), while the our method addresses all six components.Secondly, even for the overlapping storage component (component 6: prediction logits), the compression targets differ between FKD and our method, as shown in Table 22 . The total stored prediction logits can be approximated by the formula: number_of_condensed_images × number_of_augmentations × dimension_of_logits. FKD's label quantization focuses on compressing the dimension_of_logits, whereas the proposed label pruning method focuses on compressing the number_of_augmentations. Although FKD's approach is orthogonal to our method, a comparative analysis was conducted to better understand their relative performance. Table 23 presents a detailed comparison between FKD's two label quantization strategies (Marginal Smoothing and Marginal Re-Norm) and the proposed method. It is important to note that FKD only compresses component 6, with the compression rate related to hyper-parameter K. Components 1-5 remain uncompressed (1× rate) in FKD. Additionally, FKD's quantized logits store both values and indices, so their actual storage is doubled, and their compression rate is halved. This analysis has yielded two key observations. First, our method demonstrates higher accuracy at comparable compression rates. For IPC10, our method achieves 32.70% accuracy at 10× compression, while FKD only reaches 18.10% at 8.2× compression. Second, our method exhibits better compression at similar accuracy levels. On IPC10, our method attains 20.20% accuracy at 40× compression, whereas FKD achieves 19.04% at just 4.5x compression. We determine labels according to the statistics of the auxiliary information:1. correct: the number of correctly classified images [31] 2. diff: the absolute difference between the Top-2 outputs 3. signed_diff: the signed difference between Top-2 output [45] 4. cut_ratio: the cut-mix ratio 5. confidence: the value of the largest output [26] .These metrics serve for the baselines compared to random label pruning in Table 5 After knowing the metric, knowing which data type to prune (i.e., "easy", "hard", or "uniform") is important. Additionally, FerKD [26] argues the reliability of generated soft labels and proposes to use neither too easy nor too hard samples. Our experiments are grounded in a careful analysis of the number of updates required for stable Batch Normalization (BN) statistics. We begin by examining the derived bound from Eq. 8:n ≥ max      -2 ln T 2 δ 2 min(q c ) Chernoff Bound , ln C τ (1 -δ)ε min(q c ) BN Convergence      .To evaluate this bound, we substitute the following values:• T = 0.05 (acceptable total failure probability, corresponding to 95% confidence) • δ = 0.2 (acceptable relative deviation from the expected number of batches) • ε = 0.1 (momentum parameter in BN)• min(p c ) = 732 1, 281, 167 ≈ 0.0005711 (ratio of the least number of images in a class to total images) • B = 256 (batch size) • min(q c ) = 1 -(1 -min(p c )) B (minimum probability that any class appears in a batch) First, we compute min(q c ):min(q c ) = 1 -(1 -min(p c )) B = 1 -(1 -0.0005711) 256 = 1 -(0.9994289) 256 ≈ 1 -e -256×0.0005711 (since min(p c ) is small) = 1 -e -0.1462 ≈ 1 -0.8639 = 0.1361.Thus, min(q c ) ≈ 0.1361.Next, we compute the two parts of the bound separately.From Chernoff Bound Term: Given that we allocate the total failure probability T equally between the two events, we have T /2 = 0.025.n ≥ -2 ln T 2 δ 2 min(q c ) = -2 ln(0.025) (0.2) 2 × 0.1361 = -2 × (-3.6889) 0.04 × 0.1361 (since ln(0.025) = -3.6889) = 7.3778 0.005444 ≈ 1, 355.2.From BN Convergence Term: We need to specify C and τ . Let's assume:• C = 1 (an upper bound on BN t c -µ c at initialization, as the running mean is typically initialized to zero) • τ = 0.01 (desired convergence tolerance) Compute the numerator:ln C τ = ln 1 0.01 = ln(100) = 4.6052.Now, compute the denominator:(1 -δ)ε min(q c ) = (1 -0.2) × 0.1 × 0.1361 = 0.8 × 0.1 × 0.1361 = 0.010888.Compute the second part: The additional storage allocation for class-specific statistics is detailed in Table 25 . It is observed that this storage requirement escalates with an increase in the number of classes. However, this is a one-time necessity during the recovery phase and becomes redundant once the synthetic data generation is completed.n ≥ ln C τ (1 -δ)ε min(q c ) = 4.Experiments are performed on 4 A100 80G GPU cards. We notice that for Tiny-ImageNet, there is a slight performance drop when multiple GPU cards are used with DataParallel in PyTorch. Therefore, we use 4 GPU cards for ImageNet-1K and ImageNet-21K-P experiments and 1 GPU card for all Tiny-ImageNet experiments.We recognize that there are several limitations and potential areas for further investigation. Firstly, while our work significantly reduces the required storage, the process for generating the soft labels is still necessary, as we randomly select from this label space. Secondly, reducing the required labels may not directly lead to a reduced training speed, as the total training epochs remain the same in order to achieve the best performance. Future work is warranted to reduce label storage as well as the required training budget simultaneously.Our research study focuses on dataset distillation, which aims to preserve data privacy and reduce computing costs by generating small synthetic datasets that have no direct connection to real datasets. However, this approach does not usually generate datasets with the same level of accuracy as the full datasets.In addition, our work in reducing the size of soft labels and enhancing image diversity can have a positive impact on the field by making large-scale dataset distillation more efficient, thereby reducing storage and computational requirements. This efficiency can facilitate broader access to advanced machine learning techniques, potentially fostering innovation across diverse sectors.In this section, we present visualizations of the datasets used in our experiments. Due to the different matching objectives, datasets of different IPCs should have distinct images. Therefore, we provide the visualization of different IPCs. Figure 6 shows randomly sampled images from ImageNet-1K at various IPC. Figure 7 depicts the Tiny-ImageNet dataset with IPC50 and IPC100. Figure 8 provides visualizations of ImageNet-21K-P at IPC10 and IPC20. • The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.Guidelines:• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. , with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 


Decision: This paper received all positive recommendations.  Reviewers acknowledged the motivation of this paper,  the thorough analysis, and the significant results. Most questions were well addressed during the rebuttal and discussion. The AC agrees with the acceptance recommendation.

Reviews: review: paper_summary: paper_summary: This paper discovers that the conventional method generates images with high similarity. To solve this, the authors introduce class-wise supervision during the image-synthesizing process by batching the samples within classes. Thanks to the increase in diversity, the soft labels can be pruned to reduce the storage size. Extensive experiments are performed on ImageNet with different compression ratios of labels to validate the effectiveness of the method.
strength_weakness: strength_weakness: 1. There are confusing sentences. The authors mention that ”The high similarity of images within the same class requires extensive data augmentation to provide different supervision” and then “To address this issue, we propose Label Pruning for Large-scale Distillation”. How does label pruning address the issue? I see that increasing the diversity of synthetic samples does improve the performance and label pruning is proposed to reduce the storage size. These are not highly related.
2. The authors should compare the performance of the proposed Label pruning with other compression methods (e.g., Marginal Smoothing/Re-Norm with Top-K, using different K when targeting at 10x/20x…) proposed in FKD. Without comparing these methods, it is unclear whether the proposed approach is better than previous pruning methods.


FKD: A Fast Knowledge Distillation Framework for Visual Recognition, Zhiqiang Shen et al
questions: questions: See weaknesses
limitations: limitations: N/A
paper_summary: paper_summary: The paper focuses on reducing the size of soft-label storage in large-scale dataset condensation. The authors discussed why the labels are large-scaled and then proposed to prune the labels by increasing the diversity of synthetic images. Extensive experiments are conducted to validate the effectiveness of the proposed method.
strength_weakness: strength_weakness: The motivation of the work is well-grounded. I don’t observe major drawbacks but do have some minor concerns towards its technical contributions.

1. The proposed method seems to be a class-wise version of SRe^2L, with a soft label-reuse mechanism during training. Thus the technical contributions seem to be weak.
2. How is Equation (8) used in the proposed method? Or is it just a theoretical result?
3. I have doubts on the computational cost (memory and time) of the proposed method (not storage), doing class-wise optimization may increase the distillation time, especially on ImageNet.
4. How is the random label pruning done? E.g. Is the same number of soft-labels removed for each epoch?
5. Proposition 2 is quite confusing, is “higher MMD” a typo?
questions: questions: Please see Weaknesses.
limitations: limitations: Yes, the limitations are addressed.
paper_summary: paper_summary: This paper addresses the challenge of large-scale dataset distillation, specifically focusing on reducing the storage requirements for auxiliary soft labels in ImageNet-level condensation. The authors propose Label Pruning for Large-scale Distillation (LPLD), which aims to achieve state-of-the-art performance while significantly reducing the storage needed for soft labels. The main contributions include:
1. Identifying that high within-class similarity in condensed datasets necessitates large-scale soft labels.
2. Introducing class-wise supervision during image synthesis to increase within-class diversity.
3. Demonstrating that simple random pruning of soft labels is effective when data diversity is improved.
4. Achieving SOTA performance with significantly reduced label storage, validated across various networks and datasets.
strength_weakness: strength_weakness: Limited theoretical analysis: While the paper provides comprehensive empirical evidence, a more rigorous theoretical analysis could strengthen it. I know this is a common challenge in dataset distillation research. However, the detailed empirical study presented here lays valuable groundwork for potential future theoretical analyses in this direction, particularly regarding the relationship between class-wise supervision and increased diversity.
questions: questions: 1. How sensitive is the method to the choice of hyperparameters, particularly in the label pruning process? Is there a way to automatically determine the optimal pruning rate? 

2. Have you explored the potential of combining your class-wise supervision approach with other dataset distillation techniques? Could this lead to further improvements? 

3. How does the performance of your method scale with the number of classes? Is there a point where the benefits of class-wise supervision diminish? 

4. Have you investigated the impact of your method on the training time of the distilled dataset compared to previous approaches?

5. In Table 6(c), why the performance on Swin-V2-Tiny is the worst even if the architecture has the largest size (28.4M)?
limitations: limitations: Yes. Limitations are provided in Appendix E.5.

Comments: title: Thank you for raising the score
Thank you for your positive feedback
Grateful for any feedback
Grateful for any feedback
Thank you for the acceptance
Thank you for the response
After rebuttal
The reasons why label quantization in FKD is orthogonal to our method
comment: Thank you for your strong support. We greatly appreciate your recognition of our work's significance to the field. Your opinion is extremely important to us, and we are deeply grateful.
Thanks for your follow-up. After careful consideration, I have decided to raise my score and strongly support the acceptance of this paper. I believe this work is significant to the field.
Thank you for your prompt response and for reviewing our additional experiments. We're delighted that our efforts have addressed most of your concerns and appreciate your decision to increase the score.

Your feedback has been invaluable in improving our work. If you have any further questions, please don't hesitate to ask.
Thank you again for your time and expertise.
Thanks for providing additional experiments using label quantization proposed in FKD. Most of my concerns have been addressed. Thus, I increase the score.
Thank you sincerely for your positive feedback on our rebuttal. We find it very encouraging that our paper is considered high-quality and that our soft label compression technique is recognized as a significant contribution to the field.

As we are approaching the discussion deadline, we wanted to provide a brief update: one reviewer has already raised their score after reading the rebuttal, while another has not yet responded. Given your intention to adjust the score and your positive assessment, we were wondering if you might consider updating your score at this time. Your support could be crucial in the final evaluation of our paper. 

We deeply appreciate your time and expertise throughout this review process. Thank you again for your thoughtful consideration.
Dear Reviewer S5vb,

I apologize for reaching out again after my previous message. With the discussion deadline now just one day away, I wanted to very gently follow up on our earlier correspondence regarding our paper and rebuttal.

We completely understand if you haven't had the opportunity to review our rebuttal yet, given the many demands on your time. However, if you have had the chance to look it over, we would be immensely grateful for any feedback you could provide. Your insights, no matter how concise, would be valuable in helping us engage in a constructive discussion before the deadline closes.

We are sincerely thankful for your continued involvement in this process, and we respect your time and professional commitments. Thank you for your understanding and consideration.
Dear Reviewer S5vb,

We greatly appreciate the time and effort you've invested in reviewing our work. As we are now two days away from the discussion deadline, we wanted to reach out regarding our rebuttal, which we submitted earlier.

If possible, we would be grateful for any feedback you might be able to provide, as this would give us the opportunity to engage in a productive discussion before the deadline. We're eager to address any remaining questions or concerns you may have about our paper.

We understand that you likely have a busy schedule, and we truly appreciate your valuable insights and expertise in this process.

Thank you for your time and consideration.
We are deeply grateful for your acceptance of our paper and truly appreciate your insightful feedback. Your comment that our discussion "will be insightful for the community" is particularly encouraging to us.

We are committed to improving our manuscript based on your thoughtful suggestions.
Thank you for the detailed response and my concerns are addressed. I now vote for acceptance of this paper. However, I strongly recommend the authors to include the discussions on time cost and detailed pruning process into the main paper/appendix. These will be insightful for the community.
Thanks for providing the detailed rebuttal. All of my concerns have been properly addressed. In addition, I have also read the other reviews’ comments and corresponding rebuttals. These efforts are appreciated.

After careful consideration, I believe this is a high-quality paper that merits acceptance. The proposed soft label compression technique is valuable and contributes significantly to the field.

I will adjust my score accordingly once the discussions between other reviewers and the authors have concluded.
Thank you for bringing up the questions related to FKD.

We would like to emphasize that label quantization mentioned in FKD is orthogonal to our method for the following reasons:

1. **We consider more** **components** **as shown in Table B.** There are **six** **components** related to soft labels. FKD only compresses the prediction logits (component 6), while our method addresses all six components.
2. **We have different compression targets, as shown in Table D**: Even for the overlapping storage component (component 6: prediction logits), our compression target differs from FKD's. The total stored prediction logits can be approximated by the `number_of_condensed_images × number_of_augmentations × dimension_of_logits`.

- FKD's **Label Quantization** focuses on compressing the `dimension_of_logits`.
- Our **Label Pruning** method focuses on compressing the `number_of_augmentations`.

**Table** **B**: Different storage components between FKD and ours. FKD, originally for **model** distillation, requires storage only for components 1, 2, 6. Adapting it to **dataset** distillation requires additional storage for components 3, 4, 5.
|  | FKD | Ours |
|---|---|---|
| 1. coordinates of crops | ❌ | ✔ |
| 2. flip status | ❌ | ✔ |
| 3. index of cutmix images | ❌ | ✔ |
| 4. strength of cutmix | ❌ | ✔ |
| 5. coordinates of cutmix bounding box | ❌ | ✔ |
| 6. prediction logits | ✔ | ✔ |

**Table D:** Breakdown explanation for component 6 (prediction logits) storage between FKD’s label quantization and our label pruning. The number of condensed images is computed by `N = IPC x number_of_classes`. FKD's compression target is `dimension_of_logits`, while our target is `number_of_augmentations`.

|                           | Number of condensed images | Number  of augmentations per image | Dimension of logits per augmentation | Total storage for prediction logits |
| ------------------------- | ------------------------ | ------------------------------- | ------------------------------- | ----------------------------------- |
| Baseline (no compression) | N                        | 300                             | 1,000                           | N $\times$ 300 $\times$ 1000        |
| Label Quantization (FKD)  | N                        | 300                             | 10                              | N $\times$ 300 $\times$ 10          |
| Label Pruning (Ours)      | N                        | 3                               | 1,000                           | N $\times$ 3 $\times$ 1000          |