DRUPI: Dataset Reduction Using Privileged Information

## Abstract

Abstract Dataset reduction (DR) seeks to select or distill samples from large datasets into smaller subsets while preserving performance on target tasks. Existing methods primarily focus on pruning or synthesizing data in the same format as the original dataset, typically the input data and corresponding labels. However, in DR settings, we find it is possible to synthesize more information beyond the data-label pair as an additional learning target to facilitate model training. In this paper, we introduce Dataset Reduction Using Privileged Information (DRUPI), which enriches DR by synthesizing privileged information alongside the reduced dataset. This privileged information can take the form of feature labels or attention labels, providing auxiliary supervision to improve model learning. Our findings reveal that effective feature labels must balance between being overly discriminative and excessively diverse, with a moderate level proving optimal for improving the reduced dataset’s efficacy. Extensive experiments on ImageNet, CIFAR-10/100, and Tiny ImageNet demonstrate that DRUPI integrates seamlessly with existing dataset reduction methods, offering significant performance gains. The code will be released after the paper is accepted.

## 1 Introduction

Dataset Reduction (DR) has attracted considerable attention in recent years, with the primary aim of compressing large datasets into smaller subsets while maintaining comparable statistical performance. Existing methods for DR can be broadly classified into two main categories: coreset selection and dataset distillation. Coreset selection methods focus on selecting a subset of samples from the original dataset (Har-Peled & Mazumdar, ; Welling, ; Toneva et al., ), while dataset distillation involves synthesizing unseen samples from the dataset (Wang et al., ; Zhao et al., ; Zhao & Bilen, ; Cazenavette et al., ; Wang et al., ).

In typical real-world scenarios, training models for target tasks is generally constrained to input data (*e.g.*, images) and their corresponding labels, as these are the most readily available elements. While existing DR methods have shown strong performance (Wang et al., ; Zhao et al., ; Zhao & Bilen, ; Cazenavette et al., ; Yin et al., ), they typically do so by compressing datasets in the same or similar format, such as the conventional data-label structure. Even advanced dataset distillation techniques, which re-parameterize images or labels to create alternative representations (Kim et al., ; Zhao et al., ; Deng & Russakovsky, ; Liu et al., ; Wei et al., ), are limited by this conventional framework. As illustrated in Figure 1(a), this reliance on fixed data-label structures restricts the capacity of such methods to incorporate richer information that could further enhance model training and improve generalization.

In fact, DR settings offer the potential to create more diverse compressed datasets that extend beyond simple input data $x_{i}$ and labels $y_{i}$, incorporating richer forms of information. A notable example is the concept of privileged information, first introduced in the context of statistical learning (Vapnik & Vashist, ; Pechyony & Vapnik, ). Figure 2 provides a illustration for privileged information. Let us consider a more concrete example, where $x_{i}$ might represent a biopsy image, and the privileged information $f_{i}^{\star}$ for $x_{i}$ could be the oncologist’s written assessment of the image (Vapnik et al., ). The label $y_{i}$ would then indicate whether the tissue in the image is malignant or benign. By leveraging this privileged information $f_{i}^{\star}$, a medical expert can make more informed decisions, benefiting from additional insights that improve diagnostic accuracy.

However, none of the existing methods compress the original dataset beyond the traditional data-label structure or synthesize privileged information for auxiliary supervision. To address this gap, we introduce a novel approach that, for the first time, incorporates not only images and labels but also privileged information. Our method, called Dataset Reduction Using Privileged Information (DRUPI), is illustrated in Figure 1(b). We primarily synthesize feature labels for the reduced dataset, as these labels capture richer, high-dimensional information, enhancing dataset quality. These feature labels generalize effectively across various neural network architectures and provide a unified representation of latent statistics across multiple models, offering additional supervision during training. Additionally, we propose a more efficient form of feature labels, *i.e.*, attention labels. As shown in Figure 1(c), the incorporation of privileged information produces gradients more aligned with those of the original dataset, ultimately improving the model’s generalization capabilities.

Figure: Figure 1: A comparison between conventional dataset reduction pipelines and our proposed DRUPI framework. (a) Previous dataset reduction methods distill or select a subset $\mathcal{D}_{\mathcal{S}}$ from the original dataset $\mathcal{D}_{\mathcal{T}}$, maintaining the original “data-label” structure. (b) In contrast, DRUPI synthesizes auxiliary privileged information from $\mathcal{D}_{\mathcal{T}}$, enriching further supervision to models trained on the reduced subset $\mathcal{D}_{\mathcal{S}}$. (c) Cosine similarity between the gradients of a pre-trained model on synthetic datasets w/ and w/o privileged information (feature labels) and the real dataset. Synthetic datasets are generated using DC with 10 IPC. We used the same pre-trained ConvNet for gradient extraction.
Refer to caption: x1.png

A key challenge for DRUPI lies in determining appropriate feature labels to synthesize for the reduced dataset. To address this, we employ an arbitrary methods of dataset distillation to synthesize the feature labels. During each step of the bi-level optimization, we match the statistical information of models trained on reduced datasets with and without feature labels. Furthermore, we observed that synthesized feature labels cannot be overly discriminative or diverse, which degrade the overall quality of the reduced dataset. This finding suggests that an optimal balance between discriminability and diversity is crucial for synthesizing effective feature labels. Our contributions are summarized as follows:

- 1.
We propose a new paradigm, *i.e.*, DRUPI, for dataset reduction. In particular, privileged information, such as feature labels, can be synthesized in addition to traditional data-label pairs. This privileged information provides additional generative supervision during model training, thereby improving the generalization ability of the reduced dataset.
- 2.
We observe that effective feature labels should balance the trade-off between diversity and discriminability. Overly discriminative feature labels, such as those directly extracted from a pre-trained neural network, can even degrade the quality of the reduced dataset.
- 3.
We further provide a theoretical analysis of our DRUPI pipeline based on VC theory (Vapnik, ) from statistical learning, which rigorously ensures its effectiveness.
- 4.
Our experiments demonstrate that DRUPI can be seamlessly integrated into state-of-the-art DR methods. Particularly, for coreset selection methods, applying DRUPI to Herding on CIFAR10 (with a fraction of 0.4%) improves performance by 24.3%, while applying it to K-center in cross-architecture evaluations leads to an improvement of up to 23.4%. For dataset distillation methods, integrating DRUPI with the DC method on CIFAR100 (with 10 images per class) yields a 4% improvement, and further cross-architecture evaluations of DC show gains of up to 18.3%.

## 2 Background and Related Work

Similar to prior works on dataset reduction (DR), we focus on classification tasks (Welling, ; Zhao et al., ; Shin et al., ). Consider a multi-classification problem. Let $\mathcal{X}\in\mathbb{R}^{d}$ represent the input space and $\mathcal{Y}$ denote the set of possible labels. Our dataset, $\mathcal{D}_{\mathcal{T}}=\{(x_{i},y_{i})\}_{i=1}^{n}\subseteq\mathcal{X}
\times\mathcal{Y}$, consists of $n$ training samples, where each $x_{i}\in\mathcal{X}$ is an input vector and $y_{i}\in\mathcal{Y}$ is its corresponding label. In typical DR settings, the goal is to obtain a smaller dataset, $\mathcal{D}_{\mathcal{S}}$ with size $m$, where $m\ll n$. DR methods are typically divided into two categories: coreset selection, where the reduced dataset is a subset of the original, and dataset distillation, where the reduced dataset consists of synthesized data not present in the original set but learned through optimization.

Coreset selection techniques. Coreset selection techniques are designed to identify a representative subset $\mathcal{D}_{\mathcal{S}}$ from the complete dataset $\mathcal{D}_{\mathcal{T}}$. This process typically seeks to optimize a criterion that quantifies the informativeness of $\mathcal{D}_{\mathcal{S}}$, which matches that of the original dataset $\mathcal{D}_{\mathcal{T}}$. The informativeness can be gauged by various metrics, including gradients (Paul et al., ; Mirzasoleiman et al., ; Killamsetty et al., ), loss values (Toneva et al., ), predictive uncertainties (Coleman et al., ), proximity to decision boundaries (Ducoffe & Precioso, ; Margatina et al., ), and the sharpness of the learned model (Shin et al., ).

Dataset distillation methods.
Dataset distillation approaches offer an alternative strategy by focusing on synthesizing a distilled dataset $\mathcal{D}_{\mathcal{S}}$ from the original dataset $\mathcal{D}_{\mathcal{T}}$, rather than directly selecting a subset. This is typically accomplished through a bi-level optimization process that aligns the performance of $\mathcal{D}_{\mathcal{S}}$ with that of $\mathcal{D}_{\mathcal{T}}$. A distance metric $\mathbf{D}$ is employed to quantify the statistical divergence between datasets, guiding the learning of $\mathcal{D}_{\mathcal{S}}$ via gradient descent. Specifically, the distilled dataset is updated as follows: $\mathcal{D}_{\mathcal{S}}\leftarrow\mathcal{D}_{\mathcal{S}}-\eta\cdot\nabla_{
\mathcal{D}_{\mathcal{S}}}\mathbf{D}(\mathcal{D}_{\mathcal{S}},\mathcal{D}_{
\mathcal{T}})$. The choice of distance metric $\mathbf{D}$ is versatile and can encompass various aspects such as gradients (Zhao et al., ; Lee et al., ; Zhao & Bilen, ), feature representations (Zhao & Bilen, ; Sajedi et al., ), training trajectories (Cazenavette et al., ; Du et al., ; Cui et al., ; Guo et al., ), and kernel information (Nguyen et al., ; ; Zhou et al., ).

In addition to direct performance matching, certain methodologies endeavor to re-parameterize input data to enhance compression efficiency. Techniques employed in this context include exploiting data regularity, as discussed in various studies (Kim et al., ; Zhao et al., ; Son et al., ), factorizing images to capture intrinsic structures (Liu et al., ; Deng & Russakovsky, ), and employing sparse coding to represent data effectively (Wei et al., ).

## 3 Method

Figure: Figure 2: Comparison between (a) the traditional “data-label” structure and (b) Different forms of privileged information. Non-target classes of soft labels provide additional information, can be considered a form of privileged information. Feature labels encapsulate high-dimensional information. Attention labels are obtained by applying average pooling to feature labels.
Refer to caption: x2.png

### 3.1 Determining Privileged Information

Although prior research on dataset reduction has shown impressive results, it mainly focuses on generating reduced datasets in conventional data-label formats, as depicted in Figure 2(a).
However, more informative data, such as privileged information, can be utilized to enhance both the utility and representational performance of reduced datasets, as illustrated in Figure 2(b). Below, we briefly explore several forms of privileged information that can be incorporated to achieve this goal.

Soft labels. We first claim that soft labels are a form of privileged information, as they offer richer insight into how an expert model interprets predictions, providing soft probabilities rather than a single hard label. Specifically, the non-target logits can be viewed as additional information for supervision. Several works have previously explored the effectiveness of soft labels in dataset reduction (Guo et al., ; Cui et al., ; Bohdal et al., ; Qin et al., ). However, while soft labels enhance the available information, they are limited to low-dimensional discriminative representations and fail to capture more complex, high-dimensional statistics. Moreover, they do not fundamentally alter the data-label structure of the reduced dataset representation.

Feature labels. Beyond soft labels, we propose feature labels as a more effective form of privileged information. These labels, derived from unified intermediate representations across well-trained models, encapsulate rich, high-dimensional latent statistics. By providing additional supervision, feature labels enhance model training on datasets with privileged information. Unlike approaches that focus primarily on soft labels, assigning a unified feature label to each input enriches the supervision signal for downstream tasks, effectively addressing the limitations of prior methods.

Attention labels. Alongside feature labels, we propose attention labels as an alternative form of privileged information that provides a more memory-efficient representation. Attention labels can be derived from either spatial or channel attention of feature labels (Woo et al., ). For example, given a feature label of size $Ch\times H\times W$, spatial attention reduces the $Ch$ dimensions through pooling operations (*e.g.*, average pooling, max pooling), resulting in an attention label of size $1\times H\times W$. Similarly, channel attention applies pooling along the $H\times W$ dimension to produce a reduced representation, *i.e.*, $Ch\times 1\times 1$. Both feature and attention labels are valuable, with attention labels offering a more efficient representation but with a possible trade-off in the richness of information. We provide more discussion on attention labels in Appendix B.2.

In this work, we primarily focus on generating additional feature labels(^1^11We consider attention labels as a specific form of feature labels. Therefore, for simplicity, we use the term “feature labels” as a unified description for both feature labels and attention labels.) for the reduced dataset, as these forms of privileged information provide more complementary and useful insights for model training. However, privileged information can take various forms beyond attention and feature labels. Depending on the task and the model architecture, other types of information, such as learned embeddings, domain-specific signals, or task-related metadata, could be equally beneficial in enhancing the informativeness and performance of reduced datasets. The flexibility to incorporate different kinds of privileged information allows us to tailor the dataset to specific needs and maximize the potential of the reduced data.

### 3.2 Synthesizing Privileged Information

In this section, we discuss the process of generating privileged information for a given reduced dataset $\mathcal{D}_{\mathcal{S}}=\{(\tilde{x}_{i},\tilde{y}_{i})\}_{i=1}^{m}$, with the goal of obtaining a more informative dataset $\mathcal{D}_{\mathcal{S}}^{\star}=\{(\tilde{x}_{i},\tilde{y}_{i},f_{i}^{\star}
)\}_{i=1}^{m}$. Here, $\mathcal{D}_{\mathcal{S}}$ is the reduced dataset of a larger dataset $\mathcal{D}_{\mathcal{T}}=\{(x_{i},y_{i})\}_{i=1}^{n}$, where $m\ll n$. Our primary focus is on incorporating feature labels as the form of privileged information. While various methods can be employed to synthesize privileged information, they can generally be categorized into two strategies: direct assignment and learning-based methods.

Direct Assignment of Feature Labels.
A straightforward approach to obtaining feature labels is by using a pre-trained model, *e.g.*, $g_{\mathcal{T}}$, to extract intermediate features for each input data $\tilde{x}_{i}\in\mathcal{D}_{\mathcal{S}}$. Specifically, this is formalized as $f_{i}^{\star}=g_{\mathcal{T}}(\tilde{x}_{i})$, resulting in an extended dataset represented as $\mathcal{D}_{\mathcal{S}}^{\star}={(\tilde{x}_{i},\tilde{y}_{i},f_{i}^{\star})}$. This method is computationally efficient but relies heavily on the generalization ability of the pre-trained model $g_{\mathcal{T}}$. While the feature labels, which capture the implicit biases of $g_{\mathcal{T}}$, may enhance the quality of the reduced dataset, they could also introduce potential drawbacks. In fact, directly assigned feature labels are often overly discriminative, reducing diversity. However, we find that suitable feature labels should strike a balance between these two properties.

Learning Feature Labels.
A more robust approach to obtaining feature labels is through learning-based methods. Many dataset distillation techniques can be adapted for learning feature labels. For instance, we can employ the Dataset Condensation (DC) (Zhao et al., ) method as an illustrative example to guide the process of learning synthetic feature labels. Suppose we have a learned synthetic dataset $\mathcal{D}_{\mathcal{S}}$. In the typical DC method, we initialize a random model $g$ with parameters $\theta=\theta_{0}$ and train it for $T$ epochs on both $\mathcal{D}_{\mathcal{T}}$ and $\mathcal{D}_{\mathcal{S}}$ separately. The synthetic dataset $\mathcal{D}_{\mathcal{S}}$ is updated by matching the category gradients between the two datasets, which can be expressed as follows(^2^22We ignore the category symbol for simplicity.):

$$ $\displaystyle\mathcal{D}_{\mathcal{S}}=\mathop{\arg\min}_{\mathcal{D}_{ \mathcal{S}}}$ $\displaystyle\underset{\theta_{0}\sim P_{\theta}}{\mathbb{E}}\left[\sum_{t=0}^ {T}\mathbf{D}\left(\nabla_{\theta}L(\mathcal{D}_{\mathcal{S}};\theta_{t}), \nabla_{\theta}L(\mathcal{D}_{\mathcal{T}};\theta_{t})\right)\right]$ (1) where $\displaystyle L(\mathcal{D}_{\mathcal{T}};\theta_{t})=\underset{(x_{i},y_{i}) \in\mathcal{D}_{\mathcal{T}}}{\mathbb{E}}\ell_{ce}\left[\left(y_{i},\sigma(g(x _{i};\theta_{t}))\right)\right],$ and $\displaystyle L(\mathcal{D}_{\mathcal{S}};\theta_{t})\triangleq\mathcal{L}_{ cls}=\underset{(\tilde{x}_{i},\tilde{y}_{i})\in\mathcal{D}_{\mathcal{S}}}{ \mathbb{E}}\left[\ell_{ce}\left(\tilde{y}_{i},\sigma(g(\tilde{x}_{i};\theta_{t }))\right)\right],$ $$

where $\ell_{ce}(\cdot,\cdot)$ denotes the cross-entropy (CE) loss, and $\sigma(\cdot)$ represents the softmax function. In contrast, we aim to match the performance between $\mathcal{D}_{\mathcal{T}}$ and $\mathcal{D}_{\mathcal{S}}^{\star}$, where privileged information is synthesized to capture additional informativeness. Let $\ell_{mse}$ represent the mean square error (MSE) loss, and let $\psi(\cdot)$ denote the intermediate output of model $g$, *i.e.*, $g=\psi\circ\kappa$, where $\kappa(\cdot)$ is the classifier component of $g$. Therefore, our objective becomes:

$$ $\mathcal{D}_{\mathcal{S}}^{\star}=\mathop{\arg\min}_{\mathcal{D}_{\mathcal{S}} ^{\star}}\underset{\theta_{0}\sim P_{\theta}}{\mathbb{E}}\left[\sum_{t=0}^{T} \mathbf{D}(\nabla_{\theta}L_{c}(\mathcal{D}_{\mathcal{S}}^{\star};\theta_{t}), \nabla_{\theta}L_{c}(\mathcal{D}_{\mathcal{T}};\theta_{t}))\right],$ (2) $$

$$ where $\displaystyle L(\mathcal{D}_{\mathcal{S}}^{\star};\theta_{t})\triangleq \mathcal{L}_{cls}+\lambda_{reg}\cdot\mathcal{L}_{reg},$ (3) $\displaystyle\mathcal{L}_{cls}=\underset{(\tilde{x}_{i},\tilde{y}_{i})\in \mathcal{D}_{\mathcal{S}}^{\star}}{\mathbb{E}}[\ell_{ce}\left(y_{i},\sigma(g( \tilde{x}_{i};\theta_{t}))\right)],$ and $\displaystyle\mathcal{L}_{reg}=\underset{(\tilde{x}_{i},\tilde{y}_{i},f_{i}^{ \star})\in\mathcal{D}_{\mathcal{S}}^{\star}}{\mathbb{E}}[\ell_{mse}\left(f_{i} ^{\star},\psi(\tilde{x}_{i};\theta_{t})\right)],$ $$

where $\lambda_{reg}$ is a hyper-parameter to determine the scale of using privileged information. In addition to DC, other dataset distillation methods can also be employed to synthesize feature labels $f_{i}^{\star}$. We provide further resuls on coreset selection methods like Herding (Welling, ), K-center (Har-Peled & Mazumdar, ), Forgetting (Toneva et al., ), and dataset distillation methods like DC (Zhao et al., ), MTT (Cazenavette et al., ), and DATM (Guo et al., ).

In addtional, we introduce additional supervision to enhance the discriminative power of these feature labels while preserving their diversity.

$\bullet$ Task-oriented synthesization.
To improve the discriminative power of the feature labels, we adopt a task-oriented approach by feeding the synthesized feature labels into the classifier of the model used to extract gradients during bi-level optimization. We achieve this by performing additional CE loss between the feature label $f_{i}^{\star}$’s prediction and the ground-truth label $\tilde{y}_{i}$. Specifically, we have

$$ $\mathcal{L}_{task}=\underset{(\tilde{f}_{i}^{\star},\tilde{y}_{i})\in\mathcal{ D}_{\mathcal{S}}^{\star}}{\mathbb{E}}[\ell_{ce}\left(\tilde{y}_{i},\sigma( \kappa(f_{i}^{\star};\theta_{t}))\right)].$ (4) $$

This allows the feature labels to contribute directly to the final prediction, ensuring they become better aligned with the task at hand. The scale of task supervision is controlled by the hyper-parameter $\lambda_{task}$. Our observations indicate that the preferred feature labels should strike a balance between discriminability and diversity, *i.e.*, they should neither be overly discriminative nor completely lack discriminative power. As shown in Figure 3(a)(c), increasing $\lambda_{task}$ tends to cluster the feature labels, reducing their diversity while increasing their discriminability. We find that the optimal feature labels are not achieved with the highest task supervision. Instead, a moderate level of task supervision, as shown in Figure 3(b), strikes the right balance between diversity and discriminability. This also suggests a possible explanation for the drawbacks of directly assigning feature labels from a well-trained model, as these labels tend to be overly discriminative.

$\bullet$ Versatility synthesization.
Beyond task-specific supervision, we aim to preserve the generative versatility of the feature labels, *i.e.*, a set of feature labels $f_{i}^{\star}\in F_{i}$, where $F_{i}$ represents the possible feature label set for input $x_{i}$. This approach involves synthesizing multiple feature labels for a single data-label pair. Appropriate versatility enhancement ensures that the synthesized feature labels remain informative across different tasks and applications, providing a richer and more comprehensive representation of the data. When using multiple feature labels, we primarily employ two strategies: randomly selecting one or using the average feature label from $F_{i}$. Further discussions demonstrating the benefits of this versatility are presented in Figure 4(a) and Appendix B.3.

We define the overall loss function for a model trained on the reduced dataset $\mathcal{D}_{\mathcal{S}}^{\star}$ as follows:

$$ $L(\mathcal{\mathcal{D}_{\mathcal{S}}^{\star}};\theta_{t})=\mathcal{L}_{cls}+ \mathbb{E}_{f_{i}^{\star}\in F_{i}}\left[\lambda_{reg}\cdot\mathcal{L}_{reg}+ \lambda_{task}\cdot\mathcal{L}_{task}\right],$ (5) $$

where $F_{i}$ denotes the feature label set, containing multiple $f_{i}^{\star}$ for a single data-label pair $(\tilde{x}_{i},\tilde{y}_{i})$. The pseudocode for learning privileged information is provided in Appendix D.

Figure: Figure 3: Feature labels learned under varying levels of task supervision. (a) t-SNE visualization of feature labels learned with different task supervision coefficients $\lambda_{task}$. (b) The most effective feature labels are produced with a moderate level of task supervision, avoiding excessively high or low supervision. (c) Increasing task supervision makes the feature labels more discriminative but less diverse. Diversity is measured by the negative mutual information between the feature labels and the ground truth labels, while discriminability is measured by the classification accuracy of a linear classifier trained on the feature labels.
Refer to caption: x3.png

### 3.3 Learning Using Privileged Information

We have previously discussed how to synthesize appropriate privileged information $f_{i}^{\star}$ for a given reduced dataset $\mathcal{D}_{\mathcal{S}}$ and extend it into $\mathcal{D}_{\mathcal{S}}^{\star}$. We now focus on leveraging the new reduced dataset $\mathcal{D}_{\mathcal{S}}^{\star}$ to enhance a model’s performance on unseen test data. Following the learning using privileged information (LUPI) framework (Pechyony & Vapnik, ; Lopez-Paz et al., ), we incorporate the additional privileged information $f_{i}^{\star}$ during training to build a classifier that outperforms those trained solely on the regular reduced dataset $\mathcal{D}_{\mathcal{S}}$.

Given an arbitrary model $h$ with parameters $\theta\in\Theta$, we now formally discuss how to train $h$ with synthesized feature labels. The same loss function $L(\mathcal{D}_{\mathcal{S}}^{\star})$, as shown in Eq. (5), is applied to train $h$, with hyper-parameters (*e.g.*, $\lambda_{reg},\lambda_{task}$) kept consistent for fairness.

$$ $\theta^{\star}=\mathop{\arg\min}_{\theta\in\Theta}L(\mathcal{D}_{\mathcal{S}}^ {\star};\theta)$ (6) $$

Besides feature labels, we can store attention labels, which can be generated by performing average pooling on the spatial or channel dimensions of learned feature labels. Attention labels contain more condensed information, which can further reduce storage cost. During LUPI, we apply the same pooling strategy for intermediate features of the given model to calculate the MSE loss between the intermediate features and given feature labels.

### 3.4 Theoretical Analysis

We propose a theoretical analysis to elucidate the mechanisms by which the DRUPI framework enhances the quality of reduced datasets. Let $g_{\mathcal{T}}\in\mathcal{G}_{\mathcal{T}}$ denote the oracle function for the original dataset $\mathcal{D}_{\mathcal{T}}$, and $|\cdot|_{C}$ represent a function class capacity measure, *i.e.*, a measure of model performance. Consider two models: one trained on the pure reduced dataset $\mathcal{D}_{\mathcal{S}}$, represented by $g_{\mathcal{S}}\in\mathcal{G}_{\mathcal{S}}$, and another trained on the reduced dataset with privileged information, $\mathcal{D}_{\mathcal{S}}^{\star}$, denoted by $g_{\mathcal{S}^{\star}}\in\mathcal{G}_{\mathcal{S}^{\star}}$.

We begin by briefly reviewing the well-known VC theory (Vapnik, ), a fundamental analytical tool in statistical learning theory that forms the basis of our theoretical framework. The VC-dimension defines the performance of a model to limited data points. Specifically, for a model $g$ belonging to a function class $\mathcal{G}$, with a finite VC-dimension $|\mathcal{G}|_{\text{VC}}$, the expected error $R(g)$ with probability $1-\delta$ is bounded as follows:

$$ $R(g)\leq R_{m}(g)+O\left(\left(\frac{|\mathcal{G}|_{\mathrm{VC}}-\log\delta}{m }\right)^{\alpha}\right),$ (7) $$

where the $O(\cdot)$ term is the estimation error, and $R_{m}(g)$ is the training error over $m$ data points, and $\alpha$ lies between $\frac{1}{2}$ and $1$. The parameter $\alpha$ represents the difficulty of the task. For more difficult, non-separable tasks, $\alpha\approx\frac{1}{2}$, yielding a slower learning rate of $O(m^{-1/2})$. For easier, separable tasks, where the model makes no training errors, $\alpha\approx 1$, yielding a faster learning rate of $O(m^{-1})$. Given a student learning from a fixed dataset of size $m$, a good teacher model can ease the learning process by accelerating the learning rate from $O(m^{-1/2})$ to $O(m^{-1})$.

Next, we provide theoretical analysis for DRUPI, showing how incorporating privileged information accelerates the learning process and improves the quality of the reduced dataset. Building on the top of (Lopez-Paz et al., ), we extend the results in dataset reduction scenarios. First, assume that the model trained on the pure reduced dataset $g_{\mathcal{S}}$ learns the true function $g_{\mathcal{T}}$ at a slower rate $\alpha_{\mathcal{S}}$:

$$ $R(g_{\mathcal{S}})-R(g_{\mathcal{T}})\leq O\left(\frac{|\mathcal{G}_{\mathcal{ S}}|_{\text{C}}}{m^{\alpha_{\mathcal{S}}}}\right)+\varepsilon_{\mathcal{S}},$ (8) $$

where $\varepsilon_{\mathcal{S}}$ is the approximation error of $\mathcal{G}_{\mathcal{S}}$ with respect to $g_{\mathcal{T}}\in\mathcal{G}_{\mathcal{T}}$. Second, assume that the model trained on the dataset with privileged information $g_{\mathcal{S}^{\star}}$ learns at a faster rate $\alpha_{\mathcal{S}^{\star}}$:

$$ $R(g_{\mathcal{S}^{\star}})-R(g_{\mathcal{T}})\leq O\left(\frac{|\mathcal{G}_{ \mathcal{S}^{\star}}|_{\text{C}}}{m^{\alpha_{\mathcal{S}^{\star}}}}\right)+ \varepsilon_{\mathcal{S}^{\star}},$ (9) $$

where $\varepsilon_{\mathcal{S}^{\star}}$ is the approximation error of $\mathcal{G}_{\mathcal{S}^{\star}}$ with respect to $g_{\mathcal{T}}$. Finally, assume that the performance difference $g_{\mathcal{S}}$ learns from the model with privileged information $g_{\mathcal{S^{\star}}}$ is

$$ $R(g_{\mathcal{S}})-R(g_{\mathcal{S}^{\star}})\leq O\left(\frac{|\mathcal{G}_{ \mathcal{S}}|_{\text{C}}}{m^{\alpha}}\right)+\varepsilon,$ (10) $$

where $\varepsilon$ is the approximation error of $\mathcal{G}_{\mathcal{S}}$ with respect to $g_{\mathcal{S}^{\star}}$, and $\frac{1}{2}\leq\alpha\leq 1$. Combining Eq. (9) and Eq. (10), the learning rate for the model without privileged information learning the oracle function $g_{\mathcal{T}}$ is then given by

$$ $\displaystyle R(g_{\mathcal{S}})-R(g_{\mathcal{T}})=R(g_{\mathcal{S}})-R(g_{ \mathcal{S^{\star}}})+R(g_{\mathcal{S^{\star}}})-R(g_{\mathcal{T}})$ (11) $\displaystyle\leq O\left(\frac{|\mathcal{G}_{\mathcal{S}}|_{\textrm{C}}}{m^{ \alpha}}\right)+\varepsilon+O\left(\frac{|\mathcal{G}_{\mathcal{S^{\star}}}|_{ \textrm{C}}}{m^{\alpha_{\mathcal{S}^{\star}}}}\right)+\varepsilon_{\mathcal{S^ {\star}}}\leq O\left(\frac{|\mathcal{G}_{\mathcal{S}}|_{\textrm{C}}}{m^{\alpha }}+\frac{|\mathcal{G}_{\mathcal{S^{\star}}}|_{\textrm{C}}}{m^{\alpha_{\mathcal {S}^{\star}}}}\right)+\varepsilon+\varepsilon_{\mathcal{S^{\star}}},$ $$

where the final inequality arises because $\alpha\leq 1$. Therefore, for $\frac{1}{2}<\alpha\leq 1$ in dataset reduction settings, the inequality becomes:

$$ $O\left(\frac{|\mathcal{G}_{\mathcal{S}}|_{\textrm{C}}}{m^{\alpha}}+\frac{| \mathcal{G}_{\mathcal{S^{\star}}}|_{\textrm{C}}}{m^{\alpha_{\mathcal{S}^{\star }}}}\right)+\varepsilon+\varepsilon_{\mathcal{S}^{\star}}\leq O\left(\frac{| \mathcal{G}_{\mathcal{S}}|_{\text{C}}}{{m}^{\alpha_{\mathcal{S}}}}\right)+ \varepsilon_{\mathcal{S}}.$ (12) $$

This inequality highlights the advantages of models trained with privileged information: training using privileged information exhibit lower generalization and approximation errors compared to those trained without privileged information. More importantly, it emphasizes that the privileged information is most beneficial in low-data regimes, which is the typical DR scenario. These benefits align with the principles of LUPI as outlined in  (Vapnik et al., ; Lopez-Paz et al., ).

## 4 Experiments

**Table 1: Application of DRUPI to representative pruning methods on CIFAR-10/100. We initialized the dataset with the baseline methods, and utilized DC for synthesizing feature labels.**
| Dataset | CIFAR-10 | CIFAR-100 |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Fraction (%) | 0.02 | 0.1 | 0.2 | 0.4 | 1 | 0.2 | 1 | 2 | 4 | 10 |
| Random | 13.5$\pm$0.4 | 20.0$\pm$0.5 | 27.1$\pm$0.6 | 35.8$\pm$0.6 | 43.0$\pm$0.5 | 4.3$\pm$0.2 | 9.5$\pm$0.2 | 14.5$\pm$0.2 | 19.5$\pm$0.4 | 29.5$\pm$0.3 |
| L-Conf | 10.7$\pm$0.4 | 10.5$\pm$0.4 | 10.8$\pm$0.4 | 17.9$\pm$0.4 | 23.1$\pm$0.6 | 2.1$\pm$0.1 | 3.6$\pm$0.1 | 6.6$\pm$0.2 | 9.0$\pm$0.2 | 16.4$\pm$0.3 |
| Entropy | 12.2$\pm$0.6 | 14.1$\pm$0.5 | 14.8$\pm$0.5 | 19.6$\pm$0.4 | 23.8$\pm$0.7 | 1.7$\pm$0.1 | 3.7$\pm$0.2 | 6.7$\pm$0.2 | 9.0$\pm$0.3 | 17.1$\pm$0.3 |
| Margin | 8.9$\pm$0.4 | 15.8$\pm$0.7 | 20.3$\pm$0.4 | 24.8$\pm$0.5 | 31.3$\pm$0.5 | 3.0$\pm$0.2 | 6.2$\pm$0.2 | 9.0$\pm$0.2 | 12.7$\pm$0.3 | 20.7$\pm$0.3 |
| Glister | 11.5$\pm$0.3 | 16.9$\pm$0.5 | 23.0$\pm$0.4 | 28.4$\pm$0.5 | 30.0$\pm$0.5 | 2.9$\pm$0.5 | 7.1$\pm$0.4 | 10.4$\pm$0.6 | 13.3$\pm$0.7 | 27.2$\pm$0.7 |
| Graig | 18.1$\pm$0.3 | 19.5$\pm$0.4 | 19.0$\pm$0.5 | 27.8$\pm$0.3 | 30.2$\pm$0.4 | 4.3$\pm$0.5 | 9.0$\pm$0.4 | 13.6$\pm$0.7 | 14.6$\pm$0.5 | 20.1$\pm$0.6 |
| Herding | 15.3$\pm$0.5 | 23.5$\pm$0.3 | 25.1$\pm$0.5 | 26.7$\pm$0.4 | 34.9$\pm$0.6 | 4.0$\pm$0.1 | 6.2$\pm$0.2 | 8.1$\pm$0.4 | 13.1$\pm$0.6 | 18.4$\pm$0.6 |
| +DRUPI | 27.9$\pm$0.6 | 37.3$\pm$0.6 | 45.8$\pm$0.7 | 51.0$\pm$0.4 | 54.0$\pm$0.6 | 14.0$\pm$0.3 | 20.4$\pm$0.5 | 25.5$\pm$0.4 | 28.9$\pm$0.3 | 31.4$\pm$0.6 |
| ↑ | 12.6 | 13.8 | 20.7 | 24.3 | 19.1 | 10.0 | 14.2 | 17.4 | 15.8 | 13.0 |
| k-Center | 16.4$\pm$0.6 | 22.4$\pm$0.6 | 23.1$\pm$0.5 | 30.4$\pm$0.4 | 36.7$\pm$0.5 | 4.8$\pm$0.2 | 6.7$\pm$0.4 | 10.0$\pm$0.5 | 15.9$\pm$1.1 | 21.8$\pm$1.0 |
| +DRUPI | 29.7$\pm$0.6 | 40.0$\pm$0.6 | 46.2$\pm$0.6 | 50.8$\pm$0.6 | 54.3$\pm$0.6 | 13.5$\pm$0.3 | 20.0$\pm$0.5 | 25.9$\pm$0.3 | 29.1$\pm$0.5 | 32.0$\pm$0.5 |
| ↑ | 13.3 | 17.6 | 23.1 | 20.4 | 17.6 | 8.7 | 13.3 | 15.9 | 13.2 | 10.2 |
| Forgetting | 15.3$\pm$0.6 | 19.1$\pm$0.7 | 23.9$\pm$0.7 | 26.9$\pm$0.7 | 39.5$\pm$0.5 | 4.1$\pm$0.1 | 7.8$\pm$0.3 | 10.4$\pm$0.5 | 14.1$\pm$0.6 | 22.3$\pm$0.4 |
| +DRUPI | 30.0$\pm$0.6 | 39.7$\pm$0.7 | 46.6$\pm$0.6 | 51.3$\pm$0.5 | 54.5$\pm$0.5 | 14.0$\pm$0.4 | 20.1$\pm$0.6 | 25.8$\pm$0.3 | 29.3$\pm$0.4 | 32.2$\pm$0.5 |
| ↑ | 14.7 | 20.6 | 22.7 | 24.4 | 15.0 | 9.9 | 12.3 | 15.4 | 15.2 | 9.9 |
| Full Dataset | 84.8$\pm$0.1 | 56.2$\pm$0.3 |  |  |  |  |  |  |  |  |

**Table 2: Results of DRUPI on distillation methods across CIFAR-10/100, and Tiny ImageNet. We initialized reduced datasets with corresponding baseline methods, and synthesized feature labels for these datasets with the same baseline methods.**
| Dataset | CIFAR-10 | CIFAR-100 | Tiny ImageNet |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| IPC | 1 | 10 | 50 | 1 | 10 | 50 | 1 | 10 |
| Random | 15.4$\pm$0.3 | 31.0$\pm$0.5 | 50.6$\pm$0.3 | 4.2$\pm$0.3 | 14.6$\pm$0.5 | 33.4$\pm$0.4 | 1.4$\pm$0.1 | 5.0$\pm$0.2 |
| KIP | 49.9$\pm$0.2 | 62.7$\pm$0.3 | 68.6$\pm$0.3 | 15.7$\pm$0.2 | 28.3$\pm$0.1 | - | - | - |
| DM | 26.0$\pm$0.8 | 48.9$\pm$0.6 | 63.0$\pm$0.4 | 11.4$\pm$0.3 | 29.7$\pm$0.3 | 43.6$\pm$0.4 | 3.9$\pm$0.2 | 12.9$\pm$0.4 |
| DSA | 28.8$\pm$0.7 | 52.1$\pm$0.5 | 60.6$\pm$0.5 | 13.9$\pm$0.3 | 32.3$\pm$0.3 | 42.8$\pm$0.4 | - | - |
| DCC | 32.9$\pm$0.8 | 49.4$\pm$0.5 | 61.6$\pm$0.4 | 13.3$\pm$0.3 | 30.6$\pm$0.4 | 40.0$\pm$0.3 | - | - |
| DSAC | 34.0$\pm$0.7 | 54.5$\pm$0.5 | 64.2$\pm$0.4 | 14.6$\pm$0.3 | 33.5$\pm$0.3 | 39.3$\pm$0.4 | - | - |
| CAFE | 30.3$\pm$1.1 | 46.3$\pm$0.6 | 55.5$\pm$0.6 | 12.9$\pm$0.3 | 27.8$\pm$0.3 | 37.9$\pm$0.3 | - | - |
| IDM | 45.6$\pm$0.7 | 58.6$\pm$0.1 | 67.5$\pm$0.1 | 20.1$\pm$0.3 | 45.1$\pm$0.1 | 50.0$\pm$0.2 | - | - |
| DC | 28.3$\pm$0.5 | 44.9$\pm$0.5 | 53.9$\pm$0.5 | 12.8$\pm$0.3 | 25.2$\pm$0.3 | 29.8$\pm$0.3 | - | - |
| +DRUPI | 31.5$\pm$0.9 | 47.4$\pm$0.9 | 55.0$\pm$0.5 | 14.9$\pm$0.4 | 29.2$\pm$0.5 | 30.9$\pm$0.5 | - | - |
| ↑ | 3.2 | 2.5 | 1.1 | 2.1 | 4.0 | 1.1 |  |  |
| MTT | 46.2$\pm$0.8 | 65.4$\pm$0.7 | 71.6$\pm$0.2 | 24.3$\pm$0.3 | 39.7$\pm$0.4 | 47.7$\pm$0.2 | 8.8$\pm$0.3 | 23.2$\pm$0.2 |
| +DRUPI | 47.4$\pm$0.5 | 65.8$\pm$0.6 | 71.7$\pm$0.2 | 25.6$\pm$0.4 | 40.8$\pm$0.3 | 48.8$\pm$0.3 | 11.2$\pm$0.1 | 24.9$\pm$0.2 |
| ↑ | 1.2 | 0.4 | 0.1 | 1.3 | 1.1 | 1.1 | 2.4 | 1.7 |
| Full Dataset | 84.8$\pm$0.1 | 56.2$\pm$0.3 | 37.6$\pm$0.4 |  |  |  |  |  |

### 4.1 Experimental Setup

In this section, we investigate the effectiveness of our proposed method, DRUPI, through a series of experiments on diverse datasets and tasks. We begin by evaluating the efficacy of DRUPI when applied to coreset selection and dataset distillation tasks. Specifically, we followed prior works to conduct
experiments on CIFAR-10/100 (Krizhevsky et al., ) for coreset selection methods, where ResNet-18 (He et al., ) is utilized for extracting importance score. For the dataset distillation methods, we conducted experiments on CIFAR-10/100, Tiny ImageNet (Le & Yang, ), and subsets of ImageNet (Russakovsky et al., ).

For coreset selection, we benchmarked against several representative baselines, including Random, L-conf, Entropy, Margin (Coleman et al., ), Glister (Killamsetty et al., ), Graig (Mirzasoleiman et al., ), Herding (Welling, ), k-Center (Har-Peled & Mazumdar, ), and Forgetting (Toneva et al., ). More detailed settings are provided in Appendix A.2.

For dataset distillation, we evaluated a range of advanced methods, including KIP (Nguyen et al., ), DM (Zhao & Bilen, ), DSA (Zhao & Bilen, ), DCC, DSAC (Lee et al., ), CAFE (Wang et al., ), IDM (Zhao et al., ), DC (Zhao et al., ), and MTT (Cazenavette et al., ). In line with prior studies, we used networks with instance normalization as the default setting. Unless otherwise specified, distillation was performed with a depth-3 ConvNet for CIFAR-10/100, a depth-4 ConvNet for Tiny ImageNet, and a depth-5 ConvNet for ImageNet subsets. See Appendix A.3 for more details.

It is worth noting that for both pruning and distillation methods, we initialized all data-label pairs using the baseline method and employed a weakly-trained model (trained for one epoch) to extract feature labels, which were then used to synthesize privileged information. By default, we utilized DC to synthesize one feature label per data-label pair, aligning it with features extracted from the final layer of a ConvNet during bi-level optimization. We employed $\lambda_{reg}=0.5$ and $\lambda_{task}=0.1$, and learning rate = 0.1 for feature optimization.

### 4.2 Main Results

Coreset selection. In our experiments, DRUPI utilizes the reduced dataset initialized with the Herding, k-Center, and Forgetting methods to assess its performance across diverse fraction on CIFAR-10/100. As shown in Table 1, by incorporating privileged information, these methods consistently outperformed the baseline across a range of fraction settings on CIFAR-10/100. Particularly, on the CIFAR-10 (fraction = 0.4%) , DRUPI achieved a performance increase of 24.4% on the Forgetting method and 24.3% on the Herding method. We find that for datasets without optimized instances (*e.g.*, selected coresets), the performance gain is much higher than those with optimized samples.

Dataset distillation. We employed DRUPI in several classical dataset distillation methods, where privileged information is obtained with DC and MTT. Table 2 summarizes the classification performances of ConvNets trained with different distillation methods. Specifically, applying DRUPI to DC on CIFAR-100 (10 IPC) resulted in a 4% improvement. For MTT, DRUPI delivered a 2.4% gain on Tiny ImageNet (1 IPC). Additionally, we evaluated its effectiveness on ImageNet subsets, as shown in Table 3, where DRUPI applied to MTT led to a 3.4% improvement on ImageMeow with 10 IPC, demonstrating strong performance even on larger datasets. Notably, learning both feature labels with DRUPI outperforms simply extracting features alone. Further results for DATM are provided in Appendix B.1.

**Table 3: Results on ImageNet subsets when integrating DRUPI into dataset distillation methods. Reduced datasets are initialized with MTT, and feature labels are synthesized with MTT.**
| Dataset | ImageNette | ImageWoof | ImageFruit | ImageMeow | ImageYellow |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| IPC | 1 | 10 | 1 | 10 | 1 | 10 | 1 | 10 | 1 | 10 |
| MTT | 47.7$\pm$0.9 | 63.0$\pm$1.3 | 28.6$\pm$0.8 | 35.8$\pm$1.8 | 26.6$\pm$0.8 | 40.3$\pm$1.3 | 30.7$\pm$1.6 | 40.4$\pm$2.2 | 45.2$\pm$0.8 | 60.0$\pm$1.5 |
| +DRUPI | 50.5$\pm$0.1 | 65.7$\pm$0.5 | 31.3$\pm$0.2 | 37.5$\pm$1.0 | 29.1$\pm$1.4 | 43.0$\pm$0.9 | 34.0$\pm$1.6 | 43.8$\pm$0.9 | 46.6$\pm$0.6 | 62.2$\pm$0.9 |
| ↑ | 2.8 | 2.7 | 2.7 | 2.7 | 2.5 | 2.7 | 3.3 | 3.4 | 1.4 | 2.2 |

Table: Table 4: Cross-architecture evaluation of coresets on unseen networks. Reduced datasets are initialized with different pruning methods on CIFAR-10 (0.2%). Feature labels are learned with DC. We utilized ConvNet for synthesizing feature labels.

|  | LeNet | MLP | ResNet | VGG | ConvNet | AlexNet |
| --- | --- | --- | --- | --- | --- | --- |
| Herding | 23.0$\pm$1.3 | 21.3$\pm$0.4 | 26.2$\pm$0.9 | 24.1$\pm$0.7 | 25.1$\pm$0.5 | 23.3$\pm$1.3 |
| +DRUPI | 32.4$\pm$1.9 | 30.4$\pm$0.4 | 36.9$\pm$1.0 | 36.4$\pm$0.7 | 46.3$\pm$0.6 | 33.2$\pm$2.0 |
| ↑ | 9.4 | 9.1 | 10.7 | 12.3 | 21.1 | 9.9 |
| Forgetting | 25.5$\pm$1.6 | 23.8$\pm$0.3 | 24.7$\pm$1.0 | 21.0$\pm$0.4 | 24.0$\pm$0.5 | 25.5$\pm$0.9 |
| +DRUPI | 35.0$\pm$1.6 | 33.1$\pm$0.5 | 38.1$\pm$0.9 | 35.3$\pm$0.6 | 47.1$\pm$0.7 | 35.7$\pm$1.1 |
| ↑ | 9.5 | 9.3 | 13.4 | 14.3 | 23.1 | 10.2 |
| k-Center | 21.6$\pm$1.1 | 19.7$\pm$0.4 | 24.0$\pm$0.8 | 21.5$\pm$0.8 | 23.1$\pm$0.7 | 21.3$\pm$0.7 |
| +DRUPI | 34.4$\pm$1.3 | 31.4$\pm$0.3 | 36.0$\pm$1.5 | 34.3$\pm$0.6 | 46.5$\pm$0.6 | 36.0$\pm$1.6 |
| ↑ | 12.8 | 11.7 | 12.0 | 12.8 | 23.4 | 14.7 |

Table: Table 5: Cross-architecture evaluations of distilled datasets on unseen networks. Reduced datasets are initialized with DC on CIFAR-10 (IPC=10). Feature labels are learned with DC.

|  | DC | +DRUPI | DC | +DRUPI | DC | +DRUPI |
| --- | --- | --- | --- | --- | --- | --- |
| Train\Test | LeNet | ConvNet | ResNet |  |  |  |
| LeNet | 23.3$\pm$5.3 | 28.8$\pm$4.1 (5.5) | 35.8$\pm$0.6 | 46.6$\pm$0.7 (10.8) | 29.7$\pm$2.0 | 36.1$\pm$1.3 (6.4) |
| MLP | 28.0$\pm$1.1 | 28.5$\pm$4.1 (0.5) | 29.0$\pm$1.1 | 46.5$\pm$0.8 (17.6) | 21.8$\pm$1.8 | 36.5$\pm$1.2 (14.7) |
| ResNet | 22.0$\pm$1.5 | 25.9$\pm$2.6 (3.9) | 36.2$\pm$0.8 | 45.9$\pm$0.6 (9.7) | 33.6$\pm$1.3 | 36.1$\pm$1.8 (2.5) |
| VGG | 22.7$\pm$2.4 | 28.3$\pm$1.3 (5.6) | 33.5$\pm$1.1 | 46.4$\pm$0.6 (12.9) | 17.5$\pm$1.9 | 35.8$\pm$1.5 (18.3) |
| ConvNet | 16.8$\pm$1.6 | 24.7$\pm$4.3 (7.9) | 44.5$\pm$0.9 | 47.1$\pm$0.9 (2.6) | 36.2$\pm$1.6 | 37.5$\pm$1.1 (1.3) |
| AlexNet | 32.2$\pm$1.9 | 34.6$\pm$1.6 (2.4) | 36.9$\pm$0.9 | 46.2$\pm$0.6 (9.3) | 31.5$\pm$1.5 | 36.1$\pm$1.4 (4.6) |
| Train\Test | VGG | MLP | AlexNet |  |  |  |
| LeNet | 34.0$\pm$0.7 | 34.9$\pm$0.7 (0.9) | 32.6$\pm$0.5 | 34.3$\pm$0.5 (1.7) | 25.6$\pm$2.1 | 32.4$\pm$2.9 (6.8) |
| MLP | 22.3$\pm$1.0 | 34.6$\pm$0.7 (12.3) | 36.6$\pm$0.5 | 37.1$\pm$0.4 (0.5) | 29.9$\pm$2.5 | 30.0$\pm$0.9 (0.1) |
| ResNet | 31.6$\pm$0.9 | 32.6$\pm$0.7 (1.0) | 28.3$\pm$0.9 | 30.0$\pm$0.6 (1.7) | 24.4$\pm$3.3 | 27.2$\pm$3.6 (2.8) |
| VGG | 30.1$\pm$0.7 | 34.5$\pm$0.9 (4.4) | 27.0$\pm$0.7 | 31.8$\pm$1.1 (4.8) | 22.5$\pm$2.5 | 27.1$\pm$3.2 (4.6) |
| ConvNet | 35.4$\pm$0.6 | 35.8$\pm$0.8 (0.4) | 28.0$\pm$0.8 | 32.3$\pm$1.0 (4.3) | 20.1$\pm$4.5 | 27.3$\pm$3.2 (7.2) |
| AlexNet | 33.8$\pm$0.7 | 34.3$\pm$0.8 (0.5) | 31.0$\pm$0.6 | 32.3$\pm$0.5 (1.3) | 34.0$\pm$4.6 | 38.2$\pm$1.7 (4.2) |

### 4.3 Cross-architecture generalization

Cross-architecture evaluation is a critical step toward ensuring robust generalization across previously unseen architectures. We measured the quality of reduced dataset with privileged information on both pruning and distillation settings. To address the misalignment between the shapes of the learned feature/attention labels and the intermediate features of different network architectures, we introduced an additional fully connected layer that is trained alongside the evaluation model.

For pruning methods, we utilized ConvNet to synthesize feature labels for selected coresets, and benchmarked their performance across 6 distinct network architectures. As illustrated in Table 4, on the CIFAR-10 (fraction = 0.2%), all 3 methods yielded performance gains exceeding 20% on ConvNet. These methods consistently exhibit improvements exceeding 10% in most cases.

For distillation methods, we applied DRUPI to synthesize feature labels for reduced datasets initialized by DC and MTT. As detailed in Table 5, we conducted experiments on distilled dataset initialized by DC on CIFAR-10 (10 IPC), and trained models on 6 distinct network architectures, and evaluated the model performance across them. Notably, DRUPI achieved an 18.3% performance improvement over the baseline when training on VGG and evaluating on ResNet. Additional results on attention labels are provided in Appendix B.2. Table 9 presents the results of applying DRUPI to MTT on CIFAR-10 (1 IPC), showing significant gains, such as an 11.1% improvement when training on ConvNet and evaluating on AlexNet.

## 5 Discussion

Different ways for synthesizing privileged information. As discussed in Section 3.2, there are multiple methods for synthesizing feature labels. A straightforward approach is to assign feature labels using a pre-trained model. However, as shown in Figure 4, this can sometimes degrade dataset quality. In contrast, learning feature labels offers greater flexibility and adaptability. As demonstrated in Figure 4(a), reduced datasets with learned feature labels significantly outperform those with directly assigned features. This is because feature labels extracted from a pre-trained model often lead to overly discriminative features with low diversity. The empirical results also support the observation that overly discriminative feature labels with strong task supervision can hurt performance in Figure 3. More detailed results are provided in Table 10 in Appendix B.3.

Figure: Figure 4: (a) Comparison of different methods for obtaining feature labels in datasets initialized with various distillation methods. Our results indicate that learning-based methods yield the best performance. (b) Impact of feature label versatility and the utilization of multiple feature labels. We find that incorporating more feature labels produces a more robust reduced dataset, with averaging the features outperforming random selection. (c) Evaluation of supervision using different layers from a depth-3 ConvNet for synthesizing feature labels. Results show that, across different IPCs and datasets, using the final layer features for supervision generates the most effective reduced dataset.
Refer to caption: x4.png

Impact of feature label versatility and methods for utilizing feature labels. We investigated the effect of synthesizing multiple feature labels for a single data-label pair. As shown in Figure 4(b), experiments demonstrate that increasing the number of feature labels enhances performance, likely due to the greater versatility captured by additional features. However, too many feature labels for a single input can introduce excessive diversity, leading to degraded performance. This verifies the trade-off between the diversity and discriminability of feature labels. Furthermore, averaging multiple feature labels outperforms random selection, which enables us to save only the averaged feature labels. Hence, increasing the number of feature labels does not bring more storage overhead.

Layer choice for supervision features. We conducted an in-depth analysis to determine which ConvNet layer’s features are most effective for supervision. Specifically, we compared features extracted from the first, second, and final layers of a depth-3 ConvNet. Figure 4(c) shows that deeper layers consistently yielded better performance. This is likely due to the final layer’s ability to capture more complex and discriminative information, effectively representing high-level semantics. Therefore, we used the last layer’s features to supervise the synthesis of feature labels by default.

For more details on different methods of initializing feature labels, please refer to Appendix B.4. Ablation studies on the regression magnitude of MSE are provided in Appendix C.1. Additionally, we explored the potential of feature regression using losses beyond MSE in Appendix C.2.

## 6 Conclusion

In this paper, we introduced DRUPI, a novel framework that synthesizes privileged information for reduced datasets. To the best of our knowledge, DRUPI is the first approach to go beyond the traditional data-label paradigm by utilizing synthesized feature labels. Extensive experiments on ImageNet, CIFAR-10/100, and Tiny ImageNet validate the effectiveness of DRUPI, demonstrating significant improvements in model performance when integrated with existing reduction techniques. Additionally, we showed that achieving a balance between the discriminability and diversity of the synthesized feature labels is crucial for maximizing the quality of the reduced dataset.

## Appendix A Detailed Experimental Settings

### A.1 Computational Resources

The training was conducted on NVIDIA GPUs, specifically RTX 4090 and A100. All coreset selection experiments were run on A100 GPUs, while all DC-related experiments were carried out with RTX 4090. For MTT and DATM experiments on CIFAR-100 with IPC 10 and 50, as well as Tiny ImageNet, we utilized four NVIDIA A100 GPUs. Results of smaller datasets and lower IPC settings were conducted with RTX 4090.

### A.2 Coreset Selection

We first employed several coreset selection methods to initialize our reduced dataset, specifically using Herding (Welling, ), k-Center (Har-Peled & Mazumdar, ), and Forgetting (Toneva et al., ), where each data point was selected based on scores from a pre-trained ResNet-18. Next, we synthesized feature labels for the coreset using DC Zhao et al. (), assigning these labels to the intermediate features of a ConvNet trained for just one epoch. We also fine-tuned the initial images using the same method as for feature label learning, although we found that simply synthesizing feature labels could already bring performance improvement for reduced datasets.

Then we provide detailed settings for hyper-parameters. The hyper-parameters include:

- •
$\lambda_{reg}$: regularization coefficient, which controls the strength of the regularization term in the loss function.
- •
$\lambda_{task}$: task supervision coefficient, determining the discriminative power of the synthesized feature labels.
- •
$n_{feat}$: number of feature labels synthesized for a single data-label pair, which controls the diversity.

Unless otherwise specified, for the CIFAR-10 dataset, we set $\lambda_{reg}$ to 0.5, while for CIFAR-100, it was set to 5. Across all configurations, the task supervision coefficient $\lambda_{task}$ was set to 0.001, and we only synthesized one feature label for a single data-label pair ($n_{feat}=1$) in the reduced dataset.

### A.3 Dataset Distillation

For dataset distillation methods, we initialized the reduced datasets (images and labels) using distilled datasets and applied the same distillation method to synthesize feature labels. Specifically, we used DC (Zhao et al., ), MTT (Cazenavette et al., ), and DATM (Guo et al., ) for both data-label initialization and feature label synthesis.

We followed the original image-label synthesis settings of these distillation methods to generate feature labels. Table 6 provide the detailed hyperparameter settings used for experiments on CIFAR-10, CIFAR-100, Tiny ImageNet. Table 7 provides the parameter settings for the MTT method on the ImageNet subsets. By default, we set $n_{feat}$ to 1 and $\lambda_{reg}$ to 0.01. We explored different configurations of Images Per Class (IPC), specifically IPC = {1, 10, 50} for CIFAR-10 and CIFAR-100, and IPC = {1, 10} for Tiny ImageNet.

**Table 6: Hyperparameter settings for dataset distillation methods. ^† denotes that soft labels were synthesized in this set of experiments to further enrich the privileged information in the synthetic dataset. We used a pre-trained model to synthesize soft labels.**
| Dataset | IPC | Method | $\lambda_{reg}$ | $\lambda_{task}$ | $n_{feat}$ |
| --- | --- | --- | --- | --- | --- |
| CIFAR-10 | 1 | DC | 1.5 | 0.1 | 1 |
| MTT^† | 0.5 | 0.001 | 1 |  |  |
| DATM | 0.5 | 0.001 | 5 |  |  |
| 10 | DC | 0.5 | 0.1 | 1 |  |
| MTT | 0.0005 | 0.01 | 3 |  |  |
| DATM | 0.05 | 0.1 | 3 |  |  |
| 50 | DC | 0.01 | 0.01 | 1 |  |
| MTT | 0.05 | 0.001 | 1 |  |  |
| DATM | 0.05 | 0.001 | 1 |  |  |
| CIFAR-100 | 1 | DC | 1.5 | 0.1 | 1 |
| MTT | 0.5 | 0.01 | 3 |  |  |
| DATM | 0.05 | 0.01 | 1 |  |  |
| 10 | DC^† | 0.001 | 0.005 | 1 |  |
| MTT | 0.005 | 0.001 | 1 |  |  |
| DATM | 0.05 | 0.001 | 1 |  |  |
| 50 | DC^† | 0.5 | 0.1 | 1 |  |
| MTT^† | 0.5 | 0.01 | 3 |  |  |
| DATM | 0.0005 | 0.001 | 1 |  |  |
| Tiny ImageNet | 1 | MTT^† | 0.5 | 0.0001 | 3 |
| DATM | 0.005 | 0.001 | 1 |  |  |
| 10 | MTT^† | 0.005 | 0.001 | 1 |  |
| DATM | 0.005 | 0.001 | 1 |  |  |

**Table 7: Hyperparameter settings for ImageNet subsets used in MTT experiments.**
| Dataset | IPC | $\lambda_{reg}$ |
| --- | --- | --- |
| ImageNette | 1 | 0.005 |
| 10 | 0.005 |  |
| ImageWoof | 1 | 0.005 |
| 10 | 0.5 |  |
| ImageFruit | 1 | 0.005 |
| 10 | 0.5 |  |
| ImageMeow | 1 | 0.00005 |
| 10 | 0.5 |  |
| ImageYellow | 1 | 0.005 |
| 10 | 0.5 |  |

For each method, the hyperparameters for feature synthesis are fine-tuned across different datasets and IPC settings to achieve optimal performance. For instance, in CIFAR-10 (1 IPC), the DC method utilizes $\lambda_{reg}=1.5$, $\lambda_{task}=0.1$, and $n_{feat}=1$. In contrast, for 50 IPC, the same method adjusts its hyperparameters to $\lambda_{reg}=0.001$, $\lambda_{task}=0.005$, and $n_{feat}=1$. Similar fine-tuning is performed for all datasets and IPC values across each method.

For distilled datasets with feature labels, we also tried to incorporate additional forms of privileged information, such as soft labels, to further enrich the privileged information in the synthetic dataset. Specifically, we used a pre-trained network to generate soft labels for the distilled dataset. Some methods like DATM have already learned soft labels. We only synthesized soft labels for DC and MTT based reduced datasets. We provide futher results on soft labels in Table 12.

## Appendix B Additional Results on DRUPI

### B.1 Further Performance Results

In Section 4.2, we presented results for two dataset distillation methods, DC and MTT. Here, we provide the results for DATM on the CIFAR-10 and CIFAR-100 datasets. We initialized the images and labels with DATM and used it to synthesize feature labels for the reduced dataset. Experiments were conducted using a ConvNet for both distillation and evaluation tasks. We additionally provide baselines such as (Zhou et al., ; Loo et al., ; Cui et al., ; Du et al., ; Guo et al., ) .Table 8 shows the performance improvements of DRUPI over various existing methods under different IPC settings. Specifically, the datasets used in these experiments include CIFAR-10 with IPC = {1, 10, 50} and CIFAR-100 with IPC = {1, 10}. Each dataset was further evaluated with varying data fractions, allowing us to assess the generalization of the methods across different data availability scenarios.

**Table 8: The application of DRUPI to DATM across CIFAR-10, CIFAR-100, accompanied by a comparative analysis with existing methods. ConvNet is utilized for both distillation and evaluation. Our methodology demonstrates enhanced performance compared to previous results.The ↑ symbol signifies performance enhancements compared to random selection.**
| Dataset | CIFAR-10 | CIFAR-100 |  |  |  |
| --- | --- | --- | --- | --- | --- |
| IPC | 1 | 10 | 50 | 1 | 10 |
| Fraction (%) | 0.02 | 0.2 | 1 | 0.2 | 2 |
| Random | 15.4$\pm$0.3 | 31.0$\pm$0.5 | 50.6$\pm$0.3 | 4.2$\pm$0.3 | 14.6$\pm$0.5 |
| KIP | 49.9$\pm$0.2 | 62.7$\pm$0.3 | 68.6$\pm$0.3 | 15.7$\pm$0.2 | 28.3$\pm$0.1 |
| FRePo | 46.8$\pm$0.7 | 65.5$\pm$0.4 | 71.7$\pm$0.4 | 28.7$\pm$0.2 | 42.5$\pm$0.4 |
| RCIG | 53.9$\pm$1.0 | 69.1$\pm$0.4 | 73.5$\pm$0.3 | 39.3$\pm$0.4 | 44.1$\pm$0.4 |
| DM | 26.0$\pm$0.8 | 48.9$\pm$0.6 | 63.0$\pm$0.4 | 11.4$\pm$0.3 | 29.7$\pm$0.3 |
| DSA | 28.8$\pm$0.7 | 52.1$\pm$0.5 | 60.6$\pm$0.5 | 13.9$\pm$0.3 | 32.3$\pm$0.3 |
| DCC | 32.9$\pm$0.8 | 49.4$\pm$0.5 | 61.6$\pm$0.4 | 13.3$\pm$0.3 | 30.6$\pm$0.4 |
| DSAC | 34.0$\pm$0.7 | 54.5$\pm$0.5 | 64.2$\pm$0.4 | 14.6$\pm$0.3 | 33.5$\pm$0.3 |
| CAFE | 30.3$\pm$1.1 | 46.3$\pm$0.6 | 55.5$\pm$0.6 | 12.9$\pm$0.3 | 27.8$\pm$0.3 |
| IDM | 45.6$\pm$0.7 | 58.6$\pm$0.1 | 67.5$\pm$0.1 | 20.1$\pm$0.3 | 45.1$\pm$0.1 |
| TESLA | 48.5$\pm$0.8 | 66.4$\pm$0.8 | 72.6$\pm$0.7 | 24.8$\pm$0.4 | 41.7$\pm$0.3 |
| FTD | 46.0$\pm$0.4 | 65.3$\pm$0.4 | 73.2$\pm$0.2 | 24.4$\pm$0.4 | 42.5$\pm$0.2 |
| DATM | 46.9$\pm$0.5 | 66.8$\pm$0.2 | 76.1$\pm$0.3 | 27.9$\pm$0.2 | 47.2$\pm$0.4 |
| +DRUPI | 48.0$\pm$0.4 | 67.8$\pm$0.3 | 76.4$\pm$0.3 | 28.4$\pm$0.5 | 47.6$\pm$0.2 |
| ↑ | 1.1 | 1.0 | 0.3 | 0.5 | 0.4 |
| Full Dataset | 84.8$\pm$0.1 | 56.2$\pm$0.3 |  |  |  |

### B.2 Results on Cross-Architecture Evaluation with Attention Labels

In addition to the results presented in Table 4 and 5 of the main paper, which summarize the performance of utilizing the synthesized datasets from pruning and dataset distillation to initialize the feature labels, we provide additional results in Table 9 to evaluate the cross-architecture generalization by using both feature labels and attention labels on different reduced datasets.

Table 9 illustrates the cross-architecture evaluations conducted for both dataset condensation (DC) and MTT across various network architectures. In these experiments, we propose DRUPI-F (feature labels learned by DC) and its variant DRUPI-A, which is further enhanced by pooling through spatial attention. Similarly, feature labels synthesized with MTT are also average pooled into channel attention labels. We utilized several networks for cross evaluatoin, including LeNet, MLP, ResNet, VGG, ConvNet, and AlexNet.

We provide examples to illustrate the operation. For example, the first layer feature of a depth-3 ConvNet can be used to supervise the learning of feature labels. In this case, a single feature label takes the form of $Ch\times H\times W$ (*e.g.*, $128\times 16\times 16$), which is reduced to a $128\times 1\times 1$ channel attention label after average pooling. Attention labels provide a more efficient way for using privileged information.

Additionally, during the cross-architecture process, if the features obtained from the training and testing models differ in shape or dimensionality, we employ an additional fully connected (FC) layer to align the features. For example, if the feature representation from the training model has a shape of $128\times 16\times 16$ and the testing model’s feature representation is $64\times 16\times 16$, the FC layer reshapes the $128\times 16\times 16$ feature into the $64\times 16\times 16$ format. The FC takes the input feature from the source architecture and transforms it into a format compatible with the target architecture by adjusting the dimensionality of the feature space.

The table demonstrates that initializing the reduced dataset with feature labels assigned through intermediate features of pre-trained networks leads to significant performance improvements across all architectures. Further gains are observed when channel attention pooling is applied. By leveraging Eq. (5) to learn and update the feature labels, the reduced dataset consistently yields competitive or improved performance across different networks. This highlights the benefits of feature and attention labels in enhancing model generalization, with particularly notable improvements observed in ConvNet and AlexNet settings. Channel attention pooling further contributes to these gains, reinforcing the effectiveness of this approach across architectures.

**Table 9: Cross-architecture performance comparison using feature labels and channel attention labels for initialization and learning on reduced datasets. We utilized reduced datasets initialized by both the DC and MTT methods. DC was employed for learning both feature labels and attention labels, while MTT was used exclusively for attention label learning through average pooling.**
|  | LeNet | MLP | ResNet | VGG | ConvNet | AlexNet |
| --- | --- | --- | --- | --- | --- | --- |
| DC | 16.8$\pm$5.2 | 28.0$\pm$0.6 | 36.2$\pm$1.6 | 35.4$\pm$0.6 | 44.5$\pm$0.5 | 20.1$\pm$4.5 |
| +DRUPI-F | 24.7$\pm$4.3 | 32.3$\pm$1.0 | 37.5$\pm$1.8 | 35.8$\pm$0.8 | 47.1$\pm$0.9 | 27.3$\pm$3.2 |
| +DRUPI-A | 23.7$\pm$6.6 | 27.9$\pm$0.5 | 37.7$\pm$1.1 | 35.6$\pm$0.7 | 45.6$\pm$0.5 | 25.3$\pm$3.0 |
| MTT | 29.1$\pm$1.5 | 29.1$\pm$0.4 | 35.1$\pm$1.1 | 31.4$\pm$0.9 | 45.6$\pm$0.7 | 24.0$\pm$0.8 |
| +DRUPI-A | 31.6$\pm$1.7 | 29.4$\pm$0.5 | 36.7$\pm$5.2 | 37.9$\pm$2.8 | 47.0$\pm$0.7 | 34.1$\pm$4.4 |

### B.3 Additional Results on Different Methods for Synthesizing Privileged Information

As a complement to Figure 4(a), Table 10 provides a more comprehensive overview of the experimental results on different methods for synthesizing privileged information. Specifically, it presents a detailed comparison between direct feature label assignment and the DRUPI framework across three dataset distillation methods: DC, MTT, and DATM. These evaluations were conducted on CIFAR-10, CIFAR-100, and Tiny ImageNet datasets with varying images per class (IPC) values, and include results for different fractions of the full dataset.

For each method, the results are provided for various IPC settings, showing the performance both with direct feature assignment and with the inclusion of DRUPI (feature label). The DRUPI framework consistently improves performance across different methods and datasets, as indicated by the higher accuracy values. Specifically, DRUPI shows substantial improvements over direct assignment, particularly in the low-data regime, such as CIFAR-100 1 IPC and Tiny ImageNet 1 IPC. These findings underscore the effectiveness of DRUPI in enhancing the generalization and performance of dataset distillation techniques, even with limited data.

**Table 10: Experimental results comparing direct feature label assignment and the DRUPI framework across three dataset distillation methods (DC, MTT, DATM) on various datasets with different IPC values.**
| Dataset | CIFAR-10 | CIFAR-100 | Tiny ImageNet |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| IPC | 1 | 10 | 50 | 1 | 10 | 50 | 1 |
| Fraction (%) | 0.02 | 0.2 | 1 | 0.2 | 2 | 10 | 0.2 |
| DC | 28.3$\pm$0.5 | 44.9$\pm$0.5 | 53.9$\pm$0.5 | 12.8$\pm$0.3 | 25.2$\pm$0.3 | 29.8$\pm$0.3 | - |
| Directly Assign | 28.3$\pm$0.8 | 45.1$\pm$0.5 | 54.1$\pm$0.4 | 12.7$\pm$0.4 | 25.1$\pm$0.4 | 29.8$\pm$0.4 | - |
| DRUPI | 31.5$\pm$0.9 | 47.4$\pm$0.9 | 55.0$\pm$0.5 | 14.9$\pm$0.4 | 29.2$\pm$0.5 | 30.9$\pm$0.5 | - |
| MTT | 46.2$\pm$0.8 | 65.4$\pm$0.7 | 71.6$\pm$0.2 | 24.3$\pm$0.3 | 39.7$\pm$0.4 | 47.7$\pm$0.2 | 8.8$\pm$0.3 |
| Directly Assign | 40.8$\pm$1.8 | 56.2$\pm$1.1 | 66.1$\pm$0.5 | 23.9$\pm$0.5 | 38.1$\pm$0.4 | 47.4$\pm$0.2 | 8.1$\pm$0.4 |
| DRUPI | 47.4$\pm$0.5 | 65.8$\pm$0.6 | 71.7$\pm$0.2 | 25.6$\pm$0.4 | 40.8$\pm$0.3 | 47.9$\pm$0.3 | 11.0$\pm$0.1 |
| DATM | 46.9$\pm$0.5 | 66.8$\pm$0.2 | 76.1$\pm$0.3 | 27.9$\pm$0.2 | 47.2$\pm$0.4 | 55.0$\pm$0.2 | 17.1$\pm$0.3 |
| Directly Assign | 46.3$\pm$0.7 | 64.5$\pm$0.5 | 73.7$\pm$0.4 | 26.6$\pm$0.3 | 36.2$\pm$0.5 | 55.5$\pm$0.3 | 15.6$\pm$0.1 |
| DRUPI | 48.0$\pm$0.4 | 67.8$\pm$0.3 | 76.4$\pm$0.3 | 28.4$\pm$0.5 | 47.6$\pm$0.2 | 55.0$\pm$0.1 | 17.6$\pm$0.1 |
| Full Dataset | 84.8$\pm$0.1 | 56.2$\pm$0.3 | 37.6$\pm$0.4 |  |  |  |  |

### B.4 Effects on Feature Label Initialization

We further explored the issue of feature label initialization. Specifically, initialization can be performed either by using random noise or by feeding synthetic images into a ConvNet to extract intermediate layer features for initialization.

Figure 5 presents a performance comparison between two feature initialization approaches—random noise and assigned features—across the CIFAR-10 and CIFAR-100 datasets with varying images per class (IPC) settings. In this comparison, noise (yellow bars) refers to randomly initialized feature labels, while real (blue bars) refers to initialization using features extracted from synthetic datasets passed through the network.

For CIFAR-10 (1 IPC) setting, the initialization with assigned features method achieves significantly higher performance (around 33%) compared to the noise-based initialization (approximately 30%). A similar trend is observed in the CIFAR-10 (10 IPC) case, where initialization with assigned features substantially outperforms noise.

On the CIFAR-100 (1 IPC) dataset, initialization with assigned features also demonstrates better performance, with a clear margin over noise initialization, where using assigned features reaches around 29% while noise remains below 28%. The same pattern holds for CIFAR-100 (10 IPC), where initialization with assigned features achieves a accruacy of over 47%, far exceeding the noise initialization’s performance of approximately 45%.

Figure: Figure 5: Comparison of noise initialization (yellow) and initialization with assigned features (blue) from a pre-trained ConvNet on CIFAR-10 and CIFAR-100 across different IPC settings.
Refer to caption: x5.png

## Appendix C Ablation on Regression Supervision

### C.1 Sensitivity of Regression Magnitude

We further investigated the influence of $\lambda_{reg}$ on the model’s performance. The results in Table 11 demonstrate that varying the MSE regularization parameter $\lambda_{reg}$ does not significantly impact the final accuracy. While there is a slight improvement as $\lambda_{reg}$ increases, from 28.91% at $\lambda_{reg}=0.05$ to 30.74% at $\lambda_{reg}=10$, the overall effect remains relatively small. This suggests that the model’s performance is not highly sensitive to the regularization weight for MSE loss within the tested range, indicating that other factors may have a more dominant influence on accuracy. In this case, regularization helps prevent overfitting but does not drastically change the model’s ability to generalize within the 1 IPC setting for CIFAR-10.

**Table 11: Results on CIFAR-10 1 IPC for different $\lambda_{reg}$.**
| $\lambda_{reg}$ | 0.05 | 0.1 | 0.5 | 1 | 5 | 10 |
| --- | --- | --- | --- | --- | --- | --- |
| Acc (%) | 28.91 | 29.2 | 30.42 | 30.53 | 30.67 | 30.74 |

### C.2 Further Regression Supervision Objectives

We extended the use of feature labels by integrating additional supervision mechanisms, including CE loss and InfoNCE loss, to enhance the distillation process. Additionally, we performed ablation studies to evaluate the impact of using soft labels. These components contribute to a more structured and informative representation learning framework.

First, we introduced CE loss and InfoNCE loss to provide direct regularization between the feature labels and the supervision features. In line with previous work, we also incorporated soft labels as privileged information.

The combination of CE loss, InfoNCE loss, and soft labels consistently yielded the best performance across our ablation studies, as shown in Tables 12. We found that incorporating InfoNCE or CE loss marginally improved the quality of the reduced dataset, while soft labels, when combined with feature labels, provided a more significant boost in performance.

**Table 12: Ablation study of different components in DRUPI across CIFAR-10 and CIFAR-100 (1 IPC). The table compares the effects of feature labels, CE loss ($\mathcal{L}_{CE}$), contrastive loss ($\mathcal{L}_{IN}$), and soft labels on model performance.**
| Feature label | $\mathcal{L}_{CE}$ | $\mathcal{L}_{IN}$ | Soft label | Acc (%) |
| --- | --- | --- | --- | --- |
|  |  |  |  | 28.30 |
| ✓ |  |  |  | 31.13 |
| ✓ | ✓ |  |  | 31.54 |
| ✓ |  | ✓ |  | 31.47 |
| ✓ | ✓ | ✓ |  | 31.90 |
| ✓ |  |  | ✓ | 32.28 |

## Appendix D Pseudo Code of DRUPI

Algorithm  1 outlines the process where we initialize the reduced dataset with assigned feature labels based on the synthetic dataset from dataset condensation (DC). Subsequently, we update the reduced dataset by learning the feature labels, which are progressively refined during the training process.

Figure: Algorithm 1 Dataset Reduction Using Privileged Information (DRUPI) For DC

## Peer Reviews

confidence: 5
contribution: 2
initial_rating: 3
presentation: 2
questions: 1.	Since the attention label is average pooled from the feature label, are these two labels redundant?
2.	Equation 6 lacks detail on how feature labels are used in training. The statement, "we apply the same pooling strategy for intermediate features of the given model to calculate the MSE loss between the intermediate features and given feature labels." is unclear and seems to be missing context.
3.	The theoretical analysis relies on existing VC theory without specifics related to the proposed method.
soundness: 2
strengths: 1.	The use of feature labels and attention labels as privileged information is straightforward and easy to implement.
summary: This paper explores the potential to synthesize additional information beyond the standard data-label pairs as a learning target to enhance model training. The authors introduce Dataset Reduction Using Privileged Information (DRUPI), which augments dataset reduction by generating privileged information alongside a reduced dataset. This privileged information may include feature labels or attention labels, providing auxiliary supervision to strengthen model learning. The study finds that effective feature labels strike a balance between being too discriminative and too diverse, with a moderate level yielding good results in enhancing the efficacy of the reduced dataset. Extensive experiments on ImageNet, CIFAR-10/100, and Tiny ImageNet demonstrate that DRUPI integrates smoothly with existing dataset reduction techniques, yielding substantial performance improvements.
weaknesses: 1.	The paper does not sufficiently acknowledge prior work, for instance, it should compare and discuss the proposed method with previous dataset distillation methods that use data-soft label structures, which are similar in approach.
2.	The novelty of the privileged information (feature and attention labels derived from feature labels) is limited and similar to approaches such as Re-labeling ImageNet and FKD.
3.	The theoretical analysis using VC theory is general and not specific to the proposed method, making it insufficient to substantiate the approach.
4.	The baselines in Table 1 are weak. In Table 2 of many cases, only marginal improvements are shown over prior methods. Additionally, the lack of results on full ImageNet weakens the validity of the experiments and the effectiveness.
5.	Baseline approaches like DC and MTT are somewhat outdated, newer stronger methods such as SRe2L and RDED are not included in the comparison.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. Does the method show cross-architecture generalization on ViT models?
2. Can HeLlO support strong performance at higher IPCs, such as 100 or 200?
3. How does the proposed framework perform on downstream tasks like object detection or segmentation?
4. It would be interesting to see an ablation study examining the effects of soft labels, ZCA, and EMA on DRUPI.
5. Could the authors include visualizations of some synthetic images for different IPCs? I am curious to see how the synthetic images look.
soundness: 2
strengths: [S1] The paper presents the use of privileged information, such as feature and attention labels, to improve dataset reduction, demonstrating performance gains on datasets like CIFAR-10/100 and ImageNet.

[S2] The approach integrates well with existing dataset reduction methods, highlighting its flexibility.

[S3] It is supported by a solid theoretical foundation based on VC theory, which explains how privileged information enhances generalization and learning.

[S4] The paper is well-written and easy to understand.
summary: This paper introduces a novel method for enhancing dataset reduction by incorporating privileged information, such as feature and attention labels, to improve model training. This approach goes beyond traditional data-label compression, providing additional supervision that leads to better generalization and performance. The method demonstrates significant improvements in datasets like CIFAR-10/100 and ImageNet, outperforming baseline reduction techniques. While the results are promising, the paper does not fully address distillation costs or generalization to tasks like object detection. Overall, DRUPI offers a fresh perspective on dataset reduction with meaningful contributions to the field.
weaknesses: [W1] The paper lacks a thorough analysis of computational costs, which is critical for assessing the method’s efficiency. I suggest reporting specific metrics such as training time, memory usage, and storage overhead when generating and storing privileged information. It would be helpful to compare these costs with baseline methods to provide a clear understanding of DRUPI’s performance in terms of resource consumption. This analysis will clarify the computational trade-offs involved in using privileged information versus traditional dataset distillation methods.

[W2] Concerns about generalizability remain, particularly regarding the method's applicability to high-resolution datasets and different model architectures. I recommend testing DRUPI on 256x256 ImageNet-1k and on practical datasets such as clinical or aerial images to address this. Additionally, evaluating its performance on ViT models for cross-architecture generalization would provide further insights into the method's broader applicability across deep architectures.

[W3] While the practical applications of dataset distillation are not explored in depth, I understand this may be beyond the scope of a single paper. However, it would strengthen the paper to include a discussion on DRUPI’s potential applications, particularly in areas like neural architecture search (NAS) or continual learning. If feasible, adding an experiment or case study in one of these domains could demonstrate the method’s utility in real-world scenarios.

[W4] The paper does not include comparisons with key state-of-the-art methods like DREAM, DataDAM, and SeqMatch in Table 2. Including these methods would provide a more comprehensive evaluation of DRUPI’s effectiveness. If there are reasons why these methods were not compared, it would be helpful for the authors to explain this and clarify how DRUPI relates to these recent approaches.

**References**:
- [a] Liu, Yanqing, et al. "DREAM: Efficient dataset distillation by representative matching." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.
- [b] Sajedi, Ahmad, et al. "DataDAM: Efficient dataset distillation with attention matching." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.
- [c] Yin, Zeyuan, et al. "Squeeze, recover and relabel: Dataset condensation at ImageNet scale from a new perspective." Advances in Neural Information Processing Systems 36 (2024).

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. The description of the main method is elaborated based on bi-level optimization distillation methods, DC. However, I'd like to know details of how privileged information is learned with coreset selection methods, as improvements shown in Table 1 are significant. The code in the supplementary material also doesn't include this part. (related to weakness 1)

2. As the synthetic dataset now receives more information when being updated, how does the newly introduced feature label affect the distilled dataset? For example, Figure 3(a) shows the t-SNE visualization of feature labels under different $\lambda_{task}$, and the conclusion is that feature labels should balance diversity and discriminative power. However, how does this affect the data? Some visualization could be made to further discuss the impact of different feature labels on the synthetic data. (related to weakness 2)

3. Since the ultimate objective of dataset reduction is to find a smaller set of data on which the model can achieve similar or even better performance,  and lossless performances have only been seen on large IPCs (DATM[1]), it is recommended to show DRUPI can consistently improve baselines even when IPC goes larger (based on reported results, the improvement brought by DRUPI becomes smaller when IPC increases). (related to weakness 3)

4. This question is for discussion. Authors argue that the data-label format should be deprecated for reduced datasets. To improve this, feature labels are introduced to improve performance, and evaluation models are required to adopt a new training objective. 
However, at the end of the day, dataset reduction should focus primarily on the **dataset** itself instead of improving the test accuracy by introducing a lot more information. Thus,  to fill this gap, I am curious about the evaluation performance of the model trained on $D^{*}\_{S}$ following the same evaluation method of MTT/DC (without feature label matching) to see if feature labels improve the distilled dataset's quality. 


[1] Towards Lossless Dataset Distillation via Difficulty-Aligned Trajectory Matching, ICLR 2024
soundness: 3
strengths: 1. All the charts in the text are clear and easy to understand.
2. The theoretical analysis further supports the proposed method
3. The distilled dataset can generalize well across various architectures
summary: This paper proposes learnable feature labels in the dataset pruning/distillation task. The feature labels are initialized by intermediate outputs of a pre-trained model and are optimized jointly with synthetic data. On several benchmarks, the proposed method improves baseline methods, especially under the pruning setting.
weaknesses: 1. Some details of the method are not clarified, especially on the pruning part. The authors described the algorithm in the context of DC but have yet to explain how pruning can incorporate learnable feature labels.
2. Analysis in section 5 is trivial and **doesn't** provide enough insights. More discussion can be included.
3. The scalability on **large** IPCs may not be good enough according to the reported results in Table 2 and Appendix.

For the weaknesses mentioned above, I'll raise the corresponding questions below.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: Please see weakness.
soundness: 3
strengths: The introduction of feature labels as privileged information goes beyond the traditional data-label paradigm, providing additional supervision that improves model generalization and robustness.

The paper conducts an extensive set of experiments across a variety of datasets (CIFAR-10/100, Tiny ImageNet, and ImageNet subsets) and methods (coreset selection and dataset distillation), clearly demonstrating the efficacy of DRUPI in improving performance.

The authors provide a rigorous theoretical framework based on VC theory, which justifies the efficacy of the proposed approach, enhancing the contribution and ensuring the generalizability of the paper.
summary: This paper introduces DRUPI, a novel approach for dataset reduction that synthesizes privileged information, such as feature labels, alongside traditional data-label pairs. By leveraging privileged information, DRUPI aims to improve the efficacy of reduced datasets across various machine-learning tasks. The paper thoroughly evaluates the method on multiple benchmarks, including CIFAR-10/100, Tiny ImageNet, and ImageNet subsets, demonstrating significant performance improvements compared to baseline dataset reduction techniques.
weaknesses: DRUPI uses the output of a pre-trained model as privileged information and further applies an MSE loss to enforce consistency between the model’s predictions and the privileged information. This approach shares similarities with knowledge distillation techniques. While the method is straightforward and yields competitive results, it inherently limits the upper bound of DRUPI's performance to the quality of the pre-trained model. Consequently, if the pre-trained model is suboptimal, the privileged information may not fully capture the necessary features, potentially constraining the overall effectiveness of DRUPI.


Given the conceptual overlap between DRUPI and knowledge distillation, it would have been beneficial for the paper to include a direct experimental comparison with established knowledge distillation techniques. Such comparisons would provide more insight into how DRUPI differentiates itself from or improves upon these methods, as both share the goal of leveraging additional information to enhance model performance.

The core innovation of DRUPI lies in the introduction of the output from a pre-trained model and the use of consistency constraints to guide model training. However, this concept has been extensively explored in the domain of knowledge distillation. While DRUPI's application of this idea within the context of dataset reduction is novel, the paper does not sufficiently acknowledge the wealth of prior work in knowledge distillation, where similar strategies have been widely adopted.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: No
soundness: 2
strengths: 1. The core contribution of this paper is the introduction of additional supervised information, specifically privileged information, alongside input data and soft labels. This paradigm effectively extends the capabilities of the dataset condensation algorithm.

2. The authors validate DRUPI on multiple datasets and demonstrate its integration with several classical dataset condensation algorithms, confirming its generalizability to some extent.
summary: This paper extends dataset condensation—referred to by the author as dataset reduction, a more general term I prefer, as the author's definition in the introduction does not encompass all dataset condensation methods—from conventional image-label pairs to image-label-privileged information pairs. It provides an in-depth analysis of the effectiveness of this approach in various settings, including different label types (feature and attention labels) and task-oriented synthesis to balance discriminability and diversity. The experiments demonstrate that the algorithm can be integrated into several existing dataset condensation methods.

This approach offers an intriguing extension of feature-based knowledge distillation. As a scholar in the fields of knowledge distillation and dataset condensation, I have naturally considered this idea and attempted to implement it. However, I encountered several challenges that ultimately led me to abandon it. Therefore, I believe the authors must address key challenges related to the introduction of privileged information, particularly regarding generalizability. For instance, does the use of privileged information violate the core principles of dataset condensation? (A similar question arose during the ICLR submission process for paper [1].) Additionally, is the extra storage overhead introduced by privileged information manageable?

All in all, this is an interesting approach and I read the article very carefully. However, this article is missing a lot in terms of experimentation, which I'll point out in a follow-up WEakness.

[1] Scaling Up Dataset Distillation to ImageNet-1K with Constant Memory, ICML, 2022
weaknesses: Overall, the biggest drawback of this work is what I mentioned in Summary, the generalization of this paradigm, does it violate the concept of dataset condensation? Is the additional storage overhead introduced acceptable? Besides that, some expressions in the paper need to be improved.

**Presentation:**

1. The concept of dataset distillation is misunderstood. The authors classify dataset reduction into coreset selection and dataset distillation (line 125). However, this raises the question: how should algorithms that synthesize data without training, such as RDED [1], be categorized? The authors argue that all dataset distillation methods adhere to the distillation paradigm and synthesize only unseen data from the training set. In contrast, coreset selection is restricted to algorithms that select a subset of the original dataset.

2. The authors later make a definitional mistake by assuming that all dataset condensation that requires training is bi-level optimization paradigm, which is false. None of the researches [1,2,3,4,5,6] require bi-level optimization.

3. On line 253, Figure 3(a)(c) is a typo.

4. The theoretical analysis provided by the authors was unclear to me, and I am interested in seeing how other reviewers perceive it. Overall, it did not provide any significant insights and felt as though it was included primarily to meet the expectation of a theoretical component.

[1] On the Diversity and Realism of Distilled Dataset: An Efficient Dataset Distillation Paradigm, CVPR, 2024

[2] Squeeze, Recover and Relabel: Dataset Condensation at ImageNet Scale From A New Perspective, NeurIPS, 2023

[3] Generalized Large-Scale Data Condensation via Various Backbone and Statistical Matching, CVPR, 2024

[4] Information Compensation: A Fix for Any-scale Dataset Distillation, CVPR Workshop, 2024

[5] Elucidating the Design Space of Dataset Condensation, NeurIPS, 2024

[6] Diversity-Driven Synthesis: Enhancing Dataset Distillation through Directed Weight Adjustment, NeurIPS, 2024


**Method**

1. The full paper, including the appendices, does not address the additional storage overhead introduced by the use of privileged information. This omission is significant, as the storage burden imposed by the model's intermediate layer outputs is much greater than that of soft labels. For example, on the full ImageNet-1k dataset, the soft labels for IPC 50 alone amount to tens of gigabytes.

2. The generalizability of DRUPI is a significant concern. The results presented in Table 4 are based solely on experiments using convolutional model. I believe this paradigm may not be applicable when using a Vision Transformer as the teacher model and convolutional model as the student model. Furthermore, I think the paper should detail how the authors solved the gap in the feature map due to dimensional inconsistency.

3. The authors likewise need to clarify whether this paradigm violates the classical paradigm of dataset condensation.

**Experiment**

1. The authors tested DRUPI with only a few algorithms, specifically MTT and DC, which is insufficient. To date, dataset condensation algorithms that require training have consistently outperformed dataset selection algorithms in terms of performance. The authors should consider evaluating DRUPI with additional algorithms, such as gradient matching and distribution matching.

2. In addition, decoupled distillation (as referenced in [1-6]) represents the current state-of-the-art (SOTA) algorithm for large-scale datasets, including the full 224x224 ImageNet-1k and the full 224x224 ImageNet-21k. The authors should build on these algorithms to apply DRUPI. Restricting the experiments to small-scale datasets, such as the subset of ImageNet-1k used in the paper, does not significantly advance the field of dataset condensation. This limitation also hinders the potential for dataset condensation to play a meaningful role in the era of foundation models.