Grounding and Enhancing Informativeness and Utility in Dataset Distillation
Authors: Shaobo Wang
Yantai Yang
Guo Chen
Peiru Li
Kaixin Li
Yufa Zhou
Zhaorun Chen
Linfeng Zhang
Venue: ICLR 2026 Poster (2026)
Keywords: Explainable AI
Generative AI
Primary area: generative models
TL;DR: We propose a theoretically principled dataset distillation method which achieves SOTA performance.

Abstract: Dataset Distillation (DD) seeks to create a compact dataset from a large, real-world dataset. While recent methods often rely on heuristic approaches to balance efficiency and quality, the fundamental relationship between original and synthetic data remains underexplored. This paper revisits knowledge distillation-based dataset distillation within a solid theoretical framework. We introduce the concepts of Informativeness and Utility, capturing crucial information within a sample and essential samples in the training set, respectively. Building on these principles, we define \textit{optimal dataset distillation} mathematically. We then present InfoUtil, a framework that balances informativeness and utility in synthesizing the distilled dataset. InfoUtil incorporates two key components: (1) game-theoretic informativeness maximization using Shapley Value attribution to extract key information from samples, and (2) principled utility maximization by selecting globally influential samples based on Gradient Norm. These components ensure that the distilled dataset is both informative and utility-optimized. Experiments demonstrate that our method achieves a 6.1\% performance improvement over the previous state-of-the-art approach on ImageNet-1K dataset using ResNet-18.

Published as a conference paper at ICLR 2026
GROUNDING AND ENHANCING INFORMATIVENESS AND UTILITY IN DATASET DISTILLATION
Shaobo Wangf 112
Yantai Yang', Guo Chen'
Peiru Li
Kaixin Li
Yufa Zhou
Zhaorun Chen
Linfeng Zhangi 1,2
EPIC Lab, STTU
Shanghai Jiao Tong University National University of Singapore
Duke University
The University of Chicago 'Corresponding authors
E-mail:
[shaobowang1000
/haggiiggggiagggggggggggggge edu. cn
Code: https://githul. wttp:hu...ooo....eea.eeaaeee htt.p//.....omg.taaaa///ttttttt.
ABSTRACT
Dataset Distillation (DD) seeks to create a compact dataset from large, real- world dataset. While recent methods often rely heuristic approaches to bal- ance efficiency and quality, the fundamental relationship between original and synthetic data remains underexplored. This paper revisits knowledge distillation- based dataset distillation within a solid theoretical framework. We introduce the concepts of Informativeness and Utility, capturing crucial information within a sample and essential samples in the training set, respectively. Building on these principles, we define optimal dataset distillation mathematically. We then present InfoUtil, framework that balances informativeness and utility in synthesizing the distilled dataset. InfoUtil incorporates two key components: (1) game-theoretn theoreic informativeness maximization using Shapley Value attribution to extract key infor- mation from samples, and (2) principled utility maximization by selecting globally influential samples based Gradient Norm. These components ensure that the distilled dataset both informative and utility-optimized. Experiments demon- strate that our method achieves 6.1% 1% performance improvement over the previ- ous state-of-the-an approach on ImageNet- KK dataset using ResNet-18.
Cock
Brambling
Otter
(a) RDED
Random Cropping & Loss Scoring (not interpretable or theorctically principled)
(b) InfoUtil (Ours)
Attribution Cropping GradNorm Scoring (both interpretable and theoretically principled)
Figure 1: Comparison of visualization results between previous method (a) RDED Sun et al. 2024 and (b) our InfoUtil. Unlike prior methods relying on random selection and intuitive scoring, In- foUtil is both interpretable and theoretically grounded. It synthesizes images that more accurately capture semantically meaningful regions with principled scores. Prioritizing core content over irrel- evant details like background elements ensures more focused and meaningful representation.
INTRODUCTION
Dataset distillation (DD) WWagg al. 2018 Sachdeva & McAuley 2023 has emerged as promis- ing approach for enabling vision models to achieve performance comparable to training on large datasets, but with only small set of synthetic samples. The core idea behind DD is compress large

---

Published as a conference paper at ICLR 2026
Step1: Maximuze Jobemuathmes (Definition
Step 2: Maximise Unley (Definition 3)
sample as set of players
Coogerative game
Reward attabution
Infueeiial players
Whole dataset
Unilty Judger
Grad North
Ioflucntial Samples
Score Set
aibibute
sokett
Budge
Figure 2: InfoUtil's 's pipeline for optimal dataset distillation involves two key steps: (i) Step I max- imizes informativeness via the Shapley Value (a game- -hee-theo attribution method), retaining the most informative patches to form compressed samples. (ii) Step 2 maximizes utility by scoring these candidates with a judge model-using Gradient Norm (proven as a utility upper bound) and retaining top samples. The final distilled dataset contains only the most informative, high-utility compressed samples. Image reconstruction and soft label generation phases are omitted here.
datasets by synthesizing and optimizing smaller, representative dataset. Models trained on distilled dataset are expected to match the performance of those trained on the original, larger dataset.
Currently, two primary lines of approaches are used to tackle DD: ie., matching -based meth- ods [Wang et al., 2018 Zhao & Bilen 2022, Zhao et al., [202] Cazenavette et al., (022) Zhou et al.) 2022). which aim to align the performance between the distilled dataset and the original dataset by matching gradients, features, distributions, or trajectories, and knowledge distillation-
meth- ods (Yin et al., 2023] Shao al. 2024), which decouple dataset distillation into two stages. In the first stage, the real data is compressed into a teacher model. In the second stage, the teacher model transfers knowledge the distilled images through deep inversion-like methods (Yin et al. 2020 Despite their success, these existing methods face two challenges:
Challenge 1: Efficiency-Perfoon Performance Trade-off. Most matching mutching-based -based methods require sig- nificant GPU memory and time, making them impractical for real-world applications.
For bi-level matching based methods, the key challenge lies in the trade-off off between performance and efficiency (Zhao et al. |021 Zhao & Bilen |021 Lee et al. 2022: Wang al. 2024a] Guo et al.
2023 Cui et al. 2023). For example, the state-of-the-an (SOTA) trajectory matching method (Guo al. 2023) requires more than 4 NVIDIA A A100 80GB GPUs to synthesize 50 image-per-class,
(IPC) dataset on Tiny-ImageNet. Such high resource demands severely limit scalability of these methods, making extremely challenging to apply to larger datasets like ImageNet-IK.
For knowledge distillation-basel methods, although they often perform better, the lack of a solid theoretical foundation impairs their interpretability (Yin al. 2023 Shao al. 20224 [Sun al. 2024) and prevents principled solution. This limitation leaves practitioners with limited insight into why certain samples are selected for compression or how the distillation process relates to underlying data. Therefore, despite demonstrating impressive empirical results, they fall short in providing the transparency required for high-stakes, or regulated applications.
Lack of Interpretability. Current methods are largely heuristic, lacking a prin- cipled framework to ensure the resulting distilled datasets are interpretable.
Challenge 2:
To rethink previous methods within a principled framework, we reconsider the knowledge distillation-based dataset distillation process by introducing Optimal Dataset Distillation (Defini- tion 4) The concept is built on Informativeness (Definition (T) and Utility (Definition 33) for desired distilled dataset. Intuitively, Informativeness captures essential information in each sample, while Utility reflects the importance of each sample for model training, whether included or excluded.
Built on the theoretical framework, we propose InfoUtil, Informativeness and Utility-enhanced Dataset Distillation (InfoUtil), method that balances both aspects. As illustrated in Figure 2 Step focuses on extracting key information from each sample, compressing it into representation that captures its most informative components. This is achieved by maximizing the game-theoretic informativeness of each sample, which we measure using the Shapley Value (Shapley et al. 1953), a principled attribution method first introduced in game theory. In Step 2, we maximize the utility of each sample, which is critical for model training. This is done by measuring the gradient norm of each sample and selecting those with the highest values, ensuring that only the most valuable samples retained. The main contributions of this work summarized as follows:
2

---

Published as a conference paper at ICLR 2026
1. We propose Optimal Dataset Distillation (Definition 4.. which builds on the concepts of patch- wise Informativeness and sample-wise Utility for distilled datasets. This approach addresses the lack of interpretability in existing methods by providing solid theoretical framework.
2. We introduce InfoUtil, a novel method balancing informativeness and utility in distilled dataset synthesis. It employs same-theoretic informativeness maximization via the Shapley Value and utility maximization retain the most informative and valuable samples using the Gradient Norm.
3. InfoUtil demonstrates outstanding performance across various models and datasets. For instance, our method yields 16% improvement in performance approach on the ImageNeet1000, and 6.1 % improvement on ImageNet-IK.
the previous state-of-the-an
2 PRELIMINARIES
Given dataset D = {(0;, 3i) H=z dataset distillation (DD) aims to synthesize a smaller dataset D = [x y,) ) with < The desired D> should enable model achieve comparable, lossless, performance to one trained on D, evaluated on a held-out test dataset Drest. Specifically, for model f parameterized by 0 trained with cross-entropy loss f, the condition is:
min
1e (z). e(Se (x),
(1)
where d denotes the fixed parameters trained on D. Crucially, 0 represents the parameters trained on the synthetic dataset D. Consequently, the term ([So (zr), y) depends on D through the optimiza- tion trajectory of 0.
This paper focuses knowledge distillation-base DD methods, which recently showed superior performance (Yin et al. 2023 Sun et al., 2024 Shao et al. 2024). Here, D's information is first learned by a teacher model Sep, which then synthesizes D. notable work, RDED (Sun et al. 2024), uses random cropping generate candidate patches, pruned via cross-entropy scoring. The final image contains multiple compressed images, each cropped and retained in prior steps. While RDED achieves high performance efficiently, lacks principled guarantees. As Figure shows, RDED's randomly selected patches often miss key ground truth category information.
3
METHOD
3.1 OPTIMAL DATASET DISTILLATION
To theoretically analyze the above problems, we first propose the following properties before for- mally defining the optimal dataset distillation mathematically.
Definition (Informativeness) Given an arbitrary sample D and the compressed size d' d, the informativeness of x E Rdd for the model Jo is defined as:
[xx; fo) == -1 x)) fo(z)]|-
where E 1 and |s| = d' is d-dimensional binary mask to be optimized, is the HadamarrDelementtwiss product, and s o x denotes the imput æ with a mask s.
The informativeness captures the key information for a given sample. Intuitively, maximizing the informativeness of sample x of given compression size d' be regarded learning the best informative mask vector that maximize the similarity of the performance between the original sample z and the masked sample
Next, we introduce Gradient Flow, key concept we use to define the Utility function.
Definition 2 (Gradient Flow) Let l, be the cross-entropy loss for the model ((4) iteration I. We define the gradient flow computed on mini-batch B as:
(foco (x). y: B) = ae, Hli(foro (2).
(3)
Øt

---

Published as a conference paper at ICLR 2026
The gradient flow (fore (æ), y; (B) represents the instantaneous rate of change of the loss for spe- cific example (a, y) during training, providing a continuous-time approximation of training dynam- ics. Unlike discrete SGD updates, which introduce noise, gradient flow offers a smooth, analytical framework for quantifying data importance. By leveraging this, we assess the impact of removing a single data point (Ti, yt) and define utility function below as dataset pruning metric.
Definition 3 (Utility) Let the gradient flow & be defined as in Definition 2 For data point (2t, yi) in dataset D, let B C 'D be the mini-batch at iteration t; define B-; =- B {( 9:) We measure the importance of (Zi, Ye) by how much its removal changes the gradient flow over all relevant pairs:
(3a ye Soco) =
max (x)0)))
( B) Re(Joco (1zz), ys; 4)
This utility definition captures the worst-case impact of removing a data point on gradient flow, ensuring it reflects data importance. By maximizing the change in (fave (rz). My; B) over all (Iy, My) E D, it identifies points that most influence training dynamics. This aligns with dataset pruning by preserving critical samples while discarding those with minimal effect.
Based on Definition | and Definition we propose the optimal dataset distillation Definition
Definition 4 (Optimal Dataset Distillation) Let fe be the classifier model with parameter 0 and D the original training dataset. Let Dust be the test dataset. Define DD' C D as a compressed subset, and D D' as the final distilled dataset. Let (x, y; f%) measure the utility of S'g on a test example (x.y) y) defined in Definition 3 Let I (2x fa) measure the informativeness of original samples defined in Definition | and s be the informative mask with compressed size d'. The goal is to find the optimal pruned dataset D that maximizes both informativeness and utility on Drest
arg max PCD' D
Σ (B.))
U(
s.t. D' =
24 0 8.:
arg max
[(x;; fo)
10 88 =d
i=1
This formulation establishes the dataset distillation problem. The key challenge is then to define rigorous utility function that effectively quantifies (i) the importance of each component within a sample for model prediction and also (ii) the importance of each sample for model training.
3.2 INFOUTIL
In this subsection, we introduce InfoUtil, built upon the optimal dataset distillation formulation in Definition The pipeline has two main steps: (i) game-theoretic informativeness maximization and (ii) principled utility maximization. Detailed algorithm pseudocode is in Appendix A
3.2.1
GAME-THFORETIC INFORMATIVENES MAXIMIZATION
As in Definition 1 InfoUtil is to maximize the informativeness of each sample æ to obtain a com- pressed sample
represented by mask 8. This task can be framed as a feature attribution problem (Zhou et al.] 2016 Selvaraju et al., 2020 Binder et al., 2016 Shapley et al., [953]. Qin
al. 2023), where the model attributes decisions input variables based on their importance.
Among attribution methods, the Shapley Value (Shapley et al. 1953) is regarded as a robust ap- proach grounded in game theory. Specifically, given an input æ with d input variables 1c(2),x.2 x(()) we can view a deep neural network as game with d players [d] {1, .1......d]. d). Each player i corresponds to an input variable z+()) Thus, the task of fairly assigning the reward in the game translates to fairly estimating attributions of input variables in the deep neural network f. Formally, the Shapley value á can be defined as:
dy (x)
Σ 8:8,=0
d dd11
(Tx (1-1)
(s (s
(4)
where e, Rd denotes the vector with one in the i-th position but zeros in the rest positions. Notably, the Shapley Value is renowned for satisfying four key axioms (Young 1985):
For detailed technical derivations, including the complete proof, please refer to Appendix D
4

---

Published as a conference paper at ICLR 2026
Axiom 1 (Linearity. Proof in Appendix D.I) If two games can be merged into game, then the Shapley Values in the two original games can also be merged. Formally, if fwerged = fi + f2. then @f Fmerged (xl6)) = Ub (x:()) + &3 (x()), Vi E [d].
Axiom 2 (Dummy. Proof in Appendix D.2) A dummyy player is a player that has no interactions
with other players in the game f. Formally, if : = 0, f(x o (s + +e)) f(r f(z ex).
Then, the dummy player's Shapley Value computed as f(xoe,)
If players contribute equally in every case, then their Shapley values the game will be equal. Formally, is s; = 8j f(x + c,)) f(z o (s + ej)), then oy (x(1) = ) (z(0).
Axiom 3 (Symmetry. Proof in Appendix D.3)
Axiom 4 (Efficiency. Proof in Appendix D.4 The total reward of the game f is equal to the sum of the Shapley values of all players. Formally, - ((0)= Qy (z(()))
The Shapley value is the unique attribution method that satisfies the four key axioms YYungg 1985 However, directly computing the Shapley value is computationally expensive in practice. For in- stance, calculating the Shapley value for an image with 4 x patches requires 2 inferences, as- suming each patch is a player. To address this issue, prior works (Charnes al.) 1988 Lundberg & Lee 2017) have proposed using kernel- -ased estimation of the Shapley value, as follows:
Φ = arg min Beng(e) ( (f(ros)- f(0) - -s- ) s.t. 1 f(x) - f(0), (5)
where (ss) = (d - 1)/ (G (3 11 < 1 1 < d denotes the Shapley Kernel. We follow KernelShap (Lundberg & Lee 2017) to achieve fast estimation of the Shapley value based on Eq. 5), making possible to be adept in practice.
3) 1))))
After obtaining the Shapley value Φ5 (x:(1()) of each sample x!1) we apply average pooling of the Shapley value map (x) (øb (zx(1), d, (x. .... (x((0)) obtain the most informative gion inside a image. This step would generate a d < d size compressed image (e.g., d' = d/4) with the maximized informativeness, resulting a compressed dataset with n compressed samples D'.
Diversity control. The Shapley value attribution typically identifies only the most informative patch. To introduce diversity in the patch selection process, we incorporate random noise where a is the standard deviation fixed. Specifically, the random noise is employed on the average pooled attribution heatmap, resulting in diverse informative patches considered in the next phase.
(0, o²),
3.2.2 PRINCIPLED UTILITY MAXIMIZATION
After obtaining the compressed dataset, the next step is selecting samples to maximize dataset utility. Computing utility (Definition 3 is challenging, as it requires training models with and without each sample to assess its utility. We show the utility function can be upper-bounde bounded by the gradient norm (Theorem T). simplifying computation. We now define the gradient norm.
Definition 5 (Gradient Norm) The gradient norm of a training example (x., v) for model f param- eterized by git) at time t is denoted as
| ge We Effcco (z), v)ll
Given the definition Gradient Norm, we then show that Utility can upper bounded by the gradient norm through detailed analysis here.
Theorem (Utility is bounded by Gradient Norm. Proof in Appendix E} Let the utility func- tion U be defined as in Desinition3) Then there exists constant 0 such that
U(x ys: fac) & ell ooe Lelfer (I:)), v:lll-
Proof of Theorem For detailed technical derivations, including the complete proof of Theorem and auxiliary lemmas, please refer to the supplementary materials. The full proof includes step-by- step expansions of gradient flow decompositions, rigorous bounds under SGD updates, and verifica- tion of assumptions underfying the utility-gradient norm relationship.
5

---

Published as a conference paper at ICLR 2026
Table 1: Performance comparison between InfoUtil and SOTA methods on seven datasets. We evaluate dataset distillation using ResNet-18, ResNet-101, and ConvNet, reporting top-1 accuracy (%). Datasets were distilled with ResNet-18 and ConvNet, then evaluated on matching architectures. Additionally, datasets distilled by ResNet-18 were also evaluated with ResNet-101.
Re
ResNet-101
ConvNet
Dataset
IPC
SRe2L
RDED
InfoUtil
SRc2L
RDED
InfoUri
MTT
IDM
TESLA
DATM
RDED
InfoUtil
1666=99
22,99b04
25.3+04
13.7
18.7=0.11
19.6
46.3-b08 45.6=207
48.5208
469205
23.5+003
28.5
CIFAR-10
10
29.3=05
37.1
538
24.3
33.7
1.0
65.3
58.6
66.4208
668802
50,2+003
54.1200
50
45.0=007
62.14:0.1
71004:08
34.94-0.1
51.6400,4
67.1
71,64:0.2
67.5
72.640077
76.1::03
68.440.1
69.8:-0.1
6.622:022
11,0
22.9::0.4
6.2:200
10.8:0011
16.52:0.5
24.3
20.1::03
24.8:0
27.9202
19.6:20.3
33.1
CIFAR-100
10
27.0=004
42,6++02
47.5+=077
30,7++0.3
41.1+112
41.9+0006
40.1++004
45.1
41.7+03
47.2+044
48.1++03
50,5+003
50
50.2:004
62.6.10.1
64.71:02
56.91:0.1
63.4.:0.3
66.01:0.2
47.711022
50.0:102
47,91003
55.0102
$7.0.10.1
57.8
1
19.1
35.8.1100
43.810.7
15.81006
25.1.12.
28.2.1...5
47.7.10.9
33.8.10.88
42.3
ImageNNtte
10
29.4:3.0
61.4+04
686
23.4+:08
54.0
19.8
63.0+11.
63.2+0077
66.6004
50
40.9..03
80.4.10.4
86.21.006
36.51/077
75.0.1122
82.4....33
83.8.10.2
14.4
1
13.3:00
20.8-11.2
25.0--00
13.4-3-0.1
19.6-11.8
20.2:10.4
28.61:0.88
18.5
22.8 2:.4
ImageWoof
10
20.2200
38.522..
2:.4
12:72:09
11
1.2
35.8218
40.642..0
43.8
50
23.3+00
68.5++0.77
69.6+-03
21.2++0.2
59.1++0.7
67.2++00.
61.5++0.3
62.6+004
2.61::011
9.7.10.4
1..0
1.9:0011
3.8:0.11
11.9
8.8.0033
10.1:102
17.1103
12.0.10.1
19.61:055
Tiny Ting-emageNet
10
16.12:02
41.9.10.2
45.61:03
14.61:1
22.91333
34.4
23.2.10.2
21.9103
31.1103
39.6.10.1
40.21:03
50
41.1
58.2+b0.1
58.5=0.
42.5+:02
41.2+t0.4
54.7+00.
28.0g003
27.7
39.7#003
47.6
48.0=005
1
3.011:33
8.1.10.3
15.71:02
2.1
6.1.108
11.41.022
11.2.103
7.1.1..22
15.01:08
ImageNet-100 100
10
9.5::0
36.020.3
50.52:04
6.420.1
33.9tt..1
49.9
17.1=06
29.6-b0.1
42.2g007
50
27.0=04
61.6440.1
68.3+-044
25.75-03
66.0:10.66
69.7++0.4
26.3+404
50.24-0.2
60.8-009
1
0.1+au11
6.64102
12.8+007
0.6++..1
5.9+04
6,8++0.7
7,7+022
6.4+tu1
6.6+00.3
ImageNet- IK
10
21:3226
42.010.1
44.21:04
30.92:0
1 48.32110
51.41:0.3
17
20.410.1
21.52:03
50
46.8=02
56.5+b0.1
58,0=0.
60.8+:0.5
61.2+t014
63.8++0.
27,9
38.4+10.2
40.2=0.4
Given Theorem 1 we can efficiently calculate the utility of each sample using the upper bound of the gradient norm. Then, we can directly select the most influential samples with the highest gra- dient norms to maximize utility. Specifically, we employ gradient norm scoring for all compressed samples in D' with size n, and selected samples with top norm scores, resulting D
with size m << n.
Image Reconstruction. Following prior works (Yin et al. 2023 Sun et al. 2024 Shao et al. 2024
reconstruct normal-sized images by combining compressed samples. Low-resolution datasets use single image per category, while high-resolution datasets merge four 1/ 1/4-resolution images from the same category into one full-size image. For soft label generation, patch-specific logits are assigned by resizing the compressed samples. Inspired by (Qin et al., 2024] Wang et al., 2024b), intermediate checkpoints of pretrained model are used balance discriminativity and diversity, improving performance. Further details are in Section 6
4 EXPERIMENTS
4.1 EXPERIMENTAL SETTINGS
Datasets and network architectures. We evaluated our approach using widely recognized datasets. For lower-resolution datasets, we employed CIFAR-10 and CIFAR-100 (Krizhevsky al. 2009)
(32 x 32) and Tiny-ImageNet (Deng et al. 2009) (64 x 64). For highlt- experiments, we used ImageNet- (Deng al. 2009) (224 224) along with three commonly used ImageNet subsets: ImageNette, Image
and Image 100 (all at 224 x 224). In line with previous works on dataset distillation, we adept the following backbone architectures: ConvNet (Liu et al.] [0022). ResNet- 18, 50, 101 (He et al. 2016), MobileNet-V2 (Howard et al. 2019), VGG-11 (Simonyan & Zisserman 2014). and Swin- V2-Tiny (Liu et al. (021). Specifically, dataset distillation is performed using a 3-layer ConvNet for CIFAR-10/TO0 4-yyerr ConyNet for Tiny-Imageh and ImageNet- 1K, a 5-layer ConvNet for ImageWoof and ImageNette, and a 6-layer ConvNet for ImageNet- 100.
-000.N
Baseline methods. Following previous studies, assessed the quality of the condensed datasets by training neural networks from scratch using them. We reported the resulting test accuracies on the actual validation sets. Baseline include vraecttoy--atccie approaches such MTT (Cazenavettel

---

Published as a conference paper at ICLR 2026
TESLA (TB)
Method classification
Model
TB (Trainng Based)
1ooNes-18
MoNiieNa
(Taining-Free)
SRe2L (TB)
2,29GB
11,79GB
>142G
Method
RDED (TF)
1,75GB
Time Cost (s)
>241GB
1,45BB
Peak
Model
ResNet-18
MobileNet
Info (TF)
1.76GB
Memory
TESLA (TB)
47180.3
56882.9
1,46GB
SRa2L (TB)
3042.6
3622.3
RDED (TT)
617.9
1004.2
Util (TF)
L.7GB
2GB 10GB
100CB
Info (TF)
852.3
1167.4
1,47GB --
Util (TF)
725.9
1112.3
InfoUtil (TF)
1447BB e
L78BB
InfoUtil (ours)
InfoUhil (TTF)
960.3
1275.5
0
500
1000
1500
2000
2500
3000
3500
4000
47000
47500
56500
57000
Time Cost (s)
Figure 3: Performance comparison ResNet-18 and MobileNet. (a) Time cost in seconds (lower is better): "TB" denotes training-based methods (TESLA and SRe2I. fall into this category); "TF" denotes training-free methods (others belong to this type). (b) Peak memory in GB (lower is better): InfoUtil performs competitively with far lower costs than training-basek methods. "Info" denotes Informativeness only, while "Util" denotes Utility only.
ct al. 2022). TESLA (Cui et al., 2023), and DATM (Guo et al., 2023), and distributionn-machinn methods like IDM (Zhao et al. (023). For primary comparison, also include SOTA knowl- edge distillation-base methods, SRe2L (Yin et al., 2023) and RDED (Sun et al. 2024).
Implementation details of InfoUtil. Our setup follows RDED, using pretrained networks for dataset synthesis. For small IPC, we adopt the approach in (Qin et al., 2024), extracting training- stage soft labels to capture rich semantics. For larger IPC, fully converged networks from RDED are used. Details are in Appendix [A] For low-resolution datasets, one synthetic image per class used, while high-resolution datasets use four per class. The 300-image subset matches RDED's configuration. As in Table [] AutoAug (Cubuk et al.) 2018) is applied to enhance synthetic dataset performance. All experiments ran on single NVIDIA A100 GPU.
4.2
MAIN RESULTS
We verified InfoUtil's 's effectiveness benchmark datasets across mmage-per-clas (IPC) settings.
Hegher-resolution datasets. We benchmarked InfoUtil against state-of-the-ar methods on higher- resolution datasets like ImageNet- 1K and its subsets. As Table Tablle[shoe shows, InfoUtil achieves superior or comparable performance across IPC settings. Notably, on ImageNet- 100 (ResNet-101, IPC=10), it outperforms RDED (Sun al. 2024 by 16% in accuracy; on ImageWoof (ResNet-18, IPC=10), it gains 12.9% over RDED. Moreover, on ImageNet- K (ResNet-18, IPC=1), InfoUtil surpasses RDED by 6.1%, highlighting its effectiveness in small IPC scenarios.
CIFAR-10/100 and Tmay-ImageNee. We evaluated InfoUtil on rower-resolution datasets with addi- tional experiments on CIFAR-10/101 and Tiny- ImageNet. Our method continues to show superior performance across most scenarios, highlighting robustness and generalizability of InfoUtil. Specif- ically, as Table [ on Tiny- ImageNe,, using ResNet-101 at IPC = 50 yields a 13.5% improvement; on CIFAR-10, 10, ResNet- 18 at IPC = 10 obtains 16.7% % improvement.
We evaluated InfoUtil's cross-architecture generalization across ResNet- 18/50 (He et al., 2016). VGG-11 (Simonyan & Zisserman, (014)) MobileNet- V2 (Howard al. 2019), and Swin- Ve-Tiny (Liu et al. 2001). Table 2 shows InfoUtil outperforms SOTA (SRe2L, RDED) by 10% in the VGG- (teacher) vs. Swin- V2-Tiny (student) setting, con- firming versatility. Further validation with baselines SCDD, G- G-VBSM (structural regularization) and D3S (data efficiency) ImageNet-1K across BesNee-188/10 (Table shows InfoUtil consistently outperforms SRe2L/RDED and these baselines across all IPC settings.
Cross-architecture generalization.
Efficiency Analysis. We carefully measured InfoUtil's runtime and GPU usage single NVIDIA A100. (i) It is highly efficient: time is 50x lower and memory 100x smaller than TESLA across all distillation stages (Figure 3). (ii) For large-scale datasets like ImageNet-21K, distillation com-
7

---

Published as a conference paper at ICLR 2026
Table 2: Cross-architecture performance (%) Table 3: Cross-architecture comparison of InfoUtil on ImageNet-IK (IPC=10). Using ResNet- with additional baselines on ImageNet- IK. Results 18/50, VGG-I1, MobileNet-V2, and Swin-V2- are shown across ResNet-18 and ResNet-101 archi- Tiny as teachers; ResNet- 18, MobileNet-Ve V2, tectures under varied IPC settings.
and Swin- V2-Tiny as students.
Model
ResNet-18
NesNet-101
Squeezed Evaluntion R8 Vo Swin-V2-Tiny
IPC
10
50
10
50
21.740.
15.4+02
R8
RDED 42.3+0.
40.4=0.1
1
SRc21.
0.1
21,3406
06.62..0666.00
30.9201
60.8
198+04
SCDD
32.1::02
53.1
39.6:044 61,01033
G GBBS
31,410.
$1.810.4
38,2 61.0
ResNet-50
RDED
13.9+0.5
26.0+0.3
17.3=02
D35
5.3 37.2400
50.320.3
3.2 42.3 60.6
InfoUtii
15.6+04
RDED
6.6:022 42.0.:0.1 56.5....1
48.3 61.2.044 5.9.0.44
InfoUtil
12.7::077 44.0:00..50000...
01.4 63.7.1066
19.7+0.1
1022426
Mob
RDED
34,4+002
13.8+0.8
118+00
Table 4: Comparison of downstream tasks for dis- tilled samples in 5-step continual learning. Higher
39,240.3
38.5+005
20.6=02
SRc2L
16.5+0.1
10.6+00.1
VGG-11
RDED
22.7+0.1
21.6+022
7,8+011
values indicate better performance.
Method Stage
Stage 2
Stage 3
Stage
Stage
7.4001
Swin-V2-TB
RDED
17,8+0.1
18.1+002
12.1+022
RDED 0.5153
0.2918
0.201
0.1967
0.2191
InfoUtil
18.4:0..4
19.7
InfoUti
0.6560
0.5659
0.4927
0.4617
0.4739
Table 5: Comparison with baseline methods under large IPC settings. We used ResNet- 18 for dataset synthesis on Tiny-ImageNet and ImageNet-IK, K, and evaluated on ResNet- 18 and ResNet-50 models. Note that TESLA (Cui et al. 2023) used the downsampled ImageNet-1KK dataset.
Dataset
IPC
TESLA (R18)
SRe21. (RI8)
RDED (R18)
InfoUtill (R18)
SRe21. (RS0)
InfoUtil (R50)
50
41.1+0.4
58.2+00.1
58.5=20.3
42.2+055
48.3=044
Tiny-ImageNe
100
49.7.:033
59.91:0.4
60.62005
51,2204
53.7-014
200
51.2+:0.
61.5#00.
62.0=20.3
58.0=033
10
178-13
21.31:06
42.0+:0.1
43.5-104
28.420.1
ImageNet-IK
50
27.9=12
46,8+
$6.5+t0.1
$7.6=20.3
55.6#03
63.1=04
100
52.84003
58.2:10.6
58.82104
61.0204
65.5=0.5
200
57.0+004
62.5#0088
63.4
64.6203
68.0=04
pletes in just 5.83 hours. This combination of remarkable efficiency and strong performance makes InfoUtil a practical, scalable solution for modern dataset distillation.
Performance large IPC settings. We tested Tiny- ImageNet and ImageNet- under large IPC scenarios, comparing with bi-level Tesla (Cui et al. 2023) and uni-level SRe2L (Yin et al. 2023). RDED (Sun et al., 2024). Table 5 shows
method significantly outperforms existing SOTA in large IPC cases, demonstrating strong scalability and superior performance. For IPC=200 on ImageNet-1K,, 1K, we used full images (not 2x2 cropped patches as prior work) mitigate imbalance (following (Sun et al. 2024)); image count before scoring was 600 instead of 300.
Downstream tasks of distilled samples. We explored the effectiveness of distilled samples in downstream tasks via experiments on ImageNette (50 IPC) with 5-step continual learning, where new classes are incrementally introduced each stage. To ensure the robustness of results, experi- ments were repeated 5 times with varied class orders. As shown in Table 4 our method (InfoUtil) consistently surpasses the SOTA method RDED across all stages.
Visualization. InfoUtil shows significant improvements in visual quality over existing methods. First, vs. optimization-baste based methods like SRe2L (Yin et al.] (023). it produces more realistic representations by preserving intricate details and maintaining natural color fidelity. Second, vs. optimization-fret methods like RDED (Sun et al., 2024), InfoUtil more interpretable and princi- pled, effectively capturing key informative semantic content while minimizing focus irrelevant regions. Due to space constraints, visualization images are provided in Appendix
4.3 ABLATION STUDIES
To analyze the individual contributions of InfoUtil's components, conducted comprehensive ab- lation studies comparing three configurations: (1) the baseline RDED method (Rand. Crop Loss Scoring), (2) Utility Maximization alone (GradN Scoring), and (3) the complete InfoUtil (GradN Scoring + Attri Cropping). Results across multiple datasets (Image ImageNette, ImageNet- 1K) with varying IPC values are presented in Table
8

---

Published as a conference paper at ICLR 2026
ImageWoom Woof
ImageNette
ImageNet-1K
Table 6: Ablation study of
InfoUtil components" impact on image classification. Top- accuracy (%) on ResNet-18 across datasets are reported.
GradNorm Scoring
Altribution Cropping
IPC=1
IPC=50
IPC=50
IPC=10
38.5
68.5
80.4
42,0
68.8
45.2
69.6
86.2
44.2
Effect of Utility Maximization. Replacing Loss Scoring with GradN Scoring while maintaining random cropping brings significant performance improvements. shown in Table 6 Utility Maxi- mization alone achieves a 4.6% performance boost on ImageNette (IPC=50, from 80.4% to 85.0%) and a 1.5% improvement on ImageNet-Ike IK (IPC=10, from 42.0% to 43.5%). These results demon- strate that gradient norm-based scoring plays crucial role in selecting more informative samples.
Figure 4: Analysis of teacher networks for soft label gen- eration. ConvNet performance on ImageWoof using labels from five training stages (IPC=1/10), "Full" denotes pre- trained teacher. (a) IPC=1: Early high-entropy labels beat full model, aiding low-data scenarios. (b) IPC=10: Full model's 's low-entropy labels excel in data-rich conditions.
Ascunacy (%)
00 =1
(9) DPC
Effect of Combined Components. The integration of both Utility Maximization and Informa- tiveness Maximization through Attri. Cropping yields the best performance. InfoUtil achieves addi- tional gains of 1.2% on ImageNette (reaching 86.2% and 0.7% on ImageNet- IK (reaching 44.2%) compared to using Utility Maximization alone. This synergistic combination demonstrates that attribute- -uiidedgudddddd cropping effectively captures the most discriminative regions while gradient- based scoring ensures the selection of pedagogically valuable samples, together producing high-quality synthetic data that consistently outperforms the baseline across all experimental settings.
Effect of Noise Injection. We investigate the role of noise injection in serriibutoonguuddee patch selection, which is essential for maintaining synthetic data diversity. Without noise, the selection process becomes deterministic and greedy, leading redundant synthesis that hinders model gen- eralization. As shown in Table 7 removing noise ("w/o Noise") results in significant performance degradation across all settings. Notably, the performance gap exceeds 15% at higher IPC (e.g.. 86.2% vs. 70.6% on ImageNette at IPC=50), confirming that noise-induced diversity is critical for effective feature space coverage. Even at IPC=1, the consistent gains suggest that noise helps identify robust central features rather than brittle local maxima in the attribution map.
Table 7: Ablation Study on Noise Injection: Top-1 Accuracy across various settings.
Dataset IPC
ImageNette Standard
Woof Standard
ImageNer-100 Standard
ImageNet-IK w/o Noise
w/o Noise
w/o Noise
w/o Noise
Standard
43.8
12.6
10
68.6
59.8
51.4
40.0
50.5
43.8
44.2
38.5
69.6
56.3
48.3
Attribution Method: Shapley vs. Grad-CAM. We justify the selection of Shapley Value over Grad-CAM by comparing their theoretical rigor and empirical performance. Unlike heuristic gradient-based methods like Grad-CAM, which often suffer from gradient saturation, the Shapley Value is the unique attribution method satisfying fundamental axioms such as Efficiency and Sym- metry. This theoretical robustness ensures fair distribution of "IIformativeness." accurately cap- turing the marginal contribution of each patch. Empirical results in Table confirm this advantage, with Shapley-based selection consistently outperforming Grad-CAM on ImageNet-IK. Notably, IPC=10, Shapley achieves 43,88% surpassing Grad-CAM by 13.49%, demonstrating its superior ability identify semantically robust patches for effective distillation.
RELATED WORK
Dataset Distillation. Dataset Distillation aims to reduce a large dataset into smaller one. Current
methods can be categorized into two main approaches: i.e., matching-basee methods (Zhao et al,
2021 Lee al. 2022 Zhao & Bilen 202T Wang et al. 2024a Cazenavette et al. 2022 Cui al.

---

Published as a conference paper at ICLR 2026
Table 8: Empirical Comparison of Attribution Methods: Shapley Value
Grad-CAM
ImageNet- 1K (ResNet-18 Student Model).
Model
Dataset
IPC
Grad-CAM
Shapley (Ours)
4.418
7.154
ResNet-18
ImageNet-IK
10
30.394
43.880
50
52.610
56.920
2023
Guo et al. 2023; Zhao & Bilen 2022; Kim et al. 2022 Du et al. 2023 Zhou et al. 2022
Wang et 2025c| Min et al., 2026), and wwwwleegediistllatiionn
methods (in et al. 2023
Shao et al. 2024 Sun et al., 2024 Wang et al., 2025a). Matching-basee methods are typically formulated as bi-level optimization problems but struggle with the trade-off between efficiency and methods decouple the problem into a two-step process but often lack theoretical interpretability. Therefore, deeper in- methods in principled manner to
the quality of the distilled dataset. In contrast, snowleege-distillation-b-
vestigation is needed to formalize ensure their reliability in practical scenarios with theoretical support, which we address in this paper.
Attribution Methods in Explainable AI. Attribution methods essential for post-hoc expla- nations of black-box models, revealing each input variable's 's contribution to the final prediction. Among them, the Shapley Value is considered principled tool due to its key axioms: i.e., linearity, dummy, symmetry, and efficiency (Shapley et al., 1953 Young, 1985). To reduce the computational burden, KernelShap (Lundberg & Lee| 2017) was introduced efficiently approximate the Shapley Value using Linear LIME (Ribeiro et al.J (016) or neural network (Wang et al., (025b)). However, since none of the previous works have explored the application of attribution methods in dataset distillation, there is an opportunity to develop attribution- approaches for extracting key infor- mation for dataset distillation.
DISCUSSION
Soft labels encode richer probabilistic supervision in dataset distillation. Prior works (Guo et al,
2023 Yin et al. 2023 Wang et al., 2024b) Qin al., 2024 [Sun al. 2024) show they capture
inter-class relationships. (Qin et al. 2024 finds early high-entropy labels help low-data regimes, while late low-entropy labels suit data-rich settings. (Wang al. 2024b) notes effective labels balance diversity and discriminability. However, these focus on matching- based distillation, leaving knowledge-distinee distillation-baal based DD with soft labels unexplored.
To investigate this further, we explored the effectiveness of teacher model for soft label generation using ConvNet on Image Woof. For small IPC settings, we extracted soft labels from models at an intermediate training stage (10-th epoch), leveraging the high-entropy. diverse information charac- teristic of early epochs. In contrast, for large IPC settings, we used fully pretrained networks from RDED, leveraging the low-entropy, precise labels typical of later training phases.
Our findings, as shown in Figure 4 clearly highlight the effectiveness this strategy. In small IPC scenarios (e.g., IPC 1). synthetic images with soft labels generated with models at 10-th epoch outperformed those from pretrained networks, emphasizing the importance of rich label information when limited data are provided. Conversely, in larger IPC scenarios (e.g., IPC 10 or IPC 50), labels from fully pretrained networks yielded superior results.
CONCLUSION
In this paper, we present a principled approach to dataset distillation, grounded in rigorous theo- retical framework for modeling optimal distillation. We introduce Informativeness and Utility, cap- turing, the critical information within sample and essential samples for effective training. Building on these, we propose InfoUtil, a framework that synergistically combines game- game-theoretic informa- tiveness maximization with principled utility maximization. Specifically, InfoUtil leverages Shapley value attribution to extract informative features and employs gradient norm-based optimization to select samples optimized for utility. InfoUtil demonstrates superior performance in dataset distilla- tion and cross- architectrre generalization. Future work includes extending InfoUtil to more complex and diverse datasets, focusing on scalability and robustness in real-world applications.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Alexander Binder, Grégoire Montavon, Sebastian Lapuschkin, Klaus-Robert Müller, and Wojciech Samek. Layer-wise relevance propagation for neural networks with local renormalization layers. In Artificial Neural Networks and Machine Learning- ICANN 2016: 25th International Conference on Artificial Neural Net- works, Barcelona, Spain, September 6-9, 2016, Proceedings, Part II 25, pp. 63-71. Springer, 2016.
George Cazenavette, Tongzhou Wang, Antonio Torralaa, Alexei A Efros, and Jun-Yan Zhu. Dataset distillation by matching training trajectories. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition. pp. 4750-47559 4759, 2022.
Charnes, B Golany, Keane, and J Rousseau. Extremal principle solutions of games characteristic function form: core, chebychev and shapley value generalizations. Econometrics of planning and efficiency, 123 133, 1988.
Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quee V T.c. Autoaugment: Learning augmentation policies from data. arXiv preprint arXiv:1805.0900 2018.
Justin Cui, Ruochen Wang, Si Si, and Cho-Jui Hsieh. Scaling up dataset distillation to imagenet- Ik with constant memory. In International Conference on Machine Learning, pp. 6565-6590. PMLR, 2023.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248-255, 2009. doi: 10..109/CVR..2009222000
Jiawei Du, Yidi Jiang. Vincent YF Tan, Jocy Tianyi Zhou, and Haizhou Li. Minimizing the accumulated trajectory eπrо to improve dataset distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3749 3758, 2023.
Ziyao Guo, Kai Wang, George Cazenavette, Hui Li, Kaipeng Zhang, and Yang You. Towards lossless dataset distillation via difficulty-aligned trajectory matching. arXiv preprint arXiv:2310.05773, 2023.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEFE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Andrew Howard, Mark Sandler, Grace Chu, Liang- Ching Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang. Vijay Vasudevan, et al. Searching for mobilenetv3. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 1324,, 2019.
Hang-Hyun Kim, Jinuk Kim, Seong Joon Oh, Sangdoo Yun, Hwanjun Song. Joonhyun Jeong, Jung Woo Ha. and Hyun Oh Song. Dataset condensation via efficient synthetie-data parameterization. In International Conference on Machine Learning, pp. 11 1120-11118.. PMIRR, 2022.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Sachyung Lee, Sanghyuk Chun, Sangwon Jung, Sangdoo Yun, and Sungroh Yoon. Dataset condensation with contrastive signals. In International Conference on Machine Learning, pp. 12352-12364. PMLR, 2022.
Ze Liu, Yutong Lin, Yue Cao. Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF inter- national conference on computer vision, pp. 10012-10022 2021.
Zhuang Liu, Hanzi Mao, Chao- Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In Proceedings of the IEEF/CVF conference computer vision and pattern recognition, pp. 11976-1 1986, 2022.
Scott M Lundberg and Su-In Lee. A unified approach to interpreting model predictions. Advances in neural information processing systems, 30, 2017.
Yue Min, Shaobo Wang, Jiaze Li. Tianle Niu, Junxin Fan. Yongliang Miao, Lijin Yang, and Linfeng Zhang Imagebindde: Compressing multimodal data with imagebind- based condensation. Annual AAAI Conference on Artificial Intelligence, 2026.
Dong Qin, George Amariucai, Daji Qiao, Yong Guan, and Shen Fu. A comprehensive and reliable feature attribution method: Double-sided remove and reconstruct (dorar), 2023.
Tian Qin, Zhiwei Deng, and David Alvarez-Melis. A label is worth thousand images in dataset distillation. arXiv preprint arXiv:2406. 10485, 2024.
11

---

Published as a conference paper at ICLR 2026
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. why should i trust you?" explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pp. 1135-1 1144, 2016.
Noveen Sachdeva and Julian McAuley. Data distillation: A survey. arXiv preprint (.Xiv:2301.04277 2023.
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: visual explanations from deep networks via gradient- based localization. International journal of computer vision, 128:336-333 359, 2020.
Shitong Shao, Zeyuan Yin, Muxin Zhou, Xindong Zhang. and Zhiqiang Shen. Generalized large-scale data condensation via various backbone and statistical matching. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 16709-16718 2024.
Lloyd S Shapley et al. A value for n-person games. 1953.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint 14Xiv 1556, 2014.
Peng Sun, Bei Shi, Daiwei Yu, and Tao Lin. On the diversity and realism of distilled dataset: An efficient dataset distillation paradigm. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3399, 2024.
Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, and Geoffrey J Gordon. An empirical study of example forgetting during deep neural network learning. arXiv preprint arXiv:1812.05159, 2018.
Shaobo Wang. Yantai Yang, Qilong Wang, Kaixin Li, Linfeng Zhang, and Junchi Yan. Not all samples should be utilized equally: Towards understanding and improving dataset distillation. arXiv preprint arXiv:2408. 12483, 2024a.
Shaobo Wang. Yantai Yang, Shuaiyu Zhang, Chenghao Sun. Weiya Li, Xuming Hu, and Linfeng Zhang, Drupi: Dataset reduction using privileged information. arXiv preprint arXiv:2410011611 2024b.
Shaobo Wang, Tianle Niu, Runkang Yang, Deshan Liu, Xu He, Zichen Wen, Conghui He, Xuming Hu, and Linfeng Zhang. Videocompressa: Data-efficient video understanding via joint temporal compression and spatial reconstruction. arXiv preprint arXiv:2511.1882 2025a.
Shaobo Wang, Hongxuan Tang, Mingyang Wang. Hongrui Zhang, Xuyang Liu, Weiya Li, Xuming Hu, and Linfeng Zhang. Gnothi seauton: Empowering faithful self-interrretability in black-box transformers. In The Thirteenth International Conference on Learning Representations, 2025b.
Shaobo Wang, Yieun Yang. Zhiyuan Liu, Chenghao Sun, Xuming Hu, Conghui He, and Linfeng Zhang. Dataset distillation with neural characteristic function: A minmax perspective. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 35570-25580. 2025c.
Tongzhou Wang, Jun-Ya Yan Zhu, Antonio Torralba, and Alexei A Efros. Dataset distillation. arXiv preprint 1rX.v:1811.10959, 2018.
Max Welling, Herding dynamical weights to learn. In Proceedings of the 26th annual international conference on machine learning, pp. 1121-1128, 2009.
Hongxu Yin. Pavlo Molchanov. Jose M Alvarez, Zhizhong Li, Arun Mallya, Derek Hoiem, Niraj K Jha, and Jan Kautz. Dreaming distill: Data-free knowledge transfer via deepinversion. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8715-8724, 2020.
Zeyuan Yin, Eric Xing, and Zhiqiang Shen. Squeeze, recover and relabel: Dataset condensation imagenet scale from a new perspective. Advances in Neural Information Processing Systems, 36:73582 73603, 2023.
II Peyton Young. Monotonic solutions of cooperative games. International Journal of Game Theory, 14(2): 65-72, 1985.
Bo Zhao and Hakan Bilen. Dataset condensation with differentiable siamese augmentation. In International Conference on Machine Learning, pp. 12674-12683 PMLR, 2021.
Bo Zhao and Hakan Bilen. Dataset condensation with distribution matching, 2022.
Bo Zhao, Konda Reddy Mopuri, and Hakan Bilen. Dataset condensation with gradient matching. In Interna- tional Conference Learning Representations, 2021.
12

---

Published as a conference paper at ICLR 2026
Ganlong Zhao, Guanbin Li, Yipeng Qin, and Yizhou Yu. Improved distribution matching for dataset densation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7856-7865, 2023.
Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, Antonio Torralb.. Learning deep features for discriminative localization. Proceedings of the IEEE conference on computer vision and pottern recognition, pp. 2921-2929, 2016.
Yongchao Zhou, Ehsan Nezhadarya, and Jimmy Ba. Dataset distillation using neural feature regression. Ad- vances in Neural Information Processing Systems, 35:9813 9827, 2022.
13

---

Published as a conference paper at ICLR 2026
Algorithm InfoUtil Pipeline
Input: original dataset D, pre-trained teacher model fop teacher model at early 1-th epoch fa. compressed size d', noise variance o, distilled dataset size m, number of patches for each class in D do
De = { y1) E D 1/
=
// Stage 1: Informativeness Maximization
for (v., Ui) E 'De do
Compute (x(x;) using fop
Apply average pooling to (2;))) to obtain pooled heatmap
Add noise E ~ (0, o²) to the pooled heatmap
Extract of size d' from z; based on the highest heatmap value
end for
D' {(E: 34))
// Stage 2: Utility Maximization
for (E,, 31) D' do
Compute 9i VV (5:),
end for
Select top-k x IPC samples {Ei...... EAPPCR by for - to IPC do
Combine 11,(-1))kk+1 to Eijjkk into zz
For each Eit in ry. set Ujk = fo, (Est)
1/2 Vji. Yjkk
D = DU [イ)
end for end for Output: Distilled dataset D
A DETAILED IMPLEMENTATION
In this section, detail the implementation specifics of InfoUtil, including the computation of informativeness, tuning settings teacher models, and provide the corresponding pseudocode in the Algorithm
A.1 COMPUTATION OF INFORMATIVENESS
In our implementation of InfoUtil, we leveraged the PyTorch framework together with the Captum package to compute Shapley values. Captum provides a robust and flexible interface for model interpretability, allowing us to quantitatively assess the contributions of individual features to the model's 's predictions. By utilizing Captum's 's KernalShap"l method, we could accurately determine the importance of each feature within sample, which turn guides the data refinement process during dataset distillation. Moreover, in the first four cropping, we injected Gaussian noise drawn from the normal distribution (00 2)), where a is defined as the product of the overall standard deviation of the Shapley values after average pooling and a hyperparameter that controls the noise intensity. In our experiments, we set the kernal size to 2 x with stride 1, and the hyperparameter 2. The final (5th) cropping maintained the original Shapley values. This approach effectively reduced the probability repeatedly cropping the same location.
Besides. in most scenarios, divided each original image into 4 x 4 grid of patches, computed the Shapley value for each individual patch and subsequently identified the center of the patch with the highest Shapley value the optimal cropping center.
A.2 PRETRAINED TEACHER MODEL.
When generating soft labels, utilized teacher models from the early stages of training. Specifi- cally, for 1IFAR-10 and C0F,R-100, the teacher models were pretrained for 10 epochs using learn-
https://captumm.at / /api htappp/shppllyyyyppe laapappeeyyllll_aa_aeeeeeeeee Le__smpl____ppeeeppppeep____p11m heampling.hta html
14

---

Published as a conference paper at ICLR 2026
Table Comparison of Top-1 Accuracy (%) InfoUtil vs. Coreset Selection Methods (ConvNet).
Model
Dataset
IPC
Random
Hending
Forgetting
InfoUtil (Ours)
1
14.4
2.0
21.5 + 1.2
13.5 + 1.2
28.5 + 1.4
CIFAR-10
10
26.0
=0.0=10 1.2
31.6
23.3
+ 1.0
54.1 = 0.5
50
43.4
1 1.0
40.4 0.6
23.3 1111111 1.1
69.8 1 0.1
1
4.2
8.4
4.5.2.2
33.1 = 0.3
ConvNet
CIFAR-100
10
14.6 0.5
17.3
1..1
50.5 1: 0.3
50
30.0=0.4
33.7 0.5
30.5 + 0.3
57.8 + 0.2
1
1.4
1.1
1.6
19.6 + 0.5
Tiny ImageNet
10
5.0
5.0
5.1
40.2 = 0.3
50
15.0
15.0 0.4
1..
48.0 + 0.5
Table 10: Comparison Top-1 Accuracy (%) of InfoUtil vs. Coreset Selection (ResNet-18)
Model
Dataset
IPC
Random
Herding
K-Means
InfoUtil (Ours)
Tiny ImageNet
10
7.5
- 0.1
0.3 5.8 6.8
8.9 1.920.2
45.6 0.3
ResNet-18
ImageNet-I IK
10
4.4 + 0.1
5.5
44.2 0.4
ing rate of 0.001 on IPC = 1 and 10. Meanwhile, for other datasets (Tiny-ImageNet, ImageNette, ImageWoof, ImageNet-1000 and ImageNet- we trained the teacher models for 10 epochs using a learning rate of 0.01 on IPC and 10. For IPC = 50 scenarios, we employed fully converged teacher models across all datasets ensure that soft labels generated could reflect the comprehen- sive and stable representations learned from the entire training dataset. Compared to teacher models from early training stages, fully converged models provide richer, more accurate semantic infor- mation, which significantly benefit the distillation process, especially when synthesizing a larger number representative images.
B CORESET SELECTION COMPARISON AND INFORMATION DENSITY
We investigate the fundamental distinction between data synthesis (InfoUtil) and traditional coreset selection methods, which aim to construct a compact dataset by selecting unaltered real samples. While both approaches pursue dataset compression, InfoUtil's s ability to synthesize highly informa- tive, compressed knowledge yields significant performance gap, especially under extreme data scarcity (IPC 10).
B. FUNDAMENTAL DISTINCTION AND EMPIRICAL ADVANTAGE
Traditional coreset selection methods (such as Random, Herding Welling 2009) and Forget- ting (Toneva et al. 2018)) are constrained by the quality and content of the original training sam- ples. InfoUtil overcomes this limitation by dynamically synthesizing samples that are optimized for knowledge transfer, extracting informative patches, and utilizing soft labels to condense teacher knowledge. As shown in Tab. Pnd Tab. 10 to empirically demonstrate this advantage, we compare InfoUtil against coreset selection baselines across CIFAR, Tiny- ImageNet, and ImageNet-1K.
B.2 PERFORMANCE ON LARGE-SCALE DATASETS
As shown in Tab. TO we further validate the results using a deeper architecture (ResNet-18) on challenging large-scale datasets, comparing against K-Means coreset selection.
The empirical results show massive performance gap. For example, on ImageNet-1K (IPC: = 10), InfoUtil achieves 44.2%, which is nearly 7.6 times higher than the best coreset method (Herding, 5.8%). This clearly illustrates that simply selecting real images is insufficient for training deep net- works from scratch on such limited budgets. Coreset methods inherently suffer from background noise and reliance hard labels. In contrast, InfoUtil's synthesis mechanism which incorporates at- tribution cropping (Infornativeness)) and soft labels, effectively condenses the necessary knowledge, making it far more efficient and powerful than standard subset selection.
15

---

Published as a conference paper at ICLR 2026
Table 11: Accuracy (%) under the Controlled "Fully Converged Teacher" Setting on ImageWoof.
Model
Method
IPC=1
IPC=10
IPC=50
RDED
18.5
40.6
61.5
ConvNet
InfoUtil (Ours)
20.0
42.4
62.6
RDED
20.8
38.5
68.5
ResNet-18
InfoUtil (Ours)
21.4
43.6
69.2
RDED
19.6
31.3
59.1
ResNet-101
InfoUtil (Ours)
19.8
35.0
67.0
C ANALYSIS OF SOFT LABELING STRATEGY ROBUSTNESS
To ensure fair performance assessment, we rigorously isolate the contribution of our proposed InfoUtil from potential advantages conferred by the soft soft-labeling strategy employed by the teacher model. While previous distillation literature has explored utilizing "early-stage teacher" models for maximizing performance at low Images Per Class (IPC) settings, we demonstrate the intrinsic robustness of InfoUtil by unifying the teacher protocol.
C.I CONTROLLED EXPERIMENT WITH FULLY CONVERGED TEACHER
We conducted a controlled experiment on the ImageWoot dataset where both the baseline (RDED) and InfoUtil method were strictly constrained to the exact same Fully Converged Teacher model across all tested IPC settings (1, 10, and 50). This experimental setup eliminates any poten- tial performance artifact stemming from differences teacher model convergence stages, ensuring that measured gains are attributed solely to InfoUtil's data synthesis mechanism (Shapley-based informativeness and GradNorm utility).
The results across different student architectures (ConyNet, ResNet- 18, and ResNet- 101) are re- ported in Table I1. As evidenced by Table IT our method consistently maintains clear perfor- mance advantage over the RDED baseline under this strict controlled setting, with improvements observed across every student architecture and IPC configuration. The gains are particularly sub- stantial in deeper architectures and higher compression rates (e.g., 7.9% margin for ResNet-101 at IPC = 50). This data strongly validates that the performance gains are not an artifact of the label- ing strategy but directly attributable InfoUtil's core mechanism of selecting and synthesizing high-informativeness data.
D PROOFS OF SHAPLEY VALUE AXIOMS
Building upon the game-theorreti theoretic formulation in Section we now formally show that our feature attribution method-whis maximizes informativeness Shapley salies-satisfies the four iomatic properties of Shapley values. These properties ensure that the attributions assigned to input variables theoretically sound and fair.
Consistent with our informativeness maximization framework, we define:
Neural network as characteristic function: The deep neural network acts as the characteristic function in a cooperative game, mapping each coalition of features to a predictive score.
Players: Each input variable (+) (1 E [d] := (1,2...., d}} is treated as a distinct player in the game.
Coalitions: A binary mask E 1] represents coalition of active features, with =
indicating inclusion of ( and = 0 indicating its exclusion.
Reward: The informativeness score f(s .)) is regarded as the reward contributed by the coalition
The Shapley value for variable (+) computed as:
d-1)
())
Σ
(f(xo et)) -(r)) f(ros))
ITs
=
16

---

Published as a conference paper at ICLR 2026
where e, Rd denotes the vector with one in the i-th position but zeros in the rest positions, and 8 is binary mask indicating active input variables.
D.I PROOF OF AXIOM I(LINBARITY)
Axiom 1 (Linearity) two games can be merged into game, then the Shapley Values in the two original games can also be merged. Formally, if smerged = -1 + f2, then & fmergod (x(())
$5. (x(6) + 012( (x(1), Vi [d].
Proof of Axiom 1: For merged game fmerged = fi + f2. by definition we have fmerged (xr o t) f1 t) + f2(1 t) for any mask Substituting into the Shapley value formula:
d dd11)
) IT
Sancraed (z:1))
Σ
(Imorgod (++ - [morgea s))
d
8:s,
0
1
Σ
d - 1)
(f)( f2(22 (s))))))) (fi(oos) fa(cos))
d
1T
=
1
Σ
(d-1) 1)
d
1Ts
#:a=0
d dd-1
Σ
f2( - o2)
1Ts
$:5==0
Φ1 (z) + 1/ (z)
Thus, if finerged f1 f2, then I 6y (z)() vi ld.
D.2 PROOF OF AXIOM 2(DUMMY)
Axiom 2 (Dummy) A dummy player i is player that has interactions with other players in the (s
game f. Formally, is : 8; 0, [x (a ex)) = (() + f(x e4). Then, the dummy player's 's Shapley Value is computed as fx
Proof of Axiom 2: For a dummy player satisfying Vs : 0, (zoo(t)))) = f(roe)), substitute the condition into the Shapley value formula:
))((0 =
1
Σ
d
f(xo
d
1
=:s==0
1
Σ
d - 1)
IT (f(x
0
f(xoe)) 1
Σ
d - 1
1T 9
=:
Note that the sum over all 8: 8: 0 (subsets of the remaining d - variables) satisfies:
d
d-1
d
1
d
Σ
Σ
2d-1
d.
ITs
k
d
=t
0=0
where we use the identity Ei (1) 2n with n Thus:
() = f(xoo)) d d = f(xoee).
D.3 PROOF OF AXIOM 3(SYMMETRY)
Axiom 3 (Symmetry) two players contribute equally in every case, then their Shapley values in the game ff will be equal. Formally, if 8, 8j = 0. f(r (s + e.)) f(x (s ez)), then dy (x)())) th (x11))
17

---

Published as a conference paper at ICLR 2026
Proof of Axiom 3: For symmetric players i and j satisfying Vs =, f(ro f( o (s + ej)), consider their Shapley values:
(1=3)
x)(x0 - Σ
(((zos+t)))-
(Ts
8:a;=0
d=1)
d Σ s:s;
f(z
ITs
Define bijection between masks 8 8; 0 and l 0 j and L= (s) (1)
if j By symmetry, fx + e:)) fx {2)) and f(xos) = fx)t). Since 1 88 ==1 1 the binomial coefficients are equal. Thus:
))(x(0)) d t:ty
Σ
(-1)
o)) = ()(x
1
D.4 PROOF OF AXIOM 4(EFFICIENCY)
Axiom 4 (Efficiency) The total reward of the game f is equal to the sum of the Shapley values of all players. Formally, f(x f()) = ='eed Φ (x)
Proof of Axiom 4: Summing Shapley values over all players:
d d-1
(f(x (f(xo(s+e))) f(x f(xo
1
iE|d
iE|d]
s:s
1
d d-1) ITs
- o()
d
For fixed mask with 1 k there are d k players not in The inner becomes:
Σ Ef(roo + +:)) - f(ro = Σ ((r) (d k))
igs
Summing over all and telescoping the series, all intermediate terms cancel, leaving:
Σ
2)(x f(x) -(())
E
E PROOFS OF THEOREMS
This appendix presents the full derivation to formally establish Theorem complementing the par- tial analysis in the main text.
Recall the definition of utility:
Theorem Utility is bounded by Gradient Norm. Let the utility function 24 be defined as in Definition 3 Then there exists constant e >> 0 such that
U(x ye faw) < cll Vecc (x.), willl-
Using the chain rule for gradient flow, we have
ag
(ffoco (2z), Uy: B) = V &s (zz),
Ot ls'
and similarly for B-g. Thus, the change in gradient flow is
(0(()
a01
(eCCooo (2;), yo; EsJoroo (1z)), M5; B.:) Vores (foce
Øt 1
18
18

---

Published as a conference paper at ICLR 2026
Under SGD with learning rate 1/. the update step is
ah(*)
7
B (x,g)EB
Removing (2i, 31) gives
O0(1)
Σ
Öt 18.
(z
Taking the difference,
00(0)
200))
= - Veco (I;))
0t 8
Ot
B
Substituting this into the gradient flow change gives
è, (Joco (11) Mo; & &e(oo U1;) B-:)
V axc & (aac (zz) 11) V yen 6 (fpce (2:),
S nl V 9c (foce (2,) 8/ || (focs (x,))
where the last step follows from the Cauchy- Schwarz inequaaltyy Let
7
max
V Vfooo (zz). w;)ll
be constant independent of (24, y.). Taking the maximum over (zj, u;) E D, we obtain
U( Us: ) I Voce Celoors (I;), villl-
Note that c = max(sy II ore (ffore (zz). v1 II satisfies the following properties:
1. is independent the current measured data (I,, Vi), ensuring that the bound Theorem holds uniformly for all training examples.
2. It only assumes that the gradient norm Va (3yy), u,)ll has upper bound, which is reasonable assumption for any successfully converged model.
3. Since the learning rate can be chosen to be small in practice, the value of remains controlled and does not become excessively large.
F ADDITIONAL VISUALIZATIONS OF SYNTHETIC DATA
Compared to optimization-based SRe2I., InfoUtil creates more realistic images by preserving details and color consistency. Compared to optimization-fue methods like RDED, InfoUtil stands out with its enhanced interpretability and structured framework, emphasizing key semantic details while reducing focus irrelevant images generated by SRe2L, RDED, and InfoUtil at IPC = 10. Specifically, Figures 6 7 and illustrate results across three representative classes, clearly highlighting the superior visual quality attained by InfoUtil. As intuitive evidence, Figure 5 shows condensed images for ImageNet-IK's IK's indigo bunting category, including results from Original, Herding, SRe2L, RDED, and InfoUtil (Ours). InfoUtil focuses on the most discriminative object parts, yielding more informative results. Additional visualizations are provided in the supplements due to space constraints.
We present further visual comparisons of synthetic ImageNet- 1K
19

---

Published as a conference paper at ICLR 2026
Original
Herding
SRe21
RDED
InfoUtil (Ours)
Figure 5: Visualization of condensed images for the indigo bunting category on ImageNet-IK.
(a) SRe2L
(b) RDED
(c) InfoUtil (ours)
Figure 6: We visualized synthesized images generated by SOTA methods and InfoUtil on ImageNet- IK. These images are distilled from the "Welsh Springer Spaniel" category.
20

---

Published as a conference paper at ICLR 2026
(a) SRe2L
(b) RDED
(c) InfoUtil (ours)
Figure 7: We visualized synthesized images generated by SOTA methods and InfoUtil on ImageNet- IK. These images are distilled from the "schooner" category.
(a) SRe2L
(b) RDED
(c) InfoUtil (ours)
Figure 8: We visualized synthesized images generated by SOTA methods and InfoUtil on ImageNet- 1K. These images distilled from the "indigo bunting" category.
(a) SRe2L
(b) RDED
InfoUtil (ours)
Figure 9: We visualized synthesized images generated by SOTA methods and InfoUtil on ImtgeNet- 1K. These images are distilled from the "Siamese cat" category.
21


Reviews:
[{"review_id": "delwElmYP4", "reviewer": "Reviewer_fiHj", "summary": "The paper propose a new knowledge distillation-based dataset distillation approach that is better theoretically motivated compared to previous method. Their proposed method InfoUtil, which compresses the original dataset at a feature level and at a sample level. At the feature level, InfoUtils utilizes KernelShap to locate salient features within an image to crop on. At the sample level, InfoUtils calculate the gradient norm of each sample and filter samples with low values. InfoUtil demonstrate superior performance compared to previous methods on most benchmarks except for the popular dataset distillation setting: training simple CNNs on CIFAR-10.", "questions": "1. What is the connection between this work and works on data selection, data pruning, and corset selection? Dataset distillation typically optimize the distilled data itself, but the method provided seems to only crop photos and select a subset from real data.  It is unclear whether this constitutes as synthetic data.\n2. How important is adding noise to the patch selection process?\n3. How connected is the utility function, which utilizes gradient norm, to the empirical fisher information? i.e. does maximizing the utility function finds a subset with the maximal empirical fisher information?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. To the best of my knowledge, the proposed method is novel.\n2. The work is well motivated as many dataset distillation methods lack theoretical foundations.\n3. The proposed method outperforms two existing knowledge distillation-based dataset distillation methods: SRe2L and RDED.", "weaknesses": "**1.** The storage budget for the resultant distilled dataset is missing in the paper. One of the key problem with knowledge-based dataset distillation methods is that the resulting distilled dataset can become considerably larger than expected due the soft-labeling [1,2], defeating the whole purpose of dataset distillation. With the lack information on storage size, the usefulness of the proposed method in practice is unclear. \n\n**2.** The performance in the most standard dataset distillation setting, training a simple Convent on CIFAR-10, is very subpar. For 1 image per class, the proposed method only achieves 28.5% compared to MTT 46.3%. BPTT and memory address [3], a previous work that is over 3 years old, achieves 49.1% and 66.4% respectively. While the standard dataset distillation setting maybe too simple and unrealistic, since this work positions itself as being theoretically grounded, the considerable gap in performance in this setting is concerning. To put into perspective, 28% is comparable to gradient matching [4], a dataset distillation work that is 5 years old. \n\n**3.** Missing more recent methods such as TEDDY [5] or EDF [6]. All of the baselines used for comparisons are from 2023 or earlier. \n\n**4.** Missing qualitative examples on what the distilled data looks like\n\n**5.** The presentation of the work should be improved. While the problem formulation is good, the argument in the paper does not provide adequate support as outlined below:\n\n**5.1.** The paper describes gradient flow and how it provides a better approximation of the training dynamics unlike SGD updates. However, the algorithm provided approximates the gradient flow measure with gradient norm, motivated by a bound derived with the assumption of SGD updates. It is unclear from the paper why this can be done as the argument seems very circular. Similarly, as the DATM paper shown, different patterns are learned at different stages of training. The paper lack any discussion on this topic. \n\n**5.2.** The paper outlines challenge 1 with the efficiency performance trade-off between matching-based dataset distillation method, but lack any further discussion as the method proposed is a knowledge-distillation based method\n\n**5.3.** Since the paper position itself as being more theoretically grounded, it needs more details on image reconstruction, soft label generation, and diversity control. For instance, adding random noise during the patch selection process is not grounded in the proposed theory at all. \n\n**5.4.** Minor typos: nclude -> include line 323,  table 6 ablation is confusing (row 2 and 3 are the same?)\n\n[1] Qin, Tian, Zhiwei Deng, and David Alvarez-Melis. \"A label is worth a thousand images in dataset distillation.\" Advances in Neural Information Processing Systems 37 (2024): 131946-131971.\n\n[2] Xiao, Lingao, and Yang He. \"Are Large-scale Soft Labels Necessary for Large-scale Dataset Distillation?.\" Advances in Neural Information Processing Systems 37 (2024): 16406-16437.\n\n[3] Deng, Zhiwei, and Olga Russakovsky. \"Remember the past: Distilling datasets into addressable memories for neural networks.\" Advances in Neural Information Processing Systems 35 (2022): 34391-34404.\n\n[4] Zhao, Bo, Konda Reddy Mopuri, and Hakan Bilen. \"Dataset Condensation with Gradient Matching.\" Ninth International Conference on Learning Representations 2021. 2021.\n\n[5] Yu, Ruonan, et al. \"Teddy: Efficient large-scale dataset distillation via taylor-approximated matching.\" European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2024.\n\n[6] Wang, Kai, et al. \"Emphasizing discriminative features for dataset distillation in complex scenarios.\" Proceedings of the Computer Vision and Pattern Recognition Conference. 2025.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DFR14bbNE3", "reviewer": "Reviewer_tEAx", "summary": "This paper revisits knowledge-distillation-based dataset distillation and introduces a principled framework named InfoUtil, which jointly optimizes Informativeness and Utility in the synthesis of compact datasets. The authors formally define informativeness at the patch level using a Shapley-value–based game-theoretic attribution, and utility at the sample level using gradient flow. The proposed two-step pipeline first selects the most informative image regions and then retains high-utility samples via gradient-norm scoring. Extensive experiments on CIFAR, Tiny-ImageNet, and ImageNet-1K show consistent improvements over prior methods such as RDED and SRe2L.", "questions": "1. Could the authors further elaborate on the key differences between the proposed InfoUtil and RDED? From the visualizations, it appears that the improvement mainly comes from the occurrence of complete targets. However, in RDED, similar results can be achievable by simply adjusting the difficulty level of the selected patches. In that case, RDED could potentially produce images visually comparable to those from InfoUtil. I am therefore curious whether InfoUtil still provides additional information gain beyond what RDED can achieve through such adjustments.\n\n2. Is the Shapley-based patch selection the only possible approach to maximize informativeness? Could alternative attribution or saliency methods, such as Grad-CAM, achieve comparable results in practice? It would be helpful if the authors could clarify why Shapley values are particularly suited for this task and whether other approaches were considered or tested.\n\n3. I am curious whether the images distilled by InfoUtil could further improve the performance of training-based (TB) methods if used as initialization. Since InfoUtil is designed to generate informative and utility-optimized synthetic data, it would be interesting to investigate whether integrating it with TB approaches could lead to additional performance gains or faster convergence compared to using RDED for initialization.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. Presents a clear theoretical framework that connects informativeness and utility under a unified definition of optimal dataset distillation.\n2. The use of Shapley-value–based attribution provides interpretability and theoretical grounding to the distilled sample selection.\n3. Demonstrates strong empirical performance across multiple datasets and architectures, showing robustness and scalability.\n4. Ablation studies and cross-architecture evaluations validate each component’s contribution and highlight the method’s generalizability.", "weaknesses": "1. Although informativeness is formally defined as an optimization over binary masks (Eq. 2), the method substitutes this process with a Shapley-value attribution heuristic. The paper does not show that Shapley-based patch selection approximates or lower-bounds the true informativeness objective, nor does it provide conditions under which this substitution is valid. This leap from optimization to attribution lacks formal grounding, especially given nonlinear interactions among image regions.\n\n2. The theoretical analysis of Utility (Definition 2, Equation (3)) relies on a continuous-time gradient flow formulation to describe training dynamics. In this setup, parameter updates are modeled as smooth differential equations. However, in practice, training is conducted via discrete stochastic gradient descent (SGD) with randomness from data augmentation and normalization layers. The paper does not specify under what assumptions the continuous gradient flow accurately approximates the discrete SGD process (e.g., small learning rate, smooth loss landscape). Consequently, the theoretical derivations based on gradient flow may not faithfully represent real training behavior, which weakens the practical validity of the proposed Utility-based analysis.\n\n3. Theorem 1 establishes only an upper bound  $U(x, y) \\le c \\|\\nabla_{\\theta} \\ell(f_{\\theta}(x), y)\\|.$ However, using the gradient norm as a surrogate for utility in sample selection implicitly assumes a monotonic or bounded relationship between the two. Without a corresponding lower bound or a proof of ranking consistency, this substitution lacks theoretical justification; large gradient norms do not necessarily imply high utility. As a result, the theoretical foundation of the utility maximization step remains incomplete.\n\n4. The baseline methods used for the main comparison, namely RDED (CVPR 2024) and SRe2L (NeurIPS 2023), are relatively outdated. Several more recent dataset distillation techniques, such as EDC (NeurIPS 2024) and DELT (CVPR 2025), have demonstrated significantly improved performance and scalability. Without comparisons to these stronger and more up-to-date baselines, it is difficult to substantiate the claim that the proposed method achieves state-of-the-art or optimal performance.\n\n5. (Minor) In line 414, there should be a space between \"samples\" and \"We\".\n\n6. (Minor) Visualization is important; the authors are encouraged to move the results from Appendix F to the main paper for better clarity.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0zJwU1m9AN", "reviewer": "Reviewer_987g", "summary": "The paper introduces InfoUtil, a theoretically grounded framework for dataset distillation that aims to generate compact yet highly representative datasets. The method operates in two key stages. In the first stage, it employs the Shapley value, a game-theoretic attribution method, to identify the most informative image patches, ensuring that each synthetic sample captures the essential visual and semantic information from the original data. In the second stage, InfoUtil selects samples based on their gradient norms, which serves as a principled measure of utility by estimating the influence of each sample on model training dynamics. This dual optimization of informativeness and utility enables the distilled dataset to retain both rich content and strong learning potential. Extensive experiments across multiple benchmark datasets and architectures demonstrate that InfoUtil not only achieves superior performance but also provides interpretability and computational efficiency compared to existing methods, making it a robust and scalable approach to dataset distillation.", "questions": "A writing suggestion. In line 181-184, the authors should clarify which components are functions of \\tilde{D}. Explicitly specifying the elements that depend on the distilled dataset would improve the mathematical clarity of the formulation and help readers better understand how \\tilde{D} influences the optimization process.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "The strength of this work lies in its solid theoretical foundation and practical effectiveness. Unlike prior heuristic or empirical approaches, InfoUtil unifies the concepts of informativeness and utility within a principled mathematical framework, providing interpretability and transparency to dataset distillation. Its combination of game-theoretic Shapley value attribution and gradient norm-based sample selection ensures that the distilled data are both highly informative and influential for model training. Furthermore, InfoUtil achieves state-of-the-art performance across multiple benchmarks while being significantly more computationally efficient, requiring far less memory and training time than previous methods. This balance of theoretical rigor, interpretability, and scalability makes InfoUtil a robust and impactful contribution to the field of dataset distillation.", "weaknesses": "Despite its strong theoretical grounding and empirical performance, the paper contains a few minor weaknesses. There are some typo errors. \n\n1. In lines 89–90, where the phrase \"we reconsider the knowledge distillation-based dataset distillation process by introducing Principled Dataset Distillation (Definition 4)\" should read \"Optimal Dataset Distillation\" to maintain consistency with the terminology used throughout the paper. \n2. Similarly, in line 323, the word \"nclude\" in “Baseline nclude trajectory-matching\" should be corrected to \"include.\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ut8u8fYTiF", "reviewer": "Reviewer_CGAE", "summary": "This paper proposes InfoUtil including two key concepts: Informativeness (how much useful information a sample carries) and Utility (how essential a sample is for training). InfoUtil maximizes these objectives via (1) Shapley Value-based informativeness attribution and (2) Gradient Norm-based utility selection. Experiments on ImageNet-1K with ResNet-18 show a 6.1% improvement over prior SOTA.", "questions": "pls see the weaknesses.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The presentation is clear and well-structured.\n\nThe theoretical analysis  is reasonably justified.", "weaknesses": "I noticed that the authors carefully adopt different levels of teacher models to generate soft labels for different IPC settings. However, since this labeling strategy is directly borrowed from prior works rather than being a core contribution of this submission, it should be consistently applied to all comparison methods to ensure fairness, at least to RDED. Otherwise, part of the reported performance gain may be attributed to better soft labeling rather than the proposed method itself.\n\nThe performance on ConvNet with CIFAR is significantly weaker than that of recent competitors. In addition, several of the compared distillation-based baselines are outdated. Pls consider some newer and stronger baselines such as \n- Elucidating the Design Space of Dataset Condensation\n- Dataset Distillation via the Wasserstein Metric\n- Breaking Class Barriers: Efficient Dataset Distillation via Inter-Class Feature Compensator.\n- Heavy Labels Out! Dataset Distillation with Label Space Lightening\n\nThe current ablation is also insufficient. The contributions of GradNorm Scoring and Attribution Cropping should be clearly isolated from the carefully designed soft labeling strategy in order to demonstrate their real effectiveness. Furthermore, in large IPC settings, it is unclear why SRe²L and RDED are missing, especially given that both are already open-sourced and widely used. \n\nLines 435–436 in Table 6 report identical settings (both GradNorm Scoring and Attribution Cropping), yet the numbers differ. Please verify and correct.\n\nFinally, there are typos, e.g., “scriptsizeImageNet.” in line 364.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)