ROME is Forged in Adversity: Robust Distilled Datasets via Information Bottleneck
Authors: Zheng Zhou
Wenquan Feng
Qiaosheng Zhang
Shuchang Lyu
Qi Zhao
Guangliang Cheng
Venue: ICML 2025 poster (2025)
Keywords: Dataset distillation
adversarial robustness
information bottleneck
synthetic subsets
adversarial attacks
Primary area: deep_learning->robustness
TL;DR: We introduce ROME, a method that enhances the adversarial robustness of dataset distillation by leveraging the information bottleneck principle, leading to significant improvements in robustness against both white-box and black-box attacks.

Abstract: Dataset Distillation (DD) compresses large datasets into smaller, synthetic subsets, enabling models trained on them to achieve performance comparable to those trained on the full data. However, these models remain vulnerable to adversarial attacks, limiting their use in safety-critical applications. While adversarial robustness has been extensively studied in related fields, research on improving DD robustness is still limited. To address this, we propose ROME, a novel method that enhances the adversarial RObustness of DD by leveraging the InforMation BottlenEck (IB) principle. ROME includes two components: a performance-aligned term to preserve accuracy and a robustness-aligned term to improve robustness by aligning feature distributions between synthetic and perturbed images. Furthermore, we introduce the Improved Robustness Ratio (I-RR), a refined metric to better evaluate DD robustness. Extensive experiments on CIFAR-10 and CIFAR-100 demonstrate that ROME outperforms existing DD methods in adversarial robustness, achieving maximum I-RR improvements of nearly 40% under white-box attacks and nearly 35% under black-box attacks. Our code is available at https://github.com/zhouzhengqd/ROME.

ROME is Forged in Adversity: Robust Distilled Datasets via Information Bottleneck
Zheng Zhou
Wenquan Feng'
Qiaosheng Zhang 23 Shuchang Lyu'
Qi Zhao
Guangliang Cheng
Abstract
Sege
Inelovai
DC DSA,
Releneer Incommation
Arplass
Dataset Distillation (DD) compresses large datasets into smaller, synthetic subsets, enabling models trained on them to achieve performance comparable to those trained on the full data. How- ever, these models remain vulnerable to adver- sarial attacks, limiting their use in safety-critical applications. While adversarial robustness has been extensively studied related fields, research improving DD robustness is still limited. To address this, propose ROME. novel method that enhances the adversarial RObustness of DD by leveraging the InforMation BottlenEck (IB) principle. ROME includes two components: a performance-aligned term to preserve accuracy and a robustness aallune term improve robust- ness by aligning feature distributions between syn- thetic and perturbed images. Furthermore, we in- troduce the Improved Robustness Ratio (I-RR), refined metric to better evaluate DD robustness. Extensive experiments on CIFAR-10 and CIFAR- 100 demonstrate that ROME outperforms existing DD methods in adversarial robustness, achieving maximum I-RR improvements of nearly 40% un- der white-box attacks and nearly 35% under black- box attacks. Our code is available at https: ROME.
NO
Apple Aapleas 3ou
Protectin
Spsliii
Doy
(a) Previous Method
Trannce Stape
Louneg Sage
Ligh Accerse
Relevore Information
Arrlor
Disillate
Datunel
Infermaation
Coni
Coni
Anylaan
Batteeeck
Bae
Spileii
(b) Our Method
Figure I. Comparison of previous DD methods and the pro- posed ROME under adversarial attacks: (a) Previous meth- ods align representations between original and synthetic datasets remain vulnerable to adversarial attacks to neglecting the mutual information among input X, latent representations. 2, and output Y, Icading to reduced accuracy under perturbations. (b) Our method, ROME, employs the information bottleneck principle minimize mutual information between X' and Z, while maximiz- ing it between y and Z. thereby enhancing adversarial robustness and maintaining high accuracy under perturbations.
/ //github.c. cwm/
Chang et al., 2024) and large vision-language models (Rad- ford et al., 2021), enabling models to achieve high accuracy and generalization across diverse domains (Thirunavukarasu et al., 2023; Kirchenbauer et al., 2023). However, training on such massive datasets presents substantial challenges, including high computational costs, excessive memory us- age, and prolonged training times, particularly in resource- constrained environments (Lei & Tao, 2024; Yu et al., 2023).
1. Introduction
The rapid expansion of large datasets has driven significant advancements in computer vision and deep learning appli- cations, including large language models (Vaswani, 2017;
'Beihang University. Beijing, China "Shanghai Artificial Intelli- gence Laboratory, Shanghai, China 'Shanghai Innovation Institute, Shanghai, China UUniversity of Liverpool, Liverpool, UK. Corre- spondence to: Shuchang Lyu <lyushuchange @ buaa.edu.cn>.
Dataset Distillation (DD) (Wang et al., 2018) addresses these challenges by compressing large datasets into small, syn- thetic subsets, allowing models to achieve performance com- parable to those trained on full datasets. Despite the success
Proceedings of the nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

ROME is Forged in Adversity: Robust Distilled Datasets via Information Bottleneck
of methods like DC (Zhao et al., 2021), MTT (Cazenavette et al., 2022), DM (Zhao & Bilen, 2023), IDM (Zhao al., 2023), and BACON (Zhou et al., 2024b), models trained on distilled datasets remain vulnerable to adversarial at- tacks (Zhou et al., 2024a; Wu et al., 2024), as illustrated in Figure I(a). These vulnerabilities pose significant risks to critical applications such as face recognition (Wei et al., 2022a;b), object detection (Zhou et al.. 2024c: Hu et al., 2021), and autonomous driving (Wang et al., 2021a; Yuan et al., 2023), underscoring the need for robust dataset distilla- tion techniques that enhance adversarial robustness without compromising performance.
2024a), better evaluate adversarial robustness. Extensive experiments CIFAR-10 and CIFAR-100 demonstrate that ROME outperforms existing DD methods, achieving max- imum I-RR improvements of nearly 40% and nearly 35% under white-box and black-box attacks, respectively.
Our main contributions are summarized as follows:
Theoretically, we first introduce the IB to DD for de- riving robust distilled datasets. By leveraging the con- ditional entropy bottleneck, we obtain a numerically feasible lower bound and reformulate the IB principle, incorporating adversarial perturbations as a prior for feature learning. This approach is termed RObust dis- tilled datasets via inforMation bottlenEck (ROME).
While adversarial robustness distillation (Goldblum et al., 2020; Kuang et al., 2023; Wang et al., 2024) has been ex-
tensively explored to enhance Adversarial Robustness (AR) in knowledge distillation (Hinton, 2015; Gou et al., 2021), its application to DD remains limited. Recent efforts have made strides in addressing this gap. For instance, Wu et al. (2024) introduced DD-Robustbench, a framework for evalu- ating AR across distilled datasets, while Zhou et al. (2024a) proposed BEARD, a game-theoretie framework with three metrics for AR evaluation, including adversarially trained models. Additionally, Xue et al. (2024) developed GUARD, which enhances AR through curvature regularization. De- spite these advances, many methods rely on adversarial training (Madry et al., 2018), which retrains models with adversarial examples to improve AR. However, applying ad- versarial training to DD presents two significant challenges:
Algorithmically, to implement ROME, we propose two key training terms: a performance-alignel term to preserve the accuracy of models trained on distilled datasets, and a robustness-aligned term that maximizes the margin between synthetic images and the decision boundary improve adversarial robustness.
Experimentally, we introduce I-RR, a refined metric for more effective evaluation of DD robustness. Extensive experiments on benchmark datasets including CIFAR- 10 and CIFAR- 100 demonstrate that our method out- performs existing DD approaches in adversarial robust- ness under both white-box and black-box attacks.
a. High computational cost of retraining.
2. Related Work
b. Trade-off' between model performance and robustness.
2.1. Dataset Distillation
Dataset distillation, introduced by Wang et al. (2018), is a bi-level optimization problem that computationally ex- pensive due to nested recursion. Zhao et al. (2021) pro- posed Dataset Condensation (DC), which improves perfor- mance by aligning gradients between original and synthetic datasets. Zhao & Bilen (2021) introduced DSA to enhance distillation via data augmentation, while Cazenavette et al. (2022) proposed MTT to match training trajectories. Zhao & Bilen (2023) introduced Distribution Matching (DM), later refined by Zhao al. (2023) as Improved Distribution Matching (IDM). Zhou et al. (2024b) proposed BACON, Bayesian method for DD that improves performance.
To address these challenges, we propose novel method called ROME, which enhances the adversarial RObustness of DD using the InforMation BottlenEck (IB) (Tishby et al., 2000) principle. ROME reduces computational costs for downstream tasks while maintaining both robustness and performance. By leveraging the Conditional Entropy Bot- tleneck (CEB) (Fischer, 2020), a variant of IB with label priors, ROME minimizes the mutual information between input X' and latent representations Z, while maximizing it between output y and Z to enhance the relevant infor- mation, as shown in Figure 1(b). This design leads to two key components: a performance-aligned term to preserve accuracy, and a robustness- aligned term to improve adver- sarial robustness by aligning feature distributions between synthetic and perturbed images. Additionally, we intro- duce robust prior for dataset generation by applying ad- versarial perturbations based on the CEB theory, which can be further strengthened by incorporating robust pretrained models (Hendrycks et al., 2019; Goldblum et al., 2020). Furthermore, we propose the Improved Robustness Ratio (I- RR), an enhanced metric derived from BEARD (Zhou et al.,
2.2. Adversarial Robustness Distillation
Adversarial robustness distillation, introduced by Goldblum et al. (2020), was initially proposed to improve AR in knowl- edge distillation and demonstrates that smaller models can achieve enhanced robustness without incurring additional training costs. Zi et al. (2021) showed that soft labels from the teacher model significantly improve the robustness of the student model. Furthermore, Kuang et al. (2023) addressed
2

---

ROME is Forged in Adversity: Robust Distilled Datasets via Information Bottleneck
the limitations of adversarial training (Madry et al., 2018) by proposing Information Bottleneck Distillation (IBD), which leverages the information bottleneck principle to en- hance robustness. Wang et al. (2024) introduced DFARD, a method that enables robust model training without the need for additional data. Despite notable progress, research on enhancing AR DD remains relatively scarce. Promi- nent works in this field include DD-Robustbench (Wu al., 2024), benchmark tailored for evaluating AR in distilled datasets, and BEARD (Zhou et al., 2024a), which provides thorough AR evaluation using a game-theoretic frame- work and three distinct metrics. BEARD also examines models adversarially trained on distilled datasets, thereby boosting AR. However, this approach necessitates retrain- ing and introduces a trade-off between model performance and robustness. Furthermore, Xue et al. (2024) introduced GUARD, method that enhances AR through curvature regularization. Nonetheless, GUARD requires specific as- sumptions, such as the convexity of the loss function and the linearity of the feature extractor, which may restrict its applicability across diverse models and tasks.
Neural Network
Original Dataset
Synthetie Dalaset
Maximize
F('yy) 8(x,)
KY.Z)
Entray)
CE [s
(a) Performance-alignee Term
Original
Noural Network
Dataset
Synthetic
(K,,) Miiimiieeee
Datasct
K0: 11 Exccs || | lcto) E |o(9]
(b) Robustness-alinnet Term
In contrast, we propose ROME, a novel method that en- hances both performance and robustness by integrating the information bottleneck principle. ROME improves AR while maintaining high accuracy under adversarial attacks, eliminating the need for retraining. Inspired by Kuang et al. (2023), ROME leverages the conditional entropy bottleneck within DD, providing a flexible and general framework that effectively balances model performance and robustness. To implement ROME, introduce two key components: performance-alinnee term to ensure the accuracy of models trained on distilled datasets, and robustness-aligne aliinned term that maximizes the margin between synthetic images and the decision boundary to enhance AR. Extended background and formal definitions are provided in Appendix B.
Figure 2. The framework of ROME: ROME utilizes the informa- tion bottleneck to frame the robust dataset distillation problem a min-max optimization of mutual information. It consists of two key components: (a) The performance-aligee allignd term maximizes the mutual information between the latent space Z and the output y by aligning the logits with the true labels. (h) The robustness-aligeed term minimizes the mutual information between 2 and the input X, conditioned on robust prior X' (the adversarially perturbed dataset), by aligning the embeddings to reduce the discrepancy.
this by minimizing mutual information between input and hidden layers while maximizing it between hidden layers and output, effectively filtering irrelevant input information and enhancing adversarial robustness. Prior work (Wang et al., 202 lb; Xu al.. 2022; Kuang et al., 2023) has demon- strated IB's s effectiveness in improving robustness. Inspired by IBD (Kuang et al., 2023), which applies IB to knowledge distillation for robustness enhancement, we integrate IB into DD to address the challenges of adversarial training. The framework of our method is illustrated in Figure 2.
3. Robust Dataset Distillation via Information Bottleneck
3.1. Preliminary
Motivation. Dataset distillation compresses large datasets into compact subsets while preserving comparable perfor- mance. While adversarial robustness is well studied in re- lated areas such as knowledge distillation (Goldblum al., 2020), it remains underexplored in DD, which focuses efficiency and accuracy. Adversarial training, a common approach to enhance robustness, suffers from two key limi- tations: (a) high computational cost of retraining, and (b) trade-off between model performance and robustness.
Information Bottleneck (IB). The information bottleneck principle, introduced by Tishby et al. (2000) for informa- tion compression, has been applied deep learning mod- els (Tishby & 7aslavsky, 2015). IB aims to find repre- sentation Z that preserves much information as possible about the target labels y, while reducing its dependence on the input X. The IB objective is formulated as:
RIB max IQ; 3I(X;
(I)
Adversarial examples are non-robust features (Ilyas al., 2019) that can be mitigated through their removal. The infor- mation bottleneck principle (Tishby et al., 2000) addresses
where denotes the mutual Information and 8 controls the trade-off between T(): 2) and T(X;
3

---

ROME is Forged in Adversity: Robust Distilled Datasets via Information Bottleneck
Variational Information Bottleneck (VIB). The varia- tional information bottleneck (Alemi et al., 2017) extends the IB principle by leveraging variational inference to ap- proximate mutual information with a tractable lower bound. The VIB objective is formulated as:
Algorithm 1 RObust Dataset Distillation Infor
BottleNeck (ROME)
Require: Perturbed data &, one-hot labels y², classes c
{0, I,...,C 1}. pretrained logits f(-), embeddings
e(.), learning rate 1). iterations T. Ensure: Robust distilled dataset X'. 1: Initialize synthetic dataset X' by randomly sampling from X with corresponding class labels;
T(); - ( (I;
p(z[z) ((2)
2 E (p(a))((=())) log ((vl2) 3 log
(2)
for t to T' do
where p(z|z) represents the latent space distribution, ( approximates the true conditional distribution p(yl=), and ((2) is a fixed K-dimensional spherical Gaussian (i.e., ((2) = (= 1)). VIB leverages deep neural networks to parameterize these distributions, enabling efficient han- dling of high-dimensional, continuous data like images and overcoming earlier constraints to díscrete or Gaussian cases. The proof of Eq. 2 is provided in Appendix A.I.
3: for c = 0 to C- do
4: Select Xe as the subset of X for class c, and & as the subset of X for class c;
5:
end for
6:
Compute the performance--lignedd term Pent Alig us- ing Eq. 10;
Compute the robustness-aligned term CRob using Eq. 11;
8:
Compute total loss LOOTAI. as in Eq. 12;
Conditional Entropy Bottleneck (CEB). Building on
9: Update synthetic dataset: X' f X v/ x CTOTAL:
Fischer (2020) proposed the conditional entropy bot- tleneck, which leverages label prior information to better approximate (z). The CEB can be formulated as:
10:
end for
11:
return robust distilled dataset X'
1V;: z) (I (
where p(=[ar) represents the conditional probability distribu- tion of latent information & E Z given the synthetic dataset x E X, and (=[() denotes the variational distribution of = E 2 given the perturbed source dataset E &.
logglul2 - Blog p(z|z)
2 E (((as)((aaa))
(=1l))
3.2. Information Bottleneck Perspective for Robust Dataset Distillation
Theorem 3.4. The variational lower bound of ROME can be computed follows (Proof in Appendix A.4):
ROME = 1)); - (1 2
Definition 3.1 (Robust Distilled Datasets via Information Bottleneck (ROME)). Let )' represent a random variable corresponding to the output information, and Z represent a random variable corresponding to the latent space informa- tion in a neural network. Let X' denote the synthetic dataset, and & denote the source dataset. To introduce a robust prior for the source dataset, adversarial perturbations are applied to X. The parameter 3 controls the relationship between X, X,, and Z. The ROME can be defined as follows:
2 Extrrs..nnn(aala.... logg(ulz) - :logg p(=|z)
(=(Ø))
Remark 3.5. By leveraging variational methods, we aim to refine our understanding and utilization of mutual informa- tion in the context of ROME. The variational lower bound Ep(y.=) [log q(vl2)] as detailed in Theorem 3.2, serves as fundamental measure. It optimizes the expectation of log (2)]2) with respect to the distribution ((v]z), where (z]22) acts as a variational approximation to the true con- ditional distribution p(yl=). This bound provides a conser- vative estimate of 7 I(); Z), ensuring that (yylz) effectively captures dependencies between output information ) and hidden variables Z. Simultancously, the variational upper bound E (p(r,))((slr.a)) [log p(=[z) as elucidated in Theo- q([())
ROME = 2):;2) -T 2)
(4)
Theorem 3.2. The variational lower bound of I()) Z) can be computed as follows (Proof in Appendix A.2):
I(); 2) 2 Ep(g.a) (log ((v) []]]
(5)
where p(v, 2) is the joint distribution of output information )) and latent information Z with y E y and 2 E Z, and ((yl=) is the variational distribution of y conditioned on 2.
rem 3.3, offers an upper bound for / [(x; 2)))). ). quantifies the expected log ratio between the true conditional distribu- tion p(=[zr) of latent variables 2 given the synthetic dataset X' and the variational distribution g(2(2). given the per- turbed dataset &. These bounds help optimize variational distributions in ROME, improving the distillation of robust datasets while maintaining key information.
Theorem 3.3. The variational upper bound of 1(X;; Z)²
can be calculated as follows (Proof in Appendix A.3):
p(2[x)]
T(x:: 2 < Ep(c.
log
(6)
q(2 5+)

---

ROME is Forged in Adversity: Robust Distilled Datasets via Information Bottleneck
Comparison of model robustness when trained using various DD methods with IPC settings {1, 10, 50), against both white-hox targeted and untargeted attacks the CIFAR-10 10 and CIFAR- 100 datasets. Robustness evaluation metrics include RR and CREL, well as their improved versions I-RR and I-CREL The best results between the baseline and proposed methods are highlighted in bold, while the second-bes results are underlined. Improvements in metrics compared the second second-best results are highlighted in red.
Table
Targeted Attack
Untargeted Attack
Dataset
Method
RR
CREI
I-RR
I-CREI
RR
CREI
I-RR
I-CREI
Full-size
20.42%
24.98%
67. 67.24%
48.39%
28.33%
25.12%
28.82%
25.36%
DC 2020
30.79%
29.35%
88.51%
58.21%
31.87%
26.70%
56.02%
38.78%
CIFAR-II 10
DSA 2021
45.22%
36.43%
86.81%
57.22%
36.53%
27.75%
53.66%
36.32%
MTT
36.00%
32.26%
83.95%
56.24%
33.30%
26.26%
48.34%
33.77%
DM 2023
46.01%
36.01%
85.76%
55.89%
34.50%
28.32%
56.19%
39.16%
IDM 2023
32.35%
27.75%
87.07%
55.11%
33.03%
28.46%
53.43%
38.66%
BACON
2024
36.83%
33.05%
84.37%
56.82%
32.87%
27.20%
50.49%
36.01%
81.36%
55.28%
97.44%
63.32%
49.86 %
35.05%
67.01 %
43.62
ROME
(35.35 f)
(18.85
(8.93
(5,11
(13.33 f)
(6.59 1)
(10.82 f)
(4.46 1)
Full-size
6.77%
18.18%
65.50%
47.55%
19.91% %
18.60%
20.08%
18.69%
DC 2020
33.11%
30.31%
77.14%
52.32%
28.74%
22.40%
32.33%
24.19%
CIFAR- 100
DSA 2021
43.97%
35.01%
72.97%
49.51
28.53%
20.40%
33.29%
22.77%
MTT 2022
36.06%
31.16%
74.54%
50.40%
26.07%
19.65%
31.10%
22.17%
DM
39.32%
31.32%
71.29%
47.30%
26.72%
19.78%
29.74%
21.28%
IDM 2023
34.44%
27.16%
74.57%
47.23%
26.28%
20.36%
30.83%
22.63%
BACON 2024
31.81%
29.78%
69.96%
48.86%
25.26%
19.30%
27.42%
20.38%
103.09
66.18%
100.65%
64.96%
44.10%
28.29 %
46.24%
29.36%
ROME
(59.12 f)
(31.17 1)
(23.51 1)
(12.64 1)
(15.36 1)
(5.89 1)
(12.95 f)
(5.17
3.3. Optimization Framework for Performance and Robustness Alignment
where X' and X' are class-aligned sample sets (i.e., X' con- tains synthetic samples and X' perturbed original samples, both partitioned by the label y), p(z, & y) is the joint distri- bution, e(-) is the embedding layer output, and || denotes the squared Total Variation distance.
To achieve optimal performance and adversarial robustness, we maximize Ep(a...v)e(all.s... [log [(][2)] and minimize
Simultaneously. maximiz- ing the first term aims to enhance classification accuracy, while minimizing the second term helps reduce the discrep- ancy between the probability distributions of the synthetic dataset X' and the perturbed source dataset X. For clarity, we refer to these objectives as the performance-aligned term and pouustness-alignee term, respectively.
Epta, (ytee log
p(aiz) ((]
Monte Carlo Approximation for ROME. To approxi- mate the expectations in Eq. and Eq. 9, we apply Monte Carlo sampling. Specifically, for each class c E C = {0, 1,..., 1}. we draw synthetic samples x and cor- responding perturbed original samples å under class c. We then aggregate the sampled pairs across all classes with equal weighting to construct empirical estimates. The performance-aligee aligned term is approximated as:
Theorem 3.6. The performance-aligner term can also be expressed as follows (Proof in Appendix A.5);
C-1
CpeifAlle ())( CE [g' f())]]
(8)
Opert
Σ
Σ
CE
f(r)),
(10)
ZER,
where f(-) is pretrained model robust adversarial at- tacks, and (x) denotes its logits output for input y is the one-hot true label vector, and CE denotes cross-entropy.
while the robustness-aligned term is estimated by
The robustness-alignee term can also be expressed as the following lower bound, derived by scaling Pinsker's inequality (Proof in Appendix A.6):
Theorem 3.7.
Loob
Σ AEX
c(t)
TEX.
(11)
L Allg Bp((,.2.)) | [e(x)] B [e(+)]]|
where X. and & are the synthetic and perturbed sample subsets of category e, with sizes [X, and [&C], respectively.
(9)
5

---

ROME is Forged in Adversity: Robust Distilled Datasets via Information Bottleneck
4.1. Experiment Settings
Table Comparison of model robustmess measured by I-RR for various dataset distillation methods with IPC-50 under targeted and untargeted transfer- based and query-base black-box attacks on CIFAR-10. Best results are in bold, second-best underlined, and improvements over the second-best highlighted in red.
Datasets and Baseline Methods. To systematically eval- uate method,
conduct experiments using the BEARD (Zhou et al., 2024a) benchmark, which is specifi- cally designed to assess the adversarial robustness of dataset distillation methods. The datasets used in our evaluation are CIFAR-10 (Krizhevsky, 2009) and CIFAR- 100 (Krizhevsky, 2009). We compare the performance of our method against six baseline dataset distillation techniques: DC (Zhao et al., 2021), DSA (Zhao & Bilen, 2021), MTT (Cazenavette et al., 2022), DM (Zhao & Bilen, 2023), IDM (Zhao et al., 2023), and BACON (Zhou et al., 2024b).
Targeted Attack
Untargeted Attack
Method
Transfer
Query
Transfer
Query
DC 85.84%
88.71% %
83.97%
43.81 %
DSA
94.09%
94.95%
92.31% %
54.60%
MTT
91.40%
92.76%
89.02%
48.71%
DM
92.22%
93.86%
90.36%
57.53%
IDM
92.17%
94.37%
89.22%
63.23%
BACON
92.46%
94.67%
89.25%
63.26%
Evaluation Attack. We evaluate the robustness of ROME against both white-box and black-box adversarial attacks. For white-box attacks, we adopt FGSM (Goodfellow et al., 2015), PGD (Madry et al., 2018), DeepFool (Moosavi- Dezfooli et al., 2016), C&W (Carlini & Wagner, 2017), and AutoAttack (Croce & Hein, 2020). For black-box attacks, we consider (1) transfer- attacks, where adversarial examples generated from a surrogate model are used to evaluate models trained with different dataset distillation methods; and (2) query-based attacks, where adversarial examples are crafted by querying the model using methods such as Square (Andriushchenko et al., 2020) and SPSA (Ue- sato et al., 2018) 8). Both targeted and untargeted scenarios are evaluated for comprehensive analysis.
99.90%
99.79 %
98.44%
78.46% %
ROME
(5.81 1)
(4.84 1)
(6.13 1)
(15.2 1)
3.4. Overall Framework and Pseudocode
In summary. the overall loss function of ROME combines the terms in Eq. 10 and Eq. 11. The total loss function is defined as follows:
LOOTAL. = (1 CPetLAll + A.ob
(12)
where the hyperparameter a serves as the weighting factor for the total loss function and is adjustable. By tuning a, we can customize the loss function to optimize performance.
To ensure a fairer evaluation, we introduce the Improved Ro- bustness Ratio (I-RR), which refines the original Robustness Ratio (RR) from BEARD (Zhou et al., 2024a). Since RR can overestimate robustness when the Attack Success Rate (ASR) is dominated by strong attack, I-RR incorporates model accuracy (ACC) to better balance robustness and per- formance. Additionally, we propose the Improved Compre- hensive ciasstness-Effciinne Index (I-CREI,, combining I-RR with the Attack Efficiency Ratio (AE) from BEARD to jointly assess robustness and efficiency. These metrics reduce sensitivity to outlier attacks, providing more stable and fair evaluation. In our experiments, I-RR is used for black-box robustness, where efficiency metrics (c.g., com- putation time) are unavailable, making I-CREI inapplicable. In white-box settings, I-CREI is preferred for its compre- hensive assessment of robustness and efficiency. Formal definitions and further details are provided in Appendix C. I. Detinition 4.1 (Improved Robustness Ratio (I-RR)). Given neural network model m E . and an adversarial attack function a E A, the I-RR is defined as:
Pseudocode Description. The pseudocode for ROME is shown in Algorithm 1. ROME guides the distillation pro- cess effectively enhance both performance and robustness. Inputs include the perturbed dataset X, one-hot true labels y' for synthetic dataset X, classes E {0, 1......., C -1), pretrained logits f-). embeddings e(-), learning rate and total iterations T. The synthetic dataset X is initialized by sampling from de with class labels. For each itcration t= 1 to T, class- wise subsets X, and & are selected, The cetemmance-allinnee term CrertAlig and robustness- aligned term LR Alig are computed via Eq. 10 and Eq. 11, respec- tively. The total loss LTOAAL is calculated using Eq. 12, and the synthetic dataset X' is updated via gradient descent.
4. Experiments
In this section, we first outline the experimental setup and evaluation procedure in Section 4. We then assess the adversarial robustness and accuracy of various DD methods under both white-box and black-box attacks in Section 4.2. Section 4.3 presents a comparison of training efficiency for adversarially distilled datasets. Ablation studies of the pro- posed ROME method are provided in Section 4.4, followed by visualizations of the results in Section 4.5.
ASR ASR"
I-RR(m; a) = 100 x
(13)
ACC
where ASR is the average attack success rate, ASR" is the maximum attack success rate, and ACC is the average accuracy without adversarial attacks.
6

---

ROME is Forged in Adversity: Robust Distilled Datasets via Information Bottleneck
Adversarial Robustness Under Black-Box Untargeted Attacks
Adversarial Robustness Under Black-Box Untargeted Attacks
DC
$1.78
94.44
91.73
92.76
92.72
93.07
99.94
DC
29.15
92.43
90.70
90.93
99.72
97.5
92:71
DSA
81.85
34.88
89.10
$9.95
89:67
90.26
99.56
Artack Models
MTT
$4.53
93.71
83.61
$1.69
91.71
92.13
99.91
Attack Models
MTT-
81.66
91,50
31.33
$8.85
88.02
88.58
99.55
92.5
DM
84.72
93.82
90,99
$7.20
91.92
92:42
99.99
DM
81.45
91.21
K7.62
35.35
87.30
88.39
99.47
60
IDM
83.76
93.16
89.61
90.68
$7.80
90.96
99.88
IDM
79.06
$9.03
94.42
$5.75
36.87
82.26
98.72
87.5
50
BACON
83.91
BACON
79.01
$9.49
K3 K4
$5.92
81,48
36.53
98,
ROME
$4.96
93.52
90.24
91.99
91.01
91.34
97,62
90.69
82.5
DC
DSA
MTT
DM
IDM
BACON
ROMB
DC
DSA
MTT
DM
IDM
BACON
ROME
Transfer Models
Transfer Models
(a)
(b)
Figure 3. Robustness heatmap of models trained using diverse dataset distillation methods with IPC-50 on CIFAR-10 under targeted and untargeted attacks. The vertical axis represents attacked models, and the horizontal axis shows models used for transfer attacks. Heatmap values represent 1-RR, with darker colors indicating higher I-RR and thus better robustness against adversarial attacks.
Remark 4.2. The I-RR metric is designed to assess adversar- ial robustness under a single attack E A, similar to the RR introduced by BEARD (Zhou et al., 2024a). When extended to multiple adversarial attacks {.A). the multi-adversary ver- sion, denoted I-RRM, provides a more comprehensive evaluation of model robustness. Similarly, I-CREI extends I-RR by incorporating the AE to jointly assess robustness and efficiency, and can be generalized to multi-adversary settings (I-CREIM). In the following experiments, I-RR and I-CREI denote I-RRM and I-CREIM by default, as all evaluations involve multiple adversarial attacks.
4.2. Adversarial Robustness Evaluation
White-box Robustness. Table shows the white-box ro- bustness of ROME and baseline methods under targeted and untargeted attacks on CIFAR-10 and CIFAR-100 1000 Best results are bold, second-best are underlined, and improve- ments over the second-best are highlighted in red. The pro- posed ROME outperforms all methods in BEARD across all adversarial attacks both datasets. Specifically,
CIFAR-
10, ROME surpasses DC by 8.93% in I-RR and 5. 1.11 1% in I-CREI under targeted attacks, and by 10.99% in I-RR and
4. 4.64% in I-CREI under untargeted attacks. On CIFAR-100,
ROME outperforms DC by 23.51 % in I-RR and 12.64% in I-CREI under targeted attacks, and by 13.91% % in I-RR and 5.17% in I-CREI under untargeted attacks. To further validate the results in Table 1, we record the accuracies of models trained on various distilled datasets under PGD attacks with perturbation budgets from 0 to 0.05, and plot the robustness curves following the protocol in (Dong et al., 2020; Liu et al., 2024). The complete curves and additional analysis are provided in Appendix C.2.
Implementation Details. All experiments use ConvNet (Gidaris & Komodakis, 2018). I-RR and AE measure the average top-1 accuracy over five runs under attacks and GPU time for attacks, respectively. The images-per-clant (IPC) values are set to 50, 10, and with models trained using stochastic gradient descent (SGD) with learning rate of 0.01, momentum of 0.9, and weight decay of 0.0005. Robust priors are generated with PGD, using perturbation budget of 253 under targeted attack settings, and a = 0.2 in Eq. 12, except in ablation studies. Adversarial robustness assessed under both targeted and untargeted attacks, with perturbation budget of 8 For black-box attacks, the Square attack uses random search with 5000 queries, and the SPSA attack evaluates gradients with 128 random samples per iteration, both using the same budget. The setup follows BEARD guidelines, with experiments conducted on NVIDIA RTX 3090 GPUs. More experimental settings, evaluation metrics, and implementation details are provided in Appendix C.I.
&
The significant improvement in adversarial robustness achieved by ROME can be attributed to the introduction of the conditional entropy bottleneck principle, which incor- porates robust priors into the distilled dataset. Additionally, the use of a pre-trained robust model further strengthens these priors. Regarding the CIFAR- 100 targeted attack re- sults, where I-RR exceeds 100%, we hypothesize that this phenomenon, which we refer to the Phenomenon" arises due to the application of the informa- tion bouteneck principle. During model training on ROME,
le| = 255
7

---

ROME is Forged in Adversity: Robust Distilled Datasets via Information Bottleneck
Table Comparison of adversarial rohustness (I-CREI, %) and training time (bours) of ROME and baseline dataset distillation methods on CIFAR-I0 10 (IPC-50) under targeted attacks. "Base" indicates standard distillation training, while "+Adv Train" refers to the additional time required for adversarial training to improve robustness. Best results, balancing robustness and efficiency, are highlighted in bold, and f denotes consistent results from "Base" "+AvTTrain", indicating no need for adversarial fine-tuning.
Table 4. Ablation studies on the Robust Pretrained Model (RPM) and Adversarial Perturbation (AP) under both targeted and untar- geted attacks, evaluated by 1-RR and I-CREI on the CIFAR-I0 10 dataset with IPC-50. Best results are highlighted in bold.
Targeted Attack
Untargeted Attack
Configuration
I-RR
I-CREI
I-RR
I-CREI
Baseline +RPM +AP
81.86%
55.26%
32.45%
29.29%
I-CREI
Training Time
84.50%
56.53%
34.89%
30.45%
Method
94.66%
61.67%
47.64%
36. 78%78
Base
+Adv Train
Base +AdvTrain
+RPM&AP
97.73%
63.23%
51.73%
38.95%
DC
58.21%
63.43%
0.425
1.088
DSA
57.22%
63.46%
0.437
1.103
MTT
56.24%
62.44%
0.444
perfect robustness, with performance exceeding 99.79% for 1.088
DM
55.89%
63.21%
0.452
1.109
targeted and query- attacks, and shows up to 15.2% improvement in untargeted query-based attacks compared to the second-best DD method. Figure 3 presents robust- ness heatmap illustrating the performance of ROME and baseline methods against the second transfer-based attack. Darker colors indicate higher robustness, while lighter col- ors indicate lower robustness. ROME achieves 97. 97.62% the best performance under targeted attacks, and 50.36%, the best performance under untargeted attacks. These results show that ROME performs strongly in both white-box and black-box attack scenarios, indicating that the robustness improvement is not due to obfuscated gradients. The con- sistent performance across different attack types highlights ROME's ability to generalize well, ensuring stability and effectiveness in various adversarial settings.
IDM
55.11%
63.11%
0.414
1.055
BACON
56.82%
62.68%
0.442
1.101
ROME
63.32%
63.32% 1
0.418
0.418 1
both the original dataset information and non-robust fea- tures, which serve as robust priors, are effectively utilized. In the distillation process, ROME compresses substantial image information from the original dataset, leading to a higher proportion of non-robust features in the compressed dataset. Consequently, models trained with ROME may outperform clean models, exhibiting higher accuracy under adversarial attacks than in the absence of such attacks. This Phenomenon" explains why I-RR ex- ceeds 100%, occurring exclusively under targeted attacks due to robust priors generated via targeted PGD.
4.3. Training Efficiency Comparison with Adversarially Distilled Datasets
Black-box Robustness. To evaluate the black-box robust- ness of ROME and baseline methods, we conduct compar- ison experiments using distilled datasets with IPC-50 on CIFAR-10. We assess performance under both targeted and untargeted transfer-based and query-hased attacks. For transfer-based attacks, we perform two experiments: (1) ad- versarial examples generated from an adversarially trained model are transferred evaluate the robustness of models trained with different DD methods, shown in Table 2, and (2) adversarial examples generated from a model trained with a specific DD method are transferred to evaluate the robustness of models trained with other DD methods, as shown in Figure 3. Additional results are in Appendix C.2.
Table 3 presents a comparison of various DD methods in terms of adversarial robustness (measured by I-CREI un- der targeted attacks) and training time on CIFAR-10 with IPC-50. Among all methods, ROME achieves the highest I-CREI score (63.32% in the "Base" setting, surpassing both traditional and recent distillation approaches. Most baselines rely on adversarial finettnnig ("+AdvTrain") to improve robustness, often requiring significant additional training time. In contrast, ROME achieves strong robust- ness without adversarial retraining. Even in the "+AdvTrain" setting. ROME still outperforms the majority of methods
In terms of efficiency, ROME requires only 0.418 hours of training in the "Base" setting, slightly more than IDM (0.414 hours) but still among the fastest methods. In the "+AdyTrain" setting, ROME incurs no additional training time, as it does not require adversarial fine-tuning. In con- trast, methods like DM require 0.452 hours for base train- ing and 0.658 hours for adversarial training, totaling over 1.1 hours. This makes ROME the most efficient method overall, offering strong robustness with the shortest total training time. Identical values in both "Base" and "+Adv-
Table 2 shows the adversarial robustness measured by I-RR of ROME and baseline methods against the first transfer- based and query-based attacks. The best results between the baseline and proposed methods are highlighted in bold, while the second-best results are underlined. Improvements in metrics compared to the second-best results are high- lighted in red. ROME outperforms all baseline methods in both attack types. Specifically, ROME achieves near-
8

---

ROME is Forged in Adversity: Robust Distilled Datasets via Information Bottleneck
Impact of Hyperparameter a on Accuracy
4.5. Visualization
60
Figure 4(b) illustrates how varying the hyperparameter affects the generated images. As increases, the images become brighter and exhibit more figh-frequeney details, which may correspond to stronger adversarial features con- tributing to improved robustness. However, this increase in robustness comes with a decrease in clean accuracy (Fig- ure 4(a),, indicating a trade-off between robustness and standard performance. Additional visualizations for abla- tion studies and comparisons across different IPC settings with other dataset distillation methods are in Appendix C.5.
Accuracy (%)
50-
45
40
30
0.0
0.2
0.4 Hyperparameter a
0.6
0.8
1.0
5. Conclusion, Limitations, and Future Work
In this work, we introduce the information bottleneck prin- ciple into dataset distillation and propose ROME, a robust method for dataset distillation. ROME combines two key components: a performance-aliguee alligge term preserve accu- racy, and a abbutnnsss-lligned term that enhances adversar- ial robustness without compromising overall performance. To more effectively evaluate adversarial robustness, we in- troduce I-RR, a metric that balances attack success with model accuracy. Experiments on CIFAR-10 and CIFAR- 100 show that ROME consistently outperforms existing methods in both white-box and black-box settings, achiev- ing up to 38.19% higher I-RR than the worst baseline and 23.51 % higher than the second- best under white-bor box at- tacks, and 34.65% and I 15.2% improvements, respectively, under black-box attacks. Notably, these improvements are achieved without the need for adversarial training, signifi- cantly reducing training costs while maintaining robustness.
0.0 = 0.2
a = 0.4
a= 0.6 a= 0.8 a == 1.0 (b)
Figure 4. Ablation study of the hyperparameter a. (a) Displays the accuracy (y-axis) as function a (X-axis) for different values of a, and (b) shows the corresponding visualizations for these values.
Train" columns (marked by !) confirm no additional steps are needed. Further details on the comparison with mod- els trained on adversarial datasets and the corresponding training time are provided in Appendix C.3.
4.4. Ablation Studies
Impact of the Robust Prior. We incorporate the robust prior into the source dataset as å E X via Adversarial Pertur- bation (AP), as derived in Theorem 3.7, and further enhance it using a Robust Pretrained Model (RPM) f(-), as intro- duced in Theorem 3.6. To evaluate their contributions, we perform ablation studies on ROME under IPC-50 on CIFAR- 10. As shown in Table 4, the combination of RPM and AP achieves the highest adversarial robustness, outperforming either component used in isolation. Further ablation results on the robust prior are presented in Appendix C.4, with corresponding visualizations in Appendix C.5.
Limitations and Future Work. While ROME effectively distills robust datasets without adversarial retraining, its scalability is limited by the large search space for complex datasets like ImageNet, restricting its applicability to large- scale language and vision-language models. Future work will explore more efficient search algorithms and compres- sion methods scale ROME to complex tasks and datasets.
A Acknowledgements
This work was supported by the Alan Turing Institute (UK) through the project "Turing- Labs Singapore Collabo- ration' (SDCIP2)
Impact of the Hyperparameter a. Eq. 12, the hyperpa- rameter a balances the performance-alignee and robustness- aligned terms in the loss function. As shown in Figure 4(a), varying significantly affects model performance. Al- though the optimal accuracy on CIFAR-10 with IPC-50 is achieved at a = 0.2, and may vary for other datasets or IPC settings (e.g., CIFAR- 100; IPC-1 and IPC- models trained with a = 0.2 consistently demonstrate the strongest robustness against both targeted and untargeted attacks.
Impact Statement
This paper advances machine learning security, impacting critical areas like face recognition, autonomous driving, and healthcare. It mitigates adversarial risks, promoting safer Al in high-stakes domains and supporting robust, trustworthy AI focused on faimess, accountability, and transparency.

---

ROME is Forged in Adversity: Robust Distilled Datasets via Information Bottleneck
References
Hendrycks, D., Lee, K., and Mazeika, M. Using pre-training improve model robustness and uncertainty. In Inter- national Conference Machine Learning (ICML), pp. 2712-2721 2019.
Alemi, A. A., Fischer, I., Dillon, J. V., and Murphy, K. Deep variational information bottleneck. In International Conference on Learning Representations (ICLR), 2017.
Hinton, G. Distilling the knowledge in a neural network. 2015.
Andriushchenko, M., Croce, F., Flammarion, N., and Hein, M. Square attack: a query-efficient black-box adversarial attack via random search. In European Conference on Computer Vision (ECCV), pp. ,
arXiv preprint
Hu, Y.-C.-T., Kung, B.-H., Tan, D. S., Chen, J.-C., Hua, K.-L.,, and Cheng, W.-H. Naturalistic physical adversarial patch for object detectors. In IEEE/CVF International Conference on Computer Vision (ICCV), pp. 7848-8888877 2021.
Carlini, N. and Wagner, D. Towards evaluating the robust- ness of neural networks. In IEEE Symposium on Security and Privacy (S&P), pp. 39-57, 2017.
Ilyas, A., Santurkar, S., Tsipras, D., Engstrom, L., Tran, B., and Madry, A. Adversarial examples are not bugs, they are features. Advances in Neural Information Processing Systems (NeurlPS), 32, 2019.
Cazenavette, G., Wang. T., Torralba, A., Efros, A. A., and Zhu, J.-Y. Dataset distillation by matching training tra- jectories. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4750-4444444 4759047599220 2022.
Kirchenbauer, J., Geiping, J., Wen, Y., Katz, J., Miers, I., and Goldstein, T. A watermark for large language mod- els. In International Conference on Machine Learning (ICML), pp. 17061-17084, 2023.
Chang, Y., Wang, X., Wang, J., Wu, Y., Yang, I.., Zhu, K., Chen, H., Yi, X., Wang, C., Wang, Y., et al. A survey on evaluation of large language models. ACM Transactions on Intelligent Systems and Technology. 15(3):1 2024.
Krizhevsky, A. Learning multiple layers of features from tiny images. Master's thesis, University Tront, 2009.
Croce, F. and Hein, M. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free at- tacks. In International Conference on Machine Learning (ICML), pp. 2206-2216, 2020.
Kuang, H., Liu, H., Wu, Y., Satoh, S., and Ji, R. Improving adversarial robustness via information bottleneck distilla- tion. Advances in Neural Information Processing Systems (NeurIPS), 16:10096-100 2023.
Dong, Y., Fu, Q.-A., Yang, X., Pang, T., Su, H., Xiao, Z., and Zhu, J. Benchmarking adversarial robustness on image classification. In JEEE/CVF Conference on Computer Vision and Pattern Recognitio (CVPR), 321 331, 2020.
Lei, and Tao, comprehensive survey of dataset distillation. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 46(1):17-32, 2024.
Liu, C., Dong, Y., Xiang. W., Yang, X., Su, H., Zhu, J., Chen, Y., He, Y., Xue, H., and Zheng, S. A comprehen- sive study on robustness of image classification models: Benchmarking and rethinking. International Journal of Computer Vision (ICC). pp. 1-23, 2024.
Fischer, I. The conditional entropy bottleneck. Entropy, 22 (9)9999 2020.
Gidaris, S. and Komodakis, N. Dynamic few-shot visual learning without forgetting. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4367-4375, 2018.
Madry, A., Makelov, A., Schmidt, I.., Tsipras, D., and Vladu, A. Towards deep learning models resistant to adversarial attacks. In International Conference on Learn- ing Representations (ICLR), 2018
Goldblum, M., Fowl, I.., Feizi, S., and Goldstein, T. Adver- sarially robust distillation. In Innovative Applications of Artificial Intelligence Conference (AAAI), pp. 3996-4003, 2020.
Moosavi-Dezfoolli S.-M., Fawzi, A., and Frossard, P. Deep- fool: a simple and accurate method to fool deep neural networks. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2574-2583 2016.
Goodfellow, J., Shlens, J., and Szegedy, C. Explaining and harnessing adversarial examples. In International Conference on Learning Representations (ICLR), 2015.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning (ICML), pp. 8748-8763, 2021.
Gou, J., Yu, B., Maybank, S. J., and Tao, D. Knowledge distillation: A survey. International Journal of Computer Vision (IJCV), 18896))1789-88881 1819, 2021.
10

---

ROME is Forged in Adversity: Robust Distilled Datasets via Information Bottleneck
Szegedy, C., Zaremba, W., Sutskever, I,, Bruna, J., Erhan, D., Goodfellow, I. J., and Fergus, R. Intriguing proper- ties of neural networks. In International Conference Learning Representations (ICLR), 2014.
Xu, M., Zhang, T., Li, Z.., and Zhang, D. Infoat: Improv- ing adversarial training using the information bottleneck principle. IEEE Transactions Neural Networks and Learning Systems (TNNLS), 35(1):1255-122) 1264, 2022.
Thirunavukarasu, A. J., Ting, D. S.J., Elangovan, K., Gutier- rez, L., Tan, T. F., and Ting, D. S. W. Large language models in medicine. Nature Medicine, 1930):133-- 2023.
Xue, E., Li, Y., Liu, H., Shen, Y., and Wang, H. Towards adversarially robust dataset distillation by curvature regu- larization. arXiv preprint arXiv: 10045, 2024.
Yu, R., Liu, S., and Wang, X. Dataset distillation: A compre- hensive review. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2023.
Tishby, N. and 7aslavsky, N. Deep learning and the infor- mation bottleneck principle. In IEEE Information Theory Workshop (ITW). pp. 1-5, 2015.
Yuan, X., Hu, S., Ni, W., Wang, X., and J Jamalipour, A. Deep reinforcement learning-driven reconfigurabl intelligent surface-assisted radio surveillance with a fixed-wing uav. IEEE Transactions on Information Forensics and Security (TIFS), 2023.
Tishby, N., Pereira, F. C. N., and Bialek, W. The information bottleneck method. CoRR, physics/0004057, 2000.
Uesato, J., O'donoghue, B., Kohli, P., and Oord, A. Ad- versarial risk and the dangers of evaluating against weak attacks. In International Conference on Machine Learn- ing (ICML), pp. 5025-5034, 2018
Zhao, B. and Bilen, H. Dataset condensation with differen- tiable siamese augmentation. In International Conference on Machine Learning (ICML), pp. 12674-12685, 2021.
Vaswani, A. Attention is all you need. Advances in Neural Information Processing Systems (NeurIPS), 2017.
Zhao, B. and Bilen, H. Dataset condensation with distri- bution matching. In IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp. 6514-6525 2023.
Wang, T., Zhu, J.-Y., Torralba, A., and Efros, A. A. Dataset distillation. arXiv preprint 1rXiv:1811. 2018.
Wang, Y., Sarkar, E., Li, W., Maniatakos, M., and Jabari, S. E. Stop-and-go- Exploring backdoor attacks on deep reinforcement learning- based traffic congestion control systems. IEEE Transactions on Information Forensics and Security (TIFS), 16:4772-4787 2021a.
Zhao, B., Mopuri, K. R., and Bilen, H. Dataset condensation with gradient matching. In International Conference on Learning Representations (ICLR), 2021.
Zhao, G., Li, G., Qin, Y., and Yu, Y. Improved distribu- tion matching for dataset condensation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7856-7865, 2023.
Wang, Y. Chen, Z., Yang, D., Guo, P., Jiang, K., Zhang, W., and Qi, L. Out of thin air: Exploring data-free adversar- ial robustness distillation. In Innovative Applications of Artificial Intelligence Conference (AAAI), volume 38. pp. 5776-5788 5784, 2024.
Zhou, Z.,, Feng, W., Lyu, S., Cheng, G., Huang, X., and Zhao, Q. Beard: Benchmarking the adversarial robustness for dataset distillation. arXiv preprint arXiv:2411.09265, 2024a.
Wang. Z.. Jian, T.. Masoomi, A., Ioannidis, S., and Dy, J. Revisiting hilbert-schmidt information bottleneck for adversarial robustness. Advances in Neural Information Processing Systems (NeurIPS), 34:586-597 2021b.
Zhou, 7., Zhao, H., Cheng, G., Li, X., Lyu, S., Feng, W., and Zhao, Q. Bacon: Bayesian optimal conden- sation framework for dataset distillation. arXiv preprint arXiv:2406.00 01 / 12, 2024b.
Wei, X., Guo, Y., and Yu, J. Adversarial sticker: A stealthy attack method in the physical world. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 45(3):2711-2725 2022a.
Zhou, Z., Zhao, H., Liu, J., Zhang, Q. Geng, L., Lyu, S., and Feng, W. Mvpatch: More vivid patch for adversarial camouflaged attacks on object detectors in the physical world, arXiv preprint arXiv: 12.22.17333 17431, 2024c.
Wei, X., Guo, Y., Yu, J., and Zhang, B. Simultaneously optimizing perturbations and positions for black-box ad- versarial patch attacks. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2022b. 2022b.
Zi, B., Zhao, S., Ma, X., and Jiang, Y.-G. Revisiting ad- versarial robustness distillation: Robust soft labels make student better. In IEEF/CVF International Conference on Computer Vision (CVPR), pp. 16443-16452, 2021.
Wu, Y., Du J., Liu, P., Lin, Y., Cheng, W., and Xu, W. Dd-robustbench: An adversarial robustness benchmark for dataset distillation. arXiv preprint arXiv:2403.1332. 2024.

---

ROME is Forged in Adversity: Robust Distilled Datasets via Information Bottleneck
Appendix
This appendix presents supplementary details that further support the main paper, including proofs for theorems, background on dataset distillation, and experimental setup and results. The contents are organized as follows:
Appendix A contains proofs for all theorems and definitions presented in this paper.
Appendix provides additional background information and preliminary details on dataset distillation.
Appendix C provides implementation details of experiments and visualizations.
A. Proof
A.1. Proof of Variational Information Bottleneck (VIB)
A.1.1. VARIATIONAL LOwER BOUND OF VIB
We derive variational lower bound of [)); Z) by leveraging the non-negativity of IKL divergence. This bound allows us to approximate the mutual information using tractable variational distribution q(v/ (z).
I(): / plu. 2) log p(u.2) 2) dy dz p(u)p(z)
(14)
/ ply, 2) log p(ylz) p(y)
dy dz.
(15)
(L60012))((012) 20 + * log dy dz 2 / plulz) log ((vl2) dy dz.
(16)
(17)
I)V:2) 2 / plu, 2) log (ullz) ply) dy dz
(18)
/ p(u. log (3)12) dy dz / plu. 2) log p(y) dy dz
(19)
I p(u, () log ((vl=) dy dz / plu) plyg dy
(20)
/ p(u, =) log (()l)) dy dz + H()
(21)
2 / p(u, 2) log (()12) dy
(22)
/ (p(a,y,2) 2) dx] log ((vl2)) dy dz
(23)
= / (,(,,))(=1r;; y) dæ) log [(vمz) 2) dy dz
(24)
/ ()((())((= da] log <(ul=) dy dz
(25)
I p(a, )>> log ((vl=) 2)) da dy dz
(26)
E [log ((vl=))
(27)
12

---

ROME is Forged in Adversity: Robust Distilled Datasets via Information Bottleneck
A. 1.2. VARIATIONAL UPPER BOUND OF VIB
Similarly, we provide variational upper bound of (2;; 2) using a surrogate marginal q(z), which is critical for enforcing the information bottleneck constraint in practice.
1( / plu, 2) log p(x, 2)
dx dz
(28)
p(x))(z)
/ p(xz, 2) log
p(z[z)
dx dz.
(29)
p(2)
KT.()(2)(((2)) 20
(30)
+ 1
/ p(z) p(z) log (() dz.
(31)
1)x;2) < / p(a, log p(=|z) ((2) dr dz
(32)
ff (2(2,2,)) dyl log ((z)
p(z|x)
dx dz
(33)
/ (0(,,))((=l,, v) dyj log p(z(z)
da dz
(34)
((2)
/ (p(z,v) dyl log p(z[ar
dz dz
(35)
((2)
/ p(a, v)p(=/<)) log p(=[2) (=))
da dy dz
(36)
log p(=| ()]=
(37)
((2)
Combining the above variational lower bound on 7()); 2) and the variational upper bound on [(X; 2), we derive the following variational objective for the VIB framework:
- (0);2)-8((222) 2 Epta log ((vl())== - 33 log p(z[x)
(38)
{(2))
A.2. Proof of Theorem 3.2
In the ROME setting, there are a synthetic dataset X and a perturbed source dataset å. ROME aims to learn the feature representations of X, which is comparable to the representations of X. Additionally, we assume Z, & and y are independent given X, where denotes as IL {&,))) depends only on X' because of Markov chain X' + (). We can obtain the lower bound of 2)):
Theorem A.I. The variational lower bound of IVV; can be computed as follows:
T(): 2) 2 Ep(..s) [log {(v1=)]
(39)
13

---

ROME is Forged in Adversity: Robust Distilled Datasets via Information Bottleneck
Proof.
2)) = H(O) - - & - - -
(40)
= =-
(41)
= / ply, 2) log p(u.2) 2) dydz
(42)
p==)
[ p(v,:) log p(ul= z)ddzz
(43)
= I p(v,=) logp(ul=)) dydz
/ p(v, =) log (yll))ddz + / plu, log ((ul))
(44)
/ p(v.2) log a 2) dydz
KL (p(ol=))la((12)) dydz
(45)
2 / plu, 2) log q (lul))de dydd
(46)
E) [log [b]1))]
(47)
A.3. Proof of Theorem 3.3
Theorem A.2. I(X;; 2(8) can also be simplified as follows:
(48)
Proof.
1(2 = -1(2 + - -
(49)
(50)
(( + ( -H(0) - (H(X, H(()]
(51)
H(2))) + ()(, - H(X 2)]
(52)
(((2)))+ - - +((Z)I,)))
(53)
H(2 H)
(54)
1(x,8;2) =H(X, () H(2) & [H({)) H(2) - H(&,
(55)
(56)
- -H(X, (2)) 11x, x) H R)] - [[1((2) + H(3)
(57)
= -14,, 112)- -( H1S -HEE
(58)
(59)
Therefore, learning intermediate features Z of X' is equivalent to minimizing 2(x)
1(x:; Z|²) 1(x [ z) = =) 1(x; 2) 1(x; Z).
(60)
Theorem A.3. The variational upper bound of T(X: 2 can be calculated as follows:
log p(z[=)
T(x:; 2 < E ()(c))))()
(61)
9(2 ())
14

---

ROME is Forged in Adversity: Robust Distilled Datasets via Information Bottleneck
Proof.
2() - H(2) H( (H(2)-H(212)))
(62)
(63)
--H(Z)X) + H(Z
(64)
/ p(2,x)) log p(2,,) drdz p(z)
p(z, Bog 0(2,3t dōdz
(65)
p(2)
/ p(z,a) log (
p(z, 8) log Ogg(( (z) dddzz
(66)
/ p(2,z) (
(zz, 3) logp(= 2 ($) didz
(67)
+ / p(2,2) log (=|0))
/ p(2, 2) log diidz
(68)
/ p(2,x)) log ((2r))ddd
p(z, .) log g( (=) - f KIL [=(e[)))((())))
(69)
S / p(z,z) pog
/ p(z, .) log ((=
(70)
p(s)p(z(>) logg(sls)drdl / p()))(z)()) log (=1())dddd
(71)
= (r, p<l i) log p(=((, t)dodz p())p(=10) log {(=ld)dād=
(72)
a) log (( ddddidz / p(t) [p(x|()p(222z d) log g d(=l))
(73)
/ ((),))p(21)) it log p(: drddddd
/ ((xx))p(2 2) log a(2) dlddddd
(74)
/ p(a, (p((a)) log {(
p(=(+)
drdids
(75)
p(= p(2(x) x))
(p log (2)
(76)
A.4. Proof of Theorem 3.4
Theorem A.4. The variational lower bound of ROME can be computed as follows:
ROME
(77)
[ogg((l==) - 3 log p(2[z)
2 Bp(a, 2.8,0)
(78)
q(=(,))
Proof.
T()2)= = H() H()1)) x -H()2))
(79)
15

---

ROME is Forged in Adversity: Robust Distilled Datasets via Information Bottleneck
2)
(80)
f ply.2) log p(u,2) dydz p22)
(81)
/ p(v,2) log p(yll)ddyd
(82)
= / p(v.2) log p(g)( dyd=
/ plu, 2) log ((ll))ddz + / plu, 2) log log ((ll))dddz
(83)
= J p(u. 2) log (lll))ddddd -
/ IKL (p(or=)] dydz
(84)
= Ep(v.e) Dlgg [v]l))]
(85)
= - [o(2, & y, =)dxd) log (yl=))ddz
(86)
= J p(x, &, v)p(2 2, 2. v) log q ddddddddd
(87)
= Ep(e....w)maall..... [log [1)l)]]
(88)
I(x;21&) -
(89)
[1) - (())) ([())-1(Z)))
(90)
= --H(ZIX)+
+ H()
(91)
= / p(z,a) log p(2, x) drdz
p(z, 8)
p(z, 8) (2,,0loog log
död=
(92)
plz)
p()))
/ p(2,z))og (2
p(z, b) logp(= 2 ddidz
(93)
/ p(2,z) (lg(((2())ddd
p(z, .) log p(2 (8) ddid
(94)
/ p(2,0) log (2)3))
/
p(z, () log {==
(95)
p(z, 5) log ( dd)d= f KKL [[]]a)]l(aa]
= p(z,0) log
(96)
log.)))op((())))) log p() / p(z, log ((=]))(=(d)))
(97)
/ p(z)p(2|0) log d()
= p(z)p(2(x) (tpp(2//))ogp(paap)
(98)
((z, p(212, 2) log p0gp(222 ddddd p())p(2()) (ogg)
(99)
((x,))p(21)) lx, a) log p(2 (xdddddd / p(() [p(a(())((1 dx log g( dd)d
(100)
/ p(a,, (((,,))(()x, +) log al (2 (dddddd
p(a, #) log (o(((()))ddddd
(101)
p(a, p(a,,))(z1o,,5 8) log p(=[z) dxdidz
(102)
{((())
/ (l u)dy) p(2|0, 2, w) log
p(2 (z)xx) dædidz
(103)
q(= (+)
/ p(a, g)p(2))) log
p(z x
dxdddydz
(104)
(2(Ø)
log p(=
(105)
((1()t)
16

---

ROME is Forged in Adversity: Robust Distilled Datasets via Information Bottleneck
Therefore, the lower bound of ROME can be defined as:
ROME -
(106)
p(z[2) (=
2 B x, logg(v|>) 3 log
(107)
Remark A.5. By leveraging variational methods, we aim to refine our understanding and utilization of mutual information in the context of ROME. The variational lower bound Ep(g.=) [log [(y[2)] as detailed in Theorem 3.2, serves as a fundamental measure. It optimizes the expectation of log ((ylz) with respect to the distribution q(ylz), where ((rl2) acts as a variational approximation to the true conditional distribution p(y[z). This bound provides conservative estimate of I(V; 2), ensuring that g(3/22) effectively captures dependencies between output information y and hidden variables Z. Simultaneously. the
variational upper bound (p(x,.))(e1zz log q(=(a)
((=|=) elucidated in Theorem 3.3, offers upper bound for T(X; 2|2²).
quamtifies the expected log ratio between the true conditional distribution p(= (z) of latent variables Z given the synthetic
dataset X' and the variational distribution q==(,), given the perturbed dataset X. These bounds help optimize variational distributions in ROME, improving the distillation of robust datasets while maintaining key information.
A.5. Proof of Theorem 3.6
Theorem A.6. The performance-aligned term can also be expressed follows:
[CE [ ff))]
(108)
where f(-) is pretrained model robust adversarial attacks, and (xx denotes its logits output for input æ. s the one-hot true label vector, and CE denotes cross-entropy.
Proof.
E Ep (ppp [log {(]|2)] = pp((,a [log (b]2))
(109)
Ep(r, (p(aa)) [log [b]l=)]
(110)
Ep(a,).)) [log (()l(())
(111)
[ log (()))())
(112)
= -- -Epaa.))
[CE))
(113)
To maximize the iefoormance-aliineee term, we can equivalently minimize the negative of this term. To facilitate optimization using gradient descent, formulate the term follows:
Leert_Aiig
[CE [y' [[)]]]
(114)
A.6. Proof of Theorem 3.7
Definition A.7 (Total Variation Distance), The total variation distance can be defined as follows:
TV(P.Q) EIP(s) - - Q(x))
(115)
Definition A.8 Balllback-eeibles Divergence). The Kullback-Leibler divergence can be defined as follows:
P(x)
(L((PII) Σ
P(x) log
(116)
(
17

---

ROME is Forged in Adversity: Robust Distilled Datasets via Information Bottleneck
Definition A.9 (Pinsker's Inequality). The Pinsker's inequality can be defined follows:
TV(P,Q) S 2
(117)
Theorem A.10. The robustness-aligned term can also be expressed as the following lower bound, derived by scaling Pinsker's inequality:
CRob_Alis Bp( [BeN [e(x)] - -B [e(
(118)
where and & are class-aligned sample sets (i.e., X' contains synthetic samples and X' perturbed original samples, both partitioned by the label y). p(r, 2, y) is the joint distribution, e(-) the embedding layer output, and || 1|2 denotes the squared Total Variation distance.
Proof.
log p<2[2)]
Ep(a 20)(sla)))
log q( [)
(119)
(2|())
Bp( pp(((()) log p(2(a)
(120)
g(=(a)
-pp( KIL [p( = [[]
(121)
> Bpla, (2TV² [=(=)z)) [=[[a]
(122)
Pinsker's incquality
Ep( |B (Bxwx
[e(x)] E [e(
(123)
Therefore, the abbustness-aligned term can be also computed as:
Coo Alig = Ep(r,a.)) Exxx [e(x)] - Enn [e.
(124)
B. Extended Background
B.1. Dataset Distillation
Let T [1x,, 91)1 denote the real dataset, where each 2; E X Rdd is d-dimensional input and y: Y CO.... 13 is the corresponding class label. The distilled (synthetic) dataset is represented as S = { 3i } where T; E RR, D: E V1 and ||| < [T). In this work, we adopt the notation X' = S for the synthetic dataset, and use X' to denote the adversarially perturbed version of T. This deviates from the conventional dataset distillation literature, where X' typically refers to the clean source data. The modified definitions are introduced to align with our focus on adversarial robustness, an aspect that has been underexplored DD research. Notations in cited methods are retained for clarity and consistency.
The goal of dataset distillation to construct compact dataset S that captures the essential information in such that training a model S yields similar performance training on T. Let de x + y denote model parameterized by 0, and let L be a loss function (e.g., cross-entropy). The distillation objective is formulated as:
E B v) E. (Z)o 0)).
(125)
B.I.I. META-LEARNING BASED METHOD
predominant approach to dataset distillation formulates the task as a bi-level meta-optimization problem (Wang et al., 2018). The inner optimization minimizes the training loss on the synthetic dataset to obtain model parameters 0(S), while the outer optimization updates minimize the generalization loss on the original dataset T:
S* arg min Ea-e
s.t. (*) arg min C(S,0).
(126)
S
0
where denotes the task specifc loss function. Despite its effectiveness, this bi-level framework introduces substantial computational overhead, spurring research into more tractable alternatives.
18

---

ROME is Forged in Adversity: Robust Distilled Datasets via Information Bottleneck
1.1.2. GRADIENT MATCHING BASED METHODS
To mitigate the memory and time overhead introduced by gradient unrolling in meta-learning, Zhao et al. (2021) propose gradient matching approach. This method optimizes the synthetic data S by aligning its gradient with that of the original data T using cosine similarity. At each iteration, class wise mini-batches Se C. S and Te C. T are sampled, where e C indexes the classes, and updates are performed independently for each class. The objective is given by:
TC-1 Σ
min Eage 8
(127)
where :): denotes the cosine distance between gradients. While this formulation avoids full backpropagation through time, it remains computationally intensive due to repeated gradient computations and per-class updates.
1.1.3. DISTRIBUTION MATCHING BASED METHODS
To further reduce the computational burden of bi-level optimization, Zhao & Bilen (2023) introduce a distribution matching approach based on the Maximum Mean Discrepaney (MMD) metric. This method minimizes the Euclidean distance between the feature distributions of synthetic data S and real data T under a given model øs. The objective is defined as:
ISI
1 Σ de (2)))) IT
min min Ee
||
Σ
(o().)
(128)
|S|
where (z) and de (z) denote the feature embeddings of synthetic and real samples, respectively. By aligning the feature distributions, this approach bypasses gradient- -ased unrolling and offers improved efficiency.
B.2. Adversarial Robustness Distillation
B.2.1. ADVERSARIAL TRAINING
Deep neural networks have been found to be vulnerable to adversarial examples, which are generated by adding imperceptible adversarial perturbations (Szegedy al., 2014). Goodfellow al. (2015) proposed the Fast Gradient Sign Method (FGSM) adversarial attack method, which generates adversarial perturbation in the direction of the gradient of the loss function. Madry et al. (2018) presented a Projected Gradient Decent (PGD) adversarial attack method, which is multi-step optimal first-order attack method. Following these works, series of works have been proposed to improve the performance of the adversarial attack, such as DeepFool (Moosavi-Dezfooll et al., 2016), C&W (Carlini & Wagner, 2017), AutoAttack (Croce & Hein, 2020). To overcome the effect of adversarial examples on deep neural networks, Madry et al. (2018) proposed adversarial training by adopting multistep adversarial attack, i.e., PGD, to generate adversarial examples to improve adversarial robustness. adversarial training has been demonstrated as one of the most effective approaches improve adversarial robustness, which can be formulated mini-max optimization problem as follows:
arg min 1 (D max [(fa(= + +c) v)
s.t.
||||p S c,
(129)
where fe(-) represents DNN with wights 0, D is a data distribution with the benign sample x and the correponding ground truth label y. the generated adversarial perturbation, e the maximum perturbation strength and L(fo(x + c), represents the cross entropy loss function. The adversarial perturbation can be defined as:
G = II [G-1 + (. +-1))
(130)
[-c,c]
where e is the adversarial perturbation for iterations and a is the step size.
B.2.2. INFORMATION BOTTLENECK
Information Bottleneck (IB) (Tishby et al., 2000; Alemi et al., 2017) aims to encode the maximally informative representation Z for target labels y while restrain input informmation X. The objective function of IB can be dedined as:
R max I ()) Z) - (1;; Z).
(131)
where 7 denotes the mutual Information and B3 controls the trade-off between (V:3) and T(X;: 2).
19

---

ROME is Forged in Adversity: Robust Distilled Datasets via Information Bottleneck
C. Experiment
C.1. Experiment Settings
Datasets. In our experiments, we use two standard image classification datasets: CIFAR-I0 (Krizhevsky, 2009) and CIFAR-100 (Krizhevsky, 2009). Each dataset has been selected for its relevance and complexity in the context of dataset distillation and adversarial robustness evaluation.
C-1 (Krizhevsky, 2009) contains 60,000 32 x 32 color images in 10 classes, with 50,000 images for training and 10,000 for testing. The images are preprocessed to normalize pixel values to the range [0, 1].
R-F0R-100 (Krizhevsky, 2009) Similar to CIFAR-10 but with 100 classes, this dataset contains 60,000 images, divided into 50,000 training and 10,000 testing images. Each image is resized to 32 x 32 pixels and normalized.
Dataset Distillation Methods. We six representative dataset distillation methods as baselines: DC (Zhao al., 2021), DSA (Zhao & Bilen, 2021), DM (Zhao & Bilen, 2023), MTT (Cazenavettee al., 2022), IDM (Zhao et al., 2023), and BACON (Zhou et al., 2024b). These methods span several common categories in recent dataset distillation research, including gradient matching methods (Zhao et al., 2021; Zhao & Bilen, 2021), distribution matching methods (Zhao & Bilen, 2023; Zhao et al., 2023; Zhou et al., 2024b), and trajectory matching methods (Cazenavette et al., 2022).
DC (Zhao et al., 2021) formulates dataset distillation as a bi-level optimization problem, focusing on matching the gradients of deep neural networks trained on the original dataset T and the synthetic dataset S.
DSA (Zhao & Bilen, 2021) improves distillation by incorporating data augmentation, enabling the generation of more informative synthetic images, which enhances the performance of models trained with these augmentations.
DM (Zhao & Bilen, 2023) offers a straightforward yet impactful method for generating condensed images by aligning the feature distributions of synthetic images S with those of the original training set T across multiple sampled embedding spaces.
MTT (Cazenavette et al., 2022) introduces trajectory matching as a distillation technique, condensing large datasets into smaller ones by aligning the training trajectories of models trained on both the synthetic S and original T datasets.
IDM (Zhao et al., 2023) proposes novel dataset condensation approach based distribution matching, which proves to be both efficient and promising for dataset distillation tasks.
BACON (Zhou et al., 2024b) leverages Bayesian framework for dataset distillation, formulating as risk minimization to substantially improve performance and efficiency.
Adversarial Attack Methods. All attacks are implemented using the BEARD framework (Zhou et al., 2024a), which provides comprehensive suite state-of-the-an adversarial attack methods, including both white-box and black-box attacks. To ensure fair comparisons, consistent parameters are applied across all models. The attack library includes FGSM (Goodfellow al., 2015), PGD (Madry et al., 2018), C&W (Carlini & Wagner, 2017), DeepFool (Moosavi-Dezfooli et al., 2016), and AutoAttaac (Croce & Hein, 2020) for white-box attacks, as well as Square (Andriushchenko et al., 2020) and SPSA (Uesato al., 2018) for black-box attacks. During evaluation, adversarial perturbations are applied to assess the robustness of distilled datasets generated by various methods. Both targeted and non- non-targeted attacks are conducted to comprehensively evaluate adversarial robustness. For consistency, all trained models are tested under the same parameters, with the perturbation budget set to le 255 for all methods except DeepFool and C&W.
FGSM (Goodfellow et al., 2015) generates adversarial examples by perturbing the input in the direction of the gradient of the loss function, with perturbation size set to 8/255.
PGD (Madry et al., 2018) extends FGSM by applying iterative steps to create adversarial examples. The perturbation budget and step size adjusted for each dataset to enhance attack strength.
C&W (Carlini& & Wagner, 2017) focuses on optimizing adversarial examples to minimize perturbation while ensuring misclassification, providing a robust evaluation of model robustness.
20

---

ROME is Forged in Adversity: Robust Distilled Datasets via Information Bottleneck
DeepFool (Moosavi-Dezfooli et al., 2016) estimates the minimal perturbation required induce misclassification, offering insights into the model's 's sensitivity adversarial changes.
AutoAttack (Croce Hein, 2020) combines multiple strong attacks to provide a comprehensive evaluation of model robustness, ensuring thorough assessment of adversarial robustness.
Square (Andriushchenko et al., 2020) utilizes a random search strategy without relying on gradient approximation. is configured with multiple queries and a fixed perturbation budget.
SPSA (Uesato et al., 2018) estimates gradients by drawing random samples and performs multiple iterations, each constrained by fixed perturbation budget.
Evaluation Metric.
Definition C.1 (Average Attack Success Rate). Let m .M represent neural network model and a E A an adversarial attack function. The average attack success rate is defined to evaluate the average-case attack scenario as follows:
ASR A
(132)
Definition C.2 (Maximum Attack Success Rate). Given neural network model m E M and an adversarial attack function E A. The maximum attack success rate is defined to capture the worst-case attack scenario as follows:
ASR
max ASR(m;a). mEM,cAA
(133)
Definition C.3 (Average Accuracy). Let E M represent neural network model and E A an adversarial attack function. The average accuracy is defined to normalize both the average- and worst-case attack conditions as follows:
ACC = B
(134)
Definition C.4 (Improved Robustness Ratio (I-RR)). Given a neural network model m E .M and an adversarial attack function a E A, the improved robustness ratio is defined as:
ASR ASR"
I-RR(m; a) = 100 x
(135)
ACC
where ASR is the average attack success rate, ASR" is the maximum attack success rate, and ACC is the average accuracy without adversarial attacks.
Remark C.5. The I-RR metric is designed to assess adversarial robustness under single attack E A. similar to the RR introduced by BEARD (Zhou et al., 2024a). When extended to multiple adversarial attacks (AA), the multi-adversary version, denoted as I-RRM, provides more comprehensive evaluation of model robustness. Similarly, I-CREI extends I-RR by incorporating the AE to jointly assess robustness and efficiency, and can be generalized to multi-adversary settings (I-CREIM). In the following experiments, I-RR and I-CREI denote I-RRM and I-CREIM by default, as all evaluations involve multiple adversarial attacks.
Motivation for Introducing I-RR and I-CREL The Robustness Ratio (RR) (Zhou et al., 2024a) evaluates adversarial robustness by calculating the relative difference between the average (ASR) and worst-case (ASR*) attack success rates, aiming quantify the discrepancy between these values as follows:
1
ASR
RR(m; a) 100
(136)
ASR'
However, RR may overestimate robustness when dominant attack disproportionately inflates ASR". The Improved Robustness Ratio (I-RR) addresses this issue by ensuring that both the average (ASR) and worst-case (ASR" attack success rates are minimized, while maintaining high clean accuracy (ACC). Furthermore, I-CREI extends the Comprehensive Robustness- Efffciees- Index (CREI) by replacing RR with I-RR, providing more reliable assessment that balances both attack effectiveness and computational efficiency. Together, I-RR and I-CREI establish a more robust and fair evaluation framework, particularly in scenarios where adversarial attacks vary widely in strength or dominate specific cases.
21

---

ROME is Forged in Adversity: Robust Distilled Datasets via Information Bottleneck
Black-box Evaluations. We conduct two types of transfer-based attacks and one query-based attack evaluate the robustness of DD methods in black-box settings:
Transfer-based Attack from Adversarially Trained Models: Adversarial examples are generated from an adver- sarially trained model and transferred to evaluate the robustness of models trained with various DD methods. This simulates an attacker who cannot access the distilled model but can use adversarial examples from a different model. Results are shown in Table 2.
Transfer- basee Attack across Distilled Models: Adversarial examples are generated from model trained with a specific DD method and transferred to models trained with other DD methods. This tests the transferability of adversarial perturbations and investigates the impact of DD method choice on robustness. Results are shown in Figure 4.
Query-based Attack: Adversarial examples are generated using Square (Andriushchenko et al., 2020) and SPSA (Uesato et al., 2018) through multiple rounds of querying, assessing model robustness under query-based attacks. Results are shown in Table 2.
Implementation Details. We use ConvNet architecture (Gidaris & Komodakis, 2018) for the dataset distillation experiments. The performance of the synthetic datasets is evaluated by I-RR and AE, representing the average top-1 accuracy over five runs and the average GPU time required for adversarial attacks on the validation set, respectively. The experiments are conducted with different IPC values, specifically IPC-50, IPC-10, and IPC-1. Models are trained using the SGD optimizer with a learning rate of 0.01, momentum of 0.9, and weight decay of 0.0005. All robust priors are generated using PGD with perturbation budget of 255 under targeted attack settings. In our experiments, we set in Eq. 12 to 0.2, except in the ablation studies. Both targeted and untargeted attacks are used to evaluate adversarial robustness. To & is applied to maintain consistency, all models are trained with identical parameters, and a perturbation budget of |c| = 255
all methods except DeepFool (Moosavi-Dezfoolli et al., 2016) and C&W (Carlini & Wagner, 2017). For black-box attacks, the Square attack employs random search without gradient approximation, configured with a maximum of 5000 queries and a perturbation budget of 8 Similarly, the SPSA attack conducts full gradient evaluations by drawing 128 random 255
samples per iteration, utilizing a perturbation budget of 255 in a single iteration. For fair comparisons in generalization, we incorporate DSA (Zhao& & Bilen, 2021) data augmentation during the evaluation model training process. Other parameters are set in accordance with those used in BACON (Zhou et al., 2024b), ensuring consistency across experiments. The overall experimental setup follows the guidelines outlined in BEARD (Zhou et al.. 2024a). All experiments, including synthetic dataset generation and model training, are conducted on NVIDIA RTX 3090 GPU clusters.
C.2. Adversarial Robustness Evaluation
White-box Robustness. The robustness curves are shown in Figure 5. We evaluate the adversarial robustness of models trained on distilled datasets with different IPC settings using PGD attacks under both targeted and untargeted settings, across various perturbation budgets. As the perturbation budget increases, the adversarial robustness of all models decreases, with the rate of decline serving as an indicator of robustness. Notably, the model trained with ROME exhibits the slowest decrease in accuracy, demonstrating superior robustness compared to other methods. Furthermore, the robustness curves under targeted attacks decline more gradually than those under untargeted attacks, indicating that ROME is particularly more robust against targeted PGD attacks. This enhanced robustness aligns with the incorporation of targeted PGD as robustness prior during training, which likely contributes to its improved defense against such attacks.
Black-box Robustness. Table 5 reports the I-RR of ROME and baseline methods under transfer- based black-box attacks on CIFAR-10, 10, evaluated at IPC levels 1, 10, 50, and the aggregated metric M. While robustness generally decreases with increasing IPC for all methods and is higher under targeted than untargeted attacks, ROME consistently achieves the highest robustness with the smallest variation across IPC settings and attack types. This demonstrates ROME's superior and stable adversarial robustness across different dataset distillation scales and attack scenarios.
C.3. Additional Results on Adversarially Distilled Dataset Training Efficiency
CREI Comparison and Robustness Analysis. We compare ROME against adversarially distilled datasets using the BEARD benchmark and I- I-CREI as a unified robustness metric. As summarized in Table 6, ROME delivers strong adversarial robustness under targeted attacks across different dataset compression levels (IPC-1, 10, 50) and their aggregation (denoted
22

---

ROME is Forged in Adversity: Robust Distilled Datasets via Information Bottleneck
Targeted Attacks (IPC-1)
Targeted Attacks (IPC-10)
Targeted Attacks (IPC-50)
1.0
1.0
DC
IDM
DC
10M
DC
DDM
DSA
BACON
DSA
BACON
DSA
BACON
0.8
MTT
ROME
0.8
MTT
ROME
0.8
MTT
ROME
DM
DM
DM
Accuracy
0.6
0.6
Accuracy
0.66
0.4
0.4
0.2
0.2
0.2-
0.0
0.00
0.01
0.02
0.03
0.04
0.05
0.00
0.01
0.02
0.03
0.04
0.05
0.00
0.01
0.02
0.03
0.04
0.05
Perturhation Budget
Perturbation Budget
Perturhation Budget
(a)
(b)
Untargeted A ttack (IPC-1)
Untargeted Attack (IPC-10)
Untargeted Athack (IPC-50)
1.0
1.0
1.0
DC
IDM
DC
IDM
DC
IDM
DSA
BACON
DSA
BACON
DSA
BACON
MTT
ROME
MFT
ROME
MIT
ROME
DM
DM
DM
Accuracy
0.6
Accuracy
0.6
Accuracy
0.6
0.4
0.2
0.2
0.2
0.0
0.0
0.0
0.00
0.01
0.02
0.03
0.04
0.05
0.00
0.01
0.02
0.03
0.04
0.05
0.00
0.01
0.02
0.03
0.04
0.05
Perturbation Budget
Perturbation Budget
Perturbation Budget
(d)
(f)
Figure 5. Robustness curves of models trained on distilled CIFAR-10 datasets with varying IPC settings under white-hox attacks. Subfigures (a), (b), and (c) present the results for targeted attacks with IPC-1, IPC-10, and IPC-50, respectively, while (d), (e), and (f) show the corresponding results for untargeted attacks with the same IPC settings.
Table Comparison of I-RR across IPC settings 1, 10, 50, and M (a unitied metric aggregating these settings) for various DD methods under black-box attacks on CIFAR-10 10. Best results are highlighted in bold.
Targeted Attack
Untargeted Attack
Method
IPC-1
IPC-10
IPC-50
IPC-M
IPC-1
IPC-10
IPC-50
IPC-M
DC 99.52%
92.54%
85.84%
90.67%
99.28%
90.03%
83.97%
89.31%
DSA 98.65%
93.81%
94.09%
90.92%
98.30%
91.11%
92.31%
89.13%
MTT
95.31%
93.40%
91.40%
89.80%
94.05%
90.63%
89.02%
87.52%
DM 99.15%
94.53%
92.22%
90.08%
99.12%
91.02%
90.36%
88.45%
IDM
93.86%
92.12%
92.17%
89.91 %
91.88%
88.99%
89.22%
86.97%
BACON
93.80%
91.90%
92.46%
89.24%
91.18%
88.46%
89.25%
85.66%
ROME
99.98% %
99.98%
99.90 %
99.44%
99.60 %
98.94% %
98.44 %
97.34%
23

---

ROME is Forged in Adversity: Robust Distilled Datasets via Information Bottleneck
Table Comparison of adversarial robustness between ROME and adversarially distilled datasets using DDD methods under CIFAR-10
IPC-1, 1, IPC-10, 10, and IPC-50, evaluated using I-CREI against both targeted and untargeted attacks. Best results are highlighted in bold.
Targeted Attack
Untargeted Attack
Mrthod
IPC-1
IPC-10
IPC-50
IPC-M
IPC-1
IPC-10
IPC-50
IPC-M
DC
64.51 %
64.27%
63.96%
63.43%
56.44% %
51.52%
49.60%
48.08%
DSA
64.15%
64.09%
64.47%
63.46%
53.54%
50.09%
50.73%
47.19%
MTT
63.05%
63.90%
63.98%
62.44%
47.50%
49.58%
49.52%
44.39%
DM
64.34%
64.00%
64.54%
63.21%
53.52%
50.62%
51.89%
46.58%
IDM 64.47%
64.09%
64.29%
63.11%
51.19%
51.63%
51.99%
47.96%
BACON 63.84%
64.03%
64.22%
62.68%
47.37%
50.02%
51.32%
44.66%
ROME
64.46%
64.32%
63.66%
63.32%
51.24%
48.29%
43.67%
43.62%
Table Comparison of training time for ROME and adversarially distilled datasets using DD methads CIFAR-10 IPC-50. Best results are highlighted in bold. The training time is measured in hours, with total time including both standard training and adversarial retraining.
Method
Standard Training
Adversarial Training
Total
DC
0.425
0.663
1.088
DSA
0.437
0.666
1.103
MTT
0.444
0.644
1.088
DM
0.452
0.658
1.109
IDM
0.414
0.641
1.055
BACON
0.442
0.659
1.101
ROME
0.418
0.000
0.418
as IPC-M), achieving comparable or better results than prior methods such DC, DSA, MTT, DM, IDM, and BACON. Notably, unlike these baselines which require adversarial retraining reach competitive robustness, ROME attains its performance without any additional adversarial fine-tuning.
Under untargeted attacks, ROME exhibits a decline in robustness at higher IPCs (e.g., 43.67% at IPC-50), likely due to its reliance on robust priors generated via targeted PGD during distillation, which biases the model toward robustness against targeted threats. As further discussed in Section 4.2, this design can lead to an Over-Robustness Phenomenon, where improvements in targeted robustness do not fully generalize untargeted settings. Consequently, ROME demonstrates weaker robustness under untargeted attacks compared adversarially trained DD methods. Despite this limitation, ROME strikes favorable balance between robustness and training efficiency, making it lightweight and practical distillation approach. Future work will aim to design distillation objectives that improve robustness across various threat models.
Training Time Comparison and Efficiency Analysis. Table compares the training times for neural network models trained on distilled datasets generated by ROME and several other dataset distillation methods, including DC, DSA, MTT, DM, IDM, and BACON, under CIFAR- 10 IPC-50. ROME requires 0.418 hours for model training on the distilled dataset, which is slightly higher than IDM at 0.414 4 hours. However, ROME does not involve adversarial retraining. In contrast, methods such as DC, DSA, MTT, DM, and IDM necessitate additional adversarial retraining to achieve comparable robustness. For example, DM requires a total 1.109 hours, with 0.658 hours spent on adversarial retraining, more than doubling the training time compared standard training.
The primary computational cost for ROME stems from distilling adversarially robust features, which forces the model to learn from broader set of adversarial examples during the training process. While this adds complexity to the distillation process, ROME continues to perform excellently during standard training. Unlike methods such as DC, DSA, MTT, DM, and IDM, which require additional adversarial retraining to achieve comparable robustness, ROME eliminates this retraining step, significantly reducing computational overhead. This makes ROME a more efficient approach, as achieves strong adversarial robustness without the time-conuuiing retraining phase.
24

---

ROME is Forged in Adversity: Robust Distilled Datasets via Information Bottleneck
Table Ablation study of Robust Pretrained Model (RPM) and Adversarial Perturbation (AP) CIFAR-10 with IPC-50, evaluated under targeted and untargeted attacks using 1-RR, AE, and I-CREI. Best results are highlighted in bold.
Targeted Attack
Untargeted Attack
Baseline RPM (f(-)) AP (0)
I-RR
AE
I-CREI
I-RR
AE
I-CREI
81.86%
28.66%
53.26%
32.45%
26.13%
29.29%
84.50%
28.56%
56.53%
34.89%
26.00%
30.45%
94.66%
28.68%
61.67%
47.64%
25.92%
36.78%
V
97.73%
28.73%
63.23%
51.73%
26.16%
38.95%
C.4. Ablation Study
More details are provided in Table 8, which also includes the calculation of the Attack Efficiency Ratio (AE). The ablation studies confirm that ROME incorporates the best-performing results.
C.5. Visualization
Figure 6 shows the distilled datasets generated by ROME under varying robust prior configurations, highlighting their impact on the synthetic data distribution. Figures 7, 8, and 9 illustrate the distilled datasets generated by ROME, BACON, and IDM on CIFAR-I0 and CIFAR-I00 with IPC settings of 50, 10, and respectively.
a Baseline
(b) Baseline+RPM
(c) Baseline+AP
(d) Baseline+RPM+AP
Figure 6. Visualization of distilled datasets generated by ROME under different robust prior configurations, showcasing the impact of varying settings on the synthetic data distribution.
25

---

ROME is Forged in Adversity: Robust Distilled Datasets via Information Bottleneck
CIFAR-10
ROME
BACON
IDM
CIFAR- 100
ROME
BACON
IDM
Figure 7. Visualizations of distilled datascts generated by diverse DD methods with IPC-50 settings on the CIFAR-10 and CIFAR-100.
26

---

ROME is Forged in Adversity: Robust Distilled Datasets via Information Bottleneck
CIFAR-10
ROME
BACON
IDM
CIFAR-100
ROME
BACON
IDM
Figure Visualizations of distilled datasets generated by diverse methods with IPC-10 settings on CIFAR-10 10 and CIFAR-100.
CIFAR-10
ROME
BACON
IDM
CIFAR-100
ROME
BACON
IDM
Figure Visualizations of distilled datasets generated by diverse DD methods with IPC-1 settings the CIFAR-10 and CIFAR-100.
27


Reviews:
[{"review_id": "9J8gSiEkui", "reviewer": "Reviewer_dcbW", "summary": "This paper aims to improve adversarial robustness in dataset distillation. Inspired by the Information Bottleneck principle, this paper proposes a novel framework which is able to balance model performance and robustness. Various kind of experiments demonstrate the effectiveness of the proposed framework.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "- **Q1: It would be better if authors can further discuss the main purpose of using RR, CREI, I-RR and I-CREI in evaluating adversarial robustness separately, so that evaluation results could be more easier to understand.**\n- **R2:** \nThank you for your suggestion. We have clarified the purpose of using RR, CREI, I-RR, and I-CREI in evaluating adversarial robustness. RR, originally introduced in BEARD, measures robustness by considering both **the average and worst-case** attack success rates (ASR), aiming to keep both values low. However, when a single attack method is exceptionally strong, RR can become disproportionately high, leading to an **overestimation of robustness**. To address this issue, we propose I-RR, which refines RR by incorporating model accuracy (ACC) to **ensure that both the average and worst-case ASR remain low while maintaining high ACC**, making it a more balanced robustness metric. CREI is a comprehensive index that combines RR with Attack Efficiency (AE), and in ROME, we introduce I-CREI by replacing RR with I-RR, further improving robustness evaluation. The primary motivation for using I-RR and I-CREI in ROME is to overcome the limitations of RR and CREI, ensuring a more reliable and fair assessment of adversarial robustness, especially in cases where a single attack method dominates. These distinctions have been added to the manuscript for better clarity.\n\n---\n\n- **Q2: Does the proposed method require a longer time for executing, comparing with baseline methods? If so, which part consumes the most resources?** \n- **R2:** \nThank you for your question. ROME does introduce **additional computational overhead** compared to standard dataset distillation methods, mainly due to the robust prior generation. As shown in **Table 1**, ROME requires more training time than other methods. However, it does not necessitate full **adversarial retraining**, which is typically much more computationally expensive. The primary computational cost stems from the **adversarial perturbation** step used to generate the **robust priors**. Despite ROME having higher training time, its total time is lower compared to methods requiring adversarial retraining. We have included a detailed analysis of the computational costs compared to baseline methods in the supplementary material.\n\n    **Table 1: Comparison of Training Time for ROME and Adversarially Distilled Datasets Using DD Methods under CIFAR-10 IPC-50.**\n    | Method | Training Time (hrs) | +Adversarial Training (hrs) | Total (hrs) |\n    |--------|---------------------|-----------------------------|-------------|\n    | DC     | 1.009               | 2.019                       | 3.028       |\n    | DSA    | 0.898               | 1.796                       | 2.694       |\n    | MTT    | 0.882               | 1.764                       | 2.647       |\n    | DM     | 0.963               | 1.925                       | 2.888       |\n    | IDM    | 0.895               | 1.790                       | 2.685       |\n    | BACON  | 0.874               | 1.748                       | 2.622       |\n    | **ROME**   | **1.014**               | **1.014**                       | **2.027**       |\n\n---\n\n- **Q3: As shown in Figure 4, looks like the robustness-aligned term has a higher impact on model performance, comparing with performance-aligned term. Any thoughts on it?**\n- **R3:** \nThank you for your question. The observed difference in impact between the **robustness-aligned** and **performance-aligned terms** can be attributed to their distinct roles in model optimization. According to *Eq. 12*, $\\alpha$ controls the trade-off between these two terms, where a higher $\\alpha$ increases the weight of the robustness-aligned term, thereby enhancing the model's ability to generalize under adversarial perturbations. However, as shown in *Figure 4*, while emphasizing robustness improves performance under adversarial attacks, an excessive focus on robustness (i.e., a high $\\alpha$) can suppress **non-robust features** critical for classification, potentially **reducing overall accuracy**. This trade-off is further emphasized in Table 3, where the robustness-aligned term (AP) contributes more significantly to **improving robustness** than the performance-aligned term (RPM). We have updated the discussion to clarify this trade-off and its impact on model performance.", "claims_and_evidence": "In this work, authors' claim that they proposed a new method to enhance the adversarial robustness of data distillation. The experimental results demonstrate the superiority of the proposed method, which support the authors' claim sufficiently.", "methods_and_evaluation_criteria": "Authors utilize most common benchmark datasets and evaluation metrics in their experiments, hence the evaluation criteria makes sense for the problem studied in this paper.", "theoretical_claims": "After going through theoretical proofs provided in supplementary material, I don't find any issues.", "experimental_designs_or_analyses": "Yes, from my point of view, the experimental design is reasonable and complete.", "relation_to_broader_scientific_literature": "This work utilize the Information Bottleneck principle to improve the adversarial robustness in dataset distillation, which further illustrate that the Information Bottleneck principle could be an effective way to help defense adversarial attack.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Pros: 1. Authors proposed a novel idea to improve the adversarial robustness in dataset distillation; although the backbone technique Information Bottleneck principle has been widely utilized in similar settings, adopting it into dataset distillation is still very innovative；2. A series of experiments involving both targeted attack and untargeted attack, show that the proposed method is able to improve the adversarial robustness by a large margin, comparing with baseline methods.\n\nCons: 1. It would be better if authors can further discuss the main purpose of using RR, CREI, I-RR and I-CREI in evaluating adversarial robustness separately, so that evaluation results could be more easier to understand.", "other_comments_or_suggestions": "No.", "questions_for_authors": "1. Does the proposed method require a longer time for executing, comparing with baseline methods? If so, which part consumes the most resources? \n2. As shown in Figure 4, looks like the robustness-aligned term has a higher impact on model performance, comparing with performance-aligned term. Any thoughts on it?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TLoMVKEt96", "reviewer": "Reviewer_yn3H", "summary": "This paper proposes a new method -- ROME for dataset distillation that uses the Information Bottleneck principle to create small, synthetic datasets with improved resistance to adversarial attacks. Traditional adversarial training is slow and often reduces accuracy. ROME incorporates the Conditional Entropy Bottleneck into the distillation process. It optimizes two parts: one that preserves accuracy by strengthening the link between latent features and true labels, and another that boosts robustness by weakening the link between the input and latent features when adversarial noise is added. Tests on CIFAR-10 and CIFAR-100 show that ROME outperforms previous methods under various attack scenarios.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "- **Q1: Curious about the performance of the proposed method on larger dataset.**\n- **R1:** \nThank you for your question. Scaling ROME to larger datasets like ImageNet requires significantly **more computational resources and time**, which was not feasible within the **limited rebuttal period** and **available resources**. Additionally, prominent dataset distillation baselines (e.g., DC, DSA, MTT, DM, IDM, BACON) have not been trained on larger datasets due to the computational challenges posed by **feature alignment strategies**. As the dataset size increases, the search space for representative features expands, requiring considerably **more computational power**. For instance, generating ROME on CIFAR-100 with IPC-50 settings takes 3-5 days in our lab, and ImageNet or its subsets, which are hundreds of times larger, would require significantly more time. Furthermore, methods like DM, IDM, and BACON have not publicly released results on large datasets, so a fair comparison would require re-running baselines, which is not feasible within the rebuttal period. Nevertheless, we believe ROME can perform well on larger datasets. As shown in *Table 1* of our paper, ROME demonstrates better performance on CIFAR-100 compared to CIFAR-10, suggesting that its robustness benefits could **extend to larger datasets**, though further investigation is needed. As discussed in the *Limitations and Future Work* section, we acknowledge this limitation and plan to explore more scalable approaches to enhance **adversarial robustness** on **larger datasets** in future work.\n\n---\n\n- **Q2: Can the IB-based robust distillation framework be extended to other network architectures, such as transformers or hybrid models?**\n- **R2:** \nThank you for your question. We believe the **IB-based robust distillation (ROME)** framework can extend beyond ConvNet-based architectures to models such as **transformers**. Currently, dataset distillation methods like **DC**, **DSA**, **MTT**, **DM**, **IDM**, and **BACON** are primarily designed for **ConvNets**. Since ROME enhances the adversarial robustness of distilled datasets, it has the potential to generalize to other architectures as dataset distillation evolves beyond ConvNets. Regarding **generalization**, ROME has demonstrated strong robustness against both **transfer-based** and **query-based black-box attacks**, as shown in *Table 2* and the *Figure 3 in the paper*. As discussed in the *Limitations and Future Work* section, we plan to explore its applicability to **more complex models**, such as transformer-based architectures, particularly for tasks like **adversarially robust vision-language learning**. Additionally, we aim to assess adversarial robustness across **diverse architectures** to validate ROME's broader effectiveness.\n\n---\n\n- **Q3: Do you have any quantitative comparisons (e.g., specific training time reductions) but rather emphasizes the conceptual benefit of lower computational demands?**\n- **R3:** \nThank you for your question. While we emphasize the conceptual advantage of ROME, which inherently provides adversarial robustness without the need for **additional adversarial training**, we also present quantitative comparisons. Methods like DC, DSA, MTT, DM, IDM, and BACON require **adversarial retraining** to achieve similar robustness, which typically doubles the training cost due to the additional retraining on adversarial examples. In contrast, ROME eliminates the need for repeated adversarial training by **integrating robust priors directly into the distillation process**, thus significantly reducing computational overhead. Although ROME requires more training time, its total time is lower than methods that rely on adversarial retraining. We have included these quantitative comparisons in Table 1 and provided further experimental results in the revised version to highlight the training time differences.\n\n    **Table 1: Comparison of Training Time for ROME and Adversarially Distilled Datasets Using DD Methods under CIFAR-10 IPC-50.**\n    | Method | Training Time (hrs) | +Adversarial Training (hrs) | Total (hrs) |\n    |--------|---------------------|-----------------------------|-------------|\n    | DC     | 1.009               | 2.019                       | 3.028       |\n    | DSA    | 0.898               | 1.796                       | 2.694       |\n    | MTT    | 0.882               | 1.764                       | 2.647       |\n    | DM     | 0.963               | 1.925                       | 2.888       |\n    | IDM    | 0.895               | 1.790                       | 2.685       |\n    | BACON  | 0.874               | 1.748                       | 2.622       |\n    | **ROME**   | **1.014**               | **1.014**                       | **2.027**       |", "claims_and_evidence": "Claim: ROME significantly improves adversarial robustness of distilled datasets without needing costly adversarial retraining.\nEvidence: The experimental results (Tables 1–3) consistently show improvements in I-RR and related metrics under both targeted and untargeted attacks. For example, on CIFAR-10, ROME achieves up to a 40% improvement in I-RR compared to baselines such as DC, DSA, and BACON.\n\nClaim: Incorporating the IB (and specifically the Conditional Entropy Bottleneck) into the distillation process yields a favorable balance between accuracy and robustness.\nEvidence: The paper presents both theoretical derivations (Theorems 3.2–3.7, with proofs in the appendix) and ablation studies that validate the effectiveness of the two loss components—the performance-aligned and robustness-aligned terms.", "methods_and_evaluation_criteria": "Method: ROME reframes dataset distillation using the IB principle by introducing a robust prior via adversarial perturbations. The method defines an objective function that combines: 1). A performance-aligned term ensuring that distilled data retain sufficient label-related information. 2).A robustness-aligned term that minimizes the discrepancy between the synthetic dataset and its adversarially perturbed counterpart, thereby pushing the model’s decision boundary away from potential adversarial examples.\n\nEvaluation: The authors evaluate robustness using both white-box and black-box attack scenarios (e.g., FGSM, PGD, C&W, Autoattack for white-box; transfer-based and query-based for black-box). They introduce the I-RR metric as a refined measure of robustness. Experiments are conducted on CIFAR-10 and CIFAR-100 using a ConvNet architecture, and ablation studies explore the impact of robust pretraining, adversarial perturbations.", "theoretical_claims": "The theoretical claims are supported by detailed proofs provided in the appendix. While the derivations are mathematically involved, they offer a solid foundation for the proposed objective and justify the design choices in ROME.", "experimental_designs_or_analyses": "The experimental setup is comprehensive. The authors compare ROME with several leading dataset distillation methods (DC, DSA, MTT, DM, IDM, BACON) under various attack settings on standard benchmarks (CIFAR-10 and CIFAR-100). \nAblation studies show the contributions of key components, such as the robust pretrained model (RPM) and adversarial perturbations (AP).", "relation_to_broader_scientific_literature": "By introducing an IB-based formulation into dataset distillation, ROME offers a new perspective on how to achieve both accuracy and robustness in compact datasets.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strengths:\n1. The visualization is clear and understandable.\n2. The experiments are solid and comprehensive.\n3.The theoretical proof is solid.\n\nWeaknesses:\n1. Need more discussion about dataset scaling.\n2. Need more discussion about different model architectures.\n3. Lack of experiments about computation cost.", "other_comments_or_suggestions": "See questions.", "questions_for_authors": "1. Curious about the performance of the proposed method on larger dataset.\n2. Can the IB-based robust distillation framework be extended to other network architectures, such as transformers or hybrid models?\n3. Do you have any quantitative comparisons (e.g., specific training time reductions) but rather emphasizes the conceptual benefit of lower computational demands?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uioiGzKJMl", "reviewer": "Reviewer_UEJd", "summary": "The authors proposed an adversarially robust distillation optimization framework for dataset distillation. They also provided the optimization method for this framework.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "- **Q1: The theoretical derivation has been checked, and no issues were found. However, there may be some minor notation misuse, such as the use of $\\mathbb{CE}[p(z|x)||q(y|z)]$ in the proof of Theorem 3.6.**\n- **R1:** \nThank you for your valuable feedback. We have carefully reviewed and revised the proof of Theorem 3.6 to correct the notation and ensure consistency. Additionally, we have provided more details in the proof to enhance clarity and readability.\n\n---\n\n- **Q2: Moreover, Theorem 3.7 is a lower-bound objective obtained by scaling an inequality, but this is not stated in the main text.**\n- **R2:** \nThank you for your suggestion. We have updated the main text to explicitly state that Theorem 3.7 is a lower-bound objective derived by scaling an inequality. This clarification have been included in the revised manuscript to ensure better readability and understanding of the theoretical derivation.\n\n---\n\n- **Q3: Under the dataset distillation framework, the attacker appears to be somewhat disadvantaged. The defender can carefully select features that are robust. Perhaps an adaptive attack should be considered during evaluation, assuming that the attacker is aware that the defender employs dataset distillation methods.**\n- **R3:** \nThank you for your suggestion. Our method is designed without prior knowledge of specific attacks, with the robust prior derived solely from **PGD ($\\frac{8}{255}$ perturbation budget)**. As mentioned in *Evaluation Attack (L278)*, we assess robustness using PGD and **six additional attack methods**: **FGSM**, **C&W**, **DeepFool**, **AutoAttack**, **Square**, and **SPSA**, none of which were used to construct ROME. This ensures a balanced information setting between the attacker and the defender. Notably, **AutoAttack**, a **representative adaptive attack**, ensures a rigorous evaluation. Our experiments follow a **white-box attack setting**, where attackers have full knowledge of the model parameters, giving them an advantage. As shown in Table 1, ROME consistently outperforms other methods in all robustness metrics. Finally, developing more effective adaptive attacks for dataset distillation remains an open challenge and a promising direction for future work.\n---\n\n- **Q4: The authors claim that the proposed method avoids the high computational cost of retraining but do not provide a detailed explanation.**\n- **R4:** \nThank you for your suggestion. Models trained on **ROME-distilled datasets** inherently exhibit adversarial robustness, unlike methods such as DC, DSA, MTT, DM, IDM, and BACON, which require **additional adversarial training**, effectively **doubling the training cost**. ROME mitigates this overhead by **embedding robust priors directly into the distillation process**, thus eliminating the need for retraining. The experimental results quantifying this difference are shown in **Table 1**. While ROME requires slightly more initial training time, its overall computational cost remains significantly lower than methods that involve adversarial retraining.\n\n    **Table 1: Comparison of Training Time for ROME and Adversarially Distilled Datasets Using DD Methods under CIFAR-10 IPC-50.**\n    | Method | Training Time (hrs) | +Adversarial Training (hrs) | Total (hrs) |\n    |-|-|-|-|\n    | DC | 1.009 | 2.019 | 3.028 |\n    | DSA | 0.898 | 1.796 | 2.694 |\n    | MTT | 0.882 | 1.764 | 2.647 |\n    | DM | 0.963  | 1.925 | 2.888 |\n    | IDM | 0.895 | 1.790 | 2.685 |\n    | BACON| 0.874 | 1.748 | 2.622|\n    | **ROME** | **1.014** | **1.014** | **2.027** |\n\n---\n\n- **Q5: The proposed framework does not discuss the sampling of $\\hat{x}$. Intuitively, the iterative training of dataset distillation and the sampling of $\\hat{x}$ resemble the process of training a model and sampling adversarial examples in standard adversarial training.**\n- **R5:** \nThank you for your comment. The sampling of $\\hat{x}$ in ROME is similar to generating adversarial examples in standard adversarial training, but there are two key differences. First, unlike adversarial training, where adversarial examples are generated in each iteration to update model weights, ROME **integrates adversarial perturbations directly into the dataset distillation process**, producing a distilled dataset that inherently possesses adversarial robustness, eliminating the need for repeated adversarial training and reducing computational overhead. Second, while adversarial training results in robust model weights that require retraining for new tasks, ROME does not have this requirement. This distinction is discussed in the revised manuscript.\n\n---\n\n- **Q6: Some typos. Line 153 misses I. Line 197 misdescribe joint distribution. Line 940 misses Expectation notation.**\n- **R6:** \nThank you for your review. We have made the following corrections: added the missing \"I\" in Line 153, revised the joint distribution in Line 197, and added the Expectation notation in Line 940. These changes have been included in the revised manuscript.", "claims_and_evidence": "yes", "methods_and_evaluation_criteria": "yes", "theoretical_claims": "The theoretical derivation has been checked, and no issues were found. However, there may be some minor notation misuse, such as the use of CE[p(z|x)||q(y|z)] in the proof of Theorem 3.6.\n\nMoreover, Theorem 3.7 is a lower-bound objective obtained by scaling an inequality, but this is not stated in the main text.", "experimental_designs_or_analyses": "Under the dataset distillation framework, the attacker appears to be somewhat disadvantaged. The defender can carefully select features that are robust. Perhaps an adaptive attack should be considered during evaluation, assuming that the attacker is aware that the defender employs dataset distillation methods.", "relation_to_broader_scientific_literature": "1. Conditional Entropy Bottleneck (CEB).\n2. Adversarial examples are not bugs, they are features.", "essential_references_not_discussed": "no", "other_strengths_and_weaknesses": "The authors claim that the proposed method avoids the high computational cost of retraining but do not provide a detailed explanation. The proposed framework does not discuss the sampling of \\hat{x}. Intuitively, the iterative training of dataset distillation and the sampling of \\hat{x} resemble the process of training a model and sampling adversarial examples in standard adversarial training.", "other_comments_or_suggestions": "Some typos. Line 153 misses I. Line 197 misdescribe joint distribution. Line 940 misses Expectation notation.", "questions_for_authors": "no", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "I07eia9Lal", "reviewer": "Reviewer_gfFq", "summary": "Dataset Distillation (DD) compresses large datasets into smaller synthetic subsets but remains vulnerable to adversarial attacks. To address this, the paper proposes ROME, a method leveraging the Information Bottleneck principle to enhance adversarial robustness by aligning feature distributions, demonstrating significant improvements in robustness metrics on CIFAR-10 and CIFAR-100.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "- **Q1: Theorem 3.6: The expression of equation (111) is confusing, what is this \"||\" means for CE loss? Also, how to derive from equation (111) to (112) is not very clear to me.**\n- **R1:** \nThank you for your feedback. We have updated the derivation in Theorem 3.6 to remove the \"||\" symbol in Equation (111) for clarity. The revised version provides a clearer derivation, leading to the cross-entropy loss.\n$$\n    \\begin{align}\n    \\mathbb{E} _ {p(x,\\hat{x},y)p(z|x,\\hat{x},y)} \\left[\\log q(y|z)\\right] \n    &= \\mathbb{E} _ {p(x,\\hat{x}, y)p(z|x)} \\left[ \\log q(y|z)\\right]\n    \\\\\\\\\n    &= \\mathbb{E} _ {p(x,\\hat{x}, y)}\\mathbb{E} _ {p(z|x)}\\left[ \\log q(y|z)\\right]\\\\\\\\\n    &= \\mathbb{E} _ {p(x,\\hat{x},y)} \\left[\\log q(y|e(x))\\right]\n    \\\\\\\\\n    &= - \\mathbb{E} _ {p(x,\\hat{x},y)} \\left[- y^t \\log q(y^t|e(x))\\right]\n    \\\\\\\\\n    &= - \\mathbb{E} _ {p(x,\\hat{x},y)} \\left[\\mathbb{CE}[y^t, f(x)]\\right].\n  \\end{align}\n$$\nThe derivation starts with the joint distribution $p(x, \\hat{x}, y)$ and $p(z | x, \\hat{x}, y)$, simplifying to an expectation over $p(x, \\hat{x}, y)$ and $p(z | x)$, as $z$ depends only on $x$. The term $p(z | x)$ is replaced by $q(y | z(x))$ since $z(x)$ is deterministically determined by $x$. The term $e(x)$ represents the latent embeddings from the model's embedding layer, which is equivalent to $z(x)$. The classifier $f(x)$ models $q(y^t | e(x))$, and $y^t$ is the one-hot encoding of the true label. The final result is the cross-entropy loss $\\mathbb{CE}[y^t, f(x)]$, which minimizes prediction error.\n\n---\n\n- **Q2: Theorem 3.7: It seems that the authors use the embedding to estimate the total variation distances. Why does this make sense in practice?**\n- **R2:** \nThank you for your question. The use of embeddings to estimate total variation distances in Theorem 3.7 aligns with feature alignment strategies in dataset distillation, helping capture robust features for improved adversarial robustness. This approach is **consistent with prior works like DM, IDM, and BACON**. The transition from $p(z|x)$ and $q(z|\\hat{x})$ to $e(x)$ and $e(\\hat{x})$ in Equations 121-122 reflects the use of embeddings to ensure robustness. This has been clarified in the revised manuscript.\n\n---\n\n- **Q3: What is IPC, which seems to not be defined in the paper?**\n- **R3:** \nThank you for your suggestion. We have added a definition of **Images Per Class (IPC)** in the revised manuscript to ensure clarity.\n\n---\n\n- **Q4: Are there any real-world applications of ROME?**\n- **R4:** \nThank you for your question. We are exploring ROME's potential in **real-world applications** like **autonomous driving**, **facial recognition**, and **edge computing**, where **robustness** and **efficiency** are crucial. As mentioned in *Lines 57-59 of the Introduction*, these fields need robust dataset distillation techniques. ROME can speed up training on compact, robust datasets, improving security without costly adversarial retraining. While still in early stages, we believe ROME has great potential to enhance safety and efficiency in these areas.\n\n---\n\n- **Q5: The RR metric sometimes has a value over 100%, which seems to be weird to me. A good metric should usually fall into [0,100]% range.**\n- **R5:** \nThank you for your observation. While I-RR values typically range from 0 to 100, ROME exhibits an **\"Over-Robustness Phenomenon\"**, where models trained with ROME achieve **higher accuracy under attack than in a clean setting**, causing I-RR values to exceed 100%. We propose that ROME’s information bottleneck framework enhances robustness by **amplifying non-robust features** under specific adversarial conditions like PGD. This is discussed in *Line 374 of the Adversarial Robustness Evaluation* section. A similar effect is observed in BEARD’s RR metric for CIFAR-100 with targeted white-box attacks. We plan to explore more effective metrics to better capture this phenomenon in future work.\n\n---\n\n- **Q6: Could the authors do the comparison with DD baselines by using an adversarial version of the original dataset?** \n- **R6:** \nThank you for your suggestion. We compared **ROME** with **adversarially distilled datasets** using the **BEARD benchmark** and applied **CREI** for a fair comparison. As shown in **Table 1**, ROME outperforms adversarially distilled methods in both **adversarial robustness** and **efficiency** under targeted and untargeted attacks. ROME achieves comparable robustness **without the need for retraining**, significantly **reducing computational costs**. We have clarified this and provided additional experimental details in the revised manuscript.\n\n    **Table 1: Comparison of Adversarial Robustness of ROME and Adversarially Distilled Datasets Using DD Methods under CIFAR-10 IPC-50 with CREI (%).**\n    |Attack Type|Full-size|DC|DSA|MTT|DM|IDM|BACON|ROME|\n    |-|-|-|-|-|-|-|-|-|\n    |Targeted Attack|50.54|52.30|55.56|50.96|57.21|54.67|56.18|**63.32**|\n    |Untargeted Attack|41.33|37.39|37.59|33.13|37.34|39.05|37.25|**43.62**|", "claims_and_evidence": "Yes, the claims are well supported by evidence. The authors use information bottleneck to enhance the adversarial robustness of the distilled datasets.", "methods_and_evaluation_criteria": "Yes, it mostly makes sense. The problem I got is: why not generate adversarial examples on the original dataset first, then use DD methods to distill the dataset? In this way, is ROME still outperforming others?", "theoretical_claims": "Yes, I checked all of them. I have questions on Theorem 3.6 and 3.7.\n\n- Theorem 3.6: The expression of equation (111) is confusing, what is this \"||\" means for CE loss? Also, how to derive from equation (111) to (112) is not very clear to me.\n\n- Theorem 3.7: It seems that the authors use the embedding to estimate the total variation distances. Why does this make sense in practice?", "experimental_designs_or_analyses": "Yes, the experimental design is generally sound and valid. However, I am not sure whether it is fully fair to compare ROME with DD methods without any adversarial example-generating procedures. I wonder whether it would be to compare ROME with adversarial distilled datasets of other DD methods for a fairer comparison.", "relation_to_broader_scientific_literature": "The key contribution of this paper would be improving the adversarial robustness of distilled datasets in an efficient way.", "essential_references_not_discussed": "References are discussed comprehensively.", "other_strengths_and_weaknesses": "- The paper is well-written, easy to follow, and well-motivated.\n- The results are comprehensive and show significant improvements.", "other_comments_or_suggestions": "- What is IPC, which seems to not be defined in the paper?\n- Are there any real-world applications of ROME?\n- The RR metric sometimes has a value over 100%, which seems to be weird to me. A good metric should usually fall into [0,100]% range.", "questions_for_authors": "- Could the authors do the comparison with DD baselines by using an adversarial version of the original dataset?\n- Could the authors explain more on the theorems 3.6 and  3.7? (See above)", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper presents a dataset distillation approach that, by incorporating a information bottleneck penalty, enhances adversarial robustness. The latter is validated both through theoretical guarantees as well as experiments.

Reviewers overall appreciated the contribution. Both the detailed performance analysis w.r.t. training time that was reported in the rebuttal, as well as the adversarial DD benchmarks should absolutely be added to the paper. Additional feedback by the reviewers (e.g., in the clarification around Eq. (111)) should also be incorporated.
Decision: Accept (poster)