PRISM: Diversifying Dataset Distillation by Decoupling Architectural Priors
Authors: Brian Bernhard Moser
Shalini Sarode
Federico Raue
Stanislav Frolov
Krzysztof Adamkiewicz
Arundhati Shanbhag
Joachim Folz
Tobias Christian Nauen
Andreas Dengel
Venue: Accepted by TMLR ()

Abstract: Dataset distillation (DD) promises compact yet faithful synthetic data, but existing approaches often inherit the inductive bias of a single teacher model. As dataset size increases, this bias drives generation toward overly smooth, homogeneous samples, reducing intra-class diversity and limiting generalization. We present PRISM (PRIors from diverse Source Models), a framework that disentangles architectural priors during synthesis. PRISM decouples the logit-matching and regularization objectives, supervising them with different teacher architectures: a primary model for logits and a stochastic subset for batch-normalization (BN) alignment. On ImageNet-1K, PRISM consistently and reproducibly outperforms single-teacher methods (e.g., SRe2L) and recent multi-teacher variants (e.g., G-VBSM) at low- and mid-IPC regimes. The generated data also show significantly richer intra-class diversity, as reflected by a notable drop in cosine similarity between features. We further analyze teacher selection strategies (pre- vs. intra-distillation) and introduce a scalable cross-class batch formation scheme for fast parallel synthesis. Code: https://github.com/Brian-Moser/prism

PRISM: Diversifying Dataset Distillation by Decoupling Architectural Priors
Brian Moser', Shalini S Federico Raue', Stanislav Frolov', Krzysztof Adamkiewiez!2, Arundhati Shanbhag Joachim Folz Tobias Nauen Andreas Dengel
'German Research Center for Artificial Intelligence (DFKI)
2RPTU Kaiserslautern-Lanaau
first. perst.lasse last@dfki.ddd
Abstract
Dataset distillation (DD) promises compact yet faithful synthetic data, but existing ap- proaches often inherit the inductive bias of single teacher model. As dataset size increases, this bias drives generation toward overly smooth, homogencous samples, reducing intra-class diversily and limiting generalization. We present PRISM (PRlors from diverse Source Mod- els), a framework that disentangles architectural priors during synthesis. PRISM decouples the logit-matching and regularization objectives, supervising them with different teacher architectures: a primary model for logits and stochastic subset for natthliormalizitionn (BN) alignment. On ImageNet-1K,, PRISM consistently and reproducibly outperforms single- teacher methods (e.g., SRe2L) and recent multi-teacher variants (e.g., G-VBSM) at low- and mid-IPC regimes. The generated data also show significantly richer intra-class diversity, as reflected by a notable drop in cosine similarity between frat We further analyze teacher selection strategies (pre- vs. intra-distillation) and introduce a scalable cross-class batch for- mation scheme for fast parallel synthesis. Code: https://githin http:tb.....a//raaaaoooeerrrrrra
1 Introduction
Dataset distillation (DD) has emerged as a critical and controllable data generation method in modern deep lcarning, motivated by three core objectives: enhanced robustness against adversarial attacks (Lai et al.) 2025 improved privacy through membership and model inversion safeguards (Dong et al. 2022 Carlini
et al. 2022), and efficient data compression (Zhao et al. 2020; Zhao Bilen 2023). Unlike large generative
visual models such diffusion models, DD produces images whose class semantics are guaranteed by gradient supervision, i.c., they produce
samples (Dhariwal & Nichol 2021 Fort & Whitaker
2025
Cazenavette et al. 2022).
As DD techniques have matured, significant challenge remains unresolved: the failure to synthesize diverse, large-scale datasets that capture the complexity of their real-world counterparts. Current DD methods, including gradient matching (Yin al., 2023 or parameter matching (Cazenavette et al. 2022 Cui al. 2023), also in combination with generative priors (Cazenavette al., 2023 Moser et al.] 2024
al. 2024
Duan et al.) 2023), predominantly concentrate on high-resolution synthetic images in ssallescale settings, such 50 100 images- perccsss (IPC) and fail to close the gap between the full and compressed datasets.
In this work, we question the motivation of compressing a dataset, since memory is cheap, in favor of the other motivations, namely robustness and privacy. Yet, naively upscaling DD techniques for an identical dataset. size often results in overly smooth, feature-restricted datasets lacking sufficient diversity (Shao et al. 2024a Sun al. 2024 Shen et al. 2025). as illustrated in Figure 1] This leads to homogeneous representations devoid of sufficient intra-class variability, thus impairing robustness as well as safety and ultimately limiting the practical utility of DD (Sorscher et al. 2022

---

SRe2L
PRISM (ours)
SRe2L and others
(vpll X Y,)
X,
same
L
BN (x,)
PRISM
(ours)
(x),
ReN(
X
L
R (N,
Figure 1: UMAP visualization of synthetic images from ImageNet-1K (10 classes), comparing SRe2L with our pro- posed multi-teacher alignment. Our approach, PRISM, gen- erates significantly greater intra-class diversity, contrasting the overly uniform clusters of SRe2T. that can lead to model overfitting more casily.
The core idea behind PRISM (PRIors from diverse Source Models): Use mul- tiple, diverse models for decoupling the logit maximization and regularization through BN alignment. instead of one, like in SRe2L, and related work.
Figure 2:
Our central argument is that knowledge within trained network is inseparable from the architecture that contains it (Clyanov et al. 2018 Shao et al. 2024a).. Any single model possesses a strong inductive bias, i.c., "view" of the world. Distilligg a dataset through one model inevitably imprints this single, limited view onto the synthetic data, resulting in a homogenous dataset that fails to train generalizing models. In order to create a truly generulizable synthetic dataset, we must synthesize it from distribution of "world views".
Addressing this limitation, propose diversifying the distillation process through prior framework that simultaneously optimizes logit matching @Yin et al. 2023 and regularization through batch normalization (BN) alignment (Yin et al. 2020) with at least two decoupled teacher models, which we coin PRISM (PRIors from diverse Source Models). As such, our teacher decoupling is orthogonal to all existing scaling attempts, as shown in Figure 2 Instead of relying on training schedules (Shen et al., 2025). data initialization (Sun al. 2024)). or post-ovaluation pipclines (Shao al. 2024b), PRISM introduces an orthogonal mechanism by decoupling the logil-malching objective from the BN-aligmment regularization. This allows different architectural priors from multiple, distinct teacher models to simultancously contribute to different aspects of the synthesis, complementing rather than replacing prior innovations.
In summary, our contributions are clear and well-scoped within the ongoing evolution of dataset. distillation:
First, we introduce PRISM, novel framework that tackles the lack of diversity in dataset distillation by decoupling the architectural priors used for logit supervision and BN-alignment regularization.
Second, we provide systematic analysis of teacher-selection strategies, demonstrating that a pre-distillation selection diverse teachers is highly effective.
Third, we show that PRISM not only sets new stato-of-the-art results, achicving up to 70.4% top-1 accuracy with ResNet-1101 at IPC=100, but also generates datasets with quantifiably greater intra-class diversity, directly addressing the critical challenge of homogeneity in modern dataset. distillation, while still maintaining simple and massively parallelizable synthesis pipeline that scales efficiently to large datasets like ImageNet-1K.
2

---

2 Related Work
Dataset Distillation. The field has largely bifurcated into two main strategies. Early and prominent methods relied on batch-to-batch matching, which involves expensive bi-level optimization to align gradients iZhao et al. 2020). training trajectories (Cazenavette et al. 2022: Cui et al. 2023 or feature distributions (Zhao & Bilen 2023) between real and synthetic data batches. While effective, the computational overhead of these methods makes them challenging to apply to large-scale datasets.
Thus, the community has shifted towards sitch-to-global matching, a paradigm pioneered by SRe2L Yin ct al. 2023). This approach uses a pre-trained teacher model to generate global statistics (e.g., from batch normalization layers) and then optimizes synthetic images to match these global targets. This strategy is significantly more efficient and has enabled distillation at the scale of ImageNet- IK. However, because every synthetic image in a class is optimized against the same global signal, this approach often suffers from a lack of intra-class diversity (Shao el al. 2024a
Strategies for Enhancing Diversity. Recognizing the diversity challenge in batch-to-globat matching. several methods have proposed alternative solutions. G-VBSM (Shao et al. 2024a] was key step in this direction, introducing the idea of using multiple teacher models and statistical metrics (bcyond just BN statistios) to create a richer, more varied supervision signal. EDC (Shao et al. 2024b) further refined this multi-teacher approach by identifying a suite of critical, and often overlooked, design choices in the post-training and evaluation pipeline, such as smoothing learning rate schodules and EMA-basod evaluation, that are necessary to unlock the full potential of a diverse distilled dataset.
Another related direction is D3S (?), which employs ensemble of independently trained ResNet-18 teachers and rotates them during optimization to reduce single-model bias. While similar in spirit, it keeps fixed architecture family and couples the same ensemble for both soft-labeling and feature-matching, whereas PRISM decouples both the teacher roles and the architectural priors, enabling a richer and more diverse supervision signal. CV-DD (?) improves diversity by letting a committee of heterogeneous teachers vote on soft labels, reducing the bias of any single model. Unlike PRISM, which diversifies the training signal by decoupling logit and BN supervision during synthesis, CV-DD focuses on the target labels rather than the optimization dynamics themselves. As such, it is complementary but orthogonal to our architectural decoupling strategy.
Other methods have approached diversity from different angles. RDED (Sun et al., 2024 focuses on the initialization data itself, using multi-crop image concatenation to create varied starting points for distillation. While this improves performance and speed, it sidesteps the goal of generating purely syuthetic data, as the resulting images are composites of real images, which may not satisfy the privacy and robustness motivations of DD. In contrast, DELT (Shen al. 2025) introduces the "EarlyLate" training scheme, where images are synthesized for varying numbers of iterations to create spectrum of synthetic samples from realistic to ract.
3 Methodology
In the following, we will derive the method behind PRISM step-by-step, starting from classical DD and SRe2L. Consider a real dataset. T = (X,, Yr) comprising N images, where X, E NxHxXXC are the real images, DD aims to distill this dataset into a smaller synthetic set S = (X,, Y.), where X, E Mx with M & N. Conventionally, M C. IPC, where C the number of classes and IPC are the specified Formally, classical DD seeks the optimal synthetic dataset S* that minimizes the distillation loss T(S
S* = aargmin L(S,T) T)
S
Here, we follow the formulation of SRe2L Fs with parameters Φ:
Yin et al.
2023 by optimizing over an output of teacher model
X3 arg min E ( (X.), Y.) + XRngg
(2)
X
3

---

where Rzeg is the regularization term to avoid noise-like artifacts, d.e., regularize synthetic images to look more natural, and À is its weighting hyperparameter. While there are multiple valiable options for Reeg such as L2 or TV regularization, the authors of SRe2L found that using the deep inversion (Yin et al. 2020 inspired BN alignment Rº BN of the model alone led to the best overall performance:
Rieg~ R
RR DN (X,)
Σ [l,,, E E T) + _)lo(o(x)) E (0 T))l2
(3)
where is the index of BN layer in the model with parameters 0, (1,0 and of are mean and variance, which can be conveniently approximated by the running mean and running variance in a pre-trained model at the 1-th layer.
3.1 Dual-Teacher Decoupling
The standard SRe2T. framework (Yin et al. 2023 uses a single, pre-trained teacher model for both parts of the objective function. This means the same model architecture and weights provide the supervision for both the logit-matching term (governed by paramcters Φ) and the BN-alignment regularization (which depends on the BN layer parameters within 9). We refer to this standard approach, where a single model fulfills both roles, single-teacher alignment. Thus, 0.
The core idea of PRISM is to challenge this coupling. We propose to decouple the architectural priors by allowing different models to supervise each term, a strategy we coin mulli-teacher alignment. In this setup, the logit teacher's parameters (ø) and the BN teacher's parameters (0) belong to distinct models. For instance, one could use an EfficientNet as the logit teacher and a standard ResNet as the BN teacher. This leads to an optimization where the gradient is a composite of two different architectural perspectives:
Vx, (x, Vx, ((p)(X.).Y)) + XVx, A (Xx)
Teacher
Teacher
As a result, the optimization of X, is guided by two distinct and potentially complementary objectives derived from different architectural priors:
(x..(pp(-..)): This term pushos X, to have features that are effective for classification from the perspective of the logit teacher. If optimized in isolation, this objective is known to produce adversarial-lis patterns that lack semantic realism and, therefore, lead to poor generalization (Yin ct al. 2020).
Vx, (..(..):: This term pushes X, to have low-level global feature statistics (mean and variance) that are considered "natural" from the perspective of the BN teacher model with parameters 0, a countermeasure against adversarial-like patterns.
Unlike prior ausemble-based methods (e.g., G-VBSM (haa al. 2024a), D3S (?), and CC-DD (?)), which aggregate identical architectures within single objective, PRISM introduces structural decoupling: different architectures supervise separate loss terms (i..., logit matching and BN regularivation). This decoupling changes the optimization dynamics, enabling complementary architectural priors rather than redundant ones.
3.2 Generalized
Alignment
As natural next step, more generalized version of our proposed decoupling is to apply multiple models for BN alignment. Thus, we obtain PRIors from diverse Source Models (PRISM). More concretely, let M (......., sPe. be l models used for BN aligument, the objective becomes:
X5 arg arg min é (x,) Y.) + XRreg
Xs
arg min é (xx( Y.) + Σ R
(4)
Xs
WEM
4

---

Class
Class
First Batch
IPC
Last Batch
First
Last
Batch
Batch
G-VBSM, EDC, DELT
SRe2L, PRISM
Figure 3: Batch formation and optimization strategies. (Left) Methods like G-VBSM, EDC, and DELT optimize jointly over all classes simultaneously. (Right) Methods like our PRISM and SRe2L process each IPC index independently.
This core principle is visualized in Figure? 2 using more than one BN alignment. for each distilled image. To formalize this, let Mpnol = ( :P0 the full set of kootll available source models for BN alignment, and let kmax be the maximum number of BN tcachers use simultancously, constrained by VRAM. We define the set of all valid, non-cmpty subsots of teachers as Mivalid {Maub Mpool 1 [Mauul S kmax).
To further boast diversity, we recommend including probabilities of
Thus, sample subset Maub from distribution P over all possible valid subsets, Maub P(Mlvatia). A simple and effective choice for P is the uniform distribution. The overall objective is then to minimize the expected loss over this random selection:
X = arg min (Pp((.)), Y,) + E P(Muatia) X3
Σ
REN (X+) (Xa)
(5)
WE W.M.out
By following the insights of Tran et al. (201)). we further claim improved robustness by this generalized multi-teacher alignment by providing a proof sketch in the appendix.
3.3 Teacher-Selection Strategy
To further dissect the role of teacher model selection in distillation, we explore two distinct teacher-selection strategies: (1) a pre-distillation selection strategy, in which a fixed set of teacher models is determined before the distillation begins, and (2) an distidiitillation selection strategy, where teachers are dynamically selected during the distillation process itself.
More specifically, in the pre-distillation strategy, the set of active teachers is determined once for each synthetic image before the optimization process begins. For given image Xs, we sample single gradient teacher & and a corresponding subset of BN alignment teachers Maub- This fixed ensemble then guides the entire distillation process for that specific image.
Conversely, the intra- distillation strategy leads to a more dynamic distillation process by re-selecting teachers during the optimization itself, method heavily inspired by G-VBSM (Shao al., (024a).. Within PRISM, this means that at each distillation step, a new of teachers, both the gradient teacher Pp and the BN alignment subset. Msub, can be re-sampled from their respective pools.
5

---

Table 1: Comparison with state-of-the-art dataset distillation on ImageNet-1K. We report top-1 accuracy [%]] for MosNNet-18/501111 trained on distilled datasets IPC = 10, 50, 100 reporting mcan over three seeds. While competitive at low IPCs, our method, PRISM, consistently and reproducibly establishes new state-of-the-art performance at higher IPCs (50 and 100) across all architectures and evaluation protocols. Results marked with $ use DELT's evaluation procedure (Shen et al. 2025 Bold indicates the best value per column. All other results follow our validation protocol.
denotes not reported.
ResNet-18
ResNet-50 (IPC=50)
ResNet-101
Method
(IPC=10)
(IPC=50)
(IPC=100)
(IPC=10)
(IPC=100)
(IPC=10)
(IPC=50)
(IPC=100)
SRe2L
21.3:0.6
46.8+0.2
52.8+0.3
28.4:0.1
55.640.3
61.040.4
30.9:001
60.8=0.5
62.8+0.2
G-VBSM
31.4+0.5
51.8+0.4
55.7=0.4
35.440.88
58.7+0.3
62.240.3
38.2+00.4
61.0=0.4
63.7+0.2
RDED
42.020.1
56.540.1
59.8+0.1
30.910011
60.8:0.5
62.840.2
EDC
48.620.3
68.010.2
54.110.2
61.310.2
51.7=1033
64.9:0.2
PRISM
49.4:10.2
59.0+t..1
60.9-0.2
51.141.2
65.1=0.1
67.5+0.2
48.5+1.7
65.9:1.2
68.6+0.4
DELT
46.1+0.4
59.2+0.4
62.4+0.2
1..5+11.6
66.1++0.5
67.6++0.3
PRISM f
46.9+0.1
59.6+0.2
62.7+0.1
46.3+0.7
66.5-0.2
69.4+0.1
40.7+3..
66.7+0.2
70.4+0.2
3.4 Batch Formation and Parallelization Strategy
A key design choice that distinguishes PRISM from other recent methods like G-VBSM (Shao et al., 2024a), ) EDC [Shao et al. 2024b), and DELT (Shen et al. 2025) is the batch formation strategy during data synthesis. PRISM, following SRe2L, processos each image-per-class (TPC) index independently. This creates cross-class batches where cach batch consists of single IPC slice across multiple classes (e.g., the i-th image from each class). The primary ad advantage of this stralegy is its high efficiency and soraightforward parallelizability; the synthesis of each IPC can be treated independently and easily distributed across multiple GPUs.
In contrast, other approaches often form intra-class batches, which contain multiple images from the same class, as shown in Figure 3 This enables specific regularization, such as the data densification in G- EVBSM/EDC optimization in DELT, which operates on the same-class images within a batch. While
or diversity-dring this improves diversity, this also comes the cost of increased complexity, as such regularizations introducc intra- batc-batchaae dependencies during optimization, e.g., through explicitly pushing images during distillation apart. Our method prioritizes a simple and massively parallelizable pipeline, achieving diversity not through complex intra-batch information exchange between images, but through our core contribution of BN decoupling and diversifying architectural priors.
4 Experiments
4.1
Setup
We follow the well-established SRe2L pipeline (Yin et al. 2023) and its standard configurations for the recovery stage to ensure methodological continuity. Moreover, we initialize the synthetic dataset by selecting each real image exactly once from the ImageNet training set, ensuring a direct and consistent comparison across different configurations in our identical datasct-size distillation (no coresct selection like in DELT or multi-inage initialization as in RDED (EDC) During the distillation process, we evaluate multiple teacher models to diversify synthetic data representations. We set maximum of 4000 optimization iterations.
While most dataset distillation papers include CIFAR-10 results, we deliberately focus on ImageNet-1K 1K and CIFAR-100 for two reasons. First, PRISM is designed to address the architectural bias and diversity limitations that become pronounced only in large-scale, high-class-count settings. Since the goal of PRISM is to scale diversity, ImageNet-1K and CIFAR-100 the minimal scale at which its contributions are meaningful. We emphasize that all basclines are reproduced and compared under identical conditions, ensuring fair benchmarking at the appropriate scale.
6

---

Table 2: Comparison with state-of-the-ar dataset distillation methods on CIFAR-100 using ResNet--18
We report top-1 accuracy [%] IPC 1, 10, 50. PRISM (ours) will be filled once evaluated.
IPC
SRe2L
G-VBSM
RDED
EDC
PRISM (ours)
1
2.0 0.2
25.9
11.0
3.1
41.5 -
10
31.6 + 0.5
59.5
4.2
63.7 0.3
64.7 + 0.2
50
49.5 = 0.3
65.0 0.5
6.1
68.6
69.4 + 0.3
4.2 Classical Dataset Distillation
Table 1 summarizes our main results, offering a balanced comparison between PRISM and prior statc-of-tho-art
methods on ImageNet-1K. Table summarizes our results on CIFAR-100. To ensure a fair and comprehensive analysis on ImageNet- IK, we present results under two distinct evaluation protocols: our own, which is optimized for low-to-mid IPCs (more details follow in Section 4.5). and the protocol used by DELT, which excels at higher IPCs.
Performance with PRISM's Optimized Evaluation. Under our primary evaluation protocol, PRISM establishes new SOTA results compared to EDC in Table 1] While EDC shows strong performance IPC=10 larger backbones, PRISM consistently and reproducibly outperforms all prior methods IPC=50 and IPC=100 across all (ested architectures For instance, ResNet-18, PRISM achieves an accuracy of 49.4% at IPC=10, surpassing EDC (48.6%), and extends its lead at IPC=50 (59.0%) and IPC=100 (60.9%). This trend holds for larger models, where PRISM's performance of 65.1% (ResNet-50, IPC=50) and 68.6% (ResNett-101, IPC=100) confirms that decoupling architectural priors highly effective strategy for generating diverse and generalizable synthetic data.
Performance with DELT's Evaluation Procedure. When evaluated using the DELT protocol, PRISM's advantages become even more pronounced for mid-to-high IPC scenarios, setting new SOTA results across multiple settings in Table 1 For ResNett- PRISM outperforms DELT al. IPC=50 (59.6%) and IPC=100 (62.7%). The benefits of our diverse architectural priors scale to larger models, where PRISM achieves a remarkable 69.4% accuracy ResNet-50 and 70.4% ResNet-101 at IPC=100. PRISM's ability to excel under an evaluation pipeline tailored for different method underscores the fundamental quality of the dataset it generates, proving its robustness across varied training configurations, especially for larger IPCs.
Performance CIFAR-100. On CIFAR-100, we observe similar trend when evaluating ResNet-18 student, summarized in Table 2 PRISM consistently outperforms prior methods across all IPC regimes, improving over EDC by up to 1.8 points at IPC=1 and maintaining a solid margin at IPC-10 and IPC-50.
4.3 Analysis on Diversity
Current dataset distillation methods, particularly those building upon SRe2L, (Yin et al. 2023 heavily rely on post-recovery strategics such as Knowlodge Distillation (Qin et al. 2024 Li al. 2025) and highly specialized validation settings to achieve their reported performance (Shao al. 2024b Shen et al. 2025 Yin & Shen 2023). To ensure unbiased diversity assessment of our proposed architectural decoupling, and to isolate its direct impact on synthesized data diversity, we intentionally diverge from these post-processing techniques in this analysis (contrasting the setting in the previous setting).
Our rationale is that if increased diversity, stemming from our aulti-architectural priors, is a truly orthogonal axis for scaling DD, then its pronounced performance improvement should be significantly observable even in this largo-scale, simplified setting, free from the confounding factors of post-recovery optimization. Experimental details are outlined in the Appendix C

---

Table 3: Study teacher alignment and selection strategies on ImageNet-1K, IPC=1200: final validation accuracy [%] of ResNet-18. In addition, we report the max. VRAM consumption during distillation under distillation batch size of 100. Note that this is the result for recovery-only (no knowledge distillation).
Variant
Model Selection
BN Teachers
Acc. [%]]
VRAM [GB]
Baseline (real data)
70.0
SRe2L
17.9
6.5
+ single-teacher alignment
intra-distillation
18.3
6.5
+ dual-teacher decoupling
intra-distillation
19.0
13.0
+ single-teacher alignment
pre-distillation
32.4
6.5
+ dual-teacher decoupling
pre-distillation
36.2
13.0
+ multi-teacher aligmment
pre-distilaation pre-distillation pre-distillation
37.4
18.5
+ multi-teacher alignment
3
38.7
26.0
+ multi-teacher alignment
4
39.1
32.5
0.925
Method
Cosine Simllarity
0.900
SRe2L
0.875
G-VBSM
DELT
0.850
PRISM
0.800
0.775
0.750
0
200
400
600
800
1000
Class Index
Figure 4: Intra-class semantic cosine similarity with pretrained ResNet-18 model on ImageNet-1K dataset applied on the respective distilled images showing higher diversity as indicated by lower mean values and higher variance.
4.4 Analysis on Recovery-Only Diversity
Recent advances in dataset distillation increasingly couple the core data synthesis process with powerful
post-recovery optimizations, such as knowlodge distillation (Qin al. 2024 Li et al. 2025 and highly specialized validation schedules (Shao et al.) 2024b Shen et al. 2025 Yin & Shen 2023 While effective at boosting final accuracy, this entanglement. can obscure the intrinsic quality of the generated data, making it difficult to attribute performance gains directly to the synthesis method itself. Therefore, to perform a rigorous and unbiased assessment of our architectural decoupling, this analysis intentionally isolates the synthesis stage from these downstream optimizations. Our rationale is that truly effective diversity-enhancing method like PRISM demonstrates a clear advantage in this controllod sctting, directly linking its architoctural design to the quality of the distilled data. Experimental details for this setup without knowledge distillation are outlined in the appendix.
Table 3 summarizes the results of our recovery-only setting, from which we derive the following three positive observations: (1) Sampling multiple teacher models during distillation helps to improve the performance, but pre-selecting them for the whole distillation process is better. (ii) Decoupling gradient matching and BN statistics alignment further improves the performance. (iii) The best performance is achieved by using multiple models for BN statistics alignment. Taken together, all best performing configurations form our proposed method PRISM, namely multi-teacher alignment with 4BN priors and pre-distillation selection.

---

SRe2L
PRISM
Figure 5: Qualitative comparison of synthetic images from ImageNet-1K generated by SRe2L and PRISM. Both methods start from the ezact same initial real images to ensure a fair comparison. The images generated by SRe2L (Ieft) exhibit significant homogencity, with samples within each class (goldfish, rooster, shark, frog) converging to similar colors and textures. In contrast, PRISM (right) produces a wider variety of contexts and colorations.
To quantitatively validate that method generates more diverse dataset, compute the intra-class semantic cosine similarity, This metric measures the average feature similarity between all pairs of synthesized images within the same class, using pretrained ResNet-18 as feature extractor; a lower similarity score thus indicates higher intra-class diversity. Figure 4 shows a clear separation between the methods. While existing approaches like SRc2L, G-VBSM, and DELT produce highly similar images (between 0.86 and 0.92), PRISM consistently achieves the lowest cosine similarity across all classes by a significant margin (mean values of 0.83 and below). This result provides strong quantitative evidence for our central claim: by decoupling and diversifying architectural priors, PRISM effectively breaks the homogencity constraint inherent in single-teacher distillation, leading to the superior downstream performance reported in Table 1
This quantitative evidence is further supported by direct qualitative comparison, as shown in F Figure To ensure a fair assessment, both PRISM and SRe2L start from the exact same initial images. While SRe2L consistently produces homogeneous images where samples from the same class converge on similar textures and poses, PRISM generates a visibly more diverse set.
During this study, also experiment with alternative regularization approaches besides BN aligument, which can be found in the appendix. In more detail, we try multiiresolution synthesis alternative regularization to BN alignment with CLIP embeddings (see Appendix D) as well using large-scale pro-trained generative text-to-image models for identical datasct-size generation (see Appendix E
4.5 Recovery and Post-Recovery Strategies
To identify the optimal configuration for PRISM, we conduct systematic analysis beyond DELT's evaluation procedure, beginning from the vanilla SRe2L bascline. Our analysis starts with the generation of soft labels, then progresses to the image recovery process, and concludes with teacher model alignment and learning rate
9

---

Table 4: Study relabeling and additional recovery strategies on ImageNet-1K with ResNet-18,
IPC=10: final validation accuracy [%].
(a) Lobe-Label Targets
(b) Add. Recovery Strategies
(c) Batch Size (Relabeling)
Variant
Acc. [76]
Variant
Acc.
[%]]
Batch Size
Acc. [%]]
ResNet-18 Only
21.22
Baseline from (a)
29.53
1024 from (b)
31.30
Ensemble Average
23.75
+ Resize Schedule
29.93
16
40.41
+ MAE (.05)
28.97
+ Var. Iterations
31.30
32
43.40
+ MSE (GT==0055)
28.07
+ Both
30.89
50
44.04
+ MAE (GT=0.1)
23.85
64
43.79
+ MSE (GT=0.1)
29.53
128
42.59
(d) Teacher Strategies
(e) LR (Recovery)
(f) Backbones
Variant
Acc.
[%]]
LR
Acc. [%]]
Model Pool
Acc. [%]
Baseline from (c)
44.04
0.25 from (d)
45.73
Models from (e)
47.35
01 +
0%1
45.73
0.10
46.63
EfficientNet (Rel.)
47.53
0.05
47.35
+ AlexNet (Rel)
AlexNet (Rec.) 47.77
schodule refinements. We employ ResNet18, ResNet34, ShuffcNNetV-00.5, MobileNet and EfficientNet-BO during multi-teacher selection, using ResNet18 for logit maximization ensure comparability with G-VBSM.
Optimizing Soft-Label Generation. Our first step is to refine the soft-label targets used for distillation. We find that moving beyond a single relabeler to an ensemble of the recovery models significantly improves performance, an idea inspired by G-VBSM (Shao el al. is most effective when using a Mean Squared Error (MSE) loss combined with a 0.1 ground-truth (GT) addition, which proves superior to both MAE and standard KL divergence.
2024a). As shown in Table 4a this ensemble approach
Refining the Recovery Process. With improved soft labels, we turn our attention to the recovery stage itself. We investigate two key strategics: scheduling the augmentation strength and varying the number of distillation iterations. Although adjusting the minimum crop size of the random resize augmentation provides only minor improvement, varying the number of distillation iterations image, following DELT (Shen
et al. 2025). yields the best performance Tablle 4b/.
Batch Size during Relabeling. With an effective loss function established, we next examine the batch size used during this relabeling phase. Our experiments, summarized Table 4c reveal that batch size of 50 is optimal, aligning with similar findings in recent literature (Yin & Shen 2023 Shao al., 2024b). Even with distilled data, the relabeling process remains highly sensitive to batch statistics.
First BN teacher. Finally, address the composition of the teacher models themselves. Confirming key insight from SRe2L (Yin et al. 2023), we find it crucial that the primary logit matching teacher (Φ) and the primary BN alignment teacher (01) align with the model used for relabeling. This alignment, once fixed, yields notable performance increase Table 4d).
Learning Rate in Recovery. Next, examine the recovery learning rate. While the process begins with standard learning rate of 0.25, our ablations demonstrate that much lower learning rate leads to more stable convergence and better final accuracy. As detailed in Table 4e we identify optimal value of 0.05.
Backbone Models. Replacing the heavier EfficientNet with AlexNet in both relabeling and recovery teacher pools yields an additional performance boost. without reducing architectural diversity Table 4f).
Learning Rate Schedule. The culminating improvement comes from adopting the learning rate schedule proposed by EDC (Shao et al. (2044b). Applying the SSRS decayed cosine schedule with slowdown coefficient. of = 2.5 provides the final significant leap in performance, leading to state-of-the-art results.
10

---

Table 5: Ablation between a fixed set of BN alignment teachers and PRISM's variable BN teacher set on ImageNet-lk at IPC=10. We report top-1 accuracy [%] for ResNct-18/50/101.
BN alignment strategy
RcsNet-18
ResNet-500
ResNct-101
Fixed BN teacher set
48.7 =8.7=0.3 0.3
48.3 0.7
42.7 = 1.1
Variable BN teacher set
49.4 0.2
51.1 + 1.2
48.5 + 1.7
Table PRISM performance on ImageNet-1K for ResNet-18 and ViT. We report top-1 accuracy [%] IPC 10, 50, 100 as meantstd over three seeds.
Backbone
IPC=10
IPC=50
IPC=100
PRISM (ResNet-18)
49.4 0.2
59.0
60.9 = 0.2
PRISM (ViT)
43.4 + 0.3
52.2 + 0.2
56.2 + 0.3
4.6 Fixed vs. variable BN teacher sets.
Table isolates the effect of sampling BN alignment teachers instead of relying on fixed set. Across all backbones at IPC=10, using PRISM's variable BN teacher set consistently improves performance the fixed-teacher variant: from 48.7% to 49.4% on ResNet-18, from 48.3% to 51.1% on ResNet-50, and most notably from 42.7% to 48.5% on ResNet-101. These gains support our claim that stochastic, diverse BN priors provide a stronger and more generalizable regularization signal than a single fixed alignment set.
4.7 PRISM with transformer backbones.
Our architectural decoupling naturally extends to models that do not rely on BN, such as Vision Transformers (ViTs), by keeping BN regularization confined to a CNN ensemble. Table 6 reports the performance when distilling for both a ResNet-18 and a VIT (ViT/S-16). While PRISM consistently yields lower accuracy on the ViT than on the CNN at IPCs, the gap narrows as more synthetic data become available: from 6.0 points IPC=10 just 4.7 points at IPC=100. We attribute this behavior to the well-known data hunger of transformers. With limited samples per class, ViTs underutilize the distilled supervision, but IPC increases, their relative disadvantage diminishes.
5 Limitations and Future Work
While PRISM establishes distinct and verifiable axis for scaling dataset distillation, it also opens several well-defined avenues for future investigation.
VRAM Constraints on Teacher Ensembles. PRISM's cross-class batching parallelizes cleanly across multiple GPUs, making the synthesis of individual IPCs highly efficient. VRAM capacity currently limits the number of simultaneous BN teachers per image. This invites exploration memory-efficintt teacher ensembles, including model offloading and parameter-officien fine-tuning.
Reliance on Batch Normalization. Our current formulation leverages the rich statistical priors available in batch normalization layers, which are prevalent in many standard CNN architectures. Extending decoupling framework to directly regularize using priors from models with alternative normalization schemes, such LayerNorm or GroupNorm, represents natural next step.
Cross- Architecture Evaluation. While PRISM was not evaluated with separate downst ream architectures such as Vision Transformers or ConvNeXt, this omission is intentional. Since PRISM already integrates a heterogencous set of CNN backbones BN teachers, synthotic data inherently captures diverse inductive biases across architectures. Evaluating with additional backbones would therefore conflate the supervision diversity already embedded within PRISM, rather than isolate a source of generalivation.
11

---

6 Conclusion
This work addresses a critical bottleneck in datasct distillation: the tendency of singlo-tcacher methods to generate homogeneous datasets with poor intra-class diversity due to a constrained inductive bias. We introduce PRISM, a simple yet powerful framework that diversifies the synthesis process by decoupling architectural priors. By separating the logit-matching objective from the batch normalization alignment and supervising each with different teacher models, PRISM effectively injects richer, facti-faceted training signal into the synthetic data.
Our extensive experiments on ImageNet-1K empirically confirm that PRISM not only achieves state-of-the-art performance but also produces datasets with quantifiably greater semantic diversity. Ultimately, PRISM establishes architectural decoupling as a new, orthogonal axis for scaling dataset distillation, paving the way for principled, generalizable synthetic datasets for robust and privacy-pree prevacypresoovingg machine lcarning.
Broader Impact Statement
Even when data are synthetic, there remains possibility that distilled datasets inadvertently encode sensitive attributes or enable membership inference and related privacy attacks, especially if the teachers are trained on data that contain such information. Moreover, the teacher models themselves encode societal biases (e.g., along gender, race, or socio-economic lines), PRISM can inherit and even amplify these biases through more diverse synthetic samples (but still biased). This risk is especially salient high-stakes applications such healthcare, hiring, or surveillance, where biased synthetic datasets can reinforce unfair outcomes.
12

---

References
Nicholas Carlini, Vitaly Feldman, and Milad Nasr. No free lunch in' privacy for free: How does dataset condensation help privacy". arviv preprint arXiv:2209. 14987, 2022.
George Cazenavette, Tongzhou Wang, Antonio Torralba, Alexei Efros, and Man Zhu. Datasct distillation by matching training trajectories. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 4750-4759, 2022.
George Cazenavette, Tongzhou Wang, Antonio Torralba, Alexei A Efros, and Yun-Yan Zhu. Generalizing dataset distillation via deep generative prior. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 3739-3748, 2023.
Justin Cui, Ruochen Wang, Si Si, and Cho-Jui Hsieh. Scaling up dataset distillation to imagenet- 1k with constant memory. In International Conference on Machine Learning, pp. 6590. PMI.R, 2023.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780 8794, 2021.
Tian Dong, Bo Zhao, and Lingjuan Lyu. Privacy for free: How does dataset condensation help privacy? In International Conference on Machine Learning, pp. 5378-5377700 5396. PMLR, 2022.
Yuxuan Duan, Jianfu Zhang, and Liqing Zhang. Datasel distillation in latent space. arXiv preprinl arXi::231.115447, 2023.
Stanislav Fort and Jonathan Whitaker. Direct ascent synthesis: Revealing hidden generative capabilities in discriminative models. arXie preprint or Xxxx:2502.07553, 2025.
Wei Lai, Tianyu Ding, Lei Wang, Jing Huo, Yang Gao, Wenbin Li, et al. Robust dataset distillation by matching adversarial trajectories. arXiv preprinl Stl:2553311003 2025.
Zekai Li, Xinhao Zhong, Samir Khaki, Zhiyuan Liang, Yuhao Zhou, Mingjia Shi, Ziqiao Wang, Xuanlei Zhao, Wangbo Zhao, Ziheng Qin, et al. Dd-ranking: Rethinking the evaluation of dataset distillation. arXiv preprint Xiv:2500.. 13300, 2025.
Brian B Moser, Federico Rauc, Scbastian Palacio, Stanislav Frolov, and Andreas Dengel. Latent datasct distillation with diffusion models. arXiv preprint arXiv:2403.0355 038811 2024.
Tian Qin, Zhiwei Deng, and David Alvarez-Melis. A label is worth thousand images in dataset distillation. Advances in Neural Information Processing Systems, 37:131946-131999 2024.
Shitong Shao, Zeyuan Yin, Muxin Zhou, Xindong Zhang, and Zhiqiang Shen. Generalized large-scale data condensation via various backbone and statistical matching. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pp. 16709-16718, 2024a.
Shitong Shao, Zikai Zhou, Huanran Chen, and Zhiqiang Shen. Elucidating the design space of dalaset. condensation. Advances in Neural Information Processing Systems, 37:99161-9929 2024b.
Zhiqiang Shen, Ammar Sherif, Zeyuan Yin, and Shitong Shao. Delt: simple diversity earlylate training for dataset distillation. In Proceedings of the Computer Vision and Pottern Recognation Conference, pp. 4797-4806, 2025.
Ben Sorscher, Robert Geirhos, Shashank Shekhar, Surya Ganguli, and Ari Morcos. Beyond neural scaling laws: beating power law scaling via data pruning. Advances in Neural Information Processing Systems, 35: 19523 19536, 2022.
Duo Su, Junjie Hou, Weizhi Gao, Yingjic Tian, and Bowen Tang. D Datasct distillation via discntangled diffusion model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pp. 4809-5818, 2024.
13

---

Peng Sun, Bei Shi, Daiwei Yu, and Tao Lin. On the diversity and realism of distilled dataset: An efficient dataset distillation paradigm. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pp. 9390 9399, 2024.
Cuong Tran, My H Dinh, Kyle Beiter, and Ferdinando Fioretto. A fairness analysis on private aggregation of teacher ensembles. arXiv preprint arXiv:2109.08653 2021.
Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Deep image prior. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 9446 9454, 2018.
Hongxu Yin, Pavlo Molchanov, Jose M Alvarez, Zhizhong Li, Arun Mallya, Derek Hoiem, Niraj K Jha, and Jan Kautz. Dreaming to distill: Data-free knowledge transfer via deepinversion. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8715-8724, 2020.
Zeyuan Yin and Zhiqiang Shen. Dataset distillation in large data era. Xr preprinl ar Xrx::23311. 188338 2023.
Zeyuan Yin, Eric Xing, and Zhiqiang Shen. Squeeze, recover and relabel: Datasel condensation at imagenet scale from a new perspective. Advances in Neurol Information Processing Systems, (7773522-3660) 2023.
Bo Zhao and Hakan Bilen. Dataset condensation with distribution matching. In Proceedings of the TEEE/CY /VF Winter Conference on Applications of Computer Vision, pp. 6514 6523, 2023.
Bo Zhao, Konda Reddy Mopuri, and Hakan Bilen. Dataset condensation with gradient matching, arXiv preprint Xro::20066. 05929, 2020.
14

---

Table 7: Summary of our different configurations used our ImageNet-11K distillation experiments.
(a) Validation settings
(b) Recovery settings
config
value
config
value
optimizer
AdamW
QBN
0.01
base learning rate
0.001 (all)
optimizer
Adam
weight decay
0.01
base learning rate
0.05
50 (TPC 10)
momentum
31, 82 0.5, 0.9
batch size
100 (IPC 50)
batch size
100
100 (IPC 100)
learning rate schedule
cosine decay
learning rate schedule
dec. cosine decay
recovery iteration
4,000
training epoch
300
augmentation
RandomResizedCrrp
Renzedo/eesieedCrrrp
augmentation
RandomHorizontalFiii
A On the Potential Privacy Benefits of Architectural Decoupling
We outline how PRISM's multi-architecturall distillation setup may implicitly enhance privacy by reducing the sensitivity of updates to individual data samples. Differential privacy (DP) ensures robustness to the inclusion exclusion of any single data point. Formally, randomized algorithm M is (e,6)-DP if for any adjacent datasets D, D' differing by one element and all subsets S:
P(M(D) E) e P(M(D) E
While PRISM does not explicitly satisfy this condition (as it adds no calibrated noise clipping); its architectural setup introduces natural source of stochasticity analogous to the Private Aggregation of Teacher Ensembles (PATE) framcwork (Tran et al. 2021
In PATE, privacy emerges from aggregating predictions across disjoint teacher models, with added noise ensuring DP guarantees. In contrast, PRISM partitions not the data but the architectural priors of its teachers. Gradients from distinct architectures exhibit disagreement, introducing what we term architectural noise (March), which acts as an intrinsic regularizer.
For synthetic data point 8j optimized using teachers and 0, the total gradient is:
g(s,, T)
1
Σ (z S,ET
+ - ((,,,
[T]
For adjacent datasets T and T' = [2'), the resulting gradient difference Ag = g( T') g(8;, depends on x', but the disagreement between architectures reduces this dependence:
Viotal V + Barch:
where Narch captures model-induced variance. This implicit noise weakens the influence of any single data point on the optimization trajectory, resembling the privacy-preserving aggregation in PATE, though without formal guarantees. Hence, while PRISM docs implement differential privacy the strict sense, its architectural decoupling may contribute privacy robustness by introducing model-level stochasticity that dilutes sample-specific gradients.
B More Training Details & Experiments
A brief overview of all the settings used during validation and recovery for ImageNct-IK is provided Table 7aland TTble 7b] for CIFAR-100 Table Sa and Table 8b To better contrast our settings with those of related work, we provide Table Tn Table 10 also summarize the influence of the min. crop size for
15

---

Table 8: Summary of our different configurations used in our CIFAR-100 distillation experiments.
(a) Validation settings
(b) Recovery settings
config
value
config
value
optimizer
AdamW
QBN
0.01
base learning rate
0.001 (all)
optimizer
Adam
weight decay
0.01
base learning rate
0.05
50 (TPC 10)
momentum
31, 82 0.5, 0.9
batch size
50 (IPC 10)
batch size
100
100 (IPC 50)
learning rate schedule
cosine decay
learning rate schedule
dec. cosine decay
recovery iteration
4,000
training epoch
1,000
augmentation
RandomResizeeCrrp
RenzedrnResizedCCropp
augmentation
RandomHHorizonallFiip
Table Configurations of various dataset distillation methods compared to ours (PRISM). Different
colors in each highlight the differences.
Config
SRe2L.
RDED
CDA
DWA
D4M
EDC
G-VBSM
DELT
PRISM (ours)
Batch Size (Etelabel)
1024
100
128
128
1024
100
1024
IPC depend.
IPC depend. AdamnW
Optimizer
AdamW
AdamW
AdanW
Adamn
AdamW
AdamW
AdamW
AdamW
LR Scheduler
cosine
eceine
decayed
cosine
decaged coeine
Loes Function (Relahel) Toucher
KL
KL
KL
KL
KL
MSE
MSE
KL
MSE
single
single
singlo
single
single
single
single, ensomble
CropRange (Rerovery)
0.08, 1.0
0.5, 1.0
0.08, 1.0
0.08, 1.0
0.08,
0.08, 1.0
0.08, 1.0
0.08, 1.0 Uniform
CropRange Schedule (Recovery)
Uniform
Uniform
Cosine
Uniform
Uniform
Uniform
Uniform
Cosine
PatchShufßle
Table 10: Performance comparison for different min. crop size for the randomly resized crop augmentation during relabeling and validation ImageNet-1kK, IPC-10.
Min. Crop Size
0.1
0.15
0.2
0.25
0.3
0.4
Acc. [%]
45.0
45.1
45.3
45.7
45.4
45.1
Table 11: Performance comparison for different batch sizes during recovery on ImageNet-IK, IPC=10.
Batch Size
40
80
100
Acc. [%]
44.8
45.2
45.7
the randomly resized crop augmentation during relabeling and validation, which resulted in found optimal value of 0.25. in Table 11 summarize experiments different batch sizes during the recovery stage, which resulted in keeping 100 as the optimal batch size. However, there is trend of increased performance with increasing batch size, so we expect even higher performance if more VRAM is available for larger batch sizes. We assume that increased batch size leads to a better mean and variance estimation of the global feature statistics for the batch normalization alignment.
C Experimental Details on Teacher Alignment and Selection
To isolate the impact of distinct distillation methods clearly, we adopt simplified experimental setup without employing soft-labeling additional training augmentation techniques. Specifically, we deviate from the original approach by reducing the batch size from 1024 to 256 and adopting an initial learning rale of 10-1 instead of 10-3 for faster convergence. Moreover, we employ a linear learning rate schedule rather than the conventional cosine anncaling, and we limit the training duration to 90 epochs, compared the original 300. Also, no relabeling was applied. The architectures employed during multi-teacher selection were ResNet18, ResNet34, ShuflleNetV2 (x1.0), MNASNet1.0 and EfficieettNet-RO.
16

---

Table 12: Study of direct ascent synthesis and varying time steps on ImageNet-1K, 12Cx1200: final validation accuracy [%] of ResNet-18.
Variant
Final val. acc. [%]]
Baseline (real data)
70.0
SRe2L
17.9
+ CLIP
8.8
Rellii-Reooltion
Deeplnversion
7.6
CLIP Multi-Resolution DeepInversion
5.2
D Alternative Regularization
) we logit-based based gradient matching and batch normalization (BN) alignment. Specifically, we experimented with semantic priors dcrived from CLIP embeddings and multi-resolution synthesis techniques proposed in DAS. Table 12 summarizes our findings.
Motivaled by recent. developments such as Direct Ascent Synthesis (DAS; Fort & Whitaker (2025
explored several alternative regularization strategies beyond conventional logit-
We follow the same experimental setup in (Section 4.4 Despite the intuitive appeal of these alternative regularizers, we obscrved that integrating various combinations of CLIP-based supervision and multi- resolution synthesis strategies did not result in performance improvements. Indeed, these configurations notably underperformed compared to our baseline SRe2[. method. These results suggest. that the intrinsic characteristics DAS-inspired methods, while effective in their original context, may not directly transfer addition, using multi-resolution synthesis
DD scenarios without further adaptation or refinement. introduces additional parameters since we distill images at multiple resolutions. Also, also tried the varying time steps proposed by Shen et al. (2025). but also observed decline in performance with this optimization strategy.
E
Alternative Synthesis Approaches
We further explored alternative data synthesis methods using popular text-to-image models, aiming to assess their viability for DD tasks. Specifically, we evaluated several state-of-tho-art models including FLUX (Schnell), Stable Diffusion (SD) versions 1.0, 2.1, 3.5 Turbo, SDXL, and SDXL Turbo. We follow the same experimental setup as in Section 4.4 Table 13 summarizes the anticipated results these experiments.
Preliminary observations suggest. these text-to-image models, despite their high generative capabilities other contexts, are unlikely surpass the baseline established by real datasets and conventional DD methods. This indicates inherent limitations in directly applying text-to-image generative models to distillation tasks without significant method modifications.
Table 13: Study of applying classical text-to-image models on ImageNet-1K, IPC=1200 with Text- To-lmage Models: final validation accuracy [%] of ResNet-18.
Variant
Final val. acc. [%]
Baseline (real data)
70.0
SRe2L
17.9
SD 1.0
17.1
SD 2.1
14.5
FLUX (Schnell)
9.8
SDXL
9.7
SD 3.5 Turbo
6.4
SDXT, Turbo
4.7
17


Reviews:
[{"review_id": "FqAQLYbGuF", "reviewer": "Reviewer_37qo", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces a framework called PRISM for dataset distillation. The authors argue that current dataset distillation methods produce homogeneous datasets and are limited in diversity. In order to solve this problem, PRISM uses multiple different teacher models for the logit and regularizing objectives. The authors also experiment with different selection strategies for the selection of teacher models in the training process and show results for the same. \n\nStrengths:\n1. The paper's key idea is simple and effective in combining the \"views\" of a dataset from different teacher models.\n2. The paper has good empirical results and ablation studies. PRISM consistently shows better performance on Imagenet-1k compared to baselines. The paper also provides the computational cost studies associated with it.\n\nWeakness:\n1. This has been addressed in the limitations section, but the key weakness seems to be the experimentation on only CNN based architectures. Since nowadays, CNNs have been replaced with vision transformers, it is not clear if this performance is applicable there as well. For instance, it might be that all the CNN based teacher models used learn different variations of the dataset in the feature space. But transformer architectures might not have that pattern, so using multiple priors could prove to have no benefits at all. I do understand that that would mean extending PRISM to layernorm and dealing with more memory requirements, however, what about showing the beneftis of distilled dataset on transformer architectures, i.e train a vision transformer on the distilled dataset? \n2. There is not a lot of information on how the teacher models are selected. How diverse in architectures should the teacher models be?", "claims_and_evidence_yn": "Yes", "claims_explanation": "The paper has a good evaluation section. The authors show that PRISM outperforms other baselines on ImageNet-1K. They also show evidence such as UMAP visualizations and cosine similarity between features to prove that PRISM is providing a much better distilled dataset.", "audience_yn": "Yes", "audience_explanation": "The paper's key idea is interesting, i.e atleast in CNN based architectures, different architectures learn something different in the dataset. Using multiple such priors can ensure that data distillation is much more diverse. The findings of the paper is something that would definitely interest other researchers in the data distillation space.", "requested_changes": "1. Provide empirical results with vision transformers trained on the distilled dataset. How is the performance on the distilled dataset?\n2. The experiment section is limited to ImageNet-1k, the authors argue that PRISM is suitable for large scale diverse datasets, but how serious of a limitation is that? Does it mean on small scale datasets such as CIFAR-10 and CIFAR-100, PRISM does not provide any added advantage? It would be good to have multiple different datasets to understand this. (Critical for accept)\n3. Similarly, are there any constraints on the teacher models to select? Do they have to be varied in sizes or diverse in architectures for PRISM to work? The paper does not clearly specify or perform experiments to understand this selection. In order to put PRISM to practical use, this information is critical. (Critical for accept)", "broader_impact_concerns": "I do not have any concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "lWqdSaJRT1", "reviewer": "Reviewer_6eU7", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes PRISM, a dataset distillation (DD) method which improves upon previous work by addressing the \"homogeneity\" problem present in single-teacher distillation methods. The authors argue that distilling from a single model results in low intra-class diversity and poor generalization. The authors fix this by building on top of the SRe2L method and decouple the two objectives of logit matching and regularization into different teacher models. The main logit matching objective is optimized using a single teacher model and the regularization objective is optimized using multiple teacher models which can belong to different architectures. This essentially helps learn several different views of the data due to the different architectures used for optimization. The paper shows state-of-the-art performance on the ImageNet-1k benchmark for IPCs >= 50 and improves upon several existing DD baselines. It achieves 70.4% Top-1 accuracy with ResNet-101 at IPC 100. The paper also shows qualitative samples generated compared with SRe2L and show that their method achieves the lowest intra-class cosine similarity suggesting that the synthetic dataset generated using PRISM has the highest diversity. Some weaknesses include: \n1. The method relies heavily on minimizing the BatchNorm based objective. Since BatchNorm is traditionally only used with CNN based architectures, the recent ViT based architectures traditionally use LayerNorm/RMSNorm etc, it might not generalize to these different architectures. \n2. The authors only show results on one dataset, ImageNet-1k. While they argue that the main benefits of PRISM can only be observed at much higher IPC and hence they need a large dataset, it would be good to show results on some additional datasets/domains such as for robustness on ImageNet-C which has several corruptions. It might also help analyze the robustness of data generated using PRISM as the authors mention one of the benefits of DD is \"enhanced robustness against adversarial attacks\". \n3. The teacher selection strategy for the BN objective seems a bit arbitrary as the authors randomly choose k teachers from a pool of multiple teachers. It might help to provide some more justifications as to why the uniform sampling is the best choice here. It is also not clear as to how the selection of different teachers affects downstream performance. For example, if all model selected were larger (eg. 4X ResNet-101) vs a mixture of smaller and larger models, what exactly changes. Currently the authors only show results in terms of the number of teachers used.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The authors have shown several clear and convincing experiments which validate their hypothesis. \n1. Decoupling logit supervision from batch normalization (BN) statistics enhances synthetic data diversity is validated through rigorous ablation studies (Table 2). These experiments effectively isolate the specific contributions of \"dual-teacher decoupling\" versus \"multi-teacher alignment\"\n2. SoTA performance on ImageNet-1K (Table 1). The results convincingly demonstrate that PRISM outperforms strong, recent baselines (such as SRe2L, EDC, and DELT) in the mid-to-high IPC regimes (IPC 50 and 100) across multiple backbone architectures (ResNet-18/50/101).\n3. Dataset Diversity: By employing a cosine similarity analysis (Figure 4) and conducting \"recovery-only\" experiments (Section 4.4), the paper successfully controls for confounding factors like soft-labeling or complex post-processing. This demonstrates that the performance gains are indeed driven by the intrinsic quality and diversity of the generated data. The qualitative comparisons (Figure 5) also show improvements over SRe2L.", "audience_yn": "Yes", "audience_explanation": "I think the paper is useful to audience around data distillation and synthetic data generation. The paper covers several interesting insights around decoupling objectives for DD, using multi-teacher ensembles and improving the general diversity of the synthetic data generated.", "requested_changes": "I believe the following would strengthen the work: \n1. Adding some discussion on more recent architectures which do not use BN such as transformers and how the objective function might change for them. If some experiments can be conducted showing whether or not those architectures work will be good. \n2. Analyzing adversarial robustness of different methods such as SRe2L, PRISM, etc to see how different methods perform on robustness. It will be interesting to see whether approaches will lead to high diversity and stronger performance on accuracy on ImageNet-1k also translate to higher robustness say on benchmarks like ImageNet-C, CIFAR-10C etc. \n3. Adding some additional analysison how the type of architectures used in the regularization objective impacts downstream performance and diversity. Currently, the authors only show how the number of teachers impact downstream performance. It will be great if the authors can show for a fixed set of teacher number, say 2, which 2 from their pool of teachers result in the best performance.\n4. It will be great if the authors can include a section on the different baselines and what were each of their contributions, it might help readers get a better sense of what additions PRISM makes to existing methods available.", "broader_impact_concerns": "The authors have included a statement and it looks good to me.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "mMnSWIj7vr", "reviewer": "Reviewer_ANs1", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "PRISM introduces a simple yet powerful idea: diversify dataset distillation by decoupling architectural priors across two roles (logit supervision and BN alignment), so that different teacher models guide different aspects of the synthesis. Instead of relying on a single model for everything, PRISM uses a logit teacher to shape classification signals and a BN teacher (or a pool of BN teachers) to regulate global feature statistics. This separation yields complementary gradients, producing synthetic data that support not only high accuracy but also richer intra-class diversity. The approach scales well because the batch formation is designed for parallelization across IPC indices, avoiding heavy intra-batch dependencies.  \n\nEmpirically, PRISM pushes state-of-the-art at mid to high IPC on ImageNet-1K across ResNet-18/50/101, while also showing noticeably more diverse intra-class representations, as evidenced by lower intra-class cosine similarity. The work also provides a systematic look at teacher selection, finding that pre-distillation with a fixed diverse set of teachers tends to yield more stable BN alignment signals than dynamic intra-distillation. Additionally, the generalized multi-teacher BN alignment, where multiple BN priors are used and even sampled per image, underpins the reported gains in diversity and robustness.  \n\nIn short, the key contribution is framing architectural decoupling as an orthogonal axis for scaling dataset distillation: by distributing logit and BN supervision to distinct, diverse priors, PRISM achieves richer synthetic data and stronger downstream performance, while maintaining a simple, massively parallelizable synthesis pipeline that scales to large datasets.", "claims_and_evidence_yn": "Yes", "claims_explanation": "Yes. The submission provides convincing and clear evidence that PRISM’s architectural decoupling—separating logit supervision from BN alignment across diverse teachers—yields higher IPC performance on ImageNet-1K (SOTA at IPC 50/100) and substantially greater intra-class diversity (lower intra-class cosine similarity). The evidence includes comprehensive quantitative results (Table 1–3), diversity analyses (Figure 4, 5), and ablations on teacher selection, BN teacher counts, and batch strategies.  While some aspects (cross-architecture tests, broader datasets) await further validation, the core claims are well-supported by the presented data.", "audience_yn": "Yes", "audience_explanation": "Yes. The findings are relevant to TMLR readers interested in dataset distillation, model diversity, and scalable training. PRISM’s architectural decoupling and multi-teacher BN alignment offer a clear, novel contribution with strong empirical results on ImageNet-1K, plus thorough ablations that illuminate design choices.", "requested_changes": "- Provide full, reproducible experiment details: list all hyperparameters, random seeds, data splits, IPC-specific evaluation protocols, and exactly how BN statistics are handled so others can replicate results. Release complete code and repo structure: ensure the public code covers multi-teacher BN alignment, Msub sampling, VRAM-constrained setups, recovery settings, and matches the paper’s descriptions.   \n- Expand BN teacher ablations: report results for a wider range of kmax values and give a clearer rationale for BN teacher pool choices.  \n- Cross-architecture validation or justification: provide at least one non-CNN backbone (e.g., Vision Transformer) or give a strong justification for focusing on CNNs to validate architectural decoupling.  \n\nAnd below are some strengthening adjustments (not required for acceptance, but would strengthen the paper)  \n- More datasets to test generalization: show results on at least one additional large-scale dataset beyond ImageNet-1K.  \n- Stability and uncertainty measures: report standard deviations across seeds or confidence intervals for key results.  \n- Richer visualizations: add more diversity-focused visuals (distributions of intra-class diversity, BN statistic distributions, more UMAPs) to reinforce the claims.", "broader_impact_concerns": "The authors have included a Broader Impact Statement addressing ethical implications related to their work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)