Asymmetric Synthetic Data Update for Domain Incremental Dataset Distillation
Authors: Minyoung Oh
Jae-Young Sim
Venue: ICLR 2026 Poster (2026)
Keywords: Dataset Distillation
Continual Learning
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Dataset distillation (DD) attempts to construct a compact synthetic dataset that serves as a proxy for a large real dataset under a fixed storage budget, thereby reducing the storage burden and training costs.
Prior works assume the full dataset is available upfront which is distilled at once, although real datasets are collected incrementally over time in practice.
To alleviate this gap, we introduce a new problem setting, *Domain Incremental Dataset Distillation*, that continually distills datasets from different domains into a single synthetic dataset.
The conventional DD sequentially processes arriving datasets in order, overwriting the old knowledge with new one, causing catastrophic forgetting problem.
To overcome this drawback, we propose *Asymmetric Synthetic Data Update* strategy that adjusts the per-sample update rates for synthetic dataset while balancing the stability-plasticity trade-off. Specifically, we design a bi-level optimization method based on meta-learning framework to estimate the optimal update rates, which allows each sample to focus on either stability or plasticity, thereby striking a balance between them.
Experimental results demonstrate that our approach effectively mitigates the catastrophic forgetting and achieves superior performance of DD across continually incoming datasets compared with existing methods.

Published as a conference paper at ICLR 2026
ASYMMETRIC SYNTHETIC DATA UPDATE FOR DOMAIN INCREMENTAL DATASET DISTILLATION
Minyoung Oh & Jae-Young Sim
Graduate School of Artificial Intelligence
Ulsan National Institute of Science and Technology (UNIST)
Ulsan, Republic of Korea (mmyy2513, jysim Bunist. ac.
ABSTRACT
Dataset distillation (DD) attempts to construct a compact synthetic dataset that serves proxy for large real dataset under fixed storage budget, thereby re- ducing the storage burden and training costs. Prior works assume the full dataset is available upfront which is distilled at once, although real datasets collected incrementally over time in practice. To alleviate this gap. we introduce a new problem setting, Domain Incremental Dataset Distillation, that continually distills datasets from different domains into a single synthetic dataset. The conventional DD sequentially processes arriving datasets in order, overwriting the old knowl- edge with new one, causing catastrophic forgetting problem. To overcome this drawback, we propose Asymmetric Synthetic Data Update strategy that adjusts the per-sample update rates for synthetic dataset while balancing the stability- plasticity trade-off. Specifically, we design bi-level optimization method based meta-learning framework to estimate the optimal update rates, which allows each sample to focus on either stability or plasticity, thereby striking a balance between them. Experimental results demonstrate that our approach effectively mitigates the catastrophic forgetting and achieves superior performance of DD across continually incoming datasets compared with existing methods.
INTRODUCTION
Dataset Distillation (DD) (Wang et al.] 2018) aims to generate small synthetic dataset that encap- sulates the essential information learned from a large real dataset by matching the training dynamics of models between the synthetic and real datasets. Highlighting the importance of the large scale dataset to guarantee good performance of deep learning models, DD has attracted significant atten- tion to streamline the model training process and reduce storage costs. These advantages of DD facilitate more efficient applications in areas such as continual learning (Masarczyk & Tautkute) 2020 Yang al. 2023] Gu et al., 2024), neural architecture search (Such al. 20200 Medvedev & D D'yakonov] 2021). and federated learning (Goetz & Tewari 2020 Jia et al. 2024). while si- multancously addressing privacy issues (Dong et al. [022] Chai et al. 2024; Zheng et al. (025). Notably, a recent method (Guo et al. 2024) even surpasses the performance of the whole dataset, while requiring significantly less storage compared to the original dataset.
However, existing methods of DD only consider that the entire dataset is available in advance, even though large datasets are often collected incrementally over time practice. In such a case, a naive approach that distills each newly arriving dataset into seperate synthetic dataset in order, linearly increases the storage burden accumulate the resulting synthetic datasets, as shown Figure (a). Moreover, the training costs also increases since the models should be trained on all accumulated synthetic datasets.
To address those storage and training cost issues, formally define new problem setting, Domain Incremental Dataset Distillation (DIDD), where datasets from different domains arrive sequentially while sharing the same label space. To tackle this, introduce novel framework that continually distills these arriving datasets into a single fixed-size synthetic dataset, as shown in Figure [ (b). It keeps efficiency of the storage and training costs by consistently maintaining single fixed-size

---

Published as a conference paper at ICLR 2026
Real Datasets
Storage
Storage
Storage
Storage
expansive
Distilled Datasets
slow
Storage
Storage
Storage
Storage
compact
fast
(b)
Figure I: Comparison of (a) existing Dataset Distillation and (b) the proposed Domain Incremental Dataset Distillation. (a) DD considers per-dataset distillation, which increases storage and training costs depending the number of datasets. (b) DIDD maintains fixed-size synthetic dataset over time, thereby keeping storage and training costs comparable to those of a single dataset.
synthetic dataset over time. However, existing DD methods only focus on adapting the synthetic dataset to the most recent dataset, which is known as plasticity, overlooking the importance of preserving knowledge from previously seen datasets, referred to as stability. Consequently. the knowledge of earlier datasets in the synthetic dataset can be easily overwritten by that of the most recent one, known as catastrophic forgetting (McCloskey & Cohen 1989
French 1999).
To alleviate catastrophic forgetting, we introduce stability loss, which preserves the knowledge of the previously seen datasets by encouraging the features of the updated synthetic dataset to re- main consistent with those of the previous one. However, the stability and plasticity inherently conflicting to each other, since the datasets collected at different time instances contain different knowledge. To mitigate this conflict, adjusts per-sample update rates for the stability and plasticity objectives, respectively, using bi-level optimization within meta-learning framework (Finn al. (017). In this way, each sample is encouraged to focus on either stability or plasticity, thereby achieving balance between the two. Experimental results on Rotated MNIST (Larochelle al. (2007), Seq-CORe50 (Lomonaco & Mal- toni 2017 Shi & Wang 2023), and PACS (Li et al. 2017) datasets demonstrate that the proposed method effectively mitigates catastrophic forgetting, achieving superior DD performance compared with existing methods.
propose asymmetric synthetic data update strategy that
Our contributions are summarized as follows:
We formally define Domain-Incrreeent Dataset Distillation (DIDD) novel prob- lem setting and introduce a framework that distills the knowledge continually arriving datasets into single synthetic dataset under fixed storage budget while preserving knowl- edge from previous data.
We design an asymmetric synthetic data update strategy that adjusts per-sample update rates via bi-level optimization to effectively strike balance between stability and plasticity.
2

---

Published as a conference paper at ICLR 2026
Experimental results demonstrate that the proposed method effectively mitigates catas- trophic forgetting while accurately representing the most recent dataset, achieving superior performance compared with existing methods.
2
RELATED WORK
2.1 DATASET DISTILLATION
Dataset Distillation was first introduced by (Wang et al. 2018) and has since been actively studied to reduce both storage and training costs for large datasets. The main idea is to synthesize a compact surrogate dataset whose training dynamics induce model updates comparable to those obtained from the full dataset. Recent approaches realize this objective by matching aspects of the training dynam- ics between models trained synthetic and real data. Broadly, they fall into three categories: (i)
gradient matching, (ii) trajectory matching, and (iii) distribution matching. Gradient Matching (GM) methods align the parameter gradients computed on synthetic data with those computed real data, typically by minimizing the discrepancy between the two gradient vectors (Zhao & Bilen (021a)) Building on GM, DSA introduces differentiable data augmentation improve the generalization of the distilled dataset (Zhao & Bilen 2021b). Trajectory Matching (TM) methods encourage the opti- mization path of parameters trained on the synthetic data to follow the path observed when training on the real data (Cazenavette et al.) 2022). Recent TM- MM--asedd methods achieve superior performance by analyzing the knowledge contained in the early or late segments of the trajectory (Guo et al.) 2024) or by emphasizing class-discriminative regions during image updates (Wang et al. 2025) However, they typically require substantial computation, as distillation involves unrolling and back- propagating through multiple optimization steps. Distribution Matching (DM) methods align feature distributions computed from synthetic and real data (Zhao & Bilen 2023) and are generally the most computationally efficient among the three categories. Recent studies leverage intermediate features for alignment (Wang et al. (202)) and develop improved distance metrics to measure distributional similarity (Zhang al.) 2024), leading more faithful distilled datasets. However, existing methods usually assume that the entire dataset is available upfront and can be revisited repeatedly, which is often unrealistic in practice. As result, they do not directly address scenarios where datasets arrive sequentially over time under fixed storage budget.
2.2 DOMAIN INCREMENTAL LEARNING
Domain Incremental Learning (DIL) is subfield of continual learning (Van de Ven et al. 2022) that considers settings where datasets from different domains arrive sequentially while sharing the same label space. In this scenario, the set of classes remains fixed, while the data distribution shifts across domains due to differences in acquisition conditions, sensors, styles, or backgrounds. Such domain shifts often lead catastrophic forgetting, where knowledge acquired on earlier domains is overwritten by that of recent domains. Approaches to DIL are commonly grouped into three cat- egories: (i) eggularization-based (ii) replay- and (iii) dynamic rechttecture-based methods. Regularization-ban based methods constrain updates on parameters deemed important for previous do- mains that those parameters do change excessively. For example, EWC estimates parameter importance using the Fisher information (Kirkpatrick et al. 2017). while MAS leverages parameter sensitivity to achieve a similar effect (Aljundi et al. 2018). Replay-based methods store a sub- set of past data and train with it alongside current data to resist distribution shift. GEM imposes constraints that prevent gradients from harming past performance (Lopez-Paz. & Ranzato 2017). whereas ER mixes exemplars from past and present domains directly in minibatches (Rolnick et al.) 2019). Dynamic architecture-based methods expand model capacity when new domains arrive, help- ing isolate knowledge and reduce interference. PNN grows the architecture roughly linearly with the number of domains (Rusu et al., (016),, while other approaches expand capacity adaptively to balance sharing and specialization (Hung et al. 2019). Despite these advances, it remains underex- plored how to mitigate catastrophic forgetting in DD when heterogeneous domain datasets continue to arrive.
3

---

Published as a conference paper at ICLR 2026
stubilaty plasticity
---1
Upulate
stability
D
plasticity
x
Agyon Update
stubilaty
D
plasticity
x
(b)
Figure 2: Overview of (a) the proposed framework for Domain Incremental Dataset Distillation
utilizing (b) the asymmetric synthetic data update strategy.
3 PRELIMINARY
The conventional dataset distillation aims to synthesize a small dataset DD which serves as a proxy for a large real dataset DD where [D] < |DD,, such that the models trained on D achieves comparable performance to those trained on DD Concretely, the goal of DD can be formulated as follows:
min B [(( D
where P(D*) denotes target data distribution, L(a, b) measures the difference between and b, and Φ and ΦD indicate the models trained on DD and D, respectively. To achieve this goal, recent meth- ods synthesize D by matching training dynamics (i.e., gradient, trajectory, and feature distribution) of models trained on D and DD respectively. In this paper, we focus on the DM method (Zhao & Bilen 2023), which matches the feature distributions of to those of D in a class-wise manner, computed using randomly initialized model Ø.
4 METHODOLOGY
Figure illustrates the proposed Domain Incremental Dataset Distillation framework, which asym- metrically updates the per-sample rates. This strategy balances stability and plasticity, allowing each image to prioritize one over the other.
4.1 PROBLEM FORMULATION
Conventional dataset distillation generates a synthetic dataset from each dataset independently, as- suming that the entire dataset D is fully available in advance. However, when sequential datasets (D')7= arrive over time. such per-dataset distillation scheme increases the storage and training costs. To overcome these drawbacks, we formally define new problem setting, Domain Incremen- tal Dataset Distillation, where datasets from different domains arrive sequentially while sharing the same label space. To address this problem, propose a novel framework that continually distills these datasets into a single synthetic dataset. Specifically, let the dataset D' with C' classes arrives at the t-th time instance. The goal of DIDD is to synthesize an optimal dataset D' that not only con- tains the knowledge of the current dataset D' but preserves the knowledge of the previous datasets, Dlit-1, subject to contraint of fixed storage budget, |D' IPC x C where IPC means the Images Per Class. Consequently, we formulate the problem of DIDD as follows:
min E Bew D
( (æ), )D).( (x))]
s.f. | = IPC x C.
(2)
Note that DIDD does incur increasing storage training costs time, since it consistently maintains single fixed-size synthetic dataset D'.
4

---

Published as a conference paper at ICLR 2026
4.2 ASYMMETRIC UPDATE VIA BI-LEVEL OPTIMIZATION
Plasticity and Stability. We construct a baseline following (Zang et al., 2024), minimizing the Maximum Mean Discrepancy (MMD) between two feature distributions of D and D', defined as the plasticity loss:
Cplastic =d(
(3)
Here d(A, B) is the MMD with the Gaussian kernel between two sets of and B. F(a:.) (o(2",)) and F( {)( where z²,, and æ indicate the i-th image of class in D' and De, with N < M. We omit the class index c throughout this paper for simplicity. Minimizing Lpassicc increases plasticity P making D a faithful proxy for D.
However, existing DD methods focus solely on maximizing P adapting given datasets. Therefore, directly applying them the DIDD setting results in D' only containing the knowledge of D'. while forgetting that of D which known as catastrophic forgetting (McCloskey & Cohen 1989 French 1999). To mitigate this problem, it is essential to promote the stability S of the synthetic dataset. Thus, we introduce stability loss Listable that matches the features of D' to those of D
Lstable di (()( F()")).
Minimizing Latable increases of Di by regularizing its features to remain consistent with those of Di-1, thereby preserving knowledge from earlier datasets.
By jointly minimizing Eplastic and Letable each image de is updated while considering both plasticity and stability simultancously, formulated as:
Osm
Oass
05
a + क na (gs.s + gr,).
(5)
where The is the learning rate for the synthetic data and
-0.0
gs., @Estable Oá
gp @Cplastic (Ø'
(6)
Oá.
300
33:00
2002
However, shown in Figure gs.; and gr, often con- flict with each other, as indicated by their negative cosine similarity. This is because gs preserves knowledge from
Figure 3: Cosine similarity between Is,i and gp, on PACS.
D while gp. adapts to D', leading to conflicting ob- jectives. Therefore, applying the same update rate to gs.; and gr,; is potentially suboptimal.
Asymmetric Synthetic Data Update. To address this issue, we propose an asymmetric synthetic data update strategy that assigns distinct update rates to gs.i and gr, for each sample aç. Specifi- cally, we introduce scale parameters a N. and 3 {B.] to scale gs,i and gr.i, respec- tively. Accordingly, Eq. 5 is modified as
+ al The (a. gs, B, gp.)).
Here 0j and B, are bounded values by sigmoid functions within (amin amax) and (Amin Blax), respectively, which are formulated as:
Omax Omin
Bi = Amin + Smax Buin
C₁ Omin 1 cxp(-a;;)
1+ 1+exp(-83)
(8)
This asymmetric update strategy enables each sample to prioritize either stability or plasticity by adaptively adjusting ās and Be. For instance, át emphasizes stability over plasticity if a; > 3. and it emphasizes plasticity if ā; < B. In this way, the conflict between stability and plasticity can be mitigated.
However, it is not trivial to determine the optimal values of a, and B for each sample. To this end, we propose a bi-level optimization method based on the meta-learning framework (Finn al.) 2017]. treating that treats a; and Bi as learnable parameters. Here a; and B are initialized as O. Specifically, to find the optimal as and Bi, we first perform a meta-update of # by using Eq.
5 meta, #/ (as gs.t + B, gP.:)-
5

---

Published as a conference paper at ICLR 2026
Using 2 intta, we compute the meta-loss Lueta (#
Lmeta (Lineta) Catable (eeta) + plastiic ( metaa).
(10)
Minimizing Lmeta guides the search for optimal and / that maximize S and P, respectively.
However, this process may lead to a trivial solution in which both ā; and 3. are maximized indis- criminately. To analyze this, we consider the first-order Taylor expansion of Letta Lmeta( (2meta) around :²:
Lueet ( ( meta)) R (meeta(')) - na Fmeta (+')
(ā: gs. 3: ...gp.))
(11)
Oāt
==1
This implics that large values of āg and 3, for all can significantly decrease in Cmeta mmeta) if the inner products & Emeta xs and ( Cmeta (t'), gr..) are positive, To prevent this trivial
solution and enforce selectivity, we penalize the magnitude of the average per-sample update rates, ag and 3, by minimizing:
N
N
1
1
Sponalty
04,
Sponalty-8
B
(12)
N
N
This regularization discourages üs and Be from growing indiscriminately. thereby encouraging an asymmetric update strategy.
Consequently, we formulate the overall meta-objective as:
Chenalty
Lmeta -mmeta (#'meta) + >a Cpenalty-a Xg Epenalty
(13)
By minimizing Cpenally we update the learnable parameters 0;; and 3; as follows:
ppenslty
2 penalty
Ta
Da;
meta
B: t- B; 7)8 0s,
meta
(14)
where and Πy are the learning rates for a; and Bú, and Aa and X are hyperparameters. In this way, obtain and that maximize S and respectively, while keeping their average magnitudes minimal. Finally, updating & with the learned a and G encourages each sample to specialize in either S or P, yielding synthetic dataset D' that strikes an optimal balance between them.
Theoretical Interpretation via KKT Conditions. We view DIDD as a constrained optimization problem in which the synthetic data &' is required to satisfy stability and plasticity requirements. Let éstable, and épasticc. denote desired tolerance thresholds for each sample a!. Conceptually, the primal problem can be written as
find o' s.t. Lstable S Estable.i; Eplastie (æ.) S plastico,
Wi.
(15)
Here the thresholds éstable,i and éplastic, describe the desired upper bounds on the two losses.
Introducing non-negative Lagrange multipliers a. 3 20. we obtain the Lagrangian function:
N
Σ 1=1
aa (Cstable(E)) Cstable.s) + B3 (Cplasticc <plastic..)
(16)
where a; and 3, are the bounded versions of a; and B, defined in Eq. Note that the gradient of L with respect to át! aligns with the proposed asymmetric update direction in Eq. 7 This provides an interpretation of oi; and Be as quantities analogous to sample-wise Lagrange multiplerrs, controlling the extent to which each constraint influences the update of áς.
In the ideal constrained problem, the KKT conditions imply the complementary slackness relations
(Cstabble (x.) Estable,)) = 0, (Cplaatti - Eplastic,)) = O,
(17)
meaning that the multipliers become non-zero only when the corresponding constraints are active. Our meta-learning objective approximates this behavior by optimizing and / that enforces the samples with relatively close to violating stability or plasticity constraints to receive larger update rates. Therefore, the KKT formulation serves an interpretive lens that connects the asymmetric update rule to a constrained optimization perspective and clarifies the role of a and B3 as stability and plasticity weights.

---

Published as a conference paper at ICLR 2026
5 EXPERIMENTS
5.1 EXPERIMENTAL SETUP
5.1.1 DATASETS
To verify the effectiveness of our method, we use three datasets: Rotated MNIST (R- MNIST) (Larochelle et al. (007), Seq-CORe50 (Lomonaco & Maltoni 2017 Shi & Wang 2023),
and PACS (Li et al. 2017). 7. R-MNIST containing the TO digit classes; the images in each domain are rotated by domain specific angle that increases across domains. Seq-CORe50 contains 50 object classes collected over 11 different locations, which are treated distinct domains. PACS contains 7 classes from 4 domains: Photo, Art painting, Cartoon, and Sketch. We describe more details about datasets in Appendix A.2
variant of MNIST with 20 sequential domains, each
5.1.2 IMPLEMENTATION DETAILS
All experiments are implemented with PyTTorc and run single NVIDIA RTX 3090 GPU. For all datasets, images are resized to 32 x 32 and a simple 3 layer ConvNet is used as the feature extractor with instance normalization for distillation, following prior DD work. For distillation, we use the SGD optimizer and run 3,000 iterations per domain. We apply data augmentations including ColorJitter, RandomCrop, and Cutout (DeVries & Taylor 2017) following (Zhao & Bilen 2021b),
and the multi formation function following (Kim et al.) [022
2 Zhang et al. 2024). Hyperparameters
are set as Qminn = Smin O, Qmax 8max 2. 1/x 7)
le-2, and Aa = Å = le-4.
More detailed hyperparameters are provided in the Appendix A.5 After distillation, we evaluate
the distilled dataset by training model from scratch for 1,000 epochs using SGD with learning rate
le-2 and batch size 64.
5.1.3 METRICS
After distillation from the T-th domain dataset, we train a network on DE and evaluate the distilled dataset using two metrics, average accuracy AA" and average forgetting J" The average accuracy is the mean of per domain accuracies after distillation of D':
T
AT
(18)
T
where of denotes the accuracy on the t-th domain, evaluated with the model trained on DA after distillation from the k-th domain dataset. The average forgetting FT measures the mean drop from the best accuracy previously achieved on each domain to its accuracy after the T-th domain dataset distillation:
1 max at
FJ -
T-
ai
(19)
ISkkT
1=1
All results are reported as the mean and standard deviation over three runs.
5.2 PERFORMANCE COMPARISON
We compare our method on three datasets when IPC is 1, 10, and 20 on R-MNIST, and 10 and 20 on Seq-CORe50 and PACS, using the following baselines (i) Whole dataset: training on the entire real datasets across all domains, where the storage budget is Σ | (ii) Gistil-Gather (Gather): training on the union of per domain distilled datasets as conventional DD methods (as shown in Fig- ure (a)), where the storage budget is Ex [D']. (iii) Joint: training on a dataset distilled jointly from all domains, where the storage budget is 12'', thereby serving
upper bound. (iv) Fine- tune: training on a dataset distilled sequentially from each domain without additional techniques. For more diverse baseline construction, additionally implement Gather and Joint methods based on DSA (Zhao & Bilen 2021b) and DC (Zhao & Bilen 2021a). which are most widely used DD methods.
Since this is the first study on domain incremental dataset distillation, there is no existing method for direct comparison. Therefore, also compare several regularization-base DII. methods, LwF (Li
7

---

Published as a conference paper at ICLR 2026
Dataeer IFC
R-MNIST 10
Seq CORRe50
PMCS
20
10
20
10
Metne
1 5e
AT (1)
yr (1)
A
(t)
A ct:
AP
A7 (t)
57 ())
Real Dutasee (Budget -
Whilslats
98.9ge:
98 Ses-
98.0
98 Bess
98.6
66.1
66, Iosa
Convertional Dunset Distilution (Budgec. Lanes
(2%))
Gather (M3D)
92.8e2
94.0uca
93,9um
96.5a3
97.5
56.5.14
59.5.04
Gather (DSA)
74.1111
92.9uc
69 69.9 71 Baax
76.0144
50.2.13
-4.1.11
Gather (DC)
87 4ers
501 leen
52. $2.5
Join Dutuset Disiilacoon (Budget: (2")
Joint (M3D)
50-24ea
50,9k64
36.3
84-7
52.7.M
54.5-14
Joint (DSA)
55.413
62.3
33. 23.2
41.2137
46.1-18
45.6.11
Joine (DC)
67.4
34 Pas
42.4
Domaio Iacremernal Dutueet Distilation (Bulge:: (P))
Fisetuse
38.9as
50 10
39.4.3
99.7.a
41.9uaa
567eca
79.0.32
26.4432
26.
443
27.4.1a
48.5:1
35.0
39.013
19.3119
42.6.4
42.911
55.0.14
25.7
79.0139
25.912
79.7184
274
42.9
44,0114
MAS
45:21:5
10.2113
45.611
45.0.1
518134
60.3:12
12.0.44
66.118
35.0.13
Pripucaa
3K
21.0eaa
6277
34.-3a
39,8-a
30 30.3am
35.4
18.811
38.74m 7m
48.0gm
11.2
52.1
16.0
Table 1: Performance Comparison on R-MNIST, Seq-CORe50, and PACS datasets.
& Hoiem 2017), EWC (Kirkpatrick et al., 2017). MAS (Aljundi et al. 2018). and LF (Jung et al. 2016). by substituting the regularization target from model parameters to synthetic data. Note that we do not consider other methods that require extra memory such as replay and dynamic irchitecture-based methods, which conflicts with the goal of DD minimize storage.
Table T reports performance on R-MNIST, Seq-CORe50, and PACS, respectively. Us- ing all real datasets across domains, Wholedata achieves the best performance as expected, but it incurs the largest storage and training costs. Gather is second best because uses all per domain distilled datasets, however it still suf- fers from increasing storage and training costs as the number of domains grows. Joint can be regarded as an upper bound for the DIDD task since it maintains storage and training effi- ciency comparable to a single synthetic dataset by distilling from all domains jointly. Finetune serves as a lower bound and performs the worst in most cases due catastrophic forgetting. We observe that EWC and LF fail to preserve knowledge from earlier datasets, resulting in performance similar Finetune. MAS and LwF both perform better on PACS, which has relatively short quence, but LwF degrades on the longer sequence datasets R-MNIST and Seq CORe50 whereas MAS only slightly better than Finetune on those datasets. Although some methods report the lowest JJ in certain cases, this largely results from low overall accuracy, which reduces forgetting scores under Eq. 19 In contrast, our method outperforms Finetune and the other methods by a clear margin across all settings. Remarkably, the proposed method surpasses even the Joint upper bounds of established baselines (DSA and DC) on R-MNIST (at IPC = 1). Seq-CORe50 and PACS (at IPC- 10, 20). In particular, on the PACS dataset, our method recovers over 91% % and 95% of the Joint performance at IPC 10 and IPC = 20, respectively, validating its high data efficiency.
Joint
Finetune
MAS
Proposed
Figure 4: Synthesized images (guitar) on PACS.
We also visualize synthesized images PACS in Figure 4 Images synthesized by the Joint method show that multiple domains are well captured together, whereas Finetune mainly reflects the most recent domain, Sketch. MAS produces images biased toward earlier domains such Photo and Art painting, indicating poor plasticity. By contrast, images synthesized by our method closely resemble those of Joint, which verifies that our method effectively strikes balance between stability and plasticity. More qualtitative results are provided in Appendix A.9
Figure shows the average accuracy as increases on each dataset for IPC 10 and IPC 20. Baselines including Finetune and other continual learning methods exhibit substantial degradation as grows due to catastrophic forgetting. Meanwhile, our method preserves knowledge from earlier datasets and achieves the best performance across t on all datasets.
8

---

Published as a conference paper at ICLR 2026
Jeint
Rineuune
LuF
EWC
LF
MAS
Propeeed
149
13
Figure 5: A' across R-MNIST (left), Seq-CORe50 (middle), and PACS (right) when IPC 10 (first row) and IPC = 20 (second row).
Distillation Cost Analysis. In Table we analyzed the distilla- tion cost of the proposed method compared that of other base- lines, evaluated on the PACS dataset when IPC=10. The pro- posed method incurs higher distillation cost compared to other baselines due to the addi- tional process of bi-level opti- mization. However, it is important note that DD is fundamentally offline procedure executed only once prior to model training, and thus does not incur the complexity associated with real-time processing. In addition, number iterations as 1,000 during the distillation process. Note that, even with a smaller number of iterations, the proposed method (Proposed (1,000 iter)) still outperforms other baselines signifi- cantly, while achieving the lowest distillation cost.
Methed
(s) C
Performance (9)
A
Total
A' (1)
Fr (C)
Finetune
112.2
102.1
101.9
94.6
410.7
26.9a08
44.3a03
LwF
109.4
145.2
130.4
139.0
524.0
35,0s0a
13.0a03
EWC
112.5
121.9
109.8
114.1
458.3
26.4s0.7
43.7=00
LF
103.8
151.0
143.9
141.4
540.0
27.4
42.9224
103.6
425.8
35.0a17
1.
Progosed
111.4
342.5
334,2
335.1
1,123.3
48.0
11.211x
Proposed (1,000 iter)
37.5
109.6
99.6
97.4
344.1
4633ta
8.5
Table 2: Distillation cost analysis PACS under IPC=10.
conducted an experiment to scale the distillation cost by adjusting the
5.3 ABLATION STUDY
To assess the effectiveness of our method, we conduct ablation studies on R-MNIST, which has the longest sequence among the three datasets. To examine each component with respect to the stability-plasticin trade-off, we split R-MNIST into four segments of five sequential domains (D1:5 D6:10 D 16:220 and analyze the average accuracy on each segment. Table reports results for IPC = 10 and IPC = 20. denote by Af's the average accuracy on D1:5 after dis- tillation of the 7-th domain, where AT, 5-14I El= af We observe that the Finetune performs poorly on earlier segments due to catastrophic forgetting. Adding Latable improves performance on recent segments, but the earliest segment ATTs degrades. This suggests that when the sequence length i' large, Latable mainly preserves recent knowledge but struggles to retain information from earlier domains. Moreover, the most recent segment Af6:20 also degrades TPC 20, suggesting that Lsttble can hinder the plasticity of the synthetic dataset. In contrast, applying the asymmetric update strategy via bi-level optimization improves performance across all segments, including Af.s and Af indicating that method effectively balances stability and plasticity.
Moreover, to verify the effectiveness of the proposed bi-level optimization method, we implement a simple alternative that assigns per sample ā; and B,. The most naive approach is assign linearly decreasing and increasing values to a and 3, for the sample indexed by in the synthetic dataset, respectively. Specifically,
define ā; amax N (a Qmin)) and 3: Smin + Xー (Bmax-
8min), where E {O...., N - 13. The results are shown in the third row Table 3 and we observe that this linear assignment fails to consistently improve performance across all segments, implying that our bi-level optimization method more effectively assigns ā; and 8, than the linear assignment scheme.

---

Published as a conference paper at ICLR 2026
IPC
10
20
Method
Af.
A6:10
Af1:15
A16:20
Afs
A
A
A18:20
Finetune
33,4
19.411.1
23.7:26
80.9166
31.3=20
19.7
31.0148
85.613M
+ Listable
26.9=0.7
27.5+22
59.3+29
87.1+12
29,4=37
25.5+27
45.0235
77.5+45
Asym. (Linear)
31.0=19
33.4.13.9
56.618.3
82.7406
27.3
27.3
48.7
81.1426
Asym. (Bi-level)
36
43.8-47
76.3+33
94.54103
32.9
34.8
72.9
95.24001
Table Ablation study on R-MNIST when IPC 10 and IPC 20. Asym. denotes the proposed
asymmetric update method, with two strategies: linear assignment and the proposed bi-level opti- mization.
3
3
3
3
3
3
(a)
C
6
6
5
5
5
7
7
A
4
/
/
/
1
1
3
3
3
3
3
3
3
3
(b)
5
5
5
5
5
5
5
5
7 7
7
7
Figure 6: Synthesized images on R-MNIST for classes 3, 5, and across (a) and (b) show images with the smallest and largest values of a; 3; for each class each I, respectively.
We also visualize synthesized images prioritized for stability and plasticity, respectively. Specifi- cally, samples with the smallest values of ā; - 33 are regarded as plasticity focused, whereas those with the largest values are stability focused. Figure shows results on R-MNIST for classes 1, 3, 5, and 7 across domains. (a) displays plasticity focused samples, which mainly capture knowledge from the most recent domain and exhibit the domain specific rotation angles. By contrast, stability focused samples in (b) appear more consistent across domains, indicating preservation of earlier knowledge. These observations support that our bi-level optimization method assigns ā; and 3 effectively to balance stability and plasticity.
CONCLUSION
In this work, introduced a novel problem setting, Domain Incremental Dataset Distilla- tion (DIDD), which aims to continually distill sequence of datasets from different domains into single synthetic dataset. By maintaining fixed-size synthetic dataset across time, DIDD keeps both storage and training costs efficient regardless of the number of datasets. Since exisiting DD methods only focus on the plasticity of the synthetic dataset, they suffer from catastrophic forget- ting. To address this problem, we introduced stability loss that preserves knowledge from earlier datasets. Moreover, to balance between stability and plasticity, we proposed an asymmetric update strategy that assigns different update rates for plasticity and stability to each sample. To effectively determine those update rates, we presented a bi-level optimization method based on meta- meta-learning. It encourages each sample to be specialized in either stability or plasticity, effectively reducing their inherent conffict. Extensive experiments validated the effectiveness of our method, showing consis- tent improvements over existing continual learning methods adapted the DIDD task.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENTS
This work was supported in part by the National Research Foundation of Korea (NRF) grant funded by the [Ministry of Science and ICT (MSIT) under Grant RS-2024-00392536, in part by the Insti- tute of Information and Communications Technology Planning and Evaluation (IITP) grants funded by the Korean Govermment (MSIT), including the Leading Generative Artificial Intelligence (AI) Human Resources Development Program under Grant the Artificial Intelligence Graduate School Program [Ulsan National Institute of Science and Tech- nology (UNIST)) under Grant RS- AS-2200--1201333 and in part by the Al Star Fellowship Program (UNIST) under Grant RS-2025-25442824
in part by
REFERENCES
Rahaf Aljundi, Francesca Babiloni. Mohamed Elhoseiny, Mareus Rohrbach, and Tinne Tuytelaars. Memory aware synapses: Learning what (not) to forget. In Proceedings of the European confer- ence on computer vision (ECCV), pp. 139-154, 2018.
George Cazenavette, Tongzhou Wang, Antonio Torralba, Alexei Efros, and Jun-Yan Zhu. Dataset distillation by matching training trajectories. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10718-10727, 2022.
Ze Chai, Zhipeng Gao, Yijing Lin, Chen Zhao, Xinlei Yu, and Zhiqiang Xie. Adaptive backdoor attacks against dataset distillation for federated learning. In IEEE International Conference Communications (ICC). pp. 4614-4619, 2024.
Terrance Devries and Graham W. Taylor. Improved regularization of convolutional neural networks with cutout. arXiv preprint arXiv: nen::170..05555 2017.
Tian Dong, Bo Zhao, and Lingjuan Liu. Privacy for free: How does dataset condensation help privacy? Proceedings of the International Conference on Machine Learning (ICML), pp- 5378-5396 2022.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. International conference on machine learning, pp. 1126-1135. PMLR, 2017.
Robert French. Catastrophic forgetting in connectionist networks. Trends in cognitive sciences, 3(4):128-135, 1999.
Jack Goetz and Ambuj Tewari. Federated learning via synthetic data. Ttl::2008.044883 2020.
arXiv preprint
Jianyang Gu, Kai Wang, Wei Jiang, and Yang You. Summarizing stream data for memory-restricted online continual learning. In Proceedings of the AAAI Conference Artificial Intelligence (AAAI), 2024.
Ziyao Guo, Kai Wang, George Cazenavette, Hui Li, Kaipeng Zhang, and Yang You. Towards lossless dataset distillation via aifficultty-ligned trajectory matching. In Proceedings of the International Conference on Learning Representations (ICLR), 2024.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common cor- ruptions and perturbations. ICLR, 2019.
Ching-Y Yi Hung, Cheng- Hao Tu, Cheng-En En Wu, Chien-Hung Chen, Yi-Ming Chan, and Chu-Song Chen. Compacting, picking and growing for unforgetting continual learning. Advances in neural information processing systems, 32, 2019.
Yuqi Jia, Saeed Vahidian, Jingwei Sun, Jianyi Zhang, Vyacheslav Kungurtsev. Neil Zhenqiang Gong, and Yiran Chen. Unlocking the potential of federated learning: The symphony of dataset distillation via deep generative latents. In Proceedings of the European Conference on Computer Vision (ECCV), 2024.
Heechul Jung, Jeongwoo Ju, Minju Jung, and Junmo Kim. fess-torge learning in deep neural networks. arXiv preprint arXiv: 160..00/2,, 2016.

---

Published as a conference paper at ICLR 2026
Hyyn Kim, Jinuk Kim, Seong Joon Oh, Sangdoo Yun, Hwanjun Song. Joonhyun Jeong, Jung- Woo Ha, and Hyun Oh Song. Dataset condensation via efficient yythetic-data parameterization. In International Conference on Machine Learning, pp. 11102- 11118. PMLR, 2022.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Bard Barwiiskka, et al. Overcom- ing catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):3521-3526, 2017.
Hugo Larochelle, Dumitru Erhan, Aaron Courville, James Bergstra, and Yoshua Bengio, An empir- ical evaluation of deep architectures on problems with many factors of variation. In Proceedings of the 24th international conference on Machine learning, pp.
2007.
Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M. Hospedales. Deeper, broader and artier domain generalization. In Proceedings of IEEE International Conference on Computer Vision (ICCV), Oct 2017.
Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE transactions pattern analysis and machine intelligence, 40(12):2935-2947, 2017.
Vincenzo Lomonaco and Davide Maltoni. Core50: a new dataset and benchmark for continuous object recognition. In Conference robot learning, pp. 17-26. PMLR, 2017.
David Lopez-Paz and Mare' Aurelio Ranzato. Gradient episodic memory for continual learning. Advances in neural information processing systems, 30, 2017.
Wojeiech Masarczyk and Ivona Tautkute. Reducing catastrophic forgetting with learning on syn- thetic data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition (CVPR), Workshop, pp. 4321-4326, 2020.
Michael McCloskey and Neal J Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, volume 24, pp. 109-165. Elsevier, 1989.
Dmitry Medvedev and Alexander D' D'yakonov. Learning to generate synthetic training data using gradient matching and implicit differentiation. In Proceedings of the International Conference on Analysis of Images, Social Networks and Texts (AIST), pp. 138-150 2021.
David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy Lillicrap, and Gregory Wayne. Experience replay for continual learning. Advances in neural information processing systems, 32, 2019.
Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. arXiv preprint arXiv:1606.04677 2016.
Haizhou Shi and Hao Wang. A unified approach to domain incremental learning with memory: Theory and algorithm. Advances in Neural Information Processing Systems, 36:15027-15059, 2023.
Felipe Petroski Such, Aditya Rawal, Joel Lehman, Kenneth Stanley, and Jeffrey Clune. Genera- tive teaching networks: Accelerating neural architecture search by learning to generate synthetic training data. In Proceedings of the International Conference on Machine Learning (ICML), pp- 9206 9216,921,,111 2020.
Minh-Tuan Tran, Trung Le, Xuan-May Le, Thanh-Toan Do, and Dinh Phung. Enhancing dataset distillation via non-critical region refinement. In CVPR, pp. 10015-10024, 2025.
Gido M Van de Ven, Tinne Tuytelaars, and Andreas S Tolias. Three types of incremental learning. Nature Machine Intelligence, 4(12):1185-1197, 2022.
Kai Wang, Bo Zhao, Xiangyu Peng, Zheng Zhu, Shuo Yang. Shuo Wang, Guan Huang, Hakan Bilen, Xinchao Wang, and Yang You. CAFE: Learning to condense dataset by aligning features. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 12195-12205. 2022.
12

---

Published as a conference paper at ICLR 2026
Kai Wang, Zekai Li, Zhi-Qi Cheng, Samir Khaki, Ahmad Sajedi, Ramakrishna Vedantam, Kon- stantinos N Plataniotis, Alexander Hauptmann, and Yang You. Emphasizing discriminative fea- tures for dataset distillation in complex scenarios. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025.
Tongzhou Wang, Jun- Yan Zhu, Antonio Torralba, and Alexci A. Efros. Dataset distillation. arXiv preprint arXiv:1811. 2018.
Enneng Yang, Li Shen, Zhenyi Wang, Tongliang Liu, and Guibing Guo. An efficient dataset conden- sation plugin and its application to continual learning. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), 2023.
Hansong Zhang, Shikun Li, Pengju Wang, and Shiming Zeng, Dan Ge. M3D: Dataset condensation by minimizing maximum mean discrepancy. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), 2024.
Bo Zhao and Hakan Bilen. Dataset condensation with gradient matching, In Proceedings of the International Conference Learning Representations (ICLR), 2021a.
Bo Zhao and Hakan Bilen. Dataset condensation with differentiable siamese augmentation. In Proceedings of the International Conference on Machine Learning (ICML), pp. 12674-12685, 2021b.
Bo Zhao and Hakan Bilen. Dataset condensation with distribution matching. In Proceedings of the IEEF/CVF Winter Conference on Applications of Computer Vision (WACV), pp. 6514-6523, 2023.
Runkai Zheng, Vishnu Asutosh Dasu, Yinong Oliver Wang. Haohan Wang, and Fernando De la dataset distillation. In Proceedings of the
Torre Improving noise efficiency in IEEE/CVF International Conference Computer Vision (ICCV), 2025.
A APPENDIX
Large language models (LLMs) were used solely for language refinement, and all research content was generated entirely by the authors.
A.1
ALGORITHH
Algorithm summarizes the overall procedure of our proposed method.
A.2 SPECIFICATION OF DATASETS
We summarize the specifications of each dataset in Table We also visualize examples of R- MNIST, CeqeCO and PACS in Figure 7 and Each column corresponds different domain and each row corresponds to different class.
Dataset
R-MNIST
Seq-CORe50
PACS
# of classes
# of domains
20
11
4
Average of images per domain (train)
60,000
11,990
2,244
Average of images per domain (test)
10,000
2,999
715
Total # of images (train)
1,200,000
131,888
8.977
Total # of images (test)
200,000
32,987
1,014
Table 4: Specification of datasets.
13

---

Published as a conference paper at ICLR 2026
Algorithm Asymmetric Synthetic Data Update for DIDD
Require: Datasets (Dr) with label set IPC N; randomly initialized model d; total iterations
T; learning rates Y/a, V/a 113; B bounds (Omin (max), (pmin; fmax); hyperparameters λa, As-
1: for = 1 to 1 do
2: D' <- D freeze 1
3: for iter to / do
4:
for each class C do
5:
Sample class e mini-batch from Dz
6:
Compute ((x'), F()²), F()+-11 by d.
7:
Compute plastic, Listable using Eqs. and
8:
Get gradients ip,ig. via Eq
9:
Meta update: 2 Ameta,s by Eq. 9
Compute meta-losses Lineta (Eq. 10p and penalty Speenalty 3 (Eq. 12
11:
Update scale parameters: 8 by Eq 14
Asymmetric update: a by Eq.
13:
end for end for
14:
15:
end for
16: return D
2
2
3
3
3
Figure 7: Examples for each domain in R-MNIST.
Figure 8: Examples for each domain in Seq- COR-O.
Figure 9: Examples for each domain in PACS.
A.3 COMPARISON OF STORAGE COST
We compare storage time R-MNIST, -c0-,, and PACS with IPC 20, assuming new dataset arrives at each time step t. Figure plots time f on the z-axis and storage in MB on log scale on the y-axis. As / increases, the size required store the original datasets grows rapidly.
14

---

Published as a conference paper at ICLR 2026
Conventional DD greatly reduces storage, yet its efficiency diminishes time because distilled sets accumulate for each domain. In contrast, our method maintains a fixed size synthetic dataset across time, so the storage remains constant regardless of t.
Original Datasets
Existing
incremental DD
Figure 10: Comparison of storage cost on R-MNIST (left), Seq- ce0- (middle), and PACS (right)
when IPC = 20. x-axis indicates and y-axis indicates the storage cost in MB (log scale).
A.4 EFFECT OF DIFFERENT SEQEUNCE ORDERS
We conducted additional experiments with different orders of training sequences. Table reports the results with randomly generated two different orders (2, 3), in addition to the default order (1), for R-MNIST and PACS datasets. We observe that the proposed method consistently outperforms the baselines with different sequence orders, demonstrating its robustness to the sequence orders.
Datasct Order
R-MNIST 2
PACS
Metelo
1F (t)
FT(() (1)
AT
(1)
FF (4)
(t)
(1)
AUT (t)
JTT (1)
(?)
F (L)
A (t)
FT (4)
Finctune
39.422
56.7a20
42.7a13
53,6e16
40,5e03
56,0s04
26.9568
33,2493
35.409
24.0ans
25.7=20
32.4-20
50.5
35.0a1a
49:13
Proposed
62.7420
33.1+20
62.5=20
33.3e10
63.7=24
32.1
48.0a11
8.4s1s
48.0e11
8.2
30.7=00
7.1+14
Table 5: Performance comparison on R-MNIST and PACS datasets under IPC=10 with different
dataset orders.
A.S HYPERPARAMETERS
For simplicity, most hyperparameters are shared across datasets and IPC settings. We summarize them in Table 6 Only for PACS, we set the batch size to 64 because there exists a class with fewer than 128 images. The image update learning rate 7h is set equal to the IPC. For example, 7 when IPC = and n,, = 10 when IPC = 10. following (Zhang et al. 2024)
Iteration
Batch size
flo
1)8
Aa
Orainn
Omaa
Smax
3,000
128"
same IPC
le-2
le-2
le-4
le-4
2
2
For PACS only, the batch size 64.
Table 6: Hyperparameter settings.
A.5.1 ANALYSIS ON À AND A3
We analyze the impact of 7a and Ag. Table 7epport results on R-MNIST with IPC = 10. Increas- ing Àa with 7, improves performance updates for plasticity. As a result, the synthetic dataset more focuses on stability. Conversely, in- creasing λą with fixed A improves performance recent segments, since update rates for stability shrinks while increasing plasticity relatively. These results support that our bi-level optimization method works intended to balance between stability and plasticity by adjusting X, and As-
earlier segments, because a stronger penalty on 3; reduces
15

---

Published as a conference paper at ICLR 2026
IPC
10
IPC
10
A
Ag
A
A:10
A
A 16:20
Au
Ag
AT
A
A 11:15
A16:20
5e-5
36.0
44.5
75.6
95.1
5e-5
39,9
42.0
76.2
94.1
1e.4
le-4
36.3
43.8
76.3
94.5
le-4
le-4
36.3
43.8
76.3
94.5
5c-4
37.5
48.8
79.0
93.9
5c-4
35.1
41.5
78,9
95.2
Table 7: Perfomance on R-MNIST when IPC = 10 to study the impact of X and As-
A.6
SCAILABILITY TO LARGER BACKBONES AND DATASET
Table 8 presents the performance of the proposed method when scailing up to larger back- bone (ResNet) and higher image resolution (128x 128). Seq-CORe50 dataset can be regarded as medium scale one, which is originally in 128x 128 resolution and has 50 classes. The results show that the proposed method outperforms MAS method, which the second-best baseline on most cases, with clear margins, thereby demonstrating its effectiveness and scalability to larger back- bones and datasets.
Resolution Backbone
32x32
128x 128
ConvNet
ResNet
IPC
10
10
Metric
AT (t)
F (4)
AT (7)
JF (L)
Finetune
25.4
79.9
26.5
65.3
MAS
26.4
60.3
26.5
11.7
Proposed
55.4
38.8
61.7
28.2
Table Performance on Seq-CORe50 under IPC=10 with different backbone and image resolution.
We additionally evaluated the performance of the proposed method on six ImageNet-based subsets: ImageNette, ImageWooff ImageMeow, ImageFruit, Image Yellow, and ImageSquawk, under IPC=1. Since ImageNet does not provide explicit domains, we constructed five domain sequences by se- lecting five corruption types from ImageNet-C Henddrycks Dietterich (2019) (defocus blur, fog, speckle noise, contrast, and frost), We tested two high-capacity backbone models, ResNet and ViT. The results are summarized in Table g Although VIT has larger model capacity, its dataset dis- tillation performance is generally lower than that of ResNet, which aligns with the observation in NRR-DD Tran al. (2025) that stronger backbones not necessarily yield better distillation per- formance. Nevertheless, the proposed method outperforms the Finetune baseline in most cases and both backbone models, demonstrating that our framework generalizes well to large-scale datasets and modern architectures.
Dataset Backbone
ImugeNetie
Image Woof
ImageMeow
ResNet
VIT
ResNet
IPC
Metric
A7 (1)
5 (4)
A7 (f)
JT (1)
AT (t)
J (4)
A" (t)
JT (L)
A (t)
(4)
A (t)
(U)
40.6
18.6
25.2
15.6
Finetune Proposed
32.4
10.0
18.0
9.8
21.6
1.3
12.4
6,3
21.0
9,3
14.8
8.5
42.8
19.5
23.0
0.5
15.8
2.3
18.2
6,3
15.2
Dataset Backbone
ImageFruit
Image Yollow
ImageSquawk
ResNet
VIT
ResNet
VIT
ResNet
ViT
IPC
Metric
AT (1)
F (4)
A" (1)
JT (4)
AT (t)
F (4)
A" (t)
FT (1)
A" (t)
FT (4)
A" (t)
J (U)
27.2
27.8
Finetune Proposed
25.0
4.3
20.2
7.8
43.0
4.3
20.6
14.5
36.6
6.0
25.2
2.3
31.2
22.2
3.5
43.4
0.0
28.2
8.0
42.8
0.5
27.6
1.3
Table Performance on ImageNet-C: subsets, under IPC=1 with two different backbones.
A.7 OPTIMIZATION DIFFICULTY IN META-LEARNING FRAMEWORK
Conventional meta-learning frameworks often encounter optimization difficulties arising from the nested structure of their objectives, where large number of inner loop updates may lead to vanishing
16

---

Published as a conference paper at ICLR 2026
exploding gradients. Let wy be the initial parameters that aim to optimize, Lir be the training loss for inner loop updates, and Lmeta the meta-loss for outer loop updates. In the inner loop, the parameters are updated as:
ut = W:-1 V/ w. (w-
t= 1,,,.....
(20)
where y is the inner loop learning rate and T is the number of inner loop updates. the outer loop, wo updated based on the meta-loss with respect to wr:
Wo wo VV wo (WT)
(21)
where 3 is the outer loop learning rate. Here, the gradient V Vw, Cmeta(Wr)) can be expressed using the chain rule as:
II (r (o
V we Lineta (WTT) = V wy metta (WT): t=1
(22)
We can observe that the gradient can either vanish or explode with increasing T. However, the proposed method employs only a single inner loop update (7 1), and thus effectively mitigates these optimization challenges, leading to more stable and efficient training.
A.8 LIMITATIONS AND FUTURE WORK
this work, we focus on the domain incremental setting, where datasets from different domains arrive sequentially and share the same label space. However, approach does not yet address other continual learning settings, such as class incremental learning where new classes are introduced over time. Extending the DIDD framework to handle the arrival of new classes is an important direction for future work. We will explore strategies to expand the synthetic dataset to incorporate new classes while preserving knowledge of existing ones, aiming for more versatile and widely applicable dataset distillation framework.
A.9 MORE VISUALIZATION
We visualize more synthesized images on R-MNIST, Seq-CORe50, and PACS comparing Finetune and our proposed method in Figures
12 and B respectively.
17

---

Published as a conference paper at ICLR 2026
3
5
3
O
Domain
W
S
S
S
S
5
5
3
9
3
6
4
5
55
5
5
5
Domain
2
2
JC
6S
C
Domain 15
7
Domain
G
(a) Finetune
6
T
Domain
S
S
7
9
G
Q
7
3
R
%
Domain
5
S
A
Q
3
ON
C
ON
RO
SSY
Domain 15
NO
ES
NO
V
/
3
S
MA
Z
Domain 20
S
S
S
S
Z
A
4
(b) Proposed
Figure 11: Synthesized images on R-MNIST for (a) Finetune and (b) the proposed method.
18

---

Published as a conference paper at ICLR 2026
Domain
Domain
Domain
(a) Finetune
Domain
Domain
Domain 10
(b) Proposed
Figure 12: Synthesized images on Seq-CORe50 for (a) Finetune and (b) the proposed method.
19

---

Published as a conference paper at ICLR 2026
Domain
Domain 2
Domain
(a) Finetune
Domain
Domain
Domain
(b) Proposed
Figure 13: Synthesized images on PACS for (a) Finetune and (b) the proposed method.
20


Reviews:
[{"review_id": "BF4MSU9fX3", "reviewer": "Reviewer_XAFs", "summary": "This paper investigates Domain Incremental Dataset Distillation, where data arrives sequentially. The authors propose Asymmetric Synthetic Data Update, which introduces a stable loss that constrains representation during updates. Additionally, they introduce a meta-learning method and regularization term to balance the weights of the two losses and prevent them from growing excessively.", "questions": "1. Does the domain sequence order affect the results?\n2. Could you provide non-DD fine-tuning results to help us understand how the gap changes between DD and full incremental domain training?\n3. What’s the training cost comparing to baselines?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-written and easy to follow\n2. The setting is novel", "weaknesses": "1. Meta-learning makes optimization more difficult.\n2. Constraining the label space to be the same makes the setting less general.\n3. DD is already challenging to train. Adding meta-learning raises concerns about training instability and tuning difficulty.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BCm9Osz4gE", "reviewer": "Reviewer_sWfL", "summary": "The paper proposed a new problem in dataset distillation: Domain Incremental Dataset Distillation (DIDD). The problem assumed that a sequence of domain-shifted datasets (classification datasets that shared the same label space). The difference between this problem and the Continual Learning is the data storage budget is fixed in DIDD. The authors introduce a stability loss to preserve prior knowledge and an asymmetric per-sample update learned via bi-level meta-optimization. The authors conducted experiments to verify their method's effectiveness.", "questions": "1. What is the definition of Domain Incremental Dataset Distillation (DIDD)? It was defined as a problem in the abstract, but it was also defined as a framework in contributions. \n\n2. What is the main contribution of this paper? The proposed new problem DIDD? or the new proposed method to address the DIDD problem? \n\n3. Why are there no baselines listed under dataset distillation? Any dataset distillation (DD) method that condenses an accumulated dataset can serve as a valid baseline.\n\n4. How is the computational cost ? There should be an experiments to dicuss the computational cost.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The authors propose a new problem setting of dataset distilaltion.\n\n2. The proposed method seems address the proposed problem well.", "weaknesses": "1. The paper defines a new hybrid setting DIDD by combining dataset distillation and continual learning. However, this formulation appears somewhat artificial and tailored to the proposed method, rather than motivated by a clearly established real-world need or widely recognized benchmark.\n\n2. The paper lacks the baseline of dataset distillation, which condense the accumulated datasets directly. Additionally, the performance comparision is not so fair as the continue learning baselines are not designed for the new DIDD setting. \n\n3. Even in R-MNIST dataset, there is a huge performance loss compared to the whole dataset.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mqVrPsE934", "reviewer": "Reviewer_1Hqu", "summary": "The authors identified that the existing dataset distillation assumes the availability of the entire dataset, while in reality, real datasets are collected incrementally over time. To solve this issue, the authors propose Domain Incremental Dataset Distillation to continuously distill datasets from multiple sources into a single synthetic dataset. Additionally, traditional DD methods sequentially process arriving datasets and cause catastrophic forgetting. The authors propose an Asymmetric Synthetic Data Update strategy to balance the stability-plasticity trade-off:  a bi-level optimization approach based on the meta learning framework to estimate the optimal update rates. The authors perform evaluations to demonstrate the mitigation of the catastrophic forgetting problem and the efficacy of DD in their proposed approach.", "questions": "Please see \"Weaknesses.\" Mainly I'm concerned why aren't $\\bar{\\alpha}_i$ and $\\bar{\\beta}_i$ updated jointly since they are used jointly?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "+ The idea is simple but effective, as demonstrated by the evaluation results that the proposed work achieves the state-of-the-art level performance. \n+ The idea is clearly and effectively formulated in mathematical terms, and the writing is very straightforward and easy to follow.", "weaknesses": "- While there's beauty in simplicity, the idea is \\textit{too} simple. For example, Eq 12 is just a standard gradient-based optimization with the defined losses.\n- It seems to me (please correct me if I'm wrong) that $\\bar{\\alpha}_i$ and $\\bar{\\beta}_i$ are separately optimized yet they are used jointly (Eq 9). Why aren't they updated jointly instead?\n- The theoretical evidence on why this method is working is missing. The existing mathematical formulation is simply descriptive of the approach, not highlighting any insights on why the approach works.\n- In a paper that is 9 pages, only 2 pages are the methodology. This balance is a little bit off as I was expecting a longer methodology section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yP67zhNLC0", "reviewer": "Reviewer_7mfs", "summary": "This paper introduces Domain Incremental Dataset Distillation (DIDD) — a new problem setting where synthetic data must be updated continuously as new domains arrive, under a fixed memory budget.\nUnlike conventional dataset distillation, which assumes access to all data at once, DIDD requires the synthetic set to maintain past knowledge while integrating new information, analogous to continual learning but at the *data level* rather than *parameter level*.\n\nTo tackle this, the authors propose Asymmetric Synthetic Data Update, a bi-level optimization framework where each synthetic sample learns its own *stability–plasticity* trade-off coefficients. These coefficients adaptively balance gradients from old and new domain objectives, allowing the synthetic dataset to evolve without catastrophic forgetting.\n\nExperiments on R-MNIST, Seq-CORe50, and PACS demonstrate significant gains and reduced forgetting over standard distillation and continual learning baselines. Ablation analyses show the importance of asymmetric updates and bi-level learning for maintaining cross-domain knowledge.", "questions": "1. How does training time scale with dataset size or number of domains? Can the authors provide runtime or complexity analysis?\n2. Could the asymmetric update be approximated with parameter-efficient strategies (e.g., low-rank updates, meta-network sharing) to improve scalability?\n3. How would the approach behave on *class-incremental* or *multi-modal* tasks rather than domain-incremental ones?\n4. Is it possible to replace full bi-level optimization with a single-level approximation or gradient truncation without significant performance loss?\n5. Could this framework generalize to larger backbones (e.g., ResNet, ViT) or to non-vision modalities (text or graph distillation)?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* **Novel problem definition:** The paper bridges the gap between dataset distillation and continual learning, providing a fresh research direction.\n* **Conceptually elegant solution:** The asymmetric update mechanism offers an intuitive and interpretable way to balance stability and plasticity at the data level. The per-sample α/β weighting is original and theoretically motivated.\n* **Strong empirical gains on small-scale benchmarks:** The proposed method outperforms both conventional dataset distillation methods (e.g., MTT, DSA) and continual learning baselines (e.g., EWC, LwF, MAS), validating the effectiveness of the asymmetric strategy.", "weaknesses": "* **Scalability concerns:** The method relies on *bi-level optimization for each sample*, which is computationally expensive. The current experiments use small networks (3-layer ConvNet) and small datasets (R-MNIST, PACS). It is unclear whether the approach can scale to modern architectures (ResNet, ViT) or larger datasets (CIFAR-100, ImageNet).\n* **Lack of large-scale validation:** The paper would be significantly stronger if it included experiments or runtime analysis on medium-scale settings to demonstrate practical feasibility.\n* **Limited diversity of tasks:** The benchmarks are all domain-incremental vision tasks; it would be interesting to see if this framework can handle class-incremental or multimodal distillation scenarios.\n* **Ablation on efficiency missing:** Although the method is theoretically well-motivated, there is no quantification of time or memory overhead compared to one-shot distillation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)