Multimodal Dataset Distillation via Phased Teacher Models
Authors: Shengbin Guo
Hang Zhao
Senqiao Yang
Chenyang Jiang
Yuhang Cheng
Xiangru Peng
Rui Shao
Zhuotao Tian
Venue: ICLR 2026 Poster (2026)
Keywords: Dataset Distillation
multimodal learning
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Abstract: Multimodal dataset distillation aims to construct compact synthetic datasets that enable efficient compression and knowledge transfer from large-scale image-text data. However, existing approaches often fail to capture the complex, dynamically evolving knowledge embedded in the later training stages of teacher models. This limitation leads to degraded student performance and compromises the quality of the distilled data. To address critical challenges such as pronounced cross-stage performance gaps and unstable teacher trajectories, we propose Phased Teacher Model with Shortcut Trajectory (PTM-ST)—a novel phased distillation framework. PTM-ST leverages stage-aware teacher modeling and a shortcut-based trajectory construction strategy to accurately fit the teacher’s learning dynamics across distinct training phases. This enhances both the stability and expressiveness of the distillation process. Through theoretical analysis and comprehensive experiments, we show that PTM-ST significantly mitigates optimization oscillations and inter-phase knowledge gaps, while also reducing storage overhead. Our method consistently surpasses state-of-the-art baselines on Flickr30k and COCO, achieving up to 13.5\% absolute improvement and an average gain of 9.53\% on Flickr30k. Code: \url{https://github.com/Previsior/PTM-ST}.

Published as a conference paper at ICLR 2026
MULTIMODAL DATASET DISTILLATION VIA PHASED TEACHER MODELS
Shengbin Guo Hang Zhao" Senqiao Yang', Chenyang Jiang'
Yuhang Cheng', Xiangru Peng', Rui Shao', Zhuotao Tian'
Haarbin Institute of Technology, Shenzhen The Chinese University of Hong Kong
shengbingue 202 02w.022220m022e 200110431@stu. hwt.:sttt..i....u.....
ABSTRACT
Multimodal dataset distillation aims to construct compact synthetic datasets that enable efficient compression and knowledge transfer from large-scale image-text data. However, existing approaches often fail to capture the complex, dynamically evolving knowledge embedded in the later training stages of teacher models. This limitation leads to degraded student performance and compromises the quality the distilled data. To address critical challenges such as pronounced cross-stage performance gaps and unstable teacher trajectories, we propose Phased Teacher Model with Shortcut Trajectory (PTM-ST)-2 novel phased distillation frame- work. PTM-ST leverages stage-aware teacher modeling and a shortcut-based trajectory construction strategy to accurately fit the teacher's learning dynamics across distinct training phases. This enhances both the stability and expressive- ness of the distillation process. Through theoretical analysis and comprehensive experiments, we show that PTM-ST significantly mitigates optimization oscilla- tions and inter-phase knowledge gaps, while also reducing storage overhead. Our method consistently surpasses state-of-the-ar baselines on Flickr30k and COCO, achieving up to 13.5% absolute improvement and an average gain of 9.53% on Flickr30k. Code: hitps:
INTRODUCTION
Large-scale multimodal models such CLIP (Radford et al., 2021), BLIP (Li et al., 2022), and Flamingo (Alayrac et al., 2022) have established image-text pairs as a fundamental aspect of visual- language pre-training (VLP). Their successors, e.g., BLIP-2 (Li et al., 2023) and LLaVA (Liu al., 2023), further extend this paradigm to multimodal large language models (MLLMs) (Jia et al., 2021; Chen al., 2023c). However, the application of such extensive multimodal data incurs substantial storage as well as computational overhead (Hoffmann et al., 2022; Kang et al., 2024), highlighting an urgent need for efficient data utilization and improved model training efficiency.
Dataset Distillation (DD) (Wang et al., 2018; Lei & Tao, 2023) offers a promising solution to this
challenge by synthesizing compact surrogate dataset that compresses the dataset by orders of magnitude without hurting performance (Zolfaghari et al., 2021; Lin & Hu, 2022). Despite notable progress in unimodal tasks across text (Li & Li, 2021), video (Wang et al., 2024) and graph (Xu et al., 2023) domains, directly extend DD to multimodal scenarios presents considerable difficulties.
Although a few recent studies (e.g., MIT-VLL (Wu et al., 2023) and LoRS (Xu et al., 2024)) have attempted to extend dataset distillation methods to multimodal tasks, these approaches primarily focus on directly adapting existing unimodal distillation strategies to multimodal scenarios or en- hancing cross-modal alignment through the incorporation of low-rank similarity matrices. Overall, such methods largely remain at a superficial level. emphasizing the design of data structures or dis- tance metrics, and concentrate mainly representation and alignment challenges in multimodal learning (Du et al., 2023; 2024). However, they fall short of investigating the underlying mecha- nistic differences between multimodal and unimodal settings in the context of dataset distillation, differences that likely govern the stability, efficiency, and ultimate effectiveness of any practical
"Equal contribution. *Corresponding author: tiaazhuotaaghi edu. en.

---

Published as a conference paper at ICLR 2026
distillation strategy. This gives rise fundamental scientific question: What the essential dis- tinctions between multimodal and unimodal dataset distillation tasks and how should we improve?
Key Observation. To explore the fun- damental distinctions between multimodal and unimodal dataset distillation, duct a comparative study under a unified MTT framework. Our results reveal that, teacher
Performance on Flickr30k
TR000100)
RAND
con-
$2.7
38.0
LOKS
MTTT-V
PTV-57
(2000)
unlike unimodal guidance remains effective throughout the whole training process- molless-multimodall dis- tillation benefits primarily from the early 20-30% of training epochs. As shown in Figure (a), although teacher model con- tinues to improve in later stages, using directly for distillation leads to a sharp drop in student performance.
(Rg5)
65.5
1.
(0/10/1100)
IRg10/200
00.1
13..
7.0
(R:26/000)
(Rg3((100)
TRi11200)
19.8
2044
36.6
27.5
26.3
TRS210/500)
64.8
TR@52200)
16.0
TRi5(5000
T8@10(200)
We hypothesize that this limitation stems from the inherent sparsity of multimodal data and the absence of explicit semantic constraints, which lead the teacher model to encode substantially different forms of knowledge different training stages. As a result, existing distillation methods struggle to effectively leverage the knowl- edge embedded in the teacher model across stages to support the continual learning of the student model. These findings underscore the necessity of developing distillation framework that can dynamically adapt to the evolving knowledge representation of the teacher model, thereby enabling student to absorb intermediate signals more effectively, mitigate representation drift, and achieve stable, continual improvements in downstream tasks.
(R:g1(500) 22
TRI211500)
(R0250000) 51.1
54.0 (Rg100550))
Figure I: PTM-ST has achieved far superior perfor- mance than other selection or distillation methods in different metrics Flickr30k.
Our Solution. To address the limitations of conventional distillation strategies in modeling the evolving dynamics of teacher models, we propose novel framework: Phased Teacher Modeling with Shortcut Trajectory (PTM-ST). Built upon the dual-loop architecture of the original MTT framework, PTM-ST introduces two core components: (1) Phased Teacher Modeling (PTM), which decomposes the overall distillation objective into a sequence of sub-tasks, each associated with a phase-specific teacher model that adapts to the semantic evolution across training stages; and (2) Shortcut Trajectory (ST), which constructs a smoothed and stabilized trajectory to guide student learning, mitigating the instability and noise often present in late-stage teacher updates.
By capturing stage-specific knowledge more precisely and ensuring stable knowledge transfer, PTM-ST significantly improves the quality of distilled data and facilitates more effective student training. As shown in Figure 1, we conducted extensive experiments on the Flickr30K (Plummer al., 2015) and MS-COCO (Lin al., 2014) datasets, validating the superior performance and robustness of our method compared to existing state-of-the-art approaches.
To summarize, our contributions are as follows:
We identify and analyze the phased knowledge gap in multimodal dataset distillation, validating the phenomenon through visualization and supporting theoretical analysis.
We propose the PTM-ST framework, combining a phased distillation strategy with trajectory endpoint matching, significantly improving distillation stability and performance.
Our method outperforms existing techniques in multimodal image- text retrieval distillation, push- ing the boundaries of multimodal distillation research and its applications.
2

---

Published as a conference paper at ICLR 2026
2
PRELIMINARIES
In this section, briefly review foundational concepts relevant to Section 2.1 establish back- ground for our work. Then, Section 2.2 details our key insights and design motivations.
2.1 BACKGROUND
Multimodal Dataset Distillation. Dataset distillation seeks to synthesize a compact, representa- tive subset that effectively approximates the statistical properties (Wang al., 2022; 2025c; Zhao & Bilen, 2023; Lee et al., 2022; Deng & Russakovsky, 2022) and training dynamics (Chen et al., 2023b; Du et al., 2024; Shao al., 2024a; Yin al., 2023) of large-scale datasets. Multimodal dataset distillation (MDD) extends this concept from unimodal data (e.g., images) to multimodal data (e.g., image-text pairs). Current approaches predominantl fall into meta-learning-basse et al., 2022) and distribution matooonng-ttttttaaaaaee based (Lee & Chung, 2024) strategies. While these methods perform well on unimodal benchmarks (Lei et al., 2024), their extension multimodal settings re- mains challenging due to the intrinsic complexity and heterogeneity of multimodal data (Chen et al., 2023a; Peng et al., 2024; Aljundi et al., 2019b;a), requiring further exploration.
(Zhou
The Baseline. We adopt LoRS (Xu et al., 2024) as the baseline for MDD, which based on the Match- Training-Traaeccors
(MTT) frame- work (Cazenavette al., 2022: Wu et al., 2023; Cui al., 2023b). MTT optimizes a synthetic dataset by aligning the parameter trajectories of models trained separately on the syn- thetic data and the real dataset D.
Remean
Epoch/
(a) R-mean Teacher Student
Gradient Directions
Concretely, a teacher model is trained on D for n epochs, producing a tra- jectory - = (0o, 0i,..., 0,) that captures the parameter evolution. Let T' be the lower bound and T be the upper bound on the sampling range of T, where T is generally set 0. The student model is then trained on synthetic dataset D match the teacher's trajectory from fr, minimizing the discrepancy between their parameter states after additional training steps. Formally, the objective is to minimize:
Figure 2: (a) shows that using late-stage, high- pelforriffrmgggggg teacher during distillation leads to a decline in student per- formance. (b) shows the directions of different matching range gradients (after PCA dimensionality reduction).
||0r+ 0r OT+ATi
CMTT (D,0r)
100 - 0T+
where 6r+z denotes the model parameters obtained by training steps on D starting from oy. and AT represents the length of matching range in r. Building upon this foundation, LoRS applied an augmentation technique to the synthetic dataset to enhance its expressive capability. More details about previous work can be found in Appendix A.
2.2 MOTIVATION
Knowledge gap across learning stages. Despite recent progress in multimodal dataset distilla- tion (MDD), existing methods predominantly rely on the early-stage checkpoints of the teacher model modcl-typically within the first 20-30% of training epochs (I* < 0.3n). However, shown in Figure (a), while the teacher's performance continues to improve in later epochs, directly incorpo- rating these late-stage checkpoints often leads to degraded student performance. This suggests that knowledge discrepancies emerge across different training stages, and the student model struggles to assimilate the teacher's late-stage stage knowledge effectively, compromising the distillation results.
To further investigate this phenomenon, inspired by DDGM (Zhao et al., 2020), we analyze the gradients propagated from the teacher the synthetic dataset various epochs. As illustrated
3

---

Published as a conference paper at ICLR 2026
Trained All Data
Stage
Stage 2
Stage P
Di
Dz
Dp
Dr
(a) Normal
(b) Phased Teacher Model
(c) Evaluation
(d) Shortcut Trajectory
Orr
Buudent Net
Stage
Stage
BT:
Original Trajectory
Original Match
Combine D, Dr.
8T.
Figure 3: (a) shows the conventional single-stage training with a fixed teacher model and uniform data use. In contrast, (b) depicts our Phased Teacher Model (PTM), which employs different teacher models across multiple training stages to distill knowledge to specific data subsets. (c) illustrates the aggregation of distilled subsets for final student evaluation. Additionally, (d) presents our Short- cut Trajectory (ST) strategy that dynamically generates stage-adaptive teacher models, improving distillation effectiveness and robustness.
in Figure 2 (b), the gradient norm increases progressively throughout training, implying intensified update signals. However, the gradient directions exhibit high variability and inconsistency, revealing substantial instability the learning signals over time. These observations highlight two critical challenges: pronounced stage-wise knowledge shifts and unstable optimization dynamics in later training botses-both of which hinder effective knowledge transfer in current MDD frameworks.
Limitations of existing methods. Existing multimodal dataset distillation (MDD) approaches (Xu al., 2024) typically rely on a single synthetic subset, which subjects the distilled dataset to persis- tently high-magnitude and unstable gradient updates. Such instability compromises the consistency and reliability of knowledge transfer, ultimately exacerbating the performance gap between teacher and student models. Moreover, recent findings (Guo al.. 2023; Chen et al., 2023b) highlight that the generalization capability of deep models emerges from their progressive assimilation of train- ing samples across different learning stages. This suggests that conventional single-subset MDD strategies are fundamentally constrained in their ability to capture the full spectrum of knowledge distributed throughout the training process. For more analysis, see Appendix B.
Motivated by these insights, we propose a phased distillation mechanism that enables the student model to incrementally align with the teacher's knowledge trajectory across successive learning stages. By decomposing the distillation process into multiple temporal phases, our approach facili- tates more stable gradient updates and improves the fidelity of knowledge transfer from the original dataset to the synthetic one, therefore enhancing the distillation effect.
METHOD
This section introduces the Phased Teacher Model (PTM) with Shorteut Trajectory (ST), ap- proach designed to address the challenge highlighted Section 2.2. The overall framework is illus- trated in Figure 3. Section 3.1 show the design of PTM, followed by the ST strategy in Section 3.2.

---

Published as a conference paper at ICLR 2026
Algorithm Distillation: Phased Teacher Model with Shortcut-Trajectory
Input: Real dataset X.Y. Size, iteration, matching range, interpolation endpoint of each subset: {(Np. pr Ap. T Ipt
[p)) }=1 p==1.Dcca parameter a,learning rate 7.Synthesis steps t, expert epochs AT'. Output: A series of synthetic datasets: D1, D........ Dp.
1: Generating synthetic subsets:
2: for porp= 1 ... Pd do
3: Initialize Xz. yiP by real data, Sp LN, (identity matrix). D. = (Fp: Sp: " 0 based on On and Ot,
4: Calculate Shortcut- Trajector (
5: for 2,...... Ip do
6: Sample an initial network parameter 0% {oP .p.... T
AP .
7: Train the network for t steps on D,, to 6
8:
Compute MTT loss LMTT |16 f 1/2 2/10//11- 0 AT 112
9:
Gradient descent: Xg x Rs 7x V X LMTT: Dp s Dp My V LMTT.
Sp + Sp - 1s V LMTT
10:
EMA: DJ + (Rp. D Sp), DJ + aD + (1- a)
11: end for
12: Dp + D//
13: end for
3.1 PHASED TEACHER MODEL.
As mentioned in the Section 2.2, simply switching the teacher model according to the training phase does not yield satisfactory results. To better capture stage-dependent learning dynamics and provide smoother guidance, we propose the Phased Teacher Model (PTM). This strategy progressively in- troduces different teacher models during training, allowing the distilled dataset to absorb as much information as possible from different teacher models each stage.
Specifically, as shown in Figure (b), we divide the distillation process into P stages, with each stage independently distilling small subset Dp. where p 1,2, .... P denotes the stage id.
In our method, we dynamically adjust the sampling range of the trajectory matching starting point. The sampling range for stage p is .... j. and this range evolves training progresses. This means that, at each stage, we use a different teacher model to guide the update of the distillation dataset, allowing each subset focus on distilling different aspects of knowledge. As analysed in Section 4.2, the union of all subsets, D -.....UU -UD, thus capture the complete training dynamics.
During testing, we adopt progressive training scheme. Specifically, the model is first trained on D,, then D., and on. This corresponds to the different stages of our distillation process, allowing students to gradually extract the teacher model's 's training dynamics.
Formally, We generated the dataset for stage p follows:
D- = arg min E TT, CPTM(Dp. 0P::
(2)
D,
where LPTM denotes the trajectory matching loss defined Eq. 3:
LPTM (Dp. 07)) = 1107.r
OF+
s.t. 07--- = arg min [(Dp: 87),
(3)
100 OT.ATIB
9
where 1(D; 8) denotes the loss of the student model trained on the distillation dataset D with param- eter 6, and 82, denotes the updated teacher model parameters generated according to the Shortcut Trajectory (ST) strategy, which will be elaborated in the subsequent section.
5

---

Published as a conference paper at ICLR 2026
3.2 SHORTCUT TRAJECTORY
PTM enables the distilled dataset to capture multi-stage knowledge throughout the training process, yet does not guarantee optimal distillation quality at every stage. To investigate the influence of teacher models at different stages, we compute the trajectory loss gradients of the distilled data under various alignment scopes and visualize their cosine similarities.
As shown in Figure (a), the gradient similarities across different alignment starting points remain low along the original trajectories, indicating high variance the update directions of the distilled data, even in the later, more stable training stages. This instability hinders effective and efficient knowledge transfer, further corroborating the observations Section 2.
To address this, propose Shortcut Trajectory. Rather than fitting the full trajectory directly, it retains key endpoint information and introduces intermediate teacher models with stronger structure and guidance. These intermediates assist in aligning endpoints, resulting in new trajectories with more stable optimization targets and clearer update directions.
endpoint tp for stage p. The trajectory Teanv = {0 0 & t $ tp} is defined
Formally,
80= (1 (t) (1)00,
(4)
where 3p(() E 1) serves weighting function that governs the placement of way- points along the trajectory. Here, the start and end points, 8g and 0r,, are exactly of and of, from Tooms respectively. In essence, Toonv pro- vides direct interpolation from 0g to Oz.
422
4.75
1.00
normal
shortcut
Figure 4: Gradient cosine similarity on the syn- thetic dataset across different epochs for the orig- inal and shortcut trajectories.
The weighting function (t) is computed pro- portionally based on the accumulated distance along the original trajectory Timss as follows:
Norm (0141 0t) Norm(O141 - 0r)
(₂(0) = 0, Bp(t) =
(5)
where Norm(-) denotes the f2-norm. To mitigate inconsistencies across different network layers, we compute layer-wise fz-normalization, yielding vector (t) = (t)) (1) where each component corresponds to distinct layer parameter in the network.
Unlike Matching Convexified Trajectory (Zhong al., 2025) interpolation strategy, which uses only the last point of the teacher's trajectory, use different interpolation endpoints for each stage. As illustrated Figure (d), this strategy dynamically generates matching trajectories each stage by utilizing the earliest and latest teacher models of the current phase, thereby effectively capturing inter-epoch variations in the teacher parameters.
As shown in Figure 4 (b)), the gradient similarity of the trajectory generated by our method is significantly improved, noting the optimization process of distillation dataset becomes more stable.
In addition, have mathematically proven this gradient phenomenon, which be summarised in the following proposition. The details and proof of this proposition are given Appendix C.
Proposition 1. Let E(D, 0) denote the comparative learning loss the distillation dataset D when the model parameter is 0. Let L, (D), (D) denote the matching loss different ranges the interpolated trajectory, with the difference between the starting points as At, M is the maximum length of matching trajectory, then under the following assumptions:
1. The loss ((D,0) is twice continuously differentiable in 0, and its Hessian H; = V2, (D.00)) 0.) along the trajectory {0.} satisfies a Lipschitz condition [H Hell S
LH t), and in addition [|H Hmax for some constant Huaxi
2. The mixed Hessian V2 ((D,0) has spectral norm bounded by L.po-

---

Published as a conference paper at ICLR 2026
Then the following conclusion holds:
(V (DLi(D)) < KAI+ O(AP), where K = M | (bml|A01) + Loo)
77
(6)
This proposition states that the gradient difference of the interpolated trajectory converges linearly with At, but the original trajectory does not have this guarantee. This demonstrates that our method can effectively control gradient differences and enhance the stability the distillation process.
To further improve the stability of the distillation process, also used exponential moving average smoothing strategy (Hunter, 1986). The distillation dataset Dj after the i-th iteration was smoothed and updated as follows.
- -z -1--a)) a));
(7)
In summary, our procedure is shown in Algorithm 1.
4 EXPERIMENT
4.1
EEXPERIMENTAL SETTINGS
Dataset and task. We evaluate our method Flickr30K (Plummer et al., 2015) and COCO (Lin et al., 2014), using the Karpathy split (Karpathy & Fei-Fei, 2015) for training, validation, and testing. Flickr30k and COCO are image captioning datasets with 31K and 123K images respectively, and each image is paired with five captions. The model performance is commonly measured by the recall of top-K retrieval (R@K): given query from one modality, we retrieve the closest k matches from the other modality and measure the cor- rectness. We denote the text-to-image retrieval as IR K and image-to-text retrieval as TR @
standard vision-language benchmarks:
Baseline method. We compare method against several baselines. For coreset selection, include random subset selection (Random), Herd (Welling, 2009), K-center (Farahani & Hekmatfar, 2009), and the forgetting strategy (Forgetting) (Toneva al., 2018). For dataset distillation, we consider MTT-VL Wu et al. (2023), LoRS Xu et al. (2024) and EDGM (Zhao et al., 2025). A detailed description of these methods can be found in the Appendix
Implementation Details. We evaluate data-pair counts N {100, 200, 500} } with EMA decay of 0.99. All experiments run on a single 3090 GPU, highlighting the method's efficiency and low memory use. For details regarding time and memory overhead, please refer to Appendix D.6.
Following prior work (Xu et al., 2024), we mainly employ a trainable Normalizer-Free ResNet (NFNet) (Brock al., 2021) as the image encoder with ImageNet-pretrainee weights (Deng et al., 2009). For the text encoder, we use a pretrained and frozen BERT (Devlin et al., 2019), followed by trainable projection head. We also tried other encoders such RegNet (Radosavovic al., 2020) and DistiIBERT' (Sanh et al., 2019) in Appendix D.5. For additional training details and more analysis, please refer Appendix D.
4.2 MAIN RESULTS
The main results are summarized in Table 1, Table 2 and Table 3. Experimental results demonstrate that our PTM-ST method consistently yiclds significant gains across all metrics, as shown in our empirical evaluations below. Under equal compression ratios, the multi-stage distillation strategy outperforms conventional single-stage baselines, confirming the effectiveness and robustness of the phase-aware mechanism in MDD. This advantage grows with the number of synthetic samples, indi- cating that decomposing distillation into semantically aligned phases allows each subset to capture the teacher model's evolving dynamics at different training stages. Scaling experiments 1 1,000 distilled pairs further validates the scalability and efficacy of our approach. Comparative analysis of core-set selection methods shows negligible or worse performance relative to random sampling. un- derscoring their limitations in modeling complex cross- modal dynamics, under our standard metrics
We have provided the drawing details for Figure 2 and Figure in Appendix B.2.
7

---

Published as a conference paper at ICLR 2026
Table 1: Results on Flickr30k. The metrics for training model on the full dataset are IR@ @R@1=22 1=27.3, IR 5-571, IR @ 00-9.77; TR @ 1-33.9, TR@5-65.11 @ 10=75.2.
Pairs
Ratio
Metric
Corc-set Selection
Dataset Distillation
RAND
HERD
K-CENT
FORGET
MTT-VL
LoRS
EDGE
PTM-ST
A
1.0
4.7
7.8a02
9,6ana
+13
IR @
4.0
2.8
3.1
2,4
15.74a5
24.4
28.4:00
+40
100
0.3%
IR 10
6.5
5.3
6.1
5.6
24.6+10
35.5+00
41.5:005
1.3
9.9.0.1
10.2403
TR@5
5,9
4.7
4.2
4.2
28. 28.3
30.9
38.6::95
+77
TR 10
10.1
7.9
7.6
9.7
39.1
44.9s06
52.7=0.7
IR 60
1.1
1.5
1.2
4,62n9
10.8
12.5.:0.1
+17
IR
4,8
5.5
5.4
3.1
16.0+16
29 81a4
34.7::9.2
200
0.7%
I 10
9.2
8.4
25.5+26
40.0+07
48.5
TR
2.1
2.3
1.5
10.2+a8
14.1
19.3::02
+52
T
8.7
8.4
8.2
8.4
28.7410
36. 36.1
45.9.:94
+93
41.9219
50.0gax
+89
IR 1
2.4
3.0
3.5
1.8
6.6203
12.7+94
6.7
16.0::02
+33
IR 5
10.5
10.0
10.4
20.2412
32.9203
21.0
40.5..0.1
1.7%
IR 10
17.4
17.0
15.9
30.0+2.
44.9a0s
30.5
54.0=9.3
+9.1
T 1
5.2
5.1
4.6
3.6
13.3106
14.7496
13.3
22.2::94
+75
T
18.3
16.4
12.3
32.8+18
37.6
35.6
51.1=03
TR 25.7
24.3
23.3
19.3
46.8298
51.1204
47.5
64.6:833
-138
Table 2: Results the COCO dataset.
The metrics for training model on the full dataset are IR @ 1=16,9, IR@5=41.9, IR@ 10=55.9; TR @ 1=19.,, TR@5=45.6, 10=59.5.
Pairs
Ratio
Metric
Core-set Selection
Dataset Distillation
RAND
HERD
K-CENT
FORGET
MTT-VL
LoRS
EDGE
PTM-ST
IR@1
0.3
0.5
0.4
0.3
1.3
1.7101
2.3am1
+06
IR@5
1.3
1.4
1.5
5,4003
6.9=02
9,0=a1
100
0.8%.
IR 10
2.7
3.5
2.5
2.5
9.5
11,940
15.6:93
+37
TR@:
0.8
0.8
1.4
0.7
2.5
3.0m03
4,1:n2
3.0
10.01a
13.4:0.2
TR 10
5.0
4.9
5.5
4.8
15,7404
18.5
22.0
+33
IR@1
0.6
1.7=0.1
2.2+02
IR 685
2.3
2.4
2.1
2.8
6.5204
8.62n1
13.7
+51
200
1.7%
I 10
4.4
4.1
5.8
4.9
12,34a8
14,7
22,2::0.
TR@I
1.0
1.
3.3
3,6203
5.7=002
1R@5
4.0
3.6
3.5
11,9206
12.1
18.2::0.1
+6.1
TR @ 10
7.2
7.7
7,5
7.0
19,4122
20.8104
27.8
+700
@
1.1
1.7
2.5
3.3
1.8
6,6:aa11
+33
IR 585
5.0
5.3
6.3
5.8
8.9209
11,8194
6.5
20.5::0.1
+83
500
4.4%
IR@ 10
8.7
10.5
8.2
15.8
19.2496
11.2
30.7
TR@1
3.2
5.0=04
4.1=03
2.9
6,9:n3
+19
TR
7.5
7.8
8.7
8.2
17.2+13
12.8+04
9.5
20.1::022
+29
TR
12.5
13.7
13.8
13.0
26.0a19
20.2409
15.7
30.02..3
and settings. In contrast, our progressive framework explicitly transfers multi early and late teacher stages, significantly enhancing the distilled dataset's representational capacity. Notably, on Flickr30K, PTM-ST achieves 76% of full training performance using only 1.7% of the original data, highlighting its potential improve compression efficiency in multimodal distillation. In addition, we extend our method to the larger dataset LLa (Liu et al., 2023). This dataset is prepared for LLaVA pre-trainet rreitrannnng and contains 595k image-text text pairs, We split into training, val- idation, and test sets in 3:1:1 ratio. We also explore as the image encoder and BGE-1.5 (Xiao et al., 2024) as the text encoder in Appendix D.4. The results shown in Table 3 and Table 16 reveal that method remains effective with scaled training data size and model capacity, and it consistently outperforms the previous SOTA.
multi-phase phase knowledge from
powerful DiNo-v2 (Oquab et al., 2023)
4.3 ABLATION STUDY
This section presents ablation study on the three core components of our framework: Progressive Teacher Modeling (PTM), Shortcut Trajectory (ST), and Exponential Moving Average (EMA). Ex- periments conducted on 500 distilled pairs from Flickr30K and COCO are summarized in Table 4. The results demonstrate that each component contributes positively to the overall performance, val- idating their individual effectiveness in enhancing distillation quality.
IIterceppeed from the original paper. Reeroduuedd by ourselves.
8

---

Published as a conference paper at ICLR 2026
Table 3: Results on
dataset. The metrics for training model on the full dataset are IR @ 199.3, IR @ 5-25.9, 9, IR @ 10=36.5; T 19988 TR@5-26.4, TR @ 10=37.3.
Pairs
LoRS
Ours
IR@1
IR@5
@ 10
TR@1
TR@5
TR 10
IR
TR@:
TR@5
TR
100
1.2
4.6
7.7
1.7
6.9
11.4
2.3
8.2
13.2
2.9
10.0
15.9
200
1.4
5.3
13.6
2.7
9.7
15.8
3.7
11.9
18.4
500
1.7
6.2
10.1
2,5
8.7
13.8
3,3
11,4
17.9
4.1
13,2
19.9
Table 4: Various ablation studies with 500 pairs on Flickr30k and COCO.
No.
Model
Flickr TR@K Flickr IR@K
COCO IR@K
COCO TRØK
10
10
(I) BASE (N/A)
12.2
33.0
16.2
39.4
54.0
3.4
11.7
19.1
4.2
13.8
20.8
(2) EMA
12.9
33.7
46.3
16.2
40.6
54.3
3.5
12.3
19.8
4.0
13.6
20,8
(3) PTM
13.4
35.2
48.1
19.6
43.2
55.5
4.2
22.6
4.8
23.4
(4) ST
14.2
37.8
50.8
19.5
45.1
59.3
49
15.9
24.7
4.8
15.1
23.8
(5) PTM + EMA
14.3
37.7
50.7
18.8
45.0
58.5
4.5
14.8
23.1
4.8
15.7
24.3
(6) EMA
15.4
38.3
19.7
46.7
60.3
26.3
5.0
15.8
24.0
(7) PTM ST
15.4
38.8
52.2
22.3
50.5
64.6
6.3
20.1
30.2
6.2
19.9
29.8
(8) OURS(PTM + ST EMA)
15.5
39.6
53.6
22,9
31.6
64.9
6.6
20.5
30.7
6.9
20.1
30.0
Among the modules, Shortcut Trajectory (ST) provides the largest improvement by stabilizing and guiding the teacher model's optimization path, which is crucial for reliable and representative knowl- edge transfer. PTM further boosts performance by dividing the teacher's training into phases to bet- ter align semantic information across stages. EMA helps smooth training dynamics and improves robustness in multimodal settings. Together, these components work synergistically best distillation results. Quantitatively, Table 4 shows monotonic gains from EMA to PTM to ST, with PTM+ST contributing most of the improvement. Adding EMA top yields small but consis- tent benefits, giving the hest IR @ K/TR K on both datasets.
achieve the
4.4 GENERALIZATION TO VQA AND ZERO-SHOT CLASSIFICATION
To more comprehensively evaluate the generalization capability our distillation method, we fur- ther conducted experiments Visual Question Answering (COCO-QA) (Lu et al., 2016) and Im- ageNet classification (Deng et al., 2009). Specifically, in COCO-QA, we enumerate all possible answers and construct the template "Question: {question) Answer: (answer)." for each ques- tion-answer pair, and then compute the Top-K retrieval performance. For ImageNet, we randomly sample 50 categories from ImageNet- 1 K and report the Top-K zero-shot classification accuracies.
In these comparisons, teacher denotes the model trained on the full COCO dataset and evaluated directly in zero-shot manner both tasks. In contrast, LoRS and Ours trained under the same distillation budget, using only 499 COCO image-text pairs. As summarized in Table 5, under identical supervision budgets, our method significantly outperforms LoRS and further narrows the performance gap relative to the full-data teacher model.
4.5 DISTILLATION FOR MORE PAIRS
To further validate the scalability of our approach, we extend the distillation process to 1,000 syn- thetic data points distributed across three distinct training phases (details in Appendix D). Table 6 presents a comparative analysis between our proposed method and the baseline LoRS. While con- ventional methods such LoRS tend to saturate the number of distilled samples increases-evidentt by the declining IR @K scores-our phased
experience performance degradation as
Table Performance on additional downstream tasks.
VQA COCO-QA
ImageNet-50 Classification
Methods
ACC@
ACC@5
ACC@ 10
ACC@1
ACC@5
ACC@
LoRS
10.8
33.0
39.9
18.6
42.1
55.9
Ours
16.3
38.9
47.4
22.1
52.7
67.9

---

Published as a conference paper at ICLR 2026
Table Performance comparison Flickr30k and COCO with 1000 pairs.
Dataset
Method
IR @
IR@5
IR @ 10
TR @
TR@S
TR @ 10
LoRS
9.8
29.0
41.6
14.9
39.8
53.5
Flickr30k
EDGE
9.9
28.2
40.5
14.5
38.3
51.7
Ours
17.5
42.8
56.3
23.8
53.6
67.2
LoRS
2.5
9.9
16.6
4.2
15,2
24.1
COCO
EDGE
2.8
9.8
16.2
3.9
13.0
21.0
Ours
7.0
21.8
32.3
6.8
20.9
30.8
distillation strategy consistently yields performance improvements. This demonstrates the effec- tiveness of our approach in capturing the evolving knowledge the teacher model across different training stages, thereby enhancing the quality of the distilled dataset even at larger scales.
4.6 VISUALIZATION
man purple rides bike.
man shorts and pink polo shirt, with a lorge backpack, riding bike.
Asian lady selling fruit.
A woman wears a hat while presenting variety fruits front of her.
Figure 5: Examples of initial (left) and synthetic (right) image-text pairs.
We visualize the images and texts of the synthetic pairs from Flickr30k to illustrate the distilled data. Figure presents the images and texts before (initial) and after distillation. The distilled images exhibit DeepDream-like style (Zeiler & Fergus, 2014), phenomenon commonly observed in dataset distillation, often accompanied by repetitive textures, exaggerated details, and hallucinatory patterns. The distilled texts, in contrast, are richer and more informative. Other examples are in Appendix F.
CONCLUSION
In this study, we propose PTM-ST, phased distillation framework that addresses a key limita- tion multimodal dataset distillation (MDD): the inability of conventional methods capture and transfer the complex, evolving knowledge from teacher models during later training stages, It intro- duces Phased Teacher Modeling for stage-aware guidance and a Shorteut Trajectory for stabilizing teacher evolution. Experiments on Flickr30K and IKS-COCO show that PTM-ST outperforms ex- isting methods in both performance and robustness.
Future Work. PTM-ST requires manual specification of interpolation endpoints and matching ranges for each stage, which may result in excessive parameterisation and difficulty in tuning. Fu- ture research could focus on how to adaptively adjust these parameters the distillation process progresses, transforming it into an automated, continuous procedure. Furthermore, to provide new perspectives, future work could explore distillation methods not based on MTT (see Appendix E).
ACKNOWLEDGEMENT
This work was supported by the Guangdong Basic and Applied Basic Research Foun- dation (2025A151501 1546) and by the Shenzhen Science and Technology Program (CC110040883105900 KIZDD00200001122900000 ZDCY20250901 13300000).
10

---

Published as a conference paper at ICLR 2026
REPRODUCIBILITY STATEMENT
We describe the methodology we applied in detail Section 3, and provide parameter information in Appendix D. The dataset Flickr30k was downloaded from here, while COCO was obtained from here. Our code is primarily built upon the foundations laid by previous work (Xu et al., 2024; Cazenavette et al., 2022). We currently organising the code and will make it publicly available in the near future.
REFERENCES
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lene, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35:23716- 23736, 2022.
Rahaf Aljundi, Eugene Belilovsky, Tinne Tuytelaars, Laurent Charlin, Massimo Caccia, Min Lin, and Lucas Page- Page-Caccia. Caccia. Online continual learning with maximal interfered retrieval. Advances in neural information processing systems, 32, 2019a.
Rahaf Aljundi, Min Lin, Baptiste Goujaud, and Yoshua Bengio. Gradient based sample selection for online continual learning, Advances in neural information processing systems, 32, 2019b.
Andy Brock, Soham De, Samuel L Smith, and Karen Simonyan. High-performance large-scale image recognition without normalization. International conference on machine learning, pp. 1059-1071. PMLR, 2021.
George Cazenavette, Tongzhou Wang, Antonio Torralba, Alexei A Efros, and Yan 7hu. Dataset distillation by matching training trajectories. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4750-4759, 2022.
Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multimodal I1m's referential dialogue magic. arXiv preprint arXiv:2306. 15795, 2023a.
Xuxi Chen, Yu Yang, Zhangyang Wang, and Baharan Mirzasoleiman, Data distillation can be like vodka: Distilling more times for better quality. arXiv preprint arXiv:2310.0660 2023b.
Yuxiao Chen, Jianbo Yuan, Yu Tian, Shijie Geng, Xinyu Li, Ding Zhou, Dimitris N Metaxas, and Hongxia Yang. Revisiting multimodal representation in contrastive learning: from patch and token embeddings to finite discrete tokens. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15095-15104, 2023c.
Jiequan Cui, Shu Liu, Zhuotao Tian, Zhisheng Zhong, and Jiaya Jia. Reslt: Residual learning for long-tailed recognition. IEEE transactions on pattern analysis and machine intelligence, 45(3); 3695-3706, 2022.
Jiequan Cui, Zhisheng Zhong, Zhuotao Tian, Shu Liu, Bei Yu, and Jiaya Jia. Generalized parametric contrastive learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(12); 7463-7474, 2023a.
Justin Cui, Ruochen Wang, Si Si, and Cho-Jui Hsieh. Scaling up dataset distillation to imagenet- 1k with constant memory. In International Conference on Machine Learning, pp. 6565-6590. PMLR, 2023b.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hier- archical image database. 2009 JEEE Conference on Computer Vision and Pattern Recognition, pp. 248-255, 2009. doi: 00.11099CPRR.2299520665.
Zhiwei Deng and Olga Russakovsky. Remember the past: Distilling datasets into addressable mem- ories for neural networks. Advances in Neural Information Processing Systems, 55:34391-34400 2022.
11

---

Published as a conference paper at ICLR 2026
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume (long and short papers), pp. 4171-4186, 2019.
Jiawei Du, Qin Shi, and Joey Tianyi Zhou. Sequential subset matching for dataset distillation. Advances in Neural Information Processing Systems, 36:67487-67504, 2023.
Jiawei Du, Juncheng Hu, Wenxin Huang, Joey Tianyi Zhou, et al. Diversity-driven synthesis: En- hancing dataset distillation through directed weight adjustment. Advances in neural information processing systems, 37:1 19943-114999999 119465, 2024.
Reza Zanjirani Farahani and Masoud Hekmatfar. Facility location: concepts, models, algorithms and case studies. Springer Science & Business Media, 2009.
Ziyao Guo, Kai Wang, George Cazenavette, Hui Li, Kaipeng Zhang, and Yang You. Towards lossless dataset distillation via difficulty-lligned trajectory matching. arXiv preprint 10X0523310005773, 2023.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Train- ing compute-optimal large language models. arXiv preprint arXiv:2203.15533 2022.
J Stuart Hunter. The exponentially weighted moving average. Journal of quality technology, 18(4): 203-210, 1986.
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In International conference on machine learning, pp. 4904-4916. PMLR, 2021.
Feiyang Kang, Hoang Anh Just, Yifan Sun, Himanshu Jahagirdar, Yuanzhi Zhang, Rongxing Du, Anit Kumar Sahu, and Ruoxi Jia. Get more for less: Principled data selection for warming up fine-tuning in llms. arXiv preprint arXiv:2405.02774. 2024.
Andrej Karpathy and Li Fei-Fei. Deep semual-semantic alignments for generating image descrip- tions. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 3128-3137, 2015.
Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. Lisa: Rea- soning segmentation via large language model. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 9579-95998 2024a.
Xin Lai, Zhuotao Tian, Yukang Chen, Senqiao Yang, Xiangru Peng, and Jiaya Jia. Step-dpo: Step- wise preference optimization for long-chain reasoning of llms. arXiv preprini arXiv:2406. 18629, 2024b.
Hae Beom Lee, Dong Bok Lee, and Sung Ju Hwang. Dataset condensation with latent space knowl- edge factorization and sharing. arXiv preprint arXiv:2208. 10494, 2022.
Yongmin Lee and Hye Won Chung. Selmatch: Effectively scaling up dataset distillation via selection-basee initialization and partial updates by trajectory matching. arXiv preprint arXiv:2406.1 18561, 2024.
Hongyang Lei, Xiaolong Cheng, Dan Wang, Kun Fan, Qi Qin, Huazhen Huang, Yetao Wu, Qingqing Gu, Zhonglin Jiang. Yong Chen, et al. M3-jepa: Multimodal alignment via multi-directional moe based on the jepa framework. arXiv preprint arXiv:2409.0S5: 05999, 2024.
Shiye Lei and Dacheng Tao. A comprehensive survey of dataset distillation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-ima -mage pre- training for unified vision-language understanding and generation. In International conference on machine learning, pp.
PMLR, 2022.
12

---

Published as a conference paper at ICLR 2026
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference machine learning. pp. 19730-19742. PMLR, 2023.
Yongqi Li and Wenjie Li. Data distillation for text classification, arXiv preprint arXiv:2104.08448, 2021.
Yunxin Li, Zhenyu Liu, Zitao Li, Xuanyu Zhang. Zhenran Xu, Xinyu Chen, Haoyuan Shi, Shenyuan Jiang, Xintong Wang, Jifang Wang, et al. Perception, reason, think, and plan: A survey on large multimodal reasoning models. arXiv preprini arXiv:2505.0499 2025.
Ronghao Lin and Haifeng Hu. Multimodal contrastive learning via uni-modal coding and cross- modal prediction for multimodal sentiment analysis. arXiv preprint 1XXi::2200.14556 2022.
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Ession-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp. 740-755. Springer, 2014.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36:34892-3499 2023.
Jiasen Lu, Jianwei Yang, Dhruv Batra, and Devi Parikh. Hierarchical question-image co-attention for visual question answering. Advances in neural information processing systems, 29, 2016.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv preprint 1rXbv:1807.037448 2018
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193. 2023.
Bohao Peng, Zhuotao Tian, Xiaoyang Wu, Chengyao Wang, Shu Liu, Jingyong Su, and Jiaya Jia. Hicrarchical dense correlation distillation for few-shot segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 23641-23633 2023.
Shangpin Peng, Senqiao Yang. Li Jiang, and Zhuotao Tian. Mitigating object hallucinations via sentence-level early intervention. arXiv preprint arXiv:2507. 12455, 2025.
Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, Qixiang Ye, and Furu Wei. Grounding multimodal large language models to the world. In The Twelfih Interna- tional Conference on Learning Representations, 2024.
Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svet- lana Lazebnik. Flickr30k entities: Collecting region to to-phrase correspondences for richer image- to-sentence models. Proceedings of the IEEE international conference on computer vision, pp. 2641-2649, 2015.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp- 8748-8763. PmLR, 2021.
Ilija Radosavovic, Raj Prateek Kosaraju, Ross Girshick, Kaiming He, and Piotr Dollár. Designing network design spaces. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10428-10436, 2020.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv: 0008, 2019.
Shitong Shao, Zeyuan Yin, Muxin Zhou, Xindong Zhang, and Zhiqiang Shen. Generalized large- scale data condensation via various backbone and statistical matching. In Proceedings of the IEFF/CV Conference Computer Vision and Pattern Recognition, pp. 16709-16718, 2024a.
13

---

Published as a conference paper at ICLR 2026
Tong Shao, Zhuotao Tian, Hang Zhao, and Jingyong Su. Explore the potential of clip for training- free open vocabulary semantic segmentation. In European Conference on Computer Vision, pp. 139-156. Springer, 2024b.
Zhuotao Tian, Hengshuang Zhao, Michelle Shu, Zhicheng Yang. Ruiyu Li, and Jiaya Jia. Prior guided feature enrichment network for few-shot segmentation. IEEE transactions on pattern analysis and machine intelligence, 44(2):1050 2020.
Zhuotao Tian, Pengguang Chen, Xin Lai, Li Jiang, Shu Liu, Hengshuang Zhao, Bei Yu, Ming- Chang Yang, and Jiaya Jia. Adaptive perspective distillation for semantic segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(2): 1387, 2022a.
Zhuotao Tian, Xin Lai, Li Jiang, Shu Liu, Michelle Shu, Hengshuang Zhao, and Jiaya Jia. General- ized few-shot semantic segmentation. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp.
2022b.
Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, and Geoffrey Gordon. An empirical study of example forgetting during deep neural network learning. arXiv preprint arXiv:1812.05159) 2018.
Junjie Wang, Bin Chen, Yulin Li, Bin Kang, Yichi Chen, and Zhuotao Tian. Declip: Decoupled learning for open-vocabulary dense perception. In Proceedings of the Computer Vision and Pat- tern Recognition Conference, pp. 14824- 14834, 2025a.
Junjie Wang, Keyu Chen, Yulin Li, Bin Chen, Hengshuang Zhao, Xiaojuan Qi, and Zhuotao Tian. Generalized decoupled learning for enhancing open-vocabulary dense perception. arXiv preprint arXiv:2508. / /256, 2025b.
Kai Wang, Bo Zhao, Xiangyu Peng, Zheng Zhu Shuo Yang, Shuo Wang, Guan Huang, Hakan Bilen, Xinchao Wang, and Yang You. Cafe: Learning condense dataset by aligning features. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp-
12196-12205, 2022.
Shaobo Wang, Yicun Yang. Zhiyuan Liu, Chenghao Sun, Xuming Hu, Conghui He, and Linfeng Zhang. Dataset distillation with neural characteristic function: A minmax perspective. In Pro- ceedings of the Computer Vision and Puttern Recognition Conference, pp. 25570-25580, 2025c.
Tongzhou Wang, Jun-Yan Zhu, Antonio Torralba, and Alexei A Efros. Dataset distillation. arXiv preprint 1rxiv:1811.1090 2018
Ziyu Wang, Yue Xu, Cewu Lu, and Yong-Lu Li. Dancing with still images: Video distillation via static-dynamic disentanglement. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6296-6304, 2024.
Max Welling. Herding dynamical weights to learn. In Proceedings of the 26th annual international conference on machine learning, pp. 1121-1128, 2009.
Xindi Wu, Byron Zhang, Zhiwei Deng, and Olga Russakovsky, Vision-language dataset distillation. arXiv preprint arXiv:2308 07545, 2023.
Shitao Xiao, Zheng Liu, Peitian Zhang, Niklas Muennighoff, Defu Lian, and Jian-Yun Nie. C-pack: Packed resources for general chinese embeddings. In Proceedings of the 47th international ACM SIGIR conference on research and development in information retrieval, pp. 641 641-649, 2024.
Yue Xu, Zhilin Lin, Yusong Qiu, Cewu Lu, and Yong- Li. Low-rank similarity mining for multi- modal dataset distillation. arXiv preprint arXiv:2406.03793. 2024.
Zhe Xu, Yuzhong Chen, Menghai Pan, Huiyuann Chen, Mahashweta Das, Hao Yang, and Hanghang Tong. Kernel ridge regression based graph dataset distillation. In Proceedings of the 29th ACM SIGKDD Conference Knowledge Discovery and Data Mining, pp. 2850-2861, 2023.
Senqiao Yang, Tianyuan Qu, Xin Lai, Zhuotao Tian, Bohao Peng. Shu Liu, and Jiaya Jia. Lisa++: An improved baseline for reasoning segmentation with large language model. arXiv preprint 2...2222..7744 2023.
14

---

Published as a conference paper at ICLR 2026
Senqiao Yang, Zhuotao Tian, Li Jiang, and Jiaya Jia. Unified language-drive driven zero-shot domain adaptation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recogni- tion. pp. 23407-23415, 2024.
Senqiao Yang. Yukang Chen, Zhuotao Tian, Chengyao Wang, Jingyao Li, Bei Yu, and Jiaya Jia. Visionzip: Longer is better but not necessary in vision language models. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 19792-19802, 2025a.
Senqiao Yang, Junyi Li, Xin Lai, Bei Yu, Hengshuang Zhao, and Jiaya Jia. Visionthink: Smart and efficient vision language model via reinforcement learning. arXiv preprint urXiv:2507.33348
2025b.
Zeyuan Yin, Eric Xing, and Zhiqiang Shen. Squeeze, recover and relabel: Dataset condensation at imagenet scale from perspective. Advances in Neural Information Processing Systems, 36: 73582-73603, 2023.
Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In European conference on computer vision, pp. 818- 188-833.22 Springer, 2014.
Bo Zhao and Hakan Bilen. Dataset condensation with distribution matching. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 6514-6523, 2023.
Bo Zhao, Konda Reddy Mopuri, and Hakan Bilen. Dataset condensation with gradient matching. arXiv preprint #rXiv:2006.09229 2020.
Zhenghao Zhao, Haoxuan Wang, Junyi Wu, Yuzhang Shang, Gaowen Liu, and Yan Yan. Efficient multimodal dataset distillation via generative models. arXiv preprint arXiv:2509. 2025.
Wenliang Zhong, Haoyu Tang, Qinghai Zheng, Mingzhu Xu, Yupeng Hu, and Weili Guan. Towards stable and storage-efficien dataset distillation: Matching convexified trajectory. In Proceedings of the Computer Vision and Puttern Recognition Conference, pp. 25581-25589, 2025.
Yongchao Zhou, Ehsan Nezhadarya, and Jimmy Ba. Dataset distillation using neural feature re- gression. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), 2022.
Mohammadreza Zolfaghari, Yi Zhu, Peter Gehler, and Thomas Brox. Crossclr: Cross-modal con- trastive learning for multi-modal video representations. Proceedings the tional conference on computer vision, pp. 1450-1459, 2021.
the IEEE/CVF interna-
15

---

Published as a conference paper at ICLR 2026
A RELATED WORK DETAILS
Multimodal learning. Multimodal Learning (ML) has demonstrated superior representational power and holds substantial promise for wide range of applications (Jia et al., 2021; Chen al., 2023c; et al., 2025; Lai et al., 2024b), owing to its ability to integrate complementary information from diverse modalities. Recent works have further extended this capability to reasoning segmen- tation (Lai et al., 2024a; Yang et al., 2023) and open-vocabulary perception (Wang et al., 2025a;b). Existing multimodal learning approaches predominantly seek project inputs into unified fea- ture space, thereby facilitating the alignment of matched samples while distinguishing unmatched ones (Shao al., 2024b; Cui et al., 2023a). However, compared unimodal learning tasks such as semantic segmentation and object detection (Tian et al., 2020; 2022b; Cui et al., 2022; Peng et al., 2023), multimodal data is not only larger in scale and richer in information, but also significantly more complex (Lei et al., 2024). This increased complexity often compromises data quality and continuity, thereby exacerbating the challenges associated with training, such as object hallucina- tions (Peng et al., 2025). Consequently, there is urgent need to explore more efficient learning strategies (Yang et al., 2025a; Tian et al., 2022a; Yang et al., 2025b) that are better tailored to the unique characteristics of multimodal scenarios (Zolfaghari et al., 2021; Lin & Hu, 2022; Yang al., 2024).
Image- Text Contrastive Learning. In multimodal data set distillation (Wu et al., 2023), the ba- sic task is image-text contrast learning. Let the image-text pair dataset be D = (X,)), which contains m pairs of images X {a:) and text y {ye Jm- Image-Text Contrastive models (e.g. CLIP) include an image encoder fv and text encoder fr, which encode image and text into one-dimensional vector representations, respectively, i.e. u; fv(z)), v. = fr(1).. The model performs cross-modal retrieval using cosine similarity 8.g = cos(u;, Uj). During training, the InfoNCE loss (Oord al., 2018) typically used:
exp(Sig/Te)
exp(8is/Te) Σn exp(6xs/
LNCE(B) b Σ log(P!)) + +og log(P!))
PT
E. Exp((B///Te) Pg
where B C D is a small batch of data with size of b. Pj and pf are the softmax probabilities of similarity Kej, respectively, and Te the temperature factor. InfoNCE assumes that the paired text y for each image z; is positive sample, while other texts yk, k # i are negative samples, thereby achieving contrastive alignment.
Match Training Trajectory. The MTT (Cazenavette et al., 2022) method guides the optimization of the synthetic dataset 2> by training two separate models on D and the real dataset D, respectively, and matching their weight trajectories of different lengths-namely, trajectory of length t on D and a trajectory of length AT on D.
Low-Rank Similarity Mining. The recent method LoRS (Xu et al., 2024) further improves the expressive power of the distillation dataset by introducing learnable similarity matrix. And use wBCE loss when training student models:
(B,)) Hs
Σ 1
( (Big/T) + Ksi S BH
Σ 188881888
B)
where C is small batch of data in synthetic dataset, 3 is the positive/negative threshold, set 0.5, the sigmoid function, Siy the learnable similarity between the i-th image and the j-th text in D. and f(y.p) p) = yy log(p) - (1 y) log(1 - p).
They perform low-rank decomposition on the similarity matrix, i.e., Š = wI + LR., where w, Lrxr. xr. X xr are learnable parameters, and a, are hyperparameters. The number of parame- ters saved by using low-rank decomposition is very small, and our method only uses the complete similarity matrix. As shown in Table 12 (sim type=full).
16

---

Published as a conference paper at ICLR 2026
Efficient Multimodal Dataset Distillation via GEnerative Models. Zhao al. (2025) proposes a generative multimodal dataset distillation framework called EDGE, which efficiently replaces the original large-scale graphic dataset with very small number of generated samples while maintain- ing the retrieval performance by training graphic generation model and introducing a bi-directional contrast loss, diversity loss and caption synthesis strategy.
Although this form of supervision primarily aims to capture cross-modal relationships, it inadver- tendy introduces additional information such as intra-modal similarities. The inconsistency of these signals between the early and late stages of training significantly increases the training difficulty. Moreover, existing methods like LoRS only capture early training dynamics, synthesizing samples using weights from the first two epochs. While the teacher model continues to improve with further training, the student model rapidly collapses during the distillation process.
B MOTIVATION
Section B. primarily introduces the motivation behind our entire project and the process of explor- ing methods. Section B.2 introduces the specific settings of the visualisation experiment.
B.1 MOTIVATION DETAILS
Our overall approach stems from an important observation: LoRS only utilises the teacher model from the first 20-30% of the training phase during the distillation of synthetic datasets. This dif- fers significantly from traditional single-modas dataset distillation methods, which typically rely on teacher information from the entire training process. However, when we attempted to directly in- troduce the teacher model from the later training stages into the distillation process, we observed severe instability or even collapse in the training process of the student model, as shown in Table 7. This phenomenon prompted us to delve deeper into the question: why does the introduction of the later teacher model lead to such severe performance degradation in multimodal tasks?
Table 7: Performance of teacher and student models different epochs on COCO and Flickr30K.
Dataset
Model
4
8
10
Teacher Student
35.4
39.0
40.4
41.9
42.8
COCO
11.4
7.9
6.9
5.7
5.1
Teacher
45.6
48.9
51.2
53.6
55.3
Flickr30K
Student
33.8
35.1
33.5
31.4
30.1
DATM Experiment. We referenced the dynamic alignment strategy proposed by DATM (Guo al., 2023) single-modal data distillation: as distillation proceeds, dynamically increase the sam- pling upper bound I of the matching starting point. This gradually captures the training dynamics of the teacher at different training stages. We directly applied DATM to existing multimodal data set distillation frameworks, and the experimental results on COCO are shown in Table 8. Simply ex- panding the sampling range of matching starting points through the distillation process still reduces the distillation effect.
More Visualization. To better understand the root causes of these challenges, we conducted a de- tailed analysis by visualizing the loss gradients that were back- pack-propagated from the teacher model to the synthetic dataset across multiple training epochs, as shown in Figure 6. Our observations revealed that, training progressed, the gradient magnitudes steadily increased. More critically, the direction of these gradients oscillated violently between epochs, exhibiting no discernible pat- tern. This erratic behavior suggests that the training dynamics of the teacher model are inherently unstable, and such instability is directly inherited by the synthetic dataset.
Remark.
The above analysis inspires us: due to the complexity of multimodal data, the informa- tion in different training stages of the teacher model has significant differences. It is necessary to develop a distillation method that capture the training dynamics of different stages.
17

---

Published as a conference paper at ICLR 2026
Table 8: Results on the COCO dataset. The metrics for training the model the full dataset are
IR @ 1=16.9, IR IR@5=41.9, IR @ 10=55.9; TR @ 1=19.6, TR@5=45.6, TR 10=59.5.
Dataset Distillation
Pairs
Ratio
Metric
MTT-VL
LoRS
DATM
PTM-ST
I
1.3+0.1
1.7:0.1
1,2+0.1
2.310.11
I
5.4+0.3
6.9+0.2
5.1=0.2
9.0=0.1
100
0.8%c
IR @
9.5+
11.910.3
8.9:0.4
15.6=0.3
TR@1
2.5+0.33
3.0:1..3
2.5
4.1
TR@5
10.0:10.5
11.0=0.2
9.120.3
13.4=0.2
T 10
15.7+0.4
18.5+0.3
14.9+0.3
22.0+-022
I@@1
1.740.1
2.2+0.2
1.6+0.2
3.9+0.1
I@@5
6.5+0.4
8.6+0.11
6.3+0.2
13.7=0.2
200
1.7%e
R
12.340..8
14.740.2
11.800.
22.2--.2
TR@1
3.3+0.2
3.6:1..3
2.8+0.2
5.7
TR@S
11.9:10.6
12.1=0.2
11.1#0.4
18.2-0.3
T 10
19.4+1.2
20.8+0.4
17.8+0.3
27.8 0.3
I
2.5
3.3+0.2
2.2+0.3
6.6+0.1
IR@5
8.9+0.7
11.8+0.4
8.1+0.4
20.5=002
500
4.4%o
IR
15.8
19.2+0.6
14.3+0.7
30.7+0.2
TR@1
5.0+0.4
4.1:0..5
3.200.4
6.9
TR@5
17.211.3
12.8-0.4
10.920.6
20.1
TR 10
26.0+1.9
20.2+00.9
18.3+1.0
30.0 0.3
Greditm and Among Origina Teccher
Teaccter Mtdel
CM locder
(a)
(b)
(c)
Figure 6: (a) shows the gradients of the teacher model on the synthetic dataset across different epochs. (b) illustrates the parameter differences between various training trajectories of the teacher model. (c) present visualization of the new teacher model constructed using our proposed method.
B.2 EXPLAIN OF FIGURE
Explain of 3D. As illustrated in Figure 6, (a) presents a detailed visualization of the gradients propagated from the teacher model to the synthetic dataset at various training epochs. It is apparent that the training process progresses, the magnitude the gradients increases substantially. More importantly, the directions of these gradients exhibit erratic fluctuations, with discernible pattern, highlighting the inherent instability the teacher model's 's learning trajectory as advances through later training epochs. This behavior further reinforces becomes increasingly volatile during the later stages of training, a phenomenon that significantly complicates the distillation process for the student model.
earlier assertion that the teacher model
In (b), we depict the parameter differences between teacher models that have been derived from distinct training trajectories. The substantial divergence observed between these trajectories serves as an additional confirmation of the inconsistencies and instability inherent independently trained teacher models. This variability across trajectories further emphasizes the challenges associated
18

---

Published as a conference paper at ICLR 2026
with applying standard distillation techniques, as it introduces significant misalignment between the knowledge encapsulated in different teacher models.
(c) provides visualization of the parameter trajectory of the new teacher model generated using our proposed method. The resulting shortcut trajectory, constructed with the strategic interpolation of key teacher checkpoints, exhibits remarkable stability when compared traditional teacher tra- jectories. This enhanced stability directly contributes to the consistency of the distilled knowledge, enabling the student model receive more reliable and coherent supervision signals throughout the training process. Consequently, this improvement in trajectory consistency facilitates a more effective knowledge transfer, ultimately leading better performance and robust distillation results.
Figure in Main Paper. Figure (a) in the paper illustrates the performance differences between the teacher model and the student model the Flickr30k test set. The specific values are shown in Table 7. The teacher model was trained directly on the real dataset, with the x-axis coordinates representing the number of epochs trained. The student model was trained on the distilled dataset, where the epoch count on the x-axis represents the maximum value of the trajectory matching start- ing point when generating this distilled dataset imax_start_eppee generating this distilled dataset. The remaining parameters are shown in Table 9.
T PTM-ST was not used when
Table 9: Hyper-parameters for the distillation run.
Parameter
Flickr30k
COCO
syn steps expert.epochs
8
1
Ir_img
1000
1000
Ir_txt
1000
1000
Ir_Ir
lc-2
le-2
Ir teacher img
0.1
0.1
Ir teacher txt
0.1
0.1
Ir_sim
10
50
sim_type
lowrank
lowrank
sim_rank
20
40
alpha
0.01
1.0
num queries
499
499
mini mini-batch. size
40
30
type
wBCE
wBCE
In Figure (b), we used the original expert trajectories to calculate the gradient of the trajectory matching loss on the distilled dataset under different matching ranges, and visualised using PCA dimensionality reduction. Specifically, tialisation for the synthetic dataset, and calculated the trajectory matching loss LMTT using different trajectory matching starting points (as indicated by the numbers in the figure). We then backpropa- gated this loss to the distilled dataset to obtain the gradient.
selected the first 200 images from Flickr30K as the ini-
In Figure 4, we calculate the gradient on the synthetic dataset using the original expert trajectory and the trajectory after interpolation. The interpolation endpoint the endpoint of the original trajectory. We extract the gradient of the image part and calculate the cosine similarity between each pair, and plot the result in Figure 4 in the paper.
C PROOF OF THE PROPOSITION
Notation. Let the original teacher trajectory be {Bu. O,....,on} We reparameterize this trajectory by linear interpolation:
0, = 0, + tA e n]. A = 6 - 6o.
Thus the direction is constant all times (i.e. straight line with uniform speed).
For given distilled dataset D,, one SGD step with step size 17 is denoted:
(D 0, 79t, It == Vol(D,
where l is the training loss (e.g. InfoNCE wBCE).
Intercepted from the original paper.
Reproduced by ourselves.
19

---

Published as a conference paper at ICLR 2026
Trajectory-Matcchnn Loss. We wish to match trajectory segment of length M that the stu- dent move approximates the "target step" MAR. For two starting points and t + At, the matching losses are
(D( = 100 0, M2 A@l1
(9)
M1 |
CD( |10+ta1 02 As M2 A0
(10)
M2 ||0
Substituting the SGD update gives:
U1 == 1): + M1 A0,
U2 := 79++at + M2 20,
(11)
so that:
|elll
1
L1 (D) M1 AO
C2(D)
M2 M2llA0i
Gradients w.r.t the distilled dataset D. one obtains:
By differentiating the norm and applying the chain rule,
DDL((D) M₁
(-n V (.:)),
(12)
AMl
11
1/2
VoC2(D)= |2llAa
(-n V (9(+2r).
(13)
|022||
Main Proposition: Linear convergence of gradient-difference on the interpolated trajectory. Proposition 2. Under the following assumptions:
1. The loss (D,0) is twice continuously differentiable in 0, and its Hessian
:= Voo E(D,0t)
along the trajectory {0:} satisfies a Lipschitz condition |H H.ll S Lu It' th. and in addition |Hell S H for some constant Hmaxi
2. The mixed Hessian V2 E(D,0) has spectral norm bounded by L
Then for the interpolated trajectory 0 6u + tA0 (AA 8,, Gu). one has
(Z) - -V < KAt + O(AP), k= M | (LH || + L (14)
Proof. For brevity set
It == Vo((D,)), G::= V Vi ((D,0c).
:::= Vi. (().
We bound |V V in three steps.
(i) Taylor Expansion. Along the interpolation path 01+ = At+ A9, we write g = Vol(S,0) and expand to second order:
91+A: g(0, + A1 g(0,) + / + ( (1)) ds,
where H()) Vi (SS,). 0). Split the integral at H(:)):
g++ g: + H, (AtA0) +
1( + - H.| (s At 20) ds.
/0
20

---

Published as a conference paper at ICLR 2026
Since H is LH -ipsschit along the path, |H(0, At Hell S LH (s At) || we bound the remainder:
[Blll < S/" LH (8 11) x ||| ds = LH (01)) /'" dd
LH
|150| (A1)²
3
(15)
So that:
9t+A: - 9r AL H, A0 1/11)² Res
|e S Lu ||
(16)
Since || < Mmxx, the term At H, A0 is itself ((At).
(ii) Gradient expressions. Using the definition
0: - 79:: and the chain rule,
C((D) Ing. + M, |
VDC.
17
0. G.,
Mi A0l
where 1; 199 M, and 0; = -;/l|u ubll Hence
- - VDL = |
1 Mg
02
GI+AI
0. GT.. Mi
(iii) Bounding the difference. From equation 16 and normalization one shows 82 = 01 + O(At), and similarly
GH+I G. + AtG. + O(AP), || s Lig ||0A|.
Applying the triangle inequality and Cauchy Cauchy-Schwarz Schwarz yields
DV - V&Cill S M| (1e:ll|Galll + +|G dill) At + O(AP).
Here M M M2). so that the denominator M M A0| scales the bound by the size of that segment. Since 0 = and ||G,ll S Lio setting
M || (LH ||00|| || + I.po)
"
completes the proof of equation 14.
Remark. On the non -oniinttrpll trajectory, each segment direction Adp may differ substantially from the global A0, This effectively amplifies ||β₂ - أ by a factor a > 1, leading to a larger constant in the bound and thus explaining the empirically observed lower gradient cosine-similarity.
D EXPERIMENT
For LoRS and PTM-ST, similarity matrices require additional memory. To ensure fairness, we remove one data pair, saving 151K parameters (3 x 224 + 768). In PTM-ST, no phase split is used for 100 pairs; we use (99+ 100) for 200 and (200 299) for 500. For 500 pairs, the similarity matrix size is 129K, smaller than one sample's footprint. This ensures memory- comparison across all settings.
D. SYPER-PARAMETER
This section shows the specific settings we obtained for the teacher trajectory and distillation process hyper-parameters, as shown in Table 10 and Table 12. The ema_decay is 0.99 for all experiment. We use the SGD optimizer to train the model with momentum=0.9, weight. decaa=000005.. For the student training, adopt syper-parametee distillation process, the learning rate of the training student model is learnable, and this learning rate is Ir 1==000,, and the table shows the values taken at the time of initialization.
shown in Table 11. It should be noted that during the
21

---

Published as a conference paper at ICLR 2026
Table 10: Hyperparameters for teacher training.
Table 11: Hyperparameter for student training.
Flickr-30K
MS-COCO
Flickr-30K
MS-COCO
epoch / num cem_experts
10
10
epoch (per subset)
50
50
20
20
batch_size
128
128
batch_size size
128
128
Ir_mg
0.1
0.1
ir_teacher_in img
0.1
0.1
Ir_txt
0.1
0.1
ir_teacher .txt
0.1
0.1
image_size
224x224
224x224
image_sia
224x224
224
Table 12: Hyper-parameter settings for distillation.
Flickr-30K
MS-COCO
100
200
500
1000
100
200
500
1000
subset_nume /um /
syn_steps
expert_epochs/
min start cpech/ ITp
0,
0,
0, 1.2
0. 1
0. 1,2
max_ssarrt.coch T
2.
2,
3, 4,
2,
2, 3,
iteration ip
2000
2000"2
2000*2
2000"3
2000
2000*2
2000
2000,500,
interpolation endpoints / t
6, 8
6,
8,8.
6, 8
6, 8
8, 10 1000
img
1000
1000
1000
1000
1000
1000
1000
Ir.txt
1000
1000
1000
1000
1000
1000
5000
1000
1e-2
1e-2
le.2
Ir teacher ling
0.1
0.1
0.1
0.1
0.1
0.1
0.1
0.1
Tr.lecher..a
0.1
0.1
0.1
0.1
0.1
0.1
0.1
sim
10.0
50.0
50.0
50.0
50.0
sim simltype
[ull
full
full
full
full
full
full
Tull
num queries/ Nip
99,100
200.299
332.332.333
99.100
200.299
200,299,499
mini batch.size
20
40
40
20
20
20
loss type
wBCE
wBCE
wBCE
wBCE
wBCE
wBCE
wBCE
wBCE
D.2 ADDITIONAL ABLATION STUDY
In our method, the distillation process is divided into stages, with each stage having different sam- pling ranges and interpolation endpoints for trajectory matching. As shown in Table 12, the T,- Tp. and lp values for each stage are shifted backward. To demonstrate this, conducted addi- tional ablation experiments on Flickr30K, keeping all other parameters constant while omitting this shift, and compared the distillation results. The results are shown in Table 13, where PTM without shift indicates that Tp and T are the same for each stage, and ST without shift indicates that tp is the same for each stage.
Table 13: Ablation on whether to shift under different data sizes.
Pairs
Setting
IR@1
IR@5
IR@ 10
TR @
TR@5
TR@ 10
Mean
PTM w/o shift
11.8
33.2
46.3
18.1
44.3
59.0
35.5
200
ST w/o shift
11.6
33.3
47.0
18.5
43.5
57.8
35.3
full
12.5
34.7
48.5
19.3
45.9
58.9
36.6
PTM w/o shift
15.2
39.6
53.1
21.6
48.6
62.9
40.2
500
ST w/o shift full
15.3
39.9
53.5
21.1
49.6
63.1
40.4
16.0
40.5
54.0
22.2
51.1
64.6
41.4
PTM w/o shift
16.9
41.9
55.4
21.6
49.3
62.5
41.3
1000
ST w/o shift full
16.8
42.7
56.5
23.6
51.4
64.9
42.7
17.5
42.8
56.3
23.8
53.6
67.2
43.5
As can be seen from the table, both the sampling range and the offset of the interpolation endpoint can improve the distillation effect. This confirms that our method is conducive to capturing the dynamics of the model's later training in the synthetic dataset.
D.3 COMPARISON OF DIFFERENT INTERPOLATION ENDPOINTS AND MATCHING RANGES
Interpolation endpoints. Our approach necessitates specifying interpolation endpoints for each stage in advance. Consequently, we established several sets distinct interpolation endpoint pa- rameters for each pair of distilled models and conducted experiments the Flickr-30k dataset. All
22

---

Published as a conference paper at ICLR 2026
other parameters remained identical to those in Table 12, with the experimental results presented in Table 14.
Table 14: Comparison of distillation results different interpolation endpoints.
Pairs
Interpolation endpoint
IR @
IR @
IR 10
TR
TR@5
TR
@ 10
Mean
6
9.6
28.4
41.5
14,4
38.6
52.7
30.9
100
8
8.4
26.6
39.7
15.4
39.9
54.6
30.8
10
8.8
26.8
39.7
14.9
39.6
53.6
30.6
4,
11.7
33.2
46.3
18.5
43.5
58.6
35.3
200
6,
12.5
34.7
48.5
19.3
45.9
58.9
36.6
8, 10
11.2
32.3
45.4
18.1
46.4
60.2
35.6
4,
15.3
39.6
53.1
22.6
49.2
63.4
40.5
500
6, 8
16.0
40.5
54.0
22.2
51.1
64.6
41.4
8, 10
14.9
38.8
52.2
22.1
51.2
64.4
40.6
4, 7, 10
16.6
42.5
56.2
23.7
50.2
62.5
42.0
1000
6. 8, 10
17.5
42.8
56.3
23.8
53.6
67.2
43.5
8, 9, 10
17.1
43.2
57.1
23.2
52.8
65.7
43.2
The result demonstrates that the selection of interpolation endpoints also exerts a certain influence the final results. It is necessary to select appropriate values ensure the distilled dataset can capture the training information corresponding to the teacher model's 's respective stages.
Matching ranges. For the matching range . we adopt simple linear strategy in experiments: T,- Tí + Ad(p 1), T Ti + Ad(p 1). To further assess the effect of varying Ad, we conduct additional ablation studies under the Flickr30k 500-pair setting and the results are shown Table 15.
Table 15: Effect of Ad on retrieval performance.
Method
IR @ 1
IR@5
IR @ 10
TR@1
TR@5
TR@ 10
Mean
Ad
15.2
39.6
53.1
21.6
48.6
62.9
40.2
Ad
16.0
40.5
54.0
22.2
51.1
64.6
41.4
Ad = 2
15.9
40.1
53.2
21.0
50.1
63.7
40.7
LoRS
12.7
32.9
44.9
14.7
37.6
51.1
32.3
The results reveal that although different parameter choices introduce variation, the overall in- fluence is limited; more importantly, our method consistently outperforms the LoRS baseline across all configurations. This demonstrates the robustness and practicality of our approach.
D.4 DISTILATION ON POWERFUL VISION-LANGUAG MODELS
To further validate the applicability of method larger-scale models, conduct additional experiments using the more powerful DINO-v2 (Oquab et al., 2023) and BGE-1.5 (Xiao et al., 2024) the image and text encoders, respectively. Due computing resource constraints, both encoders are kept frozen, and a single-layer trainable projection head was added on top of them. Table 16: Results DiNo-v2 BGE-1.5 MS-COCO. The model trained on the full dataset performs: @ 1=22.5, @@55=50. IR @ 10=65.0,
TR@ 1=31.7, TR @ 5=61.4, TR 10=74.0.
Pairs
LoRS
Ours
IR@1
IR@5
TR@1
TR@5
IR@5
IR@10
TR@1
TR
100
5.8
20.6
30.7
10.2
29.4
39.8
8.1
23.5
34.9
13.5
33.8
46.4
200
20.9
12.0
43.1
8.4
24.1
35.4
14.1
35.6
48.4
500
7.7
22.9
33.3
12.7
33.5
46.1
9.1
25.9
37.2
15.0
36.5
50.1
As shown in Table 16, evaluation the COCO dataset shows that method remains effective even when scaling up the model capacity. This demonstrates that the proposed approach generalizes well across different backbone sizes.
23

---

Published as a conference paper at ICLR 2026
D.5 CROSS ARCHITECTURE GENERALIZATION
To validate the generalisation performance of the synthetic dataset, we conducted cross-model eval- uation. The data is distilled with NFNet+BERT and evaluated on other architectures such as RegNet (Radosavovic et al., 2020) and DistilBERT (Sanh et al., 2019). As shown in Table 17, method demonstrates good generalisation performance and outperforms the baseline method.
Table 17: Cross architecture generalization.
Method
Model
IR@1
IR@5
IR@ 10
TR@1 1
TR@5
TR
Mean
NENET + BERT
12.1
33.2
45.9
16.6
42.2
55.7
34.3
LoRS
NEÑET + DISTILBERT
7.6
24.5
34.7
12.7
34.5
46.1
26.7
REGNET BERT
4.5
15.0
23.7
5.4
21.0
32.4
17.0
NENET + BERT
14.6
39.1
52.5
21.5
50.6
64.0
40.4
Ours
NENET + DISTILBERT
11.3
29.8
41.3
19.0
41.3
56.5
33.2
REGNET BERT
4.2
14.4
22.3
8.7
23.8
36.0
18.2
D.6
TIME AND MEMORY OVERHEAD
We recorded the computational time and memory consumption of our proposed method and the LoRS method across key experiments, as shown in Table 18. Specifically, we documented the average time required per iteration to update the synthetic dataset during distillation (in seconds per iteration) and the peak and average memory consumption throughout the process (in gigabytes). It can be observed that our method incurs slightly lower time and VRAM costs than the original approach. This is attributable to our phased division, wherein each phase distils only a subset of data, eliminating the need to store computationally intensive information across entire phase.
Table 18: Runtime and memory comparison across different numbers of pairs.
Method
time (s/iter)
peak memory (GB)
avg memory (GB)
200
500
100
200
200
500
LoRS
4.522
4.913
7.392
13.713
16.333
20.942
5.339
5.586
7.985
Ours
4.570
4.554
7.250
13.768
13.818
20.363
5,390
5.404
7.592
E AN ATTEMPT AT THE NCFM METHOD
Introduction. Neural Characteristic Function Matching (NCFM) (Wang et al., 2025c) has recently demonstrated outstanding performance in uni-modal dataset distillation. first uses an encoder to extract nigh-dimeessonal vector representations of images, then calculates the Neural Characteris- tic Function Discrepancy (NCFD) as the matching loss between the real and synthetic datasets. employs minmax adversarial update strategy optimise the frequency sampling network max- imise this loss, while updating the synthetic dataset to minimise it. The distillation process can be represented follows:
min max L(D,D.f.e) min max
Zm D
D
(17)
min max E D.DDD /. [Chilt:
D
where Chf(t) = (1, 22 (1))
2 (t)), and can reformulated as:
Chf(t: = a (1 - 4 (t)
(18)
+ (1 - a)
1 f(,w)) | (ta) cos a(aaa)))))
Here: (t) = Eg [ei(t.a)]] = J, (t.a)) dF(x) is the characteristic function of æ; is the frequency argument; (1 cos (a)())() - (()) denotes the phase difference; and f is the image encoder.
Our Attempts. Uni-modal data set distillation is mainly used for classification tasks, where NCFM distills ipe synthetic data for each category. Multi-modal data distillation is used for
24

---

Published as a conference paper at ICLR 2026
image-text matching retrieval tasks, which do not have category concepts, it is not possible to directly apply this method.
To migrate it multimodal data set distillation, we first perform clustering the original image-text paired data set. Specifically, for each data point (I,, y1), we use the trained teacher models ft and ff extract its high-dimensional representations u, (x;) and = (v)), and concatenate these two modalities' nigh-dimensional representations to obtain the vector representation of this data point. Then, we perform Mini-Batch K-Means clustering using this vector representation. During distillation, we treat each cluster as a category and extract ipc data points from it for initialisation. We match feature functions of the data points using the same method as NCFM. After passing the encoder, the concatenation of the two modal vector representations corresponds to xr in Eq. 17.
Results. We tested this method on Flickr30K. The learning rate for the sampling net and syn- thetic dataset was set to 0.1, the sampling frequency was set to 512, and the weight of the phase difference term in the loss function was set to 0.5. The results are shown Table 19.
Table 19: Results of the NCFM method on Flickr30K.
Clusters
ipc
IR @ 1
IR@5
IR@ 10
TR@1 1
TR@5
TR@ 10
Mean
100
1
0.7
3.2
5.7
2.1
7.1
10.5
4.9
100
2
1.4
5.5
9.6
2.9
9.3
14.5
7.2
100
5
3.0
10.7
17.9
4.8
15.8
25.8
13.0
200
1
1.4
5.5
9.6
2.9
9.3
14.5
7.2
200
2
2.5
9.6
15.6
5.6
15.9
23.8
12.2
500
1
3.2
11.7
18.9
6.1
18.1
26.0
14.0
The effectiveness of NCFM inferior to that of methods based on trajectory matching. We speculate that this may be due to the sparsity of multimodal datasets and the more difficult retrieval tasks.
F VISUALIZATION OF DISTILLED DATA
In this section, we present visualizations of subset of the synthetic dataset distilled by our proposed method. As illustrated in Figure 7 and Figure 8, the synthesized samples exhibit two notable char- acteristics. First, the textual descriptions become significantly more informative and semantically rich, providing more nuanced guidance for downstream learning. Second, the visual content con- tains subtle and complex perturbations that difficult for the human eye interpret, yet potentially essential for enhancing student model learning.
These findings suggest that method does not simply replicate the teacher's outputs, but rather generates enriched supervision signals that span both linguistic and visual modalities. This fur- ther demonstrates the effectiveness of our approach in constructing high-quality synthetic data for multimodal knowledge distillation.
G LLM USAGE
We primarily utilise large language models to assist with writing tasks. Specific applications include:
Inquiry regarding how adjust the layout, such spacing between elements like formulas, headings, algorithms, etc.
Auxiliary table generation: Copy data from an Excel spreadsheet to an LLM to generate LaTeX table source code.
To avoid excessive blank lines at the end of each line, inquire how the LLM might add or remove existing text.
For some formulas, employ large language models generate preliminary versions, which are then manually revised.
Enquire about LaTeX syntax, such as importing relevant packages and setting colours.
25

---

Published as a conference paper at ICLR 2026
man in a purple shirt rides a bike.
shorts and pink polo shirt, with large black backpack and headphones, riding bike.
A band sings and play their guitar.
performing with playing guitar and singing into
microphone, another holding harmonica.
a rural village in India, two men are energetically playing stick- ball game that resembles casual form baseball.
Two people sitting around campfire.
Two people, one standing and one sitting, look at the camera, as they are situsted around campsite fire.
Two men playing a game of baseball.
woman wears while presenting a variety of fruits in front of her, engaging with potential customers.
Man using laptop at table in empty outdoor dining area.
A man in a suit and tie sits alone at table, possibly working on his laptop in the serene setting.
Asian lady selling fruit.
A Violinist plays he is decorated clown.
A dressed formal attire with glasses and tie passionately plays the violin.
Five young men in a grassy playing game.
A group of boys young play field game together in grassy open space,
Figure 7: Flickr30KK before and after distillation. (Left) The original image-text pairs before the distillation. (Right) The image-text pairs after distillation.
26

---

Published as a conference paper at ICLR 2026
meal is presented in brightly colored plastic trays.
A colorful assortment of food is presented in close-up inside bright plastic trays.
A young man riding a skateboard into the air.
young man soars into the air on his skateboard, while nearby group of teenagers also take turns jumping ramp.
A tall giraffe stretches its neck to A giraffe standing up feed from the branches of nearby tree
big airplane flying in the big blue sky
large airplane with four engines soars through the vast blue sky, showcasing its size and flight capabilities.
tree, standing gracefully its natural habitat.
flower vase sitting porch stand.
decorative white vase filled with various colored flowers is neatly placed on stand out on porch.
oven with stove on top of it in kitchen.
kitchen scene shows stove with oven below and lighted hood above.
delicious-lookin piece of chocolate cake sits atop clean white plate, accompanied by fork.
A train coming stop on the tracks out side.
A long train is approaching stop on outdoor tracks, slows down on the railway line.
A chocolate cake and fork ready be eat
Figure 8: COCO before and after distillation. (Left) The original image-text pairs before the distil- lation. (Right) The image-text pairs after distillation.
27


Reviews:
[{"review_id": "FmGoyixEoP", "reviewer": "Reviewer_dydj", "summary": "This paper identifies a issue called \"phased knowledge gap\" in multimodal dataset distillation where student models fail to learn effectively from teacher models in the later stages of training. This paper then proposes Phased Teacher Model with Shortcut Trajectory (PTM-ST) to address this challenge. The core idea is to decompose the distillation process into phases, using different teacher models at each stage to provide more stable guidance. Additionally, it introduces a \"Shortcut Trajectory\" to create a smoothed and stabilized learning path for the student model. The proposed method is evaluated on Flickr30K and COCO datasets for image-text retrieval, demonstrating significant improvements over existing state-of-the-art methods.", "questions": "The authors are encouraged to compare with the following NeurIPS 2025 paper.\n\nEfficient Multimodal Dataset Distillation via Generative Models.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Novelty. The paper identifies the \"phased knowledge gap\" as a critical issue in multimodal dataset distillation, where student performance degrades when using teacher models from later training stages. The proposed PTM-ST framework is a novel and effective solution that addresses this problem through phased learning and trajectory stabilization.\n\n- Strong empirical performance. The experimental results on Flickr30K and COCO datasets show that PTM-ST consistently and significantly outperforms existing baselines across all metrics.\n\n- Thorough analysis and ablations. The paper provides a comprehensive analysis of the problem, supported by visualizations of gradient instability and theoretical arguments for the proposed solution's stability. The ablation studies in Tables 3 effectively demonstrate the contribution of each component of the PTM-ST framework (PTM, ST, and EMA) to the overall performance improvement.", "weaknesses": "- The PTM-ST framework introduces additional complexity, requiring manual specification of interpolation endpoints and matching ranges for each distillation stage. This may make the method difficult to apply to new datasets or tasks and raises concerns about hyperparameter sensitivity.\n\n- Limited Scope of Evaluation: The experiments are limited to image-text retrieval. It would be beneficial to evaluate the generalizability of PTM-ST on other multimodal tasks, such as Visual Question Answering (VQA) or image captioning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wkaLv97a8P", "reviewer": "Reviewer_nf5n", "summary": "- The paper addresses dataset distillation, a key topic for enabling fast and cost-effective training of student models.\n- It proposes a novel dataset distillation method specifically designed to be effective in multimodal settings.\n- The core contribution is using intermediate checkpoints from a teacher model to distill datasets that capture the teacher's learning trajectory.\n- The method utilizes sets of teacher checkpoints to create the distilled dataset, rather than sampling from individual steps, to provide a more global view.", "questions": "See above.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The method demonstrates promising empirical results, achieving strong and consistent improvements over the compared baselines.\n- The approach of using sets of teacher parameters (checkpoints) effectively provides the student model with a global perspective on the teacher's training dynamics.", "weaknesses": "- The method requires access to intermediate teacher checkpoints. For very large models (the bigger CLIP variants), these are often unavailable, and replicating the teacher training to generate them introduces significant computational overhead.\n- The student model's architecture and specific training details are missing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UkzUPhBCcW", "reviewer": "Reviewer_Bjes", "summary": "This paper presents a data distillation technique that aims to compress large multimodal datasets into a small set of synthetic samples that can train new models to achieve comparable performance for multimodal learning purposes. The authors propose the Phased Teacher Model with Shortcut Trajectory (PTM-ST) approach, which divides the training process into different phases by splitting the teacher trajectory and applying shortcut alignment between them, achieving a 13.5% improvement in retrieval metrics and an average gain of 9.53% on Flickr30K.", "questions": "It remains unclear how the proposed phased-teacher strategy generalizes under large-scale training. How do the observed “phase gaps” in representation or gradient dynamics behave when the dataset size grows substantially?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "* Strong motivation. The authors explore phased knowledge gaps in multimodal dataset distillation by investigating the insight differences in distillation dynamics between multimodal and unimodal data distillation techniques, offering valuable finding into modality-specific learning behavior.\n\n* Proposes a simple yet effective phased distillation strategy (PTM-ST), which divides the learning process into semantic phases and aligns them with the teacher’s learning dynamics for trajectory endpoint matching, improving overall distillation stability and performance.", "weaknesses": "- **Limited data scale and generalizability evidendence:** The experiments are conducted on small datasets (e.g., Flickr30K, COCO) with limited computational resources, which may not capture the knowledge-gap behavior at large scale. As a result, the observed early-phase effectiveness and multi-stage training strategy may not generalize to full-scale CLIP-style pretraining.\n\n- **Additional training cost and model complexity:**  Requires more training and model complexity. Although the method aims to improve efficiency, it still requires fully training a large teacher model and dividing it into multiple phases, which substantially increases both computational and storage costs\n\n- **Narrow experimental scope and weak comparison**: The experiments are confined to small benchmarks and a single teacher architecture (CLIP-base), without comparisons against stronger or larger vision–language models. This narrow setup limits the evidence supporting that “information-gap phasing” or session partitioning is a generally effective approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CaRznE4MbP", "reviewer": "Reviewer_SUGG", "summary": "This paper proposes PTM-ST, which aims to address the instability of knowledge transfer from the teacher model during different training phases in MDD. The core idea lies in combining PTM and ST to capture the evolving knowledge of the teacher model across different training stages and to smooth its optimization trajectory, thereby enhancing the stability and effectiveness of knowledge distillation.", "questions": "See wekness.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper conducts the first systematic analysis of the phenomenon of “phase-wise knowledge drift” in multimodal distillation, and validates its existence through both theoretical derivation and empirical evidence. The proposed method is evaluated on two mainstream datasets, Flickr30K and COCO, with consistent and significant performance gains demonstrated through comprehensive ablation studies.", "weaknesses": "1. The PTM-ST framework appears to be highly sensitive to the choices of phase partition ( P ), matching intervals, and endpoint selection. If these hyperparameters are not properly tuned, will the model’s performance collapse or degrade significantly? Can these parameters be made adaptive rather than manually specified?\n2. Your Proposition 1 relies on the second-order smoothness assumption, which is often invalid for large Transformer architectures. How do you justify the practical relevance of this assumption, and have you conducted any empirical validation or relaxation of it?\n3. The proposed method still requires access to the entire teacher training trajectory, which seems computationally expensive. If only partial teacher checkpoints or commercial API-based teacher models are available, can your method still be applied effectively, or does it fundamentally depend on the full training process?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)