Asynchronous Matching with Dynamic Sampling for Multimodal Dataset Distillation
Authors: Ding Qi
Jian Li
Shuguang Dou
Zifan Song
Junyao Gao
Yabiao Wang
Chengjie Wang
Cairong Zhao
Venue: ICLR 2026 Poster (2026)
Keywords: Trajectory Matching
Dataset Distillation
Primary area: optimization

Abstract: Multimodal Dataset Distillation (MDD) has emerged as a vital paradigm for enabling efficient training of vision-language models (VLMs) in the era of multimodal data proliferation. Unlike traditional dataset distillation methods that focus on single-modal tasks, MDD presents distinct challenges: (i) the effective distillation of heterogeneous multimodal knowledge, complicated by feature space misalignment and asynchronous optimization dynamics; and (ii) the lack of discrete class guidance, which hinders the distribution coverage and representativeness of synthetic data due to the vastness and continuity of the semantic space. To address these challenges, this paper proposes an Asynchronous Matching with Dynamic sampling (AMD) framework. AMD enables asynchronous trajectory matching by decoupling the selection of starting points for image and text trajectories. Additionally, a Semantics-Aware Prototype Mining module is introduced, which replaces random initialization by leveraging feature-space clustering to identify representative prototypes, enhancing the coverage and representativeness of the distilled samples. Extensive experiments demonstrate that AMD achieves superior distillation performance on Flickr30k and COCO (e.g., IR@1, IR@5, and IR@10 \textbf{gains of 4.5\%, 9.6\%, and 10.9\%}, respectively, on Flickr30k 200 pairs.) with negligible computational overhead.

Published as a conference paper at ICLR 2026
ASYNCHRONOUS MATCHING WITH DYNAMIC SAM- PLING FOR MULTIMODAL DATASET DISTILLATION
Ding Qi'
Jian Li
Shuguang Dou'
Zifan Song'
Junyao Gao
Yabiao Wang
Chengjie Wang²
Cairong Zhao
Tongji University
Tencent YouTu Lab
ZZejjang University
NNanjing University
ABSTRACT
Multimodal Dataset Distillation (MDD) has emerged as vital paradigm for en- abling efficient training of vision-language models (VLMs) in the era of multi- modal data proliferation. Unlike traditional dataset distillation methods that focus single-modal tasks, MDD presents distinct challenges: (i) the effective distilla- tion of heterogeneous multimodal knowledge, complicated by feature space mis- alignment and asynchronous optimization dynamics; and (ii) the lack of discrete class guidance, which hinders the distribution coverage and representativeness synthetic data due to the vastness and continuity of the semantic space. To address these challenges, this paper proposes an Asynchronous Matching with Dynamic sampling (AMD) framework. AMD enables asynchronous trajectory matching by decoupling the selection of starting points for image and text trajectories. Addi- tionally, Semantics- places random initialization by leveraging feature-space clustering to identify rep- resentative prototypes, enhancing the coverage and representativeness of the dis- tilled samples. Extensive experiments demonstrate that AMD achieves superior distillation performance on Flickr30k and COCO (e.g., IR@ 1, IR @5, and IR @ 10 gains of 4.5%, 9.6%, and 10.9%, respectively, Flickr30k 200 pairs.) with negligible computational overhead..
Prototype Mining module is introduced, which re-
INTRODUCTION
In the era of massive data, the substantial storage, transmission, and computational expenses sociated with large-scale datasets pose a significant bottleneck for deep learning model training and iteration. Dataset distillation (DD) (Wang al., 2018; Zhao et al., 2020; Zhao & Bilen, 2023; Cazenavette et al., 2022) emerged to address this challenge, with the aim of distilling a small amount of synthetic data that allows models trained this reduced set achieve performance similar to the original. This approach significantly reduces data volume, lowers training costs, accelerates research, and aids in data privacy protection (Dong et al., 2022: Loo et al., 2023).
However, most existing DD research is mainly focused on single-modal tasks. With the explosion of multimodal data like image-text pairs and the rise of Vision-Language Models (VLMs) (Radford et al., 2021), efficiently processing and utilizing this massive multimodal data presents a new and critical challenge. Consequently, research specifically targeting Multimodal Dataset Distillation (MDD) is becoming exceptionally crucial, offering vital pathway for efficient multimodal model training and deployment.
Despite its vital role, MDD faces unique challenges distinct from prior dataset distillation paradigms such as image classification (Zhao Bilen, 2023; Wang et al., 2022; Zhao et al., 2023; Du et al., 2023) and text classification (Li & Li, 2021: Maekawa et al., 2025). MDD presents two main chal- lenges: (i) Efffcttive distillation of heterogeneous multimodal knowledge. The core of MDD lies in extracting and condensing effective joint knowledge from heterogencous modalities like im- age and text into synthetic data. This process is much more complex than single-modal tasks, as
*Corresponding author (zhaocal rongêtangjt 1- edu. en). Work done when Ding Qi is an intern in Tencent YouTu Lab, and the advisor is Jian Li.

---

Published as a conference paper at ICLR 2026
Ima
KFTTAR
AND
Text
Expest Traoctorics
+
iRat1
19
Previous Synchromous Trajeckory Matching
(b) Asynchronous Trujectocy Matching
(c) Performance Comparison
Figure 1: Synchronous vs, Asynchronous Matching: (a) Previous synchronous matching only paired same- time expert parameters. (b) Our asynchronous matching explores more flexible and richer cross-modal expert
parameter combinations. (c) Performance Flickr30k: synchronous methods (Wu et al., 2023; et al., 2024)
vs. our AMD.
it involves misalignment between modality-specifie feature spaces and asynchronous optimization dynamics during training: the latter. in particular, has been largely overlooked in previous work (Wu et al., 2023; Xu et al., 2024) but is crucial for capturing precise cross-modal correlations. (ii) Distri- bution coverage and representativeness without discrete class guidance. Traditional DD meth- ods often benefit from the natural guidance and structure provided by discrete classes. However, MDD lacks such clear class distinctions. Coupled with the vastness and continuity of the image-text data's semantic space, simple random initialization struggles to effectively cover the diverse joint distribution of the original data. Without clear guidance, initial points selected by random meth- ods in previous work may lack representativeness (e.g.. corresponding ambiguous descriptions or low-quality images), affecting distillation quality and subsequent optimization.
To address the aforementioned challenges and limitations, this paper proposes novel Asynchronous Trajectory Matching (AMD) framework for MDD. In contrast to prior approaches that typically synchronize image and text trajectories by selecting model parameters from the same training stage, as shown in Figure I(a), our preliminary experiments suggest that such rigid synchronization suboptimal for synthesizing modality-specific data representations. This limitation stems from the inherent heterogeneity in the learning dynamics of different modalities. To overcome this, our framework adopts an asynchronous trajectory matching strategy, illustrated in Figure 1(b), that de- couples the sampling stages of image and text trajectories, allowing for more diverse combinations of image and text model parameters drawn from different training epochs. This increased flexibility facilitates the optimization of synthetic image-text pairs. Additionally, address the absence of discrete set of classes to guide the distillation process, we introduce a Semantics-Awar Prototype Mining module that constructs cluster centers serve grounding references. This module per- forms clustering in the feature space to identify representative sample prototypes. These prototypes replace the randomly selected initial points used in prior methods and are employed initialize the synthesis process, thereby substantially enhancing the diversity and representativennss of the dis- tilled samples. Notably, these improvements are achieved with negligible additional computational overhead compared to existing methods.
Our main contributions are summarized as follows:
We propose a novel asynchronous matching with dynamic sampling for MDD that ad- dresses the limitations of synchronous methods by enabling asynchronous sampling of im- age and text trajectory points to explore richer cross-modal learning dynamics.
We introduce semantics-aware prototype mining module that identifies representative prototypes via clustering in the joint semantic space to provide high-quality initialization, significantly enhancing the coverage and diversity of distilled samples.
Extensive experiments demonstrate that our method achieves significant performance im- provements on Flickr30k and COCO. For instance, on Flickr30k (200 pairs), Image Re- trieval metrics IR @ IR @55, and IR @ 10 improve by 4.5%, 9.6%, and 10.9%, respectively.
2 PRELIMINARY
Multimodal Dataset Distillation. We first provide a formal definition of Multimodal Dataset Dis- tillation (MDD). Given large-scale image-text dataset T ((x,, y: where 2; denotes an
2

---

Published as a conference paper at ICLR 2026
image sample, Y/ represents the corresponding text deseription, and N IT] is the size of the original dataset. The goal of MDD to compress T into a budget-constraing synthetic dataset S = (ā új) ر with M < N,, such that models trained on S approximate the performance of those trained on T. This objective can be formulated as:
(T |(00] (x), -(0 s.
where 03 and 0% are parameters of multimodal model trained on T B and 0 are trained on S, Test is the test data distribution, denotes the performance measure function, e a small tolerance.
Since Aision-languaee datasets lack the category-leve labels found in traditional classification tasks, methods such as gradient matching (Zhao et al., 2020) and distribution matching (Zhao & Bilen, 2023), which rely on intra-category data compression, struggle to be effective.
Matching Image- Trajectories. Existing works (Wu et al., 2023; Xu et al., 2024) employ MTT- based methods (Cazenavette et al., 2022; Cui et al., 2023) to compress key information and cross- modal relationships through: (1) expert trajectories buffering and (2) image-text pairs distilling.
During buffering, the multimodal model is first trained on dataset TT using the bi-directional In- foNCE loss, which consists of symmetric image-to-text and text-to-image contrastive terms. The image-to-text contrastive loss can be formulated as:
exp (s(0v(z.), (c(()))))((()/)))
LInfONCE =
log
Dj=1 exp (s(0v(z;), 0c (2:))//)
where .(:-)) measures similarity between positive pair (Ia, v1) and K negative pairs (.;, Uj) in batch, with temperature т. To ensure the generalization capability the expert trajectories, is common practice to perform multiple rounds of retraining and periodically save the parameters of the image encoder By and the text encoder 0c at different training steps, thereby constructing the expert trajectories. One of them can be formalized as: image trajectories = " , ",")) and text trajectories = (02 (("")..,"!"),,
). r total training epochs.
During the distilling phase, matching is performed between student and expert trajectories in both the vision (V) and language (C) modalities. At initialization step t, the student and expert networks share identical parameters (0 00()). The expert trajectory undergoes M optimization steps to reach (oL (f) of( ((+++))), while the student network performs N gradient descent updates (N < M)
to obtain its final parameters (00H+,,,,(+++ The matching objective minimizes the normalized fz-distance between corresponding student and expert trajectories across both modalities:
(7, i) arg min 100 of
12
|
+
)
110 0 (1
112
| 0 M1)
12
where (0 1190000)000 (6 0(4)) It is noteworthy that, to optimize memory efficiency, LoRS (Xu et al., 2024) leverages TESLA (Cui et al., 2023) technology, enabling the this framework to be exccuted on single GPU.
3
METHODOLOGY
3.1 EXPLORATION
Existing MDD methods typically adopt a synchronous sampling strategy for trajectory matching, perhaps extending the synchronized processing of image and text data in standard VLM training. However, this paper questions the validity such a synchronous matching assumption:
First, the inherent architectural differences between the image and text networks lead to asyn- chronous evolution of their parameter trajectories. Taking NFNet+BERT, a commonly used back- bone in MDD, followed by a linear layer for optimization) exhibit significantly different parameter update dynam- ics. Second, from the perspective of data distillation, the optimization spaces of synthesized images
example, the visual encoder and the text encoder (with BERT often frozen and
3

---

Published as a conference paper at ICLR 2026
Trajectorics
Image Enocder Text Encoder
1000
LOoSs VAVAA
Optimintiion Path
1.00
2019
Image MTT
Loss
"MEECE Lock
Ibe Meration
Esoch Taarittinn Nill
(a) 3D Expert Trajectories Visualization
(b) Parameter Divergence
(c) Distilled Pair Optimization
Exploring the Asynchronous Phenomenon: (a) Visualizes the expert trajectories during the buffering stage; (b) analyzes the parameter update magnitudes; (c) MTT loss curves during the distillation stage.
Figure 2:
(3x224x224 pixel space) and synthesized texts (768-dimensional embedding space) possess funda- mentally different topological properties, making synchronization of their optimization processes difficult.
To empirically investigate these theoretical reasons and validate our skepticism, we conducted sys- tematic experimental analyses on the Flickr30k dataset. Our observations provide strong evidence:
Observation 1. Asynchronous Expert Trajectories. Consistent with the notion of inherent archi- tectural differences, our analysis of expert model training reveals clear asynchrony. As visualized in Figure 2 (a), expert trajectories during the buffering stage show initial synchronization followed by clear decoupling in middle and later stages. Furthermore, analysis of parameter update magni- tudes (Figure 2 (b)) reveals distinct dynamics: the text network undergoes intense initial fluctuations but quickly converges, whereas the image network maintains consistently high update intensity throughout training. This evidence highlights the asynchronous evolution of image and text network parameters.
Observation 2. Asynchronous Synthetic Data Optimization Speed. Supporting the perspective that the optimization spaces of synthesized images and texts are fundamentally different, Figure 2 (c) demonstrates that during the distillation stage, the synthesized text optimizes significantly faster than the synthesized image. This discrepancy in optimization speed provides direct evidence that the optimization processes of synthesized image and text data fundamentally asynchronous.
Based on these empirical findings, we draw two important conclusions:
The asynchrony of expert trajectories is inherent characteristic of visual-language mod- els;
Synthetic text converges significantly faster than synthetic images, further validating the asymmetry in the cross-modal optimization process.
Building on these insights, in Section 3.2, we further propose asynchronous trajectory matching decoupling the distillation paths of image and text modalities realize distillation process that better aligns with the actual optimization dynamics.
3.2 ASYNCHRONOUS MATCHING WITH DYNAMIC SAMPLING.
Building upon the empirical findings presented in the previous section, which revealed the funda- mental asynchronous nature of both expert trajectories and the optimization of synthesized data, we propose novel Asynchronous Matching with Dynamic sampling (AMD) Framework for mul- timodal Dataset Distillation. Unlike conventional synchronous methods that ignore this inherent asynchrony by strictly aligning expert trajectories based the same training steps (l,, = 44), AMD, as shown in Figure 3, enables the independent and flexible selection of starting points (t,,, tt) for the image (8(.) and text (B()) expert trajectories (where t is the training step).
The expert and student trajectories whose states are matched in AMD are generated by the standard visual- language model training process using contrastive loss like InfoNCE. The expert trajectory
4

---

Published as a conference paper at ICLR 2026
is generated by training the network on the real dataset T:
(0l+1) ( = (Bl)) 2 as LINIONCE (T; : ,0()
(4)
Similarly, the student trajectory corresponds training a network on the synthetic dataset S: ( (0++)) 0(++)) = (85)) 2 V LItONNEE ( 0
(5)
Asynchronous Trajectories Matching. AMD then aims to optimize the synthetic data such that points on its resulting student trajectory match points the expert trajectory. Specifically, it minimizes the asynchronous trajectory matching loss (LMM)), which compares student parameters (ž. (corresponding to expert state) expert parameters after
after N steps of optimization M steps, starting from ip and 4, Extending the standard MTT-VL. (Wu et al., 2023) formulation, this objective minimizes the normalized 1.2 distance between student parameters and expert parameters:
LAMD = 1105 +1) 1
|)(++N) 0 00
10 00 1|2
[10(0) f
(6)
s.t. L. E [0, Rv). l E [0, RT].
where Ry and Rg, denote the sampling ranges for visual and text.
Maximum Mean Discrepancy based Dynamic Sampling. We leverage the convergence speed differences between the visual and text expert trajectories to determine dynamic sampling ranges. As shown in Figure (b), first compute the Maximum Mcan Discrepancy (MMD) of trajectory parameters between consecutive epochs. Under a linear kernel, the MMD reduces to the squared Euclidean distance between the average parameter vectors of consecutive epochs. For the visual modality, we define:
MMDv, M ,00)) =
1 ny
(t-1) V.
V.
nV
1=1
Similarly, MMMD2, is computed for the text modality. In practice, this quantity reflects the update magnitude between consecutive epochs and serves surrogate measure of optimization dynamics.
We then compute the median of the trajectory ratio over the complete stored expert trajectory:
{{ MMDY, MMDDE c.t +e |te (L.11)))
Tmedian Median
where is a small constant for numerical stability. The visual and text sampling ranges Ry and R are then determined according to the relative position of the trajectory ratio with respect to Tmediani
MMDv,
Ry = min
-Tmediinn
MMD te +e
MMDva.
R3 = max
I Tmodiaan
MMD +
Intuitively, once the ratio exceeds the median, the text modality has largely stabilized. We therefore truncate text sampling before the crossing point, while allowing visual sampling to extend beyond it. This asymmetric strategy aligns sampling with modality-specin convergence speeds, reducing cross-modal asynchronicity without extra hyperparameters and improving AMD stability and gen- eralization.
3.3 SEMANTIC-AWARD PROTOTYPE MINING
Multimodal Dataset Distillation for non-categorical data like image-text pairs faces the critical chal- lenge of insufficient synthetic data coverage and diversity. Due the data's 's continuous and complex
5

---

Published as a conference paper at ICLR 2026
Algorithm Asynchronous Matching with Dy- namic Sampling
Start
0.0.
match
Input: Real set T: Expert Trajectories
{0e Yzzo [0L Vz Max start epochs Rz, Rv; Synthetic data learning rate y/s-
Output: Synthetic dataset S
2 match
I: Initialize S
(t. 1) using Semantics-
aware Prototype Mining
2: repeat
Select the start epoch separately: [0, Ry] and t E Ry]
3 match
4: Get expert trajectories (0
) Jt=t.
+ and
( +.+ t=
Tmage-Text Experi Trajecterics
Studecl
Initialize the student (0 Bl1)
network
Trajectories
Train (0 oll)) on S for N steps and Figure 3: Illustration of Asynchronous Trajec- tory Matching: Dashed lines represent expert tra- jectories saved during buffering. while solid lines are student trajectories matched during distilla- pairs: tion. Columns represent decoupled expert start- ing points ((t, = 0, f = 1) left, (t, = 1, 0) right), and rows depict subsequent matching steps.
get ( (000 1 and (0 It=l.
fiet,
Compute AMD loss using Eq. 6
8: Update synthetic image-text
(i,9) + (ž, Ø) ns VE.)
9: until convergence
10: return S
nature, traditional random initialization of synthetic samples often selects semantically redundant pairs, significantly harming coverage and diversity. To mitigate this, we propose a novel Semantics- aware Prototype Mining (SPM) module. SPM analyzes the joint semantic feature space of the original dataset to identify representative prototypes, which are utilized as initialization points for the synthetic samples (( 0=1)
SPM involves several steps. First, we extract corresponding visual features = 0y(x;) and text features Oc(w: for every image-text pair (æ4, W:) in the original dataset {x; v !2 using trained image and text encoders 0y and lg. Second, to capture the joint semantic information of each pair, we construct a joint image-text feature fü by simple feature concatenation: fi [v,; 4..
This builds a representation {.fs 12 [== of the original dataset in the joint feature space. Next, we perform K-means clustering on {fi] } in this space, setting the number of clusters K equal to the synthetic dataset budget B (K = B). This yields B cluster centroids (c+)en each representing semantic prototype from the original data distribution:
- K= B), (x vi) = arg min - exl2 Vk E (1...., B}. (10)
For each cluster 1: E {1,.... B). we select the original dataset sample (aī. vi) whose joint feature f& is closest to the centroid eą the representative prototype for that cluster.
This prototype-based approach leverages the semantic structure of the original data to guide the initialization process. By selecting B initialization pairs { ) based on diverse semantic clusters identified in the joint feature space, it maximizes the coverage of the original data distri- bution and ensures the initial synthetic samples highly diverse and representative. This directly addresses the semantic redundancy issue associated with random initialization, providing a high- quality starting for the subsequent distillation optimization.

---

Published as a conference paper at ICLR 2026
Table 1: Performance comparison with four coreset selection and current multimodal dataset distillation meth- ods on Flickr30k (Plummer et al., 2015) dataset. In line with the fair comparison setting of LoRS (Xu et al., 2024), both LoRS and AMD use 99, 199, and 499 pairs, while others uses 100, 200, and 500 pairs. The NFNet (Brock et al., 20RT)+BERT (Vaswami ct al., 2017) model trained on the full dataset yields: IR@ RR@1=27.33 IR @5=57.1, IR @ 10=69.7 for 12T, and TR@ 1339,, TR@5=65.1, TR @ 10=75.2 25. for T2I. Best results are in bold.
Coreset Selection
Dataset Distillation
Pairs
Ratio
Metric
Rand
Herd
K-Cent
Forget
MTT-VL
LoRS
AMD (Ours)
IR11
1.0
0.7
0.7
0.7
4.7+0.2
8.3+0.2
10,4=0.3
IR
4,0
2.8
3.1
2.4
15.720.5
24.120.2
30.5=0.77
100
IR @
6.5
5,3
6.1
24.6=1.0
35.1+
43.0=0.6
0.3%
(99)
TR
L.3
1.1
9.9x0.3
11.8+0.2
14.4:0.5
TR
5.9
4.7
5.0
4.2
28.320.5
35.820.6
39.1=0.6
TR 10
10.1
7.9
7.6
9.7
1=07
49.2+0.5
52.620.6
IR
1.1
1.5
1.5
1.2
4,640.9
8.640.3
13.1
IR
5.5
16.0=1.6
25.320.2
34.9:0.6
200
IR@00 10
9.2
9.3
9.9
8.4
25.5=2.6
36.620.3
47.5=00.7
0.7%
(199)
TR@1
2.1
2.3
2.2
1.5
10.2=0.8
14.5+0.5
16.9=00.4
TR
8.7
8.4
8.2
8.4
28.7=1.0
38.720.5
42.3=0.6
TR
13.2
13.5
41.9=1.9
53.4+0.5
56.2+0.8
@
2.4
3.0
3.5
6.64003
10.0+0.2
15.8=0.4
1R55
10.5
10.0
10.4
9.0
20.2+1.2
28.9+0.7
39.8=0..4
500
@ 10
17.4
17.0
17.3
15.9
30.0=2.1
41.620.6
53.2=0.5
1.7%
(499)
5.2
5.1
13.3=0.6
15.5+0.7
19.3+0.5
T
18.3
16.4
16.4
12.3
32.8=1.8
39.8+0.4
46.4=0..4
T 10
25.7
24.3
23.3
19.3
46.8=0.8
53.7=0.3
60.0=0.6
EXPERIMENTS
4.1 EXPERIMENTS SETUP
Datasets and Metries. Following established MDD protocols (Wu et al., 2023; Xu et al., 2024), we evaluate our approach on the Flickr30K (Plummer et al., 2015) and COCO (Lin et al.. 2014) datasets, which are standard cross-modal retrieval benchmarks containing 31,783 and 123,287 im- ages, respectively, each paired with five aunaan-anotatte captions. We assess retrieval performance using standard Recall (R @ K) metrics with {1, 5, 10), reporting results in both directions: Image-to- Text (I2T),, denoted as IR @ K, which measures the hit rate of retrieving correct captions among the top-K results, and Imaae (T2I), denoted as TR @ K, which evaluates the accuracy of finding matching images based on text queries.
Implementation Details. Following the setup of the LoRS (Xu et al., 2024) baseline, we utilize an NENet (Brock et al., 2021) (Normalizer-Firee Free ResNet) pretrained ImageNet (Deng et al., 2009) as the image encoder, along with a pretrained BERT (Vaswani et al., 2017) model that includes an appended linear layer as the text encoder. In accordance with the protocols of previous work (Xu et al., 2024; Wu et al., 2023), the BERT weights remain frozen during training and distillation, with only the parameters of the linear layer being optimized. Adhering to the MTT (Cazenavette al., 2022), during the buffer phase, we train the image and text encoders on the original dataset for 10 epochs, repeating this process 20 times to generate 20 expert trajectories. We optimize the distilled data using SGD with momentum 0.5. The reported results are calculated as the mean + standard deviation over 15 independent evaluations: we generate 3 synthetic datasets, and for each dataset, we retrain the model 5 times. All experiments are conducted on single NVIDIA V100 / RTX 4090 GPU. More detailed are provided in the Appendix.
Counterpart Methods. We compared two main categories of methods: Coreset Selection and Dataset Distillation. Coreset Selection includes commonly used techniques such as Random (Re- buffi et al., 2017), Herding (Welling, 2009), K-center (Farahani & Hekmatfar, 2009), and For- getting (Toneva et al., 2018). Dataset Distillation encompasses MTT-VL (Wu et al., 2023) and LoRS (Xu et al., 2024). MTT-VL is the first work to apply MTT (Cazenavette et al.. 2022) (training trajectory matching) in the multimodal area, while LoRS enhances similarity mining and incorpo- rates TESLA (Cui et al., 2023) technology reduce memory overhead.
7

---

Published as a conference paper at ICLR 2026
Table 2: Performance comparison with coreset selection and dataset distillation methods COCO (Lin et al., 2014). For fair comparison, LoRS (Xu et al.. 2024) and AMD use 99, 199, and 499 pairs, while others use 100, 200, and 500 pairs. The NFNet (Brock et al., = (Vaswani et al., 2017) model trained on the full dataset achieves: IR @ 1=16.9, 9, IR @5=41.9, 19,, IR @ 10=55.9 , for 12T, and TR @ 1916., TR @ 5=45.6, 6,,,, TR @ 10=59.5 for T2I. Best results are in bold.
Coreset Selection
Dataset Distillation
Rand
Herd
Cenent
Forget
MTTVL
LoRS
AMD (Ours)
IR@1
0.3
0.5
0.4
0.3
1.3+0.1
18800.1
2.8+0.2
1.5
5.4+0.3
7.1+0.2
IR 10
2.7
3.5
2.5
2.5
9.5=0.5
12.2+0.2
17.2:0.4
0.02%
(99)
TR 1
0.8
0.8
1.4
0.7
2.5+0.3
3.3+0,2
4.1=0.3
TR
3.0
3.7
2.6
10.0a0.5
12.240.3
13.840.3
5.5
4.8
15.7+0.4
21.8+0.4
IR@1
0.9
0.7
0.6
1.7+0.1
2,480.1
3.8:0.2
IR
2,3
2.4
2.1
2,8
6.5=0.4
9.3a0.2
13.4
200
0.04%
IR@00
4,4
4.1
5.8
4.9
12.3+0.8
15.5+0.2
21.4:0.4
(199)
1.2
3.320.2
4.620.2
T
4.0
3.6
3.8
3.5
11.940.6
14.2x0.3
15.5
TR 10
7,2
7.7
7.5
7.0
19.421.2
22.6+0.2
24.1=0.5
IR01
1.1
1.7
1.1
0.8
2.5g0.5
2.840.2
4.2:0.2
14.2+0.5
500
0.09%
IR 10
8.7
9.9
10.5
8.2
15.8+1.5
16.5+0.7
22.3+0.4
(499)
TR
1.9
1.9
2.5
2.1
5.0+04
5,3+0.5
5.7=0.6
TR@5
8.7
8.2
17.2=1.3
18.3+1.5
19.3+1.2
@
12.5
13.7
14.3
13.0
26.0+1.9
27.9+1.4
28.7=1.0
4.2 QUANTITATIVE RESULTS
As shown in Tables and 2, our AMD demonstrates significant advantages over existing approaches across both datasets. On the Flickr30k dataset, AMD achieves new state-of-the performance, significantly surpassing traditional Coreset Selection methods and existing dataset distillation tech- niques. For instance, under the setting of 200 image-text pairs, AMD improves I2T over the LoRS baseline with gains of +4.5%, +9.6%, and + -00.9% IR I, @, and IR @ 10, respectively. For T2I, AMD achieves improvements of +2.4%, +3.6%. and +2.8% in TR @ 1, TR@5, and TR@ 10, spectively. Given that the COCO dataset is 3.9x larger than Flickr30k and contains richer semantic relationships, the performance for all methods superiority on the more complex COCO dataset. For example, under the setting of 200 pairs, AMD achieves improvements of +1.4%, +4.1 ,, and +5.9% in IR @ IR@5, and IR@ @ 10, respectively, compared to the LoRS baseline.
relatively lower. Nevertheless, AMD maintains its
Additionally, we observed two noteworthy phenomena. First, the distillation budget (the num- ber of image-text pairs) increases. the performance gains from the asynchronous trajectory become more pronounced. In the case of Flickr30k, AMD outperformed LoRS by 2.1%, 4.5%, and 5.8% in the IR @ metric for 99, 199, and 499 pairs, respectively. This suggests that previous synchronous trajectory strategies may create performance bottlenecks when handling large-scale data, as they struggle to adapt to the increasing complexity of image-text pairs. Second, the improvements in 12T retrieval metrics are particularly significant, indicating that the previous synchronous trajec- tory approach, which foreed the matching of imbalanced image and text expert trajectories, led to optimization challenges for synthetic image-text pairs. In contrast, our proposed asynchronous tra- jectory technique enables flexible optimization of image and text combinations at different stages, resulting in substantial performance enhancements.
4.3 ABLATION STUDY
Component Analysis. We conduct comprehensive ablation study on the proposed AMD frame- work to analyze the contribution of its key components: (1) Baseline (using the codebase of LoRS (Xu et al., 2024)), (2) Asynchronous Trajectory Matching (AMD), and (3) Semantic-aware Prototype Mining (SPM). As shown in Table 9, the Baseline (LoRS) provides a solid starting point. Adding the AMD component alone yields significant improvements across all metrics (c.g.. IR @ increases from 8.6% to 12.1 1)), highlighting the effectiveness of explicitly modeling and leveraging asynchronous learning dynamics during distillation. Adding the SPM component the baseline also provides noticeable gains (c.g., IR @ increases from 8.6% to 9.1%), demonstrating the importance of semantic-aware initialization for enhancing synthetic data coverage and diversity. Importantly, the
8

---

Published as a conference paper at ICLR 2026
Table 3: Ablation study of proposed modules. Experiments conducted on the Flickr30k 200pairs setting. The best results are bold.
Baseline
AMD
SPM
IR@1
IR@5
IR@10
TR@1
TR@5
TR@ 10
8.6+0.3
25.3=0.2
36,6+0.3
14.5+0.5
38.7+0.5
53,4+0.5
12.1=0.3
33.9=0.6
46.7=0.5
16.5=0.4
41 41.6+0.6
55.7+0.5
4
9.1
26.4=0.3
38.5+0.4
15.3+0.4
53.9+0.4 40.1+0.5
13.1+0.3
34.9=0.6
47.5+0.7
16.9+0.4
42.3+0.6
56.2+0.8
Table 4: Cross-archieccture generalization: We utilize NENT as the dataset distillation model to gener- ate synthetic images and evaluate performance across various architectures.
Flickr30k
Pairs
Method
Evaluate Model
IR
IR
IR@10
TRØ1
TR@5
TR@10
NFNet-BERT
10.000.2
28.940.7
41.620.6
15.540.7
39.820.4
53.7.0.3
499
LoRS
ResNet-BURT
3.3e0.2
12.740.3
20.4=0.2
6.840.2
19.6.1.3
31.140.3
RegNet+BERT
3.5a0.1
12.640.3
21.120.4
6.840.3
20.820.3
30.240.3
NENG
15,8204
39.120.4
53.2=0.5
19.340.5
46.450.4
60.020.6
499
AMD
BesNettBBRRT
4.1+0.3
14.240.5
22.6+0.4
7.6004
22.3+0.4
33.240.6
RegNet-BEKT
4.0=0.2
14.4+0.3
23.1=0.5
7,540.3
22.8+0.5
32,740.4
Table 5: Upper Bound Analysis on Synthetic Data Scale.
Method
Image Encoder
Text Encoder
Ratio
TR@1
TR@ 10
IR@1 1
IR@ 10
AMD
NFNet
BERT
10%
32.5
73.9
24.7
67.3
Upper Bound
NENet
BERT
100%
33.9
75.2
27.3
69.7
AMD
NENE
CLIP
10%
60.5
91.7
47.9
86.8
Upper Bound
NFNet
CLIP
100%
61.2
92.8
49.8
88.3
Macc bocking bronco
grvsh with the dig green taking lead
Iwo haree are right raca
band perfirmu crond of
ruas help coubby who fell off care during rodeo
deutenee, galariat and vialelit play their insorunnets stage
sttcepling to back off cantay the redio
Figure 4: Visualization of initial (Left) and synthetic (Right) image-text pairs, with the synthetic data under- going 3000 distillation steps.
improvement from AMD alone is generally larger than that from SPM alone, particularly for Im- age Retrieval metrics. When combining both AMD and SPM, full framework achieves the best performance across all metrics, reaching 13.1 1//4,999 4.44%% /47.5% % for IR and 16.9% /42.3%/56.2 9%/5..2% % for TR. These results underscore the complementarity of the proposed AMD and SPM components, and their combined effect leads to superior performance in MDD.
Cross-Architecture Generation. Cross architecture generation aims assess the ability of syn- thetic datasets to generalize to unseen architectures. We employed NFNet+BERT as the distillation model to generate the synthetic dataset, and then evaluated it using various architectures. Since BERT is frozen, we followed the baseline approach of LoRS by modifying the image encoder ar- chitecture, including ResNet (He et al., 2016) and RegNet (Radosavovic et al., 2020). As shown in Table 4, the dataset generated by AMD method outperforms the baseline LoRS in cross- architecture performance.
Upper Bound Analysis. We investigated the AMD performance upper bound by scaling up syn- thetic data quantity. As detailed in Table 5, with the CLIP encoder, AMD trained on a 10% synthetic subset achieves 47.9 IR @ recovering over 96% of the full dataset's upper bound (49.8). The 10% synthetic subset, with the BERT encoder, recovers 90.4% of its upper bound (24.7 vs. 27.3 IR@ I). This demonstrates the high performance ceiling and potential of the AMD approach.
4.4 QUALITATIVE RESULTS.
Synthetic Image-Texx Text Pairs Visualization. Figure 4 provides qualitative examples of synthetic image-text pairs generated by our method. Similar to prior work (Wu et al., 2023; Xu et al., 2024), synthetic images exhibit deepdream-like style (Zeiler & Fergus, 2014), characterized by real

---

Published as a conference paper at ICLR 2026
a black dog runs through 8 held)
teams, wearing green
one weening rec. playing a geme
brown fonoo
of soccer:
mar standing nest to motorgycle:
two dogs are running next to each other on the graee
ancwwbaarder flying
Figure 5: 1-SNE visualization initialization strategies. (Left) Random selects samples haphazardly, lcad- ing to semantic redundancy and poor coverage. (Right) SPM (ours) identifies representative prototypes via clustering, ensuring uniform, diverse coverage of the semantic manifold. Each cluster is color-coded.
(4) Initial Similarity Matrix
(b) Syne Matching (LoRS)
Asyne Matching (Ours)
Diagonal Similarity Evolution
Figure 6: Qualitative analysis of similarity matrices and their evolution. (a) The initial similarity matrix, where only diagonal elements are 1.0. (b) The similarity matrix synchronous matching using the LoRS baseline. (c) The similarity matrix after asynchronous matching with our AMD framework. (d) The evolution of mean diagonal similarity over iterations, demonstrating the superior convergence of AMD compared to the LoRS.
images overlaid with learned feeh-frequency components. The examples demonstrate that our syn- thetic text, even if not longer, is significantly more effective at capturing salient objects and rela- tionships compared to the initial synthetic text (i.e., before optimization). For instance, the last case clearly describes the performers and their relationships, detail often lacking initially but crucial for high-quality VL understanding.
Initialization Strategies Analysis. As shown in Figure 5, t-SNE visualization reveals that randomly initialized prototypes (red stars) cluster around few similar semantics (e.g.. multiple "dogs running on grass"). leading limited and biased coverage. In contrast, SPM employs K-means select representative prototypes that uniformly span distinct semantic clusters, such as "soccer matches," "motocross," and "snowhoarding." This diversity-driven initialization improves semantic coverage and fidelity, providing stronger foundation for subsequent distillation.
Similarity Matrix Evolution. As shown in Figure 6, we visualize the evolution of the low-rank similarity matrix during distillation. Figures (a)-(c) correspond to initialization. LoRS, and AMD, respectively. By decoupling image and text trajectories, AMD enables stable text optimization asynchronous matching at optimal convergence and frees image synthesis from stage- wise straints, allowing optimization along more informative gradients. This results stronger diagonal dominance and better suppression of off-diagondl clements. Figure (d) further shows that AMD converges faster and achieves higher final diagonal values than LoRS.
5 CONCLUSION
In this study, we introduce the Asynchronous Matching with Dynamic Sampling framework for Mul- timodal Dataset Distillation, AMD. We propose novel components: asynchronous trajectory matching strategy that decouples image and text parameter sampling to leverage differential conver- gence rates, and semantics-aware prototype mining module leveraging clustering for representative initialization. Experimental results demonstrate AMD achieves superior distillation performance on Flickr30k and COCO with negligible computational overhead, offering an efficient and scalable solution for mitigating data bottlenecks.
10

---

Published as a conference paper at ICLR 2026
6 ACKNOWLEDGMENT
This work was supported by National Natural Science Fund of China (No. U25A20527, 62473226)) This work was also supported by Shanghai Municipal Science and Technology Major Project (No. 2025SHZDZX025G10),
REFERENCES
Andy Brock, Soham De, Samuel L Smith, and Karen Simonyan. High-performance large-scale image recognition without normalization. International conference on machine learning, pp- 1059-1071. PMLR, 2021.
George Cazenavette, Tongzhou Wang, Antonio Torralba, Alexei Efros, and Jun-Yan Zhu. Dataset distillation by matching training trajectories. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4750-4759, 2022.
Justin Cui, Ruochen Wang, Si, and Cho-Jui Hsich. Scaling up dataset distillation to imagenet- Ik with constant memory. In International Conference on Machine Learning, pp. 6565-6590. PMLR, 2023.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-2442255 225.2.55222 Ieee, 2009.
Tian Dong, Bo Zhao, and Lingjuan Lyu. Privacy for free: How does dataset condensation help privacy? ? In International Conference on Machine Learning, pp. 5378-5396. PMLR, 2022.
Jiawei Du, Yidi Jiang, Vincent YF Tan, Joey Tianyi Zhou, and Haizhou Li. Minimizing the accumu-
lated trajectory error to improve dataset distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3749-3758, 2023.
Reza Zanjirani Farahani and Masoud Hekmatfar. Facility location: concepts, models, algorithms and case studies. Springer Science & Business Media, 2009.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Jang- Hyun Kim, Jinuk Kim, Seong Joon Oh, Sangdoo Yun, Hwanjun Song, Joonhyun Jeong, Jung- Woo Ha, and Hyun Oh Song. Dataset condensation via efficient synthetic-data parameterization. In International Conference on Machine Learning, pp. 11102-11118. PMLR, 2022.
Yongqi Li and Wenjie Li. Data distillation for text classification, 2021. URL https://arxix: org/abs/ /aaas//1000 244....... 08448.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer vision-ECCV 20/4: 13th European conference, zurich, Switzerland, September 6-12, 2014, pro- ceedings, part 13, pp. 740-755. Springer, 2014.
Noel Loo, Ramin Hasani, Mathias Lechner, Alexander Amini, and Daniela Rus. Understanding reconstruction attacks with the neural tangent kernel and dataset distillation. arXiv preprint arXiv:2302.01442. 2023.
Aru Maekawa, Satoshi Kosugi. Kotaro Funakoshi, and Manabu Okumura. Dilm: into language model for text-level dataset distillation. Journal of Natural Language Processing, 32(1):252-288 2025.
Distilling dataset
Bryan A Plummer, Liwei Wang. Chris M Cervantes, Juan Caicedo, Julia Hockenmaier, and Svet- lana Lazebnik. Flickr30k entities: Collecting region-to-ppln -t---hasse correspondences for richer image- to-sentence models. In Proceedings of the IEEE international conference on computer vision, pp. 2641-2649, 2015.
11

---

Published as a conference paper at ICLR 2026
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pp- 9748-8763. PmLR, 2021.
Ilija Radosavovic, Raj Prateek Kosaraju, Ross Girshick, Kaiming He, and Piotr Dollár. Designing network design spaces. In Proceedings of the IEEE/CVF conference on computer vision and patter recognition, pp. 10428-10436, 2020.
Sylvestre-Alviss Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 2001-2010, 2017.
Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, and Geoffrey Gordon. An empirical study of example forgetting during deep neural network learning. arXiv preprint arXiv:18/2.051/33 2018
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- tion processing systems, 30, 2017.
Kai Wang. Bo Zhao, Xiangyu Peng. Zheng Zhu, Shuo Yang, Shuo Wang, Guan Huang, Hakan Bilen, Xinchao Wang, and Yang You. Cafe: Learning condense dataset by aligning features. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 12196-12205, 2022.
Tongzhou Wang, Lun-Yan Zhu, Antonio Torralba, and Alexei Efros. Dataset distillation. arXiv preprint arXiv: 109599 2018.
Max Welling. Herding dynamical weights to learn. In Proceedings of the 26th annual international conference on machine learning, pp. 1121-1128, 2009.
Xindi Wu, Byron Zhang, Zhiwei Deng, and Olga Russakovsky. Vision-language dataset distillation. arXiv preprint 13Xix::338.0754:: 2023.
Yue Xu, Zhilin Lin, Yusong Qiu, Cewu Lu, and Yong-Lu Li. Low-rank similarity mining for multi- modal dataset distillation. arXiv preprint arXiv:2406.0377 2024.
Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part I 13, pp. 318-833. Springer, 2014.
Bo Zhao and Hakan Bilen. Dataset condensation with differentiable siamese augmentation. In International Conference on Machine Learning, pp. 16674-168551 PMLR, 2021.
Bo Zhao and Hakan Bilen. Dataset condensation with distribution matching. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 6514-6523, 2023.
Zhao, Konda Reddy Mopuri, and Hakan Bilen. Dataset condensation with gradient matching. In International Conference Learning Representations, 2020.
Ganlong Zhao, Guanbin Li, Yipeng Qin, and Yizhou Yu. Improved distribution matching for dataset condensation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7856-7865, 2023.
12

---

Published as a conference paper at ICLR 2026
Table 6: Ablation study Text Encoder.
Text Encoder
Method
TR@1
TR@ 10
IR@1
IR
MTT-VL
9.9
39.1
4.7
24.6
BERT
AMD
14.4
52.6
10.4
43
MTT-VL
31.4
72
17.1
56,2
CLIP
AMD
41.7
81.2
29.7
76.4
A APPENDIX
The supplementary material is organized as follows: distillation and multimodal dataset distillation. Section A.2 provides a performance comparison using different text and visual encoders. Section A.3 analyzes the time overhead of the AMD. Section A.4 provides implementation details and hyperparameter settings.
Section A.1 reviews related works on dataset
A.1 RELATED WORK
Dataset Distillation. Dataset distillation Wang (2018); Zhao et al. (2020) has witnessed rapid advancements in recent years, with the primary goal of generating a compact set of highly informative synthetic data replace massive original datasets for model training. thereby dramat- ically reducing demands on data storage and computational resources. Current research predomi- nantly focuses on classification tasks, giving rise to three mainstream approaches: gradient match- ing Zhao al. (2020), distribution matching Zhao & Bilen (2023), and training trajectory match- ing Cazenavette et al. (2022).
Gradient matching methods (DC) optimize synthetic samples by minimizing the gradient discrep- ancies between synthetic and real data during model training. Subsequent improvements include DSA's Zhao & Bilen (2021) introduction of differentiable data augmentation to enhance generaliza- tion, and IDC's Kim et al. (2022) adoption of multi-formation synthesis techniques for better per- formance. Distribution matching methods (DM) aim to align the statistical distributions of synthetic data with real data in feature space, where early works employed Maximum Mean Discrepancy (MMD) as the distance metric, while advanced approaches like CAFE Wang et al. (2022) extend the alignment to intermediate network layers beyond final features. Training trajectory matching meth- ods (MTT) optimize synthetic data by minimizing parameter differences between models trained on synthetic versus real data across training stages. To address the prohibitive computational and memory costs of long-horizon trajectory matching, TESLA Cui et al. (2023) reduces memory con- sumption through loss reparameterization.
Multimodal Dataset Distillation. In multimodal dataset distillation, current research primarily targets image-text paired datasets. MTT-VL Wu et al. (2023) pioneered the first framework by ex- tending conventional single-trajectory matching to dual visual-textual trajectory alignment. Building upon this, LoRS Xu et al. (2024) introduces cross-modal similarity mining with low-rank matrices to reduce computational overhead, while incorporating TESLA's memory management techniques to enable efficient single-GPU training.
A.2 FURTHER ARCHITECTURE EXPERIMENTS
Following the evaluation protocol of MTT-VL Wu et al. (2023), we systematically replaced the text encoder and the image encoder to verify the generalization performance of AMD.
As shown in Table 6, the performance is significantly stronger when the text encoder utilizes CLIP. This is attributed to the fact that pre-trained CLIP inherently possesses a more powerful cross-modal alignment capability and richer semantic information. Furthermore, the performance gain of AMD compared to MTT-VI VL (c.g., in terms of the IR@ I metric) is more pronounced when using CLIP as the text encoder, which fully validates the effectiveness of our proposed method.
In addition, as demonstrated in Table 7, different visual encoders have a more pronounced impact on the IR performance synthetic data, where better visual encoder consistently brings cer- tain degree of improvement. For instance, the overall performance is superior when using the more advanced ViT the visual encoder compared using NENet, NeResNet and NFRegNet. This ob-
13

---

Published as a conference paper at ICLR 2026
Table 7: Ablation study Image Encoder.
Image Encoder
Method
TR@1
TR@ 10
IR
IR@ @ 10
MTT-VL
10.4
38.7
5.4
27.4
VIT
AMD
52.6
MTTVVL
9.9
39.1
4.7
24.6
NENet
AMD
14.1
51.9
9.7
40.8
MTTVL
18.7
NFResNet
AMD
10.9
43.4
7.9
33.6
MTT-VL
7.8
33.3
3.3
20.5
NetegNet
AMD
11.8
46.2
8.6
35.7
Table 8: Comparison of per-iteration training time between AMD and LoRS.
Datasct
Method
100 pairs
200 pairs
500 pairs
LoRS
6.44s
6.63s
6.56s
Flickr30k
AMD
6,52s
6.67s
6.61s
LoRS
6.13s
6.04s
6.09s
COCO
AMD
6.21s
6.15s
6.21s
Table 9; Hyperparameters for different experiments.
Dataset
Flickr30k
COCO
Pairs
200
100
500
Ir Ir_image
100
1000
1000
1000
1000
5000
Ir_text
100
1000
1000
1000
1000
5000
Ir_rr
0.001
0.01
0.01
0.01
0.01
0.01
Ir similarity
10
10
100
5
500
synth steps
8
8
8
servation suggests that AMD can effectively leverage more powerful visual feature representations enhance the matching accuracy for synthetic data.
A.3 TIME COST ANALYSIS
We base our implementation on the LoRS codebase, thus comparing our time cost with LoRS. As shown Table the per-iteration time costs of AMD and LoRS are highly consistent across datasets and pair counts. Crucially, the average per-iteration time reported for AMD explicitly includes the amortization of the one-time SPM initialization cost (which is only 5-10 minutes). For a complete distillation run (3000 iterations), the total time is approximately 5.4 hours (c.g., 6.52seconds x 3000 xs 5.43 hours for Flickr30k 100 pairs), which is consistent with the baseline. This confirms significant performance improvements achieved with virtually additional computational cost during distillation.
A.4 MORE IMPLEMENTATION DETAILS
We've Re've released the hyperparameter configuration for AMD, which is aligned with the baseline LoRS Xu et al. (2024).
A.5 MORE VISUALIZATION
We provide more visual comparisons of synthesized images and text before and after distillation, as shown in Figure 7-10.
14

---

Published as a conference paper at ICLR 2026
Figure 7: Visualization for Initialized images, interation=0.
aroop relasina ankoy rig vanns the player
puayer oppostte
rengore soending medy
troer
childowo
mask
lodd playting ball in park
voconn und a cann aliing as table nating two young hoys casual playing twiseer childhon
people
ticpoles alouz
botting play
beading finish lime onggle o'young kids durahhooding
cuamalenn
mun jimape in the str us tus stateheers
ron withi Nig hur lpwcct walking fhoouhh gress trail up currying bea, standinu
rann wallin crow lacabred
rop
saandin
clay
bunch oyeli riding Ulei takes slosn thcroad
two irtte buys play with soys
owo virecc looking information is esyinal booklet
rann with stauding grues
belping women wits clouue
with heard Naz städ starding
bey playing woth wheel ou stock.
voungg child playing with umpply bocket the graaa
owo roce and noo sounen are propaiing lirge meal in the sinkko
parpis clintinuu purtabie
enpepiiing ig teockey clayens
pock opposinn
warching
yernng powzicing rundaa
wath lage luile
weeres are working with bacbens of thod
two minunn drinking boer
todbler boldtin park place ytyor
up
dresal yallow lp shailler peol people mivyinng food at crouded restermat
grezs sporo car with the mamber driving en track lxbde aad groces outsále with thei gaesto
two teinal dags maniing graw.
boots langinn on
with
wastis
aaltv vp and:
Figure Visualization for Initialized texts, interation=0
B THE USE OF LARGE LANGUAGE MODELS (LLMS)
We use LLMs solely for checking grammar and polishing writing.
15

---

Published as a conference paper at ICLR 2026
Figure 9; Visualization for distilled images, interation=3000.
aroop rdasing iig vanny prohercarggt pioc, body prse
caidoon
vegroite labhoee
conteticiaa in simaag dacikmal nulfi wasing while maak lntle gart sisking meks beach with bur
worsee sitingg os coud drikkin box
woaring how sun-dirt playing with omango pirats without Hindfflld Lookey
seveal air
with disiag and stodoel
casnon mumaleen
heading the finidh lime
dogs meniing and playtig grecy
cuus jaeape in the sir on toe staaheers
old couple walking thorugh fild
youns white made
bloca--tur is coloring with cracoea
gitt
garing
buildra
chiltree
world diniig
people
about dee raking likes vailinng
mother and dingheer plating heare gane
owo werecc looking fnformatio is espinal bookiet
aniling ran en hane trees and woods
being helped
wadding one bridewsaids, who wauting a
wending
interection
baian schowiggf
her
musical inatrumnn
sid plaa oear water firetain
chilénes playing ball in yruce fidd en vanny day
ladies oue las box bacds her tipe smiling and other oue leeiing scceeting with ber other bacd heliad her slinhing en ioe
givnn buy
biidhensr yeurticing
muddic uped lisuasic wemas verziiu svidowalk
book weesus make cletse their booee
fanala pelioc- offiaae warring sargltino
mother holding nontorr inher berseen grnequreris while caring on couch
propp goopleat teads jamping
simulaneelly canemn
beahh
child blu shit mamsnng
Seanaie
oski axa drinking boch is restaner
fermals and care racing which the sal scume io winning likée and groces cotting calor thei wedding
menning proyhoend
ledge looging
of papli outiide the cite volentazion aluore
Figure 10: Visualization for distilled texts, interation=3000.
16


Reviews:
[{"review_id": "0DzRVQNlrh", "reviewer": "Reviewer_x1Yv", "summary": "The authors tackle the problem of Multimodal Dataset Distillation.\n\nThe authors first make two observations that are unique to the bimodal setting. First, text and image optimization trajectories are not synchronous and second, the optimization of the text and image encoders happen at different speeds. \n\nTo address these observations, the authors propose an asynchronous matching with dynamic sampling framework.", "questions": "None", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is clearly organized, I especially enjoyed Observations 1 and 2 along with Figure 2. They give a nice background for why the authors are proposing a different solution. \n- Thank you for including visuals of the distilled data in Figure 4. \n- The baselines are fairly comprehensive.\n- The authors give a substantial amount of background before diving into their method.\n- I appreciate the upper bound analysis, although it would be better presented as a graph.", "weaknesses": "I do not see major weaknesses, some minor nit-picks:\n\n- MMD is not defined in Eq 7\n- Table 1: I'm confused why you have to use ones less sample for your method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OKZdQrS0rk", "reviewer": "Reviewer_E75p", "summary": "This paper proposes Asynchronous Matching with Dynamic Sampling (AMD), a new framework for Multimodal Dataset Distillation (MDD). The key idea is to address the misalignment between image and text modalities during distillation by decoupling their optimization trajectories, rather than synchronizing them as in prior work. AMD dynamically samples starting points for visual and textual expert trajectories based on their distinct convergence behaviors and incorporates a Semantics-Aware Prototype Mining module that clusters feature spaces to select representative initialization points. Together, these techniques improve the representativeness and coverage of synthetic data. The paper reports consistent performance gains on Flickr30k and COCO datasets with minimal computational overhead, showing AMD’s efficiency and scalability for multimodal learning.", "questions": "How sensitive is the AMD framework to the choice of trajectory sampling ranges, and does the dynamic sampling strategy remain stable when applied to models with very different convergence behaviors?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper identifies and empirically validates the inherent asynchronous learning dynamics between visual and textual modalities, a key factor overlooked by prior MDD works.\n\n- The proposed Asynchronous Matching with Dynamic Sampling (AMD) framework introduces a principled way to decouple and dynamically align image-text trajectories, supported by a data-driven sampling strategy rather than fixed heuristics.\n\n- The addition of the Semantics-Aware Prototype Mining module effectively enhances synthetic data diversity and representativeness by leveraging clustering in the joint semantic space. The method achieves improvements over baselines across multiple datasets and scales, demonstrating both robustness and negligible additional computational cost.", "weaknesses": "- The evaluation relies on relatively small-scale datasets (Flickr30k, COCO), which may limit conclusions about scalability to larger, more complex multimodal datasets.\n\n- The qualitative analysis of synthetic data is minimal, providing limited insight into how AMD affects semantic fidelity or interpretability of the distilled samples. More discussion on this will further improve the paper quality.\n\n- Broader downstream evaluations (e.g., captioning) will better demonstrate the general utility of the distilled data.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LfQwrKShha", "reviewer": "Reviewer_RY53", "summary": "This paper studies Multimodal Dataset Distillation (MDD), where the goal is to compress large image–text datasets into a small set of synthetic samples while retaining downstream retrieval performance. The authors identify a previously overlooked issue in MDD: existing approaches assume synchronous training dynamics for vision and language encoders, despite empirical evidence showing fundamentally asynchronous learning behaviors across modalities. The paper decouples trajectory matching between visual and language encoders, and uses semantic-aware prototype mining to initialize synthetic samples based on clustering in joint embedding space. Experiments on Flickr30k and COCO show consistent improvements over prior MDD methods, with negligible additional computational cost.", "questions": "1. Would asynchronous matching still work if using large-scale encoders (e.g., CLIP ViT-L/14)? Does the effect diminish with stronger pretrained models?\n2. For prototype mining, did you try joint contrastive clustering or CLIP-score-based sampling instead of K-means?\n3. Can you explain more intuitively how dynamic sampling boundaries are formed and whether simpler heuristics work equally well?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. This paper identifies and validates the fundamental asynchronous nature of vision–language learning dynamics.\n2. The proposed modification is minimal, which yields consistent improvements and results strong gains on Flickr30k and COCO dataset.", "weaknesses": "1. No visualization of learned prototype clusters or intuitive qualitative examples to further support semantic coverage claims.\n2. Dynamic sampling strategy relies on MMD heuristic—although it works well, theoretical justification could be strengthened.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pk0Z6XSBt6", "reviewer": "Reviewer_EB68", "summary": "This paper focuses on Multimodal Dataset Distillation (MDD), a task that aims to distill a small amount of synthetic data capable of achieving performance comparable to the original dataset. To address the challenge of discrepancy between image and text distillation stages, this work proposes an asynchronous matching strategy. Furthermore, it introduces the Semantics-aware Prototype Mining (SPM) module for initializing the searching space and identifying representative prototypes. Experimental results on the Flickr30K and COCO datasets demonstrate the effectiveness of the proposed AMD method.", "questions": "1. I am confused by the statement in L198 regarding \"decoupling the distillation paths of images and text modalities.\" From Figure 1, the proposed asynchronous matching appears to couple the two modality trajectories at different timestamps. In contrast, previous synchronous methods seem to operate in a more decoupled manner, as image and text trajectories are matched strictly pair by pair. \n\n2. What specific value is chosen for $B$ (the number of clusters) in the SPM module? Furthermore, are there any ablation studies investigating the impact of different $B$ values?\n\n3. Could the authors provide a more detailed explanation of Figure 3? Specifically, what do the x and y axes represent, and what is the meaning of each point within the trajectories shown?\n\n4. The qualitative results presented in Figure 4 are not particularly convincing. Are there any stronger qualitative evidence to support the advancements of the proposed AMD method?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 1, "strengths": "1. The motivation is strong, underpinned by a solid experimental analysis (Sec 3.1) that effectively highlights the discrepancy problem in previous multimodal distillation methods. This makes the proposed asynchronous method (AMD) well-justified.\n\n2. The proposed algorithm is commendably simple yet effective. The substantial improvements over prior LoRS work (Tables 1 & 2) clearly demonstrate the effectiveness and superiority of the asynchronous matching strategy.", "weaknesses": "1. My primary concern is the sufficiency of the contribution. While the paper identifies its core contribution as asynchronous matching with dynamic sampling for MDD, the algorithm appears to be summarized as matching teacher and student networks within controlled ranges $R_L$ and $R_V$. Could the authors clarify if this understanding is accurate, or if I am oversimplifying the approach?\n\n2. Regarding the Semantic Prototype Mining (SPM) module, it appears to function more as an initialization trick than a core MDD-specific method. SPM uses K-means to identify $B$ representative prototypes based on concatenated features, which then serves as an initialization for the subsequent matching process. Critically, this module seems unrelated to the 'asynchronous' matching strategy. Moreover, it comes across as an effective heuristic rather than a genuinely learnable component.\n\n3. The authors claim (L86) that the achieved improvements incur negligible additional computational overhead. However, this assertion lacks supporting evidence in the main paper. While Appendix Table 8 compares running times, it notably omits the computational cost of SPM initialization, specifically the K-means clustering time.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)