Efficient Multimodal Dataset Distillation via Generative Models
Authors: Zhenghao Zhao
Haoxuan Wang
Junyi Wu
Yuzhang Shang
Gaowen Liu
Yan Yan
Venue: NeurIPS 2025 poster ()
Keywords: Dataset distillation
Primary area: deep_learning

Abstract: Dataset distillation aims to synthesize a small dataset from a large dataset, enabling the model trained on it to perform well on the original dataset. With the blooming of large language models and multimodal large language models, the importance of multimodal datasets, particularly image-text datasets, has grown significantly. However, existing multimodal dataset distillation methods are constrained by the Matching Training Trajectories algorithm, which significantly increases the computing resource requirement, and takes days to process the distillation. In this work, we introduce EDGE, a generative distillation method for efficient multimodal dataset distillation. Specifically, we identify two key challenges of distilling multimodal datasets with generative models: 1) The lack of correlation between generated images and captions. 2) The lack of diversity among generated samples.
To address the aforementioned issues, we propose a novel generative model training workflow with a bi-directional contrastive loss and a diversity loss. Furthermore, we propose a caption synthesis strategy to further improve text-to-image retrieval performance by introducing more text information.  Our method is evaluated on Flickr30K, COCO, and CC3M datasets, demonstrating superior performance and efficiency compared to existing approaches. Notably, our method achieves results 18$\times$ faster than the state-of-the-art method. Our code will be made public at https://github.com/ichbill/EDGE.

Efficient Multimodal Dataset Distillation via Generative Models
Zhenghao Zhao
Haoxuan Wang!
Junyi Wu! Yan Yan
Yuzhang Shang?
Gaowen Liu
University of Illinois Chicago University of Central Florida $Cisco Research
(zzhao72, hwang339, jwu834, yyan55) edu
yuzhang- shangducf edu, gaoliuEcisco.co com
Abstract
Dataset distillation aims to synthesize small dataset from large dataset, enabling the model trained on it to perform well on the original dataset. With the blooming of large language models and multimodal large language models, the importance multimodal datasets, particularly image-text datasets, has grown significantly. However, existing multimodal dataset distillation methods constrained by the Matching Training Trajectories algorithm, which significantly increases the computing resource requirement, and takes days to process the distillation. In this work, we introduce EDGE, generative distillation method for efficient multimodal dataset distillation. Specifically, multimodal datasets with generative models: 1) The lack of correlation between generated images and captions. 2) The lack of diversity among generated samples. To address the aforementioned issues, we propose a novel generative model training workflow with bi-directional contrastive loss and diversity loss. Furthermore, we propose caption synthesis strategy to further improve text-to- nex--to-image retrieval performance by introducing more text information. Our method is evaluated on Flickr30K, COCO, and CC3M datasets, demonstrating superior performance and efficiency compared to existing approaches. Notably, our method achieves results 18x faster than the state-of-the-art method. Our code will be made public at htttss///githr. com/ichbil1/EDGE
identify two key challenges of distilling
Introduction
Dataset distillation 53 seeks synthesize compact dataset from larger one, enabling model trained on the distilled dataset to achieve strong performance on the original dataset. Typical kerne- based dataset distillation [53 64 aims match the performance of the synthetic dataset with the have been 63 even achieve lossless
original dataset. The matching training trajectories (MTT) approaches (2] [9] 5
63 12]
proven effective for the distillation of small datasets. Some works [2
performance from the original dataset. Recently, aiming the large-scale dataset, decoupled dataset distillation methods [45] [57), and generative distillation methods [44
49 have been proposed,
which enhances scalability by improving the distillation efficiency.
With the emergence of large language models (LLM) and multimodal large language models 23, 24
16 38), multimodal datasets become essential, especially image-text datasets. The first work Vision Language Dataset Distillation (VLDD) (55) matches the expert training trajectories for both the image and text encoders and applies Low-Rank Adaptation [[4] to reduce the computing requirements. The following work, LoRS (56). utilizes similarity mining during the distillation to improve the performance. However, these MTT-based methods require substantial time
(MLLM) 23]
"Correponding Author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

and computational resources to distill datasets. As illustrated in Figure | it takes about four days for
MTT-VL to distill a dataset. The memory peak usage of MTT-VL reaches 200 GB for distilling 500 pairs and 320 GB for 100 pairs. For L.RSS [56], it takes about week (150 hours) to distill only 500 image-text pairs. Therefore, time consumption becomes one of the main drawbacks of current VI.DI> methods.
Generative dataset distillation methods leverage large-scale pre-trained diffusion mod- els [34] to distill image datasets for image clas- sification tasks. However, such single- modality generative distillation methods are unsuitable for multimodal scenarios because they are designed for a limited number of classes and cannot be generalized to image-caption datasets where the captions are not constrained by fixed number of categories. Additionally, directly applying multimodal generative models [34] to VLDD will encounter two main issues: 1) The lack of correlation between generated images and captions. We observe that directly applying gen- erative models to distillation yields a suboptimal performance. This is mainly because the diffu- sion model training concentrates on sample-wise noise prediction instead of image-text correspon- dence, which is the most important aspect of the image-text contrastive learning (ITC) task on vision language datasets. 2) The lack of diversity among the generated samples. In dataset distillation, the synthetic dataset only contains less than 5% of the original data, the generalization of the dataset is essential for the distilled dataset generation.
18x faster
3.5
10x faster
single-modality
Validation IR@1
3.0
2.5
Oupt-1000
2.0
LoAS-500
1.5-
Ours-500
SETVV-1000
MTENLL300
1.0-
0.5
Ours
MTT-VL
LoRS
20G
200G
8.5 Distillation Time (h) Figure 1: Performance vs. resource usage. Exist- ing methods require substantial computation time and memory to distill datasets, whereas our method achieves up to 18x faster processing and 16x lower memory usage while delivering competitive performance. The experiments are conducted on NVIDIA RTX A5000 GPUs.
0.0g
0
7.7
85 89
151 155
To address the aforementioned issues, in this work, we introduce Efficient Multimodal Dataset Distillation via GEnerative Models (EDGE), which utilizes generative priors for distilling vision- language datasets. Our approach incorporates a novel training workflow that integrates a bi-directional contrastive loss inspired by InfoNCE (31) and diversity loss inspired by Minimax [1]] improve image-text correlation and increase the diversity of distilled datasets. The EDGE framework begins by introducing noise to the image latent representation, predicting the noise with conditioning embedding, and obtaining the denoised image latent. Using the denoised image latent and the corresponding text embedding, we apply the proposed losses as follows: 1) Contrastive loss supervises the generative model to produce content that is highly relevant to the given conditioning, aligning image and text representations effectively. 2) Diversity loss increases the variability of the generated samples by pushing apart the sample features, encouraging the synthetic dataset to reflect the same distribution as the original dataset. Additionally, to further improve performance in text-to-image retrieval tasks, we incorporate a caption synthesis strategy. This involves generating additional captions for images in the synthetic dataset using MILMMs, providing sufficient text information for the evaluation model training and boosting retrieval performance.
We evaluate our methods on multiple vision language datasets, including Flickr30K (33, COCO (22), and Conceptual Captions 3 Million (CC3M) [42] Compared to existing VLDD methods, our method not only achieves more efficient distillation on Flickr30K and COCO, but also extends the capability to distill the large-scale CC3M dataset. As illustrated in Figure T our method significantly reduces the distillation time, achieving comparable performance to existing methods within just a few hours. compared to the days required by existing approaches. Furthermore, we successfully distill CC3M. a dataset containing approximately three million image-text pairs. Notably, most existing dataset condensation methods, including coreset selection, are infeasible for CC3M due to their extremely high computational resource requirements.
Our contributions are summarized as follows:
We identify the efficiency issue of current MTT-based vision language dataset distillation methods and propose generative vision language distillation method, EDGE, to efficiently distill the vision language datasets.
2

---

We identify two key challenges in applying generative methods to VLDD, and introduce novel training workflow incorporating contrastive loss and diversity loss. We further introduce caption synthesis to improve the text-to-image retrieval performance.
We conduct empirical comparisons with existing methods, demonstrating competitive per- formance while reducing computational costs by times compared to the SOTA approach.
Related works
2.1 Dataset Distillation
Dataset distillation (DD) [53] aims to compress a large dataset into small synthetic dataset while
preserving the performance of the models trained on it. Dataset distillation methods can be categorized
into several classes: performance matching [
64 27
29
28
30
3
parameter matching [6]
18 15
2
5
63
12
21 (50,, distribution matching 600,
51
39
62
36
58
52
6
26
(0) and
decoupled distillation methods [57 40 45
41
8
59 43
48
4.
Recently, generative distillation
methods [IIl 44, 49] synthesize images using generative models to efficiently distill large-scale
datasets. Minimax Diffusion [I] introduces additional minimax criteria in the generative training to enhance the performance of the generated images of the diffusion model. D'M [4] has explored the use of diffusion models to improve cross- arsss-archieecture generalization and reduce computational overhead by constraining the consistency of the real and synthetic image spaces.
While generative distillation methods have proven successful for image classification tasks, existing tongle-mddalty approaches are not suitable for multimodal scenarios. These methods are designed for fixed number of classes in both training and test datasets and cannot be generalized image-caption datasets, where captions are not constrained to predefined categories.
2.2 Vision Language Dataset Distillation
While the majority of the dataset distillation community focuses on the image classification task in dataset distillation, recent works [55 (66] began explore the new task of image-text contrastive learning (ITC) on vision language dataset. The first work of multimodal dataset distillation (55) matches both the image and text encoders and applies LoRA [14] reduce the computing require- ments. In particular, to overcome the challenge that the image-text contrastive learning dataset does not have discrete classes, this work first proposed jointly distilling image-text pairs in a contrastive manner. The following work, LoRS (56). utilizes similarity mining during distillation for performance improvement. Specifically, a similarity matrix for synthetic image-text pairs is calculated during data synthesis to explicitly describe the relevance of each image-text pair in the distilled dataset.
However, existing VLDD methods struggle with the distillation efficiency and the scale of the datasets, which is mainly due to the use of the Matching Training Trajectories (MTT) algorithm (2. MTT obtains synthetic data through bi-level optimization process, which involves inner loop for model updates and an outer loop for synthetic data updates. Not only does the required large number of unrolled iterations during optimization cause immense computational costs, but generating expert trajectories itself is also extremely time-consuming. In this work, we propose EDGE to efficiently distill datasets via the generative method and scale up the large-scaled CC3M 42 dataset.
3 Methods
3.1 Problem Formulation
Given a large target dataset T {I,, y4 ) where x; denotes the training data and yi denotes the corresponding annotation, general dataset distillation method aims to synthesize small dataset = (z,, y1 |1 where |S] << T, so that the model trained on S has the optimal performance on the target dataset T. Specifically for image-text contrastive learning (ITC), the target dataset consists of [T] image-text pairs. Note that in practice, in dataset, an image can have multiple captions, and caption also have multiple corresponding images. We consider the size of the datasets the number of image-text pairs since the number of image-text pairs determines the actual training time.
In Section 3.2
we begin by introducing the advantages of introducing diffusion model priors and highlighting the limitations of directly applying them to vision-language dataset distillation. Then present the workflow of EDGE to overcome the limitations. Section 3.3 presents the proposed
3

---

EDGE
Ingat I
Noiey
Diffusion Model
Preducted
Wy
Nio.1 4
Denossed
push
Random Noise
Latent
push
pull
pull
2:g Mr
Text Condition
Embedding
Denaised
Latent
Vi
Fmbhodinng
Sorward
backwward
encoder
conirastive space
Embedding
The workflow of our method. Given the input image æ, text condition, we first get image latent Zo and text embedding via the image encoder and text encoder Tg. Random noise <~ (O,, I) is added to the image latent, and the noise is then predicted using U-Net B5 conditioned on y. The denoised latent representation z is obtained as z = (z: - V VV/va' where é is the predicted noise. With the denoised latent and text embedding y, employ contrastive loss Le to align corresponding image latents and text embeddings and diversity loss LD to encourage the diversity and generalization between concatenated image-text features.
Figure 2:
contrastive loss and diversity loss. Finally, in Section 3.4 to further improve the text-to-image retrieval performance, we illustrate the caption synthesis process.
3.2 EDGE Diffusion for VLDD
Diffusion models learn the distribution of dataset by progressively adding Gaussian noise images and then reversing this process to reconstruct the original data. For instance, in the latent diffusion model (LDM) [4 the training process is described follows for given training image In forward noising process, a vision encoder ε encodes the image into latent representation Zo =(x). Gaussian noise (0, I) is incrementally added to the initial latent code 2g, resulting in: Z, Vā Zo + vI ãę e. where āz is a hyperparameter controlling the noise level at step t. Next, decoder reconstructs this latent code back to the image space. With conditioning vector encoding additional information, the diffusion model is trained to minimize the mean squared error between the predicted noise ? co(2), y) and the added noise NN( T):
L ||co (2.. y) <l12.
where is noise prediction network parameterized by 0.
Diffusion models are proven efficient dataset distillation methods for image classification tasks ||| 44). However, when applying diffusion models to dataset distillation for ITC tasks, there are two main limitations to the distillation process: 1) The correspondence between image and text features is not fully explored. As mentioned above, the training of the diffusion model sample-win wise, which concentrates noise prediction. Such training scheme lacks the supervision of the correspondence of image and text features. 2) The lack of diversity among the synthetic samples. The number of samples is much smaller than the original datasets, the diversity among the distilled samples is essential for dataset distillation.
To address the aforementioned issues, propose the EDGE finetuning workflow illustrated Figure 2 Instead of calculating the mean squared error between the predicted noise eœ (2,, c) and the ground truth we proposed two novel criteria for the vision language dataset distillation. Given the input image æ, text condition, the encoders E and ти transform them into a compact latent representation, image latent Zj and text embedding y, respectively. Then add random noise e~ (0, I) to the image latent and predict the noise with U-Net [35] with the conditioning y. With predicted noise é,
2,
3
Va
to get the de-noised latent z, where a and 3 are hyperparameters. The de-noised latent and text embedding y used to calculate the contrastive loss and the diversity loss, introduced in Section 3.3

---

The contrastive loss is used to encourage the generative model to synthesize images that are highly relevant to the given text conditions, while the diversity loss is used push away the image-text embeddings among different image-text pairs.
3.3 EDGE Diffusion Criteria
In this section, we begin by presenting the motivation behind the contrastive loss, followed by its design and impact. Next, we discuss rationale for the diversity loss and provide formulation. We then introduce the combined EDGE loss and conclude by explaining the process of generating the distilled dataset using our proposed method. Contrastive Loss. The goal of image-text contrastive learning (ITC) is to establish meaningful relationships between images and text. However, generative models, such as diffusion models, are typically trained to predict noise, without explicit supervision to capture image-text alignment. Instead of predicting the noise, we want to connect the latent space 2; and y; for corresponding image T; and text y:- Inspired by InfoNCE [1I, we propose the contrastive loss that encourages aligned embeddings. Specifically, with the given image embedding and the text embedding y. the similarity score between normalized features z; and y; can be defined as Sú "; where is the temperature parameter, which is set 0.5 in experiments. Then calculate the cross-entropy loss for both directions:
CITT
Σ log . N
1
exp(Sat
LT
log
exp(Si))
(3)
Σj=1 exp(S.))
exp(S
where Sig-
1. represents the similarity score between 2; and yj, and Syz = I. represents the similarity score between zy and ye. With À controlling the weight ratio, the overall contrastive loss is defined as:
= = CT
(4)
In equation the loss encourages alignment between each image and its corresponding caption while also ensuring that each text matches the correct image. By incorporating this bi-directional contrastive objective, the diffusion model learns to capture the semantic relevance between images and text. Consequently, during sampling, the model generates image-text pairs with high relevance, which is beneficial for evaluation model training.
Diversity Loss. Compared to the original dataset, the distilled dataset distilled by generative methods [ has limited diversity. This problem is even more evident in image-text contrastive (ITC) tasks compared to image classification, since the captions are not constrained to predefined categories. Diffusion model training is in sample-wiss manner and neglects this requirement for dataset distillation. To address this issue and ensure the distilled dataset accurately represents the original distribution, we focus increasing the diversity of generated samples and enhancing the generalization of models trained on the synthetic dataset. To achieve this, we introduce a diversity loss that leverages the de-noised embedding and the text embedding y:
2
N
N
[26 vi]
1 M5l [[
LD = N(N
x
(5)
1)
Uy
1=1
j=
where [2y] vi] represents the concatenation of the image and text embeddings 2 and y/. The term indicates the similarity matrix of normalized concatenated embeddings. The diversity loss is designed to maximize the distance between these concatenated embeddings, effec- tively pushing them away in the embedding space. This encourages diversity in the representations of image-text pairs, promoting improved generalization and robustness.
[se:gu]
[[
Finally, by incorporating the contrastive loss Le and the diversity loss Lp together, we formulate the EDGE loss as:
LEDGE Lc + XLD.
(6)
The EDGE loss is used to train the diffusion model on the target dataset, to get the corresponding generative model. To synthesize the distilled dataset, we directly use diffusion model sampling for the required number of images. We first randomly select captions from the original dataset, and then use each caption as condition to sample the image.
5

---

3.4 Caption Synthesis
We observed that when training a model on the synthetic dataset, it is often more challenging for the model to retrieve the correct image with given text than to retrieve the correct text for given image. As discussed in Section [I applying the generative- based dataset distillation method to ITC tasks will counter the issue that the captions are not sufficient for the model training. To address this issue, we propose caption synthesis to generate more captions for each image. We developed a scalable approach to create many captions we want. Our method involves crafting specific prompt engineering templates that guide the Multimodal Large Language Models (MLLM) 25 produce the captions for given images.
to
We start by gathering the images from the syn- thetic dataset. For each image, we consider straightforward prompt template to generate cap-
Context
LLaVA
tions effectively. As illustrated in Figure 3 can use MLLM such as LLaVA [25] or GPT 32]
User Describe Assistant dog is laying on douch with red pillow en It.
sensence.
model to generate any number of captions for
given image. The prompts we pass to different MLLMs are little bit different. For LLa directly pass the prompt "Describe the image in sentence" and can get the required caption. GPT models tend to generate long se- quences of text, and it often contains the phase that is used for interactions, such as "the image contains". Thus we change the prompt to "De- scribe the image briefly in one sentence. Do not start with 'the image."
GPT model
User -> Deacribe the image briefly in short start with 'the image".
Assistant relaxed yellow labrador lies on couch, resting its head on bright orange toy.
Augmented Caption
dog ia laying on a couch with a red pillow
Figure 3: Illustration of the caption synthesis.
An image and prompt passed to a Multimodal Large Language Model (MLLM) to generate cap- tions. Then the dataset contains [8] /s/ images with [S] captions, where v) is the caption per image (CPI) and each image is captioned by 1j captions. Caption synthesis aims to provide sufficient text information for the model training.
During the model training stage, even though method produces fewer images than cap- tions, we consider the same image with different caption pairs distinet image-text pairs. This is due to the similar computational resources required from the model training perspective.
4 Experiments
4.1 Datasets and Metrics
Following previous methods 55| 56]. we evaluate our method on Flickr30k 33 and COCO 22 datasets for fair comparison with existing methods. Flickr30K and COCO are image-text datasets with 31K and 123K images, where each image is paired with five captions. We also evaluated methods larger image-caption dataset, Conceptual Captions Million (CC3M) 4g) to validate the effectiveness of our method on large-scale datasets. CC3M contains about 3.3 million image- text pairs. The evaluation involves the metrics of top-K retrieval from both the image and the text perspectives. We denote the text-to-image retrieval as IR @ K and image-to-text retrieval as TR @ K.
4.2 Implementation Details
Following previous works [55 56 on Vision Language Dataset Distillation, for the evaluation model, we adopt NFNet I as image encoder and BERT-base [17] as text encoder. The pre-trained weights used, and the text encoder is frozen during evaluation. Our entire distillation process be done on a single NVIDIA RTX A5000 GPU. For evaluation on the CC3M dataset, since the dataset is constructed with URLs and by the time we conduct experiments, some URLs are longer available. Eventually, we collected 2.3M of 3.3M images for the CC3M training data. In addition, since the validation set is large, we first subsample 1000 image-text pair validation subset and then evaluate the validation subset. In the distillation stage, following previous works [56], the images are resized to 224x 224 resolution, and text embeddings of 768 dimension.
4.3 Main Results
We compare our method with coreset selection methods (54 and pre-trained generative model, Stable-diffusion v1.5
10
46), existing VL.D methods [55 56
34
on Flickr30K and COCO datasets.

---

Table 1: Results of 500 pairs Flickr30K and COCO dataset. For Flickr30K, the dataset
condensation ratio is 1.7%. For COCO, the condensation ratio is 4.4%.. Bold values indicate the best results, while underlined values denote the second-bess results.
Coreset Selection
Dataset
Distillation LoRS
Dutaset
Ratio
Metric
Rand
HERD
K-Cent
Forget
MTT-VL
TESLA
SD
EDGE
IR@1
2.4
3.0
3.5
1.8
6.6
1.1
10.0
3.1
6.7
IR55
10.5
10.0
10.4
9.0
20.2
7.3
28.9
11.5
21.0
IR@
17.4
17.0
17.3
15.9
30.0
12.6
41.6
18.5
30.5
Flickr30K
1.7%
TR@1
5.2
4,9
3.6
13.3
5.1
15.5
4.6
13.3
TR@S
18.3
16.4
16.4
12.3
32.8
15.3
39.8
15.1
35.6
TR@10
25.7
24.3
23.3
19.3
46.8
23.8
53.7
22.2
47.5
IR@1
1.1
1.7
1.1
0.8
1.4
0.8
1.2
IR55
5.0
5.8
5.1
3.6
5.1
IR @ 10
8.7
9.9
10.5
8.2
8.9
6.9
13.7
9.0
11.2
COCO
4.4%
TR@1
1.9
1.9
2.5
2.1
2.3
1.7
2.8
2.2
2,9
TR@5
7.5
7.8
8.7
8.2
8.4
5.9
9.9
8.7
9.5
TR@10
12.5
13.7
14.3
13.0
13.8
10.2
16.2
14.9
15.7
Table 2: Results of 1000 pairs Flickr30K and COCO dataset. Bold values indicate the best results, while underlined values denote the second-ber best results.
Table 3: Results on CC3M. We compare our method with random selection and a pre-trained Stable Diffusion v1.5 model. The synthetic dataset contains 1000 image-text-pairs, which is only 0.3%o of the original dataset. Other ex- isting methods are not applicable due to long computation time high memory requirements.
IR61
3.9
11.2
3.3
9.9
IR65
13.0
IR@ 10
20.7
42.9
19.1
40.5
Plick30K 3.4%
Methods
IRel
IR85
IR@10
TR@1
TR85
TR@10
TRØ5
17.1
37.5
13.9
38.3
TR 10
27.7
50.5
24.8
51.7
0.1
0.9
0.3
0.9
IR@1
1.6
2.5
1.8
2.8
EDGR
0.2
0.5
1.0
0.1
0.7
1.1
IR
Table 4: CLIP score.
Table 5: FID score.
COCO
8,8%
I 10
TR @0
11.5
2.4
13.4
11.9
2.8
16.2
Metheds
COCD
CLSM
Methods
Flick-30KK
0OC0
TR 10
TR 08
16.7
9.9
11.8
19.1
10.2
16.8
21.0
13.0
WETEVI
ToRS
0:70m6
8.1277
0.304
0:003
0.3196
MTTVLL
Ours
2100
276.3
Due to extremely long computing time, coreset selection methods 10, 46) are hard to be applied on large number of pairs. As for MTT-based DD methods [55] 5), distilling a large number of pairs will lead to extremely high GPU memory usage.
Flickr30K results. We evaluate method the Flickr30K dataset and compare it with existing approaches. The distilled dataset contains 500 and 1000 image-text pairs, representing 1.7% and 3.4% of the original dataset, respectively. As shown in Table and Table 2 method achieves performance comparable to or even outperforms that of time-intensive SOTA methods across several metrics. Notably, compared to the baseline- pre-trained Stable Diffusion v1.5 model-oul approach demonstrates significant improvements, highlighting the effectiveness of the proposed techniques.
COCO results. We also evaluate methods in Table and Table 2 The distilled dataset contains 500 and 1000 image-text pairs, representing 4.4% and 8.8%o of the original dataset, respectively. With 500 pairs, it achieves competitive performance relative to SOTA methods, which require significantly longer distillation time. When distilling 1000 image-text pairs, method shows obvious advantage from both performance and efficiency perspectives. Similar to the Flickr30K results, our method substantially improves over the baseline pre-trained Stable Diffusion model, further validating its effectiveness.
method the COCO dataset in comparison with existing
CC3M results. For larger datasets like Conceptual Captions 3M (CC3M) [42]. existing dataset distillation methods not applicable due the high resource demands of trajectory matching algorithms. Even coreset selection methods are difficult to apply, as their time consumption and memory usage scale rapidly. In contrast, our method is capable of distilling the CC3M dataset efficiently using only a single GPU. The distilled dataset contains 500 image-text pairs, representing 0.3% of the original dataset. We compare pre-trained Stable Diffusion v1.5 model in Table Our method outperforms the baseline methods, demonstrating the effectiveness of method CC3M dataset.
approach with the random selection method and the
More aspects of evaluation. Furthermore, we provide the CLIP scores on Flickr30K, COCO, and CC3M, comparing with existing dataset distillation methods in Table Table 5 assesses image

---

Table 6: Comparison of required computing resources. We provide the GPU hours required for different method to distill different datasets. "OOM" in the table indicates the method is not applicable due to extremely high memory usage.
Dataset Pairs
Flickr30K
COCO
CC3M
500
1000
1500
500
1000
1500
500
1000
1500
MTT-VL [55
77.8
109.1
141.9
162.3
194.1
0OM
OOM
OOM
OOM
LoRS [56
54.2
57.6
59.0
151.6
155.6
172.5
OOM
OOM
OOM
EDGE (ours)
13.6
14.3
14.9
7.7
8.5
9.2
31.7
32.5
33.2
Table 7: Cross architecture evaluation. For baseline method, NFNet+BERT to distill, and evaluate with various architectures. Compared to matching training trajectory based approaches, the dataset generated by our method has better generalization.
COCO
# Pairs
Ratio
Method
Evaluation model
IR @
IR@5
IR
10
TR @!
TR@5
TR 10
NeNet+Bert
2.5
9.4
15.4
3.6
11.8
19.1
1000
8.8%
LoRS
ResNet+Bert
0.1
0.6
1.1
0.4
1.6
2.8
RegNettBBet
0.1
0.5
0.9
0.1
0.5
0.9
NfNet+Bert
2.8
9.8
16.2
3.9
13.0
21.0
1000
8,8%c
EDGE
ResNet+Bert
2.5
9.5
16.0
3.7
12.7
20.1
RegNettBBet
2.3
8.4
14.1
2.6
10.7
17.7
quality using FID. Results show our method improves both text-image alignment and image quality, where the former is more crucial for retrieval tasks.
Computational efficiency. We provide the computing efficiency comparison of our method and the SOTA methods [65] As depicted in Table 6 for Flickr30K, our method is approximately 9 times faster than MTT-VL, and times faster than LoRS. For the COCO dataset, our method is 22 times faster than MTT-VL. and 18 times faster than LoRS. Not even mentioning that both existing methods [55 56 are not applicable for large dataset CC3M 42. For CC3M, our method is the only applicable dataset distillation method, and our method is able to distill the dataset in an efficient manner. The GPU hour usage is even less than the baseline methods on the Flickr3KK dataset.
Cross-architectree experiments. In this work, we also evaluate the cross-architecture generalization of our method. As shown T we evaluate performance by changing the image encoder, and our method significantly outperforms the baseline. The results highlight that the cross- crossaarchitecture performance of the baseline method [56] is highly dependent on the model used during distillation. For example, when synthetic datasets are distilled with NFNet and BERT. evaluation using the exact same model architectures yields strong performance. However, substituting NFNet with ResNet [3] or RegNet [37 causes a significant performance drop in both image and text retrieval tasks. In contrast, our method, which is not related specific model architecture during training, maintains consistent performance across various model architectures.
4.4 Ablation Study
Ablation on different components. Table g states the effectiveness of each component of our method. The experiments are performed on the COCO [22] dataset, distilling into 500 image-text pairs. The baseline "SD v1.5" in the table indicates the baseline model, a pretrained stable-diffusion model v1.5 (34] used in method. "+ Lc" and "+ Lp" indicates the generative model fine-tuned with our proposed workflow in Figure 2 and corresponding losses.
The pre-trained stable diffusion model shows low performance on the evaluation dataset, indicating that the vanilla generative models, which are not designed for dataset distillation tasks, do not guarantee the correspondence for image-text pairs. By adding our proposed fine-tuning workflow and the corresponding losses, both IR and TR have been improved. By further adding the post- training caption synthesis, the performance of image-text retrieval is further improved, indicating the effectiveness of proposed methods.
Effect of different diffusion models. The diffusion model used to distill the dataset is an important aspect our method. We evaluate two widely used diffusion models: Stable Diffusion v1.5 (denoted as "SD v1.5" in the table) and Stable Diffusion 3 (denoted as "SD 3") in this work. We compare two models from both perspectives of performance and efficiency. The performance of the methods depicted in Table 9 where the performance of two diffusion models is similar. However, the sampling

---

Table 8: Effect of different components. The ex- periments are conducted on COCO dataset. The synthetic dataset contains 500 image-text pairs. The evaluation model is NFNct+BERT. "CS" de- notes the Caption Synthesis strategy.
running through sone dead
Cap 2: A white dog is running through the ground.
field with
Cap 1: A notocross rider in the process of wiping out
Cap 2: A nan is riding dirt bike on diet track, with crowd watching hin.
Methods
IR81
IR
IR@10
TR@1
TRØ5
TR@10
1.2
9.0
1.3
5.1
9.2
2.3
83
13.7
1: A nan is walking bahind tree strest.
Lo En
1.4
5.6
10.1
2.3
8.9
14.4
-mar
1.8
6.5
112
2,9
9.5
15.7
cap 2: à nan vearing a hooded sweatshirt is valking down tree-lined street.
Table 9: Comparison of different generative models. SD represents Stable Diffusion. Stable Diffusion v1.5 takes about 0.4 hour distill 500 images, while it takes approximately 1.6 hours to
Cap sports mascot during pertornance.
Cap 2: A maacot in yellow jacket and standing field.
Figure 4: Examples of images and captions of the distilled dataset. For each example image,
distill 500 images for Stable Diffusion 3.
Methods
IR@1
IR65
IR@10
TR@1
TR@5
TR@10 10
two corresponding captions are provided. The
SD
1.2
9.0
2.2
8.7
14.9
green captions are generated by caption synthesis.
SD3
1.2
5.2
9.4
2.4
8.6
14.7
Table 11: Comparison of different MLLMs.
Table 10: Ablation study caption per image Post-training strategies perform better than pre- (CPI). On COCO dataset, when CPI=2, the model processing strategics. With captions by LLaVA, achieves the best performance.
the model achieves the best performance.
Methods
IR@1
I885
IR 10
TR@1
TRØ5
TR 10
Methods
IRe1
IR
IR 10
TRØ1
TR85
TR
CP1=1
1.9
11.8
LLaVA
2.8
9.8
16.2
3.9
13,0
21.0
CPI=2
28
9.8
16.2
3.9
13,0
21.0
GPT-4o-mini
2.5
8.8
14.9
3.7
12.5
20,0
5
1.2
4.7
8.2
2.0
7.0
12.3
Llama
1.2
13.6
time of stable diffusion 3 is four times longer than that of Stable Diffusion v1.5. Consequently, we selected Stable Diffusion v1.5 as the primary model for our results.
Effect of caption per image. For caption synthesis, the number of captions per image (CPI) is essential. To investigate its impact on the model's performance, vary the CPI during caption synthesis while keeping the total number of image-text pairs constant. The evaluation is done on COCO dataset, distilling the dataset into 500 image-text pairs. For CPI=1, CPI=2, CPI=5, we sample 500, 250, 100 images.
There are two main observations from Table TO 1) Compared to the "vanilla" synthetic dataset, where each image is paired with only one caption, generating additional captions assists with the evaluation model training. This observation is consistent with the motivation of our caption synthesis, harm the performance of the evaluation model. This occurs because, with a fixed number of image-text pairs, increasing the CPI leads to the reduction of the total number of unique images. Consequently, the reduced number of images becomes insufficient for effective model training.
Effect of different MILMs. The MLLM used for caption synthesis is also essential in our method. In this work, we compare three MLLM/LLM models with two different strategies. The first strategy, pre- processing. involves rephrasing captions before image generation. The second strategy, post-training caption synthesis, involves rephrasing captions after image generation, as illustrated Section 3.4
The models evaluated in our study include LLava [25] J. GPT-4o-mini (32), and Llama [47]. For
LLava, we adopt the post-training strategy, while for Llama, we use the pre-processing strategy. For the GPT model, both pre- pre-processing and post- toatttraiiing strategies applicable.
As illustrated in Table IT the pre-processing strategy, regardless of the model used for rephrasing captions, proves to be less effective compared to post-training strategies. For the post-training methods, captions generated by the LLaVA model are more beneficial for training the evaluation model than those generated by the GPT model. Therefore, we adopt the post-training caption synthesis strategy with the LLaVA model as our primary approach.
4.5 Dataset Visualization
We visualized the images with corresponding captions generated by our method in Figure Compared to previous VLDD methods 55 56), the images generated by our method exhibit realistic, high- quality appearance. The green captions are generated by our caption synthesis. The captions exhibited

---

in the figure are generated by LLaVA [25]. We can observe that compared to the image caption from the original dataset, the synthesized caption tends to capture more features from the image.
Conclusion
In this work, we identify key limitations in current vision language dataset distillation methods, including poor efficiency and limited scalability. To address these issues, we propose EDGE, generative vision language dataset distillation approach designed to efficiently distill large-scale scale multimodal datasets. Specifically, we introduce a novel training workflow for generative models tailored to the image-text dataset distillation task. Additionally, we implement a post-training caption synthesis step to further enhance performance. Our method not only outperforms existing approaches on small datasets but also enables effective distillation of large datasets.
Acknowledgments: This research is supported by NSF IIS-2525840, CNS- 2NS-2432534, NIH IRF IMH133664-00 and Cisco Research unrestricted gift. This article solely reflects opinions and conclusions of authors and not funding agencies.
ECCS-221455
References
[1] Andy Brock, Soham De, Samuel L Smith, and Karen Simonyan. High-performance large-scale image recognition without normalization. In ICML. PMLR, 2021.
[2] George Cazenavette, Tongzhou Wang, Antonio Torralba, Alexei Efros, and Jun-Yan Zhu. Dataset distillation by matching training trajectories. In CVPR, 2022.
[3] Yilan Chen, Wei Huang, and Lily Weng. Provable and efficient dataset distillation for kernel ridge regression. In NeurlPS, 2024.
[4] Jiacheng Cui, Xinyue Bi, Yaxin Luo, Xiaohan Zhao, Jiacheng Liu, and Zhiqiang Shen. Fadrm: Fast and accurate data residual matching for dataset distillation. arXiv preprint 2.Xiv:2506.2422 2025.
[5] Justin Cui, Ruochen Wang, Si Si, and Cho-Jui Hsieh. Scaling up dataset distillation to imagenet-
1k with constant memory. In ICML, 2023.
[6] Xiao Cui, Yulei Qin, Wengang Zhou, Hongsheng Li, and Houqiang Li. Optical: Leveraging optimal transport for contribution allocation in dataset distillation. In CVPR, 2025.
[7] Wenxiao Deng, Wenbin Li, Tianyu Ding, Lei Wang. Hongguang Zhang, Kuihua Huang, Jing Huo, and Yang Gao. Exploiting inter-sample and inter-feature relations in dataset distillation. In CVPR, 2024.
[8] Jiawei Du Juncheng Hu, Wenxin Huang, Joey Tianyi Zhou, et al. Diversity-driven synthesis: Enhancing dataset distillation through directed weight adjustment. In NeurIPS, 2024.
[9] Jiawei Du, Yidi Jiang, Vincent YF Tan, Joey Tianyi Zhou, and Haizhou Li. Minimizing the accumulated trajectory error to improve dataset distillation. In CVPR, 2023.
[10] Reza Zanjirani Farahani and Masoud Hekmatfar. Facility location: concepts, models, algorithms and case studies. Springer Science & Business Media, 2009.
[11] Jianynn Gu, Saeed Vahidian, Vyacheslav Kungurtsev, Haonan Wang, Wei Jiang, Yang You, and Yiran Chen. Efficient dataset distillation via minimax diffusion. CVPR, 2024.
[12] Ziyao Guo, Kai Wang, George Cazenavette, Hui Li, Kaipeng Zhang, and Yang You. To- wards lossless dataset distillation via difficulty-alignee trajectory matching. arXiv preprint arXiv:2310.05773, 2023.
[13] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
[14] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models, 2021.
10

---

[15] Zixuan Jiang, Jiaqi Gu, Mingjie Liu, and David Z Pan. Delving into effective gradient matching for dataset condensation. In COINS, 2023.
[16] Weitai Kang, Haifeng Huang, Yuzhang Shang, Mubarak Shah, and Yan Yan. Robin3d: Improv- ing 3d large language model via robust instruction tuning, arXiv preprint arXiv:2410.00255, 2024.
[17] Jacob Devlin Ming- Weig Chang Kenton and Lee Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT, Minneapolis, Minnesota, 2019.
[18] Saehyung Lee, Sanghyuk Chun, Sangwon Jung. Sangdoo Yun, and Sungroh Yoon. Dataset condensation with contrastive signals. In ICML, 2022.
[19] Guang Li, Ren Togo, Takahiro Ogawa, and Miki Haseyama. Dataset distillation using parameter pruning. IEICE Transactions, 2023.
[20] Wenyuan Li, Guang Li, Keisuke Maeda, Takahiro Ogawa, and Miki Haseyama. Hyperbolic dataset distillation. arXiv preprint arXiv:2505. 246233, 2025.
[21] Zekai Li, Ziyao Guo, Wangbo Zhao, Tianle Zhang. Zhi-Qi Cheng, Samir Khaki, Kaipeng Zhang, Ahmad Sajedi, Konstantinos N Plataniotis, Kai Wang, et al. Prioritize alignment in dataset distillation. arXiv preprint arXiv:2408.03360 2024.
[22] Tsung-Yo Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV. Springer, 2014.
[23] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning, 2023.
[24] Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024.
[25] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, 2023,
[26] Haoyang Liu, Yijiang Li, Tiancheng Xing, Vibhu Dalal, Luwei Li, Jingrui He, and Haohan Wang. Dataset distillation via the wasserstein metric. arXiv preprint arXhv:2311.18531 2023.
[27] Noel Loo, Ramin Hasani, Alexander Amini, and Daniela Rus. Efficient dataset distillation using random feature approximation. In NeurlPS, 2022.
[28] Noel Loo, Ramin Hasani, Mathias Lechner, and Daniela Rus. Dataset distillation with convexi- fied implicit gradients. In ICML. PMLR, 2023.
[29] Timothy Nguyen, Zhourong Chen, and Jaehoon Lee. Dataset meta-learning from kernel ridge-regression. arXiv preprint arXiv:2011.0000 2020.
[30] Timothy Nguyen, Roman Novak, Lechao Xiao, and Jaehoon Lee. Dataset distillation with infinitely wide convolutional networks. In NeurIPS, 2021.
[31] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv: 2018
[32] OpenAl. Chatgpt.
openai. com. 2024. [Online; accessed 2024-07-18).
[33] Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models. In ICCV, 2015.
[34] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. In CVPR, 2022.
[35] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In MICCAI, 2015.

---

[36] Ahmad Sajedi, Samir Khaki, Ehsan Amjadian, Lucy Z Liu, Yuri A Lawryshyn, and Konstanti- nos N Plataniotis. Datadam: Efficient dataset distillation with attention matching. In ICCV, 2023.
[37] Nick Schneider, Florian Piewak, Christoph Stiller, and Uwe Franke. Regnet: Multimodal sensor registration using deep neural networks. /V. IEEE, 2017.
[38] Yuzhang Shang. Mu Cai, Bingxin Xu, Yong Jae Lee, and Yan Yan. Llava-prumerge: Adaptive token reduction for efficient large multimodal models. arXiv preprint arXiv:2403.15388 2024.
[39] Yuzhang Shang. Zhihang Yuan, and Yan Yan. Mim4dd: Mutual information maximization for dataset distillation. In NeurlPS, 2024.
[40] Shitong Shao, Zeyuan Yin, Muxin Zhou, Xindong Zhang, and Zhiqiang Shen. Generalized large-scale data condensation via various backbone and statistical matching. In CVPR, 2024.
[41] Shitong Shao, Zikai Zhou, Huanran Chen, and Zhiqiang Shen. Elucidating the design space of dataset condensation. In NeurlPS, 2024.
[42] Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In ACL., 2018.
[43] Zhiqiang Shen, Ammar Sherif, Zeyuan Yin, and Shitong Shao. Delt: A simple diversity-driven earlylate training for dataset distillation. In CVPR, 2025.
[44] Duo Su, Junjie Hou, Weizhi Gao, Yingjie Tian, and Bowen Tang. D 4: Dataset distillation via disentangled diffusion model. In CVPR, 2024.
[45] Peng Sun, Bei Shi, Daiwei Yu, and Tao Lin. On the diversity and realism of distilled dataset: An efficient dataset distillation paradigm. In CVPR, 2024.
[46] Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, and Geoffrey J Gordon. An empirical study of example forgetting during deep neural network learning. arXiv preprint / 2018
[47] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Manie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13977, 2023.
[48] Minh-Tuan Tran, Trung Le, Xuan-May Le, Thanh-Toan Do and Dinh Phung, Enhancing dataset distillation via non non-critical region refinement. In CVPR, 2025.
[49] Haoxuan Wang, Zhenghao Zhao, Junyi Wu, Yuzhang Shang, Gaowen Liu, and Yan Yan. Cao 2: Rectifying inconsistencies in diffusion-based dataset distillation. arXiv preprint arXiv:2506. 22637, 2025.
[50] Kai Wang, Zekai Li, Zhi-Qi Cheng, Samir Khaki, Ahmad Sajedi, Ramakrishna Vedantam, Konstantinos N Plataniotis, Alexander Hauptmann, and Yang You. Emphasizing discriminative features for dataset distillation in complex scenarios. In CVPR, 2025.
[51] Kai Wang, Bo Zhao, Xiangyu Peng, Zheng Zhu, Shuo Yang, Shuo Wang, Guan Huang, Hakan Bilen, Xinchao Wang, and Yang You. Cafe: Learning to condense dataset by aligning features. In CVPR, 2022.
[52] Shaobo Wang, Yicun Yang, Zhiyuan Liu, Chenghao Sun, Xuming Hu, Conghui He, and Linfeng Zhang. Dataset distillation with neural characteristic function: minmax perspective. In CVPR. 2025.
[53] Tongzhou Wang, Jun-Yan Zhu, Antonio Torralba, and Alexei A Efros. Dataset distillation. arXiv preprint arXiv: 18/ 10959, 2018
[54] Max Welling. Herding dynamical weights to learn. ICML, 2009.
[55] Xindi Wu, Byron Zhang, Zhiwei Deng, and Olga Russakovsky. Vision-language dataset distillation. arXiv preprint 20855 2023.
12

---

[56] Yue Xu, Zhilin Lin, Yusong Qiu, Cewu Lu, and Yong-Lu Li. Low-rank similarity mining for multimodal dataset distillation. arXiv preprint NXXi::2406003792 2024.
[57] Zeyuan Yin, Eric Xing, and Zhiqiang Shen. Squeeze, recover and relabel: Dataset condensation at imagenet scale from new perspective. In NeurIPS, 2024.
[58] Hansong Zhang, Shikun Li, Pengju Wang, Dan Zeng, and Shiming Ge. M3d: Dataset condensa- tion by minimizing maximum mean discrepancy. In AAAI, 2024.
(59] Xin Zhang. Jiawei Du, Ping Liu, and Joey Tianyi Zhou. Breaking class barriers: Efficient dataset distillation via inter-class feature compensator. arXiv preprint arXiv:2408. 06927, 2024.
[60] Bo Zhao and Hakan Bilen. Dataset condensation with distribution matching. WACV, 2023.
[61] Bo Zhao, Konda Reddy Mopuri, and Hakan Bilen. Dataset condensation with gradient matching. arXiv preprint arXiv:2006..0592, 2020.
[62] Ganlong Zhao, Guanbin Li, Yipeng Qin, and Yizhou Yu. Improved distribution matching for dataset condensation. CVPR, 2023.
[63] Zhenghao Zhao, Haoxuan Wang, Yuzhang Shang, Kai Wang, and Yan Yan. Distilling long-tailed datasets. arXiv preprint arXiv:2408. /4506, 2024.
[64] Yongchao Zhou, Ehsan Nezhadarya, and Jimmy Ba. Dataset distillation using neural feature regression. In NeurIPS, 2022.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The claims in the abstract and introduction accurately reflect the contributions the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: See Section 4
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
14

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: The are fully discussed.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
15

---

Answer: [Yes]
Justification:
The code will be released publicly.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https guwdes/ddeeeeee Cooeeeoodeesssseeeeee //nips.com / ssiniiiiccypyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
They are discussed in details.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Error bars not reported because would be too computationally expensive for baselines.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
16

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
is properly discussed.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
The research conducted in this paper adheres to the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: See Appendix C
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
17

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All external assets such as datasets and code libraries used in the research are properly credited, and their licenses are respected and clearly mentioned.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the Datasets.paaersstt paperswi thcode. cww/ datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
18

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
19

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [Yes]
Justification: The usage of LLMs is detailed in Section
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy fattps://neur//: httpp////.e//oooeeeeeee///////// for what should or should not be described.
20

---

A Training details
We provide the detailed hyperparameter settings used during diffusion model training. Following the approach in [1] we fine-tune only a small subset of the diffusion model's parameters. The weight À in Equation 6 is set to I. For Flickr30K dataset, the learning rate is set to 1 x 10 with batch size of 8 and total of 16 training epochs. For COCO dataset, the learning rate is set to x 10-4, with a batch size of and a total of training epochs.
B Qualitative results
Cap woman is ho holdding plate with piece of cake it.
Cap brown suitcase is sitting next to bed with pillow.
Cap 2: woman is about to eat huge slice of cake.
Cap 2: A luggage bag is next to dresser in bedroom.
Cap 1: A man is walking in the snow, with a car behind him.
Cap 1: A brown bear and two cubs are standing in grassy area.
Cap 2: Some people standing out in the snow.
Cap Some baby bears having fun on a sunny day.
Cap living roon with brown couch, a coffee table, and a flat screen TV.
Cap 1: A dirty toilet sitting dirty bathroon with broken nirror on the floor. Cap 2: Urine filled toilet in a state of disrepair.
Cap 2: A nice big living room with a te television and a table.
Cap 1: Three ducks
sitting the grass, looking at the camera.
Cap 1: Three slices of pizza are white plates, with one slice missing.
Cap 2:
Some very cute looking ducks in the grass.
Cap piece of pizza sits on white plate that has gold accents.
Cap 1: horse is tied to a wooden bui lding, with saddle its back,
Cap yellow white train is traveling down the tracks, passing under bridge.
cap 2: A horse strapped to the front of a carriage stop in front of wooden building.
Cap 2: A doub le decker train is going down the tracks.
wooden bench is located a park setting with cenent ground.
Cap 1:
Cap Two clock towers lit up night, standing in front of a building
Cap 2: red wooden bench sits in park with concrete ground.
Cap 2: Two tall clock towers above multiple circular Light fixtures.
Cap 1: A snall bird with a blue head and yellow body perched netal fence.
Cap 1: A white plate with a varisty of fruits on it, inc luding raspberries, and an blueberries, cap cooked lunch with fruit sits on table.
Cap bird sits the ground near fence
Cap Two elpphants are walking through auddy river, with one then looking back.
Cap 1: brown white dog laying on a bed with its head on pillow.
Cap 2: Two brown elephants wa walking through mud in jungle.
Cap 2: A dog that is laying down on a bed.
Figure 5: More qualitative results of the distilled dataset.
C Broader Impacts and Limitations
The societal impacts of our work are multifaceted. Positively, it contributes to greater resource efficiency, accelerates advancements in artificial intelligence, improves accessibility, and can enhance data privacy. However, also introduces potential drawbacks, such as the possibility of information loss, the introduction of security vulnerabilities, and the complication of intellectual property rights. careful balance between these advantages and challenges paramount to ensure the responsible and ethical application of this technology.
We recognize that the current approach may exhibit limitations when applied to highly specialized do- mains such as medical imaging, where the data distribution, structural characteristics, and annotation
21

---

protocols differ substantially from those of general-prrpsse datasets used in our experiments. These domain-specifit speciiic complexities may not be fully captured by the existing framework. Future research could address this limitation by incorporating domain-informed priors, adapting the optimization objectives, or designing customized representations that better align with the underlying structure and semantics of such specialized data.
22


Reviews:
[{"review_id": "cp2qwQonDa", "reviewer": "Reviewer_Hjt5", "summary": "This paper proposes an efficient multimodal dataset distillation method named EDGE, which is designed to compress large-scale vision-language datasets using generative models, thereby improving model training efficiency and performance. The EDGE method introduces a bidirectional contrastive loss and a diversity loss to enhance the correlation between images and text and to increase sample diversity. Furthermore, a post-training caption synthesis strategy is proposed, which utilizes multimodal large language models (MLLMs) to further boost performance on text-to-image retrieval tasks. Experimental results show that compared to existing methods, EDGE not only achieves superior performance on datasets of varying scales such as Flickr30K, COCO, and CC3M, but also demonstrates significant advantages in speed and computational resource consumption, especially when processing large-scale datasets. This proves its dual advantages in both efficiency and effectiveness.", "questions": "1. Could you provide a plot showing the relationship between the performance achieved and the computation time required for the training samples distilled by various methods?\n\n2. Regarding the method's extensibility, could it be applied to other tasks, and what do you see as its most promising application areas?", "limitations": "Please provide a limitation section.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### **Strengths:**\n\n1.  This study introduces EDGE, an efficient multimodal dataset distillation method that compresses large-scale vision-language datasets using generative models, thereby improving model training efficiency and performance.\n\n2.  Experimental results demonstrate that, compared to existing methods, EDGE not only achieves superior performance on datasets of varying scales such as Flickr30K, COCO, and CC3M, but also shows significant advantages in speed and computational cost. This proves its dual advantage in both efficiency and effectiveness, especially when processing large-scale datasets.\n\n### **Weaknesses:**\n\n1.  The study lacks a feature analysis of the distilled data samples. \n\n2.  A clearer explanation is needed for the performance gap between the proposed method and LoRS, as seen, for example, in Table 2.", "was_revised": true, "final_justification": "The authors answer my questions and fix the weakness.", "rebuttal": "We sincerely thank you for the valuable questions and comments. For the concerns and questions, here are our responses:\n### **W1. The study lacks a feature analysis of the distilled data samples.**\n\n**Table 1.** Feature analysis compared to SOTA methods.\n\n| Methods           | MMD ↓   | NNO ↑   | Entropy ↑ |\n|-------------------|---------|---------|------------|\n| MTT-VL            | 0.0293  | 71.04%  | 7.79       |\n| LoRS              | 0.0261  | 78.11%  | 7.87       |\n| Stable-diffusion  | 0.0286  | 70.55%  | 8.01       |\n| EDGE (ours)       | 0.0242  | 79.68%  | 8.34       |\n\nWe appreciate the reviewer’s constructive suggestion and have carried out a comprehensive feature-level evaluation of the distilled dataset. We conducted an extensive analysis in Table 1 that benchmarks our method, against MTT‑VL, LoRS, and a pre-trained Stable Diffusion baseline. Specifically, we include a table reporting Maximum Mean Discrepancy (MMD), Nearest‑Neighbor Overlap (NNO), and Entropy. EDGE achieves the lowest MMD, indicating the closest alignment to the original distribution, while simultaneously attaining the highest NNO  and Entropy, demonstrating superior semantic coverage and lexical richness. These results corroborate the previously reported FID and CLIP‑score improvements, collectively showing that EDGE preserves data fidelity without sacrificing diversity or image–text coherence.\n\nTo make the findings more intuitive, we also plotted t‑SNE visualizations that reveal EDGE samples forming compact clusters largely overlapping the original data manifold, whereas competing methods exhibit noticeable drift or mode collapse. We will add these plots and the full numerical results to the revised manuscript.\n### **W2: A clearer explanation is needed for the performance gap between the proposed method and LoRS.**\nThank you for your comments. LoRS is fundamentally a trajectory‑matching method [1] that optimizes a synthetic set so that the parameter trajectory taken by a specific network trained on that set mirrors its training trajectory on the original dataset. Such a model-specific method performs outstandingly when using the exact same model architecture, but compared to our method, it has two fatal drawbacks:\n\n1\\. LoRS suffers from cross-model evaluation. Because the LoRS optimisation explicitly relies on gradients from a single architecture, the resulting distilled set is highly specialised: it performs well when evaluated with the same backbone but loses predictive power when the evaluation network changes. As shown in Table 7, when using model architectures that are different from the model used during distillation, the performance of LoRS drops dynamically.\n\nOur proposed EDGE aims to explore the dataset representation regardless of which network is used. From Table 7, it is observed that although LoRS performance drops dynamically, EDGE can still achieve a decent performance when using other network architectures.\n\n2\\. A second factor is the computing budget. Trajectory matching back‑propagates through many unrolled training steps, which becomes prohibitive on large datasets and limits the number of outer‑loop updates LoRS can afford. The trajectory-based distillation methods suffer from a huge computational requirement. As we highlighted in Figure 1 and Table 6, our proposed generative-distillation method EDGE is significantly efficient than LoRS, which is 18 times faster.\n\nIn conclusion, the performance gap comes from LoRS sacrificing the performance of cross-model evaluation. Furthermore, our proposed method is more efficient.\n\n[1] Dataset Distillation by Matching Training Trajectories. CVPR 2022.\n\n### **Q1: Could you provide a plot showing the relationship between the performance achieved and the computation time required for the training samples distilled by various methods?**\n\n**Table 2.** Performance *vs.* GPU hour usage.\n\n| Method       | Validation IR@1 ↑ | Validation TR@1 ↑ | GPU hour |\n|--------------|------------------|------------------|----------|\n| MTT-VL       | 1.8              | 2.5              | 194.1    |\n| LoRS         | 2.5              | 3.6              | 155.6    |\n| EDGE (ours)  | 2.8              | 3.9              | 8.5      |\n\nThank you for the suggestion. As image uploading is disabled this year, we instead provide Table 2 summarizing the relationship between performance and GPU time for the distilled training samples. The experiments are conducted on NVIDIA RTX A5000 GPUs. We distilled the COCO datasets to 1000 image-text pairs.\n\nFrom the table, it is clear that our method achieves higher accuracy than state-of-the-art baselines while requiring only a fraction of the computational cost. Specifically, EDGE delivers better performance while using just 5% of the GPU time, demonstrating both superior accuracy and substantial efficiency gains.\n\n### **Q2: Regarding the method's extensibility, could it be applied to other tasks, and what do you see as its most promising application areas?**\n\n**Table 3.** VQA metrics evaluation.\n\n| Methods           | Tifa-score ↑ | DSG score with dependency ↑ | DSG score without dependency ↑ |\n|-------------------|------------|----------------------------|-------------------------------|\n| LoRS              | 0.856   | 0.852                   | 0.876                   |\n| Stable-diffusion  | 0.824     | 0.771                    | 0.805                      |\n| EDGE (ours)       | 0.877     | 0.874                      | 0.894                         |\n\nWe appreciate the reviewer’s interest in our work and address the question of extensibility in two parts: technical adaptability and practical application. \n\nEDGE is designed around a task‑agnostic generative prior and a modality‑aligned objective, so adapting it to new task settings primarily requires substituting the downstream loss or the diffusion backbone. With minor changes, our method can be applied to video-text datasets and other image-text tasks such as VQA. We perform a VQA evaluation on the distilled dataset of COCO with TIFA[1] and DSG[2], compared with other distilled datasets to show the effectiveness of our method in Table 3.\n\nThe distilled dataset by our method has multiple promising applications. First of all, the distilled dataset can speed up the training efficiency and protect sensitive content leakage of the original dataset. Besides, the generated dataset can be a part of supplemental data for large generative model training and mLLM model training. The distilled data can also serve as supplemental, diversity‑enhancing supplements for large generative‑model pre‑training or for recent multimodal LLMs, which benefit from additional image–caption pairs yet are often bottlenecked by curation costs.\n\n[1] TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answering. ICCV 2023.\n\n[2] Davidsonian Scene Graph: Improving Reliability in Fine-grained Evaluation for Text-to-Image Generation. ICLR 2024.\n\n### **Limitation of our method**\nThank you for your thoughtful comment. We acknowledge that our current method may have limitations when applied to highly specialized domains such as medical imaging, where data characteristics can differ significantly from the general datasets used in our evaluation. These domains may involve unique structural patterns or annotation constraints that are not fully captured in our current experiments. We believe future work could explore adapting our approach to such specialized settings by incorporating domain-specific priors or tailoring the optimization process accordingly. We will add a detailed discussion of these limitations in the revised manuscript."}, {"review_id": "2m5yGHw8at", "reviewer": "Reviewer_nNDs", "summary": "This paper introduces EDGE, a method for efficient multimodal dataset distillation using generative models. The authors tackle two main challenges: poor correlation between generated images and text, and lack of diversity in generated samples. Their solution combines a bi-directional contrastive loss with a diversity loss, plus a caption synthesis strategy. Experiments on Flickr30K, COCO, and CC3M show the method works 18× faster than current approaches while maintaining competitive performance.", "questions": "See weakness", "limitations": "See weakness", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n1. It successfully distills the large-scale CC3M dataset, which previous methods couldn't handle. The caption synthesis strategy is very practical.\n\n2. The proposed method dramatically reduces computational requirements compared to existing approaches. The ability to distill datasets 18× faster is a substantial contribution.\n\n3. It works consistently across different architectures, unlike baseline methods that show severe performance drops when switching models.\n\n4. The experiments are comprehensive: good ablation studies on different components, caption-per-image ratios, and caption synthesis techniques.\n\n**Weakness**\n\n1. While the paper emphasizes EDGE's efficiency advantages over baseline methods like LoRS (which performs quite well across most setups), it lacks a controlled comparison based on FLOPs. The authors primarily focus on runtime and memory usage comparisons, which can be influenced by implementation details and hardware configurations. A more rigorous efficiency comparison controlling for computational resources would strengthen the claim that EDGE is both efficient and effective.\n\n2. In Table 10, the proposed method performance drops a lot when setting CPI=2 to CPI=5. These results may suggest that the model is bit sensitive and not robust enough to apply to different setups", "was_revised": true, "final_justification": "It addressed my concerns. I will raise the score accordingly", "rebuttal": "We sincerely thank you for the valuable questions and comments. For the concerns and questions, here are our responses:\n### **W1: A rigorous efficiency comparison.**\n\n| Method        | FLOPs Required |\n|---------------|----------------|\n| LoRS          | 2528 PFLOPs     |\n| EDGE (ours)   | 149 PFLOPs     |\n\nThank you for your comment. In the revised manuscript, we have added a new Table, which reports the total floating-point operations required by each method when distilling the COCO dataset used in our study. FLOP counts were computed with the PyTorch profile. All experiments were executed on a single NVIDIA RTX A5000 GPU with CUDA 12.4, and the host system provided 48 CPU cores and 256 GB of RAM.\n\nLoRS consumes 2.5 EFLOPs, whereas EDGE requires only 149 PFLOPs. This metric substantiates the claim that EDGE is intrinsically more compute efficient rather than merely faster due to implementation details.\n\n### **W2: Explanation of performance drops when setting CPI=2 to CPI=5.**\n\nThank you for raising this point. In our setup, as CPI increases, the unique images in the generated dataset decrease to preserve the consistency of the total number of image-text pairs. CPI refers to “captions per image.” When we increase CPI from 2 to 5, we intentionally keep the total number of image–text pairs constant for a fair comparison. This means the number of unique images in the generated dataset shrinks from 250 to 100 (2 × 250 = 5 × 100 = 500 pairs). Such a substantial reduction in image diversity naturally lowers performance, so the observed drop reflects the altered dataset composition rather than a lack of robustness in our method."}, {"review_id": "ORIoa1elMM", "reviewer": "Reviewer_3o9X", "summary": "This paper proposes EDGE, an efficient generative method for multimodal dataset distillation, aiming to address the high computational costs and long distillation time of existing MTT-based methods. It identifies two key challenges in distilling multimodal datasets with generative models: the lack of correlation between generated images and captions, and the lack of diversity among generated samples. To tackle these, EDGE introduces a novel training workflow with a bi-directional contrastive loss and a diversity loss, as well as a caption synthesis strategy to improve text-to-image retrieval performance. Evaluations on Flickr30K, COCO, and CC3M datasets show that EDGE achieves superior performance and efficiency, being up to 18× faster than the state-of-the-art method while distilling large-scale datasets effectively.", "questions": "1. Why is the CC3M dataset particularly challenging for existing dataset distillation methods, and how does EDGE address this?\n2. What is the role of the diversity loss in EDGE, and how does it differ from traditional dataset distillation approaches?\n3. What is the purpose of the caption synthesis strategy in EDGE, and how does it work?\n4. What is the role of the diversity loss in EDGE, and how does it differ from traditional dataset distillation approaches?", "limitations": "This paper has two limitations. It relies on pre-trained generative models like Stable Diffusion, which may introduce biases and limit generalizability to specialized domains. The diversity loss, while effective empirically, lacks a rigorous theoretical foundation for matching the original dataset’s distribution.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Strengths\nThe paper presents EDGE, a generative approach for multimodal dataset distillation that efficiently addresses the computational inefficiencies of existing MTT-based methods. By integrating bidirectional contrastive loss and diversity loss, EDGE enhances image-text correlation and sample diversity, demonstrating superior performance on Flickr30K, COCO, and the large-scale CC3M dataset. Notably, it achieves distillation up to 18 times faster than state-of-the-art methods with 16× lower memory usage, making it feasible for resource-constrained scenarios.\nWeaknesses\nWhile EDGE shows significant efficiency gains, the reliance on pre-trained diffusion models like Stable Diffusion may limit its generalizability to specialized domains where model biases exist. The theoretical foundation of the diversity loss lacks explicit justification for matching original dataset distributions, and the study does not validate the method on other multimodal tasks (e.g., video-text).", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for the valuable questions and comments. For the concerns and questions, here are our responses:\n### **W1. While EDGE shows significant efficiency gains, the reliance on pre-trained diffusion models like Stable Diffusion may limit its generalizability to specialized domains where model biases exist.**\n\nThank you for your comment. Although our implementation employs a Stable Diffusion backbone that is pre-trained on LAION-5B, we fine-tune the generator jointly with EDGE on every target corpus. Our primary experiments are conducted on Flickr30K and COCO, whose visual styles and caption vocabularies differ from LAION. Before finetuning, Stable Diffusion achieves 3.3 and 1.8 recall@1 on these datasets, respectively; after applying EDGE, the scores rise to 9.9 and 2.8. These gains, together with consistent improvements in MMD, demonstrate that EDGE can adapt a biased or domain-mismatched generator to unfamiliar content libraries with only a modest compute budget.\n\n### **W2 & Q2 & Q4: The diversity loss lacks explicit justification for matching original dataset distributions. What is the role of the diversity loss in EDGE, and how does it differ from traditional dataset distillation approaches?**\n\n**Table 1.** Feature analysis compared to SOTA methods.\n\n| Methods           | MMD ↓   | NNO ↑   | Entropy ↑ |\n|-------------------|---------|---------|------------|\n| MTT-VL            | 0.0293  | 71.04%  | 7.79       |\n| LoRS              | 0.0261  | 78.11%  | 7.87       |\n| Stable-diffusion  | 0.0286  | 70.55%  | 8.01       |\n| EDGE (ours)       | 0.0242  | 79.68%  | 8.34       |\n\nThank you for the thoughtful feedback. In EDGE, the diversity loss complements the contrastive objective by explicitly maximizing the pairwise angular distance between distilled image–text embeddings. This design is inspired by the maximum entropy principle: among all low-cardinality datasets that satisfy the contrastive constraints, we prefer the one whose embeddings occupy the largest feasible volume in the representation space. By discouraging co-location of different samples, the loss mitigates the tendency of small distilled sets to collapse onto a few modes and thus helps the synthetic distribution approximate the support of the original data. While the contrastive loss is responsible for aligning corresponding image-text embeddings to ensure semantic consistency, the diversity loss serves as a complementary objective that actively disperses the embeddings of different image-text pairs. In this way, EDGE explicitly encourages inter-sample variation through diversity loss, thereby enhancing generalization and distributional fidelity.\n\nTraditional trajectory-matching-based dataset distillation approaches (MTT-VL, LoRS) usually do not consider the diversity, since they strictly focus on the performance of a specific model architecture, ignoring the evaluation on other models.\n\nAdditionally, our feature analysis experiments in Table 1 demonstrate the effectiveness of the diversity loss, where we observe that incorporating it leads to a reduction in MMD and an increase in NNO and Entropy, further supporting its role in promoting broader distributional coverage.\n\n### **W3: The study does not validate the method on other multimodal tasks.** \n\n**Table 2.** VQA metrics evaluation.\n\n| Methods         | Tifa-score ↑ | DSG score with dependency ↑ | DSG score without dependency ↑ |\n|----------------|------------|----------------------------|------------------------------|\n| LoRS           | 0.856      | 0.852                      | 0.876                        |\n| Stable-diffusion | 0.824    | 0.771                      | 0.805                        |\n| EDGE (ours)    | 0.877      | 0.874                      | 0.894                        |\n\nWe appreciate the reviewer’s suggestion. To evaluate EDGE beyond image–text retrieval, we have added experiments on the VQA TIFA [1] and DSG [2] benchmarks in Table 2. These extensions confirm that the advantages conferred by EDGE are not confined to static imagery, and we commit to releasing our code for video distillation to facilitate future research.\n\n[1] TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answering. ICCV 2023.\n\n[2] Davidsonian Scene Graph: Improving Reliability in Fine-grained Evaluation for Text-to-Image Generation. ICLR 2024.\n\n### **Q1: Why is the CC3M dataset particularly challenging for existing dataset distillation methods, and how does EDGE address this?**\n\nThank you for the question. CC3M poses a substantially tougher testbed than Flickr30K and COCO because of its scale: it contains roughly 3.3 million image–text pairs, compared with 31 K and 123 K images (each with five captions) in Flickr30K and COCO, respectively. Existing distillation approaches rely on trajectory‑matching procedures whose memory and compute demands grow with dataset size. This limitation stems from the objective function of Matching Training Trjectories (MTT) [1], which involves unrolling T steps of SGD using synthetic images and aligning the resulting model weights with a reference point obtained from training on the original dataset. Because this process requires differentiating through T optimization steps, it entails constructing and retaining T gradient computation graphs in memory. As a result, the memory overhead becomes prohibitive when applied to large-scale datasets.\n\nEDGE sidesteps this bottleneck by introducing a lightweight generative distillation strategy that eliminates the need for full‑dataset gradient matching. Instead, it learns a compact generative prior that can be optimized with a small, fixed computational budget, enabling us to distill CC3M efficiently while retaining competitive downstream performance.\n\n[1] Dataset Distillation by Matching Training Trajectories. CVPR 2022.\n\n### **Q3: What is the purpose of the caption synthesis strategy in EDGE, and how does it work?**\n\nThank you for the question. The purpose of the caption synthesis strategy in EDGE is to improve performance on the text-to-image retrieval task, which we found to be more challenging than its image-to-text counterpart. As discussed at the beginning of Section 3.4, there exists an inherent asymmetry in retrieval difficulty between these two directions. Specifically, generated images may not always have sufficiently rich or semantically grounded captions, which can hinder the model's ability to learn strong text-to-image alignment.\n\nTo address this, we introduce a caption synthesis strategy that augments the captions associated with the generated images. This is achieved by leveraging a pretrained captioning model to generate more informative and diverse textual descriptions, thereby enriching the training data and improving the alignment quality. These synthesized captions supplement the original ones and help the model better associate generated images with a broader set of possible textual queries. Empirically, we observe that this strategy leads to notable improvements in text-to-image retrieval accuracy, as shown in Table 8. We believe this component plays an important role in balancing the retrieval performance across modalities and enhancing the generalization capability of the distilled dataset.\n\n### **Formatting Concerns**\nThank you for pointing this out. We will carefully revise all entries to ensure that titles, author lists, publication dates, and venues are complete and accurate."}, {"review_id": "iguEjECWft", "reviewer": "Reviewer_r1pa", "summary": "The paper introduces a novel generative method (EDGE) for multimodal dataset distillation, addressing the limitations of existing approaches that are computationally intensive and lack scalability for large image-text datasets. The gist of the method is in the reformulation of the contrastive loss inspired by InfoNCE that alignes text-image embeddings, and added diversity loss. The authors show that they can match or outperform existing methods at a fraction of computational cost, including distilling large datasets that were out of reach for previous methods.", "questions": "* One of the 2 objectives is to increase the diversity of the distilled dataset - while that indeed seems to help as demonstrated with Table 8, do you have any other evals that could further substantiate the claims about increased diversity of the dataset? (eg. using Vendi score, DPP or some other diversity metric) \n   * Also the same question for text-image alignment evals (have you considered using TIFA/VNLI/Gecko/DSG?)\n\n* Do you have any intuition about why are CLIP scores for LORS so much lower compared to EDGE? This seems particularly surprising given that performance is on-par.  Similarly, FID scores are very high, would be good to have some intuition on why (also why did you not include LoRS results here?)\n\n* Limitations: Are there any pathological corner cases for your method that may have not been captured with the datasets you're looking at?", "limitations": "yes", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths: \n* Strong results substantiated with multiple experiments, the proposed method achieves results comparable to related work (LoRS) yet at a much lower computational footprint to the extent that it makes it possible to distill very large datasets (such as CC3M)\n* Novel proposal for generative loss \n* Comprehensive ablation studies (different losses, different MLLMs) to demonstrate robustness\n* The paper is very transparent about methods/datasets it uses, including highlighting where the competing methods perform better\n\nWeaknesses:\n* Evals could be enriched by adding a few more different methods to evaluate text-image alignment and diversity (for example, by considering VQA-based text-to-image alignment metrics) \n* A minor point, but it would be nice to gain some intuition about how are distilled datasets different/similar between EDGE and other methods (eg examples that would substantiate \"images generated by our method exhibit a realistic, high-quality appearance\")\n* No clear discussion of limitations of your method  \n* A bit more clarity on the scale of improvements would be fair: For example in Table 3: \"our method outperforms the baseline methods\" --> That \"outperform\" here carries a lot of weight here. The numbers are still very close to random baseline and in total those are very small numbers so I would advise to reframe this sentence to highlight that distilling CC3M is indeed still an open challenge.", "was_revised": true, "final_justification": "The authors adequately addressed my questions and have provided further experimental data that solidifies the existing results. I am happy to maintain the existing recommendation.", "rebuttal": "We sincerely thank you for the valuable questions and comments. For the concerns and questions, here are our responses:\n### **W1: Evals could be enriched by adding a few more different methods to evaluate text-image alignment and diversity.**\n\n**Table 1.** Feature analysis compared to SOTA methods.\n\n| Methods           | MMD ↓   | NNO ↑   | Entropy ↑ |\n|-------------------|---------|---------|------------|\n| MTT-VL            | 0.0293  | 71.04%  | 7.79       |\n| LoRS              | 0.0261  | 78.11%  | 7.87       |\n| Stable-diffusion  | 0.0286  | 70.55%  | 8.01       |\n| EDGE (ours)       | 0.0242  | 79.68%  | 8.34       |\n\n**Table 2.** VQA metrics evaluation.\n\n| Methods         | Tifa-score ↑ | DSG score with dependency ↑ | DSG score without dependency ↑ |\n|----------------|------------|----------------------------|------------------------------|\n| LoRS           | 0.856      | 0.852                      | 0.876                        |\n| Stable-diffusion | 0.824    | 0.771                      | 0.805                        |\n| EDGE (ours)    | 0.877      | 0.874                      | 0.894                        |\n\nThank you for your suggestions. To analyze dataset diversity, we now report Mean Distribution Discrepancy (MDD), Nearest Neighbor Overlap (NNO), and Entropy in Table 1, alongside the FID and CLIP‑based scores already in the paper. In the revised manuscript, we have also incorporated TIFA [1] and DSG [2], as shown in Table 2. These additional metrics consistently confirm that the distilled dataset maintains strong semantic alignment while exhibiting higher visual and textual variety than competing approaches. We will place the full numerical results and implementation details in the appendix and reference them in the main text to make these contributions clear to readers and reviewers.\n\n[1] TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answering. ICCV 2023.\n\n[2] Davidsonian Scene Graph: Improving Reliability in Fine-grained Evaluation for Text-to-Image Generation. ICLR 2024.\n\n### **W2: A minor point, but it would be nice to gain some intuition about how are distilled datasets different/similar between EDGE and other methods.**\n\nThank you for your suggestions. In the revised version, we will include a concise qualitative comparison that places representative samples from EDGE next to those generated by MTT‑VL and LoRS. As the NeurIPS rebuttal format prohibits the inclusion of new figures, we summarize the main observations here. In addition, both MTT-VL and LoRS also provide qualitative examples in their paper. Compared to these baselines, EDGE produces images with sharper edges and more faithfully preserved fine details such as fur, foliage, and specular highlights, whereas baseline outputs often exhibit noticeable blur and noise. Quantitative results corroborate the visual differences: EDGE attains substantially lower FID scores, indicating closer alignment with the original data distribution and higher perceptual quality.\n\n### **W3 & Q3: No clear discussion of limitations of your method. Limitations: Are there any pathological corner cases for your method that may have not been captured with the datasets you're looking at?**\n\nThank you for your thoughtful feedback. We acknowledge that our current method may have limitations when applied to highly specialized domains such as medical imaging, where data characteristics can differ significantly from the general datasets used in our evaluation. These domains may involve unique structural patterns or annotation constraints that are not fully captured in our current experiments. We believe future work could explore adapting our approach to such specialized settings by incorporating domain-specific priors or tailoring the optimization process accordingly. We will add a detailed discussion of these limitations in the revised manuscript.\n\n### **W4: Revise the statement of the scale of improvements.**\nThank you for the constructive feedback. We appreciate your observation regarding the phrasing in Table 3 and fully agree that clarity in interpreting the scale of improvements is important. In our revision, we will adjust the wording to more accurately reflect the relative improvements and acknowledge the overall difficulty of the task. Rather than stating that our method \"outperforms\" the baselines in absolute terms, we will highlight that the performance gains, though modest in magnitude, consistently demonstrate the advantages of our distillation approach over existing methods. We will also explicitly note that distilling CC3M remains a challenging open problem, which our method takes a step toward addressing. This reframing should provide a more balanced and accurate interpretation of the results.\n\n### **Q1: Evals that could further substantiate the claims about increased diversity of the dataset.**\n\nThank you for the thoughtful comment. In addition to Table 8, we have further evaluated the diversity of the distilled dataset using three complementary metrics: Minimum Distance Distribution (MDD), Nearest Neighbor Overlap (NNO), and Entropy. MDD quantifies the compactness of sample representations, NNO reflects neighborhood uniqueness, and Entropy measures the overall uncertainty in token usage. As shown in the table below, our method EDGE consistently achieves lower MDD, higher NNO, and higher Entropy compared to other baselines, indicating that EDGE generates more diverse and less redundant samples. These results provide additional evidence supporting our claim that enhancing diversity contributes to the improved performance of our distilled dataset. We appreciate the suggestion of using Vendi score or DPP and will consider including those in future work to further strengthen the analysis.\n\n### **Q1.1: Text-image alignment evals.**\n\nThank you for the valuable suggestion. We have now evaluated our method using the TIFA benchmark to better assess text-image alignment performance. As shown in the table below, EDGE achieves a TIFA score of 0.8773, outperforming both LoRS and Stable Diffusion. Additionally, we have evaluated our method using DSG scores, both with and without dependency annotations. EDGE again demonstrates superior performance, achieving scores of 0.874 and 0.894, respectively. These results reinforce the strength of EDGE in aligning image and text modalities. We appreciate the reviewer’s insight and will consider including additional benchmarks such as VNLI and Gecko in future work to provide a more comprehensive evaluation.\n\n### **Q2: Do you have any intuition about why are CLIP scores for LORS so much lower compared to EDGE? This seems particularly surprising given that performance is on-par. Similarly, FID scores are very high, would be good to have some intuition on why (also why did you not include LoRS results here?)**\n\n**Table 3.** FID score comparison.\n\n| Methods      | Flickr-30K | COCO  |\n|--------------|------------|-------|\n| MTT-VL       | 210.0      | 276.3 |\n| LoRS         | 114.7      | 91.6  |\n| EDGE (ours)  | 88.1       | 83.1  |\n\nThank you for the thoughtful comment. Regarding the FID scores, we believe the notably higher FID of MTT-VL and LoRS compared to EDGE stems from fundamental differences in how the datasets are constructed. MTT-VL and LoRS rely on gradient-based updates to synthetic images guided by trajectory matching loss, which can lead to artifacts or unrealistic samples. In contrast, EDGE first learns a compact representation of the target dataset and then generates images using the fine-tuned generative model conditioned on this representation. This design allows EDGE to produce more visually realistic samples, resulting in significantly lower FID scores. We included the FID scores for LoRS in Table 3.\n\nAs for the lower CLIP scores observed for LoR, we believe this arises from how LoRS updates text embeddings. Specifically, LoRS perturbs the original text embeddings by adding gradients from the trajectory matching loss. While this may help align synthetic image-text pairs for training, it can disrupt the semantic structure of the original text embeddings, which negatively impacts CLIP scores that depend on the global alignment between image and text representations. Nevertheless, LoRS maintains a learned image-text similarity matrix that guides evaluation model training, compensating for this distortion and enabling it to achieve reasonable performance despite the lower CLIP alignment."}]

Author rebuttal:

Meta-review: This paper proposes an efficient and scalable multimodal dataset distillation method that meaningfully advances the field. The key strengths are the combination of generative-based distillation with contrastive and diversity loss functions, as well as a practical caption synthesis step, together with strong experimental validation showing both superior accuracy and dramatically improved efficiency.  

The reviewers did note some issues: evaluation could be broadened with more alignment/diversity metrics, qualitative examples would strengthen intuitions about dataset quality, and some claims are phrased more strongly than warranted given the absolute performance numbers. During the rebuttal, the above concerns have been addressed and are promised to be revised accordingly. The reviewers are all positive and vote to accept the paper.
Decision: Accept (poster)