Vision-Language Dataset Distillation
Authors: Xindi Wu
Byron Zhang
Zhiwei Deng
Olga Russakovsky
Venue: Accepted by TMLR ()

Abstract: Dataset distillation methods reduce large-scale datasets to smaller sets of synthetic data, preserving sufficient information to quickly train a new model from scratch. However, prior work on dataset distillation has focused exclusively on image classification datasets, whereas modern large-scale datasets are primarily in the vision-language space. In this work, we design the first vision-language dataset distillation method, building on the idea of trajectory matching. A key challenge is that vision-language datasets do not have a set of discrete classes. To overcome this, our proposed method jointly distills image-text pairs in a contrastive formulation. Further, we leverage Low-Rank Adaptation (LoRA) matching to enable more efficient and effective trajectory matching in complex modern vision-language models. Since there are no existing baselines, we compare our distillation approach with three adapted vision-language coreset selection methods. We demonstrate significant improvements on the challenging Flickr30K and COCO retrieval benchmarks: for example, on Flickr30K, the best coreset selection method selecting 1000 image-text pairs for training achieves only 5.6% image-to-text retrieval accuracy (i.e., recall@1); in contrast, our dataset distillation almost doubles that to 9.9% with just 100 training pairs, an order of magnitude fewer.

Published in Transactions on Machine Learning Research (08/2024)
Vision-Language Dataset Distillation
Xindi Wu' Byron Zhang' Zhiwei Deng²
Olga Russakovsky'
Princeton, University Google Deed
(rindin, zishuoz, olgorus) Sprinncttprnne shiueideng@@pooge.ccom
Reviewed on OpenReview:
setearevvewnnee/oorreeee forum? id- 6L6cD65pot
Website: https: httpss///rrncttoovsssslllas github. so/ io/multimddal, datasstadddddddssett.eeaaeae distillation/
Abstract
Dataset distillation methods reduce large-scale datasets to smaller sets of synthetic data, preserving sufficient information to quickly train a new model from scratch. However, prior work on dataset distillation has focused exclusively on image classification datasets, whereas modern large-scale datasets are primarily vision-language datasets. In this work, design the first vision-language dataset distillation method, building on the idea of trajectory matching. A key challenge is that rision-languas datasets do not have a set of discrete classes. To overcome this, our proposed method jointly distills image-text pairs in a contrastive formulation. Further, we leverage Low-Rank Adaptation (LoRA) matching to enable more efficient and effective trajectory matching complex modern laiiunla models. Since there are existing baselines, we compare our distillation approach with three adapted vision-language coresct selection methods. We demonstrate significant improvements the challenging Flickr30K and COCO retrieval benchmarks: for example, best coreset selection method selecting 1000 image-text pairs for training achieves only 5.6% image-to-text. retrioval accuracy (i.c., recall@1); in contrast, our dataset distillation almost doubles that to 9.9% with just 100 training pairs, an order of magnitude fewer.
Flickr30K, the
1 Introduction
Data Information + Irrelevant Data
(Wright & Ma, 2022)
Dataset distillation aims to create concise sunmaries of data that preserve most of the critical information of the entire datasct. It holds paramount importance in the cra of big data as addresses the challenge posed by " DDaaa = Information + Irrelevant Data" (Wright & Ma, 2022), where we often need to learn the useful information in an ocean of non-critical data. The recent growth of dataset distillation methods, c.g., (Wang et al., 2018; Cazenavette et al., 2022; Nguyen et al., 2020) has primarily focused image classification datasets, capturing class-specific information to build discriminative boundaries. Considering the recent progress in multinodal machine learning, where we witness the explosion of vision-language datasets in which the majority of image pixels may belong to irrelevant contextual elements and may further lack corresponding textual descriptions, a significant necessity arises to efficiently distill this vast amount of data. A well-distiledd multimodal dataset simplifies complex vision-language interactions and emphasizes the most. salient connections, making more effective for models to learn cross-modal representations.
Why is it hard? The first key challenge and the main difference from prior dataset distillation methods (Wang et al., 2018; Cazenavette al., 2022) is that vision-languag datasets do not contain discrete set of classes to ground the distillation process. Instead, these datascts contain complex cross-modal connections and redundancies, requiring a co-distillation approach to capture their interdependencies effectively. Second, the complexity of cross-modal representations and lision-language models (VLMs) leads to computational challenges. Prior datasct distillation methods operate on low-rcsolution images (typically 28x28 or 32x32, as in MNIST (LeCun et al., 1998) or CIFAR (Krizhevsky et al., 2009)) and nevertheless suffer from significant computational costs, even with ConyNet when creating distilled datasets. Vision-language datasets often

---

Published in Transactions on Machine Learning Research (08/2024)
Image-Label
Image-Bare
Vision-Languaa
labels
distilled Images
labels
Basis
distilled Images
distiled text embeds
distilled Images
"a cat figurine set in the bathroom by toilet'
"cat" "cat"
"brown dog running through shallow water*
"dog"
"dogg
"surfer surfing in a beautiful with birds around and waves with beautiful texture"
"bird"
"bird"
addressing matrices
Figure 1: Dataset Distillation Comparison. (Left) Prior dataset distillation methods (Wang et al., 2018; Cazenavette et al., 2022; Nguyen et al., 2020) are class-specific: they distill the key information for each individual discrete class. (Center) Even the recently developed method (Deng & Russakovsky, 2022), which enables information sharing between classes through Icarned bases, still assumes a discrete set of classes. (Right) In contrast, we set out to distill vision-language datasets with no discrete classes; we do so via a novel method which jointly distills images and texts.
contain higher-resolution images, and models designed for vision-language tasks are substantially more complex, such as Vision Transformers (ViTs) (Dosovitskiy et al., 2020). Lastly, unlike continuous data, text is inherently non-differentiable, making direct gradient- optimization impossible on discrete text tokens.
Our work. We propose the first Vision-Language Dataset Distillation method. Concretely, given a datasel of images with corresponding text descriptions, our method creates a much smaller synthetic set of (image, text embedding) pairs which can then be used to efficiently train a model that aims to learn the image-text aligument. Given the infeasibility direct information extraction, our co-distillation is achieved by implicitly matching the by-products of the target vision-language data and the synthetic ones. In our case, the by-product is the long-range training bi-trajectory- Additionally, as finetuning pretrained models is widely used in vision-languee tasks, match the trajectories of the low-rank matrices (Hu et al., 2021) for complex models to effectively distill critical information.
Contributions. To the best of our knowledge, this is the first work to tackle vision-language dataset. distillation. In doing so, we make the following key contributions:
We highlight the challenges of vision-language dataset distillation and establish the first set of baselines for this task by adapting three coreset selection methods (Welling, 2009; Toneva al., 2018; Farahani & Hekmatfar, 2009; Sener & Savarese, 2017).
2. We propose the Bi-Trajectory Vision-Language Co-Distillation method. Different from prior image classification dataset distillation methods, our method is not restricted to discrete classes and distills vision-languae language pairs jointly. We leverage Low-Rank Adaptation (LoRA) matching to make it computationally feasible for training with complex models (e.g., ViTIs) on high-resolution images. Our method significantly improves image-text retrieval with training set constraints on the challenging Flickr30K (Plummer et al., 2015) and COCO (Lin el al., 2014) datasets. For example, the best coreset. selection method (adapted K-center) achieves 5.6% image-to-text retrieval performance (R@1) after selecting 1000 image-text pairs for training. In contrast, our method almost doubles that performance on the same task to 9.9% with an order of magnitude fewer (just 100) distilled image-text pairs.
The growing interest in multimodal datascts makes it even more crucial to develop mochanisms that efficiently and effectively distill insights from different modalities. We hope this work jump-starts further research into the important and challenging space of vision-language dataset distillation.
2

---

Published in Transactions on Machine Learning Research (08/2024)
2
Related Works
The concept of datasct distillation has demonstratod that a handful of synthetic images, although not drawn from the training distribution, can achieve comparable performance to that of the original dataset (Wang et al., 2018). Meta-learning based data distillation approaches (Nguyen al., 2021; Zhou et al., 2022; Deng & Russakovsky 2022; Nguyen et al., 2020; Vicol et al., 2022; Zhou et al., 2022) typically use bilevel optimization, where the inner loop trains on the distilled data samples and the outer loop optimizes meta datasets. Several works (Zhao & Bilen, 2021b;a; Cazenavette et al., 2022; Jiang et al., 2022; Du et al., 2023; Liu et al., 2023) explored by-product matching approaches, such as matching the gradient or trajectory of the gradient with respect to the model trained on the real and distilled data.
Dataset Distillation.
Our work is mostly inspired by the trajectory matching method (Cazenavette al., 2022; Cui et al., 2022), which is more efficient for optimization since they mostly do not involve long unrolling of computation graphs. Rather than aligning model gradients, another thread of work (Zhao & Bilen, 2021b; Wang el. al., 2022; Lee et al,, 2022) has been developed to align feature distributions between real and distilled data using a distribution divergence metric in the latent space. While most of the prior works focus on image classification dataset distillations, (Sucholutsky & Schonlau, 2021) explored dataset distillation on text datasets. Our work the first to scale up dataset distillation methods to vision-language datasets, which involves creating distilled data that capture critical features and complex relationships within and between two modalities.
Cross-modal Retrieval. Most cross-mode retrieval methods function at the representation level and encourage a joint embedding space by measuring the similarities between learned representations across different modalities (Liang et al., 2022; Zhu et al., 2022; Pokle et al., 2022; Chun et al., 2021; Wu et al., 2023). Image-text retrieval focuses on retrieving images given captions, or of captions given images (Wang et al., 2020b; Wu et al., 2019). Many techniques have been developed to produce representations that are semantically similar for imago-text pairs (Huang et al., 2018; Gu et al., 2018). More advanced imago- text alignment hods (Li et al., 2022; Lin et al., 2023; Pandey el al., 2022) that incorporate pretraining have shown promising results image-text retrieval tasks. We evaluate our vision-language dataset distillation method on image-text retrieval tasks.
Vision-language Knowledge Distillation. Prior efforts on vision-language distillation are primarily centered around knowledge distillation, which transfers knowledge from a larger teacher model to a smaller student model to improve the latter's performance (Xue et al., 2023; Radenovic et al., 2023; Valverde et al., 2021). Our dataset distillation study focuses on the orthogonal question and is fundamentally a pragmatic compression problem. We aim to find equivalent bits that can represent the entire vision-language datasets.
3 Method
We propose a rision-languae dataset distillation method for distilling a large-scale dataset consisting of (image, text) pairs into a smaller dataset, while maintaining much of the original datasct's information relevant to training vision-language models (VLMs). The detailed method is in Fig. 2.
3.1 Problem Formulation
Consider large-scale dataset D = (( y:) ) where each z; denotes an image and each yi denotes its corresponding text descriptions; note that in practice, Yi may be a set [ Vi2. yik. } where K is the number of descriptions associated with each image. Our goal is to learn smaller dataset D= {($). 0,) 1=1 =
with siguificantly fewer data pairs M & N that still captures most of the essential information needed to train VLM effectively. For Di, we aim to use one (instead of K) sentence per image in the distilled set for a more compact representation. Concretely, consider a VLM with vision encoder f(; 0smg) and language encoder g(:: This model can be trained by optimizing the similarity loss which encourages alignment between the image and text embeddings:
1 N
8" = arg min
Σ 1=1
E 0l(g) 0tze))
(1)
N
3

---

Published in Transactions on Machine Learning Research (08/2024)
contrastive
image trajectories
Image Encoder
loss
D
joint
distilled pairs
embedding space
bi-trajectory matching loss
distilled data
text trajectories
+
*
"the cat looks
Text Encoder
away from the person next
contrastive
loss
pretrained and frozen
pretrained and trainable
projection leyer, train from scratch
expert trajectory
student trajectory
update distilled set
Figure 2: Vision-Language Dataset Distillation. Both the image and text encoders are pretrained and followed by a trainable projection layer, and the text encoder is frozen. We use contrastive loss to measure the distance between the paired image-lext embeddings, which influences the trajectory updates during distillation. The right panel shows how the distilled data aligns its training trajectory with the expert's, from random starting point on the expert trajectory. The distilled dataset is updated based on bi-trajectory matching loss between the student and expert parameter trajectories.
Our goal is to distill a dataset D such that the model trained with D obtains comparable vision-langunge matching performance as the one trained on D. More specifically, consider a metric m defined to quantify the correlation between the model's representation fa; 0mgg) of given image æ and the representation g(y: 0mgg) of given text y, this representation should match the actual similarity between the image and text pairs. The correlation calculation is based whether the image-text pair is positive (matching) or negative (non-matching) pair. Given the test dalaset Drest; our objective can be defined as follows:
E (aay))Deent [ff(r; (. gly; ( E(z.y))D.... [m(f(a; 0mmg) g(y; (tmz))]
(2)
where H represents the optimal model parameters from training on the entire dataset, and 0 denotes parameters from training on the distilled dataset. Importantly, even when the model is trained on the distilled dataset B, we still evaluate its performance on the original Dtest for a fair measurement. When creating the dataset D, the pairs (á, 9) can be subsampled from the original D, as described in the coreset selection methods below (Sec. 3.2). We propose much more effective strategy in Sec. 3.3 to learn synthetic image-text pairs (f, 2), which can be more information--rchh.
Connection with Image-only Dataset Distillation. Traditionally, dataset distillation is tailored for classification tasks with discrete labols, cach of which possesscs a distinctive set of distilled data that enables efficient learning while preserving important information. We take this concept a step further to the multimodal scenario, where we distill information from both vision and language data. This involves creating synthetic data that capture critical relationships within and between these two modalities. As opposed to mercly classifying discrete labels, we examining more complex, interconnected dataset where the relation between modalities is crucial. Our method considers the image-text correlation and how they influence each other. It is worth noting that distillation would be impossible with single modality optimization (see Sec. 4.3).
3.2 Baselines: Coreset Selection
Since, lo the best. of our knowledge, there is no pre-existing work in the domain of vision-language dataset distillation, we begin by formulating a set of baselines to construct the smaller dataset D. These baselines are basod on coresct selection mothods, where subset of the training pairs (I,,)) is chosen, up given budget of M pairs, as to maximize the "informativeness" of the selected subset. We consider three such methods, adapted from prior work.
4

---

Published in Transactions on Machine Learning Research (08/2024)
Herding (Welling, 2009) Herding selects data points based on the distance between the coresct center and the original dataset center in the feature space. It greedily adds one sample each time into the coreset to minimize the distance between two centers. We use pre-trained encoders to extract features from the image-text pairs, concatenate the features, and calculate the dataset center in the feature space by averaging all feature vectors. We start with an emply coresel and for each iteration, add the image-lext pair that closest to the current. center of the coreset in Euclidean distance. We recalculate the coresct center after adding each data point.
K-center (Farahani & Hekmatfar, 2009; Sener & Savarese, 2017) Different from computing single center in Herding, K-center selects the training examples that maximally separated. Concretely, concatenate the features of the image and text pairs and start by randomly selecting a single data point. Then, for cach iteration, until K points are selected, we add a new image-text pair that is furthest in Euclidean distance from the nearest example. The drawback of this method is its high computational cost, especially with large datasets, as it involves heavy distance calculations between data points each iteration.
Forgetting (Toneva et al., 2018) The core idea is to identify reliable training data that the original model consistently learns well. During each training epoch, we check how accurately the models predict every image-text pair for specific task (i.e., image-text retrieval). A forgetting event is registered for an image-text pair when the model correctly predicts the data in one epoch but fails in the next. Throughout training, we continually track these forgetting events for each pair, to identify the ones with the fewest forgetting events.
3.3 Bi-trajectory Guided Vision-Language Co-Distillation
The coreset selection methods described above, while effective to some extent, demonstrate certain limitations as they only roly on selecting a subset of the training datasct D. This restriction leads to less effective results compared to our method, as ours provides the flexibility to generate an optimized distilled dataset D, and the learning process efficiently helps extract the most essential information embedded in D. Not only does this lead to decreased storage and computational requirements, but it also optimizes the performance of the model trained on this distilled dataset.
Here we describe our vision-language dataset distillation framework, building off of the idea of matching training trajectories (MTT) (Cazenavette et al., 2022) developed for distilling image classification datasets. The core idea of trajectory matching is that the dataset distillation can be achievod by implicitly matching the by-product, which is the parameter trajectory of the distilled dataset and the original full dataset, given direct information extraction is not feasible. We can compute a loss function on the cumulative discrepancy between the expert parameter trajectory 0* obtained from the model trained on the full dataset D and the parameters 9 obtained from the model on the distilled dataset D, and use that loss to guide the creation of a better D one that can match the parameters 0 more closely. The approach consists of two stages:
1. Obtaining the expert training trajectories (7*), with each trajectory 7" {0:) by training multiple models for T epochs on the full dataset D. For our multimodal setting, the models are trained using bidircctional contrastive loss, described below.
2. Training a set of student models on the current distilled dataset D using the same bidirectional contrastive loss, and then updating D based on the bi-trajectory matching loss of the student. models' parameter trajectories and the expert trajectories T".
Bidirectional Contrastive Loss. We train both expert and student VLMs using the bidirectional contrastive loss, following the formulation of (Radford et al., 2021) as it is effective for learning shared image-text representation. Concretely, given a batch of n image-text pairs {(x,y)), either from the real dataset D or from the synthetic distilled dataset D we jointly learn the encoders f(x; 0lmg) and g(v: Otee) such that the cosine similarity of all n correct image-text pairs is high and that of the (n² n) incorrect (f(rr00 (ag.llmymggggg00.. 1.0g9: pairs is low. We define cosine similarity between image and text y as: azyy [T((:))mg)) (= compute bidirectional contrastive losses composed of an image-to-text matching loss and a text-to-image matching loss, following the form of the InfoNCE loss (Oord et al., 2018):
We then
5

---

Published in Transactions on Machine Learning Research (08/2024)
Algorithm 1 Bi-Trajectory Co-Distillation
(Ting ) expert trajectories over epochs, trained on D. M: distilled dataset size. T max start epoch. R: # updates between starting and target expert parameters. R: # updates student network per distillation step. Ag: imitial learning rate.
Input:
1: Initialize distilled data D ~
2: Initialize trainable learning rate a == ag for training student models on D
for each distillation step do
4: Randomly sample an expert trajectory (Fimgs Tiz); corresponding to {10mmg Fint, ))
D Choose random start epoch, S T
6: > Initialize the student network with expert params 0ang. = 00 and Orot,s == ffsrt,
7: for ft iterations do Sample mini batch of distilled dataset D Use the contrastive loss of Eqn. 3 to update the parameters 0ang and Orat 10: end for
11: D Compute loss Etrajectory between 02 O++ and 0 0.tR using Equ. 4
12: & Update distilled datasct and learning rate a with respect to ltrajectory
13: end for
14: Output: Distilled data D and learning rate a.
Lcontrastive
1
Σ
log
exp aay En'ty Ay exp azy
log
exp Cuay
(3)
2n
En exp Cry' expas'
(a.1) batch
To imitate the effect of training data on parameter trajectories, we use the same objective function to guide the update of parameters 0img. Otzt during both expert training (stage 1) and distillation (stage 2). Notably, while hard negative mining is typically used in conjunction with contrastive loss, here we rely fully on the dataset distillation process itself without additional intervention. This process inherently considers hard negatives; distills samples that are hard negative samples for others, which are eventually effective samples for learning. Dataset distillation can potentially by-pass the traditional hard negalive mining complexities through the learning process.
Bi-Trajectory Matching Loss. Following the formulation of MTT (Cazenavette et al., 2022). we randomly sample M image-text pairs from D to initialize the distilled dataset D (more details can be found in the Sec 4.1). We sample an expert trajectory -" - {0 and random starting epoch to initialize Ú, 0z. We train the student model on the distilled dataset for R steps to obtain 0 We then update the distilled dataset based on bi-trajectory matching loss ltrajectory computed on the accumulated difference between student trajectory and expert trajectory:
110 10.gg.-A 0 0ing. + 12l12
1: 10e0r,048 - 0fnt, +
ltrajectory
(4)
110ing. A 0igg.+Rll
| 6" Rill,
We update the distilled dataset by back-propagating through multiple (ft) gradient descent updates to D, specifically in the image pixel space and text embedding space with respect to Equ. 4. We initialize the continuous sentence embeddings using a pretrained BERT model and update the distilled text in the continuous embedding space. For the distilled image optimization, we directly update the pixel values of the distilled images. The full details are in Algorithm 1.
Low-Rank Adaptation Matching. Further, for complex image encoders like Vision Transformers (VITs) (Dosovitskly et al., 2020), the bi-trajectory matching does not work well due to the high dimensionality of the embeddings and large number of parameters saved in the trajectorics compared to models like NFNet. To mitigate this issue, we propose Low-Rank Adaplation (LoRA) matching by matching the trajectories of only a small subset of the model's parameters through low-rank matrices. LoRA is effective for finetuning protrained models (Hu et al., 2021), and it introduces trainable low-rank matrices the weight matrices of specific layers of the pretrained models. LoRA matching optimizes the trajectories of low-rank adapters instead of the full parameters.

---

Published in Transactions on Machine Learning Research (08/2024)
Given the weight matrix W of certain layer in the ViT model, we introduce two low-rank matrices A E R and B e Rerd to each layer's weight matrix W, where d is the dimension of W and r & d represents the rank. The adaptation is performed by modifying W to W = W + AB",, where W denotes the adapted weight matrix. We only train and save the weights from A and B in the expert trajectories and match the student trajectories with Eqn. 4. For example, the ViT model we used vit_base _aasse___tchh____1_ 224,, which has 86 million parameters, but with LoRA and r = 4, the parameters are reduced to 18 million, cutting 78.71% of the parameters. This allows for efficient adaptation of the model with minimal additional parameters. With LoRA matching, can focus on smaller set of parameters and efficiently optimize the ltajectory during distillation while maintaining the capacity to save critical information.
4 Experiments
In this section, we first describe the cross-modal retrieval test-bed in Sec. 4.1. We use it to evaluate our vision-language dataset co-distillation performance. We then compare our method to bascline coresct selection approaches and provide the key quantitative, qualitative results, and cross-architecture generalization results in Sec. 4.2. We further conduct a set of ablation studies in Sec. 4.3.
4.1 Vision-Language Distillation Setup
Datasets and Tasks. We evaluate our method on standard vision-language datasets: Flickr30K (Plummer et al., 2015) and COCO (Lin et al., 2014), which are widely used for image-text retrieval tasks. We use them for expert training (stage 1) and distillation (stage 2). We adopt the Karpathy split (Karpathy & Fei-Fei, 2015) for Flickr30K (29k/1k/1k) and COCO (113/5k/5k) for main/validation /test respectively. Each image is paired with five captions. We retrieve the closest matches using cosine distance from one modality based on query from the other. We use R@K (for Ke 5, 10}) to compute the fraction of times the correct result appears among the top K items. To move from distilling image-only datasets to vision-language datasets, we validate in appendix Sec. B if our method has potential in the classic image classification setting.
Network Architectures. We primarily use pretrained and trainable NormalizerFroe ResNet (NFNet) (Brock et al., 2021b) as the image backbone following Flamingo (Alayrac et al., 2022) as well as Vision Transformer (ViT), for the text backbone we use pretrained and frozen BERT (Devlin et al., 2018). Ablation studies different backbones are in Appendix Sec. F.2. While both the encoders are pretrained, they are only pretrained unimodal data with exposure to the other modality. Each encoder is followed by a trainable linear projection layer with random initialization. Using trainable BERT adds additional complexity which is orthogonal to vision-language dataset distillation and is out of the scope of this work. Pretrained models serve as a common foundation and good starting point and see Appendix Sec. E.3 for details.
Implementation. For expert training, we train on a single RTX 3090 GPU for 10 epochs, where a single epoch takes 40 minutes of wall-clock time. Sampling from a set of trajectories encourages the distilled dataset to include diverse information and avoid overfitting to a particular step, thus we save 20 image-text bi-trajectories. For distillation, it takes 6 - 15 GPU hours depending on the settings (e.g. number of distilled pairs) with a 8-GPU A6000 node. We initialize a trainable learning rate a at 0.1 for the student model. We followed the data augmentation techniques in (Li et al., 2022), including resizing, cropping, flipping, and Augmen. We use SGD with momentum=0.55 the learning rate for updating a, distilled image pixels, and distilled text embeddings are le-02, 1000, and 1000, respectively.
Initialization. Following prior studies (Nguyen et al., 2020; Zhou et al., 2022), we initialize the distilled set with randomly selected real samples. We randomly select original dataset, with images at 224 224 resolution, and 768-dimensional sentence embeddings obtained via pretrained BERT. Our findings in Appendix Sec. E.1 show that initializing images from Gaussian distribution results in significantly lower performance. The complexity of images makes learning from random initialization challenging. In contrast, there is little difference in performance between using real and randomly initialized text emboddings. Surprisingly, despite the initial lack of semantic meaning between 'noisc' texts and real images, we found notable semantic similarity between distilled text and real images, suggesting potential applications of our method in Visual Question Answering.
{100, 200, 500, 1000} image-text pairs from the
7

---

Published in Transactions on Machine Learning Research (08/2024)
Table Baseline comparisons on Flickr30K (top) and COCO (bottom) with NFNet and BERT. We compare our distillation method to four corcsct selection methods: random sclection of training examples (R), Herding (H) (Welling, 2009), K-center (K) (Farahani & Hekmatfar, 2009) and Forgetting (F) (Toneva et al., 2018). We consider different selected sizes (100, 200, 500, and 1000) and report the image-to-text (TR) and text-to-image (IR) R@1 retrieval performance on Flickr30K and COCO datasets. We report our distillation results along with standard deviation, they are calculated from the performance of five differently initialized models after training on the same distilled dataset. Full details with R@5/10 are in Appendix Tab, In the very small budget regime, retrieval accuracy is much lower compared to the full data performance. As the budget grows, performance reaches full performance as shown in Tab. in the appendix.
TR
IR
Coreset Selection
Coreset Selection
Dataset
#pairs
R
H
K
F
Dist (ours)
R
H
K
F
Dist (ours)
100
1.3
1.1
0.6
1.2
9.9
0.3
1.0
0.7
0.7
0.7
4.7
0.2
200
2.1
2.3
2.2
1.5
10.2
i
0.8
1.1
1.5
1.5
1.2
4.6
0.9
Flickr30K
500
5.2
5.1
4.9
3.6
13.3
+ 0.6
2.4
3.0
3.5
1.8
6.6
+
0.3
1000
5.2
5.6
3.1
13.3
1.0
3.8
4.1
4.4
3.2
7.9
0.8
100
0.8
0.8
1.4
0.7
2.5
0.3
0.3
0.5
0.4
0.3
1.3
0.1
200
1.0
1.0
1.2
1.1
3.3
+ 0.2
0.6
0.9
0.7
0.6
1.7
0.1
COCO
500
1.9
1.9
2.5
2.1
5.0
+
0.4
1.1
1.7
1.1
0.8
2.5
- 0.5
1000
1.9
2.4
2.4
1.9
6.8
0.4
1.5
1.3
1.5
0.7
3.3
0.1
4.2 Key Results
Quantitative Results. As shown in Tab. 1 and Tab. 6 in Appendix Sec. A, we observe that although there is relatively little variation in performance across each of the coreset selection baselines which we compare to, dataset distillation outperforms the best alternative by anywhere between 138% (improving R@1 from 5.6 of K-center (Farahani & Hekmatfar, 2009) to 13.3 of our model) to 661% (improving R@1 from 1.3 of random selection 9.9 of our model). The relative improvement increases when fewer pairs are used for training.
Morcover, shown in Tab. 6, we note that with 1000 pairs, almost 30 times fewer examples than in the original dalaset, our data distillation approach reaches 43.7 R@10 for TR, relative to practical upper bound of 75.2, and 34.4 for IR R@10, relative to an upper bound of 69.7. We also observe that the performance among the bascline coresct selection methods varies only slightly, with no single method consistently outperforming the others across all pair sizes and retrieval metrics, often matching or underperforming random selection. This suggests coreset selection limitations multimodal settings. In comparison, our bi-trajectory co-distillation method is optimized for vision-language alignment settings and thus performs significantly better. Our results show the effectiveness of distilled data, achieving umparalleled efficiency with significantly fewer examples.
We compare the performance of the VIT model (vit_ (vit_base sasse__atchi___11 224) with and without the LoRA trajectory matching using BERT as the language encoder on the Flickr30K dataset in Tab. 2. Interestingly, vanilla ViT struggles in distillation, potentially due to attention mechanisms. For 100 pairs, the TR score jumps to 10.4 and the IR score 5.4. With 1000 pairs, the improvement even more noticeable: the TR score inercases to 15.8 and the IR to 8.1. Those results show that the LoRA trajectory matching is much more effective for distilling critical information. We report the practical upper/lower performance in Tab. 3.
Qualitative Results. Here we provide distilled image-text pairs visualizations out of 100 distilled pairs from Flickr30K after 2000 distillation steps in Fig. 3. We visualize the distilled text embeddings via their nearest neighbor sentences (cosine similarity) in the training set embedding space for more intuitive understanding. Additional visualizations are in Appendix Sec. G. The distilled images, compared to the original ones, add high-frequency components that help improve the generalization performance (Wang et al., 2020a). While the distilled texts maintain semantic components associated with the distilled images and capture the key attributes e.g. "couple", "kiss", "man", "surf", "huge wave", they also deviate from

---

Published in Transactions on Machine Learning Research (08/2024)
Table 2: Performance Comparisons of VIT with and without LoRA on Flickr30K (top) and COCO (bottom). We report the distilled performance on Flickr30K and COCO using ViT with and without LoRA, we BERT the language encoder.
Without LoRA
With LoRA
Dataset
/ Pairs
TR
IR.
TR
IR.
R05
R@10
R@1
R
R@10
R@1
R
R@10
R@1
R35
R@10
100
1.5
2.3
4.5
0.6
1.2
2.3
10.4
23.6
38.7
5.4
18.8
27.4
200
1.8
3.9
6.4
0.8
1.5
2.7
11.2
24.5
41.5
6.4
19.4
29.4
Flickr30K
4.3
2.1
13.4
43.4
7.6
21.1
32.7
1000
3.3
5.8
7.9
1.5
2.3
3.9
15.8
29.7
45.9
8.1
23.4
35.8
100
0.5
0.9
2.1
0.3
0.7
1.4
5.1
17.4
27.1
2.3
8.1
14.5
1.5
3.5
6.8
28.5
2.9
9.5
18.4
COCO
500
1.2
2.3
4.1
0.5
1.1
2.3
7.4
21.4
29.4
3.8
11.2
19.6
1000
1.5
2.7
4.5
0.7
1.5
2.9
9.9
22.5
32.8
4.7
12.7
20.2
newly married
a couple kissed front afa beautiful three- tiered cake with blue ribbon and
man a black wet man surfs over huge wave in
a woman a teboggan sledding with child that toboggan
skntebcarder jeans and t-shiet performing jump
man sitting in swing on carnival ride
buy toy room wearing hatman pajamas building
couple sharing a kiss front of convertible
suite is surfing luge wave in the the beautiful
something with
little boy in white shärt ts rock climbing
a buy with a blue helmet and gray clinshing
Two boys are watching another perform
kid in hoodie
two dugs run thruugh mud
the brown and white dog nips the yellow
two men are competiing for the ball in soccer
four football players look from the sideline while two formation acstmmaage
Figure 3: Before and After Distillation. (Teft) The image-text pairs before the distillation. (Right) The image-text pairs after 2000 distillation steps. Note that the texts visualized here are the nearest sentence decodings in the training set of the distilled text embeddings.
Table 3: Practical Limits Comparison. A side- byds comparison of the image-to-text (TR) and text-to-image (IR) retrieval results obtained from random ranking, full dataset training with NENet and BERT, and full dataset training with VIT (with LoRA) and BERT on Flickr30K and COCO.
Lower Bound: Random Ranking
Upper Bound: NENet
BERT
Upper Bound: VIT (LoRA) BERT
TTl.
IR
TR
IR
TR
IR
Dataset
R01
RM5
R010
R01
RM5
R010
R01
R35
R010
RÓ1
R05
R810
R81
R05
R810
R01
RMS
Ry110
Flickr30K
0.1
0.6
1.1
0.1
0.5
1.0
33.9
65.1
75.2
27.3
57.1
69.7
42.7
72.9
83.5
31.8
62.8
74.5
COCO
0.02
0.1
0.2
0.02
0.1
0.2
19.6
45.6
59.5
16.9
41.9
55.9
22.6
50.8
64.8
19.1
44.7
58.7
original sentence embeddings, they not in the original five captions paired with the images. The improved performance indicates that both high-frequency components and semantic ones are perceived by models and these significantly help in aligning vision-language modalities.
Arcss-Archieeturee Generalization. Following previous works (Cazenavette et al., 2022; Cui al., 2022; Zhao & Bilen, 2023), we evaluate the arcstarcchiecture generalization ability of our distilled data training unseen architectures. The experiments are conducted on Flickr30K with 100 distilled pairs. Distilling with NFNet model, report the cross-architecture generalization performance NF-ResNett0 (Brock al.,

---

Published in Transactions on Machine Learning Research (08/2024)
Table 4: Cross-architecture Generalization. Comparison of image-text retrieval performance when
models are trained directly vs. transferred the specified architecture. Evaluated on Flickr30K with 100 pairs.
TR
TRt
Distill
Evalmate
REa
R
Ra10
R&1
R555
R8:10
Distill Evaluate
RX1
R@5
R310
R@1
R
R:810
NeNet
9.9
28.3
39.1
4.7
15.7
21.6
VIT
10.4
23.6
38.7
5.4
18.8
27.4
NPNet
Ne55o
5.2
14.7
21.2
4.5
13.8
21.2
VIT
NF- NF-ResNec50
2.8
8.3
12.2
2,0
6.7
11.5
NF-RogNet
3.6
2.5
NF- NF-RogNer
3.7
5.9
9.2
VIT
3.1
13.2
2.3
13.3
NFNet
44
12.6
20.3
7.3
13.9
Table 5: Ablation with single modality distillation. We reported the image-text retrieval performance of text-only distillation (T), image-only distillation (I), and co-distillation (Ours) on Flickr30K. The results demonstrated the effectiveness of jointly distilling both image and text.
TR
R@1
R@5
RÓ10
R@1
R
R@10
# pains
T
T
Ouns
T
Ours
T
Ours
T
Ours
T
T
Ours
T
T
Ours
100
1.3
3.5
9.9
3.5
11.5
28.3
5.9
17.4
39.1
0.5
1.6
4.7
2.1
5.6
15.7
3.4
9.7
24.6
4.5
8.2
21.7
0.7
4.6
2.7
16.0
4.7
13.0
25.5
6.5
19.5
30.4
28.9
46.8
3.8
6.6
13.5
12.4
20.2
20.8
19.9
30.0
7.7
5.0
13.3
20.7
17.4
34.8
31.2
24.9
45.7
4.0
9.1
13.3
13.1
24.1
20.1
20.1
33.8
2021a), NF-RegNet (Xu et al., 2022), and VIT (Dosovitskly al., 2020) (LoRA), As shown in Tab. 4, our method transfers well across different models.
4.3 Ablation Studies
We conduct set of ablation studies to understand unimodal distillation co-distillation, distilled dataset. initialization (Sec. E.1), different encoder backbones (Sec. E.2), pretraining (Sec. E.3), synthetic steps (Sec. E.4), and their influence on distillation.
We compare co-distillation with unimodal distillation, where we keep one of the modalities fixed during distillation. Tab. shows the retricval performance of text-only distillation, image-only distillation, and co-distillation. Across all lasks and metrics, the co-distillation approach clearly outperforms the others. We observed that the performance of text-only distillation is worse than that of image-ouly distillation. This may not be surprising: text descriptions typically contain only salient but small portion of visual information. However, descriptions in the evaluated datasets typically contain no information that cannot be inferred from the images. By distilling images to text-relevant aspects, it can highlight essential image features. Thus, if we interpret each original image as having substantially more information than its original sentence, we would expect image-only distillation to perform better a smaller scale regime (removing spurious information) and text-only distillation perform better in a larger-scale regime (adding useful details).
In contrast, co-distillation allows the synthetic dataset to further optimize for compact representation and efficient storage, removing redundant information between examples in the smaller-scale contexts and adding information not present in the selected original images in larger-scale contexts. Our co-distillation method, combining text and image modalities during training, consistently outperforms single-modality distillation across different nunbers of training pairs and metrics. While the improvement from co-distillation is consistent, it is particularly substantial with fewer pairs: in the 100 and 200 pairs rows, co-distillation outperforms its unimodal alternatives by over 2x. In fact, co-distillation with 100 pairs consistently outperforms unimodal distillation with 1000 pairs. These results demonstrate the effectiveness of jointly distilling across modalities and highlight the complementary nature of multimodal data.
10

---

Published in Transactions on Machine Learning Research (08/2024)
5 Conclusion
In this work, we propose the first vision-language dataset distillation method. By co-distilling both vision and language modalities, we can progressively optimize and distill the most critical information from vision- language dataset. Our experiments show that co-distilling different modalities via bi-trajectory matching and using LoRA matching for complex model finetuning hold promisc. We hope that the insights we gathored can serve as a roadmap for future studies exploring more complex settings. Furthermore, we believe our work lays the groundwork for future research aimed at understanding the minimum information required for a vision-language model achieve comparable performance quickly, thereby building better understanding of the compositionality of compact visual-linguistic knowledge.
Limitations. We make note of two limitations of our approach. Firstly, dataset distillation is not exempt from the "No Free Lunch" theorem (Wolpert & Macready, 1997). As discussed in (Sachdeva & McAuley, 2023), we also observed that the effectiveness of the distilled data is highly influenced by learning algorithms and models used during distillation, which could potentially lead to poor transferability. Furthermore, many dataset distillation methods are computationally intensive, i.e. the bi-level optimization in meta-learning distillation approaches, which is another major challenge. In contrast, our trajectory matching approach is significantly less computationally demanding, yet we observed that the larger syuthetic steps often result in improved performance, and exploring closed-form solutions, i.c. implicit gradient-based methods (Lorraine et al., 2020) could be promising future directions to pursue.
Broader Impact Statement. Our exploration focuses on scientific understanding and practical applications of vision-language dataset distillation. While our work does not directly imply negative impacts, it may indirectly propagale existing biases in the original datasets. Therefore, it important to incorporate rigorous bits-mitigation measurements for dataset distillation. Discussion on these critical aspects should remain priority as further explore the potential of vision- langune dataset distillation.
Acknowledgements
This material is based upon work supported by the National Science Foundation under Grants No. 2107048 and No. 2112562. Any opinions, findings, conclusions, or recommendations expressed in this material are those of the author(s) and do not necossarily reflect the views of the National Science Foundation. We thank many people from Princeton Visual AI lab (Allison Chen, Jihoon Chung, Tyler Zhu, Ye Zhu, William Yang and Kaiqu Liang) and Princeton NLP group (Carlos E. Jimenez, John Yang), as well as Tiffany Ling, George Cazenavette and Ilia Sucholutsky for their helpful foedback.
References
Jean-Baptiste Alayrac, Jeff Donahuc, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lene, Arthur Mensch, Katherine Millican, Malcolm Reynolds, al. Flamingo: visual language model for few-shot learning. NearIPS, 2022.
Andrew Brock, Soham De, and Samnel I. Smith. Characterizing signal propagation to close the performance gap in unnormalized resnets. arXiv preprint arXiv:2101.06600 2021a.
Andy Brock, Soham De, Samuel Smith, and Karen Simonyan. High-performant largo-scale image recognition without normalization. In ICML, 2021b.
George Cazenavette, Tongzhou Wang, Antonio Torralba, Alexei A. Efros, and Jun- Zhu. Dataset distillation by matching training trajectorics. In CVPR, 2022.
Sanghyuk Chun, Scong Joon Oh, Rafael Sampaio De Rezende, Yannis Kalantidis, and Diane Larlus. Proba- bilistic embeddings for cross- modll retrieval. In CVPR, 2021.
Justin Cui, Ruochen Wang, Si Si, and Cho-Jui Hsich. Scaling up dataset distillation imagenet- 1k with constant memory. Xin preprint or 2rXi::2211.105:: 2022.
11

---

Published in Transactions on Machine Learning Research (08/2024)
Zhiwei Deng and Olga Russakovsky. Remember the past: Distilling datasets into addressable memories for neural networks. In NewrIPS, 2022.
Jacob Devlin, Ming-Wer Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint 1X0::1810004805, 2018.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition al scale. Xiw preprinl arxiv:2010. 1929,, 2020.
Jiawei Du, Yidi Jiang, Vincent T. F. Tan, Joey Tianyi Zhou, and Haizhou Li. Minimizing the accumulated trajectory error to improve dataset distillation. In CVPR, 2023.
Reza Zanjirani Farahani and Masoud Hekmatfar. Facility location: concepts, models, algorithms and case studies. Springer Science & Business Media, 2009.
Jiuxiang Gu, Jianfei Cai, Shafiq R Joty, Li Niu, and Gang Wang. Look, imagine and match: Improving textual-visual cross-modal retrieval with generative models. In CVPR, 2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106. 09685, 2021.
Yan Huang, Qi Wu, Chunfeng Song, and Liang Wang. Learning semantic concepts and order for image and sentence matching. In CVPR, 2018.
Zixuan Jiang, Jiaqi Gu, Mingjie Liu, and David Z Pan. Delving into effective gradient matching for dataset. condensation. arXiv preprint arXiv:2208.0033 2022.
Andrej Karpathy and Li Fei-Fei. Deep visual-semantic alignments for generating image descriptions. In CVPR, 2015.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document. recognition. Proceedings of the IEEE, 1998.
Sachyung Lee, Sanghyuk Chun, Sangwon Jung, Sangdoo Yun, and Sungroh Yoon. Dataset condensation with contrastive signals. In ICML, 2022.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip:
Bootstrapping language-image pro-training for unified vision-language understanding and generation. In ICML, 2022.
Paul Pu Liang, Amir Zadeh, and Louis-Philippe Morency. Foundations and recent trends in multimodal machine learning: Principles, challenges, and open questions. arXin preprint Sfw:2209.034900 2022.
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014.
Zhiqiu Lin, Samuel Yu, Zhiyi Kuang, Deepak Pathak, and Deva Ramanan. Multimodality helps unimodality: Cross-modal few-shot learning with multimodal models. arXiv preprint arXic:2301. 062667, 2023.
Haoyang Liu, Tiancheng Xing, Luwci Li, Vibhu Dalal, Jingrui He, and Haohan Wang. Datasct distillation via the wasserstein metric. arXiv preprinl ar Xiv:23311. 1.1851181,, 2023.
Jonathan Lorraine, Paul Vicol, and David Duvenaud. Optimizing millions of hyperparameters by implicit differentiation. In AISTATS, 2020.
12

---

Published in Transactions on Machine Learning Research (08/2024)
Timothy Nguyen, Zhourong Chen, and Jachoon Lee. Dataset meta-learning from kernel redge-regression. In ICL.R, 2020.
Timothy Nguyen, Roman Novak, Lechao Xiao, and Jaehoon Lee. Dataset distillation with infinitely wide convolutional networks. In NeurIPS, 2021.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv: mate:1887.0330 807.08748, 2018.
Rohan Pandey, Rulin Shao, Paul Pu Liang, Ruslan Salakhutdinov, and Louis-Philippe Morency. Cross-modal attention congruence regularization for vision-language relation alignment. ACL, 2022.
Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicodo, Julia Hockenmaier, and Svetlana Lazebník. Flickr30k entities: Collecting region-lo-phrase correspondences for richer image-to-sentence models. In ICCV, 2015.
Ashwini Pokle, Jinjin Tian, Yuchen Li, and Andrej Risteski. Contrasting the landscape of contrastive and non-contrastive learning. 2022.
Filip Radenovic, Abhimanyu Dubey, Abhishek Kadian, Todor Mihaylov, Simon Vandenhende, Yash Patel, Yi Wen, Vignesh Ramanathan, and Dhruv Mahajan. Filtering, distillation, and hard negatives for vision-language pre-training. In CVPR, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Adilya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, 2021.
Noveen Sachdeva and Julian McAuley. Data distillation: A survey. arXiv preprint vxXi:2201.04272 2023.
Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-set approach. arXiv preprint arXic:1708 00489, 2017.
Ilia Sucholutsky and Matthias Schonlau. Soft-label dataset distillation and text dataset distillation. IJCNN. IEEE, 2021.
Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, and Geoffrey J Gordon. An empirical study of example forgetting during deep neural network learning, arXiv preprint arXiv:1812.055
1.50511,,. 05159,111111111111011111010 2018.
Francisco Rivera Valverde, Juana Valeria Hurtado, and Abhinav Valada. There is more than meets the eye: Self-supervised multi-object detection and tracking with sound by distilling multimodal knowledge. In CVPR, 2021.
Paul Vicol, Jonathan P Lorraine, Fabian Pedrogosa, David Duvenaud, and Roger B Grosse. On implicit bias in overparameterized bilevel optimization. In ICML, 2022.
Haohan Wang, Xindi Wu, Zeyi Huang, and Eric P Xing. high-frequency component helps explain the generalization of convolutional neural networks. In Proceedings of the IFEE/CVF conference on computer vision and pattern recognition, pp. 8684 8694, 2020a.
Haoran Wang, Ying Zhang, Zhong Ji, Yanwei Pang, and Lin Ma. Consensus-awart visual-semantic embedding for image-text matching. In ECCV, 2020b.
Kai Wang, Bo Zhao, Xiangyu Peng, Zheng Zhu, Shuo Yang, Shuo Wang, Guan Huang, Hakan Bilen, Xinchao Wang, and Yang You. CAFE: Learning to condense dataset by aligning features. In CVPR, 2022.
Tongzhou Wang, Jun Yan Zhu, Antonio Torralba, and Alexei A Efros. Dataset distillation. arXiv preprint Xrv 10951 2018.
Max Welling. Herding dynamical weights to learn. In JCML, 2009.
13

---

Published in Transactions on Machine Learning Research (08/2024)
David H Wolpert and William G Macready. No free lunch theorems for optimization. IEEE transactions evolulionary compulation, 1997.
John Wright and Yi Ma. High-dimensiondd data analysis with low-dimensional models: Principles, computation, and applications. Cambridge University Press, 2022.
Hao Wu, Jiayuan Mao, Yufeng Zhang, Yuning Jiang, Lei Li, Weiwei Sun, and Wei- Ma. Unified visual-semantic embeddings: Bridging vision and language with structured meaning representations. CVPR, 2019.
Xindi Wu, KwunFung Lau, Francesco Ferroni, Aljoša Ošep, and Deva Ramanan. Pix2map: Cross-modal retrieval for inferring street maps from images. In CVPR, 2023.
Jing Xu, Yu Pan, Xinglin Pan, Steven Hoi, Zhang Yi, and Zenglin Xu. Regnet: image classification. TEEE Transaclions Newral Networks and Learning Systems, 2022.
self-regulated network for
Zihui Xue, Zhengqi Gao, Sucheng Ren, and Hang Zhao. The modality focusing hypothesis: Towards understanding crossmodal knowledge distillation. In The Eleventh International Conference on Learning Representations, 2023. URL https://opeurrviev..eett wttp://ww..eeettoo///ddddo/o0--0
Bo Zhao and Hakan Bilen. Dataset condensation with differentiable siamese augmentation. In JCML, 2021a.
Bo Zhao and Hakan Bilen. Dataset condensation with gradient matching. In ICLR, 2021b.
Bo Zhao and Hakan Bilen. Dataset condensation with distribution matching. In WACV, 2023.
Yongchao Zhou, Ehsan Nezhadarya, and Jimmy Ba. Dataset distillation using neural feature regression. In NewrIPS, 2022.
Ye Zhu, Yu Wu, Nicu Sebe, and Yan Yan. Vision+ x: A survey on multimodal learning in the light of data. Xiviv preprint ar Xiv:22200 02884, 2022.
14

---

Published in Transactions on Machine Learning Research (08/2024)
Appendix
In this Appendix, we first we provide the full baseline comparison (with R@1/5/10) on Flickr30K and COCO (Sec. A). Then we show the challenges of vision-language distillation (Sec. B) by transitioning the wrajectory-matching pipeline from image-only to image-text retrieval. We provide analysis on distilled images (Sec. D) and lossless distillation (Sec. C). We further extend the ablation study, analyzing components of our pipeline, i.e. distilled dataset initialization (Sec E.1), encoder backbones (Sec. E.2), pretraining (Sec. E.3) and synthetic steps (Sec. E.4). Lastly, show additional visualizations of the distilled samples, as well the ones under different backbones (Sec. G).
A Full Details for Distilled Performance
We provide full distillation results following Section 4.2, including image-to-text and text-to-image retrieval results R and R@10 with NFNet in Tab. 6.
Table 6: Detailed Baseline comparisons of NFNet on Flickr30K (top) and COCO (bottom).
Following Tab. 1, here report the full details of the distilled performance Flickr30K and COCO with NENet and BERT.
TR
IR
Coreset Selection
Dist (ours)
Coreset Selection
Dist
(ours)
Dataset
#pairs
Metrics
R
H
R
K
F
R@1
1.3
1.1
0.6
1.2
9.9
0.3
1.0
0.7
0.7
0.7
4.7 1 0.2
100
R@5
5.9
4.7
5.0
4.2
28.3
1 0.5
4.0
2.8
3.1
2.4
15.7 1 0.5
R@10
10.1
7.9
7.6
9.7
39.1
+ 0.7
6.5
5.3
6.1
5.6
24.6 + 1.0
R@1
2.1
2.3
2.2
1.5
10.2
+ 0.8
1.1
1.5
1.5
1.2
4.6 + 0.9
200
R@5
8.7
8.4
8.2
8.4
28.7
+ 1.0
4.8
5.5
5.4
3.1
16.0 + 1.6
R@10
13.2
14.4
41.9
9.2
8.4
25.5 2.6
Flickr30K
R@1
5.2
5.1
4.9
3.6
13.3
- 0.6
2.4
3.0
3.5
1.8
6.6 = 0.3
500
R
18.3
16.4
12.3
32.8 +
10.5
10.4
9.0
20.2 1.2
R@10
25.7
24.3
23.3
19.3
46.8 1 0.8
17.4
17.0
17.3
15.9
30.0 + 2.1
R@1
5.2
5.6
3.1
13.3 - 1.0
3.8
4.1
4.4
3.2
7.9 - 0.8
1000
R@5
15.6
14.6
16.1
14.9
34.8 - 1.9
11.8
12.1
12.8
9.5
24.1 - 1.6
R@10
21.4
20.4
20.8
18.9
45.9 + 2.5
19.9
20.0
20.4
18.7
33.8 + 2.0
R@1
0.8
0.8
1.4
0.7
2.5
0.3
0.3
0.5
0.4
0.3
1.3 1 0.1
100
R@5
3.0
2.1
3.7
2.6
10.0 1 0.5
1.3
1.4
1.4
1.5
5.4 1 0.3
R@10
5.0
4.9
5.5
4.8
15.7 1 0.4
2.7
3.5
2.5
2.5
9.5 1 0.5
R@1
1.0
1.0
1.2
1.1
3.3
0.2
0.6
0.9
0.7
0.6
1.7 + 0.1
200
R
4.0
3.6
3.8
3.5
11.9 + 0.6
2.3
2.4
2.1
2.8
6.5 + 0.4
R@10
7.2
19.4 1.2
4.9
0.8
COCO
R@1
1.9
1.9
2.5
2.1
5.0 = 0.4
1.1
1.7
1.1
0.8
2.5 - 0.5
500
R@
7.5
7.8
8.2
17.2 + 1.3
5.0
6.3
5.8
8.9 0.7
R@10
12.5
13.7
14.3
13.0
26.0 1 1.9
8.7
9.9
10.5
8.2
15.8 1 1.5
R@1
1.9
2.4
1.5
0.7
1000
R@5
7.6
9.0
9.0
7.7
21.9
+ 1.2
5.6
5.7
7.1
4.6
11.9
+ 0.5
R@10
12.7
14.0
14.1
13.0
31.0
1.5
9.6
10.1
10.9
8.0
22.1
0.9
15

---

Published in Transactions on Machine Learning Research (08/2024)
B CIFAR10 Classification vs Retrieval Distillation
Prior work has shown remarkable distillation results on CIFARIO (Krizhevsky et al., 2009) classification. To move from distilling image-only datasets lision-language datasets, we first check if our method has potential in simple settings. Concretely, convert CIFAR10 labels to captions that pair with their corresponding images. Under this formulation, the objective of classification is equivalent to that of image-to-text retrieval (TR): finding the best text given an image.
In Tab. 7, we compare CIFARIO distillation performance for dataset size of 1, 10, 50 images per class (IPC), under three different settings: classification, single-caption retrieval, and multi-capiion retrieval. For classification, we demonstrate results from MTT (Cazenavetle et al., 2022), where they distill an image-only dataset using expert trajectories trained on image-label pairs. In single-caption TR, we distill image-caption pairs using expert trajectories trained when each image is paired with single caption "This is [label}". In multi-caption TR, we distill image-caption pairs but the expert trajectories are trained when each image is paired with five captions that are generated with varies prompts from (Radford et al., 2021). For consistency, all image trajectories are obtained with the 3-layer ConvNet backbone as specified in (Cazenavette et al., 2022), and text trajectories are from lincar projection layers over protrained BERT (Devlin et al., 2018) embeddings. Although the performance of vision-language distillation trails behind that of image-only distillation, the gap closes larger IPCs. However, this gap highlights the challenge of the continuous label space in vision-language datasets. Moreover, the performance gap between single and multi-caption retrieval demonstrates the challenge of capturing the variability within human language descriptions.
Table 7: CIFAR10 Classification vs Retrieval. We provided ipc=1/10/50 classification performance vs. image-lo-text retrieval R@1, which both measure whether image has been matched with the correct class.
image- saage-to-text retrieval
IPC
Classification
Single Caption
Multi Caption
46.3
0.8
27.4 1.0
22.3 1.0
10
65.3
1 0.7
35.9 1 0.7
33.2 i 0.5
50
71.6
0.2
66.8 1.1
62.0 = 0.8
Full
84.8 + 0.1
79.6
+ 0.6
80.3 + 0.4
C Upper Bound Performance
We further increase the distilled size to be 10% of the original Flickr30K dataset size and we provide the comparisons for distillation performance with the upper bound results (Tab. 8). The distillation performance are closely approaching the upper bound results.
Table Matching Upper Bound Performance. With only 10% of the original size of Flickr30K, models trained on this distilled data show remarkable performance, closely approaching the upper bound results. For certain metrics, such as NFNet CLIP with Text Retrieval (TR) R@1, they reach high 98% of the upper bound.
Result
Vision
Language
Ratio
TR
IR
Type
Backbone
Backbone
R&1
R@5
R@10
R@1
R35
R@00
Distillation
NFNet
BERT
10%
32.1
60.0
73.2
21.1
53.9
66.5
Upper Bound
NeN
BERT
100%
33.9
65.1
75.2
27.3
57.2
69.7
Distillation
NeNet
CLIP
10%
60.0
86.3
91.4
47.4
78.2
86.5
Upper Bound
NeNet
CLIP
100%
61.2
87.5
92.8
49.8
79.8
88.3
16

---

Published in Transactions on Machine Learning Research (08/2024)
D Analysis on Distilled Images
We have found that increasing the learning rate and distillation time lead to more noticeable changes in the images within the distilled dataset (distilled images: Fig. 4, original images: Fig. 5). However, it is important note that higher learning rate or longer distillation time does not necessarily translate to improved performance of the distilled dataset, even the images appear deviate more drastically from the human perception perspective. Changes in image pixels alone may not reliably predict distillation performance. It is rather measurement. of the distillation strength. More distorted images suggest uneven pixel updates, while even updates yield results similar to the visualization we provided before in Fig. 3.
In line with previous studies, we initially expected more obvious changes in images would lead to better performance, but our findings suggest a different behavior of lision-language distillation with trajectory matching framework, reflecting how models capture vision-language interaction. From a human perception perspective, the distilled images appear be moving less compared to previous classification works, yet those small vectors are still meaningful and contain useful information, opposed to artifacts like noisy patterns. Our algorithm achieves clear and consistent improvement over random basclines indicated by the results. We hope this discussion can inspire more research on vision-language dataset distillation.
Figure 4: Distilled Images, iteration 7000, Ir image 5000.
Figure 5: Original Images, ileration 0.
17

---

Published in Transactions on Machine Learning Research (08/2024)
E
Additional Ablation Studies
In this section, we provide additional ablation studies. Unless specified, these distillation experiments are conducted on the Flickr30K dataset to distill 100 image-text pairs, and we use pretrained NFNet and BERT as backbones, with synthetic step set to 8 during distillation.
E.1 Distilled Dataset Initialization
In the main paper, we provided experiments with real sample initialization. Here we experiment and evaluate initializing with Gaussian noise. Our findings in Tab. show that initializing images from the Gaussian distribution results in significantly lower performance. which encodes a high degree and rich information of colors, shapes, textures and spatial relationships between objects, can make it difficult for models learn effectively from randomly initialized images. On the other hand, using real text sampled from the training bring significant difference. We assumee that the pretrained language models good generating transforming 'noise' text embedding into meaningful sentences during the learning process, partly due to the inherent structure and predictability of language. We provide visualizations of real images and 'noise' texts combination below in Fig. and Fig. and Tab. E.1. To our surprise, even though the initialized 'noise' texts are not semantically meaningful to the initialized real images, we discovered substantial degree of semantic similarity between the initialized real images and the learned distilled text. This suggests the probability of future application of our method in Visual Question Answering (VQA).
is worth noting that the complexity of images,
randomly initialized text embeddings does not
Table 9: Image-Text Pair Initialization. We compare the retrieval performance achieved with different combinations of image and text initialization strategies. The v denotes the use of real images texts directly sampled from the training set, otherwise indicates the use of randomly initialized image or text, following Gaussian distribution. We can see that if we initialize the image from scratch, the performance will be pretty low. On the contrary, the performance did not drop too much if we start with 'noise' texts and real images, which indicates the importance of image signal for the small distilled set.
Distillation
TR
IR
Real Image
Real Text
R31
R@5
R@10
R@1
R@5
R& 10
9.9
28.3
39.1
15.7
24.6
9
27.2
40.1
3.9
13.2
20.6
0.2
0.7
1.1
0.1
0.5
1
0.1
0.3
0.4
0.1
0.4
0.8
Figure 6: Initialized Images, iteration 0.
18

---

Published in Transactions on Machine Learning Research (08/2024)
Figure Distilled Images, iteration 1000.
'Noise' Texts, iteration = 0.
30 randomly initialized text
from Gaussian distribution, we use nearest neighbor to find their closest sentence in the training set in Flickr30k for visualization purposes.
this man is fit and sell tonad running anthusiant
music concert juat started the giant stadiun
nan in a beige shirt and tan slacks site in chair next to hospital patient wearing a blue gown who is eitting cross-logged
hospital
and wonan enployed nongolian barbecue stand at counter
wonan cupping water with hands over bathreon sink as child stande beside her snovflake sign, sits while another stands wearing badge and headphones very brave anou akiar doing a flip off a cliff guy,
nicely, painting mural dog charing brown cov and black cou
vall,
ladder sitting beside him
seens to looks
people in
room or working they all using laptop computers apple seens
pepper soda and vater bottle
olyspian performing on the ringe salking bahind distracted-looking
carrying bags and
three nen in capa sit at fireside near cabin, reading at night
dog the
white, black,
miner lesgue pitcher
man chair laughing the with shirt, resches into bucket extract the subetance inside emall brown leash vonan sitting at a park bench reading a book violin soloista
talking others,
handling
the stage during the archeatra's 's opening the theatar
one backpack on hands black dog aitting shile eating with neon yellow band around shoulders starding joans pockets shile the other one boy searing
several people
tarp ladies are facing
jumps guarterback is looking to set up pass from the end shile teamnate provides blocking veman in red coat takes a picture near narble colunns at tuilight
people anti-innigration signa
the outside of restaurant called el triuneo cheerlenders build pyranid near goal-line baby sears green frog big and nakes grotesque face yellou, auspanded roller coaster on yellow track mideay through loop
E.2 Encoder Backbone Selection
In this section, we evaluate the impact of different language/visin backbones on the distillation performance.
E.2.1 Language Backbones
Perhaps not surprisingly, CLIP (Radford et al., 2021) text encoder significantly outperforms BERT in all evaluation metrics, with a striking peak performance in TR R@10 at 92.8% for expert training. This exceptional performance can be mainly attributed to the fact that the pre-trained, off-the-shelff CLIP model is designed to learn a shared embedding space across multi-modalitics.. Although CLIP also shows a performance drop during distillation, it still retains relatively high performance recovery ratio. In Sec. G provide visualization of synthetic data distilled via NFNet and CL.P.
E.2.2 Vision Backbones
The vision encoders carry the main gradient flows for the distillation process. We experimented on several vision backbones, and found that the architecture choice strongly influences the distillation quality. Similar
19

---

Published in Transactions on Machine Learning Research (08/2024)
Distilled Texts, iteration = 1000.
Starting with randomly initialized text from Gauseian distribution. here the synthetic after distillation.
superhero leaping in plaza
guy in a blue shirt listens to nusic as he ekateboarde along the edge of a remp
tiger goods about nake patt
littla boy pulling a grean sagon searing aveatshirt and boota young girl with blond-hair and glasses sitting
three black young man are vorking in seni area vith pile of construction naterial and juge, one of then is digging
wonan buying cups fruit fron street vendor
six men in blue jumpsuits and man in an orange junpsuit valk by shipyard
young girl balances voman filleta fiah, as part of preparing a recipa that includea broscoli, calery, and agga young vearing
reclined nan's loga
of performance in front audiance
red contorticnist in strange checkered outfit vearing white mask man plays acoustic guitar, sale vearing brovn shirt holding nicrophone with expression of singing
kicking
another accompanies
accordion
young lady in colorful dress, holds white stuffed animal stands in the rain hold plaid unbrella with blue and polka-dot socks jumpa bad with dazaged black color car on the atreet ekateboarder junping
chite blankat.
skateboard
lege
venan with a guitar tings in front of building and grass vonan sitting on
together, facing vater
street with building Lined and people valking dovn the strest outaide and nighttine parasailer doing flip in nidair
crouded
lots peopla searing yallox, carrying red flags
nen in turbane leying down and exanining cloth
venan white apron prepares various neats large grill
niddle-aged man in trench coat sleeps on bus or train
people, sone standing and
sitting,
waiting on platform train
three playing druns, bass, and plano
dirt-blande girl in whita top with a kay necklace holds a hag, standing in front sidevalk of street cattle-drawn vagons traveling down paved and loaded with sticks
Table 10: Ablation Analysis Language Backbones. We provide expert training and distillation performance evaluation for both pretrained BERT and CLIP models. CLIP text encoder demonstrates a strong capacity for high-recall retrieval.
Expert
Distillation
Language Model
TR R@55
IR
TR
IR
R@1
R@00
R@1
R05
R@10
R@1
R@5
R@10
R@1
R@5
R@10
BERT CLIP
33.9
65.1
75.2
27.3
57.2
69.7
9.9
28.3
39.1
4.7
15.7
21.6
61.2
87.5
92.8
49.8
79.8
88.3
31.4
58.8
72.0
17.1
41.9
56.2
dataset distillation by gradient matching (Zhao Bilen, 2021b), batch normalization has an impact on the gradinn/pparameter paarameter matching framework. This is mainly because batch normalization incorporates a non-parametric component that can only be accumulated with batches and can not be trained.
Table 11: Ablation Analysis on Vision Backbones. We provide an extensive evaluation of several pretrained vision backbones including NFNet 10 (Brock et al., 2021b), NF ResNet50 (Brock al., 2021a), NF Re_Nett (Xu et al., 2022), and ResNet50 (He et al., 2016). This underscores the inffuence of architecture and highlights the potential negative impact of batch normalization for distillation.
Expert
Distillation
Vision Model
TR R
IR
TR.
IR.
R@1
R@10
R@I
R:05
R@10
R@1
R@5
R@:10
R@1
R:5
R&10
VIT (LoRA)
40.7
69.8
80.1
28.8
59.3
73.4
10.4
23.6
38.7
5.4
18.8
27.4
NFNet-l0
33.9
65.1
75.2
27.3
57.2
69.7
9.9
28.3
39.1
4.7
15.7
24.6
NF Re_ee5e055
28.9
56.6
71
22.8
50.1
63.4
6.5
18.2
28.1
3.5
11.6
18.7
NF RegNet
26.9
57.2
70.2
21.1
50.1
62.9
7.8
21.9
33.3
3.3
12.7
20.5
ResNet50
43.5
59.5
13.4
36.6
49.9
3.8
1.6
3.6
20

---

Published in Transactions on Machine Learning Research (08/2024)
E.3 Pretrained Non-pret pretrained
Tab. 12 demonstrates the pretraining influence of the backbone encoders. Optimal performance is observed when both language and vision backbones pretrained. This emphasizes the importance of pretraining before the expert training stage for large models and datasets.
Pretraining Impact. Expert performance comparison for different, pretraining configurations of vision and language backbones. The checkmark (v) indicates the model was pretrained.
Table 12:
Expert
Language Backbone
Vision
TR R@5
IR.
Backbone
R@1
R@00
R@1
R
R@10
33.9
65.1
75.2
27.3
57.2
69.7
4.4
14.1
20.7
3.5
11.4
18.8
0.5
1.1
1.8
0.3
0.7
1.4
E.4 Synthetic Steps
The synthetic step size plays an important role in optimizing the dataset distillation performance, as shown in Tab. 13. Using larger synthetic steps tends achieve better distillation performance.
Table 13: Synthetic Steps Impact. Larger syuthetic steps greatly improve performance. For 100 pairs with a synthetic step of 1, the performance is even below random selection. Setting the synthetic steps to a low value typically takes longer to optimize the distilled set and challenging with very small sets (e.g., # Pairs=100).
Distillation
#Pairs
#Sy Steps
TR.
IR
R@1
R
R@10
R&1
R@10
1
0.5
2.1
4.4
0.3
1.5
2.8
2
7.1
23.4
32.9
3.0
10.2
16.4
100
8.2
24.9
35.2
3.5
12.2
20.7
8
9.9
28.3
39.1
4.7
15.7
24.6
1
3.2
9.3
14.1
1.6
5.2
8.8
2
6.5
19.2
1.6
10.0
200
2.2
8
10.2
28.7
41.9
4.6
16.0
25.5
1
6.6
18.1
25.5
2.1
10.1
16.3
2
21.7
31.3
3.8
14.9
23.2
500
4
23.6
34.9
4.4
15.2
23.7
8
13.3
32.8
46.8
6.6
20.2
30.0
1
7.3
20.6
29.7
3.9
13.2
20.7
2
8.8
26.8
36.6
5.7
17.4
26.4
1000
4
10.4
29.1
37.9
6.6
19.5
29.5
8
13.3
34.8
45.7
9.1
24.1
33.8
F Beyond Trajectory Matching
In this section, we further provide experiment results of distribution matching (Zhao & Bilen, 2023) baseline adapted to the vision-language setting. To use distribution matching for vision-language datasct distillation, concretely, we minimize the maximum mean discrepancy (mmd) between two distributions by sampling NFNet with different initialization and pretrained BERT, Similar to the distribution matching setting for image classification, we update the distilled data via. mmd for vision and language modalities to match the original data distribution in family of embedding spaces. We provide the comparison of Our method w/ DM (distribution matching) and Our method w/ TM (trajectory matching) Flickr30K (R@1) in Tab. 14.
21

---

Published in Transactions on Machine Learning Research (08/2024)
TR
TR
# pairs
Ours w/ DM
Ours w/ TM
Ours w/ DM
Ours w/
TM
100
3.2 = 1.8
9.9 + 0.3
1.4 + 0.7
4.7 + 0.2
200
3.3 = 1.3
10.2
+ 0.8
1.4 + 0.4
4.6 + 0.9
500
5.8 = 1.5
13.3
+ 0.6
4.1 + 0.9
6.6 + 0.3
1000
6.1 = 2.7
13.3
1.0
4.9 + 1.8
7.9 0.8
Table 14: Comparison of our method using Distribution Matching (Ours w/ DM) and Trajectory Matching (Ours w/ TM) on the Flickr30K dataset. The table shows retrieval performance (R@1) across different numbers of pairs. The results indicate that our method with trajectory matching consistently outperforms distribution matching, particularly in scenarios with smaller data budgets.
Looking forward, we hope our method could serve as roadmap for future studics exploring more complex settings with slate-of-the-art (SOTA) methods. New SOTA dataset distillation methods can adopt low-rank adaptation matching to scale efficiently with large and complex models, and incorporate bi-trajectory co-distillation to handle textual data more effectively. By doing so, these methods can extend their applicability to previously infeasible models for distillation, such as those involving ViTs, thus improving the scalability and efficiency of the distillation process. New approaches that distill from both text and image data can consider using methods similar to bi-trajectory matching with contrastive loss to learn the interactions and redundancies across multinodalities.
22

---

Published in Transactions on Machine Learning Research (08/2024)
G Additional Visualizations
Here we include number of visualizations of the data we distilled from the multimodal dataset (both Flickr30K Tab. G and Fig. 8, and COCO Tab. G and Fig. 10, 11) for a more intuitive understanding of the distilled set. We provide 50 distilled image-text paired examples including their visualization before the distillation process. Unless otherwise stated, these experiments are conducted using 100 distilled pairs, with pretrained NFNet (Brock al., 2021b) and BERT (Devlin al., 2018) as backboncs and the synthetic stop is set to 8 during distillation We provide visualization of distilled data using NFNet and CLIP in Tab. G and Fig. 12, 13 in the end.
Figure 8: Flickr30K Initialized Images, iteration = 0.
Figure 9: Flickr30K Distilled Images, ileration 2000.
23

---

Published in Transactions on Machine Learning Research (08/2024)
Flickr30k Initinlized Texts, iteration
construction vorker stares down flight of stairs being built nan in suft valking acrosa city atrest
child hita bassball into nat
larga croud of people
glass many political advertisenents an eld nan with white hair in red hat
old nan, behind
road, playing tuo man dreased before big avent dog is running through field with its tongue hanging out gray shirt netorcycle is parked along side a nountain road while another goes down the road through hoop
young girl,
older
long-sleave shirt under it holding up small wooden cabinet
nan orange shirt tuo people sit on the end of dock
man perforns
flip shile preparing for
performance compatition
little boy plays with
bunch young children are riding on the back trolley while being carried by a black and white horee ledge
clinbing up
young nan sita on beach in dovatovn setting
ground covered in baskethall players practicing for their game
black dog
orange ball, walking
man juat undersear jumping on surrounded by
paopla
vearing lots of plaid riding bike through the streets men are trying to cut down trees
black getting baircut
this was big new yeare event the people were sing and dancing sll night shirt dribbles basketball series of nen taking break from riding their motarcycles
boy yearing
brosn-haired nan patterned shirt
purple tie ia ainging into nicrophone holds enall cancorder and films person in front bin
curly dark-haired yellow shirt holds rether large snail in her hande next to her check young dancing the streat
young girl
girl standing in a shallow creek, vearing stilts park talking to
people starding
group young nen in colorful uniforns playing with shite ball tuo guys are on the side of the street playing guitar and druns
nine applying his nakeup
child decorates a shoe with colorful sticks
up stage with nicrophones their hands
tuo basketball players opposing teans, one in shite, the other in red, are mid-gane, running down the court, white-mnifor player ball-in-hand
one young lady with black hair in ponytail yearing black bracelet and white shirt, taking pictures with black canera that has shoulder strap laying boy skateboard is vall near the and
sculptor is carving a picture of knight into a brick sall
in blua an eld nan vearing glasses is holding stick
is walking
sidezalk
playing with doll-like toy
an old nan vearing hooded aveatahirt is crouched over fish that bas been open
young girl in black drees is holding red flag and covering a happy expression
brown dog vith baseball its nouth
man in shite and rad tackling nan in green shirt for the ball
t-abirt holding abovel
24

---

Published in Transactions on Machine Learning Research (08/2024)
Flickrā0k Distilled Texts, iteration 2000.
construction workers repair walls subway ship barbor night
city skyline behind baseball pitcher throving pitch
group peopla sitting around
nesting
nan points sonething ae is talking vonan vearing white panta, they stand Tront of atore man in red eveater with backvards hat crossing the the bridal party poses with the bride and grocn, all searing black except for the bride
vearing
parked and walking down street
old lady, searing
hat, standing the sidevalk park
nan grilling hotdogs and esusages
notocroaa
kicks up dirt as being ridden around wearing purple shirt and black pants, clap while man dressed in black
circuit
mine wonen in blue and purple dresses and dances
sitting down boat while playing soccar, man in yallow starta to fall, shile a man in white trips over him, stapping on his ankle in the proceas
young and
anchor and watching the
little boy is valking on fallen tree in the woods Jockey and horse in the niddle other jockeys horses during race, in the middle jumping over hurdle extrene snowboarding one man, in blue jacket, is sitting in the rain under green unbrolla the brown and shita boy takes a bath with diving mask and anorkel valking
side down nountain
running to catch somathing
angry looking person on bax bike, leaping ento bench
sidevalk
tuo people and dog are in the enov couch with his hand his chest
young shirtlers boy sleeping person spins a sparkler around at night and sparks fly through the air and across the ground vonan, searing sunglasses,
athletic top, and running shorta compates narathon ballet dancer vearing a blue tutu doing the splite, nid-leap milee policenan taping off an area by group of firemen
woman
her cellphone
nan with beer and another nan facing each other, talking
and two children reading outside man on notorcycle riding in dry field searing helmat and backpack
bench
in white shirt black
sidevalx,
located bahind under construction
in front of a tvo garage house
trap, sone young adults are playing saxophones and clarinsts outdoors
man hitting golf
green grase around
young man eitting on a rock on the thore of body of water looking contemplative
young bay, covered in nud, plays the beach shirlay nanson poses in front of a nicrophone on stage shile holding a large blue, red, and white flag bohind her
hockey players pleying group of friende, 3 boys and girle, juzp in the air holding hands for a photo boy skateboards
defense
a jump over ginger baby playing with train setup mada out of counterfeit lago
skateboard
group choreographed rollerskatera dancing
little bland girl her blond little girl orapped up in a pink care bears blanket:
sticking
tongue while holding red ballaon
oriental nan, vearing one boys sits an giant nortar gun as another boy walks tovard his puppy toes
apron cooking
dog trying bite white
yollou, green
soman standing on the shore of beach
nales, one red shirt and one in yellow sbirt, cleaning a room
25

---

Published in Transactions on Machine Learning Research (08/2024)
Figure 10: COCO Init ialized Images, ileration = 0.
Figure 11: COCO Distilled Images, iteration = =2000.
26

---

Published in Transactions on Machine Learning Research (08/2024)
COCO Initialized Texts, iteration = 0.
photo taken night young nan playing frisbee
bathroon toilet surroundad with silver bandrails
pink floser is sticking out of green vase
fridge small group of theep en the coast
poses
photo shoving large
tover on building
the people are saiting be picked up
band holding black television remote control
suinging tannis rackat
nan holda anall animal to bis face
air yellow and red street signs warning larger vehicles in large city
people
man riding
notorcycle
the train is traveling doun the railroad tracks
building with large clock on
of snall bathroon vith toilet, sink and window buildings
standing
shop line of notercycles parked on street store looking track
street
young girla
snall toy model train couple of giraffes are outside the vild
pouring coffee
cups
people riding en top of elephante across river swings at bazeball
an old picture of
holding tennis racqust
there are people vatching three nen on horseback table with laptope
people sitting
there are nany things laying on the ground
batter avings hard low ball the catchar reaches his five sheep stand around large dirt field sink, glazed donut sitting on wooden table
bathroon
donut shop
buge chinees aircraft sitting at airport with people unloading bunch notorcycles parked together outeide cutting board with scissors and dried food on it decorated bedrean pictured this inage
nan standing on skis at the of hill under high tension vires
long dog
on plate
pan of dough is going into the dirty toaster oven road sign with both english and arabic directions kite being flown the beach shile people watch a yellow fire hydrant is on the corner of an old sidevalk
head herse struggles to draz loaded cart through piles of snow voman sitting betwean two large taddy
bicycle
aubuay
kid skateboarding while other kids atand and satch there is a dog in the back of the truck
largo assortment fruits lie display narket he's taking a picture of his friands at the restaurant
27

---

Published in Transactions on Machine Learning Research (08/2024)
COCO Distilled Texts, Iteration 2000.
dog flying mid-air running bath tub with metal shover head, vanity nirror, and small utilities compartment pink flovers large green
frishes
chita
this apertnent has kitchen with refrigerator, stove, dishuasher, and cabineta ski resort with many people gethered around the outside lodge grandfather clock hanging sall the kitchan is brightly 1it from the xindow nintendo venan ewinging tennis racket, while standing on tennis court jet plane taking off into the air
grandfather clock
playing
controller
sign varning people stop at red lights man swinging baseball bat with ball in air and crowd watching
hitching trailer with
sports equipnent sports utility vehicle
four trains Lined at the train station
large
top
people standing at bus stop about to board bus
snall bathroon with sink nirror
sdmiring notorcycle in parking lot,
large building
notorcyclist in a red and white suit riding red and white notorcycle little boy playing loconotive train resting on some tracks next to a building plant
tannis court
giraffes graze voman looking at camera while lying in bed
herd adult elephants vith baby elephant waling through forest baseball batter in vide
vaiting for pitched ball
the cars were parked along the street near the traffic light travalers gatting cluttered deak with laptop opened to flickr lady eitting the xood beach
baseball player preparing to sving baseball bat field kettle and microsave
tuo lanbe eating hay from ground brow cabinets black the raception ifs full of profossional people
baseball batter ready strike arriving ball unpire waiting catch nissea
there lot of notorcyclee park in a line with shite car, a red car and van park not far from the notorcycles while there is riding on
clean roon and pair of scissors
the bedroon has wood closet and bookcase near the bed line of skiers heading towards cabin in the plate topped with oniens rings next to hanburger and hot dog pizza
personal
topped with vegetables
sign letting people knov about the castle rising castle standing there is obite fire dyrant the corner streat bus
girl
man skateboarding on atreet in front vintago black and white photograph of too baseball players raggedy ann doll sitting in a chair with pooh bear blonde boy doing juzp shile riding dog driving car down street
board
bananas, apples oranges light with the green light 1it boxl,
28

---

Published in Transactions on Machine Learning Research (08/2024)
Figure 12: CLIP, Flickr30K Initialized Images, iteration = 0.
Figure 13: CLIP, Flickr30K Distilled Images, itcration = 2000.
29

---

Published in Transactions on Machine Learning Research (08/2024)
CLIP, Flickr30k Initialized Texts, iteration =
holding child atanding in front tank
venan and nan with child in the center of circle of children broon dog eyaing a fly
four Xids next to blue house are valking down street
tourists look at nerchandise on street vendors dieplay in england riffling through carde and nape people vearing blue clothing four workers in field harvesting
naking hand gestures
another
dog approacha vezan holding baby and man in glasses holding snall boy emile at the canera for their fanily photo four men are harvesting crop on farn
barren, snovy field
musicians stage frent of nicrophone stand
nan in a suit is walking under a netal bridge
girl gotting her hair dyed
child in in a ball pit while three adulta vatch har kayak boy in a blue shirt holds a toy helnst in his hands while standing path in park
is aitting
diving
boy doing flip in the sir at the beach dressed up man valking underneath bridge glances at the camera heads getting the ball voman la at an art studio, painting a mural from her art suppliea vetsuit stands
saccer players
about
boy vearing
crovded chef prepares table with food behind it
tvo blond girls in white dreeses, one much amaller than the other, stand on the bank large body of water black nountain bike rides through nan in a hat stands with decorated horses around hin asian narching band with uniforned nenbars in beiga, yallou, and red play in the streat
filled with other bikers
two nen with angry faces drink out of white cupa green person siting against vall with dog one fighter delivers hard blow the face of another fighter young group of children sitting teacher standa in front of a projector and student at the front of the class has her hand raised large body vater causing
young boy plays
against the vall
dog is running
aplash
football playera are stretching together
playing video gane instead of working group of people at dining table playing card game
young voman celebrating her greduation
dog looks
nan vearing a flamel ahirt and black pants in vorking on a now reed roof on tap of house nevly narried couple having their
kiss
vezan playe aide-and-go-ee go-eekk with check scarf as she site with man in dark colored jacket
blond-hair sniling acoustic guitar
veman
singing into microphone while
striped shirt, back, plays
group astan teenagers kearing red t-shirts ait on steps
gets lots of air tine ha sakehoards
black and white dog is running through the field to catch sonething in ita nouth lega
small dog caught young man tends chicken vings on barbecue while young vonan in a pink shirt watches
person
black dog jumping over
with supports
group of uoman vearing shirts that habe is standing table with food on it
30

---

Published in Transactions on Machine Learning Research (08/2024)
CLIP, Flickr30k Distilled Texts, Iteration = 2000.
young asian girl patting an aninal some sort the aninal 1a laying dovn enjoying group in othnic dress dancing
brown and tan dog, mouth open with tongue hanging out, running in the grass
older
vearing colorful print
directly
woman in red thirt shopping in outdoor market
veman
nan holding bag valking dovn long staircasa
nan in black valking doun a street
holds baby
blue jumper
snall car in an open field musiciens
playing instrumente including guitar nan is standing inside subvay train with his mouth vide open
group
shaving somcone's
group people are shopping in what looka to be christmas store filled with colorful toya
nan in high rubber boots and plaid shirt is puehing breon over the noesy blacktop the third standing, what night be park during sunny day girl jumping up in the air with her hands above her bead people, one of group of people valking through an alley along cobblestons street, between two buildings grean jarsay nan painting over graffiti
other,
sitting
costune, pose wearing funny glasses
soccer playar
and yallow
tuo dogs running near river while one dogs in svinning in it surprised venan sits in chair on the beach, backdropped by the ocean
black
deli counter
young man popping
bicycle black coe while riding horse
riding
country
person attempts to
nen dressed red shite playing musical inatrunents
drink beer out tall drinking glasses
little boy esting en sidewalk
vall painted with mural nan vearing hat has his eyes closed, as another man in a red shirt is licking bis face poses young bay in aports uniforn standa in front of group of chi Idren
is standing inside doorvay that
family
together
little boy plays outdoors vater spurting up from an inground fountain red
do acrobatics with ladder
young caucasian man sits at a desk using a laptop computer
group people sharing neal group of people protest with one holding dog
large table
restaurant
cardboard sign
their dogs nan with plaid shirt is working on sone wood in his workshop
group people
the fellow in the black suit valking veman singer holding a ni nicrophone singing at concert
formal occasion has a salnon rose in his lapel field of
and
young astan female aitting
pase the nan is standing by creek in blue flannel shorts whita dog leaps nan is feeding two exotic birds
vallas
being photographed
an ssian chef is the foreground, working over a steeming grill while younger man is bebind bis young ckateboarding down the railing of stairs
female chaf exanines a piece of bread while shoving it to the canera
31


Reviews:
[{"review_id": "rQp0HEpCMp", "reviewer": "Reviewer_aeRf", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "* The paper does a good job of introducing the new setting that is considered, and the method is described in reasonably good detail. One thing that is unclear is how one goes from the distilled token embeddings to the discrete tokens. This should be made more clear\n* Experimental evaluation looks at several different aspects, including comparison with prior methods, an ablation, qualitative results, and generalisation of the distilled datasets to new architectures. In that sense, it is quite a good evaluation. Moreover, the method is shown to outperform core-set methods by a substantial margin.\n* I like that the submission also considers the case of dataset distillation in the case of model adaptation---i.e., when training Low Rank Adaptations of the model.\n* However, there are some things missing from the evaluation: there should be a comparison with other dataset distillation methods. The authors claim that this is not possible because text is discrete and therefore cannot be optimised via gradient-based optimisation. However, the method used in this paper optimises token embeddings, not discrete tokens, so it should be possible to compare with other methods by using the same token embedding to discrete token transform employed by the current method.\n* The presentation of the evaluation could be improved. It is currently quite hard to understand how meaningful the results are, and the tables in the main paper do not provide the full picture.\n* The main issue with the submission is overclaiming of novelty; the method is from a previous paper, and the only new thing is the evaluation in the vision-language setting. This is fine, but the way the paper is written is not in keeping with the contributions being made.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The submission gives an overview of a method for distilling a dataset for vision-language pretraining. The method is evaluated on a number of vision-language tasks, and the results are compared with several core-set methods. The method is shown to outperform the core-set methods by a substantial margin.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "* The authors should make it clear how they go from the distilled token embeddings to the discrete tokens.\n* The authors should compare their method with other dataset distillation methods.\n* The authors should improve the presentation of the evaluation, so that it is easier to understand the results. In particular, I would recommend that Table 1 is removed and replaced by plots summarising the results in Table 6.\n* Please add performance of the models trained on the whole dataset to each of the tables, so that it is easier to understand the practical significance results.\n* The writing needs some work to tone down the overclaiming and to generally better reflect the contributions actually being made in this paper.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "oYOOR6HEs8", "reviewer": "Reviewer_zbg5", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. The research problem of vision-language dataset distillation is interesting and novel. \n2. The paper is well-written and well-motivated. The proposed method is reasonable and easy to re-produce.\n3. Experimental results prove that the new method works much better than the baselines.\n\nWeaknesses：\n\n1. The performance of the model trained on distilled dataset is poor, which prevents the utility. \n2. As shown in Figure 3, the distilled image/text pairs are somewhat unexplainable. For example, the unexist concepts in texts and the artifacts in images.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper extends the research on classification dataset distillation to vision-language dataset distillation. The authors adapts the popular trajectory matching method for vision-language model optimization by introducing the bi-trajectory matching. Experiments show that the proposed method works much better than the coreset selection baselines. Visualization of the learned synthetic data is given for analysis.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "No.", "broader_impact_concerns": "No.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "vlEECqqA73", "reviewer": "Reviewer_CQga", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strength\n- The paper is well written and easy to follow\n- The method is quite novel such as adopting LoRA as methods like MTT cannot easily scale to large number of parameters due to heavy memory usage.\n- The experiment results prove that the proposed method is very effective in distilling image together with text.\n- This is the first work to tackle the problem of dataset distillation for vision-language models to the best of my knowledge.\n\n## Weakness\n- Since the method is based off MTT, can the proposed method easily extend to using other DD method as the backbone?\n- Have the authors explored other methods for distilling text other than the input embedding space?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces a novel approach for dataset distillation specifically tailored for vision-language models, a domain that has not been previously explored. The main contributions are as follows:\n\n- Building upon the state-of-the-art trajectory matching method, the authors adapt this technique to vision-language datasets to co-distill both images and text.\n\n- Unlike previous dataset distillation efforts that have concentrated on image classification in a continuous space with fewer parameters, the authors propose using Low-Rank Adaptation (LoRA) to address the challenge of managing the large number of parameters in the new domain.\n\n- The paper demonstrates the effectiveness of the proposed method by comparing it to three core-set selection methods.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The paper could be improved by discussing how the proposed method can be adapted by new state-of-the-art methods that go beyond trajectory matching. [not critical for my recommendation]", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers found various positive elements in the article, such as the presentation (CQga, zbg5), the addressed problem (aeRf, CQga, zbg5), the experimental results (aeRf, CQga, zbg5), and the idea behind the proposed approach (CQga, zbg5). 

At the same time, they raised some concerns about the low overall performance (zbg5), lack of explainability (zbg5), the generality of the framework (CQga), novelty claims on the method (aeRf), and lack of clarity on some aspects of the evaluation and embeddings discretization (aeRf).

The authors' reply fully addressed the concerns of CQga and zbg5, who recommended full acceptance. While aeRf also leans to accept the manuscript, they raise some points on the presentation (i.e, tables vs plots) and on overclaiming novelty on the method (i.e., trajectory matching). The AE finds the latter concerns minor as i) the presentation quality of the results might be subjective and the AE finds no critical issues in the current version; ii) the article stresses in multiple points that the proposed method is based on trajectory matching, providing proper references and descriptions (i.e., Section 3.3).

Considering the revised article and the reviewers' feedback, the AE recommends the acceptance of the work.
Decision: Accept (poster)