CovMatch: Cross-Covariance Guided Multimodal Dataset Distillation with Trainable Text Encoder
Authors: Yongmin Lee
Hye Won Chung
Venue: NeurIPS 2025 poster ()
Keywords: Dataset distillation
Dataset condensation
Multimodal dataset
Image-text contrastive learning
Primary area: deep_learning
TL;DR: We propose CovMatch, a scalable dataset distillation method for image-text contrastive learning that aligns cross-covariance and feature distributions between real and synthetic data.

Abstract: Multimodal dataset distillation aims to synthesize a small set of image-text pairs that enables efficient training of large-scale vision-language models. While dataset distillation has shown promise in unimodal tasks, extending it to multimodal contrastive learning presents key challenges: learning cross-modal alignment and managing the high computational cost of large encoders. Prior approaches address scalability by freezing the text encoder and update only the image encoder and text projection layer.  However, we find this severely limits semantic alignment and becomes a bottleneck for performance scaling.
We propose CovMatch, a scalable dataset distillation framework that aligns the cross-covariance of real and synthetic features while regularizing feature distributions within each modality. Unlike prior approaches, CovMatch enables joint optimization of both encoders, leading to stronger cross-modal alignment and improved performance. Evaluated on Flickr30K and COCO, CovMatch outperforms state-of-the-art multimodal distillation methods and achieves up to 6.8\% absolute gains in retrieval accuracy using only 500 synthetic pairs.

CovMatch: Cross-Covariance Guided Multimodal Dataset Distillation with Trainable Text Encoder
Yongmin Lee School of Electrical Engineering KAIST
Hye Won Chung
School of Electrical Engineering KAIST
Lym75550kaistt.ac
hwchung@kaist.acc .r
Abstract
Multimodal dataset distillation aims to synthesize a small set image-text pairs that enables efficient training of large-scale vision-language models. While dataset distillation has shown promise unimodal tasks, extending it to multimodal contrastive learning presents key challenges: learning cross-modal alignment and managing the high computational cost of large encoders. Prior approaches address scalability by freezing the text encoder and updating only the image coder and text projection layer. However, we find this severely limits semantic alignment and becomes bottleneck for performance scaling. We propose Cov- Match, a scalable dataset distillation framework that aligns the cross-covariances real and synthetic features while regularizing feature distributions within each modality. Unlike prior approaches, CovMatch enables joint optimization of both encoders, leading to stronger cross-modal alignment and improved perfor- mance. Evaluated on Flickr30K and COCO, CovMatch outperforms state-of- the-art multimodal distillation methods and achieves up to 6.8% absolute gains in retrieval accuracy using only 500 synthetic pairs. Our code is available at https://github.com http://hhhhloooooooooaa///////// Yongalls/CovMutch
Introduction
Dataset distillation aims to synthesize compact and representative dataset from much larger one, enabling efficient model training with significantly reduced computational cost. While this approach has shown strong results in unimodal settings, particularly in image classification, its extension to multimodal tasks remains underexplored. The success of large-scale multimodal models such as CLIP 366 has relied training with massive image-text datasets, often comprising hundreds of millions of examples, which poses serious computational and storage challenges. This gap motivates the study of multimodal dataset distillation, which seeks to generate small, high-quality set of image-text examples that enables efficient multimodal training. However, distillation in the multimodal setting introduces new challenges that are both algorithmic and computational.
central difficulty in vision-language distillation lies in learning accurate cross cross-modal modal correspon- dences between image and text, which typically involves powerful pretrained encoders and contrastive learning frameworks, Incorporating such setups into distillation pipeline significantly amplifies the computational and memory burden, especially under the standard bi-level optimization framework of dataset distillation [41], where the inner loop updates model parameters using the synthetic data, and the outer loop updates the synthetic dataset based the model's 's performance real data. Computing gradients for the outer loop requires tracking how model parameters evolve with respect synthetic data, often necessitating full unrolling of the inner loop, which is expensive even in unimodal settings. In multimodal contrastive learning, scalability is even more constrained, as it involves significantly larger models, such NFNet (140MB) and BERT (450MB), compared the lightweight ConvNets (e.g.. 1.24MB) commonly used in unimodal distillation.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

framework
framework
Xime
Kang
Image encoder
Image encoder
Dicslistion loss
Dielllation
ancoder
and sooe
Text encoder
Next
X
Trandormer layers
Input enbbedding layer
Distiled (ta)
+
frozen
Enear projection layer
forward pass
backward
Figure 1: Comparison of multimodal dataset distillation frameworks. Prior methods freeze the entire text encoder and pass distilled text only through linear projection. In contrast, freeze only the input embedding layer (i.e., token and position embedding module) and include the transformer layers in the distillation process.
Table I: Resource requirements for long-term trajectory matching methods on a single A100 80GB GPU. For the unimodal case, we apply MTT [5] to distill CIFAR-10 using a 1.24MB ConvNet. In the multimodal setting, we distill COCO [26] using NENet (1 (140MB) and BERT (450MB) as encoders, applying MTT and Tesla 8 by matching trajectories for both modalities. The cost of storing expert trajectories scales with model size and checkpoint count, while distillation cost scales with model complexity, synthetic steps, and synthetic data batch size |, Despite using reduced hyperparameters, long-term matching in the multimodal case remains costly. In contrast, CovMatch requires no expert trajectories and achieves distillation with much lower cost, scaling only with model complexity and the batch sizes [B and |
Hyper parameters
Expert
Distill
Method
checkpoints
syn steps
1yya
[Bral]|
storage
time
memory
time
MTT (unimodal)
5000
50
128
6.2GB
1.9h
15GB
1.0 sec/it
MTT (multimodal)
200
20
120GB
132h
71GB
16.9 see/it
Tesla (multimodal)
200
8
20
120GB
132h
22GB
19.2 sec/it
CovMatch
100
128
0GB
Oh
15GB
1.2 sec/it
To address scalability issues, initial multimodal distillation methods such as MTT-VL 43 and
LoRS [45] adopt Matching Training Trajectories (MTT) [5] surrogate for bi-level optimization, while freezing the text encoder and updating only the image encoder and the projection layers. MTT precomputes and stores expert training trajectories from real data, and optimizes the synthetic dataset to match segments of these trajectories. While MTT avoids full unrolling of the inner optimization loop, still requires storing expert checkpoints and unrolling gradients with respect to the synthetic data to align with the expert trajectories. Applying MTT to both image and text encoders in multimodal setups exacerbates scalability issues, as summarized in Table For instance, storing expert trajectories for large backbones such as NFNet (image encoder) and BERT (text encoder) can require over 120GB of storage and 5 days of training on a single A100 GPU. Moreover, synthesizing just 100 image-text pairs for COCO dataset [26] requires over 70GB of memory for distillation, necessitating the use of high-end GPUs like the A100 80GB. Even memory- -fffcceenf variants like Tesla [8] remain costly due to expert trajectory storage. As result, current approaches typically freeze the text encoder, as shown in Figure left), and synthesize image-text pairs that update only the image encoder and the image/text projection layers.
However, we find that freezing the text encoder and relying solely on the projection layer for aligning modalities severely limits the capacity for semantic alignment in multimodal contrastive learning. In particular, this design becomes bottleneck scaling the performance of dataset distillation. In models using LoRS-ge -eRg-geneetted synthetic image-text pairs 45
our analysis, we train corresponding to the same image in the original Flickr30K dataset [35 and observe that captions fail to form coherent clusters in the text embedding space, even as the size of the synthetic dataset increases (Figure a)). This indicates that a frozen text encoder insufficient to support the semantic alignment necessary for effective cross- modal learning. As result, image-text retrieval performance with the LoRS saturates with increasing synthetic data size and, beyond = 1000 image-text pairs, underperforms models trained on randomly sampled real pairs (Figure c)))

---

Motivated by these observations, we propose new vision- vision-language dataset distillation framework (Figure (Fight)[[rggh)) [right) that synthesizes image-text pairs update both image and text encoders, enabling stronger cross-modal alignment while remaining scalable even with large pretrained models. The core idea to simplify the inner optimization of the bi-level distillation framework into a closed- form solution, thereby avoiding the high memory and computational cost associated with unrolled optimization or trajectory matching. Inspired by prior works in the unimodal setting such as KIP [32] and FrePo (54), we fix the encoders during each distillation step and optimize only the linear projection layers. By adopting the linear multimodal contrastive loss [3I, the inner optimization admits a closed-form solution, reducing the bi-level objective to maximizing the trace of inner product between the covarccovee matrices of real and synthetic image-tex features. This enables efficient outer optimization without unrolling, as gradients are simply backpropagated through the fixed encoders.
Building on this insight, we introduce Cross-Covariance Matching (CovMatch), an efficient algorithm for multimodal dataset distillation. CovMatch aligns crvss-cov matrices between real and synthetic pairs sampled at each distillation step, and includes a regularization term that matches feature distributions within each modality to prevent trivial solutions. To ensure the synthetic data remains informative for encoder training, we incorporate a lightweight online model update, where the encoders are updated using small batch of real image-text pairs before each distillation step, keeping the alignment statistics in syne with the evolving representations.
We evaluate CovMatch on image-text retrieval tasks using the Flickr30K 35 and COCO [26] benchmarks. CovMatch consistently outperforms state-of-the-as multimodal distillation methods, including MTT-VI. 4 and LoRS
45). Remarkably, with just 500 synthetic pairs, CovMatch achieves absolute improvements of 6.8% on Flickr30K and 6.1 on COCO in average retrieval accuracy over the best- hest-performing baseline. These gains are largely attributed to CovMatch's 's ability to jointly optimize both image and text encoders without compromising scalability.
2 Motivation
2.1 Preliminaries
We introduce the problem of multimodal dataset distillation for image-text contrastive learning and briefly summarize existing approaches. A detailed review of related works is provided in Appendix A
Image-Texx Text Contrastive Learning Image-text contrastive learning aims to map visual and textual data into shared embedding space using a bidirectional contrastive loss [36]. Given a dataset of paired image-text samples T = {(x), xx)] ) where x² E I' and 2 E R denote the i-th image and text inputs, the goal is to train an image encoder fo IR" + Rd= and a text encoder Ji : R - R4, each followed by a trainable linear projection layer, G. E Rexd, and G3 E R=xd: These projection layers map the modality-speciff embeddings into a shared space F= that captures semantic similarity, For given pair (z... 21). the encoders produce intermediate representations h2 = (zv: 0.) and f = fi(z;; 0t), which are then projected as 2 = G.,,, and 2 = G1h,, The similarity between image ær, and text ./f is computed using cosine similarity, 8ij == cos( (z = (=t. = |/ in the shared embedding. The model is trained using the InfoNCE loss [34] with temperature 7 >
exp(sn/T) Σ log Σ exp(s1//T) log Ejer exp(s5s/7)
M
exp(su/T)
LNCE
(1)
M
Main Goal of Vision-Language Dataset Distillation The goal of vision- languae dataset dis- tillation is to compress a large dataset T = {Ir, =f)= M
into much smaller synthetic dataset S = { # with N < M, such that a model trained on S achieves comparable image- text alignment performance to one trained on T. Following prior work 4
45 we evaluate S using standard mage-too-text and text-to-image retrieval metrics. Let d : R
x R - R de- note a similarity metric in the shared embedding space. The image and text representations are computed as (0,, G.) = fzz; 6.) and (1) Gt) = (zz given model parameters @ (0., 9r, Go,, G.). Given a test set Diest, the objective to ensure that the expected similarity remains consistent between models trained on the full dataset T and on the synthetic dataset S:
(((...2)) (d(zz (0,, G.),, (0 GT) ~ B(ze.a)) (d(zz. Gv), (0,,, Gi))].
(2)
where O* (0%, 07, G G7) and é (0., 0r, Go, Gi) are the parameters trained on T and resp.
3

---

0.6
0.4
LoRs
CovMetch
0.2
LoRS
random
LoRS
CovMatch
N-100
N-500
Number pairs (c) Scalability
1000 1500 2000
(a) T-SNE visualization
[b) Painwise similarity
Figure 2: (a) T-SNE visualization of text embeddings from the Flickr30K test set. Models are trained
with N = 500 synthetic pairs distilled by LoRS (left) and CovMatch (right). (b) Intra- and inter-pair
average similarity of test captions for models trained with N 100 and 500 synthetic samples. (c)
Retrieval performance as function of the number of synthetic pairs N.
Bi-Trajectory Matching As an initial approach to vision-language dataset distillation, recent works such as MTT-VL and LoRS 43 45 originally proposed for image classification. MTT constructs synthetic datasets by matching the training trajectories of models trained on real data. To extend this to the multimodal setting, MTT-VL introduces bi-trajectory matching, where both image and text encoders are jointly trained using the contrastive loss (T. and the synthetic dataset is optimized to reproduce their joint training dynamics. Let r" = (or 1== denote the expert trajectory with model parameters θ = (0 Gaa,)) obtained from training on the real dataset T. A student model initialized at random epoch with θ, O and trained on synthetic data S for It' steps to yield 2..R. The synthetic dataset then updated by minimizing the bi-trajectory matching loss:
build on Matching Training Trajectories (MTT) (S),
||0img,s+ ++ O ing.
11Qu,.e O
L Chi-trajectory
+ 10tr, O x
(3)
||0 il0..e= O +
where R is target step for matching. This objective encourages the model trained on S follow the update trajectory of the model trained on real data. However, computing this loss requires unrolling R' steps of gradient descent for both encoders, leading significant memory and compute overhead (Table DT. To mitigate this, prior work freezes the pre-trained text encoder (c.g., BERT) and updates only its projection layer Gj. As result, only θ GI. is trainable on the text side, while iigg. (0o.,, G..)) remains trainable. We discuss the limitations of this partial freezing strategy.
2.2 Limitations of Bi- Bi-Trajectory Matching Methods with a Frozen Text Encoder
We find that freezing the text encoder and updating only the projection layer for modality alignment severely limits semantic alignment in multimodal contrastive learning, creating a bottleneck for scaling the performance of dataset distillation. To investigate this, we train vision-language models using synthetic image-text pairs generated by LoRS [45). which freezes the text encoder during distillation. We visualize the resulting text embeddings from the Flickr30K test set 335 in Figure a). Captions associated with the same image (red for the top image, blue for the bottom) fail to form tight clusters in the shared embedding space, suggesting that the frozen text encoder cannot adapt to visual context. In contrast, our method, CovMatch, which jointly optimizes both encoders, yields more semantically coherent text embeddings that better reflect shared visual semantics.
To quantify this observation, we measure the average intra- and inter-pair cosine similarities between text embeddings. Intra-similarity is computed among captions corresponding to the same image, while inter-similarity is computed among captions of different images. As shown in Figure b). LoRS yields relatively small gap between intra- and snter-similarrty, whereas CovMatch produces significantly larger gap, indicating stronger semantic alignment. This pattern remains consistent as the number of synthetic samples increases from N 100 to N 500.
This limited alignment in LoRS leads to poor performance scaling: as shown in Figure c)), retrieval accuracy saturates or even degrades beyond N = 1,000 synthetic pairs, eventually falling below models trained on randomly sampled real pairs. In contrast, CovMatch maintains steady performance gains with more synthetic data, enabled by its efficient cross-covariance alignment objective. This scalable formulation allows the synthetic dataset effectively support the joint training of both encoders, key factor in its superior alignment and retrieval performance.

---

3 Method
3.1 Cross-Covariance Alignment: Closed-Forte Solution for Linearized Contrastive Learning
To develop a compute- multimodal dataset distillation method that updates both image and text encoders, we revisit the original formulation of dataset distillation. In the context of image-text contrastive learning, the distillation objective can be framed as bi-level optimization problem:
S* arg min LNCE where θ = arg min LNC
Here, the inner loop learns model parameters Ô from the synthetic dataset S, while the outer loop updates S to improve performance the real dataset T. However, computing gradients of the outer loss with respect to requires backpropagating through the entire inner optimization, which is costly in both memory and computation. To mitigate this, prior work in unimodal distillation has proposed simplifying the inner loop to admit a closed-form solution. For instance, FrePo [54] considers training only the final linear layer while fixing the feature extractor. Under this setting, the outer objective can be approximated as a kernel ridge regression problem based on the Gram matrix of neural features from real and synthetic inputs. This allows the synthetic dataset be updated by backpropagating only through the conjugate kernel and fixed feature extractor, greatly reducing the computational cost.
Following this insight, we propose distillation framework for image-text contrastive learning in which the inner optimization admits closed-form solution, enabling direct gradient updates to the synthetic dataset S. To achieve this, we fix the image and text encoders fu (= 0.)) and fi (x:;0) (t) and optimize only the linear projection layers G, and Gt at each distillation step. Given an input pair (z,, æi), we extract features h, = (x 0g) and f = fz 01), and project them to shared embedding space as 20 = Goh, and 21 = G.hy. We adopt the linear contrastive loss from [3 15
[D]
12
Lia(Ge Gr; 2)DI((D) 1)
Σ
sty sh)+ 2|DI([D) 1)
>E (s
+ 2
GIl
(5)
where Stj := (G (Grht) and D = (1h h This loss is equivalent to trace-based
cross-covariance formulation:
Lim (G Gr; = TT(G. + IIc G.l.
(6)
where the cross-covariance matrix CP) is given by
D
C
Σ hh,
ph. (b phx)
[
with Mhs and Ph, denoting the empirical means of the image and text features, respectively. Under this setup, the dataset distillation objective can be written as
S arg min Lia (G,, Gú; T) where Gy,, G. arg min Lio (Gn Gr: Gu.G,
(8)
which is equivalent to
S* arg min CrG GT) (TO) where G., G = arg min -TG. GIG.
(9)
G..G.
with C and C denoting the cross covariance matrices of the real and synthetic datasets, respectively. The inner loss further admits the reformulation:
Tr(G, (
/ ||G] G. 12 |.. G. 1 c
2p |
where the optimal solution satisfies G G. = 1C Substituting this into the outer loss in ( yields
S* arg max Tr(CT C s
(10)
indicating that the optimization reduces to aligning the cross-covariances of real and synthetic data. We provide theoretical justification for linear contrastive loss and complete derivation in Appendix C
5

---

Image cnooder
Image feature matching loss
Image
Image (real)
Image (59m))
Cress coveriancs mabrix
Cross covarlance matching loss
Cross- covariance
mattix
Synthetic dataset
datanct
Text ancoder
Text feature matthing loss:
Text cncoddr
Thp perure
Text (real)
Text (syn)
Shared
Encoder
Linar projection layer
Image feature
Tost feature
Cross covariance mattk:
Forward pess
Badioropagation
Figure 3: Illustration of our proposed method, CovMatch. At each distillation step, CovMatch updates the synthetic image-tett pairs using matching loss computed with an alternately updated online model. The loss consists of two components: 1) Cross-covariance matching loss (II). computed as the distance between the cross-covariance matrices of real and synthetic features (extracted before the projection layers); and 2) Feature matching loss (12). computed as the distance between the mean feature vectors of real and synthetic data after the projection, applied independently to each modality.
3.2 CoyMatch: Cross-Covariance Matching Algorithm
Cooss-oovrriance Matching Loss As shown in (10), multimodal dataset distillation for linear contrastive learning can be reformulated aligning the cross-covariance matrices real and synthetic datasets. However, directly optimizing the trace objective in (IO) can lead to training instability, the trace is unbounded. To mitigate this, instead frame the objective Frobenius norm minimization between the real cross-covariance CT and the ideal projection-based approximation G G ICs:
(T) -T 1S.
(11)
Here, the hyperparameter adjusts the scale differences between real and synthetic data, possibly arising from dataset size mismatches. We note that similar objective is considered in ClipCov 17] for selecting a subset of image-text pairs for data-efficient contrastive pretraining. However, due to the combinatorial complexity of subset selection, ClipCov approximates the objective using heuristics such as maximizing CLIP similarity scores [13]. In contrast, CovMatch directly optimizes the synthetic dataset continuous space minimize the cross-covariance alignment loss.
Feature Matching Loss While align- ing the cross-covariance matrices be- tween the real and synthetic datasets is central to multimodal dataset distillation, relying solely on this can lead to subop- timal solutions. As shown in Fig. a). the cross-covariance matrix CT derived from the full Flickr30K dataset exhibits a low-rank structure, indicating that match- ing it alone may impose insufficient constraints, particularly the synthetic dataset grows. Fig. b) further shows that optimizing only for cross-covariance can result in notable mismatches in the mean feature embeddings.
1500
Sinquiar values
1000
1
250
500
0-0
5000
10000
Index
Distiliation steps
(a) Sinuglar values
(b) Matching loss
Figure 4: (a) Singular values of the cross-covariance ma- trix CT computed on the full Flickr30K dataset, show- ing its low-rank structure. (b) Evolution of the cross- covariance matching loss Leov I and the feature match- ing loss Cfear + Lfcai 12 during distillation with only the cosariavee objective for 500 synthetic pairs.
To mitigate this, we introduce a feature matching loss as regularizer, which minimizes the maximum mean discrep- ancy between projected real and synthetic features, computed separately for each modality. For each

---

Algorithm Dataset Distillation via Cross-Covariance Matching (CovMatch)
1: Require: Full training set T, pretrained weights 0peetriined for image encoder Sv, pretrained
weights 0% oprctraince for text encoder fi, the learning rate for the distilled data
2: repeat
3: Initialize o,, 0g + @pretrained @fretrinee
4: Randomly initialize projection layers Gu. G1
5:
for t = 0 to T 1 do
6:
Sample mini-batch pairs BJ ~ and 1 S
7:
Compute the matching loss [CovMatch
13 on (B")B)
8: Update S + S a V CCrMahhh
9:
Train the model (0u, l1, Go, Gi) with T for one step.
10:
end for
11:
until convergence
12:
Output: S
modality m E {v,1} (vision, language), the loss is defined as:
1 m
1 [S] |s|
Bm)]| :m)) E {v,1). (12)
Clcal (T,
Σ G. fm (m mm)
Σ Gm fm
Here, (2 x and ((t), ( denote the real and synthetic datasets, respectively.
Training Algorithm for CovMatch Our final objective combines cross-covariance alignment and feature matching losses:
CooMaachh = Loov + A (Etcat + C(
(13)
where À balances the contributions of the two terms. At each distillation step, the synthetic dataset updated to minimize this combined loss.
To prevent overfitting to a fixed model state, we apply an online model update the image and text encoders using batch of real data for single gradient step before each distillation step. For additional stability, we periodically reinitialize the model: every steps, the encoders are reset to their pretrained weights and the projection layers are randomly reinitialized. The full training algorithm is presented in Algorithm Figure F an overview of CovMatch, aligning cross-covariance and feature distributions between real and synthetic data.
Experimental Results
4.1 Experiment Setup
Dataset and Tasks We evaluate the effectiveness of CovMatch on the Flickr30K [35] and COCO (66 datasets, following prior work [43 45). Both datasets consist of image-caption pairs: Flickr30K contains approximately 31K images and COCO contains 123K images, each annotated with five captions. We adopt the Karpathy split [88] for both datasets, yielding train/validation/test splits of 29K/IK/IK for Flickr30K and 113K/5K/5K for COCO, respectively. We assess model performance using cross-modal retrieval task, where the objective is to retrieve the correct item from modality (image or text) given a query from the other. Specifically, we compute cosine similarity between embeddings retrieve the top-K closest matches, and evaluate how often the correct match is included. We denote text-to-image retrieval as IR KKKK (i.e., retrieving the relevant image given a text query), and image-to-text retrieval TR @ K (i.e., retrieving the relevant text given image query).
Network Architectures Following 43
we use an ImageNet- Normalizer- ResNet (NFNet) [4] as the image encoder and pretrained BERT-base model 9 as the text encoder. Fach encoder followed by trainable linear projection layer that maps features into a shared embedding space. Additional results with alternative architectures are reported in the Appendix E.

---

Table 2: Performance comparison on Flickr30K and COCO with various number of synthetic pairs. Using 100, 200, 500 pairs corresponds to approximately 0.3%, 0.7%, 1.7% of the full dataset for Flickr30k, and 0.1 %, 0.2%, 0.4% for COCO. The performance achieved by training on the full dataset as follows: IR @ 188.7, -RR.227,., IR 10=87.2, T 1=61.6, TR @ 5=85.9, and TR @ 10=91.5 for Flickr30k, and IR @ 1=25.1, 1,, IR @ 5-53.9,, IR@ 10=67.5, TR@ 1=33.0, TR TR @ 10=75.0. for COCO. The reported values are averages of five runs, and the full results with standard deviations
are provided in the Table 14 and Table 15
Flickr50k
COCO
Pains
Method
IRG
IRW2
TR
TR.65
TR 10
IR@1
IR
IR@10
TR
TR
TR
Ramdom
2.0
12.6
3.3
10:1
16.0
8.6
0.7
2.8
1.0
4.0
3/1
Herding
2.2
8.0
13.4
3.0
9.9
15.6
8.7
0.7
2.9
5.3
1.1
4.1
6.8
3.5
100
MTEVL
C-Ceneer
2.0
13.0
2.8
9.7
16.4
8.6
0.7
0.9
4.2
7.6
3.8
LoRS
8.3
24.1
35.1
11.8
35.8
49.2
27.4
1.8
7.1
12:2
3.3
12.2
19.6
9.4
CorMatch
10.1
28.6
40.9
14.8
38,0
30.6
30.5
2.8
10.5
17.7
3.8
13.1
21.1
11.5
Random
3.3
15.8
1.7
6.5
Herding
3.0
11.3
18.3
4.7
15.4
22.9
12.6
1.2
1.6
6.6
11.2
200
MTTVL
K-Conler
4.6
3.2
16.0
11.1
25.5
17.7
10.2
5.3
28.7
15.2
25.2
41.9
21.2
12.6
1.7
1.2
5.1
12.3
8.9
3.3
1.9
11,9
6.7
19.4
11.6
5.9
CovMatch
12.3
33.6
45.8
17.4
41.7
$3.8
34.4
3.8
13.4
21.8
5.3
17.3
27.0
14.8
Random
6.9
21.0
31.2
10.0
28.0
38.7
22.6
2.2
8.8
14.9
3.5
11.9
19.2
10.1
Herding
6.8
9.3
26.4
2.3
2.9
500
MTEVLL
K.Center
6.9
6.6
20.2
30.0
13.3
10.6
29.5
32.8
46.8
25.0
2.5
8.9
13.8
5.0
3.6
17.2
26.0
12.6
LoRS
10.0
28.9
41.6
15.5
99.8
53.7
31.6
2.8
9.9
16.1
5.3
18.3
27.9
13.5
Table 3: Cross- Arositrrcite evaluation results on Flickr30K using 100 synthetic pairs. Reported values are averaged over six retrieval metrics: IR@1, 1, IR@5, IR@ 10, TR@ 1, TR@5, and TR@ Full results are provided in Table
Text encoder
BERT
DistilBERT
Image encoder
NENet
Ne-ResNet
NE-RegNet
VIT
NFNet
NE-ResNet
NE-RegNet
VIT
Random
8.4
8.9
8.3
10.8
9.4
10.2
8.7
11.5
MIT-VL
20.4
8.4
7.5
9.6
20.2
7.5
7.0
8.5
LoRS
8.8
8.4
9.3
23.5
8.9
CovMatch
30.2
15.5
15.1
27.1
14.6
13.4
Baselines We compare CovMatch against both coreset selection and dataset distillation methods. For coreset selection, include Random (uniform sampling), Herding [7], and K- K-Center (38), all adapted to the multimodal setting as in 433 For dataset distillation, we evaluate MTT-VL 4S and LoRS [45]. Following their original protocols, these methods freeze the text encoder during both distillation and evaluation due to computational and optimization constraints. contrast, all other baselines fine-tur the entire network. Further implementation details are available in Appendix B
4.2 Main Results
Flickr30K and COCO We evaluate CovMatch on both Flickr30K and COCO, comparing its performance against several baselines across varying numbers of synthetic image-text pairs, presented in Table key observation is that performance gains from existing dataset distillation methods quickly saturate the number of synthetic pairs increases-even in the extremely low-data regime (i.e., under 2% of the full dataset). This result indicates that fine-tuning the text encoder is critical for effective multimodal contrastive learning, particularly when scaling to larger synthetic datasets. By jointly optimizing both the image and text encoders through cross-covariance alignment, CovMatch consistently outperforms prior methods and establishes new state-of state-of-the-arr the-art results across all settings. Notably, on Flickr30K with 500 synthetic pairs, CovMatch achieves a 6.8% absolute improvement over the strongest baseline.
Cross-Architecture Gemeralization One of the essential characteristics of a distilled dataset is its ability to generalize across different, unseen architectures. demonstrate CovMatch's 's cross- architecture generalizability, we distill the dataset with NFNet and BERT, and then evaluate it with alternative architectures Ne-ResNet [ NF-RegNet 44
ViT 10 for the image encoder, and
DistilBERT 377 for the text encoder. As shown in Table 3 the dataset distilled by CovMatch

---

Retrieval aecuracc (N6
Retrirra accuracy (N)
-5
30
8.0
Ratroo
25
30
20
Fixed
2.
N-200
25
Update with
N-500
Update with
1.5.5
p. scaling factor
0.25 0.50 0.75
matching
1.00
N=100
N=200
N=500
(a) Ablation on
(b) Ablation on
(c) Ablation on model update
Figure 5: (a) Ablation on the scaling factor in Scaling the real cross-covariance CT is particularly important when N is small. (b) Ablation on the feature matching weight À in (13). The optimal value of À tends to increase as N increases, (c) Ablation on the online model update strategy. Updating with real data exhibits the best performance. All ablation studies are conducted on Flickr30k with varying N.
LoRS
CouMatch
young male in white larate outfi with blue beft wrapped around his west perfonns benale move ir front of the judges of the competition
bay jumps high
grabbing
young shoulder
white karate outfit practicing mowe displays his tighting technn que
younng practices martial stards stiking postion While holding his agconents head the buy is jamping the ai
gny
young wearing tunic, white and
1
standing his left leg with right extended karate youung wnien jamp the wirking mantial
Figure 6: Comparison of the five retrieved captions from models trained with LoRS- and CovMatch-generaal synthetic image-text pairs, given image query. Each model is trained on 500 synthetic pairs, and queries are from the Flickr30K test set. Ground-truth captions (i.e., five captions paired with the query image Flickr30K test set) are highlighted in blue.
generalizes effectively to both unseen image and text encoders, whereas baseline methods perform comparably to or worse than random selection. We attribute this the fact that prior methods only optimize the image encoder and projection layers during distillation, which induces overfitting to the specific architecture used in training.
4.3 Ablation Study and Further Analysis
Scaling Factor CovMatch introduces a scaling factor p in [I when aligning the cross-covariance matrix of the synthetic dataset with that of the real dataset, where originates from the regularization term in the inner optimization of the original bi-level optimization problem the impact of varying P across different numbers of synthetic image-text pairs. The results indicate that appropriate scaling becomes particularly important when the number of synthetic samples is
(9). Figure 55aa presents
small. This finding consistent with the intuition that stronger regularization is required in low-data regimes to prevent overfitting.
Feature Matching We also investigate the effect of the feature matching objective by varying its weighting coefficient À in (13). As shown in Figure 5Jb). incorporating feature matching significantly enhances performance, particularly when larger number of synthetic pairs is used. Moreover, the optimal value of À tends to increase with the number of pairs. This observation is consistent with our intuition: as the number of synthetic samples grows, aligning mross-ovvriiacc--- second- order hecistic--ecomes easier, even when the underlying feature distributions remain misaligned. Consequently, stronger feature-level regularization is required to constrain the optimization toward semantically meaningful representations.
Online Model Update Figure (5.) shows the impact of the online model update. We compare three update strategies: (1) fixing the encoder, (2) updating the encoder with the synthetic set S, and (3) updating it with the real data from T-the latter being the strategy adopted our method. The result reveals that updating the online model is crucial for preventing overfitting to fixed model

---

state. Additionally, further improvements are achieved by updating the online model with real data from T, rather than with the synthetic dataset S.
Qualitative Analysis In contrast to previous methods, CovMatch incorporates the text encoder into both the distillation and evaluation processes. To illustrate the impact of this design, we provide text-retrieval examples in Figure 6 For each image query, we show the top five retrieved captions using models trained on synthetic image-text pairs generated by LoRS and CovMatch, respectively; the ground-truth caption (from the five true captions paired with the image) is highlighted in blue. As shown, CovMatch leads to stronger alignment between visual and textual modalities, retrieving more ground-truth captions. While LoRS often captures only basic semantics (e.g., "boy"), CovMatch enables alignment with more nuanced concepts (e.g., "karate"), reflecting the tighter clustering of text representations seen in Figure aa). More examples are provided in Appendix
Conclusion
In this work, we revisit the original bi-level optimization formulation of dataset distillation and, under the assumption of linear contrastive learning, derive a simplified objective that facilitates the inclusion of the text encoder in the multimodal dataset distillation framework. We present CovMatch, lightweight and scalable algorithm for multimodal dataset distillation that aligns the cross-covariance of image-text embeddings between real and synthetic datasets, with additional regularization via feature distribution alignment within each modality. CovMatch outperforms existing algorithms with significant performance improvements, enhancing cross-modal retrieval ability, cross- pross-architecture generalization and scalability.
Limitations and Future Works We assume scenario in which image and text encoders, pre- trained within their respective modalities, are available. Accordingly, our method focuses on dataset distillation to effectively fine-tune image-text contrastive models. However, we have not yet explored scenarios where pretrained image and text encoders are unavailable, requiring training from scratch. Distilling datasets for pretraining image-text contrastive models presents promising direction for future work.
Broader Impact This paper introduces computationally and meemory-efficiee multimodal dataset distillation method. CovMatch demonstrates strong generalization across different network archi- tectures, highlighting its versatility and robustness. These features, combined with its enhanced scalability, make a highly practical solution for real-world applications. Additionally, our new framework, which freezes only the input embedding layer while incorporating the transformer layers of the text encoder in the distillation process, provides a solid foundation for future research and serves as a starting point for the development of more efficient and scalable multimodal learning methods.
Acknowledgments and Disclosure of Funding
This work was supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (No. RS-2024-00408003, AS-2005-00051611 and 04-2024-004448622
References
[1] Andrew, G., Arora, R., Bilmes, J., Liveseu, K.: Deep canonical correlation analysis. In:
International Conference on Machine Learning (ICML) (2013)
[2] Bain, M., Nagrani, A., Varol, G., Zisserman, A.: Frozen time: A joint video and image encoder for end-to-end retrieval. In: Proceedings of the IEEE/CVF international conference on computer vision (2021)
[3] Brock, A., De, S., Smith, S.L.: Characterizing signal propagation close the performance gap in unnormalized resnets. arXiv preprint arXiv:2101 .86922 (2021)
[4] Brock, A., De, S., Smith, S.I., Simonyan, K.: perf-peeformance large-scale image recognition without normalization, In: International conference machine learning (2021)
10

---

[5] Cazenavette, G., Wang, T., Torralba, A., Efros, A.A., Zhu, J.Y.: Dataset distillation by matching training trajectories. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2022)
[6] Chen, Y., Guo, S., Zheng, B., Wang, L.: large-scale study on video action dataset condensation. arXiv preprint arXiv:2412.21119 (2024)
[7] Chen, Y., Welling, M., Smola, A.: Super-samples from kernel herding. In: Proceedings of the Twenty-Sixit Conference on Uncertainty in Artificial Intelligence. pp. 109-116 (2010)
[8] Cui, J., Wang, R., Si, S., Hsich, C.J.: Scaling up dataset distillation to imagenet- Ik with constant memory. In: International Conference Machine Learning (2023)
[9] Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: Bert: Pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume (long and short papers) (2019)
[10] Dosovitskiy, A., Beyer, L.,, Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., al.: An image is worth 16x 16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.1 1192 (2020)
[11] Faghri, F., Fleet, D.J., Kiros, J.R., Fidler, S.: Vse++: Improving visual-semantic embeddings with hard negatives. arXiv preprint arXiv: 707.77077.111 006612 (2017)
[12] Frome, A., Corrado, G.S., Shlens, J., Bengio, S., Dean, J., Mikolov, T.: Devise: A deep visual- semantic embedding model. In: Advances in Neural Information Processing Systems (NeurIPS) (2013)
[13] Gadre, S.Y., Ilharco, G., Fang, A., Hayase, J., Smyrnis, G., Nguyen, T., Marten, R., Wortsman, M., Ghosh, D., Zhang, J., al.: Datacomp: In search of the next generation of multimodal datasets. Advances in Neural Information Processing Systems (2023)
[14] Jacot, A., Gabriel, F,, Hongler, C.: Neural tangent kernel: Convergence and generalization in neural networks. Advances in neural information processing systems (2018)
[15] Ji, W., Deng, Z., Nakada, R., Zou, J., Zhang, L.: The power of contrast for feature learning: A theoretical analysis. Journal of Machine Learning Research (2023)
[16] Jia, C., Yang. Y., Xia, Y., Chen, Y.T., Parekh, Z., Pham, H., Le, Q., Sung, Y.H., Li, Z., Duerig, T.: Scaling up visual and vision-languad representation learning with noisy text supervision. PMLR (2021)
In: International conference on machine learning. pp.
[17] Joshi, S., Jain, A., Payani, A., Mirzasoleiman, B.: Data-efficient contrastive language- image pretraining: Prioritizing data quality over quantity. In: International Conference on Artificial Intelligence and Statistics (2024)
[18] Karpathy, A., Fei-Fei, L.: Deep visual-semantic alignments for generating image descriptions. In: Proceedings of the IEEE conference on computer vision and pattern recognition (2015)
[19] Kushwaha, S.S., Vasireddy, S.S.N., Wang, K., Tian, Y.: Audio-visual dataset distillation.
Transactions on Machine Learning Research (2024)
[20] Lee, K.H., Chen, X., Hua, G., Hu, H., He, X.: Stacked cross attention for image-text matching. In: European Conference on Computer Vision (ECCV) (2018)
[21] Lee, S., Chun, S., Jung, S., Yun. S., Yoon, S.: Dataset condensation with contrastive signals. In: International Conference on Machine Learning (2022)
[22] Li, Li, D., Xiong, C., Hoi, S.: Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In: International conference on machine learning (2022)
11

---

[23] Li. Selvaraju, R., Gotmare, A., Joty, S., Xiong, C., Hoi, S.C.H..: Align before fuse: Vision and language representation learning with momentum distillation. Advances neural information processing systems (2021)
[24] Li, K., Zhang, Y., Li, K., Li, Y., Fu, Y.: Visual semantic reasoning for image-text matching. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 4654-4662 (2019)
[25] Li, Y., Li, W.: Data distillation for text classification. arviv preprint arXiv:2104.08448 (2021)
[26] Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., Zitnick, C.L.: Microsoft coco: Common objects in context. In: Computer vision -CCv 2014: 13th European conference, zurich, Switzerland, September 6- 12, 2014, proceedings, part 13 (2014)
[27] Loo, N., Hasani, R., Lechner, M., Rus, D.: Dataset distillation with convexified implicit gradients. In: International Conference on Machine Learning (2023)
[28] Maekawa, A., Kobayashi, N., Funakoshi, K., Okumura, M.: Dataset distillation with attention labels for fine-tuning bert. Journal of Natural Language Processing (2025)
[29] Malladi, S., Wettig, A.. Yu, D., Chen, D., Arora, S.: A kernel-based view of language model fine-tuning, In: International Conference on Machine Learning (2023)
[30] Mu, N., Kirillov, A., Wagner, D., Xie, S.: Slip: Self-supervision meets language-image pre- training. In: European conference on computer vision (2022)
[31] Nakada, R., Gulluk, H.I., Deng, Z., Ji, W., Zou, J., Zhang, L.: Understanding multimodal contrastive learning and incorporating unpaired data. In: International Conference on Artificial Intelligence and Statisties (2023)
[32] Nguyen, T., Chen, Z., Lee, J.: Dataset meta-leaming from kernel ridge-regression. In: Interna- tional Conference on Learning Representations (2020)
[33] Nguyen, T., Novak, R., Xiao, L., Lee, J.: Dataset distillation with infinitely wide convolutional networks. Advances in Neural Information Processing Systems (2021)
[34] Oord, A.v.d., Li, Y., Vinyals, O.: Representation learning with contrastive predictive coding. arXiv preprint arXiv: wwXivv1807.00m 037403348 (2018)
[35] Plummer, B.A., Wang, L., Cervantes, C.M., Caicedo, J.C., Hockenmaier, J., Lazebnik, S.: Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to- taage-to-sentenc models. In: Proceedings of the IEEE international conference on computer vision (2015)
[36] Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: International conference on machine learning. pp. 8748-8763. PmLR (2021)
[37] Sanh, V., Debut, L.,, Chaumond, J., Wolf, T:: Distilhert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01 108 (2019)
[38] Sener, O., Savarese, S.: Active learning for convolutional neural networks: A core-set approach. In: International Conference on Learning Representations (2018)
[39] Tran, D., Wang, H., Torresani, I.., Ray, J., LeCun, Y., Paluri, M.: A closer look at spatiotemporal convolutions for action recognition. In: Proceedings of the IEEE conference Computer Vision and Pattern Recognition (2018)
[40] Wang, K., Zhao, B., Peng, X., Zhu, Z., Yang. S., Wang, S., Huang, G., Bilen, H., Wang, X., You, Y.: Cafe: Learning to condense dataset by aligning features. In: Proceedings the IEEE/CVF Conference Computer Vision and Pattern Recognition (2022)
[41] Wang, T., Zhu, J.Y., Torralba, A., Efros, A.A.: Dataset distillation. arXiv preprint arXiv:1811.10955 (2018)
12

---

[42] Wang. Z., Xu, Y., Lu, C., Li, Y.L.: Dancing with still images: video distillation via static- dynamic disentanglement. In: Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 6296-6304 (2024)
[43] Wu, X., Zhang, B., Deng, Z., Russakovsky, O.: Vision-language dataset distillation. Transactions on Machine Learning Research (2023)
[44] Xu, J., Pan, Y., Pan, X., Hoi, S., Yi, Z., Xu, Z.: Regnet: self-regulated network for image classification. IEEE Transactions Neural Networks and Learning Systems (2022)
[45] Xu, Y., Lin, Z., Qiu, Y., Lu, C., Li, Y.L.: Low-rank similarity mining for multimodal dataset distillation. In: International Conference on Machine Learning (2024)
[46] Yao, T., Pan, Y., Li. Y., Mei, T.: Filip: Fine-grained interactive language-image pre-training, In: International Conference on Learning Representations (ICLR) (2022)
[47] Yu, J., Wang, Z., Vasudevan, V., Yeung, L., Seyedhosseini, M., Wu, Y:: Coca: Contrastive captioners are image-text foundation models. Transactions on Machine Learning Research (2022)
[48] Zeiler, M.D., Fergus, R.: Visualizing and understanding convolutional networks. In: Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part 13 (2014)
[49] Zhai, X., Mustafa, B., Kolesnikov, A., Beyer, L.: Sigmoid loss for language image pre-training. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 11975- 11986 (2023)
[50] Zhai, X., Wang. X., Mustafa, B., Steiner, A., Keysers, D., Kolesnikov, A., Beyer, L.: Lit: Zero-shot transfer with locked-image text tuning. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (2022)
[51] Zhao, B., Bilen, H.: Dataset condensation with differentiable siamese augmentation. In: Inter- national Conference on Machine Learning (2021)
[52] Zhao, B., Bilen, H.: Dataset condensation with distribution matching. In: Proceedings of the IEEE/CVF Winter Conference Applications of Computer Vision (WACV) (2023)
[53] Zhao, B., Mopuri, K.R., Bilen, H.: Dataset condensation with gradient matching. In: Interna- tional Conference on Learning Representations (2020)
[54] Zhou, Y., Nezhadarya, E., Ba, J.: Dataset distillation using neural feature regression. Advances Neural Information Processing Systems (2022)
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction clearly highlight the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of work in Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
14

---

Justification:
Our paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide detailed information of dataset, performance metric, network
architectures, and implementation details in Section 4.1 and Appendix B
Guidelines:
The answer NA means that the paper does include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
15

---

Answer: [Yes]
Justification: We provide all the code and scripts required to reproduce our main experimental results in the supplementary material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurIPS code and data submission guidelines (https
hetpolliiiggiddee/ooooooo http://uddd//oeeSuubntn ssionpolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide all detailed information of our experimental setting in Section 4.] and Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification; We report the standard deviations for our main results in Table 14 and Table
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
16

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We report the memory usage and execution time of our method with the GPU in Table
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
Our research is conducted with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer:
[Yes]
Justification: discuss broader impacts our work in Section
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
17

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We cite the original paper that produced the code package dataset.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
18

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We provide the code and our distilled dataset in the supplementary material.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/cood/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
19

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification:
We used LLMs solely for writing and editing purposes.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy fattps://neur//: http://pp....ccooffeeeeee/////// for what should or should not be described.
20

---

A Related Works
Traditional Dataset Distillation Dataset distillation, introduced in 41], aims to create small synthetic S, so that a model ( trained on S achieves good generalization performance, performing well on the full dataset T
S arg min CT (08), with 0 arg min argmin (3)
Here, CT and L are losses on T and S, respectively. To address the bi-level optimization's 's computational complexity and memory demands, existing works have employed two methods: closed-form approximation and surrogate-based matching.
Closed-form approximation methods obtain meta- meta-gradient V C (BS) in closed-form by approxi- mating the inner optimization. For example, KIP (2, and FrePo [54] obtain closed-form solution for inner optimization by approximating it to kernel ridge regression. RCIG (27] assumes that the inner optimization is convex and computes the meta-gradient via implicit gradient.
Surrogate- based matching methods replace the original objective with simpler surrogate objective. It can be further categorized into two groups: short-term matching and long-term matching. Short-term matching methods aim to optimize the synthetic dataset such that it emulates the short-range behavior of the model with given parameter 0. For example, DC [53]. DSA (5T], and DCC [21 matches gradient of neural network computed with synthetic and real dataset V VoL (0) and VoL (0). DM [52] and CAFE 40 matches distribution of S and TT in the feature space. Since these approaches do not involve unrolled optimization, they are computationally efficient. In contrast, long-term matching methods, such as MTT [ and TESLA [8], aim to emulate long-term behavior by matching network parameters after several steps of training on the synthetic dataset. Although long-term matching has demonstrated superior performance compared to short-term matching methods, it introduces substantial costs. While TESLA (8 improves memory efficiency by making memory usage constant with respect to the number of synthetic steps, it still inherits the drawbacks of trajectory preparation and high computational overhead inherent to long-term matching.
Image- Text Retrieval Image-text retrieval, the canonical multimodal retrieval task, aims to align visual and textual representations within a shared semantic space. Early methods such De-
ViSE 12 and Deep CCA [ introduce joint embedding frameworks, later enhanced by VSE++ [T]] through hard-negative mining. Fine-grained cross-encoder models like SCAN (20) and VSRN [4 capture region-word interactions to improve alignment accuracy, though at the expense of effi ciency. The paradigm has shifted with large-scale contrastive dual-encooers- notaall CLIP (66 and ALIGN which trained image-text encoders on web-scale noisy data, establishing a foundation for efficient zero-shot retrieval. Building on this, LiT (50) enhances data efficiency by freezing the image encoder, SLIP (30 integrates self-supervised objectives, and SigLIP (49) introduces a pairwise sigmoid loss for improved stability. To retain fine-grained matching without full cross-attention,
FILIP 446 employes late interaction between loken-pateh pairs, while ausion-based pretraining frameworks such as ALBEF (23). BLIP [22), and CoCa (47 combine contrastive learning with image-conditioned language modeling to mitigate noisy supervision. Beyond empirical successes, a few recent works have provided theoretical insights into multimodal contrastive learning. The work in 31 showes that for linear models, each step of loss minimization via gradient descent can be interpreted as performing singular value decomposition (SVD) on the contrastive cross-covariance matrix between modalities, which forms a key insight motivating our work.
Multimodal Dataset Distillation Recent efforts have extended dataset distillation techniques to the image-text contrastive learning setting. MTT-VL. 43 adapts the MTT (5) framework in a naive manner, matching the parameters of the image and text encoders independently. LoRS [45] further improves performance by jointly distilling the ground-truth similarity matrix between image-text pairs, effectiiely capturing cross-modal correspondence. In addition to contrastive tasks, other works have explored dataset distillation in different domains and problem settings. The work in [19 addresses a multimodal classification task with audio and video modalities by matching cross- modal distributions, while [25 28 investigate dataset distillation for natural language processing tasks and [2 6 extend it to video classification. In parallel, ClipCov I7 proposes a dataset selection method for multimodal contrastive learning, aiming to preserve the cross-covariance structure of the original dataset. While ClipCov relies on heuristic surrogate objectives to approximate cross-covariance preservation, our work, CovMatch, directly optimizes the synthetic dataset to achieve this objective.
21

---

B Implementation Details
Distillation The synthetic dataset is initialized with randomly selected real image-texi pairs from the training set. During distillation, both the synthetic image pixels and text input embeddings optimized using SGD with momentum 0.5 and a learning rate 1.0. At each distillation step, the cross-covariance matrix and feature means computed using a batch of 128 real samples. For the synthetic data, the entire set is used for these computations, except in the 500-pair setting, where a batch of 256 synthetic samples is used to reduce memory consumption. We set the scaling factor to = 2 for 100 synthetic pairs and p = for 200 or more pairs. The feature matching weight is fixed 0.1 for 100 and 200 pairs, and increased 0.5 0.6 for 500 pairs to impose stronger regularization on cross-covariance alignment. Note that all network components-includinn
the image encoder, text encoder, and projection layers-are updated with one step of training on the real dataset each distillation step, and re-initialized every 50 updates. We distill for 10,000 iterations by default; for the 500-pair setting, extend this 20,000 iterations ensure full convergence, even after reaching 95% of the final performance. A summary of the hyperparameter used in CovMatch is provided in Table
Table 4: Plyper-paaamete used for CovMatch. A dash (-) in the synthetic batch size denotes that the entire synthetic set used for computing the matching loss.
Dataset Pairs
Flickr30k
COCO
100
200
500
100
200
500
Scaling factor p
2
2
Feature match weight À
0.1
0.1
0.6
0.1
0.1
0.5
Batch size (real)
128
128
128
128
128
Batch size (syn)
256
256
Learning rate (img)
1
1
Learning rate (txt)
1
1
1
Iteration
200
200
400
200
200
400
Evaluation During the evaluation stage, we train the model using SGD optimizer with momentum 0.9, weight decay 5e-4, batch size 128, and learning rate 0.01 for the image and text encoders and 0.1 for the projection layers. Training conducted for 100 epochs, and we employ a multi-step learning rate scheduler that decays the learning rate by factor of 0.1 at the 50th epoch.
Network Architectures In Table 5 provide detailed information on all the models used in our experiments. For image encoders, we use NFNet, Ne-ResNet, NF-RegNet, and ViT, and for text encoders, we employ BERT and DistilBERT.
Table 5: Detailed information of the models.
Network Architecture
Model
Parameter Count
NFNet [4]
nfnet 10
26.4M
NE-ResNet S
nf resnet50
25.6M
NF-RegNet
nf_regnet_bl
21.0M
VIT [0]
vit wwl_baa_techhhle_ 224
85.8M
BERT 9
bert- bert-base-uncased
110.0M
DistilBERT 37
seiilbert-base-uncee
66.0M
C Discussion about Linearized Contrastive Learning
we derived the cross-covariance alignment objective under the linearized contrastive learning assumption. In this section, we provide theoretical justification for using the linear contrastive loss (5) and full derivation for Equation (
In Section 3.1
22

---

Theoretical Justification for Linearized Contrastive Objective note that the linear contrastive loss (5) can be interpreted high- -esperrerr approximation of the InfoNCE loss (T). Specifically, under the assumption of a large temperature > 0, the softmax in the InfoNCE loss flattens, and
first-order Taylor expansion yields
log( exp(si//T)) log
Σ jzti
jyht
+ -22 + log(M - 1)
-log
jti
Σ
Stz + log(M - 1).
T TM 1)
jyf
Substituting this into the InfoNCE loss leads to the linearized form
M
M Σ sa) + Am JØl
1
LNCE ~ TM(M 1)
Σ
Si sn) + TM(M -M 1)
(sji
Constant,
2=1
which is equivalent to (5), up constant scaling and shift. This derivation provides justification for using the linearized objective as an approximation of InfoNCE when is large.
Also, recent work has shown that fine-tuning large neural networks (e.g.. foundation models) often operates the Neural Tangent Kernel (NTK) regime [4), where the training dynamics are well approximated by linear model over a high-dimensional feature space derived from the model's gradients [29] This connection suggests that our analysis in the linear regime can mcaningfully inform behavior in more general, non-linear settings.
Equivalence between Eq. (5) and Eq. (6) Remind that the cross-covariance matrix of the dataset D = { At) is defined as
eD (ID
2 (he - mh. ph,) (ID) 1)
ht(h))) (Dlmn (Mnr)) (14)
with Mh. and μıı denoting the empirical means of the image and text features, respectively. The cross-covariance alignment term can be written as
TT(G,CPGT) = Σ CTCC9..
Ive
gi
(15)
k=1
where G.T ......... Iv.1 E Rd. xa and GT
[9 ...,.....90 -9.1 E Rdixa Note that the similarity
terms in can be expressed as
st sy = (Gohtt) (G+) (G
(gr) Th. (h/( gt.
Then, the linear contrastive loss
5
can be written
0 |G Gill
(Gn, Gr;
[D](12] 1)
8ij
D
8ji +
2
i=1
1=1
D
DD]
90
h (2()
h (ht)
91. + + L.
D ( 1)
D -
1=1
1=1
1D)
[D]
Jos
(1D)
1) Mh, (MmT) T
D
94 +
2
|G Gill G.....l..
1=1
- P
G.
2
Hence, the linear contrastive loss (5) is equivalent to cross-covariance alignment term with regular- ization term the projection matrices
23

---

D Generalization to Video-Text Retrieval Tasks
To further evaluate the generalizability of CovMatch beyond image-texi data, we conduct experiments on the video-tex text retrieval task. For this study, we construct a cnmpuuttiooaallymanagge
subset of the Web IOM dataset [2]. consisting 49K training and IK test video-text pairs. We use pretrained VideoResNet (r3d_ 18) 39 the video encoder and BERT the text encoder. For the synthetic dataset, we distill 500 video-text pairs, sampling four frames per video.
As shown in Table 6 CovMatch outperforms both MTT-VL 4 and LoRS [45]. demonstrating superior retrieval performance. The result indicates that CovMatch generalizes effectively to the video-text retrieval task, underscoring its robustness across modalities, Its strong performance on this more complex task also highlights the CovMatch's computational efficiency.
Table 6: Performance comparison on video-text retrieval task with N 500. The performance achieved by training on the full dataset is as follows: @R@ 1=15.,, VR@5=38.9, VR@ 10=53.6, TR 1=14.2 TR@5=38, TR@ 10=52.8.
Method
VR@1
VR@5
VR @ 10
TR
TR@5
TR@ 10
Avg
Random
1.1
5.2
9.6
1.8
5.8
10.2
5.6
MTT-VL
2.0
8.3
13.9
2.2
8.8
13.7
8.1
LoRS
2.4
8.7
14.1
2.8
8.5
13.4
8.3
CovMatch
3.0
11.4
19.0
3.4
11.1
18.0
11.0
E More Ablation Studies
E.1
Distillation with Other Networks
We use NFNet the image encoder and BERT the text encoder for main results. In this section, we evaluate the generalization ability of our method by applying it to alternative network architectures for both the image and text encoders. The results, summarized in Table demonstrate that CovMatch consistently outperforms all baseline methods across a range of encoder choices, confirming its robustness to architectural variations.
Table 7: Distillation with various network architectures for both image and text encoders Flickr30k using 100 synthetic pairs. The average retrieval performance is reported. Note that for MTT-VL with the VIT model, the low-rank adaptation matching technique is employed, which involves matching the trajectories of small subset of model parameters using low-rank matrices.
Image encoder
Text encoder
Random
MTT-VL
LoRS
CovMatch
NFNet
BERT
8.6
20.4
27.4
30.5
NF-ResNet
BERT
9.1
14.4
20.7
25.3
NE-RegNet
BERT
7.7
16.6
22.0
26.6
ViT
BERT
11.3
20.7
29.9
35.9
NFNet
DistiBBERT
9.5
24.0
26.0
29.1
E.2 Ablation on Frozeness of Text Encoder
As shown in Figure T our method distills the text modality into synthetic embeddings used inputs to the transformer layers of the text encoder, whereas baseline methods distill text embeddings as inputs to the text projection head, bypassing the text encoder entirely. Consequently, under the baselines, the text encoder is never involved-ent even at evaluation. To enable fairer comparison in which the text encoder can trained during evaluation for the baselines, we adapt our framework (Figure (T to them: instead of feeding distilled text embeddings directly into the projection head, we modify the baselines to distill synthetic embeddings that serve as inputs to the text encoder's transformer layers. In this setup, freeze the text encoder during distillation and unfreeze it only at

---

evaluation. We made this choice for two reasons: (i) unfreezing the text encoder during distillation incurs substantial compute and storage overhead; and (ii) despite the high cost, when we did unfreeze it and included it in trajectory matching, the text-encoder trajectory- matectogymmachhnng loss did not decrease (unlike the projection head), leading to optimization failure and unstable training.
As reported in Table & unfreezing the text encoder only for evaluation yields a small improvement for the baselines, but our method still significantly outperforms them. This observation confirms that baseline methods, which do not incorporate the text encoder during distillation, cannot fully benefit from unfreezing it during evaluation, resulting in only marginal performance gains.
Also, we conduct an additional ablation study that investigates the effect of freezing the text encoder for our method during both the distillation and evaluation stages. As shown in Table freezing the text encoder during evaluation leads to a significant performance drop, highlighting the importance of training only the image encoder and projection layers but also the text encoder for strong image-text retrieval ability. Moreover, we observe that freezing the text encoder during distillation also causes substantial drop in performance (from 38.4 29.4 in average score). These findings suggest that involving the text encoder during distillation is critical for maximizing the effectiveness of our method.
Table 8: Results of unfreezing text encoder during the evaluation stage of baseline method on Flickr30k with N 500.
Method
IR@1
IR@5
IR@ 10
TR@1
TR@5
TR 10
Avg
LoRS (frozen)
10.0
28.9
41.6
15.5
39.8
53.7
31.6
LoRS (unfrozen)
10.0
28.0
41.5
17.7
42.5
57.0
32.8
CovMatch 14.7
38.4
51.4
19.9
46.7
59.5
38.4
Table 9: Effect of freezing the text encoder during the distillation and evaluation stages of CovMatch on Flickr30k with N 500.
Distill
Eval
IR @
IR @5
IR@ 10
TR @
TR @5
TR @ 10
Avg
Frozen
Frozen
4.3
13.4
21.1
6.5
20.6
30.6
16.1
Frozen
Unforzen
9.8
27.8
40.2
15.1
35.8
48.0
29.4
Unfrozen
Frozen
4.5
14.7
22.6
8.4
23.2
33.4
17.8
Unfrozen
Unfrozen
14.7
38.4
51.4
19.9
46.7
59.5
38.4
E.3
Ablation Batch Size
While our original objective aims to match the cross-covariance matrix and mean feature the entire training dataset with those of the synthetic set, computing the exact statistics over the full training set at every distillation step is computationally prohibitive, since the embeddings are continuously updated by the online model. To address this, approximate the full-dataset statistics using mini- batches sampled from the training set at each step. This strategy introduces some variance, but allows the method to remain tractable. We investigate the impact of the size of this mini- in Table TO The result shows clear trend: larger batch sizes consistently yield better retrieval performance. This improvement arises because larger batches produce more accurate estimates of the true gross-covariance enabling stronger alignment between real and synthetic features and thus more effective distillation. However, this performance gain comes at the cost of increased per-iteration computation time, highlighting an inherent trade-off between statistical fidelity and computational efficiency in our cross-covariance matching framework.
E.4 Ablation on Projection Layers
We adopt two-layer projection network with GELU activation, following prior baseline designs. To explore the design space of projection layers, we investigate two key architectural factors: projection dimension (i.e., dimension of projected output) and projection depth (i.c., number of layers in the
25

---

Table 10: Effect of batch size on performance and distillation time on Flickr30k with N = 100. Our default configuration uses batch size of 128.
Batch size
64
128
256
512
1024
Performance
29.1
30.5
30.4
30.9
31.1
Time (Sec/iter)
1.02
1.22
1.36
1.79
3.50
Table 11: Effect of projection dimension on
Table 12: Effect of projection depth on Flickr30k with N=100.
Flickr30k with N=100.
Projection Dim
256
768
2304
4096
Projection Depth
3
Performance
16.1
26.0
30.5
28.9
Performance 29.5 30.5
19.9
projection network). Note that default configuration uses projection dimension of 2304. As shown in TT1 reducing the projection dimension leads to noticeable performance degradation, likely due loss of information. Also, as shown in Table [2] single-layer projection network achieved performance comparable to our default two-layer setup, but deeper networks (3 layers) significantly degraded the performance. These results highlight that both the dimensionality and depth of the projection head play an important role, and an overly compact or overly deep design can hinder retrieval effectiveness.
E.5 Ablation with Single Modality
We conduct an ablation study to assess the impact distilling each modality individually. Specifically, we perform distillation while freezing either the synthetic image text inputs. As shown in Table [3] freezing either modality leads substantial drop performance, indicating that both image and text play essential roles in effective model training. Notably, the performance degradation is more severe when the images are frozen, suggesting that learning the image modality more critical.
Table 13: Ablation study on single-modality distillation using 100 image-text pairs on Flickr30k. The results show that jointly optimizing both modalities crucial for effective performance.
Method
IR @ 1
IR@5
IR@ 10
TR@1
TR@5
TR@ 10
Avg
Image-only
7.1
22.0
32.9
11.0
30.7
43.1
24.5
Text-only
5.8
20.0
30.6
8.2
25.1
35.3
20.8
Both
10.1
28.6
40.9
14.8
38.0
50.6
30.5
F
More Qualitative Results
In Figure 7
we provide more examples of qualitative analysis described in Section 4.3
G
Visualizations
We present visualizations of N 100 distilled image-text pairs from the COCO dataset in Figure The image-text pair on the left represents the initial pair before distillation, while the image-text pair on right corresponds to the distilled pair after distillation. We observe that images are transformed into DeepDreame-like style [48), exhibiting high-frequency components. The text representations are visualized by retrieving nearest caption from the training set in the text embedding space.
26

---

LoRS
CovMatch
? shaggy dog runs in front of three people on
ba man in black shirt with sitting on Hi shoulders
beach
walking the street
gray shirt jumps dune desert weoden rall
?
man in groen shirt is walking en the beach carrying his
sneakers
man grewn t-shirts aneekors while walking alone
seach
66 tWo men wearing swim trunks jump the
grewn walking
holding his shows
at mocersbaly populittd beach
helding walking the
man in black shirt fshing on rocky shore
LoRS
CovMatch
white
feshionable
shirt and white skirt playing tennis
between valleys
with blue racket
the young ady hitting her tennll ball the fleld
00
youing female sennis player in red shirt vitting the ball
00
woman in red shirt and white skirt playing tennis
2
a female soccer player yellow unilorm in soccer
with a blue racket
0
young lady white holdingg a tennis racket standing
pro basketball playar
09
woman people green
LoRS
wearing cowboy hers the fence of rodeo field while contentant the of roping an onimal
2 four horees with cmwbay rders ninning near 6ith home that has no rider
3 a cowboy is viciousy throun about as he attempts to hold on horse tee rictin rodeo competition
two men are trying to rape in bull who is crashing down on one of them. contest comal trynng wrangle free horse
hispanics are
CovMatch
thrown
rodeo scane with big crowd watching thrww mun being thrown from bucking brancox
bwo men are trynng to rope bull who crathing cown on one of them, contest conal
man falling off of buok ng buil at rodec, front of spactators
a cewbbo bucked steer while other man watch from the stands
Figure Comparison of the top five retrieved captions from models trained with LoRS- and GooMatch-generaae synthetic image-tex pairs, given image query. Each model is trained on 500 synthetic pairs, and queries are from the Flickr30K test set. Ground-trutt captions (i.c., five captions paired with the query image in Flickr30K test set) are highlighted in blue.
BBB
bay glasses and cap skateboarde rdingg down some stalrs
men on skateboard jumping ramp
kitchen has window view of staircase
cat laying down on top of a cardboard box
pristine white bathroom with
white tile bathroom with tollet silppers
two people are playing on wil in house
man playing the wil in reom people
modern fridge pulled away from the wall
metallic refrigerator freezer kitchen next to food
with bear standing beautiful woman
wearing
Figure 8: Examples of distilled image-text pairs from the COCO dataset with N = 100 pairs. (Left) Initial image-text pair before distillation. (Right) Distilled image-text pairs.
27

---

H Full Results
Table I4 and Table [3
present the full results for Flickr30k and COCO, respectively. Table T6 shows the results of the cross architercee generalization experiment on Flickr30k using N = 100 synthetic pairs.
Table 14: Performance comparison Flickr30K. The performance trained by full dataset is
@ 1-48.7, IR 5=79.2 IR@ 10-87.2, TR@ 1=61.6, TR@5=85.9, and TR@ 10=91.5%.
Coreset Selection
Distillation LoRS
Pairs
Ratio
Metrics
Random
Herding
K-Center
MTT-VL
CovMatch
IR @1
2.0+0.2
2.2+00.3
2.0+0.2
4.7+-..2
8.3+
10.1+0.2
IR @s
7.5
8.0.0.88
7.6.0.77
15.7.10.5
24.1100.2
28.6.10.4
IR @ 10
12.6.1.00
13.41.1.
13.011.0
24.6.11.00
35.1110.3
40.911..
100
0.3 %
TR@:
3.340.11
3.0.1..7
2.8.0.55
9.910.3
11.810.2
14.8.10.9
TR TR@5
10.4:0.5
9.911.0
9,741.0
28.3=00.5
35.84:0.6
38.0:10.4
TR@ 10
16.0#0.6
15.6#1.1
16.4=099
39.1=
49.2+0..5
50.6+0.6
IR @1
3.340..2
3.0
3.240.11
4.6.1..9
8.6
12.3:10.4
IR @5
11.5+:0.4
11.3+0.4
11.1=0.5
16.0-11.6
25.3+00.2
33.6+10.3
@ 10
18.440..5
18.340.7
17.7+0.4
25.5++2.6
36.6+-0.3
45.8+10.2
200
0.7
TR@1
5.740..5
4.710.4
5.310.7
10.2:10.8
14.5::0.5
17.410.5
TR@5
15.810.5
15.410.4
15.2=0.9
28.721.
38.710.55
41.7:1055
TR 10
23.9
22.91100
23.210.4
41.911.9
$3,410.5
55,810.55
IR @1
6.9+0.44
6.840.4
6.9+0.2
6.6+0.3
10.0
14.7+-0.3
@
21.011.4
20.810.5
22.110.4
20.2.11.2
28.910.7
38.410.4
IR 10
31.2.10.6
30.910.6
32.2.1..6
30.0.12.1
41.6.10.6
51.4.0.3
500
1.7 %
TR@1
10.0.0.6
9.310.6
10.6.10.7
13.3.0.6
15.5.0.7
19.910,6
T
28.0+0.88
26,4+0.5
29.5+0.7
328+1188
39.8+0.4
46.7+0099
TR@ 10
38.7#0.9
36,8+0.7
40.6=#0.3
46.8=0088
53,7+00.
59.5+0.7
Table 15: Performance comparison on COCO. The performance trained by full dataset is IR@ 1=25.1, IR @ 5=53.9, IR @ 10=67.5, TR @ 1=33.0, TR @ 5=62.8, TR@ 10=75.0.
Coreset Selection Herding
Distillation LoRS
Pains
Ratio
Metrics
Random
K-Center
MTT-VI.
CovMatch
IR @ I
0.7
0.7++0.1
0.7+0.11
1.3+0.11
1.8+0.1
2.8+10.1
IR @5
2.8=0.1
2.9:0.11
3.2+0.11
5,410.3
7.1=0.2
10.5=:0.2
IR @ 10
5.1-0.3
5.3++0.2
6.0+0.22
9.5+-0.5
12,240.2
17.7-10.3
100
0.1 %
@R@1
1.0=0.1
1.100.1
0.9+0.1
2.5+0.3
3.3-1.2
3.81:0.1
TR
4.0=0.2
4.1:10.2
4.2+00.2
10.0+0.5
12.240.3
13.1-10.3
TR@ 10
6.9-0.3
6.8+0.22
7.6+0..3
15.7+0.4
19.6#0.3
21.1::0.2
IR@1 1
1.1=0.1
1.240.1
1.20.11
1.700.1
2.410.1
3.8
IR@5
4.6=0.33
4.7.0..1
5.1
6.5+0..4
9.3
13.4:1.11
IR@10
8.3-0.6
8.5+0.2
8.940.2
12.3+0.8
15.5+0.2
21.8:10.2
200
0.2 %
T
1.7=0.2
16600.2
1.940.1
3.3+0.2
4.3.1..1
5.311..2
TR@5
6.5=0.5
6.6.0..2
6.7+0.2
11.9+0.6
14.240.3
17.3+0.2
TR@10 10
11.140.66
11.240.4
11.6.0.3
1.,4
22.6:10.2
27.0
IR@1 I
2.2-0.2
2.340.11
2.400.2
2.5+0.55
2.8-10.2
5.400.1
IR@5
8.8=0.6
8.8+0.11
9.0+0.33
8.9+0.7
9.9-0.5
18.0
IR@10
14.94..8
14.8-10.2
15.410.4
15.8+11.5
16.5-10.7
28.2-10.1
500
0.4 %
TR@1
3.5-0.4
2.940.2
3.6+0.22
5,0+0.44
5.340.5
8.110.33
TR@5
11.940.77
11.2-:0.4
12.410.3
17.2+113
18.3111.5
23.5-10.3
TR@
19.2+0.5
18.9+0.3
20.0
26.0=1.9
27.9+1.4
34.611.66
28

---

Table 16: Cross-architccture cvaluation results on Flickr30k with N
100 synthetic pairs.
Method
Text Encoder
Image Encoder
[R@1
IR@5
IR @ 10
TR@:
TR@5
TR @ 10
NFNet
2.1
7.4
12.2
3.6
9.6
15.6
NF-ResNet
2.2
8.0
13.4
3.0
10.1
16.4
BERT
NF-RegNet
2.2
7.6
12.4
2.6
9.7
15.3
ViT
2.8
9.3
15.1
4.2
13.3
19.8
Random
NFNet
8.6
14.4
3.2
16.4
NF-ResNet
2.6
9.6
15.3
3.6
11.4
18.4
DistilBERT
NF-RegNet
2.3
8.0
13.4
2.9
10.0
15.4
ViT
2.9
10.6
17.4
4.1
13.1
20.6
NFNet
4.7
15.7
24.6
9.9
28.3
39.1
NE-ResNet
1.8
7.0
11.9
2.9
10.5
16.4
BERT
NE-RegNet
10.6
2.9
15.0
ViT
2.1
7.8
13.1
3.9
12.2
18.6
MTT-VL
NFNet
22.5
10.2
29.0
40.8
NF-ResNet
1.2
5.0
9.0
3.2
10.1
16.7
DistilBERT
NF-RegNet
0.8
3.7
7.1
3.1
10.2
17.0
VIT
11.0
3.2
10.9
17.7
NFNet
7.4
22.8
34.0
14.2
37.9
52.2
NF-ResNet
12.1
2.9
18.0
BERT
NF-RegNet
1.6
6.4
11.2
2.5
11.0
17.7
ViT
2.0
7.3
12.7
3.2
11.7
19.0
LoRS
NENet
5.2
17.3
27.2
12.5
34.0
45.0
NF-ResNet
1.7
6.3
10.8
4.0
11.9
18.8
DistilBERT
NF-RegNet
1.3
5.2
8.9
3.6
12.0
18.6
ViT
1.4
5.8
9.9
3.2
12.7
20.4
NENet
28.9
41.0
13.6
36.9
50.6
NE-ResNet
4.7
15.4
23.4
5,5
17.3
26.6
BERT
Ne-RegNet
3.9
13.8
21.7
5,6
16.8
26.0
ViT
3.7
13.5
21.7
5.2
18.6
28.0
CovMatch
NFNet
8.5
25.1
36.7
12.6
33.4
46.2
NE-ResNet
5.3
16.2
25.1
6.2
17.6
26.4
DistilBERT
NE-RegNet
21.9
5.3
25.6
ViT
3.0
11.9
19.5
5.0
16.3
24.6
29


Reviews:
[{"review_id": "SWNBhY497g", "reviewer": "Reviewer_xbh4", "summary": "This paper introduces CovMatch, a novel framework for multimodal dataset distillation (MDD). The authors first unfreez the text encoder for learning efficacy. By analyzing the linearized model, CovMatch addresses the MDD problem by matching the covariance matrices between modalities. With additional online model updating technique, CovMatch achieves both significant efficiency and performance on multimodal retrieval tasks.", "questions": "1. The text encoder is unfrozen in only distillation stage or both distillation and network training (evaluation) stages? The latter one involves unfair comparison to baselines and the authors could unfreeze the text encoder of the baselines during evaluation.\n2. There seems not to be a clear ablation comparison of CovMatch without unfrozen encoder (due to Q1). If the blue bars in Fig.5(c) are, their performance is comarable to baselines, which reduces the performance contribution of CovMatch itself.\n3. In Sec.2.2, the authors discuss the limitation of MDD baselines at performance scaling (starts from line 142). Thus, a comparison at 1000 pairs would support the claim.\n\nI would be glad to raise my rating if my concerns are addressed.", "limitations": "yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths\n\n- The paper is well-motivated. The proposed approach is reasonable, with both theoretical novelty and interesting technical contributions.\n- The empirical evaluation is strong, exhibiting both distillation efficiency and good performance.\n- The writing is clear.\n\n## Weakness\n\nThe main weaknesses come from the ablation. The paper's central claim is that unfreezing the text encoder helps MDD and improves the performance when synthetic data scales up. This, however, could potentially lead to unfair comparison. Please see my \"Questions\" for details. Overall, I think the method is self-contained and addressing these concerns would enhance the clarity of the paper.", "was_revised": true, "final_justification": "I appreciate the authors' feedback. My questions are sufficiently addressed, so I would keep my positive rating.", "rebuttal": "# Response to Reviewer xbh4\n\nWe sincerely appreciate the constructive feedback provided by the reviewer. We have made every effort to address the concerns and questions raised. We hope that this response addresses the reviewer's questions.\n\n>**1. The text encoder is unfrozen in only the distillation stage or both the distillation and network training (evaluation) stages? The latter one involves unfair comparison to baselines and the authors could unfreeze the text encoder of the baselines during evaluation.**\n\nWe thank the reviewer for raising this important point. We unfreeze the text encoder during **both the distillation and evaluation stages** in our method. We understand the reviewer's concern regarding the fairness of comparison. However, it is infeasible to unfreeze and fine-tune the text encoder during the evaluation stage for existing baselines due to structural limitations in how they treat the distilled text.\n\nTo elaborate, consider a baseline method that distills a synthetic dataset (e.g., 100 image-text pairs from Flickr). When training with the distilled set at the evaluataion stage, the image component (e.g., shape `[100, 3, 224, 224]`) is passed through the image encoder, while the distilled text component (e.g., `[100, 768]`) is directly used as input to the text projection layer, bypassing the text encoder entirely. As a result, the text encoder is never involved or updated during training. Once training is complete, the learned image encoder and projection layers are used in conjunction with the original frozen text encoder (e.g., pretrained BERT) to process real test samples.\n\nHence, to ensure a fair comparison, we adapt our evaluation framework (Fig. 1) for the baseline methods. Specifically, instead of distilling text embeddings as direct inputs to the projection layer, we modify the baselines to newly distill synthetic embeddings that serve as inputs to the transformer layers within the text encoder. This allows the text encoder to be effectively trained on the synthetic set generated by the baselines. In this modified setup, we keep the text encoder frozen during distillation and unfreeze it only during the evaluation stage. This design choice is driven by two key challenges: (1) unfreezing the text encoder during distillation incurs substantial computational and storage overhead; and (2) despite the high cost, we attempted to unfreeze the text encoder and apply trajectory matching that includes it, but observed that trajectory matching loss for the text encoder fails to decrease, unlike the projection layers. This results in optimization failure and unstable training dynamics.\n\nAs shown in Table R1, unfreezing the text encoder for evaluation improves baseline performance slightly, but our method still significantly outperforms them. This observation confirms that baseline methods, which do not incorporate the text encoder during distillation, cannot fully benefit from unfreezing it during evaluation, resulting in only marginal performance gains. \n\nTable R1. Results of unfreezing text encoder during the evaluation stage of baseline method on Flickr30k with N=500. \n| Method           | IR@1 | IR@5 | IR@10 | TR@1 | TR@5 | TR@10 | Avg  |\n|------------------|------|------|-------|------|------|-------|------|\n| LoRS [1] (frozen)    | 10.0 | 28.9 | 41.6  | 15.5 | 39.8 | 53.7  | 31.6 |\n| LoRS [1] (unfrozen)  | 10.0 | 28.0 | 41.5  | 17.7 | 42.5 | 57.0  | 32.8 |\n| CovMatch         | 14.7 | 38.4 | 51.4  | 19.9 | 46.7 | 59.5  | 38.4 |\n\n[1] Xu Y, Lin Z, Qiu Y, et al. Low-Rank Similarity Mining for Multimodal Dataset Distillation. ICLR 2024.\n\n>**2. There seems not to be a clear ablation comparison of CovMatch without unfrozen encoder (due to Q1). If the blue bars in Fig.5(c) are, their performance is comarable to baselines, which reduces the performance contribution of CovMatch itself.**\n\nThank you for raising this important point. We would like to first clarify a possible misunderstanding regarding the interpretation of Figure 5(c). The term *“fixed”* in Figure 5(c) does not refer to freezing the text encoder during the evaluation stage, as discussed in Q1. Instead, it refers to whether the online model—i.e., the image and text encoders—is updated during the distillation process.\n\nTo prevent the synthetic data from overfitting to a static encoder, our method introduces an online model update strategy during distillation: before each distillation step, the encoders are updated using a batch of real data. In Figure 5(c), the *blue bars* represent the setting where both the image and text encoders are fixed at their pretrained initialization throughout the distillation phase. In contrast, the *orange* or *green bars* represent settings where encoders are updated with syntehtic or real data during distillation. Importantly, in all three settings shown in Figure 5(c), the text encoder is unfrozen during the evaluation stage. Therefore, this figure should not be interpreted as an ablation on whether text encoder is frozen or not, but instead as an analysis of the effect of updating the online model during distillation. \n\nTo more directly examine the role of the text encoder, we conducted an additional ablation study that investigates the effect of freezing it during both the distillation and evaluation stages. As shown in Table R2, freezing the text encoder during evaluation leads to a significant performance drop, highlighting the importance of training not only the image encoder and projection layers but also the text encoder for strong image-text retrieval ability. Moreover, we observe that freezing the text encoder during distillation also causes a substantial drop in performance (from 38.4 to 29.4 in average score). These findings suggest that involving the text encoder during distillation is critical for maximizing the effectiveness of our method.\n\nTable R2. Effect of freezing the text encoder during the distillation and evaluation stages of CovMatch on Flickr30k with N=500.\n| Distill  | Eval     | IR@1 | IR@5 | IR@10 | TR@1 | TR@5 | TR@10 | Avg  |\n|----------|----------|------|------|-------|------|------|-------|------|\n| Frozen   | Frozen   | 4.3  | 13.4 | 21.1  | 6.5  | 20.6 | 30.6  | 16.1 |\n| Frozen   | Unfrozen | 9.8  | 27.8 | 40.2  | 15.1 | 35.8 | 48.0  | 29.4 |\n| Unfrozen | Frozen   | 4.5  | 14.7 | 22.6  | 8.4  | 23.2 | 33.4  | 17.8 |\n| Unfrozen | Unfrozen | 14.7 | 38.4 | 51.4  | 19.9 | 46.7 | 59.5  | 38.4 |\n\nFor a fair comparison where text encoder is unfrozen in both our method and baselines, we refer the reviewer to **Table R1 in Response 1**, which directly addresses this point. We will include this ablation study in the revised version to support a fair comparison with existing baselines.\n\n\n>**3. In Sec.2.2, the authors discuss the limitation of MDD baselines at performance scaling (starts from line 142). Thus, a comparison at 1000 pairs would support the claim.**\n\nWe thank the reviewer for this helpful suggestion. We agree that comparison at more than 500 pairs is required to support our strong ability at performance scaling. As shown in Table R3, our method continues to improve as N increases until N=2000. In contrast, baseline methods such as MTT-VL [2] and LoRS [1] tend to saturate at smaller scales, typically around N=500, and in some cases even exhibit performance degradation as N increases further. We note that we conducted extensive hyperparameter tuning for these baselines at each scale, including learning rates, maximum start epoch, and similarity rank. Despite these efforts, the observed degradation suggests that the optimization processes of the baselines are inherently less effective on large number of synthetic pairs.\n\nThese results highlight the strong scalability of our method compared to existing approaches. We will include this extended comparison, along with scaling curves, in the revised version of the paper.\n\nTable R3. Scalability comparison on Flickr30K.\n| Method   | N=100 | N=200 | N=500 | N=1000 | N=1500 | N=2000 |\n|----------|-------|-------|-------|--------|--------|--------|\n| Random   |  8.6  | 13.1  | 22.6  | 31.7   | 38.2   | 42.1   |\n| MTT-VL   | 20.4  | 21.2  | 25.0  | 19.0   | 18.2   | 18.2   |\n| LoRS     | 27.4  | 29.5  | 31.6  | 31.3   | 29.6   | 27.3   |\n| CovMatch | 30.5  | 34.4  | 38.4  | 41.2   | 41.8   | 43.7   |\n\n[2] Wu X, Zhang B, Deng Z, et al. Vision-language dataset distillation. TMLR 2024."}, {"review_id": "TSZUFtE8c9", "reviewer": "Reviewer_NCKk", "summary": "This paper presents CovMatch, a multimodal dataset distillation method for image-text contrastive learning. The key improvements over existing approaches include: (1) avoiding trajectory storage and unrolling through cross-covariance matching, which provides a closed-form solution for the inner optimization; (2) enabling joint optimization of both image and text encoders rather than freezing the text encoder; and (3) achieving computational efficiency gains (1.2 sec/it vs 16.9 sec/it compared to prior methods). The approach reformulates the bi-level optimization as cross-covariance matrix alignment between real and synthetic features.", "questions": "## Questions\n\n1. The paper claims computational efficiency by fixing encoders and only training projection layers to obtain a more tractable bi-level objective. However, given the extensive work in the VLM community on projection layer design and feature mapping [3], have the various hyperparameters, complexity considerations, and architectural choices for the projection layers been sufficiently explored and discussed?\n\n2. This represents my main concern regarding the relationship with MTT-VL [1]. While the paper does compare with MTT-VL, I believe **the discussion of similarities and differences should be expanded in Section 2.2, rather than only discussing LORS.** The innovation appears somewhat limited relative to MTT-VL - both methods employ trainable projection layers after the two encoders. Although the InfoNCE linearization transformation is mathematically elegant, the paper would benefit from additional experiments or more insightful analysis to strengthen the theoretical contributions beyond what appears to be primarily a computational optimization trick of \"fix encoders during distillation, then train them afterwards.\" The fundamental algorithmic differences from [1] beyond this computational efficiency gain need to be more clearly articulated with supporting experimental evidence.\n\n3. Given the significant computational savings demonstrated by this method, could it be extended to accelerate video-text matching scenarios involving multiple video frames [4]? This exploration would be valuable and could potentially improve the evaluation of this work.\n\n4. What are the conditions required for the linearization in lines 167-173? It appears that equation (5) holds under two conditions: \"large τ / bounded inner product magnitudes for flatting softmax\" and \"zero-mean preprocessing of features.\" This needs to be clarified to meet NeurIPS community standards. I suggest placing the main conclusions in Section 3.1, with step-by-step derivations in the appendix to help readers from other fields follow along, as this work could potentially inspire other domains.\n\n# minor revision\n1. Add commas after formulas (if the following word is lowercase like \"where\") or periods (if the following word is capitalized)\n2. Considering the method's effectiveness, Table 1 could be accompanied by a 2D plot to visualize performance with other baselines for more clear representation, as this method should be on the Pareto frontier.\n3. The derivation in lines 167-173 **could be written in a more easy-to-follow manner,** particularly the transition to the cross-covariance matching objective -Tr(G_v C^D G_l^T) + (ρ/2)||G_v^T G_l||_F^2, and step-by-step derivations should be added in the appendix.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Strengths:**\n\nMultimodal dataset distillation is an important topic [1][2], and this work addresses a limitation of existing approaches by enabling text encoder optimization during distillation. The motivation is well-supported through convincing experimental analysis showing that frozen text encoders create semantic alignment bottlenecks, as evidenced by the t-SNE visualizations and intra/inter-similarity analyses in Figure 2. The theoretical derivation leading to the cross-covariance matching objective is mathematically sound and provides an elegant closed-form solution that significantly reduces computational overhead compared to trajectory matching methods.\n\n**Weaknesses:**\n\nWhile the paper does compare with MTT-VL [1], the innovation appears quite limited relative to this prior work. The theoretical foundations are not original - linear contrastive learning comes from prior work [2], and the importance of cross-covariance has already been recognized in multimodal learning. The core insight is essentially a computational trick: \"fix encoders during distillation, then train them afterwards.\" This feels more like an engineering solution rather than an algorithmic breakthrough, and the fundamental differences from [1] beyond this computational optimization are not sufficiently substantial.\n\nAdditionally, given the extensive work in the VLM community on projection layer design and feature mapping [3], the paper would benefit from more thorough analysis of projection layer architectural choices, hyperparameters, and complexity considerations.\n\n**References:**\n[1] Wu X, Zhang B, Deng Z, et al. Vision-language dataset distillation[J]. TMLR, 2024.\n[2] Xu Y, Lin Z, Qiu Y, et al. Low-Rank Similarity Mining for Multimodal Dataset Distillation[C]//International Conference on Machine Learning. PMLR, 2024: 55144-55161.\n[3] Li J, Li D, Savarese S, et al. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models[C]//International conference on machine learning. PMLR, 2023: 19730-19742.\n[4] https://huggingface.co/datasets/TempoFunk/webvid-10M", "was_revised": true, "final_justification": "I've read the rebuttal and raised my score  accordingly", "rebuttal": "# Response to Reviewer NCKk\n\nWe sincerely appreciate the constructive feedback provided by the reviewer. Below, we addresses the reviewer's questions.\n\n>**1. While the paper does compare with MTT-VL [1], I believe the discussion of similarities and differences should be expanded in Section 2.2....The paper would benefit from a clearer articulation of the algorithmic differences from MTT-VL and stronger experimental or theoretical justification to support the claimed novelty.**\n\nWe thank the reviewer for the comment and the opportunity to clarify the novelty of our method.\n\nFirst, we acknowledge that Section 2.2 primarily compares our method only with LoRS [2]. To address this more thoroughly, we have extended our analysis of the pairwise similarity (Fig. 2b) and scalability (Fig. 2c) plots to include MTT-VL. As shown in Table R1, CovMatch consistently exhibits a larger gap between intra- and inter-pair similarity across synthetic set sizes, compared to MTT-VL and LoRS. We also report additional scalability results with MTT-VL in Table R3 of the Reviewer xbh4 response, further supporting the strong scalability of CovMatch. These comparisons will be incorporated into the revised manuscript.\n\nTable R1. Extended comparison of gap between intra- and inter-pair average similarity.\n|Method|N=100|N=500|\n|-|-|-|\n|MTT-VL|0.387|0.284|\n|LoRS|0.389|0.378|\n|CovMatch|0.575|0.592|\n\nSecond, we would like to highlight the key algorithmic distinctions that differentiate our method from MTT-VL.\n* **Distillation with large pretrained encoders:** A major limitation of MTT-VL is the difficulty to incorporate unfrozen large pretrained encoders such as BERT due to the high computational cost. In contrast, CovMatch introduces a lightweight cross-covariance matching objective, enabling efficient distillation with large, unfrozen encoders. The reviewer may be concerned that our method also assumes frozen encoders, which could diminish the distinction from prior work. However, we emphasize that while we assume frozen encoders to derive the cross-covariance matching objective, we do not fix them during distillation. Instead, we employ an online model update strategy to adapt the distilled set to evolving, unfrozen encoders.\n* **Distillation of text data:** Prior methods distill synthetic text that is directly fed into the text projection layer, which forces the text encoder to remain frozen. In contrast, our method distills inputs to the transformer layers of the text encoder, which maximizes the performance gain achievable through distillation.\n* **Cross-covariance matching algorithm:** We find that aligning the cross-covariance matrices between real and synthetic pairs is critical for effective multimodal distillation. However, as showin in Table R2, the baselines fail to successfully align the cross-covariance, and in fact, MTT-VL even increases the misalignment from initial state. In contrast, our method directly maximizes the alignment using cross-covariance matching loss term, resulting in state-of-the-art performance.\n\nTable R2. Comparison of the Frobenius distance between the cross-covariance matrices of the real and distilled sets, $||C^\\mathcal{T}-C^\\mathcal{S}||_F$. \n|Method| Frobenius distance |\n|-|-|\n|Random (initial)|53.9|\n|MTT-VL|157.6|\n|LoRS |57.1|\n|CovMatch| 7.8 |\n\nWe introduce a scalable and principled framework for multimodal dataset distillation that departs from conventional practices in the field. We believe this work constitutes a meaningful advancement in the field, beyond simple computational trick.\n\n[1] Wu X, Zhang B, Deng Z, et al. Vision-language dataset distillation. TMLR 2024.\n\n[2] Xu Y, Lin Z, Qiu Y, et al. Low-Rank Similarity Mining for Multimodal Dataset Distillation. ICLR 2024.\n\n\n>**2. The paper claims computational efficiency by fixing encoders and training only projection layers for a tractable bi-level objective. But given extensive work on projection design in the VLM community [3], have the architectural choices and hyperparameter tuning been sufficiently explored?**\n\nWe thank the reviewer for this insightful suggestion. In our implementation, we adopt a two-layer projection network with GELU activation, following prior baseline designs. While this setup has demonstrated strong performance, we agree that exploring the architectural design space of projection layers is a meaningful direction.\n\nTo address this, we conducted ablation studies on two aspects: projection dimension (i.e., dimension of projected output) and projection depth (i.e., number of layers in the projection network). Note that our default configuration uses a projection dimension of 2304. As shown in Table R3, reducing the projection dimension (e.g., 768, 256) leads to noticable performance degradation, likely due to loss of information. Also, as shown in Table R4, single-layer projection network achieved performance comparable to our default two-layer setup, but deeper networks (3 layers) significantly degraded the performance. These results suggest that careful design of the projection network is essential for achieving strong performance.\n\nTable R3. Effect of projection dimension on Flickr30k with N=100.\n|Projection Dim|256|768|2304|4096|\n|-|-|-|-|-|\n|Performance (avg)|16.1|26.0|30.5|28.9|\n\nTable R4. Effect of projection depth on Flickr30k with N=100.\n|Projection Depth|1|2|3|\n|-|-|-|-|\n|Performance (avg)|29.5|30.5|19.9|\n\nIn this context, the approach proposed in [3]—which freezes the encoder and instead trains a more expressive projection module, such as a transformer-based adapter (e.g., Q-former)—offers a promising alternative. We believe integrating such ideas into distillation setting, where they have yet to be systematically explored, could open up meaningful future direction. We thank the reviewer again for this valuable suggestion.\n\nWe will incorporate these ablation results and the related discussion in the revised version.\n\n[3] Li J, Li D, Savarese S, et al. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. ICML 2023.\n\n>**3. Given the significant computational savings demonstrated by this method, could it be extended to accelerate video-text matching scenarios involving multiple video frames [4]?**\n\nWe thank the reviewer for this insightful suggestion. We conducted additional experiments on the video-text retrieval task using pretrained VideoResNet (r3d_18) [5] as the video encoder and BERT as the text encoder. Since it is computationally impractical to distill full WebVid-10M dataset [4] within our current time constraint, we instead used reduced dataset which consists of 49K training video-text pairs and 1K test pairs. For the synthetic set, we distilled 100 video-text pairs, using 8 sampled frames per video.\n\nAs shown in Table R5, CovMatch outperforms both MTT-VL [1] and LoRS [2], demonstrating superior retrieval performance. The result indicates that CovMatch generalizes effectively to the video-text retrieval task, underscoring its robustness across modalities. Its strong performance on this more complex task also highlights the CovMatch’s computational efficiency.\n\nTable R5. Performance comparison on video-text retrieval task with N=100.\n|Method|Random|MTT-VL|LoRS|CovMatch|\n|-|-|-|-|-|\n|Performane (avg)|3.0|5.2|5.4|**7.0**|\n\nWe will incorporate this new experiment and analysis into the revised version of the paper to highlight the potential applicability of our method to video-text retrieval. \n\n[4] Bain, Max, et al. Frozen in time: A joint video and image encoder for end-to-end retrieval. ICCV 2021.\n\n[5] Tran, Du, et al. A closer look at spatiotemporal convolutions for action recognition. CVPR 2018.\n\n\n>**4. What are the conditions required for the linearization in lines 167-173?**\n\n>**Minor Revision 3. The derivation for the transition to the cross-covariance matching objective $-Tr(G_v C^D G_l^T) + (\\rho/2)||G_v^T G_l||_F^2$.**\n\nTo clarify, in our main analysis, we do not impose any assumptions to directly linearize the InfoNCE loss. Instead, we adopt the linear contrastive loss defined in Eq. (5) as a standalone objective to simplify the bi-level optimization in multimodal distillation. This linear form enables a natural connection to the cross-covariance alignment term in Eq. (6), ultimately leading to the simplified trace-based objective in Eq. (8), $Tr((\\mathcal{C}^{\\mathcal{T}})^\\top \\mathcal{C}^{\\mathcal{S}} )$.\n\nThat said, as the reviewer correctly points out, the linear loss in Eq. (5) can also be derived as a high-temperature approximation of the InfoNCE loss. \nWhen $\\tau$ is large, the softmax flattens, and a first-order Taylor expansion gives:\n$$\\log(\\sum_{j\\neq i} \\exp(s_{ij}/\\tau))\\approx \\frac{1}{\\tau(|\\mathcal{D}|-1)}\\sum_{j\\neq i} s_{ij}+\\log (|\\mathcal{D}|-1).\n$$\nSubstituting this into the InfoNCE loss (1) leads to the linearized form Eq. (5). This derivation provides further justification for using the linear objective in Eq. (5).\n\nFurthermore, the equivalence between Eq. (5) (the linear contrastive loss) and Eq. (6) (the cross-covariance matching objective) holds without any assumptions. The similarity terms in Eq. (5) can be expressed using the projected embeddings:\n$$s_{ij} := (G_v h_v^i)^\\top (G_l h_l^j)=\\sum_{k=1}^z (g_{v_k})^\\top h_v^i ({h_l^j})^\\top g_{l_k},$$\nwhere $G_v^\\top=[g_{v_1},\\dots, g_{v_z}]$ and $G_l^\\top=[g_{l_1},\\dots, g_{l_z}]$.\n\nThe cross-covariance matrix $C^\\mathcal{D}$ is defined as:\n$$\nC^{\\mathcal{D}}:=\\frac{1}{(|\\mathcal{D}|-1)}\\sum_{i=1}^{|\\mathcal{D}|}(h_v^i-\\mu_{h_v})(h_l^i-\\mu_{h_l})^\\top,\n$$\nand the term $-Tr(G_v C^{\\mathcal{D}} G_l^\\top)$ expands as\n$$\n-\\sum_{k=1}^z g_{v_k}^\\top  C^{\\mathcal{D}}g_{l_k}.\n$$\nBy aligning the summation forms, we establish the equivalence between the linear contrastive loss and cross-covariance matching objective.\nWe will revise Section 3.1 for clarity and include the full derivation in the appendix. \n\n>**We will incorporate Minor Revision 1 and 2 accordingly.**"}, {"review_id": "kalwovpqC8", "reviewer": "Reviewer_29i1", "summary": "The paper introduces CovMatch, a framework for multimodal dataset distillation that addresses the key challenge of cross-modal alignment in vision-language models. The core innovations include:\n- A cross-covariance matching loss that aligns the cross-covariance matrices of real and synthetic data, enabling joint optimization of image and text encoders.\n- A feature matching regularizer to ensure distribution consistency within each modality.\n- An online model update strategy to prevent overfitting during distillation.\nExperimental results on Flickr30K and COCO demonstrate that CovMatch outperforms state-of-the-art methods, while reducing computational costs significantly.", "questions": "See Weakness Part. If the concerns can be resolved, I will reconsider the final rating.", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths\n\n- Novel Approach to Multimodal Distillation: By relaxing the assumption of frozen text encoders, CovMatch addresses a critical bottleneck in prior work. The cross-covariance alignment theoretically grounds the method in capturing semantic dependencies between modalities.\n- Comprehensive Evaluation: The paper validates CovMatch across multiple datasets, metrics (IR@K, TR@K), and model architectures (NFNet, ViT, BERT, DistilBERT). Ablation studies on scaling factors, feature matching weights, and update strategies provide clear insights into the method’s effectiveness.\n\nWeaknesses\n\n- Theoretical Depth: While the cross-covariance alignment is motivated by linear contrastive learning, the paper lacks a formal proof of identifiability or convergence guarantees. For instance, it is unclear how the method handles non-linear feature spaces or complex causal structures in latent variables.\n\n-  Limited Modality Scope: The current framework focuses on image-text pairs, ignoring other multimodal scenarios (e.g., video-text, audio-visual). Extending CovMatch to these domains would strengthen its generality.\n\n- Insufficient comparison methods: Only two dataset distillation methods are used for performance comparison, and only one is published in 2024. \n\n- Insufficient related works: For Multimodal Distillation, it is weird no studies for image-text retrieval are mentioned, such as [1-4]. \n\n[1] Karpathy, Andrej, and Li Fei-Fei. \"Deep visual-semantic alignments for generating image descriptions.\" Proceedings of the IEEE conference on computer vision and pattern recognition. 2015.\n\n[2] Li, Kunpeng, et al. \"Visual semantic reasoning for image-text matching.\" Proceedings of the IEEE/CVF international conference on computer vision. 2019.\n\n[3] Liu, Chunxiao, et al. \"Graph structured network for image-text matching.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020.\n\n[4] Wang, Haoran, et al. \"Coder: Coupled diversity-sensitive momentum contrastive learning for image-text retrieval.\" European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022.", "was_revised": true, "final_justification": "The additional theoretical analysis and more experiments on more modalities resolve my main concerns, I raise my score. I also hope the theoretical analysis given in rebutal will be inporporated into the final version of paper if accepted.", "rebuttal": "# Response to Reviewer 29i1\n\nWe sincerely appreciate the constructive feedback provided by the reviewer. We have made every effort to address the concerns and questions raised. We hope that this response addresses the reviewer's questions.\n\n>**1. Theoretical Depth: While the cross-covariance alignment is motivated by linear contrastive learning, the paper lacks a formal proof of identifiability or convergence guarantees. For instance, it is unclear how the method handles non-linear feature spaces or complex causal structures in latent variables.**\n\nThank you for this valuable comment. As the reviewer noted, our formulation of multimodal dataset distillation as a cross-covariance alignment problem is grounded in a linearized contrastive learning framework. Specifically, at each distillation iteration, we fix the image and text encoders and consider training the final linear projection layers to convergence. To mitigate overfitting to a static model state, we incorporate an online model update step: before each distillation step, the encoders are updated using a small batch of real data via a single gradient step.\n\nOn the theoretical side, our linearized formulation enables a closed-form solution for the inner optimization. The optimal projection layers $G\\_v$ and $G\\_l$ minimize the following objective,\n$$-Tr(G\\_v C^\\mathcal{S} G\\_l^\\top ) + \\frac{\\rho}{2} \\||G\\_v^\\top G\\_l\\||\\_F^2 = \\frac{\\rho}{2}\\||G\\_v^\\top G\\_l - \\frac{1}{\\rho} C^\\mathcal{S} \\||\\_F^2 - \\frac{1}{2\\rho}\\||C^\\mathcal{S}\\||\\_F^2,$$\nwhich admits a closed-form solution:\n$$\\hat{G}\\_v^\\top \\hat{G}\\_l = \\frac{1}{\\rho} C^\\mathcal{S}.$$\nThis provides identifiability and convergence guarantees for the projection layers under the linear setting.\n\nWhile our theoretical formulation assumes linear representations—i.e., $z\\_v=G\\_vh\\_v$ and $z\\_l=G\\_l h_l$, with  $h\\_v=f\\_v(x\\_v;\\theta\\_v)$ and $h\\_l=f\\_l(x\\_l;\\theta\\_l)$ for fixed encoder parameters—the insights extend more broadly to non-linear settings. In particular, recent work has shown that fine-tuning large neural networks (e.g., foundation models) often operates in the Neural Tangent Kernel (NTK) regime [1], where the training dynamics are well approximated by a linear model over a high-dimensional feature space derived from the model's gradients [2]. This connection suggests that our analysis in the linear regime can meaningfully inform behavior in more general, non-linear settings.\n\nEmpirically, our experiments in Section 4 confirm that this formulation generalizes well: CovMatch, which is designed based on the linearized analysis, achieves state-of-the-art performance even when used with non-linear, high-capacity encoders like NFNet and BERT. The intuition is that synthetic data that effectively trains the linear projection layer tends to also provide strong training signals for the full network, particularly when the network is decomposed into a feature extractor and a task-specific head.\n\nWe will incorporate these theoretical insights and supporting references in the revised manuscript.\n\n[1] Jacot, A., Gabriel, F., and Hongler, C. Neural tangent kernel: Convergence and generalization in neural networks. NIPS 2018.\n\n[2] Malladi, S., Wettig, A., Yu, D., Chen, D., and Arora, S. A kernel-based view of language model fine-tuning. ICML 2023. \n\n\n>**2. Limited Modality Scope: The current framework focuses on image-text pairs, ignoring other multimodal scenarios (e.g., video-text, audio-visual). Extending CovMatch to these domains would strengthen its generality.**\n\nWe thank the reviewer for this insightful suggestion. To investigate the generality of CovMatch beyond image-text settings, we conducted additional experiments on the video-text retrieval task. We used a pretrained Video ResNet (r3d_18) [3] as the video encoder and BERT as the text encoder. Due to the time constraint, we constructed a reduced version of the WebVid-10M dataset [4], consisting of 49K training video-text pairs and 1K test pairs.\n\nAs shown in Table R1, CovMatch consistently outperforms both MTT-VL [5] and LoRS [6] across nearly all evaluation metrics, demonstrating superior retrieval performance. These results indicate that CovMatch generalizes effectively to the video-text retrieval task, highlighting its strength as a robust and adaptable framework for multimodal dataset distillation across diverse modalities.\n\nTable R1. Performance comparison on video-text retrieval task.\n| Method    | VR@1 | VR@5 | VR@10 | TR@1 | TR@5 | TR@10 | Avg  |\n|-----------|------|------|-------|------|------|-------|------|\n| Random    | 1.1  | 2.7  | 5.2   | 0.7  | 3.3  | 5.0   | 3.0  |\n| MTT-VL    | 1.3  | 4.9  | 9.2   | 1.4  | 5.6  | 9.0   | 5.2  |\n| LoRS      | 1.2  | 5.4  | 9.3   | **1.9**  | 5.7  | 8.7   | 5.4  |\n| CovMatch  | **1.8**  | **7.6**  | **11.9**  | 1.8  | **6.9**  | **12.1**  | **7.0**  |\n| Full      | 15.2 | 38.9 | 53.6  | 14.2 | 38.4 | 52.8  | 35.5 |\n\nWe will incorporate this new experiment and analysis into the revised version of the paper to highlight the potential applicability of our method to video-text retrieval. We thank the reviewer again for this valuable suggestion.\n\n[3] Tran, Du, et al. A closer look at spatiotemporal convolutions for action recognition. CVPR 2018.\n\n[4] Bain, Max, et al. Frozen in time: A joint video and image encoder for end-to-end retrieval. ICCV 2021.\n\n[5] Wu X, Zhang B, Deng Z, et al. Vision-language dataset distillation. TMLR 2024.\n\n[6] Xu Y, Lin Z, Qiu Y, et al. Low-Rank Similarity Mining for Multimodal Dataset Distillation. ICML 2024\n\n\n>**3. Insufficient comparison methods: Only two dataset distillation methods are used for performance comparison, and only one is published in 2024.**\n\nWe thank the reviewer for the comment. We acknowledge the concern regarding the limited number of baseline methods used for comparison. However, we would like to clarify that multimodal dataset distillation is still a very nascent research area, and to the best of our knowledge, there currently exist only two publicly available methods that perform dataset distillation in the multimodal (specifically, vision-language) setting.\n\nIt is worth noting that both existing baselines—MTT-VL [5] and LoRS [6]—are based on adaptations of the trajectory matching framework [7]. In principle, one could attempt to adapt other unimodal dataset distillation methods (e.g., DC[8], DM[9]) to the multimodal case. However, such extensions face significant challenges due to the importance of maintaining cross-modal alignment and the larger scale of multimodal models. As reported in [2], these adaptations often result in unstable optimization and performance degradation (see Table 1 in [6]).\nWe agree that as the field evolves and more methods are introduced, future work should expand the range of comparisons accordingly. We will also make this limitation and justification more explicit in the revised version of the paper.\n\n[7] Cazenavette, George, et al. Dataset distillation by matching training trajectories. CVPR 2022.\n\n[8] Zhao, Bo, Konda Reddy Mopuri, and Hakan Bilen. Dataset Condensation with Gradient Matching. ICLR 2021.\n\n[9] Zhao, Bo, and Hakan Bilen. Dataset condensation with distribution matching. WACV 2023.\n\n\n\n>**4. Insufficient related works: For Multimodal Distillation, it is weird no studies for image-text retrieval are mentioned, such as [1-4].**\n\nWe thank the reviewer for pointing out this important oversight. We agree that prior work on image-text retrieval and matching provides valuable context for our study, especially given that our method builds upon contrastive learning paradigms similar to those employed in retrieval systems.\n\nThe works cited by the reviewer [10–13] have significantly advanced the field of image-text alignment and retrieval through various architectures such as visual-semantic embedding [10], semantic reasoning [11], graph-based matching [12], and contrastive momentum learning [13]. While these methods do not address dataset distillation directly, they are indeed relevant as they form the foundation for many modern image-text pretraining frameworks.\n\nTo address this, we will add a new subsection titled “Image-Text Retrieval and Matching” under the Related Work section of our paper. In this subsection, we will briefly review these representative works and clarify how our setting differs—namely, in focusing on synthesizing compact datasets for training rather than improving retrieval performance directly.\n\nWe appreciate the reviewer’s suggestion and will incorporate this discussion in the revised manuscript to provide a more comprehensive overview of related research.\n\n[10] Karpathy, Andrej, and Li Fei-Fei. Deep visual-semantic alignments for generating image descriptions. CVPR 2015.\n\n[11] Li, Kunpeng, et al. Visual semantic reasoning for image-text matching. ICCV 2019.\n\n[12] Liu, Chunxiao, et al. Graph structured network for image-text matching. CVPR 2020.\n\n[13] Wang, Haoran, et al. Coder: Coupled diversity-sensitive momentum contrastive learning for image-text retrieval. ECCV 2022."}, {"review_id": "FIZ8h2e7nO", "reviewer": "Reviewer_cGeV", "summary": "This paper studies dataset distillation problem aiming to distill a smaller dataset from a large dataset to achieve the best performance,  for vision-language model training. The authors analyze the bottlenecks of existing bi-trajectory matching methods. They further propose a new method to involve text encoder training in the distillation framework. To reduce computational overhead, they propose cross-covariance matching loss with a feature matching regularization. The experiments show that the proposed framework beats previous methods on two large datasets.", "questions": "In the weaknesses, I list a bunch of questions about the method and experiments. The authors just need to respond to my questions listed. I’ll also refer to the comments of other reviewers to decide final ratings.", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths: \n1. This paper includes thorough analysis on widely-used bi-trajectory macing method, which shows more insights into the deficiency of previous methods.\n2. This paper shows evidence for the proposed method using equations and experiments. 3. The improvement over previous baselines is prominent on the two datasets.\n\nWeaknesses: \n1. I don’t quite understand how to update the dataset during training. This paper says the goal is to synthesize a smaller dataset. How to under the word “synthesize” here? Selecting a subset of samples from the large dataset or generating completely new samples? How to backpropagate the gradient to update the dataset? \n2. What’s the input embedding layer in Fig.1? Is it the encoding layer to map each word to a vector? Why not involve it in gradient backpropagation either? \n3. In Eq.10, the expectation is over the whole real and synthetic datasetes. However, the loss is calculated based on batches during training. What’s the impact of training batch size to the final performance? \n4. What’s upper bound of the proposed method by increasing the scale of the distilled datasets? It will show more insights if the authors can show the saturation point. 5. What’s the performance of train the VLM on entire Flickr30K and COCO datasets? I’m wondering the current gap between training using the distilled dataset and training using the full set.", "was_revised": true, "final_justification": "After reading the rebuttal and the response to other reviewers, my concerns have been well addressed. I tend to keep my positive rating for this submission. I hope the author can futher polish this paper by involving the experiments and discussions in rebuttal.", "rebuttal": "# Response to Reviewer cGeV\n\nWe sincerely appreciate the constructive feedback provided by the reviewer. We have made every effort to address the concerns and questions raised. We hope that this response addresses the reviewer's questions.\n\n>**1. I don’t quite understand how to update the dataset during training. This paper says the goal is to synthesize a smaller dataset. How to understand the word “synthesize” here? Selecting a subset of samples from the large dataset or generating completely new samples? How to backpropagate the gradient to update the dataset?**\n\nIn our work, the term “synthesize” refers to learning a set of synthetic image-text pairs, which are initialized from a randomly selected subset of real samples in the original dataset. These synthetic samples are parameterized as learnable tensors and are directly optimized through gradient-based methods.\n\nSpecifically, each synthetic image and text sample is represented as a learnable tensor with shapes `[3,H,W]` and `[L,D]`, respectively, where L is the sequence length and D is the hidden dimension. During training, we compute the cross-covariance and feature-matching loss (as defined in Equation (11)) using these synthetic pairs. Since the synthetic samples are directly involved in the loss computation, we can backpropagate the gradient of the loss with respect to the synthetic inputs. This allows us to update the synthetic dataset $\\mathcal{S}$ as follows:\n$$ \\mathcal{S} \\leftarrow \\mathcal{S} - \\eta \\nabla_\\mathcal{S} \\mathcal{L}^\\textrm{covmatch}(\\mathcal{S}), $$\nwhere $\\eta$ is the learning rate.\n\n\n>**2. What’s the input embedding layer in Fig.1? Is it the encoding layer to map each word to a vector? Why not involve it in gradient backpropagation either?**\n\nWe thank the reviewer for pointing this out. Let us first clarify the process of text encoding in our framework. The input text, which is a sequence of words, is first tokenized into a sequence of token IDs using a tokenizer. These token IDs are then converted into dense vector representations by summing three types of embeddings: token embeddings, segment embeddings, and positional embeddings. The resulting embedding sequence is then fed into the transformer layers of the text encoder.\n\nIn Figure 1, we refer to this entire process—i.e., from raw text to the input of the transformer layers—as the input embedding layer. We do not involve this part in gradient backpropagation because the input tokens are discrete and non-differentiable. Since gradients cannot be propagated through discrete inputs, we instead directly optimize continuous synthetic embeddings that bypass the input embedding layer, as explained in our method.\n\nWe will clarify this terminology in the revised manuscript to avoid any confusion.\n\n\n>**3. In Eq.10, the expectation is over the whole real and synthetic datasetes. However, the loss is calculated based on batches during training. What’s the impact of training batch size to the final performance?**\n\nWe thank the reviewer for the thoughtful question. While our original objective aims to match the cross-covariance matrix and mean feature of the entire training dataset with those of the synthetic set, computing the exact statistics over the full training set at every distillation step is computationally prohibitive—especially since the embeddings are continuously updated by the online model. To address this, we approximate the full-dataset statistics using mini-batches sampled from the training set at each step. This strategy introduces some variance, but allows the method to remain tractable and scalable during training.\n\nAs shown in Table R1, increasing the training batch size leads to consistently improved retrieval performance. This trend suggests that larger batches enable more accurate approximation of the full-dataset statistics, particularly the cross-covariance between real and synthetic features, thereby facilitating more effective learning.  However, this performance gain comes at the cost of increased computational overhead. Specifically, we observe that the per-iteration time (sec/iter) rises substantially with larger batch sizes. This highlights a fundamental trade-off: while larger batches provide better statistical approximations and improved final accuracy, they require significantly more compute resources.\n\nWe will include this analysis and discussion in the revised version of the paper.\n\n\nTable R1. Effect of training batch size on performance and distillation speed on Flickr30k with N=100.\n| Train batch size | 64   | 128  | 256  | 512  | 1024 |\n|------------------|------|------|------|------|------|\n| Performance      | 29.1 | 30.5 | 30.4 | 30.9 | 31.1 |\n| sec/iter         | 1.02 | 1.22 | 1.36 | 1.79 | 3.50 |\n\n\n>**4. What’s upper bound of the proposed method by increasing the scale of the distilled datasets? It will show more insights if the authors can show the saturation point.**\n\n\nWe thank the reviewer for the helpful suggestion. We agree that analyzing the scaling behavior of distilled dataset size can provide further insight into the effectiveness and limitations of our method.\n\nTo investigate this, we conducted additional experiments on the Flickr30K dataset by increasing the number of distilled image-text pairs N beyond 500. As shown in Table R2, our method continues to improve as N increases, and reaches a saturation point around **N = 2000**, where it closely matches the performance of training with randomly selected real samples of the same size.\n\nIn contrast, we observe that baseline methods such as MTT-VL[1] and LoRS[2] tend to saturate at smaller scales, typically around N=500, and in some cases even exhibit performance degradation as N increases further. We note that we conducted extensive hyperparameter tuning for these baselines at each scale, including learning rates, maximum start epoch, and similarity rank. Despite these efforts, the observed degradation suggests that the optimization processes of the baselines are inherently less effective on large number of synthetic pairs.\n\nThese results highlight the **strong scalability** of our method compared to existing approaches. We will include this new analysis, along with saturation curves, in the revised version of the paper.\n\nTable R2. Scalability comparison on Flickr30K.\n| Method   | N=100 | N=200 | N=500 | N=1000 | N=1500 | N=2000 |\n|----------|-------|-------|-------|--------|--------|--------|\n| Random   |  8.6  | 13.1  | 22.6  | 31.7   | 38.2   | 42.1   |\n| MTT-VL   | 20.4  | 21.2  | 25.0  | 19.0   | 18.2   | 18.2   |\n| LoRS     | 27.4  | 29.5  | 31.6  | 31.3   | 29.6   | 27.3   |\n| CovMatch | 30.5  | 34.4  | 38.4  | 41.2   | 41.8   | 43.7   |\n\n[1] Wu X, Zhang B, Deng Z, et al. Vision-language dataset distillation. TMLR 2024.\n\n[2] Xu Y, Lin Z, Qiu Y, et al. Low-Rank Similarity Mining for Multimodal Dataset Distillation. ICLR 2024.\n\n\n>**5. What’s the performance of train the VLM on entire Flickr30K and COCO datasets? I’m wondering the current gap between training using the distilled dataset and training using the full set.**\n\nWe thank the reviewer for the question. We report the performance of full dataset and compare with the performance of our distilled dataset in Table 1. Note that the performance of full-dataset training is also reported in the caption of Table 2 in the main paper. As shown in Table R3, our distilled dataset with 2,000 samples (less than 7% of the original size) achieves around 60% of the full-dataset performance on average across metrics. Notably, for the TR@10 metric, our method achieves approximately 75% of the full-data performance. This highlights the strong scalability and data efficiency of our proposed method. \n\nWe believe that these results demonstrate the ability of our approach to significantly reduce the reliance on large-scale annotated datasets. That said, a non-negligible performance gap still remains, and we consider closing this gap through improved distillation objectives and architectures to be an important direction for future work.\n\nTable R3. Comparison of the CovMatch's performance with full dataset on Flickr30k.\n| # Synthetic pairs | IR@1 | IR@5 | IR@10 | TR@1 | TR@5 | TR@10 | Avg  |\n|--------------|------|------|-------|------|------|-------|------|\n| 100     | 10.1 | 28.6 | 40.9  | 14.8 | 38.0 | 50.6  | 30.5 |\n| 500      | 14.7 | 38.4 | 51.4  | 19.9 | 46.7 | 59.5  | 38.4 |\n| 1000     | 16.3 | 41.0 | 53.2  | 22.3 | 51.4 | 63.3  | 41.2 |\n| 2000     | 18.7 | 44.7 | 56.5  | 25.7 | 54.2 | 68.2  | 44.7 |\n| Full set     | 48.7 | 79.2 | 87.2  | 61.6 | 85.9 | 91.5  | 75.7 |"}]

Author rebuttal:

Meta-review: This paper received ratings (4, 4, 4, 4). It introduces CovMatch a framework for multimodal dataset distillation. All reviewers agree on acceptance. The reviewers find it well-motivated and technically sound about the method of unfreezing the text encoder and matching covariance matrices between modalities. Some initial questions raised regarding the experimental comparison and scalability were addressed during the rebuttal. The AC panel recommends accept.
Decision: Accept (poster)