Beyond Modality Collapse: Representation Blending for Multimodal Dataset Distillation
Authors: Xin Zhang
Ziruo Zhang
Jiawei Du
Zuozhu Liu
Joey Tianyi Zhou
Venue: NeurIPS 2025 poster ()
Keywords: multimodal dataset distillation
modality collapse
representations blending
symmetric projection trajectory matching
Primary area: general_machine_learning
TL;DR: We propose RepBlend, a multimodal dataset distillation framework that mitigates modality collapse via representation blending and symmetric projection matching, achieving superior performance and generalization.

Abstract: Multimodal Dataset Distillation (MDD) seeks to condense large-scale image-text datasets into compact surrogates while retaining their effectiveness for cross-modal learning. Despite recent progress, existing MDD approaches often suffer from ***Modality Collapse***, characterized by over-concentrated intra-modal representations and enlarged distributional gap across modalities. In this paper, at the first time, we identify this issue as stemming from a fundamental conflict between the over-compression behavior inherent in dataset distillation and the cross-modal supervision imposed by contrastive objectives. To alleviate modality collapse, we introduce **RepBlend**, a novel MDD framework that weakens overdominant cross-modal supervision via representation blending, thereby significantly enhancing intra-modal diversity. Additionally, we observe that current MDD methods impose asymmetric supervision across modalities, resulting in biased optimization. To address this, we propose symmetric projection trajectory matching, which synchronizes the optimization dynamics using modality-specific projection heads, thereby promoting balanced supervision and enhancing cross-modal alignment.
Experiments on Flickr-30K and MS-COCO show that RepBlend consistently outperforms prior state-of-the-art MDD methods, achieving significant gains in retrieval performance (e.g., +9.4 IR@10, +6.3 TR@10 under the 100-pair setting) and offering up to 6.7$\times$ distillation speedup.

Beyond Modality Collapse: Representations Blending for Multimodal Dataset Distillation
Xin Zhang!? Ziruo Zhang' Jiawei Du Zuozhu Liu
Joey Tianyi Zhou 12 "Centre for Frontier Al Research, Agency for Science, Technology and Research, Singapore
2 Insttittee of High Performance Computing, Agency for Science, Technology and Research, Singapore National University of Singapore, Singapore *Zhejiang University, China {zhangx7, , duju, Joey_Zhou\Oa-staae edu. ag ziruo zbu. mus mus.eduu edu suozhuliuSinta zju. wdw.ttt......uu........... edu. enu.cn
Abstract
Multimodal Dataset Distillation (MDD) seeks condense large-scale image-text datasets into compact surrogates while retaining their effectiveness for cross-modal learning. Despite recent progress, existing MDD approaches often suffer from Modality Collapse, characterized by ceeeennncentated intra-modal representa- tions and enlarged distributional gap across modalities. In this paper, for the first time, we identify this issue as stemming from fundamental conflict between the over-compression behavior inherent dataset distillation and cross-modal supervision imposed by contrastive objectives. To alleviate modality collapse, we introduce RepBlend, novel MDD framework that weakens overdominant cross- moosl supervision via representation blending, thereby significantly enhanc- ing intra-modal diversity. Additionally, we observe that current MDD methods impose asymmetric supervision across modalities, resulting in biased optimiza- tion. To address this, we propose symmetric projection trajectory matching, which synchronizes the optimization dynamics using modality-specinn projection heads, thereby promoting balanced supervision and enhancing cross-modal alignment. Experiments Flickr-30K and MS-COCO show that RepBlend consistently outperforms prior state-of-the-art MDD methods, achieving significant gains retrieval performance (e.g.. +9.4 IR @ 10, +6.3 TR@ 10 under the 100-pair setting) and offering up to 6.7 distillation speedup. Our code is publicly available at https:/ htt///gtttttttt www http//hangxinn-ie weeeiiinnndd/Reee Red/RppBeend
Introduction
The unprecedented expansion of large-scale datasets has catalyzed recent breakthroughs in deep learning [6, 2, 1], but has also introduced considerable storage and computational overhead [20, 22]. Thus, reducing dataset size to streamline the development process has emerged as an important research focus. Among various solutions, Dataset Distillation (DD) [50] has emerged as com- pelling strategy, achieving high compression ratios by synthesizing a compact surrogate dataset that approximates the training efficacy of the original dataset. The effectiveness of DD has been demonstrated across various modalities, including images [4, 57], text [30, 32], videos [11, 51], and graphs [29, 58]. These unimodal successes motivate its extension to increasingly prominent multimodal scenarios [37, 28, 35, 5].
The pioneering effort in multimodal dataset distillation (MDD) is MTT-VL [53], which first validates the feasibility of extending existing vanilla DD techniques to the image-text setting. Building this baseline, LoRS [55] further proposes to mine cross-modal similarity to calibrate the supervision
Corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Modality Collapse
Random
LoRS
Ours
Image embedding
embedding
Figure I: Multimodal embedding distributions across various distillation methods. We extract image and text embeddings from finetuned CLIP [37] and project them into a shared representation space using DOSNES [31]. Red triangles and blue circles denote image and text embeddings, respectively. Left: Embeddings from randomly sampled data in the original dataset exhibit well-spread and modality-aligned distribution. Middle: The distilled dataset generated by a SOTA MDD method (LoRS [55]) suffers from Modality Collapse, where image and text embeddings are poorly aligned and concentrated in separate regions. Right: Our method effectively mitigates modality collapse, resulting in distribution with improved cross-modal alignment and higher representational diversity.
from matched and mismatched pairs, thereby achieving better adaptation to high-variance image-text data. Despite achieving promising results, existing studies remain confined to the data structure level, without probing the underlying conflict between DD and contrastive learning. Specifically, to prevent significant performance deterioration, vanilla DD prioritizes capturing representative features under limited distillation budgets, often sacrificing diversity and distributional coverage [14, 18, 15]. While this compromise tolerable in unimodal classification tasks, naively applying such strategies to multimodal contrastive learning, which places great importance instance-level discriminability. inevitably leads to Modality Collapse. As illustrated in Figure (middle), the distilled dataset exhibits pronounced intra-modality aggregation and inter-modality separation.
This modality collapse leads to two critical issues. First, induces excessive intra-modal similarity, where embeddings within each modality become increasingly concentrated as distillation progresses. This over- werrconnentratton gradually suppresses representational diversity, making semantically distinct instances harder to separate, and eroding the fine-grained discrimination ability within each modality, Second, it widens the inter-modal gap, resulting in a large divergence between the feature distributions of different modalities. Insufficient mrosll interaction fragments the embedding spaces and weakens semantic alignment, compromising the correct matching of positive pairs and the separation of negative pairs across modalities.
Recognizing these limitations, we propose RepBlend, novel framework for MDD aimed at alleviat- ing modality collapse. First, we theoretically identify that the collapse results from the joint effect of the over-compeessive nature of DD, where optimization converges toward a small set of dominant fea- tures, and the cross-modas contrastive supervision, which further reinforces this convergence, leading to intra-modal collapse. To address this issue, RepBlend introduces Representation Blending within each modality to weaken the overly strong cross-modal supervision, thereby promoting intra-modal diversity. Furthermore, we observe that existing MDD approaches exhibit asymmetric supervision between modalities, with the image branch receiving significantly weaker update signals than the text branch. To address this, we propose Symmetric Projection Trajectory Matching, a mechanism that aligns the optimization trajectories of both projection heads, thereby enhancing cross-modal alignment and improving overall distillation efficiency. Extensive evaluations on Flickr-30K and MS-COCO demonstrate that RepBlend consistently surpasses existing MDD methods. Notably, under the 100-pair setting on Flickr-30K, it achieves improvements of +9.4 in IR @ 10 and +6.3 in TR @ 10, along with 6.7% distillation speedup over the SOTA baseline. Beyond these benchmarks, RepBlend also exhibits strong generalization to other multimodal scenarios, such as audio-text.
Our contributions are summarized follows:
For the first time, we identify the modality collapse issue in current MDD solutions, where the distilled dataset exhibits high intra-modal similarity and large inter-modal gap. Through theoretical analysis, we attribute this to mutually reinforcing effect between the over- compression behavior of dataset distillation and the cross-modal supervision enforced by contrastive objectives.

---

Random
CR:0.95
Learned
nitro-modol Similority
0.520
LRRS
collagsing
0.510
Heape Viler
Groundtruth Matrix
0.514
CR:0.42
maintainig
CR:0.57
0.512
0.510
600
1200
1800
2400
3000
Text Andes
Text index
Iteration
LORS
Ours
Figure 2: Left: Increasing intra-modal similarity as distillation progresses. We run optimization for 3000 iterations and track the intra-modal cosine similarity, which increases from 0.512 to 0.522 (red curve). Though small in magnitude, this rise leads to a more than twofold increase in concentration ratio (CR)² due to the high dimensionality of the embedding space. Right: Modality collapse undermines the efffectiveness of learned soft cross-modal correspondence. The non-matching image- text pairs exhibit nearly uniform similarity scores, forming horizontal and vertical stripes.
We propose Representation Blending to mitigate modality collapse by weakening the overly strong cross-modal supervision and enhancing intra-modal representational diversity, Furthermore, we introduce Symmetric Projection Trajectory Matching to enable more balanced multimodal distillation, which not only strengthens cross- modal alignment but also improves overall distillation efficiency.
2 Preliminaries and Related Works
Dataset Distillation (DD) [50] aims to synthesize a compact surrogate dataset that emulates the key properties of the original large-scale dataset. These properties include distributional characteristics, such as feature-level statistics [60, 48, 49] and batch normalization parameters [57, 42, 15], and training dynamics, including gradients [61, 59] and optimization trajectories [4, 7, 14, 18, 25]. While DD achieves promising results on unimodal benchmarks, extending it to multimodal scenarios remains challenging due to unique data structures and learning strategies [53, 55]. We begin by formalizing the problem of Multimodal Dataset Distillation (MDD).
Problem Formulation. Given large-scale image-text dataset D [(x,, T:), ).) where a; E Rd and E Rd denote the i-th image and its paired caption representation', and each pair is independently sampled from natural data distribution P. Each y, (0, 1)Pi is one-hot vector indicating the correspondence between I; and the caption set [T,) with the i-th entry activated. Similar to DD, MDD also aims to minimize the loss original dataset using the model trained on its distilled synthetic counterpart S
( Fx), 9
S arg min S
[[[Ss (xr, T), y) (2,T)~/P
E
S.L. Os arg min
E (0,F)~
(() (1)
0
where |S| < [D), and L denotes the contrastive learning loss. The model fo(.) represents a CLIP- style network parameterized by 0. Each distilled sample consists of synthetic image-text pair (æ;, f.), where i; E Idas and 7; E IR accompanied by learned soft label ÿ..
MDD vs. Vanilla DD. According to the Equation 1, the generalization from vanilla DD to MDD involves two key modifications: 1) introducing soft ground- vectors ÿs and 2) optimizing under contrastive learning loss L for image-text alignment. While learning soft labels is common in vanilla DD [7]. optimizing Ù; in MDD is more challenging. as both image and text representations updated simultaneously. Besides, in practice, the contrastive loss L is typically instantiated as InfoNCE [33], extended InfoNCE (eNCE), or weighted BCE (wBCE) [55], all aiming to strengthen positive alignments while penalizing mismatched pairs. However, these extensions only make the multimodal adaptation feasible, overlooking the essence of dataset distillation: effective information
GGivn the discrete nature of text, all subsequent analysis is conducted in the representation space, while images remain processed the pixel space. Here, ding
x H and duex 768 (for BERT [10]. "CR measures how tightly the features are clustered, based on how much of the hypersphere is covered at the given cosine similarity. (Refer Appendix C for more calculation details).
3

---

condensation. More specifically, they prioritize cross-modal alignment, while failing to preserve intra-modal diversity and discriminability under severe data compression.
3 Methodology
In this section, we introduce RepBlend, a novel approach for MDD. We begin by identifying the phenomenon of Modality Collapse, which emerges when vanilla DD methods are naively applied to multimodal settings. Through theoretical and empirical analysis, we uncover its underlying causes. To address this issue, we propose Representation Blending to enhance intra-modal diversity. In addition, introduce Symmetric Projection Trajectory Matching, which balances the distillation process across modalities and further strengthens cross-modal alignment. The overall pipeline of RepBlend is outlined in Algorithm 1.
3.1
Modality Collapse
LoRS [55] is representative MDD method built upon Equation 1, where L is defined as:
18
Tūo > 8 Waj E (Bij, a (9t//))) Way (((,,) Dyj B) ((,55) By S BY
Tūy S 33
WBCE
Σ 2
(2)
Here, B C. S denotes sampled batch. yi represents the cosine similarity between the normalized image and text embeddings, where ă' Normalize( fimeE (i,)) and +/ N (e,x)(++))))))))))) (f,)). )).
with pimgE()) (.) and fixxtP (.) denoting the image encoder and text projection head, respectively. The threshold 3 is used to determine positive and negative pairs, (--) denotes the sigmoid function, and the temperature. -(.-)) refers the binary cross-entropy loss. While this supervision primarily aims to mine cross-modal relationships, it inadvertently reinforces intra-moda similarities, ultimately leading to Modality Collapse, as shown in Figure 1, where instances within each modality excessively concentrate. Without loss of generality, the following analysis focuses on the image modality.
Cross-modal supervision reinforces intra-modal similarity. During dataset distillation, (R,, 3n and (ă.,, Fm.) exhibit some non-negligible similarity, i.e., Dum R Ymn > 8, then the direction of their subsequent updates Oa Oas m
Proposition:
ac ac is determined by
Wrm Wynn
(B
Io (ūmm))) Gmn) | T/
(3)
which indicates that the optimization guided by positive pairs 7.T 7% promoting concentra- tion in similar directions. A detailed derivation is provided in Appendix B. When distilling large dataset into a compact one, the optimization pro- cess tends to be dominated by few salient fea- tures [9, 15, 18, 43]. Once this convergence trend emerges, cross-modal supervision further rein- forces it: modality-speciff diversity is implicitly suppressed, and intra-modal representations are increasingly aligned toward a limited set of dom- inant directions. As illustrated in Figure 2 (left), the intra-modal similarity consistently increases throughout the distillation process.
Intro-modal Similarity
0.36
0.5225
0.5200
Moodllin Gop
0.5175
0
0.5150
0.33
0.5125
0.32
0.3075
=
= 0.01
= 0.05
Figure 3: As the noise level X increases, intra- modal similarity (blue bars) shows slight de- cline, while the modality gap (yellow bars) rises markedly. In contrast, our representation blend- ing (RB) leverages in-distribution samples to si- multaneously reduce intra-modal similarity and mnter-modal gap, effectively mitigating modality collapse during distillation.
In addition to the aggravated mntra-modal similar- ity, modality collapse also exacerbates the cross- modal representation gap, as features from each modality become increasingly centralized within compact regions of the shared embedding space. image-text pairs converge toward a uniform distribution. Such behavior undermines the utility of soft label distributions, which are designed to encode fine-grained relational information beyond
Consequently, the similarities between non-matching
In LoRS [55], no image projection head is used.

---

the binary supervision provided by one-hot labels. As illustrated in Figure 2 (right), non-diagonal similarity values exhibit a near-uniform pattern, where image embeddings produce nearly constant similarity scores across all non-matching text embeddings (manifesting as horizontal stripes), and vice versa for text samples (vertical stripes).
3.2 Mitigating Modality Collapse via Representation Blending
As analyzed in Equation 3, modality collapse arises from overly strong cross-modal supervision, which implicitly encourages intra-modal concentration and undermines representational diversity. To alleviate this constraint, one potential approach to inject directional signals that deviate from F' and F. To empirically validate this hypothesis and explore a viable remedy, we conduct a controlled perturbation experiment on Flickr-30K [36]. In particular, we adopt two key metries following [26]: the intra-modal similarity (Sim) and the modality gap (Gap), defined as,
[S]
[S]
Sim
Gap
+/122
(4)
S 1)
S
ig
2=1
We inject Gaussian noise into the text representations, a/tnoise Normalize (fuxt ((1 Am + AS..)) m
Normalize (FrouP(( ((1-)) + +AB.):
An
where Am and An independently sampled random noise from from../(.. /V 1), and À controls the noise level. We evaluate Sim and Gap under varying levels of X. As shown in Figure 3, a slight increase in noise reduces intra-modie similarity (blue bars), indicating enhanced modality-sppeciffe diversity. These results support our hypothesis that perturbing in the representation space can effectively counteract modality concentration.
However, as noise level continues to grow, the injected perturbation begins to introduce semantically meaningless signals, which hinders cross-modal alignment. This is evidenced by the growing modality gap (yellow bars), accompanied by a performance drop of 1.9% in IR@ and 2.1% 11% in TR@ 1 at À 0.01 under 100 distilled pairs on Flickr-30K dataset. To mitigate this issue, we propose replacing the random perturbation with a structure- preterrirrerviigg variant using in-distribution samples. Specifically, we blend representations from different synthetic instances:
Normalize (Jeaa((1 Am Ar.)) Tn
//blind
Normalize (neaP((1 A)=.. (rj)). (5)
m
where < أزم [S]. This operation resembles the idea of MixUp, but is applied in the representation space. As shown in the last group of Figure 3, we can maintain a low level of intra-modal similarity and small modality gap. Note that although here we illustrate the formulation on text, the same operation is also applied to image side in practice.
3.3 Enhancing Cross-modal Alignment via Symmetric Projection Trajectory Matching
In prior MDD practices, methods such as MTT-VL [53] and LoRS [55] follow a de facto protocol wherein the text encoder is frozen and the image projection layer is omitted. The image encoder and the text projection head are trained to generate expert trajectories for distillation. In this setup, the image encoder is initialized with pretrained weights from ImageNet- IK [8], while the text projection head is trained from scratch. This design is motivated by two key considerations: 1) the prohibitive computational and memory cost of optimizing and storing expert trajectories for large-scale text encoders such as BERT [10]: and 2) the fact that text distillation operates in the representation space, where supervision is applied only through the projection head, thus, matching at the encoder level cannot propagate supervision to the representation space. LoRS [55] minimize the objective in Equation through trajectory matching, which is formulated follows:
IT M 2
1105- 0 lt)) (108 gl+M
# 4" 3 = arg min argmin 2.7.
10h_0
(2)
where T and O' denote the T-step finetuned weights of the image encoder and text projection head using S, initialized from b and 0bpe respectively. The objective is to align the T-step synthetic trajectory with the M-step real trajectory by minimizing the &g distance between their terminal weights, given the same initialization.
5

---

LoRS
Loft SM
2.0
0.85
1.00
Imoge
Updbte Horm Balative Eo
Gag 2.29
Trojectory Marching
Masching
0.00
Infoobertin
0.
20
0.5
0.7
iteration
1.000 1900
2000
2003
5000
LoAS
LORSASM
Figure 4: Current MDD methods adopt asymmetric distillation. Left: The loss on the image side shows much smaller variation than that of the text side, fluctuating mildly around 1.0 without notable reduction. Right: The update norm relative modality in LoRS (0.69) compared to the text modality (0.90), suggesting insufficient representation transfer. The update norm is computed in the shared representation space for both modalities. After incorporating symmetric matching (SM), both image and text modalities exhibit more balanced and synchronized update dynamics, leading to more effective cross-modal alignment (reduced Gap).
initialization significantly lower for the image
However, the aforementioned trajectory matching is asymmetric. As shown in Figure 4 (left), the trajectory matching losses of the image and text modalities exhibit divergent trends: the text-side loss decreases steadily, whereas the image-side loss quickly plateaus and remains relatively high. This is primarily because the image encoder contains significantly more parameters than the text projection head, thus, even small per-parameter errors accumulate into large overall mismatch. This imbalance is further evidenced Figure (right), the norm of updates relative to initialization for the image modality is significantly smaller than that of the text, indicating insufficient distillation on the image side. While the representation blending introduced in Section 3.2 helps narrow the modality gap, its effect is still constrained by the inherently asymmetric distillation. To address this imbalance and further enhance cross moddll alignment, we propose a symmetric distillation strategy by matching trajectories of projection head for both modalities:
2 arg min #.T.9
110
(6)
Here, the image encoder is initialized with ImageNet- IK pretrained weights and kept frozen. While the added image projection head incurs slight computational overhead, it enables projection based matching that significantly enhances the overall efficiency of the distillation process (as discussed in Section 4.4). As shown in Figure 4, symmetric projection matching leads to a more consistent decrease in loss for both image and text branches. Moreover, the increased magnitude of updates suggests stronger supervision signals across modalities, resulting in a more balanced and effective distillation process. With symmetric distillation, the modality gap is further narrowed from 0.318 (in Figure 3) to 0.290, indicating enhanced cross-modal alignment.
Experiments
In this section, we conduct extensive experiments on multiple benchmark datasets to demonstrate the effectiveness of the proposed RepBlend framework. We first present the experimental setup, including the datasets, baseline methods, and implementation details. The main results are summarized in Table 1, Table 2, and Table 3. In addition, we also provide detailed ablation studies to evaluate the individual contribution of each component. All experiments are conducted using two NVIDIA RTX 3090 GPUs and one NVIDIA H100 GPU.
4.1 Experimental Setup
We evaluate our method on two widely-used image captioning datasets: Flickr-30K [36] and MS-COCO [27], which contain approximately 3 3lk and 123k images respectively, with each image paired with five human-annotated captions. For the image encoder, we experiment with NENet [3], RegNet [38], ResNet-50 [19], and ViT [12]. For the text encoder, we consider both BERT [10] and DistilBERT [40]. To further demonstrate the generalizability of our approach across modalities, we extend our evaluation to the AudioCaps [23] audio-text benchmark, utilizing EfficientAT [41] as the audio encoder. Model performance is primarily evaluated using Recall at (R @ K) in cross-modal retrieval tasks. Given a query from one modality, we retrieve the top-K most similar samples from the other modality and measure the retrieval accuracy. We denote text-to-image retrieval as IR @ K, and image-to-text retrieval as TR @ K.
Datasets and Networks.

---

Algorithm Blending Representations to Mitigate Modality Collapse in MDD
Require: Original large dataset D; CLIP-style network { (rime,, fiext,, fimee, fixxp); j: real trajectories
set Diup and D real trajectory matching length M, synthetic trajectory matching length T:: total optimization itcration number Iter
1: Initialize S with |S] randomly sampled image-text pairs and one-hot groundtruth labels
2: Load pretrained weights into encoders (frozen); randomly initialize projection heads
3: for it - 1 to Iter do
4: Sample Prmge Drece and 1 M. 0 from Drgg and O Dior
"Dang"
5: Initialize 0's and 0 SwMP using 0% and 0 DAP
6: for i = to do
7:
for mini-batch
(( n.); Db b=1 E S do Calculate image representaion [fimeE((a)) (#t)}
8:
9:
> Blending in representation space
(fimgE ( } RepBlend ( ((ii (at): +t))
11:
Compute loss L LBBE using Equation 2
12:
Update projection head weights 0 g++ end for
Singt and 0
13:
14:
> Symmetric projection trajectory matching
15:
Optimize S { Ry), ūs b= according Equation 6
[S]
16:
end for
17:
end for
Ensure: Synthetic dataset S
Baselines. The comparison encompasses a range of SOTA approaches, including coreset selection methods such Random sampling. Herding [52], K-Center [16], and Forgetting [47], as well as recent advances in dataset distillation tailored for vision- Lasiun-language models, including MTT-VL [53], TESLA- VL [55], and LoRS [55]. A detailed description of these methods be found in the Appendix E. For fairness, both LoRS [55] and our method synthesize one fewer pair per distillation budget (e.g.. 99 pairs for budget of 100) to account for the additional similarity -marrix overhead.
Implementation Details. We construct CLIP-style architecture using the aforementioned image and text encoders. The image encoder is initialized with ImageNet-pretrained weights [8], while the text encoder is initialized with the official pretrained weights provided by the corresponding language model. After feature extraction, the outputs from both branches passed through separate linear projection layers to obtain the final embeddings. During buffer generation, distillation, and evaluation training. the encoders are frozen and only the projection layers optimized. We collect 20 expert trajectories, each consisting of 10 training epochs. The hyperparameter settings follow those used in LoRS [55] and can be found in Table 7 and Table in Appendix F.
4.2
Main Results
The results on Flickr-309 [36] and MS-COCO [27] are presented in Table and Table 2, respec- tively. Our method consistently outperforms all baseline methods, across all distillation budgets and evaluation metrics. Notably, on Flickr-30k, under the extremely low-data regime of 100 training pairs (0.3%), our method achieves an IR @ of 11.5%, substantially surpassing LoRS (8.3%) and MTT-VL. (4.7%). Similarly, our TR @ 10 reaches 35.5%, a considerable gain over the best baseline LoRS (49.2%) These trends hold consistently across all pair settings. Under the 500-pair scenario (1.7%), our method improves the IR @ 10 from 41.6% (LoRS) to 55.9% and TR @ 10 from 53.7% to 66.7%, reflecting relative gain of over 30%. On MS-COCO, dataset known for higher complexity and variability, our method continues to exhibit superior performance. Under the 100-par -oir setting (0.8%o). our approach achieves IR @ 10 22.3% and TR @ 10 28, 28.0%, substantially outperforming LoRS, which attains 12.2% and 19.6%, respectively. At higher budget of 500 training pairs (4.4% (4.4%)). our method maintains its advantage, achieving the highest IR 10 (30.6%) and TR @ (32.9%) among all evaluated methods. Besides, we also extend our method to larger-scale setting using the L.a dataset, which serves as the pretraining dataset for L.LaVA and consists of 558k image-text pairs. We use approximately 60% of the data (about 334k pairs) for training and reserve
7

---

Table 1: Results on Flickr-30k [36]. Both distillation and validation are performed us- ing BERT.+BERT. The model trained on full dataset performs: IR@ 1=23.16, IR@5=53.95
@ 10=66.62; TR 1=33.8, TR ==6.7, TR@ 10=76.9.
Coreset Selection
Dataset Distillation
Rand Herd [52]
K K-Cent [16]
Forget [47]
MTT.VI. [53]
TESLA-VI. [55]
LoRS [55]
Ours
IR@1
1.0
0.7
0.7
0.7
4,7
0,5g0.
8,3
11.3
15.7+0.6
2.3+0.2
24.1+002
32.0+0.7
IR 10
6.5
5.3
6.1
5.6
24.621.0
4.7
35. 35.1
44.5 44.5
100
0.3%
@
9.9+0.5
5.540.5
11.8+0.2
16.2+00.
TR@5
5,9
4.7
5.0
4.2
28.3
19.5
35.8
41.7+:0
TR@10
10.1
7.9
9.7
39.1
28.9
49.2
55.5 1..4
IR@1
1.1
1.5
4,6
0.2
8.6
12.7=n0.
IR
4.8
5.5
3.1
16.0=1.
1.3
25.3+0.2
34.7++0.
200
0.7%
@
IR@10
2.1
9.2
9.3
2.3
8.4
25.5
10.2.0.8
2.5
2.8.
36.6 36.6.0.3
14.5
47.61995
18.6
TR@5
8.7
8,4
8.2
8,4
28.7=110
10.421.6
38.7
46.0
1R@10
419=19
17.4a1.
53.4+0.8
60.0
IR@1
6.6
1.
10.0
17.0
IR5
10.5
10.0
10.4
9.0
20.2+1.2
7.3+004
28.9+0.7
42.5
500
1.7%
TR 69
5.2
5.1
4.9
3.6
30.0
13.3
5.1
gu.6
41.6 1o.8
15.5
55.9 L0:R
22.5
TR
164
32.8
15.3
39.8
53.2
TR@10
25.7
24.3
23.3
19.3
46.8=0.8
23.8
53.7+0.8
66.7+00.
Table 2: Results MS-COCO [27]. Both distillation and validation performed using
NFNet+BERT. The model trained on full dataset performs: IR@1=14.6 IR@5=38.9, IR@ 10=53.2;
TR @ 2026,, TR@5=46.8. TR 10=61.3.
Coresct Selection
Dataset Distillation
Pairs
Ratio
Metric
Rand Herd [52]
K-Cent [16] Forget [47]
MIT-VL[S3]
TESLA-VLI [SS]
LoRS [55]
Ours
TR
0.3
0.5
0.4
0.3
1.3
0.3
1.8
4.1 4.1
5.4
1.0.0.4
7.1
13.9
IR@10
2.7
3.5
2.5
2.5
9.5
1884..
12.2+0.2
22.3+0.5
0.8%
TR@1
0.7
2.5
2.0.0.2
3,3
5.2-0.5
TRGS
2.1
3.7
10.0
1.7
12.2
17.9
TR 10
5.0
4.9
4.8
15.7
13.5
19.6
28.0
IR@
0.6
0.9
0.7
0.6
1.7
0.1
2.4
6.1 6.1.
IR@5
6.5
0.2
9,3
19.3
IR
4.1
4.9
12.3
0.5.0.1
15.5
29.8
200
1.75
TR@1
1.2
3.3
0,7
4.340.1
6.9....5
TR@S
4,0
3.6
3.8
3.5
11.9
3.1
14.2
21.8 Loa
1R610
19.4
5.3g0.
22.6go
32.3
IR@1
2.5
0.8
2.840.2
6.2-0.1
IR85
5.0
5.3
6.3
5.8
8.9
3.6
9.9.0..
19.9
500
4,4%
T
1.9
1.9
2.5
2.1
15.8
5.0
6.7
1.720.4
16.5
5,3
30.6
7.0
TR@5
17.2=1a
5.940.8
18.3+1.
22,0+0.8
TR@10
12.5
13.7
14.3
13.0
26.0=1.
10.2
27.9
32.9
non-overlapping set of 10k pairs for validation". In addition, we evaluate our approach with more powerful encoders, including DiNo-v2 [34] (85,798,65 parameters) for vision and BGE- [54] (109,482,240 parameters) for text. The results (shown Table 3) demonstrate that our method remains effective when scaling both model capacity and training data, and it significantly outperforms the SOTA competitor. Moreover, our method also demonstrates strong generalizability to other multimodal settings, such audio-text benchmark. See Appendix H for details.
4.3 Ablation Study
Representation Blending & Symmetric Matching. We conduct an ablation study on the Flickr-30K dataset using NERT to evaluate the individual and combined contributions of the proposed components: Representation Blending (RB) and Symmetric Projection Trajectory Matching (SM). As shown in Figure 5, removing either module leads to consistent performance degradation across all retrieval metrics (IR @ 1/5/10 and TR @ 1/5/10) and distillation budgets (100, 200, 500 pairs). RB contributes by mitigating intra-modal collapse; as illustrated in Figure 3, it effectively reduces
see http://///gaatteeoe/////////////

---

Table 3: Results when sealing to larger dataset and model. NFNet + BERT on LLaVA-ce3m: the model
trained on the full dataset performs: IR@ 19.13, @R@5=25.94 IR @ 10=36.34, TR@ 1-9.49, TR@5=26.08
T 10=37.07. DiNo-v2 BGE-1.5 .5 on MS-COCO: the model trained on the full dataset performs: @ 1=22.70, IR @ 551.13,, @ 10=65.26, TR @ 331 04, TR @ 5=61 .96, TR @ 10=74.1.
Methods
NFWer
BERT LLa
DiNo-12
BGB-1.5 MS-COCO
IR
IR
IR 10
TR@1 1
TR@5
TR@10
IR@1
IR
IR 10
TR
TR@5
TR 10
2010
LoRS [55]
1.56
5.33
8.77
1.01
4,04
6.81
1.59
6.12
10.59
1.62
6.15
10.47
Ours
3.53
12.38
19.33
4.39
13:45
20.18
12.52
31.96
44.39
16.06
36.84
49,34
LoRS [55]
1.96
6.72
10:55
1.1
5.11
8.51
2.48
8.46
13.42
3.18
10.36
16.37
500
4.45
15.09
22.31
5.14
14.89
22.97
13.37
33,09
45.69
16.90
39.38
52.14
LoRS [55]
1.67
5.87
9.69
1.68
6.11
10.25
2.95
9.90
15.69
4,56
13.66
20.63
800
24.16
13.68
33.58
45.93
17.14
39.76
RB
SM
SM
AB
SM
18
57-
IR:91 (96)
14-
48-
33
4%
24
Nuri
TR:81 (96)
48
51
6:1
60
66
200
500
300
500
Pairs
Num
Figure 5: Ablation study of Representation Blending (RB) and Symmetric Projection Trajectory
Matching (SM) on Flickr-30K with NFNet+BERT.
intra- modal similarity and enhances representational diversity. SM further balances the learning dynamics across modalities and improves cross-modas alignment, as evidenced in Figure When combined, RB and SM achieve the best overall performance, highlighting their complementary roles in enhancing intra- modal diversity and cross-modal alignment.
Cross-Architecture Generalization. We further validate the generalization capability of RepBlend across diverse architectures. Following the protocol of LoRS [55], we keep the text encoder fixed and evaluate the dataset distilled with NENT using alternative image encoders, including ResNet- 50 and RegNet. As shown in Table 4, RepBlend consistently maintains strong performance across different encoder architectures. Moreover, we extend the evaluation to a broader set of architecture combinations, such as 50+EERT, BiT+BERT, RegNet+BERT, and NFNet+DistilBERT as illustrated in Figure 6 and Figure in Appendix I. Across all architectures, datasets, and distillation budgets, RepBlend consistently outperforms the sota baseline, demonstrating its robustness and architectural adaptability.
Zero-Shot Generalization. To further validate the effectiveness of distilled dataset, fur- ther evaluate zero-shot ImageNet [8] classification and OCR-relevant retrieval on TextCaps [45]. Specifically, we randomly select 10 classes from ImageNet- IK and report Top-1 and Top-5 zero-shot accuracies. For Tex(Caps, we measure retrieval performance on 3,166 validation samples. The results, summarized in Table 5, show that under the same budget, models trained on our distilled dataset outperform LoRS and narrow the performance gap to the full-dataset baseline.
4.4 Computational Efficiency
In the proposed method, the training trajectories of image and text projection layers are used for matching optimization. Although we introduce an additional image projection, it incurs negligible computational overhead. In fact, as shown in Table our method achieves significantly better computational efficiency compared to prior work. Specifically, the time required to construct expert trajectories reduced from 70 minutes to 40 minutes per trajectory (1.75x speedup), and the corresponding memory footprint decreases from 1.63 GB to 0.73 GB (2.23x reduction). During the distillation phase, our method accelerates training iterations from 11.5 seconds to 1.71 seconds per iteration, yielding 6.7x speedup. Moreover, it lowers the peak GPU memory usage from 21.78 GB

---

Table 4: Cross-architecture generalization. The distilled data synthesized with NENet+BERT and evaluated across architectures on Flickr-30K under the 500-pair setting.
Evaluate Model
Methods
IR@1
IR@5
IR@ 10
TR@1
TR@5
TR
TESLA-VL [55]
3.0
10.8
17.0
6.0
18.8
27.7
ResNet-BERT
LoRS [55]
3.3
12.7
20.4
6.8 6.8
19.6
31.1
Ours
4.2
14.1+0.2
23.6
8.4
23.1
35.0
TESLA-VL [55]
3.2
11.1118
17.5
5.8
18.6.0.6
28.1 11.0
RegNet-BERT
LoRS [55]
3.5.
12.610.3
21.1
6.8 10.3
20.8 10.3
30.2
1o.3
Ours
3,9 3.2
13.9
10.3
24.0
10.6
7.9 7.9.
24.2 10.3
36.2 16.1
Table 5: Zero-Shot Generalization. Models trained on the distilled MS-COCO dataset under the 500-pair setting are evaluated on zero-shot ImageNet classification and TextCaps retrieval tasks.
ImageNes-10 Classification
TexcCas Retrieval
Methods
ACC@1
ACc@5
IR@1
IR@5
IR@
TR@1
TR@5
TR@ 10
LoRS [55]
21.4
74.4
1.7
5.1
8.4
0.4
1.7
3.1
Ours
27.6
76.2
3.1
9.4
14.5
1.9
6.2
10.3
10.17 GB (2.14x reduction). These results show that our projection-based design not only enables effective multimodal distillation, but also leads to substantially improved computational efficiency.
Conclusion
In this work, we investigate the underexplored challenge of modality collapse in multimodal dataset distillation (MDD), where intra-modal similarity is excessively am- plified and inter-modal alignment is degraded. Through theoretical analysis and empirical evidence, we attribute this phenomenon to the inherent over-compression be- havior of dataset distillation and interplay with cross- modal contrastive supervision. To mitigate these issues, we propose RepBlend, a novel MDD framework incor- porating two key components: Representation Blending for enhancing intra-modal diversity and Symmetric Pro- jection Trajectory Matching for achieving balanced and effective supervision across modalities. Extensive experiments on Flickr-30K and MS-COCO confirm the superiority of RepBlend both retrieval performance and distillation efficiency.
Table 6: Study of computational efficiency.
LoRS [55]
(IR TR 1) (3)
(8.3, 11.8)
(11.5, 16.2)
Buffer
Speed (minstraj) Momory (GB/traj)
70
40
1.63
0.73
Distillation
Speed (sfiter)
Peak GPU VRAM (GB)
21.78
10.17
Limitations and Future work. Despite the promising results RepBlend, current MDD frame- works, including ours, remain limited to pair-level modeling, which restricts fine-graines alignment between text tokens and visual objects. Additionally, insufficient cross-instance interaction hampers representation expressiveness and limits further gains in compression. In the future, will explore awwre, relation-cnhance strategies to overcome these challenges.
Acknowledgement
This research is supported by Xin Zhang's A*STAR Career Development Fund (CDF) (Project No. C243512009), and Jiawei Du's A*STAR Career Development Fund (CDF) (Project No. (2333.2004). This research also supported by the Japan Science and Technology Agency (JST) and the Agency for Science, Technology and Research (A *STAR) under the Japan-Singapore Joint Call (Project No. R2416IR133)
10

---

References
[1] Samira Abnar, Mostafa Dehghani. Behnam Neyshabur, and Hanic Sedghi. Exploring the limits of large scale pre-training. In International Conference on Learning Representations, 2022.
[2] Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx. Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint 1iX2:22088.07258, 2021.
[3] Andy Brock, Soham De, Samuel L Smith, and Karen Simonyan. High-performance large-scale image recognition without normalization. In International Conference Machine Learning, pages 1059-1071. PMIR, 2021.
[4] George Cazenavette, Tongzhou Wang, Antonio Torralba, Alexei A Efros, and Jun-Yan Zhu. Dataset distillation by matching training trajectories. In CVPR, 2022.
[5] Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multimodal Ilm's referential dialogue magic. arXiv preprint arXiv:2306.15195, 2023.
[6] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1- 113, 2023.
[7] Justin Cui, Ruochen Wang, Si Si, and Cho-Jui Hsieh. Scaling up dataset distillation imagenet- Ik with constant memory. In ICML, 2023.
[8] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. CVPR, 2009.
[9] Wenxiao Deng, Wenbin Li, Tianyu Ding, Lei Wang, Hongguang Zhang, Kuihua Huang, Jing Huo, and Yang Gao. Exploiting inter-sample and inter-feature relations in dataset distillation. In CVPR, 2024.
[10] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 10.0. R10.04805, 2018.
[11] Guodong Ding, Rongyu Chen, and Angela Yao. Condensing action segmentation datasets via generative network inversion. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, 2025.
[12] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. In ICLR, 2021.
[13] Jiawei Du, Yidi Jiang, Vincent F. Tan, Joey Tianyi Zhou, and Haizhou Li. Minimizing the accumulated trajectory error to improve dataset distillation. In CVPR, 2023.
[14] Jiawei Du, Qin Shi, and Joey Tianyi Zhou. Sequential subset matching for dataset distillation. In NewrlPS, 2023.
[15] Jiawei Du, Xin Zhang, Juncheng Hu, Wenxin Huang, and Joey Tianyi Zhou. Diversity-driven synthesis: Enhancing dataset distillation through directed weight adjustment. Ady. Neural Inf. Process. Syst. (NeurIPS), 2024.
[16] Reza Zanjirani Farahani and Masoud Hekmatfar. Facility location: concepts, models, algorithms and case studies. Springer Science Business Media, 2009.
[17] Jort F Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R Channing Moore, Manoj Plakal, and Marvin Ritter. Audio set: An ontology and human-labeled dataset for audio events. In 2017 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 776-780, IEEE, 2017.

---

[18] Ziyao Guo, Kai Wang, George Cazenavette, HUT LI, Kaipeng Zhang, and Yang You. Towards lossless dataset distillation via difficult--aligned trajectory matching. In ICLR, 2024.
[19] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
[20] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203./5553 2022.
[21] Andrew G Howard. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:
2017.
[22] Feiyang Kang, Hoang Anh Just, Yifan Sun, Himanshu Jahagirdar, Yuanzhi Zhang. Rongxing Du, Anit Kumar Sahu, and Ruoxi Jia. Get more for less: Principled data selection for warming up fine-tuning in LLMs. In The Twelfth International Conference on Learning Representations, 2024.
[23] Chris Dongjoo Kim, Byeongchang Kim, Hyunmin Lee, and Gunhee Kim. Audiocaps: Generat- ing captions for audios in the wild. In NAACL-HI.T, 2019.
[24] Sachyung Lee, Sanghyuk Chun, Sangwon Jung, Sangdoo Yun, and Sungroh Yoon. Dataset condensation with contrastive signals. In ICML, 2022.
[25] Yongmin Lee and Hye Won Chung. Selmatch: Effectively scaling up dataset distillation via selection-based initialization and partial updates by trajectory matching. In ICML, 2024.
[26] Weixin Liang. Yuhui Zhang. Yongchan Kwon, Serena Yeung, and James Zou. Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
[27] Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740-755. Springer, 2014.
[28] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances neural information processing systems,
2023.
[29] Yang Liu, Deyu Bo, and Chuan Shi. Graph distillation with eigenbasis matching, In Forty-first International Conference Machine Learning. 2024.
[30] Huimin LU, Masaru Isonuma, Junichiro Mori, and Ichiro Sakata. Unidetox: Universal detox- ification of large language models via dataset distillation. In The Thirteenth International Conference on Learning Representations, 2025.
[31] Yao Lu, Jukka Corander, and Zhirong Yang. Doubly stochastic neighbor embedding on spheres. Pattern Recognition Letters, 125:581-55 1-78 2019.
[32] Aru Maekawa, Satoshi Kosugi, Kotaro Funakoshi, and Manabu Okumura. Dilm: Distilling dataset into language model for text-level dataset distillation. Journal of Natural Language Processing, 32(1):252-282, 2025.
[33] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv 03748, 2018
[34] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez. Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193. 2023.
[35] Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang. Shuming Ma, Qixiang Ye, and Furu Wei. Grounding multimodal large language models the world. In The Twelfth International Conference Learning Representations, 2024.
12

---

[36] Bryan A Plummer, Liwei Wang. Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. Flickr30k entities: Collecting eegion-to-phrase correspondences for richer image-to-sentence models. In Proceedings of the IEEE international conference on computer vision, pages 2641-2649, 2015.
[37] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning transferable visual models from natural language supervision. In Proceed- ings of the 38th International Conference on Machine Learning, pages 8748-8763 2021.
[38] Iija Radosavovic, Raj Prateek Kosaraju, Ross Girshick, Kaiming He, and Piotr Dollár. Design- ing network design spaces. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages
2020.
[39] Ahmad Sajedi, Samir Khaki, Ehsan Amjadian, Lucy Z. Liu, Yuri A. Lawryshyn, and Konstanti- nos N. Plataniotis. DataDAM: Efficient dataset distillation with attention matching. In ICCV, 2023.
[40] Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01006 2019.
[41] Florian Schmid, Khaled Koutini, and Gerhard Widmer. Efficient large-scale audio tagging via transformer-to-cnn knowledge distillation. In ICASSP 2023-2023 IEEE international Conference on acoustics, Speech and signal processing (ICASSP), pages 1-5. IEEE, 2023.
[42] Shitong Shao, Zeyuan Yin, Muxin Zhou, Xindong Zhang, and Zhiqiang Shen. Generalized large-scale data condensation via various backbone and statistical matching. In CVPR, 2024.
[43] Zhiqiang Shen, Ammar Sherif, Zeyuan Yin, and Shitong Shao. Delt: A simple diversity-driven earlylate training for dataset distillation. In CVPR, 2025.
[44] Seungjae Shin, Heesun Bae, Donghyeok Shin, Weonyoung Joo, and Il-Chul Moon. Loss- curvature matching for dataset selection and condensation. In AISTAS, 2023.
[45] Oleksii Sidorov, Ronghang Hu, Mareus Rohrbach, and Amanpreet Singh. Textcaps: a dataset for image captioning with reading comprehension. In European conference on computer vision, pages 742-758. Springer, 2020.
[46] Peng Sun, Bei Shi, Daiwei Yu, and Tao Lin. On the diversity and realism of distilled dataset: An efficient dataset distillation paradigm. In CVPR, 2024.
[47] Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, and Geoffrey J Gordon. An empirical study of example forgetting during deep neural network learning. arXiv preprint
2018.
[48] Kai Wang, Bo Zhao, Xiangyu Peng, Zheng Zhu, Shuo Yang, Shuo Wang, Guan Huang, Hakan Bilen, Xinchao Wang, and Yang You. Cafe: Learning to condense dataset by aligning features. In CVPR, 2022.
[49] Shaobo Wang, Yicun Yang, Zhiyuan Liu, Chenghao Sun, Xuming Hu, Conghui He, and Linfeng Zhang. Dataset distillation with neural characteristic function: A minmax perspective. In CVPR, 2025.
[50] Tongzhou Wang, Jun- Yan Zhu, Antonio Torralba, and Alexei A Efros. Dataset distillation. arXiv preprint arXiv: /8/ 10959, 2018
[51] Ziyu Wang, Yue Xu, Cewu Lu, and Yong-Lu Li. Dancing with still images: video distillation via static-dynamic disentanglement. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6296-6304, 2024.
[52] Max Welling. Herding dynamical weights to learn. In Proceedings of the 26th Annual Interna- tional Conference on Machine Learning, pages 1121-1128, 2009.
[53] Xindi Wu, Byron Zhang, Zhiwei Deng, and Olga Russakovsky. Vision-language dataset distillation. In TMLR, 2024.
13

---

[54] Shitao Xiao, Zheng Liu, Peitian Zhang, and Niklas Muennighoff. C-pack: Packaged resources advance general chinese embedding, 2023.
[55] Yue Xu, Zhilin Lin, Yusong Qiu, Cewu Lu, and Yong-Lu Li. Low-rank similarity mining for multimodal dataset distillation. In ICML, 2024.
[56] Zeyuan Yin and Zhiqiang Shen. Dataset distillation in large data 2023.
[57] Zeyuan Yin, Eric Xing, and Zhiqiang Shen. Squeeze, recover and relabel: Dataset condensation at imagenet scale from a new perspective. In NeurlPS, 2024.
[58] Yuchen Zhang, Tianle Zhang, Kai Wang, Ziyao Guo, Yuxuan Liang, Xavier Bresson, Wei Jin, and Yang You. Navigating complexity: Toward lossless graph condensation via expanding window matching. In Forty-first International Conference on Machine Learning, 2024.
[59] Bo Zhao and Hakan Bilen. Dataset condensation with differentiable siamese augmentation. In ICML, 2021.
[60] Bo Zhao and Hakan Bilen. Dataset condensation with distribution matching. In WACV, 2023.
[61] Bo Zhao, Konda Reddy Mopuri, and Hakan Bilen, Dataset condensation with gradient matching.
In ICLR, 2021.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer:
[Yes]
Justification:
The abstract and introduction clearly state the proposed method and its core contribution.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The paper discusses the limitations of the work.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
15

---

Justification:
The paper provides the full set of assumptions and a complete and correct
proof.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The paper fully discloses the information needed to reproduce the main experimental results of the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
16

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
This paper provides implementation details and algorithm descriptions for reproduction. We also release our codes for reproduction in camera-ready version.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.c
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: / /nips. cctpublic//eeeccccccccccccccecct habblccggiidss//// /Coo//ooeeemeeeee sesssuumiisioooi ssiinnolicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: The paper specifies the training and test details necessary to reproduce the results, including datasets, hyperparameters, optimizer type, and how they were chosen.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Our experiments were rigorously conducted with five repetitions each, and we meticulously reported both the mean values and standard deviations for each experimental trial.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
17

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: The paper comprehensively details compute resources in both the experiments
section and supplementary materials, covering GPU type, memory, and storage specifics.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't iidn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https attps://nmt //tt:://eeeupeeeeeee wwwpurips.ccppuuueee cc/public/
http//uublc//bhheeeece EEthii csGuidelines?
Answer: [Yes]
Justification: The research aligns with the NeurIPS Code of Ethics, ensuring ethical standards are upheld throughout the study.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Our work foundational research, and therefore, it does not have direct societal impacts discuss.
Guidelines:
18

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (c.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e-g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: The paper does not involve the release of data or models that have high risk for misuse.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
The paper provides proper credit to asset creators, citing relevant papers and explicitly mentioning license and terms of use. URLs are included where possible, and all licenses are respected.
Justification:
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
19

---

The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. . com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The new assets introduced in the paper are well-documented, providing com- prehensive details alongside the assets, including training procedures, licenses, limitations, and consent processes, ensuring transparency and reproducibility.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsoureing or research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
20

---

The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: LLMs were not used in the development of the core methods; their usage was limited to language polishing and did not influence the scientific contributions of this work.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to LLM policy (https: http://rppppscccooooeeeeeeeeeeee /LLM) for what should or should not be described.
21


Reviews:
[{"review_id": "K8yeduV7hg", "reviewer": "Reviewer_bSSh", "summary": "### **Strengths**\n\n1. **Problem Significance**:\n   - First to formally define and analyze *modality collapse* in MDD, linking it to over-compression and contrastive supervision (Sec. 3.1). Theoretical analysis (Prop. in Sec. 3.1, Appendix B) rigorously justifies the cause.\n2. **Innovative Solutions**:\n   - **Representation Blending**: Effectively diversifies intra-modal features via in-distribution MixUp-like blending (Eq. 5, Fig. 3), outperforming naive noise injection.\n   - **Symmetric Projection Matching**: Resolves asymmetric supervision (Fig. 4) by aligning image/text projection trajectories (Eq. 6), improving cross-modal alignment.\n3. **Strong Empirical Validation**:\n   - RepBlend consistently outperforms SOTA (LoRS, MTT-VL) across datasets (Tables 1–2), metrics (e.g., +30% TR@10 on Flickr-30K at 500 pairs), and modalities (audio-text in Table 7).\n   - Ablations (Fig. 5) confirm each component’s contribution; cross-architecture tests (Table 3, Fig. 6–7) show robustness.\n4. **Efficiency Gains**:\n   - Achieves 6.7× speedup and 2.14× memory reduction via lightweight projection-head matching (Table 4).\n\n------\n\n### **Weaknesses**\n\n1. Altough the authors propose an informable algorithmatic issue, the modality collapas, it would be more clear if the authors could provide a detailed example to facilitate the reader's understanding. \n2. The relationship between Section 3.2 and 3.3 is not clearly stated. Are they two different ways of method implementation? or do they need be utilized simultaneously?\n2. **Visualization Insufficiency**:\n   - Fig. 9–10 show distilled images/texts but lack quantitative or qualitative analysis (e.g., diversity metrics, attention maps). Claims about \"enhanced diversity\" need stronger visual proof.\n3. Since LoRS is an important baseline throughout the experiment section, there should be explicit definition of LoRS in the formal part of paper. \n4. The structure of Section 3.3 can also be improved. It may be better to use a prelimiary section to introduce LoRS, while the method section should focus on the different between LoRS and RepBlend.\n\n------", "questions": "1. Although the authors propose an informative algorithmic issue, the modality collapses, it would be clearer if the authors could provide a detailed example to facilitate the reader's understanding. \n2. The relationship between Section 3.2 and 3.3 is not clearly stated. Are they two different ways of method implementation? or do they need to be utilized simultaneously?\n2. **Visualization Insufficiency**:\n   - Fig. 9–10 show distilled images/texts but lack quantitative or qualitative analysis (e.g., diversity metrics, attention maps). Claims about \"enhanced diversity\" need stronger visual proof.\n3. Since LoRS is an important baseline throughout the experiment section, there should be an explicit definition of LoRS in the formal part of paper. \n4. The structure of Section 3.3 can also be improved. It may be better to use a preliminary section to introduce LoRS, while the method section should focus on the difference between LoRS and RepBlend.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### **Strengths**\n\n1. **Problem Significance**:\n   - First to formally define and analyze *modality collapse* in MDD, linking it to over-compression and contrastive supervision (Sec. 3.1). Theoretical analysis (Prop. in Sec. 3.1, Appendix B) rigorously justifies the cause.\n2. **Innovative Solutions**:\n   - **Representation Blending**: Effectively diversifies intra-modal features via in-distribution MixUp-like blending (Eq. 5, Fig. 3), outperforming naive noise injection.\n   - **Symmetric Projection Matching**: Resolves asymmetric supervision (Fig. 4) by aligning image/text projection trajectories (Eq. 6), improving cross-modal alignment.\n3. **Strong Empirical Validation**:\n   - RepBlend consistently outperforms SOTA (LoRS, MTT-VL) across datasets (Tables 1–2), metrics (e.g., +30% TR@10 on Flickr-30K at 500 pairs), and modalities (audio-text in Table 7).\n   - Ablations (Fig. 5) confirm each component’s contribution; cross-architecture tests (Table 3, Fig. 6–7) show robustness.\n4. **Efficiency Gains**:\n   - Achieves 6.7× speedup and 2.14× memory reduction via lightweight projection-head matching (Table 4).\n\n------\n\n### **Weaknesses**\n\n1. Although the authors propose an informative algorithmic issue, the modality collapses; it would be clearer if the authors could provide a detailed example to facilitate the reader's understanding. \n2. The relationship between Section 3.2 and 3.3 is not clearly stated. Are they two different ways of method implementation? or do they need to be utilized simultaneously?\n2. **Visualization Insufficiency**:\n   - Fig. 9–10 show distilled images/texts but lacks quantitative or qualitative analysis (e.g., diversity metrics, attention maps). Claims about \"enhanced diversity\" need stronger visual proof.\n3. Since LoRS is an important baseline throughout the experiment section, there should be an explicit definition of LoRS in the formal part of paper. \n4. The structure of Section 3.3 can also be improved. It may be better to use a preliminary section to introduce LoRS, while the method section should focus on the difference between LoRS and RepBlend.", "was_revised": true, "final_justification": "Most of the concerns are addressed. We believe this is a reasonable paper and will keep our positive score.", "rebuttal": "Thank you for your constructive comments! We give point-to-point replies to your questions in the following.\n\n**Q1 (Weaknesses 1 & Questions 1):** Although the authors propose an informable algorithmatic issue, the modality collapse, it would be more clear if the authors could provide a detailed example to facilitate the reader's understanding.\n\n**A1:** Thank you for the question. \n - Definition of ***Modality Collapse***\nIn our paper, *modality collapse* refers to a phenomenon in multimodal dataset distillation (MDD) where the distilled embeddings within the same modality become overly concentrated (i.e., exhibit low intra-modal variance), while the alignment between cross-modal embeddings (e.g., image-text pairs) is insufficient. This is visually illustrated in ***Figure 1***, where intra-modal representations cluster tightly, yet fail to align across modalities. \n- Detrimental effect of ***Modality Collapse***\nSuch collapse results in degraded representational diversity and weakened multimodal correspondence (as shown in ***Figure 2***), ultimately leading to significant performance degradation when models are trained on the distilled data, compared to those trained on the original data, as quantitatively demonstrated in Tables I and II.\n---\n**Q2 (Weaknesses 2 & Questions 2):** The relationship between Section 3.2 and 3.3 is not clearly stated. Are they two different ways of method implementation? or do they need be utilized simultaneously?\n\n**A2:** Thank you for the question. Representation Blending (Section 3.1) provides a theoretically sound solution to mitigate modality collapse by encouraging intra-modal diversity. While this strategy can effectively reduce intra-modal concentration and slightly narrow the cross-modal gap (from 0.327 in LoRS to 0.318), its impact remains limited due to the inherently asymmetric nature of the distillation process.  \n\nTo address this asymmetry and further enhance cross-modal alignment, we introduce a Symmetric Projection Matching strategy (Section 3.2), which aligns the projection-head trajectories of both modalities during distillation. This method significantly reduces the modality gap (from 0.327 in LoRS to 0.230), demonstrating its effectiveness in improving cross-modal alignment.  \n\nCombining these two components achieves a synergistic effect: Representation Blending enhances intra-modal diversity, while Symmetric Projection Matching ensures cross-modal alignment, thereby significantly improving the quality of the distilled dataset.  \n\n---\n\n**Q3 (Weaknesses 3 & Questions 3):** Fig. 9–10 show distilled images/texts but lack quantitative or qualitative analysis (e.g., diversity metrics, attention maps). Claims about \"enhanced diversity\" need stronger visual proof.\n\n**A3:** Thank you for the constructive suggestion. \n- Quantitative results:\nThe enhanced diversity of our distilled dataset is reflected in two aspects: **(1) reduced intra-modal similarity**, and **(2) a narrower inter-modal gap**. We quantify them using the **Concentration Ratio (CR)**, **intra-modal similarity (Sim)**, and **modality gap (Gap)** as defined in the main paper. The following tables summarize these quantitative results, showing that our method substantially improves diversity compared to LoRS.\n\n| Metric | LoRS | Ours |\n|--------|------|------|\n| **CR (Sim)** – *Image* | 0.95 (0.522) | **0.57 (0.512)** |\n| **CR (Sim)** – *Text*  | 0.888 (0.519) | **0.45 (0.5104)** |\n\n| Metric | LoRS | Ours |\n|--------|------|------|\n| **Gap** - *(Image ↔ Text)* | 0.327 | **0.230** |\n\n- Qualitative Visualizations:\nIn addition to the quantitative results, **the qualitative visualizations in Figure 1** clearly demonstrate that the distilled embeddings are more dispersed within modalities and better aligned across modalities. Furthermore, **the right subfigure in Figure 2** depicts a cross-modal similarity heatmap (functionally equivalent to an attention map), where each image exhibits distinct attention patterns toward the text, similarly, each text also attends differently to the images. This highlights the increased instance-wise diversity in cross-modal interactions.\n\nAs figures are not permitted during the rebuttal, we will revise the final version to better highlight and elaborate on these visualizations and their implications, and provide further qualitative illustrations to substantiate our claims.\n\n---\n\n**Q4 (Weaknesses 4 & 5 & Questions 4 & 5):** Since LoRS is an important baseline throughout the experiment section, there should be explicit definition of LoRS in the formal part of paper. The structure of Section 3.3 can also be improved. It may be better to use a prelimiary section to introduce LoRS, while the method section should focus on the different between LoRS and RepBlend.\n\n**A4:** Thank you for the valuable suggestions. In the current version, Section 2 (Preliminaries and Related Work) introduces the Multimodal Dataset Distillation (MDD) problem, contextualized by two representative baselines: LoRS and MTT-VL. The details of LoRS are also described at the beginning of Section 3.1.\nTo improve clarity and coherence, we have restructured these sections by explicitly formalizing LoRS in the preliminaries (see below), while leaving the method section to focus on the distinctions between LoRS and RepBlend.\n\n-  Preliminaries and Related Works.\n\nGiven a large-scale image-text dataset $\\mathcal{D} = \\{(\\mathbf{x}\\_i, \\boldsymbol{\\tau}\\_i), \\mathbf{y}\\_i\\}_{i=1}^{|\\mathcal{D}|}$,\nwhere $\\mathbf{x}\\_i \\in \\mathbb{R}^{d{\\_\\text{img}}}$ and $\\boldsymbol{\\tau}\\_i \\in \\mathbb{R}^{d{\\_\\text{text}}}$ denote the $i$-th image and its paired caption representation, and each pair is independently sampled from a natural data distribution $\\mathcal{P}$. \n\nEach $\\mathbf{y}_i \\in \\lbrace 0,1 \\rbrace^{|\\mathcal{D}|}$ is a one-hot vector indicating the correspondence between $\\mathbf{x}\\_i$ and the caption set ${\\boldsymbol{\\tau}\\_j}\\_{j=1}^{|\\mathcal{D}|}$, with the $i$-th entry activated.\nSimilar to Dataset Distillation (DD), Multimodal Dataset Distillation (MDD) also aims to minimize the loss on the original dataset using the model trained on its distilled synthetic counterpart $\\mathcal{S} = \\{(\\tilde{\\mathbf{x}}\\_i, \\tilde{\\boldsymbol{\\tau}}\\_i), \\tilde{\\mathbf{y}}\\_i \\}\\_{i=1}^{|\\mathcal{S}|}$:\n$$\\mathcal{S}^* = \\arg\\min\\_{\\mathcal{S}} \\mathbb{E}\\_{(\\mathbf{x},\\boldsymbol{\\tau}) \\sim \\mathcal{P}} \\left[ \\mathcal{L}(f\\_{\\boldsymbol{\\theta}\\_{\\mathcal{S}}}(\\mathbf{x}, \\boldsymbol{\\tau}), \\mathbf{y}) \\right]\\quad \\text{s.t.} \\quad \\boldsymbol{\\theta}\\_{\\mathcal{S}} = \\arg\\min\\_{\\boldsymbol{\\theta}} \\mathbb{E}\\_{(\\tilde{\\mathbf{x}},\\tilde{\\boldsymbol{\\tau}}) \\sim \\mathcal{S}} \\left[ \\mathcal{L}(f\\_{\\boldsymbol{\\theta}}(\\tilde{\\mathbf{x}}, \\tilde{\\boldsymbol{\\tau}}), \\tilde{\\mathbf{y}}) \\right]  [1]\n$$\nwhere $|\\mathcal{S}| \\ll |\\mathcal{D}|$, and $\\mathcal{L}$ denotes the contrastive learning loss. The model $f\\_{\\boldsymbol{\\theta}}(\\cdot)$ represents a CLIP-style network parameterized by $\\boldsymbol{\\theta}$. Each distilled sample consists of a synthetic image-text pair $(\\tilde{\\mathbf{x}}\\_i, \\tilde{\\boldsymbol{\\tau}}\\_i)$, where $\\tilde{\\mathbf{x}}\\_i \\in \\mathbb{R}^{d\\_\\text{img}}$ and $\\tilde{\\boldsymbol{\\tau}}\\_i \\in \\mathbb{R}^{d\\_\\text{text}}$, accompanied by a learned soft label $\\tilde{\\mathbf{y}}\\_i$.\n\nLoRS is a representative MDD method built upon Equation [1], where the loss function $\\mathcal{L}$ is defined as:\n$$\\mathcal{L}\\_{\\mathrm{wBCE}}^{\\mathcal{B}} = \\sum\\_{i,j}^{|\\mathcal{B}|} w\\_{ij} \\cdot \\ell\\left(\\tilde{\\mathbf{y}}\\_{ij}, \\sigma\\left(\\hat{\\mathbf{y}}\\_{ij} / \\gamma\\right)\\right), \\quad w\\_{ij} = \\frac{\\mathbb{I}[\\tilde{\\mathbf{y}}\\_{ij} > \\beta]}{|{(i,j): \\tilde{\\mathbf{y}}\\_{ij} > \\beta}|} + \\frac{\\mathbb{I}[\\tilde{\\mathbf{y}}\\_{ij} \\leq \\beta]}{|{(i,j): \\tilde{\\mathbf{y}}\\_{ij} \\leq \\beta}|}.\n$$\nHere, $\\mathcal{B} \\subset \\mathcal{S}$ denotes a sampled batch. $\\hat{\\mathbf{y}}\\_{ij}$ represents the cosine similarity between the normalized image and text embeddings, where: $\\tilde{\\mathbf{x}}\\_i^\\prime = \\operatorname{Normalize}(f^{\\text{imgE}}(\\tilde{\\mathbf{x}}\\_i))$ and $\\tilde{\\boldsymbol{\\tau}}\\_j^\\prime = \\operatorname{Normalize}(f^{\\text{textP}}(\\tilde{\\boldsymbol{\\tau}}\\_j))$ with $f^{\\text{imgE}}(\\cdot)$ and $f^{\\text{textP}}(\\cdot)$ denoting the image encoder and text projection head, respectively. Note that In LoRS [1], no image projection head is used. The threshold $\\beta$ is used to determine positive and negative pairs, $\\sigma(\\cdot)$ denotes the sigmoid function, and $\\gamma$ is the temperature. $\\ell(\\cdot, \\cdot)$ refers to the binary cross-entropy loss. While this supervision primarily aims to mine cross-modal relationships, it inadvertently reinforces intra-modal similarities, ultimately leading to Modality Collapse.\n\n*[1] Low-Rank Similarity Mining for Multimodal Dataset Distillation, ICML 2024.*"}, {"review_id": "s8UVM3h5Hv", "reviewer": "Reviewer_DAj6", "summary": "This paper addresses the \"Modality Collapse\" problem which is common in Multimodal Dataset Distillation (MDD), where compressing datasets may lead to low-diversity representations and poor alignment across modalities. This paper proposes RepBlend framework, including two key techniques: 1) Representation Blending (RB): Mixes instance representations (instead of data examples) to enhance intra-modal diversity; 2) Symmetric Projection Trajectory Matching (SM): Balances the optimization process across modalities, by adding an image projection head (with image encoder frozen) and matching optimization trajectories for both modalities symmetrically.\n\nThe result shows improvement against previous SOTA methods on Flickr-30K and MS-COCO in image-text retrieval tasks. Moreover, its computational efficiency is also impressive, achieving 6.7x speedup in distillation time by avoiding training the entire image encoder, achieving 6.7x speedup in distillation time (due to frozen image encoder).\n\nOverall, this paper presents a simple, intuitive, yet effective and efficient method for a rapidly evolving domain (though a bit narrow at this moment), which could provide valuable insights for the multimodal research community.", "questions": "How many parameters do the models have in the experiments?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n\nMDD is a rapidly advancing and promising area of research. This paper identifies the \"modality collapse\" problem by providing a convincing analysis of its root cause, as illustrated in Figure 1, which depicts how the common methods such as LoRS produce poorly aligned and overly concentrated embeddings. In Figure 2, it further demonstrates how cross-modal supervision reinforces intra-modal similarity and leads to collapse. This problem formulation and explanation are all valuable and insightful.\n\nThe proposed solutions RB and SM are elegant and effective. The idea of RB is similar to MixUp but applied in the representation space, which smartly weakens the overly strong cross-modal supervision. SM is to mitigate the asymmetric optimization, by simply freezing the large image encoder and adding a lightweight image projection head.\n\nThe efficiency is also largely improved, thanks to the freezing of the image encoder. This leads to 6.7x speedup during the main distillation phase, and reduces peak GPU memory usage by more than half.\n\nThis paper is very well written and easy to read.\n\nWeaknesses\n\nIt’s unclear whether the method is still effective when scaling the model and data, because all the presented experiments were conducted on small models and datasets. The IR numbers are too small compared to SOTA models (we don’t need to achieve SOTA as well, but better to be more comparable).\n\nGiven the proposed method is to reduce the modality gap, demonstrated in image and text embeddings, more zero-shot tasks could be more convincing, such as INET classification, OCR-relevant retrieval (e.g. textcaps) etc.", "was_revised": true, "final_justification": "My concerns regarding scaling and insufficient evaluations were addressed during the rebuttal process. Considering that 1) the gap between baseline and the distilled data is still non-trivial, and 2) the proposed method could be insightful for the community, I would like to maintain my previous score.", "rebuttal": "Thank you for your time and constructive comments. We have responded to all your questions as follows and hope these address your concerns.\n\n**Q1 (Weaknesses 1):**\nIt’s unclear whether the method is still effective when scaling the model and data, because all the presented experiments were conducted on small models and datasets. The IR numbers are too small compared to SOTA models (we don’t need to achieve SOTA as well, but better to be more comparable).\n\n**A1:** Thank you for raising this important point. To address the concern, we have extended our method to a larger-scale setting using the ***LLaVA-cc3m*** dataset, which serves as the pretraining dataset for LLaVA and consists of 558k image-text pairs. We use approximately 60% of the data (about 334k pairs) for training and reserve a non-overlapping set of 10k pairs for validation. In addition, we evaluate our approach with more powerful encoders, including ***DiNo-v2*** (85,798,656 parameters) for vision and ***BGE-1.5*** (109,482,240 parameters) for text. The following results demonstrate that our method remains effective when scaling both model capacity and training data, while also showing clear superiority over the state-of-the-art competitor. We note that a performance gap against the baseline still exists, which remains a longstanding challenge. Addressing this gap will be a key focus of future work toward achieving lossless or near-lossless distillation.\n\n- *NFNet + Bert on LLaVA-cc3m dataset. The model trained on the full dataset performs: IR@1=9.13, IR@5=25.94, IR@10=36.34, TR@1=9.49, TR@5=26.08, TR@10=37.07.*\n\n| pairs | Methods | IR@1 | IR@5 | IR@10| TR@1 | TR@5 | TR@10\n| -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- |\n| 199 | LoRS     |   1.56|5.33|8.77|1.01|4.04|6.81\n| | **ours**     | **3.53**     | **12.38**    |**19.33**|**4.39**|**13.45**|**20.18**\n| 499 | LoRS     |1.96|6.72|10.55|1.41|5.11|8.51 \n| | **ours**     |**4.45**|**15.09**|**22.31**|**5.14**|**14.89**|**22.97** \n| 799 | LoRS     |1.67   |  5.87    |9.69|1.68|6.11|10.25 \n| | **ours** |**5.26**|**16.31**|**24.23**|**5.42**|**16.13**|**24.16**\n\n- *DiNo-v2 + BGE-1.5 on COCO dataset. The model trained on the full dataset performs: IR@1=22.70, IR@5=51.13, IR@10=65.26, TR@1=31.04, TR@5=61.96, TR@10=74.1.*\n\n| pairs| Methods | IR@1 | IR@5 | IR@10| TR@1 | TR@5 | TR@10\n| --------| -------- | -------- | -------- | -------- | -------- | -------- | -------- |\n| 199| LoRS     |1.59|6.12|10.59|1.62|6.15|10.47\n|      | **ours**     |  **12.52**    |**31.96**|**44.39**|**16.06**|**36.84**|**49.34**\n| 499| LoRS     |2.48|8.46|13.42|3.18|10.36|16.37\n|      | **ours**     |  **13.37**    |**33.09**|**45.69**|**16.90**|**39.38**|**52.14**\n| 799 | LoRS    |  2.95     |  9.9    |15.69|4.56|13.66|20.63 | \n| | **ours** |**13.68**|**33.58**|**45.93**|**17.14**|**39.76**|**52.92**\n\n*LLaVA-cc3m: https://github.com/haotian-liu/LLaVA/blob/main/docs/Data.md*\n\n*DiNo-v2: https://huggingface.co/timm/vit_base_patch16_224.dino*\n\n*BGE-1.5: https://huggingface.co/BAAI/bge-base-en-v1.5*\n\n---\n\n**Q2 (Weaknesses 2):**\nGiven the proposed method is to reduce the modality gap, demonstrated in image and text embeddings, more zero-shot tasks could be more convincing, such as INET classification, OCR-relevant retrieval (e.g. textcaps) etc.\n\n**A2:** Thank you for raising this point. To further validate the effectiveness of the reduced modality gap in our distilled dataset, we additionally evaluate zero-shot ***ImageNet*** classification and OCR-relevant retrieval on the ***TextCaps*** dataset [1]. Specifically, we randomly select 10 classes from ImageNet-1K and report the Top-1 and Top-5 zero-shot classification accuracies. For TextCaps, we measure retrieval performance over 3,166 validation samples.\nHere, '*Baseline*' denotes the model trained on the full COCO dataset and directly evaluated on these two tasks in a zero-shot manner. LoRS and Ours correspond to models trained with only 499 distilled COCO image-text pairs. The results, summarized below, show that under the same budget, models trained on our distilled dataset outperform LoRS,  narrowing the performance gap with the full-dataset baseline.\n\n- *ImageNet-10 zero-shot classification*\n\n| Methods     |Acc1 | Acc5 \n| -------- | -------- | -------- |\n| Baseline     |   36.4 | 83.2|\n| LoRS    |  21.4 | 74.4| \n| **Ours**    |  **27.6**  | **76.2**|\n\n\n- *TextCaps zero-shot retrieval*\n\n| Methods     |IR@1 | IR@5         | IR@10     |  TR@1  | TR@5         | TR@10 \n| -------- | -------- | -------- |-------- | -------- |-------- | -------- |\n| Baseline     |   7.4|18.1  | 26.2   |5.1|14.7|21.3\n| LoRS    |    1.7  |  5.1   |8.4|0.4|1.7|3.1|\n| **Ours**    |   **3.1**   |   **9.4**   |**14.5**|**1.9**|**6.2**|**10.3**\n\n---\n**Q3 (Questions 1):** How many parameters do the models have in the experiments?\n\n**A3:** Thanks for the question. In our experiments, we have considered four image encoders, including ***ViT, RegNet, ResNet-50***, and ***NFNet***, and two text encoders including ***BERT*** and ***DistilBERT*** (same as baseline LoRS). To evaluate the scalability of our method, we further extended it to more powerful encoders, namely ***DiNo-v2*** for vision and ***BGE-1.5*** for language. The selected models encompass representative architectures with varying capacities, and their parameters are summarized below.\n\n|  | Models | Paras |Official link|\n| -------- | -------- | -------- |-------- |\n|Image encoders |ViT|5,717,416 | https://huggingface.co/WinKawaks/vit-tiny-patch16-224 |\n||RegNet|9,262,984|https://huggingface.co/timm/nf_regnet_b1.ra2_in1k |\n||ResNet-50|23,508,032 |https://huggingface.co/microsoft/resnet-50\n|     | NFNet     | 32,769,488     |https://huggingface.co/timm/nfnet_l0.ra2_in1k |\n|| DiNo-v2|85,798,656|https://huggingface.co/timm/vit_base_patch16_224.dino|\n| Text encoders     | BERT     |   109,482,240   | https://huggingface.co/google-bert/bert-base-uncased\n||DistilBERT|66,362,880| https://huggingface.co/distilbert/distilbert-base-uncased\n||BGE-1.5|109,482,240|https://huggingface.co/BAAI/bge-base-en-v1.5|\n\nThe superiority demonstrated across these diverse models not only validates the effectiveness of the proposed method, but also highlights its scalability and generalization capability.\n\n*[1] TextCaps: a Dataset for Image Captioning with Reading Comprehension, ECCV 2020.*"}, {"review_id": "068QWQwTt1", "reviewer": "Reviewer_t2RZ", "summary": "The authors examine the problem of Multimodal Dataset Distillation (MDD), where the goal is to generate a synthetic dataset distilled from a real one, that can be used to train a CLIP style model while using fewer data points. The authors identify issues with existing techniques for this goal, and through representation blending and improvements on the optimization process demonstrate improvements over existing methods.", "questions": "- I would be grateful if the authors could elaborate on the point I raised above regarding Appendix B, and whether there are any other underlying assumptions in the arguments made there.\n\n- Also, I would be grateful if the authors could let me know whether they have tried Representation Blending in the unimodal setting (as it seems that it might be useful there as well).", "limitations": "The authors have adequately described the limitations and potential impact of their work.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n\n- The paper is clearly written, and the points made by the authors are mostly well supported.\n\n- Both ideas presented by the authors are solid and novel in this topic as far as I am aware:\n\n  - Representation blending is well motivated. The authors use an example where they blend the representations of the model with noise, and show that blending by itself alleviates the identified modality collapse issue. This is a good foundation for blending representations with other samples, which is clearly shown to improve the intra-modal similarity between samples in Figure 3.\n\n  - Symmetric Projection Trajectory Matching is a simple idea at its core (using a projection head for the image side of the architecture, instead of just the image encoder). Nevertheless, this is shown to greatly improve the loss in the image encoder in Figure 4. This is important, as matching trajectories is key to the synthetic dataset having similar behavior to the real one, in both image and text domains.\n\n- Experimental evaluation is also convincing. The authors use a similar set of experiments to state-of-the-art prior work (LoRS) and clearly improve upon that. This demonstrates the solid performance of the methodology presented in this paper. \n\nWeaknesses\n\n- The authors make a theoretical argument for modality collapse, with the analysis present in Appendix B. Their argument is that the way the synthetic dataset is updated, embeddings eventually become closer to each other, even across samples that are only slightly similar. However, some steps of the analysis imply some approximations (specifically, that the encoders are trained well enough for negative samples to have approximately orthogonal embeddings). I believe that the authors can improve this analysis by clearly stating hidden assumptions about the embeddings in Appendix B, and even expanding upon the visual analysis in Figure 2 (for example, by explicitly including a CR-Iteration plot, as CR is the most salient metric).\n\n- In Table 3, the NFNet+BERT evaluation model case is excluded. For completeness, it should be added to this table as well (although it can be inferred from other tables).\n\n- Some more minor points:\n\n  - Tables 1 and 2 should float to the bottom of the page for readability.\n\n  - Table 4 can also become horizontal instead of vertical, so that it can float to top after Table 3.\n\n  - I would recommend highlighting (bolding / different color) the difference between Equations 5 and 6, since at first glance they are too similar (the difference being the use of image encoder or image projection head).\n\n  - In line 168, “blends” -> “blend”.", "was_revised": true, "final_justification": "As mentioned in my original review, I think this is a solid paper, and reading the authors' responses to both mine and other Reviewers' comments further solidifiy this belief (with my only concern regarding implicit assumptions for modality collapse having been addressed). As such, I am keeping my recommendation of \"Accept\" for this work.", "rebuttal": "Thank you for your time and insightful feedback. We have provided detailed responses to all your questions below and hope they effectively resolve your concerns.\n\n**Q1 (Weaknesses 1 & Questions 1):** The analysis in Appendix B relies on implicit assumptions (e.g., negative sample embeddings being approximately orthogonal) that should be explicitly stated.\n\n**A1:** Thanks for the insightful suggestion. This deviation indeed relies on the assumption that both the image and text encoders are well-pretrained, such that negative pairs are approximately orthogonal in the high-dimensional embedding space. This assumption is consistent with our practical setup, where we employ ImageNet-1K–pretrained image encoders (e.g., NFNet) and BookCorpus + English Wikipedia–pretrained text encoders (e.g., BERT). The validity of this assumption has also been supported by several prior works [1, 2]. To further substantiate it, we empirically computed the relevant values, obtaining a cross-modal negative pair mean cosine similarity of 0.0003 and an intra-modal (image) mean cosine similarity of 0.004.  For clarity, this assumption is explicitly emphasized in the revised version.\n\n---\n    \n**Q2 (Weaknesses 1):** Suggests expanding the visual analysis, for instance by adding a CR‑Iteration plot, to strengthen the argument.\n\n**A2:** Thank you for the suggestion. We chose intra-modal similarity for illustration because it is more intuitive to understand compared with CR. The CR–Iteration plot shows the same trend as the current similarity–Iteration plot, since CR is positively correlated with intra-modal similarity; higher intra-modal similarity corresponds to higher CR, and in particular, CR provides a more numerically salient perspective. As figure updates are not permitted during rebuttal, we will incorporate this additional plot in the final version.\n\n---\n**Q3 (Weaknesses 2):**\nIn Table 3, the NFNet+BERT evaluation model case is excluded. For completeness, it should be added to this table as well (although it can be inferred from other tables).\n\n**A3:** Thanks for pointing this out. We have included the NFNet + BERT case in Table 3 to ensure completeness and consistency across the reported results.\n\n*Table 3. Cross-architecture generalization. The distilled data are synthesized using NFNet+BERT and evaluated across different architectures. Evaluations are conducted on Flickr-30K under the 500-pair setting. For fairness, both LoRS [3] and ours synthesize one fewer pair, e.g., 499 pairs.*\n| Evaluate Model  | Methods        | IR@1         | IR@5         | IR@10        | TR@1         | TR@5         | TR@10        |\n|-----------------|----------------|--------------|--------------|--------------|--------------|--------------|--------------|\n| NFNet+BERT | TESLA-VL      | 1.1±0.2 | 7.3±0.4 | 12.6±0.5 | 5.1±0.2 | 15.3±0.5 | 23.8±0.3\n|                 | LoRS          | 10.0±0.2 | 28.9±0.7 | 41.6±0.6 | 15.5±0.7 | 39.8±0.4 | 53.7±0.3\n|                 | **Ours**       | **17.0±0.6** | **42.5±0.5** | **55.9±0.6** | **22.5±0.4** | **53.2±0.3** | **66.7±0.3** |\n| ResNet+BERT | TESLA-VL       | 3.0±0.2    | 10.8±0.5   | 17.0±0.8   | 6.0±0.9    | 18.8±0.7   | 27.7±1.2   |\n|                 | LoRS           | 3.3±0.2    | 12.7±0.3   | 20.4±0.2   | 6.8±0.2    | 19.6±1.3   | 31.1±0.3   |\n|                 | **Ours**       | **4.2±0.2**| **14.1±0.2**| **23.6±0.6**| **8.4±0.2**| **23.1±0.8**| **35.0±1.3**|\n| RegNet+BERT | TESLA-VL       | 3.2±0.8    | 11.1±1.8   | 17.5±1.3   | 5.8±0.1    | 18.6±0.6   | 28.1±1.0   |\n|                 | LoRS           | 3.5±0.1    | 12.6±0.3   | 21.1±0.4   | 6.8±0.3    | 20.8±0.3   | 30.2±0.3   |\n|                 | **Ours**       | **3.9±0.2**| **13.9±0.3**| **24.0±0.6**| **7.9±0.3**| **24.2±0.3**| **36.2±1.1**|\n\n\n***\n**Q4 (Weaknesses 3):** Some more minor points. 1) Tables 1 and 2 should float to the bottom of the page for readability. 2) Table 4 can also become horizontal instead of vertical, so that it can float to top after Table 3. 3) I would recommend highlighting (bolding / different color) the difference between Equations 5 and 6, since at first glance they are too similar (the difference being the use of image encoder or image projection head). 4) In line 168, “blends” -> “blend”.\n\n**A4:** Thank you for the helpful suggestions. We will (1) adjust the float positions of Tables 1–2, (2) reformat Table 4 horizontally, (3) highlight the difference between Eq. 5 and Eq. 6, and (4) fix the typo “blends” → “blend” in line 168 in the final version.\n\n***\n**Q5 (Questions 2):**\nAlso, I would be grateful if the authors could let me know whether they have tried Representation Blending in the unimodal setting (as it seems that it might be useful there as well).\n\n**A5:** Thank you for the insightful suggestion. We have evaluated the proposed Representation Blending (RB) in the unimodal setting. Specifically, we adopt SRe2L [4] as the baseline and incorporate RB into its distillation process. RB effectively mitigates feature redundancy and encourages the distillation of more class-discriminative representations. The results below show that integrating RB consistently improves the performance of SRe2L across different IPC budgets.\n\n- *Distill CIFAR-100 with ResNet-18*\n    \n| IPC | SRe2L [4] | SRe2L + RB |\n| -------- | -------- | -------- |\n| 1     | 7.3     | **7.7**    |\n| 10     | 26.3     | **27.4**     |\n| 50     |  51.6    |  **52.8**    |\n    \n*[1] Relaxing Contrastiveness in Multimodal Representation Learning, WACV 2023.*\n\n*[2] Uncovering Meanings of Embeddings via Partial Orthogonality, NeurIPS 2023.*\n\n*[3] Low-Rank Similarity Mining for Multimodal Dataset Distillation, ICML 2024.*\n\n*[4] Squeeze, Recover and Relabel: Dataset Condensation at ImageNet Scale From A New Perspective. NeurIPS 2023.*"}, {"review_id": "DvfVy9rShI", "reviewer": "Reviewer_nvV9", "summary": "The paper first identifies the \"modality collapse\" in multimodal dataset distillation and then provides theoretical and empirical analysis on the cause. To mitigate, the authors propose RepBlend to enhance intra-modal diversity and inter-modal similarity. And lastly they have the appraoch tested on a bunch of datasets including Flickr-30K and MS-COCO, showing the best performance among the compared methods.", "questions": "Q1. To reach the full performance, roughly how many samples are needed? How does this number vary across different methods? Hope to see the plot with more numbers.\n\nQ2. How's the performance with stronger base models, like DinoV2, or BGE, NV-Embed?\n\nQ3. How's the zero-shot performance, how well can the trained model generalize to other datasets?", "limitations": "See weaknesses and questions", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n\nS1. The authors identified the \"modality gap\" issue in the realm of multimodal dataset distillation, and attributed the issue to the over-compression behavior through analysis. Motivation is intuitive and reasonable.\n\nS2. The multi-modal results presented in the paper demonstrate a noticeable improvement over previous methods\n\nS3. Overall writing is good, easy to follow.\n\n**Weaknesses**\n\nW1. Similar approaches have already been introduced in prior work FuseMix [C1] and should be discussed. \n\nW2. The authors employed the intra-modal similarity and modality gap metrics but didn't not discuss this in detail, e.g., how will these individually affect the performance. \n\nW3. The experiments are focused on multi-modal performance only. However, some experiments on unimodal task will strengthen the paper. To support the claim that the approach reduces both intra-modal similarity and modality gap, showing evidence that this improves not only multi-modal but also uni-modal performance is helpful.\n\nC1. Noël Vouitsis, Zhaoyan Liu, Satya Krishna Gorti, Valentin Villecroze, Jesse C. Cresswell, Guangwei Yu, Gabriel\nLoaiza-Ganem, and Maksims Volkovs. Data-efficient multimodal fusion on a single gpu. , CVPR 2024", "was_revised": true, "final_justification": "I'll maintain my score of 4. The author's response resolved my questions on the metric correlation, zero-shot and unimodal performance. But I'm still questioning the originality of the idea (similar to FuseMix). However, the proposed method significantly outperforms the listed baselines which gives me more confidence in accepting the paper. After revisiting all these aspects, I'm more comfortable with maintaining my score.", "rebuttal": "**Q1 (Weaknesses 1):** Similar approaches have already been introduced in prior work FuseMix [1] and should be discussed.\n\n**A1:** Thank you for mentioning this important prior work. FuseMix is a latent data augmentation approach proposed to facilitate efficient multimodal alignment. The distinction between FuseMix and our method lies in both motivation and mechanism. In the final version, we will include a dedicated section for this discussion.\n\n- **Motivation**:\nFuseMix assumes that well‑pretrained unimodal encoders already capture rich semantics, leveraging them for multimodal alignment without expensive full‑model training. It employs latent data augmentation to mitigate data scarcity and the limited capacity of lightweight adapters. In contrast, our method targets the practical challenge of modality collapse in multimodal dataset distillation. Through theoretical analysis, we show that Representation Blending alleviates this issue, enhancing intra‑modal diversity and cross‑modal alignment.\n       \n- **Mechanism**:\nFuseMix is essentially a data augmentation method, where even modality‑agnostic perturbations such as Gaussian noise can also boost baseline performance. In contrast, our approach directly addresses modality collapse. As shown in Section 3.2, Gaussian noise introduces semantically meaningless signals that enlarge the modality gap and degrade cross‑modal alignment, leading to a performance drop. These results underscore the distinctiveness of our design for multimodal dataset distillation (MDD).\n\n---\n**Q2 (Weaknesses 2):** The authors used intra-modal similarity and modality gap but did not detail their individual impact on performance.\n\n**A2:** Thank you for the question. The intra-modal similarity (Sim), concentration ratio (CR), and modality gap (Gap) are key metrics for evaluating the quality of the distilled dataset. In general, smaller Sim and CR indicate greater intra-modal diversity, while a smaller Gap reflects stronger cross-modal alignment. Together, these factors lead to better retrieval performance. However, as these metrics are interdependent, it is difficult to vary one without affecting the others. So here we provide two cases to generally assess their individual effects. \n\n- *Distilling 100 pairs on Flickr-30k using NFNet+BERT.*\n\n|| Sim (CR) | Gap | IR@1         | IR@5         | IR@10        | TR@1         | TR@5         | TR@10\n|-------- | -------- | -------- | -------- |-------- |-------- |-------- |-------- |-------- |\n|baseline| 0.522 (0.95)    | 0.327     | 8.30 | 24.10 | 35.10 | 11.80 | 35.80 | 49.20\n|case 1| 0.511 (0.54)    | 0.318     |8.64|25.64|37.24|13.03|37.63|51.6\n|case 2| 0.512 (0.57)    | 0.23     |11.50 | 32.00 | 44.50 | 16.20 | 41.70 | 55.50\n\n- Baseline vs. Case 1: A substantial reduction in intra-modal similarity (CR from 0.95 → 0.54) led to a 2.1% increase in IR@10 and a 2.4% increase in TR@10.\n\n- Case 1 vs. Case 2: A further notable reduction in the cross-modal gap (0.318 → 0.23) yielded an additional 7.3% increase in IR@10 and 3.9% in TR@10 compared with Case 1.\n\nThese results indicate that reducing intra-modal concentration, and narrowing the modality gap, provides complementary benefits and leads to substantial performance gains.\n\n---\n**Q3 (Weaknesses 3):** Some experiments on unimodal task will strengthen the paper. \n\n**A3:** Thank you for the insightful suggestion. We evaluated the proposed Representation Blending (RB) in the unimodal setting by adopting SRe2L [2] as the baseline and integrating RB into its distillation process. RB effectively reduces feature redundancy and promotes more class-discriminative representations. As shown below, RB consistently improves SRe2L’s performance across various IPC budgets.\n\n- *Distill CIFAR-100 with ResNet-18*\n\n| IPC | SRe2L [2] | SRe2L + RB |\n| -------- | -------- | -------- |\n| 1     | 7.3     | **7.7**    |\n| 10     | 26.3     | **27.4**     |\n| 50     |  51.6    |  **52.8**    |\n\n---\n**Q4 (Questions 1):** To reach the full performance, roughly how many samples are needed? How does this number vary across different methods? Hope to see the plot with more numbers.\n\n**A4:** We extended the comparison to larger pair settings and provide a detailed table since figures are not allowed in the rebuttal. Both LoRS and our method show only marginal gains as pairs increase, reflecting the well-known saturation issue in dataset distillation where added pairs bring limited diversity. Nevertheless, our method maintains consistent superiority over LoRS under the same conditions. In the future, we will further explore ways to overcome this saturation toward lossless or nearly lossless MDD.\n\n- *NFNet + Bert on Fliker-30k dataset. The model trained on the full dataset performs: IR@1=23.16, IR@5=53.98, IR@10=66.62; TR@1=33.8, TR@5=65.7, TR@10=76.9.*  \n\n| pairs |Methods| IR@1 | IR@5 | IR@10| TR@1 | TR@5 | TR@10\n| -------- | -------- | -------- | -------- | -------- | -------- | -------- |-------- |\n|  199  |LoRS|  8.6    | 25.3     |36.6|14.5|38.7|53.4\n||**Ours**|**12.7**   |  **34.7**    |**47.6**|**18.6**|**46.0**|**60.0**\n|  299  |LoRS|   9.6 |   27.7   | 39.3|13.82|39.32|53.5\n||**Ours**|**14.3**  |   **36.1**   |**50.6**|**20.6**|**49.2**|**62.3**\n|  499  |LoRS |  10.0    |  28.9    |41.6|15.5|39.8|53.7\n||**Ours**|**17.0**    |   **42.5**   |**55.9**|**22.5**|**53.2**|**66.7**\n|  699  | LoRS|  11.6   |    31.5  |44.1|16.3|41.9|57.0\n||**Ours**|**17.4**  |   **43.5**   |**56.8**|**22.8**|**53.4**|**67.1**\n|  799  | LoRS|  11.5   |   31.5   |44.1|15.4|40.6|56.8\n||**Ours**|**17.6**      |**44.0**      |**56.9**|**22.2**|**53.7**|**67.0**\n\n- *NFNet + Bert on COCO dataset. The model trained on the full dataset performs: IR@1=14.6, IR@5=38.9, IR@10=53.2; TR@1=20.6, TR@5=46.8, TR@10=61.3.*  \n\n|pairs| Methods | IR@1 | IR@5 | IR@10| TR@1 | TR@5 | TR@10\n| -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- |\n|  199  |LoRS| 2.4     |  9.3    |15.5|4.3|14.2|22.6\n||**Ours**|  **6.1**    |   **19.3**   |**29.8**|**6.9**|**21.8**|**32.3**\n|  299  |LoRS|   2.5   |   9.5 |15.9|4.4|15.8|25.1\n||**Ours**|   **6.6**  |   **20.5**   |**29.5**|**6.9**|**21.7**|**32.2**\n|  499  |LoRS|  2.8    |   9.9   |16.5|5.3|18.3|27.9\n||**Ours**| **6.2**    |  **19.9**    |**30.6**|**7.0**|**22.0**|**32.9**\n|  699  |LoRS|  3.0    |   10.9   |17.7|6.0|19.0|28.1\n||**Ours**| **6.5**     |    **20.1**  |**31.0**|**7.5**|**22.2**|**33.4**\n|  799  |LoRS|3.1|10.7|17.4|5.9|19.1|28.5\n||**Ours**| **6.7**    |     **20.8** |**31.7**|**7.7**|**22.5**|**33.7**\n\n---\n**Q5 (Questions 2):** How's the performance with stronger base models, like DinoV2, or BGE, NV-Embed?\n\n**A5:** Thank you for the question. We have evaluated our approach with more powerful encoders, like ***DiNo-v2*** (85,798,656 parameters) for vision and ***BGE-1.5*** (109,482,240 parameters) for text. The results demonstrate that our method remains effective when scaling model capacity.\n\n- *DiNo-v2 + BGE-1.5 on COCO dataset. The model trained on the full dataset performs: IR@1=22.70, IR@5=51.13, IR@10=65.26, TR@1=31.04, TR@5=61.96, TR@10=74.1.*\n\n| pairs| Methods | IR@1 | IR@5 | IR@10| TR@1 | TR@5 | TR@10\n| --------| -------- | -------- | -------- | -------- | -------- | -------- | -------- |\n| 199| LoRS     |1.59|6.12|10.59|1.62|6.15|10.47\n|      | **ours**     |  **12.52**    |**31.96**|**44.39**|**16.06**|**36.84**|**49.34**\n| 499| LoRS     |2.48|8.46|13.42|3.18|10.36|16.37\n|      | **ours**     |  **13.37**    |**33.09**|**45.69**|**16.90**|**39.38**|**52.14**\n\n*DiNo-v2: https://huggingface.co/timm/vit_base_patch16_224.dino*\n\n*BGE-1.5: https://huggingface.co/BAAI/bge-base-en-v1.5*\n\n---\n**Q6 (Questions 3):** How's the zero-shot performance, how well can the trained model generalize to other datasets?\n\n**A6:** Thank you for raising this point. We further validated the proposed method with zero-shot ImageNet classification (10 randomly selected classes) and OCR-relevant retrieval on TextCaps. Compared with LoRS, our method trained on only 499 distilled pairs achieves higher accuracy and retrieval performance, thereby narrowing the gap with the full-data baseline.\n\n- *ImageNet-10 zero-shot classification*\n\n| Methods     |Acc1 | Acc5 \n| -------- | -------- | -------- |\n| Baseline     |   36.4 | 83.2|\n| LoRS    |  21.4 | 74.4| \n| **Ours**    |  **27.6**  | **76.2**|\n\n- *TextCaps zero-shot retrieval*\n\n| Methods     |IR@1 | IR@5         | IR@10     |  TR@1  | TR@5         | TR@10 \n| -------- | -------- | -------- |-------- | -------- |-------- | -------- |\n| Baseline     |   7.4|18.1  | 26.2   |5.1|14.7|21.3\n| LoRS    |    1.7  |  5.1   |8.4|0.4|1.7|3.1|\n| **Ours**    |   **3.1**   |   **9.4**   |**14.5**|**1.9**|**6.2**|**10.3**\n\nBeyond zero-shot performance, we also extend our method to the large-scale ***LLaVA-cc3m*** dataset, which contains 558k image-text pairs. We train on ~60% (334k pairs) and validate on a separate 10k-pair set. The results demonstrate that our method generalizes well to large-scale datasets.\n\n- *NFNet + Bert on LLaVA-cc3m dataset. The model trained on the full dataset performs: IR@1=9.13, IR@5=25.94, IR@10=36.34, TR@1=9.49, TR@5=26.08, TR@10=37.07.*\n\n| pairs | Methods | IR@1 | IR@5 | IR@10| TR@1 | TR@5 | TR@10\n| -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- |\n| 199 | LoRS     |   1.56|5.33|8.77|1.01|4.04|6.81\n| | **ours**     | **3.53**     | **12.38**    |**19.33**|**4.39**|**13.45**|**20.18**\n| 499 | LoRS     |1.96|6.72|10.55|1.41|5.11|8.51 \n| | **ours**     |**4.45**|**15.09**|**22.31**|**5.14**|**14.89**|**22.97** \n| 799 | LoRS     |1.67   |  5.87    |9.69|1.68|6.11|10.25 \n| | **ours** |**5.26**|**16.31**|**24.23**|**5.42**|**16.13**|**24.16**\n\n*[1] Squeeze, Recover and Relabel: Dataset Condensation at ImageNet Scale From A New Perspective. NeurIPS 2023.*\n\n*[2] TextCaps: a Dataset for Image Captioning with Reading Comprehension, ECCV 2020.*\n\n*LLaVA-cc3m: https://github.com/haotian-liu/LLaVA/blob/main/docs/Data.md*"}]

Author rebuttal:

Meta-review: The paper addresses Modality Collapse in Multimodal Dataset Distillation (MDD). Modality Collapse is a phenomenon in which  dataset compression leads to low diversity and poor alignment across modalities. The authors propose RepBlend, which consists of two components: Representation Blending (RB) to enhance intra-modal diversity by mixing instance representations, and Symmetric Projection Trajectory Matching (SM) to balance optimization across modalities by using a frozen image encoder. RepBlend shows superior performance in image-text retrieval tasks, improving over state-of-the-art methods while achieving a 6.7x speedup in distillation time by not requiring the training of the entire image encoder.

Reviewers generally agree that this paper makes significant contributions by being the first to formally define and analyze Modality Collapse in Multimodal Dataset Distillation (MDD), and by linking it to over-compression and contrastive supervision. They find that the paper introduces novel solutions like Representation Blending, which diversifies intra-modal features effectively, and Symmetric Projection Trajectory Matching, which resolves asymmetric supervision to improve cross-modal alignment. Reviewers recognize the strong empirical validation, as the proposed method consistently outperforms state-of-the-art techniques across multiple datasets and modalities. Additionally, reviewers appreciate the substantial efficiency gains achieved (6.7x speedup and significant memory reduction) thanks to lightweight projection-head matching. Finally, reviewers unanimously praise the writing and technical exposition as being clear and well-supported.

Despite the generally positive initial opinions from the reviewers, paper has several areas pointed out for improvement:

+ The lack of explicit definition for the baseline method LoRS and insufficient visualization, such as diversity metrics or attention maps, weakens the claims about enhanced diversity. In rebuttal, the authors provided a quantitative comparison between LoRS and the proposed approach based on the Concentration Ration, demonstrating superior diversity after distillation.
+ The experiments are conducted only on small models and datasets, which raised concerns about scalability, and that there is a lack of zero-shot tasks to showcase the effectiveness. In rebuttal, the authors provided a number of new experimental results on different datasets, with larger models, and on zero-shot tasks. These results further demonstrate the effectiveness of the proposed approach and the importance of modality collapse.

The reviewers were satisfied with the author clarifications and rebuttals, and in summary are in agreement that both main ideas presented are solid and novel, with experimental evaluation convincingly demonstrating the superior performance of the proposed approach. The recommendation is thus to Accept.
Decision: Accept (poster)