Towards Multimodal Active Learning: Efficient Learning with Limited Paired Data
Authors: Jiancheng Zhang
Yinglun Zhu
Venue: Accepted by TMLR ()

Abstract: Active learning (AL) is a principled strategy to reduce annotation cost in data-hungry deep learning. However, existing AL algorithms focus almost exclusively on unimodal data, overlooking the substantial annotation burden in multimodal learning. We introduce the first framework for multimodal active learning with unaligned data, where the learner must actively acquire cross-modal alignments rather than labels on pre-aligned pairs. This setting captures the practical bottleneck in modern multimodal pipelines, where unimodal features are easy to obtain but high-quality alignment is costly. We develop a new algorithm that combines uncertainty and diversity principles in a modality-aware design, achieves linear-time acquisition, and applies seamlessly to both pool-based and streaming-based settings. Extensive experiments on benchmark datasets demonstrate that our approach consistently reduces multimodal annotation cost while preserving performance; for instance, on the ColorSwap dataset it cuts annotation requirements by up to 40% without loss in accuracy.

Published in Transactions on Machine Learning Research (02/2026)
Towards Multimodal Active Learning: Efficient Learning with Limited Paired Data
Jiancheng Zhang
jzhan74550 edu
University of Colifornia, Riverside
Yinglun Zhu
yahu edu
University of California, Riverside
Reviewed on OpenReview:
Abstract
Active learning (AL) a principled strategy to reduce annotation cost in data-hungry deep learning. However, existing AL algorithms focus almost exclusively unimodal data, overlooking the substantial annotation burden in multimodal learning. We introduce the first framework for multimodal active learning with unaligned data, where the learner must actively acquire cross moosl aligmments rather than labels on pre-aligned pairs. This setting captures the practical bottleneck in modern multimodal pipelines, where unimodal features are easy obtain but high-quality alignment is costly. We develop a new algorithm that combines uncertainty and diversity principles in a modality-aware design, achieves linear- time acquisition, and applies scamlessly to both pool-based and streaming- based settings. Extensive experiments on benchmark datasets demonstrate that approach consistently reduces multimodal annotation cost while preserving performance; for instance, on the ColorSwap dataset it cuts annotation requirements by up to 40% without loss in accuracy.
1 Introduction
Deep learning has achieved remarkable success across a wide range of applications, but its effectiveness often hinges on access to large amounts of annotated training data. Active learning (AL) has long been viewed as a promising approach to reduce annotation cost by selectively querying the most informative instances for laboling (Settles, 2009). Thcoretically, AL can exponentially reduce the amount of labeled data required (Zhu & Nowak, 2022a), and empirically, it has delivered consistent gains in data efficiency for deep models (Sener & Savarese, 2018; Ash al., 2020; Citovsky et al., 2021; Saran et al., 2023; Zhang al., 2024a), as well more recently for large language models (Margatina et al., 2023; Bhatt et al., 2024; Yuan et al., 2024).
Despite recent progress, most AL methods operate in unimodal and unidirectional setting, where the learner queries class labels for given features inputs. Multimodal extensions remain limited: Shen et al. (2023) studies multimodal AL with pre-aligned vision-language pairs, effectively reducing the task to unidirectional AL on composite inputs. Zhang et al. (2024b) applies AL to a specific multimodal similarly treats one modality input and the other as annotation. As result, existing approaches sidestep the core challenge of actively discovering cross- modal correspondences in unaligned data.
captioning- wwich
In this paper, we introduce the first setting of multimodal active learning with unaligned data, where the learner begins with independent vision and language features and must actively acquire cross-modal alignments. Unlike unimodal AL or pro-aligned multimodal AL, our setting requires deciding both which modality to query from and how to align instances across modalilies. This formulation raises two qualitatively new challenges: (i) bidirectional alignment, since annotation can begin from either vision-to-language or language-to-vistt and different choices lead to distinct annotation sets and lcarning trajectories; and (fi) large cross-modal
Prrjeec lead and corresponding author.

---

Published in Transactions on Machine Learning Research (02/2026)
candidate space, since evaluating the utilityy of an instance requires reasoning over potential matches across the entire other modality, which may contain millions of unique candidates (Gadre et al., 2024). Naively scoring all image text pairs scales quadratically, making classical AL strategies computationally infeasible.
Our setting is directly motivated by modern multimodal pipelines, where raw modality-speciffc features can be obtained cheaply at scale, but high-quality alignment is expensive, domain-specific, and often the true bottleneck (Gadre et al., 2024; Bai et al., 2024). This challenge is especially acute in specialized domains (or when adapting pretrained models to such domains), such as medical imaging (Chen & Hong, 2024) and autonomous driving (Ge et al., 2023), where multimodal annotation is both costly and essential.
Our contributions. Our main contributions are as follows:
(i) We introduce the problem of multimodal active learning with unaligned data, clarifying how it differs fundamentally from unimodal AL and existing multimodal AL with pre-aligned data.
(ii) We develop a new algorithm that integrates uncertainty and diversity principles in a modality-aware design, achieves linear-lime complexity in the number of unaligned instances, and applies seamlessly to both pool-based and streaming-based scenarios.
(iii) We conduct extensive experiments on benchmark datasets, demonstrating consistent annotation savings (up to 40% on ColorSwap) while maintaining competitive performance.
Paper organization. The remainder of the paper is organized as follows. Section 2 reviews related work. Section 3 introduces our problem formulation and highlights the unique challenges of multimodal active learning with unaligned data. Section 4 presents our algorithm, logether with its complexity analysis and extensions. Section 5 reports the main experimental results, and Section 6 provides additional analysis. Section 7 concludes the paper. Proofs, implementation details, and supplementary experiments are deferred to the Appendix.
2 Related Work
Active Learning. Active learning (AL) studies how to train accurate models with fewer annotations by selectively querying informative samples (Settles, 2009). A substantial body of theoretical work establishes provable advantages of AL over passive learning under various assumptions (Castro & Nowak, 2007; Balcan et al., 2007; Dasgupta et al., 2009; Hanneke, 2014; Zhang & Chaudhuri, 2014; Krishnamurthh et al., 2019; Puchkin & Zhivotovskiy, 2021; Zhu & Nowak, 2022b,a). On the empirical side, AL has been shown to improve data efficioncy in deep learning, including batch-mode methods and strcaming scttings (Sener & Savarese, 2018; Ash et al., 2020; Citovsky et al., 2021; Ash et al., 2021; Wang et al., 2022; Saran et al., 2023; Zhang el al., 2024a), and more recently for large pretrained models (Margatina et al., 2023; Bhatt et al., 2024; Yuan et al., 2024).
Most existing AL methods focus on the unimodal sctting with unidirectional annotation: given unlabeled feature representations, the learner selectively queries class labels from small, fixed label set. Multimodal AL is far less explored. The most relevant effort is the pre-aligned multimodal AL. framework of Shen et al. (2023), which assumee that vision-language pairs are already aligned and therefore restricts queries to class labels for these aligned pairs. Since the alignment is free, the problem effectively reduces to unidirectional AL on composite inputs rather than addressing the core challenge of discovering cross-modal correspondences. Other applications of AL to multimodal tasks, such as vidco captioning (Zhang et al., 2024b), follow a similar path: one modality (e.g., video) is treated as input while the other (e.g., text) serves as annotation, keeping the process strictly unidirectional and structurally unimodal.
In contrast, we introduce the first multimodal AL sctting that supports bidirectional alignment with unaligned data: the learner is provided with independent vision and language features and must actively acquire meaningful cross-modal correspondences, either from images to text or from text to images. This setting is directly motivated by modern multimodal pipelines such as CLIP (Radford et al., 2021) and SigL.IP (Zhai et al., 2023), where unimodal features are easy to obtain at scale, but high-quality aligument is expensive, domain-specific, and often the true bottleneck.
2

---

Published in Transactions on Machine Learning Research (02/2026)
Multimodal learning. Multimodal learning seeks to integrate information from diverse modalities such text, images, and audio to improve learning performance (Baltrušaitis et al., 2018; Liang et al., 2024). Farly approaches relied on supervised labels, where multimodal features were annotated with classification labels, making the process closely resemble standard unimodal learning. More recently, multimodal learning has shifted toward supervision from paired multimodal data, where one modality provides a supervision signal for another (Zong et al., 2024). A prominent example is CLIP (Radford et al., 2021) and its variants (Zhai et al., 2022, 2023), which leverage paired image-text data to train contrastive objectives that align representations across modalities.
Many recent works study problems related to multimodal aligmment but with goals that differ from active learning with unaligned data. For example, Han et al. (2024) focuses on correcting mismatched cross-modal pairs under the assumption that most pairs are already correctly aligned, while Kimura et al. (2025) studies aligning independently pretrained unimodal encoders without performing cross-modal data alignment.
While large-scale, noisily aligned multimodal data can be scraped from the web, it is increasingly recognized that training hirf-performing multimodal models requires high-quality, well-aligned datasets (Gadre et al., 2024; Bai et al., 2024). This need is even more pronounced in specialized domains such medical imaging (Chen & Hong, 2024) and autonomous driving (Ge et al., 2023), where careful multimodal annotation is both critical and costly. These challenges highlight the importance of developing efficient methods that can learn effectively from fewer paired examples. Although recent work has explored active learning for multimodal tasks with label annotations (Shen et al., 2023), to the best our knowledge, our work is the first to design a multimodal active learning algorithm specifically tailored for pairing annotation.
Data selection. Data selection is closcly related to active learning, aiming to construct a high-quality subset of data for more efficient or effective model training. The key distinction lies in the availability of labels or pairings: active learning selects data points before annotation, whereas data selection assumes a fully labeled paired datasct. From this perspective, active learning is strictly more challenging, as it must operate without access to labeling or pairing information.
Data selection methods have been shown to reduce training cost (Schreiber et al., 2020; Mindermann et al., 2022; Sorscher et al., 2022; Yang et al., 2023; Shen et al., 2024), and in some cases even improve performance by removing duplicated or noisy data (Lee et al., 2022; Tirumala et al., 2023; Xia et al., 2024). In the multimodal domain, researchers have proposed metries such as CLIPScore (Hessel et al., 2021) and its extensions (Wang et al., b024a,b; Joshi et al., 2024) to evaluate the quality of image-ext pairs, ena caabling filtering of low-quality examples for data selection. While CLIPScore-basee filtering has proven useful for multimodal data selection (Schuhmann et al, 2021), it assumes access to pre- pre-paired multimodal data and is therefore unsuitable for our setting with unaligned modalities.
3 Problem Setting
We study multimodal learning with a dataset D = (D' D'), where De = {x]]]== denotes the collection of raw vision features and D' {x!)== denotes the collection of raw textual features. Unlike standard multimodal setups, the vision and language features are initially unaligned. The learner may query a subset of instances to obtain their aligned pairs at an annotation cost. Specifically, for any data point of E {.] a.), the learner can spend one unit of annotation cost to reveal its aligned pair x; := (x",a)) We use = {()),,))) mm to denote the set of annotated pairs obtained with a total of m units of cost.
The goal, under a fixed annotation budget, is to actively and strategically select an informative subset S to maximize the quality of multimodal model & == (ø", (6",0" Φ¹), where Ø² 0' Rdd + Rd are encoders that map raw features into shared representation space. We adopt CLIP-style contrastive training for multimodal models (Radford et al., 2021), and, following standard practice (Zhai el al., 2022, 2023), evaluate model quality on downstream tasks.
1 For simplicity, we focus the lisionllanuage case. Our setting and algorithms naturally extend general multimodal learning with more than two modalities; see Section 4.2.
3

---

Published in Transactions on Machine Learning Research (02/2026)
We refer to this setup multimodal active learning with unaligned data, which not only extends classical unimodal active learning (Sener & Savarese, 2018; Ash et. al., 2020; Citovsky et al., 2021; Saran et al., 2023) but also departs fundamentally from prior multimodal active learning frameworks that assume pre-aligned data (Shen et al., 2023). The active learning algorithm procoeds over E 24 iterations. At iteration t, the learner selects and annolates a batch of B data points {(r], at, ))= and updates the annotation set as St - S {(a = )- The multimodal model Φb = =6', is trained on St and then used to guide data selection in the next iteration. This iterative process enables the learner, under a fixed budget, build high-quality multimodal model from strategically chosen alignments.
Depending on how the learner accesses the unaligned pool, we study two regimes:
Pool-based multimodal active learning. The learner has full access to (D", D') throughout the process and can query any instances at any t (T) (Ash et al., 2020).
Streaming-based multimodal active learning. The data arrives as disjoint subsets D= = (De with UT_, Dr = D and Dr, n Di, = At iteration t, the learner only observes D; = (D" (D) and must select and annotate data within this batch. Unqueried data from Dr cannot. be revisited future iterations (Saran et al., 2023).
Additional notation. For any N E Σ., we denote [N] == [1....,,)).. For multimodal sets D = D= and S = (S", )'), we write D D == (D" S", D' \ s.). For any modality & E {v, we denote (S = {of (x²)) : z E J. When clear, we use the shorthand (s(t)) = (5+) or ((S)) = (St).
(D", D'}
3.1 Unique Challenges with Unaligned Multimodal Data
Compared to unimodal active learning (Sener & Savarese, 2018; Ash al., 2020) and multimodal active learning with pre-aligned data (Shen al., 2023), our setting multimodal active learning with unaligned data poses qualitatively new challenges.
In unimodal active learning, each instance r has single feature vector and annotation assigns a class label from a small predefined set. In multimodal active learning with pre-aligned data, the learner selects from pre-aligned pairs {(x,, )²) for label queries. Because modalities are already aligned, this effectively reduces to unimodal active learning on composite inputs.
By contrast, in our unaligned setting with vision features {= and language features {=))== the learner must simultaneously decide which modality to query from and how to align instances across modalities. This creates two distinctive challenges:
Bidirectional alignment. With unaligned data, annotation may begin from either vision-to-languse language-to-vinn to-igeiooovvvinn. Crucially, the learner does not know in advance which instance from the other modality will be paired. Different alignment directions can lead to entirely different annotation sets, adding an extra decision layer absent in unimodal or pre-aligned multimodal AL.
Large cross-modal candidate space. Even after choosing modality to query, the utility of candidate must be evaluated against the entire other modality. For instance, querying an image requires scoring potential matches across all texts, which effectively serves as an enormous candidate label space. Unlike conventional class labels, these candidates are instance-specific and extremely numerous (e.g., 12.8M unique texts in DataComp (Gadre et al., 2024)). Naively evaluating all pairs scales quadratically with dataset size, quickly bocoming infeasible.
Thus, multimodal active learning with unaligned data requires acquisition algorithms that handle both bidirectional alignment and the large cross- moos- search space efficiently. We present our approach these challenges in Section 4.
4

---

Published in Transactions on Machine Learning Research (02/2026)
4 Methods
We present our approaches to address the unique challenges mentioned Section 3.1. We first introduce our multimodal active learning algorithm for the pool-based setting Section 4.1, which further extended to the streaming- based selting and settings beyond vision-language models in Section 4.2.
4.1 Multimodal Active Learning
We present our multimodal active learning algorithm in Algorithm 1, which proceeds iteratively for T iterations. At each iteration [2]. Algorithm selects a batch of B data points for annotation, based on the multimodal model Φ1-1 (6) 1, -1) trained with respect to previously annotated data points St-1- The annotation set is then updated to S,, and the model Φr = (d] of) is retrained on the updated annotation dataset to guide data selection in the next iteration.
The core of Algorithm lies in how to actively select data for annotation. This achieved via three integrated steps: (1) modality selection, (2) coresel construction, and (3) uncerlainty-based selection. At each iteration, Algorithm 1 first selects a modality that is underrepresented by the already annotated data S;-1 (Step 1). It then constructs a coreset of Be data on the selected modality to ensure coverage (Step 2), and finally selects B Bc highly uncertain data points from this coreset using a cross-modal uncertainty score (Step 3), which quantifies how confidently feature in one modality matches candidates from the other.
Algorithm Multimodal Active Learning
Input: Unaligned multimodal dataset = (D", D'), number of iterations T, per-round selection size B, coreset hyperparameter Bc 2 B.
Initialize multimodal model de- {dj, %/ with random or pretrained weights. Initialize the annotation set S. = 0.
3: for 1. do
4: Consider unaligned data pool D, = D S,-1.
5: Step 1: Modality selection. Scloct modality hy == arg (v..) di that is less-covered by previous annotations St-1, where df
max
min
(5))
(1)
2. (P( (0St))
6: Step 2: Coreset construction. On modality kn construct a coreset c C D.e of size Be such that, together with Stn,, it maximally covers Div
cha
arg min
min =B (D (6
max
dist(= 2z).
(2)
// Eq. (2) can be approximated with efficient greedy algorithm (Algorithm 2).
7: Step 3: Uncertainty-basee
selection. Within coresct Cha select the top-B most uncertain data points using multimodal model ét-1.
8: Let m = {v,1} {kt} denote the unselected modality in Step 1.
9: For each data x E Che, compute its margin score u(a = w/1) - w/2) which serves as an uncertainty measure. Here w E RP!! denotes the vector of similarity scores between af and all unaligned features in the other modality Di" and wty) denotes the j-th largest entry of wt. Compute margin score uncertainty measure.
10: Select the subset A: )BB C Chie with the top- uncertainty scores (i.e., lowest margins), annotate them, and update S, + St-1 f(x), d))
11: Model update. Train multimodal model di = (0i, ()) on the updated annotation set S.. Actively trained multimodal model ØT = of).
Output:
Our Algorithm 1 intogrates both diversity and uncertainty principles into multimodal active learning with unaligned data. By restricting cross modal uncertainty evaluation the coresct constructed in Step 2, our algorithm enables efficient data selection with per-round runtime that scales linearly in [D]. In contrast,
5

---

Published in Transactions on Machine Learning Research (02/2026)
naive uncertainty-based selection over the entire dataset would require quadratic time in [D], which is computationally prohibitive.
We next explain the details of each of the three steps in Algorithm 1.
Step Modality selection. This step selects the modality that underrepresented with respect to the current annotation S,1 (line 5 in Algorithm 1). To assess coverage, for each modality k {u,(), we compute the maximum distance of unaligned features D. = DR S to their nearest neighbors in S/-
di = max
min dist(z;, 2j);
2.E (1
where dist a distance motric, and ((x) denotes normalized emboddings to cnsure comparability across modalities. The modality with the largest distance value (i.e., least covered), k == arg ma di is selected for coresct construction in the next step. This step has a runtime upper bound of O( (St-1))-
Step 2: Coreset construction. Given the selected modality k in Step 2 (line 6 of Algorithm 1), we construct coreset C/ C Dfe of size Bc 2 B that, when combined with already annotated data points S
maximally covers the unaligned data D.v. This is formalized in Eq. (2) of Algorithm 1. Solving Eq. (2) exactly is NP-Hard (Cunningham & Schrijver). Following prior work in diversity- based active learning in the unimodal setting (Sener & Savaresc, 2018), we employ greedy approximation algorithm (Algorithm 2) that guarantees 2 x OPT solution. Algorithun 2 can be implemented in O ((Bc + (S,-1)) (Pal) time using a distance caching strategy: initially, we compute and cache the minimum distances between all candidate points in D, and the selection set St-1, incurring a runtime of O( - |S,- For subsequent iterations in the greedy selection process, we only need O(Dr)) operations to update the cache and select the next point.
Step 3: Uncertainty-based selection. With the constructed coreset C in Step (lines 7-10 of Algorithm 1), compute the margin score the mcasure of uncortainty and select the top-B data points with the highest uncertainty (i.e., lowest margin scores) for multimodal annotation." ? For each x E C we compute vector of similarity scores w² [P"" between of* and all unaligned features in the other modality m = {u. 4 \ {ke)- The calculation of similarity score is usually model-dependent: but a simple example is the inner product between representations (or its variants), e.g., wz. := (0 ( )) th (=(")) for s""' E 'Dir. The uncertainty score is computed as the margin between the top two similarity scores: u(x;") = w/a) w(2) We select the B data points Ch with the lowest margin scores. This step has a runtime of O O (Drl).
Algorithm 2 Greedy Approximation for Coreset Construction
Input:
Selected modality kt, unaligned multimodal dataset D.t, selection size Bc, multimodal model Φ1-1,
current annotation set S,-1- Initialize coreset set c²: = (0. while cha Be do Select data point zu such that:
Zu == argmax max 2, (D" 28$2))
min dist(z,, 2z)-
Update ch + cha U {zu).
Output: Coreset Ch
Computational complexity. Summing the runtime of Steps 1-3, the per-round data acquisition complexity of Algorithm is upper bounded by O (Sr [Dr)), where |S,_1l = ((BB) and [D, < [D). Across T rounds, the total complexity is upper bounded by O(T (Bc + TB) [D]), as formalized in Section 4.1.
2 Although adopt margin the uncertainty measure in this step, our algorithm be casily extended to alternative choices such entropy or confidence. We use margin because has been observed consistently yield strong performance across active learning strategies (Zhang et al., 2024a).

---

Published in Transactions on Machine Learning Research (02/2026)
Assuming T == and Bc < D); both per-round and total complexity is linear in [D], which is typically large in real-world multimodal learning tasks (Gadre et al., 2024). In contrast, naive uncertainty-basen approach would compute margin scores for all pairs across modalities in the unaligned data, resulting in a per-round complexity of O(1D)²), which is computationally expensive. Our algorithm avoids this quadratic bottleneck by computing cross-modal uncertainly only over the coresel (Slep 3).
The per-round data acquisition complexity of Algorithm upper bounded by O ( + (S--l) (Pr)),
resulting in overall complexity of O (Bc +TB) (D))
4.2 Extensions of Algorithm 1
Streaming-based multimodal active learning. While Algorithm is originally designed for the pool- based setting, it can be easily adapted to streaming- based multimodal active learning. As introduced in Section 3, in the streaming- based setting, the learner only has access to the current batch of stream data D,. To adapt Algorithm to this setting, we simply replace line 4 with the current batch stream data De, and leave all other parts of the algorithm unchanged. As result, aside from potential variation the size of Dr, the per-round computational complexity remains the same as in the pool-based case.
Active learning beyond vision-language models. Although our focus this paper is on multimodal learning with vision-language data, A Algorithm can be naturally extended to general multimodal settings with m 2 3 unaligned modalities D {D(1), ((2))... D(m)). To support this generalization, we modify Step and Step 3 of the algorithm while keeping Step 2 unchanged, since it operates solely within the selected modality. We discuss these changes below:
For Step 1, we generalize the modality selection in Eq. (1) to select the least-covered modality among all m modalities, i.c., hn == arg max (1.2.....)) d:
For Step 3, given any data point at E Df in the selected modality, let we (x**t denote its cross-modal uncertainty score with respect to each unselected modality j [m]] \ {kr]. We then define the overall uncertainty score as u(>"1)= Ezemm] (1"1,, and use this score for uncertainty- based selection.
Assuming m = 0(1,, the computational complexity of this generalized algorithm remains order- wise the same as the analysis provided in Section 4.1.
5 Experiments
We conduct extensive experiments to evaluate the effectiveness of our proposed algorithm. The experimental setup is described in Section 5.1, followed by the main results and analyses in Section 5.2. We defer additional implementation details and experimental results to Appendix B.
5.1 Experimental Setups
Datasets. We conduct experiments on three multimodal datasets: ColorSwap (Burapachoep et al., 2024), MS-COCO (Lin et al., 2014), and DataComp (Gadre et al., 2024). ColorSwap is designed to evaluate object-color matching with color-swapped image-caption pairs. MS-COCO is a large-scale image-caption dataset designed for object detection. DataComp consists of large-scale image-text pairs collected from Common Crawl. See Table for detailed descriptions of these datasets. We use ColorSwap for pool-based AL, and MS-COCO and DataComp for streaming- AL. For ColorSwap and MS-COCO, we initialize models from pretrained weights and study multimodal active learning in the finetuning regime. For DataComp, we initialize modcls from random weights to evaluate multimodal active learning in the pretraining regime.
We focus the data acquisition complexity and its dependency on the data pool size [D], which is usually large for multimodal learning; note that the model training complexity the same for all active learning algorithms and it's proportional the training data St-1l-
7

---

Published in Transactions on Machine Learning Research (02/2026)
Table Datasets used in our experiments and their corresponding evaluation metrics.
Datasets
Evaluation metrics
# Data samples
ColorSwap
Scores: text, image, group (Burapacheep al., 2024)
1400
MS-COCO
R@1: I T, T I (Zhai et al., 2023)
118K
DataComp
Average score over 38 tasks (Gadre et al., 2024)
12.8M
Text Score
Image Score
Group Score
64
52
57
Score
48
56
54
44
51
40
48
5%
10%
15%
20%
25%
30%
5%
10%
15%
20%
25%
30%
5%
10%
15%
20%
25%
30%
84
80
81
80
76
score
76
Random Coreset Uncertainty
68
72
64
68
Ours
5% 10% 15% 20% 25%
30%
5%
10%
15% 20% 25% 30%
5% 10% 15% 20% 25% 30%
Percentage of Aligned Data
Percentage of Aligned Data
Percentage of Aligned Data
Figure 1: Results of pool-based multimodal active learning on the ColorSwap dataset with CLIP-B32 (top) and SigLIP-B16 (bottom). We report text score (left), image score (middle), and group score (right) as learning progresses.
Baselines and models. We compare our algorithm against three baselines: RAND, CORESET, and UNCERTAINTY. RAND serves as passive learning baseline, randomly selecting data pairs for annotation. Since multimodal actíve learning with unaligned data is problem, existing basclines directly apply. To better assess the performance of our Algorithm 1, construct two additional baselines by adapling widely used unimodal AL methods- diversity- method (CORESET, (Sener & Savarese, 2018)) and an uncertainty-based method (UNCERTAINTY, (Settles, 2009) to our multimodal sctting with unaligned data.*
We implement our algorithu and all baselines using the CLIP model (Radford al., 2021) and its variants SigLIP (Zhai al., 2023) and LiT (Zhai et al., 2022), evaluating across models of different sizes. Detailed hyperparameter settings are reported in Appendix B.2.
Evaluation metrics. We adopt standard evaluation metrics for each dalaset. For ColorSwap, report the text score, image score, and group score, proposed by Burapacheep et al. (2024). For MS-COCO, we report recall@1 for both image-to-text and text-to-image retrieval, as commonly used in the literature (Zhai et al., 2023). For DataComp, we follow Gadre et al. (2024) and report the average score across 38 downstream tasks. All results are averaged over 4 random runs, with shaded regions in plots indicating 2/3 of standard deviation.
We use ColorSwap dataset to study the posl-based setting, and MS-COCO and DataComp study the streaming-baset setting. The streaming buffer size [Dr] is set 1024 for MS-COCO and 2048 for Data additional experiments with different strcaming buffer sizes are provided in Appendix C.2. The per-round data acquisition size B is 70 for ColorSwap, 256 for MS-COCO, and 512 for DataComp.
FFull algorithmic details CORESET and UNCERTAINTY provided Appendix B.I. We also report additional resulta other adapted unimodal active learning algorithms Appendix C.4.
8

---

Published in Transactions on Machine Learning Research (02/2026)
Image-to-ter Retrieval
Text-to-image Retrieval
Average Score
42
12
43
41
11
Score
40
42
10
Coreset
Random
9-
Uncertainty
41
Ours
5900 11800 17700 23600 29500
5900 11800 17700 23600 29500 Number of Aligned Data
0.5
1.0
1.5 2.0
210
Number of Aligned Data
Number of Aligned Data
Figure/ 2: Streaming-based multimodal active learning with the MS-COCO (left and middle) and DataComp (right) datasets using CLIP-B32. We report R@1 (image-to-text) (left), R@1 (text-lo-image) (middle), and the average score across 38 downstream tasks (right). We report algorithm performance as learning progresses.
CLIP-L14
LIT-L14
SigLIP-L16
66
30
1o00.0 Score
60
27
80
24
Random
48
21
72
Coreset
18
Uncertainty
36
15
68
Ours
5% 10% 15% 20% 25% 30%
5%
10%
15%
20%
25% 30%
5%
10%
15%
20% 25% 30%
Percentage of Aligned Data
Percentage of Aligned Data
Percentage Aligned Data
Group scores on the ColorSwap dataset in the pool- based setting, using CLIP-L14 (left), LiT-L14 (middle), and SigLIP--L16 (right).
Figure 3:
5.2 Main Results
Pool-based multimodal active learning. We compare Algorithm 1 against three baselines on the ColorSwap datasot. As shown in Fig. 1, our algorithm generally outperforms the basclines across all three metrics. Notably, with CLIP-B32, Algorithm achieves a group score of 49.42 using only 15% of the data, which is comparable to the group score reached by RAND at 25%, corresponding to 40% reduction in annotation cost. Relative to UNCERTAINTY, data, while UNCERTAINTY requires 30% to reach a similar score-repprie reveeerrrreeentigg a 33% reduction in cost. In addition to superior data efficiency, Algorithm is computationally more efficient than UNCERTAINTY, which incurs higher complexity (Section 4.1).
algorithm achieves group score of 51.42 with 20% of the
Streaming-based multimodal active learning. We next evaluate Algorithm 1 in the streaming based sctting on MS-COCO and DataComp datasets. On MS-COCO (left and middle pancls Fig. 2), our algorithm consistently outperforms all baselines across both retrieval metrics. For example, on R@1 (image--o- txxeet),, Algorithm 1 achieves a score of 41.47 using 23,600 samples (20% of the data), matching the performance of CORESET and RAND with 25% of the data. This corresponds to 20% reduction in annotation cost..
To examine performance at larger scales, we further conduct experiments on the DataComp dataset (right panel of Fig. 2), reporting the average score across 38 downstream tasks. Our algorithm outperforms all baselines, with particularly large margins over RAND and UNCERTAINTY. Relative to CORESET, Algorithm achioves clear improvements as the number aligned pairs grows, reaching a score of 11.59 with 2.5M pairs. For context, training on the full 12.8M aligned pairs (the performance skyline) yields score of 13.20 (Gadre et al., 2024). Thus, our method attains 87.80% of the skyline with just 2.5M pairs, whereas CORESET reaches only 85.68% of the skyline (with score of 11.31).
Robustness across CLIP variants and model sizes. To evaluate the robustness of Algorithm across architectures and model sizes, we conduct additional experiments with multiple CLIP variants, including

---

Published in Transactions on Machine Learning Research (02/2026)
Table 2: Ablation study of Algorithm 1 with different modality selection strategies in the pool- based set using CLIP-B32 and SigLIP-B16. We report group scores with 30% of aligned data.
Model
Random
Text-only
Image-only
Ours
CLIP-B32
52.4+3.
52.8=1.4 77.67
50.7+2.3 76.8911.9
54.3+1.2
SigT.IP-B16
75.5012.1
79.42:129
Table 3: Case study of Algorithm recording margin scores Step under the pool-based setting with CLIP-B32. We report average margin scores (scaled by x)0-2) for both correctly and incorrectly matched groups (with respect to the ground-truth alignments) across different percentages of aligned data pairs.
Percentage of Aligned Data
5%
10%
15%
20%
25%
30%
Correctly matched group
3.8+0.2
4.0
4.2=0.4
4.740.2
4.8+0.2
5.240.2
Incorrectly matched group
0.9+0.1
1.0+0.2
1.3#0.1
1.4+0.2
1.4+0.1
1.5+0.5
SigLIP (Zhai et al., 2023) and LiT (Zhai et al., 2022), as well as models of different sizes. Given the high runtime cost the streaming- based setting, most experiments are performed in the pool-based scenario. Fig. 3 reports group scores for largo-scale variants, where Algorithm consistently outperforms the baselincs. Results for text and image scores are deferred to Appendix C.I, and show similar trends.
6 Analyses and Ablations
Effectiveness of modality selection. We first assess the effectiveness of the modality selection strategy in Algorithm (Step 1). Following the same experimental setup as in Fig. 1, we compare our approach against three alternatives: randomly choosing a modality, always selecting the text modality, and always selecting the image modality. As reported in Table 2, our proposed strategy achieves the best performance across both CLIP and SigLIP models. For example, it attains a group score of 54.3 with the CLIP model, representing up to 7% improvement over the image-only strategy, with all other components held fixed.
Effectiveness of margin score in data selection. To evaluate the effectiveness of the margin score in Algorithm 1 (Step 3), we analyze its behavior across different percentages of aligned data pairs. At each iteration, Step 3 computes pseudo-aliguments for all unaligned data points by selecting their most likely match based on similarity. We then partition the data into correctly matched and incorrectly matched groups (with respect to ground-truth alignments) and report their average margin scores in Table 3. As expected, incorrectly matched samples consistently exhibit lower margin scores than correctly matched ones, reflecting higher uncertainty and thus greater value for active selection. Since these aseudo-alignments are computed unaligned data not yet used in training, the results confirm that the margin score provides robust and meaningful uncertainty signal even in noisy, unaligned settings.
Synergy of uncertainty and diversity in data selection. As shown in Section 5.2, Algorithm consistently outperforms all basclines. We attribute this advantage to its ability to prioritize data points that are both uncertain (Step 3) and diverse (Step 2). To test this hypothesis, we visualize the ColorSwap dataset using t-SNE on image-modality embeddings, comparing data selected by Algorithm with those chosen by UN- CERTAINTY. As shown in Fig, 4 (left), data points selected by UNCERTAINTY (blue circles) tend to concentrate within narrow uncertain region. In extreme cases rases-for example, when multiple highly similar or duplicated data points are all uncertain- pure uncertainty-baser selection may repeatedly select redundant samples, providing little additional learning benefit. In contrast, data points selected by Algorithm 1 (red stars) not only concentrate near high-uncertainty regions but also spread more broadly across the embedding space, reflecting greater diversity. This joint emphasis on uncertainty and diversity provides a key advantage Algorithm 1.
Empirical data acqisititinn runtime comparison. Fig. 4 (middle) reports the per-round data acquisition time of each algorithm using CLIP-B32. As expected, Algorithm 1 is slower than the CORESET baseline
10

---

Published in Transactions on Machine Learning Research (02/2026)
(-SNE Visualization on ColorSwap
Runtime Comparison
Group Score
1.00
Coreset
72
Uncertainty
Runtime (s)
Score
64
0.50
Randam Coreset
60
Uncertainty Ours
Uncortainty
512 1024
2048
4096
5%
10%
15%
20%
25%
30%
Candidate Size
Percentage Aligned
Figure 4: Left: (-SNE visualization of image-modality embeddings from the ColorSwap dataset, comparing our method (Algorithm 1) with an uncertainty-basee baseline (UNCERTAINTY), Points selected by the baseline are shown as blue circles, while points selected by our method are shown as red stars. Blue density contours represent the distribution all data. Middle: Per-round data acquisition time (seconds) across candidate set sizes for different acquisition strategies using CLIPP-33.. Right: Robustness to missing data and alignments ColorSwap (SigLIP-B16): we remove 50% of images while keeping all captions, creating many unmatched captions. All methods query labels from the image modality.
Table 4: Pseudo-labelingg accuracy and GroupMatch score on ColorSwap using CLIP-B32.
Metric
Random
Coreset
Uncertainty
Ours
Eseudo-labeliigg accuracy
79.81 20.73
78.52+0.80
70.67
80.39
GroupMatch score
86.57
10.29 10.29
85.57 10.15
16.43
87.14.0.22
because it explicitly includes coreset construction step. However, the candidate set size increases, algorithm becomes substantially more efficient than the UNCERTAINTY bascline. These runtime results are consistent with our theoretical analysis in Section 4.1 and confirm that our algorithm is more efficient than the Uncertainty baseline.
Evaluation under different metrics. To further assess alignment quality beyond the metrics used in ColorSwap, we additionally report pscudo-labeling accuracy and the GroupMatch score (Zhu et al., 2026). Table summarizes results with CLIP-B33 where our method continues outperform all baselines on both pscudo-labeling accuracy and GroupMatch, further supporting the effectiveness of the proposed approach.
Robustness to coreset hyperparameter Be. To examine robustness to the coreset hyperparameter Be (Step 2), we conduct experiments across model families (CLIP and SigLIP) and scales (CIIP--33 and SigLIP-L16). As shown in Fig. 5, performance remains stable across different values of Be and annotation costs, demonstrating that our mothod is robust to this parameter. practice, we seloct the value of Be from the set (1.5B, 2B, 2.5B), depending on the dataset and model. We provide detailed hyperparameter selections in Appendix B.2.
Robustness to missing data and alignments. While our primary experiments focus on settings where valid eross-modal alignments exist for all aallowiis-alloiing us to isolate the core challenges of multimodal active learning with unaligned data (Section 3.1) werl-odooorld datasets often contain missing data and alignments. To evaluate robustness under such conditions, we conduct an experiment on a deliberately perturbed version of the ColorSwap dataset. Specifically, we retain all captions but remove half of the images, creating a setting where many captions no longer have valid visual counterparts. We further restrict annotation to proceed only from the image modality (for our method and all baselines), since certain captions cannot be aligned any image. The results, reported in Fig. (right), show that our method continues achieve the best. performance under this noisy setting, suggesting robustness to missing data and alignments.
11

---

Published in Transactions on Machine Learning Research (02/2026)
Group Score
80
70
Bc:1.5B-CLIP
Bc:2B-CLIP
Score
60
Be:2.58-CLIP
Bc:1.58-SigLIP
50
Bc:28-SigLIP
40
Bc:2.58-SigLIP
30
5%
10%
15%
25%
30%
Percentage Aligned Data
Figure 5: Parameter study of Algorithm with different values of Be in the pool-based setting using CLIP-B33 and SigLIP-L16. We report group scores as learning progresses.
7 Conclusion
We presented the first study of active learning in multimodal settings with unaligned data, addressing the key challenges of bidirectional alignment and annotation across large cross-modal candidate spaces. By integrating uncertainty- and diversity-based selection in modality-aware design, we developed an efficient algorithm applicable to both pool-based and streaming-based scenarios. Experiments on benchmark datasets show that our approach reduces annotation requirements by up to 40% while maintaining model performance, highlighting the promise of active learning for scalable and cost-cffective multimodal learning.
Our current method specifically designed for multimodal representation learning, where supervision is method
provided through rross-modal alignments and training is typically contrastive. Extending to other multimodal paradigns- as multimodal generative learning-wond likely require non-trivial modifications to the acquisition strategy and uncertainty estimation. We view this as an important and promising direction for future work.
References
Jordan Ash, Surbhi Goel, Akshay Krishnamurthy, and Sham Kakade. Gone fishing: Neural active learning with fisher embeddings. Advances in Neural Information Processing Systems, 34:8927-899 2021.
Jordan T. Ash, Chicheng Zhang, Akshay Krishnamurthy, John Langford, and Alekh Agarwal. Deep batch active learning by diverse, uncertain gradient lower bounds. Tn International Conference on Learning Representations, 2020.
Tianyi Bai, Hao Liang, Binwang Wan, Yanran Xu, Xi Li, Shiyu Li, Ling Yang, Boxhou Li, Yifan Wang, Bin Cui, et al. A survey of multimodal large language model from data-centric perspective. arxiv preprint 2ix::2405.166400 2024.
Maria-Florina Balcan, Andrei Broder, and Tong Zhang. Margin based active learning. In International Conference on Computational Learning Theory, pp. 35 50. Springer, 2007.
Tadas Baltrušaitis, Chaitanya Ahuja, and Louis- Morency. Multimodal machine learning: A survey and taxonomy. IEEE transactions on pattern analysis and machine intelligence, 41(2):423-443, 2018.
Gantavya Bhatt, Yifang Chen, Arnav M Das, Jifan Zhang, Sang T Truong, Stephen Mussmann, Yinglun Zhu, Jeffrey Bilmes, Simon Du, Kevin Jamieson, al. An experimental design framework for label-efficient supervised finetuning of large language models. Findings of the Association for Computational Linguistics, 2024.
12

---

Published in Transactions on Machine Learning Research (02/2026)
Jirayu Burapacheep, Ishan Gaur, Agam Bhatia, and Tristan Thrush. Colorswap: A color and word order dataset for multimodal evaluation. In Findings of the Associalion for Compulational Linguisties ACT 2024, pp. 1716 1726, 2024.
Rui M Castro and Robert D Nowak. Minimax bounds for active learning. In International Conference on Compulational Learning Theory, pp. 5 19. Springer, 2007.
Qiuhui Chen and Yi Hong. Alifuse: Aligning and fusing multimodal medical data for computer-aided diagnosis. In 2024 IEEE International Conference Bioinformatics and Biomedicine (BIBM), pp. 3082-3087, IEEE, 2024.
Gui Citovsky, Giulia DeSalvo, Claudio Gentile, Lazaros Karydas, Anand Rajagopalan, Afshin Rostamizadel, and Sanjiy Kumar. Batch active learning at scale. Advances in Neural Information Processing Systems, 34: 11933 11933-119444 11944, 2021.
William J Cook William Cunningham and William Pulleyblank Alexander Schrijver. Combinatorial optimization.
Sanjoy Dasgupta, Adam Tauman Kalai, and Adam Tauman. Analysis of perceptron-based active learning. Journal of Machine Learning Research, 10(2), 2009.
Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, et al. Datacomp: In search the next generation of multimodal datasets. Advances in Neurol Information Processing Systems, 36, 2024.
Chongjian Ge, Junsong Chen, Enze Xie, Zhongdao Wang, Lanqing Hong, Huchuan Lu, Zhenguo Li, and Ping Luo. Metabev: Solving sensor failures for 3d detection and map segmentation. In Proceedings of the LEEE/CVF International Conference on Computer Vision, pp. 8721-8731, 2023.
Haochen Han, Qinghua Zheng, Guang Dai, Minnan Luo, and Jingdong Wang. Learning to rematch mismatched pairs for robust modal retrieval. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 26679-26688, 2024.
Steve Hanneke. Theory of active learning. Foundations and Trends in Machine Learning, 7(2-3), 2014.
Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference free evaluation metric for image captioning. In Proceedings of the 2021 conference on empirical methods in natural language processing, pp. 7514-7528, 2021.
Siddharth Joshi, Arnav Jain, Ali Payani, and Baharan Mirzasoleiman. Data-efficient contrastive language- image pretraining: Prioritizing data quality over quantity. Tn International Conference on Artificial Intelligence and Statistics, pp. 1000-1008. PMLR, 2024.
Tomoyoshi Kimura, Xinlin Li, Osama Hanna, Yatong Chen, Yizhuo Chen, Denizhan Kara, Tianshi Wang, Jinyang Li, Xiaomin Ouyang, Shengzhong Liu, et al. Infomae: Pair-efficient cross-modal aligument for multimodal time-series sensing signals. In Proceedings of the A ACM on Web Conference 2025, pp. 3084-3095, -095,, 2025.
Akshay Krishnamurthy, Alekh Agarwal, Tzu-Kuo Huang, Hal Daumé III, and John Langford. Active learning for cost-sensitive classification. Journal of Machine Learning Research, 20(65):1-50, 2019.
Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Ballison-Burch and Nicholas Carlini. Deduplicating training data makes language models better. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 8424-8445, 2022.
Paul Pu Liang, Amir Zadeh, and Louis-Phllippe Morency. Foundations & trends in multimodal machine learning: Principles, challenges, and open questions. ACM Compuling Surveys, 56(10):1 1-(10))1-42, 2024.
13

---

Published in Transactions on Machine Learning Research (02/2026)
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zätnick. Microsoft coco: Common objects in context. In Europeon conference on computer vision, pp. 740 755. Springer, 2014.
Katerina Margatina, Timo Schick, Nikolaos Aletras, and Jane Dwivedi- Yu. Active learning principles for in- context learning with large language models. In Findings of the Association for Computational Linguisties: EMNLP 2023, pp. 5011-5034, 2023.
Sören Mindermann, Jan M Brauner, Muhammed T Razzak, Mrinank Sharma, Andreas Kirsch, Winnie Xu, Benedikt Höltgen, Aidan N Gomez, Adrien Morisot, Sebastian Farquhar, et al. Prioritized training on points that are learnable, worth learning, and not yet learnt. In International Conference on Machine Learning, pp. 15630-15649. PMLR, 2022.
Nikita Puchkin and Nikita Zhivotovskiy. Exponential savings in agnostic active learning through abstention. In Conference on learning theory, pp. 3806-3832. PMLR, 2021.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748 8763. PmLR, 2021.
Akanksha Saran, Safoora Yousefi, Akshay Krishnamurthy, John Langford, and Jordan T Ash. Streaming active learning with deep neural networks. In International Conference on Machine Learning, pp. 30005 30021. PMLR, 2023.
Jacob Schreiber, Jeffrey Bilmes, and William Stafford Noble. apricot: Submodular selection for data summarization in python. Journal of Machine Learning Research, 21 (161)) 2020.
Christoph Schuhmann, Robert Kaczmarozyk, Aran Komatsuzaki, Aarush Katta, Richard Vencu, Romain Beaumont, Jenia Jitsev, Thco Coombes, and Clayton Mullis. Laion-400m: Open dataset of clip-filterdd 400 million image-text pairs. In NewrIPS Workshop Dalacentric AI, number FZJ-2022-00923,, Jülich Supercomputing Center, 2021.
Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-set approach. In International Conference on Learning Representations, 2018.
Burr Settles. Active learning literature survey. 2009.
Li Shen, Yan Sun, Zhiyuan Yu, Liang Ding, Xinmei Tian, and Dacheng Tao. On efficient training of large-scale deep learning models. ACM Computing Surueys, (33: 2024.
Meng Shen, Yizheng Huang, Jianxiong Yin, Heqing Zou, Deepu Rajan, and Simon See. Towards balancod active learning for multimodal classification. In Proceedings of the 31st ACM International Conference on 2023.
Mullimedia, pp.
Ben Sorscher, Robert Geirhos, Shashank Shekhar, Surya Ganguli, and Ari Morcos. Beyond neural scaling laws: beating power law scaling via data pruning. Advances in Neural Information Processing Systems, 35: 19523-19536, 2022.
Kushal Tirumala, Daniel Simig, Armen Aghajanyan, and Ari Morcos. D4: Improving 11m pretraining via document de-duplication and diversification. Advances in Neural Information Processing Systems, 36: 53983 53995, 2023.
Haonan Wang, Wei Huang, Ziwei Wu, Hanghang Tong, Andrew J Margenol, and Jingrui He. Deep active learning by leveraging training dynamics. Advances in Newral Information Processing Systems, 35:25171- 25184, 2022.
Weizhi Wang, Khalil Mrini, Linjie Yang, Sateesh Kumar, Yu Tian, Xifeng Yan, and Heng Wang. Finetuned multimodal language models are high-quality image-text data filters. arxiv preprint arXiv:2403.02877 2024a.
14

---

Published in Transactions on Machine Learning Research (02/2026)
Yiping Wang, Yifang Chen, Wendan Yan, Alex Fang, Wenjing Zhou, Kevin Jamieson, and Simon S Du. Cliploss and norm-based data selection methods for multimodal contrastive learning. Advances in Newral Information Processing Systems, 37:15028 15069, 2024b.
Mengzhou Xia, Sadhika Malladi, Suchin Gururangan, Sanjeev Arora, and Danqi Chen. Less: selecting
infiuential data for targeted instruction tuning. In Proceedings of the 41st International Conference Muchine Learning, pp. 54104 54132, 2024.
Shuo Yang, Zeke Xie, Hanyu Peng, Min Xu, Mingming Sun, and Ping Li. Datasct pruning: Reducing training data by examining generalization influence. In The Fleventh International Conference Learning Representations, 2023.
Bo Yuan, Yulin Chen, Yin Zhang, and Wei Jiang. Hide and seek in noise labels: Noise-robust collaborative active learning with mons-powered assistance. In Proceedings of the 62nd Annual Meeting of the Association for Compulational Linguislics (Volume 1: Long Papers), pp. 10977 11011, 2024.
Xiaohua Zhai, Xiao Wang, Basil Mustafa, Andreas Steiner, Daniel Keysers, Alexander Kolesnikov, and Lucas Beyer. Lit: Zero-shot transfer with locked-image text tuning. In Proceedings of the IEEE/CVE conference on computer vision and pottern recognátion, pp. 18123 18123-18133, 2022.
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre- training. In Proceedings of the LEEE/CV international conference on computer vision, pp. 11975 11986, 2023.
Chicheng Zhang and Kamalika Chaudhuri. Beyond disagreemennt-aseed agnostic active learning, Advances in Neural Information Processing Systems, 27, 2014.
Jifan Zhang, Yifang Chen, Gregory Canal, Arnav Mohanty Das, Gantavya Bhatt, Stephen Mussmann, Yinglum Zhu, Jeff Bilmes, Simon Shaolei Du, Kevin Jamieson, et al. Tabelbench: A comprehensive framework for benchmarking adaptive label-efficient learning. Journal of Data-centric Machine Learning Research, 2024a.
Yiqian Zhang, Buyu Liu, Jun Bao, Qiang Huang, Min Zhang, and Jun Yu. Learnability matters: active learning for video captioning. Advances Neural Information Processing Systems, 1187928-377955 2024b.
Yinglun Zhu and Robert Nowak. Active learning with neural networks: Insights from nonparametric statistics. Advances in Neural Information Processing Systems, 35:142-155, 2022a.
Yinglun Zhu and Robert Nowak. Efficient active learning with abstention. Advances in Neural Information Processing Systems, 35:35379 35391, 2022b.
Yinglun Zhu, Jiancheng Zhang, and Fuzhi Tang. Test-Time Matching:
Unlocking compositional reasoning in multimodal models. The Fourleenth International Conference on Learning Representations, 2026.
Yongshuo Zong, Oisin Mac Aodha, and Timothy M Hospedales. Self-supervised multinodal learning: A survey. IEEE Transactions on Pottern Analysis and Machine Intelligence, 47(7):5299 5318, 2024.
15

---

Published in Transactions on Machine Learning Research (02/2026)
A Supporting Results from Section 4
The per-round data acquisition complexity of Algorithm 1 upper bounded by O ( + (St-1)) (D+)),
resulting in an overall complexity of O(T (Bc+TB) (D)).
Proof. We analyze the data acquisition complexity of Algorithm as follows.
Per-round complerity. We analyze the runtime of each major stop in the algorithm:
Line 5. The main computational cost arises from evaluating Eq. (1); which involves iterating over both Dr and S,-1. The resulting runtime is upper bounded by -(--))(Ss-Il). Line 6. For coreset construction, we the greedy approximation algorithm in Algorithm 2, which can be implemented in O ((Bc {S-1)) (Del) time using distance caching strategy: initially, we compute and cache the minimum distances between all candidate points in Dr and the selection set St-1, incurring a runtime of O([2a) (S,-) For subsequent iterations in the greedy selection process, we only need O((+e) operations to update the cache and select the next point. Lines 7-10. These steps perform uncertainty-base based selection. Computing uncertainty scores over Be coreset candidates against the other modality requires O(Bc (Dxl) operations.
|S--1)).
Summing the contributions from each step, the per-round data acquisition complexity is upper bounded by
0 ( + (8--1) [D+)).
Overall complerity. Since [St-1] = O(BB) and |D,| - [D), the total complexity over T rounds is
T Σ (Bc + +S, [Prl O(T (Bc + +TB) (D)).
B Other Details for Experiments
B.1 Additional Details and Baselines
Additional details on datasets. We use the 2017 release of MS-COCO, which contains approximately 118K training images (train split) and 5K validation images (val split). 5 Since only these two splits provide captions, we use the train split for training and the val split for lesting. Each image in MS-COCO is paired with five captions; in experiments, use the first caption for each image.
Baselines. We provide full implementations of the two baseline methods used in our experiments: CORESET (Algorithm 3) and UNCERTAINTY (Algorithm 4). Both are adapted from diversity-basse and uncertainty- active learning algorithms originally developed for unimodal settings.
In the multimodal setting, Algorithm 3 randomly selects modality and then constructs coreset within that modality using greedy algorithm. Algorithm computes margin-based uncertainty scores in both directions (text + image and image - text), and selects the top-B most uncertain instances for multimodal annotation.
To extend these baselines to the streaming setting, we adopt the same strategy as in Algorithm 1: line in both Algorithm 3 and Algorithm is replaced with the current batch of stream data De, while the remainder of each algorithm is left unchanged. We use Euclidcan distance the default metric Algorithm 1, Algorithm 3, and Algorithm Additional results using alternative distance metrics in our algorithm are provided in Appendix C.3.
B.2 Hyperparameter Settings
Tables 5 to 7 list the hyperparameters used in our experiments across datasets and model variants.
Snttps://cacddaaaastt.rr
16

---

Published in Transactions on Machine Learning Research (02/2026)
Algorithm Multimodal Coresct Sclection
Input:
Unaligned multimodal dataset D = {De, D',, number of iterations T, per-round selection size B.
1: Initialize multimodal model Φo = {0%, % with random or pretrained weights.
2: Initialize the annotation set So = 0.
for 1. do
Consider unaligned data pool D, : D St-1- Randomly select modality k E (v.
for c 1.... do
7: Zu = arg max EEO(!!") min = dist(2r,
Annotate 2u and add (=, to St-
Train multimodal model o; = (df Φf) the updated annotation set S..
Output: Actively trained multimodal model ør = (/).
Algorithm Multimodal Uncertainty-baseed Data Selection
Input: Unaligned multimodal dataset D {D",'', D'), number of iterations T, per-round selection size B.
1: Initialize multimodal model do {0%, d6 with random or pretrained weights.
2: Initialize the annotation set Sg = 0.
3: for 1. do
4: Consider unaligned data pool Dj = D'S:111
For each modality E {u,1} and for each data x E Ct2, compute its margin score u(x{) == w(t) - waay which serves an uncertainty measure. Here w' RIPTI denotes the vector of similarity scores between 2 and all unaligned features in the other modality := {r,() \ [k}, and w(y) denotes the j-th largest entry of w'. // Calculate the margin scores from both directions.
Select B data points with smallest margin scores with respect to {-] u (1))) annolate them, and update S; - S U {[>], + //If, within the top-B selection, selected image is paired with another selected text, then continue the selection process until getting B aligned pairs.
7: Train multimodal model Φ; = (o),o] on the updated annotation set S,. Output: Actively trained multimodal model Ør = 0/).
Table 5: Hyperparameter settings for the ColorSwap dataset using CLIP model.
Hyperparameters
CLIP-B32
CLIP-L14
Epochs
Batch Size
70
35
Optimizer
AdamW
AdamW
Weight Decay
0.1
0.1
Learning Rate
2 x 10-5
x 10-5
Bc
2.5B
2.5B
Table 6: Hyperparameter settings for the ColorSwap dataset using SigLIP and LiT.
Hyperparameters
SigLIP-B16
LiT-L14 SigLIP-L16
Epochs
Batch Size
35
35
Optimizer
AdamW
AdamW
AdamW
Weight Decay
0.1
0.1
0.1
Learning Rate
2 x 10-5
1 1 10-5
1 x 10-5
Be
1.5B
2.0B
2.5B
17

---

Published in Transactions on Machine Learning Research (02/2026)
Table Hyperparameter settings for the MS-COCO and Data PataComp datasets using CLIP model.
Hyperparameters
MS-COCO
DataComp
Epochs
10
1
Batch Size
256
512
Optimizer
AdamW
AdamW
Weight Decay
1 x 10-4
0.2
Learning Rate
1 x 10-5
6.25 x 10-5
Be
2B
2.5B
C Additional Experimental Results
C.1
Additional Results on Different CLIP Variants and Model Sizes
We present additional experimental results in Figs. to The conclusions from the main results in Section 5.2 remain consistent across different CLIP variants and model sizes: baselines.
Algorithm continues to outperform all
Text Score
Image Score
Group Score
66
72
65
60
66
60
54
60
Random
55
48
Coreset
50
54
42
Uncertainty Ours.
48
5% 10% 15% 20% 25%
30%
5%
10%
15% 20% 25% 30%
10% 15% 20% 25% 30%
Percentage of Aligned Data
Percentage Aligned Data
Percentage Aligned Data
Figure 6: Results of pool-based multimodal active learning on the ColorSwap dataset with CLIP-L14. We report text score (left), image score (middle), and group score (right) as learning progresses.
Text Score
Image Score
Group Score
86
88
80
84
84
80
Random
72
Coreset
76
Uncertainty
68
Ours
72
5% 10% 15% 20% 25%
30%
5%
10% 15%
20% 25% 30%
5%
10%
15%
20% 25% 30%
Percentage of Aligned Data
Percentage of Aligned Data
Percentage of Aligned Data
Results pool-based multimodal active learning on the ColorSwap dataset with SigLIP-L16. We report text score (left), image score (middle), and group score (right) as lcarning progresscs.
Figure 7:
C.2 Streaming Active Learning with Different Streaming Buffer Sizes
We conduct experiments on the DataComp dataset under two streaming buffer sizes, 1024 and 2048. In both cases, we select 25% of the data the buffer, resulting training batch sizes of 256 and 512, respectively. As shown in Fig. 9 (left and middle), our method achieves the best performance in both cases, indicating robustness to the choice of streaming buffer size.
18

---

Published in Transactions on Machine Learning Research (02/2020)
Text Score
Image Score
Group Score
44
44
30
42
40
27
40
36
24
Random
32
21
Coreset
36
28
18
Uncertainty
34
24
Ours
5%
15% 20% 25% 30%
5%
10% 15% 20% 25% 30%
10% 15% 20% 25% 30%
Percentage of Aligned Data
Percentage of Aligned Data
Percentage of Aligned Data
Figure 8: Results of pool-based multimodal active learning the ColorSwap dataset with LiT-L14. We report text score (left), image score (middle), and group score (right) as learning progresses.
C.3 Learning with Different Distance Metrics
our main experiments, we use the Euclidean distance as the default metric. To examine the effect of alternative distance functions, we additionally conducted experiments using cosine distance, defined as deos(z, y) (osineSimilarity(( [ ) As shown in Table our method exhibits comparable performance under cosine distance, achieving similar trends and performance levels to those obtained with Euclidean distance. This suggests that our method is robust the choice of distance metric.
Table 8:
Results of pool- bool-based multimodal active learning under different distance metrics on the ColorSwap dataset using SigLIP-B16.
Distance Metric
5%
10%
15%
20%
25%
30%
Euclidean Distance
2..34
69.0012.02
71.83
74.2512.50
76.9211..9 74.67+1196
19.42
Cosine Distance 65.17
68.1742.14
74.17+2.09
77.83+1.73
78.67+:2227
C.4 Comparison against Additional Adapted Uniform AL Baselines
Beyond adapting the unimodal CORESET and UNCERTAINTY baselines to our setting, we additionally evaluate an adapted version of the unimodal BADGE algorithm (Ash et al., 2020). We adapt BADGE in the most direct manner: first randomly select modality and then apply BADGE as if operating in unimodal setting with classification losses. Results in Fig. 9 (right) indicate that our method consistently outperforms this adapted BADGE bascline.
Although both BADGE and our approach combine uncertainty and diversity principles, we hypothesize that BADGE weaker performance stems from a fundamental mismatch with our problem structure. Specifically, BADGE performs clustering in a gradient embodding space of dimension O(K)), where d is the embedding dimension and K is the number of classes. While K is typically small in unimodal classification tasks, in multimodal learning with bidirectional alignment, K effectively scales with the number of data points, since texts and images generally unique. This issue is highlighted in Section 3.1 as The resulting extremely high-dimensional gradient embeddings make BADGE 's clustering step ineffective in ice In addition, BADGE incurs quadratic per-round computational complexity in the number of data points, whereas our method explicitly designed to achieve linear per-round complexity, which critical for scalability in large multimodal datascts.
challenge of our setting,
19

---

Published in Transactions on Machine Learning Research (02/2026)
Average Score
Average Score
Group Score
11
48
Random
10
Random Coreset
Coreset
44
Uncertainty
Uncertainty
BADGE
Ours
40
Ours
0.5
1.0
1.5
2.0
2.5 0.5
1.0
1.5
2.0
2.5 x10
5%
10%
20%
25%
30%
Number of Aligned Data x10*
Number of Aligned Data
Percentage of Aligned Data
Figure Left and Middle: Results of streaming- multimodal active learning on DataComp (CLIP- B32) under streaming buffer sizes: 1024 (left) and 2048 (middle). We report the average score across 38 downstream tasks learning progresses. Right: Results of pool-based multimodal active learning ColorSwap (CLIP-B32): comparison with an adapted BADGE baseline.
20


Reviews:
[{"review_id": "85D7epIC3q", "reviewer": "Reviewer_PyWL", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper explores a new problem setting in multimodal active learning where the number of multimodal pairs are limited. The problem itself is non-trivial and interesting. The authors propose an active learning algorithm that leverages cross-modal uncertainty to iteratively refine and construct pseudo-annotation of multimodal pairs to continuously train a more pair-efficient multimodal model. The writing presents a clear, well-structured algorithm on the proposed framework as well as optimization techniques. The authors evaluate data efficiency of the proposed methods with three baselines on three image-text datasets. Finally, the authors perform ablation studies and additional analysis to understand the contribution of each module.\n\nAlthough the problem is novel with clearly written method, I feel like the evaluation falls short and lack key details that obscure the understanding of the method and significantly limit the impact of the work. I have summarized the strengths and weaknesses as the following:\n\nStrengths:\n- The formulation of multimodal active learning with unaligned data is very interesting and non-trivial.\n- The methodology is clearly written and easy to follow.\n- The current evaluation on the two settings (pool-based and streaming-based) with different model families and parameter sizes are critical and solid.\n\nWeakness:\n\n- The margins between the proposed methods and the baselines are fairly inconsistent, where the uncertainty and coreset baselines sometimes match or even outperform the poropsed methods with limited number of modality pairs. Even the random pairing can sometimes outperform other methods. This raises quesitons about whether the proposed method can be meaningful in more complicated scenarios.\n- The authors leveraged pretrained multimodal models like CLIP and SigLIP for some of the datasets evaluated. These models are already well-aligned on large-amount of paired multimodal data. It is unclear on how these priors could affect the performance during the finetuning with limited multimodal pairs. A study comparing pretrained models and randomly initialized models on the same dataset could provide more insights.\n- The datasets evaluated already consist of full pairings between the modalities. These serve as natural ground truth pairing labels to evaluate the quality of the constructed modality pairs. The current evaluation only focuses on the downstream performance as an implicit metric for the constructed pairing sets, but lacks a deeper analysis against the ground truth pairing. A more thorough analysis of how well the selected annotation achieves cam strengthen the claim that active selection provides true annotation efficiency, where we can achieve real pairing with a few pair priors only.\n- The baselines are relatively old and the datasets are narrow. Although the authors state that the problem setting is new, there are many existing works in recent years that tackle multimodal alignment with limited pairs with either cross-modal data pairing [1] and cross-modal model alignment [2]. The authors should discuss these and consider including them in the evaluation. The datasets are limited to image and text modalities only, and more diverse datasets covering differet number of modalities could enhance the validity. \n    [1] Learning to Rematch Mismatched Pairs for Robust Cross-Modal Retrieval\n    [2] InfoMAE: Pair-Efficient Cross-Modal Alignment for Multimodal Time-Series Sensing Signals\n- Although the authors have considered the imbalance between modalities with the modality selection module, I am curious on how well the model could perform when there is a significant imblance between the two modality sets. The current setup assumes that the number of modality samples are similar, but it would improve the impact and more practical if the authors could discuss and analyze the scenario where we have significantly more unimodal samples than other modalities. Since many multimodal corpora are highly unbalanced (e.g., millions of images but only thousands of texts), it would be valuable to evaluate how the algorithm over- or under-samples certain modalities.\n- The authors have included a good theoretical time-complexity analysis of the optimized method (algo 2). However, the paper misses actual training time evaluation. It is unclear on the time spent on each of the proposed modules and misses comparison of the overall latency compared to prior works. An evaluation on the empirical efficiency could improve the practicality of the work.", "claims_and_evidence_yn": "No", "claims_explanation": "The evaluation results show fairly marginal and inconsistent improvement across different scenarios. In some setups, the baselines including random pairing can outperform the proposed methods with limited number of pairs. The paper also lacks some critical evaluation that limit the evidence as a SOTA baselines.", "audience_yn": "Yes", "audience_explanation": "The new problem setting on multimodal active learning with limited pairs is critical and can be an interest for the TMLR audience.", "requested_changes": "Most of the requested changes match the weakness above. Below are the most critical changes suggested from most critical to least critical:\n\n1. Evaluate alignment quality against ground-truth pairings.\n2. Isolate the effect of pretrained multimodal models on ColorSwap and MS-COCO\n3. Expand or justify baseline selections, especially regarding recent multimodal rematching and alignment methods discussed in the weakness section.\n4. Analyze performance under strong modality imbalance.\n5. Empirical runtime measurement.", "broader_impact_concerns": "No ethical concerns.", "additional_comments": "Some additional questions I have regarding the design choice:\n\n1. Existing method is evaluated primarily using CLIP-style models, where the encoders are already aligned through pretraining. How would the proposed method perform when the image and text encoders are **pretrained independently** (e.g., ViT trained on image and BERT trained on text) such that the embedding spaces are not at all aligned initially?\n2. Recent work has shown that **intra-modality structural relationships** (e.g., distances, neighborhoods, or relational geometry within each modality) tend to be preserved across modalities [1]. For instance, if text sample A is more similar to text sample B than to text sample C, the corresponding image samples often exhibit a similar relational ordering. Could such structural consistency be exploited to guide the pairing process?\n[1] Understanding and Constructing Latent Modality Structures in Multimodal Representation Learning", "extra_scores": {}, "extra_text": {}}, {"review_id": "8LPQuXXQDq", "reviewer": "Reviewer_TrcH", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors propose an Active Learning strategy to sample cross-modal pairs that are diverse and uncertain. Their method first selects the modality with fewer annotation pairs, then greedily estimates a coreset in the selected modality, and then computes margin-based similarity scores between the embedded representation of each coreset sample in the first modality and any such vector of any sample in the second modality.\n\nThe main contribution is the design of the modality-aware sampler, that applies established methods in a new data setting that suffers from the interesting modality alignment problem.\n\nA key strength of this paper is the development of the authors' idea and the clear explanation of the concepts and algorithms.\n\nA weakness is the experimental section with the mono-criteria baselines Coreset and Uncertainty that are either diverse or uncertain but do not combine both like Badge or BatchBALD (or this paper's contribution). Furthermore, the paper repeats certain text segments without going much deeper, which would be much preferable.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The experimental section could also compare with other methods that combine uncertainty and diversity, such as Badge or BatchBALD. The two chosen baselines besides random have very well established weaknesses and the mentioned algorithms may be more suitable. Yes, they would need to be adapted similarly like you described in B.1.\n\nThe ablation studies on the modality selection, acquisition size and on the uncertainty estimation seem to show the robustness of the proposed method on the provided datasets, models, and learning settings (batch/stream).\n\nThe complexity discussion shows well how costly this method would be theoretically. However, please include runtime costs, as the operations for different acquisition sizes B for large datasets may be much more costly than simpler random sampling.", "audience_yn": "Yes", "audience_explanation": "Yes, the proposed three-step algorithm addresses an interesting problem.", "requested_changes": "1. The experimental section could compare with other methods that combine uncertainty and diversity, such as Badge or BatchBALD. The two chosen baselines besides random have very well established weaknesses and the mentioned algorithms may be more suitable. Yes, they would need to be adapted similarly like you described in B.1.\n\n2. Figure 5 includes only Uncertain and the proposed method to make the point that the proposed method is sampling more representatively. Why not include also Coreset (designed to be representative)? \n\n3. Discuss the limitations of the proposed method.\n\n4. Discussion the literature on AL for multimodal similarity may strengthen the RW section.\n\n5. The complexity discussion shows well how costly this method would be theoretically. However, please include runtime costs, as the operations for different acquisition sizes B for large datasets may be much more costly than simpler random sampling.\n\n6. typo: headline 4.1: \"Mutlimodal\"\n\n7. Discuss whether your version of margin sampling is biased towards dense regions of the embedding space where different pairs are closer compared to sparse regions where the margins are naturally larger. Can your algorithm's coreset step counter this effect, and would you investigate this experimentally?\n\n8. Have you studied different distance functions for the distance metrics in steps 2 and 3 and where there any effects?\n\n9. Please include a short paragraph on the essential settings of the experimental setup in the main text, such as the number of repetitions of the experiments (hopefully more than 5), and the acquisition sizes.\n\n10. Some citations are for preprints that are already published, please update the list with current versions of the papers.\n\n11. Finally, some text segments repeat throughout the paper with minimal changes, such as the related work section. A deeper discussion in these segments, e.g., the related work, would be preferable, or else the manuscript just repeats the texts without adding anything new.", "broader_impact_concerns": "I don't see broader ethical implications from this contribution.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "LwsGbfQD4n", "reviewer": "Reviewer_961k", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper investigates an important yet previously underexplored problem: active learning on unaligned multimodal data (e.g., images and texts that are not paired). In contrast to prior work that typically assumes image–text pairs are already aligned or focuses solely on labeling downstream tasks, this paper redefines the annotation action as acquiring the true cross-modal alignment between an image and its corresponding text. The goal is to substantially reduce the expensive human effort required for constructing large-scale paired datasets used in contrastive multimodal learning.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The main claims of the paper are generally supported by the experimental evidence. Specifically: (1) The authors conduct systematic and comprehensive evaluations across three representative settings—pool-based active learning, stream-based active learning, and large-scale contrastive pre-training on tens of millions of image–text pairs—covering different model sizes, data regimes, and task configurations. (2) The baselines are well-chosen, including random sampling, multimodal coreset methods, and uncertainty-based active learning strategies, all of which effectively highlight the performance gains introduced by the proposed approach. The paper also provides extensive ablation studies—such as analyses of the modality selection strategy, coreset parameters, and the effectiveness of the margin-based uncertainty measure—which further validate the necessity of each component and the soundness of the overall design.", "audience_yn": "Yes", "audience_explanation": "The topics addressed in this paper—multimodal contrastive learning, data-efficient learning, and active learning—are all central research areas within the current machine learning community. Moreover, the problem of actively selecting and aligning unpaired multimodal data remains a practical bottleneck that has not been adequately solved. In addition, the proposed method demonstrates substantial reductions in alignment costs for large-scale pre-training, making the findings directly valuable for researchers and practitioners working on multimodal representation learning, data acquisition strategies, and efficient model training.", "requested_changes": "Although the experiments cover the streaming setting, the paper provides limited analysis regarding the potential long-term performance degradation caused by “missing” certain data batches. It is recommended to include additional experiments or discussions on data ordering, batch size, or delayed sampling strategies to strengthen the credibility of the streaming results.\n\nWhile the task is novel and the method is effective, the theoretical analysis is largely limited to computational complexity. It would be beneficial to include further discussion on why the combination of coresets and uncertainty sampling is particularly effective, even without formal proofs, to better convey the design intuition.\n\nReal-world multimodal datasets often contain duplicated text, incomplete descriptions, or weakly related images. It would be valuable to analyze the robustness of the proposed method under such noisy conditions, for example by providing qualitative discussions or small-scale comparative experiments.", "broader_impact_concerns": "Overall, the work focuses on improving data efficiency in multimodal contrastive learning and does not raise substantial or broad ethical concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)