CoIDO: Efficient Data Selection for Visual Instruction Tuning via Coupled Importance-Diversity Optimization
Authors: Yichen Yan
Ming Zhong
Qi Zhu
Xiaoling Gu
Jinpeng Chen
Huan Li
Venue: NeurIPS 2025 poster ()
Keywords: Large Language Model
Visual Instruction Tuning
Data Selection
Primary area: deep_learning
TL;DR: CoIDO is a lightweight dual-objective data selection framework that jointly optimizes importance and diversity, achieving 98.2% of full-data performance for multimodal instruction tuning using only 20% of the data.

Abstract: Multimodal large language models (MLLMs) rely heavily on instruction tuning to align vision and language capabilities, yet the computational cost of training on large-scale datasets remains a major bottleneck. Existing data selection methods aim to mitigate this by selecting important and diverse subsets, but they often suffer from two critical drawbacks: high computational overhead from processing the entire dataset and suboptimal data selection due to separate treatment of importance and diversity.

We introduce CoIDO, a novel dual-objective framework that jointly optimizes data importance and diversity to overcome these challenges. Unlike existing approaches that require costly evaluations across the whole dataset, CoIDO employs a lightweight plug-in scorer. This scorer is trained on just a small random sample of data to learn the distribution of the candidate set, drastically reducing computational demands. By leveraging a homoscedastic uncertainty-based formulation, CoIDO effectively balances importance and diversity during training, enabling the scorer to assign CoIDO scores to all data points. This unified scoring approach allows for direct ranking and selection of the most valuable subsets, completely bypassing the need for specialized algorithms.

In our experiments, we trained the CoIDO Scorer using only 20% of randomly sampled data. Once trained, CoIDO was applied to the entire dataset to select a 20% subset for instruction tuning. On the widely used LLaVA-1.5-7B model across ten downstream tasks, this selected subset achieved an impressive 98.2% of the performance of full-data fine-tuning, on average. Moreover, CoIDO outperforms all competitors in terms of both efficiency (lowest training FLOPs) and aggregated accuracy. Our code is available at: https://github.com/SuDIS-ZJU/CoIDO

ColDO: Efficient Data Selection for Visual Instruction Tuning via Coupled Importance-Diverstyy Optimization
Yichen Yan', Ming Zhong', Qi Zhu', Xiaoling Gu2, Jinpeng Chen', Huan Li
College Computer Science and Technology, Zhejiang University Hangzhou Dianzi University, Hangzhou, China
3 School of Computer Science (National Pilot Software Engineering School), BUPT {yichen, yan, chime, lihuan. ez) edu cn, qizhu. zju. research@@mail. com, guxl@hdu. edu. en, jpchendbupt. edu.
Abstract
Multimodal large language models (MLLMs) rely heavily on instruction tuning to align vision and language capabilities, yet the computational cost of training on large-scale datasets remains a major bottleneck. Existing data selection methods aim to mitigate this by selecting important and diverse subsets, but they often suffer from two critical drawbacks: high computational overhead from processing the entire dataset and suboptimal data selection due to separate treatment im- portance and diversity. We introduce ColDO, novel dual-objective framework that jointly optimizes data importance and diversity to overcome these challenges. Unlike existing approaches that require costly evaluations across the whole dataset, ColDO employs a lightweight plug-in in scorer. This scorer is trained on just small random subset of data to learn the distribution of the candidate set, drastically ducing computational demands. By leveraging a homoscedastic uncertainty-batt formulation, ColDO effectively balances importance and diversity during training, enabling the scorer to infer ColDO scores for all samples. This unified scoring approach allows for direct ranking and selection of the most valuable subsets, completely avoiding the need for specialized algorithms. In our experiments, we train the CoIDO Scorer using only 20% of randomly sampled data. Once trained, ColDO is applied to the entire dataset to select a 20% subset for instruction tuning. On the widely used LLaVA-1 1.5-7B model across ten downstream tasks, this selected subset achieves an impressive 98.2% of the performance of full-data fine-tuning, on average. Moreover, ColDO outperforms all competitors terms of both efficiency (lowest training FL.OPs) and aggregated accuracy. Our code is available //ttps:///gttaii com/ Soom/SSDIS... 2JUU/CIDD
Introduction
Instruction tuning has become fundamental for aligning Multimodal Large Language Models
(MLLMs)) with human intent, empowering models such as GPT-40 1. Gemini (2) and LLaVA B to
handle diverse downstream tasks including visual question answering 4 6. image-text retrieval [7.9
and visual grounding [10H12). While large-scale visual instruction datasets (e.g., LLaVA-6655 [13
have enabled impressive performance, they introduce substantial redundancy, high computational costs, and optimization inefficiencies. For example, fine-tuning single epoch of LLaVA-1.5-7B
typically requires over 20 GPU hours on 8xA100 40GB GPUs [13] Such computational demands
"Corresponding author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

(a) Previous stale-of-lbe-at metthods:
Training Stage: Train the MLLM evalunte importance
Selectio Stage: Select data diversity adjustment
Optiiesize Importance
Importance
Optimize Diversity
Importance
MILMM
Selection algeritm
Selected
Data
Trsining
Diversity
Selecting
Diversity
Full Data
Full Data
(b) Our lramework:
Training Stage: Train ColDO Scorer with 1% random data
Selection Stage: Select ColDO Scorer
ColDO
Coupical Optimization
Importance
Diversity
CalDO
Selected
Scorer
Data
MILMM
Scoring Selecting
10%
Data
Full Data
Co-training
Figure 1: Previous state-of-the-ane methods (c.g., TIVE [18] and ICONS (19] treat importance and diversity as decoupled and independent components, using the entire data the training stage. Our CoIDO integrates importance and diversity in a coupled and reciprocal optimization, achieving superior data selection by utilizing only a fraction (p 100) of the full dataset for model training and without any specialized algorithm in the selection stage.
pose significant challenges for efficient deployment, particularly for research groups organizations with limited access to large-scale hardware resources. Recent works, such as LIMA [14), indicate that carefully selecting small subset of high-quality instructions significantly reduces computational costs while preserving model performance. To mitigate these challenges, data selection methods have been proposed to identify high-quality instruction data that can enhance the performance of large language models (LLMs).
Early data selection methods 114 have primarily focused text-only instruction tuning for LLMs, with limited exploration in the multimodal domain. Unlike text-only data, visual instruction tuning introduces additional complexity due to its bimodal nature. This complexity brings two more challenges for the data selection task: (1) processing image-text data pairs demands greater computational resources, and (2) evaluating the semantic alignment between images and text is inherently more complex. Consequently. it becomes more challenging to account for both the importance of individual samples and the diversity of the dataset whole. As illustrated in Figure (a), current approaches tailored for MLLM [8] 20 typically involve two stages: the training stage and the data selection stage. For importance optimization, existing approaches typically require the target MILM to process the entire candidate dataset in the training stage, either by evaluating importance indicators such as gradients [T8 19, , by leveraging intermediate layer features for clustering (21) to compute transferability. which can be regarded as importance. For diversity optimization, they also need design complicated selection algorithm in the selection stage, such as soft sample [18 or a similarity penalty (20 to optimize the diversity of the subset.
However, these methods have major drawbacks: (1) The training and selection stages decou- pled processes. Data importance is assessed during training, while diversity is handled separately through specialized algorithms in the selection stage. This separation fails to jointly optimize the two objectives and often results in a suboptimal trade-off. Specifically, placing too much emphasis on importance may reduce overall dataset diversity, while prioritizing diversity could exclude high-value samples. (2) They require the target MILMM training to process the entire dataset to compute sample importance. This results in computational costs comparable to full fine-tuning. This fundamentally contradicts the objective of data selection, which is to reduce training overhead. Moreover, any addition of new data necessitates reprocessing the entire dataset, severely limiting scalability and efficiency. Therefore, an ideal data selection framework should satisfy two key criteria:
(i) jointly optimize data importance and diversity, and (ii) significantly reduce computational overhead by utilizing only a limited subset of candidate data before the selection.
Building on these considerations, we propose novel and efficient data selection framework, named COlDO (Coupled Importance-Diversity Optimization), explicitly designed to overcome the two critical issues identified above. As illustrated in Figure (b), CoIDO utilizes only p% (c.g., up

---

to 20%) of randomly sampled data to train a lightweight scorer, instead of fine-tuning the target MLLM extensively evaluate the entire dataset. Specifically, in the training stage, we introduce a plug-in CoIDO Scorer. Unlike previous methods that require evaluating each data sample during training, CoIDO Scorer learns the data distribution from a small subset. Additionally, we move the optimization of data diversity, which was traditionally handled in the selection stage by selection algorithm, to the training stage. We design an importance loss based on the backpropagation and a diversity loss integrating the spectral clustering. These two objectives are jointly optimized during the training stage via a homoscedastic uncertainty-base optimization [22]. The scorer assigns each data sample CoIDO score that takes both importance and diversity into consideration.
In the selection stage, we select the top-scoring samples from each task's 's candidate pool in identical proportions. This ensures balanced coverage and prevents task bias that might arise from global ranking and selection. Our method only requires examining a small amount of data, which greatly reduces the training cost. More importantly, since our scorer learns the underlying data distribution efficiently from a small subset, it transfers seamlessly to new in-domain candidate data without retraining, substantially enhancing the scalability in data selection. In summary, our contributions are highlighted in three aspects:
(i) We introduce a novel dual-objective optimization approach for MLLM visual instruction data, using coupled optimization for data importance and diversity. (ii) We propose lightweight scorer pipeline that learns the candidate data distribution from a small amount of data in the training stage, enabling efficient and accurate ranking across the entire dataset at significantly reduced training cost. (iii) In the selection stage, ColDO Scorer can directly select data from the entire dataset without additional diversity optimization algorithms. Experiments show that fine-tuning the LLaVA-1.5 with only 20% of the samples selected by ColDO achieves 98. 98.2% of the performance of using the full dataset.
2
Related Work
Data Selection for LLMs. Data selection is critical for optimizing both pre-training and fine-tuning of LLMs. For pre-training, several methods 23 26
curate diverse large-scale datasets improve diversity rather than instruction-specae -secifiic quality. During instruction tuning, where instruction quality is crucial, recent methods like LIMA [14] and ALPAGASUS [15] employ gradients or an L.LM-based evaluator to filter low-quality samples. Similarly, NUGGETS (27) utilizes zero-shot and one-shot evaluation, while CAR (28 incorporates expert-aligned scoring with clustering to preserve diversity. Though effective, these methods are designed for text-only datasets and fail to account for more complex semantics in multimodal datasets.
foundation models, but focus on seneral-purpont
Data filtering methods based on static metrics 29 33 are not exclusive to textual data. However, they demonstrate suboptimal performance in the context of visual instruction tuning (see Table I).
Data Selection for Visual Instruction Tuning. MLLMs such as Flamingo (34. LLavA B. and BLIP2 35 rely on instruction tuning to align visual and textual modalities. Early efforts like INSTRUCTIONGPT-4 (36] target small-scale datasets (e.g., MiniGPT-4 (37), 3.4K instructions) and require evaluation across multiple downstream tasks as labels, limiting scalability. More recent methods attempt to address this by prioritizing high-value samples from larger instruction datasets, such as LLaVA-6655
[3]. Typical methods include SELF-FILTER (20) that employs a scoring network, TIVE [8] based on gradient to compute data influence, ICONS [1)], which applies bradient-based selection to identify representative data from tasks, and COINCIDE 21,, which employs clustering techniques based on concept-skill composition representations.
These methods [8] 19 typically require full full-dataset processing by the target MLLM, incurring high computational cost. They also decouple the optimization of importance and diversity, often leading to suboptimal subsets. In contrast, our proposed CoIDO reduces overhead by training lightweight scorer on a small subset, while jointly optimizing importance and diversity. Moreover, approach can transfer scamlessly to new in-domain candidate data without retraining
3

---

u22
Mean
Coupled optimization
After optiiiationn
High Li-
Weighting
Mean
Weighling
2
W
Mean
Variance
Loss
Importance Loss
Weighling
333
Average score
optimize
loptimzee
Mean
clusters
Wi
Weighling
Spectral Clustering
W1
2
3/
203
3
ColDO
Loss of samples
ColDO Scorer
Target Multimodal
Large Language Model
Image+ Text+ Image-taxt features
Image-Texx Text- Image-lext features
Fetch reievent features the minibatoh
Feature Extraction
Minibatch
batch size S
Q: How might girl's choice of activity inflnence her social her attire may influence her social
young girl's activity,
Raw Dataset
interactions and life style?
interactions in various ways, such ii.....
Figure 2: Framework overview of ColDO (also see Appendix A.2). Basic features (image, text, and image-text) are extracted for spectral clustering and scorer training. The plug-in ColDO Scorer, co-trained with the target MLLM (e.g., LLaVA-1.5-788 scores are used to compute the importance loss &, and diversity loss Ln. which are jointly optimized to capture both data importance and diversity via backpropagation using batches of random samples.
outputs a score w; for each sample. These
Methodology
3.1 Task Description, Framework, and Key Modules
Task Description. Given a large-scale visual instruction dataset D = (= where each sample 2y (zj. yj) includes a visual instruction Zj captured by an image-tex, pair (see Figure bottom- right), and a target response Yj, the task of data selection for visual instruction tuning is to efficiently select a subset Dr C 'D with [D,] = NN. The subset should reduce visual instruction tuning costs while maintaining at least Rel. percentage of the performance achieved using the full dataset.
Typically, E 0.4] and Rel. 2 95% are desired by real-world applications [20 18 19 21
Overall Framework. Our proposed ColDO framework efficiently selects high-quality subsets by leveraging only fraction 5% (e.g., 20%) of random samples for training. As shown in Figure 2 we first extract multimodal features and scoring metrics (evaluating text, images, and image-text alignment) from the raw dataset. These features are used for two purposes: (i) training the lightweight ColDO Scorer to assess data importance and diversity in one pass, and (ii) clustering to obtain class assignments for each data sample, which will be subsequently used in modeling the diversity loss.
This selection framework, centered around the ColDO Scorer, is then trained using the sampled data and extracted features in a minibatch mode: basic features are fused and fed into the scorer, while the raw samples are simultaneously used fine-tune the target MILM.
During training, the ColDO Scorer assigns scalar score w; referred to as the CoIDO score, to each sample. This score reflects both data importance and diversity due to the coupled optimization of these objectives (see Section 3.2). ). In the selection phase, the trained scorer is applied to the full dataset to infer the ColDO score for each sample. The final subset is obtained by ranking the samples based on their scores and selecting the top fraction ? per downstream task.
Feature Extraction. As depicted in Figure 2 we extract set of basic features for samples to train the scorer and perform clustering. These include: 1) Texi Features: Captured using an L.LM- based evaluator, which assesses dimensions like spelling and grammar quality. The resulting score,

---

LD. which minimizes the variance of average weights referred to as the LLM Score, is detailed in Appendix A. 2) Image Features: Measured using the ImageReward Score [8), which evaluates image fidelity and semantic clarity. 3) Image- Text Features: Derived from a pre-trained CLIP encoder [39, including the low-dimensional multimodal representations of image-text pairs and the corresponding CLJP Score [29] on image-text alignment. These features collectively provide comprehensive assessment of text quality, image quality, and image-text alignment. The combined features are concatenated as input for clustering and training.
Spectral Clustering, We use spectral clustering 40 to group data samples into a total of M classes. Unlike K-means, spectral clustering better suited for capturing complex, non-linear distributions in multimodal data. The resulting clusters represent fundamental class information, which is used to compute diversity loss during optimization. Further implementation details and experiments on feature usage and clustering methods are provided in Appendixes A.I C.I and
ColDO Scorer Training. The scorer co-trained with the target MILLM to assign ColDO scores to data samples for the subsequent selection stage, where the features of samples are fetched from the previous clustering stage. During training, we randomly sample fraction p% of the dataset. Each batch serves two purposes: (1) fine-tuning the target MLLM using standard instruction learning, and (2) training the plug-in ColDO Scorer, which processes corresponding basic features via lightweight architecture, like a multilayer perception (MLP). For each batch, the CoIDO Scorer outputs CoIDO score per sample, while the target MLLM computes a cross-entropy loss. These outputs are jointly optimized to balance data importance and diversity. as detailed in Section 3.2 We further explore alternative CoIDO Scorer architectures and optimization methods in Section 4..
Importance Loss. Training loss is a well-established indicator of instruction difficulty, as higher losses typically correspond more challenging and valuable samples for improving model general- ization 42 20) Building on this insight, we introduce learnable CoIDO scores, predicted by the scorer, to re-whight the prediction loss for each data sample.
Since more difficult instructions generally result in higher losses, the corresponding learnable scores expected to decrease during training, thereby implicitly capturing sample importance. For each batch, the ColDO scores are processed using Softmax and subsequently applied to modulate the cross-entropy (CE) loss of the target MLLM, producing weighted sum over all samples (see top-right of Figure
Let represent the number of samples in a batch, drawn from m clusters (where m S M, as some clusters may not appear in this batch). For the i-th cluster, with n, samples (s = D/ n.), the importance loss L is defined as:
4 where you and du denote the ground truth and predicted labels for the k-thh sample in the i-th cluster, respectively. We apply a Softmax operation to obtain normalized ColDO scores Wike which will be used in coupled optimization of the framework (see Section 3.2).
Wis CE(y,:: 9sk),
(1)
A high cross-entropy CE(Uae, Dik) implies the sample is difficult to learn and thus more important. According to the principle of backpropagation, to minimize the overall loss, the model naturally adjusts the weight was downward for high-difficulty samples. In other words, lower was corresponds to a more important sample. The formulation L, is therefore designed to capture and optimize the importance of individual samples in the data selection process.
Diversity Loss. While the importance loss L, optimizes sample weights wis from the data importance perspective, it can lead to imbalanced weight distributions across clusters. Certain clusters may consistently dominate with high or low weights, particularly when they group overly simple or excessively difficult samples. This imbalance risks selecting subsets dominated by few clusters, reducing diversity and potentially impairing the model's 's generalization performance.
To address this, we introduce diversity loss across clusters:
CD = Var({ (1, 02 Dm)))
w. =
Wik:
(2)
k=1
where Var(-) denotes variance, and W; is the average weight of samples in the i-th cluster. Only clusters with samples in the current batch included in the computation.
This loss encourages balanced weight distribution across clusters, promoting inter-cluster diversity while preserving intra-cluster variation. For clusters with high average weights (c.g., red an-d green
5

---

in Figure 2 top-left), W; is reduced, while for clusters with low weights (e.g., purple and blue), W, is increased. This prevents overrepresentation of any single cluster the final selection, ensuring more diverse subset.
Notably, LD only acts the cluster-level mean values {w:) and does penalize the variance within each cluster, thereby preserving the intra-cluster ranking of sample importance. By jointly optimizing importance and diversity losses, CoIDO scores reflect both the importance and diversity of samples, ensuring balanced and representative data selection.
3.2 Coupled Optimization
Importance and diversity somehow inherently conflict in data selection. Prior methods like SELF- FILTER (20) and TIVE 17 address these objectives in separate optimization stages but often fail to achieve optimal balance. Unlike these approaches,
method considers both importance and diversity in modeling wak- However, direct summation requires heuristic tuning of scalar weights between objectives, which can be sensitive to scale differences and lead unstable or biased optimization. Moreover, fixed weights cannot adapt to varying task uncertainties or objective difficulties during training, potentially resulting in suboptimal trade-offs between importance and diversity. To address this, we propose dynamic and automatic balancing mechanism during training.
We formulate this problem using maximum likelihood estimation (MLE) framework under task- specific uncertainty. Each loss term is treated as the negative log-likelihood of a probabilistic model, where learnable parameters capture the inherent uncertainty or noise in optimizing each objective. This concept, termed homoscedastic uncertainty [22], assumes that uncertainty is constant for a given objective but varies across objectives, aligned well with our multimodal scenario.
We introduce ør and on as learnable parameters representing the uncertainty for the importance and diversity objectives, respectively. Let 0 denote the model parameters, y the outputs of the target MILM, and w the ColDO scores from the scorer. Using the multi-task likelihood framework, the total likelihood is modeled as:
log ply, w 0, (101) =E,, log p(ysk Tzk. 0,0,) + Σ, log log p(ū;
(,
(3)
Importance Objective. We begin with the importance objective L,, formulated within an instance- weighted likelihood framework. For sample (æ, Vik), we directly introduce temperature pa- rameter σe and sample-wi wise score Wik scale logits, yielding weighted Boltzmann (Gibbs) distribution [3)
plya Take O, I wa) = Softmax
where a controls the sharpness of the distribution, fo(r) the model output, and was adjusts the sample's importance. Taking the negative log-likelihood yields:
- llog = Wik
Sa + log E, exp
Wikk
(5)
of
where fo the ground-truth logit of class c. Defining a = Wisk approximated as:
the second term Equation 5 can be
g(a) = logg Σ, explaf)) log S + log E, PS.
(6)
where S = =3
Ejeb eb and pj = =ela/S. Applying second-order Taylor expansion of log E, 12f around
1:
log E, pj ~ log E, (a - 1)
Given that Σ₁p} = the zero-order term is eliminated, and O((a - 1)2) refers second-order infinitesimals. Besides, (p)) = E, Pj log the entropy. Substituting this approximation back to g(a). we have:
g(a) = a log S (a (p).
(8)
We justify why the first-order error term (a - 1) Hp)(p) can safely neglected in practice: According to the definition of entropy, H (p) reaches its maximum value log O when the output distribution is

---

uniform over C' classes. However, in practice, model outputs are highly concentrated: the effective number of candidate tokens T is typically much smaller than the vocabulary size C, reducing the upper bound of H (p) to log 7
Empirical studies
suggest T typically ranges from 5 to 10 for LLMs, keeping H (()) low. Combined with the fact that (a 1) is close to zero, the contribution of the first-order term to the gradient is negligible. Hence, the final approximation of the negative log-likelihood for single sample simplifies to:
44
45
loga
Tat 0, 01 Wx) R
Wik of
CElUiks 9ik)-
For minibatch samples from m clusters (each with n, samples), the final importance loss in optimizations becomes:
E
i.e log plyex 0u,0,,0 0,))) o7 + logar.
1
(10)
The log oy term arises from the normalization constant of the Softmax-bast likelihood under temperature scaling, as a direct consequence of MLE.
Diversity Objective. Following the multi-task learning framework from Kendall et al. [22] we interpret the diversity loss probabilistically. We consider a regression perspective by modeling the cluster-level mean weights wi as Caussian-distribute random variables with a common mean μ and variance oi:
p( 1,, op) = NN μ1, (b)
(11)
In this formulation, the variance o reflects the homoscedastic uncertainty of the diversity objective, acting as a learned task-dependent weight that downscales LD when the diversity signal is noisier or less reliable. The negative oikeeeiid for this regression objective thus takes the form:
(wo 14) +
1
log pegp((; 0, (D)
1ogo
(12)
2
By considering all clusters and letting the μ be the mean of wú summing across all clusters, and omitting constant terms, we arrive at the diversity loss expressed as:
Σ, logp(w;; W. 0m
1 Ln + logøn. 20
(13)
where LD Var(wi,..... W.) is exactly our diversity loss derived Section 3.1
Coupled Objective. Combing Equation 10 and Equation 13 we obtain our final optimization goal:
Ciotal - 20 LD
CD + log + log ap.
(14)
07
In Equation| 14 σi and ap regulate the homoscedastic incertainty associated with the objectives of importance and diversity, facilitating the balancing and adaptive optimization of the selection. At inference time, we discard the uncertainty parameters and directly use the trained ColDO Scorer to assign selection scores to candidate samples. To construct the final subset, for each downstream task, we select the top-ry fraction of samples with the lowest (i.e., most ColDO scores within that task's candidate pool. This ensures the selected set maintains balanced task coverage while preserving the optimization goals of importance and diversity.
4 Experiments
Model Implementation. We conduct experiments on the LLaVA-665K visual instruction tuning dataset [13), using LLaVA-1.5-7B-LLF [ the target MLLM. Our framework trains p 20% of the dataset, sampled randomly, with training configuration aligned with the standard LLavA. The ColDO Scorer is four-layer ML.. More implementation details in Appendix B.2
we evaluate CoTDO with a wide range of multimodal benchmarks that test different capabiliiies of MLLMs. The benchmarks include: I)
Evaluation Tasks. Following the previous work [19 21
7

---

Table 1: Overall performance and efficiency comparison of selection approaches across various multimodal evaluation benchmarks, with the best measures in bold and the second-best underlined.
Method
VQA=2
GQA
VirWir
SQA-I
TeNVVA
POPE
MME
MMBench
LLaVA- Bench
Rel (5)
MILLM Training
Total
en
cn
Data Cost (%)
FLOPs
Full Data
79.1
63.0
47.8
68.4
58.2
86.4
1476.9
66. 58.9
67.9
100
10.35
Model-tree Methods
RANDOM
75.9
90.3
43.6
68.6
55.3
85.9
1461.0
60.3
53.3
64.5
95.1
SCURE
13.4
43,0
65.0
85.3
55.2
52.0
91.2
ELAN
53.0
47.4
PERPLEXITY
47.8
68.1
52.8
82.6
1341.4
91.6
SEMDEDUP
74.2
54.5
16,9
65.8
55.5
84.7
1376.9
52.2
48.5
70.0
92.6
D2-PRUNING 11
73.0
58.4
41.9
69:3
$1.8
85.7
1391.2
6d.7
57.6
63.9
94.8
SELF-SUP 30
74,9
39.5
46.0
67.8
49.3
83.5
1335.9
62.4
53.
63.3
93.4
Model-Invoreed Methods
SELF FILTER
13.7
58.3
53.2
61.4
52.9
83.8
1306
48.8
45.3
90.9
31.2E
17
48.6
69.8
53.3
58.7
11.7E
ICONS
1p
22.0
60.4
45.5
70.4
54.5
86.1
1447.7
64.6
54.0
66.9
97.1
300
12.0E
COINCIDE 21
76.5
59.3
46.8
69.2
55.6
86.1
14956
03.1
54.5
67.3
97.4
100
4.9E
ColDO (Ours)
77.2
60.4
47.1
69.4
55.6
81.4
14502
6388
56.7
70.1
98.2
20
4.2k
These methods need additional data train the target MLLM warm-up or reference set. reproduced
the relevant LoRA
results included in the paper. Reproduced with the optimal subset LLAVA-ICUUS-11 released ICONS within unified evaluation framework (scc an examination of the evaluation framework in Appendix [B3). corresponding to the original reports Rel. 98.6%.
QA: Sci- enceQA [49] 3) Optical Character Recognition (OCR): TextVQA [50]: 4) hallucination: POPE [5I]; 5) multiple-choies MME [52]. MMBench (33;; 6) free- form generation: LLaVA- -erncBBechh (In-the- Wild) [] More information about these benchmarks is provided in Appendix B.3 Since each evaluation benchmark has different scale, we compute average relative performance, denoted as Rel., across the ten benchmarks to assess the level of generalization. Relative performance on each benchmark is defined (subset model performance / full-data model performance) x 100%.
visual question answering: VQAv2 44), GQA 47. VizWiz 48 2)
4.1 Overall Performance and Efficiency Comparison
Metrics. We introduce key efficiency metric: MLLM Training Data Cost, which quantifies the proportion of data used to train the selection model relative to model fine-tuning. A 100% MLLM Training Data Cost implies no reduction in resource usage compared to full-data training. addition, we report total FLOPs, derived as the product of parameter volume and training sample number.
Baselines. [ the proposed ColDO two baseline categories: 1) model-free methods include random sampling and CLIP-Score 229]. which rely on simple criteria without resorting to the target MLLM. While computationally efficient (zero MLLM Training Data Cost and low FLOPs), their limited understanding of the data compromises effectiveness and transferability. 2) model- involved methods (20) 21 [9] require fine-tuning the target MILM during data selection, resulting in significant computational costs (usually 100% MLLM Training Data Cost). Gradient- based methods like TIVE and ICONS also require additional warm-up data (8% and 5%, respectively). with ICONS further relying on benchmark validation sets (2.2% of the total data) for gradient computations. While this provides high-quality, task-specific data, is often impractical in real- world scenarios. COINCIDE uses smaller MLLM TinyLLaVA-28 as the reference model, but the entire data traversal still incurs a high training cost compared to ours. See more details of these baselines in Appendix B.I
Performance & Efficiency Results. As shown in Table ColDO achieves Rel. of 98.2% while using only = 20 (%) of the data for framework training. Compared to model-free methods, ColDO provides significantly higher performance and robustness. Against model-involved methods, it matches the SOTA performance of ICONS but reduces MLLM Training Data Cost to just 20%. Unlike ICONS, CoIDO does not require additional validation sets, simplifying the process while ensuring optimal performance. Furthermore, ColDO achieves the lowest total computational cost, requiring only 4.2E (ExaFLOPPs) outperforming all other methods in efficiency.
4.2 Ablation Study and Design Choices
We perform ablation studies to confirm the design choice of CoIDO, focusing on coupled optimization strategies and the scorer architectures. See more ablation studies in Appendix C
8

---

Optimization Methods. ColDO employs coupled optimization strategy (Section 3.2 which jointly optimizes data importance and diversity via integrated learning objectives. assess its effectiveness, we test several optimization strategies, summarized in T As a baseline, we first adopt a conventional approach that optimizes the importance loss L alone during training, omitting the diversity loss LD. Diversity is added only at the selection stage via standalone procedure. This approach achieves the lowest Rel.=89%. We also experiment with alternative formulations, including simple summation of the two loss terms and weighted combinations using learnable coefficients (e.g.. and - A). Among all configurations, our homoscedastic uncertainty optimization proves most effective (the best in most cases, while performing rather close to the best measure).
Table 2: Ablations of optimization methods (the best in bold and the second-best underlined).
Loss Function
VQAv2
GQ4
Vizwiz
SQA-I
TextVQA
POPE
MME
MMBenchheen)
(MBBench((c)
LLAVA-B
Rel. (%)
C1
77.9
48.9
44,6
59.7
$2.5
86.2
1393.5
51.1
44.9
64.9
89.0
Ci Co
74.5
55.8
46.4
67.3
52.6
83.5
1339.7
57.0
50.9
62.3
92.0
AC, A)Cp
76.1
59.4
68.7
Ours
77.2
60.4
47.1
69.4
55.6
85.4
1450.2
63.8
36.7
70.1
98.2
Scorer Architecture. To explore alternatives to MLP-based CoIDO Scorer, we evaluate two additional designs: (1) Transformer-bused Scorer with two standard Transformer blocks, and (2) an attention-only Scorer using self-attention and cross-attention mechanisms applied directly to extracted features. The Transformer-basee variant combines stacked attention and feedforward layers for enhanced capacity, while the attention-only design explicitly captures intra- and inter- sample relationships. Figure compares their performance (Rel.) and training-tin taining-time FLOPs. The strong performance of the MLP-based scorer can be attributed to the expressiveness of the extracted multimodal features (e.g., CLIP Score and ImageReward), which already encode rich semantic alignment and quality cues, thereby reducing the need for additional modeling complexity.
100
100
98.2
Rel. (N)
96.1
$6.6
Ral. (%0
98.2
97.5
93
Seffibrr
1.50M
1.57M
TIxE
85
52
Attention
ColDO Scorer Structure
HLP
Transformrr
Training Ratio
70%
5%
10%
20%
(%)
40%
Figure 3: Ablations of different ColDO Scorer architectures.
Figure 4: Comparison of differ- ent training data ratios p%a.
Figure 5: Performance vs. selec-
tion ratios 7,
4.3 Parameter Sensitivity Study
We analyze 1) the training data ratio (p%) and 2) the selection ratio (7y defined in the task description) that denotes the proportion of data selected for training. See more parameter analyses in Appendix C
Training Data Ratio. ColDO enables effective data selection using a small fraction of the training data. To identify an optimal y%, we gradually increase p% starting from 5%. As shown in Figure performance is modest when < 10 (%), but improves clearly beyond this threshold. Notably, Rel.
stabilizes after p > 20 (%), indicating that 20% data is sufficient to capture the dataset distribution. Based on these findings, we set p = 20 (%) as the default for all experiments.
Selection Ratio. We evaluate ColDO and baselines across selection ratios (7) ranging from 5% 40%. Figure 5 shows that ColDO consistently outperforms the competitors across this range. However, as approaches 40%, the performance gap narrows since most high-quality samples in LLaVA-665X would be identified by all methods. When the selection ratio exceeds 50%, the impact of randomness increases, and even random selection strategy may yield strong results due to the inclusion of a large portion of the dataset (this shows a necessity of reducing visual instruction tuning data in practice). Hence, we focus on selection ratios between 5% and 40%, where differences in selection methods are most distinct and provide meaningful insights into method effectiveness. Notably, ColDO consistently achieves the highest Rel. across all selection ratios.

---

4.4 Generalizability and Transferability
In order to further explore the potential of COIDO, we conduct experiments about generalizability and transferability. We define generalizability as the ability of the proposed data selection frame- work to be directly applied to other models datasets. Transferability, whether a ColDO scorer trained on one domain can be reused to select informative data in another, out-of-domain corpus. To evaluate these two aspects, we conduct experiments the Vision-Flas dataset, a large-scale human-annotated visual instruction-tuning benchmark containing over 200 diverse vision- language tasks. Additional results other models and datasets (e.g.. LLaVA-13B and LLaVA-150K) are presented in Appendix C.4 and C.5 transferability are summarized in Table
the other hand, measures
Both results on generalizability and
Table 3: Performance of ColDO on the Vision-Flan dataset (20% data selection). & ColDO scorer
trained on -LaVA-6655 and applied to Vision Flan (out-of-domain transfer).
Model Setting
VQAv2
VizWiz
POPE
TextVQA
MMBench(en))
MMBenchlen)
LLaVA-B
Rel.
Full tune
74.5
47.1
52.8
61.8
46.4
85.7
1480.6
40.2
46.2
38.2
100.0
Random
74.6
44.3
50.0
59.8
40.9
81.3
14077.1
49.2
48.3
33.6
V7.8
ColDO
102.1
ColDO
75.7
46.8
53.3
66.2
42.1
85.5
1486.1
51.4
47.3
40.8
103.7
Generalizability. As shown in the first three rows of Table 3 ColDO consistently surpasses random selection and even slightly outperforms full fine-tuning under the same 20% budget, achieving relative score of 102. %. The gains are especially notable on reasoning-intensvve intensive and perception- heavy benchmarks such VizWiz and POPE, demonstrating that ColDO is capable of identifying semantically rich and instruction-relevan samples even within large human-curated datasets. These results confirm ColDO's strong generalizability which be applied across different datasets.
Transferability. To further assess knowledge transfer, we reuse the CoIDO scorer trained on LLaVA-665X directly for data selection Vision-Flas without retraining or domain adaptation. As indicated by the last row of Table 3. the scorer trained on LLaVA- 6655 achieved even better results Vision-Flas than the scorer trained directly on Vision-Fli itself. We believe this because LLaVA-6655 is a significantly larger and more diverse dataset, allowing the scorer to learn more generalizable notion of sample diffieulty and importance. This enables it to transfer effectively across domains, even outperforming in-distribution scorers trained on smaller datasets. This indicates that ColDO seamlessly be transferred new domain datasets with similar distributions, demonstrating its transferability.
5 Conclusion and Future work
In this paper, we present an efficient data selection framework for visual instruction tuning in multimodal large language models (MLLMs). By coupling the optimization of data importance and diversity, and employing lightweight scorer to learn data distribution from a small subset, our CoIDO framework achieves oftte--f-thear- performance while significantly reducing the amount of required training data and computational cost. This scalable and practical approach provides valuable contribution to improving efficiency MILM fine-tuning.
Future work includes exploring co-training with smaller, proxy MILMs, such as TinyLLaVA, to further reduce training costs and validating scalability on larger MLLMs like LLaVA-13B. Additional directions involve dynamic data selection for evolving datasets and training progress and extending the framework other multimodal scenarios, e.g., audio LLMs and video LLMs.
6
Acknowledgement
This work was supported by the Zhejiang Provincial Natural Science Foundation (No. LD24F020015), Pioneer R&D Program of Zhejiang (No. 2024C01021), and NSFC (Nos. U24A20254, 62471 1 168, 62572075,) ). Jinpeng Chen was additionally supported by BNSF (No. l.233034) and Fundamental Research Funds for BUPT (No. 2025TSQY01).
10

---

References
[1] J. Achiam, S. Adler, S. Agarwal. L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt,
S. Altman, S. Anadkat et al., "Gpt-4 technical report." arXiv preprint urXiv:2303.88774, 2023.
[2] G. Team, R. Anil, S. Borgeaud, J.-B. Alayrae, Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth, K. Mil- lican et al., "Gemini: a family of highly capable multimodal models," arXiv preprint arXiv:2312.11805 2023.
[3] H. Liu, Ci Q. Wu, and Lec, "Visual instruction tuning," Advances in neural information processing systems, vol. 36, pp. 34 892-892-88 34 3461,, 2023.
[4] S. Antol, A. Agrawal, J. Lu, M. Mitchell, D. Batra, C. L. Zitnick, and D. Parikh, "Vga: Visual question answering," in Proceedings of the IEEE international conference on computer vision, 2015, pp. 2425-2433.
[5] D. Hao. Q. Wang. L. Guo, J. Jiang, and J. Liu, "Self-bootstrapped risual-language model for knowledge selection and question answering," " in Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 2024, pp. 1857-1868.
[6] Q Wu, D. Teney, P. Wang. C. Shen, A. Dick, and A. Van Den Hengel, "Visual question answering: A survey of methods and datasets," Computer Vision and Image Understanding, vol. 163, pp. 21-40, 2017.
[7] M. Cao, S. Li, J. Li, L. Nic, and M. Zhang, "Image-text retrieval: A survey on recent rescarch and development" in Proceedings of the Thirty-First International Joint Conference Artificial Intelligence, IJCAI-22, 2022, pp. 5410-5417.
[8] L. Qu, M. Liu, J. Wu, Z. Gao, and L. Nie, "Dynamic modality interaction modeling for image-text retrieval"" in Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, 2021. pp. 1104-1113.
[9] Y. Cheng, X. Zhu, J. Qian, F. Wen, and P. Liu, "Cross-modal graph matching network for image-text retrieval," ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM), vol. 18, no. 4, pp. 1-23, 2022.
[10] J. Deng, 7. Yang. T. Chen, W. Zhou, and H. Li, "Transvg: End-to-end visual grounding with transformers," in Proceedings of the IEEF/CVF International Conference on Computer Vision, 2021, pp- 1769-1779.
[11] W. Wang. Y. Zhang, X. He, Y. Yan, Z. Zhao, X. Wang, and J. Liu, "Beyond literal descriptions: Under- standing and locating open-world objects aligned with human intentions," " in Findings of the Association for Computational Linguistics ACL 2024, 2024, pp. 762-776
[12] Y. Yan, X. He, S. Chen, and J. Liu, "Calibration & reconstruction: Deeply integrated language for referring image segmentation," in Proceedings of the 2024 International Conference on Multimedia Retrieval, 2024, pp. 451 451-459.
[13] H. Liu, C. Li, Y. Li, and J. Lee, "Improved baselines with visual instruction tuning." in Proceedings of the JEEF/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 26 296-26 306.
[14] C. Zhou, P. Liu, P. Xu, S. Iyer, Sun, Y. Mao, X. Ma, A. Efrat, Yu, I.. Yu et al., "LIMA: Less is more for alignment," Advances in Neural Information Processing Systems, vol. 36, pp. 1000050621... 2023.
[15] L. Chen, S. Li, Yan, H. Wang. K. Gunaratna, V. Yadav, Z. Tang, V. Srinivasan, T. Zhou, H. Huang, and H. Jin, "Alpagasus: Training a better alpaca with fewer data," in The Twelfth International Conference on Learning Representations, 2024.
[16] M. Li, Y. Zhang, Z. Li, J. Chen, L. Chen, N. Cheng, J. Wang, T. Zhou, and J. Xiao, "From quantity to quality: Boosting Um performance with self- gelf-guided data selection for instruction tuning," in Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), 2024, pp. 7595-7628.
[17] M. Xia, S. Malladi, S. Gururangan, Arora, and D. Chen, *LESS: Selecting influential data for targeted instruction tuning." in International Conference on Machine Learning (ICML). 2024.
[18] Z. Liu, K. Zhou, W. X. Zhao, D. Gao, Y. Li, and J.R. Wen, "Less is more: High-value data selection for visual instruction tuning," arXiv preprint arXiv:2403.099: 2024.
[19] X. Wu, M. Xia, R. Shao, Z. Deng, P. W. Koh, and O. Russakovsky, "Icons: Influence consensus for vision-language data selection." arXiv preprint arXiv:2501.0066 2024.
11

---

[20] R. Chen, Y. Wu, L. Chen, G. Liu, H. T. Xiong. C. Liu, J. Guo, and H. Huang, "Your vision-language model itself is strong filter: Towards high-quality instruction tuning with data selection." in Findings of the Association for Computational Linguistics ACL 2024, 2024, pp. 4156-4172.
[21] Lee, B. Li, and S. J. Hwang, "Concept-skill transferability-boe based data selection for large vision-language models," in Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 2024. pp. 5060-5080.
[22] A. Kendall, Y. Gal, and R. Cipolla, "Multi-task learning using uncertainty to weigh losses for scene geome- try and semantics," in Proceedings of the IEEE conference on computer vision and pattern recognition. 2018, pp. 7482-7491.
[23] H. Xu, S. Xie, P.Y. Huang, I., Yu, R. Howes, G. Ghosh, L. Zeutlemoyrr and C. Feichtenhofer, "Cit: Curation in training for effective vision-language data," in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 15 180-15 189.
[24] A. Mahmoud, M. Elhoushi, A. Abbas, Y. Yang. N. Ardalani, H. Leather, and A. S. Morcos, "Sieve: Multimodal dataset pruning using image captioning models," in Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, 2024, pp. 22 42323-22 4322.
[25] S. Y. Gadre, G. Ilharco, A. Fang, Hayase, G. Smymiss, T. Nguyen, R. Marten, M. Wortsman, D. Ghosh, J. Zhang et al. "Datacomp: In search of the next generation of multimodal datasets," Advances in Neural Information Processing Systems, vol. 36, pp. 27 092 27 112, 2023.
[26] W. Wang. K. Mrini, L. Yang, S. Kumar, Y. Tian, X. Yan, and H. Wang. "Finetuned multimodal language models are high-quality image-text data filters," arXiv preprint arXiv:2403.02677 2024.
[27] Y. Li, B. Hui, X. Xia, J. Yang, M. Yang, L.. Zhang, S. Si, I..H. Chen, J. Liu, T. Liu al., "One-shot learning as instruction data prospector for large language models," in Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Pupers), 20224, pp. 4586 4601.
[28] Y. Gc, Y. Liu, C. Hu, W. Meng, S. Tao, X. Zhao, M. Xia, Z. Li, B. Chen, H. Yang et al., "Clustering and ranking: Diversity-preserve instruction selection through expert-aligned quality estimation," in Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 2024, pp. 464-478.
[29] Hessel, A. Holtzman, M. Forbes, R. Le Bras, and Y. Choi, "Clipscore: reference- free evaluation metric for image captioning," in Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, 2021, pp. 7514-7528.
[30] A. K. M. Abbas, K. Tirumala, D. Simig, S. Ganguli, and A. S. Morcos, "Semdedup: Data-efficient learning at web-scale through semantic deduplication." in ICLR 2023 Workshop on Mathematical and Empirical Understanding of Foundation Models, 2023.
[31] A. Maharana, P. Vadav, and M. Bansal, "D22 pruning: Message passing for balancing diversity & difficulty in data pruning," in The Twelfth International Conference on Learning Representations, 2024.
[32] M. Paul, S. Ganguli, and G. K. Dziugaite, "Deep learning on data diet: Finding important examples carly in training," Advances in neural information processing systems, vol. 34, pp. 20 596-20 607, 2021.
[33] M. Marion, A. Ustün, I.. Pozzobon, A. Wang, M. Fadace, and S. Hooker, "When less is more: Investigating data pruning for pretraining Ilms scale," arXiv preprint arXiv:2309.04564 2023.
[34] J.BB Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y. Hasson, K. Lene, A. Mensch, Millican, M. Reynolds et al., "Flamingo: a visual language model for few-shot learning"" Advances in neural information processing systems, vol. 35, pp. 23 716-6233 2216-237386 736, 2022.
[35] J. Li. D. Li. S. Savarese, and S. Hoi, "Blip-2: Bootstrapping language imagee---maeeeeee pre- mre-training with frozen image encoders and large language models," in International conference machine learning. PMLR, 2023, pp. 19 730-19744 742.
[36] L. Wei, Z. Jiang, W. Huang, and L. Sun, "Instructiongpt-4: A 200-instruction paradigm for fine-tuning minigpt-4," arXiv preprint ar3iv:2308. /2067, 2023.
[37] D. Zhu, J. Chen, X. Shen, X. Li, and M. Elhoseiny, "Minigpt-4; Enhancing vision- languaee understanding with advanced large language models," in International Conference on Representation Learning, 2024, pp. 18 378-18 18 394.
12

---

[38] J. Xu, X. Liu, Y. Wu. Y. Tong, Q. Li. M. Ding, J. Tang, and Y. Dong, "Imagereward: Learning and evaluating human preferences for text-t-iimage generation." " Advances Neural Information Processing Systems, vol. 36, pp. 15 903-15935, 2023.
[39] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin,
J. Clark et al., "Leaarnin transferable visual models from natural language supervision," in International conference on machine learning. PMLR, 2021. pp. 5748-8763.
[40] A. Ng. M. Jordan, and Y. Weiss, "On spectral clustering: Analysis and algorithm." Advances in neural information processing systems, vol. 14, 2001.
[41] M. Paul, S. Ganguli, and G. K. Dziugaite, "Deep learning data diet: Finding important examples early in training," " Advances in neural information processing systems, vol. 34, pp. 20 596-22 607, 2021.
[42] Z. Zhang and T. Pfister. "Learning fast sample re-weighting without reward data," in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 725-734.
[43] A. E. Gelfand, "Gibbs sampling." " Journal of the American statistical Association, vol. 95, no. 452, pp. 1300-1304, 2000.
[44] K. Shum, M. Xu, J. Zhang, Z. Chen, S. Diao, H. Dong, J. Zhang, and M. Raza, "First: Teach a reliable large language model through efficient trustworthy distillation," in Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 2024, pp. 12 626-12 659.
[45] S. Ravfogel, Y. Goldberg, and J. Goldberger, "Conformal nucleus sampling," in Findings of the Association for Computational Linguistics; ACL 2023, 2023, pp. 27-34.
[46] Y. Goyal, Khot, D. Summers-Stap D. Batra, and D. Parikh, "Making the v in vqa matter: Flevating the role of image understanding in visual question answering," in Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, 2017, pp. 6325-6334.
[47] D. A. Hudson and C. D. Manning, "Gqa: A new dataset for real-world visual reasoning and compositional question answering," " in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 6693-6702
[48] D. Gurari, Q. Li, J. Stangl, A. Guo, C. Lin, K. Grauman, J. Luo, and J. P. Bigham, "Vizwiz grand challenge: Answering visual questions from blind people," in Proceedings of the JEEE/CVF Conference Computer Vision and Pattern Recognition, 2018, pp. 3608 3617.
[49] P. Lu, S. Mishra, T. Xia, L. Qiu, K.-W Chang, S.-C. Zhu, O. Tafjord, P. Clark, and A. Kalyan, "Learn to explain: Multimodal reasoning via thought chains for science question answering," in Advances in Neural Information Processing Systems, 2022, pp. 2507-2521.
[50] A. Singh. V. Natarajan, M. Shah, Y. Jiang, X. Chen. D. Batra, D. Parikh, and M. Rohrbach, "Towards vqa models that can read," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 8317-8326.
[51] L. Y. Du, K. Zhou, J. Wang. W. X. Zhao, andJ.RR, Wen, "Evaluating object hallucination in large vision- language models," " in Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 2023, pp. 292-305.
[52] C. Fu, P. Chen, Y. Shen, Y. Qin, M. Zhang. X. Lin, J. Yang. X. Zheng, K. Li, X. Sun et al. "Mme: A com- prehensive evaluation benchmark for multimodal large language models," arXiv preprint arXiv:2306. 13394, 2023.
[53] Y. Liu, H. Duan, Y. Zhang, B. Li, S. Zhang, W. Zhao, Y. Yuan, J. Wang. C. He, Z. Liu et al., "Mmbench: Is your multi-moda model an all- all-around player?" European conference on computer vision. Springer, 2024, pp. 216-233.
[54] A. Liu, B. Feng. B. Xue, Wang, B. Wu, C. La, C. Zhao, C. Deng, C. Zhang, C. Ruan al., "Deepscek-v3 technical report," arXiv preprini arXiv:2412. 2024.
[55] H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava.
S. Bhosale et al., "Llama Open foundation and fine-tuned chat models," arXiv preprint ariiv.2077.0002228 2023.
[56] E. J. Hu, Y. Shen, Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen, "Lora: Low-rank adaptation of large language models," in Proceedings of the 10th International Conference on Learning Representations 2022.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We clearly state our contributions and scope in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations in the appendix.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
14

---

Justification:
Each theorem comes with a clear statement of the assumptions and proofs are provided in the methodology section.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
The details and hyperparameters of the experiments are provided in the appendix.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
15

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: All the data used are open open-sourced. Our code is available in the anonymous github repository.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
The training and evaluation details are provided in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
The high computational cost of supervised fine-tuning multimodal large language models precludes multiple experimental runs necessary for reporting statistical sig- nificance. In the experiments, use consistent experimental setups across all experiments for fairness.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
16

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The hardware we use is reported in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics attpps//nneut http::pp.p..pppppi/iiiiciiiaaiaa
Answer: [Yes]
Justification: We have reviewed the NeurlPS Code Ethics and confirm that our research conforms it.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We have such discussions in the introduction and the conclusion, well as an explicit section the appendix.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
17

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This paper does not introduce new models or datasets. However, we propose novel data selection framework for visual instruction tuning.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging. and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We use open-source models and data, which have been properly cited.
Guidelines:
The answer NA means that the paper does
existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
18

---

If assets are released, the license, copyright information, and terms of use in the thcode has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssit
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
The code is well documented.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
We do not have such experiments.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification:
We do have this type of studies.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
19

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
We have reviewed the NeurlPS 2025 Policy on the Use of Large Language Models and confirm that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Justification:
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non- non-standard components.
Please refer to our LLM policy for what should or should be described.
20

---

A More Technical Details
A.1 Details in Feature Extraction and Spectral Clustering
Feature Extraction. Before our training process, we derive four categories of basic features for each training data sample. These features are intended to encapsulate the quality of images, text, and the alignment of image-text. A comprehensive overview of these basic features is provided in Table 4
while the exact prompt employed our LLM-based evaluator is depicted in Figure 6
Table 4: Explanations of our basic features extracted before the training process.
Indicator
Explanation
Image-Reet Reward Score
This score from the ImageReward model [88] which assesses the quality of images generated from text prompts by aligning them with human preferences, focusing on aspects such as text-image alignment, visual fidelity, and overall aesthetic appeal.
LLM Score
The LLM Score represents a robust measure employed by a language model to assess the response's 's quality, specifically in terms of grammar, spelling, and fluency. This score indicates how well the generated caption aligns with the model's language capabilities. In our approach, we utilize DeepSeek- V3 [54] for text evaluation, with the prompts illustrated in Figure 6
CLIP Features
Vision-language features low-dimensional space obtained by encoding images with ViT from CLIP 39 and text with Llama2 SS followed by conducting unsupervised dimensionality reduction.
CLIP Score
29
This score the Cosine similarity between the embedding of an image and that of its corresponding text. This metric serves as an indicator of the semantic alignment between visual and textual modalitics. It quantifies how well the caption represents the visual content and used to assess coherence between image-text pairs.
deepssek
System Prompt:
We would like request your feedback the performance of Al assistant in response to the user's questions the conversation displayed following.
Conversation: [text]
User Prompt:
Please rate according to the content of the responses to the questions. The assistant should receive a score on scale of to 10, where higher score indicates higher level of the content. Please first output a single line containing the value indicating the scores. the subsequent line, please provide comprehensive explanation your evaluation, avoiding any potential bias.
Figure The prompt analogous to that used ALPAGASUS to evaluate the quality of the text.
15
We employ DeepSeek- V/3
Spectral Clustering. We employ spectral clustering to group training data samples based on their basic features. The process begins by constructing a similarity matrix from the input basic features, which are illustrated in Table 7 We utilize Cosine similarity to quantify the pairwise relationships between data samples. To focus on the most relevant connections and manage computational complexity, K --eerrett Neighbors (KNN) graph is then built upon this similarity matrix, effectively
21

---

creating a sparse representation of the data's affinity structure. The spectral clustering algorithm subsequently applied to this graph, partitioning the data into a predefined number of clusters by analyzing the spectrum (eigenvalues) of the graph Laplacian. This method allows for the identification of clusters based on connectivity rather than solely relying on compactness.
Compared to K K-means, spectral clustering offers distinct advantages for analyzing complex image- text datasets. Firstly, spectral clustering is capable of identifying clusters with non-convex shapes, which K-means often struggles with due to its assumption of spherical cluster geometry. The rich semantic relationships in image-text data often manifest as intricate structures in the feature space that are better captured by spectral methods. Secondly, by operating clustering can reveal underlying manifold structures within the data. This is particularly beneficial for image-text embeddings, where meaningful relationships might not be best represented by simple Fuclidean distances to cluster centroids, but rather by the connectivity patterns between samples. This allows for a more nuanced understanding of how different image-text pairs relate to one another, leading to potentially more coherent and semantically meaningful groupings.
graph representation, spectral
A.2 Pseudocode of the Framework Training
We provide detailed pseudocode for our ColDO framework in Algorithm I
Algorithm 1 ColDO: Training the ColDO Scorer
Require:
Instruction dataset D, MLLM Scorer, sampling ratio p%, clusters M, batch size s 1: Randomly sample D, C DD |Dr +
p%[[D|
2: Extract basic features x, for each x, in D,
3: Cluster D, into M groups using spectral clustering
4: for each training step do
5: Sample minibatch B < {zy}}=1 Dr
6: for each ø; B do
7:
Compute prediction úU t f2xz)
8:
Compute loss Cj= & CE(Uj. Wú
9:
Compute weight w + Scorer(x,) (x;;
10:
end for
11:
Compute L1 - w
12:
for cluster C; in this minibatch, i i= = 1.2....., m do
Compute average weight wi +- Mean(wjj By C:) end for
14:
15:
Compute diversity loss LD +- Var(w wn)
16:
Compute total loss Listal 07 J + 20 LD + log or + log
17:
Update f, Scorer, and σ₁, DD using Lotal
18:
end for
B More Experimental Setting Details
B.1 Baseline Description
Model-free methods select training data without relying the training inference of large multimodal language models (MLLMs). Instead, they utilize static heuristics or precomputed metrics to evaluate sample quality or diversity. Due to the absence of target MLLM, the results of such methods are usually not satisfactory. These baselines including
Model-free Methods.
RANDOM SAMPLING: This baseline method involves randomly selecting samples from the dataset without any specific criteria. It serves reference point to assess the effectiveness more sophisticated selection strategies.
CLIP-SCORE 29
Utilizes the Cosine similarity between image and text embeddings generated by the CLIP model to measure the semantic alignment of image-text pairs. Higher scores indicate better alignment.
22

---

Calculates the L2 norm of the loss for each sample during the initial epochs of training. Samples with higher EL2N scores are considered more informative, as they are harder to learn early and are thus prioritized for training.
EL2N [3]
Employs language model to compute the perplexity of textual responses. Lower perplexity scores suggest that the text is more predictable and, by extension, of higher quality.
PERPLEXITY [33]
SEMDEDUP 30) Implements semantic deduplication by identifying and removing semanti- cally similar or duplicate samples from the dataset, thereby enhancing diversity and reducing redundancy.
Model-involved Methods. Model-involved methods incorporate the training of MLLMs into the data selection process. These approaches often involve tne-ttuing the MLLM, estimating gradient-base influence, or leveraging internal representations of MLLMs to evaluate the utility of each sample. Although these methods are typically more computationally expensive, they tend to outperform model-free methods by capturing more informative signals about task relevance, data quality, and alignment with downstream objectives. These baselines including
Trains a scoring network alongside the MILM to evaluate the quality of instruction samples. By filtering out low-quality data based on model feedback, it aims to enhance the overall training efficiency and performance.
SELF-FILTER 20
Estimates the value of each task- instancee pair by analyzing gradient- gradient-based influence scores. This method selects samples that expected to have the most significant impact on model performance, thereby reducing redundancy in the training data.
TIVE 17
(Influence Consensus): Aggregates influence scores across multiple tasks to identify samples that consistently contribute positively to performance. By focusing on these consensus samples, ICONS facilitates efficient multi-task learning with a compact dataset.
ICONS [99
COINCIDE 22 Combines multimodal features and task relevance to select representative samples that capture the essential characteristics of vision-language data. This approach aims to improve the model's generalization and multi-task learning capabilities.
B.2 Training Configuration of Our Method
We implement our method based on the LLaVA-1.5-78 architecture with LoRA tuning 56 All experiments are conducted using eight A5000 GPUs with 24GB of memory. The fine-tuning process utilizes LoRA (56) across 2 epochs, with each GPU processing batch size of 8 and learning rate set at 2e-5. The Adam optimizer is applied, featuring no weight decay, alongside Cosine learning rate schedule and warmup ratio of 3%.
Following LLavA 3) we use CLIP-ViT-L-336px for visual embedding and Vicuna-7B for text encoding. Spectral clustering is performed on normalized features with the number of clusters set to M 20. Two learnable uncertainty parameters. di and oD. are initialized to O. The hidden layer size of our MLP is specified as 1536 for handling CLIP features and 3 for combining LLM Score, CLIP Score, and Image-Reward Reward Score. Consistent with the original paper, we applied the same fine-tuning settings to the LLaVA-1.5-1 1.BBBB model for both LoRA and full fine-tuning.
B.3 Evaluation Setup
Benchmarks. Our MLLM undergoes evaluation through ten diverse benchmarks designed to assess range of capabilities, such visual question answering, reasoning, robustness, and following instructions. A detailed overview of these benchmarks is provided in Table 5
Evaluation Framework. In all experiments, evaluate all benchmarks in strict accordance with the official guidelines and procedures of LLaVA-1.5-74 [33. presenss the relative correctness of our unified evaluation framework. VA- VA-Bench was initially evaluated using apt-3.5-turbb- 0613. However, this model has since been deprecated by OpenAI. Based on discussions within the community and comprehensive consideration, switched to using gpt-4o-mini for evaluation.
"The related discussion http://////gggiiiioooooo/oaaataa htt.a////mmmmrraaa/sseee/////
23

---

Table 5: Descriptions of benchmarks used evaluate the MLLM performance.
Benchmark
Explanation
VQAv2
widely-used benchmark for visual question answering, consisting of natural images paired with open-ended questions that require understand- ing visual content to produce correct answers.
GQA 47
This benchmark evaluates models on questions that require reasoning about real-world images, emphasizing logical inference and relational reasoning capabilities.
VizWiz 48
A visual question answering benchmark specifically designed for visually impaired users, containing challenging real-world images with questions often needing practical reasoning about ambiguous or incomplete visual content.
SQA-1 49
Situated Question Answering on Images, benchmark designed to as- sess how well models can handle questions requiring situational and contextual visual understanding.
Tex 50
A benchmark for evaluating model capability in answering questions requiring reading and reasoning over textual information presented in images, such as signs or printed text.
POPE 51
visual understanding benchmark focused on identifying the presence or absence specific objects or entities in images, emphasizing accuracy and robustness in object recognition.
MME S2
Multimodal Evaluation, an aggregate benchmark assesses compre- hensive multimodal understanding by combining various tasks including reasoning, captioning, and classification across multiple modalities.
MMBench (en/cn) 33
A multimodal benchmark assessing the performance of models in an- swering complex multimodal questions in both English (en) and Chinese (cn), highlighting multilingual and multimodal understanding capabili- lies.
LLaVA-Bench
tailored benchmark for evaluating multimodal large language models (MLLMs), focusing on complex reasoning and understanding capabili- ties from multimodal data, particularly image-text pairs.
This change in evaluation framework is the primary reason for the notable difference observed in the metrics presented in T
To ensure the validity of our evaluation pipeline, we first verified that our unified evaluation framework accurately reproduces the official results of LLaVA-115-78 on all benchmarks. As shown Table the reproduced scores are consistently aligned with the original reports, despite the shift from ppt-3.5-ttrbb-00 to gpt-4o-mini as the evaluation backend.
However, when reproducing the ICONS baseline, we observed a notable discrepancy: our evaluation yields relative performance 97. 17.1%, while the original ICONS paper reports 98.6%. We emphasize that this difference does not stem from issues in our framework. Our result is obtained using the officially released subset (LLAVA-ICONS-1) scoring protocol applied to all methods. Given the consistency of our framework across all other benchmarks and baselines, we attribute this minor drop to changes in evaluation prompts, model versions, or post-processing details ICONS's side that were not fully disclosed. Therefore, this discrepancy should not be interpreted as flaw in our implementation, but rather as natural outcome of aligning all methods within consistent and transparent evaluation setup.
provided by ICONS and evaluated using the same
24

---

Table 6: Validation of the evaluation procedure.
Evaluation Setting
VQAzZ
GQA
Vizwiz
SQA-
TeatVQA
POPE
MMK
MMBenchi(nn)
MMBenchicn))
LLaYA-B
Rel. (%)
Original Report
63.0
Our Reproduction
79.1
62.9
48.4
68.5
58.2
86.4
1478.1
66.6
58.9
69.4
100.4
Difference (A)
0.0
-0.1
40.6
40.2
00
0.
+1.2
40.5
0.0
+1.5
40.4
Table 7: Feature components used in ColDO Scorer training.
Composition
VQAv2
GQA
Virwix
SQ4-I
TeatQQ
MMBeencheen)
MMBBencheen)
L.AVA.B
(%)
Only CLIP Features
76.5
59.0
51.1
63.6
55.1
82.9
1423,8
60.1
53.1
06.1
96.3
Only Scvres
77.0
59.6
47.4
66.1
$4.7
81.4
1.46.99
61.9
55.5
68.1
96.1
All (Ours
77.2
60.4
47.1
694
5566
85.4
1430.2
638
56.7
70.1
98.2
Table 8:
The number of clusters M (M is set to 20 by default in our main experiments).
#Clusters
VQAr2
GQA
Vizwiz
SQA-I
TexVQQ
POPR
MME
MVBBenchien)
MMBench((n)
LLAVA-B
Rel. (%)
74.3
47.1
64.7
52.1
85.9
1398.4
53.4
48.8
66.2
92.2
74.0
58.3
49.8
68.1
51.7
87.5
1415.8
59.3
49.7
62.6
94.2
20
71.2
60.4
47.1
69.4
55.6
85.4
1450.2
63.8
56.7
70.1
98.2
Table Application on full fine-tuning.
Method
VQAv2
GQA
Vizwiz
SQA-I
TextVQA
POPE
MME
MMBench(en)
MMBench((n))
LLAVA-B
Ref. (%)
Full Data
78.5
62.0
50.0
66.8
58.2
85.9
1510.7
64.3
58.3
65.4
100.0
RANDOM
74.5
56.5
48.5
66.5
54.9
83.1
1364.1
58.6
52.1
67.5
94.7
Ours
75.9
57.8
47.4
67.8
54.6
81.1
1402.9
60.6
55.0
68.8
96.1
C More Experimental Results
C.1 Feature Components
To ascertain the features employed in the training CoIDO, an ablation study is performed focusing on the components of features utilized during this process. For this analysis, we employ only the CLIP Features or the three scores (LLM Score, CLIP Score, and Image-Reward Score). The findings are presented in Table 7 These results indicate that the omission of any feature results in decline in overall performance.
C.2 Number of Clusters
The cluster number M is a crucial parameter in our framework, as the computation of diversity loss Ln for each batch depends on the number clusters included that batch (m, m S M). When M substantially exceeds the batch size, it is highly probable that the samples in each batch originate from a limited subset of the clusters. Consequently, each time the diversity loss is evaluated, certain clusters may be excluded. Conversely, if M is significantly smaller than the batch size, it is highly probable that samples from all clusters can be represented in each batch. However, choosing a too small for M may, in turn, compromise the optimization of diversity. To choose the appropriate M, we experiment on different M settings from 5 to 40, the results are reported in Table
C.3 Application on Full Fine-tuning
Table presents a comparative analysis of LLaVA-1.5-7B-LOFF
performance. To assess the efficacy of supervised fine-tuning across the entire L.VVA-1.5-7B we evaluated our selected subset, as detailed in Table The findings indicate that, even with the direct fine-tuning of LLaVA, ColDO remains effective, thereby confirming its generalizability.
C.4 Application on Larger Model
Table 10 reports the evaluation results on the larger-scale LLaVA-13B-LORA model. To further examine the scalability and generalizability of ColDO, we apply the same data selection pipeline used for the 7B model to the 13B counterpart without any modification. The results demonstrate
25

---

that CoIDO consistently outperforms existing baselines and maintains comparable performance to full-data fine-tuning under the same 20% budget. This confirms that CoIDO generalizes effectively to larger model capacities, validating its robustness and adaptability across different parameter scales.
Table 10: Evaluation results the LLaVA-13B-LORAA model under 20% data selection ratio.
Method
VQAv2
GQA
VizWix
SQA-1
TextVQA
POPE
MME
MMBench(en)
MMBenchhen))
LLaVA-B
Rel. (%)
100.0
Random
76.7
60.5
48.0
68.8
57.7
84.8
1484.9
62.8
55.2
68.6
96,0
ICONS
77.9
60.5
47.7
74.0
57.3
87.4
1503.9
65,5
59.2
65.3
95.7
COINCIDE
77.8
60.4
516
70.0
58.6
$7.1
1516.8
64,0
57.7
67.4
95,9
ColDO (Ours)
51.1
58.9
1586.4
37.5
C.5
Application on Smaller Dataset
Table 11 presents the evaluation results on the smaller-scale LLaVA-150K dataset. To further assess the generalizability of ColDO under limited data regimes, we directly apply our selection strategy without re-tuning any hyperparameters. As shown in [T] ColDO consistently achieves superior or comparable performance to full-data fine-tuning while using only 20% the data, demonstrating its robustness and adaptability in low-resource scenarios.
Notably, the accuracies of GQA and Tex are consistently below 1% across all methods (close to random guessing), and thus omitted from the table. This behavior is expected because LLaVA-150K contains very few or no samples related to compositional reasoning (GQA) or OCR-centric tasks (TextVQA), rendering these benchmarks unreliable in this setting. Interestingly, for MMBench and MMBench(en), both random selection and ColDO obtain higher scores than full-data fine-tuning, which we attribute to the presence of noisy or low-quality samples in the complete dataset- fine-tuning helps mitigate overfitting and improves generalization to evaluation benchmarks.
Table 11: Performance of ColDO on the LLaVA- 150K dataset.
Model
VQAv2
VizWiz
SQA
POPE
MME
MMBench(enn
MMBench(en)
LLaVA-B
Rel.(%)
Full Fine-tune
55.2
45.5
57.6
57.9
1234.5
22.4
27.1
65.9
100.0
Random
50.0
44.8
53.4
54.2
1184.7
30.7
30.1
62.4
101.7
CoIDO (Ours)
49.5
47.1
58.2
56.1
1214.8
29.8
32.3
63.8
104.8
C.6
Detailed Results for Main Paper Experiments
This section presents the complete per-benchmark evaluation results, as detailed in Tables 12 and 14 corresponding to Figures and 5 in the main paper, respectively.
C.7
Visual Analyses
To visually analyze the effectiveness of the scores outputed by the ColDO Scorer, we conduct case study with results presented in Figure 7and Figure
D
Limitations
While ColDO demonstrates strong performance in data-efficient instruction tuning, several limita- tions remain. First, our approach relies four handcrafted data features represent each sample. Although the computation for these features is lightweight, this dependency may limit extensibility or introduce engineering overhead. Future work will explore whether an effective scorer can be learned from fewer or even latent representations to further streamline the pipeline.
Second, our experiments currently limited LLaVA-1.5-78 The effectiveness of ColDO on larger models, such as LLaVA-13B or other recent MLLMs, is yet to be verified. Extending the evaluation to larger-scale models is crucial for assessing the scalability and generalizability of our approach. This will require significantly greater computational resources, but it is foreseeable that this task can be accomplished in near future.
Third, although ColDO achieves high selection quality with significantly reduced computational cost, COINCIDE 21 also offers low FLOPs by leveraging smaller reference model (TinyLLaVA-22))
26

---

Table 12: Detailed results for different CoIDO Scorer architectures (MLP by default in our design).
ColDO Structure
VQAv2
GQA
Vizwiz
SQA-1
TextVQA
POPE
MME
MMBenchlenn)
MMBenncheen))
LLAVA-B
Rel. (%)
Attention
Transformer
77.0
59.7
48.7
68.1
53.9
83.5
1439.7
63.4
61.2
96.6
MLP
77.2
60.4
47.1
69.4
55.6
85.4
14502
63.8
36.7
70.1
98.2
Table 13: Detailed results for different training data ratios (20% by default our design).
Ratio
VQAv2
GQA
Vizwiz
SQA-1
TexIVQA
POPE
MME
MMBench((nn)
MMBench(en)
LLAYA-B
Rel. (%)
76.1
58.4
41.8
67.4
54.6
86.1
[449.
61.9
53.9
65.4
94.8
10%
77.0
59.4
46.4
66.8
54.2
83.3
1499.4
61.9
55.7
66.9
96.4
15%
76.9
59.8
51.6
67.0
54.7
85.6
1438.6
61.1
55.0
64.1
96.9
20%
77.2
60.4
47.1
69.4
55.6
85.4
1450.2
63.8
56.7
70.1
98.2
25%
77.1
60.2
48.2
67.9
55.2
84.7
1438.9
63.4
56.5
68.0
97.5
Table 14: Detailed results for different selection ratios (20% by default in design).
Ratio
VQAv2
GQA
Vizwiz
SQA-I
TextVQa
POPE
MMF
MMBßench(en)
MMBenchienn)
LLAVA-B
Rel. (%)
73.5
55.1
42.5
68.1
51.2
1371.7
58.8
54.1
60.8
91.8
10%
75.2
58.0
42.2
68.8
54.1
83.8
[440.3
60.0
54.6
68.8
94,8
40%
78.2
61.6
47.7
69.1
56.6
85.5
[484.1
62,5
56.1
67.3
98.3
This highlights promising direction for future research replacing the target MLLM with small proxy model during scoring to further reduce training cost without compromising selection performance. This capability can be readily incorporated into ColDO with additional development effort.
E Broader Impact
From broader impact perspective, ColDO promotes efficient fine-tuning by selecting compact, diverse, and informative subsets, which may reduce energy consumption and democratize access to instruction tuning for resource-constrained users. However, data selection algorithms inherently influence model behavior. Care must be taken to ensure faimess, representation, and robustness, especially when deploying models downstream in sensitive high-stakes applications.
27

---

What the stuffed animal depicted the image?
What is the cat doing in the image?
The stuffed animal in the image is a stuffed monkey.
The cat is peeking out from underneath white conforter blanket, which is likely top bed
What type of computers present the desk?
What color are the blankets in the image?
There two types of computers present the desk: a laptop computer and desktop computer, which includes a monitor keyboard. There is also the desk.
blankets image white
What color the cat in the image?
The the image grey and white.
What is the position of the stuffed monkey on the desk?
Can you infer the cat's maad or behavior from the image?
The stuffed monkey is leaning against the laptop computer, with both the monkey and laptop situated on the desk.
Based the image, cat appears to be in curious or playful mood, as it is peeking out from under the white blankets. Cats often engage in such behavior when they feel comfortable and secure in their environment. They might be trying to play game of hide-and-seet
Can you describe the overall setup of the desk?
desk setup consists of laptop computer and desktop computer with monitor, keyboard, and mouse. The laptop computer appears to be the primary workstation,
the desktop computer secondary device. This arrangement allows
Figure 7: The best (left) and the worst (right) ColDO scores samples from LLaVA- -133K datasets.
28

---

What type of clothing is black?
what is the pattern of the shirt?
Jacket
Striped
the candles behind the wine bottles in the bowl?
Do you see any people to the left of the man?
Yes
No
what is the white clothing item in this photo?
who is walking?
Dress
Lady
Are there any umbrellas to the left of the people that walk on the sidewalk?
What is in front of the candles the image?
Wine
bottles
On which side of the image is the large bowl?
what is the lady wearing?
Left
Shirt
What is the little child standing near? Apppe c.c C.water babasket
What is the purpase of the gray object? heaeat B.pest prevention Cooo D.play music
C
A
Figure 8: The best (top left) and the worst (top right) ColDO scores samples from GQA datasets. The best (bottom left) and the worst (bottom right) ColDO scores samples from VQAv2 datasets.
29


Reviews:
[{"review_id": "AbyhxVjL1C", "reviewer": "Reviewer_oB3E", "summary": "This work presents COIDO, an innovative framework aimed at improving the training efficiency of multimodal large language models . It tackles the computational challenges of instruction tuning by simultaneously optimizing for data importance and diversity. A key innovation is the introduction of Diversity Loss and Important Loss, which helps balance these two aspects. COIDO uses a lightweight scorer trained on a small random subset of data to learn the distribution, reducing computational demands. Through a homoscedastic uncertainty-based formulation, it effectively balances importance and diversity without needing specialized algorithms for data selection. Experimental results demonstrate that COIDO achieves near-full-data performance with only a fraction (20%) of the data, outperforming existing methods in both efficiency and accuracy.", "questions": "As showned in the Weakness.", "limitations": "As showned in the Weakness.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. The framework's code is available.\n2. Present Diversity Loss and Important Loss.\n\nWeaknesses：\n1. Diversity Loss doesn't make sense and needs further explanation, especially on how to assess data diversity with a one pass.\n2. The experiments should be conducted across more models to verify the effectiveness.", "was_revised": true, "final_justification": "After reading explanations, I believe the quality of this work has been significantly improved and has resolved some of the key issues. Therefore, I have updated my recommendation to \"Weak Accept\".", "rebuttal": "**Q1: Explanation and Implementation of the Diversity Loss.** \n\n**A1:** Thank you for raising this important question. Below we explain why the Diversity Loss is well-motivated and easy to implement, from four complementary angles: **(1) intuitive motivation**, **(2) theoretical justification**, **(3) implementation**, and **(4) empirical evidence**.\n\n**(1) Intuitive Motivation**. If we optimize only the importance loss $\\mathcal{L}_I$, the learned CoIDO scores $w$ (lower is better) can become highly skewed across the $M$ semantic clusters obtained by spectral clustering.  Specifically, samples in a few clusters (e.g., the “blue” cluster in Fig.2) may all receive very low scores, while samples in other clusters (e.g., the “red” and “green” clusters) receive uniformly higher scores.  When we later select the top $\\gamma$ (e.g., $\\gamma = 20\\\\%$) of samples by ranking $w$, almost all selected items may come from those few favored clusters, leading to poor coverage and low diversity.  \n\nOur diversity loss $\\mathcal{L}_D$ explicitly counteracts this imbalance at the *cluster level*.  Let $\\bar{w}_i$ denote the mean CoIDO score of cluster $i$ within a minibatch.  By minimizing the variance of $\\{\\bar{w}_1, \\ldots, \\bar{w}_m\\}$, defined as: $\\mathcal{L}_D = \\mathrm{Var}\\big(\\{\\bar{w}_1,\\ldots,\\bar{w}_m\\}\\big)$,\nwe shrink the inter-cluster gaps: clusters whose samples all have very low scores are gently pushed upward, while clusters with uniformly high scores are pulled downward.  \n\nCrucially, $\\mathcal{L}_D$ does not alter the relative ordering within a cluster; it only balances the average scores across clusters.  \nThus, when we finally rank by $w$ and select the top $\\gamma\\%$, each cluster still contributes its best (lowest-scored) samples, achieving both high importance and balanced diversity.\n\n**(2) Theoretical Justification**. Let $S = \\sum_{j=1}^{m} \\bar{w}_j$, and normalize $\\bar{w}_i$ to $p_i = \\frac{\\bar{w}_i}{S},\\quad u_i = \\frac{1}{m}$. \nSubstituting $\\bar{w}_i = S p_i$ and $\\mu = S/m$ into $\\mathcal{L}_D$, we obtain $\\mathcal{L}_D=\\frac{S^2}{m} \\lVert \\mathbf{p} - \\mathbf{u} \\rVert_2^2 $.\nThus minimizing $\\mathcal{L}_D$ is (up to a constant factor) minimizing the squared ${\\ell}_2$ distance between the cluster-level distribution $\\mathbf{p}$ and the uniform distribution $\\mathbf{u}$, explicitly encouraging balanced coverage. This makes CoIDO Scorer tend to choose samples in a more balanced and diverse way.\n\nWe further note that the Pearson $\\chi^2$ divergence is given by:\n$$\n\\chi^2(\\mathbf{p} \\| \\mathbf{u}) = \\sum_{i=1}^{m} \\frac{(p_i - u_i)^2}{u_i} = m \\lVert \\mathbf{p} - \\mathbf{u} \\rVert_2^2.\n$$\nHence, $\\mathcal{L}_D$ is proportional to $\\chi^2(\\mathbf{p} \\| \\mathbf{u})$, connecting our variance-based loss to a well-established divergence. This equivalence reveals that our diversity loss $\\mathcal{L}_D$ does not merely reduce variance. It explicitly penalizes deviation from a uniform distribution over clusters, a well-founded strategy in statistical learning. Since the Pearson $\\chi^2$ divergence is a widely used f-divergence that measures how far is one distribution from another, minimizing it encourages balanced representation across semantic partitions, preventing over-selection from a few clusters.\n\n**(3) Implementation**. In our *one-pass optimization*, cluster assignments are calculated prior to the CoIDO optimization. Random minibatch sampling makes each $\\bar{w}_i$ an unbiased estimate of its true cluster mean.  Thus, $\\mathcal{L}_D$ provides a valid stochastic estimator of the inter-cluster variance. It is efficient ($\\mathcal{O}(m)$ per batch), fully differentiable, and easily integrated into standard forward/backward passes with no additional clustering or pairwise computations.\n\n**(4) Empirical Evidence**. In our paper, Table 2 (Sec. 4.2) shows that removing the diversity term and optimizing only $\\mathcal{L}_I$ degrades the average relative performance to 89.0%, the worst among the tested variants. Simply adding $\\mathcal{L}_D$  without proper coupling improves to 92.0%, and a naive weighted sum reaches 95.9%. Our final homoscedastic-uncertainty formulation achieves 98.2%, essentially matching full-data performance while keeping cost low. These ablations demonstrate both the necessity of having a diversity term and the effectiveness of our particular integration.\n\n**Q2: More experimental results.**\n\n**A2:** We thank the reviewer for the valuable suggestion. To verify the robustness and generality of our method, we provide additional results from two complementary perspectives:\n\n**(a) Across More Models**. We apply our method to a larger and stronger model **LLaVA-13B-LoRA** to examine its generalization capability. As shown in Table 2, our method consistently improves performance across nearly all benchmarks compared to both the Random baseline and the SOTA selection methods (ICONS and COINCIDE). Notably, our gains are significant on reasoning-heavy tasks such as GQA, SQA-I, and MME, demonstrating our method’s ability to scale with model size while enhancing both perception and cognition.\n\n**Table 1: Evaluation results on LLaVA-13B-LoRA.**\n\n| **Method**     | **VQAv2** | **GQA** | **VizWiz** | **SQA-I** | **TextVQA** | **POPE** | **MME**  | **MMB** | **MMB(cn)** | **LLaVA-WB** | **Rel. (%)** |\n|----------------|----------:|--------:|-----------:|----------:|------------:|---------:|---------:|--------:|------------:|-------------:|-------------:|\n| Full-Finetune  |   80.0    |  63.3   |    58.9    |   71.2    |    60.2     |   86.7   |  1541.7  |  68.5   |     61.5    |     69.5     |     100      |\n| Random         |   76.7    |  60.5   |    48.0    |   68.8    |    57.7     |   84.8   |  1484.9  |  62.8   |     55.2    |   68.6   |     94.0     |\n| ICONS|**77.9**|60.5|47.7|74.0|57.3|**87.4** |1503.9|**65.5**|**59.2**|65.3|95.7|\n| COINCIDE       |**77.8**    |  60.4   |  **51.6**  |   70.0    |    58.6     | **87.1** |  1516.8  |  64.0   |   **57.7**  |     67.4     |     95.9 |\n| **Ours**       |   77.5    | **61.5**|    51.1    | **74.2**  |  **58.9**   |   85.8   | **1586.4**| **64.7**|     57.5    |     **69.7**     |   **97.2**   |\n\n**(b) Across More Fine-tuning Datasets**\nWe further test our method under different data availability conditions:\n\n``LLaVA-150K``, a much smaller subset of LLaVA pretraining data (about 1/5 the size), often used for budget-limited fine-tuning.\n\n``Vision-Flan``, the largest human-annotated visual instruction tuning dataset to date (186k images), covering 200+ diverse vision-language tasks.\n\nWe provide representative evaluations using the same benchmarks to simulate small-data and diverse-data scenarios. Results are shown in Table 2 and Table 3.\n\n**Table 2: Performance on LLaVA-150K dataset (smaller fine-tuning data).**\n| **Model**         | **VQAv2** | **VizWiz** | **SQA** | **POPE** | **MME**  | **MMB** | **MMB(cn)** | **LLaVA-WB** | **Rel. (%)** |\n|-------------------|----------:|-----------:|--------:|---------:|---------:|--------:|-----------:|-----------------:|-------------:|\n| Full Fine-tune    |   55.2    |    45.5    |  57.6   |   57.9   | 1234.5   |  22.4   |     27.1    |        65.9      |     100      |\n| Random   |   **50.0**    |    44.8    |  53.4   |   54.2   | 1184.7   |  **30.7**   |     30.1    |        62.4      |     101.7    |\n| Ours |   49.5    |    **47.1**    |  **58.2**  |   **56.1**   | **1214.8**   |  29.8   |     **32.3**    |      **63.8**     |     **104.8**  |\n\nIn Table 2, we omit the results of GQA and TextVQA because their accuracies are consistently below 1% (close to random guessing) under both full fine-tuning and data selection methods. This is expected since LLaVA-150K is a small subset that contains very few or no samples relevant to the compositional reasoning tasks in GQA or the OCR-centric tasks in TextVQA, making these benchmarks unreliable indicators of model capability in this setting.\n\nInterestingly, for MMBench and MMBench (cn), both random selection and CoIDO achieve much higher scores than full-data fine-tuning. This is likely due to the presence of noisy or low-quality samples in the full dataset, where carefully selected subsets (or even random subsets) may better align with these benchmarks and reduce overfitting.\n\n\n**Table 3: Performance on Vision-Flan dataset (new dataset).**\n\n| **Model**            | **VQAv2** | **GQA** | **VizWiz** | **SQA** | **TextVQA** | **POPE** | **MME**  | **MMB** | **MMB(cn)** | **LLaVA-WB** | **Rel. (%)** |\n|----------------------|----------:|--------:|-----------:|--------:|------------:|---------:|---------:|--------:|------------:|-------------:|-------------:|\n| Full Fine-tune    |     74.5  |    47.1   |      47.1  |   52.8  |      46.4     |    61.8  |    46.4  |    85.7 |     1480.64 |         40.2 |        100.0 |\n| Random   |     74.6  |    44.3   |      44.3  |   50.0  |      40.9     |    59.8  |    40.9  |    81.3 |     1407.10 |         49.2 |         97.8 |\n| Ours|     **75.7**  |    **45.1**   |    **45.1**  |   **53.5**  | **45.3**   |    **62.3**  |    **45.3**  | **82.8** |     **1452.90** |  **52.0** |        **102.1** |\n\nTable 3 demonstrates that our method can be effectively transferred to Vision-Flan. Even under this entirely new domain dataset, CoIDO maintains strong relative performance, outperforming both random selection and the full-data baseline. Additional experiments confirm that our method generalizes well across different MLLMs and datasets, highlighting its scalability and adaptability.\n\nWe would like to note that all newly added experiments on alternative datasets and model backbones will be included in the final version of the paper, either in the main text or the appendix. We sincerely hope these results help clarify the generality and robustness of our method, and we kindly invite the reviewer oB3E to reconsider the merits of our submission in light of this additional evidence."}, {"review_id": "ottwP6fWsr", "reviewer": "Reviewer_o8Bj", "summary": "This paper introduces a novel framework called COIDO that optimizes data selection for multimodal large language models. Unlike prior methods that separately address data importance and diversity and require full-dataset processing, COIDO jointly optimizes both objectives using a lightweight scorer trained on only a small subset of data, efficiently balancing sample difficulty and dataset coverage during training. The framework significantly reduces computational overhead while achieving nearly the same performance as full-data fine-tuning. Experiments using the LLaVA-1.5-7B model show that COIDO retain 98.2% of full-set performance across ten benchmarks with only 20% training data, surpassing prior methods in both effectiveness and efficiency.", "questions": "- I am concerned that the current complex configuration might be overfitted to the specific setup using LLaVA-1.5-7B-LoRA and LLaVA-665K in the experiments. Could the authors provide additional results on other models or datasets to validate the generalizability of their approach? I would be inclined to increase my score if more experimental evidence is provided.\n\n- Could the authors include results using a larger number of clusters, e.g., 100 and 200.\n\n- In Figure 4, the performance with 25% training data appears worse than with 20%. Could the authors explain this counterintuitive result?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n- The paper is well-written and easy to follow.\n\n- The authors propose a novel method that jointly optimizes data importance and diversity, effectively addressing the instability issues observed in prior work.\n\n- Experimental results demonstrate that the proposed approach consistently outperforms baseline methods across a wide range of benchmarks.\n\n- The paper provides a thorough analysis of each module and includes a detailed discussion on hyperparameter selection.\n\n**Weaknesses**\n\n- The architecture is relatively complex and may be prone to overfitting on the current dataset and model configuration.\n\n- The method appears to heavily rely on specific hyperparameters, such as the number of clusters and training data ratios, which may affect generalizability.", "was_revised": true, "final_justification": "The experiments demonstrate that this method outperforms other baselines, and the author’s additional experiments and explanations have resolved my concerns.", "rebuttal": "**W1 & Q1: Concern about Overfitting to Specific dataset (LLaVA-665K) and model (LLaVA-1.5-7B-LoRA).**\n\n**A1:** We sincerely thank the reviewer for this insightful concern. To verify that our approach is not overly tailored to one specific dataset or model, we conducted extensive new experiments from two complementary dimensions: larger model backbones and alternative fine-tuning datasets. All additional experiments are conducted under the same settings as our paper.\n\n**(a) Across Model Backbones.** We applied CoIDO to **LLaVA-13B-LoRA**, a substantially larger vision-language model. As shown in Table 1, our method continues to deliver strong performance across 10 diverse benchmarks. Compared to both random selection and a recent SOTA methods (ICONS and COINCIDE), CoIDO achieves better or comparable results, particularly excelling in reasoning-heavy tasks like GQA, SQA-I, and MME. This suggests that our method not only scales well but also enhances both perception and cognition capabilities.\n\n**Table 1: Evaluation results on LLaVA-13B-LoRA.**\n| **Method**     | **VQAv2** | **GQA** | **VizWiz** | **SQA-I** | **TextVQA** | **POPE** | **MME**  | **MMB** | **MMB(cn)** | **LLaVA-WB** | **Rel. (%)** |\n|-|-:|-:|-:|-:|-:|-:|-:|-:|-:|-:|-:|\n| Full-Finetune|80.0|63.3|58.9|71.2|60.2|86.7|1541.7|68.5|61.5|69.5|100|\n| Random|76.7|60.5|48.0|68.8|57.7|84.8|1484.9|62.8|55.2|68.6|94.0|\n| ICONS|**77.9**|60.5|47.7|74.0|57.3|**87.4** |1503.9|**65.5**|**59.2**|65.3|95.7|\n| COINCIDE|**77.8** |60.4|**51.6**|70.0|58.6|**87.1**|1516.8|64.0|**57.7**|67.4|95.9|\n| **Ours**| 77.5    | **61.5**|    51.1    | **74.2**  |  **58.9**   |   85.8   | **1586.4**| **64.7**|     57.5    |     **69.7**     |   **97.2**   |\n\n**(b) Across Fine-tuning Datasets.**\nTo further assess generalizability under different data conditions, we evaluated CoIDO on two very different fine-tuning sets:\n\n``LLaVA-150K``, a much smaller subset (∼20%) of LLaVA data, often used in low-resource settings.\n\n``Vision-Flan``, a completely new visual instruction tuning dataset with 186k human-annotated images across 200+ tasks.\n\n**Table 2: Performance on LLaVA-150K dataset (smaller fine-tuning data).**\n| **Model**| **VQAv2** | **VizWiz** | **SQA** | **POPE** | **MME**  | **MMB** | **MMB(cn)** | **LLaVA-WB** | **Rel. (%)** |\n|-|-:|-:|-:|-:|-:|-:|-:|-:|-:|\n| Full Fine-tune|55.2|45.5|57.6| 57.9| 1234.5|22.4|27.1|65.9|100|\n| Random|**50.0**|44.8|53.4|54.2| 1184.7|**30.7**|30.1|62.4| 101.7|\n| Ours |49.5|**47.1**|**58.2**|**56.1**|**1214.8**|29.8|**32.3**|**63.8**|**104.8**|\n\nTable 2 reports the results on LLaVA-150K. CoIDO not only outperforms full-data fine-tuning on several benchmarks, but also improves the overall relative performance to 104.8%. Notably, we omit GQA and TextVQA from this table because all methods achieve <1% accuracy, likely due to the lack of in-domain examples in LLaVA-150K, making them unreliable in this setting. Interestingly, for MMBench and MMBench (cn), both random and CoIDO significantly outperform full-data training, which suggests that careful selection can help avoid overfitting to noisy data.\n\n**Table 3: Performance on Vision-Flan dataset (new dataset).**\n\n| **Model**| **VQAv2** | **GQA** | **VizWiz** | **SQA** | **TextVQA** | **POPE** | **MME**  | **MMB** | **MMB(cn)** | **LLaVA-WB** | **Rel. (%)** |\n|-|-:|-:|-:|-:|-:|-:|-:|-:|-:|-:|-:|\n| Full Fine-tune|74.5|47.1|47.1  |52.8  |46.4|61.8|46.4|85.7|1480.64 |40.2 |100.0|\n| Random|74.6|44.3|44.3|50.0|40.9|59.8|40.9|81.3 |1407.10 | 49.2 |97.8|\n| Ours|**75.7**|**45.1**|**45.1**|**53.5**|**45.3**|**62.3**|**45.3**|**82.8**|**1452.90**|**52.0**|**102.1**|\n\nTable 3 evaluates our method on Vision-Flan, a new dataset with a broader task distribution and no overlap with our development. Despite the domain gap, CoIDO still achieves the highest relative score (102.1%), outperforming both full-data and random baselines.\n\nThese results strongly support that our method generalizes beyond a single model and dataset. We believe this provides compelling evidence that CoIDO is ``not overfitting``, but rather offers a scalable and adaptable solution for sample selection in MLLM training.\n\nWe would like to note that all newly added experiments on alternative datasets and model backbones will be included in the final version of the paper, either in the main text or the appendix. We sincerely hope these results help clarify the generalizability of our method, and we kindly invite the reviewer o8Bj to reconsider the merits of our submission in light of these additional experiments.\n\n**W2 & Q2: Sensitivity to the Number of Clusters.**\n\n**A2:** We thank the reviewer for the insightful question. We acknowledge that the number of clusters $M$ is an important hyperparameter in our framework. However, we emphasize that the strong performance of our method is not the result of fine-tuning this value. Rather, our approach remains robust across a broad range of cluster settings. We provide additional results under cluster numbers of 100 and 200. As shown in Table 4, our method achieves 96.9% and 96.7% relative performance at 100 and 200 clusters, respectively. These results are highly consistent and comparable to state-of-the-art methods such as ICONS (97.1%) and COINCIDE (97.4%). This suggests that the influence of $M$ is relatively limited once it reaches a reasonable scale.\n\n**Table 4: Ablation on the number of clusters.**\n\n| Clusters  | VQAv2 | GQA  | VizWiz | SQA-I | TextVQA | POPE | MME    | MMB  | MMB(cn) | LLaVA-WB | Rel. (%) |\n|-|-|-|-|-|-|-|-|-|-|-|-|\n| 20 |**77.2**|**60.4**|**47.1**|69.4|**55.6**|**85.4**| 1450.2 |**63.8**|**56.7**|**70.1**|**98.2**|\n| 100  | 75.5  | 58.1 |45.7| 69.8  |55.1| 84.9 |**1483.1**|**63.8**| 56.6| 65.6| 96.9|\n| 200  | 75.7  | 57.5 |45.4|**70.4**|55.3|83.5|1467.9|62.4|56.4| 68.3|96.7|\n\nMoreover, as shown in Table 7 of our original Appendix C (Line 180), we previously evaluated other settings (e.g., $M=40$) and found they still outperformed COINCIDE. In fact, we observe that when $M $is small (e.g., 0–20), performance variations are more noticeable due to coarse clustering and less stable diversity loss signals. However, beyond that range, the results become much more stable and consistent. That said, our method remains robust and effective across a wide range of clustering configurations. Even under less optimal settings (e.g., 100 or 200 clusters), it still delivers competitive performance while maintaining significantly lower computational cost. \n\nOur choice of  $M=20$ is not arbitrary or obtained through exhaustive search. It is an empirically grounded trade-off between granularity and statistical reliability. In our implementation, each GPU processes 16 samples per batch, with 8 GPUs yielding 128 samples per minibatch. Setting $M=20$ ensures that each cluster has a reasonable probability of being represented in each batch, which stabilizes the computation of per-cluster means and the diversity loss. Using much larger $M$ (e.g., 200) leads to overly fine-grained partitions, where many clusters contain only a few samples per batch, weakening the quality of inter-cluster statistics. In conclusion, although the clustering granularity has some effect, our method performs consistently across a wide range of values.\n\nFinally, beyond cluster hyperparameters, the results on LLaVA-13B, LLaVA-150K, and Vision-Flan consistently show strong gains over full-data and other selection baselines. These findings collectively demonstrate that our method is not narrowly tuned to a specific data or model configuration, but rather generalizes well across different data scales, model sizes, and selection conditions.\n\n**Q3 & W2: About Counterintuitive Result in Figure 4.**\n\n**A3:** We thank the reviewer for noticing this interesting observation. Indeed, the slightly worse performance with 25% of training data compared to 20% appears counterintuitive at first glance. We analyze and explain this phenomenon from three complementary perspectives: model structure, joint optimization strategy, and intrinsic properties of the data itself.\n\n**(1) Model Structure (Limited Capacity of the Scorer).** Our CoIDO scorer is designed as a lightweight 4-layer MLP to maintain computational efficiency. While effective, this compact model has limited representational capacity. As the data selection proportion increases from 20% to 25%, the complexity and diversity of the training subset also increase significantly. Without additional regularization or early stopping, the scorer may slightly overfit on local batches, introducing gradient noise into the importance scores \n$w$. These noisy gradients can propagate back to the main model, creating minor interference and slightly degrading the final results.\n\n**(2) Training Strategy.** Our method jointly optimizes the importance and diversity terms using learned uncertainty weighting $\\sigma_I$ and $\\sigma_D$. When increasing from 20% to 25%, more \"borderline\" samples are inevitably introduced into each cluster. This reduces inter-cluster variance, leading to a flatter and less discriminative distribution of the learned importance scores. Consequently, the scoring becomes less sharp, diminishing the overall selection quality and slightly affecting downstream performance.\n\n**(3) Data Quality.** As in general MLLM training, simply increasing the amount of data does not always guarantee improved performance. Additional data may also include redundant, noisy, or misleading examples. Data selection methods face a similar challenge. Just like baseline methods such as COINCIDE and ICONS, a data selection model trained with more data does not necessarily perform better. In fact, our result strongly supports the core motivation of our method: careful and targeted selection is often more effective than indiscriminately using larger data volumes. This counterintuitive result further emphasizes the importance and superiority of our proposed data-selection approach."}, {"review_id": "8Sv2HszHpF", "reviewer": "Reviewer_28BB", "summary": "This paper proposes to solve MLLM fine-tuning cost by adaptively selecting important data. Specifically, the authors propose a CoIDO scorer that can automatically score the data samples via Coupled Importance-Diversity Optimization using only a small proportion of the data. Furthermore, it establishes theoretical justification for the proposed training objectives. Extensive results demonstrate that the proposed module can significantly reduce model training cost while preserving high performance.", "questions": "Please see the Weakness.", "limitations": "N.A.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The paper is well-written and easy to follow.\n2. The authors provide theoretical justifications over the proposed CoIDO scorer.\n3. Extensive experiments demonstrate the effectiveness of the method.\n\nWeakness & Questions:\n1. The proposed method is only tested on LLaVA-1.5-7b. More backbones such as LLaVA-1.5-13b, Qwen-VL should be tested to verify the effectiveness and generalizability of the proposed method.\n2. In line 175, the authors claim that \"A high cross-entropy $CE(y_{ik}, \\tilde{y}_{ik})$ implies the sample is difficult to learn and thus more important\". However, difficulty is a widely-studied concept in Curriculum Learning, and [1] has suggested that instead of an instantaneous hardness (e.g. loss), a dynamic hardness will be more robust to measure the actual difficulty with respect to each sample. Therefore, it is questionable that the proposed $w_i$ can optimally reflect the importance/difficulty of the data.\n3. Does the data selection strategy affect the generalization speed? I suggest the authors provide a brief analysis into this, since normally model will converge slower on more data.\n4. Does the proposed CoIDO scorer suffer from out-of-distribution problems? For example, since it is trained on LLaVA-655K, can it be used to effectively score other datasets?\n\nReferences:\n[1] Curriculum Learning by Dynamic Instance Hardness", "was_revised": true, "final_justification": "Most of my concerns have been addressed. I intend to keep my current (positive) score.", "rebuttal": "**Q1 & W1: Generalization to Other Models.**\n\n**A1:** We sincerely thank the reviewer for highlighting this important issue regarding generalization beyond a single model (LLaVA-1.5-7B). To thoroughly address this concern, we conducted extensive additional experiments along two complementary dimensions: larger model backbones and alternative fine-tuning datasets. \n\n**(a) Across Different Models**. We extended our method to a significantly larger vision-language model, **LLaVA-13B-LoRA**, to verify robustness. As demonstrated in Table 1, our approach consistently achieves competitive or superior results across 10 diverse benchmarks compared to the state-of-the-art selection methods (ICONS and COINCIDE).\n\n**Table 1: Evaluation results on LLaVA-13B-LoRA.**\n| **Method**     | **VQAv2** | **GQA** | **VizWiz** | **SQA-I** | **TextVQA** | **POPE** | **MME**  | **MMB** | **MMB(cn)** | **LLaVA-WB** | **Rel. (%)** |\n|-|-:|-:|-:|-:|-:|-:|-:|-:|-:|-:|-:|\n| Full-Finetune|80.0|63.3|58.9|71.2|60.2|86.7|1541.7|68.5|61.5|69.5|100 |\n| Random|76.7|60.5|48.0|68.8|57.7|84.8|1484.9|62.8|55.2|68.6|94.0|\n| ICONS|**77.9**|60.5|47.7|74.0|57.3|**87.4** |1503.9|**65.5**|**59.2**|65.3|95.7|\n| COINCIDE |**77.8**|60.4|**51.6**|70.0|58.6| **87.1** |1516.8|64.0|**57.7**|67.4|95.9|\n| **Ours**|77.5| **61.5**|51.1| **74.2**|**58.9**|85.8|**1586.4**| **64.7**|57.5|**69.7**|**97.2**|\n\n**(b) Across Fine-tuning Datasets**. We further validate generalization across distinct fine-tuning datasets:\n\n``LLaVA-150K``, a substantially smaller subset (~20%) of LLaVA, representative of low-resource training.\n\n``Vision-Flan``, a completely new, large-scale visual instruction tuning dataset containing 186K images across 200+ diverse vision-language tasks.\n\n**Table 2: Performance on LLaVA-150K dataset (smaller fine-tuning data).**\n| **Model**| **VQAv2**| **VizWiz**| **SQA**|**POPE**| **MME**|**MMB**|**MMB(cn)**|**LLaVA-WB** | **Rel. (%)** |\n|-|-:|-:|-:|-:|-:|-:|-:|-:|-:|\n| Full Fine-tune|55.2|45.5|57.6|57.9|1234.5|22.4|27.1|65.9|100|\n| Random|**50.0**|44.8|53.4|54.2|1184.7|**30.7**|30.1| 62.4|101.7|\n| Ours |49.5|**47.1**|**58.2**|**56.1**| **1214.8** |29.8| **32.3** | **63.8** |**104.8**|\n\nAs shown in Table 2, CoIDO consistently outperforms full-data fine-tuning and achieves a relative improvement of 104.8%. We omitted GQA and TextVQA results as both baselines and our method yielded accuracies <1%, indicating that the smaller LLaVA-150K subset lacks meaningful coverage of these datasets’ domains, rendering them unsuitable for this particular evaluation.\n\n**Table 3: Performance on Vision-Flan dataset (new dataset).**\n\n| **Model** | **VQAv2** | **GQA** | **VizWiz** | **SQA** | **TextVQA** | **POPE** | **MME**  | **MMB** | **MMB(cn)** | **LLaVA-WB** | **Rel. (%)** |\n|-|-:|-:|-:|-:|-:|-:|-:|-:|-:|-:|-:|\n| Full Fine-tune|74.5|47.1|47.1  |52.8  |46.4|61.8  | 46.4  | 85.7 |1480.64 |40.2 |100.0 |\n| Random|74.6|44.3|44.3|50.0  |40.9|59.8  |40.9  |81.3 |1407.10 | 49.2 |97.8 |\n| Ours|**75.7**|**45.1**|**45.1**|**53.5**| **45.3**   | **62.3**  |**45.3**  | **82.8** | **1452.90** |  **52.0** |**102.1** |\n\nIn Table 3, we test CoIDO on Vision-Flan, a dataset substantially different from our original training data. Remarkably, CoIDO achieves superior results, demonstrating strong adaptability and effectiveness even under significant domain shift.\n\n``Regarding the suggested evaluation on Qwen-VL``, unfortunately, Qwen-VL's instruction tuning dataset is currently not publicly available, preventing us from replicating a controlled evaluation. However, we actively seek and plan to test additional backbones and datasets as they become accessible.\n\nOverall, these comprehensive evaluations demonstrate that CoIDO is robust and broadly generalizable across multiple model scales and diverse fine-tuning datasets, clearly mitigating concerns regarding overfitting to a specific dataset or model configuration. These results will be included in the main text of our final version manuscript.\n\n**Q2 & W2：Difficulty Measurement Strategy.**\n\n**A2:**  Thank you for pointing us to Curriculum Learning by Dynamic Instance Hardness (DIH). Our method is inspired by [1], which shows that simple per‑sample scores computed very early in training (1–3 epochs) already identify the most important and difficult examples; pruning up to 50 % of data with these early‑loss scores incurs almost no accuracy drop. This empirical finding suggests that instantaneous loss is a strong, stable proxy for sample hardness. Below, we clarify why our use of early‑epoch cross‑entropy remains valid, how our optimization implicitly accumulates “dynamic hardness. \n\n**Two-epoch accumulation of sample history.** In original LLaVA fine-tuning, there is only one epoch, which means all samples are traversed only once. Our training schedule spans two full epochs over the selected p % subset. Hence, each sample is encountered at least twice. Suppose a particular sample exhibits a high cross‑entropy loss in the first epoch, indicating that it is difficult for the current model. After one epoch of learning, the main model parameters have adapted, so the same sample may become substantially easier (lower loss) when revisited in epoch 2. Because the learnable weight $w_i$ is updated every time the sample is processed, its value in the second pass already encodes the loss trajectory from the first pass. In this way the scorer naturally integrates temporal information: samples that remain persistently difficult keep higher cumulative influence, whereas samples whose loss quickly decreases are automatically down‑weighted.\n\n**Dynamic weights instead of raw CE loss values.** Crucially, we do not treat the instantaneous cross‑entropy as the hardness score itself. Instead, each sample has a dedicated score $w_i$ that is continuously adjusted by back‑propagation. The gradient of the importance loss pushes $w_i$ up or down depending on the current loss, while the diversity term further modulates \n$w_i$ at the cluster level. This iterative update is mathematically equivalent to an implicit, learnable moving average historical losses influence the present $w_i$, and the current $w_i$ in turn shapes future gradient flows. Consequently, our difficulty assessment is dynamic and self‑correcting, capturing long‑term sample behaviour rather than a single noisy observation.\n\nWhile our current design attains strong results, we agree that explicit **DIH style hardness could be an interesting extension.** Because CoIDO’s coupled objective accepts any scalar importance signal. We will explore it as part of future work. We thank the reviewer for this valuable insight, which will be discussed in the related work in our paper. \n\n**Q3 & W3: Data Selection Strategy and Generalization Speed.**\n\n**A3:** We thank the reviewer for raising this interesting question. Indeed, we observed that data selection strategies do influence the speed of model convergence. In our experiments, we found that when applying CoIDO with only 20% of the data, a single epoch of fine-tuning (similar to what is commonly used in previous methods such as ICONS and COINCIDE) was not sufficient to achieve full convergence. However, extending the training to two or three epochs revealed that convergence was typically achieved by the middle of the second epoch.\n\nImportantly, although 2 epochs were required for full convergence, the total number of training steps for CoIDO remained significantly smaller than for other baselines. Therefore, when measured by total training steps, our method converges faster, highlighting its efficiency advantage.  We are very willing to provide a brief analysis in our paper and continue discussing this question during the discussion stage.\n\n**Q4 & W4: Does the proposed CoIDO scorer suffer from out-of-distribution problems?**\n\n**A4:** We thank the reviewer for raising the important question of whether the CoIDO scorer trained on LLaVA-665K suffers from out-of-distribution (OOD) generalization issues. To directly evaluate this, we conducted an experiment where we used a CoIDO scorer trained solely on LLaVA-665K to score and select data from an entirely different dataset—Vision-Flan. There is no additional training or adaptation of the scorer to the new domain. \n\n**Table 4: Using LLaVA-665K Trained Scorer to Select Data from Vision-Flan.**\n\n| **Scorer Source**| **VQAv2** | **GQA** | **VizWiz** | **SQA** | **TextVQA** | **POPE** | **MME**  | **MMB** | **MMB (cn)** | **LLaVA-WB** | **Rel. (%)** |\n|-|-:|-:|-:|-:|-:|-:|-:|-:|-:|-:|-:|\n| Trained on Vision-Flan|**75.7**|45.1|**53.5**|62.3|**45.3**|82.8|1452.9|**52.0**|46.8|37.6|102.1|\n| Trained on LLaVA-665K|72.7|**46.8**|53.3|**66.2**|42.1|**85.5**|**1486.1**|51.4|**47.3**|**40.8**|**103.3**   |\n\nSurprisingly, the LLaVA-trained scorer achieved even better results on Vision-Flan than the scorer trained directly on Vision-Flan itself. As shown in Table 4, the relative performance reached 103.3%, exceeding the 102.1% obtained by a scorer trained on Vision-Flan. We believe this is because LLaVA-665K is a significantly larger and more diverse dataset, allowing the scorer to learn a more generalizable notion of sample difficulty and importance. This enables it to transfer effectively across domains, even outperforming in-distribution scorers trained on smaller datasets.\n\nThis result demonstrates that CoIDO does not overfit to the training distribution of its scorer, and in fact shows strong transferability across datasets. It suggests that a scorer trained once on a large-scale, diverse dataset can generalize well and be reused to effectively select valuable subsets from other datasets. We believe this is an appealing practical advantage of CoIDO, enabling scalable and efficient data selection in real-world multimodal model development. This part of the content will also be included in future manuscripts. \n\nReferences: [1] Deep Learning on a Data Diet: Finding Important Examples Early in Training, NeurIPS 2021"}, {"review_id": "sJAx7nfhDA", "reviewer": "Reviewer_Nq7i", "summary": "This paper introduces CoIDO, a novel framework for efficient data selection in visual instruction tuning of MLLMs. The key challenge addressed is the high computational cost of training on large-scale datasets, where existing methods either process the entire dataset (inefficient) or treat importance and diversity as separate objectives (suboptimal). \n\nCoIDO jointly optimizes data importance and diversity through a unified framework, using a homoscedastic uncertainty-based loss to dynamically balance the two objectives during training.\n\nInstead of evaluating the entire dataset, CoIDO trains a plug-in MLP scorer on only 20% of randomly sampled data to learn the underlying distribution. This scorer then assigns CoIDO scores (combining importance and diversity) to all samples, enabling efficient ranking.\n\nOn the LLaVA-1.5-7B model, CoIDO selects a 20% subset that achieves 98.2% of full-data fine-tuning performance across 10 downstream tasks (e.g., VQA, ScienceQA). It reduces computational costs significantly (only 4.2 ExaFLOPs vs. 12.6E for competitors like ICONS) by avoiding full-dataset processing.\n\nCoIDO provides a scalable, resource-efficient solution for MLLM instruction tuning, making high-performance model adaptation accessible even with limited computational resources. The framework’s simplicity and effectiveness are validated through extensive experiments, establishing a new state of the art in data selection for multimodal tasks.", "questions": "1. To order to demonstrate the scalability, authors should validate the proprosed method on MLLMs with larger model size, at least a 13B model.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The paper introduces a well-designed framework (CoIDO) that effectively combines importance and diversity optimization through a principled homoscedastic uncertainty-based loss. The theoretical justification is sound, and the optimization strategy is well-motivated.\n2. CoIDO addresses a critical bottleneck in MLLM fine-tuning (high computational costs) with a solution that is both effective and efficient.This could democratize access to large-scale visual instruction tuning.\n3. The coupled optimization of importance and diversity via a unified loss is a distinct contribution compared to prior decoupled approaches. \n4. While not new in multi-task learning, its application to balance data selection objectives is creative and well-executed.\n5. The paper is logically organized, with clear motivation, methodology, and experimental results. \n\nWeaknesses:\n1. Experiments are restricted to LLaVA-7B; validation on larger models (e.g., LLaVA-13B) is absent, raising questions about scalability.\n2. While ablation studies are provided, deeper analysis (e.g., varying clustering methods beyond spectral clustering) could strengthen claims about robustness.", "was_revised": true, "final_justification": "Most of my concerns have been addressed by rebuttal, I raise score to accept.", "rebuttal": "**Q1 & W1: Demonstrate the Scalability of the Method.**\n\n**A1**: We sincerely thank the reviewer for raising this important concern. To rigorously evaluate the scalability of CoIDO, we conduct additional experiments using **LLaVA-13B-LoRA**,  which provides strong evidence of CoIDO’s generalizability beyond the LLaVA-7B setting. The results are reported in Table 1.\n\n**Table 1: Evaluation results on LLaVA-13B-LoRA.**\n| **Method**     | **VQAv2** | **GQA** | **VizWiz** | **SQA-I** | **TextVQA** | **POPE** | **MME**  | **MMB** | **MMB(cn)** | **LLaVA-WB** | **Rel. (%)** |\n|-|-:|-:|-:|-:|-:|-:|-:|-:|-:|-:|-:|\n| Full-Finetune|80.0|63.3|58.9|71.2|60.2|86.7|1541.7|68.5|61.5|69.5|100 |\n| Random|76.7|60.5|48.0|68.8|57.7|84.8|1484.9|62.8|55.2|68.6|94.0|\n| ICONS|**77.9**|60.5|47.7|74.0|57.3|**87.4** |1503.9|**65.5**|**59.2**|65.3|95.7|\n| COINCIDE |**77.8**|60.4|**51.6**|70.0|58.6| **87.1** |1516.8|64.0|**57.7**|67.4|95.9|\n| **Ours**|77.5| **61.5**|51.1| **74.2**|**58.9**|85.8|**1586.4**| **64.7**|57.5|**69.7**|**97.2**|\n\n As shown in Table 1, our method continues to perform strongly across 10 challenging benchmarks. Compared to both random selection and the recent SOTA methods ICONS and COINCIDE, CoIDO maintains or exceeds performance, particularly on reasoning-intensive benchmarks such as GQA, SQA-I, and MME. This indicates that our method scales well to larger models and further enhances both perception and cognition tasks.\n\nTo further validate the scalability of CoIDO across diverse data , we additionally test it on two contrasting fine-tuning datasets:\n\n``LLaVA-150K``, a small-scale subset (~20%) often used in limited-resource settings, and\n\n``Vision-Flan``, a newly released large-scale dataset containing 186k images with 200+ instruction-following vision-language tasks.\n\n**Table 2: Performance on LLaVA-150K dataset (smaller fine-tuning data).**\n| **Model**| **VQAv2**| **VizWiz**| **SQA**|**POPE**| **MME**|**MMB**|**MMB(cn)**|**LLaVA-WB** | **Rel. (%)** |\n|-|-:|-:|-:|-:|-:|-:|-:|-:|-:|\n| Full Fine-tune|55.2|45.5|57.6|57.9|1234.5|22.4|27.1|65.9|100|\n| Random|**50.0**|44.8|53.4|54.2|1184.7|**30.7**|30.1| 62.4|101.7|\n| Ours |49.5|**47.1**|**58.2**|**56.1**| **1214.8** |29.8| **32.3** | **63.8** |**104.8**|\n\nAs shown in Table 2, CoIDO consistently outperforms full fine-tuning and achieves a relative improvement of 104.8%. We omitted GQA and TextVQA results as both baselines and our method yielded accuracies <1%, indicating that the smaller LLaVA-150K subset lacks meaningful coverage of these datasets’ domains, rendering them unsuitable for this particular evaluation.\n\n**Table 3: Performance on Vision-Flan dataset (new dataset).**\n\n| **Model** | **VQAv2** | **GQA** | **VizWiz** | **SQA** | **TextVQA** | **POPE** | **MME**  | **MMB** | **MMB(cn)** | **LLaVA-WB** | **Rel. (%)** |\n|-|-:|-:|-:|-:|-:|-:|-:|-:|-:|-:|-:|\n| Full Fine-tune|74.5|47.1|47.1  |52.8  |46.4|61.8  | 46.4  | 85.7 |1480.64 |40.2 |100.0 |\n| Random|74.6|44.3|44.3|50.0  |40.9|59.8  |40.9  |81.3 |1407.10 | 49.2 |97.8 |\n| Ours|**75.7**|**45.1**|**45.1**|**53.5**| **45.3**   | **62.3**  |**45.3**  | **82.8** | **1452.90** |  **52.0** |**102.1** |\n\nIn Table 3, we evaluate CoIDO on Vision-Flan, a new, out-of-domain dataset. CoIDO still outperforms both random selection and full-data training, achieving a relative score of 102.1%. This strongly demonstrates robust generalization and domain transferability, further affirming the scalability and versatility of our method.\n\n**Q2: Provide Deeper Analysis to Strengthen Claims about Robustness.**\n\n**A2:** We thank the reviewer for raising this important point regarding robustness. We provide two complementary pieces of evidence to demonstrate that CoIDO is both structurally and behaviorally robust across different clustering strategies and data domains.\n\n**(a) Varying clustering methods.**\nWe conducted additional ablations to test whether our approach is sensitive to the choice of clustering strategy. In addition to the spectral clustering used in the main paper, we experimented with **K-Means** and **Gaussian Mixture Model (GMM)** clustering. We specifically chose these two methods because, like spectral clustering, they allow us to explicitly control the number of clusters. This makes it feasible to conduct fair comparisons under the same cluster count. In contrast, methods like DBSCAN or Hierarchical Clustering often determine the number of clusters implicitly based on distance thresholds or linkage criteria, making controlled comparisons less reliable. We chose the number of clusters $M=20$, and the results are reported in Table 4.\n\n**Table 4: Comparison of Different Clustering Methods Used in CoIDO (Number of Clusters $M$ = 20)**\n\n| Clustering Method | VQAv2 | GQA  | VizWiz | SQA  | TextVQA | POPE  | MME    | MMB  | MMB_CN | LLaVA-WB | Rel.  |\n|-|-|-|-|-|-|-|-|-|-|--|-|\n| K-Means| 74.4| 57.0 | **47.9**   | **69.9** | 52.7    | 84.8  | **1489.8** | 51.8 | 52.8   | 57.4 | 92.9  |\n| GMM| 76.9  | 58.4 | 45.8   | 68.2 | 54.5    | 83.4  | 1408.7 | 62.1 |**56.8**| 68.6| 96.2  |\n| Ours| **77.2**  | **60.4** | 47.1| 69.4 | **55.6**    | **85.4**  | 1450.2 | **63.8** | 56.7   | **70.1**| **98.2**  |\n\nAs shown in Table 4, our spectral clustering consistently achieves the highest overall performance across the majority of benchmarks. While K-Means performs well on VizWiz and MME, it significantly underperforms on MMBench and LLaVA-WB, suggesting poor generalization. This inconsistency is likely due to K-Means’ inability to model nonlinear boundaries, which means it assumes spherical clusters and struggles with complex, manifold-structured data, often leading to poor inter-cluster separation and diversity.\n\nGMM, while more flexible than K-Means (due to soft assignments and elliptical clusters), still assumes that data comes from a mixture of Gaussians. This assumption limits its ability to capture complex decision boundaries and latent task structures, especially in vision-language embedding spaces.\n\nIn contrast, spectral clustering can capture arbitrary cluster shapes by leveraging eigenstructure in the similarity graph, making it much more suitable for the high-dimensional, nonlinear nature of multimodal representation spaces. These results empirically justify our use of spectral clustering as the most robust and effective option for partitioning samples when optimizing diversity-aware selection.\n\n\n**(b) Transferability to Out-of-Domain Datasets.** \nIn addition to testing different clustering strategies, we further validate the robustness and transferability of CoIDO by evaluating whether the learned scorer trained on one dataset (LLaVA-665K) can be **directly applied to another dataset (Vision-Flan) without retraining.**\n\n**Table 5: Using LLaVA-665K Trained Scorer to Select Data from Vision-Flan.**\n\n| **Scorer Source**| **VQAv2** | **GQA** | **VizWiz** | **SQA** | **TextVQA** | **POPE** | **MME**  | **MMB** | **MMB (cn)** | **LLaVA-WB** | **Rel. (%)** |\n|-|-:|-:|-:|-:|-:|-:|-:|-:|-:|-:|-:|\n| Trained on Vision-Flan|**75.7**|45.1|**53.5**|62.3|**45.3**|82.8|1452.9|**52.0**|46.8|37.6|102.1|\n| Trained on LLaVA-665K|72.7|**46.8**|53.3|**66.2**|42.1|**85.5**|**1486.1**|51.4|**47.3**|**40.8**|**103.3**   |\n\nAs shown in Table 5, the scorer trained on LLaVA-665K performs remarkably well when used to select training samples from Vision-Flan. Surprisingly, it even outperforms the scorer trained directly on Vision-Flan across several key metrics, including GQA, SQA, POPE, MME, MMB (cn), and LLaVA-WB, achieving a higher overall relative performance of 103.3% vs. 102.1%. We attribute this to the fact that LLaVA-665K is a larger and more diverse dataset, enabling the scorer to learn more generalizable patterns that transfer effectively to other domains.\n\nThis result provides strong evidence that our scorer does not overfit to its training domain, and instead captures domain-agnostic signals of data quality and informativeness, making CoIDO a robust and transferable framework for data selection in MLLMs. All these additional experiments will be included in our final version manuscript."}]

Author rebuttal:

Meta-review: The final ratings for this paper are unanimously positive (two "Accept" and two "Boarderline Accept").

The AC agrees with the reviewers' positive evaluation. The paper introduces CoIDO framework for efficient data selection in visual instruction tuning. The primary concerns shared across all initial reviews are the method's generalizability and robustness. 
However, the authors addressed these during the rebuttal by providing an set of additional experiments.
They demonstrated CoIDO's effectiveness on a larger model (LLaVA-13B-LoRA) and on two additional datasets (the smaller LLaVA-150K and the out-of-domain Vision-Flan). They also provided a detailed explanation for the Diversity Loss and presented new ablations on key hyperparameters. 

The authors' rebuttal and the positive engagement from all reviewers have convincingly resolved the initial weaknesses. Therefore, the AC recommends this paper for acceptance.
Decision: Accept (poster)