Title: CoIDO: Efficient Data Selection for Visual Instruction Tuning via Coupled Importance-Diversity Optimization
Venue: NeurIPS 2025
Accepted: yes
Keywords: Large Language Model

Visual Instruction Tuning

Data Selection
Abstract: Multimodal large language models (MLLMs) rely heavily on instruction tuning to align vision and language capabilities, yet the computational cost of training on large-scale datasets remains a major bottleneck. Existing data selection methods aim to mitigate this by selecting important and diverse subsets, but they often suffer from two critical drawbacks: high computational overhead from processing the entire dataset and suboptimal data selection due to separate treatment of importance and diversity.

We introduce CoIDO, a novel dual-objective framework that jointly optimizes data importance and diversity to overcome these challenges. Unlike existing approaches that require costly evaluations across the whole dataset, CoIDO employs a lightweight plug-in scorer. This scorer is trained on just a small random sample of data to learn the distribution of the candidate set, drastically reducing computational demands. By leveraging a homoscedastic uncertainty-based formulation, CoIDO effectively balances importance and diversity during training, enabling the scorer to assign CoIDO scores to all data points. This unified scoring approach allows for direct ranking and selection of the most valuable subsets, completely bypassing the need for specialized algorithms.

In our experiments, we trained the CoIDO Scorer using only 20% of randomly sampled data. Once trained, CoIDO was applied to the entire dataset to select a 20% subset for instruction tuning. On the widely used LLaVA-1.5-7B model across ten downstream tasks, this selected subset achieved an impressive 98.2% of the performance of full-data fine-tuning, on average. Moreover, CoIDO outperforms all competitors in terms of both efficiency (lowest training FLOPs) and aggregated accuracy. Our code is available at: https://github.com/SuDIS-ZJU/CoIDO


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces CoIDO, a novel framework for efficient data selection in visual instruction tuning of MLLMs. The key challenge addressed is the high computational cost of training on large-scale datasets, where existing methods either process the entire dataset (inefficient) or treat importance and diversity as separate objectives (suboptimal). \n\nCoIDO jointly optimizes data importance and diversity through a unified framework, using a homoscedastic uncertainty-based loss to dynamically balance the two objectives during training.\n\nInstead of evaluating the entire dataset, CoIDO trains a plug-in MLP scorer on only 20% of randomly sampled data to learn the underlying distribution. This scorer then assigns CoIDO scores (combining importance and diversity) to all samples, enabling efficient ranking.\n\nOn the LLaVA-1.5-7B model, CoIDO selects a 20% subset that achieves 98.2% of full-data fine-tuning performance across 10 downstream tasks (e.g., VQA, ScienceQA). It reduces computational costs significantly (only 4.2 ExaFLOPs vs. 12.6E for competitors like ICONS) by avoiding full-dataset processing.\n\nCoIDO provides a scalable, resource-efficient solution for MLLM instruction tuning, making high-performance model adaptation accessible even with limited computational resources. The framework’s simplicity and effectiveness are validated through extensive experiments, establishing a new state of the art in data selection for multimodal tasks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The paper introduces a well-designed framework (CoIDO) that effectively combines importance and diversity optimization through a principled homoscedastic uncertainty-based loss. The theoretical justification is sound, and the optimization strategy is well-motivated.\n2. CoIDO addresses a critical bottleneck in MLLM fine-tuning (high computational costs) with a solution that is both effective and efficient.This could democratize access to large-scale visual instruction tuning.\n3. The coupled optimization of importance and diversity via a unified loss is a distinct contribution compared to prior decoupled approaches. \n4. While not new in multi-task learning, its application to balance data selection objectives is creative and well-executed.\n5. The paper is logically organized, with clear motivation, methodology, and experimental results. \n\nWeaknesses:\n1. Experiments are restricted to LLaVA-7B; validation on larger models (e.g., LLaVA-13B) is absent, raising questions about scalability.\n2. While ablation studies are provided, deeper analysis (e.g., varying clustering methods beyond spectral clustering) could strengthen claims about robustness.\n\nQUESTIONS:\n1. To order to demonstrate the scalability, authors should validate the proprosed method on MLLMs with larger model size, at least a 13B model.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-19"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes to solve MLLM fine-tuning cost by adaptively selecting important data. Specifically, the authors propose a CoIDO scorer that can automatically score the data samples via Coupled Importance-Diversity Optimization using only a small proportion of the data. Furthermore, it establishes theoretical justification for the proposed training objectives. Extensive results demonstrate that the proposed module can significantly reduce model training cost while preserving high performance.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The paper is well-written and easy to follow.\n2. The authors provide theoretical justifications over the proposed CoIDO scorer.\n3. Extensive experiments demonstrate the effectiveness of the method.\n\nWeakness & Questions:\n1. The proposed method is only tested on LLaVA-1.5-7b. More backbones such as LLaVA-1.5-13b, Qwen-VL should be tested to verify the effectiveness and generalizability of the proposed method.\n2. In line 175, the authors claim that \"A high cross-entropy $CE(y_{ik}, \\tilde{y}_{ik})$ implies the sample is difficult to learn and thus more important\". However, difficulty is a widely-studied concept in Curriculum Learning, and [1] has suggested that instead of an instantaneous hardness (e.g. loss), a dynamic hardness will be more robust to measure the actual difficulty with respect to each sample. Therefore, it is questionable that the proposed $w_i$ can optimally reflect the importance/difficulty of the data.\n3. Does the data selection strategy affect the generalization speed? I suggest the authors provide a brief analysis into this, since normally model will converge slower on more data.\n4. Does the proposed CoIDO scorer suffer from out-of-distribution problems? For example, since it is trained on LLaVA-655K, can it be used to effectively score other datasets?\n\nReferences:\n[1] Curriculum Learning by Dynamic Instance Hardness\n\nQUESTIONS:\nPlease see the Weakness.\n\nLIMITATIONS:\nN.A.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-22"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a novel framework called COIDO that optimizes data selection for multimodal large language models. Unlike prior methods that separately address data importance and diversity and require full-dataset processing, COIDO jointly optimizes both objectives using a lightweight scorer trained on only a small subset of data, efficiently balancing sample difficulty and dataset coverage during training. The framework significantly reduces computational overhead while achieving nearly the same performance as full-data fine-tuning. Experiments using the LLaVA-1.5-7B model show that COIDO retain 98.2% of full-set performance across ten benchmarks with only 20% training data, surpassing prior methods in both effectiveness and efficiency.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n- The paper is well-written and easy to follow.\n\n- The authors propose a novel method that jointly optimizes data importance and diversity, effectively addressing the instability issues observed in prior work.\n\n- Experimental results demonstrate that the proposed approach consistently outperforms baseline methods across a wide range of benchmarks.\n\n- The paper provides a thorough analysis of each module and includes a detailed discussion on hyperparameter selection.\n\n**Weaknesses**\n\n- The architecture is relatively complex and may be prone to overfitting on the current dataset and model configuration.\n\n- The method appears to heavily rely on specific hyperparameters, such as the number of clusters and training data ratios, which may affect generalizability.\n\nQUESTIONS:\n- I am concerned that the current complex configuration might be overfitted to the specific setup using LLaVA-1.5-7B-LoRA and LLaVA-665K in the experiments. Could the authors provide additional results on other models or datasets to validate the generalizability of their approach? I would be inclined to increase my score if more experimental evidence is provided.\n\n- Could the authors include results using a larger number of clusters, e.g., 100 and 200.\n\n- In Figure 4, the performance with 25% training data appears worse than with 20%. Could the authors explain this counterintuitive result?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

comments: SUMMARY:
This work presents COIDO, an innovative framework aimed at improving the training efficiency of multimodal large language models . It tackles the computational challenges of instruction tuning by simultaneously optimizing for data importance and diversity. A key innovation is the introduction of Diversity Loss and Important Loss, which helps balance these two aspects. COIDO uses a lightweight scorer trained on a small random subset of data to learn the distribution, reducing computational demands. Through a homoscedastic uncertainty-based formulation, it effectively balances importance and diversity without needing specialized algorithms for data selection. Experimental results demonstrate that COIDO achieves near-full-data performance with only a fraction (20%) of the data, outperforming existing methods in both efficiency and accuracy.

STRENGTHS AND WEAKNESSES:
Strengths:
1. The framework's code is available.
2. Present Diversity Loss and Important Loss.

Weaknesses：
1. Diversity Loss doesn't make sense and needs further explanation, especially on how to assess data diversity with a one pass.
2. The experiments should be conducted across more models to verify the effectiveness.

QUESTIONS:
As showned in the Weakness.

LIMITATIONS:
As showned in the Weakness.
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-07-02