Title: Data Selection Matters: Towards Robust Instruction Tuning of Large Multimodal Models
Venue: NeurIPS 2025
Accepted: yes
Keywords: Large Multimodal Models

Data Selection

Robustness
Abstract: Selecting a compact subset of visual instruction–following data has emerged as an effective way to align large multimodal models with human intentions while avoiding the high cost of full-dataset training. Yet we observe that both full-data training and existing state-of-the-art data selection methods tend to inherit underlying dataset biases such as position bias and spurious correlations, leading to biased model behaviors. To address this issue, we introduce ARDS, a robustness-aware targeted visual instruction-selection framework that explicitly mitigates these weaknesses, sidestepping the need for access to downstream data or time-consuming gradient computation. Specifically, we first identify the worst-case evaluation subgroups through visual and textual task-specific perturbations. The robust training mixture is then constructed by prioritizing samples that are semantically closer to these subgroups in a rich multimodal embedding space. Extensive experiments demonstrate that ARDS substantially boosts both robustness and data efficiency for visual instruction tuning. We also showcase that the robust mixtures produced with a smaller model transfer effectively to larger architectures. Our code and selected datasets that have been demonstrated transferable across models are available at https://github.com/xyang583/ARDS.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper propose a gradient-free robustness-aware data selection framework, a data filtering strategy that selects the worst-case samples based on embedding distances from a lightly fine-tuned proxy model, to improve robustness and efficiency for instruction tuning. The approach is evaluated across multiple multimodal and language understanding benchmarks, and the authors report that selecting approximately 30% of the data yields comparable or even better performance than using the full training set.\n\nSTRENGTHS AND WEAKNESSES:\nStrength: \n1. The proposed ARDS method is conceptually simple yet well-motivated.\n2. The method achieves consistently strong performance across a diverse set of benchmarks, highlighting the effectiveness of the proposed selection strategy.\n3. The proposed data selection strategy is especially relevant for scenarios where computational resources are limited or training on full datasets is infeasible.\n\nWeakness:\n1. One major concern lies in the conceptual link between the data selection strategy and the robustness claims made throughout the paper. The authors heavily emphasize robustness as the central motivation, yet the core selection mechanism relies on embedding distances to identify worst cases. It is not sufficiently explained why samples that are semantically distant or harder to fit necessarily correspond to those that improve robustness. The assumption that worst-cases are aligned with robustness cases remains unsubstantiated, and the paper lacks an ablation or theoretical justification to support this connection. As a result, the method appears more as a form of relevance or difficulty-based sampling, rather than one explicitly targeting robustness.\n2. I found the results on ARC-e and BoolQ in Table 3 puzzling, the full-data training performs significantly worse than using only a random 30% subset. This seems counterintuitive unless the full training set introduces severe biases that the smaller subset helps avoid. Could the authors elaborate on this surprising behavior? It would be helpful to include further analysis or visualization, such as label distributions, training loss curves, or predictions breakdown, to confirm this effect is genuine rather than due to setup discrepancies.\n3. Figure 3 partially justifies the 30% ratio on two representative tasks, but it remains unclear whether ARDS consistently holds its advantage across tasks of varying modality complexity and reasoning depth. Given that the benchmark includes visual multiple-choice, text-based classification, and math QA, a brief discussion on whether this ratio remains effective across different cognitive requirements would strengthen the generality claim.\n4. One point that remains underexplored is the practical motivation for data selection in the current setup. Since the full training data-LLAVA665k is assumed to be available, clean, and standard across benchmarks, it is not immediately clear why discarding a large portion (70%) of it would be desirable. The authors should clarify under what practical or deployment conditions such a selection strategy is intended to be applied. For example, is the method primarily motivated by training efficiency, overfitting concerns, or robustness to spurious correlations? Making this clearer would help situate the proposed method in real-world training scenarios.\n\nQUESTIONS:\nSee weakness above. \nOverall, my main confusion with this paper lies in how the authors directly associate worst-case examples and accuracy improvements with robustness. I would appreciate further clarification on this point, and I’m open to engaging in further discussion.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper designs a novel visual instruction selection approach, named ARDS. ARDS first divides clusters based on the extracted embeddings of conversations and then calculate ranking scores for each visual instruction by taking both embedding similarity and loss into account. Since ARDS only need one forward process and no backward process for each instance, ARDS is faster than gradient-based competitive methods. Extensive comparisons and ablation experiments are conducted to validate the effectiveness of ARDS.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The designed pipeline of ARDS is clear and intuitively meaningful. Authors make sufficient explorations in three key steps of ARDS: i) representation of conversation used for clustering; ii) identifying vulnerable samples as representations of clusters; iii) scoring mechinism that combines embedding similarity and average loss.\n2. According to experimental results, ARDS outperforms competing methods across multiple benchmarks and requires less time consume.\n\nWeaknesses:\n1. The experimental section lacks an analysis of the distribution of the selected samples across the clusters.\n2. The ablation settings are to some extent hard to understand.\n\nFor a more detailed explanation, please refer to the Questions section below.\n\nQUESTIONS:\n1.\tAccording to Equation 6, the value score of an instance is computed as an aggregation of the average cluster loss and the similarity between the instance and the cluster. Typically, there are some clusters with low average loss and others with significantly higher average loss. Clearly, instances that belong to or are closer to high-loss clusters tend to receive higher value scores. As a result, the selected samples are likely to be predominantly from these high-loss instances. My concern is that, in extreme cases, if the majority of selected samples come from such high-loss clusters, the model might partially or even largely forget the knowledge associated with low-loss clusters due to their underrepresentation in training. Therefore, I believe a statistical analysis of the selected samples across clusters is necessary, but the paper lacks this component.\n2.\tI do not understand how the ablation study in Table 6 is conducted. If no perturbation is applied, then x′ does not exist—how is Equation 4 computed in that case? And if clustering is not performed, how is the value score calculated? The paper lacks necessary clarification regarding this part of the experimental setup.\n\nIf my concerns are addressed, I would consider raising my rating.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a data selection strategy for the SFT stage of MLLMs. The core idea is to utilize embeddings from a warmed-up proxy model to identify hard samples by measuring model consistency under random input variations, and then selecting data points around these identified hard samples.\n\nSTRENGTHS AND WEAKNESSES:\n+ It is encouraging to see that principles from active learning and data selection, such as coreset selection and consistency/uncertainty metrics [1, 2, 3], can be straightforwardly adapted and remain effective in the context of MLLMs.\n- The manuscript would benefit from a more thorough discussion of its relationship to the broader field of active learning and data selection. Given the direct application of concepts from this area, explicitly positioning the work within this context would provide valuable perspective.\n- The set of comparative methods is insufficient. The paper would be substantially strengthened by including comparisons against more recent and relevant baselines, such as D3M, which appears to be closely related, and ICONS [5], a direct extension of LESS. Furthermore, the evaluation should be expanded to include a wider range of standard MM benchmarks.\n- A more detailed analysis of the time consumption is needed. This analysis should critically compare the proposed method's efficiency against strong baselines, particularly random sampling. And meantime take the performance gains into consideration (eg. comparison done by COINCIDE).\n- Table 2, as the primary results table, is presented in a way that could be misleading. Following the established practice in MLLM data selection literature, the main results table should focus on performance comparisons against the full data pool and other selection methods under standard, unaltered evaluation benchmarks. It is unfair to evaluate competing methods on an augmented evaluation set where \\textbf{the specific augmentations are leveraged by the proposed method but are unseen by the baselines}. The analysis of robustness should be presented in a separate table to ensure a fair and clear comparison of core performance.\n\n[1]Coresets for Scalable Bayesian Logistic Regression\n[2]Consistency-Based Semi-supervised Active Learning: Towards Minimizing Labeling Cost\n[3]Not All Labels Are Equal: Rationalizing the Labeling Costs for Training Object Detection\n[4]box-level active detection\n[5]ICONS: Influence Consensus for Vision-Language Data Selection\n\nQUESTIONS:\n- How many augmented samples are generated for each seed instance during the selection process? What is the sensitivity of the proposed method's performance and computational cost to this number? A brief ablation study on the number of augmentations would be valuable.\n- The analysis is primarily conducted on the SEED and ScienceQA benchmarks. Could the authors provide a more detailed justification for this specific choice?\n- What is the performance of the proposed method on Vision-FLAN, the widely used dataset for selection?\n\nLIMITATIONS:\nThe Limitation section currently reads more like a concluding summary and a brief outlook on future work. I recommend that the authors revisit the limitation section after expanding their experimental evaluation. This would allow for a much deeper and more valuable discussion.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-30"}

comments: SUMMARY:
This paper focuses on developing data selection method to alleviate the data bias and supious correlation in the visual instruction tuning data mixture. The paper is well-written and the experimental section is well-organized. The proposed method ARDS is evaluated in serveral aspects and many benchmarks.

STRENGTHS AND WEAKNESSES:
Pros: 
- The proposed method is gradient-free.
- The paper is well-organized.
- The paper evaluates the ARDS extensively across various benchmarks and settings.

Cons:
- The main concern is that in a real scenario, one cannot obtain the whole training dataset at one time, if new data comes, one should recalculate the worst-case evaluation groups again, which would cost a lot. Ideally, we would like a data selection method that could directly give a decision on each candidate data before training and inference.

- If you compress the 30% data again, how small will the final data mixture be?

Small typo errors: There is no Figure 4 in the main paper. Figure 4 in Line 271 cannot be clicked.

QUESTIONS:
See weakness.

LIMITATIONS:
Please add the limitation discussion at the end of paper.
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-07-01