NeurIPS 2025

Data Selection Matters: Towards Robust Instruction
Tuning of Large Multimodal Models
Xu Yang1
Chen Liu† 1
Ying Wei† 2
1 City University of Hong Kong
2 Zhejiang University
xyang337-c@my.cityu.edu.hk
chen.liu@cityu.edu.hk
ying.wei@zju.edu.cn
Abstract
Selecting a compact subset of visual instruction–following data has emerged as
an effective way to align large multimodal models with human intentions while
avoiding the high cost of full-dataset training. Yet we observe that both full-data
training and existing state-of-the-art data selection methods tend to inherit underly-
ing dataset biases such as position bias and spurious correlations, leading to biased
model behaviors. To address this issue, we introduce ARDS, a robustness-aware
targeted visual instruction-selection framework that explicitly mitigates these weak-
nesses, sidestepping the need for access to downstream data or time-consuming
gradient computation. Specifically, we first identify the worst-case evaluation sub-
groups through visual and textual task-specific perturbations. The robust training
mixture is then constructed by prioritizing samples that are semantically closer to
these subgroups in a rich multimodal embedding space. Extensive experiments
demonstrate that ARDS substantially boosts both robustness and data efficiency for
visual instruction tuning. We also showcase that the robust mixtures produced with
a smaller model transfer effectively to larger architectures. Our code and selected
datasets that have been demonstrated transferable across models are available at
https://github.com/xyang583/ARDS.
1
Introduction
Large multimodal models (LMMs) [23, 9, 30, 123, 64, 115, 75] have garnered significant attention
due to their strong zero-shot capabilities, enabling a wide range of applications in real-world scenarios.
Visual instruction tuning [64] has proven effective in enhancing these models’ abilities to follow user
instructions and reason deeply based on visual cues. Recent efforts have demons...