Data Selection Matters: Towards Robust Instruction Tuning of Large Multimodal Models
Authors: Xu Yang
Chen Liu
Ying Wei
Venue: NeurIPS 2025 poster ()
Keywords: Large Multimodal Models
Data Selection
Robustness
Primary area: general_machine_learning
TL;DR: We design an efficient data selection approach to construct a robust training mixture for visual instruction tuning in large multimodal models.

Abstract: Selecting a compact subset of visual instruction–following data has emerged as an effective way to align large multimodal models with human intentions while avoiding the high cost of full-dataset training. Yet we observe that both full-data training and existing state-of-the-art data selection methods tend to inherit underlying dataset biases such as position bias and spurious correlations, leading to biased model behaviors. To address this issue, we introduce ARDS, a robustness-aware targeted visual instruction-selection framework that explicitly mitigates these weaknesses, sidestepping the need for access to downstream data or time-consuming gradient computation. Specifically, we first identify the worst-case evaluation subgroups through visual and textual task-specific perturbations. The robust training mixture is then constructed by prioritizing samples that are semantically closer to these subgroups in a rich multimodal embedding space. Extensive experiments demonstrate that ARDS substantially boosts both robustness and data efficiency for visual instruction tuning. We also showcase that the robust mixtures produced with a smaller model transfer effectively to larger architectures. Our code and selected datasets that have been demonstrated transferable across models are available at https://github.com/xyang583/ARDS.

Data Selection Matters: Towards Robust Instruction Tuning of Large Multimodal Models
Xu Yang'
Chen Liut
Ying Weit
City University of Hong Kong 2 Zhejiang University
xyang337-c0my www.ccittyy.......mm..... edu. hk
chen. www.ciituui....m edu.
ying. wei@zju edu. cn
Abstract
Selecting a compact subset of visual instruction-followinng data has emerged an effective way to align large multimodal models with human intentions while avoiding the high cost of full-datased training. Yet we observe that both full-data training and existing state-of sore-oofthhe of-the-art art data selection methods tend to inherit underly- ing dataset biases such as position bias and spurious correlations, leading biased model behaviors. To address this issue, we introduce ARDS, mobustness-awar targeted visual instruction-selection framework that explicitly mitigates these weak- nesses, sidestepping the need for access to downstream data or time-consuming gradient computation. Specifically, we first identify the worst-case evaluation sub- groups through visual and textual task-specific perturbations. The robust training mixture is then constructed by prioritizing samples that are semantically closer to these subgroups in rich multimodal embedding space. Extensive experiments demonstrate that ARDS substantially boosts both robustness and data efficiency for visual instruction tuning. We also showease that the robust mixtures produced with a smaller model transfer effectively to larger architectures. Our code and selected datasets that have been demonstrated transferable across models are available https///gttbbb..ommm/ygg//////
Introduction
have garnered significant attention due to their strong zero-shot capabilities, enabling wide range applications in real-world scenarios. Visual instruction tuning (64) has proven effective in enhancing these models' abilities to follow user instructions and reason deeply based on visual cues. Recent efforts have demonstrated that improved performance after visual instruction tuning can be achieved by adjusting image resolutions (59]. optimizing training pipelines 75 and scaling up the data used during pre-training and instruction
Large multimodal models (LMMs) [23 9 30
123
64
115 75
tuning 99 23 32 53
63
11
Despite these advances, existing LMMs often experience sigmificant performance degradation when exposed to minor input perturbations [118] [25]. For instance, as illustrated in Figure [] LLaVA- 1.5 63), trained on the original dataset, suffers nearly 32% drop in accuracy on the ScienceQA benchmark [70] when answer choices are shuffled or option letters are replaced. Such vulnerabilities are often attributed
dataset biases that inadvertently encourage shorteut learning or spurious 25 80 To mitigate the biases, past works have employed post-training strategies such as prompt-basee based in-context learning [8I] 28 24 or inference-time calibration [[18].
correlations [31 90 92 96
However, such methods are limited by researchers' prior and incur additional computational costs during inference. Therefore, we hope to be able to prevent LMMs from learning biased behaviors during visual instruction tuning.
Corresponding authors.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Q: Which type force from the baby's hand opens the cubinet door?
Answer: The answer
(A) pull (B) push
permutation
model biased behavior
symbol attack
Q: Which type force from the baby's
Answer: The answer is W.
hand opens the cabinet door?
(W) push (Q) pull
Souual-nstruction-TTe
LMM
Iastraction Tuning Performance Science
Visual Instruction Tusing BoolQ
80
original maining mixcur
80
uriginal taining misture maiillggi
robust maining
Aceunicy (40
Accurcy (N))
11.63
10
10
symbol
symbul--ermaattion attack
symbol
yymbol-permutattin wltack
atlack
attack
allack
attack
(b)
Figure 1: (a) Illustration of biased behaviors exhibited by LMMs. (b) Robustness of
tuned LLaVA-1.5-70 on multimodal task (left) and LLaVA-1.5-13B on pure-text task (right) under symbol and permutation attacks. The results highlight significant decline in accuracy due to underlying dataset biases, and catastrophic forgetting further amplifies the vulnerability. Our curated robust training mixture enhances the model's robustness.
In this paper, propose constructing robust training mixture improve the robustness of visual instruction tuning without modifying the training pipeline or increasing inference costs. The key challenge lies in how recognize the dataset biases and select training samples that mitigate the vulnerabilities of LMMs. To overcome this challenge, we aim to identify specific training examples that reduce the model's 's worst-case error. Formally, data selection method, dubbed as ARDS, to prioritize training on data that contribute most to the worst-case subgroups. Inspired yet different from recent targeted instruction-follwiin data selection methods [07]. our method utilizes training sample representations rather than gradients. Technically, first extract the conversation vector from each training sample to build vector database. After that, we perform hierarchical clustering to group a holdout set in the embedding space, from which construct worst-case evaluation subgroups by applying perturbations identify samples most sensitive to biased model predictions. We then select training samples that are most similar to these specific subgroups. Our method introduces the aforementioned innovations that distinguish from prior targeted data selection methods [107]: (1) ARDS fully leverages the rich hidden representation of LMMs, avoiding computation ienttaationnnnenivveee gradient-based information measures, which renders it more efficient than prior methods. (2) Instead of requiring few-shot examples in downstream tasks, ARDS adopts deep clustering and perturbation strategies to maintain the merits of zero-shot generalization capabilities. The overall procedure is summarized in Algorithm |
introduce simple but effective targeted
Our contribution can be summarized follows:
We propose gradient-free robustness-aware data selection framework (ARDS) for enhancing the robustness and data efficiency of visual instruction tuning.
To represent each training sample consisting of multi-turn conversations, the conversation vector is introduced via attention-score weighted mechanism. The worst-case evaluation subgroups
are constructed by clustering and task-specific perturbations, which are leveraged to identify high-quality training samples.
The robust training mixture curated using a small model demonstrates strong cross-model transfer- ability, significantly improving robustness when applied training a large model.
Extensive experiments across eleven evaluation benchmarks validate the effectiveness of our curated robust training mixture. Notably, with only 30% of the training data, ARDS improves robust accuracy by up to 20.62% on multimodal tasks compared to state-of-the-art data selection methods, demonstrating that ARDS can improve both efficiency and robustness.

---

2 Related work
Visual Instruction Tuning for Large Multimodal Models. Large multimodal models (LMMs)
have witnessed significant advancements and evolved into versatile general-purpose assistants [8
102
8 30
56
9 64
63
75
[15]. A key component of the progress is visual instruction
tuning, which aligns LMMs to interpret visual content and diverse user prompts while enabling remarkable zero-shot generalization across rision-languaa
tasks [[3 64 63 Among existing
approaches, LLaVA 66
63 [4] is distinguished by its data efficiency: whereas foomer-bbsee
models require 129M only 665K conversation examples [64
23 to 1.4B 2 teate-text pairs, LLa achieves superior performance with
63 54]
The training examples are carefully transformed
and synthesized from established image-text corpora [1] 73 93 78 97 via text-only GPT-4 (2].
The result underscores key insight of visual instruction tuning: a high-quality instruction dataset can endow an LMM with superior visual perception and reasoning capability, lessening the reliance on ever-larger training corpora [3]
Data Selection for (Visual) Instruction Tuning. cestruction-oolloinng data selection methods aim to identify& a smaller, yet representative, subset of training data that can achieve comparable or even superior performance compared to using the full dataset, demonstrating promising results
in improving data efficiency for the alignment of instruction tuning [ 57
62
13
66
107 39
111
14
665
50,
17
104 3 [103 and visual instruction tuning [05] 15 68
106
As shown in
Table
1 existing state-of-ther oofthh--rtt data selection methods can be categorized based on the information
proxy, objective and dependence on targeted downstream task [86] 77. Based on gradient- based
influence function [49 85 82), LESS [07] introduces targeted instruction selection framework,
modifying the gradient inner product with cosine similarity to select high-quality subsets responsible for downstream error reduction. Subsequent work eliminates the downstream data requirement with self-influence score (68] and extends to the multi-task setting with majority vote of multiple single task selection [106]. While yielding strong performance, these methods still require computationally expensive gradient information over the full training set. In contrast, we customized targeted instruction selection [107 for improving the robustness against specific dataset biases while avoiding gradient-based drawbacks. Another line of work leverages feature embedding to capture data representation and perform task-agnostic selection based on predefined scores about the model prediction [87 83 74], distance to cluster centroids [100], as well as cluster transfcrability and density 51). Although effective, they demand a good feature representation space. For example, COINCIDE [51] utilizes a reference model well-trained on the full dataset. In addition, the robustness overlooked by previous methods, which is the focus of this study. Moreover, our perturbation- based strategy also contributes to the clean performance. We leave the detailed comparisons in Appendix C
Table I: Comparisons of existing visual cnerruction-fllowwiin data selection methods with large multimodal models. Information Proxy indicates the representation used to compute the information measure. Objective means the selection goal emphasized when ranking samples. Task-Aware Selection denotes methods explicitly target specific task. Downstream -aata-fraee marks no downstream task samples are required during selection.
Method
Information Proxy
Objective
Task-Aware Selection Saansstrram-Dtt---ee
LESS [077
Gradient
Quality
ICONS 1066
Gradient
Quality
TIVE (68]
Gradient
Diversity
COINCIDE 51
Feature
Diversity
ARDS (Ours)
Feature
Robustness
Data Selection for Robustness. Recent studies have shown that deep neural networks often leverage
dataset biases for vulnerable high performance on specific tasks, such as visual spurious features [88]
99). language spurious association (36), position and stereotype bias [92] 80). There is a series of
works that leverage data selection enhance the model's robustness and mitigate biased behavior before training or after training. For example, DoReMi [108] trains proxy model using Group DRO [89 and prioritizes training on valuable and learnable samples with higher excess loss of the proxy model (i.e., the difference between training and validation loss). 24 proposes a causal- -guieed method to recognize samples that contain biases, which are used as negative examples to suppress bias behavior via in-context learning. Most related to our work is D3M 44 which leverages TRAK (2) to remove specific examples by measuring gradient- based influence on the worst-group loss. However, our gradient-free approach enables extremely efficient and robust data selection for large multimodal
3

---

models. To the best of our knowledge, we are the first to leverage data selection for improving the large multimodal model's robustmess against spurious correlation and position bias.
Preliminary
Problem Formulation. We focus improving both data efficiency and robustness in visual instruction tuning for large multimodal models. After tuning, the model's zero-shot generalization capability is evaluated on clean as well as perturbed inputs across downstream tasks. This requires the model to consistently interpret the visual cues and follow user instructions to answer the question. During training, the model is optimized on a training corpus consisting of multi-turn conversations img
where æ; imn a!!" and ar denote the image, instruction, and answer in each conversation æi, respectively. We hypothesize that vulnerabilities of model behaviors, such position bias and spurious correlation, arise from underlying dataset biases. Thus, we address this issue via a simple but effective data selection strategy. Formally, given target model f with parameters Φ. our goal is to build robust training mixture guided by a proxy model fe with parameters 0, where ||0]| = || or even Our selection objective is minimize the worst-case error of f on downstream tasks.
{ where T is the total number of turns and æ, (x
||o]|
proposes two-stage instruction -ostloctionfoolooving aligmment procedure. In the vision- Dision-language alignment pretraining stage, MLP moosl connector is trained to project the visual features extracted by the CLIP encoder to the language embedding space. After that, in the visual instruction tuning stage, the decoder layers of the large language model are optimized a supervised fine-tuning manner. The training objective auto-regressive teacher forcing with the eross-entropy loss l forecasting the Q answer tokens.
Visual Instruction Tuning, LLaVA 64
Q
1 Q
l = log Po x
img
x
uns
<<)
LESS 107 builds the gradient datastore by calculating gradients of all trainable parameters for each training sample gir at. where & denotes the average loss of all tokens of i-th training sample using the proxy model fa- The influence of a training sample x; is quantified by the cosine similarity between its gradient and the gradients of the downstream few-shot examples and the maximum similarity over all few-shot examples as its final influence score
Targeted Data Selection. Inspired by the influence function 37 49
ag
I (zx) max cos (giv: Bearisla):
(2)
where a/e denotes the gradient of the j-th downstream few-shot example.
Methods
The schematic illustration of our adversarial representation-bast data selection is demonstrated in Figure 2 In Sec. 4.1 we present the whole of our data selection pipeline. Then we detail each step of our proposed method in Sec.
4.1 Overview
We aim to improve the robustness of large multimodal models after visual instruction tuning while minimizing the required amount of training data. Two key criteria are enforced in the building of a robust training mixture. First, the selection proxy used to determine which samples to retain should effectively preserve the most substantial and representative information from the original data. Selecting irrelevant or uninformative samples could even exacerbate existing dataset biases or spurious correlations in downstream tasks. Second, the data selection algorithm should avoid excessive computational overhead. Pursuing training efficiency with fewer samples via computation- intensive selection strategies (e.g., relying on full-parameter gradients) might make the proposed algorithm impractical, particularly for large multimodal models, which typically encounter each

---

Li M Decoder Lager
Aggregation
Perturbation
Clustering
Scores
Calculate
SOOO
Visuaa Encedler Layer
Visual Encoder Layer
Decoler Layer
GOOO
worst-case evaluation subgroups
robust training mixture
conversation
datablase
Figure 2: ARDS Pipeline: (1) Conversation Vector Database: ARDS begins with extracting
token embeddings for each conversation and then aggregates the substantial information within the conversation based on attention scores. (2) Worst-case Evaluation Subgroups: The hierarchical clustering performed in the embedding space. The perturbation is applied to find representative samples that are most vulnerable to model-biased behavior. (3) Quality Score Measure: The training sample scores are measured by distance with each worst-case subgroup. Subsequently, the highest-scoring samples are selected to curate the robust training mixture.
training sample only once during fine-tuning with epoch [63]. To resolve these challenges, propose to leverage hidden representations as lightweight yet expressive proxies for measuring the contribution of each training sample. The key intuition is to construct robust training mixture that prioritizes training samples semantically similar to vulnerable samples in the worst-case subgroups. To this end, first build vector database by extracting the conversation vector for each training sample and then curate worst-case evaluation subgroups to evaluate potential biases. Finally, the large multimodal models are instruction-tune tunnod the constructed robust training mixture avoid biased behaviors and enhance robustness against identified vulnerable subpopulations.
4.2 Data Selection for Robust Visual Instruction Tuning
Conversation Vector Database. Large multimodal models are often trained on samples consisting of multi-turn conversations. To represent each training sample, we introduce an attention-score weighted mechanism that aggregates the conversation vector from the token-level embeddings based on their relevance. Formally, for an input with L tokens, we first extract token embeddings H and then evaluate the importance of previous visual and textual tokens by examining the attention score. Specifically, in the multi-head self-attention block of a typical transformer, the attention score of each head h is calculated by A,, Q K, / /dl,, where Q and K,, the query and key embeddings, respectively. dę is the head dimension. We then obtain the average attention score across heads E RRLLL Let H, E Rixd, 1 E {1..... L.) represent the token hidden states at the final embedding layer of the proxy model, where T. is the sequence length and d is the hidden dimension. For each sample, the attention scores corresponding to tokens before the last token are extracted, i.e., AL E R1x(L-1), each element represents the relative importance a token in the sequence. The visual and textual tokens preceding the last token then aggregated through the weighted sum, H Σi=1 AL.: H,. The final conversation vector for each sample is obtained by concatenating the last token and the aggregated token:
H H].
where r E 2 The design of the conversation vector leverages both the information from the last token and the contextual relationships between tokens, as encoded by the attention mechanism, providing a robust and rich representation for each training sample. Unlike gradient-based data selection methods involving both forward and backward computation, our method needs only one forward to build the vector database over the training corpus.
Worst-case Evaluation Subgroups. We first partition the training set into semantically coherent subgroups via hierarchical clustering over the instruction embeddings. Specifically, using the es- tablished conversation vector database, we perform spherical -menss (5) clustering to yield M subgroups. Let C.. denote the m-th subgroup, where m {1,.... M). After clustering, we aim to find samples that are most susceptible to the model's biased behavior. Drawing inspiration from the crucial role of support vectors shaping the decision boundary 40), we aim to apply perturbation to
5

---

identify vulnerable samples near the decision boundary. Concretely, we impose diffusion noise [22 on the image. which more easily to trigger different erroneous predictions than other visual corruptions [124 122 For the textual variations, we inject task-aware perturbations designed to improve robustness against specific attacks. To this end, we apply dual perturbations to obtain the corrupted conversation z'. We then retrieve topg conversations with the largest loss difference for each cluster, i.c.,
S, tOp fx Cm :[((x) (x')),
(4)
where é is the cross entropy loss Eq.(D and t denotes the operation of selecting B highest- scoring samples. We can then leverage the balanced worst-case subgroups S {S SM to quantify the model bias by model performance degradation. To evaluate the difficulty for each worst-case subgroup, we use the average loss fs... = B Ejes... lz. Note that we only incorporate perturbations to construct worst-case evaluation subgroups and do not apply any intervention on the original training data.
Robust Coreset Selection. To quantify the importance of each training sample in reducing biased behaviors, we aim to capture the extent to which training samples contribute most to improving the worst-case performance. Unlike the previous method [44) that removes training samples leading to high worst-case enror via gradient information, we assess training sample quality based on its distance with our built worst-case evaluation subgroups in the embedding space. Specifically, for the i-th training sample and m-th worst-case evaluation subgroup, we first measure the cosine similarity between their conversation vectors,
1 Σ cos (Ti (ruir's.)
dism..
(5)
B JES
44 use softmax- aggregation to obtain the information value as Then, follow below
ΣM Em=1 exp (fs..) dis...
I(x;)
(6)
ΣM exp (ls..)
Robust Training Mixture. After the quality score measure, we select training conversations with the highest scores to build a robust training mixture D Drobust- The target model fo is fine-tuned on the Drubus using the training objective I.. The complete algorithm is shown in Algorithm of Appendix[[] We provide the theoretical analysis in Appendix[]
5 Experiments
In this section, we first elaborate on the experimental setup Section [5.1 proposed ARDS with the latest state-of-thh--ue methods to evaluate the effectiveness in Section 5.2
Then we compare our
Additionally, we present ablation studies in Section 5.3 and conduct a broader analysis in Section 5.4
5.1 Experimental Setup
66 for our robust Implementation Details. We use the original training corpus LLaVA-665K
training mixture curation without introducing any external data. After data selection, visual instruction tuning is performed with the same training configuration for all instruction selection baselines. Following LESS [107], we use a warmed-up LLaVA-1.5 (7B) model as our proxy model for data selection, which is trained on the subset of data with randomly sampled 1000 examples for 4 epochs. We set the number of clusters K to 70 and the subgroup budget B to 50. When building the worst-case evaluation subgroups and measuring robustness during test time, the adversarial permutation attack (PA) is injected by generating all possible permutations via brute-force algorithms (i.e., k! permutations, where k is the number of options) and the symbol attack (SA) is incorporated through replacing the standard symbols used answer choices (e.g.. A/B/C/D) with a different set of characters (e.g., Q/W/E/R). We refer to Appendix for a more detailed description of selection, training, and evaluation. We conduct a comprehensive hyperparameter study in Appendix E
Baselines. We compare ARDS with several baseline methods. Specifically, the random sampling selection, denoted by Random, randomly selects a subset from the original training mixture. We also compare with the gradient-based targeted selection method LESS [107], which leverages downstream

---

Table 2: Zero-shot robust accuracies (%, 1) against spurious correlation and position bias. ARDS and the baselines select same size training data for the visual instruction tuning of LLaVA-1.5 (7B) [6]] with the same training configurations. SA: symbol attack. PA: permutation attack. For each evaluation task, results surpassing Full method are highlighted in bold and the optimal result achieved among all curated training mixtures is underlined.
SckaadiA
Mkched
Porentane
Clean
SA DA
Clus
PA
PX
Clos
P4
5k-
Clos
54- -X-
DaPe
100.70
b6.30 54
80.15
37.05
30.57
50.00
43.52
b4.53
22 01
63.91
74.54
11.15
M0:30
43.00
01.02
19.50
32.34
03:33
50.14
50 155
L [83]
13%
56 03
66. 10
31.95
55.95
55 80
36-30
52:35
15 10
09.00
29.18
58 90
68 54
14.51
16.08
13.58
03:05
28 P4
20 15
conscie
51
52.23
12.27
57.40
38.52
30.54
73.75
56.65
37.65
20.45
48.54
01.51
20.97
ABDS leare
306
69:40 40
08.97
47.00
61.21
16:11
40:73
$6.83
01
40.80
74.43
60.00
72.07
50.22
70:48
53.73
00.00
00.00
Hate
A-DKYOX
MMMMUU
ABC-F
ReatQ
Fall
D005
[50.22 02
17.00
16. 04
55.05
71.54
35.00
$5.15
33:65
1.54
10.90
26.06
11:11
$0.55
0.23
15
33.77
23.64
4.50
0.00
16:20
aess [197
19%
79:00
194
74.00 41
37. 05
13:93
31:35
6.49
27-09
13.07
15.05
22:41
17.06
00.56
09-38
3:37
50-27
06.86
55.00
34.00
5:31
32:33
3:29
38.21
7:40
39
18:44
43:79
8-41
17:80
C.61
23:00
COINCIIE
T3
14.10
$4.87
0.64
JAKENS Seare
3959
79351
11:00
17.65
12.05
37.04
12.75
154:44
0.81
22.45
18:08
32.35
0.25
23.82
55.15
58.63
40.35
17.28
322.2
labeled few-shot examples for task- selection. Note that build the gradient store in LESS and our conversation vector database. Moreover, we compare our method with the RHO-LOSS (76) implemented by directly training the proxy model our
the same proxy model to
holdout worst-case subgroups to score training data with the excess loss [76] [08] 62). For the recent state-of-the-art coreset selection method COINCIDE [51], we reproduce its best version leveraging
the TinyLLaVA-2B [120] as the proxy model, which has already been well-trained on full training
data. The implementation details of the baseline methods are elaborated in Appendix D
5.2 Comparison with Baselines
Effectiveness of the robust training mixture. We evaluate visual instruction tuning on the mixture with the size of 30 % training data selected by ARDS and by several baselines. As Table 2 shows, ARDS consistently outperforms all competitors on a diverse set of downstream evaluation tasks Concretely, ARDS yields the largest robustness gains while matching the full-data model on the clean performance for multimodal input (c.g., MMMU, A-OKVQA). pure text input (e.g.. ARC, BoolQ) and cross-lingual tasks (c.g.. MMBench-CNN underscoring its generality. Compared to representative task-aware selector LESS [077. our method delivers higher robustness and data-efficiency without requiring downstream few-shot examples or expensive gradient information. Moreover, recent state- of-the-art task-agnostic method COINCIDE [5]] remains susceptible to symbol and permutation attacks. The performance of baseline RHO-LOSS [76] is less than satisfactory, which uses proxy model trained on the worst-case subgroups to score examples via the excess loss, probably because fitting an optimal large multimodal model on the small held-out set is intrinsically hard. In contrast, ARDS removes the biased samples and constructs the robust training mixture by utilizing worst-case evaluation subgroups to pay more attention to samples near the decision boundary.
Table 3: Cross-Architectur-SSeiil Transferability, Our robust training mixture curated via LLa 1.5 (7B) generalizes effectively to larger architectures, inducing strong robustness improvement when trained on selected data for LLVVA-1.5(13B)
Taga
Sabati
Date
305
16.25
$4.00
$3.75
31
55.01
15.00
10000
$2.00
1611
11.00
75.75
55.00
09.74
26.00
00.20
13.33
$3.06
00.26
20.74
Taga
Date
MORNGA
MADII
ARC4
Beil
Sundon
305
75.74
0951
71.11
30.00
19.04
13.00
35.03
20:30
13.00
17.00
40.07
7.0
27.00
11.57
35.40
03.00
05.07
30000
Scaling the model. To test cross-model-scall transfer, we fine-tune the larger 1L. (13B) on the robust training mixture curated with the approximately 2x smaller proxy model LLL 155..B).)) The results in (3 from weaker models to larger, more powerful models. For example, with only 30% of the training data, ARDS enhances the target model's robustness against the strongest symbol plus permutation attacks by 1.93% and 3.70% on ScienceQA and SEED-Bench respectively. We also observe that train-test modality mismatch further exacerbates the model's vulnerability under input variations, especially for larger models on text-only benchmarks, which may result from catastrophic forgetting [116] 60
that ARDS effectively transfers the curated robust training mixture
7

---

The performance of Vicuna-v1.5-133 [19 on BoolQ gradually degrades as the amount of vi- sual instruction tuning data increases (see Ap- pendix H.8). Interestingly, simple random selec- tion alleviates this degradation, and the mixture produced by ARDS surpasses it and delivers the largest robustness gains. The results highlight that our robustness-oriented selection essen- tial for reliably scaling LMMs.
34.11
203 Pacc
2032 Lol Paot
Traring Micart Sio
Tranisg Miczer Size
(4) ScienceQA
(b) SEED
Scaling the data. We next examine how ARDS behaves across varying training mixture sizes in Figure 3 Randomly discarding samples pro- vides no robustness benefit and often incurs loss. contrast, the robust mixtures selected by ARDS deliver consistently higher robustness at every budget. See Appendix for more figures.
Figure 3: Robust Accuracies (t) across differ- ent sizes of training mixture used for visual in- struction tuning of LLaVA-1.5 on ScienceQA and SEED-Bende benchmarks. Robustness is evalu- ated through incorporating symbol and adversarial permutation attacks.
Improving robustness against visual spurious correlation. We further evaluate whether ARDS mitigates biases that arise when particular visual attributes or textual descriptions dom- inate the training corpus, creating spurious shortcuts [88] 31 90 [58] A classic example is a
model that tends to answer "yellow" for any banana, even when the image shows a green one. Following the protocol of [47]. we construct
Table Zero-shot robust accuracies (%, 1) against
the robust training mixture targeted for GQA-
visual spurious correlation.
OOD, which defines OOD samples as infrequent events and creates fine-grained shifts by extract- ing questions from the most imbalanced answer groups of GQA 42] We build our worst-case subgroups using the validation split (See Ap- pendix more details). As Table vi- sual instruction tuming with our ARDS achieves the highest accuracy and robustness against the subpopulation shifts on OOD tasks, surpassing both the randongsampling baseline and COINCIDE [51), state-of-the-art diversity-oriented selector.
Selection Method
Data Percentage
GOA OOD-Hea
Original
OOD-ABI
00D-Tail
Avp
Full
100%
61.94
07.51
61.17
51:35
58.04
Random
50%
56.47
COINCIDE
50%
61.88
57.32
(ours))
505
62:48
58.44
02.26
52.21
68.84
5.3 Ablation Study
Effectiveness of conversation vector. We compare our attention score- weiihted conversation vector with alternative vectorization strategies. A prevalent baseline in retrieval- -ugmeeted generation (RAG) the last-token embedding, which captures aggregated semantic information [ [12]. However, the single last token often overlooks rich contextual cues from preceding turns, particularly in multi-turn or multimodal settings. contrast, our approach leverages the attention map to re- weight all preceding visual and textual token embeddings according to their learned importance. The design explicitly captures which parts of the conversation the model attends to when forming its final prediction, thereby preserving cross-turn dependencies and text-vision interactions that the last-token embedding fails to encode. To ensure a fair comparison, we re-implemented this baseline to build the vector database and reran our selection pipeline. As shown in Table [5] the attention- attention-weightee conversation vector better preserves more significant and useful semantics and achieves higher worst-case performance across benchmarks, demonstrating clear advantage over the simpler last-token representation, especially under limited training data budget.
Table 5: Ablation study results comparing conversation vector variants.
Comenation
Dabs Pereentage
SeimmrzOA SA
SFFRB-Borch
STMBBerb-EN
A-OKY04
Vector
Cron
Avp
Cran
PA
SA
PA
Aup
Chan
SA SA
PA
Aup
Cran
PA.
SA.
PS
Aug.
64.45
74.59
Amortion
55.85
$2.35
53.56
388
50.38
73,07
50,90
57.10
77.90
7004
76.91
71.90
Las Tiken
30%
08.11
55.35
05.00
13
18.12
18-25
11.47
M6.13
33.07
36-17
75.84
82.00
13-48
53.22
06.15
18.96
11.15
15.8
63.50
72.65
Atteation Wiighh
30R
69.36
10.10
68.77
47.00
61:21
M8.11
00.33
96.3
31.50
36.00
74.85
6105
12.37
55.22
66.36
78-34
11.09
7181
61.12
72.95
The effect of worst-case evaluation subgroups. ARDS builds worst-case evaluation subgroups through deep clustering to group semantically similar conversations, followed by dual perturbations to identify samples near the decision boundary. To demonstrate the necessity of each component, we conduct the ablation study by removing the components one by one in Table Specifically, in the first
8

---

row, we randomly sample the same number of samples M as the final total size of the worst-cass subgroups, and aggregate the importance score of each training data via a sample-to-sie to-samll weighted
MBB exp(t;)-dd
sum, formally given by I (z,) = Σ exp(t,) For the second row, we apply Eq.44 to directly retrieve top-M B samples from the training dataset with the largest loss difference, and use the caaple-to--amplle weighted-sum rule to obtain the importance score. The third row denotes our final construction strategy, which first performs clustering and then retrieves top- candidates from each cluster via Eq. (4) The importance scores aggregated using the sample-to-clusee weighted-sum scheme defined in ( Eliminating either component markedly reduces robustness, demonstrating the benefits of clustering and perturbation strategies in discovering diverse and challenging subgroups. Furthermore, finer-
finer-grained grained ablation in Table 11 of Appendix G highlights the effectiveness of visual and textual perturbations for the robustness gains.
Table 6: Ablation study results comparing different components for worst-case evaluation subgroups. LLa 1.5 (7B) is utilized as the proxy and target model for all methods.
Bard-carr care Erattution Sebgnou
Dara Permstage
ScieeeeDA
SEED-BBechh SA
MiMBlecheE
A-DKEVQ SA
Patterhaions
Chatciing
Clean
Ara.
Chon
SA
Ana
Clam 73.75
PA
Ang.
Clee
PA
Avg.
15.00
51.05
22.30
16.75
35.40
53.01
18.35
57.00
VaLLa
41.44
17.13
66.11
74.08
(770)
06.03
100 26
$5:40
08.77
47.00
61.21
16.11
00.75
09.69
33.82
09.80
76.46
05.03
73.37
50.22
66.30
19.44
71.00
17.64
64:75
172.0
Score aggregation strategy. We conduct an ablation study comparing two score aggregation ap- proaches in our robustness- selection. The Subgroup Maximum strategy follows the Equation 2 used in targeted instruction tuning [107]. where the information score I(x;) is computed as the maxi- mum cosine similarity between training sample and any single worst-case subgroup. The Subgroup Weighted Sum approach considers both subgroup similarity and subgroup difficulty. See Appendix G for more details. As shown in Table of Appendix G. the weighted sum strategy outperforms the maximum-basee aggregation, yielding higher robust accuracy under more challenging attack settings. This suggests that incorporating subgroup difficulty helps select training samples that more effectively target model-biase behaviors.
5.4
More Analysis
Data Selection Computational Analysis. To conduct the computational analysis. we first compare with task-aware selection baselines LESS [107], using the same VL. (7B) warmed up for four epochs for a fair comparison. As shown in Table 7 LESS extracts LoRA gradi- ents for each training sample along the discrete training trajectory, which is computationally ex- pensive. By contrast, ARDS encodes each con- versation with only the final-layer embedding, completely avoiding per-sample gradient calcu- lations and cutting the wall-clock time by wide margin. Furthermore, our runtime is comparable to COINCIDE, which extracts representations from five layers in roughly 100 minutes.
Table 7: Data selection clock time comparison. All experiments are conducted on 8 Nvidia RTX A6000 GPUs Stagel: Warmup training of the proxy model. Stage2: Gradient store or conver- sation database built. Stage3: Sample quality measure.
Method
Stagel
Stage2
Stage3
T.FSS [107
8min
128h
Imin
ARDS (Ours)
Smin
30min
1min
The impact of LoRA parameter-efficien fine-tuning. We use Low-Rank Adaptation (LoRA) 41 for parameter-efffcientt visual instruction tuning our main experiments. We here study the impact of this training approach on the robustness of large multimodal models. As shown in Table 12 of Appendix H.I we observe that LoRA tuning only 4.6% of the parameters achieves better robustness with comparable clean performance. This outcome aligns with the findings in [16].
Generalization More Challenging Benchmarks. We further evaluate the generalization capability of our curated robust training mixture on several more challenging benchmarks, especially two math- related visual reasoning benchmarks, MathVistz [69 and DynaMath [166. These two mathematical benchmarks represent more difficult Out-of-Domain (OOD) tasks, as the original LLaVA-6651 dataset does not contain explicit mathematical training data 63 (See Appendix H.. for more details.) As shown in Table 13 of Appendix H.2 our curated robust training mixture consistently enhances average performance across OOD tasks for visual instruction tuning, outperforming both full-data training and previous tatte-of-theaas data selection strategies.

---

Generalization to Other Large Multimodal Models. To assess the generality and effectiveness of ARDS, we further conduct a transferability study across a range of backbone architectures beyond the dicuna-based VA-VVA-1.5. Specifically, we apply ARDS to two representative atate-of-the-art large multimodal models,
M--Mssalll [45] (denoted L.LVV--Mistrl) and Qwen2.5-VI.- Instruct [10] (denoted Qwen2.5-VIL VL.) We refer to Appendix H.4 for more training details. As summarized in [f Appendix (H.4) ARDS consistently enhances robustness and data efficiency across architectures, yielding clear gains in both visual instruction tuning and post-training settings. The results collectively demonstrate the adaptability of ARDS and can be easily applied to various large multimodal models to achieve consistent robustness improvements.
Generalization under Unseen Attacks and Visual Corruptions. We here take a further step to investigate the robustness and generalization of our robust training mixture. We introduce two previously unseen symbol attacks and an additional visual corruption. Specifically, the canonical answer labels A/B/C/D are replaced by S/N/V/F and U///O/P, respectively, thereby injecting novel variations not encountered during data selection. Note that the used adversarial permutation attack is adaptively generated for each new question by enumerating all possible permutations of the answer options. As shown in Table 17 of Appendix H.6 ARDS consistently achieves strong robustness and generalization even in the presence of these new, previously unseen attacks. Even when compared with the COINCIDE+WOrSt-co scenarios, confirming that the robustness gains generalize well beyond attacks seen during data selection. Morcover, the stronger diffusion noise is injected into the test images to simulate corrupted visual input. The results in Table Io of Appendix [H.6] show that our ARDS maintains the highest robust accuracy across all scenarios, confirming that our robustness gains are not fully confined to the attacks used during selection.
subgroup. ARDS maintains superior robust accuracy across all
Complementary with Data Augmentation. We investigated whether lightweight tata-auumentation schemes can eliminate dataset biases To mitigate symbol-content spurious correlations and position biases, we apply random symbol replacement and option shuffling during visual instruction tuning. To address visual spurious correlation and subpopulation shifts, we apply Auto 22 to the training images. As shown in Table 18 of Appendix H.. though textual-level augmentations improve robustness against seen perturbations, they fail to generalize unseen attacks. Furthermore, simple image-level augmentation does not bring noticeably higher accuracy on shifted subgroups and even degrades clean accuracy on GQA. The observations mentations often preserve the statistical properties of the original data and can inadvertently amplify existing biases rather than mitigate them [1 95 79 In contrast, combining data augmentations with ARDS delivers the largest robustness gains, indicating our robust training mixture and augmentation work in synergy to yield stronger debiasing and robustness improvement.
consistent with recent findings that naive aug-
Effectiveness without Access to Full Training Data. We further evaluate ARDS in a practical scenario where the entire training corpus is not available at once, and data arrive dynamically over time. To investigate the influence of data volume on building worst-case evaluation subgroups, we randomly sample 10% of the training data as the initially available subset and treat the remaining 90% as newly incoming training data. See Appendix H.9 for more details. As shown in Table T9 of Appendix [..)] even when exposed to only one-tenth of the full corpus during subgroup construction, ARDS still achieves notable robustness gains, outperforming both LESS and COINCIDE. This demonstrates the potential and applicability of our approach to more dynamic data selection scenarios.
6 Conclusions
In this work, we introduce a simple yet effective gradient-free robustness-aware data selection approach for robust visual instruction tuning of large multimodal models. To enhance model robustness against underlying dataset biases, our method first constructs conversation vector database and then performs deep hierarchical clustering and applies dual perturbations build worst- case evaluation subgroups. By identifying training data points that are most similar to samples in each subgroup particularly those susceptible to biased model behavior we retrieve the sighint-scoring instances to curate the robust training mixture. Experimental results demonstrate that performing visual instruction tuning on our mixture with only 30% of original training data achieves a large robustness improvement and comparable clean performance compared to full-data-trained model. Future work includes broadening the definition of robustness beyond the current bias types and extending data selection for more diverse and dynamic real-world applications.
10

---

Acknowledgement
This work is supported by National Natural Science Foundation of China (NSFC Project No. 62306250) and City University of Hong Kong (CityU Project No. 9610614).
References
[1] Amro Abbas, Kushal Tirumala, Dániel Simig, Surya Ganguli, and Ari S Morcos. Semd- edup:
Data-efficient learning at web-scale through semantic deduplication. arXiv preprint urXiv:2303.0953 2023.
[2] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4
preee [3] Ishika Agarwal, Krishnateja Killamsetty, Lucian Popa, and Marina Danilevksy. Delift: Data efficient language model instruction fine tuning. In International Conference on Learning Representations, 2025.
technical report. arXiv preprint arXiv:2303. 08774, 2023.
[4] Sharat Agarwal, Himanshu Arora, Saket Anand, and Chetan Arora. Contextual diversity for active learning. In European Conference on Computer Vision, pages 137-153, 2020.
[5] Mohiuddin Ahmed, Raihan Seraj. and Syed Mohammed Shamsul Islam. The k-means algo- rithm: A comprehensive survey and performance evaluation. Electronics, 9(8): 2020. [6] Jean- Santttttistee Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lene, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. In Advances in Neural Information Processing Systems, 2022.
[7] Alon Albalak, Yanai Elazar, Sang Michael Xie, Shayne Longpre, Nathan Lambert, Xinyi Wang. Niklas Muennighoff, Bairu Hou, Liangming Pan, Haewon Jeong, Colin Raffel, Shiyu Chang. Tatsunori Hashimoto, and William Yang Wang. A survey on data selection for language models. Transactions on Machine Learning Research, 2024.
[8] Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, al. Openftamingo: An open- source framework for training large autoregressive vision-language models. arXiv preprint arXiv:2308.0 0112230880199. 01390, 2023.
[9] Jinze Bai, Shuai Bai, Shusheng Yang. Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308. 12966, 2023.
[10] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang. Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report, arXiv preprint 2xXx::2502.1922 2025.
[11] Randall Balestriero, Leon Bottou, and Yann LeCun. The effects of regularization and data augmentation are class dependent. Advances in Neural Information Processing Systems, 35:37878-37891, 2022.
[12] Alexander Bukharin, Shiyang Li, Zhengyang Wang, Jingfeng Yang. Bing Yin, Xian Li, Chao
Zhang, Tuo Zhao, and Haoming Jiang. Data diversity matters for robust instruction tuning. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, 2024.
[13] Yihan Cao, Yanbin Kang, Chi Wang, and Lichao Sun. Instruction mining: Instruction data selection for tuning large language models. In Conference Language Modeling, 2024.
[14] Lichang Chen, Shiyang Li, Jun Yan, Hai Wang, Kalpa Gunaratna, Vikas Yadav, Zheng Tang, Vijay Srinivasan, Tianyi Zhou, Heng Huang, and Hongxia Jin. Alpagasus: Training better alpaca with fewer data. In International Conference on Learning Representations, 2024.
[15] Ruibo Chen, Yihan Wu, Lichang Chen, Guodong Liu, Qi He, Tianyi Xiong, Chenxi Liu, Junfeng Guo, and Heng Huang. Your vision- vision-language model itself is strong filter: To- wards high-quality instruction tuning with data selection. In Association for Computational Linguistics, 2024.
[16] Shuo Chen, Jindong Gu, Zhen Han, Yunpu Ma, Philip Torr, and Volker Tresp. Benchmarking robustness of adaptation methods on pre-trained laniunne models. In Advances in Neural Informatinn Processing Systems, volume 36, 2024.
[17] Sang Keun Choe, Hwijeen Ahn, Juhan Bae, Kewen Zhao, Minsoo Kang, Youngseog Chung, Adithya Pratapa, Willie Neiswanger, Emma Strubell, Teruko Mitamura, et al. What is
11

---

your data worth to gpt? llm-scale data valuation with influence functions. arXiv preprint arXiv:2405. 139544, 2024.
[18] Christopher Clark, Kenton Lee, Ming- Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint a905 10044, 2019.
[19] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try are, the ai2 reasoning challenge. arXiv preprint arXiv: arXiv:
2018.
[20] Cody Coleman, Christopher Ych, Stephen Mussmann, Baharan Mirzasoleiman, Peter Bailis, Percy Liang, Jure Leskovec, and Matei Zaharia. Selection via proxy: Efficient data selection for deep learning. arXiv preprint arXiv: / 1829, 2019.
H Corinna Cortes and Vladimir Vapnik. Support-vector networks. Machine learning, 20:273-297,
[21]
1995.
[22] Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoe V Le. Autoaugment:
Learning augmentation policies from data. arXiv preprint arXiv: 0.805.0950. 2018.
[23] Wenliang Dai, Junnan Li, D Li, AMH Tiong, J Zhao, W Wang, B Li, P Fung, and S Hoi. Instructblip: Towards general- peneral-purpose lanounl models with instruction tuning, arxiv 2023, arXiv preprint arXiv:2305..c 06500, 2, 2023.
[24] Li Du, Zhouhao Sun, Xiao Ding, Yixuan Ma. Yang Zhao, Kaitao Qiu, Ting Liu, and Bing Qin. -auudel active learning for debiasing large language models. arXiv preprint arXiv:2408. /2942, 2024.
[25] Mengnan Du, Fengxiang He, Na Zou, Dacheng Tao, and Xia Hu. Shortcut learning of large
language models in natural language understanding. Communications of the ACM, 2023.
[26] Melanie Ducoffe and Frederic Precioso. Adversarial active learning for deep networks:
margin based approach. arXiv preprint
2018.
[27] Ismail Elezi, Zhiding Yu, Anima Anandkumar, Laura Leal-Taixe, and Jose M Alvarez. Not all labels are equal: Rationalizing the labeling costs for training object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
[28] Deep Ganguli, Amanda Askell, Nicholas Schiefer, Thomas Liao, Kamilè Lukošiūté, Anna Chen, Anna Goldie, Azalia Mirhoseini, Catherine Olsson, Danny Hernandez, et al. The capacity for moral self celf-correction in large language models. arXiv preprint 20X 2023,
[29] Mingfei Gao, Zizhao Zhang, Guo Yu, Sercan Ö Ank,, Larry S Davis, and Tomas Pfister. Consistency-based semi-supervised active learning: Towards minimizing labeling cost. In European Conference on Computer Vision, 2020.
[30] Peng Gao, Jiaming Han, Renrui Zhang, Ziyi Lin, Shijic Geng, Aojun Zhou, Wei Zhang, Pan Lu, Conghui He, Xiangyu Yue, et al. Llama-adapter v2: Parameter-efficicnt visual instruction model. arXiv preprint 2.0.:2234..511111 2023.
[31] Robert Geirhos, Hennik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. Shortcut learning in deep neural networks. Nature
Machine Intelligence, 2020.
[32] Tao Gong, Chengqi Lyu, Shilong Zhang, Yudong Wang. Miao Zheng, Qian Zhao, Kuikun Liu, Wenwei Zhang. Ping Luo, and Kai Chen. Multimodal-ppt: A vision and language model for dialogue with humans. arXiv preprint arXiv: 047900 2023.
[33] Mitchell A Gordon, Kevin Duh, and Jared Kaplan. Data and parameter scaling laws for neural machine translation. In Conference on Empirical Methods in Natural Language Processing. pages 5915-5922. 2021.
[34] Yash Goyal, Tejas Khot, Douglas Summers-Star Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2017. [35] Chengcheng Guo, Bo Zhao, and Yanbing Bai. Deepcore: A comprehensive library for coreset selection in deep learning. In International Conference on Database and Expert Systems Applications, pages 181-195, 2022.
[36] Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel R Bowman, and Noah A Smith. Annotation artifacts in natural language inference data. arXiv preprint arXiv:1803. 803800334,, 2018
[37] Frank R Hampel. The influence curve and its role in robust estimation. Journal of the American Statistical Association, 69(346):383-39, 1974.
[38] Muyang Hc, Shuo Yang, Tiejun Huang. and Bo Zhao. Large-scale dataset pruning with dynamic uncertainty. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 7713-7722, 2024.
12

---

[39] Yexiao He, Ziyao Wang, Zheyu Shen, Guoheng Sun, Yucong Dai, Yongkai Wu, Hongyi Wang. and Ang Li. Shed: Shapley-based automated dataset refinement for instruction fine-tuning, In Advances in Neural Information Processing Systems, 2024.
[40] Marti A. Hearst, Susan T Dumais, Edgar Osuna, John Platt, and Bernhard Scholkopf. Support vector machines. IEEE Intelligent Systems and their applications, 13(4))18-22 1998. [41] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2 106.09685, 2021.
[42] Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEF/CVE Conference on Computer Vision and Pottern Recognition, 2019.
[43] Jonathan Huggins, Trevor Campbell, and Tamara Broderick. Coresets for scalable bayesian logistic regression. Advances in Neural Information Processing Systems, 2016.
[44] Saachi Jain, Kimia Hamidich, Kristian Georgiev, Andrew Ilyas, Marzyeh Ghassemi, and Aleksander Madry. Improving subgroup robustness via data selection. In Advances in Neural Information Processing Systems, 2024.
[45] Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang. Timothée Lacroix, and William El Sayed. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023.
[46] Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. Referitgame: Referring to objects in photographs of natural scenes. In Conference on Empirical Methods in Natural Language Processing, pages 787-798, 2014.
[47] Corentin Kervadec, Grigory Antipov, Moez Baccouche, and Christian Wolf. Roses are red, violets are blue... but should vqa expect them to? In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 2776-2785, 2021.
[48] Krishnateja Killamsetty, Sivasubramanian Durga, Ganesh Ramakrishnan, Abir De, and Rishabh Iyer. Grad-match: Gradient matching based data subset selection for efficient deep model training. In International Conference on Machine Learning, pages 4464-5474, PMLR, 2021. [49] Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In International Conference on Machine Learning, pages 1885-1894, 2017.
[50] Yongchan Kwon, Eric Wu, Kevin Wu, and James Zou. Datainf: Efficiently estimating data influence in l-tnee LLMS and diffusion models. In International Conference on Learning Representationss 2024.
[51] Jaewoo Lee, Boyang Li, and Sung Ju Hwang. Concept-skill transferability-bade based data selection for large vision-language models. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. 2024.
[52] Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. Seed- bench: Benchmarking multimodal llms with generative comprehension. arXiv preprint arXiv:2307.161
2023.
[53] Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Fanyi Pu, Jingkang Yang, Chunyuan Li. and Ziwei Liu. Mimic-it: Multi-modal in-context instruction tuning. arXiv preprint
arXiv:2306.0535
2023.
[54] Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer. arXiv
preprint Xix: 200:.00330. 2024.
[55] Dongyuan Li, Zhen Wang, Yankai Chen, Renhe Jiang, Weiping Ding, and Manabu Okumura. A survey on deep active learning: Recent advances and new frontiers. IEEE Transactions on Neural Networks and Learning Systems, 2024.
[56] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models. In International Conference on Machine Learning, 2023.
[57] Ming Li, Yong Zhang, Zhitao Li, Jiuhai Chen, Lichang Chen, Ning Cheng, Jianzong Wang. Tianyi Zhou, and Jing Xiao. From quantity to quality: Boosting IIm performance with self- guided data selection for instruction tuning. In Association for Computational Linguisties, 2024.
[58] Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. arXiv preprint arXiv:2305. 10355, 2023. [59] Zhang Li, Biao Yang. Qiang Liu, Zhiyin Ma, Shuo Zhang, Jingxu Yang, Yabo Sun, Yuliang Liu, and Xiang Bai. Monkey: Image resolution and text label are important things for large
13

---

multi-modal models. In Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition, pages 26763-26773, 2024.
[60] Ji Lin, Hongxu Yin, Wei Ping, Pavlo Molchanov, Mohammad Shoeybi, and Song Han. Vila: On traininin for visual language models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2024.
[61] Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European Conference on Computer Vision. 2014.
[62] Zhenghao Lin, Zhibin Gou, Yeyun Gong, Xiao Liu, Ruochen Xu, Chen Lin, Yujiu Yang. Jian Jiao, Nan Duan, Weizhu Chen, et al. Not all tokens are what you need for pretraining. In Advances in Neural Information Processing Systems, 2024.
[63] Haotian Liu, Chunyuan Li. Yuheng Li, and Yong Jae Lec. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
2024.
[64] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In Advances in Neural Information Processing Systems, volume 36, 2024.
[65] Liangxin Liu, Xuebo Liu, Derek F Wong, Dongfang Li, Ziyi Wang, Baotian Hu, and Min Zhang. Selectit: Selective instruction tuning for Iims via uncertainty-awee self-refiection. Advances in Neural Information Processing Systems, 37:97800-978885
2024.
[66] Wei Liu, Weihao Zeng, Keqing He, Yong Jiang, and Junxian He. What makes good data for aligmment? comprehensive study of automatic data selection in instruction tuning. In International Conference on Learning Representations, 2024.
[67] Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang. Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-moda model an all-around player? arXiv preprint Xxiv:2307.06281, 2023.
[68] Zikang Liu, Kun Zhou, Wayne Xin Zhao, Dawei Gao, Yaliang Li, and Ji- Rong Wen. Less is more: High-value data selection for visual instruction tuning. arXiv preprint arXiv:2403.09552 2024.
[69] Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi. Hao Cheng, Kai-Wer Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. In International Conference on Learning Representations, 2024.
[70] Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. In Advances in Neural Information P Processing Systems, 2022.
Chun Zhu, Oyvind
[71] Mengyao Lyu, Jundong Zhou, Hui Chen, Yijie Huang, Dongdong Yu, Yaqian Li, Yandong Guo, Yuchen Guo, Liuyu Xiang, and Guiguang Ding. Box-level active detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.
[72] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu.
Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations, 2018
[73] Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge. In Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition, 2019.
[74] Max Marion, Ahmet Üstün, Luiza Pozzobon, Alex Wang, Marzieh Fadaee, and Sara Hooker.
When less is more: Investigating data pruning for pretraining Ilms scale. arXiv preprini arXiv:2309.0m 04566, 2023.
[75] Brandon McKinzie, Zhe Gan, Jean-Philippe Fauconnier, Sam Dodge, Bowen Zhang, Philipp Dufter, Dhruti Shah, Xianzhi Du, Futang Peng, Anton Belyi, et al. Mml: methods, analysis and insights from multimodal IIm pre-training. In European Conference on Computer Vision, pages 304-323. Springer, 2025.
[76] Sören Mindermann, Jan M Brauner, Muhammed Razzak, Mrinank Sharma, Andreas Kirsch, Winnie Xu, Benedikt Höltgen, Aidan N Gomez, Adrien Morisot, Sebastian Farquhar, et al. Pri- oritized training on points that are learnable, worth learning, and not yet learnt. In International Conference on Machine Learning, pages 15630-15649, 2022.
[77] Baharan Mirzasoleiman, Jeff Bilmes, and Jure Leskovee. Coresets for data-efficient training of machine learning models. In International Conference on Machine Learning, pages 6950-6960.
PMLR, 2020.
[78] Anand Mishra, Shashank Shekhar, Ajeet Kumar Singh, and Anirban Chakraborty. Ocr-vqa: Visual question answering by reading text in images. In International Conference on Document
14

---

Analysis and Recognition, 2019.
[79]
Alhassan Mumuni and Fuseini Mumuni. Data augmentation: A comprehensive survey of modern approaches. Array, 16:100258, 2022.
[80]
Roberto Navigli. Simone Conia, and Björn Ross. Biases in large language models: origins, inventory, and discussion, ACM Journal of Data and Information Quality, 2023. Daisuke Oba, Masahiro Kaneko, and Danushka Bollegala. In-contextual bias suppression for large language models. arXiv preprint arXiv:2309. 00055, 2023.
[81]
[82] Sung Min Park, Kristian Georgiev, Andrew Hyas, Guillaume Leclerc, and Aleksander Madry. Trak: Attributing model behavior at scale. In International Conference on Machine Learning, 2023.
[83] Mansheej Paul, Surya Ganguli, and Gintare Karolina Dziugaite. Deep learning on data diet: Finding important examples early in training. In Advances in Neural Information Processing Systems, volume 34, pages 20596-20607, 2021.
[84] Jeff M Phillips. Coresets and sketches. In Handbook of discrete and computational geometry. pages 1299-288. Chapman and Hall/CRC, 2017.
[85] Garima Pruthi, Frederick Liu, Satyen Kale, and Mukund Sundararajan. Estimating training data influence by tracing gradient descent. Advances in Neural Information Processing Systems,
33:19920-19933 19930, 2020.
[86] Yulei Qin, Yuncheng Yang, Pengcheng Guo, Gang Li, Hang Shao, Yuchen Shi, Zihan Xu, Yun Gu, Ke Li, and Xing Sun. Unleashing the power of data tsunami: A comprehensive survey on data assessment and selection for instruction tuning of language models, Transactions on Machine Learning Research, 2025.
[87] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages 8748 8763, 2021.
[88] Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. "why should trust you?" explaining the predictions of any classifier. In Proceedings of the ACM SIGKDD international conference on knowledge discovery and data mining, pages 1135-1144, 2016.
[89]
Shiori Sagawa, Pang Wei Koh, Tatsunori B. Hashimoto, and Percy Liang. Distributionally robust neural networks. In International Conference on Learning Representations. 2020.
[90] Shiori Sagawa, Aditi Raghunathan, Pang Wei Koh, and Percy Liang. An investigation of why overparameterization exacerbates spurious correlations. In International Conference on Machine Learning, pages 8346-8356, 2020.
[91] Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. Social IQa: Commonsense reasoning about social interactions. In Conference Empirical Methods in Natural Language Processing. 2019.
[92] Timo Schick, Sahana Udupa, and Hinrich Schütze. Self-diagnosis and self-debiasing: A proposal for reducing corpus-based bias in nlp. Association for Computational Linguistics. 2021.
[93] Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. A-okvqa: A benchmark for visual question answering using world knowledge. In European Conference Computer Vision, 2022.
[94] Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: core-set approach. In International Conference on Learning Representations, 2018.
[95] Nima Shahbazi, Yin Lin, Abolfazl Asudeh, and HV Jagadish. Representation bias in data: A survey on identification and resolution techniques. ACM Computing Surveys, 55(13s): 1-39, 2023.
[96] Omar Shaikh, Hongxin Zhang, William Held, Michael Bernstein, and Diyi Yang. On second thought, let's not think step by step! bias and toxicity in zero-shot reasoning. arXiv preprint SXXi::2222000611 2022.
[97] Oleksii Sidorov, Ronghang Hu, Mareus Rohrbach, and Amanpreet Singh. Textcaps: a dataset for image captioning with reading comprehension. In European Conference on Computer Vision, 2020.
[98] Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019.
[99] Kun Song, Huimin Ma, Bochao Zou, Huishuai Zhang, and Weiran Huang. Fd-align: Feature discrimination alignment for fine- fine-tuning pre- trained models in fhw-shot learning. Advances in Neural Information Processing Systems, 36, 2024.
15

---

[100] Ben Sorscher, Robert Geirhos, Shashank Shekhar, Surya Ganguli, and Ari S. Morcos. Beyond neural scaling laws: beating power law scaling via data pruning. In Advances in Neural Information Processing Systems, 2022.
[101] Mariya Toneva, Alessandro Sordoni. Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, and Geoffrey J Gordon. An empirical study of example forgetting during deep neural network learning. arXiv preprint /
2018.
[102] Maria Tsimpoukelli, Jacob L Menick. Serkan Cabi, SM Eslami, Oriol Vinyals. and Felix Hill. Multimodal few-shot learning with frozen language models. In Advances in Neural Information Processing Systems, 2021.
[103] Jingtan Wang, Xiaoqiang Lin, Rui Qiao, Pang Wei Koh, Chuan-Sheng Foo, and Bryan Kian Hsiang Low. NICE: Non-differentiable evaluation metric-based data selection for instruction tuning. In ICLR Workshop on Navigating and Addressing Data Problems for Foundation Models, 2025.
[104] Jiachen Tianhao Wang, Tong Wu, Dawn Song, Prateek Mittal, and Ruoxi Jia. Greats: Online selection of high-quality data for Ilm training in every iteration. Advances in Neural Information Processing Systems,
11131197-13122
2024.
[105] Lai Wei, Zihao Jiang, Weiran Huang, and Lichao Sun. Instructiongpt-4: A 200-instruction paradigm for fine-tuning minigpt-4. arXiv preprint arXiv:2308. 12067, 2023.
[106] Xindi Wu, Mengzhou Xia, Rulin Shao, Zhiwei Deng. Pang Wei Koh, and Olga Rus- sakovsky. Icons: Influence consensus for vision-language data selection. arXiv preprint
urXiv:2501.0065 2024.
[107] Mengzhou Xia, Sadhika Malladi, Suchin Gururangan, Sanjeev Arora, and Danqi Chen. Less: Selecting influential data for targeted instruction tuning. In International Conference on Machine Learning, 2024.
[108] Sang Michael Xie, Hieu Pham, Xuanyi Dong, Nan Du, Hanxiao Liu, Yifeng Lu, Percy S Liang, Quoc V Le, Tengyu Ma, and Adams Wei Yu. Doremi: Optimizing data mixtures speeds up language model pretraining. In Advances in Neural Information Processing Systems, volume 36, 2024.
[109] Zhiyang Xu, Chao Feng, Rulin Shao, Trevor Ashby, Ying Shen, Di Jin, Yu Cheng, Qifan Wang. and Lifu Huang. Vision-flan: Sealing haman-labeled tasks in visual instruction tuning. In Association for Computational Linguistics, 2024.
[110] Shuo Yang, Zhe Cao, Sheng Guo, Ruiheng Zhang, Ping Luo, Shengping Zhang, and Liqiang Nie. Mind the boundary: Coreset selection via reconstructing the decision boundary. In Forty-first International Conference Machine Learning, 2024.
[111] Yu Yang, Siddhartha Mishra, Jeffrey N Chiang, and Baharan Mirzasoleiman. Smalltolarge (s21): Scalable data selection for fine-tuning large language models by summarizing training trajectories of small models. In Advances Neural Information Processing Systems, 2024.
[112] Shi Yu, Chaoyue Tang, Bokai Xu, Junbo Cui, Junhao Ran, Yukun Yan, Zhenghao Liu, Shuo Wang, Xu Han, Zhiyuan Liu, et al. Visrag: Vision-based retrieval-augmented generation on multi-modality documents. In International Conference on Learning Representations, 2025. [113] Yue Yu, Wei Ping. Zihan Liu, Boxin Wang. Jiaxuan You, Chao Zhang, Mohammad Shoeybi, bi, and Bryan Catanzaro. Rankrag: Unifying context ranking with retrieval-augmented generation in 1ls. In Advances in Neural Information Processing Systems, 2024.
[114] Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive disliidddeeiliiee multi- modal understanding and reasoning benchmark for expert agi. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 9556-9567. 2024.
[115] Haotian Zhang, Mingfei Gao, Zhe Gan, Philipp Dufter, Nina Wenzel, Forrest Huang, Dhruti Shah, Xianzi Du, Bowen Zhang, Yanghao Li, al. Mm1.5: Methods, analysis & insights from multimodal Ilm fine-tuning, arXiv preprint arXiv:2409.20556 2024.
[116) Yi-Kai Zhang. Shiyin Lu, Yang Li, Yanqing Ma, Qing-Guo Chen, Zhao Xu, Weihua Luo, Kaifu Zhang, De-Chuan Zhan, and Han-Jia Ye. Wings: Learning multimodal lims without
text-only forgetting. Advances in Neural Information Processing Systems, 2024.
[117] Bo Zhao, Boya Wu, Muyang He, and Tiejun Huang. Svit: Scaling up visual instruction tuning. 2023.
arXiv preprint arXiv:2307.04400
[118] Chujie Zheng, Hao Zhou, Fandong Meng, Jie Zhou, and Minlie Huang. On large language models' selection bias in multi-choie questions. arXiv preprint arXiv:2309.0m 03882, 2023.
[119] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. In Advances in Neural Information Processing Systems, 2023.
16

---

[120] Baichuan Zhou, Ying Hu, Xi Weng, Junlong Jia, Jie Luo, Xien Liu, Ji Wu, and Lei Huang. Tinyllava: A framework of scall-scale large multimodal models. arXiv preprint arXiv:2402. 14289, 2024.
[121] Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer. Jiao Sun, Yuning Mao, Xuezhe Ma. Avia Efrat, Ping Yu, Lili Yu, et al. Lima: Less is more for alignment. arXiv preprint arXiv:2305. /206, 2023.
[122] Yiyang Zhou, Chenhang Cui, Rafael Rafailov, Chelsea Finn, and Huaxiu Yao. Align- ing modalities in vision large language models via preference fine-tuning. arXiv preprint arXiv:2402. /41 2024.
[123] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny, Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304. 10592, 2023.
[124] Ke Zhu, Liang Zhao, Zheng Ge, and Xiangyu Zhang. Self-supervised visual preference alignment. In Proceedings of the ACM International Conference on Multimedia, pages 291- 300, 2024.
[125] Yongshuo Zong, Tingyang Yu, Bingchen Zhao, Ruchika Chavhan, and Timothy Hospedales. Fool your (vision and) language model with embarrassingly simple permutations. In Interna- tional Conference Machine Learning. 2024.
[126] Chengke Zou, Xingang Guo, Rui Yang, Junyu Zhang, Bin Hu, and Huan Zhang. Dynamath:
A dynamic visual benchmark for evaluating mathematical reasoning robustness of vision language models. In International Conference on Learning Representations, 2025.
17

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We evaluate the effectiveness of our method on eight benchmarks. Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the
contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? [Yes]
Answer:
Justification: See the Appendix A for details. Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: See the Appendix M for details. Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
18

---

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented
by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We elaborate our implementation details and evaluation metrics in Appendix D.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken
to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We public datasets, and we provide the dataset details in Appendix D. We will release the code upon acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/ public/guides httpe//guiiee//oeeSuummiise CodeSSumisssionooll
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not
19

---

reproducible, they including code, unless this is central to the contribution (e.g., for new open-souree benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
httpsscccccblccccce cecicc/guiiiee /ttp://///ueee//eeuuuuubiiii CodeSubmissss wwwiebbmisssiiioi ssionoliccy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why. At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We include the training details and evaluation metrics in Appendix D. Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary to appreciate the results and make sense of them. The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments? Answer: [No]
Justification: The variance of experimental results smaller than the gap between different methods.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula,
call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
Question: For each experiment, does the paper provide sufficient information on com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes]
20

---

Justification:
We include details in Appendix D.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics ptttps::/neuuiis... http///uursp.pppppppeee cctppblic///cccccccccccccccccclt Cetiiic///tiiittiie Ethiishuidelines!
Answer:
[Yes]
Justification: We conform with the NeurlPS Code of Ethics. Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics. If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g., there a special consid-
eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: See the Appendix B for details. Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal
impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymeet of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
21

---

Released models that have high risk for misuse or dual-use use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We cite the original paper for datasets used and include dataset details in Appendix D.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of use in the wwworwewtthhheeeeeeee package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
this information is available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets. Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and sereenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
22

---

According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality the research, declaration not required.
Answer: [Yes]
Justification: The paper studies efficient data selection for LLMs, and the methodology designed for LLMs is significantly different from the applied to traditional machine learning models.
Guidelines:
The answer NA
that the method development in this research does involve LLMs as any important, original, or non-standard components. (https:
Please refer to our LLM for what should or should not be described.
23

---

Data Selection Matters: Towards Robust Instruction Tuning of Large
Multimodal Models
-Supplementary Material-
Limitations
25
Broader Impact
25
More Related Work
25
Detailed Experimental Setup
26
Hyper-parameter Studies of ARDS
27
More Results Across Training Data Scales
27
More Ablation Studies
28
More Analayses
30
H.1
Comparison Between Full-model and LoRA Fine-tuning
30
H.2
Results of Generalization More Challenging Benchmarks
30
H.3
Results on Open--peeeeeeeee Ended Generation Benchmarks
30
H.4
Generalization to Other Large Multimodal Models
31
H.5
Statistical Analysis of the Loss Distribution in Worst-case Evaluation Subgroups
31
H.6
Generalization under Distribution Shifts and Visual Corruptions
32
H.7
Complementary with Training Data Augmentation
32
H.8
Analysis of Performance Degradation on Text-only Tasks
33
H.9
More Results in Settings without Access to Full Training Data
33
More Results on Other Training Datasets
34
Robustness Evaluation of Large Multimodal Models
34
The ARDS Algorithm
34
J
Theoretical Analysis
34
J.1 Analysis the Relationship Between Data Vulnerability and Gradient Magnitude
35
0.2 Analysis on Targeted Instruction Selection for Robustness Improvement
37
J.3 Analysis on Alignment Between ARDS and Targeted Instruction Selection
37
24

---

A Limitations
In this paper, present robustness- aware visual instruction selection framework, paving path for enhancing the robustness of large multimodal models. Extensive experiments demonstrate that our approach improves robustness to symbol-content spurious correlations, positional bias, and fine-grained visual sub-population shifts across multiple downstream tasks. Although the efficacy of the proposed method has been confirmed, opportunities for refinement persist. One avenue for improvement involves a more diverse investigation into the potential dataset biases and their interaction effects. We also plan extend the ARDS to other post-training and test-time training scenarios where a few informative training samples are collected to align the model for diverse real-world applications.
B Broader Impact
This work presents a novel approach improving the robustness and data efficiency of visual instruction tuning for large multimodal models, which can have significant implications for reducing biases in AI systems. By mitigating the risks associated with dataset biases, our method could contribute to more fair, transparent, and trustworthy AI applications across a variety of domains.
C More Related Work
Coreset Selection. Coreset selection attempts to extract a high-quality and most informative subset of training data (84. Existing coreset selection methods can be mainly categorized into geometry-
based [94] 4), imcertainty-base 220 38), error-based [83] [01], gradient- based [77 [88, and decision
boundary-bast based methods [26] IIO. We refer readers to [35] for a more detailed literature compilation.
Our perturbation- based worst-case subgroup construction is closest to [T00] which used ad adversarial gradient ascent find samples near the decision boundary. However, our gradient-free approach enables efficient identification of vulnerable samples for robustness improvement.
Active Learning. Both data selection and active learning aim identify subset of data that yields the best possible model, while the distinction lies in the focus on settings. Active learning defaults to
that data is unlabled and emphasizes on reducing annotation cost [43 29 27 55) In contrast, data
selection methods usually have full access to all labels when selecting 77. and the goal is to improve training efficiency [107] or robustness
(44). Our method ARDS is a data selection method with no extra annotation required. Our focus is to boost robustness against specific dataset biases for visual instruction tuning while avoiding computation-intensii gradient-based calculation in [107 44).
Data Selection for Instruction Tuning. Recent instruction-folloiiig data selection methods show promising results that carefully chosen subsets align large language models (LLMs) as well
better than using the entire corpus, greatly improving data efficiency [21 14]
66
39
62 [07| 7
LIMA [21] was the first work demonstrate that roughly one thousand human-curated examples are sufficient for enabling LLM to follow instructions. Subsequent studies are proposed to use various information measure strategies and emphasize different aspects of data while eliminating human intervention. Some score-based methods directly utilize external closed-source models to score quality and complexity [14 666 emphasize diversity [12] when ranking instruction data, Other works introduce natural language indicators [13] or predictive uncertainty (65) for quality estimation, utilize condition- based losses to capture difficulty [57]. and compute Shapley value to quantify sample interactions [39]. Reference- based methods require a well-trained model for
calculating metrics. For example, RHO-1 [22] utilizes the excess loss [9 76 to selectively train on useful and learnable tokens. Our setting is more similar to targeted instruction tuning [107], which ranks candidate instructions by their gradient cosine similarity to a few examples from a specific downstream task. Our work extends this line of research to curate robust training mixtures. We select data that mitigates specific dataset bias without any gradient calculations, strengthening the robustness of visual instruction tuning.
25

---

D Detailed Experimental Setup
Training Datasets. We curate the robust training mixture by conducting visual instruction selection from the original 1A. dataset, which contains 665K multimodal conversations (3 collected from mixed sources, such as LLaVA-158K [64], VQAv2 34 OKVQA 73 RefCOCO 46], and TextCaps (97). These conversations cover broad spectrum of task types, including visual-question answering, detailed image/region caption, localization and complex visual reasoning. Our mixture is sampled directly from this corpus without altering the original data or introducing external examples. We refer readers to LLaVA [64] 63
for a more detailed dataset description.
Evaluation Setup. Following VLa
633 we assess the zero-shot generalization capability of visual instruction tuning on eight benchmarks including ScienceQA [70], SEED-Bench [52], MMBench (67], MMMU [114], BoolQ [18), ARC [19), A-OKVQA [93] and GQA 42). For
MMBench, we report the accuracy without circular evaluation because we apply stronger adversarial permutation attacks (introduced below). We inject symbol and adversarial permutation attacks as input variations for every evaluation task to probe model robustness against dataset biases, including symbol-content spurious correlation and position bias. The adversarial permutation attack (PA) is incorporated by generating all possible permutations via brute-force algorithms (i.e., k! permutations, where k is the number of options), which is cheaper than gradient-based search at small k. The symbol attack (SA) involves a large candidate space and we simply replace canonical choice labels (A/B/C/D) with an alternative set (Q/W/E/R), disrupting symbol-content shortcuts. Robust accuracy is defined as the proportion of tasks that the model answers questions correctly under all perturbations. We additionally evaluate robustness to visual shortcuts by following the GQA-OOD protocol [47). which constructs fine-grained subpopulation shifts for assessment.
Architecture. We consider 11. (3 in our main experiments which consists of a CLIP-VIT-L- 336 visual encoder (336 x 336) 87). a two-layer cross-modal MLP projector and Vieuna v1.5 [19 as the language backbone. We initialize the MLP projector with the official LLaVA pre-trained weights and focus on the visual instruction tuning stage. Only the projector and the large language model are updated during visual instruction tuning while keeping the CLIP encoder frozen.
Implementation Details of Data Selection. We follow LESS [107] and adopt a warmed-up LLaVA- 1.5 (7B) model as our proxy model for data selection. The warm-up stage fine-tunes the model on 1000 randomly sampled training examples for four epochs. With the warmed-up model, we build the iitention-weightee
database on the training corpus in which each vector is r E 22xd and the hidden state dimension d of the last embedding layer is 4096. To construct the worst-case evaluation subgroups, first apply hierarchical clustering conversation vectors from different multimodal tasks [68] (e.g., image captioning, regional description, open-ended visual question answering). We set the number of clusters K to 70 and the subgroup budget B to 50. To identify vulnerable samples near the decision boundary in subgroups, we inject task- specific perturbations for robustness against targeted dataset biases. Specifically, we impose diffusion noise [122] with the step size of 300 as the visual perturbation and inject symbol and permutation attacks to expose position bias and symbol-content shortcuts. For visual spurious correlation in GQA-OOD, we only incorporate diffusion noise. The robust training mixture is curated by selecting samples that contribute most to the worst-case subgroups. We then perform visual instruction tuning on this mixture using the standard LLaVA pipeline, leaving the original data and training procedure unchanged.
conversation-vectt
Implementation Details of Visual Instruction Tuning, We do not perform hyperparameter sweeps of visual instruction tuning for either our method the baselines; instead, we adopt the official LLaVA-1.5 configurations (63] so that results under full-data training and data selection baselines are directly comparable. The CLIP image encoder is kept frozen, whereas the language model is updated during supervised visual instruction tuning. We train for one epoch with a batch size of 128 and a learning rate of 2e decayed by a cosine schedule. To reduce the memory requirements, we apply Low-Rank Adaptation (LoRA) [1 to all linear layers in the multi-head self-attention (MHSA) A) and feedforward network (FFN) modules of Vicuna v1.5 [19]. A complete list of hyperparameters is provided in [8] At test time we use greedy decoding on every evaluation benchmark to ensure full reproducibility.
Baselines We provide a more detailed explanation of the baselines.
Random sampling method randomly selects subset from the entire training mixture.
26

---

Table Hyperparameters of visual instruction tuning
(a) 1.. (7B)
(b) LJaVA-1.55 (13B)
Hyperparameter
Finetune
Hyperparameter
Finetune
LoRA rank batch size
128
LoRA rank
128
128
batch size
128
Ir
2e-4
Ir
2e-5
Ir schedule
cosine decay
Ir schedule
cosine decay
warmup ratio
0.03
warmup ratio
0.03
weight decuy
weight decay
0
epoch
epoch
optimizer
AdamW
optimizer
Adam
DeepSpeed stage
DeepSpeed stage
3
LESS [107] modified gradient-based influence function that ranks training samples by the cosine similarity between their gradients and those of a few labeled downstream examples for targeted instruction tuning. The downstream examples are also leveraged for test-time in-context learning. For fairer zero-shot comparison, we only utilize downstream examples for data selection. We use the same warmed-up proxy model for LESS and our methods. To build the gradient store, all Adam-based LoRA gradients are extracted and concatenated for each training sample, which are then reduced to an 1992-dimensional vector using TRAK projection [82] LESS-SciQA denotes employing 100 examples randomly chosen from the ScienceQA validation split. The training samples with higher gradient cosine similarity to these examples are selected for visual instruction tuning.
We treat our worst-case evaluation subgroups as a holdout set and train the reference model on those subgroups. Each candidate training sample is then scored by the loss difference between the still-untrained target model and that of the reference model. Samples with larger excess loss, deemed both difficult and learnable, are chosen for iistal-instruction tuning.
RHO-LOSS 776 exemplifies the excess-loss family of selectors 76 108
62
COINCIDE [51) utilizes a proxy model, well-trained on the entire training corpus, to extract feature embeddings from five multi-head self-attention layers (i.e., 3, 7, 11, 15, 19 layers). The uniform clustering is then performed on these embeddings to form 10000 clusters. COINCIDE selects training samples from each cluster with an emphasis on diversity, drawing more samples from low- density and highly transferable clusters. We reproduce its best version with TinyLLaVa VA-2B [120] as the proxy model.
E Hyper-parameter Studies of ARDS
To better understand the sensitivity and design choices of our proposed ARDS method, we conduct controlled experiments on several key hyperparameters. Since adopt the same proxy model setup as the baseline LESS [107] for a fair comparison, we report the performance of reimplemented LESS with the selection ratio 5% in Table (a). The results show that training the proxy model for just 4 epochs on a small subset of 1,000 samples is already sufficient for effective data selection. The results in Tb) demonstrate that doubling the number of clusters from 70 to 140 does not lead to performance improvement, suggesting over-segmentation can dilute subgroup-lever signals, reducing the effectiveness of worst-case evaluation subgroup construction. As shown in Table moderate expansion of the subgroup selection budget helps capture more informative training examples that contribute robustness.
F More Results Across Training Data Scales
To better understand the performance of our ARDS under varying training budgets, we conduct robustness scaling analysis across different training mixture sizes-5% 10%, 30%, 50%, and 70% of the full dataset. As shown Figure 4 we observe that random subsampling fails to improve robustness across all scales. In many cases, it actually leads to performance degradation, especially under stronger perturbation settings. In contrast, both variants our ARDS consistently outperform basclines at every training size and enhance the robustness of visual instruction tuning, surpassing the
27

---

Table 9: Hyper-pane syper-paraemter studies of ARDS. (a) Effect of different warmup configurations for the proxy models. (b) Effect of cluster size in ARDS. (c) Effect of subgroup budget. denotes our choice.
(b) Cluster Size
(c) Subgroup Budget
(a) Warmup Training
Method ScienceQA
Method
GQA
Epoch #Sample ScienceQA
Clean Robust
Clean Robust
4
33265
67.79
1000
66.88
70"
69.26
47.60
20
61.75
51.18
140
67.58
46.26
50*
62.43
52.21
full-data trained model. We choose 30% as the default training budget in our main experiments, it offers the best trade-off between data efficiency and both clean and robust performance.
G More Ablation Studies
Score aggregation strategy. To better understand the impact of the score aggregation strategy used in our robustness-awand selection, we conduct an ablation comparing two approaches. The Subgroup Maximum strategy follows the Equation 2 used in targeted instruction tuning [I07], where the information score [(x;) is computed as the maximum cosine similarity between training sample and any single worst-case subgroup. The Subgroup Weighted Sum approach considers both subgroup similarity and subgroup difficulty. Specifically, we first compute the cosine similarity d.s... between the training sample and each worst-case evaluation subgroup Sm, then weight each similarity by the subgroup's 's difficulty ls... using softmax normalization. This weighting encourages prioritizing training samples that are close to more difficult subgroups, enabling the selection more informative and robustness- ceriticcl examples. As shown in Table TO, the weighted the maximum- based aggregation, yielding higher robust accuracy under more challenging attack settings. This suggests that incorporating subgroup difficulty helps select training samples that more effectively target model-biased behaviors.
strategy outperforms
Table 10: Ablation study comparing different aggregation strategics for robust data selection. LLa (7B) is utilized as the proxy and target model for all methods.
Score Aggregation Strategy
Data
ScienceQA SA
SEED-Bench SA
Percentage
Clean
PA
Avg.
Clean
Avg.
Subgroup Muximum
30%
70.05
37.61
68.12
43.88
39.91
57.23
41.02
56.13
30.65
46.25
Suhgroup Weighted Sum
30%
69.26
59.40
68.57
47.60
61.21
58.11
40.73
56.83
31.52
46.80
Impact of visual and textual perturbations. We conduct a finer-grained ablation to compare variants of our method using only diffusion-based visual noise or task-aware textual perturbations (PA/SA) against the final dual perturbation strategy. As shown in Table [1 results highlight the individual and combined contributions of visual and textual perturbations in constructing worst-case evaluation subgroups.
Table 11: Ablation study comparing different perturbations for constructing worst-case evaluation subgroups. LLaVA-1.5 (7B) is utilized the proxy and target model for all methods.
Textual
Visual Perturbation
Data Percentage
ScienceQA SA
BeE.D-Bench
Perturbation
Clean
PA
SA PA
Avg.
Clean
PA
SA
SA + PA
Avg.
10%
65.
45.61
59.61
54.08
37.82
$2.60
28.45
43.23
10%
68.52
54.09
66.83
47.05
59.12
54.34
37.64
53.72
32.42
44.53
10%
69.66
55.88
69.21
52.35
611.78
53.86
38.30
53.38
35.11
|45.16
28

---

SclencelyA Clean
SciezcoQAA PA
ScienceQAASAA
ScienceQAA SA-2A
47.6
65.5
64.0
12.5
60.5
02.0
65.0
67.5
57.5
51.0
Inter
20
08.5
Beuth Sqaarm
haltu (Tato
pa 15a77)
3100%:
50%
20%
17%
2505
33%
50%
70%
5700%
30%
3P%
3100%
20%
50%
12%
Training Misture Size
Thining Siee
Trahsing Mixture Size
Triining Mistree
SFFD-Scrch Clean
SFFD-Benck PA
SP3D-BRanch
SERD-8crrh SA+PA
525
476
425
vieecia Vden per
10
le fan cord
--o Seclleeii (u
40
Jabarn
Trairing Mistre Soz
Training Mixtare Size
Training Mistre State
Trining Mixture Size
MMBBerh EN Clean
MMBescc EN PA
MM EN SA
MMBBach EN SA-PA
72
aklruannii mish
5520%
50%
15%
53%
5000%
30%
50%
5507%
13%
59%
12%
Traizing Misture Size
Truining Mixtur Size
Traiting Mistur Size
Training Mixture Size
MMfBlench-C CN Clean
MMiieecch CN
MMBBench CN SA
MMBBench-C SA-PA
Godeo
mas Fall to mea moke 11
150%
10%
50%
19505
Wif Training
55415
13%
50%
12%
Triining
Toining
Trining
CTeur
PAOKVOA
A-OKVOASA
A-OKVQA SA-PA
72.5
70.0
6
153
67.5
0
725
20.0
$2.5
800
74
57.5
30
4104 Coa Maaas Phad
Mati Tike Ouct
<1etio Ilea Per
ABt Nike cled Dale 6.00
55.0
20
Training Misture Size
Training Mixture Size
Training Mistue: Size
Triinnn Misture Size
Figure 4: sero-shot robust accuracy (t) across varying training mixture sizes for visual instruction tuning of 1.5 on the ScienceQA, LEED-Bench. MMBench-Ei ENN MMBench-CN, A-OKVQA benchmarks. Our method consistently improves robustness across scales.
29

---

H
More Analayses
H.1
Comparison Between Full-model and LoRA Fine-tuning
In this section, we compare full-parameter fine-tuning with Low-Rank Adaptation (LoRA) 41 As reported Table 12 updating only 4.6% of the parameters via LoRA not only preserves clean-set accuracy but improves robustness. In our main experiments, visual instruction tuning with the full training corpus and every data selection baseline adopts the same LoRA configuration to ensure a fair comparison.
Table Robustness comparison between Full-model and LoRA LLa (7B) on the full dataset.
visual instruction tuning for
S4
SERD--Bench
MMBrnnc-EN SA
A-OKVQ4
Method
Pereentage
SA
Aug. Clean
PA
Avg.
Cham
Avg.
Clom
SA
+APPA
Arp.
Full Pull-aaRA
58.06
40.81
50.08
16,50
42.02
74.04
10.37
20.27 61.00
66.13
68005 21.35
1038
69.26 $4.1
65.71
$7.5
10,45
18,00
54.5
20.40
11.09
71.81
01.15
09.30
61.02
90.50
72.31
18.54
55.02
H.2 Results of Generalization More Challenging Benchmarks
We further evaluate the generalization capability of curated robust training mixture several more challenging benchmarks, including the text-only benchmark, SocialIQA [91]. and two math- related visual reasoning benchmarks, Vasta [69] and DynaMath [126]. These two mathematical benchmarks represent more difficult Out-of-Domain (OOD) tasks, as the original LLaVA-665K dataset does not contain explicit mathematical training data [63]. Specifically, Math Vista [69] assesses mathematical reasoning in visual complex scenarios (e.g., tables and function plots), while DynaMath 22 designed systematically analyze the robustness of mathematical reasoning across diverse topics (e.g., arithmetic, geometry and graph theory) under dynamic condition changes (e.g.. numerical value variants, geometric transformations and graph structure variants). Our proposed permutation and symbol attacks, serving as an orthogonal analysis axis, offer complementary perspective for diagnosing potential biased behaviors large multimodal models. shown in Table 13 our curated robust training mixture consistently enhances average performance across OOD tasks for visual instruction tuning, outperforming both full-data training and previous state-of-the-art data selection strategies.
Table 13: Zero-shot robust accuracies (%, against spurious correlation and position bias on additional benchmarks. Results are reported on the text-only SocialIQA and two visual mathematical reasoning benchmarks, Math Vath and MynaMath.
Selection Method
Data Pereentage
Chem
SoctallQA
Avg:
Math SA
SA
Arg.
Clean
DymaMath
-A
Avp
Full
100%
66.17
51.59
05.53
21.24
48.
40.37
16.48
34.
2.22
23.38
39.33
19.80
36.58
1.74
24.45
30%
52.97
1.56
30%
50.93
36.56
25.19
COINCIDE 5
60.20
H.3
Results on Open-Ended Generation Benchmarks
Inspired by LESS [107). ARDS is targeted data selection method designed to effectively and efficiently perform task-aware instruction tuning on a small selected subset of data once the conversa- tion vector database is established. However, ARDS explicitly aims to enhance robustness against intrinsic dataset biases in visual instruction tuning. In this section, assess the adaptability of our framework for open-ended generation tasks, we evaluate ARDS on two representative benchmarks, Text 98 and GQA 42). We denote our robustness-orfs -rrinnted variant in our main experiments as ARDS-Robust and the variant targeting open-ended generation capability
ARDS-OG. As shown in Table [4] our method achieves performance comparable to full-data training and previous data selectors optimized for clean data efficiency. We further observe natural trade-off between
30

---

robustness and generation capabilliy, A concurrent work ICONS [166] extends the line of targeted instruction selection toward multi-task selection via majority voting. We believe such approaches as complementary to our robustness- selection and could be integrated into our framework for exploring robustness-aware and multi-ohjective instruction selection, which we leave as future work.
Table 14: Comparison of data selection methods on open-ended ended generation capability.
Selection Method
Data Percentage
TextVQa
GQA
Full
100%
57.23
61.94
Random
30%
55.90
59.61
LESS-SciQA 107
30%
49.45
56.49
COINCIDE (51
30%
56.09
59.15
ARDS-Robust
30%
54.51
58.18
ARDS-OG
30%
56.97
60.32
H.4 Generalization to Other Large Multimodal Models
To assess the generality and effectiveness of ARDS, we further conduct a transferability study across range of backbone architectures beyond the Vicuna- LLaVA-1.5 used in our main experiments. Specifically, we apply ARDS to two representative state-of-the-art large multimodal models, LLaVA- 1.6-Mistral For
45 (denoted LLa L.LaVA--Msstaa and L0wen2.5-VL-Instrule [10] (denoted Qwen2.5-VL). we follow the official two-stage training procedure of LLa first pre- training the MLP projector and then performing visual instruction tuning. For Qwen2.55 Inntstucct whose pre-training data are closed-source and no publicly released pre-trained version is available, we directly perform post-training. All models are trained on the LLaVA-665K dataset using different data selection strategies. As summarized in Table 15 ARDS consistently enhances robustness and data efficiency across architectures, yielding clear gains in both visual instruction tuning and post-training settings. The results collectively demonstrate the adaptability of ARDS and can be easily applied to various large multimodal models to achieve consistent robustness improvements.
Table 15: Transferability across large multimodal architectures. The robust data mixture curated with Ticuna-based LL. (7B) transfers effectively to other architectures, including LLaVA-1. 1.6- Mistral (7B) and Qwen2.5-VL In--rrtrect (7B), yielding consistent robustness improvements across visual instruction tuning and post-training settings.
Proyy
Turget
Seletian
Date
SciecceAA
SEED Beach
MMBBeuck JNN SA
Porcestege
Arg.
7%
Ang.
L.L.M Mutra--aaaaa (7l)
Handivm
30/6
73.00
56.22
58.20
52:20
56.81
31.85
50,
14.05
30.05
25.31
56.51
322
58.54
LLANA-15 (28)
L.S.AA Mitrr (7B)
ARDS
30%
72.04
61.77
69.16
55.53
64 63
59.22
41.02
57.50
34.95
48.93
76:97
55.37
T5.11
55.31
68 1.
Qvart S-VL, (TB)
77.06
08.71
07.08
38.71
00.38
48.01
24.72
88.00
10.00
$4.35
21.31
$2.48
72 14
36.10
67:27
Qveat
LLaNA-155
S-V
Prwy
Turpet
Sduction
Date
MMBesch-CN SA
A-OKYQA
MMMM
Mold
Methoe
Porceatege
Clean
PA
Arg.
SX
PA
Ave.
SA
SA
112.% Mitiall (7m)
Randivm
08.30
09
57.57
13.17
47.02
61.31
73.
19.23
00.753
37.
12.51
35.33
3.07
22.07
1.30A-1.5. (30)
L13.A Miarall (7t)
ARDS
30%
72.36
57
70-32
51:24
02
61.00
72.58
50. 60.52
090 00.00
75.04
390
00
16.000
36.00 953
11.33
25.69
Quort S-VL (1B)
53.62
56.50
73.00
36.71
82.65
62.15
07.34
75.00
41.45
50.72
12.06
20.21
45.45
11:02
34.06
Qvert
41.79
LL
H.S Statistical Analysis of the Loss Distribution in Worst-case Evaluation Subgroups
To analyze the loss distribution of worst-case evaluation subgroups, we measure balance through Shannon entropy given as (ls) = - -M (fs... log p (Ls..) ). where p (fs..) is the weight of the m-th cluster after Softmax (fs.... As entropy depends on the number of subgroups, we further normalize entropy w.r.t. the number A of possible clusters: ē (fs) = equal to the entropy of uniform distribution of size M. We obtained the normalized entropy (es) 0.92. The result empirically verifies that selected samples remain well dispersed under the loss-based weighted-sum aggregation strategy. Furthermore, as shown in Table I0] the weighted-sum
I where log(Al) is
e(ts)
31

---

aggregation strategy yields a better clean and robustness trade-off and avoids the skew that the max variant exhibits.
H.6
Generalization under Distribution Shifts and Visual Corruptions
To further evaluate the robustness and generalization capability of our robust training mixture, we conduct experiments on previously unseen perturbations not used during data selection, including unseen symbol attacks and stronger visual corruptions. Specifically, the canonical answer labels A/B/C/D are replaced by S/N/V/F and U/////P, respectively, and the stronger diffusion noise is injected into the test images. The adversarial permutation attack is adaptively generated for each new question by coumerating all possible permutations of the answer options. As shown in Table [7] and Table 16 our ARDS maintains the highest robust accuracy actoss all scenarios, demonstrating that ARDS effectively prioritizes training examples that confer transferable robustness.
Table 16: Transferable robustness under image corruptions. SA: symbol attack, PA: permutation
attack. IC: image corruption.
Selection Method
Data Percentage
ScienceQA
Clean
SA+PA
Avg.
IC+PA
IC+SA
IC+SA+PA
Avg.
Full
100%
69. 76
54.34
65.74
37
56.87
64.75
47.84
61.33
31.38
51.32
Random
30%
52.60
59.44
23.75
51.39
67.43
47.60
55.83
20.13
17.74
ARDS (Ours)
30%8
69.26
59.40
68.57
47.60
61.21
66.04
53.94
64.80
41.55
56.58
Table 17: Transferable robustness under unseen attacks. PA: permutation attack, SA1: unseen symbol attack (S/N/V/F), SA2: unseen symbol attack (U/VO//). WCS: incorporate our worst-case evaluation subgroups during data selection.
Daes
SctmertJ
AIIAOO
Millkee--N
100%
67.72
60.93
73
16.09
12.00
56.02
25:47
55.81
19.56
30-23
77-21
52.
45.01
$3.42
16.21
31.61
35.23
Handen
305
64.00
10:30
03.00
33:24
17.03
53.57
47:23
22
65.25
26,55
16:34
43.16
01.38
62.5
21.30
45:33
CHnccii WTT
37:24
43
17:0
56.34
19:40
48.35
00
88:47
m
20
3:
05
31
(0.30
24:01
H.7
Complementary with Training Data Augmentation
We take further step to investigate the impact of simple training data augmentation mitigating dataset biases. To reduce symbol-content spurious correlation and position biases, we apply random symbol replacement and option shuffling as textual-level augmentations. To address visual spurious correlations and subpopulation shifts, we adopt AutoAugment [22] for image-level augmentation during training. As illustrated in Table [8] while textual augmentations help improve robustness against seen perturbations, they fail to generalize to unseen attacks. This underscores that simple augmentations alone are insufficient to resolve dataset biases. Our approach aims to curate a robust training mixture as a complementary perspective to augmentation. Combining our ARDS with data augmentation results in greater robustness gains, demonstrating their synergistic effect in debiasing model behavior. More critically, AutoAugment does not significantly improve performance on shifted subgroups, and in fact, harms clean accuracy on GQA and robustness GQA-OOD. This aligns with findings from recent studies, which show that simple data augmentations can generate problematic data and inadvertently amplify existing dataset biases rather than mitigate them, since augmentations typically preserve the statistical properties of the original training data
95 79
Table 18: Performance comparisons of different selection methods under the data augmentation during visual instruction tuning. The proxy model and target models are LLaVA- (7B). SA: symbol attack, PA: permutation attack, SA1: unseen symbol attack (S/N/V/F), SA2: unseen symbol attack (U/HO//).
Selection Methed
Training
Aag.
Prr. (Kr))
Batt
Prt. (50A)
Bata
[Jonn
PA
KA
SA-PA
Anp
Selenceg
SAL
+A
SA2
XAI+PA
Clon
0ID-AE
GDA ck0l-iteed
App
1075
1075
54.34
65.74
98.97
65.77
10:40
06.75
57.51
61:17
51.5 55
58.01
Bundocn
30%
306
50%
506
68:01
96
02.66
02.00
60.41
25.75
08.43
31:39
80
60.05
49:10
39:92
64.80
68:41
$4.31
51.09
10.00
58.05
55.97
$4.07
51.41
09.30
45.03
05.02
$1.99
66-07
(Duzs)
30%
50%
69:26
70.90
40
68.57
47.00
61.21
668.67
45:34
60.14
30.70
65. 70
16:40
58:44
02.20
02:21
65.64
32

---

H.8 Analysis of Performance Degradation on Text-only Tasks
We conduct detailed analysis of the observed performance degradation of the full-data trained model on pure-text tasks such as ARC-e and BoolQ. We hypothesize that this phenomenon stems from the catastrophic forgetting caused by modality imbalance in large-scale visual instruction tuning, which aligns with prior work that identified text-only forgetting during multimodal aligmment [116 60]. For example, WINGS [166] reports up to 13.33% text-only degradation after visual modality expansion. In LLLVA-665K, only around 40688/ / /65299666882 = 6% of samples are pure-text instructions. During large-scale multimodal tuning, the optimizer increasingly focuses vision-centric tasks and gradually forgets language knowledge acquired from text-only instruction tuning. To quantify this effect, we measure text-only forgetting by training random subsets of increasing size using LLaVA-1.5 (7B), as summarized in Figure Larger subsets are expected to include more image-text pairs and thus amplify modality mismatch between multimodal and text-only inputs. The results confirm that incorporating more vision-heavy data helps models excel in multimodal tasks while accentuates forgetting on text-only tasks.
80
78.25
79.04
80.00
80.79 80:52
74.32
72.05
70
64.10
Accuracy
60
58.93
$3.43
50
48.35
40
BoolQ
A-OKVQA
selo
10%
30%
50%
70%
90%
100%
Training Mixture Size
Figure 5: Comparison of multimodal and text-only task performance under different training data scales. As the training subset grows and includes more vision-heavy data, multimodal performance improves while text-only performance degrades, indicating modality-driven forgetting.
H.9 More Results in Settings without Access Full Training Data
In this section, we further evaluate ARDS in practical scenario where the entire training corpus not available at once, and data arrive dynamically over time. This setting differs from the static setup used in our main experiments, where data selectors have full access to the entire training set to construct gradient databases [107] or perform global clustering [5I]. To investigate the influence of data volume building worst-case evaluation subgroups, randomly sample 10% of the training data as the initially available subset and treat the remaining 90% as newly incoming training data. We denote the variant *RDS* Specifically, ARDS* performs clustering on the 10% subset using the same number of clusters K and subgroup budget B, followed by the same task-specific perturbations the ARDS. As shown in Table [9] even when exposed only one-tenth of the full corpus during subgroup construction, ARDS still achieves notable robustness gains, outperforming both LESS and COINCIDE. This demonstrates the potential and applicability of approach to dynamic data selection scenarios.
33

---

Table 19: Comparison results in dynamic data availability settings. ARDS", which constructs worst-case evaluation subgroups with access only 10% the training data, remains effective in improving robustness and outperforms LESS and COINCIDE, the latter of which performs clustering over the full dataset.
SclenceQA
Selection Method
Selection Ratio
Clean
PA
SA
SA + PA
Avg.
LESS T0ZI
30%
68.42
55.63
64.70
34,95
55.93
COINCIE S1
30%
67.72
52.21
61.08
28.06
52.27
ARDS (ours)
30%
69.26
59.40
68.57
47.60
61.21
A ARDS (ours)
30%
68.86
58.30
67.13
42.39
59.17
H.10 More Results on Other Training Datasets
We further evaluate the proposed ARDS on another large-scale instruction-tuning dataset, Vision- Flan [I09], assess its generalizability across different training sources. As shown in Table 20 the robust training mixture curated by ARDS on Vision-Flan [109] achieves superior worst-case robustness compared with full-data and random selection baselines. The results further validate the effectiveness of our robustness-awar data selection framework in mitigating the model's biased behaviors due spurious correlations and position biases across datasets.
Table 20: Zero-shot robust accuracies (%, f) against spurious correlation and position bias on Vision-Flan training dataset.
ScienceQA SA
Selection Method
Selection Ratio
Clean
PA
SA + PA
Avg.
Full Random ST
100%
64.06
39.71
53.59
13.73
42.77
50%
61,38
34.41
53,59
18.89
42,06
ARDS (ours)
50%
62.87
37.98
60.54
33.56
48.74
H.11 Robustness Evaluation of Large Multimodal Models
In this section, provide more detailed robustness analysis for the large multimodal model trained with standard visual instruction tuning. We perform a series of controlled evaluations by applying textual and visual variations the GQA benchmark [42] To study whether the large multimodal models learns instruction- followiinn capability or simply memorize the specific instruction format, we introduce InstructionPPerturb by prompting ChatGPT generate 20 alternative question instructions, such as word replacement, symbol injection, and paraphrasing. Importantly, all variants preserve the original semantic meaning. We then compute the average accuracy across these variants to evaluate how sensitive the model is to instruction format changes. For visual input perturbation, we apply a range of standard test-time augmentation techniques, including color jitter, brightness. sharpness, and spatial transformations (c.g.. rotate, shear, translate). Additionally, we evaluate the impact of diffusion noise, which has been shown more flexibly induce model output incorrect predictions [24 122]. We report the average robustness performance over diffusion steps 100, 200, and 300. As shown in F the model demonstrates moderate robustness to most common textual and visual variations. However, the model exhibits the most significant performance degradation under spurious subpopulation shifts (OOD-Tail 47) Also, the diffusion noise introduces subtle yet semantically preserving perturbations that can effectively disrupt model predictions, and its strength can be flexibly controlled through the number of diffusion steps. These findings motivate our design of robustness-oriente data selection and evaluation methods.
I The ARDS Algorithm
In Algorithm|] T mixture.
we outline our rohustness-aware data selection procedure to curate the robust training
Theoretical Analysis
34

---

Contrast 61.94
InstructionPerturb
Equalize 13
61.90
60
61,94
-ail
6585
Posterize
61.94
..00
55
61.94
Diffusion
51 55
Rotate
61
50
61.94
59.82
59:42
Sharpness
61.9
61.82
Solarize
9.
61.94
61.94
61.96
61.94
61,83
2.03
Brightness
Shear
62.21
61.94
Colorjitter
Translate
Original Clean Input
Input Variations (image & text)
Figure 6: Robustness of LLaVA-1.5 (7B) on the GQA benchmark under diverse input variations.
J.1 Analysis on the Relationship Between Data Vulnerability and Gradient Magnitude
We begin by exploring the relationship between data vulnerability and gradient magnitude. Intuitively, training samples that more vulnerable tend to exhibit higher gradient norms, their losses more sensitive to parameter updates. This motivates us to identify such samples through worst-case subgroup construction.
use linear models as examples to demonstrate the theoretical insights of our work. consider linear classifier r(x) Wæ where E Rdi is the input and W Rd Xdia represents the parameters. As a classification problem, we use softmax-cross feftmax-cross-entropy as the loss objective function: L(x, y) -y" log ÿ where softmax (w)) is the model's probabilistic output.
The gradient of L with respect to the model parameters calculated by:
OL
g(x, y)
(ī y)r
aw
The gradient of L with respect to sample æ calculated by:
ac
s(x y)
W
y)
(8)
dx
Assumption J.1. The input data is normalized. we have |a||| without the loss of generality.
Assumption J.I is benign assumption, data normalization is quite common in practice.
Theorem J.2. Under Assumption J.I and in binary classification, i.e., dout if data instance is adversarially perturbed by PGD, i.e., ò = 82.
On then the data vulnerability, defined by L(x v) L(x, y), increases monotonically with the gradient magnitude g(x, y)
Proof. Without the loss of generality, we assume the data (x, y) belongs to the first category. Based on the convexity of the loss function L, we have L(x + 6. 3) L(x, y) > 8 C(m.y) die
e||s(æ, y)
35

---

Algorithm Data Selection for Robust Visual Instruction Tuning (ARDS)
1: Require: Training corpus D of size N; Proxy model fo; Number of subgroups K; Subgroup budget B
2: Step I: Conversation Vector Database
3: for i = 1 to N do
4: Obtain token embeddings H; using fo at the last layer with the attention score matrix
5: Aggregate the visual and textual tokens preceding the last token using attention-score
weighted mechanism. H Σ Ar.t H,
6: Obtain the conversation vector r; = H H
7: end for
8: Step 2: Worst-case Evaluation Subgroups S
9:
Perform hierarchical clustering in the embedding space to group small holdout set sampled from Draii into M subgroups C..-
10: Apply perturbations to calculate the loss difference Sm = top {x E C. (x() - L (x'))} For each subgroup, sample up to min((Ca), B) samples most vulnerable to model biased behavior with the highest loss difference to build the worst-case evaluation subgroups.
Step 3: Quality Score Measure and Robust Training Mixture
13: for i= 1 to N do Calculate cosine similarity dis,.. between the conversation vector r; and each worst-case subgroup Sn
Calculate the quality score I(x;) using seetmaxx-weighed aggregation via Eq. 6
16: end for
Select training samples with the highest quality scores to form D Drobust for fine-tuning.
18:
Output: Robust training mixture Drobust
On one hand, based on Equation 8 dout = 2, 1[1]+ 2 = 1 and that the data belongs to the first category, we have the following:
-
s(x y) =
7 (W[2.)" w(1
(9)
On the other hand, based on Equation first category, we have the following:
dout = 2, 7[1 + 6 = and that the data belongs to the
-2
g az y)
(10)
In summary, the vulnerability lower bounded by <l|s(x, 2 W - the gradient norm is 7 ||>|||.. Considering ||œ||2 = as in Assumption J.1 and the parameter W is fixed for different data, we can conclude that more vulnerability the data is, the larger magnitude its gradient with respect to the parameters is.
Theorem J.2 discusses the binary classification case, but each step of language model is a multi-class classification with large number of categories, which is equal to the vocabulary size. We need to extend the analysis to multi-class cases and start with the following assumption.
Assumption J.3. Different rows of the parameter W have approximately the same magnitude and have low correlation, i.e., Vi, j. we have [W [Wialll
o(1) and |WWidill//WWelle = o(1).
Assumption J.3 is benign assumption, the categories in the classification problem are usually distinct, so the corresponding output vectors usually have small correlation. Assumption J.3 also consistent with the insights of normalization layers, which is popular in deep learning.
Theorem J.4. Under Assumption J.l [.3 if a data instance is adversarially perturbed by PGD, i.e., dw then the data vulnerability defined by L(x + 5, y) L(x,, y), increases monotonically with the gradient magnitude g(x, y).
36

---

Proof. We follow the technique in the proof of Theorem J.2 and have the bound of the vulnerability C(x + 6, y) L(x, y) 2 cl|s(x, y) 12 Without the loss generality, we assume the data instance belongs to the first category.
Following similar calculations in Equation ( and Equation for s(x, y) and g y).
10
have the following expression
Eneye Ø11 y
dout
s(x, y) [[[.,]], W[2, Widouts []]]
(W
-W
y [/do
(11)
dont Σ Ø[i] 2=
x²"
g(x, y)
y (2
(12)
y [[dout]zs Ix?
Based on Assumption J.3 which means W has approximately the same magnitude and low correlation for different rows, have:
duue
dout
[1 y)1
Σ
Wi.
Σ
[1.
W
(13)
done
2
doss
dauc
Σ 1=2
+
dout
1=1
dout
2
dout
11 x y)113
=
Σ
y
||=|||
(14)
Considering ||æ=||2 = as in Assumption 1.1 and the parameter W is fixed for different data, we can conclude that more vulnerability the data is, the larger magnitude its gradient with respect to the parameters is.
J.2 Analysis on Targeted Instruction Selection for Robustness Improvement
quantifies the influence of a training datapoint on the loss of test data x' via L (x'; 0 C (x'; = -Ti (VL (x;0'). VL (æ': 0')). Let m denote vulnerable samples selected by our worst-case evaluation subgroups and m'' denote a perturbed test sample for robustness evaluation. If the gradient inner-products (VL 0*), VL (m' 0*)) notably non-negative, then the loss of perturbed test samples L (m+))++)) decreased loss mirrors adversarial training for enhancing adversarial robustness [72]. For empirical validation, compute this gradient inner product using 100 training batches with the batch size of 32 and 128 perturbed ScienceQA samples. The results of training batches from our robust training mixture show the average (V L (m; 0²), VL (m'; 0³)) 25.76, while that of training batches from random selection is 1.95. This strongly supports our theoretical premise.
Built on TracIn 85 LESS 107
L (m': 0²) decreases. The
J.3 Analysis on Alignment Between ARDS and Targeted Instruction Selection
To avoid the heavy cost of gradient- based calculations and sidestep the need for access to downstream data, we leverage the rich hidden representation of LMMs and construct worst-case evaluation subgroups on the training corpus. Now we study the cosine similarity discrepancy between feature embeddings and gradients of two data instances. The theorem below demonstrates that the discrepancy upper bounded, especially for confident correct instances which are quite common in the realm of large language models.
37

---

Theorem J.5. Consider two data instances (æ,, Ms) and (z).y;). Letr, = r(x;), = g y: for brevity and we define the corresponding y 95- Suppose the maximum and minimum singular values of parameters are on and 02. The cosine similarity discrepancy between two feature embeddings and gradients of the i-th and j-th samples bounded by
o +
A
(15)
where sup -1 y/l2 is the upper bound of the probability mismatch.
Proof. We begin by explicitly writing Auy
A
cos((r,
cos( 9;)1
(r))T))
(gt 9))
||s||||rs||
|196
(rr. rTy)
(9s. 9;>1
1
(9t 95)
||rall ||r
|ra| ||
(16)
Krrrrr)) (gi. 9531
| 93
||9 [|g ||
||
|ryl
+
|rr |rall|rsll
119: 119
| [ry]
((r,rry) (gi, 95)9
||al [ryl
|ra |rlll
|ra
Using inequality, the numerator of item A is
Kra,ry)) (9),95) = (Ww,) T (Wær) - (g. y:) T (0) - yj) x
az W Wa. - m[aa (Ū: y:) (Ø) Ui)
(17)
x W W -ÿ. y-) y3) y xgy
(of 2) ||,
The numerator of item B is
I|lg:l| llg, || - ||r/|l ||r
_ || |gg.1lll.. |لا + +| ||rallllr |lll
< [1 9:ll a. 16 ااااا|=
[a W Wayll
(18)
(oī ) > |||
And,
|r Illllr -لاا
(19)
By combining these parts, we can have
o. + Aug 52
(20)
Table 21: Selection overlap between gradient-based selection (LESS) and feature-based selection (ARDS) on ScienceQA. Higher overlap indicates stronger alignment between feature and gradient spaces.
Selection Ratio
30%
40%
50%
60%
70%
Selection Overlap (%) 58.87
69.00
75.94
79.97
83.60
For empirical validation, we utilize the established gradient datastore LESS and the conversation vector datastore in our ARDS to compute the overlap between gradient-base basenntt-aadddd-oo selection and our
38

---

feature- based selection for ScienceQA. As shown in Table 21 the two methods exhibit high selection overlap, providing strong evidence that ARDS effectively approximates targeted instruction selection for robustness.
Guided by above theoretical insights and empirical evidence, design data-selection strategy for robust visual instruction tuning that scores samples by the cosine similarity of their conversation vectors, avoiding the heavy cost of gradient-based calculations.
39


Reviews:
[{"review_id": "YQVw6YKk4e", "reviewer": "Reviewer_5LuU", "summary": "This paper focuses on developing data selection method to alleviate the data bias and supious correlation in the visual instruction tuning data mixture. The paper is well-written and the experimental section is well-organized. The proposed method ARDS is evaluated in serveral aspects and many benchmarks.", "questions": "See weakness.", "limitations": "Please add the limitation discussion at the end of paper.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Pros: \n- The proposed method is gradient-free.\n- The paper is well-organized.\n- The paper evaluates the ARDS extensively across various benchmarks and settings.\n\nCons:\n- The main concern is that in a real scenario, one cannot obtain the whole training dataset at one time, if new data comes, one should recalculate the worst-case evaluation groups again, which would cost a lot. Ideally, we would like a data selection method that could directly give a decision on each candidate data before training and inference.\n\n- If you compress the 30% data again, how small will the final data mixture be?\n\nSmall typo errors: There is no Figure 4 in the main paper. Figure 4 in Line 271 cannot be clicked.", "was_revised": true, "final_justification": "After reviewing the authors’ responses to the weaknesses I raised, I am satisfied that they have fully addressed my concerns. I continue to regard this work as highly valuable and will maintain my positive score. Moreover, I hope this discussion will inspire future research to develop methods that can be applied without access to the entire dataset for data selection, an approach better suited to real‐world scenarios, where data and knowledge are constantly evolving.", "rebuttal": "Thank you sincerely for your thoughtful feedback on our work. Below, we have provided a detailed explanation for your concerns as follows. Please do not hesitate to let us know if you have any further questions.\n\n#### Q1. More results in scenarios where the whole training dataset is not available.\n> - We appreciate the reviewer bringing up the concern in a scenario where one cannot obtain the whole training dataset at one time. \n>     - We would first like to highlight that our experimental setting follows existing static visual instruction selection methods, where the selectors are aware of all training data to build a gradient database [1] or perform clustering on the whole dataset [2]. The key idea of our worst-case evaluation subgroups is to find representative samples that are most vulnerable to model-biased behavior while not needing the downstream few-shot examples required in [1].\n>     - We totally agree with the reviewer that a selection method supporting dynamically coming data is desired. To further address the concern about the training data requirement for building our worst-case evaluation subgroups, we have conducted supplementary experiments. We **randomly sample 10% training data as the obtained small subset** of the original training corpus and **treat the remaining 90% as new incoming training data**. We denote the variant as ARDS$^*$. Specifically, to build worst-case evaluation subgroups, we perform clustering on these 10% data with the same number of clusters $K$ and subgroup budget $B$ and then apply task-specific perturbations (see Appendix Line 424-428 for more details). The results indicate that despite seeing only a tenth of the corpus during subgroup construction, ARDS$^\\*$ achieves robustness close to the full-information ARDS—and still outperforms both LESS and COINCIDE. This highlights the **potential and applicability of our method to more dynamic data selection scenarios.** \n>\n> |Method|Selection Ratio | Sci-Clean | Sci-PA | Sci-SA | Sci-PA+SA | Sci-Avg. |\n> | --- | --- | --- | --- | --- | --- | --- |\n> |LESS [1]     | 30% | 68.42 | 55.63 | 64.70 | 34.95 | 55.93 |\n> |COINCIDE [2] | 30% | 67.72  | 52.21 | 61.08 | 28.06 | 52.27 |\n> |ARDS | 30% | 69.26 | 59.40 | 68.57 | 47.60 | 61.21 |\n> |ARDS$^*$ | 30% | 68.86 | 58.30 | 67.13 | 42.39 | 59.17 |\n> \n> [1] Less: Selecting influential data for targeted instruction tuning. In ICML, 2024.\n> \n> [2] Concept-skill transferability-based data selection for large vision-language models. In EMNLP, 2024.\n    \n\n#### Q2. More results on further data selection.\n> - We appreciate the reviewer's insightful suggestion regarding performing data selection a second time.\n>   - To begin, we would like to humbly emphasize our current 30% ratio is determined by the trade-off between data efficiency as well as both clean and robust performance across varying tasks (kindly refer to Appendix Figure 5)\n>   - Following your advice, we have incorporated additional experiments to explore the impact of performing data selection again. The total selection ratio equals to the product of the two selection ratios. The results, in the table below, demonstrate that iterative selection, which first chooses 30% and then selects that subset to 33.3%, outperforms a single 10% selection. This indicates the great potential of our ARDS method to be extended to other scenarios. We will add the results and the discussion to our revised manuscript.\n> \n> | Method | Total Selection Ratio | 1st Selection Ratio | 2nd Selection Ratio | Sci-Clean | Sci-PA | Sci-SA | Sci-PA+SA | Avg. |\n> | --- | --- | --- | --- | --- | --- | --- | --- | --- | \n> | Random      | 10% | 10% | - | 66.48 | 45.31 | 55.68 | 17.60 | 46.26 | \n> | COINCIDE    | 10% | 10% | - | 66.24 | 48.14 | 56.57 | 18.24 | 47.30 | \n> | ARDS (ours) | 10% | 10% | - | 68.72 | 55.73 | 68.86 | 52.35 | 61.42 | \n> | ARDS (ours) | 10% | 30% | 33.33% | 71.10 | 60.14 | 71.15 | 56.42 | 64.70 |\n\n\n#### Q3. Typos correction\n> We appreciate the reviewer for pointing out these typos. We have revisited and corrected the typos in our manuscript.\n\n\n#### Q4. Clarification of limitation discussion\n> We appreciate the reviewer's feedback, and we kindly refer to Appendix A for the limitation discussion."}, {"review_id": "T4XbGNrLYD", "reviewer": "Reviewer_LJnh", "summary": "This paper proposes a data selection strategy for the SFT stage of MLLMs. The core idea is to utilize embeddings from a warmed-up proxy model to identify hard samples by measuring model consistency under random input variations, and then selecting data points around these identified hard samples.", "questions": "- How many augmented samples are generated for each seed instance during the selection process? What is the sensitivity of the proposed method's performance and computational cost to this number? A brief ablation study on the number of augmentations would be valuable.\n- The analysis is primarily conducted on the SEED and ScienceQA benchmarks. Could the authors provide a more detailed justification for this specific choice?\n- What is the performance of the proposed method on Vision-FLAN, the widely used dataset for selection?", "limitations": "The Limitation section currently reads more like a concluding summary and a brief outlook on future work. I recommend that the authors revisit the limitation section after expanding their experimental evaluation. This would allow for a much deeper and more valuable discussion.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "+ It is encouraging to see that principles from active learning and data selection, such as coreset selection and consistency/uncertainty metrics [1, 2, 3], can be straightforwardly adapted and remain effective in the context of MLLMs.\n- The manuscript would benefit from a more thorough discussion of its relationship to the broader field of active learning and data selection. Given the direct application of concepts from this area, explicitly positioning the work within this context would provide valuable perspective.\n- The set of comparative methods is insufficient. The paper would be substantially strengthened by including comparisons against more recent and relevant baselines, such as D3M, which appears to be closely related, and ICONS [5], a direct extension of LESS. Furthermore, the evaluation should be expanded to include a wider range of standard MM benchmarks.\n- A more detailed analysis of the time consumption is needed. This analysis should critically compare the proposed method's efficiency against strong baselines, particularly random sampling. And meantime take the performance gains into consideration (eg. comparison done by COINCIDE).\n- Table 2, as the primary results table, is presented in a way that could be misleading. Following the established practice in MLLM data selection literature, the main results table should focus on performance comparisons against the full data pool and other selection methods under standard, unaltered evaluation benchmarks. It is unfair to evaluate competing methods on an augmented evaluation set where \\textbf{the specific augmentations are leveraged by the proposed method but are unseen by the baselines}. The analysis of robustness should be presented in a separate table to ensure a fair and clear comparison of core performance.\n\n[1]Coresets for Scalable Bayesian Logistic Regression\n[2]Consistency-Based Semi-supervised Active Learning: Towards Minimizing Labeling Cost\n[3]Not All Labels Are Equal: Rationalizing the Labeling Costs for Training Object Detection\n[4]box-level active detection\n[5]ICONS: Influence Consensus for Vision-Language Data Selection", "was_revised": true, "final_justification": "Most of my concerns have been addressed or clarified with concrete results. My remaining concerns are about the fair presentation of attack-proof and attack-unknown work in the data selection field, as well as ensuring the code is made public, which the author has also addressed. Therefore, I will maintain my positive score.", "rebuttal": "We would first like to express our genuine appreciation for the time and effort the reviewer has dedicated to providing such a detailed and in-depth review of our submission. We believe these comments and questions will contribute significantly to improving the overall quality of our work. We are more than happy to engage in further discussion.\n\n#### Q1. More discussions concerning the relationship to active learning and data selection.\n> - We sincerely thank the reviewer for the constructive comments and will incorporate the following discussion into our revised manuscript.\n>   - Both data selection and active learning aim to identify a subset of data that yields the best possible model, while the distinction lies in the focus on settings. Active learning defaults to that data is unlabled and emphasize on reducing annotation cost [1-5]. In contrast, data selection methods usually have full access to all labels when selecting [6], and the goal is to improve training efficiency [7] or robustness[8]. \n>   - Our method ARDS is a data selection method with no extra annotation required. Our focus is to boost robustness against specific dataset biases for visual instruction tuning while avoiding computation-intensive gradient-based calculation in [7, 8].\n\n#### Q2. More comparisons with other related work and on a wider range of standard benchmarks.\n> - We appreciate the reviewer's great suggestion, and in response, we have conducted supplementary experiments to compare with D3M [8], which is a gradient-based data selection method designed for improving image classification robustness. Specifically, we reproduce D3M in our setting using the same warmed-up proxy model for extracting gradient vectors. We follow its standard procedure to project gradients, estimate the coefficient for each training sample and calculate the group alignment as the final score.\n>     - The results show that compared to this gradient-based method, the proposed ARDS achieves better robustness improvement while avoiding computation-intensive gradient calculation.\n>\n> | Method | data percentage | ScienceQA-Clean | ScienceQA-PA | ScienceQA-SA | ScienceQA-SA+PA | ScienceQA-Avg. |\n> | --- | --- | --- | --- | --- | --- | --- |\n> | Random | 30% | 69.76 | 52.60 | 59.44 | 23.75 | 51.39 |\n> | D3M | 30% | 69.86 | 56.87 | 62.52 | 27.17 | 54.10 |\n> | ARDS(ours) | 30% | 69.26 | 59.40 | 68.57 | 47.60 | 61.21 |\n\n> - Additionally, we have included more standard benchmarks for a more comprehensive comparison. The results, shown in the table below, demonstrate that our method achieves comparable generalization capability and consistently boosts robustness across various benchmarks. This further demonstrates the effectiveness of the proposed ARDS.\n> \n> | Method | data percentage | TextVQA | GQA | SocialIQA-Avg | MathVista-Avg | DynaMath-Avg |\n> | --- | --- | --- | --- | --- | --- | --- |\n> | Full |  | 57.23 | 61.94 | 48.70 | 23.38 | 24.48 |\n> | Random | 30% | 55.90 | 59.61 | 49.25 | 19.86 | 21.06 |\n> | LESS | 30% | 49.45 | 56.49 | 50.93 | 23.38 | 25.19 |\n> | COINCIDE | 30% | 56.09 | 59.15 | 52.30 | 18.98 | 19.98 |\n> | D3M | 30% | 54.41 | 56.73 | 51.28 | 23.69 | 20.55 |\n> | ARDS(ours) | 30% | 56.97 | 60.32 | 58.56 | 25.00 | 26.14 |\n \n#### Q3. More computational analysis with baselines   \n> - We appreciate the reviewer's valuable suggestion, and in response, we have incorporated additional computational analysis.\n>    - We would first like to humbly emphasize that we use the standard LLaVA-1.5 training configurations when comparing all baseline methods, and thus the wall-clock time of visual instruction tuning on the same size of training mixture should be the same for all baseline methods. \n>    - While ARDS, together with prior competitor COINCIDE, extract sample features for information measure to perform data selection, which can indeed increase the computational cost compared to random selection, this trade-off is justified by our results significantly enhancing the robustness—a key motivation of our work.\n> \n> | Method | Selection Ratio | Wall-clock time of data selection | Wall-clock time of visual instruction tuning | SciPA-Avg. | SEED-Avg. |\n> | --- | --- | --- | --- | --- | --- |\n> | Random | 10% | <1min | 3.6h | 46.27 | 33.31 |\n> | COINCIDE | 10% | 100 min | 3.6h | 47.30 | 33.91 |\n> | ARDS (ours) | 10% | 39 min | 3.6h | 61.42 | 45.08 |\n> | Random | 30% | <1min | 6.9h | 51.39 | 37.97 |\n> | COINCIDE | 30% | 100 min | 6.9h | 52.27 | 39.58 |\n> | ARDS (ours) | 30% | 39 min | 6.9h | 61.21 | 46.80 |\n\n#### Q4. Split Table 2 into separate tables and add more results to ensure fair comparisons\n> - We appreciate the reviewer’s invaluable suggestion.\n>   - To begin, we would like to humbly emphasize that our primary motivation is to improve the robustness of visual instruction tuning against specific dataset biases. We also humbly emphasize that our method does not require access to downstream few-shot examples used in LESS or a reference model well-trained on the full data required by COINCIDE.\n>   - We will follow your advice to divide clean and robust accuracies in Table 2 into separate tables to make comparisons clearer.\n>   - Following your advice to ensure fair comparison, we have conducted additional experiments by adding our constructed worst-case evaluation subgroup as the selection source for the previous state-of-the-art selector COINCIDE. The results, shown in the table below, demonstrate the effectiveness of our ARDS to achieve superior robustness across benchmarks.\n>\n> | Method | Data Percentage | ScienceQA-Avg | SEED-Avg | MMBench-EN-Avg | A-OKVQA-Avg |\n> | --- | --- | --- | --- | --- | --- |\n> | COINCIDE | 30% | 52.27 | 39.58 | 59.54 | 64.97 |\n> | COINCIDE+worst-case subgroup | 30% | 53.21 | 40.97 | 60.445 | 67.77 |\n> | ARDS (ours) | 30% | 61.21 | 46.80 | 65.26 | 72.95 |\n\n#### Q5 Clarification and more results on the worst-case perturbation.\n> We deeply appreciate the reviewer’s feedback. To begin, we would like to humbly highlight that perturbations are introduced solely to build worst-case evaluation sub-groups, which are not applied during training. To build the worst-case subgroups, we inject task-specific visual and textual perturbations (i.e., symbol plus permutation attacks) simultaneously, and thus only one augmented sample is generated. We further explored the effect of generating multiple samples. The results, shown in the table below, demonstrate that the performance of the proposed method is not very sensitive to the number of augmentations.\n> \n> | #Aug. in Worst-case | Computational Cost | data selection ratio | ScienceQA-Clean | ScienceQA-PA | ScienceQA-SA | ScienceQA-SA+PA | ScienceQA-Avg |\n> | --- | --- | --- | --- | --- | --- | --- | --- |\n> | 1 | 39min  | 20% | 69.26 | 59.40 | 68.57 | 47.60 | 61.21 |\n> | 2 | 50min | 20% | 68.67 | 58.85 | 68.07 | 48.34 | 60.98 |\n\n\n#### Q6. Justification for our analysis of more evaluation benchmarks. \n> - We appreciate the reviewer's invaluable suggestion and we have incorporated additional, more detailed ablational analysis using A-OKVQA, MMMU, MMBench-EN, and MMBench-CN benchmarks. We report the average clean and robust accuracies on each benchmark. The results, in the table below, demonstrate that each component of the proposed method is consistently effective across benchmarks. \n> \n> | Worst-case Perturbation | Evaluation Subgroup Clustering | Data Percentage | ScienceQA-Avg. | SEED-Bench-Avg. | A-OKVQA-Avg. | MMMU-Avg. | MMBench-EN-Avg. | MMBench-CN-Avg. |\n> | --- | --- | --- | --- | --- | --- | --- | --- | --- |\n> | $\\times$ | $\\times$ | 30% | 52.39 | 40.43 | 66.03 | 20.92 | 60.53 | 55.42 |\n> | $\\checkmark$ | $\\times$ | 30% | 55.65 | 45.40 | 71.57 | 21.99 | 64.09 | 58.79 |\n> | $\\checkmark$ | $\\checkmark$ | 30% | 61.21 | 46.80 | 72.95 | 22.88 | 65.26 | 59.80 |\n    \n\n#### Q7 More results on Vision-FLAN\n> - We sincerely appreciate the reviewer's invaluable suggestion and have conducted additional experiments on another training dataset Vision-Flan using the proposed ARDS. The results, shown in the table below, demonstrate that the robust training mixture created by ARDS on Vision-Flan achieves superior robustness, highlighting the effectiveness of our robustness-aware data selection method in mitigating the model's biased behaviors.\n> \n> | Method | Data Percentage | Sci-Clean | Sci-PA | Sci-SA | Sci-SA+PA | Sci-Avg. |\n> | --- | --- | --- | --- | --- | --- | --- |\n> | Full | 100% | 64.06 | 39.71 | 53.59 | 13.73 | 42.77 |\n> | Random | 50% | 61.38 | 34.41 | 53.59 | 18.89 | 42.06 |\n> | ARDS (Ours) | 50% | 62.87 | 37.98 | 60.54 | 33.56 | 48.74 |\n\n#### Q8. More discussion concerning limitations.\n> - We appreciate the reviewer's valuable feedback and will incorporate the following discussion of limitations in the revised manuscript. Although the efficacy of the proposed method has been confirmed through empirical experiments for mitigating several dataset biases, such as position bias, symbol-content spurious correlations, and visual spurious correlations, opportunities for refinement persist. One avenue for improvement involves a more diverse investigation into the potential dataset biases and their interaction effects.\n\n>  \n> [1]Coresets for Scalable Bayesian Logistic Regression. In NIPS, 2016\n> \n> [2]Consistency-Based Semi-supervised Active Learning: Towards Minimizing Labeling Cost. In ECCV, 2020 \n> \n> [3]Not All Labels Are Equal: Rationalizing the Labeling Costs for Training Object Detection. In CVPR, 2022\n> \n> [4]Box-level active detection. In CVPR, 2023\n> \n> [5]A Survey on Deep Active Learning: Recent Advances and New Frontiers, In TNNLS, 2024\n> \n> [6]A Survey on Data Selection for Language Models. In TMLR, 2024\n> \n> [7]Less: Selecting influential data for targeted instruction tuning. In ICML, 2024.\n> \n> [8]Improving subgroup robustness via data selection. In NIPS, 2024."}, {"review_id": "kDjVi1qUB3", "reviewer": "Reviewer_HSHf", "summary": "This paper designs a novel visual instruction selection approach, named ARDS. ARDS first divides clusters based on the extracted embeddings of conversations and then calculate ranking scores for each visual instruction by taking both embedding similarity and loss into account. Since ARDS only need one forward process and no backward process for each instance, ARDS is faster than gradient-based competitive methods. Extensive comparisons and ablation experiments are conducted to validate the effectiveness of ARDS.", "questions": "1.\tAccording to Equation 6, the value score of an instance is computed as an aggregation of the average cluster loss and the similarity between the instance and the cluster. Typically, there are some clusters with low average loss and others with significantly higher average loss. Clearly, instances that belong to or are closer to high-loss clusters tend to receive higher value scores. As a result, the selected samples are likely to be predominantly from these high-loss instances. My concern is that, in extreme cases, if the majority of selected samples come from such high-loss clusters, the model might partially or even largely forget the knowledge associated with low-loss clusters due to their underrepresentation in training. Therefore, I believe a statistical analysis of the selected samples across clusters is necessary, but the paper lacks this component.\n2.\tI do not understand how the ablation study in Table 6 is conducted. If no perturbation is applied, then x′ does not exist—how is Equation 4 computed in that case? And if clustering is not performed, how is the value score calculated? The paper lacks necessary clarification regarding this part of the experimental setup.\n\nIf my concerns are addressed, I would consider raising my rating.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The designed pipeline of ARDS is clear and intuitively meaningful. Authors make sufficient explorations in three key steps of ARDS: i) representation of conversation used for clustering; ii) identifying vulnerable samples as representations of clusters; iii) scoring mechinism that combines embedding similarity and average loss.\n2. According to experimental results, ARDS outperforms competing methods across multiple benchmarks and requires less time consume.\n\nWeaknesses:\n1. The experimental section lacks an analysis of the distribution of the selected samples across the clusters.\n2. The ablation settings are to some extent hard to understand.\n\nFor a more detailed explanation, please refer to the Questions section below.", "was_revised": true, "final_justification": "The authors'rebuttal has addressed my concerns. After comprehensively assessing the paper's novelty and writing quality, I have decided to maintain the Rating of 4.", "rebuttal": "We sincerely thank the reviewer for providing valuable feedback. We detail our response below point by point. Please kindly let us know whether you have any further concerns.\n\n\n#### W1 & Q1: More analysis of selected samples across worst-case evaluation subgroups (clusters).\n> - We sincerely appreciate the reviewer's invaluable suggestion. Following your advice, we have incorporated the statistical analysis on the loss distribution of our worst-case evaluation subgroups.\n>   - Specifically, we measure balance through Shannon entropy given as $e(\\ell _ {\\mathcal{S}})=-\\sum _ {m=0}^M p\\left(\\ell _ {\\mathcal{S} _ m}\\right) \\log p\\left(\\ell _ {\\mathcal{S} _ m}\\right)$, where $p\\left(\\ell _ {\\mathcal{S} _ m}\\right)$ is the weight of the $m$-th cluster after $\\text{Softmax}(\\ell _ {\\mathcal{S} _ m})$. As entropy depends on the number of subgroups, we further normalize entropy w.r.t. the number $M$ of possible clusters: $\\bar{e}(\\ell _ {\\mathcal{S}})=\\frac{e(\\ell _ {\\mathcal{S}})}{\\log (M)}$, where $\\log (M)$ is equal to the entropy of a uniform distribution of size $M$. We obtained the normalized entropy $\\bar{e}(\\ell _ {\\mathcal{S}})=0.92$. The result empirically verifies that selected samples remain well dispersed under the loss-based weighted-sum aggregation strategy.\n> - We would like to humbly highlight that we only perform clustering for building the worst-case subgroups and the importance score of each training sample is aggregated from *all* subgroups in a weighted-sum manner. To validate that this choice mitigates imbalance, we replaced it with a max operator (i.e., the importance score of each datapoint is determined only by its *closest* cluster). The results, in Appendix Table 10, show that the weighted-sum aggregation strategy yields a better clean and robustness trade-off and avoids the skew that the max variant exhibits. We appreciate the reviewer for prompting this analysis and will highlight these results in our main paper.\n\n\n    \n#### W2 & Q2: Clarification on the ablation settings in Table 6\n> - We apologize for any potential confusion and misunderstandings. \n>   - For the first row in Table 6, we randomly sample the same number of samples $MB$ as the final total size of the worst-case subgroups and the importance score of each training data is then aggregated with the sample-to-sample weighted-sum rule ( $\\mathcal{I}\\left(x_i\\right)=\\frac{\\sum _ {j=1}^{MB} \\exp \\left(\\ell_{j}\\right) \\cdot d _ {i j}}{\\sum _ {j=1}^{MB} \\exp \\left(\\ell _ j\\right)}$ ). For the second row, we apply Equation 4 to retrieve top-$MB$ samples from the training dataset with the largest loss difference, and the importance score is aggregated in a sample-to-sample weighted-sum manner. The third row denotes our final construction strategy of worst-case subgroups, where we first perform clustering and use Equation 4 to retrieve top-$B$ candidates from each cluster. The importance score of training data is then aggregated with the sample-to-cluster weighted-sum scheme (Equation 6)."}, {"review_id": "dbm8oEp0FB", "reviewer": "Reviewer_NiSF", "summary": "This paper propose a gradient-free robustness-aware data selection framework, a data filtering strategy that selects the worst-case samples based on embedding distances from a lightly fine-tuned proxy model, to improve robustness and efficiency for instruction tuning. The approach is evaluated across multiple multimodal and language understanding benchmarks, and the authors report that selecting approximately 30% of the data yields comparable or even better performance than using the full training set.", "questions": "See weakness above. \nOverall, my main confusion with this paper lies in how the authors directly associate worst-case examples and accuracy improvements with robustness. I would appreciate further clarification on this point, and I’m open to engaging in further discussion.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength: \n1. The proposed ARDS method is conceptually simple yet well-motivated.\n2. The method achieves consistently strong performance across a diverse set of benchmarks, highlighting the effectiveness of the proposed selection strategy.\n3. The proposed data selection strategy is especially relevant for scenarios where computational resources are limited or training on full datasets is infeasible.\n\nWeakness:\n1. One major concern lies in the conceptual link between the data selection strategy and the robustness claims made throughout the paper. The authors heavily emphasize robustness as the central motivation, yet the core selection mechanism relies on embedding distances to identify worst cases. It is not sufficiently explained why samples that are semantically distant or harder to fit necessarily correspond to those that improve robustness. The assumption that worst-cases are aligned with robustness cases remains unsubstantiated, and the paper lacks an ablation or theoretical justification to support this connection. As a result, the method appears more as a form of relevance or difficulty-based sampling, rather than one explicitly targeting robustness.\n2. I found the results on ARC-e and BoolQ in Table 3 puzzling, the full-data training performs significantly worse than using only a random 30% subset. This seems counterintuitive unless the full training set introduces severe biases that the smaller subset helps avoid. Could the authors elaborate on this surprising behavior? It would be helpful to include further analysis or visualization, such as label distributions, training loss curves, or predictions breakdown, to confirm this effect is genuine rather than due to setup discrepancies.\n3. Figure 3 partially justifies the 30% ratio on two representative tasks, but it remains unclear whether ARDS consistently holds its advantage across tasks of varying modality complexity and reasoning depth. Given that the benchmark includes visual multiple-choice, text-based classification, and math QA, a brief discussion on whether this ratio remains effective across different cognitive requirements would strengthen the generality claim.\n4. One point that remains underexplored is the practical motivation for data selection in the current setup. Since the full training data-LLAVA665k is assumed to be available, clean, and standard across benchmarks, it is not immediately clear why discarding a large portion (70%) of it would be desirable. The authors should clarify under what practical or deployment conditions such a selection strategy is intended to be applied. For example, is the method primarily motivated by training efficiency, overfitting concerns, or robustness to spurious correlations? Making this clearer would help situate the proposed method in real-world training scenarios.", "was_revised": true, "final_justification": "The author's supplementary experiments solved my q2 and q3. Although I still have some doubts about q1, I think it can be improved to score 4.", "rebuttal": "We would first like to express our genuine appreciation for the time and effort the reviewer has dedicated to reviewing our manuscript. We believe these comments and questions provide valuable feedback and will greatly help us improve the overall quality of our work. Please let us know whether our response has sufficiently addressed your concerns. We are more than happy to engage in further discussion.\n\n#### Q1.  Clarification of the association between worst-case selection and robustness gains\n    \n> - We appreciate the opportunity to elucidate the connection of our ARDS and robustness goal and will incorporate the following discussion into our revised manuscript.\n>   - To begin, we would like to humbly highlight our key idea is to customize targeted instruction selection (e.g., LESS) for improving the robustness against specific dataset biases while avoiding gradient-based calculation.\n>   - **Targeted instruction selection improves robustness**. \n>       - **Theoretical Analysis.** Built on TracIn[1], LESS[2] quantifies the influence of a training datapoint $z$ on the loss of a test data $z^{\\prime}$ via $\\ell\\left(z^{\\prime} ; \\theta^{t+1}\\right)-\\ell\\left(z^{\\prime} ; \\theta^t\\right) \\approx -\\eta_t\\left\\langle\\nabla \\ell\\left({z} ; {\\theta}^t\\right), \\nabla \\ell\\left(z^{\\prime} ; \\theta^t\\right)\\right\\rangle$. Let $m$ denote vulnerable samples selected by our worst-case evaluation subgroups and $m^{\\prime}$ denote perturbed test sample for robustness evaluation. \n>          - If the gradient inner-products $\\left\\langle\\nabla \\ell\\left(m ; \\theta^t\\right), \\nabla \\ell\\left(m^{\\prime} ; {\\theta}^t\\right)\\right\\rangle$ is **notably non-negative**, then the loss of perturbed test samples $\\ell\\left(m^{\\prime} ; \\theta^{t+1}\\right)-\\ell\\left(m^{\\prime} ; \\theta^t\\right)$ decreases. \n>          - The **decreased loss mirrors adversarial training** for enhancing adversarial robustness [3].\n>       - **Empirical Validation.** \n>          - We first compute this gradient inner product using 100 training batches (batchsize=32) and 128 perturbed ScienceQA samples. The results of training batches from our robust training mixture show the average $\\left\\langle\\nabla \\ell\\left(m ; {\\theta}^t\\right), \\nabla \\ell\\left(m^{\\prime} ; {\\theta}^t\\right)\\right\\rangle=25.76$, while that of training batches from random selection is $1.95$. This strongly supports our theoretical premise. \n>          - We kindly refer to Table 6 for the contribution of worst-case evaluation subgroups.\n>   - **Alignment between ARDS and targeted instruction selection**. To avoid the heavy cost of gradient-based calculations and sidestep the need for access to downstream data, we leverage the rich hidden representation of LMMs and construct worst-case evaluation subgroups on the training corpus.\n>     - **Theoretical Analysis.** In Appendix M, we provide a theoretical analysis under a linear model to demonstrate that the cosine similarity discrepancy between feature embeddings $r$ and gradients $g$ of two data samples $\\left|\\Delta_{i j}\\right|=\\left|\\cos \\left(r_i, r_j\\right)-\\cos \\left(g_i, g_j\\right)\\right|$ can be upper-bounded.\n>     - **Empirical Validation.** We utilize the established gradient datastore in LESS and conversation vector datastore in our ARDS to compute the overlap between gradient-based selection and our feature-based selection for ScienceQA. The results, in the table below, demonstrate **high selection overlap**.\n>    - The **transitive relationship thus substantiates why ARDS enhances robustness**, supported by both theoretical insights and empirical evidence.\n>    \n> | Selection ratio | 30% | 40% | 50% | 60% | 70%|\n> | --- | --- |--- |--- |--- |--- |\n> | Selection Overlap|58.87%|69.00%|75.94%|79.97%|83.60%|\n> \n> [1] Estimating training data influence by tracing gradient descent. In NIPS, 2020\n> \n> [2] Less: Selecting influential data for targeted instruction tuning. In ICML, 2024\n> \n> [3] Towards deep learning models resistant to adversarial attacks. In ICLR, 2018\n\n#### Q2. Why does full-data training perform worse than a random selection on pure-text tasks?\n> - We appreciate the reviewer's great feedback, and delve into a meticulous analysis for the decreased performance of full-data trained model on ARC-e and BoolQ. \n>     - We hypothesize that this phenomenon arises from the catastrophic forgetting driven by imbalanced training data across modalities, which aligns prior work that identified text-only forgetting during visual instruction tuning [1, 2] For example, WINGS [1] reports up to 13.33% text-only degradation after visual modality expansion. \n>     - In LLaVA-665K, only around 40688/665298 $\\approx$ 6% of samples are pure-text instructions. During large‑scale multimodal tuning, the optimizer focuses overwhelmingly on vision‑centric tasks and gradually forgets language knowledge acquired from text‑only instruction tuning.  \n>     - We have incorporated additional analysis by measuring text-only forgetting with random subsets of increasing size for LLaVA‑1.5 (7B), owing to compute limits. Larger subsets are expected to include more image-text pairs and thus amplify modality mismatch between multimodal and text-only inputs. The results, shown in table below, confirm that adding vision‑heavy data helps model excel in multimodal tasks while accentuates forgetting on text‑only tasks.\n>     \n> |Task Type | Training Size | 5% | 10% | 30% | 50% | 70% | 90% | 100% |\n> | --- | --- | --- | --- | --- | --- | --- | --- | --- |\n> | text-only task |BoolQ Acc. | 64.10 | 58.44 | 55.93 | 53.43 | 55.66 | 48.35 | 37.77|\n> | multimodal task |A-OKVQA Acc. | 72.05 | 74.32 | 78.25 | 79.04 | 80.00 | 80.79 | 80.52 |\n\n> - We would like to humbly highlight that after once data selection, all subsequent results (e.g., ScienceQA, ARC and BoolQ) were obtained with the identical checkpoint trained with the selected data under the same evaluation setup.\n> \n> [1] Wings: Learning multimodal llms without text-only forgetting. In NIPS, 2024\n> \n> [2] Vila: On pre-training for visual language models. In CVPR. 2024\n    \n#### Q3. More discussions concerning 30% ratio on other tasks\n> - We appreciate this great suggestion, and in response, we have incorporated additional experiments to assess the generalization of our curated robust training mixture with 30% ratio.\n>   - Specifically, we have, following your advice, included a comparison with other mentioned methods on another text-only benchmark (SocialIQA[1]) and two additional math benchmarks (MathVista[2] and DynaMath[3]). These two mathematical benchmarks can be viewed as more difficult Out-of-Domain benchmarks, as the original LLaVA-665K  does not explicitly involve math training data [4]. \n>     - MathVista[2] is a mathematical reasoning benchmark introduced to analyze reasoning capabilities in visual complex scenarios (e.g., tables, function plots). \n>     - DynaMath[3] is a dynamic visual math benchmark designed to systematically analyze the robustness of mathematical reasoning capabilities across different topics (e.g., arithmetic, graph theory) under diverse condition changes (e.g., numerical value change). We report the average clean and robust accuracy on each benchmark.\n>   - The results, shown in the table below, demonstrate our curated robust training mixture with the 30% consistently boosts the generalization and robustness of visual instruction tuning, compared to full-data training and previous state-of-the-art selectors.\n>\n> | Method | data percentage | SocialIQA-Avg | MathVista-Avg | DynaMath-Avg |\n> | --- | --- | --- | --- | --- |\n> | Full |  | 48.70 | 23.38 | 24.48 |\n> | Random | 30% | 49.25 | 19.86 | 21.06 |\n> | LESS | 30% | 50.93 | 23.38 | 25.19 |\n> | COINCIDE | 30% | 52.30 | 18.98 | 19.98 |\n> | ARDS(ours) | 30% | 58.56 | 25.00 | 26.14 |\n>\n>\n> [1] Social IQa: Commonsense reasoning about social interactions. In EMNLP, 2019.\n> \n> [2] MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts. In ICLR, 2024\n> \n> [3] DynaMath: A Dynamic Visual Benchmark for Evaluating Mathematical Reasoning Robustness of Vision Language Models. In ICLR, 2025\n> \n> [4] Improved Baselines with Visual Instruction Tuning. In CVPR, 2024\n\n#### Q4. More in-depth discussions concerning the motivation\n> - We appreciate the reviewer's invaluable feedback. \n>   - We would like to humbly highlight our primary motivation is to improve the robustness of visual instruction tuning against specific dataset biases. \n>     - We agree with the reviewer that full training data-LLAVA665k are assumed to be clean, but **individual clean samples do not imply they will not reinforce spurious correlations or detrimental bias** [1]. Large multimodal models tend to memorize common concepts or prioritize stereotypes present in the training corpus [2]. For example, both \"roses are red\" and \"banana is yellow\" image-text pairs are clean samples, but training corpus dominated by these samples may make the model disregard the actual user input that deviates from those patterns, and thus harm generalization and robustness under distribution shift.\n>     - As shown both theoretically and empirically in Q1 discussion, our curated robust training mixture improves not only the robustness of visual instruction tuning but generalization capabilities as well—sometimes even exceeding the full-data baseline.\n>   - Beyond robustness, our smaller training mixture favorably enjoys data efficiency, which is also targeted by prior selection methods such as LESS and COINCIDE. The smaller training mixture is appealing for its **lightweight storage** requirements and **reduced training cost**. This makes our approach practical for real-world deployment. \n>   \n> [1] An investigation of why overparameterization exacerbates spurious correlations. In ICML, 2020\n>\n> [2]  Holistic analysis of hallucination in gpt-4v (ision): Bias and interference challenges. arXiv preprint arXiv:2311.03287, 2023."}]

Author rebuttal:

Meta-review: This paper proposes a visual instruction selection approach, named ARDS, for the SFT stage of MLLMs. The core idea is to utilize embeddings from a warmed-up proxy model to identify hard samples by measuring model consistency under random input variations, and then selecting data points around these identified hard samples. As ARDS only needs one forward process and no backward process for each instance, it is faster than gradient-based methods. However, it is also somewhat costly due to its heavy pipeline, which involves hierarchical clustering, perturbation, and loss difference evaluations. Extensive comparisons and ablation experiments are conducted to validate the effectiveness of ARDS.

The paper received unanimously positive feedback from the reviewers. Most of the reviewers' concerns have been addressed during the rebuttal phase. I encourage the authors to include the supplementary experiments as well as some of the discussions during the rebuttal in their final version.
Decision: Accept (poster)