Title: Δ-AttnMask: Attention-Guided Masked Hidden States for Efficient Data Selection and Augmentation
Primary area: foundation or frontier models, including LLMs
Keywords: Visual Instruction Finetuning (VIF), Vision-Language Models (VLMs), post-training, data selection, Multimodal Learning, NLP, Efficient Machine Learning

Abstract: Visual Instruction Finetuning (VIF) is pivotal for post-training Vision-Language Models (VLMs). Unlike unimodal instruction finetuning in plain-text large language models, which mainly requires instruction datasets to enable model instruction-following ability, VIF also requires multimodal data to enable joint visual and textual understanding; therefore, it typically requires more data. Consequently, VIF imposes stricter data selection challenges: the method must scale efficiently to handle larger data demands while ensuring the quality of both visual and textual content, as well as their alignment. Despite its critical impact on performance, data selection for VIF remains an understudied area. In this paper, we propose Δ-AttnMask. This data-efficient framework quantifies sample quality through attention-guided masking of the model's hidden states, jointly evaluating image-text pairs without requiring domain labels, auxiliary models, or extra training. By computing loss differences (Δ) between the original states and states masked using high-attention regions, Δ-AttnMask intrinsically assesses sample quality. Experiments across multiple VLMs and datasets show that Δ-AttnMask achieves state-of-the-art performance with just 20% of data, accelerating training by 5× while surpassing full-dataset baselines by +10.1% in overall accuracy. Its model-agnostic and data-agnostic design ensures broad applicability across modalities and architectures.

TL;DR: We propose Δ-AttnMask, a data-efficient framework that quantifies sample quality through attention-guided masking of hidden states, jointly evaluating image-text pairs without domain labels, auxiliary models, or extra training.