Title: MontageAug: Enhancing Long-tail Robustness And Semantic Consistency of VLMs
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Data Augmentation, Vision Language Model, Long-tail Task

Abstract: Vision-Language Models (VLMs) have made significant strides in multimodal understanding tasks, yet their robustness faces severe challenges when dealing with the long-tail data distributions common in the real world, especially in high-stakes domains like medical image analysis. To address this challenge, we propose MontageAug, a compositional data augmentation approach designed specifically for VLMs. It strategically composes images from different sources (particularly from head and tail classes) to construct a novel visual scene (a montage image) and synchronously generates a perfectly corresponding compositional text description. This method not only fundamentally guarantees the semantic fidelity of the augmented samples but also effectively alleviates the long-tail data problem by creating information-rich "hard positive samples." We conducted extensive experimental validation on a model based on the InternVL architecture using ophthalmic medical benchmarks. The results show that MontageAug significantly enhances the model's recognition performance and generalization on tail classes, achieving state-of-the-art (SOTA) performance that surpasses existing augmentation methods on several public benchmarks. Furthermore, to explore the approach's general potential and architectural dependencies, we applied it to the general-domain LLaVA model for analysis. This experiment revealed a critical interplay between the compositional augmentation strategy and the model's underlying architecture. Our work ultimately demonstrates that MontageAug, as an efficient, low-cost, and semantics-preserving VLM augmentation strategy, holds immense practical value in solving the long-tail problem in medical image analysis. We plan to open-source our code, benchmark data, and models upon paper acceptance.