Symmetrical Visual Contrastive Optimization: Aligning Vision-Language
  Models with Minimal Contrastive Images
Authors: name: Shengguang Wu; name: Fan-Yun Sun; name: Kaiyue Wen; name: Nick Haber
Published: 2025-02-19 18:05:42

Abstract: Recent studies have shown that Large Vision-Language Models (VLMs) tend to
neglect image content and over-rely on language-model priors, resulting in
errors in visually grounded tasks and hallucinations. We hypothesize that this
issue arises because existing VLMs are not explicitly trained to generate texts
that are accurately grounded in fine-grained image details. To enhance visual
feedback during VLM training, we propose S-VCO (Symmetrical Visual Contrastive
Optimization), a novel finetuning objective that steers the model toward
capturing important visual details and aligning them with corresponding text
tokens. To further facilitate this detailed alignment, we introduce MVC, a
paired image-text dataset built by automatically filtering and augmenting
visual counterfactual data to challenge the model with hard contrastive cases
involving Minimal Visual Contrasts. Experiments show that our method
consistently improves VLM performance across diverse benchmarks covering
various abilities and domains, achieving up to a 22% reduction in
hallucinations, and significant gains in vision-centric and general tasks.
Notably, these improvements become increasingly pronounced in benchmarks with
higher visual dependency. In short, S-VCO offers a significant enhancement of
VLM's visually-dependent task performance while retaining or even improving the
model's general abilities. We opensource our code at https://s-vco.github.io/