Title: Grounding as Feedback: Vision-Language Alignment via Sampling-based Visual Projection
Primary area: generative models
Keywords: vision-language-models, visual-grounding, feedback-based-alignment

Abstract: Vision-Language Models (VLMs) have demonstrated remarkable potential in integrating visual and linguistic information, but frequently produce text that is unfaithful to the visual world, leading to object hallucinations and inaccurate descriptions that limit their real-world applicability. To address this core problem, we introduce Sampling-based Visual Projection (SVP), a framework for guided self-improvement that efficiently enhances vision-language alignment. Our key insight is to use a pre-trained grounding model as an expert guide to provide feedback on descriptions generated by the VLM itself. SVP uses this feedback in an iterative loop to score, select, and adapt the VLM on high-quality, feedback-refined samples. This process transfers the spatial reasoning skills of the expert guide into the generalist VLM without requiring new, manually curated text-image pairs or preference annotation. Our experiments show that SVP yields significant gains across a range of tasks, including a 14\% average improvement in captioning and a 12\% increase in object recall, significantly reduced hallucinations, while maintaining question-answering capabilities. The result is a more robust and reliable VLM, demonstrating that targeted, feedback-driven improvement is a powerful method for enhancing vision-language alignment.

TL;DR: Leveraging self-captioning and grounding feedback to elicit latent information in vision-language models