ViGoR: Improving Visual Grounding of Large Vision Language Models with
  Fine-Grained Reward Modeling
Authors: name: Siming Yan; name: Min Bai; name: Weifeng Chen; name: Xiong Zhou; name: Qixing Huang; name: Li Erran Li
Published: 2024-02-09 01:00:14

Abstract: By combining natural language understanding and the generation capabilities
and breadth of knowledge of large language models with image perception, recent
large vision language models (LVLMs) have shown unprecedented reasoning
capabilities in the real world. However, the generated text often suffers from
inaccurate grounding in the visual input, resulting in errors such as
hallucinating nonexistent scene elements, missing significant parts of the
scene, and inferring incorrect attributes and relationships between objects. To
address these issues, we introduce a novel framework, ViGoR (Visual Grounding
Through Fine-Grained Reward Modeling) that utilizes fine-grained reward
modeling to significantly enhance the visual grounding of LVLMs over
pre-trained baselines. This improvement is efficiently achieved using much
cheaper human evaluations instead of full supervisions, as well as automated
methods. We show the effectiveness of our approach through numerous metrics on
several benchmarks. Additionally, we construct a comprehensive and challenging
dataset specifically designed to validate the visual grounding capabilities of
LVLMs. Finally, we plan to release our human annotation comprising
approximately 16,000 images and generated text pairs with fine-grained
evaluations to contribute to related research in the community.