Enhancing Cognition and Explainability of Multimodal Foundation Models
  with Self-Synthesized Data
Authors: name: Yucheng Shi; name: Quanzheng Li; name: Jin Sun; name: Xiang Li; name: Ninghao Liu
Published: 2025-02-19 19:05:45

Abstract: Large multimodal models (LMMs) have shown impressive capabilities in a wide
range of visual tasks. However, they often struggle with fine-grained visual
reasoning, failing to identify domain-specific objectives and provide
justifiable explanations for their predictions. To address this, we propose a
novel visual rejection sampling framework to improve the cognition and
explainability of LMMs using self-synthesized data. Specifically, visual
fine-tuning requires images, queries, and target answers. Our approach begins
by synthesizing interpretable answers that include human-verifiable visual
features. These features are based on expert-defined concepts, carefully
selected based on their alignment with the image content. After each round of
fine-tuning, we apply a reward model-free filtering mechanism to select the
highest-quality interpretable answers for the next round of tuning. This
iterative process of data synthesis and fine-tuning progressively improves the
model's ability to generate accurate and reasonable explanations. Experimental
results demonstrate the effectiveness of our method in improving both the
accuracy and explainability of specialized visual classification tasks.