GEMeX-ThinkVG: Towards Thinking with Visual Grounding in Medical VQA via
  Reinforcement Learning
Authors: name: Bo Liu; name: Xiangyu Zhao; name: Along He; name: Yidi Chen; name: Huazhu Fu; name: Xiao-Ming Wu
Published: 2025-06-22 08:09:58

Abstract: Medical visual question answering aims to support clinical decision-making by
enabling models to answer natural language questions based on medical images.
While recent advances in multi-modal learning have significantly improved
performance, current methods still suffer from limited answer reliability and
poor interpretability, impairing the ability of clinicians and patients to
understand and trust model-generated answers. To address this, this work first
proposes a Thinking with Visual Grounding (ThinkVG) dataset wherein the answer
generation is decomposed into intermediate reasoning steps that explicitly
ground relevant visual regions of the medical image, thereby providing
fine-grained explainability. Furthermore, we introduce a novel verifiable
reward mechanism for reinforcement learning to guide post-training, improving
the alignment between the model's reasoning process and its final answer.
Remarkably, our method achieves comparable performance using only one-eighth of
the training data, demonstrating the efficiency and effectiveness of the
proposal. The dataset is available at
https://huggingface.co/datasets/BoKelvin/GEMeX-ThinkVG.