Listener-Rewarded Thinking in VLMs for Image Preferences
Authors: name: Alexander Gambashidze, name: Li Pengyi, name: Matvey Skripkin, name: Andrey Galichin, name: Anton Gusarov, name: Konstantin Sobolev, name: Andrey Kuznetsov, name: Ivan Oseledets

Training robust and generalizable reward models for human visual preferences
is essential for aligning text-to-image and text-to-video generative models
with human intent. However, current reward models often fail to generalize, and
supervised fine-tuning leads to memorization, demanding complex annotation
pipelines. While reinforcement learning (RL), specifically Group Relative
Policy Optimization (GRPO), improves generalization, we uncover a key failure
mode: a significant drop in reasoning accuracy occurs when a model's reasoning
trace contradicts that of an independent, frozen vision-language model
("listener") evaluating the same output. To address this, we introduce a
listener-augmented GRPO framework. Here, the listener re-evaluates the
reasoner's chain-of-thought to provide a dense, calibrated confidence score,
shaping the RL reward signal. This encourages the reasoner not only to answer
correctly, but to produce explanations that are persuasive to an independent
model. Our listener-shaped reward scheme achieves best accuracy on the
ImageReward benchmark (67.4%), significantly improves out-of-distribution (OOD)
performance on a large-scale human preference dataset (1.2M votes, up to +6%
over naive reasoner), and reduces reasoning contradictions compared to strong
GRPO and SFT baselines. These results demonstrate that listener-based rewards
provide a scalable, data-efficient path to aligning vision-language models with
nuanced human preferences. We will release our reasoning model here:
https://huggingface.co/alexgambashidze/qwen2.5vl_image_preference_reasoner.