Title: Listener-Augmented Thinking VLMs for Robust Text-to-Image Alignment
Primary area: generative models
Keywords: Text-to-Image Generation, Human Preference Modeling, Out-of-Distribution Generalization, Reward Modelling

Abstract: Improving the generalization and robustness of reward models for visual preferences is crucial-especially under limited annotated data, where collection is costly and complex. While we show that reinforcement learning with verifiable rewards (RLVR) improves generalization, we identify a key failure mode: models often produce final answers that are inconsistent with their own chain-of-thought (CoT); when the same trace is shown to a frozen vision–language model (VLM), it frequently predicts a different choice, and accuracy drops as this inconsistency grows. To solve this, we propose a listener-augmented GRPO objective: a frozen listener re-evaluates the reasoner’s CoT (with the final answer masked) and provides a dense soft reward, combined with an exact-match term for unbiased supervision. We further introduce a probability-based pairwise scoring scheme: at training time it yields calibrated confidence rewards for GRPO, and at inference time it supports an anchor-based strategy that can avoid $\mathcal{O}(n^2)$ comparisons while still producing reliable win probabilities. Under this framework, we establish a clear hierarchy - \textbf{single-image scoring $<$ pairwise comparison $<$ RL with CoT $<$ RL with CoT + listener} - showing that conditioning on both images already boosts generalization, CoT improves further, and listener shaping provides the largest gains. Despite training on only 16\% of HPDv2, our approach achieves 67.4\% on the ImageReward test set and improves OOD accuracy by up to +6 pp on a modern 1.2M-vote benchmark, while reducing contradiction rates. Integrated into a Flow-GRPO text-to-image pipeline and evaluated with the latest HPSv3 metric on prompts from HPDv2 and T2I-CompBench, it yields consistent gains. We will release code and models to facilitate reproducibility.

TL;DR: We improve text-to-image generation with a more generalizable, reasoning-aware reward. Our listener-augmented GRPO aligns explanations with outcomes, boosting OOD accuracy and image quality with minimal data.