RewardDance: Reward Scaling in Visual Generation
Authors: name: Jie Wu; name: Yu Gao; name: Zilyu Ye; name: Ming Li; name: Liang Li; name: Hanzhong Guo; name: Jie Liu; name: Zeyue Xue; name: Xiaoxia Hou; name: Wei Liu; name: Yan Zeng; name: Weilin Huang
Published: 2025-09-10 17:59:31

Abstract: Reward Models (RMs) are critical for improving generation models via
Reinforcement Learning (RL), yet the RM scaling paradigm in visual generation
remains largely unexplored. It primarily due to fundamental limitations in
existing approaches: CLIP-based RMs suffer from architectural and input
modality constraints, while prevalent Bradley-Terry losses are fundamentally
misaligned with the next-token prediction mechanism of Vision-Language Models
(VLMs), hindering effective scaling. More critically, the RLHF optimization
process is plagued by Reward Hacking issue, where models exploit flaws in the
reward signal without improving true quality. To address these challenges, we
introduce RewardDance, a scalable reward modeling framework that overcomes
these barriers through a novel generative reward paradigm. By reformulating the
reward score as the model's probability of predicting a "yes" token, indicating
that the generated image outperforms a reference image according to specific
criteria, RewardDance intrinsically aligns reward objectives with VLM
architectures. This alignment unlocks scaling across two dimensions: (1) Model
Scaling: Systematic scaling of RMs up to 26 billion parameters; (2) Context
Scaling: Integration of task-specific instructions, reference examples, and
chain-of-thought (CoT) reasoning. Extensive experiments demonstrate that
RewardDance significantly surpasses state-of-the-art methods in text-to-image,
text-to-video, and image-to-video generation. Crucially, we resolve the
persistent challenge of "reward hacking": Our large-scale RMs exhibit and
maintain high reward variance during RL fine-tuning, proving their resistance
to hacking and ability to produce diverse, high-quality outputs. It greatly
relieves the mode collapse problem that plagues smaller models.