ChARM: Character-based Act-adaptive Reward Modeling for Advanced
  Role-Playing Language Agents
Authors: name: Feiteng Fang; name: Ting-En Lin; name: Yuchuan Wu; name: Xiong Liu; name: Xiang Huang; name: Dingwei Chen; name: Jing Ye; name: Haonan Zhang; name: Liang Zhu; name: Hamid Alinejad-Rokny; name: Min Yang; name: Fei Huang; name: Yongbin Li
Published: 2025-05-29 18:15:18

Abstract: Role-Playing Language Agents (RPLAs) aim to simulate characters for realistic
and engaging human-computer interactions. However, traditional reward models
often struggle with scalability and adapting to subjective conversational
preferences. We propose ChARM, a Character-based Act-adaptive Reward Model,
addressing these challenges through two innovations: (1) an act-adaptive margin
that significantly enhances learning efficiency and generalizability, and (2) a
self-evolution mechanism leveraging large-scale unlabeled data to improve
training coverage. Additionally, we introduce RoleplayPref, the first
large-scale preference dataset specifically for RPLAs, featuring 1,108
characters, 13 subcategories, and 16,888 bilingual dialogues, alongside
RoleplayEval, a dedicated evaluation benchmark. Experimental results show a 13%
improvement over the conventional Bradley-Terry model in preference rankings.
Furthermore, applying ChARM-generated rewards to preference learning techniques
(e.g., direct preference optimization) achieves state-of-the-art results on
CharacterEval and RoleplayEval. Code and dataset are available at
https://github.com/calubkk/ChARM.