ChARM: Character-based Act-adaptive Reward Modeling for Advanced
  Role-Playing Language Agents
Authors: name: Feiteng Fang, name: Ting-En Lin, name: Yuchuan Wu, name: Xiong Liu, name: Xiang Huang, name: Dingwei Chen, name: Jing Ye, name: Haonan Zhang, name: Liang Zhu, name: Hamid Alinejad-Rokny, name: Min Yang, name: Fei Huang, name: Yongbin Li

Role-Playing Language Agents (RPLAs) aim to simulate characters for realistic
and engaging human-computer interactions. However, traditional reward models
often struggle with scalability and adapting to subjective conversational
preferences. We propose ChARM, a Character-based Act-adaptive Reward Model,
addressing these challenges through two innovations: (1) an act-adaptive margin
that significantly enhances learning efficiency and generalizability, and (2) a
self-evolution mechanism leveraging large-scale unlabeled data to improve
training coverage. Additionally, we introduce RoleplayPref, the first
large-scale preference dataset specifically for RPLAs, featuring 1,108
characters, 13 subcategories, and 16,888 bilingual dialogues, alongside
RoleplayEval, a dedicated evaluation benchmark. Experimental results show a 13%
improvement over the conventional Bradley-Terry model in preference rankings.
Furthermore, applying ChARM-generated rewards to preference learning techniques
(e.g., direct preference optimization) achieves state-of-the-art results on
CharacterEval and RoleplayEval. Code and dataset are available at
https://github.com/calubkk/ChARM.