DogeRM: Equipping Reward Models with Domain Knowledge through Model
  Merging
Authors: name: Tzu-Han Lin; name: Chen-An Li; name: Hung-yi Lee; name: Yun-Nung Chen
Published: 2024-07-01 17:01:54

Abstract: Reinforcement learning from human feedback (RLHF) is a popular strategy for
aligning large language models (LLMs) with desired behaviors. Reward modeling
is a crucial step in RLHF. However, collecting paired preference data for
training reward models is often costly and time-consuming, especially for
domain-specific preferences requiring expert annotation. To address this
challenge, we propose the Domain knowledge merged
Reward Model (DogeRM), a novel framework that integrates
domain-specific knowledge into a general reward model by model merging. The
experiments demonstrate that DogeRM enhances performance across different
benchmarks and provide a detailed analysis showcasing the effects of model
merging, showing the great potential of facilitating model alignment.