WavReward: Spoken Dialogue Models With Generalist Reward Evaluators
Authors: name: Shengpeng Ji, name: Tianle Liang, name: Yangzhuo Li, name: Jialong Zuo, name: Minghui Fang, name: Jinzheng He, name: Yifu Chen, name: Zhengqing Liu, name: Ziyue Jiang, name: Xize Cheng, name: Siqi Zheng, name: Jin Xu, name: Junyang Lin, name: Zhou Zhao

End-to-end spoken dialogue models such as GPT-4o-audio have recently garnered
significant attention in the speech domain. However, the evaluation of spoken
dialogue models' conversational performance has largely been overlooked. This
is primarily due to the intelligent chatbots convey a wealth of non-textual
information which cannot be easily measured using text-based language models
like ChatGPT. To address this gap, we propose WavReward, a reward feedback
model based on audio language models that can evaluate both the IQ and EQ of
spoken dialogue systems with speech input. Specifically, 1) based on audio
language models, WavReward incorporates the deep reasoning process and the
nonlinear reward mechanism for post-training. By utilizing multi-sample
feedback via the reinforcement learning algorithm, we construct a specialized
evaluator tailored to spoken dialogue models. 2) We introduce ChatReward-30K, a
preference dataset used to train WavReward. ChatReward-30K includes both
comprehension and generation aspects of spoken dialogue models. These scenarios
span various tasks, such as text-based chats, nine acoustic attributes of
instruction chats, and implicit chats. WavReward outperforms previous
state-of-the-art evaluation models across multiple spoken dialogue scenarios,
achieving a substantial improvement about Qwen2.5-Omni in objective accuracy
from 55.1% to 91.5%. In subjective A/B testing, WavReward also leads by a
margin of 83%. Comprehensive ablation studies confirm the necessity of each
component of WavReward. All data and code will be publicly at
https://github.com/jishengpeng/WavReward after the paper is accepted.