Diversity-Enhanced Reasoning for Subjective Questions
Authors: name: Yumeng Wang, name: Zhiyuan Fan, name: Jiayu Liu, name: Yi R. Fung

Large reasoning models (LRM) with long chain-of-thought (CoT) capabilities
have shown strong performance on objective tasks, such as math reasoning and
coding. However, their effectiveness on subjective questions that may have
different responses from different perspectives is still limited by a tendency
towards homogeneous reasoning, introduced by the reliance on a single ground
truth in supervised fine-tuning and verifiable reward in reinforcement
learning. Motivated by the finding that increasing role perspectives
consistently improves performance, we propose MultiRole-R1, a
diversity-enhanced framework with multiple role perspectives, to improve the
accuracy and diversity in subjective reasoning tasks. MultiRole-R1 features an
unsupervised data construction pipeline that generates reasoning chains that
incorporate diverse role perspectives. We further employ reinforcement learning
via Group Relative Policy Optimization (GRPO) with reward shaping, by taking
diversity as a reward signal in addition to the verifiable reward. With
specially designed reward functions, we successfully promote perspective
diversity and lexical diversity, uncovering a positive relation between
reasoning diversity and accuracy. Our experiment on six benchmarks demonstrates
MultiRole-R1's effectiveness and generalizability in enhancing both subjective
and objective reasoning, showcasing the potential of diversity-enhanced
training in LRMs.