Title: LongRM: Revealing and Unlocking the Context Boundary of Reward Modeling
Primary area: generative models
Keywords: Long context model, Reward Modeling, Long-RewarBench

Abstract: Reward model (RM) plays a pivotal role in aligning large language model (LLM) with human preferences.
As real-world applications increasingly involve long history trajectories, e.g., LLM agent, it becomes indispensable to evaluate whether a model’s responses are not only high-quality but also grounded in and consistent with the provided context.
Yet, current RMs remain confined to short-context settings and primarily focus on response-level attributes (e.g., safety or helpfulness), while largely neglecting the critical dimension of long context–response consistency.
In this work, we introduce \texttt{Long-RewardBench}, a benchmark specifically designed for long-context RM evaluation, featuring both Pairwise Comparison and Best-of-N tasks.
Our preliminary study reveals that even state-of-the-art generative RMs exhibit significant fragility in long-context scenarios, failing to maintain context-aware preference judgments.
Motivated by the analysis of failure patterns observed in model outputs, we propose a general multi-stage training strategy that effectively scales arbitrary models into robust Long-context RMs (LongRMs).
Experiments show that our approach not only substantially improves performance on long-context evaluation, but also preserves strong short-context capability
Notably, our 8B LongRM outperforms much larger 70B-scale baselines and matches the performance of the proprietary Gemini 2.5 Pro model.

TL;DR: We introduce Long-RewardBench and a training method to build LongRMs that excel at long-context reward modeling.