Robust Reward Modeling via Causal Rubrics
Authors: name: Pragya Srivastava; name: Harman Singh; name: Rahul Madhavan; name: Gandharv Patil; name: Sravanti Addepalli; name: Arun Suggala; name: Rengarajan Aravamudhan; name: Soumya Sharma; name: Anirban Laha; name: Aravindan Raghuveer; name: Karthikeyan Shanmugam; name: Doina Precup
Published: 2025-06-19 17:59:47

Abstract: Reward models (RMs) are fundamental to aligning Large Language Models (LLMs)
via human feedback, yet they often suffer from reward hacking. They tend to
latch on to superficial or spurious attributes, such as response length or
formatting, mistaking these cues learned from correlations in training data for
the true causal drivers of quality (e.g., factuality, relevance). This occurs
because standard training objectives struggle to disentangle these factors,
leading to brittle RMs and misaligned policies. We introduce Crome (Causally
Robust Reward Modeling), a novel framework grounded in an explicit causal model
designed to mitigate reward hacking. Crome employs the following synthetic
targeted augmentations during training: (1) Causal Augmentations, which are
pairs that differ along specific causal attributes, to enforce sensitivity
along each causal attribute individually, and (2) Neutral Augmentations, which
are tie-label pairs varying primarily in spurious attributes, to enforce
invariance along spurious attributes. Notably, our augmentations are produced
without any knowledge of spurious factors, via answer interventions only along
causal rubrics, that are identified by querying an oracle LLM. Empirically,
Crome significantly outperforms standard baselines on RewardBench, improving
average accuracy by up to 5.4% and achieving gains of up to 13.2% and 7.2% in
specific categories. The robustness of Crome is further testified by the
consistent gains obtained in a Best-of-N inference setting across increasing N,
across various benchmarks, including the popular RewardBench (covering chat,
chat-hard, safety, and reasoning tasks), the safety-focused WildGuardTest, and
the reasoning-specific GSM8k.