Title: SAFER: Probing Safety in Reward Models with Sparse Autoencoder
Primary area: interpretability and explainable AI
Keywords: Safety, Reward Model, Sparse Autoencoder

Abstract: Reinforcement learning from human feedback (RLHF) is a key paradigm for aligning large language models (LLMs) with human values, yet the reward models at its core remain largely opaque. In this work, we present SAFER (Sparse Autoencoder For Enhanced Reward model), a novel framework for interpreting and improving reward models through mechanistic analysis. Leveraging Sparse Autoencoders (SAEs), we uncover human-interpretable features in reward model activations, enabling insight into safety-relevant decision-making. We apply SAFER to safety-aligned preference datasets and quantify the salience of individual features by activation differences between preferred and dispreferred responses. Using these feature-level signals, we design targeted data poisoning and denoising strategies. Experiments show that SAFER can precisely degrade or enhance safety alignment with minimal data modification, without sacrificing general performance. Our approach establishes a new methodology for auditing and refining reward models in high-stakes LLM alignment tasks.