Title: Steering Evaluation-Aware Language Models To Act Like They Are Deployed
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: AI safety, Interpretability

Abstract: Large language models (LLMs) are sometimes aware when they are being evaluated and change their behavior accordingly. This poses a challenge for AI development, as it undermines the reliability of safety evaluations. In this work, we study activation steering as a technique for controlling LLM evaluation awareness. Specifically, we construct steering vectors using simple contrastive pairs of inputs such as ``You are talking to a real user'' and ``You are talking to an AI evaluator.'' We test this technique on an LLM trained using synthetic document fine-tuning and expert iteration to exhibit evaluation-aware behavior. The model recognizes a specific cue as always indicating evaluation and writes Python type hints only during evaluation, not deployment. This model almost always writes type hints when this evaluation cue is present. Steering the model towards deployment without removing the evaluation cue decreases the type hint rate to the same level as removing the cue and directly telling the model it's deployed. However, directly telling the model that it is deployed without removing the evaluation cue does not reduce the type hint rate at all. These findings suggest that activation steering is a promising technique to suppress evaluation awareness and improve our ability to conduct evaluations of frontier models.