Context Engineering for Trustworthiness: Rescorla Wagner Steering Under
  Mixed and Inappropriate Contexts
Authors: name: Rushi Wang, name: Jiateng Liu, name: Cheng Qian, name: Yifan Shen, name: Yanzhou Pan, name: Zhaozhuo Xu, name: Ahmed Abbasi, name: Heng Ji, name: Denghui Zhang

Incorporating external context can significantly enhance the response quality
of Large Language Models (LLMs). However, real-world contexts often mix
relevant information with disproportionate inappropriate content, posing
reliability risks. How do LLMs process and prioritize mixed context? To study
this, we introduce the Poisoned Context Testbed, pairing queries with
real-world contexts containing relevant and inappropriate content. Inspired by
associative learning in animals, we adapt the Rescorla-Wagner (RW) model from
neuroscience to quantify how competing contextual signals influence LLM
outputs. Our adapted model reveals a consistent behavioral pattern: LLMs
exhibit a strong tendency to incorporate information that is less prevalent in
the context. This susceptibility is harmful in real-world settings, where small
amounts of inappropriate content can substantially degrade response quality.
Empirical evaluations on our testbed further confirm this vulnerability. To
tackle this, we introduce RW-Steering, a two-stage finetuning-based approach
that enables the model to internally identify and ignore inappropriate signals.
Unlike prior methods that rely on extensive supervision across diverse context
mixtures, RW-Steering generalizes robustly across varying proportions of
inappropriate content. Experiments show that our best fine-tuned model improves
response quality by 39.8% and reverses the undesirable behavior curve,
establishing RW-Steering as a robust, generalizable context engineering
solution for improving LLM safety in real-world use.