Emergent Misalignment via In-Context Learning: Narrow in-context
  examples can produce broadly misaligned LLMs
Authors: name: Nikita Afonin; name: Nikita Andriyanov; name: Nikhil Bageshpura; name: Kyle Liu; name: Kevin Zhu; name: Sunishchal Dev; name: Ashwinee Panda; name: Alexander Panchenko; name: Oleg Rogov; name: Elena Tutubalina; name: Mikhail Seleznyov
Published: 2025-10-13 11:23:56

Abstract: Recent work has shown that narrow finetuning can produce broadly misaligned
LLMs, a phenomenon termed emergent misalignment (EM). While concerning, these
findings were limited to finetuning and activation steering, leaving out
in-context learning (ICL). We therefore ask: does EM emerge in ICL? We find
that it does: across three datasets, three frontier models produce broadly
misaligned responses at rates between 2% and 17% given 64 narrow in-context
examples, and up to 58% with 256 examples. We also examine mechanisms of EM by
eliciting step-by-step reasoning (while leaving in-context examples unchanged).
Manual analysis of the resulting chain-of-thought shows that 67.5% of
misaligned traces explicitly rationalize harmful outputs by adopting a reckless
or dangerous ''persona'', echoing prior results on finetuning-induced EM.