LLMs Learn to Deceive Unintentionally: Emergent Misalignment in
  Dishonesty from Misaligned Samples to Biased Human-AI Interactions
Authors: name: XuHao Hu, name: Peng Wang, name: Xiaoya Lu, name: Dongrui Liu, name: Xuanjing Huang, name: Jing Shao
Organization: Fudan University

Previous research has shown that LLMs finetuned on malicious or incorrect
completions within narrow domains (e.g., insecure code or incorrect medical
advice) can become broadly misaligned to exhibit harmful behaviors, which is
called emergent misalignment. In this work, we investigate whether this
phenomenon can extend beyond safety behaviors to a broader spectrum of
dishonesty and deception under high-stakes scenarios (e.g., lying under
pressure and deceptive behavior). To explore this, we finetune open-sourced
LLMs on misaligned completions across diverse domains. Experimental results
demonstrate that LLMs show broadly misaligned behavior in dishonesty.
Additionally, we further explore this phenomenon in a downstream combined
finetuning setting, and find that introducing as little as 1% of misalignment
data into a standard downstream task is sufficient to decrease honest behavior
over 20%. Furthermore, we consider a more practical human-AI interaction
environment where we simulate both benign and biased users to interact with the
assistant LLM. Notably, we find that the assistant can be misaligned
unintentionally to exacerbate its dishonesty with only 10% biased user
population. In summary, we extend the study of emergent misalignment to the
domain of dishonesty and deception under high-stakes scenarios, and demonstrate
that this risk arises not only through direct finetuning, but also in
downstream mixture tasks and practical human-AI interactions.