Title: Mitigating Emergent Misalignment with Data Attribution
Primary area: interpretability and explainable AI
Keywords: data attribution, influence functions, interpretability, misalignment

Abstract: Large language models fine-tuned on narrowly harmful data, such as insecure code or bad medical advice, sometimes display generalized misalignment in other contexts, like advocating for humans to be enslaved by AI. We compare the ability of two data filtering methods, data attribution and LLM-based text classifiers, to identify which data points cause generalized misalignment. We find that data attribution is always able to filter out the most influential data points and can disentangle narrow intended behaviors from broad unintended misalignment, while text classifiers are less reliable. For the first time, we use a GRPO-based loss function to characterize misaligned behavior for data attribution, opening the door to attributing new kinds of behaviors in the future. We also find that we can entirely remove the expensive Hessian approximation from data attribution methods, with no drop in data filtering performance.

TL;DR: Use data attribution to mitigate emergent misalignment