Title: Addressing divergent representations from causal interventions on neural networks
Primary area: interpretability and explainable AI
Keywords: activation patching, mech interp, DAS, representational divergence, faithfulness

Abstract: A common approach to mechanistic interpretability is to causally manipulate model representations via targeted interventions in order to understand what those representations encode. Here we ask whether such interventions create out-of-distribution (divergent) representations, and whether this raises concerns about how faithful their resulting explanations are to the target model in its natural state. First, we demonstrate empirically that common causal intervention techniques often do shift internal representations away from the natural distribution of the target model. Then, we provide a theoretical analysis of two classes of such divergences: "harmless" divergences that occur in the null-space of the weights and from covariance within behavioral decision boundaries, and "pernicious" divergences that activate hidden network pathways and dormant behavioral changes. Finally, to mitigate the pernicious cases, we modify the Counterfactual Latent (CL) loss from Grant (2025) that regularizes interventions to remain closer to the natural distributions, reducing harmful divergences while preserving the interpretive power of interventions. Together, these results highlight a path towards more reliable interpretability methods.

TL;DR: We show empirical representational divergence between native and causally intervened latent states, we show that this can be pernicious and propose a solution.