Title: Decoupled Classifier-Free Guidance for Counterfactual Diffusion Models
Primary area: generative models
Keywords: Counterfactual Generation; Diffusion Models; Classifier-Free Guidance

Abstract: Counterfactual generation aims to simulate realistic hypothetical outcomes under causal interventions. Diffusion models have emerged as a powerful tool for this task, combining DDIM inversion with conditional generation and classifier-free guidance (CFG). In this work, we identify a key limitation of CFG for counterfactual generation: it prescribes a global guidance scale for all attributes, leading to significant spurious changes in inferred counterfactuals. To mitigate this, we propose \textit{Decoupled Classifier-Free Guidance} (DCFG), a flexible and model-agnostic guidance technique that enables attribute-wise control following a causal graph. DCFG is implemented via a simple attribute-split embedding strategy that disentangles semantic inputs, enabling selective guidance on user-defined attribute groups. Our experiments demonstrate that DCFG significantly improves the axiomatic soundness of inferred counterfactuals on challenging medical imaging data, mitigating spurious amplification effects, and enhancing counterfactual reversibility.