CAT: Causal Attention Tuning For Injecting Fine-grained Causal Knowledge
  into Large Language Models
Authors: name: Kairong Han, name: Wenshuo Zhao, name: Ziyu Zhao, name: JunJian Ye, name: Lujia Pan, name: Kun Kuang

Large Language Models (LLMs) have achieved remarkable success across various
domains. However, a fundamental question remains: Can LLMs effectively utilize
causal knowledge for prediction and generation? Through empirical studies, we
find that LLMs trained directly on large-scale data often capture spurious
correlations rather than true causal relationships, leading to suboptimal
performance, especially in out-of-distribution (OOD) scenarios. To address this
challenge, we propose Causal Attention Tuning (CAT), a novel approach that
injects fine-grained causal knowledge into the attention mechanism. We propose
an automated pipeline that leverages human priors to automatically generate
token-level causal signals and introduce the Re-Attention mechanism to guide
training, helping the model focus on causal structures while mitigating noise
and biases in attention scores. Experimental results on our proposed Spurious
Token Game (STG) benchmark and multiple downstream tasks demonstrate that our
approach effectively leverages causal knowledge for prediction and remains
robust in OOD scenarios. Implementation details can be found at
https://github.com/Kairong-Han/CAT.