Analysing the Residual Stream of Language Models Under Knowledge
  Conflicts
Authors: name: Yu Zhao; name: Xiaotang Du; name: Giwon Hong; name: Aryo Pradipta Gema; name: Alessio Devoto; name: Hongru Wang; name: Xuanli He; name: Kam-Fai Wong; name: Pasquale Minervini
Published: 2024-10-21 15:12:51

Abstract: Large language models (LLMs) can store a significant amount of factual
knowledge in their parameters. However, their parametric knowledge may conflict
with the information provided in the context. Such conflicts can lead to
undesirable model behaviour, such as reliance on outdated or incorrect
information. In this work, we investigate whether LLMs can identify knowledge
conflicts and whether it is possible to know which source of knowledge the
model will rely on by analysing the residual stream of the LLM. Through probing
tasks, we find that LLMs can internally register the signal of knowledge
conflict in the residual stream, which can be accurately detected by probing
the intermediate model activations. This allows us to detect conflicts within
the residual stream before generating the answers without modifying the input
or model parameters. Moreover, we find that the residual stream shows
significantly different patterns when the model relies on contextual knowledge
versus parametric knowledge to resolve conflicts. This pattern can be employed
to estimate the behaviour of LLMs when conflict happens and prevent unexpected
answers before producing the answers. Our analysis offers insights into how
LLMs internally manage knowledge conflicts and provides a foundation for
developing methods to control the knowledge selection processes.