Does Time Have Its Place? Temporal Heads: Where Language Models Recall
  Time-specific Information
Authors: name: Yein Park; name: Chanwoong Yoon; name: Jungwoo Park; name: Minbyul Jeong; name: Jaewoo Kang
Published: 2025-02-20 04:52:05

Abstract: While the ability of language models to elicit facts has been widely
investigated, how they handle temporally changing facts remains underexplored.
We discover Temporal Heads, specific attention heads primarily responsible for
processing temporal knowledge through circuit analysis. We confirm that these
heads are present across multiple models, though their specific locations may
vary, and their responses differ depending on the type of knowledge and its
corresponding years. Disabling these heads degrades the model's ability to
recall time-specific knowledge while maintaining its general capabilities
without compromising time-invariant and question-answering performances.
Moreover, the heads are activated not only numeric conditions ("In 2004") but
also textual aliases ("In the year ..."), indicating that they encode a
temporal dimension beyond simple numerical representation. Furthermore, we
expand the potential of our findings by demonstrating how temporal knowledge
can be edited by adjusting the values of these heads.