Title: Translation Heads: Unveiling Attention's Role in LLM Multilingual Translation
Primary area: foundation or frontier models, including LLMs
Keywords: LLM, Multilinguistic, Interpretability

Abstract: Recently, large language models (LLMs) have made remarkable progress, with multilingual capability emerging as a core foundational strengths. However, the internal mechanisms by which these models perform translation remain incompletely understood. In this paper, we elucidate the relationship between the attention mechanism in LLMs and their translation abilities. We find that certain attention heads, which we term translation heads, are specifically responsible for mapping tokens from the source language to the target language during inference. Through a systematic investigation across various models, we confirm that these translation heads exhibit several key characteristics: (1) Universality: They are present in all LLMs we studied. (2) Sparsity: They constitute only a small fraction of all attention heads. (3) Consistency: The set of translation heads activated by the model shows strong consistency across different language pairs. (4) Causality: Interventionally removing these heads leads to a sharp decline in the model's translation performance, while randomly removing non-translation heads has little impact on translation ability. (5) Functional Specificity: Ablating translation heads disproportionately harms translation but has a varied impact on other multilingual tasks. We also traced the formation of translation heads during pre-training, revealing an evolutionary path of rapid proliferation, stabilization, and eventual pruning. Furthermore we leverage these translation heads to filter multilingual training data, and our experiments show that these datas significantly enhance translation capabilities of the models.