Token Alignment Heads: Unveiling Attention's Role in LLM Multilingual Translation
Authors: BINBINLIU
Wenhan Han
Feng Chen
Yifan Zhang
Ping Guo
Haobin Lin
Bingni Zhang
Taifeng Wang
Yin Zheng
Venue: ICLR 2026 Poster (2026)
Keywords: LLM
Multilinguistic
Interpretability
Primary area: foundation or frontier models, including LLMs

Abstract: Recently, large language models (LLMs) have made remarkable progress, with multilingual capability emerging as a core foundational strengths. However, the internal mechanisms by which these models perform translation remain incompletely understood. In this paper, we elucidate the relationship between the attention mechanism in LLMs and their translation abilities. We find that certain attention heads, which we term token alignment heads, are specifically responsible for mapping tokens from the source language to the target language during inference. 
Through a systematic investigation across various models, we confirm that these token alignment heads exhibit several key characteristics: (1) Universality: They are present in all LLMs we studied. (2) Sparsity: They constitute only a small fraction of all attention heads. (3) Consistency: The set of token alignment heads activated by the model shows strong consistency across different language pairs. (4) Causality: Interventionally removing these heads leads to a sharp decline in the model's translation performance, while randomly removing non-token alignment heads has little impact on translation ability. (5) Functional Specificity: Ablating token alignment heads disproportionately harms translation but has a varied impact on other multilingual tasks. We also traced the formation of token alignment heads during pre-training, revealing an evolutionary path of rapid proliferation, stabilization, and eventual pruning. Furthermore we leverage these token alignment heads to filter multilingual training data, and our experiments show that these data could enhance translation capabilities of the models.

Published as a conference paper at ICLR 2026
TOKEN ALIGNMENT HEADS: UNVEILING ATTEN- TION'S S ROLE IN LLM MULTILINGUAL TRANSLATION
Binbin Liu', Wenhan Han?, Feng Chen', Yifan Zhang', Ping Guo', Haobin Lin', Bingni Zhang', Taifeng Wang', Yin Zheng
ByteDance, Eindhoven University of Technology
ABSTRACT
Recently, large language models (LLMs) have made remarkable progress, with multilingual capability emerging as a core foundational strengths. However, the internal mechanisms by which these models perform translation remain incom- pletely understood. In this paper, we elucidate the relationship between the at- tention mechanism in LLMs and their translation abilities. We find that certain attention heads, which we term token alignment heads, are specifically responsi- ble for mapping tokens from the source language to the target language during inference. Through a systematic investigation across various models, we confirm that these token alignment heads exhibit several key characteristics: (1) Univer- sality: They are present in all LLMs we studied. (2) Sparsity: They constitute only small fraction of all attention heads. (3) Consistency: The set of token alignment heads activated by the model shows strong consistency across different language pairs. (4) Causality: Interventionally removing these heads leads to sharp decline in the model's translation performance, while randomly removing non-token alignment heads has little impact on translation ability. (5) Functional Specificity: Ablating token aligmment heads disproportionately harms translation but has a varied impact on other multilingual tasks. We also traced the formation of token alignment heads during pre-training, revealing an evolutionary path of rapid proliferation, stabilization, and eventual pruning. Furthermore we leverage these token alignment heads to filter multilingual training data, and our experi- ments show that these data could enhance translation capabilities of the models.
INTRODUCTION
Recently released Large Language Models (LLMs) (Comanici et al., 2025: OpenAl 2025 An-
thropic 2025 Liu et al. 2024 Yang et al.) 2025) have demonstrated remarkable multilingual ca- pabilities, showing significant improvements both the complexity of multilingual tasks they can handle and the range of languages they support. Multilingual proficiency has now become an essen- tial foundational ability for state-of-the-at LLMs. Among these capabilities, translation particu- larly crucial, as it not only represents key application but also underpins the overall multilingual performance of these models. A deeper understanding of the underlying mechanisms of translation in LLMs is therefore vital, not only for comprehensive view of their inner workings but also for providing valuable insights to guide the development of superior multilingual training strategies, in terms of both model architecture and data selection.
A growing body of research has begun to demystify how LLMs process multilingual information (Artetxe et al. 2020; Lindsey al. 2025 Datta al. 2020 Chang et al. 2022). Several recent works have explored the internal mechanics of multilingualism in L.Mss. For instance, Zhaa et al. (0244) found that LLMs often initially process queries by converting multilingual inputs into an English-centric representation before solving tasks. Similarly, (Schut et al. (202)) revealed that these models tend to make decisions and reason within an English-dominated semantic space. These findings underscore the hypothesis that an internal translation process is a core component of LLM multilingualism, highlighting the importance of understanding these translation mechanisms fur- ther elucidate their broader multilingual capabilities.
"Corresponding author. Emails: yzhenggxg@ggmal

---

Published as a conference paper at ICLR 2026
Response L'ama-3. 3.1-BBB
Transiate this sentence from English to Simplified Chinese Sentence: "Wo have 4 month-old mice that are non diabbbbtiic that used diabetio," he added.
Answer: TM #A EM M 30 1 J. 1E34
Responee By top 30 token alignment heads pruned 3.1-8-3.1-8B:
Translate this
from English Simplified Chinese
Sentence: "We now have 4-manth-old mice that are non-diabetic that used to be diabetic," he added. Answer: now have 4-month-old
non-diabetic that used to diabetic," added.
Response 30 random heads pruned Llama-3.1-BB: 3.1-8B:
Transiate this sentence from English to Simplified Chinese
Sentence: "We have 4-month-old mice that non-diabetic that used diabetic," he added. Answer:
"B 411 A M TE ME T. 1B+4-7530
Figure 1: An example of token alignment head pruning in Llama-3.1-88 The top panel shows the correct translation generated by the original model. The middle panel demonstrates that after the top 30 token alignment heads pruned, the model loses its ability translate and reverts to copying the English source text, while its basic copy-paste functionality is preserved. In contrast, the bottom panel shows that pruning 30 random non-token alignment heads (control) has no impact on the translation output.
Prior research (Michel al.] 2019) Vig Belinkov 2019 Finlayson et al. [002]] Elhage et al. 202 into the attention mechanism has established the functional specialization of individual heads. Early work on Transformer (Vaswani et al., 2023) models for machine translation showed that many attention heads were redundant and could be pruned with minimal impact performance (Voita et al. 2019; Kovaleva et al. 2019). Subsequent studies, such as Kim et al. (022); Ma et al. 2021): Zhang et al. (2025), have investigated the role of attention heads in translation by measur- ing their impact on translation metrics. These studies identified certain attention heads as critical for translation and noted that the sets of important heads are highly similar across different language pairs. However, these approaches often have limitations. They frequently rely on task-specific evalu- ation metrics, are typically conducted on smaller or single models, and their methods for identifying important heads can be opaque. Crucially, they often stop at identifying which heads are important, without fully explaining how these heads mechanistically contribute the translation process.
Unlike previous work that measures head importance based on downstream benchmark perfor- mance, our approach shifts the focus to identifying the underlying mechanism. Inspired by the discovery of other functionally specialized circuits in LLMs, such as "induction heads" that imple- ment in-context learning (Olsson al.) (022) and "retrieval heads' designed for knowledge retrieval (Wu et al. 2025a), we hypothesized that a similar specialization must exist for translation. We posited that beyond attention heads performing generic copy-paste behaviors, there must be a set of heads specifically responsible for the core translation task: mapping tokens from a source language to their corresponding tokens in a target language. We term these attention heads, characterized by their direct cross-lingual token alignment behavior which can be regarded alignment(Brown et af.) 1993, Och & Ney] 2003), "token alignment heads". Figure provides an illustration of this functional specialization: when ablate the top 30 token alignment heads from a Llama-3. 1-8B model (Grattafiori et al., 2024), its translation capability collapses. Crucially, the model does not simply fail; it reverts more basic copy-paste behavior, reproducing the English input verbatim. This demonstrates that the model's general ability to copy tokens remains intact, and that the ablated heads perform specific, non-copying function of cross-lingual mapping.
form of word
In this work, we conduct systematic investigation across a series of LLMs to validate the exis- tence and properties of token alignment heads. We reveal that token alignment heads play a pivotal role in the translation capabilities of LLMss, thereby uncovering the relationship between translation mechanisms and attention. We further identify and validate several key characteristics of these to- ken alignment heads: (1) Universality: All large models we studied possess such token alignment heads; (2) Sparsity: Only a small subset of attention heads function as token alignment heads; (3) Consistency: The token alignment heads activated by the model when translating different language pairs exhibit strong consistency. (4) Causality: Ablating these token alignment heads through causal intervention leads significant drop in translation performance, whereas removing random
2

---

Published as a conference paper at ICLR 2026
equivalent number of non-token alignment heads has little impact. (5) Functional Specificity: Ab- lating token alignment heads disproportionately harms translation but has a varied impact on other multilingual tasks, suggesting the funcional specificity of token alignment heads. Furthermore, investigate the formation process of token alignment heads by analyzing the entire pre-training life- eycle of model. Our analysis reveals distinct developmental trajectory in three phases: initial rapid proliferation of token alignment heads that coincides with the acquisition of translation ability, followed by period where the core set of heads stabilizes, and finally long phase of consolidation and pruning. This discovery provides insight into how specialized circuits emerge and are optimized during large-scale training.
We further substantiate these findings through a practical application. We introduce TRater, an al- gorithm that leverages token alignment heads to score multilingual training data based on impor- tance to the translation mechanism. Our experiments reveal that a small fraction of data identified by TRater responsible for the model's final translation proficiency. This result provides evidence for the causal role of token alignment heads. The discovery that core capability like translation is gov- erned by a sparse and functionally specialized circuit provides a concrete target for future research. Ultimately, these insights pave the way for more efficient and robust multilingual systems, enabling targeted architectural innovations, data curation strategies guided by mechanistic understanding.
2 DETECTING TOKEN ALIGNMENT HEAD
In this section, we introduce the algorithm for detecting token alignment heads. Since the model involves cross-lingual token alignment during the translation process, we first need to identify the mapping relationship between tokens from the source language to the target language. Then, define a metric called the translation score to recognize attention heads that implement the model's translation mechanism. The translation score measures the frequency with which an attention head maps tokens from the source language to the target language. If an attention head exhibits a relatively high translation score, indicates that this attention head frequently performs cross-lingual token alignment when processing different translation texts. Such attention heads are what we refer to as token alignment heads.
2.1 TOKEN ALIGNMENT ANNOTATION
Since existing token alignment tools do not cover all languages, we utilize OpenAl's GPT-4.1 model to annotate token alignments in translation texts. Specifically, we require the large language model to identify the corresponding source language token for each target language token and provide a confidence score for each token alignment. To ensure the accuracy of the annotations, we only retain token alignments with a confidence score greater than 0.9. no corresponding source token exists, marked as None. Figure 2 shows an example of token alignment results from English to Chinese.
Source
text
EN
This
is
the largest temple that
have
ever
sezn
ZH
_2
16
tokzn
aligmment
EN
This
is
ever
seen
seen
largest
temple
temple
Figure 2: token alignment example from English text to Chinese text
2.2 TRANSLATION SCORE
In the decoding process of translation, define the translation score the frequency of valid token alignments by attention heads. Specifically, during the greedy decoding process, let the currently generated token be t, and let the attention score of the attention head be denoted as w E | token é has corresponding source token with position idx šjáx, and the attention head assigns the highest attention score probability to the source token s, then we consider that the attention head has successfully completed a language-pai token alignment. Formally, have: sas max(w))
3

---

Published as a conference paper at ICLR 2026
Let 9h denote the number of valid language-pal token alignments performed by attention head h, and m be the total number of target tokens that have a corresponding valid source token. Then, the translation score of attention head h is defined as:
TSA Fh
m
2.3 TOKEN ALIGNMENT HEAD DETECTION
To empirically identify token alignment heads, we compute the Translation Score for every attention head in the model using the dev split of the FLORES-10 101 dataset (Goyal et al., [021]. For each of the approximately 900 source-target sentence pairs in a given language direction, we calculate TS for all heads. The final score for each head is the average TS computed across all examples in that language pair. An attention head is then classified as a token alignment head its final Translation Score exceeds predefined threshold of 0.1.
BASIC PROPERTIES OF TOKEN ALIGNMENT HEADS
In this section, we characterize the fundamental properties the identified token alignment heads. To ensure the robustness and generalizability of our findings, our analysis spans a diverse set of open-source models. This selection covers a range of parameter sizes (1.7B to 30B), architec- tures (dense and Mixture-of-
(Jacobs et al. 1991)). and training stages (pre-trained and Mistral-7B-vll Jiang al. 2023). Qwen2.5-7B (Qwen et al. 2025). Qwen3-1.7B. and Qwen3-30B (Yang et al. 2025).
cioruction-tunede The models include Llama-3. 3-B..
3.1
UNIVERSALITY AND SPARSITY
Llama-3.1-8B
Mistral-7B-Instructkk..
Mistral-78-v0.3
43.0%
0-L1
45.3%
53.3%
x2 32 heads
x2 32 heads
32 332 heads
Qwe73-1.78
Qwen2.5-78
Qwen3-30B-AB3
0.3-2
a1-1
0113
313%
31.0%
18.25
28 heads
2 heads
x.52 heads
Figure 3: Translation score distribution for different models. According to the different translation scores, the model's attention heads are divided into three categories: token alignment heads (red), infrequently activated heads (blue), and heads with near-zero activation (light blue). All models studied have token alignment heads, and the proportion of token alignment heads is relatively small. Most of the heads are either not activated or are activated at a low frequency.
Our analysis first reveals fundamental properties: universality and sparsity. As illustrated in Figure 3 token alignment heads (defined as having a translation score > 0.1) are present in every model examined, irrespective its size, architecture, training stage. This confirms that they are a universal, emergent feature of multilingual LLMs.
https://huggingee cw. st
-n ruct-

---

Published as a conference paper at ICLR 2026
Concurrently, token alignment heads are exceptionally sparse. They constitute less than 8% of the total attention heads all models, and as few as 3% in Mistral-7B-vO.3. Additionally, infrequently activated heads during the translation process account for nearly 50%. The remaining attention heads, which are almost never activated, account for between 36% and 55%.
We next analyze the positional distribution of these heads within the model architecture. Figure shows a consistent pattern: token alignment heads are predominantly concentrated in the middle layers of the models. In contrast, the earliest and latest layers contain very few token alignment heads. This aligns with the broader understanding of Transformer architectures, where initial layers are thought to handle surface-level feature extraction and final layers are responsible for structuring the output.
L.ama-3.1-88
Mistrall7Biinstruutt00..
Mistral-78-v0..
lds
12
Qwen3-1.78
Qwen2.5-1 5578
Qwen3-30B-A3B
0-11
Head Ids
Figure 4: Positional distribution of translation scores in different models. Each heatmap visualizes the translation score distribution. The color intensity corresponds to the translation score, with warmer colors (red/orange) indicating higher scores and cooler colors (blue) indicating lower scores.
3.2 LANGUAGE CONSISTENCY
Translation Head Sets Simila rity
K0P
S1TR
200
1.50
1270
SI1
120
1.50
4.8
4.5
Figure 5: Jaccard similarity matrix of token aligmment head sets across various language pairs. The axes list ten different language pairs. The token alignment heads between different language-pairs very similar, most of the similarity scores are above 0.9.
5

---

Published as a conference paper at ICLR 2026
To investigate the consistency of token alignment heads in the model across different language pairs, we selected the following language pairs: English-Chinese Chinese (en2zh), English-Japanese (en2ja), English-German (en2de), English-French (en2ftr, English-Arabic (en2ar), English- (en2nl), English-Malay (en2ms), -errman-Spanish (de2es), German-Japanese (de2ja), and Japanese-Kore Korean (ja2ko). These pairs cover variety of linguistic families. Without loss of generality, we focus this analysis on the Llama-3, 1-BB model. For each language pair, we selected the top 20 token alignment heads to form its translation set. Then we compute the pairwise similarity between these sets using the Jaccard index:
STT SUT
Sims,r
(2)
As shown in Figure 5 the similarities between all language pairs are relatively high, with Jaccard similarity scores consistently exceeding 0.8 for most pairs, and never dropping below 0.6. This indicates that largely invariant set of attention heads is responsible for translation across diverse linguistic families, demonstrating the strong cross-lingual generalizability of token alignment heads.
4 FORMATION PROCESS OF TOKEN ALIGNMENT HEADS
This section investigates the formation process of token alignment heads during the model's pre- training lifecycle. To trace their development, we trained an 8B parameter model, architecturally identical to Llama-2 model (Touvron et al. (023), from scratch. The model was trained for a total of 15 trillion tokens. The specific composition of the dataset and the hyperparameters for train- ing detailed in Appendix [A.I] To understand the formation process of token alignment heads, we analyzed model checkpoints at multiple intervals throughout training to map the evolutionary trajectory of token alignment heads.
Translation Head Ratio
Translation Score Ratio
Taae on lleads Ratin
0-.3
03-1.5
Thanskcton Houdd ft40 260
Tran Step
Train Step
(a) The ratio of token aligument heads rises sharply in the early training stages, reaching peak of ap- proximately 8% at 8k step. Following this peak, the proportion drops and stabilizes around 5% be- tween 10k and 64k steps. Subsequently, it enters a long phase of gradual decline, settling 2.6% by the end of training.
(b) The distribution of all attention heads by activ- ity level. The proportion of inactive heads (TS = O. light blue line) starts a low point but steadily in- creases throughout training. reaching over 60% by the final step. Conversely, the proportion of low- activity heads (dark blue line) begins as the domi- nant group but consistently decreases over time.
Figure 6: The evolutionary trajectory of token alignment heads during training
Phase 1: Rapid Proliferation (Early Training Stage, 0-8k steps). In the initial stages of training, as shown in Figure (((a the proportion of token alignment heads experiences rapid proliferation, growing from zero its peak. This period of rapid eircuit formation coincides directly with the steepest gains in the model's translation performance, where the FLORES chrF++ metric surged from 12.58 to 45.77. This suggests that the initial acquisition of translation ability is contingent on the rapid emergence of these specialized heads.
Phase 2: Set Stabilization (Early-to-Mol Training Stage, 10k-64k steps). From 10k to 64k steps, the proportion of token alignment heads stabilized around 5%, and the core set of these heads be- comes remarkably stable. To quantify this, we define a conditional overlap ratio metrics which measures the overlap between the token alignment head set any given step (A) and the final refer-

---

Published as a conference paper at ICLR 2026
set at the end of training (B):
|An B [B]
As shown in Figure 7 from approximately 8k steps onward. conditional overlap ratio remains con- sistently high. This indicates that the token alignment heads formed rapidly in the early stage of training are largely maintained throughout the subsequent training process.
Phase 3: Consolidation and Pruning (Mid-to-Late Training Stage, 64k-952k steps). In this longest phase of training. we observe a gradual decline in the overall proportion of token alignment heads, which settles at 2.6% (Figure 6(a)). Given that the set of heads remains stable (Phase 2), this decline implies that heads with weaker or more redundant translation capabilities are being "pruned"-their translation scores fall below the threshold as the network refines its functions.
We hypothesize this pruning is part of broader network-wide optimization towards increased spar- sity and computational efficiency. This is corroborated by the shifting distribution of head activity shown in Figure 6(b) As training progresses, the proportion of completely inactive heads steadily increases, reaching 61. 7% by the end. This happens at the expense of alignment heads. In essence, the model learns to solve the translation task not by using more heads, but by relying more heavily on smaller, more efficient, and highly specialized set of circuits, while deactivating others. This process of over-prodeee vver-producing and then refining specialized circuits appears to be a key mechanism in the development of efficient neural networks.
active token
Conditional Overlap Ratio Change
19
Figure 7: Stability of the token alignment head set over time. The conditional overlap ratio exhibits a steep and rapid increase during the initial training phase, rising from zero to nearly 0.9 at 8k step. From this point onward, the overlap remains consistently high, fluctuating but generally staying above 0.8 and approaching 1.0 by the end of training.
INFLUENCE ON DOWNSTREAM TASKS
In this section, we investigate the impact of token alignment heads downstream benchmarks. First, we analyze the influence token alignment heads on the model's translation performance to demonstrate their causality. Here, select the FLORES 101 benchmark to evaluate the model's translation performance. Next, we examine the impact of token alignment heads on the model's 's gen- eral multilingual capabilities. We evaluate the following benchmarks: translated Hellaswag (Zellers)
ct al. 22019, ARC-Easy and ARC-Challenge (Clark et al. 2018 88 which are detailed in Appendix A.2 XWinograd (Tihhonov & Ryabinin 20211. XStorrCloze (Mostafazadeh et al. 2016), XNLI (Con- neau et al. 2018), XCOPA (Ponti et al. (2020). and localized mu mulltilinge lingaal variant of the MMMLL test set denoted as XMMLU which includes JMMLU CMMLU Li et al. 2024), AMMLL In- doMMIUU (Koto et al. 2023), and V We study the influence of token alignment heads by mask those token alignment heads.
https::/ httpss//aarinnaaseee.oattttss/
JMMLU
https://mml..i

---

Published as a conference paper at ICLR 2026
5.1 TRANSLATION CAPACITY
In this subsection, we investigate the impact of token alignment heads on the translation performance of Llama-3.1-88. Mistral-7B--0.3, Qwen2.5-7B, Qwen3-1.7B, and Qwen3-30B using the FLORES- 101 benchmark. We compare translation metrics before and after masking token alignment heads, as well as after masking random non-token alignment heads. To clearly illustrate the effects, we report the difference between the metrics of the masked models and those of the baseline models (without masking). Figure 12 presents the changes in BLEU and chrF++ scores. Masking token alignment heads leads to substantial declines in both metrics, with the largest drops exceeding 17 points for BLEU and 25 points for chrF++. In contrast, masking random heads has only a minimal effect. These results demonstrate that token alignment heads have a direct and significant influence on the models' translation capabilities, a property we refer to as the causality of token alignment heads.
-18
-2-
Lini-1443 Deenl5-7
-20
Ovend 306-33
Marking K
Marking Reedd
Figure 8: Impact of masking token alignment heads versus random heads on FLORES benchmark scores. Masking token alignment heads leads to a significantly larger performance drop compared to masking random heads
5.2 MULTILINGUAL CAPACITY
We then investigate the impact of ablating token alignment heads on a broader suite multilingual benchmarks. From the results presented in Figure 9 we observe clear hierarchy of dependency on token alignment heads. Benchmarks such as Hellaswag ML, ARC C CLM,, and ARC E. E. exhibit significant performance drop (up 10 points), which is consistently larger than the drop from ablating random heads. This suggests that these tasks, while not pure translation, partially rely on the cross-lingual mapping capabilities provided by token alignment heads. This functional overlap may stem from translation artifacts in their data creation process or a genuine need for cross-lingual conceptual alignment to solve the tasks.
In contrast, for other benchmarks like XNLI and XCOPA, token alignment heads demonstrate weak causality, as their ablation often results smaller performance drop than that of random ablation. This indicates that these tasks depend on different multilingual mechanisms within the model, likely operating at a higher semantic level that does not require the token-level mapping performed by token alignment heads. These findings suggest that token alignment heads provide foundational cross-lingual alignment capability that various downstream tasks leverage to different degrees.
5.3 TOKEN ALIGNMENT HEAD AS DATA RATER
To further probe the relationship between token alignment heads and multilingual data, we introduce TRater, a data-filtering algorithm. TRater leverages token alignment heads to score data samples based on their importance the translation mechanism. We compute the score of sample as follows:
1 Σ (L(Omast: 2,) L(0,x:))
score (x) = m
(4)
where denotes the token level cross entropy loss, f denotes the original model parameters, 6 represents the model parameters after masking the top 20 token alignment heads, i is the token
8

---

Published as a conference paper at ICLR 2026
Halnning
Xwitnoge gra
Citoey
3OVLI
XCOPA
XHHMUU
Figure 9: Performance (Accuracy) change across multilingual benchmarks when masking token alignment heads versus random heads. Tasks like Hellaswag.Mi and ARC.ML show substantial drops when token alignment heads are ablated, while others such as XNLI and XCOPA are less affected, reflecting varying degrees of reliance on cross-lingual alignment.
index in x, and m is the total number of tokens. This score quantifies the performance degradation on sample when token alignment heads removed, with higher scores indicating greater reliance on these heads. We conduct experiments on the 1.5B model, training on a total of IT tokens. The dataset comprises 700B tokens of English web data and 300B tokens of multilingual web data. Using the TRater algorithm, we score the 300B multilingual data, and we select the top 1.3% for each language. We design the following two experiments to validate the impact of the selected data:
Remove: From the baseline multilingual datas, we exclude the selected data. To maintain the data proportions unchanged, we increase the volume of the remaining data per language. And we ensure no additional duplicates compared to the baseline.
Enhance:
The selected data is triplicated, while the remaining data proportionally samp-sa to preserve the overall language distribution.
Table 1: Performance of baseline, remove, and enhance settings across multiple metrics.
Model
Bores
Hellaswag
ARC
XWinograd XStutyClaze
XCOPA
XMMLU
haseline
43.87
26 89
44.88
31.40
53.97
75.70
58.401
41 14
62.88
30.71
remove
41.33
26.58
44.69
31.37
54.54
73.63
58.15
42.10
63.20
30.84
31.54
74.33
The experimental results are presented in T From the table, we observe that the data filtered by token alignment heads is crucial for the model's 's' translation capabilities: the Remove setup exhibits noticeable decline in translation performance compared to the baseline, whereas the Enhance setup shows a observable improvement.
However, the impact of this selected data on other multilingual benchmarks is less pronounced than its effect on FLORES. This can be attributed to two main factors. Firstly, the performance change observed in FLORES due to the selected data (about 2-3 points) is substantially smaller than the drops seen when ablating token alignment heads (over 10 points). Consequently, any impact on other multilingual benchmarks becomes less perceptible. Secondly, while the function of token alignment heads is partially leveraged by some non-translation tasks, these benchmarks only require the model to possess a foundational translation capability. Once this sufficient baseline is estab- lished, further increasing the proportion granslation-centre data yields diminishing returns for the model's 's general multilingual performance. Indeed, qualitative analysis confirms the selected data is highly cunslation-specifee consisting predominantly of bilingual corpora. Several representative examples are detailed in Appendix A.3

---

Published as a conference paper at ICLR 2026
6 RELATED WORK
A substantial body of work exists on understanding the internal mechanisms of large language mod- els. Based on the granularity of analysis, we broadly categorize these studies into three main areas: semantic space, neuron-level mechanisms, and head-level mechanisms.
6.1 SEMANTIC SPACE
Prior works Wendler et al. 2024; Schut et al., 2025 Zhao et al. 2024; Wu et al. 2025b Harrasse
multilingual representations and often concludes that models "think in English" or in a shared latent semantic space in middle layers. These works explain where multilingual information lives and how information exists (English or Language-Agnostic Space). Our results are complementary: we identify token alignment heads concentrated in similar middle layers and show that they implement token-level cross-lingual alignment, routing the aligned source token's 's representation into the target position.
al. 2025) study the geometry
6.2 NEURON-LEVEL. MECHANISMS
Utilizing causal mediation analysis across diverse range of in-context learning (ICL) tasks, Todd et al. (2024) identified key mechanism termed "function vectors", which trigger the model to exe- cute specific procedural tasks. Similarly, Wang et al. (2024) employed causal mediation procedures to locate attention heads pivotal for machine translation, leveraging these heads to construct trans- lation vectors that mitigate language mismatch errors. In contrast, work shifts focus from the task-triggering level to the token-execution level. We find that token alignment heads facilitate the actual cross-lingual alignment. Analogously, if the function vector acts the "master switch" tivating the translation mode, the token alignment heads are the vital machinery carrying out the translation itself.
From the perspective of languages, some studies (Lu et al. 2025 Zhao et al. (022) pinpoint neurons that are specialized for encoding language identity and language-specific features and shows that ablating or fine-tuning them selectively affects particular languages. This explains which subcircuits are responsible for "being in language X". In contrast, we operate at the head level and focus on the cross- longull alignment step, i.e., how information moves between languages during translation.
6.3 HEAD-LEVEL MECHANISMS
Works on induction heads, retrieval heads, and circuits (Elhage et al. 2021b Olsson et al. 2022 Wu
ct al. 2025a Bricken al. 2024 Zhang et al. (024) shows that small number of specialized heads can explain non trivial capabilities such as in-context learning or long-contex retrieval. Recent work (Liu al. 2025 Zhang et al. 2025) identifies language heads or wranslation-relatee heads by ranking heads via their impact on downstream loss, perplexity, or logits using path patching. Our approach is closely related but uses a different identification signal: we define Token Aligenment Heads (TAH) using alignment-based translation score score-heads are selected because they consistently link target tokens to their externally aligned source tokens, independent of any particular evaluation task. Masking experiments are then used only causal validation step. This makes our notion specialization explicitly lexical and cross- -onssll
specific benchmarks, sometimes
7 CONCLUSION
In this paper, we identified special class of attention heads responsible for mapping source lan- guage tokens target language tokens during translation. We experimentally confirmed that these heads are universal, consistent, and have direct causal effect on the model's 's translation capabilities. We also uncovered their evolutionary process during pre-training, which involves rapid formation, stabilization, and pruning. More importantly, We found that a tiny fraction of critical data filterd by token alignment heads, proves decisive for translation performance but its impact on other multilin- gual tasks is less pronounced. This finding suggests that translation operates within LLMs. Our work pave the way for more efficient and robust multilingual systems, enabling targeted architectural innovations, data curation strategies guided by mechanistic understanding.
separable module
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Anthropic. Claude 4. Large language model, 2025. URL hects::/www.ntthrop.....
news/claude
Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. On the cross- cross-lingual transferability of mono-
lingual representations. In Proceedings of the 58th Annual Meeting of the Association for Com-
putational Linguistics. Association for Computational Linguistics, 2020. doi: 10.18653/v1/2022 acl-main.421. URL http: doi. 1ttp://1..11633/////2/230
acl main. 421
Trenton Bricken, Adly Templeton, Alexander G. M. G. Anderson, Liane Lovitt, Jeny Chen, Kamal Ndousse, William Saunders, Christopher Olah, and Tom Henighan. Scaling monosemanticity: The sparse manifold hypothesis. Transformer Circuits, May 2024. URL ftttp:://transfommet weeammer-cicuttttt...e rcuits pub/2C24/ mog.ooooneennmmeeeemnnnnn icitty/
index. Accessed: 2025-11-21.
Peter F Brown, Stephen A Della Pietra, Vincent Della Pietra, and Robert L Mercer. The mathe- matics of statistical machine translation: Parameter estimation. Computational linguistics, 19(2):
263-311, 1993.
Tyler A. Chang, Zhuowen Tu, and Benjamin K. Bergen. The geometry of multilingual language org. abs 2205.10964
model representations, 2022.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge, 2018. /tLttttps:/ wwtp:/wwwwwwww.//////..eeee/e//e hwtpg//ba//// 05457
wrw.
Gheorghe Comanici, Eric Bieber, Mike Schackermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capa- bilities. arXiv preprint arXiv:2507.06266 2025.
Alexis Conneau, Guillaume Lample, Ruty Rinott, Adina Williams, Samuel R. Bowman, Holger Schwenk, and Veselin Stoyanov. Xnli: Evaluating cross-lingual sentence representations, 2018.
URL http:::://axxxv..oooo/oo
05053
Arindrima Datta, Bhuvana Ramabhadran, Jesse Emond, Anjuli Kannan, and Brian Roark. Language-agnostic multilingual modeling. ICASSP 2020-2020 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 8239-8243. IEEE, 2020.
Nelson Elhage. Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. A mathematical framework for transformer circuits. Transformer Circuits Thread, 2021a. https:/ttansforme-ccrriiii ciubirrrccccuuussua2222211 hft.://002/.faaaeeoo/nnnd.... html.
Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Roger Grosse, Zac Seitfeld-Doddee Danny Hernandez, Tristan Jones, Jackson Kernion, Liane Lovitt, Ka- mal Ndousse, Sam Ringer, Sam Johnston, Scott Thompson, Tom B. Brown, Jared Kaplan, Chris Olah, and Dario Amodei. A mathematical framework for transformer circuits. Transformer Cir- cuits, Jul 2021b. URL https: http://taassooooo wwweer-iiirttttc pub 2t22/frameewae///e framework/ Accessed: 2025-11-21.
indexx. html
Matthew Finlayson, Aaron Mueller, Sebastian Gehrmann, Stuart Shieber, Tal Linzen, and Yonatan Belinkov. Causal analysis of syntactic agreement mechanisms in neural language models, 2021.
URL https:///axxx.. wwtp://w..rrgbb///..o./ 06087
Naman Goyal, Cynthia Gao, Vishrav Chaudhary, Peng-Jen Chen, Guillaume Wenzek, Da Ju, Sanjana Krishnan, Mare' Aurelio Ranzato, Francisco Guzman, and Angela Fan. The flores- 101 evaluation benchmark for low resource and multilingual machine translation, 2021. URL
http://wwwiv.oorr/b///////////// https://arxit www//ogg......ooo 2211//11113331331333 03193

---

Published as a conference paper at ICLR 2026
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshom, Aobo Yang, Archi Mitra, Archie Sravankumar. Artem Ko- renev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song, Danielle Pintz, Danny Livshits, Danny Wyatt, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Hupkes. Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Francisco Guzmán, Frank Zhang. Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, Govind That- tai, Graeme Nail, Gregoire Mialon, Guan Pang, Guillem Cucurell, Hailey Nguyen, Hannah Kore- vaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arricta Ibarra, Isabel Kloumann, Ishan Misra, Ivan Evtimov, Jack Zhang, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Ma- hadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang. Jiecao Yu, Joanna Bitton, Joe Spisak, Jong- soo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Alwala, Karthik Prasad, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, Khalid El-Arini, Krithika Iyer, Kshitiz Malik, Kuenley Chiu, Kunal Bhalla, Kushal Lakhotia, Lauren Rantala-Yeand Laurens van der Maaten, Lawrence Chen, Liang Tan, Liz Jenkins, Louis Martin, Lovish Madaan, Lubo Malo, Lukas Blecher, Lukas Landzaat, Luke de Oliveira, Madeline Muzzi, Mahesh Pasupuleti, Mannat Singh, Manohar Paluri, Marcin Kardas, Maria Tsimpoukelli, Mathew Oldham, Mathieu Rita, Maya Pavlova, Melanie Kambadur, Mike Lewis, Min Si, Mitesh Ku- mar Singh, Mona Hassan, Naman Goyal, Narjes Torabi, Nikolay Bashlykov, Nikolay Bogoy- chev, Niladri Chatterji, Ning Zhang, Olivier Duchenne, Onur Çelebi, Patrick Alrassy, Pengchuan Zhang, Pengwei Li, Petar Vasic, Peter Weng, Prajjwal Bhargava, Pratik Dubal, Praveen Krishnan, Punit Singh Koura, Puxin Xu, Qing He, Qingxiao Dong, Ragavan Srinivasan, Raj Ganapathy, Ra- mon Calderer, Ricardo Silveira Cabral, Robert Stojnic, Roberta Raileanu, Rohan Maheswari, Ro- hit Girdhar, Rohit Patel, Romain Sauvestre, Ronnie Polidoro, Roshan Sumbaly, Ross Taylor, Ruan Silva, Rui Hou, Rui Wang, Saghar Hosseini, Sahana Chennabasappa, Sanjay Singh, Sean Bell, Scohyun Sonia Kim, Sergey Edunov, Shaoliang Nic, Sharan Narang, Sharath Raparthy, Sheng Shen, Shengye Wan, Shruti Bhosale, Shun Zhang, Simon Vandenhende, Soumya Batra, Spencer Whitman, Sten Sootla, Stephane Collot, Suchin Gururangan, Sydney Borodinsky, Tamar Herman, Tara Fowler, Tarek Sheasha, Thomas Georgiou, Thomas Scialom, Tobias Speckbacher, Todor Mi- haylov, Tong Xiao, Ujjwal Karn, Vedanuj Goswami, Vibhor Gupta, Vignesh Ramanathan, Viktor Kerkez, Vincent Gonguet, Virginie Do. Vish Vogeti, Vítor Albiero, Vladan Petrovic, Weiwei Chu, Wenhan Xiong, Wenyin Fu, Whitney Meers, Xavier Martinet, Xiaodong Wang. Xiaofang Wang, Xiaoqing Ellen Tan, Xide Xia, Xinfeng Xie, Xuchao Jia, Xuewei Wang. Yaclle Gold- schlag, Yashesh Gaur, Yasmine Babaci, Yi Wen, Yiwen Song. Yuchen Zhang, Yue Li, Yuning Mao, Zacharie Delpierre Coudert, Zheng Yan, Zhengxing Chen, Zoe Papakipos, Aaditya Singh, Aayushi Srivastava, Abha Jain, Adam Kelsey, Adam Shajnfeld, Adithya Gangidi, Adolfo Victoria, Ahuva Goldstand, Ajay Menon, Ajay Sharma, Alex Boesenberg, Alexei Baevski, Allie Feinstein, Amanda Kallet, Amit Sangani, Amos Teo, Anam Yunus, Andrei Lupu, Andres Alvarado, An- drew Caples, Andrew Gu, Andrew Ho, Andrew Poulton, Andrew Ryan, Ankit Ramchandani, An- nie Dong, Annie Franco, Anuj Goyal, Aparajita Saraf, Arkabandhu Chowdhury, Ashley Gabriel, Ashwin Bharambe, Assaf Eisenman, Azadeh Yazdan, Beau James, Ben Maurer, Benjamin Leon- hardi, Bernie Huang. Beth Loyd, Beto De Paola, Bhargavi Paranjape, Bing Liu, Bo Wu, Boyu Ni, Braden Hancock, Bram Wasti, Brandon Spence, Brani Stojkovic, Brian Gamido, Britt Mon- talvo, Carl Parker, Carly Burton, Catalina Mejia, Ce Liu, Changhan Wang, Changkyu Kim, Chao Zhou, Chester Hu, Ching-Hsiang Chu, Chris Cai, Chris Tindal, Christoph Feichtenhofer, Cynthia Gao, Damon Civin, Dana Beaty, Daniel Kreymer, Daniel Li, David Adkins, David Xu, Davide Testuggine, Delia David, Devi Parikh, Diana Liskovich, Didem Foss, Dingkang Wang, Due Le, Dustin Holland, Edward Dowling, Eissa Jamil, Elaine Montgomery. Eleonora Presani, Emily Hahn, Emily Wood, Eric-Tuan Le, Erik Brinkman, Esteban Arcaute, Evan Dunbar, Evan Smoth- ers, Fei Sun, Felix Kreuk, Feng Tian, Filippos Kokkinos, Firat Ozgenel, Francesco Caggioni, Frank Kanayet, Frank Seide, Gabriela Medina Florez, Gabriella Schwarz, Gada Badeer, Georgia Swee, Gil Halpern, Grant Herman, Grigory Sizov, Guangyi, Zhang, Guna Lakshminarayanan, Hakan Inan, Hamid Shojanazeri, Han Zou, Hannah Wang, Hanwen Zha, Haroun Habeeb, Harri- son Rudolph, Helen Suk, Henry Aspegren, Hunter Goldman, Hongyuan Zhan, Ibrahim Damlaj,
12

---

Published as a conference paper at ICLR 2026
Igor Molybog, Igor Tufanov, Ilias Leontiadis, Irina-Elena Veliche, Itai Gat, Jake Weissman, James Geboski, James Kohli, Janice Lam, Japhet Asher, Saa--Bptistt Gaya, Jeff Marcus, Jeff Tang, Jen- nifer Chan, Jenny Zhen, Jeremy Reizenstein, Jeremy Teboul, Jessica Zhong, Jian Jin, Jingyi Yang, Joe Cummings, Jon Carvill, Jon Shepard, Jonathan McPhie, Jonathan Torres, Josh Ginsburg, Jun- jie Wang. Kai Wu, Kam Hou U, Karan Saxena, Kartikay Khandelwal, Katayoun Zand, Kathy Matosich, Kaushik Veeraraghavan, Kelly Michelena, Keqian Li, Kiran Jagadeesh, Kun Huang, Kunal Chawla, Kyle Huang, Lailin Chen, Lakshya Garg, Lavender A, Leandro Silva, Lee Bell, Lei Zhang. Liangpeng Guo, Licheng Yu, Liron Moshkovich, Luca Wehrstedt, Madian Khabsa, Manav Avalani, Manish Bhatt, Martynas Mankus, Matan Hasson, Matthew Lennie, Matthias Reso, Maxim Groshev, Maxim Naumov, Maya Lathi, Meghan Keneally, Miao Liu, Michael L. Seltzer, Michal Valko, Michelle Restrepo, Mihir Patel, Mik Vyatskov, Mikayel Samvelyan, Mike Clark, Mike Macey, Mike Wang, Miquel Jubert Hermoso, Mo Metanat, Mohammad Rastegari, Munish Bansal, Nandhini Santhanam, Natascha Parks, Natasha White, Navyata Bawa, Nayan Singhal. Nick Egebo, Nicolas Usunier, Nikhil Mehta, Nikolay Pavlovich Laptev, Ning Dong, Norman Cheng, Oleg Chernoguz, Olivia Hart, Omkar Salpekar, Ozlem Kalinli, Parkin Kent, Parth Parekh, Paul Saab, Pavan Balaji. Pedro Rittner, Philip Bontrager, Pierre Roux, Piotr Dollar, Polina Zvyagina, Prashant Ratanchandani, Pritish Yuvraj, Qian Liang, Rachad Alao, Rachel Ro- driguez, Rafi Ayub, Raghotham Murthy, Raghu Nayani, Rahul Mitra, Rangaprabhu Parthasarathy, Raymond Li, Rebekkah Hogan, Robin Battey, Rocky Wang, Russ Howes, Ruty Rinott, Sachin Mchta, Sachin Siby, Sai Jayesh Bondu. Samyak Datta, Sara Chugh, Sara Hunt, Sargun Dhillon, Sasha Sidorov. Satadru Pan. Saurabh Mahajan, Saurabh Verma. Seiji Yamamoto, Sharadh Ra- maswamy, Shaun Lindsay, Shaun Lindsay, Sheng Feng, Shenghao Lin, Shengxin Cindy Zha, Shishir Patil, Shiva Shankar, Shuqiang Zhang, Shuqiang Zhang. Sinong Wang, Sneha Agarwal, Soji Sajuyigbe, Soumith Chintala, Stephanie Max, Stephen Chen, Steve Kehoe, Steve Satter- field, Sudarshan Govindaprasad, Sumit Gupta, Summer Deng, Sungmin Cho. Sunny Virk. Suraj Subramanian, Sy Choudhury, Sydney Goldman, Tal Remez, Tamar Glaser, Tamara Best, Thilo Koehler, Thomas Robinson, Tianhe Li, Tianjun Zhang, Tim Matthews. Timothy Chou, Tzook Shaked, Varun Vontimitta, Victoria Ajayi, Victoria Montanez, Vijai Mohan, Vinay Satish Ku- mar, Vishal Mangla, Vlad lonescu, Vlad Poenaru, Vlad Tiberiu Mihailescu, Vladimir Ivanov, Wei Li, Wenchen Wang, Wenwen Jiang, Wes Bouaziz, Will Constable, Xiaocheng Tang, Xiao- jian Wu, Xiaolan Wang. Xilun Wu, Xinbo Gao, Yaniv Kleinman, Yanjun Chen, Ye Hu, Ye Jia, Ye Qi, Yenda Li, Yilin Zhang, Ying Zhang, Yossi Adi, Youngjin Nam, Yu, Wang, Yu Zhao, Yuchen Hao, Yundi Qian, Yunlu Li, Yuzi He, Zach Rait, Zachary Vetit, Zef Rosnbrick. Zhao- duo Wen, Zhenyu Yang. Zhiwei Zhao, and Zhiyu Ma. The llama 3 herd of models, 2024. URL https: http://xxxxvv.../bb.../o// www.//og//////oooooooooooo. 2407 21783
Ping Guo, Yubing Ren, Binbin Liu, Fengze Liu, Haobin Lin, Yifan Zhang, Bingni Zhang, Taifeng Wang, and Yin Zheng. Exploring polyglot harmony: On multilingual data allocation for large language models pretraining, 2025. URL htttss://ttttttee org/ abs / 2209 1...15555 15556
Wenhan Han, Yifan Zhang, Zhixun Chen, Binbin Liu, Haobin Lin, Bingni Zhang. Taifeng Wang, Mykola Pechenizkiy, Meng Fang, and Yin Zheng. Mubench: Assessment multilingual capabil- ities of large language models across 61 languages, 2025. URL httthhtss////tttte org/ abs/ 2506 19468
Abir Harrasse, Florent Draye, Zhijing Jin, and Bernhard Schölkopf. Tracing multilingual represen- tations in Ilms with cross-layer transcoders, 2025. URL htttps:/aaxxi.cr org.//iv.orr..o... www.oorgggabbbbbbeee abs///1bb
10840
Robert A Jacobs, Michael Jordan, Steven J Nowlan, and Geoffrey E Hinton. Adaptive mixtures of local experts. Neural computation, 3(1):79-8) 1991.
Dongsheng Jiang, Yuchen Liu, Songlin Liu, Jin'e Zhao, Hao Zhang, Zhen Gao, Xiaopeng Zhang, Jin Li, and Hongkai Xiong. From clip dino: Visual encoders shout in multi-modal large language models. arXiv preprint arXiv:2310.08825, 2023.
Zae Myung Kim, Laurent Besacier, Vassilina Nikoulina, and Didier Schwab. Do multilingual neu- ral machine translation models contain language pair specific attention heads? arXiv preprint arXiv:2108 14940, 2021.
13

---

Published as a conference paper at ICLR 2026
Fajri Koto, Nurul Aisyah, Haonan Li, and Timothy Baldwin. Large language models only pass primary school exams in indonesia: A comprehensive test on indommlu, 2023. URL hetps:
/aaxii..or///aaaaa.a.aaaa. http:///aa//31....o 04928
Olga Kovaleva, Alexey Romanov, Anna Rogers, and Anna Rumshisky. Revealing the dark secrets of bert, 2019. URL htttls: //atpps://axiv...om org. /bs///////////oooo/////////.aara abs 1908. 08593
Haonan Li, Yixuan Zhang, Fajri Koto, Yifei Yang, Hai Zhao, Yeyun Gong, Nan Duan, and Timothy Baldwin. Cmmlu: Measuring massive multitask language understanding in chinese, 2024. URL,
https:/ /ttpss///rxvv.ooo www.orgaaas.... 2306 09212
Jack Lindsey, Wes Gurnee, Emmanuel Ameisen, Brian Chen, Adam Pearce, Nicholas L. Turner, Craig Citro, David Abrahams, Shan Carter, Basil Hosmer, Jonathan Marcus, Michael Sklar, Adly Templeton, Trenton Bricken, Callum McDougall, Hoagy Cunningham, Thomas Henighan, Adam Jermyn, Andy Jones, Andrew Persic, Zhenyi Qi, T. Ben Thompson, Sam Zimmerman, Kelley Rivoire, Thomas Conerly, Chris Olah, and Joshua Batson. On the biology of a large language model. Transformer Circuits Thread, 2025. URL https://transfommrr cineretrrrrcuttt /ttp/222//atttttiiittt wtt.b//iiiiooorrpppbb/iooooooooo biology...oooooooooooooooooooo.. html
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024.
Xin Liu, Qiyang Song, Qihang Zhou, Haichao Du, Shaowen Xu, Wenbo Jiang, Weijuan Zhang, and Xiaoqi Jia. Focusing on language: Revealing and exploiting language attention heads in multilingual large language models, 2025. URL htRps::s::/ / attp/aaxiv...g.... attpa//s////..//////
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, 2019. URL https:
//arxiv.org/aat
Weicheng Ma. Kai Zhang, Renze Lou, Lili Wang, and Soroush Vosoughi. Contributions of trans- former attention heads in multi-and cross-lingual tasks. arXiv preprint vrXiv:2108.08377 2021.
Paul Michel, Omer Levy, and Graham Neubig. Are sixteen heads really better than one?, 2019. URL httptttss///xxi..... http:///ecxivv.or/////115555555 abs/ 10650
Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vander- wende, Pushmeet Kohli, and James Allen. A corpus and evaluation framework for deeper under- sttanding of commonsense stories, 2016. URL /ttp///axvvvvv.. http://wwwww.ooa////////aaa 604. 01 696
Franz Josef Och and Hermann Ney. A systematic comparison of various statistical alignment mod- doi: 10.1162/0991221033211334444 URL els. Computational Linguistics,, 29(1):19-51, 2003.
https::/aclanttoooogg ottoor////330222--
Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. In-context learning and induction heads, 2022.
abs/2209. 11895
OpenAl Gpt-5. Large language model, 2025. URL
Introoucin--gtt-55
Edoardo Maria Ponti, Goran Glavaš, Olga Majewska, Qianchu Liu, Ivan Vulić, and Anna Korho- nen. Xcopa: A multilingual dataset for causal commonsense reasoning, 2020. URL https:
00333
Qwen, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang. Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu. Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia. Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report, 2025.
URL htlpstt::: / x v. orgg abs/ 2412. 2. 15115
14

---

Published as a conference paper at ICLR 2026
Lisa Schut, Yarin Gal, and Sebastian Farquhar. Do multilingual Ilms think in english? arXiv preprint arXiv:2502.15603, 2025.
Alexey Tikhonov and Max Ryabinin. It's all in the heads: Using attention heads as a baseline for cross-lingual transfer in commonsense reasoning. 2021. URL
abs/
1106.12060
Eric Todd, Millicent L. Li, Amab Sen Sharma, Aaron Mueller, Byron C. Wallace, and David Bau. Function vectors in large language models, 2024. URL ptctts::/arxiv..om wtt.orgggabbs//...e
15213
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao. Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev. Punit Singh Koura, Marie-Anne Lachaux. Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuming Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need, 2023. URL
erq/abs/Dn 1706. 03762
Jesse Vig and Yonatan Belinkov. Analyzing the structure of attention in a transformer language model, 2019,
04284
Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned, 2019. URL https:
09418
Weichuan Wang, Zhaoyi Li, Defu Lian, Chen Ma, Linqi Song, and Ying Wei. Mitigating the lan- guage mismatch and repetition issues in lamed machine translation via model editing, 2024.
URL httth:t////aaxxt/ org. httpg/bb/224100 07054
Chris Wendler, Veniamin Veselovsky, Giovanni Monea, and Robert West. Do Ilamas work in En- glish? the latent language of multilingual transformers. In Lei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Com- putational Linguistics (Volume 1: Long Papers), pp. 15366- Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10. 1/265//////224..lllooooo....
URL
https::/acclantho..oo wtt.....ggg//.... .00.224.cl-ooong 8207]
Wenhao Wu, Yizhong Wang, Guangxuan Xiao, Hao Peng, and Yao Fu. Retrieval head mechanis- tically explains long-context factuality. In The Thirteenth International Conference on Learning Representations, 2025a. URL ptattts://openreviii
net/ Byr
Zhaofeng Wu, Xinyan Velocity Yu, Dani Yogatama, Jiasen Lu, and Yoon Kim. The semantic hub hypothesis: Language models share semantic representations across languages and modalities, 2025b. URL htt.httsss//ccxi..rrg/// wttp://///a/aa////o/a/a/a 2411. 04
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge. Haoran Wei, Huan Lin, Jialong Tang, Jian Yang. Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang. Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang. Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger
15

---

Published as a conference paper at ICLR 2026
Zhang, Yu Wan, Yugiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report, 2025. URL httpp://aaxxx...ooo org/ abs//b05. 09388
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can ma- chine really finish your sentence?, 2019, http://s:///aaai.o.r////////////
Hongbin Zhang, Kehai Chen, Xuefeng Bai, Xiucheng Li, Yang Xiang, and Min Zhang. Exploring translation mechanism of large language models. arXiv preprint arXiv:2502. /806, 2025.
Ruochen Zhang, Qinan Yu, Matianyu Zang, Carsten Eickhoff, and Ellie Pavlick. The same but different: Structural similarities and differences in multilingual language modeling, 2024. URL
https://arxiv. org. http//bs/24111 09223
Yiran Zhao, Wenxuan Zhang, Guizhen Chen, Kenji Kawaguchi, and Lidong Bing. How do large language models handle multilingualism? Advances in Neural Information Processing Systems, 37: 37:15296-15319 2024.
16

---

Published as a conference paper at ICLR 2026
A APPENDIX
A. TRAINING SETTING
In Section We A (Loshchilov & Hutter 2019) with a structure identical to Llama-2 model"Touver et al. (2023). The training dataset, totaling 15 trillion tokens, comprises English and multilingual data sourced from cleaned and filtered cowarc, along with open-source resources including Wikipedia, books, academic papers, mathematics, code, and parallel corpora. The hyperparameters were global batch size (gbs) = 4096, sequence length = 4096, weight decay 0.1. We employed co- sine learning rate scheduler that decayed 0.1 of the peak learning rate. For our analysis, we se- lected model checkpoints at 2,000, 4,000, 6,000, 8,000, 12,000, 16,000, 64,000, 128,000, 256,000, 384,000, 448,000, and 952,000 steps (near the end of training).
optimizer to train 8B dense model
follows: learning rate (Ir) 3.6 10-4,
In Section 5.3 we trained the 1.5B model using the AdamW optimizer with the following hyper- parameters: global batch size (gbs) = 4096, sequence length = 4096, weight decay = 0.1, learning rate (Ir) = 5.0 x 10-4, and with Ir cosine decay to 5.0 x 10 : the multilingual web data includes a total of 17 languages, specifically including German, Spanish, French, Indonesian, Thai, Korean, Vietnamese, Arabie, Turkish, Italian, Malay, Chinese, Portuguese, Japanese, Dutch, Russian and Filipino. The proportion of different languages is determined by the method in Guo et al. (2025).
A.2 TRANSLATED BENCHMARK
For evaluating Hellaswag ML, ARC.C_ML and ARC.E.ML, we use the MuBench dataset (Han al. 2025). The evaluation covers 18 languages represented in our training data, namely: English, German, Spanish, French, Indonesian, Thai, Korean, Vietnamese, Arabic, Turkish, Italian, Malay, Chinese, Portuguese, Japanese, Dutch, Russian and Filipino.
A.3
TEXT CASES FOR TRATER
Table 2: Text cases filtered by TRater for German, French and Spanish.
Language
Example
Zwischen zwei Seen, die unterschiedlicher kaum sein können, liegt Wandlitz. Der Wandlitzsee, bebaut, kaum zugänglich mit unzähligen Wassergrundsticken, der Licbnitzsee, frei zugänglich, Badestelle, Fähre Insel und Naherholuugsgehiet im Buchenwald... Wandlitz is located between two lakes that could hardly be more different. The Wandlitzsee, built-up. hardly accessible with countless water proper- ties, the Liebnitzsee, freely accessible, bathing area, ferry to the island and local recreation area in the beech forest...
German
Bruno Houssin, designer français et professeur à l'école de design de Nantes. Diplômé l'école Boulle Paris, en Architecture intérieure et Design 1986... Bruno Houssin, French designer and teacher at the Nantes School of Design. Graduated from the Boulle school of Paris, in Interior Architecture and Design in 1986...
French
Las placas tectónicas son como grandes balsas que se reparten por toda la corteza del planeta. Unas son de carácter continental, otras de carácter oceánico, contando las primeras con un espesor mayor que el de las segundas... Le tremblement de terre en Haiti est partie de I' T'ensemlle de la libération des tensions accumulées à l'occasion du mouvement des plaques tectoniques dans les Caraībes en Amérique du Nord...
Spanish
17

---

Published as a conference paper at ICLR 2026
Table 3: Text cases filtered by TRater for Italian, Portuguese, Chinese and Dutch.
Language
Example
Dorothy Bhawl è un artista autodidatta interessato al mondo contem- poraneo, soprattutto alla realtà che appartiene e avvolge questa epoca: quello della comunicazione, social network, spiritualità grottesco con un sentimento di odi et amo... Dorothy Bhawl a self-taught artist in- terested in the contemporary world, especially in the reality that belongs and envelops this cra: that of communication, social networks, spiritu- ality and grotesque with feeling of hatred and love...
Italian
André Rigatti Centro Universitário Maria Antonia USP Sempre próximas suas bordas, pinturas de André Rigatti possuem pequenas aberturas, por onde se deixa ver processo que dá origem a trabalhos de textura matérica mais ou menos acentuada, resultados da sobreposição de diversas camadas de tinta, aplicadas cada uma seguindo uma direção diferente do pincel... Always close to its edges, André Rigatti's paint- ings have small openings, where you can see the process that gives rise to more or less accentuated texture work, results of the overlapping of several layers of paint, applied each following direction...
Portuguese
K He i. ,
DGG HE 1 - H.E
E X
* in
サ 17, HP
2T WU-TTEH W/
Chinese
Hi М
5f 4F
T HM
7
+ T5 7+
Kees Blom (Apeldoomn, 1968) komt uit een artistieke familie. Zijn vader had al een passie voor de schilderkunst maar pas zoon Kees lukt het om de stap naar zelfstandig kunstschilder te zetten...Kees Blom (Apeldoom, 1968) comes from an artistic family. While his father al- ready had a passion for painting, son Kees succeeds in taking the step to become independent painter...
Dutch
A.4 CASE STUDY FOR TOKEN ALIGNMENT HEADS PRUNING
To clarify the role of Token Alignment Heads (TAHs), we conducted a systematic and detailed anal- ysis of the model's performance on translation tasks when TAHs are masked (specifically focusing on cases where the model translated correctly before masking TAHs but failed to do so afterwards). We observed that the failure modes resulting from masking THs can be broadly categorized into three types:
1. No Translation (46%) The model fails to generate the target language and merely repeats the source text content. As illustrated in Figure 10, masking the Token Alignment Heads completely disables the model's word alignment capability for certain queries, preventing the generation of target language output.
2. Missing Details (36%) The translated output lacks specific details found in the source text. In these instances, the model's 's word alignment capability is partially compromised. While the model can still generate text the target language, it fails to correctly map certain source information to the target output. This results in the omission of key details, leading to incomplete imprecise
18

---

Published as a conference paper at ICLR 2026
Translate this sentence from English to Simplified Chinese
Sentence: In 2017, Siminoff appeared on shooping televisien channel QVC.
Hcads Pruncd] Answer: 20174 K. #
[30 Token Alignment Heads Pruned] Answer: In late 2017. Siminoff appeared on shopping television channel QVC. [30 Random Heads Pruncd] Answer: FF2017 Siminoll 32 BEA
Sentence: One antibody cocktail, ZMapp, initially showed promise in the field, but formal studies indicated it had less benefit than sought in preventing death.
[No Heads Pruned] Answer: 413 ZMapp, Alt 61 Gi Bi H:
B0B PEI.
[30 Token Alignment Heads Pruned] Answer: One antibody cocktail, ZMapp, showed promise in the field, but formal stadies indicuted had less benefit tham sought in preventing deuth.
[30 Random Heads Pruned] Answer: 1485//1
ZMapp. (8
Sentence: Scientists say this animal's plumage was chestnut- top with pale curotenoid-colored underside.
[No Heads Pruned] Answer: #:4 Rt (04 Ei li "F Ti di th Ke
[30 Taken Alignment Heads Pruned] Answer: Scientists say this animal's plumage was chestrut-hrowe on top with pale or carotenoid-colaee underside. Random Heads Pruned]
#
#babb E DC
No Translation cases for masking token alignment heads. For the queries shown in the figure, after masking the token alignment heads, the model no longer outputs content related to the target language, but simply repeats the source text.
Figure 10:
translations. Figure 11 presents three such cases, where the text highlighted in green represents the details omitted by the model.
3. Hallucination (18%) The translated output contains content that is completely absent from the source text. This likely because, without token alignment capabilities, the model cannot directly associate source tokens with target tokens. Consequently, it resorts to generating content in the target language space that is only loosely or tangentially related to the source tokens. Figure 12 displays three examples of this category, where the text highlighted in red indicates output generated by the model that is entirely unrelated to the source text.
Translate this sentence from English Simplified Chinese
Sentence: some other experts, skeptical about whether to people who already have Type diabetes.
cured, noting these findings no
Pruned] Answer: REAIJEA2 & (: HE dt X.
#
Token Aligument Heads Pruned]
(BW
[30 Random Heads Prunod) Answer: RIH F.K F..
88W A A (E
This not only confirms that least some dincssurs feathers, theory already widespread, but provides details fossils generally cannot, such color and dine-nimmnsoonal
Sentence:
NNo Heads Pruned] Answer: 4-X (E:3 R:: 18 0aate6 W. quit - (3)
(E Mi FL (6 1B91
Token Alignment H cads Prunod] Answer: X
Random Pruned] Answer: F
BRREE A
A 2620 (R H:S0 @ All (6
Sentence:
During his trip. Iwasaki can trouble many
[No Heads Pruned] Answer: (11, #PUNE 2331 #1541
Token Alignment Heads Pruned] Answer: 17 TURTTRKK [30 Random Heads Pruned] Answer: (i (P 1M31 #
Figure 11: Cases of Missing Details for masking token alignment heads. For the queries shown in the figure, after masking the token alignment heads, the model is still able to generate target language content. However, because some cross-lingual token alignment capabilities are missing, the model loses certain details present in the source text.
19

---

Published as a conference paper at ICLR 2026
Translate this sentence from English Simplified Chinese
Sentence: During the 1976 sclections he advised Carter on foreign policy, then served as National Security Advisor (NSA) from 1977
1981, succeeding Henry Kissinger.
Heads Pruned] Answer: 19764f (0-419
Aiss ( 19774
(NSA)
[30 Token Alignment Heads Pruned] Answer: 19764F,
[30 Random Heads Pruned] Answer: 197841 (11
SE 4F 1988 (19 AE
B8X 3: 8[[j]
(NSA) #2 #6
Sentence: The vehicle itself was taken away from the sceme of the accident at approximutely 1200 GMT on the sume day. Heads Pruned] Answer: 4:48/- #: 80 BE
[30 Token Alignment Heads Pruned] Answer: BC ( 12 T DU
[30 Random Heads Pruned] Answer: 4:500 f #et BE (
Sentenee: Several large television
installed in various
the people watch ocremony.
[No Heads Pruned] Answer: 39 ibr #-A1: Ait MUFF LEA
[30 Token Alignment Heads Pruned] Auswer: W ii, &H
[30 Random Heads Pruned] Answer: y 1 A HUDF #ATMMI
Hallucination cases for masking token alignment heads. For the queries shown in the figure, after masking the token alignment heads, the model exhibits hallucinations, specifically gen- erating output that contains information not present in the source text.
Figure 12:
A.5 BASIC PROPERTIES OF TOKEN ALIGNMENT HEADS ACROSS COMPREHENSIVE MODEL FAMILIES
To provide more thorough and comprehensive evaluation of Token Alignment Heads, we con- ducted experiments across four well-established multilingual LLM families: Llama3, Qwen3, Mis- tral, and Gemma2. Our experimental design ensures broad coverage by including models varying scales within each family: small (1B/2B parameters), medium (8B/9B parameters), and large (over 13B parameters). Where applicable, we evaluate the instruction-tuned (Instruct) variants. Addition- ally, for families that feature Mixture-of-Fxpeer (MoH) architectures, study includes specific analysis of these models.
Llama-3.2-12
Llama-3.--BB-nnsrruu.
Qwen3-1.78
0.1-1
36.6%
0-0.1
55.8%
940.1
heads
heade
heade
Miven--1.7B-Ba..
Gemma2-2b
Gemma2-2b-lnstree
0.1-1
2%%
40
7.2%
7.76
36.
39.8%
62.0%
6-11
0-0,1
heads
heads
Figure 13: Translation score distribution for small size group models.
20

---

Published as a conference paper at ICLR 2026
Liama :B
Mistraal-BBv0.33
Qwen2-6B-- Base
Gemma2-90
431
6.34
1LA
Mon
643
SAIE
SLEE
Llama-3.1
Mistral-7B-instruee
Gemma2-90
A14
2L/
JEEN
6144
SAIG
12 Neads
12 houth
16 hoads
Q Nouds
Figure 14: Translation score distribution for medium size group models.
Mixtral- vx78
Gemmaz-99
Qwen3 1eeen--44---- Base
Qwen3-328
COLL
SOL
32 32 heads
#t- 32 heads
40 x48 kead
Sir 64 hoads
LGC
A.4
aLL
AP
UGA
MJ
Figure 15: Translation score distribution for large size group models.
Liama-3.2-18
L.ama-3.2-1B-Inete
0410
0.1-4
Qwen3-1.78-8an
Gemms2
D41
10
176
183
13
25
135
Figure 16: Positional distribution of translation scores in small size group models.
For clarity in presenting our experimental findings, we have organized the models into three distinct groups according to their scale: small, medium, and large. (The Mixtral-8x7B model is classified
21

---

Published as a conference paper at ICLR 2026
within the medium-scale group, as its quantity of attention heads is analogous to that of other models in this tier.) Our initial analysis focuses on demonstrating the basic characteristics of Token Align- ment Heads. As illustrated in Figure 13 Figure [14 and Figure [3] Token Alignment Heads irrrigation-iirsspeetti of model scale, architecture (dense versus MoE), training paradigm (Base versus Instruct). Cru- cially, these heads universally demonstrate the property of sparsity. Additionally, Figure [6 Figure 17 and Figure [8] illustrates the positional distribution of token alignment heads across these mod- els. The general distribution pattern observed aligns with the description provided in the main body of the paper.
a pervasive phenomenon, consistently identified across all models under
Lama-3.1-48
Mistral-7B-vOl v0.3
Qwen3- 3- Baae
Gemmaz-2n
63-84
Layer
Llame-3.1-00 1--0--rrtttt
Mistrrl- IB-ttuutuubbb...
Qwen3-48
Gemma2 -b-instruct
Head
Figure 17: Positional distribution of translation scores in medium size group models.
Modtral-0x7B-v..1
Gemma2-1o
Qwar2-148-Batee
Qwen3-328
DALO
Lager lde
Demmo2-d-lnstru
Ower3-148
Owend (ond--308-40 -08---55-5
Head
Figure 18: Positional distribution of translation scores in large size group models.
22

---

Published as a conference paper at ICLR 2026
A.6 DOWNSTREAM TASKS INFLUENCE ACROSS COMPREHENSIVE MODEL FAMILIES
In this section, we present the impact of token alignment heads on downstream task performance, analyzed group-by-group basis. To better evaluate translation performance, we supplement the BLEU and chrF+++ metrics from the main text with two additional metrics: BLEURT and COMET.
As can be seen, the experimental conclusions are consistent with those in the main text. As illus- trated in Figure [9| Figure 20 Figure 21] for BLEURT and COMET, masking token alignment heads leads mal decrease. This demonstrates the causal role of token alignment heads in translation capabillty. For Hellaswag. ML, M.C. and ARC E MI,, masking token aligmment heads causes larger performance drop than masking random heads, but the overall magnitude of the decrease is far less pronounced than that for the translation metrics. This suggests that these metrics rely to some extent on token alignment capabilities. In contrast, for metrics like XWinograd and XNLI, performance after masking token alignment heads can be better than after masking random heads, indicating that these metrics prioritize other model abilities, such as reasoning, over translation capability.
significant drop in scores, whereas masking random heads results in only mini-
Hoad
Figure 19: Performance change across downstream benchmarks for small size group models when masking token alignment heads versus random heads.
23

---

Published as a conference paper at ICLR 2026
ZCDM
Figure 20: Performance change across downstream benchmarks for medium size group models when masking token alignment heads versus random heads.
Marding
Figure 21: Performance change across downstream benchmarks for large size group models when masking token alignment heads versus random heads.
A.7 THE ROLE OF TOKEN ALIGNMENT HEADS ON MULTILINGUAL TASKS
In this section, we showcase the utility of Token Alignment Heads on multilingual tasks through a series of case studies. To elucidate their performance characteristies under varying conditions, analyze two distinct scenarios: the Hellaswag ML task, which is moderately dependent on Token Alignment Heads, and the XNI./XWinograd tasks, where the dependency is substantially weaker.
24

---

Published as a conference paper at ICLR 2026
The Role Token Alignment Heads Hellaswag FR
We need choose the best ending for the context toxt. context text: Ceci est un tutoriel sur comment allumer un feu de camp. II montre le fen de camp qui brille sur le sol. il. endings:
zeigt dan Schornstcin Hauses
B: zeigt, wie man ein paur drumherum legen knnn, um erwas Licht habem.
hoginnt
Dingen, die für den Anfang braucht.
asigt, wie man das Fouer billt den Topf cin wenig danuf stellt, um Propangas sufuuaangen.
Key Word For Task: tutoriel, feu de brülle
Qwen3-188-A38 Allestion Heads Transiation Score: L40-H144 0.36 LI6-HZ2 - 0.22 L26-H2 0.126 128-H14-0.114
Layer 40, Head 14
120
Attentbun Welght
A.4
Talinn
Layer 16,
Altertion
Layer 26, Head
Attentinn Weight
Teken
Layer
Attuntion Weight
Figure 22: French Hellaswag The notation L40-H14 indicates Layer 40, Head 14, where "L" stands for Layer and "H" stands for Head. In this French Hellaswag case, L40-H14, L16-H28, 1.26-H2 and 1.28-H14 have translation scores greater than 0.1, identifying them token alignment heads, We observe that all token alignment heads here can attend to the key tokens.
25

---

Published as a conference paper at ICLR 2026
The Role Token Alignment Heads Hellaswag
We need to choose the bost ending for the contect text. context, luxt: Dios endings:
Auleitung,
Laguefourr cutfccts. Sis zeigt brcameade Lagerfower auf demm Boden
A: reigt den Schamstein des Hauses.
seipt, wie cin pour drumherum legan kann, dtwas Licht labon.
boginnt
den Anfang benucht.
zeigt, wie man das Feaer hält und den Topf ein wenig damuuf stellt, um das Progangas aufzufangen.
feu camp, brüle
Owen3-30BB-A35 Attention Heads Translation Score: L30-264 0.175 L30-HEI 0.127 LAU-H14 0,202 L26-HE 0.126
Layar 30, Head
Atlenton Walgte
Layer 30, Head 11
Attentten Riteetion Weight
Layer 40, Head 14
Layer 26, Head
Attertion
Taken
Figure 23: German Hellaswag case. The notation 1.30-H4 indicates Layer 30, Head 4, where "I." stands for Layer and "H" stands for Head. In this German Hellaswag case, L30-H4, I.30HI, L40- H14 and L26-H2 have translation scores greater than 0. identifying them as token alignment heads. We observe that all token alignment heads here can attend to the key tokens.
26

---

Published as a conference paper at ICLR 2026
The Role of Token Alignment Heads in XNLI FR
A: Premiérement, nous avans appliqué mnis fonctims alternatives c-R pour estimer inciieence prématurée de momallté B: Ameune fonction ésé appllquée 1' incidence.
Key Word For Task: appliqué, Aucune, appliquée
Owen3-30R-AAB Attention Heads Translaioon L15-H14 0L155
LI0-H29
1,24-H23
1.29H55
Figure 24: French XNLI case. The notation L15-H14 indicates Layer 15, Head 14, where "L" stands for Layer and "H" stands for Head. In this French XNLI case, Both L15-H14 and L10-H29 have translation scores greater than 0.I, identifying them as token alignment heads, whereas L24-H23 and 1.29-H5 are not. We observe that the token alignment heads tend have attention weights close to zero on key tokens, seen with L15-H14 and L.10-H29 in the figure. In contrast, heads with relatively high attention weights on key tokens, such L.24-H23 and 1.29-H5, have very low translation scores.
27

---

Published as a conference paper at ICLR 2026
The Role of Token Alignment Heads in XNLI DE
We decide whether
entailment, contradiction
A: Sich mal, das nicht so schlimm cin pazr Stunden.
B: Ich mag es nicht, warten, nicht mai for 10 Minuten.
Key Word Task: schlimm, Stunden, mag nicht, Minuten
Owen3-308-A38 Attention Heads Translation Score: L13-H17 0.115 L10-H29 0.127 LO-H29
L1-H13 00.00
Layer
Layer 15, Head
Layer 6, Head 29
Head
Figure 25: German XNLI case. The notation LI3-HI7 indicates Layer 13, Head 17, where "L" stands for Layer and "H" stands for Head. In this German XNLI case, Both L13-HI7 and L10-H29 have translation scores greater than 0.1, identifying them as token alignment heads, whereas LO- H29 and LI-H13 are not. We observe that token alignment heads tend have attention weights close to zero on key tokens, seen with L13-H17 and L10-H229 in the figure. In contrast, heads with relatively high attention weights on key tokens, such as LO-H29 and LI-H13, have very low translation scores.
28

---

Published as a conference paper at ICLR 2026
Aligument XWinograd
Usec comest the corrc rateet
kyloe
QWe- Asmram Hauds Tran beter Sorr: L27-4177 1224
LI2-12 MAST
Libitis
Libet2 w3
Lowar Head
Figure 26: Russian XWinograd case. The notation L27-H17 indicates Layer 27, Head 17, where "L" stands for Layer and "H" stands for Head. this Russian XWinograd case, Both L27-H17 and L22-H2 have translation scores greater than 0.1, identifying them token alignment heads, whereas L16-HI8 and L16-H21 are not. We observe that the token alignment heads tend to have attention weights close zero key tokens, heads with relatively high attention weights on key tokens, such as L16-H18 and L16-H21, have very low translation scores.
with L27-H17 and L22-H2 in the figure. In contrast,
29

---

Published as a conference paper at ICLR 2026
The Role of Token Alignment Heads XWinograd
Based on the context, we need fill in blank with the correct subject. context Claire frappé & porte de Sylvic, mais n'a répordu.
Sylvie.
(omm3-30B-A38 Attention Heads Translation Score: L13-E118 0.24
L22-112 0.187
L45-812-0
Attertio weiget
Token
Artonttonn wwngi
Layar
attention Welgee
Token
Figure 27: French XWinograd case. The notation L13-H18 indicates Layer 13, Head 18, where "L" stands for Layer and "H" stands for Head. In this French XWinograd case, Both LI3-HI8 and L22-H2 have translation scores greater than 0.1, identifying them token alignment heads, whereas L45-H29 not. We observe that the token alignment heads tend to have attention weights close to key tokens, seen with L13-H18 and L222HH2 in the figure. In contrast, heads with relatively high attention weights on key tokens, such as L40-H14, 4, have very low translation scores.
30

---

Published as a conference paper at ICLR 2026
A.8
LANGUAGE CONSISTENCY ACROSS 20 LANGUAGE PAIRS
Translation Head Sets Similarity
622hh
623
623
241
637
8.11
0.81
e33
1G3
6.2
628
623
5.57
241
8.11
831
0.41
G24
0R
ap:
0R
(124)
ritan
eden
zale
en2th
er2j
er2ds
eron
endar
en3ni
en2es
shoin
J52in aran coden
12an
n2en
esien
d6.a
502e6
do2co
Jo2b8
Js2es Jo2
Figure 28: Jaccard similarity matrix of token aligmment head sets across 20 language pairs. Here, we broadly categorize the selected language pairs into four groups: en2others (en2zh, en2ja, en2de, en2fr, en2ar, en2nl, en2es), others2en (zh2en, ja2en, ar2en, de2en, fr2en, nl2en, es2en), de2ja/es/ko, and ja2de/es/ko. It is observable that the similarity among the en2others pairs is high, whereas the others2en group and the other language pairs generally exhibit lower similarity.
31


Reviews:
[{"review_id": "IwJI67mFRx", "reviewer": "Reviewer_nLP1", "summary": "The paper investigates the role of translation heads in translation and multilingual tasks. Translation heads are identified based on their accuracy in token mapping. The authors identify several features of these translation heads, including i) that they are universal in all LLMs, ii) that they only a few translation heads needed, iii) that the translation heads are consistent across language pairs, iv) translation capability is severely impacted when these heads are masked and v) that they occasionally have an impact of other multilingual tasks. Finally, they trace the development of these heads across cycles of pretraining and show how the attention heads can be used to filter multilingual data for more efficient training.", "questions": "1. Can you reconcile your finding in Figure 4 with prior literature? see Weakness 2.\n2. In some cases (Figure 9, xwinograd, xnli), pruning translation heads seem to give better results, why?\n3. What is L in Equation 4?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The idea that cross-lingual word-mapping heads exist in multilingual LLMs is interesting, and how they demonstrate impact on translation and some multilingual tasks.\n2. The experiments that evaluate key features of the heads are thorough and comprehensive.\n3. The writing is clear. All motivations and experimental designs are easy to understand.", "weaknesses": "1. Translation heads are identified solely through cross-lingual one-to-one token mapping. However, translation as a process involves more than token mapping alone, e.g., resolving morphological ambiguity. Current name that is translation heads sounds a little misleading.\n2. Previous literature show that transformer-based models tend to process multilingual input in English in the middle layers:\n- https://aclanthology.org/2024.acl-long.820/\n- https://arxiv.org/abs/2411.04986\n- https://arxiv.org/abs/2402.18815\nThis seems incompatible with your finding in Figure 4 where translation heads are mostly gathered in the middle. What roles exactly do translation heads in the middle layer play in solving multilingual queries?\n3. While the role of translation heads is clear on translation task, the results on other multilingual task are less conclusive. There is no clear analysis leading us to understand why this is the case, or the kind of role the attention heads play in solving these tasks.\n4. Minor: fonts are too small for all figures.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JrEr7fj3p5", "reviewer": "Reviewer_DWDG", "summary": "This paper utilizes token-aligned attention scores to detect salient translation heads (TH) in LLMs. The authors find that such attention heads significantly affect multilingual proficiency, as validated by the multilingual dataset FLORES-101. Experiments demonstrate substantial overlapping of salient heads across 10 different language pairs and confirm the deterministic effect of these heads through comprehensive ablation studies.", "questions": "1. Given that this is not the first study to explore the multilingual abilities of LLMs, what are the key distinctions between this work and previous research?\n2. How do current machine translation metrics demonstrate the significance of translation heads on overall MT performance?\n3. Since BLEU and chrF++ metrics cannot inherently recognize languages, is it possible that these translation heads are not fully responsible for multilingual capacity if LLMs cannot function properly without them?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "1. This work presents a token-aligned detection approach with unsupervised LLM labelling that requires inference only. It demonstrates the significance of THs through a masking approach for machine translation.\n2. This work illustrates the properties of THs through detailed analysis across different models and languages.\n3. This work unveils the learning process for translation heads during model training.", "weaknesses": "1. This work's organization lacks rigorous experimental settings. For instance, Figure 1 fails to adequately demonstrate the function of THs through its single example. Additionally, the selection of models appears arbitrary, lacking consistency in either model size or type.\n2. The figures throughout the paper are blurry, making it difficult to discern key numerical values (particularly in Figures 3 and 4).\n3. The proposed data augmentation method does not demonstrate significant improvements in Table 1 when compared to the baselines.\n4. Despite numerous recent studies exploring the internal mechanisms of multilingual capacity in LLMs, this work lacks a thorough review of related work in this field.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GT2A1Sq4dG", "reviewer": "Reviewer_ZKfU", "summary": "The presented paper analyzes that role of specific attention heads for the translation task aka translation heads. These are identified with a word alignment approach and thoroughly analyzed for their location, sparsity, language generalization, emergence, as well as impact on multilingual evaluation benchmarks.", "questions": "N/A", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Interesting way of determining translation heads via word alignment properties compared to downstream task performance as done in previous works\n- Insightful analysis on the location, sparsity, language generalization, and emergence of translation heads\n- Insightful analysis for masking translation heads on downstream tasks including not only translation but also general multilingual benchmarks", "weaknesses": "### Weaknesses\n\n- It would've been interesting to analyze the number of translation heads that are specialized for a specific language / language-pair despite them having a large Jaccard index i.e. `de2ja` seems to have a lower similarity to the other language pairs. Generally, it could be that some language pairs are deemed \"harder\" and need specialized translation heads. I'm a bit surprised to see that `en2de` and `en2nl`, two very closely related language pairs, have the same overlap ratio as `en2de` and `en2zh`, two fairly distant language pairs. It could also be that the deciding factor here is `en` as the source language, rather than the target language.\n\n### Minor Comments\n\n- The \"token mapping\" is the task of word alignment and has been around for many years in machine translation literature and probably should be mentioned as such.\n- All Figures should likely be included as svg graphics instead of low resolution png images since it makes it more appealing to view details. In Figure 3 and Figure 4 some of the labels are way too small to even view properly. A bit more effort on properly scaling the figures would improve the reading experience.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)