LinguaMap: Which Layers of LLMs Speak Your Language and How to Tune Them?
Authors: J. Ben Tamo
Daniel Carlander-Reuterfelt
Jonathan Rubin
Oleg Poliannikov
Dezhi Hong
Mingxian Wang
Venue: ICLR 2026 Poster (2026)
Keywords: Multilingual Language Models
Language Consistency
Cross-lingual Transfer
Interpretability
Logit Lens
Semantic Similarity
Layerwise Fine-Tuning
Output Space Control
Model Analysis
Language Control
Primary area: interpretability and explainable AI

Abstract: Despite multilingual pretraining, large language models often struggle with non-English tasks, particularly in language control--the ability to respond in the intended language. We identify and characterize two key failure modes: the *multilingual transfer bottleneck* (correct language, incorrect task response) and the *language consistency bottleneck* (correct task response, wrong language). To systematically surface these issues, we design a four-scenario evaluation protocol spanning MMLU, MGSM, and XQuAD benchmarks. 
To probe these issues with interpretability, we extend logit lens analysis to track language probabilities layer by layer and compute cross-lingual semantic similarity of hidden states. The results reveal a three-phase internal structure: early layers align inputs into shared semantic space, middle layers perform task reasoning, and late layers drive language-specific generation. Guided by these insights, we introduce *selective fine-tuning* of only the final layers responsible for language control. On Qwen-3-32B and Bloom-7.1B, this method achieves over 98% language consistency across six languages while fine-tuning only 3–5% of parameters, without sacrificing task accuracy. Importantly, this result is nearly identical to that of full-scope fine-tuning (e.g., $>98\%$ language consistency for both methods across all prompt scenarios) but uses a fraction of the computational resources. To the best of our knowledge, this is the first approach to leverage *layer-localization of language control* for efficient multilingual adaptation.

Published as a conference paper at ICLR 2026
LINGUAMAP: WHICH LAYERS OF LLMs SPEAK YOUR LANGUAGE AND HOW TO TUNE THEM?
J. Ben Tamo', Daniel Carlander- -entterr-Reteereett- 2, Jonathan Rubin', Oleg Poliannikov Dezhi Hong", Mingxian Wang
Georgia Institute of Technology
Amazon
ABSTRACT
Despite multilingual pretraining, large language models often struggle with non- English tasks, particularly in language control-the ability respond in the in- tended language. We identify and characterize two key failure modes: the mul- tilingual transfer bottleneck (correct language, incorrect task response) and the language consistency bottleneck (correct task response, wrong language). To systematically surface these issues, we design four-scenario evaluation proto- col spanning MMLU, MGSM, and XQuAD benchmarks. To probe these issues with interpretability, we extend logit lens analysis to track language probabili- ties layer by layer and compute cross-lingual semantic similarity of hidden states. The results reveal a three-phase internal structure: early layers align inputs into shared semantic space, middle layers perform task reasoning, and late layers drive Lunguage-specffee generation. Guided by these insights, we introduce selective fine-tuning of only the final layers responsible for language control. On Qwen-3- 32B and Bloom- 7.1B, this method achieves over 98% language consistency across six languages while fine-tuning only 3-5% of parameters, without sacrificing task accuracy. Importantly, this result is nearly identical to that of full-scope fine- tuning (e.g., 98% language consistency for both methods across all prompt scenarios) but uses fraction of the computational resources. To the best of our knowledge, this is the first approach to leverage layer-localization of language control for efficient multilingual adaptation.
INTRODUCTION
The growing deployment of multilingual large language models (mLLMs) promises bridge lin- guistic divides and democratize access to information across the world's languages. Early models such as mBERT (Devlin et al. 2019). XL.M-R (Conneau al. 2020), and mT5 (Xue et al. 2020) demonstrated impressive cross- lingull generalization, while more recent large-scale L.MMs, such as PaLM-2 (Anil et al., 2023) and GPT-4, have shown even stronger multilingual capabilities, of- ten without explicit multilinuual supervision. Alongside these proprictary models, expanding ecosystem of openly available multilingual LLMs has emerged, including BLOOM (Le Scao et al,, 2022), LLaMA (Touvron et al. (023). and Qwen (Yang al., (025). Despite this progress, find that these models still exhibit persistent failures in language control, namely, the ability to respond in the intended language, even when they correctly solve the underlying task.
To systematically characterize multilingual failures, we introduce a targeted evaluation framework with four zero-shot prompt variants, each isolating different aspect of language control. (1) Mono- lingual Direct Prompting tests whether models can follow instructions and respond exclusively in the target language; (2) Code-Switched Prompting examines robustness to mixed-language input; (3) Bilingual Answer Prompting probes language preference when correct answers are presented in both the target language and English; and (4) English Distractor Prompting tests resistance to incorrect English alternatives.
WWrk completed while at Amazon. Correspondence to jtamo3ggatech. edu, (poliann, iann, mingxiiu

---

Published as a conference paper at ICLR 2026
[a] Alignment
Reasoning
(c) Outpui Language
MI 1000
410
471
201
Layer
Layer
Layer
Laper j+1
Laver
Legend:
Hidden State Similarity (En XX) Perfect Similarity (En
Layers
Early layers are frozen to pre- serve semantic alignment, mid layers maintain task reasoning, and only upper layers finetuned to introduce language-specific output control, enabling efficient multilingual adaptation with minimal disruption model capabilities.
Figure 1: Overview of Selective Finetuning for Language Control:
This evaluation reveals two failure modes: (1) language consistency bottleneck, where a model generates the correct answer but in the wrong language; (2) multilingual transfer bottleneck, where a model generates output in the correct language but fails tasks can solve in English. These failures highlight deeper disconnect between task competence and language control in mLLMs, suggesting that they are governed by distinct internal mechanisms.
Failures in language control often stem from Anglocentric pretraining, limitations in shared multi- lingual representations, and interference across typologically diverse languages (Huang et al. 2024
Zhao et al. 2024 Papadimitriou et al., 2023). While prior work has explored fine-tuning (Artetxe embeddings (Cao et al. 2020), prompt engineering (Shi et al. 2023 Vatsal et 2025), , and monolingual specialization (Dobler & de Melo 2023). these approaches of- ten face trade-ofis in scalability and coverage. Understanding how internal representations shape cross-lingual behavior remains an open challenge. We ask: Where in the model do language- specific behaviors-such as language consistency, dominance bias, and multilingual interference- reside, and can they be isolated to enable efficient and effective multilingual adaptation?
et al..| 20000).
We answer this by taking structural, mechanistic view. We apply logit lens analysis of language token probabilities and semantic similarity evaluation of multilingual hidden states. Both analyses converge threc-space structure, semantic alignment phase, reasoning phase. and language output phase, previously hypothesized in recent studies (Zhao et al., 2024, Wendler et al., [024]] Etxaniz al. 2024 Schut et al.) 2025 Lindsey et al. 2025): (i) Early layers gradually normalize language inputs into a shared semantic space, (ii) mid layers perform task reasoning, and (iii) late
layers control language-specific output.
Building on this understanding, we introduce layer-wise selective fine-tuning, a lightweight method that targets only the final output layers responsible for language control. Applied to models like Qwen-3-32B and Bloom-7. it improves language consistency from <20% to 98+% across six languages, while preserving task performance and updating fewer parameters than full fine-tuning.
Main contributions. This work (1) introduces a framework for evaluating language control in mLLMs, incorporating systematic prompt variation to diagnose multilingual failure modes, (2) un- covers and validates a three-space structure in mLLMs, where distinct layers specialize in semantic alignment, reasoning, and language generation, and (3) proposes and validates layer-wise selective fine-tuning as an efficient and effective method to correct language consistency failures without compromising performance.
RELATED WORK
Recent advances in multilingual language models have revealed deep structural asymmetries favor- ing English, even in models trained across diverse linguistic corpora.
2

---

Published as a conference paper at ICLR 2026
2.1 LATENT ENGLISH DOMINANCE IN MLLMS
Multiple studies reveal that state-of-the-an multilingual transformers often process non-English in- puts via internal English representations. [Schut et al. (2025) and Wendler et al. (2024) empirically confirm models like LLaMA-2 implicitly reason in intermediate English-based latent spaces, even when inputs and outputs are in other languages. [Wendler et al. (2024) formalize this through their multilingual workflow hypothesis, showing that multilingual reasoning commonly pivots through English representations intermediate layers. Complementing these findings, Lindsey et al. (2025) introduces the concept of multilingual circuits and further highlights that multilingual models often use English as the default internal representation, implying asymmetrical semantic spaces biased toward English. These findings collectively indicate an internal "English-thinkinn" phenomenon, contrasting the apparent multilingual capabilities observed externally.
2.2 LANGUAGE LOCALIZATION AND NEURON INSIGHTS
Building interpretability research, Tang et al
2024 show that multilingual models contain distinct clusters of neurons selectively responsive to particular languages. Wang et al. (2024) further confirm that input/output layers exhibit stronger language-agnoste concepts. Zhao et al. (2024) confirm this belief through parallel language-neuron detection and conclude that there are three spaces: input, conceptual, and output spaces. These insights sparked research in language adaptability. Preiffer et al.] (2020) introduce an invertible adapter architecture for adapting a pre-trained multilingual model new language. [Huo et al. (2025) propose deep supervision fine-tuning, explicitly aligning internal representations across lay- ers, significantly reducing latent English bias. Similarly, [Liu & Niehues (2025) emphasize explicit representational alignment during fine-tuning at intermediate layers, promoting cross- coos-llnngull seman- tic consistency and improving zero-shot transfer. Kew et al. (2024) explores how much multilingual finetuning is needed to turn English-centric models into "polyglots," and Zhong et al. (2025) inves- tigates which internal language representations non-English-centri models use during inference.
seeguage-speciiie activation, whereas middle layers encode
Collectively, these works suggest that multilingualism in L.MMs is not uniformly supported at all representational layers. While embedding layers may provide aligned token representations across languages, deeper layers exhibit emergent specialization or drift toward dominant languages. Build- ing upon these insights, our work further identifies tanguage-specin processing layers and, based on this finding, proposes efficient fine-tuning strategies enhance multilingual performance.
3
A PROMPT-BASED FRAMEWORK FOR DIAGNOSING LANGUAGE CONTROL.
3.1 PROMPT STRUCTURE AND COMPONENT DESIGN
Our framework consists of four zero-shot prompting variants, each probing a distinct aspect of lan- guage control. We define a prompt as comprising three main input components: Preamble (P)-the the metadata that frames the task; Instruction (I) the explicit directive describing the task to perform; and Question (Q)-the the task content itself, such as question or passage. The mLLMs respond with two output components: Reasoning (R) and Answer (A).
We evaluate performance across three multilingual benchmarks: MMLU Hendrycks et al. 2021 MGSM et (2023), and XQuAD/Artetxe et al. (2020a). which cover multiple-choice (MMLU), generative reasoning (MGSM), and extractive span-based answering (XQuAD). The zero-shot prompt variants (see Figure in Appendix) include:
Monolingual Direct Prompting, which tests baseline fidelity when both instructions and content are in the target language;
Code-Switched Prompting, where the instruction metadata is in one language (e.g., En- glish), while the task content (e.g., the question) written in the target language, testing the model's ability to resolve linguistic context under mixed mixed-language input;
English Distractor Prompting, which includes incorrect English answers to test rejection of misleading output.
Bilingual Answer Prompting, which presents correct answers in both the target language and English to probe language preference;
3

---

Published as a conference paper at ICLR 2026
To isolate linguistic effects, we keep the semantic content for the correct answer constant across languages and measure language consistency, whether responses are in the intended language, re- gardless of correctness. We apply each prompt variant to a standardized set of questions across six typologically and scripturally diverse languages: English, French, Spanish, Arabic, Hindi, and Japanese.
3.2 MULTILINGUAL BENCHMARK SETUP AND METRICS
To analyze multilingual model behavior more precisely. we decompose performance along two or- thogonal axes: task accuracy and language consistency. Task accuracy evaluates whether the model provides the correct answer, regardless of the output language. Language consistency refers to whether the response is delivered entirely in the intended target language. We focus the two most revealing failure modes:
Multilingual Transfer Bottleneck: The model responds the correct language but fails to provide the correct answer, despite likely being capable of solving the task in another language (e.g., English).
Language Consistency Bottleneck: The model produces a correct answer but in the wrong language, indicating difficulty in adhering to the requested linguistic context.
Language consistency is computed the proportion of responses whose primary language matches the target language. Let y; be the model output for example i, and Lang be the expected out- put language. Let Lang(y;) be the predicted language of the model output, determined using the LangDetect language identifier by (Shuyo 2010
3.3 FINDINGS: WHEN AND HOW MLLMS FAIL
We evaluate two multilingual LLMs, Qwen-3-32B, and BLOOM-7.1B, on MMLU, XQuAD, and MGSM under zero-shot settings, focusing on two core dimensions: task performance and language consistency, as presented in Table These benchmarks collectively span factual knowledge, multi- lingual reasoning, and mathematical problem solving across diverse languages.
Table TeeeTrrerrtttt average scores across all evaluated languages for each dataset, MMLU (6: en, es, fr, ar, ja), MGSM (3: en, fr, ja), and XQuAD (4: es, ar, hi). Switching from monolingual to code- switched prompts often leaves average task accuracy largely intact but can sharply degrade language consistency. For example. Owen-3-32B maintains strong average MMLU accuracy (60.5% under code-switched prompts vs. 51.77% monolingual), yet its average language consistency drops from 45.17% just 8.35%. BLO0M-7.1B shows the same pattern; the language consistency across all three datasets drops while the task performance remains comparable or slightly better.
When averaged across languages and prompt types, Qwen-3-32B consistently achieves the high- est task scores (e.g., 66.6% MGSM monolingual, 55.54 F1 on XQuAD monolingual) but suffers severe language consistency losses, often into single digits, whenever prompts mix languages in- clude English distractors. BLOOM-7.1 1,1.7.1B..B. in contrast, underperforms on both metrics, with average MGSM accuracies <0.67% and XQuAD Fl scores frequently under 7%, despite occasionally high consistency in certain monolingual conditions. These trends suggest that Qwen-3-32B is optimized for multilingual task utility, and BLOOM, despite moderate language control, fails engage with task semantics.
Stress tests expose finer-grained weaknesses. Under English- prompting (MMLU; aver- aged across languages), Qwen-3-32B's language consistency drops from 45.1 45.17% to 23.54%, while accuracy declines from 51.77% to 36.93%. Across prompt types, damage often correlates with lan- guage distance in per- per-language breakdowns (Appendix Tables 4 5 6). suggesting that shared sub- word inventory may cushion losses. BLOOM's scores remain uniformly low, with average MGSM accuracies 50.67% and XQuAD F1 scores frequently under 7%, reinforcing that its limitations capacity-driven rather than prompt-specific.
These results reflect differing model priorities: architectures, like Qwen, favor task success even at the cost of language control, while others, like BLOOM, attempt to enforce language control more strictly. Qwen-3-322 consistently achieves high accuracy across domains and languages but
4

---

Published as a conference paper at ICLR 2026
Table 1: Multilingual Trade-offs Across Prompting Strategies: Evaluated MMLU languages: en, es, fr, hi, ja), MGSM (3: en, fr, ja), and XQuAD (4: en, es, hi), Qwen-3-32B achieves the highest task performance but suffers major drops language consistency; Bloom-7.1B lags on both. Overall, robustness to cross-lingual prompt perturbations often comes at the expense of peak task accuracy. Complete breakdowns for all datasets are provided Appendix Tables 5
Prompting
Dataset
Bloom 7.1B
Qwen-3 32B
Language Consistency (%)
Task Performance (%)
Language Consistency (%)
Task Performance (%)
MMLJ
67.98
15.83
45.17
51.77
Monolingual Q- (X)
MGSM
34,00
65.56
66.60
XQuAD
98.32
4.18
81.05
$5,54
MMLU
29.49
22.31
8.35
60.50
Code-Switched P,I-(EN), Q(X)
MGSM
18.41
0.40
6.84
57.00
XQuAD
71.23
11.01
52.65
English-Distracto IX)),, Q(X & EN)
MMLU
40.00
10.51
23.54
36.93
XQuAD
69.44
0.67
41.99
15.81
Billngual-Answer I-(),, Q(X & EN)
MMLU
59.61
9.36
23.50
35.76
struggles with stable language control, particularly under mixed prompt languages. BLO0M-7.1B, despite its fluent output, lacks the semantic depth required for effective multilingual reasoning.
4 WHERE LANGUAGE CONTROL EMERGES: LAYER-WISE INTERPRETABILITT
Prompt-level behavior shows failures in language control, particularly under code-switching, but the mechanisms driving output language choice in multilingual LLMs remain unclear. We use inter- pretability tools to probe internal activations, identifying where language control emerges and how inconsistencies are encoded in hidden representations.
4.1
METHOPS FOR PROBING INTERNAL REPRESENTATION
4.1.1 DECODING INTERNAL LANGUAGE PROBABILITIES WITH THE LOGIT LENS
To trace the evolution of language preferences in multilingual LLMs, we use logit lens decoding nostalgebbraist al. 2021), which projects intermediate hidden states onto the output vocabulary via the model's language modeling head. At each layer 1, we compute pseudo-logits by projecting the intermediate state h through the unembedding matrix U R V[xd,
2 [Ub]]] u, h
(1)
where us E RRd the embedding of vocabulary token t. These pseudo-logits approximate the model's next-token distribution each layer.
For each position i in the generation, we decode the most likely token from the pseudo-logits at every layer, yielding an M-length intermediate sequence per layer when the model generates M tokens. After reconstructing full words from subword tokens, we compute language probabilities using the langdetect language identifier library (Shuyo, 2010). Operating at the word level avoids the ambiguity introduced by multilingual subword overlap. The identifier compares each recon- structed word against pre-trained language profiles derived from character-distribution statisties and returns normalized probabilities over languages:
exp (s((s ,0)
p (0)
Erec exp(s(0))).))) (())
where p (2) denotes the probability that decoded word 8 belongs to language This word- level approach ensures that language identification relies on words from full decoded sequences,

---

Published as a conference paper at ICLR 2026
providing more stable and robust signal than subword-level approaches in multilingual settings. By aggregating word-level language predictions, we estimate the language probability mass at each layer and track shifts in preference between the target language and dominant alternatives (typically English).
M Σ (O. M
(c)
(3)
which represents the average probability mass assigned to language é at layer 1. Tracking p() (t) across layers yields the trajectory language drift during generation.
4.1.2 HIDDEN STATE SIMILARITY ANALYSIS
We perform layer- wise analysis of hidden state similarity across language pairs using cosine sim- ilarity. Given a set of aligned prompts {(ah (E) (A)) 1 n==1 where each pair consists of semantically equivalent inputs in English and another language (e.g., Spanish), we pass each prompt through the model and extract hidden states at each layer E fO..... L}. including the embedding layer. We compare the internal representations layer-by- by-layer determine where they begin to diverge.
For each input, the hidden states at layer l are denoted h, (B.m) E RT...) xd and h (A(.))) E T
where d is the hidden size and 7 is the sequence length. To obtain fixed-size prompt representation per layer, we apply mean pooling over all token embeddings in the input sequence:
T(E)
T(
7. B.E..)
(E.)) 1et
T((A.))
h
(A.) Ex
(4)
T
TH
A)
We then compute the cosine similarity between the mean mean-pooled representations:
1i J.(A,n))
(5)
(E,n) |1 -.1n(a.)))
(1.1)
Aggregating across the dataset yields the average and standard deviation of similarity per layer:
S
o:
1
St (n)
5c
N
This mean-pooled prompt similarity analysis offers high-level but interpretable view of how rep- resentations evolve across layers. Mean-pooled hidden-state cosine similarity (equation 5 ad equa- tion (6) robustly captures global, sequence-level semantic alignment, even when cross-lingual tok- enization differs substantially across languages. Although this abstraction hides token-level diver- gence in attention or contextual span, loken-wise comparisons highly sensitive to tokenization mismatch and require non-trivia alignment across sequences of different lengths, often introducing noise that obscures the underlying conceptual structure.
4.2 FINDINGS ON LAYER LANGUAGE CONTROL AND REPRESENTATION
We apply these interpretability methods across monolingual and code-switched prompting, in five non-English languages (ES, FR, AR, HI, JA). English is never used the sole prompt language. Our analysis compares how much target language control vs. English dominance emerges at different network depths. Figures and Jjointly trace how multilingual LLMs control and represent language across layers. The logit lens analysis shows how word-level language probabilities evolve, while hidden-state similarity analysis examines the degree to which parallel prompts in different languages occupy shared representation space.
4.2.1 MODEL-SPECIHIC PATTERNS IN LANGUAGE CONTROL AND REPRESENTATION
In 1itoom-7.1B monolingual prompts yield high target-languaat probabilities for Arabic, Hindi, and Japanese (0.4-1 up to layer 10), with Spanish and French slightly lower but still exceeding English.

---

Published as a conference paper at ICLR 2026
d6omm77 Coate Longane
Ovend Coos Irseer www.s/ooooogeeeoo.... Formot
Ovens Lww.oagggggiiioooooo Popal
(a) Bloom-7.1BB Monolingual
(b) Qwen-3-8B. Monolingual
(c) 1-@ewen--332 3-32B: Monolingual
Obanvit Coms-ianuuec rebso then for Codl Subtred Pronging
Over Crascergage toose thiss for code, setched honggs
Owen Longasge mebed Code Swncried hoomerng
(d) Bloom-7.1B. Code-Switched
(e) Qwen-3-8B: Code-Switched
(1) Qwen-3-32B: Code-Switched
Figure 2: Cross-Language Probability by Layer under Monolingual and Switchhe Prompting on MMLU. . In Qwen, early layers are relatively biased to English, middle layers sustain English bias, and final layers shift toward Language-specingg processing. However, code-switching disrupts this control, especially in Qwen. Bloom exhibits more language-specific layers with no bias.
State
Stare
State
Freeatty
Casa Sarmarty
2n AC Reat Conts aniatty Hil Mean Caine amilttly
6A- Mune Collne dan krrty
22 AC Heat Collts miaaatty Hi Pean Carie Smilrtly
Mean
(a) Bloom- 7.1B: Monolingual
(b) (wwen-3-BB. Monolingual
(c) Owwen-3-32B: Monolingual
Figure 3: Layer-wise hidden-state cosine similarity for monolingual MMLUU prompts. Each sub- figure shows similarity between English and five target languages (ES, FR, JA, AR, HI) across the embedding output and transformer layers. Similarity rises sharply in early layers, remains stable in mid-layers where cross-lingual semantic alignment is strongest, and declines in the final layers.
These probabilities weaken in mid-layers but recover strongly after layer 20, while English remains consistently suppressed (<0.1). The wide shaded regions (standard deviations) reveal substantial variability across layers, suggesting that Bloom's intermediate layers mix cross-lingual features, producing ambiguous intermediate decodings and thus unstable language probability estimates. Un- der code-switching, however, target-language control collapses (probabilities <0.2), with only par- tial recovery for Arabic, Hindi, and Spanish. Representation-wine high cross-lingual similarity in early layers. Middle layers maintain strong similarity (0.97-0.99), implying a strongly language-invariant semantic space. Only in late layers (24-30) do sharp diver- gences appear, especially for En-Ar (to 0.36) and En-Hi (to 0.42), reflecting linguistic divergence: language control emerges. Qwen-3-8B exhibits consistently English- domiinat::mnnn::: its monolingual prompting behavior (Figure 2 shows English dominating generation bias across most layers, while larget-language probabilities (ES, FR, AR, HI, JA) start weak, remain suppressed through the mid- dle layers, and only FR recovers partially after layer 25. Similarly, under code-switched prompting (Figure [}, Qwen-3-8B's language control collapses fully: langet-aaguage probabilities remain sup- pressed in the final layers, and English becomes the sole generation language.
Bloom shows a rapid increase to
In Qwen-3-32B, English dominates early regardless of input language, with target-language proba- bilities rising only after layer 55. This layer marks the data-driven onset of anguage-specite gener- ation, defined operationally by the convergence of two independent and empirically casy-to-identify indicators: the layer where the target language probability first surpasses English, and the layer where cross-lingual hidden-state similarity sustains divergence from the stable, middle-layer align-

---

Published as a conference paper at ICLR 2026
ment. Even after this emergence, recovery is incomplete, and under code-switching, re re-grounding fails altogether. Hidden-state analysis shows very high similarity across languages in middle layers 6-55 (En-Es/Fr near 0.99, En-Ar/i 0.95-0.97, En-Hi just under 0.9). After layer 55, similarity di- verges slightly, which does not translate into effective language control: English remains entrenched the dominant generation bias.
4.2.2 CROSS-LINGUISTIC DIFFERENCES IN REPRESENTATION AND CONTROL
Across models, we observe a three-phase structure, early convergence to a shared semantic space, stable middle layers, and late divergence into language-specific generation, but the stability of lan- guage control differs. Bloom-7.1B shows high early larget-language probabilities but with large variance across layers, reflecting ambiguous intermediate decodings where hidden states straddle multiple languages. In contrast, Qwen-3-32B is stable but strongly nates early and mid layers, target-language probabilities rise only after layer 55, and recovery fails under code-switching. These contrasts suggest that instability in Bloom arises from ambiguous intermediate representations, while Qwen's consistency reflects entrenched bias.
biased: English domi-
5 LAYER- LAYER-WISE SELECTIVE TUNING FOR LANGUAGE CONSISTENCY
The analyses in Sections and 4 demonstrate that mLLMs often lose language control under ad- versarial prompts, failure linked unstable late-layer re-grounding. To address this, propose layer-wise selective supervised fine-tuning (SFT) that targets language control mechanisms without full model retraining.
5.1 How TO TUNE LANGUAGE CONTROL: SELECTIVE SUPERVISED FINE- -UNNIT
Our goal is to reinforce language consistency, the model's ability to produce outputs strictly in the intended language, while minimizing interference with general task competence coverage.
Consider a pretrained model with parameter set = {0, 00..00.... ......,, ,,. thead] where 6g corresponds layer E, and Orend is the embedding and LM head. Rather than tuning all layers, update only a subset S C (1, (1,...,L), typically the last k layers, where language-specific generation behavior emerges.
We define selective SFT as fine-tuning only a subset of parameters 0s. while keeping the remaining parameters S frozen. Given training data { y/) ) the optimization objective minimize:
S (fs) = Σ log P(us 0s).
where gradients computed only with respect to This formulation isolates adaptation to the selected components while leveraging the frozen parameters to preserve the pretrained model's 's se- mantic alignment and reasoning capacity.
To evaluate Selective SFT, we fine-tuned on a domain-focused MMLU subset covering five business subjects (ethics, marketing, management, accounting, public relations) across five languages (Span- ish, French, Arabic, Hindi, Japanese). From the pool of correctly answered examples (verified with Claude 3.5 Sonnet), we sampled 500 per subject, yielding 2,500 examples split 80/20 into training and validation sets. Each instance was augmented with chain-of-thought reasoning traces aligned with the question's 's language. Prompts followed five-part template (Preamble (P), Instruction (I), Question (Q), Reasoning (R), Answer (A)), with loss restricted to the Q, R, A tokens while P and I remained frozen context:
muskked Selective (0s) Σ m; log P(y: Qi, RG, As;
where m, E (0. 1} masks tokens outside the Q, R, A, and gradients are applied only to fs..
An ablation varying tuned last layers (1,...,n) and epochs (1-5) (see Appendix Tables 10 11 showed that the optimal configuration was the last layer at 5 epochs for Bloom-7.1 and the last two layers at 5 epochs for Qwen-3-32B.
8

---

Published as a conference paper at ICLR 2026
Table 2: Impact of fine-tuning on language consistency and task performance for Qwen-3-32B and Bloom-7.1B 1B on MGSM, MMLU, and XQuAD. Both models were fine-tuned with code-switched prompts in the MMLU Business domain across six languages, then evaluated on MMLU non- Business subjects (52 in total), MGSM, and XQuAD. Values represent averages across all evaluation languages for each dataset; full per-dataset results appear in the Appendix Tables 7
Prompting
Datasets
Model
Pre-Finetuning
Fall scope
Random Selective
Selective
Language Cons. (%)
Task (%)
Language Cons. (%)
Task (%)
Language Cons. (%)
Task (%)
Language Cons. (%)
Task (%)
Trainable Param
Qwen-3.328
NA
32B
1.5B
1.58
Aloom-7.1B
NA
7.18
0.5B
0.5B
Monolingual P. (X)
MGSM (Avg)
Qwen-3-32B
65.56
34,00
66 60
99.47
90.53
1.47
65.87
69.47
0.13
100.00
99.20
86,80
3.60
XQuAD (Avg)
Qwen-3-32D
$1.05
55.54
100.00
57.60
47.44
0.42
99.83
55.86
Bloom-7.18
98.32
4.18
99.91
16.85
54.10
0.00
99.85
20.83
(Avg)
Bloom- 7.1B
Owen-3-221
29.49
22.3]
99.87
78.84
33.72
98.30
55.56
0.00
98.66
21.14
74.44
Code Switched (EN), Q(X)
MGSM (Arg)
Bloom- 7.1B
Qwen-3-322
18.40
6.80
57.00
0.40
95.00
100
87.00
2.20
68.00
53.80
0.00
0.00
99.60
98.60
84,60
2.00
XQaAD (Avg)
Qwen-3.322
Bloom-7.188
11:01
71.23
$2.65
6.58
100.00
99.89
21.03
51.87
43.11
97.95
0.00
1:10
100.00
99.80
21.02
53.53
English Distractor (X... QXX EN)
XQUAD (Avg)
Bloom- 7.1B
Qwen-3-32B
41.99
15.81
75.99
97.03
17.77
6.96
37.78
26.16
0.00
98.23
97.62
18.05
5.2 RESULTS AND FINDINGS
Table indicates that, before fine-tuning, Qwen-3-32B shows moderate task accuracy but poor lan- guage control, achieving 66.6% on MGSM and 55.5% on XQuAD for monolingual prompts, while collapsing under code-switching with only 6-11% language consistency. Bloom-7.1B maintains higher consistency (34-98%) but far weaker in task accuracy (0.4-15.8% ),), often producing text in the target language without solving the task. Full-scope SFT substantially improves Qwen, raising consistency to nearly 100% across all regimes and boosting task accuracy (e.g., MGSM from 66.6% to 90.5%). Code-switched settings, initially unstable, are restored above 95% language consistency with 78 87% task accuracy. Bloom also reaches near- pear-perfect language consistency after full-scope SFT, though without comparable reasoning gains. Overall, full-scope SFT enforces consistent lan- guage use in both models, with Qwen uniquely leveraging this for improved task performance.
When tuning on random subset of layers, both models collapse task performance, despite retain- ing some language consistency. For example, Qwen's 's monolingual MGSM language consistency falls from nearly 100% (selective-sft) to 65.9% under random selective SFT, and Bloom shows a similar decline (from 100% to 69%). In contrast, Selective SFT recovers near-perfect language consistency across datasets. Both Qwen and Bloom maintain 99% language consistency in mono- lingual, code-switched, and English-
prompts. These results demonstrate that targeted layer adaptation preserves language consistency, whereas random selection destabilizes generation and erodes cross-lingual consistency. Selective SFT achieves nearly the same performance full- scope SFT for both Qwen and Bloom, while requiring updates to only 3-5% of the parameters, compared full-scope SFT. Random selective SFT catastrophic, reinforeing the importance of principled parameter selection. Under English distractor prompts, selective SFT substantially im- proves language consistency, yet task performance remains weak, highlighting the need for more explicit reasoning- disambiguation strategies in future work.
A critical diagnostic concern for our Selective SFT approach is whether the language control ad- justments propagate backward, altering the semantic and reasoning alignments in the frozen middle layers. To address this, we conducted full fost-iinettuning analysis using our original interpretabil- ity tools. As shown in Figure 4 the substantial increase in target-language probability is confined strictly to the late layers (the tuned region), confirming the successful localization of the interven- tion. Furthermore, Figure 3 direct empirical evidence of invariance: the high cross-lingual alignment signature in the semantically- identical to the pre-fine-tuning state. This analysis validates that Selective SFT successfully iso- lates the language control mechanism in the final layers without compromising the integrity of the model's core, language-invarian reasoning capabilities.
middle layers is fully preserved, remaining virtually

---

Published as a conference paper at ICLR 2026
Foom dww.-eetttrrrtoo Cres aaneen aidiree cerdingual
Oherd Celative
Langrngs hood
(a) Bloom-7.1B Monolingual
(b) Qwen-3-8B. Monolingual
(c) Qwen-3-3222 Monolingual
1ewe: Crmslanga irbeaiiiss for dods panced
Coench Sww.eaa--leetteeeee SIT Con-uangegg reteo ferood, suriieer
Coenh Sateereeeeeeeeeee 277 Cross Langage Frouas ths cack puicher
(d) Bloom-7.1BB Code Switched
(e) Qwen-3-8B: Code-Switched
(1) Qwen-3-32B: Code-Switched
Figure 4: Post-Selective SFT Liser-wise Language Probability Trajectories. The plots, shown un- der Monolingual and Code-Switched prompting, confirm the localization of the intervention: non- English target-language probabilities substantially increase and dominate only in the late layers (the tuned region), with minimal change observed in the early and middle layers.
Joi State
Maral wg Pervtii
cdee State Cooine Sindaatty Actrr Menciigual Parpligg
reobee Sota Costre 5m ofly Acroa Meroingud hampping
Moon
IEn Hoin tosing
Neat Coline Sam lufty
Moon casino lufty
180 sinbuity
Contne
tean Colkno siviartyy
to Cooree fan
161 3il Mooe Calhne Si brfty
10n J0 Waan codiee Sin lavitt
(a) Bloom-7.1B. 7.1B: Monolingual
(b) Qwen-3-8B. Monolingual
(c) Qwen-3-32B Monolingual
Figure 5: Post-Selective SFT Hidden State Cosine Similarity Across Layers. The results demon- strate the stability the frozen layers, maintaining the high -ingull similarity signature in the language-invariant middle layers and confirming that the language control adjustments did not propagate backward to alter the semantic alignment.
6 CONCLUSION
LinguaMap details how multilingual language control is distributed across layers in LLMs. By uncovering robust three-phase structure, from shared semantic grounding to language-specific de- coding, we pinpoint where models "think" versus where they "speak". This insight exposes distinct model tradeoffs: Qwen-3-32B excels at multilingual accurate task completion but often sacrifices language control; and Bloom-7.1B, while consistent in adhering to the intended language, strug- gles to reason reliably across languages. Guided by this structural lens, we introduce selective fine-tuning strategy that focuses exclusively the final layers responsible for language control. As L.MMs continue scaling across cultures and scripts, LinguaMap offers both a diagnostic lens and a tool for aligning them with the world's linguistic diversity.
REPRODUCIBILITY STATEMENT
Below summarize the key aspects of reproducibility, drawing our study design and the sup- porting materials.
Conceptual and Theoretical Transparency
The paper provides a clear conceptual outline and prompt template used in multilingual stress tests, enabling readers understand and replicate approach. While the paper does not introduce
10

---

Published as a conference paper at ICLR 2026
fundamentally new theory, it extends established theoretical tools with appropriate formal statements and proofs, and cites all relevant theoretical references.
Dataset Usage
Our experiments rely on publicly available datasets. We explain the motivation for choosing each dataset and provide proper citations all external data sources. No new datasets are introduced, and all datasets used are already accessible to the research community, allowing others to replicate the experimental results without restrictions.
Computational Experiments
Table specifies the number and range of hyperparameters explored during development and the criteria for selecting final parameter settings. We describe the computing infrastructure, including hardware specifications (CPU/GPU models, memory). Evaluation metrics are formally defined, and their selection is motivated.
REFERENCES
Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. Palm 2 technical report. arXiv preprint arXiv:2305. 2023.
Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. On the cross-lingual transferability of mono- lingual representations. In Dan Jurafsky, Joyce Chai. Natalic Schluter, and Joel Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp- 4623 4637, Online, July 2020a. Association for Computational Linguistics. doi: 10.18653/v1/ 4020.acl-main.420 URL
acl-minnn 421/
Mikel Artetxe, Sebastian Ruder, Dani Yogatama, Gorka Labaka, and Eneko Agiree. A call for more rigor in unsupervised cross-lingual learning. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Computa- tional Linguistics, pp. 7375-7388, Online, July 2020b. Association for Computational Linguis- URL
tics. doi: 10.18653/v 005/33/1122200 1/2020.acl-mmmmne mai..m.....6..
acl-main 658/
Steven Cao, Nikita Kitaev, and Dan Klein. Multilingual alignment of contextual word repre- sentations. In International Conference on Learning Representations, 2020. URL https:
//openzeview.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. Un- supervised cross-lingual representation learning at scale. In Dan Jurafsky. Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 1440-8451, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/v1/2022 acl-00in..mmmmam main.747. URL lttep:://aclanthologeoo org/
1020.acl. main. 7477
Jacob Devlin, Ming-Wes Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume (long and short papers), pp. 4171-4186 2019.
Konstantin Dobler and Gerard de Melo. FOCUS: Effective embedding initialization for mono- lingual specialization of multilingual models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Pro- cessing, pp. 13440-13444 13454, Singapore. December 2023. Association for Computational Linguis- URL 1tLhttss //acl cl antho wotthology.o orgg232333
tics. doi: 1tt.:6655011122221m1mmmmmom p- 8297|
Julen Etxaniz, Gorka Azkune, Aitor Soroa, Oier Lopez de Lacalle, and Mikel Artetxe. Do mul- tilingual language models think better in English? In Kevin Duh, Helena Gomez, and Steven
11

---

Published as a conference paper at ICLR 2026
Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapter of the As- sociation for Computational Linguistics: Human Language Technologies (Volume 2: Short Pa- pers), pp. 550-564 Mexico City, Mexico, June 2024. Association for Computational Linguis- URL wttph:ttps///t //acl antho wnthologyy org/2044.
tics. doi: 10. 11266///1122222.mccssoorrr naacl=short.445
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In 9th International Confer- ence on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenRe- view.net, 2021.
net www.ooomumiddddem fm 3GmQ
Kaiyu Huang, Fengran Mo, Xinyu Zhang, Hongliang Li, You Li, Yuanchi Zhang, Weijian Yi, Yulong Mao, Jinchen Liu, Yuzhuang Xu, al. A survey on large language models with multilingualism: Recent advances and new frontiers. arXiv preprint arXiv:2405. 100.200600. 2024.
Wenshuai Huo, Xiaocheng Feng, Yichong Huang, Chengpeng Fu, Baohang Li, Yangfan Ye, Zhirui Zhang, Dandan Tu, Duyu Tang. Yunfei Lu, et al. Enhancing non-english capabiliities of english- centric large language models through deep supervision fine-tuning. In Proceedings of the AAAI Conference Artificial Intelligence, volume 39, pp.
2025.
Tannon Kew, Florian Schottmann, and Rico Sennrich. Turning English-centie LLMs into poly- glots: How much multilinguality is needed? In Yaser Al-Onaizan, Mohit Bansal, and Yun- Nung Chen (eds.), Findings of the Association for Computational Linguistics: EMNLP 2024, pp. 13097-13124, Miami, Florida, USA, November 2024. Association for Computational Lin- URL
guistics. doi: 10. http://///2/...nnnnnneeeeee/ 2024. fi24.finnings emnlp. 7667
org/
Teven Le Scao, Christopher Akiki, Angela Fan, Ellie Pavlick, Francesco De Toni, and Suzana Ilić. Bloom: 176b- rrob-parameter open-access multilingual language model. Workingpaper, MIT Press, November 2022.
Lindsey et al. On the biology of a large language model. cuttpp://rransfoomee-cccrooo html, 2025.
pub/2025/ 2025/attr
Danni Liu and Jan Niehues. Middle-layer representation alignment for cross-lingual transfer in fine-tuned LLMs. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 15979-15996, Vienna, Austria, July 2025. Association for Computational Linguistics. ISBN 979-8-89176-255 exq/2025.ac1-1ee acn--ong........... 7787
URL https: bittps:/aclaatthoo.s
nostalgebraist et al. interpretting gpt: AcxTBBSwDpdaNNovrrr//nttteet hogpa//wwwaaaee//haaprrrrrnneeno
lens 2021.
Isabel Papadimitriou, Kezia Lopez, and Dan Jurafsky. Multilingual BERT has an accent: Eval- uating English influences on fluency in multilingual models. In Andreas Vlachos and Is- abelle Augenstein (eds.), Findings of the Association for Computational Linguistics: EACL 2023, pp. 1194-1200, Dubrovnik, Croatia, May 2023. Association for Computational Linguis- tics. doi: 10. 10.18653/v 1/2023 htt.2...ininggeaac.... URL htthhtts//// http:::////aaaathhhccccaaa org/2023. Findings-eacl. 89/]
Jonas Pfeiffer, Ivan Vulić, Iryna Gurevych, and Sebastian Ruder. MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfer. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Lan- guage Processing (EMNLP), pp. 7654-7673, Online, November 2020. Association for Computa- tional Linguistiss doi: (0.18653/w1/202...oo org/2020. wwn.pmeannppmmmmmmm wwt.mmainn.n.......mmmm...mm.
main.617. URL https::/aclanthollltt
Lisa Schut, Yarin Gal. and Sebastian Farquhar. Do multilingual LLMs think in english? In ICLR 2025 Workshop on Building Trust in Language Models and Applications, 2025. URL https:
7/openrevies htt.:://ooooommoooooooo0ooooo
12

---

Published as a conference paper at ICLR 2026
Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush Vosoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, Dipanjan Das, and Jason Wei. Lan- guage models are multilingual chain-of-thousee reasoners. In The Eleventh International Confer- ence on Learning Representations, 2023. URL pettps://openrevieete ew. wwt.ff/rnnwettttmmmmmmm did= FR3wGCK-
Nakatani Shuyo. Language detection library for java, 2010. RtL_nttp::/code.goolll.e. com/
3/Lanquage-dtttetiion/
detection/
Tianyi Tang, Wenyang Luo, Haoyang Huang, Dongdong Zhang, Xiaolei Wang, Xin Zhao, Furu Wei, and Ji-Rong Wen. Language-specifie neurons: The key multilingual capabilities in large language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.). Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 5701-5715, Bangkok, Thailand, August 2024. Association for Computational Lin- URLhttps: ha.lhttpps//accaattolloogo otgp//0000..g//224..//.//
guistics. doi: 1/265/////02224lllllll acl aa..ooon..09)
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.09288 2023.
Shubham Vatsal, Harsh Dubey, and Aditi Singh. Multilingual prompt engineering in large language models: A survey across nlp tasks. arXiv preprint arXiv:2505. 1665, 2025.
Weixuan Wang. Barry Haddow, Minghao Wu, Wei Peng, and Alexandra Birch. Sharing matters: Analysing neurons across languages and tasks in llms. arXiv preprint arXiv:2406.09265 2024.
Chris Wendler, Veniamin Veselovsky, Giovanni Monea, and Robert West. Do Ilamas work in en- glish? on the latent language of multilingual transformers. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 15366- 15394, 2024.
Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. m1:: A massively multilingual pre-trained text-to-text transformer. arXiv preprint arXiv:2010.1 1934, 2020.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025.
Yiran Zhao, Wenxuan Zhang, Guizhen Chen, Kenji Kawaguchi, and Lidong Bing. How do large language models handle multilingualism? Advances in Neural Information Processing Systems, 37:15296-15319 2024.
Chengzhi Zhong, Qianying Liu, Fei Cheng, Junfeng Jiang, Zhen Wan, Chenhui Chu, Yugo Mu- rawaki, and Sadao Kurohashi. What language do Non-Engiish-cenrric large language models think in? In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Findings of the Association for Computational Linguistics: ACL 2025, pp. 26333-26346, Vienna, Austria, July 2025. Association for Computational Linguistics. ISBN 979-8-89176-256 httttps//aelaantcoloog...o orgp:/2200oy..02222....2
5. doi: 10.18653/w1/2025, fitiis.f.iniin--aa ac1.
Findings-acl. 1350/
A APPENDIX
A.1 LLMS USAGE
Large Language Models (LLMs) were used solely as general-purpose assistive tools to help pol- ish the manuscript's language and to refine instructions within our multilingual prompt templates. Specifically, LLMs aided in improving grammar, clarity, and style, and in suggesting alternative phrasings for prompt templates. All scientific ideas, experimental design, and key arguments were conceived and written by the authors, and all factual statements were independently verified. Fi- nal prompt templates in English, French, Spanish, Arabic, Hindi, and Japanese were reviewed and validated by native speakers of each language ensure accuracy and cultural appropriateness.
13

---

Published as a conference paper at ICLR 2026
A.2 PROMPTING VARIANTS AND DETAILED ANALYSIS BY DATASET AND LANGUAGE
Despite impressive gains in cross-lingual generalization, multilingual LLMs often struggle with language control, the ability to produce responses in the intended language of the task. To sys- tematically assess this underexplored failure mode, we use our targeted evaluation framework to isolate and stress-test different dimensions of language consistency across diverse multilingual set- tings (Figure 6
Tables 4 present per-language results for each dataset, revealing patterns that are averaged in Table The fine-grained breakdown shows that the trade-off between reasoning ability and language consistency varies sharply by language, script, and prompt type.
For XQuAD (T, Qwen-3-32B shows strong reasoning ability but is highly sensitive to prompt perturbations. Bloom-7.1B IB maintains moderate to high language consistency (>~00% across all languages and prompting styles, though its task performance remains limited (<12%), especially in non-Englin non-English settings. In contrast, Qwen-3-32B 3-32B exhibits strong task performance. but its language consistency varies significantly depending on the language and prompt type. Notably, Spanish shows the lowest language control across all prompting variants for Qwen-3-32B, suggesting a heightened susceptibility to interference; language consistency at 41.68% in the monolingual setting and col- lapses to near 1% (1.60%) in the code-switched variant. The presence of English leads to severe language collapse, particularly for Spanish and Hindi, despite relatively preserved task performance. Overall, while Bloom displays moderate high language stability regardless of prompt structure, Qwen-3-32B's strong multilingual reasoning capabilities come with a trade-off in maintaining lan- guage control, especially when English is introduced.
In math tasks, Table S reveals that Bloom 7.1B consistently underperforns, showing both poor language control and very low task accuracy. particularly under multilingual conditions. In contrast, Qwen-3-32B exhibits a clear trade-off between language consistency and performance: it achieves high task accuracy across all prompting styles and languages, even as language consistency drops drastically, especially under code-switched prompts. For instance, under English code-switched prompts with French questions, language consistency falls to just 7.6%, while task accuracy remains high at 59.6%.
In MMLU, Table 6 reinforces the language-task trade-off seen in Qwen-3-32B: it delivers strong task accuracy across most languages, but language consistency sharply degrades under multilin- gual or mixed-language prompting. The problem is especially acute for Spanish and French, where language consistency drops below 1 % in code-switched and distractor settings, despite task accu- racy remaining above 75%. This pattern suggests that Qwen-3 32B frequently defaults to English when handling closely related languages, prioritizing task accuracy over maintaining language con- sistency. This behavior is further supported by Figure 2 which reveals that across all layers, the model performs reasoning in representations that are closely aligned with English, regardless of the input language. In contrast, Bloom-7. IB shows stronger language control, particularly for distant languages like Hindi and Arabic, but the cost of much lower task performance, particularly in non-English scenarios. These trends indicate that language similarity with English leads to higher interference and loss of control in multilingual models like Qwen-3-32B.
Overall, Qwen-3-32B delivers the strongest reasoning performance but is prone to severe language drift under language prompts. While Bloom BB maintains moderate high language con- trol across languages and prompting formats, its task accuracy remains low, highlighting its limited multilingual reasoning capabilities. Qwen-3-322 often answers accurately even when it fails to pre- serve the intended output language. This suggests that Qwen-3-32B prioritizes internal alignment with English representations. The degradation is especially pronounced for languages typologically closer English (like Spanish and French), which appear more prone to collapse under English interference.
A.3 TUNEITNNIG AND INFERENCE SETTINGS
We perform full scope and selective fine- mun-tuning on specific layers of large pre-trained language models. In the training setup, all model parameters are initially frozen, ensuring only selected layers are updated during fine-tuning. The layers to be fine-tuned are chosen from the output space. We use the Adami optimizer with learning rate of le and the OneCyclet. IIR scheduler to adjust the
14

---

Published as a conference paper at ICLR 2026
(a) Monolingual Direct Prompting
Code-Switched Prompting
You are given mutiipe choice question four answer options:
are given muliile choice question answer options: A.B.C, and D... Respanse guideline::
A.B.C, and D.. Besponse guidelines:
Think step by stap language question Inside
R splitting field of some polynomial over Q 2| There a
<thinking-/hhiike and give your final answer in the language the question inside =aneer>//nnwwr>>..
with 60 elements.
choices:
Question: EPJR 6f 55 & ai 2: 1 X a.m
False,
False,
(R:78 L 4 (202040 Ro +
<thinking- Let's analyee each statemen:: Statement 1: is splitting fleld some pelynomial over 0," Statement "There is fiald with 60 elements." with elements would finite field, also a.. Tharefore, this statement false, Since both stataments false, correct answer is false, Falee </thinking> conswer-@-/anssee
A. 1.887 i 209 5/ 224-
Anower choices:
D. 24/7 14
<thinking- 12.2
2. </thinking> vanswer And Answerr
[c] Bilingual Answer Prompting
(d) English Distractor Prompting
y pueden utilzamme en una situacóón especific para determinar quién constituye un(s)
acuerdo Evan
(1993), Principio
अाcकी oe SeR 3ie अоअस zit bt बात be shr सिय कपा Fi Jnrerr with ji8f sी tamiie vipa alt Ree Raineiiee:
Principlo de
Derechas corporatives,
poraaaiivo, interesada
qta di smar At <hinkin--/thinkng sंदर म,---.--ा.. सा iape spveai a टितs ीwत (हकreर toe?? Hich: Panthers defonse and
Negabilidad corporativs, efecto corporation, parte interesada
Derechos corpor corporativos, resporiabiiddd corporativa, interesada D. Corporate rights, Corporate effect, Stalsholder
308 points, ranking sleth interceptions, two which were retured touchdowns.
utiliaan <thinking> pregunta refiere principlos stupcinn especifica... determinar quíénes son las partes interesadas
<thinkingg 1. gea 41 eacrses 18 riah #l 48 alt quत सा 3EX 2Gt # faw vanifrr aweareff at #e aft uτरieiri Setiiiaa shthininng canower 118 </aniwers
no ajustan a esta teoria o estan en ingles, la cual no es coherente con el iiloma de la pregunta. s/thiniing candwers As/answer>
Figure 6: Overview of Multilingual Prompt Variants. Each variant isolates a specific aspect of multilingual generation: (a) Monolingual Direct Prompting tests baseline language adherence; (b) Code-Switched Prompting mixes instruction and task language to test robustness; (c) Bilingual An- swer Prompting probes language preference by offering correct answers in both the target language and English; and (d) English Distractor Prompting tests resistance to dominant- Lannnant-langugge bias.
learning rate during training, starting from a small value and gradually increasing before decaying. After each batch, the loss is computed, and only the parameters in the selected layers are updated via backpropagation.
As part of the ablation study, we perform grid search over two hyperparameters: the number of epochs (from to 5) and the number of output space layers (from to 5) fine-tuned. Table 10 Table IT shows that the best configuration for Qwen-3-32B is finetuning the last three layers. For all epochs, finetuning only the last three layers always achieves near-perfect language consistency 100% when tested on subset (150) of non-business MMLU topies. The epoch number varies the task performance.
Table 3: Fine-tuning and Inference Parameters
Hyperparameters
Values
Train Languages
ES, FR, AR, HI, JA 500
Train sample per Language
Train- Validation Split
0.8/0.2
Learning Rate
le 5
Batch Size
16
Training Epochs
1
Number of Layers to Fine-Tune
1 to
Temperature
le-ā
Top k
50
Top
0.9
Max New Tokens
512 AdamW
Optimizer
Learning Rate Scheduler GPUs
OneCycleI.
8x NVIDIA H100
15

---

Published as a conference paper at ICLR 2026
Table 4: Language consistency and task performance (F1) on XQuAD across prompting condi- tions for Bloom-7B and Qwen-3-32B. Bloom maintains moderate to high language consistency (58 99%) but fails catastrophically in task performance (FI <5% on average under monolingual prompting), revealing disconnect between staying in-language and solving the task. In contrast, Qwen demonstrates stronger task ability but with uneven and unstable language control: high con- sistency in Arabic and English, but near-total collapse under Spanish and code-switched prompts.
Prompting
Language
Bloom 7.1B
Qwen-3 32B
Language Consistency (%)
F1
Language Consistency (%)
F1 Score (%)
Score
P. Q- (EN)
99.42
11.67
71.47
P. Q (ES) P. Q (AR)
98.24
41.68
56.17
Monolingual Direct
97.23
97.05
64.99
P. Q (HI)
98.40
1.24
85.46
29.51
Average
98.32
4.18
81.05
55.54
P.I-(EN), Q(ES)
86,63
10.82
1.60
65.42
Code
P. (EEN), Q(AR)
58.57
3.42
30.08
46.31
Switched
P. (EE), Q(H))
68.49
5.50
1.34
46.23
Average
71.23
11.01
52.65
(ES), QEES & EN)
70.08
1.37
16.05
27.54
English Distractor
(AR), Q(AR & EN)
69.41
0.36
62.27
9.81
(HD. Q(I & EN)
68.82
0.28
47.65
10.09
Average
69.44
0.67
41.99
15.81
Table Language consistency and task accuracy (%) on MGSM across different prompt variants for Bloom-7B and Qwen-3-32B. Bloom collapses both axes, moderate to low language consistency, and near-zero task accuracy, indicating failure to maintain the target language and to solve the task. Qwen, by contrast, answers well, but its language control is brittle: perfect consistency in English and moderate in Japanese, yet severe collapse for French and under code-switching.
Prompting
Language
Bloom 7.1B
Qwen-3 32B
Language Consistency (%)
Task Accuracy (%)
Language Consistency (%)
Task
Accuracy (%)
I, (ENN
61.20
1.20
100
66.00
Monolingual Direct
I. (FR)
22.80
0.40
31.08
72.80
Q
18.00
0.40
65.60
60.99
Average
34.00
0.67
65.56
66.60
Code Switched
(EN), Q(FR)
22.80
0.40
7.60
59.60
(EN), QUA)
14.01
0.40
6.08
54.40
Average
18.41
6.84
57.00
16

---

Published as a conference paper at ICLR 2026
Table Language consistency and task accuracy (%) on MMLU across different prompt variants for Bloom-7B and Qwen-3-32B. Bloom exhibits high language consistency in many settings but fails catastrophically at task accuracy, sometimes near zero, even when language consistency high. Qwen flips the pattern: strong task performance in English, Spanish, and French (270%), but frag- ile language control, collapsing almost completely in French, Spanish, and code-switched inputs. Under distractors and bilingual prompts, Bloom "sticks to the language but cannot answer," while Qwen "answers well but drifts and out of the target language." The results expose a fundamental tension between being in un-anguage and being correct in current multilingual LLMs.
Prompting
Language
Bloom 7.1B
Qwen-3
Language Consistency (%)
Task Accuracy
Language Consistency
Task Accuracy
P.I.Q (EN)
99.51
21.24
100
77.08
Q- (ES)
50.22
16.59
0.31
76.33
Monolingual
Q-
85.89
11.34
48.39
6.38
Direct
P. (HII
96.79
10.76
69.09
34.23
P. Q (FR)
17.16
19.23
0.91
72.92
P. (JA)
58.33
52.32
43.68
P. R. ((EN) QEES)
33.34
27.56
0.29
75.98
((EN),, Q(AR)
32.05
10.26
1.28
43.59
Code Switched
P. -(EN), Q(II))
29.49
14.74
14.10
49.36
P. -(EN), Q(FR)
35,26
30.13
1.09
72.66
P. (EEN), QUA)
17.31
28.85
25.00
60,90
(ES). Q(ES&EN)
67.31
15.38
5.33
77.40
English
(AR). Q(AREEN)
37.82
6.41
18.58
4.05
(HI). Q(HI&EN)
41.03
3.85
47.38
14.54
Distractor
(FR). Q(FR&EN)
25.64
26.92
0.24
75.84
(JAA), QUA&EN)
28.21
0.00
13.69
15.23
(ES), QCE&&EN)
44.87
19.87
15.96
55.69
Bilingual
(FR), QUR&EN)
14.74
21.15
47.27
$2.89
(AR), Q(AREEN)
85.26
0.00
12.77
6.31
Answer
(II), QCH&EN)
96.15
1.92
31.00
23.04
(JAA), QUA&EN)
57.05
3.85
10.50
40.85
Table 7: Performance comparison of Qwen-3-32B and Bloom-7. on the MMLU dataset. Models were trained using code-switched prompts the Business domain across six languages and evalu- ated on subset of non- onnBBuineess domains.
Language
Model
Pre-Finetuning
Full scope SFT
Selective SFT
Language Cons. (%)
Acc. (%)
Language Cons. (%)
Acc. (%)
Language Cons. (%)
Acc. (%)
Qwen-3-32B
1.28
76.92
100
87.18
99.04
88.46
P. I.EN), Q(ES)
Bloom-7.11
33.34
27.56
99.36
35,90
98.08
26.92
Qwen-3-32B
1.92
71.79
100
90.38
100
85.58
P. I-ENN, Q(FR)
Bloom-7.1B
35.26
30.13
100
35.90
98.08
23.08
Qwen-3-32B
14.10
49.36
99.36
46.15
100
47.12
P. Q(HI)
Bloom-17
29.49
14.74
100
33.97
98.08
20.19
Qwen-3-32B
1.28
43.59
100
83.33
99.04
67.31
I.ENN,,), Q(AR)
Bloom-7.13
32.05
10.26
100
32.05
100
17.31
Qwen-3-32B
25.00
60.90
100
87.18
100
81.73
P, P,I-(EN), QUA)
Bloom-7.1
17.31
28.85
100
30.77
99.04
18.27
Code-Switched
Wwen33322B
8.32
60.51
99.87
78.84
99.62
74.44
Average
Bloom-7.15
29.49
22.31
99.87
33.72
98.66
21.14
17

---

Published as a conference paper at ICLR 2026
Table Comparison of Qwen-3-32B 3-32B and Bloom-7.1B across monolingual and code-switched set- tings in English, French, and Japanese for pre-finetuning, full-scope SFT, random selective SFT, and targeted selective SFT. Qwen-3-32B shows strong gains from Selective SFT, especially in cross-lingual settings, while Bloom-7.1B remains fragile despite perfect language consistency post- finetuning. highlighting limitations in multilingual task generalization. Selective SFT achieves near-parity with full-scope SFT in both consistency and accuracy, despite modifying fewer parame- ters.
Language
Model
Pre-Finetuning
Full scope SFT
Random Selective SFT
Selective SFT
Language Cons. (%)
Language Cons. (%)
Acc. (%)
Language Cons. (%)
Language Cons. (%
(%)
(%)
(%)
Qwen-3-32B
100
66.00
98.80
97.20
1.20
0.40
99.60
95.60
P. Q- (EN)
Bloom-7.1B
61.20
1.20
100
1,60
98.0
0.0
100
3,2
Qwen-3-32B
31.08
72.80
99.60
89.20
98.40
0.0
98.0
84.80
P. (PR)
Bloom- 7.1B
22.80
0.4
100
2.00
29.20
0.0
100
Q- (JA)
Qwen 3: 32B
Bloom-7.18
18.00
65.60
60.99
0.40
100
85.20
0.80
98.0
0.0
100
80.0
Monolingual
Owen-3-32B
65.56
66.60
99.47
90.53
65.87
0.13
99.20
86.80
Average
Bloom-7.1B
34.00
0.67
1.47
Qwen-3.32B
7,60
59.60
99 20.60
87.20
7.60
0.0
97.2
(EN), Q(F)
Bloom 7.1B
22.8
0.4
100
3.20
37.6
0.0
100
O.en-3.32B
6.08
$6.80
P. (EN), QUA)
Bloom-7.18
14.01
0.4
100
1.20
98.40
0.0
99.20
1.6
Code-Switched
Qwen-3-32B
6.80
57.00
95.00
10.00
53.80
0.00
98.60
84.00
Average
Bloom-7.1B
18.40
0.40
100
2.20
68.00
0.00
99.60
2,00
Table 9: Performance comparison of Qwen-3-32B and Bloom-7.1B on the XQuAD dataset in four languages (EN, ES, AR, HI). Qwen-3-32B exhibits high generalization and minimal performance degradation across SFT modes. contrast, Bloom-7.1B struggles especially under zero-shot and random SFT, with F1 scores near zero in low-resource and distractor-heavy settings.
Language
Model
Pre-Finetuning
Full scope SFT
Random Selective SFT
Selective SFT
Langunge Cons. (%)
F1 Score (%)
Language Cous. (%)
FI Score
Language Cons.
Fl Score (%)
Langunge Cons. (%)
F1 Score
(%)
P.L.Q. (EN)
Qwen-3-32B
Bloom-
99.42
11.67
100
25.08
61.76
1,21
0.0
100
24.06
Owen 3- 22B
41.68
56.17
100
69.27
22.61
0.45
100
62.09
P. Q (ES)
Bloem: 7.1
98.24
3.50
99.66
18.59
0.67
0.0
99.66
24.9
Qwen:3-328
71.09
P.I. Q C- (AR)
Bloom-7.1
97.23
0.31
11.92
72.86
0.0
100
16.58
P. Q C- (HI)
Qwen-3-32B
Bloom-7.11 B.
98. 98.40
85.46
29.51
1.24
100
100
18.
9.81
83.11
95.80
0.0
0.0
99.75
100
15.77
17.09
Monolingual Average
Qwen-3-32B
Bloom-7.11
98
4.18
99.91
54.10
99.85
P.I. (EN), Q, (ES)
Qwen-3-328
Bloom-7.1 1B
8663
1.60
65.42
10.82
100
100
68.7 23
25.02
97.06
15.13
2.53
0.0
99.83
100
23.53
19
(EN). (AR)
Qwen-3.328
Bloom-77.13
30.88
68. 67.77
96.72
17.31
0.76
0.0
99.91
100
63.55
P.I. (EN). C (LEI
Qwen-3-328
Bloom- 1B
68.4 08.49
1.34
46.23
5,50
99.66
100
18.46
18:00
96.89
100
0.0
0.0
99.66
100
20.53
30.85
Code-Switched
Owen-3.32B
11.01
52.65
100.00
51.87
97.93
1.10
100.00
53.53
Averuge
21.03
P.L.Q (ES), C- (EN)
Owen:3-320
16.05
27.54
98.66
31.98
1.08
0.0
98.99
31.86
Bloom- 7B
70.08
1,37
97.56
12.58
0.67
0.0
98.99
9.21
R.I.O (AR) C (EN)
Qwen-3-33B
Bloom-
62.27
9.81
30.81
13.25
20.67
0.0
97.73
L4.88
P.I.O Q.Q--M am. C- (EN)
Qwen-3-322
47.65
10.09
98.49
8.07
91.59
0.0
96. 06.13
7.41
Bloom-7.1 18
68.30
0.28
98.06
4.47
55.13
0.0
97.39
4.87
Einglish Distractor
Owen-3-32B
41.99
15.81
75.99
17.77
37.78
0.00
97. 62
18.05
Bloom-7.18
26.16
98.25
18

---

Published as a conference paper at ICLR 2026
Table 10: Bloom-7.1B Layer-wise Ablation across num_epochs and num_layers to analyze the effect on language consistency (LC), task accuracy (TA). Models are fine-tuned on the MMLU busi- ness domain and evaluated on non-business domains under code-switched conditions. Varying the number of last layers (1-5) and training epochs (1-5) shows that fine-tuning just 3 3-5% of parameters yields high LC (> 95%) with stable TA. The best combined scores emerge with 1-3 layers and 4-5 epochs, demonstrating that robust cross-domain, code-switched language control can be achieved with minimal parameter updates.
Epochs
of Last Layers
Lang Consistency (%)
Task Accuracy (%)
Avg-L.C
Avg-TA
Combined
ES
FR
EN
HI
AR
JA
ES
FR
EN
HI
AR
JA
Baseline
48.08
26.92
99.04
34.62
41.35
32.69
25.96
27,88
22.12
18.27
7.69
27.88
47.12
21.63
34.37
99.01
99.01
85.58
100
100
99.04
23.08
23.08
13.46
12.50
3.85
12.50
97.12
14.74
55.93
100
74.04
100
100
100
25.00
22.12
10.58
12.50
0.96
17.31
95.51
14.24
55.13
40.38
99.04
100
21.15
11.54
12.50
9.62
1
98.08
99.04
73.08
99.01
100
95.08
18:27
13,46
9.62
12.50
3.85
16.35
91.55
12.34
53.45
1
5
100
99.04
82.69
100
100
98.08
17.31
22.12
15.38
16.35
6.73
12.50
96.63
15.06
55.85
2
95304
99.04
79.81
99.04
98.08
99.04
16.35
28.85
17.31
13.46
8.65
12.50
95.67
16.19
55.93
2
100
100
67.31
99.04
100
100
34.04
27 87
13.46
8.65
7.69
16.35
94 30
16.35
55.37
69.25
99.04
12:50
16.35
14.42
93.59
34.62
99.04
97.12
23.08
6.75
24.04
19.23
19:07
2
99.04
99.04
60.58
100
100
97.12
26.92
20.19
15.38
18.27
12.50
14.42
92.63
17.95
53.29
3
99.04
100
84.62
98.08
100
97.12
35.96
23.
12.50
12.50
7.69
12.50
96.47
15.71
56.09
3
2
98.08
97.
69.25
99.04
100
97.12
28.5
25. 50
14,42
18.27
14.42
14.42
93.43
19:23
56,33
3
99.01
97.12
58.65
100
100
96.08
26.92
21.04
12.50
19.23
13.46
15:38
92.15
18.59
55.37
51.92
99.04
25.
15.38
62.50
100
99.04
98.08
20.19
10.58
20.19
18.27
56.41
4
98.08
97.12
86.54
99.04
100
98.08
25.00
20.19
11.54
15.38
3.65
13.46
96.47
15.21
56.09
4
90
1001
64.42
100
100
98.08
30.77
23.08
18.27
25.00
16.35
15.38
93.59
21 21.47
57-53
4
97.12
95.08
58.65
98.08
99.04
99.04
25.85
50.77
9.62
17.31
17.31
11.54
91.67
19:23
55.45
100
99.04
50.00
100
99.04
99.04
24.
26.92
15.38
17.31
16.35
14.42
47.12
11.54
21.15
12.50
08
87. 37.50
98. 08
100
99.04
26.92
23.08
15.38
20.10
17:31
18:27
96.79
20.19
58.49
98.08
99.04
75.96
98.08
99.01
98.08
32.69
20.19
15.38
18.27
13.46
13:46
94.71
18.91
56.81
100
97.12
73.08
99.01
99.01
100
29.81
24.04
13:46
15.38
20.19
21.15
94.71
20.67
$7.69
99.04
99.04
71.15
100
97.12
100
23.08
29
13.46
16.35
22 12
18:27
94.55 30
20.51
57.45
53.85
99.04
99.04
99.04
21.15
25,96
10.58
21.15
21.15
19.23
91.67
Table 11: Layer-wise Selective SFT analysis of Qwen-3-32B on language consistency (LC) and task accuracy (TA) across six languages. The model is fine-tuned on the MMLU business do- main and evaluated on non-business domains under code-switched conditions. Unlike the baseline, which shows strong TA (65.38%) but poor LC (24.52%), fine-tuning just 3-5% of the parameters quickly boosts LC to near-perfect levels (>99%) while preserving high TA. The best combined scores emerge with 2-3 layers and 4-5 epochs, indicating that minimal parameter updates are suffi- cient to reconcile Qwen's 's trade-off between task performance and language consistency.
Epochs
Last Layers
Lang Concistency (%)
Accuracy (%)
Avg.LC
Avg-TA Combined
FR
EN
HIl
AR
JA
ES
FR
EN
HI
AR
JA
Baseline
1.92
3.85
100
11.54
0.96
25.85
30.77
74.04
77.88
49.04
49.04
61.54
24.52
65.38
44.95
90.04
100
99.04
100
100
100
77.88
78.85
80.77
28
66.35
73.08
99.68
66.67
83,17
100
97.12
100
100
100
73.85
75.96
89.42
27.88
65.27
75.96
99.52
69.39
$4.16
100
80.77
20.
1
99.04
100
100
100
100
99.04
78.55 78.85
75.00
85.58
08
61.54
99.65 68
64.90
82.29
1
5
100
100
100
99.04
100
100
77.88
76.92
86.54
17.31
64.42
75.08
99.84
66.03
82.93
2
1
99.04
100
100
100
100
100
50.73
76.92
77.88
24.04
70.19
71.15
99.84
66.83
83.33
2
99.04
100
100
100
100
100
82.69
80.77
77.88
20.19
66.27
71.15
99.84
66.83
83.33
2
100
100
100
82.69
83.65
21.15
66.35
74.04
99.04
100
75.96
77.88
85.58
22. 12
70.19
71.15
99.54
67.15
2
99.04
100
85.58
99.04
100
100
72.12
75.00
47.12
13:46
67.31
$5.77
97.28
55.13
76.20
3
100
100
99.04
100
100
100
80.77
79.81
86. 86.54
20. 19
62.50
74.04
99 84
67.31
83.57
3
2
100
98.08
100
99:01
100
100
78.85
78.85
81.73
19.23
71.15
75.96
99 52.52
67.63
83.57
3
100
100
100
100
99.04
100
76.99 92
73.08
89.42
25.0 00
68.27
77.88
99 99.84
68.43
84.13
100
100
100
84.62
MX
99.04
100
100
100
80.77
84.62
17.31
72.12
75.00
99.52
67.47
83.49
4
99.01
100
99.04
100
100
100
$1.73
80.77
82.69
25.00
70.19
75.95
99.68
69.39
$4.54
4
99.04
100
100
100
100
100
82.69
81.73
84.62
25.96
67.31
72.12
99 84
69.07
84.66 46
4
100
99.04
100
99:04
100
100
$1.73
$1.73
84.62
25.00
73.08
75.00
99.68
70.19
84 94
4
91.35
$1.73
20.19
95.19
97.12
96,23
4.51
5.77
3.65
0.00
0.00
14.42
79.97
4.81
42.39
4
100
100
100
82.69
83.65
86.54
35.58
73.88
86-94
100
88 98.46
37.50
72.12
75.96
08
73.72
5
2
99.04
100
96.00
98,08
100
99.01
100
38.46
85.58
90.38
17.12
67.31
81.73
99.36
76.76
$3.06
5
100
98.08
29:81
98.08
100
98.08
58.65
50.00
31.73
17.31
31.73
$4.81
87.55 34
40.71
64.00
5
45.19
48.08
14.42
20.19
74.04
70.19
1.92
0.00
1.92
2,88
0.00
5.77
45.35
2.08
23,72
97.12
100
8.65
96.15
100
96.08
72.12
75.00
59,81
23.05
58.65
67.31
83.33
58.19
70.91
19

---

Published as a conference paper at ICLR 2026
Prompt: English Monolingual Direct Prompting
You are given a multiple-choice question with four answer options: B, C, and D. Please choose the best answer based on your knowledge and reasoning ability.
Response guidelines:
Your task is carefully read the question and all answer choices, then determine which option best answers the question based on your knowledge and reasoning.
Please consider the meaning of each choice and eliminate incorrect or less appro- priate options using logical deduction factual recall. If multiple answers seem plausible, select the one that is most accurate or comprehensive.
Pay close attention to subtle distinctions wording or concepts, as some questions may require domain-specine understanding nuanced interpretation.
After evaluating all options, select the single best answer and respond with only the corresponding letter: A, B, C. or D.
Think step-by-step in the language of the question inside <thinking><< /thinking> and give your final answer in the language of the question inside <answer>> /answer>.
Question: {question] Answer choices: (choices)
<thinking>
Prompt: French Monolingual Direct Prompting
Vous allez recevoir question choix multiples quatre options de réponse A, B, C et D. Veuillez choisir la meilleure réponse en vous basant sur vos connaissances et votre capacité de raisonnement.
Directives de réponse:
Votre tåche consiste à lire attentivement la question et toutes les options, puis à déterminer laquelle répond le mieux fonction de connaissances et de votre raisonnement.
Prenez en compte le sens de chaque option et éliminez celles qui sont incorrectes moins appropriées utilisant la déduction logique des faits connus. Si plusieurs réponses semblent plausibles, choisissez celle qui est la plus précise ou la plus complète.
Faites attention aux distinctions subtiles dans le libellé ou les concepts, certaines questions peuvent nécessiter une compréhension spécialisée ou une interprétation nuancée.
Après avoir évalué toutes les options, sélectionnez une seule réponse et répondez uniquement avec lettre correspondante A, B, C ou
Réfléchis étape par étape dans la langue de la question à l'intérieur de <thinking>< /thinking>> et donne ta réponse finale dans la langue de la question à I' l'intérieur de <answer><<answer>>>>>>>>>>>>>>>>
Question {question]
Choix de réponses: {choices)
<thinking>
20

---

Published as a conference paper at ICLR 2026
Prompt: Spanish Monolingual Direct Prompting
Se te presenta una pregunta de opción máltiple con cuatro posibles respuestas: A, B, y D. Por favor, elige mejor respuesta basándote en tus conocimientos y capacidad de razonamiento.
Instrucciones para la respuesta:
Tu tarea leer cuidadosamente la pregunta todas las opciones, y determinar cuál responde mejor basándote en tus conocimientos y razonamiento.
Considera el significado de cada opción y climina aquellas incorrectas o menos apropiadas utilizando la deducción lógica el conocimiento factual. Si varias op- ciones parecen plausibles, selecciona más precisa o completa.
Presta especial atención las diferencias sutiles en lenguaje o los conceptos, ya que algunas preguntas pueden requerir comprensión específica del dominio interpretación matizada.
Después de evaluar todas las opciones, selecciona una sola respuesta y responde únicamente con la letra correspondiente: A, B, C D.
Piensa paso paso en el idioma de la pregunta dentro de <thinking>>< /thinking> tu respuesta final idioma de la pregunta dentro de <answer>< /answer>.
Pregunta: (question)
Opeiones de respuesta: {choices}
<thinking>
Prompt: Japanese Monolingual Direct Prompting
h D
2  X1
# 
?. 3
チ x < H # (A. B, C. ) O
<thinking><
BO <answer><
/answer> 1 C (
: quesstion]
O i {choices)
thinking
21

---

Published as a conference paper at ICLR 2026
Prompt: Arabic Monolingual Direct Prompting
. 4 . بلاجااي البجازاتت برتتت شعاد بن الماار عغال عاية تمزف
الدنطليي الشككر غنا زشرر تبرفنة غأا بنانن الأنسب الاجاي اكتتار جاا
الأخاا: إرساتتات
الككيار خحبيد شهحة بنااي الاجاي كياراات وريع المغال ووانت بي فهقاه النبلقي الشفكر غأا ففررا ققك غلا بافنن الانشب
التاابي كبر او الشجيخ كهد الكبازات او القدغرر
واستيااد كنارر 3 شعا فراغات فرجاا
الدنجلقي الاستناح باستكمةام -فووس او يق الاكير الاخاب تحبة الغغاعي او الجياةة في الأقيق للامماافات ججتنن إنشب تفد تطلب تحد الاييز
تغفق المعكة الاجالات بات إددا
كقفف تقيين او فخين ججاال كهاقس قهقن قشطء الفقابل بلشرف زاجب الأنشب الاجاي إككز الكبارات نهع شبيع بمة بنكة
أو ج بي of
غاكبل الخنال
بكشطو كنطو قعز
<thinking>>
/thinking>,
الحشال كاكهل بنعع
الجباي الااكةة وقتم
<answer>< answer>>
ألصغاب
{question}
كبازات الاجابة
{choices} <thinking>
22

---

Published as a conference paper at ICLR 2026
Prompt: Hindi Monolingual Direct Prompting
अहरपमाी प
क पलअसहनअ सहरअग गभयक
हिं शममअरा कभअर अननेर महकअगरप अेर FT I: भ क: भहसे
4 | क अपभनप उ नमकनन अहरअ रशरककअ कप अहका
पआर मभनमप अभअनडेी अततेर कहननप HT
अतनअर कहक
सशसमप पमश पिमननन भहस मयानरर सनतेर पहकयगर रहपडशे सह शआ सहप IT पहरर सापहप A कर अभाम म.. नमशनग भउल नमरकक कडम कहकडगलप को मडबमप मअसड कयनमयाय मडं IT
केर पहकअकर कउअअ मडबलडद मयमेडाय F शहरउ लभरक
पभशश अिकनगडड कभशश शमरपमडशील कडाल शभरकल
पभशश कभम अभभपकडी पडकउलर उअअ कमश बपडन आाान कभशड
सहशेग मभहर लभााप F:: तो उनअनप A सप सयकमप मये महकक पभअश सभकमप शउउेशा अतरेर नहअनप IT
महडादाी पमक शहरहगरशाान कप पहही मपलनकरी सयe पहअ पआरे कहयशशे पाकन:: कपोनककर कडककह सभकेयाम मपा A
शहअश ममानगरशेााक पभशश नयसे ममशेगनपाशाा
सशशहरयप दा मडकअगाड रमह |
मअबशरर बहकेकापन पमर शररहडेर कशशे कार कमअर मभबमप मअहद रडकशलय पहडनप A शगरश कपकभल मयमयउमर- हशा अक मडर लरकनपा F T: H: हे. पभशअ L
<thinking>< /hinking F उसक महे F म अरभनशक नभरक कमकहरर F शगक
मग A उमड शहशागा मप शपकनयश मम शन रपकन
<answer>< answer
पहस {question) उनतअर पहकमगरश {choices)
<thinking>
23

---

Published as a conference paper at ICLR 2026
Prompt: Baggish-Frennce Code-Switched Prompting
You given multiple-choiee question with four answer options: A, B, C, and D. Please choose the best answer based on your knowledge and reasoning ability.
Response guidelines:
Your task is to carefully read the question and all answer choices, then determine which option best answers the question based on your knowledge and reasoning.
Please consider the meaning of each choice and eliminate incorrect or less appro- priate options using logical deduction or factual recall. If multiple answers seem plausible, select the one that is most accurate or comprehensive.
Pay close attention to subtle distinctions in wording concepts, some questions may require domain-specific understanding or nuanced interpretation.
After evaluating all options, select the single best answer and respond with only the corresponding letter: A, B, C, or D.
Think step-by-step in the language of the question inside <thinking /thinking> and give your final answer the language of the question inside <answer>> /answer>>
Question: Lequel des éléments suivants est la voie symplastique qui permet le déplacement du saccharose du site de photosynthèse des cellules du mésophylle vers le phloème Answer choices:
A. Les fibres, le parenchyme du phloème, la cellule compagne, le tube criblé
B. Le parenchyme du phloème, les fibres, la gaine périvasculaire, les trachéides
C. Les cellules compagnes, le parenchyme du phloème, les fibres, tube criblé
D. La gaine périvasculaire, le parenchyme du phloème, la cellule compagne, le tube criblé
<thinking>
24

---

Published as a conference paper at ICLR 2026
Prompt: English-Spanish Code-Switched Prompting
You are given a multiple-choice question with four answer options: C, and D. Please choose the best answer based on your knowledge and reasoning ability. Response guidelines:
Your task is carefully read the question and all answer choices, then determine which option best answers the question based on your knowledge and reasoning.
Please consider the meaning of each choice and eliminate incorrect or less appro- priate options using logical deduction factual recall. If multiple answers seem plausible, select the one that is most accurate or comprehensive.
Pay close attention to subtle distinctions wording or concepts, as some questions may require domain-specific understanding nuanced interpretation.
After evaluating all options, select the single best answer and respond with only the corresponding letter: A, B, or D.
Think step-by-step in the language of the question inside <thinking><< /thinking>> and give your final answer in the language of the question inside <answer>< /answer>.
Question: &Cuál de las siguientes la vía simplástica para el movimiento de la sacarosa desde el lugar de la fotosíntesis en las células del mesófilo hasta el floema? Answer choices:
A. Fibras, parénquima del floema, célula acompañante, tubo criboso.
B. Parénquima del floema, fibras, haz vascular, traqueidas.
C. células acompañantes, parénquima del floema, fibras, tubo criboso
D. Haz vascular, parénquima del floema, célula acompañante, tubo criboso.
<thinking>
Prompt: French Prompting with English-Distractor
Lequel des éléments suivants est la voie symplastique qui permet le déplacement du saccha- rose du site de photosynthèse des cellules du mésophylle vers le phloème ? Choix de réponses:
A. Fibers, phloem parenchyma, companion cell, sieve tube
B. Phloem parenchyma, fibers, bundle sheath, tracheids
C. La gaine périvasculaire, le parenchyme du phloème, la cellule compagne, le tube criblé
D. Companion cells, phloem parenchyma, fibers, sieve tube
Répondez français. Réfléchisszz étape par étape dans <thinking><< /thinking>> et don- nez votre réponse finale dans <answer><< /atf>></anssww /answer>> <thinking>
25

---

Published as a conference paper at ICLR 2026
Prompt: Spanish Prompting with English-Distractor Distractor
&Cuál de las siguientes la vía simplástica para el movimiento de sacarosa desde el lugar de la fotosíntesis en las células del mesófilo hasta el floema? Opciones de respuesta:
A. Haz vascular, parénquima del floema, célula acompañante, tubo criboso.
B. Fibers, phloem parenchyma, companion cell, sieve tube
C. Phloem parenchyma, fibers, bundle sheath, tracheids
D. Companion cells, phloem parenchyma, fibers, sieve tube
Responde en español. Piensa paso a paso dentro de <thinking>>< /thinking>> da tu re- spuesta final <answer>< aanswr>>. <thinking>>
Prompt: French Prompting with English Bilingual Answer
Lequel des éléments suivants est la voie symplastique qui permet le déplacement du saccha- rose du site de photosynthèse des cellules du mésophylle vers phloème ? Choix de réponses
A. Bundle sheath, phloem parenchyma, companion cell, sieve tube
B. Le parenchyme du phloème, les fibres, la gaine périvasculaire, les trachéides
C. Les cellules compagnes, le parenchyme du phloème, les fibres, tube criblé
D. La gaine périvasculaire, le parenchyme du phloème, la cellule compagne, le tube criblé
Répondez en français. Réfléchissez étape par étape dans <thinking><< /thinking>> et don- votre réponse finale dans <answer><> < /answer>. <thinking>>
Prompt: Spanish Prompting with English Bilingual Answer
(Cuál de las siguientes es la vía simplástica el movimiento de la sacarosa desde el lugar fotosíntesis en las células del mesófilo hasta el floema? Opciones de respuesta:
A. Fibras, parénquima del floema, célula acompañante, tubo criboso.
B. Parénquima del floema, fibras, haz vascular, traqueidas.
C. Bundle sheath, phloem parenchyma, companion cell, sieve tube.
D. Haz vascular, parénquima del floema, célula acompañante, tubo criboso.
Responde español. Piensa paso paso dentro de <thinking>>< /thinking da tu spuesta final en <answer><< < ansswr>>. <thinking>>
26


Reviews:
[{"review_id": "w9eVXkNhcW", "reviewer": "Reviewer_X6JL", "summary": "This paper addresses a central gap in multilingual large language models (mLLMs): language control, the ability to generate responses in the intended target language. It identifies two primary failure modes: the multilingual transfer bottleneck (correct language, incorrect task) and the language consistency bottleneck (correct task, wrong language). To systematically study these, the authors propose a diagnostic framework with four controlled prompting scenarios spanning tasks from MMLU, MGSM, and XQuAD. They then use interpretability techniques, layer-wise logit lens decoding and hidden-state similarity analysis, to trace where language control emerges across the model’s depth. The results reveal a three-phase internal organization: early layers align inputs semantically across languages, middle layers handle reasoning, and late layers drive language-specific generation.\n\nBuilding on this insight, the authors introduce Selective Fine-Tuning (SFT), which updates only the last few layers responsible for language control while freezing the rest. Applied to Qwen-3-32B and BLOOM-7.1B, this method improves language consistency from below 20% to over 98% across six languages while fine-tuning just 3–5% of model parameters, with minimal loss in task accuracy. The work presents both a structural understanding of multilingual layer specialization and a practical, parameter-efficient tuning method for controlling language generation.", "questions": "NA", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "- Novel diagnostic framework for multilingual failure modes: The four-scenario prompting setup provides a well-structured and reproducible way to disentangle language control from task accuracy.\n\n- Insightful interpretability analysis: The paper convincingly demonstrates a three-phase structure across layers, linking representational alignment to functional behavior in multilingual settings.\n\n- Strong empirical improvements with minimal compute cost: Selective fine-tuning significantly enhances language consistency while preserving task performance, requiring only 3–5% of parameters to be trained.\n\n- Clarity and completeness: The paper clearly presents its experimental design, prompt templates, and evaluation results, including extensive per-language tables and ablations.\n\n- Practical impact: The proposed approach offers a scalable path to adapt existing mLLMs for multilingual deployment without full retraining or specialized data.", "weaknesses": "- Limited novelty in fine-tuning method: While the interpretability analysis is insightful, the proposed selective tuning strategy builds on well-established parameter-efficient fine-tuning concepts and is not fundamentally new.\n\n- Narrow evaluation scope: The study focuses on only two models (Qwen-3-32B and BLOOM-7.1B) and a limited set of languages. Broader coverage across typologically diverse languages or other architectures would strengthen generalization claims.\n\n- No comparison with alternative lightweight methods: The paper does not benchmark against LoRA, adapters, or middle-layer alignment approaches, which would contextualize the gains from selective SFT.\n\n- Interpretability analysis could be deeper: The layer-wise similarity and logit lens analyses, while descriptive, remain qualitative. A more quantitative measure of where “language control neurons” reside would enhance rigor.\n\n- Limited real-world evaluation: The framework is confined to academic benchmarks, lacking demonstrations on open-ended generation, code-mixing robustness, or human evaluations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Df9Xi071wH", "reviewer": "Reviewer_XGPX", "summary": "The paper investigates how multilingual large language models manage their ability to generate in the intended language. The authors identify two failure modes - multilingual transfer bottleneck and language consistency bottleneck. The authors extend the logit lens technique to measure language probability trajectories across layers and apply hidden-state cosine similarity to quantify cross-lingual alignment. The authors also propose selective supervised finetuning (Selective SFT), tuning only the last few layers to restore language control efficiently. The method is empirically validated by achieving a high language consistency across six languages on Qwen-3-32B and Bloom-7.1B while fine-tuning only 3–5% of parameters.", "questions": "1.\tIn Section 4.2.1, it is mentioned that Qwen-3-32B’s target-language probabilities rise only after layer 55. How did the authors determine that this boundary (layer 55) marks the transition to language-specific control, and is this threshold consistent across tasks or languages?\n2.\tThe hidden-state cosine similarity (Eqs 6–7) uses mean-pooled token embeddings. Were layer-wise token-level divergences (eg - in attention focus or contextual span) observed, that might provide finer evidence for the semantic–reasoning–language transition?\n3.\tIn Fig 2, Bloom’s target-language probabilities exhibit high variance across layers. What could this be due to?\n4.\tIn computing language probabilities via Eqs. 2–4, how was multilingual token overlap handled, especially in cases where shared alphabets might bias the language identification model?\n5.\tSelective SFT fine-tunes the last one or two layers depending on the model. What empirical/diagnostic signals indicated that these layers were most responsible for language control? \n6.\tIn Table 2, task accuracy sometimes improves after selective fine-tuning. Why is it that adjusting the final layers for language control also improves reasoning accuracy?\n7.\tUnder code-switched prompting, Qwen fails to “re-ground” target language probabilities. Did the logit-lens traces show any mid-layer oscillation patterns indicating instability in language identity propagation?\n8.\tGiven that the similarity analyses reveal language-invariant middle layers, did the authors check whether selective fine-tuning altered these alignments? That is, did language control adjustments propagate backward into semantically aligned layers?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper makes a strong interpretability-driven contribution, linking layer-wise representational dynamics to achieve multilingual control. The integration of logit lens analysis with hidden-state similarity profiling provides a compelling explanation for language drift. The selective fine-tuning strategy seems intuitive and computationally efficient, and demonstrates that language-specific control can be restored without retraining the full model.", "weaknesses": "While the results are compelling, several aspects of the methodology needs further clarification. The precise criterion for identifying layer boundaries (e.g., layer 55 for Qwen-3-32B) is not fully justified. This raises uncertainty about whether these thresholds are architecture-specific or emergent from model dynamics. The mean-pooled cosine similarity metric may obscure finer token-level divergences, leaving open how exactly semantic alignment transitions into language control. Similarly, Bloom’s variance in cross-language probability trajectories (Figure 2) suggests that underlying architectural or tokenizer-level factors might influence the emergence of language control more than the analysis captures. The selective fine-tuning procedure itself (particularly how the tuned layers were chosen and validated) is somewhat heuristic. Finally, while the post-finetuning improvements in both language consistency and reasoning accuracy are clear, the mechanism behind this dual gain is underexplored.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2hTldfKeCE", "reviewer": "Reviewer_ha32", "summary": "This paper describes an extension to the logit lens interpretability approach to understand language representations in multimodal LLMs and identify layers that would benefit from fine-tuning to improve language control without degrading the underlying performance. The analysis focusses on two models from two families (Qwen and Bloom).", "questions": "- The abstract states that 98% language consistency whilst fine-tuning only 3-5% of the parameters — it would be beneficial to know how this compared to full fine-tuning. It is stated later in the paper (Line 087) so move this up into the abstract.\n- Why not consider models of equivalent size from different families, or different sizes within the same family?\n- Line 111: check the opening quotation marks.\n- Line 180: the equation is not needed.\n- In Table 1, the poor task performance for Bloom suggests this may just be a poor model, in which case how reliable are findings drawn from this. It seems an especially poor choice of model for the MGSM task.\n- Line 249: What should be subscript “i” is not subscript.\n- Line 255: The variable N is being reused as the number of tokens, where it was previous used for the number of samples in the evaluation set. Variables should not have multiple meanings to avoid ambiguity.\n- Line 259: talks about comparing n-gram profiles but it is not clear how. It would help to forward reference where this is discussed. Likewise it is not clear where the pre-trained language profiles are from.\n- Line 276: N appears again — this this a third definition, or is this original use (size of the evaluation set).\n- Is the language similarity score in Equation 6 not also sensitive to the specific content to? Say you took large batches of sentences for the same language — how would these similarity scores relate to the scores for different languages?\n- Line 425: Since optimality was reached after five of five epochs, why not run more to see of performance continues to improve?\n- Line 431: “Table 2 indicate that” > “Table 2 indicates that”\n- Line 458: Is “full-scope SFT” the same thing as “full fine-tuning”? Use a consistent name throughout the paper.\n- Lines 480: check the opening quotes.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "- Improving performance and control of multilingual LLMs is an important topic, especially for ensuring that all users of models, regardless of language, have an equivalent experience.\n- The approach offers a method that avoids compute-expensive fine-tuning. The results suggest that only 5% or fewer parameters need to be tuned.", "weaknesses": "- The paper considers only two models from different families, and these are not of comparable size. Specifically the Qwen model is 32B parameters whilst the Bloom model is 7B. \n- There are statements making assumptions about architectures (e.g., architectures like Qwen favor task success) but it is difficult to know if these statements are generally true with the question about the model size differences. Also, if numbers are reported from a single run then again we do not know if the observations are an artifact of that one run or if they hold true more generally.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)