Title: LinguaMap: Which Layers of LLMs Speak Your Language and How to Tune Them?
Primary area: interpretability and explainable AI
Keywords: Multilingual Language Models, Language Consistency, Cross-lingual Transfer, Interpretability, Logit Lens, Semantic Similarity, Layerwise Fine-Tuning, Output Space Control, Model Analysis, Language Control

Abstract: Despite multilingual pretraining, large language models often struggle with non-English tasks, particularly in language control--the ability to respond in the intended language. We identify and characterize two key failure modes: the *multilingual transfer bottleneck* (correct language, incorrect task response) and the *language consistency bottleneck* (correct task response, wrong language). To systematically surface these issues, we design a four-scenario evaluation protocol spanning MMLU, MGSM, and XQuAD benchmarks. 
To probe these issues with interpretability, we extend logit lens analysis to track language probabilities layer by layer and compute cross-lingual semantic similarity of hidden states. The results reveal a three-phase internal structure: early layers align inputs into shared semantic space, middle layers perform task reasoning, and late layers drive language-specific generation. Guided by these insights, we introduce *selective fine-tuning* of only the final layers responsible for language control. On Qwen-3-32B and Bloom-7.1B, this method achieves over 98% language consistency across six languages while fine-tuning only 3–5% of parameters, without sacrificing task accuracy. To the best of our knowledge, this is the first approach to leverage *layer-localization of language control* for efficient multilingual adaptation.