Language Surgery in Multilingual Large Language Models
Authors: name: Joanito Agili Lopo; name: Muhammad Ravi Shulthan Habibi; name: Tack Hwa Wong; name: Muhammad Ilham Ghozali; name: Fajri Koto; name: Genta Indra Winata; name: Peerat Limkonchotiwat; name: Alham Fikri Aji; name: Samuel Cahyawijaya
Published: 2025-06-14 11:09:50

Abstract: Large Language Models (LLMs) have demonstrated remarkable generalization
capabilities across tasks and languages, revolutionizing natural language
processing. This paper investigates the naturally emerging representation
alignment in LLMs, particularly in the middle layers, and its implications for
disentangling language-specific and language-agnostic information. We
empirically confirm the existence of this alignment, analyze its behavior in
comparison to explicitly designed alignment models, and demonstrate its
potential for language-specific manipulation without semantic degradation.
Building on these findings, we propose Inference-Time Language Control (ITLC),
a novel method that leverages latent injection to enable precise cross-lingual
language control and mitigate language confusion in LLMs. Our experiments
highlight ITLC's strong cross-lingual control capabilities while preserving
semantic integrity in target languages. Furthermore, we demonstrate its
effectiveness in alleviating the cross-lingual language confusion problem,
which persists even in current large-scale LLMs, leading to inconsistent
language generation. This work advances our understanding of representation
alignment in LLMs and introduces a practical solution for enhancing their
cross-lingual performance.