Lens: Rethinking Multilingual Enhancement for Large Language Models

## Abstract

Abstract Despite the growing global demand for large language models (LLMs) that serve users from diverse linguistic backgrounds, most cutting-edge LLMs remain predominantly English-centric. This creates a performance gap across languages, restricting access to advanced AI services for non-English speakers. Current methods to enhance multilingual capabilities largely rely on data-driven post-training techniques, such as multilingual instruction tuning or continual pre-training. However, these approaches encounter significant challenges, including the scarcity of high-quality multilingual datasets and the limited enhancement of multilingual capabilities. They often suffer from off-target issues and catastrophic forgetting of central language abilities. To this end, we propose Lens , a novel approach to enhance multilingual capabilities of LLMs by leveraging their internal language representation spaces. Specially, Lens operates by manipulating the hidden representations within the language-agnostic and language-specific subspaces from top layers of LLMs. Using the central language as a pivot, the target language is drawn closer to it within the language-agnostic subspace, allowing it to inherit well-established semantic representations. Meanwhile, in the language-specific subspace, the representations of the target and central languages are pushed apart, enabling the target language to express itself distinctly. Extensive experiments on one English-centric and two multilingual LLMs demonstrate that Lens effectively improves multilingual performance without sacrificing the model’s original central language capabilities, achieving superior results with much fewer computational resources compared to existing post-training approaches.

## 1 Introduction

In an increasingly interconnected world, large language models (LLMs) are expected to cater to a diverse range of users across various linguistic backgrounds (Ouyang et al., ; Zhao et al., ; Zheng et al., ). However, despite this global trend, most state-of-the-art LLMs remain predominantly English-centric (Brown et al., ; Touvron et al., ; ; Jiang et al., ; AI@Meta, ). These models exhibit significantly better performance in English than in other languages, leading to an imbalance in user experience and potentially excluding large segments of the global population from accessing advanced AI services (Wang et al., ; Zhu et al., ).

This disparity has directly spurred research efforts to enhance the multilingual capabilities of LLMs, aiming to provide more equitable access and performance across various linguistic communities. Current approaches are predominantly based on data-driven post-training paradigm, such as multilingual instruction tuning (Zhang et al., ; Zhu et al., ; Üstün et al., ) or continual pre-training (Cui et al., ; Kuulmets et al., ; Jaavid et al., ), which primarily seeks to inject or elicit multilingual knowledge with the supervision signals from *external* datasets.

While this paradigm is widely embraced and demonstrates certain successes, it faces several significant challenges. (1) The efficacy of multilingual enhancement heavily depend on large-scale and high-quality multilingual datasets (Zhou et al., ; Liu et al., ), which are both time-consuming and labor-intensive to obtain for each language. (2) It favors improving multilingual understanding over generation capabilities, which leaves the off-target issue inadequately addressed. As a result, the model often struggles to generate accurate responses in the intended language when prompted (Zhang et al., ; Lai et al., ; Sennrich et al., ). (3) The model’s performance in languages it previously handled well is risking at catastrophic forgetting (McCloskey & Cohen, ), such as English in the LLaMA family (Touvron et al., ; ; AI@Meta, ).

In this work, we seek to provide a new perspective on addressing the aforementioned challenges by exploring and manipulating the internal representation within the language-related latent spaces of LLMs (Zou et al., ; Park et al., ). Taking the enhancement of multilingual capabilities for English-centric LLMs as an example. This is based on the intuitive idea that the well-established English representations in existing English-centric LLMs can act as a pivot to improve the performance of other languages. More specifically, for the target language to be extended, its *language-agnostic* semantic representations should be *pulled close* to those of English, enabling it to quickly inherit general abilities in English without the need for supervision signals from external multilingual training data. Conversely, the *language-specific* linguistic representations of the target language should be *pushed away* from English to avoid off-target issues, ensuring accurate responses in the target language. Also, during this process, it is crucial to ensure that the English pivot representation remains unchanged to effectively prevent catastrophic forgetting.

To achieve this, we propose Lens, a novel multiLingual Enhancement method based on the hidden represeNtations within language Space of LLMs. To be more specific, Lens comprises two stages: Language Subspace Probing (LSP) and Language Subspace Manipulation (LSM). During LSP, the multilingual hidden space within a single layer of the backbone are decoupled into two orthogonal components: a *language-agnostic* subspace and a *language-specific* subspace. These subspaces are efficiently derived using singular value decomposition. Then in LSM, we align the parallel multilingual input representations of the target language and the central language in the language-agnostic subspace. This allows the target language to directly inherit the well-established semantic representations of the central language. Simultaneously, the projection components of the target language within the language-specific space are pushed away from those of the central language, guiding the target language toward its distinct linguistic expression and ensuring the target language is properly expressed thereby mitigating the off-target issue. Finally, we align the central language’s current representations with its original ones to preserve its proficiency during multilingual enhancement.

We conduct extensive experiments under bilingual and multilingual enhancement setups. Results on one English-centric (LLaMA-3-8B-Instruct) and 2 multilingual LLMs (LLaMA-3.1-8B-Instruct and Phi-3.5-mini-Instruct), demonstrate that Lens succeed to improve target languages on both multilingual comprehension and generation tasks without sacrificing the strong capability of central language, showing the efficacy and scalability of our method. Deeper analysis highlights the significance of steering the target language towards its unique expressions within its own language-specific subspace to fully enhance both comprehension and generation capabilities. This is overlooked by most existing approaches, which primarily focus on aligning representations across different languages to boost multilingual performance. It is crucial to note that, building on recent findings that language-related parameters are primarily concentrated in the top layers of LLMs (Wendler et al., ), Lens achieves high resource efficiency compared to baselines, with much fewer computational costs by only updating the model’s higher layers using just a few hundred data points.

The main contributions of this work are summarized as follows: (1) We provide a novel perspective for the multilingual enhancement of large language models with their internal language representation space leveraged. (2) We propose Lens, an efficient and effective multilingual enhancement method that operates within the language representation space of large language models. (3) Extensive experiments on one English-centric and two multilingual LLMs demonstrate the effectiveness, efficiency, scalability of our method to obtain truly multilingual enhanced chat-style backbones without sacrificing original central language performance.

## 2 Related Works

#### Multilingual Large Language Model

With the acceleration of globalization, multilingual large language models (MLLMs) are gaining significant attention for their ability to handle multiple languages comprehensively (Qin et al., ). Pretraining on multilingual data is a common approach to gain the multilingual capabilities (Conneau & Lample, ; Xue et al., ; Lin et al., ; Shliazhko et al., ; Wei et al., ; Xue et al., ; Le Scao et al., ; Blevins et al., ). However, due to the uneven distribution of data in pretraining corpora, current LLMs or MLLMs exhibit uneven language capabilities, with most state-of-the-art models heavily biased towards English (Jiang et al., ; AI@Meta, ; Abdin et al., ). Moreover, pretraining from scratch is computationally intensive. These limitations have directly sparked research into expanding or enhancing the language capabilities of current LLMs or MLLMs.

#### Multilingual Enhancement for LLMs

Current methods for multilingual enhancement of LLMs can be categorized into two types: 1) prompt-based methods and 2) post-training-based methods.

The former focuses on leveraging the LLMs’ own translation capabilities to translate low-resource language inputs into the central language, and then generating a response (Shi et al., ; Huang et al., ; Qin et al., ; Etxaniz et al., ; Zhang et al., ). For example, Huang et al. () introduce cross-lingual-thought prompting to minimize language disparities. However, Liu et al. () reveal the limitations of these methods, showing they are not optimal for real-world scenarios and highlighting the necessity of more comprehensive multilingual enhancement.

The latter aims to conduct further multilingual post-training to inject or elicit extensive language knowledge for specific languages, including ways of continual pre-training (Zhang et al., ; Cui et al., ; Chen et al., ; Lin et al., ; Kuulmets et al., ; Jaavid et al., ) and instruction tuning (Muennighoff et al., ; Chen et al., ; Indurthi et al., ; Ahuja et al., ; Lai & Nissim, ; Zhang et al., ; Zhu et al., ; Li et al., ; Zhao et al., ). For example, Cui et al. () attempt to inject Chinese knowledge into LLaMA by conducting continual pre-training on a large-scale Chinese corpus, while Zhu et al. () focus more on building language alignment through cross-lingual instruction tuning and translation training.

Our proposed Lens stands out from existing methods in that we seek multilingual supervision signals from the *internal* language representation space of the LLMs, rather than relying primarily on *external* multilingual datasets as in the above methods, which offers fresh insights and new opportunities for enhancing the multilingual capabilities of LLMs both efficiently and effectively.

#### Representation Engineering

Editing or manipulating representation within LLMs has garnered increasing attention due to its transparency and lightweight properties (Zou et al., ). This is theoritically rooted from Linear Representation Hypothesis (Mikolov et al., ; Nanda et al., ; Park et al., ), which posits that various human-interpretable concepts are encoded in linear subspaces of model representations. Building upon this, exist works attempt to edit representations at inference time to develop models that are more truthful (Li et al., ; Campbell et al., ; Zhang et al., ), and harmless (Lee et al., ; Uppaal et al., ). We expand and implement this paradigm for the multilingual enhancement of LLMs by focusing on representations during the training phase, ensuring that the efficiency of LLMs remains unaffected during the inference phase.

## 3 Methodology

### 3.1 Overview of Lens

We propose Lens, a novel method for effective and efficient multilingual enhancement of LLMs based on their internal language representation spaces. The overall diagram of Lens is displayed in Figure 1, consisting of two key stages: (1) Language Subspace Probing (LSP) and (2) Language Subspace Manipulation (LSM). The subsequent section offers a detailed introduction to them.

### 3.2 Language Subspace Probing

In this section, we first introduce our method to decouple and probe the language-agnostic and language-specific subspace within a single model layer in an unsupervised manner.

Assuming we aim to enhance the multilingual capabilities of a backbone model for $L$ languages, which include one central language and $L-1$ target languages to be enhanced. In each layer of the backbone, we can obtain a mean representation for each language $l$:

$$ $\bm{m}_{l}=\frac{1}{n}\sum_{i=1}^{n}\bm{e}_{l}^{i}$ (1) $$

where $\bm{e}_{l}^{i}\in\mathbb{R}^{d}$ is the embedding of the last token for the $i$-th sample in language $l$, and $n$ is the total number of samples for each language. Concatenating $\bm{m}_{l}$ of $L$ languages column-by-column results in the mean embedding matrix $\bm{M}\in\mathbb{R}^{d\times L}$ specifying the multilingual latent space.

Follow previous works (Pires et al., ; Libovickỳ et al., ; Yang et al., ), we hypothesize that such multilingual latent space $M$ could be decomposed into two orthogonal components (1) a language-agnostic subspace $\bm{M}_{a}$ representing what is commonly shared across languages and (2) a language-specific one $\bm{M}_{s}$ specifying on which different languages express different linguistic signals. Following Piratla et al. (); Xie et al. (), the objective can be formulated as:

$$ $\displaystyle\min_{\bm{M}_{a},\bm{M}_{s},\bm{\Gamma}}$ $\displaystyle\left\|\bm{M}-\bm{M}_{a}\bm{\mathbbm{1}}^{\top}-\bm{M}_{s}\bm{ \Gamma}^{\top}\right\|_{F}^{2}$ (2) s.t. $\displaystyle\text{Span}\left(\bm{M}_{a}\right)\perp\text{Span}\left(\bm{M}_{s }\right),$ $$

where $\bm{M}_{a}\in\mathbb{R}^{d\times 1}$, $\bm{M}_{s}\in\mathbb{R}^{d\times r}$ and $\bm{\Gamma}\in\mathbb{R}^{L\times r}$ is the coordinates of language-specific signals along the subspace’s $r$ components. And a lower dimensionality for $\bm{M}_{a}$ is reasonable because the semantic consistency across different languages can be captured in a simpler form. Meanwhile, $\bm{M}_{s}$ requires a higher dimensionality to account for the distinct features of each language.

The optimal solution of Equation 2 can be computed efficiently via Singular Value Decomposition (SVD), where Algorithm 1 in Appendix B presents the detailed procedure.

After obtaining the language-specific subspace $\bm{M}_{s}$, we aim to identify a direction of language expression within this subspace, which points from the projection of mean representation from target language $\bm{m}_{l}$ to that from central language $\bm{m}_{c}$. Formally, the linguistic language expression direction $\bm{\delta}_{l}\in\mathbb{R}^{d}$ for each target language $l$ is calculated as:

$$ $\displaystyle\bm{\delta}_{l}=\bm{M}_{s}^{T}\bm{M}_{s}(\bm{m}_{l}-\bm{m}_{c})$ (3) $$

Figure: Figure 1: The overall architecture of our proposed Lens for multilingual enhancement. (1) In the LSP, we begin by decomposing the multilingual latent space, which is formed by the representations of probing samples from both the target and central languages. Using singular value decomposition (SVD), we separate this space into two orthogonal components: a language-agnostic subspace, $\bm{M}_{a}$, and a language-specific subspace, $\bm{M}_{s}$. (2) Then in LSM, the parallel multilingual representations of the target languages are pushed toward their respective linguistic expression directions within $\bm{M}_{s}$, while being pulled closer to the central language in $\bm{M}_{a}$. Additionally, the representations of the central language are carefully constrained to remain largely intact.
Refer to caption: x2.png

### 3.3 Language Subspace Manipulation

To eliminate the heavy reliance on hard-to-access high-quality multilingual datasets, we leverage the well-trained hidden representations of the central language in LLMs as a pivot to derive supervision signals for multilingual enhancement within the model’s internal language space.

First, We propose to pull parallel multilingual representations closer within the shared language-agnostic subspace $\bm{M}_{a}$. This allows us to directly inherit the well-established general capabilities of the central language. Formally, this goal is accomplished by projecting multilingual representations (at the position of the last token) onto the subspace $\bm{M}_{a}$, with the optimization objective defined as:

$$ $\displaystyle\mathcal{L}_{1}=\left\|\bm{M}_{a}^{T}\bm{M}_{a}(\bm{x}_{l}-\bm{x} _{c})\right\|^{2}$ (4) $$

where $\bm{x}_{l}$ and $\bm{x}_{c}$ are parallel multilingual representations from target language $l$ and central one.

Second, to ensure that each target language can be accurately expressed and to alleviate the off-target issue, we need to push the multilingual representations in the language-specific subspace $\bm{M}_{s}$ towards their respective language-specific expression directions. This can be achieved through the projection onto the subspace $\bm{M}_{s}$ and optimizing the following objective:

$$ $\displaystyle\mathcal{L}_{2}=\left\|\bm{M}_{s}^{T}\bm{M}_{s}(\bm{x}_{l}-\bm{x} _{l}^{\text{ref}})-\lambda_{l}\bm{\delta}_{l}\right\|^{2}$ (5) $$

where $\bm{x}_{l}^{\text{ref}}$ is the representation of target language $l$ obtained from original reference model and $\lambda_{l}$ is a scalar of push strength for the corresponding language. The above process can be interpreted as directing the language-specific representations of each target language to shift a specific distance from their original positions toward a direction that enables accurate expression.

Finally, to ensure that the capabilities of the central language are not compromised and maintain a stable alignment objective for the target language, we constrain the representations of central language to remain predominantly intact:

$$ $\displaystyle\mathcal{L}_{3}=\left\|\bm{x}_{c}-\bm{x}_{c}^{\text{ref}}\right\| ^{2}$ (6) $$

where $\bm{x}_{c}^{\text{ref}}$ is the representation of central language $c$ obtained from original reference model.

The final optimization objective of Lens is:

$$ $\displaystyle\mathcal{L}=\lambda_{1}\mathcal{L}_{1}+\mathcal{L}_{2}+\lambda_{3 }\mathcal{L}_{3}$ (7) $$

where $\lambda_{1}$ and $\lambda_{3}$ are hyper-parameters to balance the impact of these two losses.

## 4 Experiments

### 4.1 Experimental Setup

#### Models

We select one representative English-centric LLMs: LLaMA-3-8B-Instruct (AI@Meta, ) and 2 MLLMs: LLaMA-3.1-8B-Instruct(^1^11[https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct](https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct)) and Phi-3.5-mini-instruct (Abdin et al., ), to fully validate the effectiveness and generality of our Lens in enhancing multilingual performance.

#### Languages to be Enhanced

We conduct experiments in both bilingual and multilingual settings to address various multilingual enhancement needs.

In the bilingual setting, English (En) serves as the central language, while Chinese (Zh) is chosen as the target language for expansion. Chinese is selected due to its growing prominence in the academic focus on multilingual enhancement for LLMs.

In the multilingual setting, we select six target languages for enhancement based on the availability of language resources. The high-resource languages are Chinese (Zh) and Japanese (Jp); the medium-resource languages are Korean (Ko) and Arabic (Ar); and the low-resource languages are Bengali (Bn) and Swahili (Sw), with English (En) continuing to serve as the central language.

It is important to note that these target languages are classified as *out-of-scope* in the official model card of the above LLMs and MLLMs, which further underscores their relevance for enhancement.

#### Training Data

The multilingual data used for the language subspace probing stage is sourced from the Aya Dataset (Üstün et al., ), a human-annotated, non-parallel multilingual instruction fine-tuning dataset with 204,000 instances in 65 languages. For the language subspace manipulation stage, we rely on parallel multilingual data from the Bactrian-X dataset (Li et al., ), which contains 3.4 million instruction-response pairs in 52 languages. These pairs are generated by translating 67,000 English instructions (derived from alpaca-52k (Taori et al., ) and dolly-15k) into 51 languages using the Google Translate API, and then obtaining natural responses from ChatGPT.

We sample 300 data points from the Aya Dataset for each language to probe the language space and 200 data points from the Bactrian-X dataset per language to manipulate the language space.

#### Benchmarks

To comprehensively measure the efficacy of our Lens on various multilingual tasks, we employ 5 mainstream benchmarks for evaluation, which can be categorized into multilingual understanding and multilingual generation:

- •
Multilingual Understanding: (1) XCOPA (Ponti et al., ), (2) XWinograd (Muennighoff et al., ), (3) XStoryCloze (Lin et al., ) and (4) M-MMLU (Hendrycks et al., ; Lai et al., ). Accuracy is adopted as the evaluation metric and we randomly sample up to 1,000 data points from each benchmark for evaluation.
- •
Multilingual Generation: (5) MT-Bench (Zheng et al., ): The dataset is designed for open-ended generation to evaluate a model’s ability to follow multi-turn instructions. The evaluation follows the LLM-as-a-judge approach, where GPT-4o is prompted to assign a score directly to a single response on a scale of 1 to 10. It is essential to highlight that the languages targeted for enhancement, as mentioned above, are all within the capability range of GPT-4o, especially given that its official model card(^2^22[https://cdn.openai.com/gpt-4o-system-card.pdf](https://cdn.openai.com/gpt-4o-system-card.pdf).) emphasizes support for low-resource languages such as Swahili (sw) and Bengali (bn). This underscores the validity and reliability of the evaluation approach. In addition, we employ Language Fidelity (Holtermann et al., ) as a metric to assess the consistency between input and output languages, offering a clear measure of how effectively different methods mitigate the model’s off-target issues.

Please refer to Appendix C for the detailed description of the benchmarks.

### 4.2 Baseline Methods

For comparison, we consider the following baseline methods that enhance LLMs’ multilingual capabilities using multilingual instruction fine-tuning technique: (1) xSFT & xSFT-Full (Ouyang et al., ): xSFT performs multilingual instruction fine-tuning using the same data volume as our Lens. In contrast, xSFT-Full utilizes the full dataset for each target language from the Aya Collection and Bactrian-X. (2) QAlign (Zhu et al., ): It explores the benefits of question alignment, where the model is trained to translate inputs into English by finetuning on X-English parallel question data. (3) SDRRL (Zhang et al., ): It is based on self-distillation from resource-rich languages that effectively improve multilingual performance by leveraging self-distillated data.

**Table 1: Detailed results on the multilingual understanding and multilingual generation benchmarks with the English-centric LLaMA-3-8B-Instruct backbone and the multilingual LLaMA-3.1-8B-Instruct backbone under the bilingual setting (English and Chinese). Accuracy serves as the evaluation metric for multilingual understanding, while GPT-4o ratings (on a scale of 1 to 10) are provided for MT-Bench. The values in parentheses represent language fidelity. Results highlighted in green indicate an improvement or performance comparable to the original backbone, while those highlighted in red signal a decline in performance relative to the original backbone.**
|  | Multilingual Understanding | Multilingual Generation |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | XCOPA | XWinograd | XStoryCloze | M-MMLU | AVG. | MT-Bench |  |  |  |  |  |  |
|  | En | Zh | En | Zh | En | Zh | En | Zh | En | Zh | En | Zh |
| LLaMA-3 | - | 83.40 | 63.50 | 54.37 | 95.40 | 88.90 | 64.90 | 49.40 | 74.60 | 69.02 | 6.99 (100%) | 2.72 (43.75%) |
| xSFT | - | 87.20 | 64.30 | 63.49 | 95.10 | 90.60 | 62.80 | 46.10 | 74.07 | 71.85 | 4.79 (100%) | 2.94 (88.75%) |
| xSFT-Full | - | 84.60 | 58.80 | 60.11 | 93.50 | 90.30 | 60.60 | 43.20 | 70.97 | 69.55 | 5.80 (100%) | 4.44 (92.50%) |
| QAlign | - | 52.20 | 55.10 | 47.02 | 89.20 | 71.90 | 56.40 | 34.00 | 66.90 | 51.28 | 3.59 (100%) | 1.23 (37.50%) |
| SDRRL | - | 85.20 | 64.80 | 55.95 | 92.60 | 84.30 | 63.80 | 47.80 | 73.73 | 68.31 | 6.60 (100%) | 3.84 (73.75%) |
| Lens (Ours) | - | 87.60 | 63.80 | 66.67 | 94.70 | 91.80 | 64.40 | 48.60 | 74.30 | 73.67 | 7.21 (100%) | 5.77 (97.50%) |
| LLaMA-3.1 | - | 90.40 | 64.10 | 68.65 | 95.80 | 91.40 | 69.30 | 52.50 | 76.40 | 75.74 | 7.31 (100%) | 5.38 (93.75%) |
| xSFT | - | 88.00 | 63.70 | 67.46 | 96.20 | 92.70 | 68.10 | 53.10 | 76.00 | 75.32 | 5.33 (100%) | 3.32 (90.00%) |
| xSFT-Full | - | 86.80 | 60.40 | 62.50 | 90.60 | 83.80 | 66.10 | 49.90 | 72.37 | 70.75 | 6.02 (100%) | 4.18 (92.50%) |
| QAlign | - | 55.00 | 56.00 | 48.02 | 94.10 | 52.30 | 64.10 | 33.50 | 71.40 | 47.20 | 4.13 (100%) | 2.65 (83.75%) |
| SDRRL | - | 87.20 | 63.20 | 58.83 | 95.30 | 89.80 | 63.50 | 45.30 | 74.00 | 70.31 | 6.49 (100%) | 3.14 (58.75%) |
| Lens (Ours) | - | 90.20 | 64.60 | 69.44 | 95.90 | 91.80 | 69.10 | 52.60 | 76.53 | 76.01 | 7.41 (100%) | 5.96 (93.75%) |

### 4.3 Implementation Details

Lens is a model-agnostic multilingual enhancement method that is compatible with different transformer-based LLM. Our experiments are implemented with PyTorch (Paszke et al., ) and Transformer library (Wolf et al., ) on a single NVIDIA A800-SXM4-80GB GPU. The training duration is set to one epoch with the learning rate of 1e-5 and batch size of 8 across all backbones. For more detailed settings, please refer to the Appendix D.

### 4.4 Overall Results

Table 1 and Figure 2 present the performance comparison between Lens and recent multilingual enhancement baselines on multilingual understanding and generation benchmarks, under bilingual and multilingual configurations, respectively. For additional results, including those on Phi-3.5-mini-instruct and multilingual configuration for the other two backbones, please see Appendix E. From the outcomes across all backbones, we have drawn the following key insights:

Lens succeed to achieve a comprehensive improvement for the multilingual capabilities of (M)LLMs without sacrificing original central language performance. Specifically, it enhances the multilingual capabilities of the backbone on both multilingual understanding and generation benchmarks, showing a marked increase in language fidelity during multilingual generation. This effectively mitigates the off-target issue. Moreover, Lens is the only approach that enhances multilingual performance across all languages. In contrast, baseline methods primarily focus on boosting multilingual understanding with little to no improvement in generation tasks. Additionally, methods like QAlign and SDRRL, which rely on translation-based training for multilingual alignment, fall short in effectively enhancing large models’ overall multilingual performance. This suggests that aligning multilingual representations alone is insufficient for fully optimizing multilingual capabilities (Hua et al., ). Finally, Lens safeguards the central language from catastrophic forgetting, allowing the resulting model to effectively serve users from diverse linguistic backgrounds.

Using the central language representations within the backbone as a supervision signal proves more effective than relying on external data for supervision. The key distinction between Lens and baseline methods lies in how multilingual performance is enhanced: Lens relies on the model’s internal representation of the central language, while baseline methods depend on external data. This difference make baselines not only fail to improve the target languages but also lead to performance degradation. This phenomenon becomes more pronounced in xSFT-Full when trained with more data. However, the Aya Dataset and Bactrain-X datasets we used are already considered high-quality multilingual resources, widely employed and proven effective in boosting multilingual capabilities in previous models such as mT5 and LLaMA-2 (Li et al., ; Üstün et al., ). This highlights that for current extensively trained LLMs such as LLaMA-3 (which has been trained on over 15T data), an over-reliance on external supervision signals may fall short of scalability needs (Cao et al., ). We hope Lens could inspire further research to explore more efficient, scalable, and automated supervision signals for multilingual enhancement of the most advanced LLMs.

Figure: Figure 2: Results on the multilingual generation benchmark with LLaMA-3-8B-Instruct backbone under the multilingual setting. GPT-4o ratings (on a scale of 1 to 10) are provided for MT-Bench.
Refer to caption: x3.png

Figure: Figure 3: The ablation results to verify the effectiveness and impact of different optimization objectives in LSM. MU Performance stands for the average performance on all multilingual understanding benchmarks, while MU Performance is the results on MT-Bench. LF represents language fidelity.
Refer to caption: x4.png

## 5 Analysis

### 5.1 Ablation Study

We further conduct ablation studies to demonstrate the effectiveness of our proposed three optimization objectives in LSM. The overall results under the bilingual enhancement setting with LLaMA-3-8B-Instruct backbone are shown in Figure 3. We can draw the following key findings.

The alignment of multiple languages within language-agnostic subspaces mainly impacts multilingual comprehension rather than generation capabilities. As we incrementally raise the coefficient $\lambda_{1}$ of the multilingual alignment objective $\mathcal{L}_{1}$ from zero, Chinese comprehension improves, but its generation ability stays largely unaffected.

Enhancing the separation between representations of different languages in the language-specific subspace is vital for boosting multilingual performance. In particular, as illustrated in the middle part of Figure 3, both comprehension and generation abilities in Chinese improve significantly as the coefficient $\lambda_{l}$ increases. This finding indicates that the commonly accepted notion of merely aligning languages to enhance multilingual capabilities (Cao et al., ; Zhu et al., ; Li et al., ; Hua et al., ) may not be sufficient for fully optimizing the multilingual performance of current LLMs. We hope this result encourages future research to focus more on eliciting and leveraging language-specific information within LLMs.

Maintaining the representations of the central language without significant changes can provide a stable and reliable alignment supervision for the target language to be enhanced. As illustrated on the right side of Figure 3, removing the objective to retain English representations leads to a significant decline in the backbone’s Chinese performance. This could be due to alterations in the English representations during optimization, which may cause misalignment in the target for Chinese, thus impacting its performance. However, regarding the English capability, since our modifications are applied to the upper layers of the backbone (layers 31 and 32 in LLaMA-3-8B-Instruct), most of the parameters remain frozen and unaffected. As a result, even without the retaining objective, the backbone’s English ability does not suffer from significant catastrophic forgetting. In the analysis shown in Figure 5, we observe that as the number of updated layers increases, the backbone’s English understanding and generation capabilities are also not impacted by catastrophic forgetting. This can manifest the effectiveness of the retain optimization objective in safeguarding the performance of the central language.

### 5.2 Impact of Varying the Number of Manipulated Layers

Recent studies on the interpretability of LLMs has sought to reveal the mechanisms underlying their multilingual capabilities (Zhao et al., ; Zhong et al., ). A growing consensus suggests that language-specific parameters or neurons are primarily concentrated in the upper layers of these models, while the middle layers tend to process inputs from various languages using a shared and language-agnostic mechanism (Chen et al., ; Wendler et al., ; Tang et al., ; Kojima et al., ; Zhang et al., ). Drawing inspiration from this, our main experiments focus on performing updates solely within the upper layers of the backbone, resulting in a notable improvement in multilingual performance. In Figure 5, we explore the effect of increasing the number of layers involved on the model’s multilingual enhancement. The horizontal axis represents the starting point of the layers where manipulation is applied, with the default endpoint being the final layer. This experiment is performed under the bilingual enhancement with LLaMA-3-8B-Instruct.

“Thinking” in English at the intermediate layers is more favorable for improving multilingual understanding. If we partition representations of target language into the language-specific subspace too early at the middle layers, it may impair its multilingual understanding capability. On the contrary, inheriting more from the shared representations at the middle layers, while emphasizing language-specific representations only at the higher layers (where most language-specific parameters and neurons are concentrated), is more beneficial for enhancing multilingual performance.

It is important to note that modifying only the final layer does not significantly improve either multilingual understanding or generation. This is because language-specific information is not sufficiently enhanced, causing the model to suffer from off-target issues and struggle to represent specific languages accurately. The lack of improvement in multilingual understanding aligns with the findings in Section 5.1, which highlight the critical role of supervision provided by the Push loss ($\mathcal{L}_{2}$).

Our proposed Lens further validates the conclusions of existing works on LLM interpretability and applies these findings to multilingual enhancement of LLMs.

Figure: Figure 4: The impact of manipulating different backbone layers on multilingual performance enhancement.
Refer to caption: x5.png

### 5.3 Comparison with Open-Sourced Multilingual-Enhanced LLMs

In Section 4.4, our main experiment primarily compares with *reproducible* baseline methods for multilingual enhancement. Additionally, we extend our comparisons to several open-source LLMs from the community that leverage private datasets and large-scale post-training to improve multilingual performance. In particular, we focus on the Chinese-LLaMA-3 series, which builds on LLaMA-3 series to enhance Chinese capabilities and includes three different versions:

- •
Chinese-LLaMA-3-Instruct-V1:(^3^33[https://huggingface.co/hfl/llama-3-chinese-8b-instruct](https://huggingface.co/hfl/llama-3-chinese-8b-instruct)) This model is continually pre-trained on 120GB of Chinese text and fine-tuned with 500 million carefully curated instruction data points, based on the LLaMA-3-8B. These training datasets is not available to the public.
- •
Chinese-LLaMA-3-Instruct-V2:(^4^44[https://huggingface.co/hfl/llama-3-chinese-8b-instruct-v2](https://huggingface.co/hfl/llama-3-chinese-8b-instruct-v2)) This version is directly fine-tuned on the same 500 million instruction data points using the LLaMA-3-8B-Instruct model.
- •
Chinese-LLaMA-3-Instruct-V3:(^5^55[https://huggingface.co/hfl/llama-3-chinese-8b-instruct-v3](https://huggingface.co/hfl/llama-3-chinese-8b-instruct-v3)) This model is created by merging V1, V2, and the original LLaMA-3-8B-Instruct, followed by fine-tuning on 5,000 instruction data points.

The experimental results and the resource consumption of different methods are presented in Figure 5 and Table 5 in Appendix E, respectively. The resulting model applied with Lens is identical to the one utilized for bilingual enhancement in Table 1. Remarkably, Lens demonstrates more comprehensive enhancement of the Chinese capabilities with extremely low resource overhead compared to these three models. This reinforces our claim that Lens is an efficient and effective approach for boosting the multilingual capabilities of LLMs. Additionally, all the data leveraged by Lens is publicly accessible, which eliminates the need for laboriously gathering extensive high-quality multilingual datasets and makes it easily shareable with the community.

Figure: Figure 6: The PCA visualization of multilingual representations projected in the obtained language-agnostic subspace (right) and the language-specific (left) subspace. The backbone model is LLaMA-3-8B-Instruct after multilingual enhanced with Lens.
Refer to caption: x7.png

### 5.4 Visualization Analysis

To further confirm whether Lens manipulates language representations within different language subspaces as anticipated, we perform a visualization analysis. Specifically, as shown in Figure 6, we perform Principal Component Analysis (PCA) to visualize the projection of multilingual representations in our obtained language-agnostic subspace and the language-specific subspace. Parallel inputs in seven languages are sourced from the MultiQ datasets (Holtermann et al., ). The visualization results indicate that representations of different languages converge within a narrow range in the language-agnostic subspace, while forming distinct clusters in the language-specific subspace, supporting our claim. This also highlights the advantages of Lens in delivering transparent, controllable, and interpretable solutions for the multilingual enhancements of LLMs.

## 6 Conclusion

In this paper, we introduce Lens, a novel method designed for the effective, efficient and comprehensive multilingual enhancement of large language models (LLMs). Lens first decouple the multilingual hidden spaces of the backbone into two orthogonal components: a language-agnostic subspace and a language-specific subspace. Then taking well-established representations of the central language as a pivot, representations of target languages are pulled closer and pushed away from them in language-agnostic subspace and language-specific subspace, respectively. Experimental results on 3 representative cutting-edge LLMs demonstrate that Lens outperforms baseline methods with much lower training costs, underscoring its efficacy, efficiency and scalability.

## Appendix A Limitation and Future Work

Despite our Lens achieving comprehensive and efficient multilingual enhancement, there are still limitations and future directions worth exploring.

First, due to limited computational resources, our experiments are not conducted on larger-scale models (larger than 8B). This remains a valuable direction to apply Lens on larger LLMs.

Second, our current operations on language representation are still relatively coarse-grained. Future work could delve into more specific parameter areas for finer operations.

Finally, as we find that relying too much on external datasets to enhance multilingual capabilities may be limited, we instead seek higher quality supervision signals from within the model itself. Future work could consider combining these two paradigms by incorporating data selection strategies (Albalak et al., ; Liu et al., ), thereby providing higher quality multilingual supervision signals to the model from both internal and external sources.

## Appendix B Probing for Language Subspace

The optimal solution of Equation 2 can be computed efficiently via Singular Value Decomposition (SVD).
Algorithm 1 presents the detailed procedure.
Readers interested in more details can consult the proof provided in Xie et al. ().
The only hyperparameter $r<L$ controls the amount of language-specific information captured by the identified subspace.
The larger $r$ is, the more language-specific signals we can identify.

Figure: Algorithm 1 Language Subspace Probing

## Appendix C Multilingual Benchmarks

We comprehensively measure the efficacy of our Lens on various multilingual tasks, including 5 mainstream benchmarks for evaluation. They can be categorized into the evaluation of multilingual understanding and multilingual generation.

For multilingual understanding:

- •
XCOPA (Ponti et al., ):(^6^66[https://huggingface.co/datasets/cambridgeltl/xcopa](https://huggingface.co/datasets/cambridgeltl/xcopa)) A benchmark to evaluate the ability of machine learning models to transfer commonsense reasoning across languages. The dataset is the translation and re-annotation of the English COPA (Roemmele et al., ) and covers 11 languages from 11 families and several areas around the globe. The dataset is challenging as it requires both the command of world knowledge and the ability to generalise to new languages. In our experimental setup, this benchmark covers both Chinese (Zh) and Swahili (Sw).
- •
XWinograd (Muennighoff et al., ):(^7^77[https://huggingface.co/datasets/Muennighoff/xwinograd](https://huggingface.co/datasets/Muennighoff/xwinograd)) A benchmark to evaluate the ability of machine learning models to transfer commonsense reasoning across languages. The dataset is the translation of the English Winograd Schema datasets and it adds 488 Chinese schemas from CLUEWSC2020 , totaling 6 languages. Formulated as a fill-in-a-blank task with binary options, the goal is to choose the right option for a given sentence which requires commonsense reasoning. In our experimental setup, this benchmark covers English (En), Chinese (Zh) and Japanese (Jp).
- •
XStoryCloze (Lin et al., ):(^8^88[https://huggingface.co/datasets/juletxara/xstory_cloze](https://huggingface.co/datasets/juletxara/xstory_cloze)) A benchmark to evaluate the ability of machine learning models to transfer commonsense reasoning across languages. The dataset consists of the professionally translated version of the English StoryCloze dataset (Spring 2016 version) to 10 non-English languages. The dataset is challenging and is designed to evaluate story understanding, story generation, and script learning. In our experimental setup, this benchmark covers English (En), Chinese (Zh), Arabic (Ar) and Swahili (Sw).
- •
M-MMLU (Hendrycks et al., ; Lai et al., ):(^9^99[https://huggingface.co/datasets/alexandrainst/m_mmlu](https://huggingface.co/datasets/alexandrainst/m_mmlu)) A benchmark to evaluate the ability of machine learning models to transfer commonsense reasoning across languages. The datasets is a machine translated version of the MMLU dataset by GPT-3.5-turbo and covers 34 languages. This is a massive multitask test consisting of multiple-choice questions from various branches of knowledge. To attain high accuracy on this test, models must possess extensive world knowledge and problem solving ability. In our experimental setup, this benchmark covers English (En), Chinese (Zh), Arabic (Ar), Korean (Ko), and Swahili (Sw).

**Table 2: Detailed hyper-parameter settings for bilingual enhancement. The number under the column of Manipulated Layer represents the starting point of the layers where manipulation is applied, with the default endpoint being the final layer.**
|  | Manipulated Layer | $\lambda_{\text{Zh}}$ |
| --- | --- | --- |
| LLaMA-3-8B-Instruct | 31 | 1 |
| LLaMA-3.1-8B-Instruct | 30 | 0.05 |
| Phi-3.5-mini-Instruct | 27 | 0.3 |

**Table 3: Detailed hyper-parameter settings for multilingual enhancement. The number under the column of Manipulated Layer represents the starting point of the layers where manipulation is applied, with the default endpoint being the final layer.**
|  | Manipulated Layer | $\lambda_{\text{Zh}}$ | $\lambda_{\text{Jp}}$ | $\lambda_{\text{Ko}}$ | $\lambda_{\text{Ar}}$ | $\lambda_{\text{Bn}}$ | $\lambda_{\text{Sw}}$ |
| --- | --- | --- | --- | --- | --- | --- | --- |
| LLaMA-3-8B-Instruct | 29 | 1 | 0.6 | 1 | 0.5 | 0.2 | 0.2 |
| LLaMA-3.1-8B-Instruct | 30 | 0.01 | 0.01 | 0.03 | 0.01 | 0.01 | 0.01 |
| Phi-3.5-mini-Instruct | 29 | 0.2 | 0.2 | 0.2 | 0.2 | 0.2 | 0.2 |

For multilingual generation:

- •
MT-Bench (Zheng et al., ): The dataset is designed for open-ended generation to evaluate a model’s ability to follow multi-turn instructions. In our experimental setup, this benchmark covers English (En), Chinese (Zh), Arabic (Ar), Japanese (Jp), Korean (Ko), Swahili (Sw) and Bengali (Bn). We collect data in English(^10^1010[https://huggingface.co/datasets/HuggingFaceH4/mt_bench_prompts](https://huggingface.co/datasets/HuggingFaceH4/mt_bench_prompts)), Japanese(^11^1111[https://huggingface.co/datasets/shi3z/MTbenchJapanese](https://huggingface.co/datasets/shi3z/MTbenchJapanese)), Korean(^12^1212[https://huggingface.co/datasets/StudentLLM/Korean_MT-Bench_questions](https://huggingface.co/datasets/StudentLLM/Korean_MT-Bench_questions)), and Arabic(^13^1313[https://huggingface.co/spaces/QCRI/mt-bench-ar/tree/main/data/mt_bench_ar](https://huggingface.co/spaces/QCRI/mt-bench-ar/tree/main/data/mt_bench_ar)) from huggingface, and Chinese(^14^1414[https://github.com/HIT-SCIR/huozi](https://github.com/HIT-SCIR/huozi)) from github. In addition, we use GPT-4o to translate the English data into Swahili and Bengali, and performed manual proofreading to ensure correctness.

## Appendix D Implementation Details

Our experiments are implemented with PyTorch (Paszke et al., ) and Transformer library (Wolf et al., ) on a single NVIDIA A800-SXM4-80GB GPU. The training duration is set to one epoch with the learning rate of 1e-5, cosine learning rate scheduler with warm up ratio of 0.05 and batch size of 8 across all backbones. And all backbones are trained with their official chat template with $\lambda_{1}=1$ and $\lambda_{3}=1$. The hyper-parameter $r$ specifying the dimension of language-specific subspace in language subspace probing stage is set to $L-1$, where $L$ is the total number of languages participated in this process. We use GlotLID (Kargaran et al., ) to identify the response language to obtain the language fidelity. GlotLID is an open-source language identification model that supports more than 1,600 languages. GlotLID returns iso_636_9 language codes, which we manually map to the language codes in this work.

More detailed hyper-parameter settings for bilingual and multilingual enhancement across different backbones are listed in Table 2 and Table 3, respectively.

Further, we carefully evaluate the official implementations of all baselines, in order to make the comparison as fair as possible. We strictly follow the hyper-parameter settings in their original code. If this could not reach the expected performance, we carry out the hyper-parameter search of the learning rate and batchsize.

**Table 4: Detailed results on the multilingual understanding and multilingual generation benchmarks with Phi-3.5-mini-Instruct backbone under the bilingual setting (English and Chinese). Accuracy serves as the evaluation metric for multilingual understanding, while GPT-4o ratings (on a scale of 1 to 10) are provided for MT-Bench. The values in parentheses represent language fidelity. Results highlighted in green indicate an improvement or performance comparable to the original backbone, while those highlighted in red signal a decline in performance relative to the original backbone.**
|  | Multilingual Understanding | Multilingual Generation |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | XCOPA | XWinograd | XStoryCloze | M-MMLU | AVG. | MT-Bench |  |  |  |  |  |  |
|  | En | Zh | En | Zh | En | Zh | En | Zh | En | Zh | En | Zh |
| Phi-3.5 | - | 81.40 | 75.80 | 67.70 | 95.40 | 89.40 | 71.70 | 47.30 | 81.00 | 71.40 | 6.18 (100%) | 4.92 (90.50%) |
| xSFT | - | 80.80 | 77.20 | 69.64 | 95.40 | 89.40 | 71.70 | 46.80 | 81.43 | 71.66 | 5.29 (100%) | 3.31 (88.75%) |
| xSFT-Full | - | 80.40 | 73.10 | 65.67 | 95.20 | 88.20 | 71.90 | 44.70 | 80.07 | 69.74 | 5.25 (100%) | 3.84 (87.50%) |
| QAlign | - | 78.00 | 69.60 | 58.73 | 95.10 | 84.70 | 70.80 | 46.60 | 78.50 | 67.01 | 5.28 (100%) | 3.15 (88.75%) |
| SDRRL | - | 81.80 | 76.30 | 66.87 | 95.60 | 90.20 | 71.60 | 46.90 | 81.17 | 71.44 | 6.15 (100%) | 4.03 (90.00%) |
| Lens (Ours) | - | 81.60 | 75.80 | 67.66 | 95.40 | 89.40 | 71.70 | 47.40 | 80.97 | 71.51 | 6.44 (100%) | 5.16 (92.50%) |

**Table 5: Resource consumption of different multilingual enhancement methods under the bilingual enhancement setup. The backbone model is LLaMA-3-8B-Instruct.**
|  | Lens | xSFT | xSFT-Full | SDRRL | QAlign | V1 | V2 | V3 |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Training time | 2m08s | 5m33s | 192m35s | 11m30s | 12m03s | - | - | - |
| Trainable parameters rate | 5.43% | 100.00% | 100.00% | 100.00% | 100.00% | 13.08% | 13.08% | - |
| Instruction data | 0.8K | 0.8K | 111.5K | 4K | 0.8K | 5M | 5M | 5K |
| Pre-training data | - | - | - | - | - | 120G | - | - |

Figure: Figure 7: Results on the multilingual understanding benchmark with LLaMA-3-8B-Instruct backbone under the multilingual setting. We report the average performance of each language on the corresponding benchmarks.
Refer to caption: x8.png

Figure: Figure 8: Results on the multilingual understanding and generation benchmarks with LLaMA-3.1-8B-Instruct backbone under the multilingual setting.
Refer to caption: x9.png

Figure: Figure 9: Results on the multilingual understanding and generation benchmarks with Phi-3.5-mini-Instruct backbone under the multilingual setting.
Refer to caption: x10.png

## Appendix E Additional Experimental Results

We report the multilingual understanding performance of LLaMA-3-8B-Instruct in Figure 7.
Experimental results of the comparison between Lens and baseline methods on Phi-3.5-mini-Instruct under bilingual and multilingual setups are shown in Table 4 and Figure 9, respectively. And the multilingual enhancement results for LLaMA-3.1-8B-Instruct are displayed in Figure 8.

The results demonstrate that our Lens is still capable of achieving the comprehensive multilingual enhancement. Similarly, Lens continues to improve the model’s multilingual generation capability, enhancing the quality of the model’s responses in specific languages. However, the improvement in language fidelity is more pronounced in the English-centric backbone than in the multilingual backbone, which the latter one undergoes more extensive multilingual alignment training. Notably, while the baseline method considerably decreases the language fidelity of the multilingual backbone, Lens has minimal impact on it. These extensive experimental results demonstrate that Lens can serve as an effective, efficient, and scalable multilingual enhancement solution. We hope that our method can provide inspiration for future work to seek multilingual supervision more from the LLM itself rather than heavily relying on external dataset.

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: How different data sizes affect Lens, and how much improvement can be achieved using more training data.
soundness: 3
strengths: -	This paper is innovative, offering a fresh perspective on enhancing the multilingual capabilities of large models. It not only provides new ideas for future multilingual research but also helps the Chinese NLP community better understand large models.
-	The proposed method verifies its effectiveness on multiple datasets including NLU and NLG and avoids problems such as catastrophic forgetting.
-	This paper is well written, and the figures and tables are well drawn, making it easy to understand.
summary: This paper presents a novel method, Lens, to enhance the multilingual capabilities of LLMs. Lens first explores the subspaces of Language-Specific and Language-Agnostic features, and introduces three training objectives—pull, push, and retain—to optimize the model's multilingual performance. Experiments across various understanding and generation tasks demonstrate that Lens effectively prevents catastrophic forgetting and significantly improves the performance of LLMs.
weaknesses: -	The experiments in the paper primarily compare Chinese and English, with additional languages including Japanese, which belongs to the same language family as Chinese, as well as low-resource languages like Bengali and Swahili, which are more distant from the representation space of English in LLMs. I am curious about the extent of improvement the method proposed in the paper can offer when the representation space of LLMs for languages within the same language family as English is closer to that of English, such as Es, Fr, and De.

-	This paper compares two SFT schemes with different data sizes, xSFT and xSFT-Full. But I still recommend that the authors compare SFT based on the LoRA version, as some experimental work [1] suggests that LoRA-based SFT can effectively prevent catastrophic forgetting, particularly when data quality is insufficient, such as when training data is sourced from automatic translation.
-	Some explanatory text needs to be added to explain the author's motivations and make it easier for readers to read. For examples, the function Span() in Equation (2)， the design of  Equation (3).
-	Typo: line 1009， “batchsize” -> “batch size”.

[1] MindMerger: Efficient Boosting LLM Reasoning in non-English Languages

---

confidence: 5
contribution: 3
initial_rating: 8
presentation: 4
questions: 1. Typical Cases Analysis:    How does the paper analyze typical cases for the language-agnostic and language-specific subspaces in Parts 4 and 5, and what implications do these analyses have for the performance of LENS in handling multilingual tasks?

2. LENS vs. xSFT-Full Performance:   In Figure 2, why does LENS not perform better than xSFT-Full for the Swahili language, and what factors might contribute to xSFT-Full's superior performance in this specific case?
soundness: 4
strengths: 1. Novelty of Approach: LENS introduces a novel perspective on multilingual enhancement by leveraging the internal language representation spaces of LLMs, offering a fresh approach compared to traditional data-driven post-training methods.

2. Efficiency and Effectiveness: LENS demonstrates high efficiency and effectiveness by achieving superior multilingual performance with significantly less computational resources, making it scalable and practical for large-scale applications.

3. Preservation of Central Language Abilities: A key strength of LENS is its ability to enhance multilingual capabilities without sacrificing the model's original central language performance, addressing the common issue of catastrophic forgetting.

4. Comprehensive Improvement: LENS shows a comprehensive improvement across various multilingual tasks, including both comprehension and generation, which is a significant advancement over methods that focus on only one aspect of language performance.

5. Transparency and Interpretability: LENS provides transparent and interpretable solutions for multilingual enhancements, allowing for better understanding and control over how language models process and represent different languages.
summary: This paper, LENS, is an innovative approach that enhances multilingual capabilities of large language models (LLMs) by manipulating their internal language representation spaces. It operates on the top layers of LLMs, aligning target languages with a central language in a shared semantic space while differentiating them in a language-specific space. This method significantly improves multilingual performance without compromising the original language abilities and does so with fewer computational resources compared to existing post-training techniques.
weaknesses: 1Typical Multilingual General or Unique Case Performance:
 The LENS approach, while effective in enhancing multilingual capabilities, may encounter challenges when dealing with languages that have unique grammatical structures or vocabularies significantly divergent from the central language. The method's reliance on internal representations might not fully capture the intricacies of such languages, potentially leading to suboptimal performance in tasks requiring deep linguistic understanding.
2 Alignment in Multilingual Tasks such as Machine Translation:
Although LENS improves multilingual performance by manipulating language representations, it might not fully address the complexities of tasks like machine translation, especially for low-resource languages. The scarcity of high-quality parallel corpora for these languages could hinder the model's ability to learn the fine-grained linguistic nuances necessary for accurate and fluent translations.

---

confidence: 3
contribution: 3
initial_rating: 5
presentation: 3
questions: 1. **Lack of explanation** Consider adding an explanation for what the bold and underlined text indicates in the caption of Table 1.

2. **Caption Error** The caption for Figure3 (Line408) contains an error, please correct it.
soundness: 2
strengths: 1. **Resource-efficient** Compared with previous resource-intensive methods like MSFT and continual pretraining, the proposed method enhances multilingual capabilities efficiently with fewer data resources and computation costs.

2. **Competitive Performance** This method demonstrates comparable performance with open-source LLMs that conduct large-scale post-training to enhance multilingual capabilities. Moreover, it surpasses current strong baselines in multilingual enhancement by a large margin.

3. **Good Interpretability** Inspired by previous findings on LLM interpretability, the authors manipulate the internal language representations in the top layers of LLMs, applying these findings to multilingual enhancement successfully. The results of the visualization analysis underscore the interpretability advantages of the proposed method.
summary: This paper introduces a method to enhance the multilingual capabilities of LLMs by leveraging the central-language internal language representation as pivot signal. Specifically, the authors decouple the internal language representation spaces into language-agnostic and language-specific subspaces. In the language-agnostic subspace, they pull the target language representations closer to those of English to inherit its capabilities, while in the language-specific subspace, they push the target language representations away from English to ensure distinct expression.
weaknesses: 1. **Missing Reference** Previous work [1] has explored how to enhance multilingual abilities through aligning internal sentence representations, but there is a lack of detailed introduction to this relevant research.

2. **The results of the ablation study do not fully support the authors' claims** The authors claim that target languages inherit capabilities from English by pull the target language representations closer to those of English. However, the left part of the figure 3 demonstrate that the performance improvement does not primarily stem from this component. There is only a slight performance variance, even when the hyperparameter is set to zero.

3. **Incorrect Color in Table1** The performance of SDRRL on XCOPA outperforms original backbone. However, the authors highlight it with red color. Additionally, I believe that comparable performance should not be indicated in green. Some results that are clearly lower than the original backbone is still marked in green, which could lead to misunderstandings about performance for readers.






[1] Improving In-context Learning of Multilingual Generative Language Models with Cross-lingual Alignment (https://aclanthology.org/2024.naacl-long.445) (Li et al., NAACL 2024)

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. Since the method doesn't leverage external data to augment knowledge, wouldn't languages closer to English theoretically benefit more? It raises the question: does the "alignment" operation inherently favor languages that are closer to or in the same family as English?
2. Why do the results in Figure 2 show greater improvement for Chinese and Japanese, but limited improvement for Arabic and Bengali?
soundness: 3
strengths: 1. The methodology is straightforward, with clear explanation and rationale.
2. Comparative results validate the effectiveness of the proposed approach.
summary: This paper proposes a novel method for enhancing multilingual performance in large language models by refining the model's text representation space. The approach applies singular value decomposition to analyze and separate cross-language similarities and differences within the representation space. During model training, this method aims to minimize the "language-agnostic" subspace and separate out the "language-specific" representation space, enabling better knowledge sharing with English (the central language) and improving the unique characteristics of each language. Experimental results indicate that this method effectively enhances multilingual capability, especially improving language fidelity.
weaknesses: 1. The technique primarily builds on existing methods (such as https://aclanthology.org/2023.findings-emnlp.190/) and lacks significant novelty.
2. Language selection for experiments could have accounted for the proximity of each language to the central language, which would add meaningful insights.
3. Performance inconsistencies are observed, such as the inferior results for the Phi dataset compared to llama's performance in the appendix.

Meta-review: This work proposes LENS (multiLingual Enhancement method based on the hidden
represeNtations within language Space of LLMs), as a new method to improve the multilingual perfromance of LLMs by modifying their internal representation spaces. Specifically, LENS consists of two steps; the first is language subspace probing, where the representations at each layer are separated into language-specific and language-agnosic subspaces using SVD. Then, in the second step, they perform language subspace manipulation, where the representations in the language-agnostic subspace are aligned and separated in the language-specific one. The experiments with 3 open-source LLMs show that LENS reduces catastrophic forgetting of the central, or source, language and improves performance on the target languages.

Strengths:
- The paper builds on prior work on aligning subspaces in LLMs by also differentiating subspaces for language-specific portions rather than just aligning them (U88a, CKFG).
- The experiments suggest that this method provides improvements on a variety of tasks (U88a, CKFG, 1RPm, yppZ). The method also reduces some drawbacks to traditional model finetuning for specialization, including mitigating catastrophic forgetting (U88a, CKFG).
- The method is also more efficient than prior work in the area, particularly compared to model-training-based approaches (CKFG, 1RPm), and is more interpretable (CKFG, 1RPm).
- The paper is well-written and easy to understand (U88a, yppZ).

The authors also addressed many of the reviewers' concerns with their response and new experiments that were added to the paper.

Weaknesses: 
- The choice of languages for adaptation is somewhat random and limited, which makes it hard to understand how this method will generalize across different choices of central and target languages (U88a, yppZ). The authors add some additional experiments with languages more similar to English (the central language) in the rebuttal. Still, there is little discussion or analysis of how different types of relatedness to the central language (typology, script, vocabulary overlap, etc.) affect downstream performance.
- The proposed method is quite similar to prior work (but include the addition of separating the language-agnostic spaces further). While LENS shows fairly consistent gains over these prior method baselines, the improvements are quite small and not significance-tested. (yppZ)

Decision: Reject