When Alignment Hurts: Decoupling Representational Spaces in Multilingual Models

## Abstract

Abstract Alignment with high-resource standard languages is often assumed to aid the modeling of related low-resource varieties. We challenge this assumption by demonstrating that excessive representational entanglement with a dominant variety, such as Modern Standard Arabic (MSA) in relation to Arabic dialects, can actively hinder generative modeling. We present the first comprehensive causal study of this phenomenon by analyzing and directly intervening in the internal representation geometry of large language models (LLMs). Our key contribution is an online variational probing framework that continuously estimates the subspace of the standard variety during fine-tuning, enabling projection-based decoupling from this space. While our study uses Arabic as a case due to its unusually rich parallel resources across 25 dialects, the broader motivation is methodological: dialectal MT serves as a controlled proxy for generative tasks where comparable multi-variety corpora are unavailable. Across 25 dialects, our intervention improves generation quality by up to +4.9 chrF++ and +2.0 on average compared to standard fine-tuning, despite a measured tradeoff in standard-language performance. These results provide causal evidence that subspace dominance by high-resource varieties can restrict generative capacity for related varieties. More generally, we unify geometric and information-theoretic probing with subspace-level causal interventions, offering practical tools for improving generative modeling in closely related language families and, more broadly, for controlling representational allocation in multilingual and multi-domain LLMs. Code will be released.

## 1 Introduction

Large Language Models (LLMs) have achieved remarkable progress in multilingual Natural Language Understanding (NLU) and Generation (NLG) tasks (Brown et al., 2020; Chowdhery et al., 2022; Scao et al., 2022; Aryabumi et al., 2024). Beyond English, these models show strong cross-lingual transfer, enabling low-resource varieties to benefit from related high-resource languages (Hu et al., 2020; Conneau et al., 2020; Xue et al., 2021).

A less understood question, however, is whether closer alignment with a dominant, high-resource variety always benefits related low-resource ones. Dialects provide a natural test case: they are linguistically distinct, socially important, yet often heavily entangled with their standardized counterpart in both data and models. Arabic exemplifies this dynamic, where Modern Standard Arabic (MSA) dominates pretraining resources while dozens of dialects remain underrepresented and underperform on benchmarks (Kantharuban et al., 2023). Similar dynamics arise in other orthographically and lexically close pairs such as Czech–Slovak or Swedish–Icelandic. Understanding representational interactions in such settings is crucial for inclusive generative modeling.

This paper challenges the assumption that alignment with a high-resource standard is always beneficial. Using Arabic dialects as a case study, chosen for their unusually rich parallel resources across 25 varieties (Bouamor et al., 2018), we show that excessive representational entanglement with MSA hinders generative performance. Since parallel corpora for other generative tasks across dialects are scarce, we focus on machine translation as a controlled proxy for dialect-sensitive generation.

Our study proceeds in two stages. First, we analyze how LLMs internally represent MSA and dialects, revealing that stronger generative performance correlates with greater representational separability from MSA. Second, we move from analysis to intervention: we introduce an online variational probing framework that continuously estimates the subspace of the high-resource standard during fine-tuning, enabling a novel subspace decoupling strategy. This causal intervention promotes orthogonal representations and improves generative capacity for dialects.

Table: Table 1: Sample of 5-way parallel sentences meaning ”How much does the breakfast cost?” in 5 different varieties of Arabic from the MADAR 26 corpus (Bouamor et al., 2018). The yellow highlights the interrogative element (roughly “how much”), the green (when present) highlights the explicit cost word, and the blue highlights the breakfast term.

Applied to 25 Arabic dialects, our approach yields consistent improvements over standard fine-tuning, up to +4.9 chrF++ on individual dialects and +2.0 on average, while trading off some performance in MSA generation. More broadly, our findings provide the first causal evidence that representational dominance by high-resource standards can limit generative modeling in closely related varieties.

Contributions.

- •
We present the first large-scale representational analysis of dialects in generative LLMs, unifying geometric and information-theoretic probing.
- •
We introduce a novel online probing-based subspace decoupling method that improves generative performance for underrepresented varieties.
- •
We empirically demonstrate consistent gains across 25 dialects, highlighting implications for related language families where orthographic and lexical similarity creates similar entanglement.

## 2 Related Works

This work investigates how LLMs internally allocate representational capacity across closely related language varieties, with a focus on Arabic dialects as a natural case study.

##### Multilingualism in Large Language Models.

Recent studies have analyzed how multilingual LLMs encode language-specific knowledge. For example, Wang et al. (2024) and Kojima et al. (2024) explore neuron sharing and language-specific activations, showing that subtle modifications can alter generation in particular languages. Our perspective differs: rather than focusing on neuron-level behavior, we ask whether dialects remain representationally distinct from their standardized counterpart and how this distinction (or entanglement) affects generative performance. This question is not limited to Arabic, but applies broadly to orthographically and lexically similar pairs with a resource imabalance.

At the representational level, Chang et al. (2022) show that languages occupy distinct subspaces in encoder-only models, while Shah et al. (2024) link geometric differences to cross-lingual transfer. We extend these insights to large generative models, showing that the degree of subspace separability between varieties correlates with downstream generation quality. Relatedly, Nigatu et al. (2023) find that models struggle to capture dialectal nuances; our results both confirm this for recent LLMs and provide causal evidence that mitigating representational entanglement improves performance.

##### Information-Theoretic Probing.

Information-theoretic probes have been used to study how linguistic signals emerge during pretraining (Voita & Titov, 2020; Müller-Eberstein et al., 2023). Building on this, we introduce probes not just for analysis but as part of training: our “dialect probes” continuously estimate the dominant standard-language subspace during fine-tuning, enabling us to directly intervene by penalizing entanglement. This extends probing from a diagnostic tool to a mechanism for causal representational control.

##### Dialectal and Low-Resource NLP.

Dialectal variation presents a persistent challenge for generative modeling. Prior work has documented large performance gaps as dialects deviate from their standardized counterpart (Kantharuban et al., 2023; Ziems et al., 2023). For Arabic, evaluation resources such as AraBench (Sajjad et al., 2020) and MADAR (Bouamor et al., 2018) have been developed, and recent studies examine MT and NLG across varieties (Kadaoui et al., 2023; Nagoudi et al., 2023). Our work departs from these by focusing not on resource creation or evaluation but on how dialects are internally represented in LLMs and how interventions on representational subspaces can improve generative capacity. While Arabic provides a uniquely rich testbed, the implications extend to other under-resourced language varieties that share high orthographic and lexical overlap with a dominant variety.

## 3 Background: Arabic Dialects

A significant challenge in developing truly multilingual models lies in handling closely-related language varieties, which often exist in a state of resource imbalance with a dominant, high-resource standard language. This scenario is not merely a linguistic curiosity but poses fundamental problems for model representation learning. We investigate this challenge through the lens of Arabic, which provides an ideal testbed due to its distinct diglossia. The language ecosystem consists of Modern Standard Arabic (MSA), a high-resource variety used in formal contexts, and numerous low-resource Dialectal Arabic (DA) varieties that are the primary spoken languages but lack substantial textual corpora. The availability of the MADAR corpus (Bouamor et al., 2018), a unique resource with parallel sentences across 25 dialects, enables a controlled study of this phenomenon, which is not feasible for many other language families with similar dialectal diversity.

Figure: Figure 1: Hierarchical clustering of Arabic varieties based on character-level distance (chrF++). Dialects cluster geographically and exhibit significant separation from the high-resource Modern Standard Arabic (MSA), quantifying their surface-level dissimilarity.
Refer to caption: x1.png

The divergence between MSA and DA is substantial, spanning lexical, syntactic, and morphological levels, as illustrated by the parallel translations in Table 1. To quantify this, we perform hierarchical clustering based on character-level similarity (chrF++ score, (Popović, 2015)) in Figure [1](https://arxiv.org/html/2508.12803v1#S3.F1). The analysis reveals that dialects form distinct, geographically-correlated clusters that are representationally distant from MSA. This dissimilarity presents a concrete technical obstacle, starting at the tokenization layer. As detailed in Appendix [A](https://arxiv.org/html/2508.12803v1#A1), standard model tokenizers trained predominantly on high-resource data yield suboptimal segmentations for dialectal words. This inefficiency leads to higher computational costs and hinders the model’s ability to learn long-range dependencies, unfairly disadvantaging low-resource varieties before any deeper processing occurs (Ali et al., 2024).

Our goal is to understand how the representational dominance of a high-resource language like MSA affects a model’s ability to generate text in closely-related, low-resource dialects. Due to the scarcity of parallel data for diverse generative tasks, we utilize machine translation (MT) from MSA to the 25 DA varieties as a controlled testbed for generation. This task allows us to probe the model’s capacity to produce dialect-specific outputs while controlling for semantic content. While our empirical study is grounded in Arabic, the core challenge is generalizable. The insights derived are relevant to other language families with similar orthographic overlap and resource asymmetry, such as Czech and Slovak, or the spectrum of Scandinavian languages, where models must learn to navigate the subtle yet critical distinctions between high- and low-resource variants.

## 4 Methodology

We present a methodology designed to first diagnose and then causally intervene in the representational geometry of multilingual models. Our approach uses a controlled generative task to probe model capabilities, analyzes the underlying representations through geometric and information-theoretic lenses, and introduces a novel training technique to mitigate representational entanglement.

### 4.1 Task Formulation: Dialectal Rewriting as a Generative Testbed

To create a controlled environment for studying dialectal generation, we formulate a task of Dialectal Machine Translation (DiaMT). Given a sentence in the high-resource standard variety (MSA), the model’s objective is to generate the semantically equivalent sentence in a target low-resource dialect. This task serves as a valuable proxy for more general conditional generation, allowing us to precisely measure a model’s ability to manipulate linguistic style while preserving meaning. This controlled setup is necessitated by the lack of comprehensive parallel corpora for other generative tasks (e.g., summarization, open-ended dialogue) across the 25 dialects. We employ zero-shot inference using a simple instructional prompt, as shown below.

Figure: Figure 2: Average character-level similarity (chrF++) between dialectal sentences and their MSA counterparts, highlighting the high surface overlap that makes this a challenging generation task.
Refer to caption: x2.png

For our causal experiments (Sec. [4.4](https://arxiv.org/html/2508.12803v1#S4.SS4)), we fine-tune models on a bidirectional rewriting objective (MSA $\leftrightarrow$ dialect). This prevents the model from simply degrading its high-resource MSA representations to favor dialects, a potential confounding factor in unidirectional training. By preserving MSA capabilities, we ensure a fairer assessment of subspace dynamics and the true impact of our intervention on dialectal generation.

### 4.2 Quantifying Performance and Representational Geometry

##### Evaluation.

We quantify generation quality using chrF++ (Popović, 2015), a character n-gram F-score. Its character-level nature is well-suited for the morphological richness of Arabic and is robust to minor lexical variations common between dialects. We acknowledge that, like many automated metrics, chrF++ cannot fully capture the nuances of ”dialectness,” especially given the high lexical overlap between Arabic varieties (Fig. [2](https://arxiv.org/html/2508.12803v1#S4.F2.fig1)). However, in the absence of better-suited metrics for this specific cross-dialectal evaluation, it serves as a reliable indicator of generative accuracy.

##### Representational Geometry.

To understand how models represent dialects, we analyze their internal geometry. We measure the Geometric Separability between sentence representations using L2 and cosine distance, anchoring all comparisons to MSA representations. This allows us to quantify how distinct dialectal representations are from the high-resource standard. Furthermore, we compute Subspace Angles (SSA) (Müller-Eberstein et al., 2023) to measure the alignment between subspaces corresponding to different dialects. Smaller angles indicate greater alignment. This allows us to track how fine-tuning and our proposed interventions reshape the model’s internal organization of linguistic information.

### 4.3 Information-Theoretic Probing

To complement the geometric analysis, we employ an information-theoretic variational linear probe (Voita & Titov, 2020; Müller-Eberstein et al., 2023). The probe is a sparsity-regularized classifier trained to identify a dialect from token-level representations. The resulting negative cross-entropy provides a tight lower bound on the mutual information $I(\mathbf{h}^{(\ell)};Y)$ between a model’s hidden states and the dialect identity. This allows us to quantify how easily dialect-specific information can be linearly decoded from the model’s representations, layer by layer, and how this changes during training. Further details are in Appendix [C](https://arxiv.org/html/2508.12803v1#A3).

### 4.4 Causal Intervention: Online Subspace Decoupling

To test the hypothesis that representational entanglement with a high-resource language harms low-resource generation, we introduce a novel training method: Online Subspace Decoupling. This method acts as a causal intervention by actively discouraging dialectal representations from overlapping with the MSA subspace during fine-tuning.

The procedure is as follows:

- 1.
Identify MSA Subspace: We train a variational linear probe (as in Sec. [4.3](https://arxiv.org/html/2508.12803v1#S4.SS3)) to distinguish MSA from all other dialects. We then use Singular Value Decomposition (SVD) on the learned probe weights to extract an orthonormal basis $\mathbf{U}_{\text{MSA}}$ for the MSA subspace and form its projection matrix: $\mathbf{P}_{\text{MSA}}=\mathbf{U}_{\text{MSA}}\mathbf{U}_{\text{MSA}}^{\top}$.
- 2.
Define Decoupling Loss: During fine-tuning on the dialectal rewriting task, we add a penalty term to the standard language modeling loss. This decoupling loss penalizes the magnitude of the projection of the model’s hidden states $\mathbf{H}$ onto the MSA subspace:
$\mathcal{L}_{\text{decouple}}=\mathbb{E}\left[\left\lVert\mathbf{H}\mathbf{P}_{\text{MSA}}\right\rVert_{2}\right]$
(1)
The total loss is $\mathcal{L}=\mathcal{L}_{\text{LM}}+\lambda\,\mathcal{L}_{\text{decouple}}$, where $\lambda$ is a hyperparameter (we use 0.01).

Crucially, the probe is periodically retrained on fresh model checkpoints during fine-tuning. This online updating of $\mathbf{P}_{\text{MSA}}$ ensures that our intervention targets the evolving MSA subspace, enabling a precise and adaptive causal manipulation of the model’s representational geometry. Training details are in Appendix [D](https://arxiv.org/html/2508.12803v1#A4).

### 4.5 Experimental Setup

##### Data.

All experiments use the MADAR 25 corpus (Bouamor et al., 2018), which contains 2,000 parallel sentences across 25 city-level Arabic dialects, MSA, English, and French. This fine-grained, multi-dialect parallel resource is unique and enables our controlled study.

##### Models.

We analyze a suite of state-of-the-art open-weight multilingual models: Jais-family 30B (Sengupta et al., 2023), Gemma 3 1B (Team, 2025a), Aya expanse 8B (Dang et al., 2024), and Qwen 3 14B (Team, 2025b). For our causal intervention experiments, we deliberately select Gemma 3 1B. Its smaller parameter count implies a more constrained representational space, making it a challenging and informative test case for the benefits of explicit subspace management. Furthermore, its weaker baseline performance provides a clear opportunity to measure improvement from our method.

## 5 Results and Analysis

We now present our empirical investigation, which first diagnoses the representational pathologies hindering dialectal generation in multilingual models and then validates our hypothesis with a causal intervention.

### 5.1 Baseline: Models Exhibit Poor Low-Resource Generative Capabilities

We first benchmark the zero-shot performance of several models on our dialectal rewriting task. As shown in Figure [3](https://arxiv.org/html/2508.12803v1#S5.F3), all models struggle significantly, with even the best-performing models, Qwen 3 14B and Aya Expanse 8B, achieving only modest chrF++ scores. Notably, performance does not correlate with model scale or specialization; the largest, Arabic-centric Jais-family 30B model is outperformed by smaller models. This poor performance, especially when contrasted with the models’ strong capabilities in high-resource language pairs (MSA to English/French), points to a more fundamental issue than a simple lack of capacity. We posit this stems from the model’s internal representations.

Figure: Figure 3: Generative performance on the dialectal rewriting task. All models perform poorly on low-resource dialects compared to high-resource languages, and performance does not correlate with model scale.
Refer to caption: x3.png

Figure: Figure 4: t-SNE of sentence representations. Higher-performing models (e.g., Qwen, Aya) exhibit clearer separation between dialectal clusters in their intermediate layers, unlike weaker models (Gemma, Jais).
Refer to caption: x4.png

### 5.2 Diagnosis I: Geometric Analysis Links Performance to Representational Separation

To investigate the underlying representational geometry, we visualize the hidden states of parallel sentences using t-SNE (Figure [4](https://arxiv.org/html/2508.12803v1#S5.F4)). The visualizations reveal a striking pattern: stronger models like Qwen and Aya learn to separate representations by dialect in their intermediate layers, whereas weaker models like Jais and Gemma maintain entangled representations. This qualitative observation suggests a link between a model’s ability to geometrically isolate dialectal subspaces and its downstream generative performance.

Figure: Figure 5: Layer-wise L2 (Top) and Cosine (Bottom) distance between dialectal representations and MSA. High-performing models show distinct geometric patterns, with Aya treating dialects more like separate languages (high L2 distance).
Refer to caption: x5.png

We quantify this by measuring the L2 and cosine distance between MSA and dialectal sentence representations across all layers (Figure [5](https://arxiv.org/html/2508.12803v1#S5.F5)). We observe that different distance metrics capture different geometric properties: L2 distance reflects the degree of spatial separation, while cosine distance measures the alignment of subspaces. To substantiate the link to performance, we compute the layer-wise correlation between these distances and the chrF++ score (Figure [6](https://arxiv.org/html/2508.12803v1#S5.F6)). A consistent negative correlation emerges between cosine distance and performance, especially in early-to-mid layers. This suggests that better alignment (lower cosine distance) in these layers is beneficial, likely facilitating the transfer of semantic information from the high-resource MSA. Conversely, the relationship with L2 distance is more complex, with models like Aya benefiting from greater spatial separation in intermediate layers. This indicates a delicate balance: subspaces must be aligned enough for knowledge transfer but separate enough to preserve unique dialectal features.

Figure: Figure 6: Layer-wise Pearson correlation between representational distance from MSA (L2-Top, Cosine-Bottom) and downstream generation performance. The consistent negative correlation with cosine distance suggests that subspace alignment is beneficial.
Refer to caption: x6.png

### 5.3 Diagnosis II: Information-Theoretic Evidence of MSA’s Representational Dominance

The geometric analysis suggests entanglement with MSA is problematic. We further interrogate this using information-theoretic probing during standard supervised fine-tuning (SFT) on the dialectal rewriting task. We track the ELBO code length required to identify dialects from the model’s hidden states (a proxy for how accessible this information is). As shown in Figure [7](https://arxiv.org/html/2508.12803v1#S5.F7), standard fine-tuning causes the code length for all dialects to increase slightly, as the model specializes for generation rather than classification. However, the increase is disproportionately large for MSA. This indicates that the model is actively making MSA-specific information less linearly accessible, suggesting its pre-trained MSA representation is oversized and detrimental to the dialectal generation task.

This “pruning” of the MSA subspace has a direct geometric consequence. As we fine-tune, the Subspace Angle (SSA) between MSA and the dialectal subspaces consistently increases (Figure [8](https://arxiv.org/html/2508.12803v1#S5.F8), left). That is, the dialectal subspaces systematically drift away from the MSA subspace. Crucially, this growing separation directly correlates with improvements in generation performance (Figure [8](https://arxiv.org/html/2508.12803v1#S5.F8), right).

Taken together, these analyses provide compelling correlational evidence for our central hypothesis: the representational dominance of the high-resource standard language (MSA) actively hinders a model’s ability to generate text in related low-resource varieties. Fine-tuning implicitly alleviates this by pushing dialectal representations away from the MSA subspace.

Figure: Figure 7: Code Length evolution over baseline training.
Refer to caption: x7.png

There are a few limitations to keep in mind of our analyses so far. Namely, that they have been observational and serve to build our hypothesis; the causal link is established specifically by the success of our decoupling intervention which we will discuss in Section [5.4](https://arxiv.org/html/2508.12803v1#S5.SS4). Finally, the MADAR dataset, while unique in its breadth of dialects, is composed of relatively short sentences. This setting may not fully capture model behaviors on tasks requiring longer-form generation, thereby defining the scope of our current findings. We hope future work addresses this gap in data availability.

### 5.4 Causal Validation: Online Subspace Decoupling Boosts Performance

To move from correlation to causation, we test our hypothesis directly using our proposed Online Subspace Decoupling method (Section [4.4](https://arxiv.org/html/2508.12803v1#S4.SS4)). By adding an explicit penalty term that pushes dialectal hidden states out of the MSA subspace, we actively enforce the representational separation that SFT appears to learn implicitly.

Figure: Figure 8: (Left) During SFT, the subspace angle (SSA) between MSA and dialects consistently increases, indicating growing representational separation. (Right) This increase in separation correlates directly with improved chrF++ scores. This provides strong evidence that disentangling from MSA is a key mechanism for improving dialectal generation.
Refer to caption: x8.png

Figure: Figure 9: Our causal intervention (Online Decoupling, Orange) consistently improves performance over baseline SFT (Blue) by actively enforcing representational separation.
Refer to caption: x9.png

The results, shown in Figure [9](https://arxiv.org/html/2508.12803v1#S5.F9), provide clear causal validation. Our intervention yields consistent and significant performance gains across nearly all dialects, achieving a +2.0 chrF++ average improvement and gains as high as +4.89 for specific dialects (Cairo) over a standard SFT baseline. The dialects that benefit least (e.g., Tunis, Beirut) were already those with the highest initial separation from MSA (Figure [8](https://arxiv.org/html/2508.12803v1#S5.F8)), suggesting they were less affected by the entanglement problem. While there is an expected trade-off, performance on MSA generation drops, the substantial boost for a wide range of low-resource varieties confirms that mitigating representational dominance is a potent mechanism for improving generative capabilities. Visualizations (tSNE) in Appendix [E](https://arxiv.org/html/2508.12803v1#A5) confirm that our method achieves a much greater degree of geometric separation than baseline SFT, directly linking the intervention to its intended structural effect on the model’s internal representations.

Our analysis, while providing strong causal evidence, has several limitations that frame opportunities for future research. Our causal claim rests on intervention experiments within a single, albeit complex, language family: Arabic. While we hypothesize that the underlying mechanism of representational dominance is a general phenomenon, empirical validation on other language families is necessary to confirm this.

## 6 Conclusion & Future Work

This work demonstrates that representational entanglement with a high-resource language is a critical and addressable bottleneck for generative modeling in closely-related, low-resource language varieties. Through a combination of geometric and information-theoretic analyses on Arabic dialects, we provided evidence that the representational dominance of Modern Standard Arabic (MSA) hinders dialectal generation. We then moved from correlation to causation, introducing a novel online subspace decoupling method that actively and dynamically separates dialectal representations from the MSA subspace during fine-tuning.

Our experiments provide the first causal evidence that explicitly managing this subspace overlap yields substantial performance gains, up to +4.9 chrF++ on individual dialects and +2.0 on average, validating our hypothesis. While our method was designed for hypothesis testing and is computationally intensive, its success illuminates a clear path forward. The results highlight the critical importance of representational allocation in multilingual models and motivate future work in several key directions:

- •
Scalable and Efficient Methods: Future research should focus on developing computationally cheaper alternatives that achieve similar decoupling effects. This includes designing parameter-efficient fine-tuning (PEFT) methods, such as subspace-aware adapters, or formulating novel pre-training objectives that encourage a more balanced representational space from the outset.
- •
Inference-Time Interventions: A particularly promising avenue is to move beyond training-based solutions. Inference-time techniques like activation steering or targeted model editing could offer a more surgical and efficient approach. For instance, identifying and patching neurons responsible for MSA-specific features could suppress the dominant language’s influence on-the-fly, without requiring any gradient-based updates.
- •
Generalization to Other Languages: A crucial next step is to investigate whether these principles of representational entanglement generalize beyond Arabic. Applying our analytical framework and interventions to other language families with similar resource imbalances and orthographic overlap, such as the Czech-Slovak or Scandinavian language continuums, is essential for establishing the broader utility of subspace management in multilingual representation learning.

#### Acknowledgments

This work was conducted during Ahmed Elshabrawy’s research internship at NICT, Japan. We gratefully acknowledge the support and computational resources provided by NICT, Japan that made this research possible.

## References

- Ali et al. (2024)
Mehdi Ali, Michael Fromm, Klaudia Thellmann, Richard Rutmann, Max Lübbering, Johannes Leveling, Katrin Klug, Jan Ebert, Niclas Doll, Jasper Buschhoff, Charvi Jain, Alexander Weber, Lena Jurkschat, Hammam Abdelwahab, Chelsea John, Pedro Ortiz Suarez, Malte Ostendorff, Samuel Weinbach, Rafet Sifa, Stefan Kesselheim, and Nicolas Flores-Herr.
Tokenizer choice for LLM training: Negligible or crucial?
In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), *Findings of the Association for Computational Linguistics: NAACL 2024*, pp.  3907–3924, Mexico City, Mexico, June 2024. Association for Computational Linguistics.
doi: 10.18653/v1/2024.findings-naacl.247.
URL [https://aclanthology.org/2024.findings-naacl.247/](https://aclanthology.org/2024.findings-naacl.247/).
- Aryabumi et al. (2024)
Viraat Aryabumi, John Dang, Dwarak Talupuru, Saurabh Dash, David Cairuz, Hangyu Lin, Bharat Venkitesh, Madeline Smith, Jon Ander Campos, Yi Chern Tan, Kelly Marchisio, Max Bartolo, Sebastian Ruder, Acyr Locatelli, Julia Kreutzer, Nick Frosst, Aidan Gomez, Phil Blunsom, Marzieh Fadaee, Ahmet Üstün, and Sara Hooker.
Aya 23: Open weight releases to further multilingual progress, 2024.
URL [https://arxiv.org/abs/2405.15032](https://arxiv.org/abs/2405.15032).
- Bouamor et al. (2018)
Houda Bouamor, Nizar Habash, Mohammad Salameh, Wajdi Zaghouani, Owen Rambow, Dana Abdulrahim, Ossama Obeid, Salam Khalifa, Fadhl Eryani, Alexander Erdmann, and Kemal Oflazer.
The MADAR Arabic dialect corpus and lexicon.
In Nicoletta Calzolari, Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi, Koiti Hasida, Hitoshi Isahara, Bente Maegaard, Joseph Mariani, Hélène Mazo, Asuncion Moreno, Jan Odijk, Stelios Piperidis, and Takenobu Tokunaga (eds.), *Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)*, Miyazaki, Japan, May 2018. European Language Resources Association (ELRA).
URL [https://aclanthology.org/L18-1535](https://aclanthology.org/L18-1535).
- Brown et al. (2020)
Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.
Language models are few-shot learners.
*Advances in neural information processing systems*, 33:1877–1901, 2020.
- Chang et al. (2022)
Tyler Chang, Zhuowen Tu, and Benjamin Bergen.
The geometry of multilingual language model representations.
In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), *Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing*, pp.  119–136, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics.
doi: 10.18653/v1/2022.emnlp-main.9.
URL [https://aclanthology.org/2022.emnlp-main.9](https://aclanthology.org/2022.emnlp-main.9).
- Chowdhery et al. (2022)
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al.
Palm: Scaling language modeling with pathways.
*arXiv preprint arXiv:2204.02311*, 2022.
- Conneau et al. (2020)
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov.
Unsupervised cross-lingual representation learning at scale.
*arXiv preprint arXiv:1911.02116*, 2020.
- Dang et al. (2024)
John Dang, Shivalika Singh, Daniel D’souza, Arash Ahmadian, Alejandro Salamanca, Madeline Smith, Aidan Peppin, Sungjin Hong, Manoj Govindassamy, Terrence Zhao, Sandra Kublik, Meor Amer, Viraat Aryabumi, Jon Ander Campos, Yi-Chern Tan, Tom Kocmi, Florian Strub, Nathan Grinsztajn, Yannis Flet-Berliac, Acyr Locatelli, Hangyu Lin, Dwarak Talupuru, Bharat Venkitesh, David Cairuz, Bowen Yang, Tim Chung, Wei-Yin Ko, Sylvie Shang Shi, Amir Shukayev, Sammie Bae, Aleksandra Piktus, Roman Castagné, Felipe Cruz-Salinas, Eddie Kim, Lucas Crawhall-Stein, Adrien Morisot, Sudip Roy, Phil Blunsom, Ivan Zhang, Aidan Gomez, Nick Frosst, Marzieh Fadaee, Beyza Ermis, Ahmet Üstün, and Sara Hooker.
Aya expanse: Combining research breakthroughs for a new multilingual frontier, 2024.
URL [https://arxiv.org/abs/2412.04261](https://arxiv.org/abs/2412.04261).
- Hu et al. (2020)
Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson.
Xtreme: A massively multilingual multi-task benchmark for evaluating cross-lingual generalization.
*arXiv preprint arXiv:2003.11080*, 2020.
- Kadaoui et al. (2023)
Karima Kadaoui, Samar Magdy, Abdul Waheed, Md Tawkat Islam Khondaker, Ahmed El-Shangiti, El Moatez Billah Nagoudi, and Muhammad Abdul-Mageed.
TARJAMAT: Evaluation of bard and ChatGPT on machine translation of ten Arabic varieties.
In Hassan Sawaf, Samhaa El-Beltagy, Wajdi Zaghouani, Walid Magdy, Ahmed Abdelali, Nadi Tomeh, Ibrahim Abu Farha, Nizar Habash, Salam Khalifa, Amr Keleg, Hatem Haddad, Imed Zitouni, Khalil Mrini, and Rawan Almatham (eds.), *Proceedings of ArabicNLP 2023*, pp.  52–75, Singapore (Hybrid), December 2023. Association for Computational Linguistics.
doi: 10.18653/v1/2023.arabicnlp-1.6.
URL [https://aclanthology.org/2023.arabicnlp-1.6](https://aclanthology.org/2023.arabicnlp-1.6).
- Kantharuban et al. (2023)
Anjali Kantharuban, Ivan Vulić, and Anna Korhonen.
Quantifying the dialect gap and its correlates across languages.
In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), *Findings of the Association for Computational Linguistics: EMNLP 2023*, pp.  7226–7245, Singapore, December 2023. Association for Computational Linguistics.
doi: 10.18653/v1/2023.findings-emnlp.481.
URL [https://aclanthology.org/2023.findings-emnlp.481](https://aclanthology.org/2023.findings-emnlp.481).
- Kojima et al. (2024)
Takeshi Kojima, Itsuki Okimura, Yusuke Iwasawa, Hitomi Yanaka, and Yutaka Matsuo.
On the multilingual ability of decoder-based pre-trained language models: Finding and controlling language-specific neurons.
In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), *Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)*, pp.  6919–6971, Mexico City, Mexico, June 2024. Association for Computational Linguistics.
doi: 10.18653/v1/2024.naacl-long.384.
URL [https://aclanthology.org/2024.naacl-long.384](https://aclanthology.org/2024.naacl-long.384).
- Müller-Eberstein et al. (2023)
Max Müller-Eberstein, Rob van der Goot, Barbara Plank, and Ivan Titov.
Subspace chronicles: How linguistic information emerges, shifts and interacts during language model training.
In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), *Findings of the Association for Computational Linguistics: EMNLP 2023*, pp.  13190–13208, Singapore, December 2023. Association for Computational Linguistics.
doi: 10.18653/v1/2023.findings-emnlp.879.
URL [https://aclanthology.org/2023.findings-emnlp.879/](https://aclanthology.org/2023.findings-emnlp.879/).
- Nagoudi et al. (2023)
El Moatez Billah Nagoudi, AbdelRahim Elmadany, Ahmed El-Shangiti, and Muhammad Abdul-Mageed.
Dolphin: A challenging and diverse benchmark for Arabic NLG.
In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), *Findings of the Association for Computational Linguistics: EMNLP 2023*, pp.  1404–1422, Singapore, December 2023. Association for Computational Linguistics.
doi: 10.18653/v1/2023.findings-emnlp.98.
URL [https://aclanthology.org/2023.findings-emnlp.98](https://aclanthology.org/2023.findings-emnlp.98).
- Nigatu et al. (2023)
Hellina Nigatu, Atnafu Tonja, and Jugal Kalita.
The less the merrier? investigating language representation in multilingual models.
In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), *Findings of the Association for Computational Linguistics: EMNLP 2023*, pp.  12572–12589, Singapore, December 2023. Association for Computational Linguistics.
doi: 10.18653/v1/2023.findings-emnlp.837.
URL [https://aclanthology.org/2023.findings-emnlp.837](https://aclanthology.org/2023.findings-emnlp.837).
- Popović (2015)
Maja Popović.
chrF: character n-gram F-score for automatic MT evaluation.
In Ondřej Bojar, Rajan Chatterjee, Christian Federmann, Barry Haddow, Chris Hokamp, Matthias Huck, Varvara Logacheva, and Pavel Pecina (eds.), *Proceedings of the Tenth Workshop on Statistical Machine Translation*, pp.  392–395, Lisbon, Portugal, September 2015. Association for Computational Linguistics.
doi: 10.18653/v1/W15-3049.
URL [https://aclanthology.org/W15-3049](https://aclanthology.org/W15-3049).
- Sajjad et al. (2020)
Hassan Sajjad, Ahmed Abdelali, Nadir Durrani, and Fahim Dalvi.
AraBench: Benchmarking dialectal Arabic-English machine translation.
In Donia Scott, Nuria Bel, and Chengqing Zong (eds.), *Proceedings of the 28th International Conference on Computational Linguistics*, pp.  5094–5107, Barcelona, Spain (Online), December 2020. International Committee on Computational Linguistics.
doi: 10.18653/v1/2020.coling-main.447.
URL [https://aclanthology.org/2020.coling-main.447](https://aclanthology.org/2020.coling-main.447).
- Scao et al. (2022)
Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Luccioni, François Yvon, Matthias Gallé, et al.
Bloom: A 176b-parameter open-access multilingual language model.
*arXiv preprint arXiv:2211.05100*, 2022.
- Sengupta et al. (2023)
Neha Sengupta, Sunil Kumar Sahu, Bokang Jia, Satheesh Katipomu, Haonan Li, Fajri Koto, William Marshall, Gurpreet Gosal, Cynthia Liu, Zhiming Chen, Osama Mohammed Afzal, Samta Kamboj, Onkar Pandit, Rahul Pal, Lalit Pradhan, Zain Muhammad Mujahid, Massa Baali, Xudong Han, Sondos Mahmoud Bsharat, Alham Fikri Aji, Zhiqiang Shen, Zhengzhong Liu, Natalia Vassilieva, Joel Hestness, Andy Hock, Andrew Feldman, Jonathan Lee, Andrew Jackson, Hector Xuguang Ren, Preslav Nakov, Timothy Baldwin, and Eric Xing.
Jais and jais-chat: Arabic-centric foundation and instruction-tuned open generative large language models, 2023.
- Shah et al. (2024)
Cheril Shah, Yashashree Chandak, Atharv Mahesh Mane, Benjamin Bergen, and Tyler A. Chang.
Correlations between multilingual language model geometry and crosslingual transfer performance.
In Nicoletta Calzolari, Min-Yen Kan, Veronique Hoste, Alessandro Lenci, Sakriani Sakti, and Nianwen Xue (eds.), *Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)*, pp.  4059–4066, Torino, Italia, May 2024. ELRA and ICCL.
URL [https://aclanthology.org/2024.lrec-main.361](https://aclanthology.org/2024.lrec-main.361).
- Team (2025a)
Gemma Team.
Gemma 3.
2025a.
URL [https://goo.gle/Gemma3Report](https://goo.gle/Gemma3Report).
- Team (2025b)
Qwen Team.
Qwen3 technical report, 2025b.
URL [https://arxiv.org/abs/2505.09388](https://arxiv.org/abs/2505.09388).
- Voita & Titov (2020)
Elena Voita and Ivan Titov.
Information-theoretic probing with minimum description length.
In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), *Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)*, pp.  183–196, Online, November 2020. Association for Computational Linguistics.
doi: 10.18653/v1/2020.emnlp-main.14.
URL [https://aclanthology.org/2020.emnlp-main.14/](https://aclanthology.org/2020.emnlp-main.14/).
- Wang et al. (2024)
Weixuan Wang, Barry Haddow, Minghao Wu, Wei Peng, and Alexandra Birch.
Sharing matters: Analysing neurons across languages and tasks in llms, 2024.
URL [https://arxiv.org/abs/2406.09265](https://arxiv.org/abs/2406.09265).
- Wu et al. (2016)
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Lukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Greg Corrado, Macduff Hughes, and Jeffrey Dean.
Google’s neural machine translation system: Bridging the gap between human and machine translation.
*CoRR*, abs/1609.08144, 2016.
URL [http://arxiv.org/abs/1609.08144](http://arxiv.org/abs/1609.08144).
- Xue et al. (2021)
Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel.
mT5: A massively multilingual pre-trained text-to-text transformer.
In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tur, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou (eds.), *Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies*, pp.  483–498, Online, June 2021. Association for Computational Linguistics.
doi: 10.18653/v1/2021.naacl-main.41.
URL [https://aclanthology.org/2021.naacl-main.41](https://aclanthology.org/2021.naacl-main.41).
- Ziems et al. (2023)
Caleb Ziems, William Held, Jingfeng Yang, Jwala Dhamala, Rahul Gupta, and Diyi Yang.
Multi-VALUE: A framework for cross-dialectal English NLP.
In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), *Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, pp.  744–768, Toronto, Canada, July 2023. Association for Computational Linguistics.
doi: 10.18653/v1/2023.acl-long.44.
URL [https://aclanthology.org/2023.acl-long.44](https://aclanthology.org/2023.acl-long.44).

## Appendix A Tokenizer Fertilities

To check how dialects are handled differently to MSA at the most basic level by models we examine various tokenizer fertilities. We include tokenizers of all the model families we explore. Furthermore, we train custom WordPiece tokenizers with an 8,000 vocabulary size (Wu et al., 2016) on the train and dev set of the MADAR 25 corpus of 6 different dialects/varieties (each having 10,000 sentences in total).

As seen in Figure [10](https://arxiv.org/html/2508.12803v1#A1.F10), there are several interesting observations. First, it is clear that there is a divide between MSA, English, and French and the non-standard dialects of Arabic which have significantly higher fertilities.

Furthermore, Across all tokenizers, dialects tend to have much higher fertilities to MSA. Interestingly, even for tokenizers trained on a single dialect, the fertility of MSA tends to be close or even lower than the dialect it was trained on. This seems to indicate that MSA serves a lowest common denominator for Arabic tokenizers whereas dialects can have more foreign/disjoint vocabularies (i.e. most dialectal varieties will feature a minimum MSA lexicon with other elements which are not shared). This is consistent with the linguistic intuition and is supported by the unique word counts of the train sets of all the dialectal tokenizers show below:

**Table 2: Unique words**
| Dialect | #Unique Words |
| --- | --- |
| DOH | 12,651 |
| BEI | 15,083 |
| MSA | 12,197 |
| CAI | 14,611 |
| RAB | 16,136 |
| TUN | 15,437 |

Interestingly, we note however that training on a single dialect seems to lead to lower fertilites on all other dialects. Especially when comparing to the LLM and MSA-only tokenizers observed. This is indicative that a majority of the training data used to train the multilingual tokenizers of all models observed is lacking in dialectal data.

This highlights that before any processing is done by models dialectal words are inherently treated differntly at the lowest stage of tokenization. Furthermore, training tokenizers on MSA data only is not sufficient for tokenizers to capture dialectal words sufficiently.

Figure: Figure 10: Tokenizer fertilities of various language model families, as well as a WordPiece tokenizer trained from scratch on a specifc dialectal set. The red bar corresponds to MSA.
Refer to caption: x10.png

## Appendix B City Names to Dialect Code

**Table 3: City Names and Their Codes**
| City | Code |
| --- | --- |
| Rabat | RAB |
| Fes | FES |
| Algiers | ALG |
| Tunis | TUN |
| Sfax | SFX |
| Tripoli | TRI |
| Benghazi | BEN |
| Cairo | CAI |
| Alexandria | ALX |
| Aswan | ASW |
| Khartoum | KHA |
| Jerusalem | JER |
| Amman | AMM |
| Salt | SAL |
| Beirut | BEI |
| Damascus | DAM |
| Aleppo | ALE |
| Mosul | MOS |
| Baghdad | BAG |
| Basra | BAS |
| Doha | DOH |
| Muscat | MUS |
| Riyadh | RIY |
| Jeddah | JED |
| Sana’a | SAN |

## Appendix C More Information About Probing

To complement geometric subspace analysis, we adopt an information-theoretic variational linear probe (Voita & Titov, 2020; Müller-Eberstein et al., 2023) to quantify how much dialect identity information is recoverable from token-level model representations. For a given token, let $\{\mathbf{h}^{(0)},\dots,\mathbf{h}^{(\ell)}\}\in\mathbb{R}^{d}$ denote its hidden states from all $\ell$ layers, including the non-contextualized layer 0. The probe computes a learned weighted average over layers:

$$ $\mathbf{h}^{\prime}=\sum_{i=0}^{\ell}\alpha_{i}\mathbf{h}^{(i)},$ $$

where $\bm{\alpha}\in\mathbb{R}^{\ell}$ are learned combination weights.

This aggregated representation is fed to a linear classifier with weight matrix $\bm{\theta}\in\mathbb{R}^{d\times c}$ for $c$ dialect classes. Following Voita & Titov (2020), each weight $w$ in $\bm{\theta}$ is drawn from a normal distribution

$$ $w\sim\mathcal{N}(z\mu,z^{2}\sigma^{2}),$ $$

where the scaling factor $z$ is also drawn from

$$ $z\sim\mathcal{N}(\mu_{z},\sigma^{2}_{z}).$ $$

The pair $(w,z)$ is given a joint normal–Jeffreys prior

$$ $\gamma(w,z)\propto|z|^{-1}\,\mathcal{N}(w\mid 0,z^{2})$ $$

which encourages sparsity by pushing weights toward zero with low variance.

The probe parameters $(\bm{\alpha},\bm{\theta})$ are trained to minimize

$$ $\mathcal{L}=\mathrm{CE}(y,\hat{y})\;+\;\beta\,D_{\mathrm{KL}}\!\left(q(\bm{\theta})\,\|\,\gamma(\bm{\theta})\right),$ $$

where $\mathrm{CE}$ is the cross-entropy loss for one-vs-rest dialect classification, and the KL term regularizes $\bm{\theta}$ toward the sparsity-inducing prior. This objective maximizes compression while preserving predictive accuracy, yielding a layer-combined, token-level estimate of recoverable dialect identity information. The one-vs-rest objective hones in on dialect specific information that can help the model discern between similar dialects and offers counter-examples. We construct the training set for each dialect/variety/language by taking all the target’s sentences in MADAR 26’s training set, we construct an equal number of counter-examples from all the other dialects and languages. We make this data available (anonymized). We include training hyperparameters for the probes in Table [4](https://arxiv.org/html/2508.12803v1#A3.T4).

**Table 4: Training hyperparameters for variational probe experiments.**
| Hyperparameter | Value |
| --- | --- |
| Model name | google/gemma-3-1b-pt |
| KL weight | 1.0 |
| Number of epochs | 30 (for analysis) |
|  | 15 (for decoupling training) |
| Early stopping patience | 5 |

## Appendix D Online Decoupling Training Details

This appendix outlines the key design decisions underlying our online MSA subspace decoupling method, as well as the exact hyperparameters used in our experiments.

### D.1 Design Choices

##### Projection Subspace Estimation.

We estimate the MSA subspace using a variational linear probe trained on a Modern Standard Arabic (MSA) vs. non-MSA dialect identification task over the MADAR corpus. We recover the subspace basis from the learned probe parameters using Singular Value Decomposition (SVD) of the parameter matrix $\theta_{\text{MSA}}$. The number of retained singular vectors equals the probe’s latent dimension.

##### Online Updating.

Rather than estimating the MSA subspace once before training, we periodically retrain the probe on the current model checkpoint during fine-tuning. This ensures that the projection matrix $\mathbf{P}_{\text{MSA}}$ remains synchronized with the evolving hidden representation geometry. The projection matrix is updated every $N_{\text{update}}$ gradient steps.

##### Layer Aggregation.

Hidden representations from all layers are combined using a learned set of attention weights $\alpha\in\mathbb{R}^{L+1}$ from the variational probe. This allows the method to focus the decoupling penalty on layers most predictive of MSA features.

##### Penalty Formulation.

We penalize the $\ell_{2}$ norm of the projection of the aggregated hidden states onto the MSA subspace:

$$ $\mathcal{L}_{\text{decouple}}=\mathbb{E}\left[\left\lVert\mathbf{H}\mathbf{P}_{\text{MSA}}\right\rVert_{2}\right],$ (2) $$

where $\mathbf{H}$ are the contextual hidden states and $\mathbf{P}_{\text{MSA}}$ is the projection matrix.

##### Loss Weighting.

The decoupling penalty is scaled by a coefficient $\lambda$ and added to the standard causal language modeling loss:

$$ $\mathcal{L}=\mathcal{L}_{\text{LM}}+\lambda\cdot\mathcal{L}_{\text{decouple}}.$ (3) $$

##### Bidirectional Training Data.

To encourage symmetric modeling of both MSA $\rightarrow$ dialect and dialect $\rightarrow$ MSA directions, we construct bidirectional rewriting prompts for each sentence pair.

### D.2 Hyperparameters

**Table 5: Hyperparameters used in online MSA decoupling experiments.**
| Parameter | Value / Setting |
| --- | --- |
| Base model | google/gemma-3-1b-pt |
| Tokenizer | Matching HF tokenizer (pad_token = eos_token) |
| Batch size (per device) | 1 |
| Gradient accumulation steps | 4 |
| Max sequence length | 512 |
| Optimizer | AdamW (via HF Trainer default) |
| Learning rate | $5\times 10^{-5}$ (default HF schedule) |
| Loss coefficient $\lambda$ | 0.01 |
| Probe update steps $N_{\text{update}}$ | 500 |
| Probe training epochs | 15 |
| Probe dataset | anonymous dataset (DID-MSA) |
| Probe input type | Sequence-level dialect identification |
| Number of probe classes | 2 (MSA vs. non-MSA) |
| Projection estimation | SVD on $\theta_{\text{MSA}}$ |
| Subspace dimensionality | Full rank of $\theta_{\text{MSA}}$ |
| Layer aggregation | Learned attention weights $\alpha$ |
| Early stopping patience | 3 epochs (validation loss) |
| Early stopping threshold | 0.01 |
| Train/validation split | 90% / 10% |

## Appendix E Decoupling Results for All Dialects

In Figure [11](https://arxiv.org/html/2508.12803v1#A5.F11), we show the comparison of the baseline SFT and the decoupling method for all 25 dialects. The overwhelming majority of dialects benefit from our novel training. Aleppo, Alexandria, Fes, Rabat, Sfax, and Tunis seem to benefit the least from the method seeing barely any changes when compared to baseline SFT. Algiers seems to be the only dialect (aside from MSA) that is actively hurt from the intervention. It is difficult to explain why exactly it stands out; however, given that the overwhelming majority dialects seem to benefit from the intervention it seems to be overall beneficial to DiaMT.

Figure: Figure 11: Baseline ChrF++ (Blue) vs. our novel online decoupling method (Orange) on Multiple dialects and on English to MSA translation.
Refer to caption: x11.png

## Appendix F tSNE post SFT Training

Figure: Figure 12: Visualizing internal model representations every 5 layers across 2 different sentences (Sentence 0 in blue and Sentence 1 in orange) written in 6 varieties of Arabic using t-SNE. Layer spaces are approximated with ellipses with a color gradient from layer 0 (blue) to the last layer (red). Left is our novel decoupling training, center is the base model, right is the baseline SFT training on DiaMT.
Refer to caption: x12.png

## Peer Reviews

confidence: 3
contribution: 4
initial_rating: 2
presentation: 3
questions: > Human annotation would be the only true alternative, but is largely infeasible given the small native-speaker populations of many varieties.

I believe human annotation could still be applied to most of the analysed varieties. Even if these are “lower-resource” varieties, there exist plenty of speakers of, e.g., Egyptian Arabic, European Portuguese, and Low German.


> To complement the geometric analysis, we employ an information-theoretic variational linear probe (similar to our online subspace decoupling intervention) (Voita & Titov, 2020; Muller-Eberstein et al., 2023). The probe is a sparsity-regularized classifier trained to identify a variety from token-level representations. The resulting negative cross-entropy provides a tight lower bound on the mutual information $I(h; Y)$ between a model’s hidden states and the variety’s identity.

As argued in Pimentel et al. (2020) and McAllester et al. (2020), to estimate the mutual information you should use the best probe possible, which would typically suggest you should not regularise it. If the goal is estimating the minimum description length of the data (as proposed by Voita et al., 2020), however, then the regularisation might be beneficial.

* Pimentel et al. (2020). Information-Theoretic Probing for Linguistic Structure. https://aclanthology.org/2020.acl-main.420/
* McAllester et al. (2020). Formal Limitations on the Measurement of Mutual Information. https://proceedings.mlr.press/v108/mcallester20a.html
soundness: 1
strengths: The paper is relatively well written and nice to read (although I believe key information is left unspecified; see weaknesses below).

The paper studies an interesting problem: how alignment between languages’ representations affects model performance.

The paper proposes an interesting method to change the "amount" of alignment between languages, which it leverages to study the relationship between alignment and performance.
summary: This paper proposes a new method to simultaneously: (i) measure the extent to which the representations of multiple varieties of a specific language are “aligned”; (ii) use this measure to explicitly train a model to reduce this alignment across training (adding it as a new term in the model’s loss function). The paper finds that adding this extra term to the loss function improves performance on a language’s low-resource varieties, and sometimes also in the high-resource ones. The paper then argues that this shows alignment between a high- and low-resource language’s representation can hurt model performance.
weaknesses: I believe this paper is very interesting: it proposes an interesting method, it runs interesting experiments, and has interesting conclusions. Issues in the writing, however, stop me from recommending its acceptance. Specifically:

1. First, one of the key contributions of this paper is unclear from the paper’s description: the online subspace decoupling method. 

The way the paper “Identifies Higher-resource Variety Subspace” is unclear. The paper states:
> We train a variational linear probe (as in Sec. 4.4) to distinguish the higher-resource variety from all other varieties in a group. We then use Singular Value Decomposition (SVD) on the learned probe weights to extract an orthonormal basis UHR for the higher-resource subspace …

The paper, however, does not more clearly specify how this variational linear probe is parametrised. If the probe distinguishes the high-resource language from others (one-vs-all), it should be implemented as a single vector. How is SVD applied to this vector? If, alternatively, the model performs a multi-way classification, it can then apply SVD to the probe’s weight matrix. Within the found directions of this multi-way classifier, however, which form the “higher-resource subspace”? Only the one corresponding to the “high-resource logit”? Or all directions, which can also distinguish between low-resource languages? 

The motivation behind “Define Decoupling Loss” is also not very clear. The paper states:
> This decoupling loss penalizes the magnitude of the projection of the model’s hidden states H onto the higher-resource subspace: $E[|\mathbf{H}\mathbf{P}|_2]$

Where $\mathbf{P}$ is a projection matrix based on the SVD above. The paper thus minimises the alignment between all hidden states and this projection. What is the intuition behind this? Why are the hidden states of the high-resource variety also being trained this way?


2. Second, other methods used in the paper are also underspecified. E.g.,:
> Furthermore, we compute Subspace Angles (SSA) (Muller-Eberstein et al., 2023) to measure the alignment between subspaces corresponding to different dialects.

The SSA method is never defined in this paper. Ideally, a paper should be self-contained, and it would be useful if this method were explicitly described here.


3. Third, I also believe some interpretations of the plots are unjustified—unless I am misreading these plots. As an example, Figure 3’s caption states:

> Figure 3: (Left) During baseline SFT, the subspace angle (SSA) between MSA and dialects consistently increases, indicating growing representational separation. (Right) This increase in separation correlates directly with improved chrF++ scores. This provides strong evidence that disentangling from MSA is a key mechanism for improving dialectal generation.
 
However, there is no “consistent” increase in SSA on the left plot; I am not even convinced there is an actually increasing trend on it. Similarly, the paper also states:

> As shown in Figure 7, standard fine-tuning causes the code length for all dialects to increase slightly, as the model specializes for generation rather than classification. However, the increase is disproportionately large for MSA.

I see no clear increasing trend in Figure 7 for the low-resource varieties, and even the MSA trend is mixed, with a strong increase followed by a decrease.

4. Finally, the paper makes clear causal claims, which I also believe are unjustified. One of the section’s titles is: “Causal validation: Online subspace decoupling boosts performance”. While I appreciate the proposed method and think it is quite an interesting method to analyse the alignment between language’s subspaces, I do not think it justifies such strong “causal” claims. Formally, a causal method *must* carefully isolate a specific property and remove *all* possible confounders. This is not the case here. The authors themselves are aware of this, as they run an extra experiment to control for one possible confounder:

> To rule out gains from generic hidden space regularization, we tested random subspace shrinking on Arabic dialects. As shown in Table 3, performance consistently dropped below baseline for MSA, the dialects, French, and English, confirming that improvements arise specifically from disentangling oversized higher-resource subspaces rather than from indiscriminate regularization.
 
While this is a good extra experiment, which gives me more confidence that there indeed exists a causal relationship between language alignment and model quality, the proposed method is not enough to measure such a causal effect and only suggests that such an effect exists.
 


I want to again highlight that I believe this paper is very interesting and that it has great potential. The issues pointed out above, however, which mostly relate to writing, stop me from recommending its acceptance.

---

confidence: 3
contribution: 3
initial_rating: 8
presentation: 4
questions: - I'd be super curious to see how these findings change at scale, given how much transfer and generalization can change at scale (https://arxiv.org/abs/2403.05530)
soundness: 1
strengths: - The specific problem with using multilingual models and co-training with higher resource related languages for dialects is an understudied yet significant problem.
- I am more familiar with multilingual representation research and think this connection to dialect MT is interesting.
- The results are reasonable and intuitive
summary: - The authors use a probing method that estimates the subspace of given languages during fine-tuning
- They do this to prevent what they term "excessive" alignment of closely related languages and improve dialect performance. 
- They show that doing this improves quality significantly across several high resource-dialect pairs of languages.
weaknesses: - I think a control of a few unrelated languages at least for analysis could strengthen your claims
- Seeing how these findings change with changing scale would have been great 

Presentation etc:
- You should make it more clear early in the paper that you also introduce a technique 
- nit: From what I understand, you used an existing technique for analysis but introduced a new technique but the abstract and intro reads a bit differently
- Figure 4 is super unclear

---

confidence: 3
contribution: 2
initial_rating: 2
presentation: 2
questions: 1. Can you further motivate the choice to use English as a pivot for Czech-Slovak and Indonesian-Malay? 
2. What are the components of Figure 4? I am not sure how to read the blue and orange lines, nor the circles in the figures 
3. To be clear, do Sections 5.2 and 5.3 include no subspace decoupling?

Typos: 

imabalance -> imbalance (line 99)

precide -> precise (line 242)

Note:

The font size in Table 2 is too small to comfortably read, it may be a good idea to change this sizing.
soundness: 3
strengths: 1. The proposed online decoupling method is well-motivated and its formulation is well explained. It is also general in its formulation and potentially applicable outside of the inter-variety MT task as well. 
2. The random subspace experiment represented in Table 3, where the question of generic vs targeted hidden space regularization task, is a great analysis to isolate the effect of the specific method proposed. 
3. The paper includes substantial testing across a number of related languauge groups. Different groups may have unique relationships, so testing on a large number helps show the breadth of the current method.
summary: This paper addresses representational entanglement in the inter-variety MT setting, and proposes a method to decouple varieties during MT fine-tuning of LLMs. This method projects varieties onto a high-variety subspace, and uses the norm of this projection as a loss penalty. Testing on numerous groups of related languages, the authors find some improvement in some related languages, and some translation degradation in others. Further analysis on Arabic varieties shows that different measures of subspace separation are aligned with performance in the inter-variety MT task.
weaknesses: 1. Motivation and contribution mismatch: The introduction to the paper focuses on the problem of "generative quality" and makes a claim that cross-lingual alignment sometimes can impede this quality. However, it is revealed much later that the test-bed of this paper is solely the inter-variety translation task. It is much more clearly the case that cross-lingual alignment would indeed harm something like inter-variety translation, but this is not clear from the motivation. Basically, if this were a paper focusing on just inter-variety translation, this would be alright, but the claims early on in the paper make the problem of alignment sound much broader than just in inter-variety translation. 
2. Context in orthogonal subspace projection: The idea of projecting hidden states onto an orthogonal subspace to try to separate subspaces has been previously explored, notably extensible by the continual learning community. While this is a new application of these methods, there is important contextualization to discuss in the related work section. For example, “Gradient Projection Memory for Continual Learning” explore a similar idea, as well as “Orthogonal Subspace Learning for Language Model Continual Learning.” 
3. Does not seem to be consistent in MSA: The degradation on MSA is severe and limits the finding of extensibility across different language varieties. Since bi-directional translation is tested in this paper, as well as claims that the method can help both low- and high-resource varieties within groups, the stark degradation of MSA weakens these claims. Also, it brings into question if the analysis on MSA+varities in section 5.2 onwards is consistent with the subspace decoupling motiation, since MSA performs much more poorly.

---

confidence: 4
contribution: 2
initial_rating: 4
presentation: 1
questions: 1) How about the results of COMET or human evaluation on some language pairs? Are they aligned with the ones of chrF++?

2) (Figure 4) How to infer that Aya exhibits clearer separation between dialectal clusters than other models?

3) Line 428: Given the negative relationship between cosine distance and performance, which means that better alignment (lower cosine distance) in these layers is beneficial, does this contradict your previous assumption?

4) Line 431: Given the different correlation pattern between Aya and other models, how to infer that "subspaces must be aligned enough for
knowledge transfer but separate enough to preserve unique dialectal features." ?
soundness: 2
strengths: It is interesting to investigate the relationship between the internal language representation alignment and multilingual performance.
summary: This paper investigates the relationship between the internal language representation alignment and multilingual performance and challenge the assumption that alignment with a high-resource standard is always beneficial. The authors first introduce a training method named online subspace decoupling and use it to fine-tune multilingual large language models on inter-variety machine translation data. They further propose a geometric analysis and information-theoretic probing to evaluate their assumption. Experimental results show that their training method advances the most of language performance excepting some high-resource languages like French and Modern Standard Arabic.
weaknesses: - Only chrF++ is used to evaluate the performance of machine translation task. Other metrics like COMET[1], which is found better correlation with human judgements, can be adopted to improve the soundness of these findings.

- The writing of this paper is poor. For example, it is hard to follow Figure 4 and its caption. How to infer that Aya exhibits clearer separation between dialectal clusters than other models? Figure 3 is presented at the 7th page but is referred at the 9th page.

- The assumption may not be well supported for missing the statistic of the separation in the model trained by Online Decoupling Training method, and some models like Qwen 3-14B is different to the Aya (Figure 6). 

**References**

[1] Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. COMET: A Neural Framework for MT Evaluation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2685–2702, Online. Association for Computational Linguistics.


Decision: Reject