RomanLens: The Role Of Latent Romanization In Multilinguality In LLMs
Authors: Alan Saji, Jaavid Aktar Husain J, Thanmay Jayakumar, Raj Dabre, Anoop Kunchukuttan, Ratish Puduppully
Accepted at: acl25

RomanLens: The Role of Latent Romanization in Multilinguality in LLMs

Alan Saji1†‡, Jaavid Aktar Husain2‡, Thanmay Jayakumar1,3, Raj Dabre1,3,4,5§ Anoop Kunchukuttan1,6, Ratish Puduppully7‡

1Nilekani Centre at AI4Bharat, 2Singapore University of Technology and Design, 3Indian Institute of Technology Madras, India, 4National Institute of Information and Communications Technology, Kyoto, Japan, 5Indian Institute of Technology Bombay, India, 6Microsoft, India, 7IT University of Copenhagen

Abstract

Large Language Models (LLMs) exhibit strong multilingual performance despite being predominantly trained on English-centric corpora. This raises a fundamental question: How do LLMs achieve such multilingual capabilities? Focusing on languages written in non-Roman scripts, we investigate the role of Romanization—the representation of non-Roman scripts using Roman characters—as a potential bridge in multilingual processing. Using mechanistic interpretability techniques, we analyze next-token generation and find that intermediate layers frequently represent target words in Romanized form before transitioning to native script, a phenomenon we term Latent Romanization. Further, through activation patching experiments, we demonstrate that LLMs encode semantic concepts similarly across native and Romanized scripts, suggesting a shared underlying representation. Additionally, for translation into non-Roman script languages, our findings reveal that when the target language is in Romanized form, its representations emerge earlier in the model’s layers compared to native script. These insights contribute to a deeper understanding of multilingual representation in LLMs and highlight the implicit role of Romanization in facilitating language transfer. Code and data are available at https://github.com/AI4Bharat/Romanlens.

1 Introduction

The majority of modern Large Language Models (LLMs) (Touvron et al., 2023; Dubey et al., 2024; Team et al., 2024) are trained predominantly on English-dominated corpora. Nonetheless, they exhibit strong multilingual performance across diverse languages (Shi et al., 2023; Huang et al., 2023; Zhao et al., 2024a; Zhang et al., 2023). This raises a fundamental question: How do LLMs develop such robust multilingual capabilities despite their English-centric training?

To address this, prior work by Wendler et al. (2024) suggests that LLMs encode multilingual information within a shared, language-agnostic latent space, albeit with an inherent bias toward English due to training data composition and architectural choices. Building on this perspective, we investigate a complementary mechanism that may underlie multilingual processing, particularly for languages written in non-Roman scripts.

We hypothesize that LLMs leverage romanized forms of non-Roman script languages as an intermediate bridge between their language-agnostic concept space and language-specific output representations. Romanization—the representation of non-Roman scripts using Roman characters—may facilitate this process by aligning non-English languages more closely with English. Supporting this, Jaavid et al. (2024) demonstrated that explicitly romanizing inputs improves model performance on multilingual tasks, suggesting an inherent alignment between romanized text and English representations. We investigate whether LLMs indeed use romanization as a bridge between language-agnostic concepts and language-specific outputs given its potential implications for understanding multilingual processing in LLMs.

Our primary experiment visualizes next-token generation using the logit lens (Nostalgiaeraiast, 2020), applying the language modeling head to intermediate layers. As illustrated in Figure 1, we prompt the LLaMA-2 7B (Touvron et al., 2023) model with “Francais: porte - हिन्दी:” to translate “door” from French to Hindi. Our results show that in the middle-to-top layers (layers 20–29), romanized Hindi subwords intermittently appear before transitioning to native script, suggesting an internal representation of romanized text as an intermediary.

†Work done during employment at AI4Bharat.
§Work done during employment at NICT, Japan.
‡ Correspondence: Alan Saji (alansaji2001@gmail.com) , Ratish Puduppully (rapu@itu.dk)
Figure 1: Logit lens visualization of Llama-2 7B model translating ‘door’ from French to Hindi. We visualize the output (दरवाजा, ‘Darwaza’ is the romanized form ) taking shape using logit lens producing a next-token distribution for each position (x-axis) and layers 14 and above (y-axis). Interestingly in the middle to top layers (20 - 29) we could observe romanized subwords of the Hindi word (बट - w; अज़ा ; ज़ -azz; ज़ - j) and dependent vowels ( र -a,ch ; फ़ - i) before they are represented in their native script. Color represents entropy of next-token generation from low (blue) to high (red). Plotting tool: (Belrose et al., 2023).

Additionally, these romanized representations increase in prominence across timesteps as the target word is generated.

To further probe this phenomenon, we employ activation patching (Ghandeharioun et al., 2024; Variengien and Winsor, 2024; Chen et al., 2024; Dumas et al., 2024), a technique that replaces activations from one forward pass with another to analyze the resulting outputs (c.f. Section 3.3). Dumas et al. (2024) found that LLMs process language and conceptual information as distinct entities. Building on this, we perform layerwise activation patching between romanized and native-script inputs to examine whether LLMs encode conceptual information similarly across scripts.

Based on our experiments, we list down a summary of our contributions below:
1. Latent Romanization Across Layers: During multilingual next-token generation, intermediate layers occasionally represent tokens in Romanized form before resolving to native script (Figure 1). We term this phenomenon Latent Romanization.
2. Consistent Semantic Encoding Across Scripts: Activation patching experiments reveal that LLMs encode semantic concepts similarly, regardless of whether the input is in native or Romanized script.
3. Earlier Emergence of Target Representations: When translating into Romanized versus native script, Romanized target representations emerge earlier in the model’s layers—typically one or two layers prior to native script representations (c.f. Figure 2).

Figure 2: Translation comparison: Romanized vs. Native script. Next-token generation is visualized using the logit lens for the Gemma-2 9B model translating "flower" from French to Malayalam in romanized (left) and native script (right). The x-axis shows next-token distributions; the y-axis covers layers 30 and above. Target language representations (e.g., “push”, “poo”) appear 1–2 layers earlier in romanized outputs compared to native script outputs.( _ol , 'pa' is its romanized representation). Push is a prefix of pushpan, the romanized form of the translation of “flower”; also, poo is another romanized translation of “flower”.Color represents entropy of next-token generation from low (blue) to high (red). Plotting tool: (Belrose et al., 2023).

2 Related Work

Recent studies have explored various aspects of LLMs’ multilingual behavior: examining whether
English emerges as a latent language in English-centric LLMs (Wendler et al., 2024), how the composition of training corpus mixtures influences latent representations (Zhong et al., 2024) and how LLMs handle multilingual capabilities (Zhao et al., 2024b). Kojima et al. (2024) describe distinct phases in multilingual information processing: initial layers map language-specific lexical and syntactic representations to a language-independent semantic space, middle layers maintain this semantic abstraction, and final layers transform these representations into language-specific lexical and syntactic forms. Interpretability tools relevant to this work include logit lens (Nostalgebraist, 2020), tuned lens (Belrose et al., 2023) and direct logit attribution (Elhage et al., 2021) which are key tools for decoding intermediate token representations in transformer models. The logit lens applies the language modeling head to earlier layers without additional training, while the tuned lens in addition to this trains an affine mapping to align intermediate states with final token predictions. Direct logit attribution attributes logits to individual attention heads. This work focuses on the logit lens (Section 4.1) to investigate whether English-centric decoder only LLMs when prompted in a non-Roman script language, processes via romanized latent states before producing native language text. Tuned lens is avoided as its training process might obscure the intermediate romanized states by aligning them to final native script outputs, potentially masking the phenomenon under investigation.

Activation patching (Meng et al., 2022) is a key interpretability technique employed in our study. This technique has been used to draw causal interpretations of LLMs representations (Variengien and Winsor, 2024; Geiger et al., 2022; Kramár et al., 2024; Ghandeharioun et al., 2024; Chen et al., 2024). Building on these approaches, we adopt an activation patching-based experimental framework to investigate and compare how concepts are encoded in romanized versus native scripts.

Previous studies have demonstrated that romanization can serve as an effective approach to interact with LLMs (Jaavid et al., 2024). Liu et al. (2024) and Xhelili et al. (2024) employ an approach based on contrastive learning for post-training alignment, contrasting sentences with their transliterations in Roman script to overcome the script barrier and enhance cross-lingual transfer.

However, our work distinguishes itself from prior research by exploring the presence of romanized representations in the latent layers of an LLM during multilingual tasks, an aspect that, to the best of our knowledge, has not yet been investigated.

3 Background

We give a quick background of the transformer’s forward pass, romanization and the basics of mechanistic interpretability approaches such as logit lens and activation patching which we leverage in this paper.

Transformer’s Forward Pass Decoder-only transformer models (Vaswani, 2017) employ a residual architecture to process input sequences through multiple layers, producing a sequence of hidden states (latents). These latents, whose dimensionality remains the same, are updated iteratively across layers through transformer blocks \( f_j \), where \( j \in [0, k] \) indicates the layer index and \( k \) is the final layer index. For next-token prediction, the final latent \( h_i^{(k)} \) is transformed by an unembedding matrix \( U \in \mathbb{R}^{v \times d} \) to produce logit scores for vocabulary tokens which are then converted to probabilities via the softmax function (c.f. Appendix A).

3.1 Romanization

Transliteration is the conversion of text written in one script to another. Romanization is a subcategory of transliteration where the target script is English/Latin. Within romanization there are multiple romanization schemes available, each based on different considerations. One key aspect of romanization schema is if it is lossy or lossless. A lossless scheme is required in cases where we have to convert the output back to native script. Typically, deterministic transliterations are lossless, whereas natural transliterations are lossy.

Example: The Hindi word for “flower” in Devanagari and its romanization:

• Devanagari (native script): फूल

• Romanization: phool

3.2 Interpretability Tool: Logit lens

Generally, in a decoder only LLM, the unembedding matrix \( U \) is multiplied with the final hidden state and a softmax is taken on the product to produce the token distributions at that token generation step. Since all hidden states of an LLM are in the same shape, it is possible to apply the unembedding
matrix and softmax on all layers, thereby generating token distributions at all layers. This method of prematurely decoding hidden states is referred to as logit lens (Nostalgebraist, 2020). Logit lens reveals how the latent representations evolve across layers to produce the final output, providing insights into the progression of computations within the model.

3.3 Interpretability Tool: Activation Patching

Activation patching involves modifying or patching the activations at specific layers during a forward pass and observing the effects on the model’s output. In this work, we adopt the activation patching setup introduced in Dumas et al. (2024).

In the context of activation patching, let \( \ell \) denote the language of a word, \( C \) denote the concept of a word and \( w(C^\ell) \) denote that word. For example, if \( C = \text{cow} \) and \( \ell = \text{en} \), then \( w(C^{\text{en}}) = \text{`cow'}. \) Similarly \( w(C^{Fr}) = \text{`vache'}. \) We use 5-shot translation prompts to create paired source \( S = (C_S, \ell_S^{\text{in}}, \ell_S^{\text{out}}) \) and target prompt \( T = (C_T, \ell_T^{\text{in}}, \ell_T^{\text{out}}) \), with different concept, input language, and output language. Unless otherwise specified, \( \ell_S \) and \( \ell_T \) refer to the output languages of \( S \) and \( T \), respectively.

For each transformer block \( f_j \), we create two parallel forward passes: one processing the source prompt \( S = (s_1, \ldots, s_{n_S}, \ldots, s_{n_S}) \) and the other processing the target prompt \( T = (t_1, \ldots, t_{n_T}, \ldots, t_{n_T}) \). It should be noted that \( n_S, n_T \) represents the position of the last token of the object to be translated whereas \( n_S, n_T \) represent the last token position of the source and target prompt to be translated. In Figure 3 in the target prompt translating sun from French to Hindi, \( n_t \) would be the position of the subword “eil” highlighted in red, whereas \( n_T \) would be the position of “ही” the last subword of the prompt. Similarly, for the source prompt in Figure 3 translating elephant from German to Italian both \( n_s \) and \( n_S \) would be the position of “ant” highlighted in red. After creating the parallel forward passes, we extract the residual stream of the last token of the word to be translated at \( n_s \) after layer \( j \), denoted as \( h_{n_s}^{(j)}(S) \) and all subsequent layers, and insert it at the corresponding layer at the corresponding position \( n_t \) in the forward pass of the target prompt, i.e. by setting \( h_{n_t}^{(j)}(T) = h_{n_s}^{(j)}(S), h_{n_t}^{(j+1)}(T) = h_{n_s}^{(j+1)}(S), \ldots, h_{n_t}^{(k)}(T) = h_{n_s}^{(k)}(S) \). We then complete the altered forward pass and analyze the next token distribution to evaluate source concept \( C_S \) encoded in the target language. An illustration of this setup is shown in Figure 3.

![Activation patching illustration. For two given concepts, say, elephant and sun, we generate multiple source prompts which translate elephant, and a target prompt for translating sun from French to Hindi. We then extract the residual stream associated with the final token of the word to be translated after a specific layer j and all subsequent layers from the source prompts. The mean residuals at each layer are computed and inserted into the corresponding positions during the forward pass of the target prompt. The resulting next token probabilities will be dominated by the source concept in target language (ELEPHANT^H, i.e., हाथिया) when patching at layers 0–15, and by the target concept in target language (SUN^H, i.e., सूरज) for layers 16–31. Adapted from Dumas et al. (2024).](page_820_172_601_496.png)

Figure 3: Activation patching illustration. For two given concepts, say, elephant and sun, we generate multiple source prompts which translate elephant, and a target prompt for translating sun from French to Hindi. We then extract the residual stream associated with the final token of the word to be translated after a specific layer \( j \) and all subsequent layers from the source prompts. The mean residuals at each layer are computed and inserted into the corresponding positions during the forward pass of the target prompt. The resulting next token probabilities will be dominated by the source concept in target language (ELEPHANT^H, i.e., हाथिया) when patching at layers 0–15, and by the target concept in target language (SUN^H, i.e., सूरज) for layers 16–31. Adapted from Dumas et al. (2024).

4 Methodology

We design our analysis setup with the intention of addressing the following research questions:

RQ1: Do LLMs exhibit latent romanization during multilingual text completion tasks? (Section 4.1)

RQ2: How does the representation of semantic concepts in LLMs compare between native and romanized scripts of non-Roman script languages? (Section 4.2)

RQ3: What are the differences in hidden layer representations when processing the same language in romanized and native scripts? (Section 4.3)

Prompt design. We design prompts that facilitate next-token (\( x_{n+1} \)) prediction from the given context (\( x_1, \ldots, x_n \)). This is adopted across all analysis setups. The prompts are designed around translation, repetition, and cloze tasks, as described below.

Translation task. We prompt the model to translate
a word given five in-context examples.

Repetition task. We prompt the model to repeat a word in the same language given five in-context examples.

Cloze task. We prompt the model to predict the masked word in a sentence given two in-context examples.

These tasks cover a range of multilingual text completion setups. Among these, the repetition task is more syntactic in nature compared to translation and cloze tasks. Appendix B provides a Hindi example of prompts across each of the three tasks, along with their English translations and romanized forms.

4.1 Latent Romanization Analysis

Translation, repetition, and cloze tasks are explored by providing the respective prompts as inputs to an LLM to generate the corresponding output word. We romanize the output word, tokenize it, retaining only the tokens present in the model’s vocabulary, and analyze the occurrence of these tokens in the latent layers across timesteps of the output word generation. The analysis is done using logit lens by examining whether the probability of a romanized token in the next token distribution at a given layer exceeds 0.1. We refer to this hereafter as the latent romanization condition. The 0.1 threshold is empirically determined to optimize detection accuracy, i.e. minimizing false positives and maximizing true positives (compared to alternative thresholds 0.05 and 0.01). Our analysis focuses on the final 10 layers of an LLM, where coherent romanized representations emerge according to logit lens visualizations (c.f. Figures 1 and 8 - 12).

We track romanized tokens using a timestep-specific tokenization scheme optimized for detection accuracy. In the first output generation timestep, we check for tokens that include the full romanized word and its prefixes. During intermediate timesteps, we check for all possible substrings of the romanized word in the latent layers. In the final output generation timestep, we probe the presence of only the full romanized word and its suffixes as potential tokens (c.f. Appendix C).

Latent romanization is analyzed under three distinct scenarios:

(a) Constrained Word Generation: Using standard prompts and the target word, we guide the model to generate the complete target word. At each layer, we track how often romanized tokens emerge during the decoding process. We do this by checking each generation timestep for latent romanization condition. The ‘latent fraction’ for a layer represents how frequently these romanized tokens appear across timesteps, averaged across all samples (c.f. Appendix D).

(b) First Subword Only: We prompt for only the initial subword and compute the latent fraction. Despite having a single timestep, we maintain the latent fraction terminology for consistency.

(c) Last Subword Only: We augment the standard prompt with all but the final subword of the target, then analyze the generation of the final subword.

We document layerwise latent fraction separately for first and last subword generation of the output. Intuitively, there is a distinction between the first and last token generation steps for a given word. In the former, the model faces a greater decision-making burden, while in the latter, the model is typically more confident in its predictions. We hypothesize that, in the latter scenario, the model may reach a decision in the layers just below the final few layers and express the output in a romanized form, as language-specific neurons, responsible for native script processing, are concentrated primarily in the last few layers (Tang et al., 2024). This could lead to romanized tokens appearing more frequently as the model progresses from the first subword to the last.

4.2 Patching With Romanized Representation Versus Native Representation

We intend to compare how concepts are encoded in native script versus romanized script using translation task. In order to do this, we patch representations from source prompt where input language is romanized and compare this with patching from source prompt where input language is in native script. We first perform patching using a single-source prompt. Then, we repeat the process using an averaged multi-source prompt, contrasting multiple romanized source input languages with multiple native script source input languages. Single-source patching might be influenced by language or script-specific characteristics. In contrast, multi-source
patching reduces such biases, leading to more robust and generalizable findings.

In all scenarios from the resulting next token distribution, we compute the probabilities \( P(C_{S}^{l_T}) \) i.e. probability of source concept in target language, and \( P(C_{T}^{l_T}) \) i.e. probability of target concept in target language. We track \( P(C^{l_T}) \), i.e., the probability of the concept C occurring in language \( l_T \), by simply summing up the probabilities of all prefixes of \( w(C^{l_T}) \) and its synonyms in the next-token distribution (c.f. Appendix G). We analyze \( P(C_{S}^{l_T}) \) to evaluate how effectively a concept is encoded in a given source input language \( l_S^{in} \).

4.3 Comparing Translations Into Romanized vs. Native Script

This analysis examines translation task with target languages in native script and their romanized equivalents. We focus on first-token generation of the output word, also considering possible synonyms.

In the next token generation step, the probability of target language and latent language (English) (Wendler et al., 2024) at each layer is examined using logit lens. Each probability is computed by summing over probabilities of all possible tokens corresponding to the answer word(s) in that respective language (c.f. Appendix E). Tokens of latent language and target language are derived using tokenization scheme for first token generation timestep mentioned in Section 4.1.

5 Experimental Settings

Languages: We focus on five Indic languages: Hindi, Gujarati, Tamil, Telugu, and Malayalam, as well as Chinese and Georgian. Among these, Hindi and Gujarati belong to the Indo-Aryan branch of the Indo-European language family and use scripts derived from the Devanagari and Gujarati scripts, respectively. Tamil, Telugu, and Malayalam, on the other hand, are part of the Dravidian language family and each has its own distinct script. Chinese belongs to the Sino-Tibetan language family and is written using logographic characters. Georgian is part of the Kartvelian language family and uses the unique Georgian script. To examine the generality of latent romanization, we perform qualitative analyses on five additional languages that use different writing systems: Greek, Ukrainian, Amharic, Hebrew, and Arabic (c.f Appendix H).

Language Models: In this study, we focus mainly on Gemma-2 9B, Gemma-2 9B instruction-tuned (Team et al., 2024), Llama-2 7B, Llama-2 13B (Touvron et al., 2023) and Mistral-7B (Jiang et al., 2023) (c.f. Appendix K) language models, some of the best performing open weights English-centric LLMs. Although the training data for these models are primarily English, these models have high multilingual capabilities (Huang et al., 2023; Zhao et al., 2024a; Zhang et al., 2023).

Romanization: We have used the IndicXlit scheme (Madhani et al., 2023) (c.f. Appendix F) for Indic languages, pypinyin (Mozillazg, 2024) for Chinese and Unidecode (Šolc, 2025) for Georgian to romanize native scripts.

Data For Logit Lens Experiments: We use a curated word-level dataset with synonyms translated from recent works in this field (Wendler et al., 2024) using the Llama 3.3 70B model (Dubey et al., 2024). The quality of translations were manually verified to ensure accuracy and relevance. The datasets are kept simple to facilitate the observation of how latents evolve at each token level.

Data For Activation Patching Experiments: We adopt the dataset used in recent studies (Dumas et al., 2024), extending it to include both native script and romanized versions of the languages considered in this study. Translations are performed using the Llama 3.3 70B model (Dubey et al., 2024) and the translations are romanized using IndicXlit (Madhani et al., 2023), pypinyin (Mozillazg, 2024) and Unidecode (Šolc, 2025). All translations were manually validated to ensure data quality.

6 Results

6.1 Latent romanization

Our analysis demonstrates that LLMs do exhibit latent romanization during text completion tasks in six out of seven quantitatively analyzed languages (c.f. Figure 4a), with Chinese being the only exception where this phenomenon is not observed. In Figure 4a we can see the latent fraction of romanized tokens for the last 10 layers of an LLM. This is across all tokens of the output word. The frequency of romanized tokens tends to increase just before the last layers. Qualitative logit lens analysis done for Greek, Ukrainian, Arabic, Amharic and Hebrew reveals similar patterns (c.f. Appendix H).
Figure 4: Distribution of Romanized Tokens Across Model Layers: Analysis of First, Last, and All Generation Timesteps. This distribution is plotted across the last 10 layers of Gemma-2 9b IT model for translation task with English as source language and is averaged across 100+ samples. X-axes represents layer index, y-axes represents latent fraction i.e. the fraction of timesteps where romanized tokens occur with a probability > 0.1 averaged over samples for a specific layer. We plot the distributions for Gujarati (gu), Tamil (ta), Telugu (te), Hindi (hi), Malayalam (ml), Georgian (ka) and Chinese (zh).

Figure 5: Frequency distribution of romanized tokens across translation, repetition and cloze task. We check if romanized tokens occur with a probability > 0.1 in the last 10 layers of an LLM and compute frequency of this occurrence across 100+ samples. Gemma 2 9B IT is the model used and English is the source language for translation task.

In Figures 4b and 4c it is observed that the range of the latent fraction of romanized tokens varies from \(0 - 0.01\) in the first token generation step to \(0 - 0.2\) in the last token generation step in most languages. This trend indicates that latent romanization increases progressively from the initial token to the final token of the output across languages. This observation supports our hypothesis that the first token generation involves more intricate decision-making processes compared to the generation of the final token within an output word.

Based on the above information, we quantify romanization across tasks for the last token generation step. Here the criteria would be if a romanized token occur at next token generation step with a probability > 0.1 in the last 10 layers, it is counted as a positive. As depicted in Figure 5, across translation, repetition and cloze task we observe a significant occurrence of romanized tokens in the latent layers.

Among the three text completion tasks considered in the latent romanization experiment (Section 4.1), a relatively high latent romanization frequency is observed for the repetition task (Figure 5). The translation and cloze tasks function at a semantic level, whereas the repetition task is purely syntactic. This means that the repetition task being less complex, the model may reach a decision of what to predict sooner, potentially in earlier layers and might express its prediction in romanized form in intermediate layers. This behavior could be attributed to language-specific neurons, responsible for native script processing, being predominantly concentrated in the initial and final few layers of LLMs, (Tang et al., 2024) leaving the intermediate layers without them.

6.2 Patching With Romanized Representation vs. Native Representation

In Figure 6, we analyze two patching scenarios: In the single-source setup, we compare patching from Hindi→Italian with Hindi(romanized)→Italian source prompt, to Malayalam→Italian target prompt. In the multi-source setup, we contrast patching from multiple native script prompts (Hindi→Italian, Gujarati→Italian ...) against their romanized counterparts (Hindi(romanized)→Italian, Gujarati(romanized)→Italian ...). We compare the probability distributions of source concept in target language \(P(C_{S'}^{t_{T}})\) across adjacent graphs where native source input language is contrasted with romanized source input language. It is evident that these probability distributions show remarkable similarity whether the source input language is in romanized or native script, consistent across both single-source and multi-source prompt setups. The similarity is quantitatively supported by the KL divergence measurements between adjacent
Figure 6: Comparative Analysis of Patching from Source Prompts: Native Script vs. Romanized Script Inputs. Concept probabilities across layers for different prompt setups are plotted in each graph. The x-axis represents the patching layer, while the y-axis indicates the probability of correctly predicting the concept in language \( l \). Curves: blue (target concept in Italian), orange (source concept in Italian), and green (source or target concept in English). Results are reported as means with 95% Gaussian confidence intervals, calculated over a dataset of 200 samples. The orange curve is compared across adjacent graphs and KL divergence \( D_{KL} \) quantifies this. Languages involved: Hindi (hi), Tamil (ta), Telugu (te), Malayalam (ml), Gujarati (gu) and Italian (it). Model: Gemma 2 9b it.

graphs, remaining below 0.01 in both setups. KL divergence value close to zero indicates that the two distributions are nearly identical.

This analysis reveals that LLMs encode semantic concepts similarly regardless of whether the input is in native or romanized script. Furthermore, this finding demonstrates that the model achieves comparable levels of language understanding when processing non-Roman script languages in their romanized form as in their native script.

6.3 Comparing Translations Into Romanized vs. Native Script

We quantify the observations from Figure 2 by analyzing next-token predictions across layers using logit lens. In Figure 7, panels 7c and 7d illustrate that for translations into native scripts, target language tokens begin to emerge from layer 40 onward. Conversely, in panels 7a and 7b, where the target language is in romanized script, target tokens appear 1–2 layers earlier. This pattern indicates that when processing non-Roman script languages, the model forms internal representations of target tokens in earlier layers for romanized script compared to native script. This trend is consistent across language pairs and models (c.f. Figures 16-18 in the Appendix). This suggests that romanization facilitates faster progression toward language-specific embeddings.

In Figure 7, in all four graphs consistent with Wendler et al. (2024), English representations emerge from the middle layers and persist until the final few layers, where the target language representations gradually take shape. It is important to note that native script curves (Figures 7c, 7d) exhibit steeper gradients than their romanized equivalents (Figures 7a, 7b).

Discussion. In our investigation of romanized representations in the latent layers, we conclusively identified romanized tokens in the last 6–7 layers of an LLM across various multilingual text completion tasks. Based on previous works in this field (Wendler et al., 2024; Zhao et al., 2024b), in an English-centric decoder only LLM this region corresponds to the transition from an English-centric language-agnostic concept space to a language-specific space where the idea conceived in the concept space is expressed in the target language. Our findings suggest that romanization serves as a bridge between the concept space and the language-specific region for non-Roman script languages, an observation strongly supported by our analysis of six diverse writing systems. Romanization acting as a bridge could explain why
Figure 7: Language probabilities for latent layers in translation from French to Malayalam and Tamil in romanized (top row) and native scripts (bottom row) across various samples using Gemma-2 9B IT model. X-axis: layer index; Y-axis: probability of correct next token (via logit lens) in a given language. Error bars: 95% Gaussian confidence intervals. English is the latent language (orange curve). For romanized script, target representations (blue curve) emerge 1-2 layers earlier than native script, appearing before layer 40.

romanization based script barrier breaking methods like Liu et al. (2024) and Xhelili et al. (2024) work. Notably, we do not observe Latent Romanization in Chinese, likely due to its logographic script and relatively high-resource status.

7 Conclusion

Our findings show that LLMs implicitly use Romanization as a bridge for non-Roman scripts, exhibiting Latent Romanization in intermediate layers before switching to native scripts. Layerwise analyses reveal that semantic concepts are encoded similarly across native and Romanized inputs, indicating a shared internal representation. Moreover, when translating into a Romanized script, target words emerge earlier, highlighting Romanization as a structural link between language-agnostic concepts and language-specific output. While our study reveals initial insights into Latent Romanization, future work could focus on applying these findings to develop training strategies that enhance performance across diverse linguistic communities.

8 Limitations

The handling of multilingual text by large language models (LLMs) remains an active area of research. Although evidence suggests that LLMs process English representations within a language-agnostic space, the specific mechanisms by which these models adjust their interactions over different timesteps during token generation are still not fully understood. In our study, we observe that romanized representations become increasingly prominent in the hidden layers as token generation progresses from the first to the final token. This trend suggests that latent romanization may help the model mitigate differences in token fertility—that is, the average number of tokens required to represent a word—between the output language and its primary latent language, English. This effect appears especially for non-Roman script languages, with high token fertility. However, further research is needed to confirm and generalize these observations.

The interpretability of non-Roman scripts at latent layers is limited when models employ tokenization schemes that split non-Roman characters into multiple bytes, complicating logit lens analysis. Extending this work to models with alternative tokenization methods would offer a more complete understanding of multilingual capabilities and representations.

This work identifies but does not explain the selective occurrence and varying intensity of latent romanization across languages—questions that merit dedicated future investigation.

9 Ethics Statement

Through this work, our aim is to democratize access to LLMs and address the issue of limited data availability for low-resource languages. We emphasize that it is not our intention to diminish the value or significance of the native scripts of the languages included in this study.

The code and datasets created in this work will be made available under permissible licenses. Generative AI systems were only used for assistance purely with the language of the paper, e.g., paraphrasing, spell-check, polishing the author’s original content, and for writing boiler-plate code.

References

Nora Belrose, Zach Furman, Logan Smith, Danny Halawi, Igor Ostrovsky, Lev McKinney, Stella Biderman, and Jacob Steinhardt. 2023. Eliciting latent predictions from transformers with the tuned lens. arXiv preprint arXiv:2303.08112.

Haozhe Chen, Carl Vondrick, and Chengzhi Mao. 2024. Selfie: Self-interpretation of large language model embeddings. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.

Clément Dumas, Chris Wendler, Veniamin Veselovsky, Giovanni Monea, and Robert West. 2024. Separating tongue from thought: Activation patching reveals language-agnostic concept representations in transformers. arXiv preprint arXiv:2411.08745.

Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, et al. 2021. A mathematical framework for transformer circuits. Transformer Circuits Thread, 1(1):12.

Atticus Geiger, Zhengxuan Wu, Hanson Lu, Josh Rozner, Elisa Kreiss, Thomas Icard, Noah Goodman, and Christopher Potts. 2022. Inducing causal structure for interpretable neural networks. In International Conference on Machine Learning, pages 7324–7338. PMLR.

Asma Ghandeharioun, Avi Caciularu, Adam Pearce, Lucas Dixon, and Mor Geva. 2024. Patchscopes: A unifying framework for inspecting hidden representations of language models. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net.

Haoyang Huang, Tianyi Tang, Dongdong Zhang, Xin Zhao, Ting Song, Yan Xia, and Furu Wei. 2023. Not all languages are created equal in LLMs: Improving multilingual capability by cross-lingual-thought prompting. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 12365–12394, Singapore. Association for Computational Linguistics.

J Jaavid, Raj Dabre, M Aswanth, Jay Gala, Thanmay Jayakumar, Ratish Puduppully, and Anoop Kunchukuttan. 2024. Romansetu: Efficiently unlocking multilingual capabilities of large language models via romanization. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15593–15615.

Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023. Mistral 7b. arXiv preprint arXiv:2310.06825.

Takeshi Kojima, Itsuki Okimura, Yusuke Iwasawa, Hitomi Yanaka, and Yutaka Matsuo. 2024. On the multilingual ability of decoder-based pre-trained language models: Finding and controlling language-specific neurons. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 6919–6971, Mexico City, Mexico. Association for Computational Linguistics.

János Kramár, Tom Lieberum, Rohin Shah, and Neel Nanda. 2024. Atp*: An efficient and scalable method for localizing llm behaviour to components. arXiv preprint arXiv:2403.00745.

Anoop Kunchukuttan. 2020. The indicnlp library.

Yihong Liu, Chunlan Ma, Haotian Ye, and Hinrich Schuetze. 2024. TransliCo: A contrastive learning framework to address the script barrier in multilingual pretrained language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2476–2499, Bangkok, Thailand. Association for Computational Linguistics.

Yash Madhani, Sushane Parthan, Priyanka Bedekar, Gokul Nc, Ruchi Khapra, Anoop Kunchukuttan, Pratyush Kumar, and Mitesh Khapra. 2023. Aksharantar: Open Indic-language transliteration datasets and models for the next billion users. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 40–57, Singapore. Association for Computational Linguistics.

Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2022. Locating and editing factual associations in GPT. In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 - December 9, 2022.

Mozillazg. 2024. pypinyin: A python library to convert chinese characters to pinyin. Accessed: 2025-01-23.

Nostalgalebraist. 2020. Interpreting GPT: The Logit Lens. LessWrong.

Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush Vosoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, et al. 2023. Language models are multilingual chain-of-thought reasoners. In The Eleventh International Conference on Learning Representations.

Tianyi Tang, Wenyang Luo, Haoyang Huang, Dongdong Zhang, Xiaolei Wang, Xin Zhao, Furu Wei, and Ji-Rong Wen. 2024. Language-specific neurons: The key to multilingual capabilities in large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5701–5715, Bangkok, Thailand. Association for Computational Linguistics.

Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. 2024. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajwial Bhargava, Shruti
Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.

Alexandre Variengien and Eric Winsor. 2024. Look before you leap: A universal emergent decomposition of retrieval tasks in language models. In ICML 2024 Workshop on Mechanistic Interpretability.

Chris Wendler, Veniamin Veselovsky, Giovanni Monea, and Robert West. 2024. Do llamas work in English? on the latent language of multilingual transformers. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15366–15394, Bangkok, Thailand. Association for Computational Linguistics.

Orgest Xhelili, Yihong Liu, and Hinrich Schuetze. 2024. Breaking the script barrier in multilingual pre-trained language models with transliteration-based post-training alignment. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 11283–11296, Miami, Florida, USA. Association for Computational Linguistics.

Wenxuan Zhang, Mahani Aljunied, Chang Gao, Yew Ken Chia, and Lidong Bing. 2023. MExam: A multilingual, multimodal, multilevel benchmark for examining large language models. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023.

Jun Zhao, Zhihao Zhang, Luhui Gao, Qi Zhang, Tao Gui, and Xuanjing Huang. 2024a. Llama beyond english: An empirical study on language capability transfer. arXiv preprint arXiv:2401.01055.

Yiran Zhao, Wenxuan Zhang, Guizhen Chen, Kenji Kawaguchi, and Lidong Bing. 2024b. How do large language models handle multilingualism? In Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 - 15, 2024.

Chengzhi Zhong, Fei Cheng, Qianying Liu, Junfeng Jiang, Zhen Wan, Chenhui Chu, Yugo Murawaki, and Sadao Kurohashi. 2024. Beyond english-centric llms: What language do multilingual language models think in? arXiv preprint arXiv:2408.10811.

Tomaž Šolc. 2025. Unicode: Ascii transliterations of unicode text. Accessed: 2025-01-23.

A Transformer’s Forward pass: Detailed

For an input sequence \( x_1, \ldots, x_n \in V \), where \( n \) is the sequence length, the initial latents \( h_1^{(0)}, \ldots, h_n^{(0)} \in \mathbb{R}^d \) are obtained from a learned embedding matrix. The update rule for the latent at position \( i \) in layer \( j \) is expressed as:

\[
h_i^{(j)} = h_i^{(j-1)} + f_j(h_1^{(j-1)}, \ldots, h_i^{(j-1)})
\]

The logit scores are computed as:

\[
z_i = U h_i^{(k)}
\]

These are converted to probabilities via the softmax function:

\[
P(x_{i+1} = t|x_1, \ldots, x_i) \propto \exp(z_{i,t})
\]

B Sample Prompts

A Hindi example, its English translation and transliteration for the translation, repetition and cloze task prompt designs mentioned in Section 4 are provided below.

Hindi example.

Translation task. A translation prompt from French to Hindi.

<table>
  <tr>
    <th>Français:</th>
    <th>हिन्दी:</th>
  </tr>
  <tr>
    <td>“poisson”</td>
    <td>“मछली”</td>
  </tr>
  <tr>
    <td>“mangue”</td>
    <td>“आम”</td>
  </tr>
  <tr>
    <td>“frère”</td>
    <td>“भाई”</td>
  </tr>
  <tr>
    <td>“odeur”</td>
    <td>“गंध”</td>
  </tr>
  <tr>
    <td>“soleil”</td>
    <td>“सूरज”</td>
  </tr>
  <tr>
    <td>“fleur”</td>
    <td>“फूल”</td>
  </tr>
</table>

Repetition task.

<table>
  <tr>
    <th>हिन्दी:</th>
    <th>हिन्दी:</th>
  </tr>
  <tr>
    <td>“मछली”</td>
    <td>“मछली”</td>
  </tr>
  <tr>
    <td>“आम”</td>
    <td>“आम”</td>
  </tr>
  <tr>
    <td>“भाई”</td>
    <td>“भाई”</td>
  </tr>
  <tr>
    <td>“गंध”</td>
    <td>“गंध”</td>
  </tr>
  <tr>
    <td>“सूरज”</td>
    <td>“सूरज”</td>
  </tr>
  <tr>
    <td>“फूल”</td>
    <td>“फूल”</td>
  </tr>
</table>

Cloze task..

एक “___” कहानियाँ पढ़ने के लिए उपयोग किया जाता है। उत्तर: “पुस्तक”
फुटबॉल और बास्केटबॉल जैसे खेल खेलने के लिए “___” का उपयोग किया जाता है। उत्तर: “बाल”
एक “___” अक्षर उपहार के रूप में दिया जाता है और यह बच्चों में पाया जा सकता है। उत्तर:

English Translation.

Translation task.
Français: “poisson”    Hindi: “fish”
Français: “mangue”     Hindi: “mango”
Français: “frère”       Hindi: “brother”
Français: “odeur”       Hindi: “smell”
Français: “soleil”      Hindi: “sun”
Français: “fleur”       Hindi:

Phutball aur baasketball jaise khel khelane ke lie “___” ka upayog kiya jaata hai. Uttar: “ball”
Ek “___” kahaaniyaan padhane ke lie upayog kiya jaata hai. Uttar: “ball” Ek “___” aksar upahaar ke roop mein diya jaata hai aur yah bageechon mein paaya ja sakata hai. Uttar:

Repetition task.

Hindi: “fish”           Hindi: “fish”
Hindi: “mango”          Hindi: “mango”
Hindi: “brother”        Hindi: “brother”
Hindi: “smell”          Hindi: “smell”
Hindi: “sun”            Hindi: “sun”
Hindi: “flower”         Hindi:

Cloze task.

A “___” is used to play sports like soccer and basketball. Answer: “ball”
A “___” is used for reading stories. Answer: “book”
A “___” is often given as a gift and can be found in gardens. Answer:

English Transliteration.

Translation task.

Français: “poisson”     Hindi: “machhalee”
Français: “mangue”      Hindi: “aam”
Français: “frère”        Hindi: “bhaee”
Français: “odeur”        Hindi: “gandh”
Français: “soleil”       Hindi: “sooraj”
Français: “fleur”        Hindi:

Repetition task.

Hindi: “machhalee”      Hindi: “machhalee”
Hindi: “aam”             Hindi: “aam”
Hindi: “bhaee”           Hindi: “bhaee”
Hindi: “gandh”           Hindi: “gandh”
Hindi: “sooraj”          Hindi: “sooraj”
Hindi: “phool”           Hindi:

Cloze task.

C Latent romanization: Tokenization scheme for the romanized word

Mathematically, we track the following romanized tokens for a given romanized word \( w \) of length \( n \):

First timestep: \( \{w[0 : i] \mid 1 \leq i \leq n\} \cup \{"_"\ + w[0 : i] \mid 1 \leq i \leq n\}\ ), “_” represents single leading space and \( w[0 : i] \) represents prefixes of \( w \)
Intermediate timesteps: \( \{w[i : j] \mid 0 \leq i < j \leq n\} \), \( w[i : j] \) represents sub-strings of \( w \)
Final timestep: \( \{w[i : n] \mid 0 \leq i < n\} \), \( w[i : n] \) represents suffixes of \( w \)

Similarly we construct token sets for the native script (\( T_{native} \)) and English (\( T_{english} \)) by including prefixes of the corresponding word, both with and without leading spaces, for all timesteps except the last. For the final timestep, we use suffixes of the corresponding word. We discard the sample if there is any overlap between romanized tokens (\( T_{romanized} \)) and either the native (\( T_{native} \)) or English tokens (\( T_{english} \)) as shown by the following condition:

\[
T_{romanized} \cap (T_{native} \cup T_{English}) = \emptyset
\]

Lets take an example with a prompt translating “rope” from French to Hindi and derive romanized, English and native tokens for its first token generation timestep. The Hindi translation of “rope” is रसी and its romanized form is “rassi”. So the romanized word tokens would be \( T_{romanized} = "r", "ra", "ras", "rass", "rassi", "_r", "_ra", "_ras", "_rass" \) and “_rassi”. The corresponding English word tokens would be \( T_{English} = "r", "ro", "rop", "rope", "_r", "_ro", "_rop" \), and “_rope”. The corresponding Hindi word tokens would be \( T_{native} = र, रस, रसी, रसी, _र, _रस, _रसी, _रसी \). Here \( T_{romanized} \cap (T_{native} \cup T_{English}) = \{r, _r\} \) which is not null. As such we will exclude
this example translating rope to Hindi from the dataset to analyze Latent Romanization.

D  Latent Fraction

Formally, we compute the latent fraction as follows:

For layer \( l \), timestep \( t \), sample \( i \) and set of corresponding romanized tokens \( R \) :
1. Latent romanization condition:
\[
r_{l,t}^{(i)} = \begin{cases}
1, & \text{if } \max_{r \in R} P(x_t = r \mid l, t) > 0.1 \\
0, & \text{otherwise}
\end{cases}
\]
2. Latent fraction for a layer \( \ell \):
\[
\mathrm{LF}(l) = \frac{1}{N} \sum_{i=1}^N \frac{1}{T} \sum_{t=1}^T r_{l,t}^{(i)}
\]
where \( N \) is the number of samples, \( T \) is the number of generation timesteps and \( P(x_t = r | l, t) \) is the probability of generating token \( r \) at timestep \( t \) and layer \( \ell \).

E  Computing Language probabilities - For translation towards native script vs translation towards romanized script task

To compute language probabilities, we search the LLM’s vocabulary for all tokens that could be the first token of the correct output word(s) in the respective language. We search the models vocabulary for all prefixes of the word(s) with and without leading space. For a language \( \ell \) with corresponding output word \( w_1 \) and its synonyms \( w_2, w_3, \ldots \), we define:
\[
P(\mathrm{lang} = \ell) = \sum_{t_\ell \in W_{\mathrm{prefix}}} P(x_{n+1} = t_\ell)
\]
where \( W_{\mathrm{prefix}} \) is the set of all prefixes of output word \( w_1 \) and its synonyms \( w_2, w_3, \ldots \), including versions with and without leading spaces. For example to get probability for english when the output word is “fast” and its synonym is “swift”, then \( P(\mathrm{lang} = \mathrm{EN}) = P(x_{n+1} = “f”) + P(x_{n+1} = “fa”) + P(x_{n+1} = “fas”) + P(x_{n+1} = “fast”) + P(x_{n+1} = “_f”) + P(x_{n+1} = “_fa”) + P(x_{n+1} = “_fas”) + P(x_{n+1} = “_fast”) + P(x_{n+1} = “s”) + P(x_{n+1} = “sw”) + P(x_{n+1} = “swi”) + P(x_{n+1} = “swift”) + P(x_{n+1} = “_s”) + P(x_{n+1} = “_sw”) + P(x_{n+1} = “_swi”) + P(x_{n+1} = “_swif”) + P(x_{n+1} = “_swift”) (all the token-level prefixes of “fast”, “_fast”, “swift” and “_swift”). “_” represents a single leading space.

F  Romanization scheme: Indic languages

We have taken into consideration two romanization schemes for Indic languages: (a) ITRANS scheme from IndicNLP library (Kunchukuttan, 2020) and (b) IndicXlit scheme (Madhani et al., 2023). Based on our initial experiments, we observed that the IndicXlit scheme produces better romanization than ITRANS scheme. Thus for romanization we have used the IndicXlit scheme (Madhani et al., 2023). It generates romanization as is commonly used by native speakers and is trained on parallel transliteration corpora.

G  Computing Probabilities : Activation Patching Experiment

Probability for a concept \( C \) in language \( \ell \) can be formulated as :
\[
P(C^\ell) = \sum_{t_\ell \in W_{\mathrm{prefix}}} P(x_{n+1} = t_\ell)
\]
where \( W_{\mathrm{prefix}} \) is the set of all prefixes of output word \( w(C^\ell) \) and its synonyms (note that a word’s tokens are its prefixes).

We keep source concept \( C_S \) and target concept \( C_T \) distinct to avoid ambiguity when both are expressed in the same target language \( \ell_{out} \).

Cases of token overlap between \( w(C_S^{\ell_T}) \) i.e. word representing source concept in target language and \( w(C_T^{\ell_T}) \) i.e. word representing target concept in target language and their synonyms are excluded. Token overlap would cause ambiguity. Therefore in the final dataset,
\[
T(w(C_S^{\ell_T})) \cap T(w(C_T^{\ell_T})) = \emptyset
\]
Where \( T(w) \) represents all the prefixes of \( w \) and its synonyms.

H  Latent Romanization Qualitative Analysis

We list qualitative logit lens analysis for Greek, Ukrainian, Hebrew, Arabic and Amharic (see Figures 8 to 12).
Languages. Greek is part of the Hellenic branch of the Indo-European language family and is written using the Greek alphabet. Ukrainian belongs to the East Slavic group of the Indo-European family and employs the Ukrainian alphabet, a variant of the Cyrillic script. Amharic is a South Ethio-Semitic language within the Afroasiatic family and is written using the Ge’ez script, an abugida where each character represents a consonant-vowel combination. Hebrew is a Northwest Semitic language within the Afroasiatic family and is written using the Hebrew alphabet, an abjad script originating from the Aramaic alphabet. Arabic is a Central Semitic language, also part of the Afroasiatic family, and utilizes the Arabic script, another abjad that evolved from the Nabataean alphabet. Notably, both Hebrew and Arabic scripts are written from right to left.

I  Latent Romanization Quantitative Analysis: Additional examples

Quantitative Analysis of latent romanization for repetition task and cloze task with gemma 2 9b it model can be seen in Figures 13 and 14 respectively. Layerwise fractional distribution of romanized tokens across output token generation timesteps for translation, repetition and cloze task with Gemma 2 9b, Llama 2 7b, and Llama 2 13b models are present in Figure 15.

J  Comparing Translations Into Romanized vs. Native Script: Additional examples

Translation towards native script is compared with translation towards romanized script for gemma 2 9b it, gemma 2 9b and llama 2 13b models (see Figures 16 to 18).

K  Other Models: Mistral

We also perform our experiments on Mistral-7B (Jiang et al., 2023), a popular LLM known for its performance and efficiency. Layerwise distribution of romanized tokens for initial, final and all token generation steps are presented in Figure 19.
Figure 8: Logit lens illustration. We input Llama-2 13b model with a prompt translating ‘love’ from French to Greek. We visualize the output (χγάπη , ‘agape’ is the romanized form ) taking shape using logit lens producing a next-token distribution for each position (x-axis) and layers 20 and above (y-axis). Interestingly in the middle to top layers (20 - 29) we could observe romanized subwords of the Greek word ( γ - ga ; ἀπη - ape ; πη -pe ; η - e ) before they are represented in their native script. Color represents entropy of next-token generation from low (blue) to high (red). Plotting tool: (Belrose et al., 2023).

Figure 9: Logit lens illustration. We input Llama-2 13b model with a prompt translating ‘good’ from French to Ukrainian. We visualize the output ( добро , ‘dobro’ is the romanized form ) taking shape using logit lens producing a next-token distribution for each position (x-axis) and layers 20 and above (y-axis).‘Українська ’ (romanized as ‘Ukrayinska’) is the Ukrainian word for ‘Ukrainian’. Interestingly in the middle to top layers (20 - 29) we could observe romanized subwords of the Ukrainian words ( бро - bro ; ська - ska ) before they are represented in their native script. Color represents entropy of next-token generation from low (blue) to high (red). Plotting tool: (Belrose et al., 2023).
<table>
  <tr>
    <th>Output</th>
    <th></th>
    <th>τ</th>
    <th>η</th>
    <th>l</th>
  </tr>
  <tr>
    <td>38</td>
    <td>-</td>
    <td>τ</td>
    <td>man</td>
    <td>l</td>
  </tr>
  <tr>
    <td></td>
    <td>_time</td>
    <td>_time</td>
    <td>man</td>
    <td>an</td>
  </tr>
  <tr>
    <td>36</td>
    <td>_time</td>
    <td>_time</td>
    <td>man</td>
    <td>an</td>
  </tr>
  <tr>
    <td>34</td>
    <td>_time</td>
    <td>_time</td>
    <td>man</td>
    <td>an</td>
  </tr>
  <tr>
    <td>32</td>
    <td>_time</td>
    <td>_time</td>
    <td>man</td>
    <td>anim</td>
  </tr>
  <tr>
    <td>30</td>
    <td>_time</td>
    <td>_time</td>
    <td>ivot</td>
    <td>anim</td>
  </tr>
  <tr>
    <td>28</td>
    <td>_time</td>
    <td>_time</td>
    <td>ivot</td>
    <td>_Keith</td>
  </tr>
  <tr>
    <td>26</td>
    <td>_time</td>
    <td>_time</td>
    <td>ivot</td>
    <td>_time</td>
  </tr>
  <tr>
    <td>24</td>
    <td>_time</td>
    <td>_time</td>
    <td>ivot</td>
    <td>HR</td>
  </tr>
  <tr>
    <td>22</td>
    <td>_time</td>
    <td>_time</td>
    <td>_times</td>
    <td>_Heb</td>
  </tr>
  <tr>
    <td>20</td>
    <td>_time</td>
    <td>_time</td>
    <td>_times</td>
    <td>_Heb</td>
  </tr>
  <tr>
    <td></td>
    <td>_time</td>
    <td>_time</td>
    <td>_times</td>
    <td>_Heb</td>
  </tr>
  <tr>
    <td></td>
    <td>_time</td>
    <td>_time</td>
    <td>_time</td>
    <td>_Heb</td>
  </tr>
  <tr>
    <td></td>
    <td>_time</td>
    <td>_time</td>
    <td>_time</td>
    <td>_Heb</td>
  </tr>
  <tr>
    <td></td>
    <td>_time</td>
    <td>_time</td>
    <td>_Isra</td>
    <td>_Heb</td>
  </tr>
  <tr>
    <td></td>
    <td>_time</td>
    <td>_time</td>
    <td>_Isra</td>
    <td>_Heb</td>
  </tr>
  <tr>
    <td></td>
    <td>_time</td>
    <td>_time</td>
    <td>_Isra</td>
    <td>_Heb</td>
  </tr>
</table>

Input

Figure 10: Logit lens illustration. We input Llama-2 13b model with a prompt translating ‘time’ from French to Hebrew. We visualize the output (‘τ’, ‘zman’ is the romanized form ) taking shape using logit lens producing a next-token distribution for each position (x-axis) and layers 20 and above (y-axis). Interestingly in the middle to top layers (20 - 29) we could observe romanized subwords of the Hebrew word (‘τ’ - man ; ‘l’ - an ). Color represents entropy of next-token generation from low (blue) to high (red). Plotting tool: (Belrose et al., 2023).

<table>
  <tr>
    <th>Output</th>
    <th></th>
    <th>ب</th>
    <th>ا</th>
    <th>ب</th>
  </tr>
  <tr>
    <td>38</td>
    <td>-</td>
    <td>ب</td>
    <td>ا</td>
    <td>ب</td>
  </tr>
  <tr>
    <td></td>
    <td>_Bab</td>
    <td>_door</td>
    <td>abs</td>
    <td>ب</td>
  </tr>
  <tr>
    <td>36</td>
    <td>_gate</td>
    <td>_door</td>
    <td>ab</td>
    <td>abs</td>
  </tr>
  <tr>
    <td>34</td>
    <td>_gate</td>
    <td>_Bab</td>
    <td>AB</td>
    <td>_Bab</td>
  </tr>
  <tr>
    <td>32</td>
    <td>_gate</td>
    <td>_door</td>
    <td>_doors</td>
    <td>_doors</td>
  </tr>
  <tr>
    <td>30</td>
    <td>_gate</td>
    <td>_door</td>
    <td>_gates</td>
    <td>_gates</td>
  </tr>
  <tr>
    <td>28</td>
    <td>_gate</td>
    <td>_door</td>
    <td>_gates</td>
    <td>_gates</td>
  </tr>
  <tr>
    <td>26</td>
    <td>_gate</td>
    <td>_door</td>
    <td>_gates</td>
    <td>_gates</td>
  </tr>
  <tr>
    <td>24</td>
    <td>_gate</td>
    <td>_gate</td>
    <td>_gates</td>
    <td>_gates</td>
  </tr>
  <tr>
    <td>22</td>
    <td>_gate</td>
    <td>_gate</td>
    <td>_gates</td>
    <td>_gates</td>
  </tr>
  <tr>
    <td>20</td>
    <td>_gate</td>
    <td>_gate</td>
    <td>_gates</td>
    <td>_gates</td>
  </tr>
</table>

Input

Figure 11: Logit lens illustration. We input Llama-2 13b model with a prompt translating ‘door’ from French to Arabic. We visualize the output ( باب , ‘bab’ is the romanized form ) taking shape using logit lens producing a next-token distribution for each position (x-axis) and layers 20 and above (y-axis). Interestingly in the middle to top layers (20 - 29) we could observe romanized subwords of the Arabic word ( باب - bab; باب - ab) before they are represented in their native script. Color represents entropy of next-token generation from low (blue) to high (red). Plotting tool: (Belrose et al., 2023).
Figure 12: Logit lens illustration. We input Gemma-2 9b IT model with a prompt translating ‘music’ from French to Amharic. We visualize the output ( ሳታችል , ‘muzika’ is the romanized form ) taking shape using logit lens producing a next-token distribution for each position (x-axis) and layers 24 and above (y-axis). Interestingly in the middle to top layers we could observe romanized subwords of the Amharic word ( ሳታችል - muzy; እን - z ) before they are represented in their native script. Color represents entropy of next-token generation from low (blue) to high (red). Plotting tool: (Belrose et al., 2023).

(a) Layerwise distribution of romanized tokens averaged across output token generation steps and samples

(b) Layerwise distribution of romanized tokens in the first token generation step averaged across samples

(c) Layerwise distribution of romanized tokens in the last token generation step averaged across samples

Figure 13: Distribution of Romanized Tokens Across Model Layers: Analysis of First, Last, and All Generation Timesteps. This distribution is plotted across the last 10 layers of Gemma-2 9b IT model for repetition task and is averaged across 100+ samples. X-axes represents layer index, y-axes represents latent fraction i.e. the instances where romanized tokens occur with a probability > 0.1 averaged over samples for a specific layer. We plot the distributions for Gujarati (gu), Tamil (ta), Telugu (te), Hindi (hi) and Malayalam (ml).
(a) Layerwise distribution of romanized tokens averaged across output token generation steps and samples

(b) Layerwise distribution of romanized tokens in the first token generation step averaged across samples

(c) Layerwise distribution of romanized tokens in the last token generation step averaged across samples

Figure 14: Distribution of Romanized Tokens Across Model Layers: Analysis of First, Last, and All Generation Timesteps. This distribution is plotted across the last 10 layers of Gemma-2 9b IT model for cloze task and is averaged across 100+ samples. X-axes represents layer index, y-axes represents latent fraction i.e. the instances where romanized tokens occur with a probability > 0.1 averaged over samples for a specific layer. We plot the distributions for Gujarati (gu), Tamil (ta), Telugu (te), Hindi (hi) and Malayalam (ml), Georgian (ka) and Chinese (zh).

Gemma 2 9b        Llama 2 13b        Llama 2 7b

Translation

Repetition

Cloze

Figure 15: Layerwise fractional distribution of romanized tokens across output token generation timesteps. This distribution is plotted across the last 10 layers of Gemma 2 9b ,LLama 2 13b and Llama 2 7b models (columns) for (a) translation task from French, (b) Repetition task and is averaged across 100+ samples. X-axes represents layer index, y-axes represents latent fraction i.e. the fraction of timesteps where romanized tokens occur with a probability > 0.1 averaged over samples for a specific layer. We plot the distributions for Gujarati (gu), Tamil (ta), Telugu (te), Hindi (hi) and Malayalam (ml).
(a) fr → te romanized
(b) fr → gu romanized
(c) fr → hi romanized
(d) fr → te
(e) fr → gu
(f) fr → hi

Figure 16: Language probabilities for latent layers in translation from French to Telugu, Gujarati and Hindi in romanized (top row) and native scripts (bottom row) across various samples using the Gemma-2 9B IT model. On x-axes, layer index; on y-axes, probability (according to logit lens) of correct next token in a given language. Error bars represent 95% Gaussian confidence intervals over input. In translations to non-English languages in romanized scripts (top row), target representations emerge slightly earlier—approximately one to two layers before layer 40—compared to their native script counterparts (bottom row), which only begin to appear from layer 40 onwards.

(a) fr → te romanized
(b) fr → gu romanized
(c) fr → hi romanized
(d) fr → ml romanized
(e) fr → ta romanized
(f) fr → te
(g) fr → gu
(h) fr → hi
(i) fr → ml
(j) fr → ta

Figure 17: Language probabilities for latent layers in translation from French to five Indic languages (Telugu, Gujarati, Hindi, Malayalam, and Tamil) in romanized (top row) and native scripts (bottom row) using the Gemma-2 9B model. On x-axes, layer index; on y-axes, probability of correct next token in a given language. Error bars represent 95% Gaussian confidence intervals over input. In translations using romanized scripts (top row), target representations emerge approximately 1-2 layers earlier than their native script counterparts (bottom row).
(a) fr → hi romanized
(b) fr → ml romanized
(c) fr → ta romanized
(d) fr → hi
(e) fr → ml
(f) fr → ta

Figure 18: Language probabilities for latent layers in translation from French to Indic languages (Hindi, Malayalam, and Tamil) in romanized (top row) and native scripts (bottom row) using the Llama-2 13b model. On x-axes, layer index; on y-axes, probability of correct next token in a given language. Error bars represent 95% Gaussian confidence intervals over input. In translations using romanized scripts (top row), target representations emerge approximately 1-2 layers earlier than their native script counterparts (bottom row).

All tokens
First token
Last token
Translation
Repetition
Cloze

Figure 19: Distribution of Romanized Tokens Across Model Layers: Analysis of First, Last, and All Generation Timesteps. This distribution is plotted across the last 10 layers of Mistral-7B model for initial, final and all token generation steps (columns) for (a) translation task from English, (b) Repetition task, (c) Cloze task (rows) and is averaged across 100+ samples. X-axes represents layer index, y-axes represents latent fraction i.e. the fraction of timesteps where romanized tokens occur with a probability > 0.1 averaged over samples for a specific layer. We plot the distributions for Tamil (ta), Telugu (te), Hindi (hi), Georgian (ka) and Chinese (zh).

Reviewer paper summaries: The authors identify a phenomenon called Latent Romanization, where LLMs internally represent non-Roman script languages in a Romanized form before converting them to their native script. Using techniques like logit lens and activation patching, the study shows that Romanization acts as an implicit bridge between the model's language-agnostic concept space and language-specific output, facilitating multilingual processing.

The paper delved into how LLMs achieve robust multilingual capabilities focusing on non-Roman script languages. Specifically, the authors investigated the role of the representation of non-Roman scripts using transliterated Roman characters. They found that the intermediate layers frequently represent target words in Romanized form before transitioning to the native script, a phenomenon denominated Latent Romanization. Furthermore, the activation patching experiments demonstrate the possibility that LLMs encode semantic concepts similarly across native and Romanized scripts. Moreover, they found that when the target language is in Romanized transliteration, its representations emerge earlier in the model’s layers than when using native script.

This work hypothesizes that LLMs leverage Romanized forms of non-Roman script languages as an intermediate bridge between their language-agnostic concept space and language-specific output representations. To verify their hypothesis, the authors apply two probing techniques, including logit lens and activation patching. Experiments are conducted on seven languages including five Indic languages, Chinese, and Georgian. Experimental results show that except for Chinese, LLMs do exhibit latent romanization during text completion tasks, which provides new insight to understand the origin of multilingual capabilities of LLMs.
Summary of strengths: - It is interesting to see how models switch between native and romanized script internally in certain layers (latent romanization). This is a novel insight into a phenomenon that enhances our understanding of LLMs' multilingual processing, and is a nice extension of the Wendler et al. study.
- The paper uses mechanistic interpretability techniques (logit lens, activation patching) to empirically validate findings.

- Although most are Indic, the authors addressed many non-Roman script languages to support their claims.
- They performed experiments on various tasks in diverse experimental settings.

1. This work conducts analysis from a different perspective, namely, analyzing whether LLMs leverage Romanized forms of non-Roman script languages as an intermediate bridge between their language-agnostic concept space and language-specific output representations.

2. Specifically, they find a novel observation that when processing non-Roman script languages, LLMs form internal representations of target tokens in earlier layers for romanized script compared to native script. This observation shows that the romanized form may be an intermediate step between abstract concepts and native languages.
Summary of weaknesses: - Please clarify an important point about the further implications of this paper: You are saying that LLMs encode semantic concepts similarly, regardless of the script, indicating a shared internal representation, correct? But do you see this as separate from the broader goal of token embeddings to represent similar concepts similarly regardless of the language (or script)? 
- Additionally, do you think that better tokenisation (as mentioned in the Limitations) would remove the effect of romanised tokens appearing earlier than their native counterparts?

- The writing and overall presentation of the paper needs improvement (see Comments Suggestions and Typos). Consider a tool like Grammarly and please export figures as PDFs for better quality.

- The study is limited to a small number of languages (Indic languages, Chinese, Georgian), which may not generalize to all non-Roman script languages.

- I wonder if using transliteration as input instead of native script leads to performance improvements on the three tasks (translation, repetition, and cloze task).
- This study used non-Roman script languages as the target language. I wonder how their use as the source language could impact the outcome.
- ll.476–477: In what way do you consider the repetition task syntactic?
- ll.478–481: To support the hypothesis, you would need to inspect the earlier and intermediate layers directly.
- ll.481–482: Could you elaborate on this point to clarify the relationship between the observed behavior and language-specific neurons?
- l.490: Why is Italian chosen as the target language instead of English in this experiment?
- l.537: What does "English representations" refer to? English subwords or Roman transliteration?
- Figure 7: Using Romanized scripts, one can observe a slight probability increase across layers 20–30. Do you have any interpretation of this phenomenon?
- ll.563–565: Do you have any idea why logographic Chinese characters do not lead to Latent Romanization? Would it be relevant for Chinese characters to represent syllables, unlike the Indic and Georgian characters, which are phonemic, and for the number of Chinese characters to be much higher than that of these languages? If you did not observe Latent Romanization for Chinese, does it mean no Roman character strings appeared across the layers?

1. It is difficult to follow since the background of romanization is missing. An example of romanization is needed (e.g., how to romanize a detailed English sentence), even though the authors have introduced the definition briefly in Lines 059-063. 

2. Rather than using English-dominant models like Llama-2, using some models with better multilingual capabilities like Llama-3 and Qwen-2 to analyze would provide more insights for future researchers.

3. A thorough proofreading is needed. Detailed information can be referred to in the following section.
Comments, suggestions and typos: Please export your figures as PDFs instead of PNGs or JPGs. It would make for much better quality of the graphics.

Figures 1 and 2: 
- please pay attention to whitespaces in the figure captions (e.g. whitespaces before brackets, missing whitespaces after full stops, …)
- add titles for x- and y-axes. The y-axis is the layer number, but this is only apparent by reading the figure caption. It should be self-explanatory by looking at the figure.

Figure 5:
- Same issue with missing x- and y-axis titles.
- Why are “Native”, “Romanized” etc. big titles added within Latex? They should be part of the figure itself (or, you can make Latex subfigures and add small captions like “a) Native”, “b) Romanized” etc.)
- The KL Divergence on the right side should either be included inside the subfigures or only stated in the Figure 5 caption.

- Line 211: typo “5-shots” -> “5-shot”
- Figure 3: your mouse is visible in the screenshot (left side, “Latents at objects last token”)
- Lines 414+415: please put the citations of pypinyin and Unidecode in full brackets (including the author name)
- Lines 435+436: unnecessary, you can delete this statement

I found many typos and incorrections, as enumerated below:

- title: Lowercase "of" and "in".
- l.7: ", we" > "and"?
- caption Figure 1: Capitalize "b" in "7b" > 7B. passim.
- caption Figure 1: Insert a space between a period and "Interestingly" in l.3. passim.
- caption Figure 2: "LLama" > "Llama"; Use quotation marks for "flower".
- l.139: "non-Roman language" > "non-Roman script language". passim.
- l.179: Put a comma after "same".
- ll.213–214: The order of the elements does not coincide with that of the preceding formulation.
- caption Figure 3: Put a closing quotation mark to German "Elefant"; You might want to eliminate the cursor left in the middle; "objects" > "objects'".
- l.227: "the" > "of the"
- caption Figure 3: You should include the description "The mean residuals at each layer are computed" in the main text body; "will dominated" > "will be dominated".
- l.232: "woul" > "would"
- ll.229–233: You should explain why n_s and n_S coincide in this case.
- l.243: "encoding" > "encoded"?
- l.266: Spell out "5".
- l.285: "e.g." > "e.g.," passim.
- l.366: Would it be "target output language l^out_S"?
- ll.414–415: "Mozillazg (2024)" > "(Mozillazg 2024)"; "Šolc (2025)" > "(Šolc 2025)".
- l.431: Using IndieXlit, did you only consider Indic languages?
- l.467: The numbering of Figure 5 and Figure 6 should be inverted following their appearance.
- Figure 7: Is the translation from French (caption) or English (legend)? If it is from French, is there any reason why you chose it?
- l.533: Indicate that the Figures 16–18 are found in Appendix J.
- l.537: "(Wendler et al., 2024)" > "Wendler et al. (2024)".
- l.562: Replace the semicolon with "and".
- l.574: "to" > "into".
- l.605: "non-roman" > "non-Roman".
- l.625: "eg:" > "e.g.,".
- caption Figure 8: "Agape" > "agape". The first letter in the Greek word is a small one.
- caption Figure 9: "Dobro" > "dobro". The first letter in the Ukranian word is a small one. Remove "(" before "Українська ’". The order of input does not coincide with that of the caption. Could you explain why the two words are fed jointly in this case?
- caption Figure10: Hebew words are written from right to left as in Arabic. However, it is from left to right in the text.
- caption Figure 11: The transliteration should be "bab" instead of "Bab" in l.2 (Arabic has no small/capital letter distinction); "ba" instead of "ab" in l.4.

## Presentation Improvement
- All figures should be created as vector illustrations to ensure printing quality.
- In Lines 403-405, it would be better to unify the style choosing 7b or 7B.

## Typos
- In Line 190, '... and a softmax is ...' -> '... and a softmax layer/activation function is ...'.
- In Lines 358 and 360, i.e -> i.e.
- In Line 405, 'Mistral- 7B' -> 'Mistral-7B'.
Overall assessment scores: 3

3.5

3.5
Metareview score: 3
Metareview: This paper investigates how LLMs use Romanized forms of non-Roman script languages by using mechanistic interpretability techniques.
presents two main interesting findings about the presence of Romanized forms of words for non-Roman script languages in the inner layers of multilingual LLMs. One finding is that LLMs use Romanization as a bridge for non-Roman scripts, before switching to native scripts in the latest layers. Another finding is that LLMs encode semantic concepts similarly across native and Romanized forms, possibly indicating a shared internal representation of meaning in both scripts of the same language.
Although these findings are very interesting thus the overall score is 3, as mentioned by multiple reviewers, this paper remains a few serious concerns to be accepted. We recommend to resubmit this paper with significant improvement of the writing and overall presentation.
Summary of reasons to publish: This paper presents two main interesting findings about the presence of Romanized forms of words for non-Roman script languages in the inner layers of multilingual LLMs. One finding is that LLMs use Romanization as a bridge for non-Roman scripts, before switching to native scripts in the latest layers. Another finding is that LLMs encode semantic concepts similarly across native and Romanized forms, possibly indicating a shared internal representation of meaning in both scripts of the same language.