Venue: NAACL 2024

# Measuring Cross-lingual Transfer in Bytes

Leandro Rodrigues de Souza

FEEC, UNICAMP, Brazil

&Thales Sales Almeida

IC, UNICAMP, Brazil

Maritaca AI, Brazil &Roberto Lotufo

FEEC, UNICAMP, Brazil

NeuralMind, Brazil &Rodrigo Nogueira

FEEC, UNICAMP, Brazil

Maritaca AI, Brazil

###### Abstract

Multilingual pretraining has been a successful solution to the challenges posed by the lack of resources for languages. These models can transfer knowledge to target languages with minimal or no examples. Recent research suggests that monolingual models also have a similar capability, but the mechanisms behind this transfer remain unclear. Some studies have explored factors like language contamination and syntactic similarity. An emerging line of research suggests that the representations learned by language models contain two components: a language-specific and a language-agnostic component. The latter is responsible for transferring a more universal knowledge. However, there is a lack of comprehensive exploration of these properties across diverse target languages. To investigate this hypothesis, we conducted an experiment inspired by the work on the Scaling Laws for Transfer. We measured the amount of data transferred from a source language to a target language and found that models initialized from diverse languages perform similarly to a target language in a cross-lingual setting. This was surprising because the amount of data transferred to 10 diverse target languages, such as Spanish, Korean, and Finnish, was quite similar. We also found evidence that this transfer is not related to language contamination or language proximity, which strengthens the hypothesis that the model also relies on language-agnostic knowledge. Our experiments have opened up new possibilities for measuring how much data represents the language-agnostic representations learned during pretraining.1

Footnote 1: The code used in our experiments is publicly available at [https://github.com/lersouza/language-transfer](https://github.com/lersouza/language-transfer). We rely on the mcC4 dataset from Huggingface, available at [https://huggingface.co/datasets/mc4](https://huggingface.co/datasets/mc4)

+
Footnote †: Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 7526–7537 June 16-21, 2024 ©2024 Association for Computational Linguistics

## 1 Introduction

The emergence of self-supervised pretraining models such as BERT has revealed a notable phenomenon of cross-lingual transfer even when these models are trained on multilingual corpora devoid of paired translation examples. For example, LLAMA Touvron et al. (2023), which was trained self-supervisedly on an English-centric corpus, exhibits surprising multilingual capabilities Yuan et al. (2023); Ye et al. (2023). The underlying mechanisms driving this behavior remain unclear, with hypotheses ranging from the presence of shared "anchor" tokens Pires et al. (2019) to language contamination Blevins and Zettlemoyer (2022), yet no scientific consensus has been reached.

Research in this area often involves the use of pre-existing language models (LMs), which are subsequently finetuned on supervised datasets in different languages de Souza et al. (2021); Yuan et al. (2023). However, when evaluating multiple languages, conventional methodologies encounter two significant challenges: firstly, the dependence on supervised finetuning datasets, which often vary in size and quality, complicating cross-lingual comparisons; secondly, the use of subword tokenizers, which do not represent all languages equally.

In this work, we avoid these problems by working with a byte-level tokenizer and by using auto-regressive language models trained in self-supervised from scratch in one language and then finetuned on another. To measure the effect of transfer learning, we employ the concept of data transfer Hernandez et al. (2021), which allows us to quantify how much each different source language contributes to the perplexity of the target language.

Our main contribution is providing a method that measures how much knowledge, in bytes, is transferred from one language to another. By applying it, our findings reveal a surprising trend: even when comparing linguistically distant languages, the data transfer metrics are of a comparable magnitude. This research contributes additional evidence supporting the language-agnostic hypothesis, which suggests that the internal representations developed

by a model are not only influenced by the linguistic surface form but also by the cultural and semantic content of the training data.

## 2 Related Work

Prior work attributed the success of multilingual models in cross-lingual transfer to "anchor" tokens Pires et al. (2019). However, subsequent research demonstrated that models could perform well even without these tokens Artetxe et al. (2020), highlighting the significance of shared parameters during training Conneau et al. (2020). Competitive results were achieved by monolingual models with minimal or no adaptation Artetxe et al. (2020); de Souza et al. (2021).

Investigations by Blevins and Zettlemoyer (2022) linked these findings to language contamination, where pretraining datasets contained target language data. Additional factors contributing to cross-lingual transfer success include dataset statistics, language attributes Lin et al. (2019), language structure Lin et al. (2019); Papadimitriou and Jurafsky (2020); Chiang and yi Lee (2020); Ri and Tsuruoka (2022), and token overlap between training and target languages Beukman and Fokam (2023). The role of language script Fujinuma et al. (2022) and model tokenizer Rust et al. (2021) was also noted, prompting the use of a byte tokenizer to address these issues Xue et al. (2022); Abonizio et al. (2022).

Recent research proposed a two-component model representation hypothesis--language agnostic and language specific de Souza et al. (2021); Zeng et al. (2023); Wu et al. (2022). While promising, no study has measured how much of the language-agnostic component is used in settings with multiple source and target languages. Additionally, existing research still applies the source language vocabulary to the target language, potentially compromising input representations and affecting results.

To address these gaps, we draw on Hernandez et al. (2021) and employ a byte vocabulary in our experiments to overcome current literature limitations.

## 3 Methodology

Inspired by Hernandez et al. (2021), our methodology focuses on quantifying the transferability of pretraining data across distributions, particularly between different languages. We select a **target** language and finetune models initialized from various **source** languages onto it. Subsequently, we evaluate each model on a **target** language test set and compare their performance. We introduce the Data Transfer (\(D_{T}\)) metric to estimate knowledge transfer, explained in more depth in 3.1. This process is repeated across different **target** languages to observe effects across a broad linguistic spectrum.

These experiments aim to quantify and compare cross-lingual knowledge transfer from different source languages. This analysis seeks to uncover the extent to which transferability depends on specific languages and the importance of language-agnostic components in learned representations. The following sections provide further details.

### Data Transfer Estimation

To quantify the knowledge transfer from pretrained models to a given target language, we utilize the Data Transfer (\(D_{T}\)) metric. This metric assesses the effectiveness of pretraining data by measuring the additional tokens required in the target language for a model initialized from scratch to match the performance of a model pretrained on a source language and finetuned on the target. Figure 1 illustrates this concept.

In our experiments, we utilize a set of \(M\) different dataset sizes, denoted as \(\mathbf{D_{F}}=\{s_{0},s_{1},...,s_{m}\}\), in the target language. Initially, we train a random-initialized model on these datasets, resulting in a set of perplexities \(\mathbf{P_{R}}=\{p_{r,0},p_{r,1},...,p_{r,m}\}\). Subsequently, we train from scratch another model on a fixed amount of tokens (e.g., 6B) in a source language \(\ell\), and then finetune it on the target language using the same dataset sizes \(\mathbf{D_{F}}\), generating another set of perplexities \(\mathbf{P_{\ell}}=\{p_{\ell,0},p_{\ell,1},...,p_{\ell,m}\}\).

To estimate the Data Effective metric (\(D_{E}\)), which represents the amount of data needed to achieve a certain performance, we utilize a linear interpolation function \(\gamma(y^{\prime},\mathbf{X},\mathbf{Y})\). This function interpolates between discrete data points \((x_{j},y_{j})\in X\times Y\), evaluated at \(y^{\prime}\). In our case, the calculation is expressed as:

\[D_{E,i}=\gamma(p_{\ell,i},\,\mathbf{D_{F}},\,\mathbf{P_{R}}) \tag{1}\]

Here, \(D_{E,i}\) signifies the Data Effective metric for the \(i\)-th perplexity value in \(\mathbf{P_{f}}\). \(\mathbf{P_{R}}\) denotes the set of perplexity values derived from the random-initialized model, while \(\mathbf{D_{F}}\) represents the dataset sizes employed during finetuning 

We utilize the linear interpolation function provided by the NumPy library to approximate \(D_{E,i}\). Further details can be found in the NumPy documentation.2

Footnote 2: See [https://numpy.org/doc/stable/reference/generated/numpy.interp.html](https://numpy.org/doc/stable/reference/generated/numpy.interp.html)

Finally, the Data Transfer metric is computed by subtracting the \(i\)-th dataset size \(s_{i}\) from \(D_{E,i}\):

\[D_{T,i}=D_{E,i}-s_{i} \tag{2}\]

By subtracting the dataset size from \(D_{E,i}\), we account only for the data coming from pretraining. Since a byte vocabulary is utilized, the amount of data transferred is measured in **bytes**.

### Task and Evaluation Metric

We adopt Language Modeling as our main task with perplexity as the performance metric for all experiments. Perplexity, derived from the model's loss (\(e^{loss}\)), facilitates future predictions of model behavior in transfer learning scenarios, following the approach in Hernandez et al. (2021). This choice also allows extensive experiments across multiple languages, leveraging datasets like mC4 Xue et al. (2021) to overcome size limitations inherent in supervised datasets.

### Tokenization Impact

In cross-lingual setups, the choice of tokenization method holds considerable significance Rust et al. (2021). While subword tokenizers are commonly employed in cross-lingual experiments, using a tokenizer trained in a source language on a distant target language may result in an increased number of tokens. This can lead to the utilization of undertrained embeddings in some instances, introducing challenges for effective sentence representation. Furthermore, dealing with different scripts introduces the issue of numerous "unknown" tokens, exacerbating the difficulty of obtaining suitable input representations for the model.

To address these challenges, we opt for a byte vocabulary based on the approach proposed by Xue et al. (2022), which allows us to standardize representations across all languages, ensuring that each model encounters the same quantity of UTF-8 bytes. By doing so, we mitigate the use of unknown tokens and undertrained embeddings, thereby minimizing the impact of tokenization issues on the performance of our experiments.

### Language Contamination

A potential reason for a pretrained model's superior performance in cross-lingual tasks is the presence of a substantial amount of data in the target language in its pretraining dataset, a phenomenon referred to as language contamination. To quantify this impact, following the approach outlined by blevins-zettlemoyer (2022), we examine the rates of target language fragments in the source language dataset and vice versa.

Specifically, for a given source language \(\ell\) and target language \(t\), we calculate the ratio of all lines

Figure 1: Example illustrating how the coeficients \(D_{T}\), \(D_{F}\) and \(D_{E}\) are calculated. Each series represents a different initialization. \(D_{T}\) is the number of additional tokens in the target language that a from-scratch model would have needed to achieve the same perplexity of a model finetuned from English. \(D_{F}\) is the size of the dataset used for finetuning and \(D_{E}\) accounts for all data, both \(D_{F}\) and \(D_{T}\).

 

classified as \(\ell\) in the target dataset (known as contamination on target) and as \(t\) in the pretraining dataset (known as contamination on source). We perform language detection using the _fasttext_ tool Bojanowski et al. (2017), employing a threshold of 0.6 for classification.

Next, we compute the Spearman correlation between the set of Data Transfer metrics \({\bf D}_{\mathbf{T}}\) and those ratios obtained from the outcomes of our experiments.

Correlating these rates with the model's data transfer indicator allows us to evaluate the impact of language contamination on model performance.

### Language Similarity

Language similarity is often cited as a crucial factor influencing cross-lingual transfer performance in natural language processing tasks. In this work, we aim to investigate the relationship between language similarity and cross-lingual transfer effectiveness based on the outcomes of our experiments.

To explore this relationship, we measure various distances between languages, including syntactic, geographic, and phonological distances. These distances are calculated based on the methodology proposed by Littell et al. (2017).

We aim to correlate these language distances with the data transfer metric (\(D_{T}\)). By employing Spearman correlation analysis, we seek to discern whether there exists a significant correlation between the observed \(D_{T}\) values and the measured language distances.

This analysis elucidates whether our experimental results can be attributed to the similarities between the languages involved in our cross-lingual experiments.

## 4 Experiments

This section presents the languages, datasets, model architecture, and training details for our experiments.

### Languages

**Source Languages Selection**. We chose three diverse languages--English, Russian, and Chinese--for the source language during the pretraining phase. This selection ensures a broad linguistic spectrum while adhering to pretraining budget constraints.

**Target Languages Selection**. Ten target languages, spanning various language families and different scripts, were chosen to establish a diverse cross-lingual setting. Details, including language codes, are provided in Table 1.

### Datasets

For training and finetuning, language subsets from the mC4 dataset Xue et al. (2021) for the selected languages were utilized.3 Pretraining datasets comprised approximately 6 billion tokens, while fine-tuning datasets ranged from 6 million to 6 billion tokens. Documents were sampled at random without replacement until the desired amount of tokens was reached.

Footnote 3: See [https://huggingface.co/datasets/mc4](https://huggingface.co/datasets/mc4) for more details.

### Model Architecture

Our model is a decoder-only Transformer Vaswani et al. (2017) that uses a byte vocabulary with 256 embeddings with a dimension of 640. The model follows closely the implementation provided in the T5x library. It consists of 10 layers, each having 10 attention heads with dimensions of 64. The intermediate dimension of Multi-Layer Perceptron (MLP) has a dimension of 2560 and GELU Hendrycks and Gimpel (2023) activations. The parameters of the embeddings matrix and the final dense layer are shared. We use relative positional embeddings Shaw et al. (2018). The resulting model has approximately 65 million parameters.

### Training details

Models were trained using a causal language modeling objective. Each batch has 512 sequences of 1024 tokens. We use the AdamW optimizer with an initial learning rate of 2e-4, which decayed to 2e-5 through cosine decay following Hoffmann et al. (2022). Finetuning employed a constant learning rate of 2e-5 over 10 epochs, except for the 6 billion dataset size where we limited it to 3 epochs.

\begin{table}
\begin{tabular}{c c l l} \hline \hline
**Code** & **Language** & **Family** & **Script** \\ \hline ar & Arabic & Afro-Asiatic & Arabic \\ en & English & Indo-European & Latin \\ es & Spanish & Indo-European & Latia \\ zh & Chinese & Sino-Thebeta & Hanzi \\ fi & Finnish & Uralic & Latin \\ de & German & Indo-European & Latin \\ ko & Korean & Koreanic & Hangul \\ id & Indonesian & Austronsen & Latin \\ ja & Japanese & Japonic & Kanji, Hirgana, Katakana \\ ru & Russian & Indo-European & Cyillic \\ \hline \hline \end{tabular}
\end{table}
Table 1: Characteristics of selected target languages.

 

This adjustment was based on preliminary experiments indicating that the model tends to overfit beyond this epoch count in larger datasets. The best model was selected based on the lowest perplexity achieved on the development set. Warmup steps varied with finetuning dataset sizes (ranging from 0 for smaller datasets to 3000 for larger ones), aligning with findings that smaller datasets completed finetuning before warmup completion Hernandez et al. (2021). We utilized the T5X framework Roberts et al. (2022) for our experiments. We used a total of 600 hours of a TPU v2-8 (seven hours of pretraining per model, and fifteen hours for the largest finetuning).

## 5 Results

Results are compiled in Table 2, where we exclusively report instances involving different source and target languages.

Given our methodology, where we vary the source language while keeping the target language constant to assess the impact of pretraining language in cross-lingual scenarios, it is essential to read the table vertically unless stated otherwise. Each row represents the results obtained by finetuning the model from a specific source language to a given target language (indicated in the column). This vertical arrangement facilitates the comparison of model performance across different source languages for the same target language. Furthermore, test sets vary significantly for each language due to the nature of the mC4 dataset. Consequently, results across target languages are not comparable.

Throughout this section, we highlight findings from models finetuned on 6 million tokens (i.e., \({\bf D_{F}=\{6MB\}}\)) unless otherwise specified.4 This extreme scenario tests models with minimal target language resources.

Footnote 4: This restriction applies only to the finetuned models. For the from-scratch ones, we need their perplexities on multiple target language dataset sizes to estimate \(D_{T}\).

### Performance with different initializations

We delve into the results of three target languages: Spanish, Arabic, and Japanese. The perplexity scores across all dataset sizes for these languages are highlighted in Figure 2, offering a chance for in-depth analysis despite the constraints of space.

A key observation is the consistent proximity of perplexity values for all three source languages in every target language. For instance, while one might expect a significant advantage for Chinese as a source language when finetuned in Japanese, or for English when paired with Spanish, this is not the case. This suggests the model leverages source language representations even when it lacks significant similarity with the target language.

Taken together, our results indicate that the model can rely on representations beyond those capturing language structure. This observation supports the recent hypothesis that these representations encompass both language-specific and language-neutral components, strengthening the latter as an important aspect.

### Data Transfer estimation for target languages

Analysis of the Data Transfer (\(D_{T}\)) metric in Table 2 reveals that values are consistently close across different source languages for a given target. Notable examples include **Arabic** (en: 101MB, ru: 99MB, zh: 90MB), **Japanese** (en: 47.5MB, ru: 47.8MB, zh: 69.48MB), and **Finnish** (en: 102.62MB, ru: 51.32MB, zh: 76.57MB).

A clear pattern emerges: one initialization stands out for most target languages, while the other two show similar \(D_{T}\) values. This pattern is visually represented in Figure 3, particularly evident for Finnish (fi), Indonesian (id), Japanese (ja), Korean (ko), and Chinese (zh). Additionally, closely clustered results are observed for Arabic (ar), German (de), Spanish (es), and Russian (ru).

With three diverse initializations, the fact that two consistently show similar \(D_{T}\) values, even when distant from the target language (e.g., Russian and Chinese for Finnish), suggests the models leverage language-agnostic representations. This observation aligns with the expected behavior, where \(D_{T}\) would be close to zero if language-specific knowledge were dominant.

English (en) demonstrates effective knowledge transfer to most target languages, potentially due to its widespread presence in corpora across languages. This hypothesis is explored further in Section 5.3. Additionally, Chinese (zh) appears to transfer effectively to Japanese (ja) and Korean (ko), likely due to their linguistic proximity.

We observe that Chinese (zh) tends to transfer effectively to Japanese (ja) and Korean (ko), both of which are considered closer languages. This, together with our other observations, indicates that, while not determinant, language-specific component also plays a role in cross-lingual transfer.

Finally, upon examination of Figure 4, we also 



\begin{table}
\begin{tabular}{c c c c c c c c c c c c} \hline \hline
* & 3.16 & 3.57 & 2.61 & 3.92 & 3.58 & 2.44 & 4.43 \\  & \(D_{T}\) & **101.02** & **95.25** &
- & **121.14** & **76.57** & **102.62** & 47.50 & 48.74 & **75.64** & **29.21** \\ \hline Russian & Perplexity & 2.83 & 3.98 & 3.66 & 3.47 & 3.80 & 2.84 & 3.89 & 3.58 &
- & 4.52 \\  & \(D_{T}\) & 99.00 & 47.87 & **174.63** & 67.88 & 50.96 & 51.32 & 47.81 & 48.69 &
- & 26.18 \\ \hline Chinese & Perplexity & 2.88 & 4.26 & 3.89 & 3.75 & 3.98 & 2.98 & 3.46 & 3.48 & 2.72 &
- \\  & \(D_{T}\) & 90.63 & 31.76 & 66.96 & 50.27 & 49.65 & 50.21 & **69.48** & **49.88** & 48.47 &
- \\ \hline \hline \end{tabular}
\end{table}
Table 2: Results for Perplexity and Data Transfer (in MB) for all target and source languages. All metrics are reported after finetuning the models in 6 million tokens of the target language.

Figure 3: Dispersion chart for Data Transfer (\(D_{T}\)) across target languages. Each series corresponds to a distinct source language. The first dashed line (top-to-bottom) indicates the average of the best results (higher transfer), while the second one represents the average of the worst results (lower transfer).

Figure 2: Results measured in Perplexity per token for three target languages. Each series represents a different initialization: train from scratch, finetune from an English, Chinese, or Russian model.



observe that most \(D_{T}\) values are clustered between 50MB and 100MB, with low variation within a source language across all target languages. While not ideal, this comparison across target languages indicates that the transfer from our source models is relatively consistent, not spanning more than one order of magnitude.

### Language Contamination Impact

Table 3 summarizes the results of assessing the language contamination effect in our experiments.

The analysis excludes the 6 billion tokens fine-tuning dataset size to mitigate the ossification effect, as observed in Hernandez et al. (2021). This effect leads to a performance drop for pretrained models with larger finetuning datasets, worsening perplexity compared to scratch-trained models. Excluding this data point helps avoid introducing noise and adverse effects on coefficient calculation, given its singular occurrence per source-target language pair. Additionally, due to the sample size (< 500 observations), the permutation test is utilized to calculate the _p-value_.

Although a correlation of 0.191 exists between \(D_{T}\) and contamination on the source dataset, this coefficient indicates a weak association, suggesting a minimal impact on cross-lingual performance, contradicting the findings by Blevins and Zettlemoyer (2022).

Exploration of language contamination in target datasets reveals a higher correlation of 0.265, particularly influenced by widespread languages like English. However, this coefficient still signifies a weak association between \(D_{T}\) and target contamination, thus not supporting the language contamination hypothesis.

### Language Similarity and Data Transfer

This subsection outlines our analysis of the correlation between language distances and the data transfer metric (\(D_{T}\)), summarized in Table 4.

We find a weak correlation between source-target language distances and Data Transfer. Since we are considering multiple language characteristics, such as syntax and phonology, the results suggest that language similarity has a minor role in knowledge transfer between distinct languages. Nonetheless, the small number of source languages necessitates a cautious interpretation of these results, especially since all obtained p-values exceed 0.7, indicating limited statistical significance.

Because of that, we conducted a controlled experiment, pretraining a language model in Portuguese and evaluating its performance on the Spanish target language -- a language known for its similarity to Portuguese. Results, depicted in Table 5, were compared across various initializations, including more distant languages like Chinese.

When initialized with Portuguese, the model achieves a lower Perplexi

Figure 4: Boxplot with Data Transfer results for the 6 million tokens datasets in all target languages.

\begin{table}
\begin{tabular}{|c|c|c|} \hline
**Measure to correlate with \(D_{T}\)** & \(\rho\) & _p-value_ \\ \hline Syntactic distance & -0.147 & \\ Geographic distance & -0.110 & \\ Phonological distance & -0.117 & \\ Genetic distance & -0.150 & \\ Inventory distance & -0.090 & \\ Featural distance & -0.145 & \\ \hline  & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} \\ \end{tabular}
\end{table}
Table 4: Spearman Correlation (\(\rho\)) and p-value assessing the correlation of \(\mathbf{D_{T}}\) with a diverse set of language distance measurements.

 

to when initialized with other languages. Additionally, \(D_{T}\) peaks among all initializations, suggesting the influence of language proximity between Portuguese and Spanish. English initialization also yields comparable results, with a \(D_{T}\) difference of around 40 MB and a perplexity variation of 0.25. Chinese and Russian show the lowest, yet similar, scores.

One possible interpretation is that the language-agnostic component accounts for 50% of the transfer, with Russian and Chinese being more distant from Spanish, while the language-specific component contributes the remaining 50%, considering closer linguistic and script systems. However, further investigation with more language pairs is necessary to determine the actual factors influencing transfer performance, including linguistic structure, script, or shared cultural knowledge in pretraining datasets.

### Commutative property exploration

We examine the commutative property of data transfer between English (en), Russian (ru), and Chinese (zh) in our cross-lingual experiments [Table 6]. Notably, the data transfer amounts exhibit non-commutative behavior, revealing variations in knowledge transfer efficiency across bidirectional language pairs.

In the English-to-Russian transfer (en, ru), data transfer is more efficient when directed from Russian to English [174.63] compared to the reverse direction [75.64], indicating an asymmetry in knowledge transfer. Similarly, in the English-to-Chinese transfer (en, zh), data transfer is more substantial from English to Chinese [66.96] than in the reverse direction [29.21].

The Russian to Chinese transfer (ru, zh) also demonstrates a non-commutative pattern, with higher data transfer from Russian to Chinese [48.47] than in the reverse direction [26.18].

The variance in mC4 subsets for each language introduces significant differences in both pretraining and evaluation datasets, potentially contributing to the absence of a commutative behavior. A more in-depth analysis would necessitate repeating experiments with equivalent datasets.

## 6 Discussion

Our study aims to measure how much knowledge, in bytes, is transferred from one language to another, enabling the investigation of the effectiveness of language-agnostic representations acquired during pretraining in cross-lingual scenarios. We hypothesize that these representations enable models to perform well on downstream tasks across diverse languages, which is observed in state-of-the-art multilingual models.

In our results, we consistently find that at least two source languages demonstrate very close \(D_{T}\) values when evaluated against a target language, despite the diverse set of script systems and linguistic characteristics involved. This observation suggests that the data transferred from these languages to the target language is not primarily related to language-specific components but rather to language-agnostic ones. For instance, as illustrated in Figure 3, both English and Russian, despite being known as distant languages, achieve nearly identical \(D_{T}\) values when evaluated on Korean, a language distinct from either. Moreover, all three source languages are remarkably close when evaluated on the Japanese test set.

Despite exposure to only a few tokens in the target language, our models demonstrate similar perplexity performance, indicating high adaptability and generalization across a broad range of languages. This reinforces the notion that the language-agnostic component plays a crucial, uniform role across source languages.

Notably, our results are not attributed to pretraining exposure to target languages, since there is a weak correlation of language contamination with the data transfer coefficient. Additionally, the observed performance is not solely dependent on language proximity, as suggested in other works.

While perplexity offers valuable insights, generalizable conclusions require evaluation in downstream tasks, especially in under-resourced languages. To explore this further, we conducted a small-scale experiment, detailed in Appendix A, finetuning our pretrained models for a low-resource language inference task. Surprisingly, we observe comparable accuracy scores between Portuguese and Russian, with good results also for English

\begin{table}
\begin{tabular}{|c|c|c|c|} \hline
**Pair (\(L_{1}\), \(L_{2}\))** & \(L_{1}\to L_{2}\) & \(L_{2}\to L_{1}\) & \(\Delta\) \\ \hline en, ru & 75.64 & 174.63 & 98.99 \\ en, zh & 29.21 & 66.96 & 37.75 \\ ru, zh & 26.18 & 48.47 & 22.29 \\ \hline \end{tabular}
\end{table}
Table 6: Analysis of the Commutative Property in terms of Data Transfer \(D_{T}\). We analyze pairs of languages (\(L_{1}\), \(L_{2}\)), reporting the observed \(D_{T}\) from \(L_{1}\) to \(L_{2}\) and vice-versa. Values are reported in megabytes.

 

and Chinese, suggesting that our initial findings with perplexity may extend broadly. However, a more detailed investigation is required to understand cross-lingual transfer mechanisms fully.

The novelty of our approach is employing a byte-level tokenizer and adapting Hernandez et al. (2021) for a cross-lingual scenario. The byte-level approach facilitates consistent model embeddings across diverse scripts, enabling effective cross-lingual knowledge transfer without language-specific tokenization or preprocessing. This is supported by the strong performance of ByT5 compared to mT5 in Xue et al. (2022).

In conclusion, our study suggests the presence of language-agnostic representations contributing to cross-lingual transferability, while also laying the foundation to measure it through the Data Transfer metric. The observed consistency in model performance across diverse languages, facilitated by the byte-level tokenizer, indicates the potential for more efficient and generalizable natural language understanding across linguistic boundaries in computational linguistics and NLP.

## 7 Limitations

Our study has certain limitations that merit consideration. Firstly, our choice of initializing models with only three languages, while diverse, leaves room for improvement. Including additional languages in the pretraining phase would enhance the robustness of our analysis by minimizing possible bias towards the selected languages while providing more samples for our correlation analysis. However, this expansion would necessitate a more substantial computational budget.

Secondly, our reliance on small models, specifically a 65 million parameter model, limits the scope of our findings as larger models may exhibit different behavior. Additionally, the capacity of very large models for few-shot learning opens avenues for further exploration in the domain of transfer learning.

Lastly, the heterogeneity of the mC4 dataset across languages introduces a potential source of variability in the models' exposure to different knowledge. While the impact of this variation on data transfer remains unclear, conducting experiments with controlled datasets would offer valuable insights. Moreover, employing a more comparable test set could help mitigate statistical variance, particularly in analyses such as the commutative property assessment.

## 8 Conclusion and Future Work

Our study delves into the transferability of knowledge in cross-lingual scenarios, leveraging a byte-level tokenizer and an adapted methodology inspired by Hernandez et al. (2021). By measuring the models' reliance on pretraining when executing tasks in diverse languages, our approach offers an understanding of the cross-lingual capabilities of language models. The results provide evidence that language-agnostic representations also play an important role in downstream tasks. This not only contributes to the current understanding of cross-lingual transferability but also serves as a catalyst for further exploration into the properties of language-agnostic knowledge transfer. For future research directions, we envision key investigations that can build upon the insights presented in this paper:

1. **Expand Experiment Range:** Use more source languages so we can draw stronger conclusions.
2. **Controlled Datasets Usage:** Employ Controlled datasets and comparable test sets to address mC4 dataset heterogeneity, offering clearer insights into varied knowledge exposure impact on cross-lingual transferability and mitigating variance.
3. **Explore Larger Models:** Investigate the use of larger models in few-shot learning downstream tasks as complementary evaluations to perplexity measurements.
4. **Measure \(D_{T}\) from Non-natural languages:** Perform experiments with non-natural language data, such as artificial languages with hierarchical structures. This exploration could shed light on whether knowledge transfer primarily occurs due to the content of pretraining or is largely influenced by the linguistic form.

## Acknowledgments

We would like to thank Hugo Queiroz Abonizio, Andrea Roque, and Luciane Kawamata Notario for the useful discussions and insights. We would like to thank Google TPU Research Cloud for the computing. This research was partially funded by grant 2022/01640-2 from Fundacao de Amparo a Pesquisa do Estado de Sao Paulo (FAPESP) 



## References

* Abonizio et al. (2022) Hugo Abonizio, Leandro Rodrigues de Souza, Roberto Lotufo, and Rodrigo Nogueira. 2022. MonoByte: A pool of monolingual byte-level language models. In _Proceedings of the 29th International Conference on Computational Linguistics_, pages 3506-3513, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
* Artetxe et al. (2020) Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. 2020. On the cross-lingual transferability of monolingual representations. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 4623-4637, Online. Association for Computational Linguistics.
* Beukman and Fokam (2023) Michael Beukman and Manuel Fokam. 2023. Analysing cross-lingual transfer in low-resourced african named entity recognition.
* Blevins and Zettlemoyer (2022) Terra Blevins and Luke Zettlemoyer. 2022. Language contamination helps explains the cross-lingual capabilities of English pretrained models. In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_, pages 3563-3574, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
* Bojanowski et al. (2017) Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. _Transactions of the association for computational linguistics_, 5:135-146.
* Chiang and Lee (2020) Cheng-Han Chiang and Hung yi Lee. 2020. Pre-training a language model without human language.
* Conneau et al. (2020) Alexis Conneau, Shijie Wu, Haoran Li, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Emerging cross-lingual structure in pretrained language models. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 6022-6034, Online. Association for Computational Linguistics.
* Rodriguez de Souza et al. (2021)Leandro Rodrigues de Souza, Rodrigo Nogueira, and Roberto Lotufo. 2021. On the ability of monolingual models to learn language-agnostic representations.
* Fujinuma et al. (2022) Yoshinari Fujinuma, Jordan Boyd-Graber, and Katharina Kann. 2022. Match the script, adapt if multilingual: Analyzing the effect of multilingual pretraining on cross-lingual transferability. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 1500-1512, Dublin, Ireland. Association for Computational Linguistics.
* Hendrycks and Gimpel (2023) Dan Hendrycks and Kevin Gimpel. 2023. Gaussian error linear units (gelus).
* Hernandez et al. (2021) Danny Hernandez, Jared Kaplan, Tom Henighan, and Sam McCandlish. 2021. Scaling laws for transfer. _arXiv preprint arXiv:2102.01293_.
* Hoffmann et al. (2022) Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Roland, Kaite Millican, George van den Driessche, Bogdan Damoc, Aurelieu Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre. 2022. Training compute-optimal large language models.
* Lin et al. (2019) Yu-Hsiang Lin, Chian-Yu Chen, Jean Lee, Zirui Li, Yuyan Zhang, Mengzhou Xia, Shruti Rijnwani, Junxian He, Zhisong Zhang, Xuezhe Ma, Antonios Anastasopoulos, Patrick Littell, and Graham Neubig. 2019. Choosing transfer languages for cross-lingual learning. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 3125-3135, Florence, Italy. Association for Computational Linguistics.
* Littell et al. (2017) Patrick Littell, David R. Mortensen, Ke Lin, Katherine Kairis, Carlise Turner, and Lori Levin. 2017. URIEL and lang2vec: Representing languages as typological, geographical, and phylogenetic vectors. In _Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers_, pages 8-14, Valencia, Spain. Association for Computational Linguistics.
* Papadimitriou and Jurafsky (2020) Isabel Papadimitriou and Dan Jurafsky. 2020. Learning Music Helps You Read: Using transfer to study linguistic structure in language models. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 6829-6839, Online. Association for Computational Linguistics.
* Pires et al. (2019) Telmo Pires, Eva Schlinger, and Dan Garrette. 2019. How multilingual is multilingual BERT? In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 4996-5001, Florence, Italy. Association for Computational Linguistics.
* Real et al. (2020) Livy Real, Erick Fonseca, and Hugo Goncalo Oliveira. 2020. The assim 2 shared task: a quick overview. In _Computational Processing of the Portuguese Language: 14th International Conference, PROPOR 2020, Evora, Portugal, March 2-4, 2020, Proceedings 14_, pages 406-412. Springer.
* Ri and Tsuruoka (2022) Ryokan Ri and Yoshimasa Tsuruoka. 2022. Pretraining with artificial language: Studying transferable knowledge in language models. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 7302-7315, Dublin, Ireland. Association for Computational Linguistics.
* Roberts et al. (2019) Adam Roberts, Hyung Won Chung, Anselm Levskaya, Gaurav Mishra, James Bradbury, Daniel Andor, Sharan Narang, Brian Lester, Colin Gaffney, Afroz Mohiuddin, Curtis Hawthorne, Attor Lewkowycz, Alex Salcianiu, Marc van Zee, Jacob Austin, Sebastian Goodman, Livio Baldini Soares, Haitang

Hu, Sasha Tsvyashchenko, Aakanksha Chowdhery, Jasmin Bastings, Jannis Bulian, Xavier Garcia, Jianmo Ni, Andrew Chen, Kathleen Keenaly, Jonathan H. Clark, Stephan Lee, Dan Garrette, James Lee-Thorp, Colin Raffel, Noam Shazeer, Marvin Ritter, Maarten Bosma, Alexandre Passos, Jeremy Matin-Shepard, Noah Fiedel, Mark Omernick, Brennan Saeta, Ryan Sepassi, Alexander Spiridonov, Joshua Newlan, and Andrea Gesmundo. 2022. Scaling up models and data with t5x and seqio. _arXiv preprint arXiv:2203.17189_.
* Rust et al. (2021) Phillip Rust, Jonas Pfeiffer, Ivan Vulic, Sebastian Ruder, and Iryna Gurevych. 2021. How good is your tokenizer? on the monolingual performance of multilingual language models. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 3118-3135, Online. Association for Computational Linguistics.
* Shaw et al. (2018) Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. 2018. Self-attention with relative position representations. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)_, pages 464-468.
* Touvron et al. (2023) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothee Lacroix, Baptiste Roziere, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models.
* Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. _Advances in neural information processing systems_, 30.
* Wu et al. (2022)Xianze Wu, Zaxiang Zheng, Hao Zhou, and Yong Yu. 2022. LAFT: Cross-lingual transfer for text generation by language-agnostic finetuning. In _Proceedings of the 15th International Conference on Natural Language Generation_, pages 260-266, Waterville, Maine, USA and virtual meeting. Association for Computational Linguistics.
* Xue et al. (2022)Linting Xue, Aditya Barua, Noah Constant, Ramil Al-Rfou, Sharan Narang, Mihiri Kale, Adam Roberts, and Colin Raffel. 2022. ByT5: Towards a token-free future with pre-trained byte-to-byte models. _Transactions of the Association for Computational Linguistics_, 10:291-306.
* Xue et al. (2021) Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. mT5: A massively multilingual pre-trained text-to-text transformer. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 483-498, Online. Association for Computational Linguistics.
* Ye et al. (2023) Jiacheng Ye, Xijia Tao, and Lingpeng Kong. 2023. Language versatilists vs. specialists: An empirical revisiting on multilingual transfer ability.
* Yuan et al. (2023) Fei Yuan, Shuai Yuan, Zhiyong Wu, and Lei Li. 2023. How multilingual is multilingual llm? _arXiv preprint arXiv:2311.09071_.
* Zeng et al. (2023)Jiali Zeng, Yufan Jiang, Yongjing Yin, Yi Jing, Fanding Meng, Binghuai Lin, Yunbo Cao, and Jie Zhou. 2023. Soft language clustering for multilingual model pretraining. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 7021-7035, Toronto, Canada. Association for Computational Linguistics.

## Appendix A Appendix: Downstream Task Experiment

To further explore the generalizability of our findings beyond casual language modeling with the perplexity metric, we conducted an additional experiment focusing on a distinct downstream task. This experiment involved finetuning our models pretrained on the selected source languages for a specific downstream task, targeting a different language.

### Experiment Details

For this experiment, we selected Portuguese as the target language and the Recognizing Textual Entailment (RTE) task from the ASSIN2 Real et al. (2020)5 dataset. The dataset comprised 6,500 training examples and 500 validation instances.

Footnote 5: [https://sites.google.com/view/assin2/](https://sites.google.com/view/assin2/) 

random classifier by 15 percentage points (ASSIN2 contains two distinct classes).

Based on our findings, cross-lingual knowledge transfer appears to occur even with more distant languages. These results underscore the importance of further exploration in this area, indicating a promising potential for measuring knowledge transfer in cross-lingual scenarios and delineating the contributions of language-agnostic and language-specific components in the models' representations.

\begin{table}
\begin{tabular}{|l|c|} \hline
**Source Language** & **Accuracy (\%)** \\ \hline _Portuguese (baseline)_ & 85.0 \\ Russian & 82.0 \\ English & 78.6 \\ Chinese & 65.6 \\ \hline \end{tabular}
\end{table}
Table 7: Results from finetuning our source models on the ASSIN2 Recognizing Text Entailment task. We report the accuracy obtained in our validation set.