Venue: NAACL 2024

# Language Agnostic Code Embeddings

Saiteja Utpala

Cohere For AI

saitejautpala@gmail.com &Alex Gu

MIT

gua@mit.edu &Pin Yu Chen

IBM Research

pin-yu.chen@ibm.com

###### Abstract

Recently, code language models have achieved notable advancements in addressing a diverse array of essential code comprehension and generation tasks. Yet, the field lacks a comprehensive deep dive and understanding of the code embeddings of multilingual code models. In this paper, we present a comprehensive study on multilingual code embeddings, focusing on the cross-lingual capabilities of these embeddings across different programming languages. Through probing experiments, we demonstrate that code embeddings comprise two distinct components: one deeply tied to the nuances and syntax of a specific language, and the other remaining agnostic to these details, primarily focusing on semantics. Further, we show that when we isolate and eliminate this language-specific component, we witness significant improvements in downstream code retrieval tasks, leading to an absolute increase of up to +17 in the Mean Reciprocal Rank (MRR).

## 1 Introduction

Large language models (LLMs) have made remarkable progress in code-related tasks, exemplified by models such as Codex, which powers GitHub Copilot and offers automated code suggestions within integrated development environments (IDEs) [3]. These models achieve their proficiency through extensive training on vast code datasets, providing them with versatile contextual understanding for a range of coding tasks [24, 5, 6]. However, it's worth noting that decoder-only models may not always be the optimal choice for retrieval tasks when compared to encoder models [28, 9, 10].

While previous studies indicate that language models trained on a variety of natural languages exhibit strong cross-lingual traits [17], their multilingual representations can be dissected into a language-specific syntax component and a language-agnostic semantic component [1]. Moreover, eliminating the language-specific elements can enhance retrieval tasks and counteract "language bias", a tendency for representations to cluster by language instead of meaning [29, 10, 11]. We aim to determine if similar patterns are evident in multilingual code models pretrained on programming languages (e.g., C, C++, Python) as opposed to natural languages (e.g., English, French, Spanish). We specifically address:

1. Can representations of these code models be categorized into language-specific and language-agnostic components?
2. If so, does removing the language-specific components enhance the consistency and comparability of code representations (alignment) across programming languages, thereby improving downstream code retrieval tasks?

Our comprehensive evaluations confirm these patterns in code language models. Our contributions are summarized as follows:

* We investigate the cross-lingual properties of pretrained multilingual code language models, examining them through the lens of both language-specific (syntax) and language-agnostic (semantic) attributes. Through various probing experiments on five models, we demonstrate that the embeddings of these code language models include both language-specific (syntax) and language-agnostic (semantic) components.
* We demonstrate that removing these language-specific components and using only the language-agnostic component in downstream tasks can significantly enhance code retrieval

tasks providing improvement upto +17 increase in MRR. Importantly, such improvements are achieved using inexpensive operations such as centering and projections, without using parallel language data or finetuning.
* Additionally, our extensive ablation studies suggest that as few as 100 samples per language suffice for these MRR improvements. We also confirm that the improvements are not restricted to a single type of embedding but can be realized across all common types, including mean, cls, and pooler embeddings.

## 2 Language Agnostic Code Embeddings

Let represent a multilingual code language model trained on a set of programming languages \(\{1,\dots,\ell\}\). Given a code snippet \(c\) in a specific programming language \(l\), this model produces an embedding \(\textbf{e}\in\mathbb{R}^{d}\), denoted as . hypothesize that the embedding \(\textbf{e}\in\mathbb{R}^{d}\) of a code snippet can be decomposed into two components: a syntax component, \(\textbf{e}^{s}\in\mathbb{R}^{d}\), which depends on the programming language \(l\), and semantic component, \(\textbf{e}^{a}\) which is language-agnostic. This relationship can be expressed as:

\[\textbf{e}=\textbf{e}^{s}+\textbf{e}^{a} \tag{1}\]

Next, we introduce the Estimation Set \(\mathcal{E}\), which is used to estimate the language-specific components \(\textbf{e}^{s}\).

**Definition 1** (Estimation Set).: _The Estimation set \(\mathcal{E}\) is defined as a collection of \(n\) code snippets \(\{c_{1}^{(l)},\dots,c_{n}^{(l)}\}\) from each programming language \(l\in\{1,\dots,\ell\}\). Importantly, the code snippets in this set need not be direct translations of one another._

Now for given model , we define embedding matrix \(\textbf{E}_{l}\in\mathbb{R}^{n\times d}\) for each language \(\ell\) as \(\textbf{E}_{l}=\left[\mathcal{M}(c_{1}^{(l)}),\dots,\mathcal{M}(c_{n}^{(l)}) \right].\)

In the subsequent sections, we explore a variety of methods designed to remove language-specific information. This analysis is conducted from the unified perspective of Equation 1, which serves as the fundamental framework for disentangling language-specific and language-agnostic components within code embeddings. Additionally, we explicitly outline the assumptions that underpin each of these methods.

### Centering

The first method we explore is centering Libovicky et al. (2020), which is grounded in the following key assumption:

**Assumption 1**.: _Given an programming language \(l\), centering method makes an assumption that language specific components \(\boldsymbol{e}^{s}\) is same for all embeddings in that programming language \(\ell\)._

Under Assumption 1, the mean of the embeddings for a programming language \(\ell\) can be expressed as:

Figure 1: Illustration of the top retrieved results for code-to-code search, where the query is in Python, and the target is in C. Language composition varies across retrieval databases: ‘Monolingual’ (C only), ’Source Excluded Multilingual’ (several languages except Python), ’Source Included Multilingual’ (several languages including Python). Demonstrates improved semantic matching and reduced language bias after removing language-specific components.

 

\[\mathbf{m}_{l} =\frac{1}{n}\sum_{i=1}^{n}\mathbf{e}_{i}=\frac{1}{n}\sum_{i=1}^{n} \left(\mathbf{e}_{i}^{s}+\mathbf{e}_{i}^{a}\right)\] \[\overset{\eqref{eq:constraint-constraints}}{=}\frac{1}{n}\sum_{i=1}^{n} \left(\mathbf{e}^{s}+\mathbf{e}_{i}^{a}\right)=\mathbf{e}^{s}+\underbrace{ \frac{1}{n}\sum_{i=1}^{n}\mathbf{e}_{i}^{a}}_{\text{small for large $n$}}~{}\approx\mathbf{ e}^{s}.\]

From the above, it is evident that for a large enough value of \(n\), the language-specific syntax embedding for a given programming language can be approximately estimated as the mean of the embeddings in that language. This method is summarized in Algorithm 1 in centering.

### Low Rank Decomposition

A significant concern with the centering method, as outlined in Assumption 1, is its presumption that the syntax embedding for each code is the same and is independent of the given code content. Instead, it is only dependent on the programming language. To address this limitation, Low Rank Decomposition (LRD) Schmidt (1907); Yang et al. (2021) introduces distinct syntax subspaces for each programming language, operating under the following assumptions:

**Assumption 2**.: _The low rank decomposition method is based on the following assumptions:_

1. _The language-specific syntax embedding varies for each individual embedding._
2. _The language-specific syntax embedding,_ \(\boldsymbol{e}^{s}\)_, is orthogonal to the language-agnostic semantic embedding,_ \(\boldsymbol{e}^{a}\)_, i.e.,_ \(\boldsymbol{e}^{s}\perp\boldsymbol{e}^{a}\)_._
3. _For each programming language, there exists a low-rank subspace of rank_ \(r\) _that captures the syntactic essence of the embeddings._

Based on the above assumptions, we determine the syntactic subspace of language \(l\) of rank \(r\), denoted as \(\mathbf{E}_{l}^{r}\in\mathbb{R}^{n\times d}\), as follows:

\[\begin{split}&\min_{\mathbf{E}_{l}^{r}\in\mathbb{R}^{n\times d}} \|\mathbf{E}_{l}-\mathbf{E}_{l}^{r}\|_{\mathbf{F}}^{2}\\ &\qquad\quad\text{ s.t.\leavevmode\nobreak\ \smash{rank}}( \mathbf{E}_{l}^{r})\leq r.\end{split} \tag{2}\]

Equation 2 can be solved using topk-vd with \(k=r\) where \(\mathbf{E}_{l}^{r}=\mathbf{U}_{r}\mathbf{\Sigma}_{r}\mathbf{V}_{r}^{T}\). The projection of the embedding \(\mathbf{e}\) onto the rowspace\((\mathbf{E}_{l}^{r})\)is given by \(\mathbf{V}_{r}\mathbf{V}_{r}^{T}\mathbf{e}\). The language-agnostic embedding is then obtained by removing this component: \(\mathbf{e}^{a}=\mathbf{e}-\mathbf{V}_{r}\mathbf{V}_{r}^{T}\mathbf{e}\). This method is summarized in Algorithm 1 in LRD.

### Common Specific Low Rank Decomposition Decomposition

The Common Specific Low Rank Decomposition is grounded on the following assumptions:

1. _The language-specific syntax embedding varies for each individual embedding._
2. _The language-specific syntax embedding,_ \(\boldsymbol{e}^{s}\)_, is orthogonal to the language-agnostic semantic embedding,_ \(\boldsymbol{e}^{a}\)_. In other words,_ \(\boldsymbol{e}^{s}\perp\boldsymbol{e}^{a}\)_._
3. _There exists a unified syntax subspace, consistent across all programming languages, that encapsulates the syntactic attributes of the code embedding._

A key distinction is that while the syntax subspace in the traditional LRD is determined for each language individually, the CS-LRD method derives a singular, unified syntax subspace that encompasses all the considered programming languages. It is formulated as:

\[\begin{split}\min_{\mathbf{m}_{c}\in\mathbb{R}^{d},\mathbf{M}_{s} \in\mathbb{R}^{d\times r},\mathbf{\Gamma}_{s}\in\mathbb{R}^{d\times c}}& \left\lVert\mathbf{M}-\mathbf{m}_{c}\cdot\mathbbm{1}_{f}^{ T}-\mathbf{M}_{s}\cdot\mathbf{\Gamma}_{s}^{T}\right\rVert_{\mathbf{F}}^{2}\\ &\qquad\quad\text{ s.t. }\mathbf{m}_{c}\perp\textsc{ colspan}(\mathbf{M}_{s}).\end{split} \tag{3}\]

where \(\mathbf{M}=[\mathbf{m}_{1},\dots,\mathbf{m}_{\ell}]\), and \(\mathbf{m}_{1},\dots,\mathbf{m}_{\ell}\) are the mean embeddings of the \(\{1,\dots,\ell\}\) programming languages. The matrix \(\mathbf{M}_{s}\), which captures the common syntactic subspace, can be obtained by the CS-LRD function in Algorithm 1. Similar to LRD, the language-agnostic embedding is obtained by removing the projection of \(\mathbf{e}\) on \(\mathbf{M}_{s}\), i.e., \(\mathbf{e}^{a}=\mathbf{e}-\mathbf{M}_{s}\mathbf{M}_{s}^{T}\mathbf{e}\).

## 3 Experiments

* This task involves identifying languages using a linear classifier. (ii) Code2Code search
- Given a piece of code in language \(L_{1}\), the objective is to retrieve the most
 

semantically relevant code in another language \(L_{2}\).

(iii) Text2Code search - The aim is to identify code that corresponds to a provided natural language query.

The first task assesses whether the procedures in Algorithm 1 effectively eliminate language-specific (syntax) components. The second and third tasks determine if language-agnostic (semantic) components are preserved.

**Datasets:** We utilize programs from the Stack dataset Kocetkov et al. (2022) to estimate language-specific components. For the Code2Code search, we employ XLCoST Zhu et al. (2022), and for the Text2Code search, we use CSN Husain et al. (2019).

**Models:** We consider five models, including encoder-only and encoder-decoder models: CodeBERT Feng et al. (2020), GraphCodeBERT Guo et al. (2020), UnixCoder Guo et al. (2022), StarEncoder Li et al. (2023), and CodeT5+ Wang et al. (2023).

**Embeddings:** For models like CodeBERT, GraphCodeBERT, UnixCoder, and StarEncoder, there isn't a standard method to obtain embeddings. In our retrieval tasks, we use mean embeddings, derived from the mean of the last hidden states. We conduct an ablation study to explore other embedding extraction methods in Section 4.2. For CodeT5+, only the pooler embedding is recommended and is given as output, and this is what we employ in our experiments.

**Retrieval Metrics:** For the Retrieval Task, we use Mean Reciprocal Rank (MRR) as our evaluation metric. MRR is calculated as \(\text{MRR}=\frac{1}{n}\sum_{i=1}^{n}\frac{1}{\text{rank}(c_{i})}\times 100\), where \(n\) represents the total number of queries, and rank(\(c_{i}\)) denotes the rank of the correct answer for the \(i\)-th query in the retrieval results. Higher MRR values indicate better performance.

### Probing

We evaluate the syntactic component of embeddings by employing a _linear_ classifier for the task of language identification, both pre and post transformations. From the Stack dataset Kocetkov et al. (2022), we allocate 10,000 code instances for estimating language components. For training, we use 24,000 code snippets from each language, and for validation, we utilize 6,000 codes from each respective language. The testing is performed on 10,000 codes for each language. The outcomes are depicted in Figure 2. Before transformation, the linear classifier yields high accuracy on the embeddings. However, after the removal of language-specific components, the accuracy declines sharply, experiencing a drop of at least 70% across all models. In particular, for the CodeT5+ model, the accuracy approaches random performance. Moreover, in the context of CS-LRD, there's an interesting relationship between the rank \(r\) and performance. As \(r\) increases, the classifier's performance diminishes. It's worth noting that this behavior is not observed with the LRD.

we also visualize PCA of CodeT5+ embeddings and show it in Figure 2f which shows embeddings are clustered by language. But after removing

Figure 2: The top row illustrates the impact on language identification accuracy before and after removing language-specific components. Meanwhile, the bottom row displays the PCA of Code T5+ embeddings for the languages: Go, Python, and C#.

 

language-specific components we see that in Figure 2g to Figure 2i there are no longer language clusters.

### Code2Code Search

Given a query \(Q\) in a source language \(\mathcal{S}\), the objective is to extract a code snippet with semantic similarity from a specific database. Depending on the language composition of the database, we consider three different variations:

* **Monolingual Database:** In this conventional setting, the database consists entirely of programs written in a single target language \(\mathcal{T}\), which is distinct from the source language \(\mathcal{S}\).
* **Source-Excluded Multilingual Database:** In this variation, the database is composed of programs in multiple languages \(\mathcal{T}_{1},\dots,\mathcal{T}_{n}\), where \(\mathcal{T}_{i}\) differs from the source language \(\mathcal{S}\).
* **Source-Included Multilingual Database:** This final variation includes the source language \(\mathcal{S}\) within its spectrum of target languages. We evaluate the language bias of models Yang et al. (2021), wherein a code from the source language \(\mathcal{S}\) is ranked higher than codes that are more semantically similar but from different languages.

For this task, we use the XLCoST dataset Zhu et al. (2022), which contains parallel translations of seven programming languages: C, C#, C++, Java, JavaScript, PHP, and Python. However, it's important to note that CodeBERT and GraphCodeBERT do not support C, C++, and C#. Therefore, we only consider Java, JavaScript, PHP, and Python for these models, while all seven languages are included for all other models.

We present the change in the Mean Reciprocal Rank (MRR) before and after the removal of the language component in Figure 3. This change is averaged over all pairwise language retrieval tasks, amounting to \(6\times 7=42\) tasks in total. Additionally, for the CodeT5+ model, we offer a detailed breakdown of the MRR for each source language. The retrieval results are averaged across the six target languages and are tabulated in Table 1.

**Discussion:** Significant improvements are observed before and after removing the language component, with an absolute increase in MRR ranging up to +17. We discuss a couple of factors below.

1. **Database Configuration:** Models exhibit substantial language bias, leading to a drastic drop in performance in the 'Source Included Multilingual' setup, with a reduction of -59.62% from 89.51 to 29.89.
2. **Centering Effects:** In three out of five cases, centering has a detrimental impact on performance. This aligns with the notion that centering may mix syntax and semantic signals, potentially removing semantic meaning as well Yang et al. (2021); Xie et al. (2022).
3. **UnXcoder Exception:** Notably, UniXcoder explicitly aligns representations from different programming languages during pretraining itself using a task involving cross-modal generation Guo et al. (2022). Consequently, none of the methods provide any improvement in this case.
4. **CS-LRD Superiority:** In most cases, CS-LRD outperforms both centering and LRD. This is attributed to the joint learning of the syntax subspace across different programming languages in CS-LRD.
5. **Effect of Rank in LRD and CS-LRD:** We examine the impact of the rank of the subspace \(r\) in Figure 5 for both LRD (Top Row) and CS-LRD (Bottom Row). Increasing \(r\) consistently enhances MRR in CS-LRD, while no such behavior is observed in LRD, which is less stable compared to CS-LRD.

### Text2Code Search

In this section, we delve into Text2Code search, a task where the objective is to find code that corresponds to a given natural language query (in English). We explore two distinct settings for Text2Code search:

* **Monolingual Database:** In this setting, we construct the retrieval database using data from a single programming language.
* **Multilingual Database:** In contrast, for this setting, we include data from all programming languages in the retrieval database. The goal is to locate the correct code snippet that matches the query, regardless of the programming language.

For this task, we utilize the CodeSearchNet dataset Husain et al. (2019), which contains data 

in six programming languages: Go, Ruby, Java, JavaScript, PHP, and Python. Retrieval database consists of codes in both val and test.

We present the change in Mean Reciprocal Rank (MRR) before and after removing the language component in Figure 4. Full view for Unixcoder can be found at Table 2.

**Discussion:** Sizable improvements are observed before and after removing language component, with an absolute increase in MRR ranging upto +8. We discuss a couple of factors below.

1. **CodeT5+ Exception:** CodeT5+ includes contrastive tuning as one of its pretraining tasks Wang et al. (2023) for text-to-code, which explicitly aligns English with programming languages. Hence, we don't observe any improvement.
2. **Centering Superiority:** Unlike in Code2Code search, in Text2Code search centering outperforms both LRD and CS-LRD.
3. **Effect of Rank in LRD and CS-LRD:** We study the influence of the subspace rank r as depicted in Figure 6. The top row illustrates the effect for LRD, while the bottom row represents CS-LRD. For both CS-LRD and LRD, increasing r either consistently improves MRR or remains stable. However, for CodeT5+, there is a consistent decrease.
4. **Effect of Projecting out English:** We conduct retrieval in two distinct ways. In the first method, we remove language components solely from programming languages, leaving the query unaffected (no English component is removed). In the second method, we transform the query by removing the English language component from it. The results are depicted in Figure 6. We find that projecting out the English language components is crucial to observe an increase in MRR.

## 4 Ablation Study

In this section, we conduct various ablation studies focusing on the estimation set's size and the effects of different kinds of embeddings.

### Effect of Estimation Set Size

In this section, we investigate the impact of estimation set size on language estimation and its influence on Mean Reciprocal Rank (MRR) in zero-shot Code2Code search. We randomly sample \(\{100,500,1000,5000,10000,25000\}\) examples from the original pool of 50,000 samples from the Stack dataset for each language used in Section 3.2. Subsequently, we conduct retrievals with the language components removed based on these samples. This study is repeated five times for each sample size, and we calculate the MRR

\begin{table}
\begin{tabular}{l|c c c c c c c c} \hline CodeTS+ & & C & C\# & C++ & Java & Javascript & PHP & Python & Avg. \\ \multirow{4}{*}{Monolingual} & Original & 86.19 & 88.12 & 90.93 & 89.95 & 90.63 & 89.46 & 91.32 & 89.51 \\  & Centering & 87.59 & 91.10 & 93.25 & 92.23 & 91.70 & 90.97 & 92.62 & 91.35 (**+1.84**) \\  & LRD(r=10) & 88.87 & 91.77 & 94.85 & 93.20 & 91.84 & 91.85 & 92.90 & 92.18 (**+2.67**) \\  & CS-LRD(r=9) & 87.37 & 90.50 & 93.12 & 91.78 & 92.09 & 90.41 & 92.69 & 91.14 (**+1.63**) \\ \hline \multirow{4}{*}{Source Excluded Multilingual} & Original & 43.73 & 24.71 & 59.51 & 31.19 & 57.43 & 61.12 & 65.26 & 48.99 \\  & Centering & 49.98 & 28.93 & 74.73 & 36.05 & 65.28 & 61.67 & 68.32 & 54.99 (**+6.00**) \\  & LRD(r=10) & 56.24 & 39.73 & 77.16 & 44.79 & 69.90 & 67.05 & 74.56 & 61.33 (**+12.34**) \\  & CS-LRD(r=9) & 57.04 & 31.13 & 76.89 & 37.75 & 66.99 & 65.10 & 72.03 & 58.13 (**+9.14**) \\ \hline \multirow{4}{*}{Source Induced Multilingual} & Original & 34.53 & 16.94 & 17.69 & 23.99 & 32.76 & 38.09 & 45.43 & 29.89 \\  & Centering & 37.03 & 16.28 & 33.23 & 21.89 & 40.14 & 40.97 & 52.81 & 34.62 (**+4.73**) \\ \cline{1-1}  & LRD(r=10) & 45.88 & 11.37 & 41.55 & 23.47 & 40.37 & 39.96 & 54.63 & 36.75 (**+6.86**) \\ \cline{1-1}  & CS-LRD(r=9) & 47.07 & 20.47 & 36.87 & 29.73 & 47.28 & 50.06 & 60.04 & 41.65 (**+11.76**) \\ \hline \end{tabular}
\end{table}
Table 1: MRR averaged across all target languages for zero-shot Code2Code search using CodeT5+ Wang et al. (2023).

Figure 3: Absolute change in MRR after removing language components in zero-shot Code2Code search.

 

change. The results can be seen in Figure 7. In this figure, the top row represents Centering, the middle row showcases LRD, and the bottom row depicts CS-LRD.

The results highlight a significant change in MRR, even with estimation sets containing as few as 100 samples per language. However, some variance is observed in certain instances. This variance diminishes considerably once the estimation set expands to 1000 samples, resulting in a steadier MRR shift. Interestingly, the variance is typically greater for Centering and LRD compared to CS-LRD. This study also reveals that specific examples in the estimation set don't play as significant a role as the overall size of the estimation set.

### Mean embedding vs [CLS] embedding vs Pooler output

In this section, we examine various kinds of embeddings and analyze the effects of removing language components from them for zero-shot code2code search. As noted in Sections 3.2 and 3.3, we utilized mean embeddings for CodeBERT, GraphCodeBERT, UnixCoder, and StarEncoder. However, other embedding types are also commonly employed in practice.

To clarify, let \(c\) be a code snippet. The function encoder\((c)\) produces the last hidden state with the shape \({\mathbb{R}^{t\times d}}\), where \(t\) denotes the number of tokens in the code. There are several methods to obtain a single \({\mathbb{R}^{d}}\) representation from the encoder's output. These methods are defined as follows:

mean-embedding\((c)\triangleq\text{encoder}(c)\).mean(0) cls-embedding\((c)\triangleq\text{encoder}(c)[0]\) pooler-embedding\((c)\triangleq\text{pooler}(\text{encoder}(x)[0])\)

Here, pooler is an MLP layer positioned atop the encoder, and its output is directed to the language modeling head.

Results are displayed in Figure 8. We observe that the improvements aren't limited to mean-embedding; they also extend to cls-embedding and pooler-embedding. Specifically, when using CS-LRD with CodeBERT's cls-embedding, there's a significant increase of +26.22. Similarly, StarEncoder's pooler embedding sees a +14.87 improvement with CS-LRD. Notably, mean-embedding remains superior to other embedding variants regardless of the presence or absence of language information.

## 5 Related Work

**Cross Linequal properties of Natural Language models**: We are the first to investigate the crosslingual properties of pretrained multilingual code language models, examining them through the lens of both language-specific (syntax) and language-agnostic (semantic) attributes. Our research is motivated by a rich body of work that probes similar behavior in multilingual natural language models Schuster et al. (2019); Libovicky et al. (2020); Kulshreshtha et al. (2020); Yang et al. (2021); Xie et al. (2022); Chang et al. (2022). While these studies

Figure 4: Absolute change in MRR after removing language components in zero-shot Text2Code search.

 

predominantly concentrate on models trained for natural languages, our emphasis lies on those designed for programming languages.

**Code Representation Learning**: The monumental success of BERT Devlin et al. (2019) and T5 Raffel et al. (2020) in natural language understanding has sparked significant interest in adapting similar architectures for programming languages. This interest has given rise to models like CodeBERT Feng et al. (2020), CodeTransformer Zugner et al. (2020), GraphCodeBERT Guo et al. (2020), ContraCode Jain et al. (2021), SynCoBERT Wang et al. (2021), UniXCoder Guo et al. (2022), and PLBART Ahmad et al. (2021). Some of these works Zugner et al. (2020); Guo et al. (2020) explore code-specific pretraining tasks, utilizing both language-specific features (e.g., Program Analysis Edges) and language-agnostic features (e.g., Abstract Syntax Trees) to improve the performance of multilingual code models. In contrast, our work focuses on examining the representations of pretrained multilingual code language models.

**LMs for Code Generation**: In recent years, there have been many language models (LMs) for code trained with various architectures, sizes, and data mixtures, inspired by the huge success of GPT Radford et al. (2019); Brown et al. (2020). Some of these include Codex Chen et al. (2021), CodeGeeX Zheng et al. (2023), SantaCoder Allal et al. (2023), PolyCoder Xu et al. (2022), CodeGen Nijkamp et al. (2022), StarCoder Li et al. (2023), WizardCoder Luo et al. (2023), and Code Llama Roziere et al. (2023). In this work, instead of focusing on code generation, we concentrate on code representations.

## 6 Discussion

In our study of multilingual code models, we find that these embeddings can be decomposed into two main components: language-specific and language-agnostic. Through extensive experimentation, we conclude that when representations are not aligned during pre-training, the removal of the language-specific component, utilizing only the language-agnostic component, significantly enhances performance in retrieval tasks.

## 7 Limitations

In this study, we have focused on exploring representations of encoder-only or encoder-decoder models. However, future work should also investigate decoder-only models.

## Acknowledgements

A. Gu is supported by the National Science Foundation (NSF) Graduate Research Fellowship under Grant No. 2141064. We sincerely thank Armando Solar-Lezama for comments and feedback on the project.

## References

* Ahmad et al. (2021)Wasi Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai-Wei Chang. 2021. Unified pre-training for program understanding and generation. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 2655-2668.
* Allan et al. (2022) Loubna Ben Allan, Raymond Li, Denis Kocetkov, Chenghao Mou, Christopher Akiki, Carlos Munoz Ferrandis, Niklas Muennighoff, Mayank Mishra, Alex Gu, Manan Dey, et al. 2023. Santacoder: don't reach for the stars! _arXiv preprint arXiv:2301.03988_.
* Athiwaratkun et al. (2022) Ben Athiwaratkun, Sanjay Krishna Gouda, Zijian Wang, Xiaopeng Li, Yuchen Tian, Ming Tan, Wasil Uddin Ahmad, Shiqi Wang, Qing Sun, Mingyue Shang, et al. 2022. Multi-lingual evaluation of code generation models. In _The Eleventh International Conference on Learning Representations_.
* Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. _Advances in neural information processing systems_, 33:1877-1901.
* Chang et al. (2022)Tyler Chang, Zhuowen Tu, and Benjamin Bergen. 2022. The geometry of multilingual language model representations. In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_, pages 119-136.
* Chen et al. (2021) Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. _arXiv preprint arXiv:2107.03374_.
* Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 4171-4186.
* Feng et al. (2019) Zhangjin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, 

Ting Liu, Daxin Jiang, et al. 2020. Codebert: A pre-trained model for programming and natural languages. In _Findings of the Association for Computational Linguistics: EMNLP 2020_, pages 1536-1547.
* Guo et al. (2022) Daya Guo, Shuai Lu, Nan Duan, Yanlin Wang, Ming Zhou, and Jian Yin. 2022. Unixcoder: Unified crossmodal pre-training for code representation. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 7212-7225.
* Guo et al. (2020) Daya Guo, Shuo Ren, Shuai Lu, Zhangyin Feng, Duyu Tang, LIU Shujie, Long Zhou, Nan Duan, Alexey Svyatkovskiy, Shengyu Fu, et al. 2020. GraphcodeBERT: Pre-training code representations with data flow. In _International Conference on Learning Representations_.
* Husain et al. (2019) Hamel Husain, Ho-Hsiang Wu, Tiferet Gazit, Miltiadis Allamanis, and Marc Brockschmidt. 2019. Code-searchnet challenge: Evaluating the state of semantic code search. _arXiv preprint arXiv:1909.09436_.
* Jain et al. (2021) Paras Jain, Ajay Jain, Tianjun Zhang, Pieter Abbeel, Joseph Gonzalez, and Ion Stoica. 2021. Contrastive code representation learning. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_, pages 5954-5971.
* Kocetkov et al. (2022) Denis Kocetkov, Raymond Li, LI Jia, Chenghao Mou, Yacine Jernite, Margaret Mitchell, Carlos Munoz Ferrandis, Sean Hughes, Thomas Wolf, Dzmitry Bahdanau, et al. 2022. The stack: 3 tb of permissively licensed source code. _Transactions on Machine Learning Research_.
* Kulshreshtha et al. (2020) Saurabh Kulshreshtha, Jose Luis Redondo Garcia, and Ching Yun Chang. 2020. Cross-lingual alignment methods for multilingual bert: A comparative study. In _Findings of the Association for Computational Linguistics: EMNLP 2020_, pages 933-942.
* Li et al. (2023) Raymond Li, Loubna Ben Allanl, Yangtian Zi, Niklas Muennighoff, Denis Koectkov, Chengao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, et al. 2023. Starcoder: may the source be with you! _arXiv preprint arXiv:2305.06161_.
* Libovicky et al. (2020) Jindrich Libovicky, Rudolf Rosa, and Alexander Fraser. 2020. On the language neutrality of pre-trained multilingual representations. In _Findings of the Association for Computational Linguistics: EMNLP 2020_, pages 1663-1674.
* Luo et al. (2023) Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xibo Beng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. 2023. WizardCoder: Empowering code large language models with evolir instruct. _arXiv preprint arXiv:2306.08568_.
* Nijkamp et al. (2023) Erik Nijkamp, Hiroaki Hayashi, Caiming Xiong, Silvio Savarese, and Yingbo Zhou. 2023. Codegen2: Lessons for training llms on programming and natural languages. _arXiv preprint arXiv:2305.02309_.
* Nijkamp et al. (2022) Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. 2022. Codegen: An open large language model for code with multi-turn program synthesis. In _The Eleventh International Conference on Learning Representations_.
* Oord et al. (2018) Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2018. Representation learning with contrastive predictive coding. _arXiv preprint arXiv:1807.03748_.
* Pirtala et al. (2020) Vihari Pirtala, Praneeth Netrapalli, and Sunita Sarawagi. 2020. Efficient domain generalization via commonspecific low-rank decomposition. In _International Conference on Machine Learning_, pages 7728-7738. PMLR.
* Pires et al. (2019) Telmo Pires, Eva Schlinger, and Dan Garrette. 2019. How multilingual is multilingual bert? In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 4996-5001.
* Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. _OpenAI blog_.
* Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. _The Journal of Machine Learning Research_, 21(1):5485-5551.
* Roy et al. (2020) Uma Roy, Noah Constant, Rami Al-Rfou, Aditya Barua, Aaron Phillips, and Yinfei Yang. 2020. Lateqa: Language-agnostic answer retrieval from a multilingual pool. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 5919-5930.
* Roziere et al. (2023) Baptise Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jeremy Rapin, et al. 2023. Code llama: Open foundation models for code. _arXiv preprint arXiv:2308.12950_.
* Schmidt (1907) Erhard Schmidt. 1907. Zur theorie der linearen und nichtlinearen integralgleichungen. _Mathematische Annalen_, 63(4):433-476.
* Schuster et al. (2019) Tal Schuster, Ori Ram, Regina Barzilay, and Amir Globerson. 2019. Cross-lingual alignment of contextual word embeddings, with applications to zero-shot dependency parsing. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 1599-1613.
* Wang et al. (2021a) Xin Wang, Yasheng Wang, Fei Mi, Pingyi Zhou, Yao Wan, Xiao Liu, Li Li, Hao Wu, Jin Liu, and Xin Jiang. 2021a. Syncobert: Syntax-guided multi-modal contrastive pre-training for code representation. _arXiv preprint arXiv:2108.04556_ 

* Wang et al. (2023) Yue Wang, Hung Le, Akhilesh Deepak Gotmare, Nghi DQ Bui, Junnan Li, and Steven CH Hoi. 2023. Codet5+: Open code large language models for code understanding and generation. _arXiv preprint arXiv:2305.07922_.
* Wang et al. (2021b) Yue Wang, Weishi Wang, Shafiq Joty, and Steven CH Hoi. 2021b. Codel5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_, pages 8696-8708.
* Xie et al. (2022) Zhihui Xie, Handong Zhao, Tong Yu, and Shuai Li. 2022. Discovering low-rank subspaces for language-agnostic multilingual representations. In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_, pages 5617-5633.
* Xu et al. (2022) Frank F Xu, Uri Alon, Graham Neubig, and Vincent Josua Hellendoorn. 2022. A systematic evaluation of large language models of code. In _Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming_, pages 1-10.
* Yang et al. (2021) Ziyi Yang, Yinfei Yang, Daniel Cer, and Eric Darve. 2021. A simple and effective method to eliminate the self language bias in multilingual representations. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_, pages 5825-5832.
* Zheng et al. (2023) Qinkai Zheng, Xiao Xia, Xu Zou, Yuxiao Dong, Shan Wang, Yufei Xue, Zihan Wang, Lei Shen, Andi Wang, Yang Li, et al. 2023. Codegex: A pre-trained model for code generation with multilingual evaluations on humaneval-x. _arXiv preprint arXiv:2303.17568_.
* Zhu et al. (2022) Ming Zhu, Aneesh Jain, Karthik Suresh, Roshan Ravindran, Sindhu Tipirneni, and Chandan K Reddy. 2022. Xlcost: A benchmark dataset for cross-lingual code intelligence. _arXiv preprint arXiv:2206.08474_.
* Zugner et al. (2020) Daniel Zugner, Tobias Kirschstein, Michele Catasta, Jure Leskovec, and Stephan Gunnemann. 2020. Language-agnostic representation learning of source code from structure and context. In _International Conference on Learning Representations_.

## Appendix A Code2Code search results

We provide more detailed results for the Code2Code search, where we calculate the mean over the target language and present the MRR (Mean Reciprocal Rank) for each source language. This is similar to what is shown in Table 1 for CodeT5+. For CodeBERT, the details can be found in Table 3. For GraphCodeBERT, see Table 4. Results for UnixCoder are in Table 6, and for StarEncoder, refer to Table 5.

 


Figure 5: Effect of the rank \((r)\) of the language subspace on MRR change in zero-shot Code2Code search. The top row shows it for LRD, and the bottom row for CS-LRD.

Figure 6: Effect of the rank \((r)\) of the language subspace on MRR change in zero-shot Text2Code search. The top row shows it for LRD, and the bottom row for CS-LRD. ’Without English’ and ‘With English’ indicate cases where query embeddings remain untransformed and transformed, respectively.

 



\begin{table}
\begin{tabular}{c|c c|c c c c c c|c c c c c} \hline CodeBERT(means) & & Java & Javascript & RIP & Python & Avg. & CodeBERT (cls) & Java & Javascript & PIP & Python & Avg. \\ \cline{2-13}  & Original & 46.90 & 56.75 & 57.89 & 45.39 & 45.15 & **2.18** & Original & 51.34 & 54.02 & 57.42 & 56.15 & 47.23 \\ Monelingual & Cosine & 55.25 & 47.29 & 43.93 & 43.32 & 45.46 (+**2.28**) & & Containers & 64.16 & 62.21 & 71.11 & 53.86 & 43.44 (**15.84**) \\  & L



\begin{table}
\begin{tabular}{c|c c c c c c c c c c c c c c c c c c c} \hline \multicolumn{12}{l}{CaffeCar/content} & C & Cv & Cora & Java & Layout & Flow & Data & Avg. & Size & Context & Grid & Cost & Our & Cars & Java & Layout & Flow & Flow & Area \\ \hline \multirow{3}{*}{Monograph} & Original & 50.76 & 89.41 & 89.12 & 89.09 & 94.04 & 97.04 & 97.04 & 98.18 & 97.04 & **ClassClass(s)** & & & & & & & & & & & & & & & & & & \\  & Classifiers & 10.59 & 89.51 & 89.22 & 89.09 & 97.04 & 97.03 & 98.18 & 97.04 & **ClassClass(s)** & & & & & & & & & & & & & & & \\  & Classifiers & 10.59 & 89.51 & 89.22 & 89.09 & 97.04 & 97.05 & 98.19 & 97.04 & **ClassClass(s)** & & & & & & & & & & & & & \\  & Classifiers & 10.59 & 89.51 & 89.22 & 89.09 & 97.04 & 97.05 & 98.19 & 97.04 & **ClassClass(s)** & & & & & & & & & & & & & \\  & Classifiers & 10.59 & 89.51 & 89.22 & 89.09 & 97.04 & 97.05 & 98.19 & 97.04 & **ClassClass(s)** & & & & & & & & & & & & & \\  & Classifiers & 10.59 & 89.51 & 89.22 & 89.09 & 97.04 & 97.05 & 97.05 & 98.19 & 97.04 & **ClassClass(s)** & & & & & & & & & & & & \\  & Classifiers & 10.59 & 89.51 & 89.22 & 89.09 & 97.04 & 97.05 & 97.05 & 98.19 & 97.04 & **ClassClass(s)** & & & & & & & & & & & & & \\  & Classifiers & 10.59 & 89.51 & 89.22 & 89.09 & 97.04 & 97.05 & 97.05 & 98.19 & 97.04 & **ClassClass(s)** & & & & & & & & & & & & \\  & Classifiers & 10.59 & 89.51 & 89.22 & 89.09 & 97.04 & 97.05 & 97.05 & 97.05 & 97.05 & 97.05 & 97.05 & 97.05 & 97.05 & 97.05 & 97.05 & 97.05 & 97.05 & 97.05 & 97.05 \\  & Classifiers & 10.59 & 89.51 & 89.52 & 89.09 & 97.04 & 97.05 & 97.05 & 97.05 & 97.05 & 97.05 & 97.05 & 97.05 & 97.05 & 97.05 & 97.05 & 97.05 & 97.05 & 97.05 & 97.05 & 97.05 \\  & Classifiers & 10.59 & 89.51 & 89.52 & 89.09 & 97.04 & 97.05 & 97.05 & 97.05 & 97.05 & 97.05 & 97.05 & 97.05 & 97.05 & 97.05 & 97.05 & 97.05 & 97.05 & 97.05 & 97.05 \\  & Classifiers & 10.59 & 10.59 & 89.52 & 89.09 & 97.04 & 97.05 & 97.05 & 97.05 & 97.05 & 97.05 & 97.05 & 97.05 & 97.05 & 97.05 & 97.05 & 97.05 & 97.05 & 97.05 \\  & Classifiers & 10.59 & 10.59 & 89.51 & 89.52 & 89.09 & 97.05 & 97.05 & 97.05 & 97.05 & 97.05 & 97.05 & 97.05 & 97.05 & 97.05 & 97.05 & 97.05 & 97.05 \\  & Classifiers & 10.59 & 10.59 & 89.51 & 89.52 & 89.09 & 97.05 & 97.05 & 97.05 & 97.05 & 97.05 & 97.05 & 97.05 & 97.05 & 97.05 & 97.05 & 97.05 \\  & Classifiers & 10.59 & 10.59 & 10.59 & 10.59 & 89.52 & 89.09 & 97.05 & 97.05 & 97.05 & 97.05 & 97.05 & 97.05 & 97.05 & 97.05 & 97.05 \\  & Classifiers & 10.59 & 10.59 & 10.59 & 10.59 & 10.59 & 89.52 & 89.09 & 97.05 & 97.05 & 97.05 & 97.05 & 97.05 & 97.05 & 97.05 & 97.05 \\  & Classifiers & 10.59 & 10.59 & 10.59 & 10.59 & 10.59 & 10.59 & 10.59 & 10.59 & 10.59 & 10.59 & 10.59 & 10.59 & 10.59 & 10.59 \\  & Classifiers & 10.59 & 10.



* [25] M. C. Bader, A. Krizhevsky, and D. Vedalov. Learning from deep learning: a large-scale hierarchical network for image recognition. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pages 1038-1049, 2017.
* [26] J. L. Bottou, Y. Bengio, and R. Girshick. Generative adversarial nets with generative adversarial networks. In _Advances in Neural Information Processing Systems (NIPS'09)_, volume 1, pages 1-10, 2009.
* [27] S. Ghahramani, P. Dollar, and Z.-J. Sun. Deep residual learning for image classification. In _International conference on computer vision and pattern recognition_, pages 1-10, 2016.
* [28] H. Chen, X. Li, and T. Lin. Image-based face detection via deep convolutional neural networks. In _IEEE International Conference on Computer Vision and Pattern Recognition (CVPR)_, pages 1-10, 2016.

[MISSING_PAGE_POST]