CodeXEmbed: A Generalist Embedding Model Family for Multilingual and Multi-task Code Retrieval
Authors: Ye Liu
Rui Meng
Shafiq Joty
silvio savarese
Caiming Xiong
Yingbo Zhou
Semih Yavuz
Venue: COLM 2025 (2025)
Keywords: Code and Text Retrieval; Code Embedding Model; Text Embedding Model; Retrieval-Augmented Code Generation
TL;DR: We introduce CodeXEmbed, a large-scale code embedding model achieving SOTA on CoIR and strong BeIR performance, enhancing code retrieval and RAG.

Abstract: Despite the success of text retrieval in many NLP tasks, code retrieval remains a largely underexplored area. Most text retrieval systems are tailored for natural language queries, often neglecting the specific challenges of retrieving code. This gap leaves existing models unable to effectively capture the diversity of programming languages and tasks across different domains, highlighting the need for more focused research in code retrieval. To address this, we introduce CodeXEmbed, a family of large-scale code embedding models ranging from 400M to 7B parameters. Our novel training pipeline unifies multiple programming languages and transforms various code-related tasks into a common retrieval framework, enhancing model generalizability and retrieval performance. Our 7B model achieves a new state-of-the-art (SOTA) in code retrieval, topping the CoIR Leaderboard. In addition to excelling in code retrieval, our models demonstrate competitive performance on the widely adopted BeIR text retrieval benchmark, offering versatility across domains. Experimental results demonstrate that improving retrieval performance significantly enhances end-to-end Retrieval-Augmented Generation (RAG) performance for code-related tasks.

Published as a conference paper at COLM 2025
Code :Embed:: A Generalist Embedding Model Family for Multilingual and Multi-task Code Retrieval
Ye Liu, Rui Meng. Shafiq Joty, Silvio Savarese, Caiming Xiong, Yingbo Zhou, Semih Yavuz Salesforce Al Research
(yeliu, yingbo. zhou, syavuz) @salesforce.com
Abstract
Despite the success of text retrieval in many NLP tasks, code retrieval has received comparatively less attention than text retrieval. Most retrieval systems are designed for natural language queries and often fail to address the structural and semantic complexities of retrieving code. Consequently, they struggle across diverse pro- gramming languages and retrieval tasks, underscoring the need for specialized models tailored code retrieval. To address this gap, introduce CODEXEM- BED, a family of large-scale code embedding models ranging from 400M to 7B parameters. Our novel training pipeline integrates multiple programming languages and reformulates diverse code-related tasks within a unified retrieval framework, enhancing model generalizability and performance. Our largest model (7B parame- ters) sets a new statc-of-the oh-tth--rtttttt (SOTA) in code retrieval, ranking first on the ColR Leaderboard. Beyond code retrieval, our models achieve competitive results on the widely adopted BelR text retrieval benchmark, demonstrating cross ross-domain ver- satility. Furthermore, our findings highlight that advancements retrieval quality directly improve end-to-end Retrieval-Augmentee Generation (RAG) performance for code-related tasks.
Introduction
Large Language Models (LLMs) have demonstrated exceptional performance across numerous Natural Language Processing (NLP) tasks. However, they often struggle to produce faithful answers and may lack up-to-date dopaif-specifie knowledge. To bridge this gap, generation (RAG) Cai et al. (2022); Cheng et al. (2024); Nguyen et al. (2024) techniques have gained prominence, integrating Information Retrieval (IR) systems with LLMs to enhance their access to relevant external information. This synergy has drawn significant attention recently, leading to the development of various retrieval models Wang al. (2022); Chen et al. (2024) based on BERT Kenton & Toutanova (2019) and other LLMs with sizes exceeding I billion parameters Wang et al. (2023); Moreira et al. (2024); Meng et al. (2024). Despite these advancements, standard IR methods, while effective in text-based retrieval, often fall short in specialized domains such as code retrieval Husain et al. (2019); Li et al. (2024b).
Code retrieval accelerates software development and improves code quality by enabling quick access to relevant snippets, explanations, and analyses. Unlike general text retrieval, it must handle syntax, dependencies, and control flow. Integrated into tools like VS Code Del Sole (2021) and GitHub Copilot Wermelinger (2023); Yetiştiren et al. (2023), code retrieval also enhances Code-RAG systems Parvez et al. (2021); Liu et al. (2020); Jimenez et al. (2024); Wang et al. (2024) by reducing LLM hallucinations. However, traditional text retrieval models struggle with code-specific elements. Existing models like CodeBERT Feng al. (2020), CodeGPT Lu al. (2021), and UniXcoder Guo et al. (2022) are based on smaller BERT models Kenton & Toutanova (2019) leading to subpar performance. Furthermore, unified model capable of handling both text and code retrieval is essential for seamless integration, enabling developers to retrieve documentation, explanations, and relevant code within a single framework.
* Now at Gongle.

---

Published as a conference paper at COLM 2025
In this work, we introduce CODEXEMBED', a family of open-souree embedding models tailored for both code and text, available sizes of 400 million, 2 billion, and 7 billion parameters. CODEX- EMBED introduces a generalizable approach that converts diverse code-related tasks with various programming languages into a unified contrastive training framework. Our approach handles 12 programming languages and five distinct code retrieval categories across eight different code tasks, including code-to-text, text-to-code, code-to-code, text-to-text and hybrid text and code tasks. To enable the model retrieve both text and code effectively, we propose novel multi-stage training method. In the first stage, we train the model on text data using LoRA adaptation. In the second stage, we continue training with both code and text data while tuning only the newly introduced LoRA adapter. This approach preserves the text retrieval knowledge learned in the first stage while enhancing code retrieval, ensuring the model retains strong performance across both domains. This comprehensive setup enables BODEXEMBERI to generalize effectively across various code domains. Our contributions can be summarized as follows:
We introduce a generalizable multi-stage training approach for code and text embedding, unifying diverse code-related tasks within retrieval framework. This leads significant improvements in retrieval performance across multiple programming languages and tasks.
Our 7B model achieves state-of-the performance on the ColR benchmark, setting a on RepoEval and
new standard for code retrieval. We furhter evaluate CODE SWE-Bench-Lite, demonstrating that improved retrieval significantly enhances end-to-end Retrieval- Augmented Generation (RAG) performance for code-related tasks.
Beyond the 7B model, we develop smaller models (400M, 2B) that surpass prior SOTA in code retrieval, while remaining competitive performance in text retrieval, highlighting their versatility across both domains.
2 Method
We transform general code-related tasks into unified retrieval framework by representing each task as a structured retrieval pair D), where Q is the query and D is the positive document. The retrieval process formulated as ranking problem, where the objective to learn a function f(2 D) that assigns higher similarity scores to relevant pairs than that of irrelevant ones.
2.1 Unified Retrieval Framework
To train a unified retrieval model for both code and text, we transform all retrieval tasks into a query- document matching problem, where a query Q retrieves relevant document D from candidate set D. This formulation supports Text-to-Texx: Text-to-Code. Code-to-Text, and Code-to-Code retrieval within a single framework.
To better capture semantic relationships across entire input sequences, we incorporate bidirectional atlention into a causal LLM BehnamGhader et al. (2024), enhancing its ability to encode contextual information for retrieval tasks. Each query Q and document are encoded using this pre-trained LLM, producing representations:
4 = Le d= A=LLM++AeDD
The similarity between query and document is measured using cosine similarity:
9.d d || 9 d
D* arg max sim(g,d), d)
(I)
DED
2.2 Multi-Stage Training with LoRA
To progressively enhance retrieval capabilities, adopt multi-stage training approach, utilizing distinct LoRA Hu et al. (2022) adapters at each stage. In each stage, the previously trained LoRA weights are merged into the base model before initializing a new adapter, as shown in Figure 1. This approach enables the model to iteratively adapt to different retrieval objectives without catastrophic forgetting Magistri et al. (2024); Han al. (2024)
Model weights: https: htt.h/hggggggggg agface. co/Sa 1es forcee Embedd https://huggingffee co/Sa force iF knbeddi Cww.-ooddeooooo
2

---

Published as a conference paper at COLM 2025
Stage
Text
LoRAA
Stage
Text
Doco
E0841
LoRAZ
Cantinu iRand
optimszer
Figure 1: The illustration depicts multi-stage training, showing the data used at each stage, the LoRA adapters applied, and the continuous optimizer with its learning rate progression.
2.2.1 Stage Supervised Contrastive Pretraining on Text Data
We initialize a LoRA adapter A0₁ and freeze the LLM and train A0₁ on text-based retrieval tasks using a contrastive learning objective:
1 Σ log exp(sim(q::47)) Σ exp(sim(qi. N
exp(sim(g:.47))
L
(2)
where d is positive document, and 7 represents negative documents, and N denotes the batch size. The model learns align text queries with their corresponding text-based documents, establishing strong foundation for language-basse retrieval.
2.2.2 Stage Code-Enhanced Retrieval Training
After training, we merge the trained adapter A0, into the base LLM model:
- 0 + A0
(3)
and initialize a new LoRA adapter 402 to learn retrieval for both code and text. This setup enables the model to mitigate text-only biases while preserving most its learned representations.
To ensure smooth transition between training phases, continue optimizing the second-stage LoRA adapter Aθ2 while retaining the gradient contributions from the first-stage adapter A6,, which remains fixed within the model. Specifically, we update A82 while preserving the learning rate p(1) and the accumulated gradient from A0
A02 <- Update(AB2, (L()),))),r())
where (L 02) represents the gradient influenced by both adapters, but only A02 is actively updated. This approach stabilizes optimization, preventing abrupt parameter shifts while allowing smooth adaptation to code retrieval. By preserving the previous learning rate and gradient continuity, we mitigate convergence instability and facilitate efficient adaptation.
This multi-stage approach enables the model to iteratively improve retrieval performance while preventing overfitting to any single modality. By leveraging LoRA adapters at different stages and maintaining a smooth learning rate transition, we ensure efficient adaptation with minimal overhead, resulting in more robust text and code retrieval.
2.3 Scaling Batch Size with GradCache
Larger batch sizes improve contrastive learning by incorporating more diverse negatives Chen al. (2022), but GPU memory limits expansion. To overcome this, we use GradCache Gao al. (2021), which reduces memory overhead, enabling larger effective batch sizes. Let Q denote the full batch of queries and documents, partitioned into sub-batches (Q = (Ôi, QM))
D = (D.,,...,, to fit memory constraints. Training follows three steps:
Graph-less Forward Pass Each sub-batch is encoded into embeddings qi and d; without computing encoder gradients, reducing memory usage.
Gradient Computation and Caching For each query qi in sub-batch Qj. we compute and cache gradients w.r.t. the representation function:
3

---

Published as a conference paper at COLM 2025
Good Agent Conversation
Codo
Code Summaay
couid heuae felwwing
Enenaaio orer of poobrin
unidg uroand plvd snlimags
Coadsos uplai ana 12421
Coda Clone Detoction
Text
Query
Ataterwa
Oada
Walie begent artueet
SQL
H18--18 A010
Code Contest
Code Completion
houree clace snting number
Cade Translation
Camrati: (apparnp
Figure 2: The code training data of CODEXEMBED contains four parts: Text-to-Code, Code-to-Code, Code-to-Text and Hybrid Code. Each Categories contains several types of code tasks.
dc
ac
u:
Vi ad;
(5)
dq:
Sub-batch Gradient Accumulation We reprocess sub-batches to collect embeddings qi and di, construct the computation graph, and link gradients:
ac de
dqi ae
ad a0
= E
Σ Q
ui
Σ
(6)
DeEDdEDD;
a denotes the LoRA adapter in either stage 2. GradCache enables larger in-batch negatives, improving contrastive learning efficiency while staying within GPU limits, essential for scaling text and code retrieval.
2.4 Unified Retrieval Training Data
We organize retrieval tasks into distinct settings by converting various code-related tasks into unified retrieval format. In this format, the input is treated as query (Q), and the expected output or relevant content treated as document (D). This transformation allows the model to generalize across various retrieval tasks in both text and code domains.
2.4.1 Text Retrieval Training Data
To improve retrieval across both code and text domains, incorporate text-to-text retrieval tasks from the BEIR datasets Thakur et al. (2021). This ensures model robustness in general document retrieval while maintaining strong performance in code retrieval.
2.4.2 Code Retrieval Training Data
We unify multiple code generation and classification tasks into the (Q, format for code retrieval training, as shown in Figure 2. Text-to-Code Retrieval involves retrieving relevant code snippets given a natural language query. Tasks such as code contest generation Billah et al. (2024); Kadir et al. (2024) take problem descriptions as Q and map them to the correct implementation as D. Similarly, in Text-to-SQ.. Finegan- -Dolakk et al. (2018); Li et al. (2024a), a user query in natural language serves as Q. and the corresponding SQL query is treated as D. Code-to-Text Retrieval retrieves human-readable descriptions for given code snippets. In code summarization Sontakke et al. (2022); Sun et al. (2024), code snippet serves Q, and the corresponding documentation or summary is D. Code-to-Code Retrieval represents retrieving functionally equivalent or semantically related code snippets. In code translation Pan et al. (2024); Karanjai al. (2024), the source code in one language is Q, and its equivalent implementation in another language is D. Code completion Ding et al. (2024); Phan et al. (2024); Liu et al. (2024) uses the incomplete code fragment as Q and the
4

---

Published as a conference paper at COLM 2025
correct completion D. Code clone detection Martinez-Gii (2024) retrieves functionally similar code, treating the reference snippet as Q and the detected duplicate as D. Hybrii Code Retrieval supports mixed queries containing both text and code. In code agent conversation Arteaga Garcia et al. (2024); Jin et al. (2024), user prompts containing explanations or code snippets act as Q. while the retrieved response (either code or text) serves as D. In code issue fixing Yang al. (2024); Jimenez et al. (2024), the hybrid query consists of an error message and buggy code as Q, while the corrected version of the code is D.
By consolidating diverse retrieval tasks into unified framework, we effectively streamline contrastive model training, thereby facilitating seamless adaptation across a wide range of programming scenarios while ensuring high retrieval accuracy.
3 Experiments
Evaluation Benehmarks We mainly use two benchmarks evaluate code and text retrieval perfor- mance. COIR Li et al. (2024b) is a comprchensive benchmark specifically designed for code retrieval tasks. COIR covers a wide range of retrieval challenges, including fine- one-grained retrieval subtasks, spanning 14 major programming languages. The dataset is composed of 10 distinct datasets, with total corpus exceeding 2 million entries. BEIR Thakur et al. (2021) is a widely-adopted benchmark designed for text retrieval tasks. BEIR encompasses a diverse set of retrieval challenges, covering 9 distinct tasks across various domains such as question answering. duplicate detection, fact-checking, and more. It supports retrieval over a wide range of datasets and provides standardized benchmark for evaluating text retrieval models across different domains.
Implementation Details We conduct general training on proposed code and text pair dataset using three model sizes: 400M, 2B, and 7B. For the CodexXEmed..pm. we use the base model Alibaba- (Li et al., 2023b), applying full model fine-tuning. For the Code CodeedEmed we initialize our embedding model from the generation model google/gemma--2 2-2--i Team et al. (2024), using low-rank adaption(LoRA) Hu et al. (2022) with rank of 8. For the CodeXEmm,,dg, we initialize our embedding model from the generation model mistralaaiMisstall Bann--7---nntttt--d using LoRA with rank 8. Following prior work Meng et al. (2024), we apply: (i) last token pooling for CodeXEmbed and CodeXEmbedry. and (ii) beginning token pooling for CodeXEmbed4400 to generate semantic vector representations Li et al. (2023b). Cosine similarity is used to compute the similarity between query and corpus for ranking. The batch size is set to 1024 across all three model sizes, with 7 hard negatives. The learning rate is set to 5e- and the end learning rate to 5e with linear decay and a 50-step warmup. To improve training efficiency and reduce GPU memory usage, we adopt gradient caching Gao al. (2021). The more implementation details can be found in Appendix A.3.
also
Baseline Models For code-domain--peecffcc models, we included UniXcoder (Guo et al., 2022), Voyage-Code-0022 and CodeSage-lagge-22 (Zhang et al., 2024), all pre-trained on code data, serving as strong baselines for comparison. For general retrieval models, we evaluated E5-Base (Wang et al., 2022), GTE-Base (Li et al., 2023b), BGE-Base (Xiao et al., 2023), Contriever (Izacard et al., 2023), E5-Mistral (Wang et al., 2023), BGE-M3 (Chen et al., 2024), NV-Embed-V V2 Moreira et al. (2024), SFR-V2 Meng* et al. (2024) and OpenAl-Ada-0022.
Evaluation Metrics In code retrieval, selecting the right metric is key for assessing both ranking sensitivity and relevance. Following prior work Wang et al. (2013), Normalized Discounted Cumula- tive Gain (NDCG) is preferred for its ability to account for both rank order and varying relevance. Therefore, we use NDCG@ 10 to evaluate performance on ColR2 and BEIR.
3.1 General Training Evaluation
In the General Training block of Table 1, we present the results of models trained exclusively on our proposed general training data, without using any ColR in-domain data. When averaged over all 10 datasets the CoIR benchmark, CodeXEmbed7s model achieves the best results, exceeding the
Pnttps://blot 'https: http::::/laattttte opwnai.......co...o... com/ docs/ tttp////giie///e/eeeeaiiieeceede *CoIR Implementation https: / /gi thub. com team/coir
redeeretrieeal/
5

---

Published as a conference paper at COLM 2025
Test-to-Code
Codesto-Text
Code-to-Code
Model
Apps
CoeQA
Teat2SQL
CSN
CSN-CCR
Code -Coneest
StackOverFlow
CodeFoedBackk'
Avg
-DI.
QA
-ST
-MT
Baselines
10M)
BGE-Baus (110M)
4.05
32.76
45.59
69.60
45.56
38.50
21.71
73.55
64.99
31.42
42.77
UniXender ( (123M)
1.56
25.14
50.45
60.220
58.
41,82
31.03
44.67
36,02
24.21
57.33
BGGEMS (567M)
7.37
48.76
43.23
47.5
31.16
39.31
OpenAl-Adn-002
8.70
28, XX
58. 32
74.21
69.13
53.34
26.04
72.40
47.12
17:74
45.59
Voynge-Code-002 Code-002
26.52
29.79
69.26
81.79
73.45
72.77
27. 28
67.68
65.35
28.74
56.26
CodeSage-large-
50.45
32.73
59.78
94,26
78.09
85.27
33.29
79.41
71.32
57.16
General Training
48.57
34.15
58.66
72.53
80.15
75.67
34,85
89.51
78.87
43.75
61.89
CodeXFonheds
74.99
36,31
99.00
73,50
85.77
86.63
33.17
90.54
81.15
53,08
67.41
CodeXBanoorr
In-domain Traiming
CodeX Eedeltanholaoo
45.91
41.28
61.29
81.23
93.74
82.72
40.81
92.35
83.36
61.51
68.42
CodeXEmbooes
76.86 86
10:17
78.12
57.87
97.66
90.30
38.57
94.47
86.36
65.51
75.65
CodeXBmisodrg
Table 1: Performance of the CODEXEMBED model family across tasks, along with average scores. CSN refers to NodeSearchNet. Baseline numbers are from the CoIR Leaderboard.
02
72
40-
Ted Code
Code to
Oo Oode
Hobris
Figure 3: The performance comparison between General Training (GT) and In-domain Training (ID) across three model sizes (400M, 2B, and 7B) on different CoIR categories and the overall average.
SOTA code-domain specific model Voyage-Code-002 by over 20%5, which shows our general code and text training stage significantly improves model performance code tasks.
As shown in Table 1, CodeXEmmeddoom and CodeXEmbedhy also provide significant improvement over Voyage-Code-002 and offer great alternative the 7B model with substantial practical advantages on the latency and cost. Moreover, their success further validates the transferability and generalizability of our proposed training recipe for code embedding models.
3.2 In-domain Training Evaluation
We further trained the model on the ColR in-domain dataset. As shown in the In-domain Training block of Table I, further training on in-domain data results consistent performance improvements across all model sizes. Specifically, CodeXEmbed4oox improves by 6.5 points, X by 8.24 points, and CodeXEmbedhp by 7.74 points on average across all 10 datasets. Figure the top of each bar represents the improvement from in-domain training. Among all categories, the Code-to-Texx category shows the largest improvement, even outperforming Voyage-Code-003 In other categories, the model also achieves over a 5-point improvement.
3.3 Unified Text and Code Retrieval
To evaluate the text and code retrieval capabilities within a single embedding model, we also present the BEIR text retrieval performance of CODEXEMBED across various sizes. As shown in Table 2, 7B model achieves an average score of over 60 across 15 datasets, placing it among the top tier on the MTEB leaderboard Compared ES-Mistral-7B-Instt
(E5-Mistral) Jiang et al. (2023), which
5 Baseline numbers are sourced from ColR Leaderboard: https / http:///rchhhaaaaeeo gi thub, i coir/.
hettps:/huuggiie co/ spaces htneees///e/////eeeee leed//baddrbbrrd

---

Published as a conference paper at COLM 2025
BM25
gte-large 400M
gte-Qwen2
ES-Mistral 7B
CODEXEMBED
CODEXEMBED
CODEXEMBED
Dataset
400M
7B
MS MARCO
22.8
42.93
43.36
43.06
42.77
41.26
42.05
TREC-Covid
65.6
77 4949
85.38
87.03
77.47
84.58
79.04
NFCorpus
32.5
36.95
39.34
38.58
35.76
41.56
43.14
NQ
32,9
56.08
56.08
63.53
63.38
67.25
74.11
HosporQA
75.72
FQQA
23.6
63.23
63.23
56.81
60.20
56.17
60.41
ArguAna
72.11
54,70
69.67
63.58
Touche-2020
2020
36.7
22.55
27.89
26.27
20.18
26.10
25,80
CQADupStack
29.9
42.16
44.76
42.97
46.07
47.46
$1.45
Quora
78.9
89.67
89.64
89.61
89.05
89.27
89.51
DBPedia
31.3
46.30
48.69
48.
46.68
47.33
49.27
Scidocs
15.8
26.35
16.32
25.05
23.36
25.25
Fever
75.3
93,81
91.57
87.84
93.86
89.03
91.94
Climate-Feever
42,91
36.93
Seifact
66.5
82.43
78.44
76.42
87.37
84.79
89.10
Average
41.7
57.91
$8.29
56.87
58.34
57.73
60.06
Table 2: Comparison of performance text retrieval BEIR datasets with different model size.
trained on both text and synthetic data, initialized from the Mistral series, our model employs single-stage training with both code and text data. Our model in performance improvement of 3.19 points over E5-Mistral.
In the 400M models, CODEXEMBED achieves 0.43 performance boost over GT'-large (Li et al., 2023b), the model it is trained on. This highlights the advantage of our approach, showing the potential to improve text retrieval by incorporating code data. Few 2B-sized language models are available; we selected Gemma 2B Team et al. (2024) for its strong code retrieval performance. For text retrieval, it performs comparably gte- nweeQ of similar size.
NV-Embed-V2
SFR-v2 7B
CODRXEMBED
Dataset
7B
CoIR
59.10
61.48
70.46
BEIR
62.65
60.18
60.06
AVG
60.88
60.83
65.26
Table 3: Comparison of code and text retrieval benchmarks between our model and the top models on the MTEB leaderboard.
We present the performance of top text retrieval models from the MTEB leaderboard on both code and text tasks, in Table 3. Compared to the top-ranked model, NV-Embed-V2 V2 Moreira et al. (2024) our model surpasses by 4.38 points, with an average score across text and code datasets.
3.4 Impact of Multi-Stage Training
To evaluate the effectiveness of our multi-stage training approach, conduct an ablation study on four settings:
Single-stage (Baseline): Trained in single phase using mixture of text and code retrieval data with single LoRA adapter.
Two-stage (Same LoRA): The first stage is trained retrieval; the second stage continues with mixed text and code retrieval, reusing the same LoRA adapter across both stages.
Two-stage (Different Optimizers): Uses different LoRA adapters for each stage, with separate optimizers for each.
Proposed Two-stage (Separate LoRA): Applies different LoRA adapters for each stage, using continuous optimizer throughout training.
7https:/ http:///hhggggggggeeeeee htt.geacceeccooooooooooooooooooe spaces setaces/ntee satee///eei /tt////t/////////////////e/e/t http////eeeebboarrrrtrr Leaderboard performnn open model of 3/28/2025.
7

---

Published as a conference paper at COLM 2025
Training Setup
BEIR
ColR
Single-stage (Baseline)
65.79
64.56
Two-stage (Same LoRA)
67.54
58.61
Two-stage (Diff, Optimizer)
66.95
66.06
Proposed Two-stage (Separate LoRA)
67.88
67.41
Table 4: Comparison of training methods on BEIR dev and ColR test sets (NDCG@) 10).
Table 4 shows Proposed Two-stage (Separate LoRA) achieves the best performance BEIR (67.88) and ColR (67.41) highlighting the importance of modular adaptation. The Single-stage (Baseline) struggles on ColR, while Two-stage (Same LoRA) improves BEIR but drops ColR. Two-stage (Different Optimizers) balances performance but falls short of our approach.
3.5
Retrieval-Augmented Code Generation
In this section, explore how different retrievers influence the final code completion and issue resolution performance repository-leve tasks.
3.5.1 RepoEval
To address this, utilize RepoEval (Zhang et al., 2023) for appository-leved code completion. While RepoEval consists of three splits (function, class, and module), we report results only on the function split, it is the only one that supports execution-based evaluation. We adopt Pass @ 1 evaluation metric, which measures the accuracy of the top-1 generated code passing the provided test cases.
For code generation, supply the top-5 retrieved code snippets input to the GPT-3.5-ture generation model. All generator parameters and decoding hyperparameters are kept frozen throughout the experiments. As shown in Table 5, all sizes of CODEX
outperform the canonical setup. While some files may not contain direct solutions, as in the canonical documents, they often include valuable function definitions or usage examples that improve code generation outcomes. This suggests that our embeddings effectively capture the repository structure and retrieve contexts that implicitly support problem-solving.
Dataset
None
BM25
Voyage
OpenAl
OpenAl
COBEXEMBED
Gold
rerank
400M
2B
7B
gpt-3.5
RepoEval
23.9
30.8
43.2
48.0
49.6
52.5
66.3
63.8
SWE-Bench-Lite
0.7
1.0
0.7
0.3
0.0
0.7
2.0
3.0
2.7
gpt-4o
SWF-Bench-LLite
2.3
21.7
19.7
21.7
25.0
30.7
Table 5: Performance of repository-level code retrieval augmented generation using 30t=3.5-turbo-011 0525 and spt-40-2024-08-00 CODEXEMBED variants denote models of size 400M, 2B, and 7B.
3.5.2 SWE-Bemch-Lite
In experiments, use SWE-bench-Lite curated subset of 300 problems from the original SWE-bench benchmark. It focuses on resolving GitHub issues by requiring models modify multiple files pass test cases, offering manageable and reproducible dataset. SWE-hench-LLi also includes a pre-configured Docker container, ensuring consistent evaluation across systems and further standardizing the testing environment.
We evaluate our retrieval- -uggeeted generation pipeline using GPT-3.5-turbo as the code generator, with all generator parameters and decoding hyperparameters kept frozen. We also conduct parallel
l.ite
8

---

Published as a conference paper at COLM 2025
Model Size
Python
Java
Go
PIIP
Javaseript
Ruby
SQL
AVG
A-AII
400M
59.70
72.09
79.28
70.96
70.18
73.01
58.87
69.16
2B
65.73
79.40
81.01
79.39
76.76
78.42
58.23
74.14
7B
67.56
81.36
85.76
81.89
78.37
83.11
65.74
77.68
400M
55,43
49.57
49.56
40.57
44.57
44.57
44.21
46,93
32.1%
Python
2B
60.86
71.46
68.69
57.88
62.95
72.21
$6.09
64.31
13.3%
78
67.02
78.35
81.56
70.88
75.73
79.19
62.56
73.61
5.2%
400M
54.79
78.37
72.94
70.85
66.56
3.8%
Java
2B
63.03
82.17
79.35
80.30
76.00
76.76
58.03
73.66
0.6%
7B
66.24
84.25
83.72
80.18
79.82
82.82
64.59
77.37
0.4%
Table 6: Python/Java indicates that only the Python and Java portions were used to train CODEXEM- BED, while All indicates that all programming languages were used. A-All represents the difference between the Python/Java AVG score and the All AVG score for the same model size.
experiments with GPT-4o and observe similar trends, confirming that the performance gains from improved retrieval are consistent across model backends.
As shown in Table 5, our results show that incorporating improved retrieval methods significantly enhances end-to-end performance of code retrieval-auue -aggmettd generation, bringing it closer using gold content and boosting problem-solving efficiency and accuracy.
3.6 Impact of the Base Models
To understand the base model's 's impact, we examine: (1) if training from a text retrieval model outper- forms generation model, and (2) if code-specific generation model offers more advantages than general language model. We present the ablation study comparing an embedding model to a generation LLM in Appendix A.2.1 and the study on code-specific vs. general LLMs in Appendix A.2.2.
3.7 Programming Language Transferability
We aim to explore the diversity of programming languages and their unique features. The details of our code training dataset, including language coverage, are provided in Appendix A.1. Our dataset comprises 12 programming languages, with Python representing the highest percentage of the data. For testing, we selected Python and Java due to their distinct programming paradigms: Python is known for its scripting capabilities and Java for its strong object-oriented design. This selection allows us to evaluate our model's performance across a range of programming styles, reflecting the versatility and adaptability of the embedding model.
To better evaluate retrieval performance across languages, we organize the ColR test set into language- specific experiments by grouping all subsets associated with the same programming language. No additional resplitting is applied. For each language, we include all ColR test subsets labeled with that language. The details are shown in Table 7.
Language
ColR Test Subsets Used APPS CoSQA CodeSearchNett Python CedeSearchNerr CodeTrans-DD CodeSearchNet-Juee CodeSearchNee
Python
Java
Go
CodeScarchNee-Goo CaatseaachNee-CCR-GGG
PHP
PodeScarchNes--HiPP CodeSScarchNe--CCR-PHP
JavaScript
CodeSearchNee-JJvaase JavaScriptt ColeSSenchiee--CCR-avaasa
Ruby
CodeSearchhNet-Ree CodeScarchNNee-CCR-Ruby
SQL
Synthetic- Text2SQL
Table 7: CoIR test subsets used for each programming language.
As shown Table 6, training with all 12 programming languages yields the best average performance across 7 target languages, compared training with single language. However, training on Java-only consistently achieves the highest performance for Java and delivers comparable results to using all languages across all model sizes. For example, the Java-only 7B model scores 77.37, while the all-languages model scores 77.68. When comparing models trained exclusively on Python or Java, the Java-trained model consistently outperforms. This may be because modern language models

---

Published as a conference paper at COLM 2025
already heavily trained on Python during the generation phase, relying solely Python in the retrieval phase may miss important nuances, resulting in suboptimal performance.
4 Related Work
4.1 Retrieval Models for Text and Code
Text retrieval models have significantly advanced by exploiting supervision from natural language in- ference tasks and labeled query-document pairs, such the MS-MARCO passage ranking dataset Ba- jaj et al. (2016), to train effective text embeddings Izacard et al. (2021); Wang et al. (2022); Xiao et al. (2024). Recently, researchers have leveraged large language models (LLMs) Jiang et al. (2023) as the base for training retrieval models, resulting in saate-of-the-aee performance. Notable examples include E5-Mistral Wang et al. (2023), 3FR--Embedding Meng et al. (2024); Meng* et al. (2024), and NV-Embedding Lee et al. (2024). While numerous models have been developed for text retrieval code retrieval Zhang al. (2024); Suresh al. (2024). Among
tasks, few have focused specifically the few are the Voyage code model AI (2024) and OpenAl's embeddings Neelakantan et al. (2022); And embedding for code issue localization Reddy al. (2025) however, both are closed-source models, limiting their accessibility and adaptability for the wider research community.
4.2 Code Retrieval Augmented Generation
Neural code generation has been an important task Lu et al. (2021), and increasingly strong code language models have been developed Roziere et al. (2023); Nijkamp et al. (2023); Li et al. (2023a); Guo et al. (2024); Team (2024) to solve various tasks Chen et al. (2021); Lai al. (2023); Jimenez et al. (2024). However, most LMs generate code solely from natural language problem descriptions and the models' parametrie knowledge, without leveraging external programming resources or using eetrieval-augmennn generation approach. While prior work has focused on text-centric tasks with general-domain corpora like Wikipedia Asai et al. (2024), some studies have used retrieved programming context from repositories Ding et al. (2024); Yang et al. (2024) or documentation Zhou et al. (2023). Code retrieval crucial for enhancing code generation RAG systems because allows models to access relevant external code resources, leading to more accurate and context-awan code outputs. Our code retrieval model demonstrates significant improvements in code RAG performance, underscoring the importance of effective code retrieval in code generation tasks.
Conclusion
In the underexplored field of code retrieval, we present CODEXEMBED, family of code embedding models ranging from 400M to 7B parameters. Our unified training pipeline integrates multiple programming languages and reformulates diverse code-related tasks within a common retrieval framework. CODEX ranks as the top model on the ColR benchmark and achieves perfor- mance comparable to SOTA text retrieval models on BEIR. By enhancing retrieval capabilities, we significantly improve end-to-end retrieval-augmented generation for code-related tasks. Bridging the gap between text and code retrieval, we release our models to foster research and innovation in developer tools and programming language understanding.
References
Voyage AI. Voyage code Elevate your code retrieval. https://bbla //blog voyageai. ccmm/024/ code- e-eeeveeeeeee levate cowor-oeoo-etrrreeaaallllleelell January 2024. [Online; ac-
01/23/ voyage- ceceggee-ood---e cessed 8- 202204.
Emily Judith Arteaga Garcia, João Felipe Nicolaci Pimentel, Zixuan Feng, Marco Gerosa, Igor Steinmacher, and Anita Sarma. How to support ml end-user programmers through conversational agent. In Proceedings of the 46th IEEE/ACM International Conference on Software Engineering, pp. 1-12, 2024.
10

---

Published as a conference paper at COLM 2025
Akari Asai, Zexuan Zhong, Danqi Chen, Pang Wei Koh, Luke Zettlemoyer, Hannaneh Hajishirzi, and Wen-tau Yih. Reliable, adaptable, and attributable language models with retrieval. arXiv preprins 2024. 2024.
arXiv:2403.03187,
Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder,, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, al. Ms marco: A human generated machine reading comprchension dataset. arXiv preprint arXiv:1611.0926 2016.
Parishad BehnamGhader, Vaibhav Adlakha, Marius Mosbach, Dzmitry Bahdanau, Nicolas Chapados, and Siva Reddy. LIm2vec: Large language models are secretly powerful text encoders. arXiv preprint 0tt1::2404.0596.6 2024.
Md Mustakim Billah, Palash Ranjan Roy, Zadia Codabux, and Banani Roy. Are large language models a threat to programming platforms? an exploratory study. arXiv preprint arXiv:2409.05824, 2024.
Deng Cai, Yan Wang, Lemao Liu, and Shuming Shi. Recent advances in eetrieval-augmennn text generation. Proceedings of the 45th international ACM SIGIR conference on research and development in information retrieval, pp. 3417-3419, 2022.
Changyou Chen, Jianyi Zhang, Yi Xu, Liqun Chen, Jiali Duan, Yiran Chen, Son Tran, Belinda Zeng. and Trishul Chilimbi. Why do we need large batchsizes in contrastive learning? gradient-biar perspective. Advances Neural Information Processing Systems, 35:33860-333 -3875, 2022.
Jianlyu Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. M3-embedding: Aulti-linguality, funtiontuncionaaity,, multi- cratuiarrauarrtty text embeddings through eelf-knowledge distillation. In Findings of the Association for Computational Linguistics ACL 2024, pp. 2318-2335. 2024.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et Evaluating large language models trained on code. arXiv preprint arXiv:2107.0333 2021.
Xin Cheng, Di Luo, Xiuying Chen, Lemao Liu, Dongyan Zhao, and Rui Yan. Lift yourself up: Retrieval-augmented text generation with self-memory. Advances in Neural Information Processing Systems, 36, 2024.
Alessandro Del Sole. Introducing visual studio code. In Visual Studio Code Distilled: Evolved Code Editing for Windows, macOS, and Linux, pp. 1-15. Springer, 2021.
Yangruibo Ding, Zijian Wang, Wasi Ahmad, Hantian Ding, Ming Tan, Nihal Jain, Murali Krishna Ramanathan, Ramesh Nallapati, Parminder Bhatia, Dan Roth, et al. Crosscodeeval: A diverse and multilingual benchmark for cross-file code completion. Advances in Neural Information Processing Systems, 36, 2024.
Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang. al. Codebert: A pre-trained model for programming and natural languages. In Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 1536-1547, 2020.
Catherine Finegan- Finegan-Dollak, Jonathan K Kummerfeld, Li Zhang, Karthik Ramanathan, Sesh Sadasivam, Rui Zhang, and Dragomir Radev. Improving text-to-sql evaluation methodology. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pp. 351-360, 2018.
Luyu Gao, Yunyi Zhang, Jiawei Han, and Jamie Callan, Scaling deep contrastive learning batch size under memory limited setup. In Proceedings of the 6th Workshop on Representation Learning for NLP (RepLANLP-202)) pp. 316-321, 2021.
Daya Guo, Shuai Lu, Nan Duan, Yanlin Wang, Ming Zhou, and Jian Yin. Unixcoder: Unified cross-modee pre- pre-training trainini for code representation. In Proceedings of the 60th Anmual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 7212-7225, 2022.
11

---

Published as a conference paper at COLM 2025
Daya Guo, Qihao Zhu, Dejian Yang. Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, YK Li, et al. Deepseek-codee When the large language model meets programming the rise of code intelligence. arXiv preprint 2411::2401.14996 2024.
Jiayi Han, Liang Du, Hongwei Du, Xiangguo Zhou, Yiwen Wu, Weibo Zheng, and Donghong Han. Slim: Let Ilm learn more and forget less with soft lora and identity mixture. arXiv preprint urXiv:2410.07739, 2024.
Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022.
Xing Hu, Ge Li, Xin Xia, David Lo, Shuai Lu, and Zhi Jin. Summarizing source code with transferred api knowledge. In Proceedings of the 27th International Joint Conference on Artificial Intelligence, pp. 2269-2275, 2018.
Hamel Husain, Ho-Hsiang Wu, Tiferet Gazit, Miltiadis Allamanis, and Mare Brockschmidt. Code- searchnet challenge: Evaluating the state of semantic code search, arXiv preprint urXiv:1909.094: 2019.
Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. Towards unsupervised dense information retrieval with contrastive learning. arXiv preprint arXiv:21 2(3), 2021.
Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. Unsupervised dense information retrieval with contrastive learning. Transactions on Machine Learning Research, 2023.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.0682. 2023.
Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. Swe-hench: Can language models resolve real-world github issues? In The Twelfth International Conference on Learning Representations, 2024.
Hyoungwook Jin, Seonghee Lee, Hyungyu Shin, and Juho Kim. Teach ai how code: Using large language models teachable agents for programming education. In Proceedings of the CHI Conference on Human Factors in Computing Systems, pp. 1-28, 2024.
Md Eusha Kadir, Tasnim Rahman, Sourav Barman, and Md Al-Amin. Exploring the competency of chatgpt in solving competitive programming challenges. International Journal, 13(1), 2024.
Rabimba Karanjai, Lei Xu, and Weidong Shi. Solmover; Smart contract code translation based on concepts. In Proceedings of the 1st ACM International Conference on Al-Powered Software, pp- 112-121, 2024.
Jacob Devlin Ming- Wei Chang Kenton and Lee Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of naact.HHLT, volume 1, pp. 2. Minneapolis, Minnesota, 2019.
Mohammad Abdullah Matin Khan, M Saiful Bari, Do Long, Weishi Wang, Md Rizwan Parvez, and Shafiq Joty. Xcodeeval: execution- based large scale multilingual multitask benchmark for code understanding, generation, translation and retrieval. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pp. 6766-6805, 2024.
Yuhang Lai, Chengxi Li, Yiming Wang, Tianyi Zhang, Ruiqi Zhong, Luke Zettlemoyer, Wen-tau Yih, Daniel Fried, Sida Wang, and Tao Yu. Ds-1000: A natural and reliable benchmark for data science code gencration. In International Conference on Machine Learning, pp. 18319-18345. PMLR, 2023.
Chankyu Lee, Rajarshi Roy, Mengyao Xu, Jonathan Raiman, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. Ny-embed: Improved techniques for training Ilms as generalist embedding models. arXiv preprint 4005 240.222255222 /7428, 2024.
12

---

Published as a conference paper at COLM 2025
Jinyang Li. Binyuan Hui, Ge Qu, Jiaxi Yang, Binhua Li, Bowen Li, Bailin Wang, Bowen Qin, Ruiying Geng, Nan Huo, et al. Can Ilm already serve as a database interface? a big bench for aegee database grounded text-to-slls. Advances in Neural Information Processing Systems, 36, 2024a.
Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Mare Marone, Christopher Akiki. Jia Li, Jenny Chim, et al. Starcoder: may the source be with you! arXiv preprint arXiv:2305.06 2023a.
Xiangyang Li, Kuicai Dong. Yi Quan Lee, Wei Xia, Yichun Yin, Hao Zhang, Yong Liu, Yasheng Wang, and Ruiming Tang. Coir: A comprchensive benchmark for code information retrieval models. arXiv preprint arXiv:2407.02880 2024b.
Zehan Li, Xin Zhang, Yanzhao Zhang, Dingkun Long, Pengjun Xie, and Meishan Zhang. Towards general text embeddings with multi-stage contrastive learning. arXiv preprint 2rXiv:2308.03281, 2023b.
Shangqing Liu, Yu Chen, Xiaofei Xie, Jing Kai Siow, and Yang Liu. Retrieval- augmented generation for code summarization via hybrid gnn. In International Conference on Learning Representations 2020.
Tianyang Liu, Canwen Xu, and Julian McAuley. Repobench: Benchmarking repository-levdl code auto-completion systems. In The Twelfth International Conference Learning Representations, 2024.
Anton Lozhkov, Raymond Li. Loubna Ben Allal, Federico Cassano, Joel Lamy- Lamy-Poirier, Nouamane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, et al. Starcoder and the stack v2: The next generation. arXiv preprint 240.:2200211977 2024.
Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin Clement, Dawn Drain, Daxin Jiang. Duyu Tang, et al. Codexglue: A machine learning benchmark dataset for code understanding and generation. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1), 2021.
Simone Magistri, Joost van de Weijer, Andew Bagdanov, et al. An empirical analysis of forgetting in pre-trained models with incremental low-rank updates. arXiv preprint arXiv:2405.1808 2024.
Jorge Martinez-Gil. Source code clone detection using unsupervised similarity measures. In International Conference Software Quality, pp. 21-37. Springer, 2024.
Rui Meng*, Ye Liu Shafiq Rayhan Joty, Caiming Xiong, Yingbo Zhou, and Semih Yavuz. Sft-embedding-2: Advanced text embedding with multi-stage training, 2024. URL https: // //huggingface...
/tt/a/allffforee/////ettaaaaaaaa -_R..
Rui Meng, Ye Liu, Shafiq Rayhan Joty, Caiming Xiong, Yingbo Zhou, and Semih Yavuz. Sfr- seeedddingmistrall Enhance text retrieval with transfer learning. Salesforce Al Research Blog, 2024. URL https:://// //bl:g/bl//.//o//.e wawesforreaireeeeee.e.......e..e ttt.eeree..........eeeeaaeeaeeae emberdeed-iistral/
Gabriel de Souza P Moreira, Radek Osmulski, Mengyao Xu, Ronay Ak, Benedikt Schifferer, and Even Oldridge. Nv-retriever: Improving text embedding models with effective hard-negative mining, arXiv preprint arXiv:2407. 15..33317...3,,, 2024.
Arvind Neelakantan, Tao Xu, Raul Puri, Alec Radford, Jesse Michael Han, Jerry Tworek, Qiming Yuan, Nikolas Tezak, Jong Wook Kim, Chris Hallacy, et al. Text and code embeddings by contrastive pre-training. arXiv preprint arXiv:2201.10000 2022.
Xuan-Phi Nguyen, Shrey Pandit, Senthil Purushwalkam, Austin Xu, Hailin Chen, Yifei Ming, Zixuan Ke, Silvio Savarese, Caiming Xong, and Shafiq Joty. Sfr-rag: Towards contextually faithful llms. arXiv preprint 2xXiv:2409.09116 2024.
Erik Nijkamp, Hiroaki Hayashi, Caiming Xiong, Silvio Savarese, and Yingbo Zhou. Codegen2: Lessons for training lims on programming and natural languages. arXiv preprint wrXv:2305..02309 2023.
13

---

Published as a conference paper at COLM 2025
Rangeet Pan, Ali Reza Ibrahimzada, Rahul Krishna, Divya Sankar, Lambert Pouguem Wassi, Michele Merler, Boris Sobolev, Raju Pavuluri, Saurabh Sinha, and Reyhaneh Jabbarvand. Lost in translation: A study of bugs introduced by large language models while translating code. In Proceedings of the IEEF/ACM 46th International Conference on Software Engineering, pp. 1-13, 2024.
Md Rizwan Parvez, Wasi Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai-Wei Chang. Retrieval augmented code generation and summarization. In Findings of the Association for Computational Linguistics: EMNLP 2021, pp. 2719-2734, 2021.
Huy N Phan, Hoang N Phan, Tien N Nguyen, and Nghi DQ Bui. Repohyper: Better context retrieval is all you need for repository-level code completion. arXiv preprint arXiv:2403.0600: 2024.
Revanth Gangi Reddy, Tarun Suresh, JaeHyeok Doo, Ye Liu, Xuan Phi Nguyen, Yingho Zhou, Semih Yavuz, Caiming Xiong, Heng Ji, and Shafiq Joty. Swerank: Software issue localization with code ranking. arXiv preprint arXiv:2505.00 07849, 2025.
Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, et al. Code Ilama: Open foundation models for code. arXiv preprint arXiv:2308. 2023.
Ankita Nandkishor Sontakke, Manasi Patwardhan, Lovekesh Vig, Raveendra Kumar Medicherla, Ravindra Naik, and Gautam Shroff. Code summarization: Do transformers really understand code? In Deep Learning for Code Workshop, 2022.
Weisong Sun, Chunrong Fang, Yuchen Chen, Quanjun Zhang, Guanhong Tao, Yudu You, Tingxu Han, Yifei Ge, Yuling Hu, Bin Luo, al. An axtrative-anddaabstraiivv framework for source code summarization. ACM Transactions on Software Engineering and Methodology, 33(3):1-39, 2024.
Tarun Suresh, Revanth Gangi Reddy, Yifei Xu, Zach Nussbaum, Andriy Mulyar, Brandon Duderstadt, and Heng Ji. Comstacck High-quality contrastive data for better code ranking. arXiv preprint arXiv:2412.01007, 2024.
Jeffrey Svajlenko and Chanchal K Roy. Evaluating clone detection tools with bigclonebench. In 2015 IEEE international conference on software maintenance and evolution (ICSME), pp. 131-140. IEEE, 2015.
CodeGemma Team. Codegemma: Open code models based gemma. arXiv:2406.1 11409, 2024.
arXiv preprint
Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based on gemini research and technology. arXiv preprint 2iV.:2400.0825,, 2024.
Together AI Team et al. Redpajama-data-v2: An open dataset with 30 trillion tokens for training large language models, october 2023. URL http:://ww: logether. aühhlogreedpiamn--dlae--
Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. Beir: heterogeneous benchmark for zero-shot evaluation of information retrieval models. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2). 2021.
Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. Text embeddings by weakly-supervie supervised contrastive pre-training. arXiv preprint arXiv:2212.0355. 2022.
Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. Improving text embeddings with large language models. arXiv preprint arXiv:2401.00368, 2023.
Yining Wang, Liwei Wang, Yuanzhi Li. Di He, and Tie- Yan Liu. A theoretical analysis of ndeg type ranking measures. In Conference on learning theory, pp. 25-54. PMLR, 2013.
Zora Zhiruo Wang, Akari Asai, Xinyan Velocity Yu, Frank F Xu, Yiqing Xie, Graham Neubig, and Daniel Fried. Coderag-bench: Can retrieval augment code generation? arXiv preprint arXiv:2406. 14497, 2024.
14

---

Published as a conference paper at COLM 2025
Michel Wermelinger. Using github copilot solve simple programming problems. In Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. I, pp. 172-178, 2023.
Shitao Xiao, Zheng Liu, Peitian Zhang, and Niklas Muennighof. C-pack: Packaged resources to advance general chinese embedding. arXiv preprint wrXiv22309.07597 2023.
Shitao Xiao, Zheng Liu, Peitian Zhang, Niklas Muennighoff, Defu Lian, and Jian- Nie. C-pack: Packed resources for general chinese embeddings. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 641-649, 2024.
John Yang, Carlos E Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. Swe-agent: Agent-come comptce interfaces enable automated software engineering. arXiv preprint arXiv:2405. 10.440555599999 15793, 2024.
Burak Yetiştiren, Işik Özsoy, Miray Ayerdem, and Eray Tüzün. Evaluating the code quality of ai-assisted code generation tools: An empirical study on github copilot, amazon codewhisperer, and chatgpt. arXiv preprint 1rXiv:2304.10778, 2023.
Tao Yu, Rui Zhang. Kai Yang, Michihiro Yasunaga, Dongxu Wang. Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, et Spider: large-scale human-laheled dataset for complex and cross-domain semantic parsing and text-to-sql task. In Proceedings of the 2018 Conference Empirical Methods in Natural Language Processing, pp. 3911-3921, 2018
Dejiao Zhang, Wasi Uddin Ahmad, Ming Tan, Hantian Ding, Ramesh Nallapati, Dan Roth, Xiaofei Ma, and Bing Xiang. Code representation learning scale. In The Twelfth International Conference on Learning Representations, 2024.
Fengji Zhang, Bei Chen, Yue Zhang, Jacky Keung, Jin Liu, Daoguang Zan, Yi Mao. Jian Stan-Gaagg Lou, and Weizhu Chen. Repocoder: Repository-level code completion through iterative retrieval and generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 2471 2471-2484, 2023.
Victor Zhong, Caiming Xiong. and Richard Socher. Seq2sql: Generating structured queries from natural language using reinforcement learning. arXiv preprint 100000070,.000033 2017.
Shuyan Zhou, Uri Alon, Frank F Xu, Zhengbao Jiang, and Graham Neubig. Docprompting: Gen- erating code by retrieving the docs. In The Eleventh International Conference on Learning Representations, 2023.
15

---

Published as a conference paper at COLM 2025
Popula Prugramming Languages KLog Suard)
1.04.00
GICC
XVCC
300,900
Pylee
LM
ICN
2
Lirgagee
Figure 4: The programming language distribution of code training data in the general training stage.
A Appendix
A.1 Dataset Details
Our training dataset contains total 3.36M training pairs, covering 12 different programming
languages. As shown in Figure 4, the distribution of programming languages is imbalanced, with the majority of the data concentrated a few popular languages. Python represents the largest portion of the dataset at 27.1%, followed by Go with 25.2% and JavaScript and PHP at 17.0% and 17. 17.2%, respectively. The remaining languages, including Java, SQL., Ruby, and others, account for smaller proportions, with Rust, Kotlin, and C# making up the smallest shares of the dataset.
We follow Text2SQL works Yu et al. (2018) and Zhong et al. (2017), converting the text part into the queries retrieval SQL as the documents, derived from Wikipedia data. In contrast, the ColR benchmark uses synthetic-tex(2sq.. generated using weak and strong LLMs. Code Contest: We extract the Code Contest tasks in the Khan al. (2024), which has 11 programming language like java, c, c# et al. While ColR benchmark use APPS, which is the interview question only in python programming language.
Text-to-Code Retrieval: Text2SQL:
Code-to-Text Retrieval: Code Summary: We follow Hu et al. (2018) and Liu et al. (2020), which summarize code based on associated documentation, while CoIR utilizes a CodeSearchNet variant with code-comment pairs for the summary task. Code Clone Detection: We follow Svajlenko & Roy (2015) and Lu al. (2021) to convert the code as query and its clone code as the does CoIR has no dataset for this task.
Code Retrieval: Code Translation: We follows Khan al. (2024) and Lu et (2021), converting code in one language into queries and corresponding code in another language into documents. CoIR has no dataset for this task. Code Completion: We use the code completion tasks from Khan al. (2024) and Lu et al. (2021), while ColR uses modified CodeSearchNet, splitting code snippets into query- mocummdo pairs.
Hybrid Code Retrieval: Code Issue Fix: We follow Khan al. (2024) and Lu et al. (2021) to involve the code issue fix task; CoIR has no dataset for this task. Code Agent Conversation: We process the conversation data from Team et al., filtering answers based favorite scores and excluding data used in CoIR.
A.2 Impact of the Base Models
We analyze the base model's 's impact by comparing (1) text retrieval vs. generation models for training and (2) code-specific generation model offers advantages general language model.
16

---

Published as a conference paper at COLM 2025
gte-Qwen2
Gemma-v2 GT
SFR-v2
Mistral
Dataset
Initial
GT
Initial
GT
GT
Size
1.5B
2B
7B
7B
ColR BEIR
68.52
67.41 57.73
61.28
69.72
70.40
58.29
59.12
60.18
60.62
60.06
Table Comparison of base models: Retrieval vs. Generation Models. GT represents our general training.
Code-Specific LLMSs
General-Domain L.MSs
StarCoder-F2
DeepSeek-Codet
Gemma-v2
Mistral
Dataset
2B
7B
ColR
66.95
71.66 50.22
67.41
70.40
BEIR
49,06
57.73
60.06
Table 9: Comparison of base models: Code-specific vs. General-domain Generation Models.
A.2.1 Embedding Models v.s. LLMs
As shown in Table 8, the text retrieval model offers stronger starting point, and additional training with our approach enhances both its text and code retrieval capabilities. For instance, gte- nre-Qwen2 10 CoIR performance improves from 62.96 68.52, while its text performance increases from 58.29 to 59. In contrast, the text generation model requires more extensive fine-tuning to reach similar performance. However, the advantage of retrieval models can sometimes hinder code retrieval performance, as seen with SFR-V2 Meng* et al. (2024) underperforming compared to Mistral in specific tasks. This suggests that prior knowledge from text-focused models may not always transfer well to code-specifie scenarios. To pursue more general training approach, we chose to train using generation model rather than a text retrieval model.
A.2.2 Code-Specific LLMs General LLMs
We evaluate whether to choose code-specific models Lozhkov al. (2024); Guo et (2024) or general LLMs Jiang et al. (2023); Team et al. (2024), As shown in Table 9, code-specific LLMs excel code tasks but underperform text tasks, while general LLMs perform well in both. This suggests that recent advancements in general LLMs have integrated code data into their training Team et al. (2024), and this capability can be effectively transferred code retrieval. This finding highlights the versatility of general LLMs, making them viable for both text and code retrieval without the need for specialized models.
A.3 Implementation Details
We summarize the base model detail in Table 10 and hyperparameters in Table 11. For the code training data, we prepend a prompt to the query in the format: "Instruct: Given Code or Text, retrieve relevant content. Query:". We use
Model Name
Model Size
Version Number
Date of Release
ste-large-nn-i... gemma momma-22. 2 2-2b-it
400M
v1.5
Approximatcly November 2023 Approximately June 2024 May 2024
2B
vl
7B
v0.3
Table 10: Model details including name, size, version, and release date.
(Phttps://www aindel s. Fyí /mwd//ooooecoooo haw.de////hggiinnscc//geeeeeeeee instruct- albbbb--n
17

---

Published as a conference paper at COLM 2025
LodeXEEmee..oo
CodeXEmbedgg
CodeXEmbed7s
base model
gte- large-en-v1.5
gemma-2- 2-----22bt
Mistral
7B- -u-tsst
max learning rate
5 10
5 x 10-5
5 10-5
random seed
42
42
42
GradCache
4
16
32
tuning parameters
Fully Model
LoRA
LoRA
Lora Rank
8
warmup steps
100
50
50
batch size
1024
1024
1024
max length
512
512
512
weight decay
0.01
0.01
0.01
hard negatives
7
bidirvctional attention
/
text:code ratio
1:1
1:3
1:3
pooling
bos
eos
eos
Table 11:
Hyperparameters for contrastive code and text training
18


Reviews:
[{"review_id": "paBJ4CQNjC", "reviewer": "Reviewer_XpJL", "summary": "CodeXEmbed is a family of embedding models, ranging from 400M to 7B parameters, designed for multilingual and multi-task code retrieval. These models utilize a training pipeline that unifies diverse code-related tasks and multiple programming languages within a single retrieval framework. CodeXEmbed's largest model (7B parameters) achieved state-of-the-art performance in code retrieval, ranking first on the COIR Leaderboard, and also demonstrated competitive results on the BeIR text retrieval benchmark. The paper highlights that improvements in retrieval quality directly enhance end-to-end Retrieval-Augmented Generation (RAG) performance for code-related tasks.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper introduces a generalizable approach that reformulates diverse code-related tasks and multiple programming languages into a unified contrastive training framework for code retrieval. This enhances model generalizability and performance across various code domains.\n- The largest CodeXEmbed model (7B parameters) achieves a new state-of-the-art (SOTA) in code retrieval, ranking first on the COIR Leaderboard. This is a significant improvement over prior models.\n- The models demonstrate cross-domain versatility by achieving competitive results on the widely adopted BeIR text retrieval benchmark, in addition to excelling in code retrieval. This shows their ability to handle both text and code effectively.\n- The findings directly highlight that advancements in retrieval quality lead to improved end-to-end Retrieval-Augmented Generation (RAG) performance for code-related tasks.", "reasons_to_reject": "- Limited novelty as all the pieces of this work already exist in the literature.\n- The paper is positioned in a confusing way. It seems like the developed embedding models were primarily targeted for code related tasks, but then the authors addressed text domain tasks as well.\n- This paper is yet another paper with lot of results (indeed they are strong). However, we know, \"more data, more training, large model\", this recipe works. What other knowledge does this paper share? I am not sure. Just a couple of lines (Line no. 34-37) to discuss the limitations of the prior works, it is a shame.", "questions_to_authors": "- What was the actual focus of the work? Code domain or text domain or both?\n- The improvements demonstrated in Table 1 is too good. Have the authors ensured decontamination of the evaluation benchmarks?", "extra_scores": {}}, {"review_id": "EhYOs3RNGO", "reviewer": "Reviewer_eN6C", "summary": "- The paper introduces CodeXEmbed, a family of multilingual and multi-task code embedding models ranging from 400M to 7B parameters, aiming to unify code and text retrieval through a multi-stage LoRA-based training pipeline\n- CodeXEmbed’s 7B model reports a new state-of-the-art on the CoIR code retrieval benchmark (https://archersama.github.io/coir/) with an average nDCG@10 of 70.46, surpassing prior open and closed-source models by over 4 points, while the 2B and 400M variants also outperform similarly sized baselines\n- CodeXEmbed's 7B model also showed competitive text retrieval, scoring 60.06 on BEIR\n- Ablation studies in the paper investigate multi-stage training, and results suggest separate LoRA adapters with a continuous optimizer yield better performance", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Solid new models (as well as the code and data to reproduce them, hopefully) the paper introduces\n- Investigation of the impact of pretraining/finetuning on specific subsets of data (e.g. just Java) and more broadly the language transfer analysis\n- The introduction and evaluation of the two-stage training pipeline with separate LoRAs", "reasons_to_reject": "- The paper lists hyper-parameters such as batch size 1024, rank 8 LoRA and learning-rate $5 \\times 10^{-5}$ in Table 10 yet gives no information on training hardware, total update steps or FLOPs, leaving reproducibility and energy cost quite difficult to verify\n- No statistical variance, confidence intervals or significance tests accompany single-run metrics on the ten CoIR datasets, leaving it unclear whether reported gains over baselines exceed random fluctuation\n- In Section 3.5.1 the authors use gpt-3.5-turbo as the code generation model but do not specify its concrete version (e.g. `gpt-3.5-turbo-0125` or `gpt-3.5-turbo-1106`) making their experiments difficult to reproduce. We would also like to note that the model is generally considered outdated as even within OpenAI's offering there are models that seem to be both cheaper and report higher performance (e.g. GPT-4o mini)\n- Analysis in Table 6 indicates that for Java, a model trained solely on Java (84.25 NDCG@10) surpasses the model trained on all 12 languages (81.36 NDCG@10 for Java) [Section 3.7]. This finding somewhat contradicts the broader narrative of multilingual training benefits for every included language and suggests specialized models could be more effective\n- This research shows imbalanced language coverage in training data with Python (27.1%), Go (25.2%), JavaScript (17.0%) and PHP (17.2%) dominating, while languages like Rust, Kotlin, and C# have minimal representation, potentially affecting cross-language performance consistency", "questions_to_authors": "1. It is unclear what dataset was the experiment in Section 3.7 done on -- could you please clarify?", "extra_scores": {}}, {"review_id": "jVMIT2T3Sn", "reviewer": "Reviewer_RZhx", "summary": "This paper introduces CodeXEmbed, a family of multilingual and multi-task embedding models (400M to 7B parameters) designed to unify code and text retrieval within a single framework. Leveraging a multi-stage LoRA-based training pipeline, the models are trained first on text retrieval and then jointly on code and text tasks across 12 programming languages and five retrieval settings. CodeXEmbed achieves state-of-the-art results on the CoIR benchmark and competitive performance on the BEIR benchmark, demonstrating strong generalizability. The paper also shows that improved retrieval performance enhances downstream code generation tasks (e.g., RepoEval, SWE-Bench-Lite), positioning CodeXEmbed as a robust foundation for retrieval-augmented generation (RAG) in software development.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper proposes a novel multi-stage contrastive training framework using LoRA adapters that effectively balances performance across both text and code retrieval tasks. This modular fine-tuning strategy avoids catastrophic forgetting and is shown to outperform single-stage and naïve two-stage baselines significantly.\n- The 7B model sets a new SOTA on the CoIR benchmark, achieving a +20% improvement over prior best models. The paper also demonstrates significant downstream benefits of improved retrieval quality on code generation tasks.", "reasons_to_reject": "- While the paper proposes a multi-stage LoRA-based training strategy, the core methodology—contrastive learning with query-document pairs—is standard in retrieval literature. The approach largely builds upon existing methods (e.g., LoRA, GradCache) without introducing fundamentally new architectures or training paradigms.\n- The retrieval-augmented generation experiments use GPT-3.5 for code synthesis, which may obscure the retriever’s standalone contribution. It's unclear how much of the performance gain is attributable to the improved retriever versus the strong decoder model.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mq913qs64S", "reviewer": "Reviewer_H4PJ", "summary": "This paper introduces CODEXEMBED, code embedding models ranging from 400M to 7B parameters. The model works for multiple programming languages and multiple code-related tasks in a unified retrieval framework. The paper claims that the 7B parameter model is SOTA for code retrieval on the CoIR Leaderboard, has competitive results on the BeIR text retrieval benchmark. The paper claims that improvements to retrieval quality directly improve RAG performance for code-related tasks.", "questions": "", "limitations": "", "rating": 9, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Section 3.4 examines the impact of multi-stage training examining 4 configurations:\n* Single stage Training on a mixture of text and code retrieval data with a single LoRA adapter. \n* Two-Stage Training (with the same LoRA) where the first stage trains on text retrieval, followed by mixed text and code retrieval in the second stage, using the same LoRA adapter in two stages. \n* Two-Stage Training (with different optimizers) using different LoRA adapters with different optimizers for each stage. \n* A two-stage approach using separate LoRAs\nThe separate LoRA approach appears to help slightly over the common LoRA variant on BEIR\nThe performance on across text retrieval BEIR datasets seems to indeed be SOTA.\nTable 1 has some pretty comprehensive tests.", "reasons_to_reject": "None. I think this paper is pretty good.", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: CodeXEmbed is a family of multilingual and multi-task code embedding models, ranging from 400 million to 7 billion parameters, designed to unify code and text retrieval within a single framework. Trained using a multi-stage LoRA-based pipeline across 12 programming languages and five retrieval tasks, CodeXEmbed achieves state-of-the-art performance in code retrieval, with its 7B model topping the CoIR benchmark and outperforming both open- and closed-source baselines. It also shows competitive results on the BEIR text retrieval benchmark (score: 60.06), while the smaller 2B and 400M variants surpass similarly sized models. Ablation studies demonstrate that using separate LoRA adapters with continuous optimization yields better performance. The paper further highlights that improvements in retrieval quality lead to better performance in downstream retrieval-augmented generation (RAG) tasks such as RepoEval and SWE-Bench-Lite, positioning CodeXEmbed as a strong foundation for code-centric RAG applications.

The work is interesting and valuable, particularly due to its support for both multilingual and multi-domain code retrieval. The two-stage training approach, using separate LoRA adapters, offers a slight but consistent improvement over the standard LoRA variant on the BEIR benchmark. The model's performance across BEIR text retrieval datasets appears to be state-of-the-art, and Table 1 presents a comprehensive evaluation across multiple settings. Overall, I find the contributions solid and the results convincing, and I would recommend the paper for acceptance.

Suggestions for Improvement:
- As suggested by a reviewer, please include the evaluation script along with a --seed flag in the released training code to enable reproducibility and support extensions for significance testing.
- As mentioned by the authors, in Section 3.4, more clearly highlight the gains from the two-stage separate-LoRA approach to better emphasize its impact. To further support reproducibility, consider releasing the model checkpoints and scripts used to produce the BEIR and CoIR results.
Decision: Accept