CoRNStack: High-Quality Contrastive Data for Better Code Retrieval and Reranking
Authors: Tarun Suresh
Revanth Gangi Reddy
Yifei Xu
Zach Nussbaum
Andriy Mulyar
Brandon Duderstadt
Heng Ji
Venue: ICLR 2025 Poster (2025)
Keywords: code representation learning
code re-ranking
contrastive learning
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: We introduce CoRNStack, a large-scale, high-quality contrastive training dataset curated using consistency filtering to eliminate noisy positives and further enriched with mined hard negatives.

Abstract: Effective code retrieval plays a crucial role in advancing code generation, bug fixing, and software maintenance, particularly as software systems increase in complexity. While current code embedding models have demonstrated promise in retrieving code snippets for small-scale, well-defined tasks, they often underperform in more demanding real-world applications such as bug localization within GitHub repositories. We hypothesize that a key issue is their reliance on noisy and inconsistent datasets for training, which impedes their ability to generalize to more complex retrieval scenarios. To address these limitations, we introduce CoRNStack, a large-scale, high-quality contrastive training dataset for code that spans multiple programming languages. This dataset is curated using consistency filtering to eliminate noisy positives and is further enriched with mined hard negatives, thereby facilitating more effective learning. We demonstrate that contrastive training of embedding models using CoRNStack leads to state-of-the-art performance across a variety of code retrieval tasks. Furthermore, the dataset can be leveraged for training code reranking models, a largely underexplored area compared to text reranking. Our finetuned code reranking model significantly improves the ranking quality over the retrieved results. Finally, by employing our code retriever and reranker together, we demonstrate significant improvements in function localization for GitHub issues, an important
component of real-world software development.

Published as a conference paper at ICLR 2025
CORNSTACK: HIGH-QUALITY CONTRASTIVE DATA FOR BETTER CODE RETRIEVAL AND RERANKING
Tarun Suresh 1,3+f Revanth Gangi Reddy Yifei Xu Zach Nussbaum" 3
Andriy Mulyar Brandon Duderstadt Heng Ji' University of Illinois Urbana- Chammaagnn (tsuresh3, revanth3, wicehh,,yiiiiiie yifeix5, heneeisihengnjeelliinnoo edu
LLapis Labs Nomic Al
ABSTRACT
Effective code retrieval plays crucial role in advancing code generation, bug fix- ing, and software maintenance, particularly as software systems increase in com- plexity. While current code embedding models have demonstrated promise in retrieving code snippets for small-scale, well-defined tasks, they often underper- form in more demanding real-world applications such bug localization within GitHub repositories. We hypothesize that a key issue is their reliance on noisy and inconsistent datasets for training, which impedes their ability to generalize to more complex retrieval scenarios. To address these limitations, we introduce CORNSTACK large-scale, high-quality contrastive training dataset for code that spans multiple programming languages. This dataset is curated using consis- tency filtering to climinate noisy positives and is further enriched with mined hard negatives, thereby facilitating more effective learning. We demonstrate that con- trastive training of embedding models using CORNSTACK leads to state-of-the- art performance across variety of code retrieval tasks. Furthermore, the dataset can be leveraged for training code reranking models, a largely underexplored area compared text reranking. Our finetuned code reranking model significantly im- proves the ranking quality over the retrieved results. Finally, by employing our code retriever and reranker together, we demonstrate significant improvements function localization for GitHub issues, an important component of real-world software development.
INTRODUCTION
The rapid advancement of software development has led to an increased reliance on automated tools for code generation (Chen et al. 2021 et al., 2022b Nijkamp al.) Ugare et 2l.44b)44b [Banejeee 2025 Ugare et al. [2024a] Suresh et al. (025), As codebases grow in both size and complexity. the ability to efficiently search for and retrieve relevant code snippets is important. Code retrieval is crucial for advancing Retrieval-Augmenter Code Generation (RACG) (Wang et al. 2024b) with large language models (LLMs)) where providing contextual examples significantly improves the rel- evance and accuracy of generated code. Effective code retrieval facilitates bug identification, enables the reuse of existing solutions, and minimizes redundancy, accelerating the overall development pro- Code embedding models (Li et al.) 2022a] Wang et al. [023bb] Zhang et al.) 2024) have gained traction for their ability to encode the semantic and syntactic properties of code into dense vector representations and help retrieve relevant snippets with high precision. However, these approaches have not demonstrated substantial success in real-world applications, particularly in complex tasks like resolving GitHub issues as evaluated by benchmarks like SWE-Bench (Jimenez et al., 2024).
We hypothesize that code embedding models often suffer from suboptimal training procedures. Most fatte-of-the-as code embedding models rely on contrastive learning (Li al., (202a)) which is powerful technique for learning representations by reducing the distance between similar code snip- pets while maximizing the distance between dissimilar ones. Yet, existing methods predominantly
"Equal Contribution.
Work done during an internship at Nomic Al.
Dataset, code and models are available
swag/ www.gnggissaaccoottaaatta cornstack

---

Published as a conference paper at ICLR 2025
fine-tune these models on noisy bimodal (text, code) datasets (Husain et al. 2019 Kocetkov et al.) heuristically sourced from open platforms like GitHub. These datasets usually lack curation and con- sisteney filtering mechanisms, leading to significant noise, including irrelevant incorrectly labeled pairs, which impairs the model's ability to learn robust representations. Further, existing approaches often fail to incorporate such challenging negatives, resulting in embeddings that struggle cap- ture fine-gre fine-grained distinctions between similar code snippets. This limitation prevents current models from effectively handling subtle semantic differences, thus compromising their retrieval accuracy in real-world scenarios.
To address these issues, we curate CORN- STACK a large-scale dataset of high- quality (text, code) pairs based on The Stack V2 (oozkov et al., 2024), refined through consistency filtering. and supplemented with mined hard negatives for effective contrastive learning. To remove noisy (text-positive) pairs, approach uses dual consistency filtering process, which ensures the positives within top-k across the corpus, while having an embedding similarity score greater than threshold. Further, we incorporate a hard negative mining strategy with softmax-bast sampling over a larger collection of negatives to promote diversity, with a curriculum con- ensure
Consiatency Filtering
Hard Negative Mining
DO
OQ
Nigalies
Nogativo
Sampling for Hard Nug Megatives
Top-k
Q
Curriculum Learning (Daingg Meda 4 Trahing!
Embedding Similarity Filter
0.55
10 1
Diverse Megetivss
trolling the sampling probability we progressively increase the difficulty of the sampled negatives across the training process. We demonstrate that contrastive learning using our dataset Icads to state-of-the- embedding models, outperforming even larger models (Zhang et al.) 2024 Wang al. 2023b)
Figure 1: Figure demonstrating the curation methodology for CORNSTACK, with consistency filtering to remove noisy positives in addition to a curriculum-based hard negative mining strategy.
code
variety of code retrieval tasks (Husain et al. 2019 Huang et al. [021] Lu et al. 2021).
In addition to embedding quality, code retrieval quality can benefit from sophisticated reranking techniques. While reranking has been extensively applied in domains like text retrieval (Zhuang
al. 2023b Sun et al., 2023) and recommendation systems (Liu et al., 2022 Gao al., 2024), its application in code retrieval remains largely unexplored. A major challenge for building effective code reranking models is the lack of high-quality data for fine-tuning on contrastive bimodal (text, code) data. In this paper, we show that our dataset, with the curated set of (text, code) pairs that involve positives and negatives, can be leveraged to finetune code generation models (Wang et al. 2023b
Hui et al., 2024) to be state-of-the-an code rerankers.
Finally, using our improved code retriever and reranker together, we show significant improvements in function localization (Xia et al. 2024), key aspect in addressing real-world software engineering challenges (Jimenez et al. 2024). Function localization refers to the ability to accurately identify the specific functions or code segments that require modification in response to particular issue, such as a bug report or feature request, especially on platforms like GitHub. Our retriever first identifies pool of highly relevant code snippets, while the reranker further refines these results, prioritizing the most contextually appropriate functions based on their relevance to the given issue.
Our main contributions be summarized as follows:
We introduce CORNSTACK, a large-scale curated high-quality (text, code) pairs dataset, refined through consisteney filtering, and supplemented with mined hard negatives for ef- fective contrastive learning.
We show that code retrievers trained using CORNSTACK have considerably higher per- formance on variety of code retrieval benchmarks, with substantial gains over current state-of-the the code embedding models, while using considerably smaller encoder.
We are the first to finetune LLMs as code rerankers. Our 7B code reranker, trained lever- aging our contrastive data, considerably improves performance over the retriever.
"CoRNSTACK stands for Consistency filtering and Robust Negatives for enriching The Stack v2.
2

---

Published as a conference paper at ICLR 2025
We demonstrate the benefit of improved code retrieval and reranking function localiza- tion, while solving real-world software development problems such as addressing GitHub issues.
CORNSTACK
The performance of code emhedding models is highly contingent on the quality of the large-scale data used for contrastive training, in the form of <query, positive, negatives> triples. Effective con- trastive training hinges on satisfying two primary conditions: 1) The positives are highly relevant to the query and not noisy, 2) The neg- atives are semantically similar to the positives but do not directly address the query, a.k.a hard negatives. Heuristically sourcing con- trastive examples from large-scale open-source code data, such as the Stackv2 (Lozhkov et al., 2024), can include irrelevant or incor- rectly labeled <query, positive>> pairs, which impair the models' ability to learn robust and accurate representations. Hence, we in- troduce two-step consistency filtering method (in $2.2) that selects positives that are among the top-k in the corpus and have an embed- ding similarity above certain threshold. Additionally, (see (2.3) to augment the <query, positive>> pairs with negatives, sampled over a larger collection ensure diversity. We also implement curriculum that adjusts the sampling probabilities, pro- gressively increasing the difficulty of the negatives throughout the training process. We call this collection of contrastive training examples CORNSTACK, with details in on leveraging this data to finetune code retrievers and rerankers. Table shows the counts of examples from different languages our contrastive dataset after filtering from the original Stackv2.
Language Original Selected
Python
46.1M
10M
JavaScript
85,5M
2.3M
Java
113.3M
3M
Go
9.2M
2M
PHP
31.8M
3M
Ruby
10.8M
0.9M
Table 1: # instances in CoRNSTACK for various languages after filtering.
employ hard negative mining strategy
2.1 DATA SELECTION
We base our dataset on the de-duplicated version of The Stack v comprehensive collection of source code in 600+ programming and markup languages. We convert this into bimodal data, ie. (text, code) pairs, by extracting the docstring of function the text, and the corresponding function as the code. Following Zhang et al. (2024), we heuristically filter out pairs when the text is not in English, is too short which removes URLs, HTML tags, and other bad Unicode characters in the text. To ensure the syntactic correctness of the code data in CoRNStack, following Guo et al. (202T) we used the Tree-sitter parsing toolkit to filter out any codes that cannot be parsed into syntax tree. However, unlike previous works, we do not filter pairs with 2 256 tokens in the text. By not filtering out these long-text pairs, we aim to improve the ability of the model to generalize well to long sequences in the queries, which are common in repository-levé code retrieval tasks. GitHub issues, for instance, often contain long, detailed descriptions of problems or feature requests.
2.2 DUAL CONSISTENCY FILTERING
In analyzing the heuristically collected text-code pairs from the Stack-v2, we observe that many docstrings inadequately describe the corresponding code behavior. Additionally, there are cases where the code does not perform the functionality described in the docstring. Such discrepancies can be harmful during training, as they provide noisy signal for relevance between the natural language descriptions and corresponding code implementations.
To address the data quality limitations, we build upon recent work in consistency filtering from training text embedding models (Wang et al. 2022; Günther et al., 2023). Consistency filtering aims to curate a refined dataset by excluding training pairs with low semantic similarity. In this work, we incorporate dual consistency filtering with two criteria. First, for given query, we ensure that the positive code snippet is among the top-k most semantically similar snippets in the dataset.
left with 16M for Java and 7M for Go after filtering, subsampled for these languages to ensure Python covered 50% of
While
since most downstream benchmarks are based on python. These languages were picked to cover those evaluated in CodeSearchNet Husain et al. (2019). htttss://huge wnn.anaaaaaaooo www.eeco/aaattttttttttooo
the- stack stack-v2--00 dedup
3

---

Published as a conference paper at ICLR 2025
This top-k retrieval step filters out irrelevant or weakly related code snippets. Next, we apply secondary filtering stage where pairs with similarity scores below a predefined threshold, ó, are discarded. This guarantees that even highly ranked pairs must surpass minimal quality threshold, reducing the retention of pairs that are only marginally relevant. The relative and absolute thresholds determined to balance dataset size and quality, ensuring that only the most consistent pairs included.
Formally, let (4i, cs) denote the heuristically col- lected (text, code) pair, T [4,, 12,.,,,,... (n] repre- sent the list of texts, and C [c1, C2.... cnl rep- resent the corresponding code snippets in the corpus D. We use an existing embedding model N to en- code both texts and code snippets into vector repre- sentations: T, (T) and C. = NCC). A simi- larity score matrix S = T.. C.¹ is computed, where each entry S represents the similarity between text and code c:- For all (t) cluded to the curated dataset D' if Si is in the top-k ranked values of S[1] and Sa > 6, the similarity threshold k= and 8=0.7.
Dataset # Examples % Correct
CosQA
20k
63.9
CSN
2M
55.8
Stack v2
200M
52.9
Ours
21M
77.1
Table 2: Evaluation of <query, positive>>
D, (tz, qi) is in-
pair correctness for different code corpora.
To demonstrate the superior quality of pairs in CORNSTACK, we perform an automated evaluation, comparing our filtered dataset against other contrastive code datasets, such CosQA HHuan et al. (2021) and CodeSearchNet (CSN) Husain et al. (2019). Specifically, we prompt Qwen2.5-Coder- code generation model judge whether a code
7B-Instruct (Hui et al. 2024). an instruction-ture snippet fully answers the corresponding query for 10k randomly sampled pairs from each dataset for 3 seeds. Table 2 shows the mean correctness evaluated by the LLM, with our large-scale contrastive collection considerably improving in quality over The Stack v2, CodeSearchNet,, and CosQA. We show the language-wise % correctness Table A.2 in the Appendix.
2.3 HARD NEGATIVE MINING
Beyond improving the semantic relevance of positive pairs, incorporating challenging negatives is critical to improving the model's ability to distinguish semantically similar instances, seen in text embedding literature (Wang et al. 2024a; Moreira et al. 2024]. Several prior works in code embedding model training attempt to mine hard negatives but face key limitations. CodeSage (Zhang jet al., 2024) weights in-batch negatives based on relevance to the query. However, since the set of in-batch negatives is chosen random, this approach is still limited by the hardness of negatives available within a single batch. CodeTT+ (Wang et al. 2023b) uses a contrastive similarity score to sample negatives from queue maintained by momentum encoder (He et al.) 2020). However, this approach is restricted by the queue size and introduces high memory overhead. Moreover, both methods risk sampling false negatives, which degrades contrastive learning by introducing noise. In this work, we introduce a hard negative mining strategy that involves sampling negatives from a large pool to promote diversity. Our hard negative mining strategy divided into 2 stages: 1) an offline stage that leverages the corpus-level similarity score matrix S pre-computed during consistency filtering to filter false-negatives, and 2) an online stage that uses softmax- based sampling strategy with curriculum to progressively select diverse, challenging negatives during the contrastive fine- tuning.
Given positive (text, code) pair in the dataset, denoted as (t,, c ), and let B; {c, } represent the set of hard negatives for text f, with S being the similarity score matrix between all the text and code snippets in the corpus. To eliminate false negatives, we follow Moreira al. (024)) to remove any oj which is sufficiently close to t,. Specifically, we filter out any negative oj for which the relevance score S.y exceeds a threshold y Si, where Sa is the similarity score text 4 and positive code snippet e; We then cache S for use in the online stage of our negative mining.
'In our experiments, we use Jina-Code-v2 Günther et al. 2023 the proxy embedding model N
4

---

Published as a conference paper at ICLR 2025
3
CODE RETRIEVER AND RERANKER
Each instance in our curated high-quality CORNSTACK dataset is of the form of the form of <query, positive, negatives>> triples, which can be used for contrastive training. Here, we describe how CONTRASTACK is leveraged for finetuning both code retriever and reranker models.
3.1
CODERANKEMBED RETRIEVER
We use a bi-encoder architecture (Reimers & Gurevych 2019) for our retriever, with weights shared between the text and code encoder. Let (42, c.')') denote a positive (text, code) pair in the dataset and (h2, ( be the respective output representations from the last hidden layer of the encoder. For a batch of size N, let H = U/ IN denote the code representations from the positive code samples in the batch. Let Hg = U'_ (he text &; in the batch sampled via online mining strategy that describe next.
LM denote the set of hard negatives code representations of all ij 5=
We use the pre-computed similarity-sone score matrix S (from $23p and for each f,, sample M negatives with probability: P(cj [4) exp(S.//T"//(2)
exp(Sim/TT)) where r' is the temperature pa- ramete Different from recent work in negative mining for text-embedding models Moreira et al. (2024) that select from top-k negatives, our roftmax-based sampling introduces diversity into the selection of negatives, increasing the exploration of the negative sample space while maintaining high likelihood of selecting challenging negatives. This strategy avoids overfitting to specific hard negatives and ensures that different negatives explored across epochs, fostering better general- ization. A key advantage of this approach lies in the gradual annealing of the temperature parameter r' during fine-tuning. As training progresses, we decrease r', thereby sharpening the softmax distri- bution and progressively increasing the difficulty of the sampled negatives. This forms a curriculum learning strategy, where the model is initially exposed to easier negatives and progressively harder ones as it becomes more adept at distinguishing semantically similar pairs.
To fine-tune the retriever, we employ contrastive learning objective based on the InfoNCE loss (Oord et al. 2018). The objective seeks to maximize the similarity between the text h; and its positive counterpart h,t while minimizing the similarity between the text h; and both hard nega- tives Hy and other positives from other in-batch examples H. In our formulation, each positive has N * (M + 1) - negatives in the contrastive loss. Specifically, the loss is formalized as:
cxp(h; hz-/T) Chu hi4) = =-log EnLE(HBUHT exp(h, hx/T)
(1)
Here, r(= 0.07) represents the temperature parameter that controls the sharpness of the softmax distribution, and the dot product h; h2 represents the cosine similarity between the text query and the code snippet in the joint embedding space. During inference, the cosine similarity between the text query and the code snippet used as the relevance score to get a ranked ordering.
3.2 CODERANKLLM RERANKER
Recently, listwise reranking approaches (Pradeep et al. 2023 Reddy et al., 2024) have gained pop- ularity for their ability score multiple passages simultaneously, opposed to pointwise Zhuang et al. (2023a c) or pairwise Qin et al. (2023) reranking, where scoring is performed in isolation, Xian al. 2023) demonstrate that listwise reranking benefits from contextually comparing multiple pas- sages at once, which helps calibrate relevance scoring better. Furthermore, [Sun et al. (2023) show that instruction- 2020 Zhuang et al.) 2023b) in zero-shot reranking settings. Due to input size limits, listwise rerank- ing with LLMs usually adopts sliding window strategy (Sun et al. (023) with a window size of M candidates and a stride For each window, passages are denoted by unique identifiers yı the LLM reranker generates as output a sequence of identifiers in decreasing order of their relevance.
L.MMs can outperform traditional supervised cross-encoders (Nogueira al.,
However, CORNSTACK cannot be directly used for finetuning listwise rerankers, they require ranked ordering data as supervision for training. Following recent work by Pradeep et al. (2023).
We use 7 = 0.95 with r' linearly decayed from 0.05 0.001
5

---

Published as a conference paper at ICLR 2025
leverage larger LLMs as teacher models to train listwise code reranker. The relevance pervision is provided in the form of an ordered sequence y = 1/1 > 1/2 > > ym. where ys is the identifier of document that has been judged more relevant to the query q than yj. for every m 2 j > i. Here, {yi k are taken from the positive and hard negative code snippets from CORN- STACK for each text query. The reranker is trained using the top M negatives from the offline stage (in 22.3), since it is relies on the ranking ordering supervision for these negatives provided by the teacher model, which is not feasible obtain an online fashion. We then train the reranker with language modeling objective, minimizing the error in predicting the true next token in the generation sequence:
lvl Σ
LLM
log( Po (y: (z, v<s))
(B (r. Ves) is the conditional probability of predicting the target y: given the instruction prompt x and the preceding tokens V-i--
4 EXPERIMENTS
CORNSTACK is high-quality curated code dataset containing <query, positive, negatives> tuples across six programming languages: Python, Java, Javascript, Ruby, Go, and PHP. This work aims to investigate two research questions: RQ1: Can CORNSTACK be leveraged to train highly performant code retrievers and rerankers?; RQ2: Can such a code retriever + reranker framework be used to assist in real-worke software development? To address RQ/, first demonstrate in $4.1 the superior performance of our code retriever on a variety of code retrieval tasks. Subsequently, in $4.2] we show the improved ranking accuracy achieved by leveraging our listwise code reranker over the retrieved results. For RQ2, $4.3 shows better function localization based on GitHub issues from using our code retriever + reranker framework.
4.1 CODE RETRIEVAL.
4.1.1 SETUP
Training We finetune our code retriever using the 21 million contrastive examples in CORN- STACK. The encoder initialized with Aretic-Embed-M (Merrick et al., 2024), text encoder supporting an extended context length of 8,192 tokens and pretrained on large-scale web query- document pairs, along with public text retrieval datasets (Yang al., 2018 Kwiatkowski et al.] 2019 Thorne et al. 20118), We finetune for three epochs using four GH200 GPUs, with a batch size of 128 and hard negatives per example. Our data filtering, negative mining, and model finetuning implemented using the contrastors package (Nussbaum et al. 2024
Evaluation Datasets To demonstrate the effectiveness of CORNSTACK, we evaluate our fine- tuned retriever on a variety of code retrieval tasks under zero-shot settings. First, we consider Code- 202 T) as benchmarks for function- SearchNet (CSN) (Husain al. 2019) and AdvTest (Lu et al.
level text-to-code retrieval, a semantic search task where natural language queries are used to retrieve relevant code snippets. Additionally, evaluate performance across diverse code retrieval tasks, consider the CoIR benchmark (Li et al. 2024). which includes code-to-text, code-to-code, and hy- brid code retrieval tasks (retrieving hybrid of code and textual documents given a hybrid query), in addition to text-to-code retrieval.
Baselines We compare our finetuned code retriever against state-of-the-art open-source and pro- prietary text and code embedding models of various parameter sizes. For open-source text em- bedding models, we include E5-Base (Wang et al., 2022) and E5-Mistral (Wang et al. 2023a). the two most performant text embedding models from the CoIR benchmark, as well as Aretic-Embed- M (Merrick et al. 2024), the base text encoder that we finetune CORNSTACK, For open-source code embedding models, we consider the Small, Base, and Large variants of CodeSage (Zhang et al.) 2024], along with C5 (Wang et al. (223b) and Lina-Code-22 (Günther et al.) (023), which are the current state-of -fte-----thhrrrrrr the-art code embedding models on text-to-code retrieval benchmarks. CodeSage is trained on an older version of the Stack (Kocetkov et al.). while CodeT5+ and Jina-Code-ves use

---

Published as a conference paper at ICLR 2025
CodeSeachNet
AdvTest
COIR
Retriever
Param.
Python
Java
JS
PhP
Go
Ruby
Avg.
Python
Avg.
Aretic-Embed-M
137M
53.8
49.5
46.3
41.1
71.9
57.9
53.4
34.1
43.0
CodeSage-Smal
130M
64.4
63.2
60.0
54.7
77.7
63.2
64.9
41.3
54.4
CodeSage-Base Base
356M
68.0
68.0
67.0
58.2
83.2
68.0
68.7
49.1
57,5
CodeSage-Large
1.3B
70.8
70.2
69.5
61.3
83.7
71.9
71.2
52.7
59.4
Jina-Code-v2
161M
64.4
66.4
61.8
55.9
84.4
70.4
67.2
37.1
58.4
CodeT5+
110M
71.7
69.2
67.8
90.7
74.4
45.9
OpenAl-Ada-002
Unknown
68.0
71.5
67.5
60.6
85.6
74.2
71.3
38.1
45.6
Voyage-Code-002
Unknown
66.8
64.8
63.4
52.0
88.9
75.0
68.5
56.3
CODERANKEMBED
137M
78.4
76.9
71.4
68.8
92.7
79.3
77.9
59.5
60.1
Table 3: Ranking performance (%) for different retrievers on various code retrieval benchmarks. We report the official metrics for each dataset: MRR @ 1000 for CodeSearchNet and Advtest, and nDCG@ 10 for COIR. Detailed task-wise numbers for COIR are in Table in Appendix.
Approach
CSN
AdvTest
Consistency Filtering + Softmax Sampling of Hard Negatives Curriculum Learning
72.7
50.8
Consistency Filtering + Softmax Sampling of Hard Negatives
72.3
49.4
Consistency Filtering + Top-K Selection Hard Negatives
71.4
48.6
Consistency Filtering
39.2
Filtering
56.7
37.6
Table 4: Ablations showing benefit of each of our proposed techniques. None represents using unfiltered Stack v2 examples with only in-batch negatives and no additional hard negatives.
the GitHub Code dataset for pretraining. We also include proprietary embedding models OpenAI- Ada-002 and Voyage-Code-002 our evaluation.
4.1.2 RESULTS
Table presents the code retrieval performance results for CodeSearchNet, AdvTest, and CoIR. The task-specific CoIR results are presented in Table 8 in the Appendix. On CodeSearchNet. our approach significantly outperforms all open-source and proprietary text and code embedding mod- els, establishing a new state-of-the-art for text-to-code retrieval. Notably, despite being evaluated in a zero-shot setting, our code retriever achieves better performance than CodeT5+, which uses CodeSearchNet for contrastive finetuning. Further, our 137M parameter encoder outperforms 1.3B CodeSage-Largy
model, which is ten times larger. In addition, on ColR, our code retriever, despite being smaller than the majority of the baselines, consistently performs well across all the tasks, leading to the highest average performance. This demonstrates the robustness of our con- trastive training data, with the trained model exhibiting superior cross-task generalization despite being trained exclusively for only text-to-code retrieval.
4.1.3 ABLATION STUDIES
Here, we conduct ablation studies for the code retriever using 10% of the training data from CORN- STACK. Specifically, we measure the benefit of our proposed techniques, namely curriculum learn- ing during training, the use hard negatives with a softmax-based sampling strategy and consis- tency filtering aimed at eliminating noisy positives. Table 4 shows the results from the ablation experiments. We can see that removing consistency filtering of positives or the use of hard negatives separately leads to significant drop in performance both CodeSearchNet (CSN) and AdvTest. We also see the benefit of curriculum learning, along with using coftmax-based sampling of hard negatives instead of top-K selection.

---

Published as a conference paper at ICLR 2025
CodeSeachNet
AdvTest
Reranker
FT Data
Python
Java
Js
PhP
Go
Ruby
Avg.
Python
None (Retriever Only)
Code
78.1
76.6
68.9
69.9
91.6
80.3
77.7
56.9
Owen-2.5-Code (zero-shot) Qwen-2.5 2.5--225Txe (finetuned)
71.6
70.7
64.0
63.1
84.0
71.6
70.2
58.7
Text
80.0
78.1
73.2
69.8
92.0
79.9
78.8
66.4
CODERANKLLM
Code
81.7
80.5
76.2
72.4
92.3
81.8
80.5
69.1
Table 5: Ranking performance (MRR@ @ 100 in %) for different models from reranking top- 100 re- trieval results on function-level text-to-code retrieval datasets. Our code reranker is finetuned from Qwen-2.5-Code with code listwise data, while Qwen-2.5-Texx is finetuned using text listwise data.
4.2
CODE RERANKING
4.2.1 SETUP
Training To create the training data for listwise reranking, we pick 50k <query, positive, negatives>> tuples from CORNSTACK by filtering for a higher similarity score and a better rank for the positive. Following Pradeep et (2023). sampling strategy with varying window sizes (between 3 to 10) and random shuffling leads to 250k training instances (more details in $A.7 of Appendix). For ranking supervision, we use the 1ron-2.5-32B-Insttuu LLM (Yang al. 2024) to obtain the ranked ordering of each example. The Qwen-2.5-Cone 1.--2-5d-odr--7B Meder-BB-Instrue model (Hui et al., 2024), which specializes in instruction- based code generation, employed as the listwise reranker. We finetune this model for one epoch using four GH200 GPUs, with batch size of 64 and maxi- mum input sequence length of 16,800
Baselines and Evaluation We compare our reranking performance with the zero-shot Qwen-2.5- Coder-7B- Instrucct model, which was used for fine-tuning. Since most text-based LLMs trained on both text and code data, we include a listwise text reranker as a baseline. Specifically, we finetune the Bwven2255-BB-nntteen
LLM using 40k listwise reranking instances labeled by GPT- 4, as described in Pradeep et al. (003)), which were created using queries from the MS MARCO dataset (Nguyen 2016). For evaluation, employ the CodeSearchNet and AdvTest text-to- code retrieval benchmarks. However, we exclude the CoIR benchmark due to its significantly larger size (containing more than 100k queries). During inference, the top 100 results from our code re- triever are passed to the reranker, with evaluation conducted using MRR @ 100. We use window size of and a step size of for the listwise LLM rerankers.
4.2.2 RESULTS
Table presents the performance of different reranking models on text-to-code retrieval datasets. Interestingly, the text reranker (Qwen-2.5-Texx) demonstrates strong performance across multiple programming languages, despite being finetuned with listwise text reranking data. This performance likely due to the presence of code examples in the LLM pretraining data, which enhances the model's 's understanding of code. Although the code LLM (Qwen-2.5-Texi) performs worse in zero-shot setting for listwise reranking, its performance improves significantly after finetuning with code-specific listwise data derived from CORNSTACK. These results suggest that listwise code rerankers can further enhance ranking performance beyond the initial retrieval step.
4.3 CODE RRTRIEVAI-+RRRAKKINN FOR FUNCTION LOCALIZATION
Having previously evaluated our code retrieval and reranker models on academic benchmarks, we now demonstrate their utility in assisting software development in real-world settings. Specifically, we focus on the task of function localization, which involves accurately identifying the specific functions that need to be modified in response bug report GitHub issue.
8

---

Published as a conference paper at ICLR 2025
File-level
Function-level
Model
Param.
Top-1
Top-2
Top-3
Top-5
Top-10
E5-Base
110M
50.0
66.1
71.9
39.4
52.2
Aretie-Embed-N)
137M
45.7
63.2
68.2
40.5
49.6
CodeSage-Small
130M
47.8
65.0
71.9
40.9
51.1
CodeSage-Base
356M
48.5
66.4
74.1
39.8
48.9
CodeSage-Large
1.3B
49.6
67.2
71.2
40.1
48.2
Jina-Code-v2
161M
42.0
62.4
70.1
43.1
55.1
CodeTS+
110M
47.8
62.4
68.2
39.8
45.3
Agentless GPT-4o-mini
Unknown
54.0
63.5
68.2
35.0
36.9
Agentless GPT-4o
Unknown
65.7
74.8
77.4
44.5
45.6
CODERANKEMBED (Ours)
137M
51.8
68.6
76.6
50.0
59.1
CODERANKLLM (Ours)
7B
68.2
81.4
85.0
67.5
73.7
Table 6: File and function localization performance (%) on SWE-Bench-Lite.
4.3. SETUP
Datasets For evaluation, we utilize SWE-Bench-Lite (Jimenez et al. 2024), a widely used eval- uation suite for automated software engineering. SWE-Bench-Lite repository-level benchmark that focuses on resolving real-world issues sourced from GitHub, requiring code patch that passes the associated cases. Following Xia al. (2024), we reformulate SWE-Bench-Lit for function localization evaluation by considering the functions to which code patches have been applied as the localized functions. Specifically, we retained 274 300 examples where patches modify existing functions or classes, with the excluded examples introducing code corresponding to new functions or import statements. The GitHub issue serves as the text query, while all functions within the files in the repository are considered as candidates for retrieval.
Baselines and Metrics Our primary baseline is Agentless (Xia et al., 2024), an automated ap- proach to solving software development problems that ranks among the top-pep pop-effformin open-source source submissions on SWE-Bench-LLid by repair. In the localization phase, uses hierarchical approach to first localize the fault to spe- cific files, then to relevant classes or functions, and finally to fine-grained edit locations. Given the considerable size of the codebase, a tree-like structure of the repository. illustrating relative lo- cation of each file, along with the GitHub issue, is used to rank and identify the files that need edits. Subsequently, the content of these files is used to identify the functions within them that require modification. For detailed description of Agentless, evaluate function localization using the output logs of Agentless obtained from the released official run. Furthermore, given the functions identified for modification, belong to for file localization evaluation. Since Agentless selects up to three files that need edits and further localizes functions within them, evaluate file localization top 1-3 and function localization at top 5 and top 10. We also consider the retrieval baselines as in Section 4.1 except for the proprietary ones due API costs.
Agentless employs two-phase process of localization followed
refer the reader to Xia et al. (2024). We
map them to the files they
4.3.2 RESULTS
Table presents the function and file localization accuracy achieved on SWE- "WE-Bench-Lite Results indicate that our code retriever significantly outperforms Agentless and other retrieval baselines on function localization. Additionally, we observe consistent improvements in both file and function localization when leveraging our code reranker on top of the retriever results. We hypothesize that the superior performance of GPT-4o on file localization, compared to the code retriever, may be due to these models having been exposed to the codebases during training, SWE-Bench-Line constructed using popular open-source Python repositories. Therefore, GPT-4o can potentially identify the file be edited without even utilizing the corresponding file content. We hypothesize that our retrieval- based approach could achieve further improvements on private repositories, which typically not included in L.LM pretraining data. We leave this investigation for future work.

---

Published as a conference paper at ICLR 2025
GitHub issue
GitHub Issue
Titl: limil, cchooosohoss___ to on a ForeignKay can render duplicate options formfiild Description: you Q object as lmmit, chhlccee__s.. to ForeignKay field involving a join, you may end up with duplicate options in your form.
Tille: Fix handling giinig empty string for RMModfiec-Sincl header Description: Eippyy string used to be ignored for I-Modiifed- Sinoe header, but now relees exception since deel3369ed3
Agentless Localization Top?
Agentless Localization Top
File Path: derggsii/mmiiisssaaaieeeeee
File Path: djanga/rpphwquees..yy
forertetekentt, e, using lone, seleargs):
oof headdrs(ssttt) (ttplleadeerseettt..ee
LsiestanceCaellf.ee remore Fiets.notel, str):
maise ValusErnant* ... (arlf-nans, self-romate, note)) return super fororield(==
CodeetankEmbadd--cddeeenntLLLL Localization Top
"fereuclass" forms Fhdesl
"quervaat"; : self.remote sasll,nodel...tffuut mmnger--ost wos..osggggoinnn)) "te_ nane": self.romote, fiele.field nane, enkeargs, "blank": self.olane,
File Path: djangriiewssstati...y
wtt.:adddfdddsiiiee (header:tene, atieent, sizent):
Miea sumetháng aodified wince cour last domitoaded 112
CodefanntEmaadeccodeeennnLLLLL Localization Top
header
This i5 the vatue of the (r-Nostrted--ee Stnce header.
File Path: dynggtoommshoddessy V
If this in None, I'1l Just neture True.
sef apply.lindt..d sapyyttntt.chotees..o www.sooooostoo.... to. www.oee.t..t..ffffeeeeeeeeeeeeee www.forfftstsffoontttt..
try:
Apply Lint__ohoiiss_to to the forwrield's is queryset needed.
heaper ValueErege
#stches re. natchie* header, ns. IENGRECASE) hrader_attre porso_ntto.deteeeme
if (esuttrrforrrieedd, "querysett) and nesettrfoorrieedd ...):
cosiitshhhissss_o to foreristd get. cheutt_chioe__oo if Liait chclees, to is not Nones
except (Attribsttttrro: Valustrror, Dvantlovtrrer): Paturn True
forefield. queryset
forefield. wttpsst.comtoxrfitttrr..
return
Figure 2:
Examples from SWE-Bench-Lite where Agentless mislocalizes the file (and function), while our
framework does function localization correctly.
Figure illustrates examples from SWE-Bench-LLite where the Agentless system localizes incor- rectly, likely influenced by the file naming conventions. In the left example, the system points to the fields folder, possibly because the issue mentions formfield. Similarly, the right example, it selects the request file within the http folder, as the issue pertains to problem with the header. In contrast, our framework accurately performs function-level localization in both cases. This attributed to the retriever and reranker integrating information from both the doestring and the code snippet, allowing it to be correctly matched to the GitHub issue serving as the query.
5 CONCLUSION
This paper presents CORNSTACK, a large-scale, high-quality dataset of contrastive training in- stances for code retrieval and reranking. Fine-tuning embedding models on this contrastive data achieves state-of-the-art performance across various code retrieval tasks, outperforming code em- bedding models that are ten times larger. We also demonstrate that a listwise code reranker, fine- tuned using CORNSTACK, can further improve the code ranking accuracy. Moreover, using our code retriever and listwise code reranker together, we show significant improvements in function localization for GitHub issues, an important component of real-world software development.
LIMITATIONS
Heuristic filtering in CoRNStack removed exact matches with evaluation datasets, but additional semantic filtering needed to catch similar queries and code. To further validate the functional correctness of the code data, one could generate test cases using LLMs and/or human annotators. However, due to the dataset size, it would be computationally intensive to generate comprehensive test cases and set up the necessary environments for execution baasdd validation.
ACKNOWLEDGEMENT
We would like to thank the BlenderNLP group and Nomic AI team for valuable feedback. This re- search is based upon work supported by DARPA ITM Program No. FA8650-23-C-7316 and DARPA SemaFor Program No. HR001120C011 the authors and should not be interpreted as representing the official policies, either expressed or implied, of DARPA, or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright annotation therein.
The views and conclusions contained herein are those of
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Debangshu Banerjee, Tarun Suresh, Shubham Ugare, Sasa Misailovic, and Gagandeep Singh. Crane: Reasoning with constrained IIm generation, 2025. URL httpp:::/eew...rrg/////e//eo
2502. 09061
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03377 2021.
Jingtong Gao, Bo Chen, Xiangyu Zhao, Weiwen Liu, Xiangyang Li, Yichao Wang, Zijian Zhang, Wanyu Wang, Yuyang Ye, Shanru Lin, et al. Lim-enhanced reranking in recommender systems. arXiv preprint arXiv:2406. 12433, 2024.
Michael Günther, Georgios Mastrapas, Bo Wang, Han Xiao, and Jonathan Geuter. Jina embeddings: A novel set of high- biigh-perforaacce sentence embedding models. In Proceedings of the 3rd Workshop for Natural Language Processing Open Source Software (NLP-OSS 2023), pp. 8-18, 2023.
Daya Guo, Shuo Ren, Shuai Lu, Zhangyin Feng. Duyu Tang, Shujie Liu, Long Zhou, Nan Duan, Alexey Svyatkovskiy, Shengyu Fu, Michele Tufano, Shao Kun Deng, Colin Clement, Dawn Drain, Neel Sundaresan, Jian Yin, Daxin Jiang, and Ming Zhou Graphcodebert: Pre-training code representations with data flow, 2021.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9729-9738, 2020.
Junjie Huang, Duyu Tang, Linjun Shou, Ming Gong, Ke Xu, Daxin Jiang, Ming Zhou, and Nan Duan. Cosqa: 20,000+ web queries for code search and question answering. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the I 1ith International Joint Conference Natural Language Processing (Volume 1: Long Papers), pp. 5690-5700, 2021.
Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Kai Dang, et al. Qwen2. 5-coder technical report. arXiv preprint arXiv:2409. 12186, 2024.
Hamel Husain, Ho-Hsiang Wu, Tiferet Gazit, Miltiadis Allamanis. and Marc Brockschmidt. Codesearchnet challenge: Evaluating the state of semantic code search. arXiv preprint 1rXiv:1909.09446. 2019.
Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. Swe-bench: Can language models resolve real-world github issues? In The Twelfth International Conference Learning Representations, 2024.
Denis Kocetkov, Raymond Li, LI Jia, Chenghao Mou, Yacine Jernite, Margaret Mitchell, Car- los Muñoz Ferrandis, Sean Hughes, Thomas Wolf, Dzmitry Bahdanau, et al. The stack: 3 tb of permissively licensed source code. Transactions on Machine Learning Research.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453 466, 2019.
Xiangyang Li, Kuicai Dong, Yi Quan Lee, Wei Xia, Yichun Yin, Hao Zhang, Yong Liu, Yasheng Wang, and Ruiming Tang. Coir: A comprehensive benchmark for code information retrieval models. arXiv preprint arXhv:2407.02883, 2024.
Xiaonan Li, Yeyun Gong, Yelong Shen, Xipeng Qiu, Hang Zhang, Bolun Yao, Weizhen Qi, Daxin Jiang. Weizhu Chen, and Nan Duan. Coderetriever: A large scale contrastive pre- training method for code search. In Proceedings of the 2022 Conference on Empirical Methods in Natural Lan- guage Processing, pp.
2022a.
11

---

Published as a conference paper at ICLR 2025
Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al. Competition-level code generation with alphacode. Science, 378(6624)
2022b.
Weiwen Liu, Yunjia Xi, Jiarui Qin, Fei Sun, Bo Chen, Weinan Zhang, Rui Zhang, and Ruim- ing Tang. Neural re-ranking in multi-stage recommender systems: A review. arXiv preprint arXiv:2202.06602 2022.
Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy-Poirier, Nouamane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, et al. Starcoder and the stack v2: The next generation. arXiv preprint arXiv:2402. 19173, 2024.
Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin Clement, Dawn Drain, Daxin Jiang, Duyu Tang, et al. Codexglue: A machine learning benchmark dataset for code understanding and generation. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1), 2021.
Luke Merrick, Danmei Xu, Gaurav Nuti, and Daniel Campos. Aretic-embed: Scalable, efficient, and accurate text embedding models. arXiv preprint arXiv:2405.05374, 2024.
Gabriel de Souza P Moreira, Radek Osmulski, Mengyao Xu, Ronay Ak, Benedikt Schifferer, and Even Oldridge. Ny-retriever: Improving text embedding models with effective hard-negative mining. arXiv preprint arXiv:2407.15831. 2024.
Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. Ms marco: A human generated machine reading comprehension dataset. In CoCo@ NIPs, 2016.
Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. Codegen: An open large language model for code with multi-turn program synthesis. In The Eleventh International Conference on Learning Representations.
Rodrigo Nogueira, Zhiying Jiang, Ronak Pradcep, and Jimmy Lin. Document ranking with pre- trained sequence-to te--nneee------unceeeeee model. In Findings of the Association for Computational Linguis- fics: EMNLP 2020, pp. 708-718, 2020.
Zach Nussbaum, John X. Morris, Brandon Duderstadt, and Andriy Mulyar. Nomic embed: Training reproducible long context text embedder, 2024.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. In Advances in Neural Information Processing Systems, pp. 07203-1021., 2018.
Ronak Pradeep. Sahel Sharifymoghaddam, and Jimmy Lin. Rankzephyr: Effective and robust zero- shot listwise reranking is breeze! arXiv preprint arXiv:2312.022724 2023.
Zhen Qin, Rolf Jagerman, Kai Hui, Honglei Zhuang, Junru Wu, Jiaming Shen, Tianqi Liu, Jialu Liu, Donald Metzler, Xuanhui Wang, al. Large language models are effective text rankers with pairwise ranking prompting. arXiv preprint arXiv:2306. /7563, 2023.
Revanth Gangi Reddy, JaeHyeok Doo, Yifei Xu, Md Arafat Sultan, Deevya Swain, Avirup Sil, and Heng Ji. First: Faster improved listwise reranking with single token decoding. arXiv preprint a4Xiv:2406. 15657, 2024.
Nils Reimers and Iryna Gurevych. Sentence-BERT Sentence embeddings using Siamese BERT- networks. In Kentaro Inui, Jing Jiang. Vincent Ng, and Xiaojun Wan (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th In- ternational Joint Conference on Natural Language Processing DOMNLP-IJCNLL pp. 3982- 3992, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/v 10118/55//1/DDI--LLL
Weiwei Sun, Lingyong Yan, Xinyu Ma, Shuaiqiang Wang, Pengjie Ren, Zhumin Chen, Dawei Yin, and Zhaochun Ren. Is chatgpt good at search? investigating large language models re-ranking agents. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Pro- cessing, pp. 14918 14937, 2023.
12

---

Published as a conference paper at ICLR 2025
Tarun Suresh, Shubham Ugare, Gagandeep Singh, and Sasa Misailovic. Is the watermarking of Ilm-generated code robust?, 2025. URL paattps://arxv..rre http///b///42// httppp///aarai...c////////...... 179883
James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal, FEVER: a large- scale dataset for fact extraction and VERification. In NAACL-IILT, 2018.
Shubham Ugare, Rohan Gumaste, Tarun Suresh, Gagandeep Singh, and Sasa Misailovic. Itergen: Iterative structured IIm generation, 2024a. URI. http:pss//rrrs.ooo///aaa//////// 072995
Shubham Ugare, Tarun Suresh, Hangoo Kang, Sasa Misailovic, and Gagandeep Singh. Syn- code: Llm generation with grammar augmentation, 2024b. URthttps:: http:::///.xvxv..././.// abs/
0403301632
Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Ma- jumder, and Furu Wei. Text embeddings by weakly-supervised contrastive pre-training, arXiv preprint arXi::2112.03533, 2022.
Liang Wang, Nan Yang. Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. Improv- ing text embeddings with large language models. arXiv preprint arXiv:2401.0035 2023a.
Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. Im- proving text embeddings with large language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Com- putational Linguistics (Volume 1: Long Papers), pp. 11897-11916 Bangkok, Thailand, August 2024a. Association for Computational Linguistics. doi: 10. 1/2.:/3///222442allologg haaps::/aacaatthol..
URL
org/ 2024. aaa-acloonnn 642
Yue Wang, Hung Le, Akhilesh Gotmare, Nghi Bui, Junnan Li, and Steven Hoi. Codet5+: Open code large language models for code understanding and generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 1069-1088, 2023b.
Zora Zhiruo Wang, Akari Asai, Xinyan Velocity Yu, Frank F Xu, Yiqing Xie, Graham Neubig, and Daniel Fried. Coderag-bench: Can retrieval augment code generation? arXiv preprint arXiv:2406. 14497, 2024b.
Chunqiu Steven Xia, Yinlin Deng, Soren Dunn, and Lingming Zhang. Agentless: Demystifying llm-based software engineering agents. arXiv preprint MxXi:24077.01480 2024.
Ruicheng Xian, Honglei Zhuang, Zhen Qin, Hamed Zamani, Jing Lu, Ji Ma, Kai Hui, Han Zhao, Xuanhui Wang, and Michael Bendersky. Learning list-level conaaininvariiat representations for ranking. Advances in Neural Information Processing Systems, 36, 2023.
An Yang. Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang. Jian Yang, Jianhong Tu, Jianwei Zhang. Jianxin Ma, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang. Mei Li, Mingfeng Xue, Na Ni, Pei Zhang, Peng Wang, Ru Peng, Rui Men, Ruize Gao, Runji Lin, Shijie Wang. Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wenbin Ge, Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Yang Fan, Yang Yao, Yichang Zhang, Yu Wan, Yunfei Chu, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zhihao Fan. Qwen2 technical report. arXiv preprint arXiv:2407. 10671, 2024.
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 2018
Dejiao Zhang, Wasi Uddin Ahmad, Ming Tan, Hantian Ding, Ramesh Nallapati, Dan Roth, Xi- aofei Ma, and Bing Xiang. Code representation learning at scale. In The Twelfth International Conference on Learning Representations, 2024.
Honglei Zhuang, Zhen Qin, Kai Hui, Junru Wu, Le Yan, Xuanhui Wang, and Michael Berdersky. Beyond yes and no: Improving sero Ilm rankers via scoring fine- ane-ggrined relevance labels. arXiv preprint 20Xiv:2310.144 /4/22, 2023a.
13

---

Published as a conference paper at ICLR 2025
Honglei Zhuang, Zhen Qin, Rolf Jagerman, Kai Hui, Ji Ma, Jing Lu, Jianmo Ni, Xuanhui Wang. and Michael Bendersky. Rankt5: Fine-tuning t5 for text ranking with ranking losses. In Proceedings of the 46th International ACM SIGIR Conference Research and Development in Information Retrieval, pp. 2308-2315 2023b.
Shengyao Zhuang, Bing Liu, Bevan Koopman, and Guido Zuccon. Open-source large language models are strong zero-shot query likelihood models for document ranking. In Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 8807-8817, 2023c
14

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 DIVERSITY OF CODE TOPICS IN CORNSTACK
To provide more insight into the variety and complexity of code tasks in CoRNStack, we analyzed the distribution of code topics for 100k randomly sampled instances using Nomic Atla a popular unstructured text visualization tool. Nomic Atlas employs cluster-based keyword identificatinn algorithm and leverages language model to generate topics. We find that the majority of examples fall into eight broad categories: object creation, data sorting, data management, API management, configuration, data validation, graphics, and math operations. The wordcloud in Figure 3 illustrates the diverse fine- seee-graied topics within these categories.
Sensor Integration Data Loading
Ne Networken king Math Problems Structured
Authenticatioon
Data Machine
Data Fornat
Learning
Up
HTTP
client
ade
Status
Management
g
r
Utility
Functions
Usuario
Data
deometry y
Retrieval
Web
Developmen
Search
Algorida
object
Context
Documentation
object
Creation
Androi
Documentation
d
Statistics
Database
Management
File
emeeettt
Android
GUI
Object Model
User Progr amning
Security
Usel
ty
Man Management
Types
Physics
Inter
Database
IDs
Boolean
Logic
f
face
Linear Algebra
Job
Methods
Financial Data
Finance
Progr amming
Methods
Time
Mana
Kuberneted tes
g
ement
Tools
Initialization Java Utilities
Sporss
Image
Management
Validation
Software Testing
re
File
Sy
S stem
Data Validation
Random Numbe
Parser Project
Management Database Deletion
Testing Failures
Figure 3: A wordcloud of popular code topics collected from 100k instances in CoRNStack
A.2 ACCURACY OF (TEXT, CODE) PAIRINGS BY LANGUAGE
In Table 7 we provide the language-wise correctness numbers for the mean results from Table in the main paper. We can see that CoRNStack has significantly higher correctness of (text, code) pairings across all languages.
Dataset
Python
Java
JavaScript
PHP
Go
Ruby
Avg
Stack v2
54.8
50.3
53.6
56.4
63.2
39.2
52.9
CSN
53.9
56.3
50.5
60.7
65.9
47.6
55.8
CosQA
63.9
63.9
CoRNStack (Ours)
76.2
80.6
74.4
77.3
82.8
71.4
77.1
Table 7: Evaluation of <query, positive> pair correctness by language for different code corpora.
A.3 FINE-TUNING TUNING VARIOUS ENCODERS ON CORNSTACK
We hypothesize that code retrievers can benefit from pretraining on supervised text ranking data, which typically abundant. validate this hypothesis, we performed the following experiment: We selected three 130M parameter text and code encoders, specifically Arctic-Embed-M Merrick ct al. (2024) and Nomic-embed Nussbaum et al. (2024) as the text encoders due to their strong performance text retrieval benchmarks like MTEB, and CodeSage-Small Zhang al. (2044) as
httpss:/at nomi cammi..i//cae httpm//m...////ttttccaaaa///////
100k
15

---

Published as a conference paper at ICLR 2025
Task (-+)
Text-to-Code
Code-to-Tent
Code-to-Code
Hybrid Code
Avg
CSN
CodeTrrans
StackOver
CodeFeedBack
Retriever
Apps CosQA SQL,
CSN
CCCR -Contest -DL
Flow
-T
-MT
E5-base
11.5
32.6
52.3
68.0
56.9
62.5
21.9
86.9
74.5
42.0
50.9
Aretie-Embed-M
27.6
18.9
37.4
62.2
28
86.9
67
28.0
43.0
E5-Mistral
21.3
51.3
66.0
54.3
65.3
82.6
33.2
91.5
72.7
33.7
55.2
CodeSage-Small
17.3
30.5
51.9
74.1
84.2
76.2
31.0
73.9
62.3
42.6
54.4
CodeSage-Base
27.6
29.4
59.4
76.9
86.9
78.9
31.9
76.2
63.0
44.6
57.5
CodeSage-Large
32.7
28.9
59.5
78.1
89.0
82.6
32.7
78.7
65.4
46.3
59.4
Jina-Code-v2
16.4
42.2
46.4
84.0
82.7
83.6
26.8
89.3
68.6
44.4
58.4
CodeT5+
3.3
23.1
78.0
83.6
52.3
31.6
59.9
53.2
32.8
45.9
OpenAl-Ada-002
8.7
29.8
58.3
74.2
69.1
53.3
26.0
72.4
47.1
17.7
45.6
Voyage-Codo-002
26.5
29.8
69.3
81.8
73.5
72.8
27.3
77.7
65.4
28.7
56.3
CODERANKEMBED
21.1
36.3
58.8
83.7
86.9
78.8
32.8
82.3
75.7
45.2
60.1
Table 8: NDCG@ 10 for different retrievers on the Code Information Retrieval Benchmark (CoIR).
the code encoder due to its overall performance on code retrieval benchmarks. We evaluated these models on code retrieval tasks before and after finetuning on CoRNStack for one epoch. shown in Table we see that finetuning CoRNStack significantly improves code retrieval performance for all models. Moreover, we observe that stronger text embedding model Arrtic-Emmedd-M M here) leads to better code retrieval performance after finetuning CoRNStack, even outperforming the pode-pretrained CodeSage- -mall. These results highlight that CoRNStack, being large-scale and high-quality, can be leveraged finetune text encoders into performant code retrievers. Therefore, we chose Pretic-Embed-M- as it provides a strong foundation from supervised text ranking pretrain- ing, which, when combined with fine-tuning on CoRNStack, leads to superior code retrieval.
Base Model
Pretrain
Params
CSN Avg.
AdyTest
CoIR Avg.
CodeSage-So -mall
Code
130M
64.9 73.9 (+9.0)
41.3 + 54.2 (+12.9)
54.4 - 60.0 (+5.6)
Nomic-Embed
Text
137M
47.2 + 76.7 (429.5)
28.6 + 54.6 (+26.0)
47.7 + 58.5 (+10.8)
Aretic-Embed-M
Text
137M
53.4 - 77.7 (+24.3) 34.1 - 57.8 (+23.7) 43.0 - 59.7 (+16.7)
Table 9: Results (Before > After) from finetuning different encoders for epoch on CoRNStack.
A.4 EFFICACY OF FINE-TUNING ON CORNSTACK vs CODESEARCHNET
CoRNStack has a significantly higher query-positive correctness, while also being upto 10x larger than existing contrastive code datasets like CodeSearchNet (CSN) (see Table in the main paper). To further highlight the impact of CoRNStack's 's scale and quality, we finetuned Arctic oeticcEmbed Merrick] et al. (2024), text embedding model, for epoch separately CoRNStack and NodeSearch To specifically show the benefit CoRNStack's 's quality, we also report results for fine-tuning on million randomly sampled datapoints from CoRNStack, the same amount of data as CodeSearchNet.
The results, shown in Table I0 clearly illustrate the improvement in code retrieval performance from finetuning on CoRNStack with both 2M subset and the full 21M examples.
Training Dataset
Examples
CSN Avg
AdyTest
CodeSearch
2M
65,8
37.5
CoRNStack Subset (Ours)
2M
71.4
48.6
CoRNStack (Ours)
21M
77.7
57.8
Table 10: Comparison of fine-tuning Arctic-Embel on CoRNStack vs CodeSearchNet.
16

---

Published as a conference paper at ICLR 2025
A.5 DETAILED EXPLANATION COMPARISON WITH CODET5+
In our paper, we compared our code retriever-an an encoder-only model-to the publicly available 1 110M parameter CodeTT+ Embedding mode (denoted as CodeTS+ in our paper). This model is also encoder-only and is listed in the official Code repository 9 but not discussed in the CodeTTS+ paper Wang al. (2023b). The repository also includes the 220M parameter CodeT5+ bimodal mode an encoder-decoder trained with text-code matching. CodeT5+ bimodal uses the 110M parameter CodeTT++ embedding model as its encoder and incorporates an additional decoder for reranking the top 32 candidates retrieved by the embedding model. Since CodeTT5+ bimodal pri- marily serves as reranker over the embedding model's 's results, we did not include this model the comparison in our paper. Our results (in Table 3 of the main paper) closely align with the perfor- mance metrics of the CodeT5+ Embedding model reported in the CodeTT5+ README. Additionally, the CodeT5+ authors note that the released CodeT5+ models are trained with multi-task data, and results differ from those their paper, which are fine-tuned for each retrieval benchmark. Table [T] shows a detailed comparison of our code retriever with both the released multi-task Code'I'++ mod- els and the single-task Code models. We observe that our zero-shot code retriever outperforms
both variants.
Model
# Params
Model
Type
CSN
AdvTTest
CodeT5+ Embedding
110M
Retriever
Multi-Tass Finetune
74.2
40.8
CodeT55 Bimodal
220M
Reranker
Multi-Task Finetune
75.9
42.9
CodeT5++ Bimodal
220M
Reranker
-See-pcciiicc Finetune
77.1
43.3
CodeT5+ Bimodal
770M
Reranker
Task- Seec-Specifiic Finetune
77.4
44.7
CODERANKEMBED
137M
Retriever
Zero-shot
77.9
59.5
Table 11: Code Retriever Fine-Tuned on CoRNStack vs. Different CodeT5+ Models
A.6 STUDENT VS TEACHER PERFORMANCE FOR LISTWISE CODE RERANKING
Here, we evaluate the teacher model for listwise code reranking, to see whether the student model reaches the teacher's performance after finetuning. Due computational limitations in running the large teacher model on the entire CodeSearchNet (CSN) and AdvTest benchmarks, we evaluated on 1,000 random sampled queries for AdvTest and each language in CSN. Table 12 compares the performance of our finetuned Qwen 2.5 7B student model with the Qwen 2.5 32B teacher model. We observe that while the finetuned student slightly outperforms the teacher on CSN, it still shows slight performance gap on the more challenging AdvTest benchmark.
CodeSeach PhP
AdvTest
Reranker
Python
Java
IS
Go
Ruby
Avg.
Python
None (Retriever Only)
76.3
77.4
71.1
70.9
91.4
80.2
77.9
58.0
Qwen-2.5-77 (Student Model)
70.9
71.8
66.0
64.7
83.6
72.4
71.5
60.6
Qwen-2.5-32B (Teacher Model)
79.8
80.4
76.8
73.3
92.0
82.6
80.8
73.6
CODERANKLLM
79.9
81.9
77.9
73.2
92.5
81.8
81.2
70.4
Table 12: Ranking performance (MRR @ 100 in %) Ik randomly sampled queries (per language) from CodeSearchNet and AdvTest for teacher model vs student model before and after finetuning.
A.7 DETAILS ON DATA AUGMENTATION FOR LISTWISE RERANKING
We follow the data augmentation methodology from Pradeep al. (202)), incorporating techniques to create more diverse and challenging training setup order to obtain more robust trained reranker. Specifically, the augmentation process consists of two key components:
55
17

---

Published as a conference paper at ICLR 2025
Variable Window Sizes: For each training instance, random subset of candidate code snippets (ranging from 3 to 10 candidates) is sampled from the original ranked list to pass input the teacher model. This introduces variability in the input size and diversifies complexity of the reranking task, ensuring that the reranker model encounters a broader range of scenarios during training.
Random Shuflling: To enhance the model's generalization ability across different doc- ument orders beyood-beyood the default order provided by the retriever-random permutations are applied to the sampled windows. This technique has demonstrated effectiveness in traditional text reranking tasks (Pradeep et al. 2023), and we extend to code reranking.
18


Reviews:
[{"review_id": "euXoxOtPVq", "reviewer": "Reviewer_g16n", "summary": "This paper proposes a new dataset CoRNSTACK for code retrieval tasks. Different from previous datasets, this dataset is a large-scale and high-quality dataset that contains hard negative samples. Trained with CoRNSTACK, the performance of models on code retrieval tasks and code reranking tasks has significantly increased. Besides, the authors also demonstrate the effectiveness of training with the proposed dataset in a real-world application (addressing GitHub issues).", "questions": "1. Why do you use pre-trained text embedding models for dual consistency filtering? What about using pre-trained code retrieval models pre-trained on previous code retrieval datasets?\n2. Fine-tuning models with contrastive learning is a common and traditional method for code retrieval tasks. Recently, there has been a new paradigm named Generation-Augmented Retrieval (GAR) framework proposed for code retrieval tasks as well [1, 2]. I think this framework makes the fine-tuning of code retrieval models somewhat unnecessary. What's your opinion on the necessity of fine-tuning code retrieval models? (Also, I think discuss this point in the paper will be better)\n\n\n[1] Haochen Li, Xin Zhou, and Zhiqi Shen. 2024. Rewriting the Code: A Simple Method for Large Language Model Augmented Code Search. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1371–1389, Bangkok, Thailand. Association for Computational Linguistics.\n\n[2] \"Generation-Augmented Query Expansion For Code Retrieval.\" arXiv preprint arXiv:2212.10692 (2022).", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. According to my knowledge, the problems in previous code retrieval datasets discussed by the authors truly exist and the proposed dataset addresses these problems well. This proposed high-quality dataset will benefit research in code retrieval tasks and real-world retrieval applications.\n2. The idea of fine-tuning LLMs as code rerankers is interesting and effective.\n3. The authors not only evaluate code retrieval tasks, but also demonstrate the effectiveness of training with CoRNSTACK in addressing real-world GitHub issues.", "weaknesses": "Overall, I think this paper is good and should be accepted. All of the following weaknesses are minor problems.\n1. There is no analysis of the reason for the improved performance of code rerankers. I guess this is also due to the hard negative samples?\n2. As stated by the authors, one of the motivations for constructing this dataset is that existing datasets contain many false negative samples. In [1], the authors also discussed this problem and proposed methods to address the problem during model fine-tuning, which I think should also be discussed in the paper.\n3. Although the authors discussed the problems of existing datasets in the introduction and some preliminaries of code retrieval in the paper, I still think it is better to write a related work section to make them clearer. Since code retrieval is not an extremely hot topic, many readers may not know many details of previous works in this area.\n4. Typo in Line 272, 'high-qualit,y'\n\n\n[1] \"Rethinking Negative Pairs in Code Search.\" Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "C7jgnIU1c0", "reviewer": "Reviewer_MeoD", "summary": "This paper presents a dataset built upon The Stack V2 dataset, proposing a high-quality contrastive training dataset called CoRNStack, which consists of (text, code) pairs. This dataset uses dual consistency filtering to eliminate noisy positives and is further enriched with mined hard negatives. The effectiveness of the proposed dataset is demonstrated through downstream code tasks, including code retrieval and re-ranking tasks.", "questions": "1. The quality of the new dataset needs to be explained, especially how accurate the pairings are between text queries and code.  \n2. It should also be clarified whether the code data in the dataset has been checked manually or tested with examples, as this is important for the quality of the code retrieval tasks.  \n3. Additionally, it would be helpful to include information about the variety of code tasks in the new dataset, particularly how simple and complex scenarios are distributed.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "A new dataset has been established on existing code datasets by filtering and selecting positive and negative samples.", "weaknesses": "The process of handling negative samples in Figure 1 needs to be explained in more detail and more directly.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "x6Ev8bURUd", "reviewer": "Reviewer_bBWX", "summary": "The paper constructs a dataset containing high-quality positive and negative samples for contrastive learning, thereby achieving better retrieval and reranking performance. They ensure a strong semantic correlation between the positive samples and queries in the dataset through filtering, and they obtain high-quality and diverse negative samples through hard negative mining. The model trained on this dataset achieves generally excellent retrieval performance across different benchmarks and retrieval tasks. Additionally, they further extend this dataset to build an ordered dataset for training the reranker, which also yields good results. Finally, they combine the retriever and reranker to achieve effective function localization for GitHub issues.", "questions": "Refer to weakness 2. What's your motivation to select Arctic-Embed-M as your initial encoder rather than other models?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is easy to follow\n\n2. Motivation is clear\n\n3. The model achieves effective results, showing significant improvements in various scenarios, including retrieval, reranking, and function localization.", "weaknesses": "There are some minor flaws in the experimental section that could be further improved:\n\n1.The motivation for selecting Arctic-Embed-M as the initial encoder for training, instead of other models, is not clearly explained.\n\n2.The setup in section 4.2.1 lacks clarity. For example, the statement “a sampling strategy with varying window sizes (between 3 to 10) and random shuffling leads to 250k training instances” could be elaborated more for better clarity.\n\n3.In training the reranker, the authors used the QWen 32B model as a teacher to annotate data. It might be beneficial to include the teacher model’s performance in Table 6, allowing us to see if the model approaches the teacher’s performance on this task.\n\n4.In section 2.3, the authors mentioned using a softmax-based sampling strategy instead of directly adopting top-k sampling for better diversity of negative samples. Including an ablation study for this strategy in the experimental section could more clearly validate the effectiveness of your approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Vg5XQ0cJ3d", "reviewer": "Reviewer_ZoFm", "summary": "The article introduces CORNSTACK, a novel, high-quality dataset for contrastive code retrieval and reranking training. The primary motivation is to address the limitations in existing code embedding models, which struggle with complex, real-world software development tasks due to the noisy and inconsistent nature of current datasets. These models, crucial for bug localization and retrieval-augmented code generation, often fail to generalize beyond small, controlled tasks. The effectiveness of the dataset was demonstrated through the training of the retriever and reranker models.", "questions": "Please refer to Weaknesses section.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The authors try to filter the high-quality code retrieval dataset from the stack dataset to solve the noise problem existing in the current code retrieval dataset.\n2. The authors train both retriever and reranker models, and the final experimental results prove the effectiveness of the authors' method and achieve state-of-the-art results. \n3. The authors evaluate the code retrieval dataset and also try to evaluate the code retrieval model for the error localization dataset.", "weaknesses": "1. This paper's contribution is mainly incremental. It improves data quality through filtering and negative sample mining methods on the Stack V2 dataset, which has been extensively explored in previous research work [1,2]. This paper only makes minor improvements based on these papers. \n2. The paper lacks a more detailed comparison and analysis between the final obtained and original datasets beyond just quantity (such as case studies and other statistical conclusions). \n3. In the writing of the paper, the construction of the dataset and the harmful sample mining methods (especially the curriculum learning technique, which is not model-independent) are coupled. In my opinion, these are two parts: one part is the construction of the dataset, and the other part is the harmful sample mining method designed by the author. I believe these two parts should be more clearly distinguished in the writing. \n4. For the codet5+ model, the results reported in the paper seem to differ from those presented in the original text. Could the author add more detailed experimental settings?\n[1]Zhang, Dejiao, et al. \"Code representation learning at scale.\" arXiv preprint arXiv:2402.01935 (2024).\n[2] Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. Text embeddings by weakly-supervised contrastive pre-training.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)