ICLR 2025

Published as a conference paper at ICLR 2025
CORNSTACK: HIGH-QUALITY CONTRASTIVE DATA
FOR BETTER CODE RETRIEVAL AND RERANKING
Tarun Suresh1,3∗†
Revanth Gangi Reddy1∗
Yifei Xu1,2
Zach Nussbaum3
Andriy Mulyar3
Brandon Duderstadt3
Heng Ji1
1University of Illinois Urbana-Champaign
2Lapis Labs
3Nomic AI
{tsuresh3,revanth3,yifeix5,hengji}@illinois.edu
ABSTRACT
Effective code retrieval plays a crucial role in advancing code generation, bug fix-
ing, and software maintenance, particularly as software systems increase in com-
plexity. While current code embedding models have demonstrated promise in
retrieving code snippets for small-scale, well-defined tasks, they often underper-
form in more demanding real-world applications such as bug localization within
GitHub repositories. We hypothesize that a key issue is their reliance on noisy
and inconsistent datasets for training, which impedes their ability to generalize
to more complex retrieval scenarios. To address these limitations, we introduce
CORNSTACK1, a large-scale, high-quality contrastive training dataset for code
that spans multiple programming languages. This dataset is curated using consis-
tency filtering to eliminate noisy positives and is further enriched with mined hard
negatives, thereby facilitating more effective learning. We demonstrate that con-
trastive training of embedding models using CORNSTACK leads to state-of-the-
art performance across a variety of code retrieval tasks. Furthermore, the dataset
can be leveraged for training code reranking models, a largely underexplored area
compared to text reranking. Our finetuned code reranking model significantly im-
proves the ranking quality over the retrieved results. Finally, by employing our
code retriever and reranker together, we demonstrate significant improvements in
function localization for GitHub issues, an important component of real-world
software development.
1
INTRODUCTION
The rapid advancement of software development has led to an increased rel...