LoRACode: LoRA Adapters for Code Embeddings
Authors: name: Saumya Chaturvedi, name: Aman Chadha, name: Laurent Bindschaedler

Code embeddings are essential for semantic code search; however, current
approaches often struggle to capture the precise syntactic and contextual
nuances inherent in code. Open-source models such as CodeBERT and UniXcoder
exhibit limitations in scalability and efficiency, while high-performing
proprietary systems impose substantial computational costs. We introduce a
parameter-efficient fine-tuning method based on Low-Rank Adaptation (LoRA) to
construct task-specific adapters for code retrieval. Our approach reduces the
number of trainable parameters to less than two percent of the base model,
enabling rapid fine-tuning on extensive code corpora (2 million samples in 25
minutes on two H100 GPUs). Experiments demonstrate an increase of up to 9.1% in
Mean Reciprocal Rank (MRR) for Code2Code search, and up to 86.69% for Text2Code
search tasks across multiple programming languages. Distinction in task-wise
and language-wise adaptation helps explore the sensitivity of code retrieval
for syntactical and linguistic variations.