Code snippets insertion approach for code completion task

Authors: Nina Kamkia, Dmitry Stanishevskiy, Alena Fenogenova, Vitaly Kadulin, Denis Kokosinskii, Dmitry Zmitrovich
Primary area: generative models
Keywords: Code completion, LLMs, Benchmark, RAG, Code snippets, GitHub files, Exact match metric

Abstract:
The purpose of the code completion task is to continue the following line of the source code. This feature helps developers to write code faster and with fewer errors by providing suggestions for completing the current line of code based on the context and the available libraries and functions.  One common but resource-expensive approach to improve model performance on the code completion task is to fine-tune Large Language Models (LLMs). This paper introduces a technique that involves feeding the additional input with ranked portions of code to the LLMs without additional fine-tuning. The proposed approach aims to replicate the development process of programmers by scanning project files for the required code snippets, making the process intuitive and efficient. The paper also discusses the lack of metrics for the task and puts forward a novel metric ClickScore, as well as a new code benchmark RealCode for the code completion task. The paper compares the insertion technique of code snippets with the existing state-of-the-art methods using standard and proposed metrics and demonstrates the approach's effectiveness.

Decision: desk reject

Reviews and discussion:

Metareview: