On Pretraining for Project-Level Code Completion
Authors: name: Maksim Sapronov, name: Evgeniy Glukhov
Organization: JetBrains Research

Repository-level pretraining is commonly used to enable large language models
for code to leverage codebase-wide context. This enhances their ability to
generate accurate and context-aware code completions. In this work, we
investigate how different repository-processing strategies affect in-context
learning in OpenCoder, a 1.5B-parameter model. We extend its context window
from 4,096 to 16,384 tokens by training on additional 1B tokens of curated
repository-level data. Despite relying on a smaller dataset than competing
models (which often use hundreds of billions of tokens), our model achieves
comparable performance on the Long Code Arena benchmark. We find that various
repository-processing techniques yield similarly strong results, with the
primary gain coming from adapting to a new rotary positional embedding (RoPE)
scaling parameter. Finally, we show that a simpler file-level training approach
at the original sequence length remains highly effective, opening up
repository-level code completion research to settings with more constrained
data and compute resources.