Scaling Granite Code Models to 128K Context
Authors: name: Matt Stallone; name: Vaibhav Saxena; name: Leonid Karlinsky; name: Bridget McGinn; name: Tim Bula; name: Mayank Mishra; name: Adriana Meza Soria; name: Gaoyuan Zhang; name: Aditya Prasad; name: Yikang Shen; name: Saptha Surendran; name: Shanmukha Guttula; name: Hima Patel; name: Parameswaran Selvam; name: Xuan-Hong Dang; name: Yan Koyfman; name: Atin Sood; name: Rogerio Feris; name: Nirmit Desai; name: David D. Cox; name: Ruchir Puri; name: Rameswar Panda
Published: 2024-07-18 17:46:02

Abstract: This paper introduces long-context Granite code models that support effective
context windows of up to 128K tokens. Our solution for scaling context length
of Granite 3B/8B code models from 2K/4K to 128K consists of a light-weight
continual pretraining by gradually increasing its RoPE base frequency with
repository-level file packing and length-upsampled long-context data.
Additionally, we also release instruction-tuned models with long-context
support which are derived by further finetuning the long context base models on
a mix of permissively licensed short and long-context instruction-response
pairs. While comparing to the original short-context Granite code models, our
long-context models achieve significant improvements on long-context tasks
without any noticeable performance degradation on regular code completion
benchmarks (e.g., HumanEval). We release all our long-context Granite code
models under an Apache 2.0 license for both research and commercial use.