XGen-7B Technical Report
Authors: name: Erik Nijkamp; name: Tian Xie; name: Hiroaki Hayashi; name: Bo Pang; name: Congying Xia; name: Chen Xing; name: Jesse Vig; name: Semih Yavuz; name: Philippe Laban; name: Ben Krause; name: Senthil Purushwalkam; name: Tong Niu; name: Wojciech Kryściński; name: Lidiya Murakhovs'ka; name: Prafulla Kumar Choubey; name: Alex Fabbri; name: Ye Liu; name: Rui Meng; name: Lifu Tu; name: Meghana Bhat; name: Chien-Sheng Wu; name: Silvio Savarese; name: Yingbo Zhou; name: Shafiq Joty; name: Caiming Xiong
Published: 2023-09-07 02:20:03

Abstract: Large Language Models (LLMs) have become ubiquitous across various domains,
transforming the way we interact with information and conduct research.
However, most high-performing LLMs remain confined behind proprietary walls,
hindering scientific progress. Most open-source LLMs, on the other hand, are
limited in their ability to support longer sequence lengths, which is a key
requirement for many tasks that require inference over an input context. To
address this, we have trained XGen, a series of 7B parameter models on up to 8K
sequence length for up to 1.5T tokens. We have also finetuned the XGen models
on public-domain instructional data, creating their instruction-tuned
counterparts (XGen-Inst). We open-source our models for both research
advancements and commercial applications. Our evaluation on standard benchmarks
shows that XGen models achieve comparable or better results when compared with
state-of-the-art open-source LLMs. Our targeted evaluation on long sequence
modeling tasks shows the benefits of our 8K-sequence models over 2K-sequence
open-source LLMs.