AstroLLaMA: Towards Specialized Foundation Models in Astronomy
Authors: name: Tuan Dung Nguyen, name: Yuan-Sen Ting, name: Ioana Ciucă, name: Charlie O'Neill, name: Ze-Chang Sun, name: Maja Jabłońska, name: Sandor Kruk, name: Ernest Perkowski, name: Jack Miller, name: Jason Li, name: Josh Peek, name: Kartheik Iyer, name: Tomasz Różański, name: Pranav Khetarpal, name: Sharaf Zaman, name: David Brodrick, name: Sergio J. Rodríguez Méndez, name: Thang Bui, name: Alyssa Goodman, name: Alberto Accomazzi, name: Jill Naiman, name: Jesse Cranney, name: Kevin Schawinski, name: UniverseTBD

Large language models excel in many human-language tasks but often falter in
highly specialized domains like scholarly astronomy. To bridge this gap, we
introduce AstroLLaMA, a 7-billion-parameter model fine-tuned from LLaMA-2 using
over 300,000 astronomy abstracts from arXiv. Optimized for traditional causal
language modeling, AstroLLaMA achieves a 30% lower perplexity than Llama-2,
showing marked domain adaptation. Our model generates more insightful and
scientifically relevant text completions and embedding extraction than
state-of-the-arts foundation models despite having significantly fewer
parameters. AstroLLaMA serves as a robust, domain-specific model with broad
fine-tuning potential. Its public release aims to spur astronomy-focused
research, including automatic paper summarization and conversational agent
development.