AstroLLaMA: Towards Specialized Foundation Models in Astronomy
Authors: name: Tuan Dung Nguyen; name: Yuan-Sen Ting; name: Ioana Ciucă; name: Charlie O'Neill; name: Ze-Chang Sun; name: Maja Jabłońska; name: Sandor Kruk; name: Ernest Perkowski; name: Jack Miller; name: Jason Li; name: Josh Peek; name: Kartheik Iyer; name: Tomasz Różański; name: Pranav Khetarpal; name: Sharaf Zaman; name: David Brodrick; name: Sergio J. Rodríguez Méndez; name: Thang Bui; name: Alyssa Goodman; name: Alberto Accomazzi; name: Jill Naiman; name: Jesse Cranney; name: Kevin Schawinski; name: UniverseTBD
Published: 2023-09-12 11:02:27

Abstract: Large language models excel in many human-language tasks but often falter in
highly specialized domains like scholarly astronomy. To bridge this gap, we
introduce AstroLLaMA, a 7-billion-parameter model fine-tuned from LLaMA-2 using
over 300,000 astronomy abstracts from arXiv. Optimized for traditional causal
language modeling, AstroLLaMA achieves a 30% lower perplexity than Llama-2,
showing marked domain adaptation. Our model generates more insightful and
scientifically relevant text completions and embedding extraction than
state-of-the-arts foundation models despite having significantly fewer
parameters. AstroLLaMA serves as a robust, domain-specific model with broad
fine-tuning potential. Its public release aims to spur astronomy-focused
research, including automatic paper summarization and conversational agent
development.