AstroLLaMA-2-7B-Base_Abstract

AstroLLaMA-2-7B-Abstract is a specialized base language model for astronomy, developed by fine-tuning Meta's LLaMA-2-7b architecture on astronomical literature. This model was originally developed by the AstroLLaMA team as part of the UniverseTBD initiative. It is designed for next token prediction tasks and is not an instruct/chat model.

Note: This model is provided for completeness in the series of AstroLLaMA models. The core AstroLLaMA team has since moved on to develop more advanced models under AstroMLab. For the original UniverseTBD version, please visit their repository.

Model Details

• Base Architecture: LLaMA-2-7b
• Training Data: Abstracts from 326,238 astronomy papers from arXiv's astro-ph category (April 1992 to July 2023)
• Fine-tuning Method: Parameter-Efficient Fine-Tuning (PEFT) with LowRank Adaptation (LoRA)
• Primary Use: Next token prediction for astronomy-related text generation and analysis
• Reference: Nguyen et al. 2023

Generating text from a prompt

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

# Load the model and tokenizer
tokenizer = AutoTokenizer.from_pretrained("AstroMLab/astrollama-2-7b-base_abstract")
model = AutoModelForCausalLM.from_pretrained("AstroMLab/astrollama-2-7b-base_abstract", device_map="auto")

# Create the pipeline with explicit truncation
from transformers import pipeline
generator = pipeline(
    "text-generation",
    model=model,
    tokenizer=tokenizer,
    device_map="auto",
    truncation=True,
    max_length=512
)

# Example prompt from an astronomy paper
prompt = "In this letter, we report the discovery of the highest redshift, " \
    "heavily obscured, radio-loud QSO candidate selected using JWST NIRCam/MIRI, " \
    "mid-IR, sub-mm, and radio imaging in the COSMOS-Web field. "

# Set seed for reproducibility
torch.manual_seed(42)

# Generate text
generated_text = generator(prompt, do_sample=True)
print(generated_text[0]['generated_text'])

Model Limitations and Biases

This model is specifically trained on astronomy abstracts and may not generalize well to other domains. Users should be aware of potential biases in the training data, which may reflect historical trends and biases in astronomical research publications.

Importantly, this model has been superseded by more advanced versions. Here's a performance comparison chart based upon the astronomical benchmarking Q&A as described in Ting et al. 2024.

Model	Score (%)
AstroSage-LLaMA-3.1-8B (AstroMLab)	80.9
AstroLLaMA-2-70B (AstroMLab)	76.0
LLaMA-3.1-8B	73.7
Gemma-2-9B	71.5
Qwen-2.5-7B	70.4
Yi-1.5-9B	68.4
InternLM-2.5-7B	64.5
Mistral-7B-v0.3	63.9
ChatGLM3-6B	50.4
AstroLLaMA-2-7B-AIC	44.3
AstroLLaMA-2-7B-Abstract	43.5

As shown, AstroLLaMA-2-7B series are outperformed by newer models. For state-of-the-art performance, we recommend using the latest models.

Ethical Considerations

While this model is designed for scientific use, users should be mindful of potential misuse, such as generating misleading scientific content. Always verify model outputs against peer-reviewed sources for critical applications.

Citation

If you use this model in your research, please cite:

@ARTICLE{2023arXiv230906126D,
       author = {{Dung Nguyen}, Tuan and {Ting}, Yuan-Sen and {Ciuc{\u{a}}}, Ioana and {O'Neill}, Charlie and {Sun}, Ze-Chang and {Jab{\l}o{\'n}ska}, Maja and {Kruk}, Sandor and {Perkowski}, Ernest and {Miller}, Jack and {Li}, Jason and {Peek}, Josh and {Iyer}, Kartheik and {R{\'o}{\.z}a{\'n}ski}, Tomasz and {Khetarpal}, Pranav and {Zaman}, Sharaf and {Brodrick}, David and {Rodr{\'\i}guez M{\'e}ndez}, Sergio J. and {Bui}, Thang and {Goodman}, Alyssa and {Accomazzi}, Alberto and {Naiman}, Jill and {Cranney}, Jesse and {Schawinski}, Kevin and {UniverseTBD}},
        title = "{AstroLLaMA: Towards Specialized Foundation Models in Astronomy}",
      journal = {arXiv e-prints},
     keywords = {Astrophysics - Instrumentation and Methods for Astrophysics, Astrophysics - Cosmology and Nongalactic Astrophysics, Astrophysics - Astrophysics of Galaxies, Astrophysics - High Energy Astrophysical Phenomena, Computer Science - Computation and Language, Computer Science - Machine Learning},
         year = 2023,
        month = sep,
          eid = {arXiv:2309.06126},
        pages = {arXiv:2309.06126},
          doi = {10.48550/arXiv.2309.06126},
archivePrefix = {arXiv},
       eprint = {2309.06126},
 primaryClass = {astro-ph.IM},
       adsurl = {https://ui.adsabs.harvard.edu/abs/2023arXiv230906126D},
      adsnote = {Provided by the SAO/NASA Astrophysics Data System}
}