File size: 4,527 Bytes
1439fc0 5c4159b 1439fc0 5c4159b 422e74f 5c4159b fdc16c0 5c4159b 84601cb 4ca0457 5c4159b 4effa83 5c4159b b709de6 5c4159b 84601cb 5c4159b |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 |
---
license: mit
language:
- en
pipeline_tag: text-generation
tags:
- llama-2
- astronomy
- astrophysics
- arxiv
inference: false
base_model:
- meta-llama/Llama-2-70b-hf
---
# AstroLLaMA-2-70B-Base_AIC
AstroLLaMA-2-70B-Base_AIC is a specialized base language model for astronomy, developed by fine-tuning Meta's LLaMA-2-70b architecture on astronomical literature. This model was developed by the AstroMLab team and is, to our best knowledge, the first specialized 70B parameter-level LLM in astronomy. It is designed for next token prediction tasks and is not an instruct/chat model.
## Model Details
- **Base Architecture**: LLaMA-2-70b
- **Training Data**: Abstract, Introduction, and Conclusion (AIC) sections from arXiv's astro-ph category papers (from arXiv's inception up to July 2023)
- **Data Processing**: The training data was derived from LaTeX source files using regex-based extraction methods to identify and extract the relevant sections (Abstract, Introduction, and Conclusion).
- **Fine-tuning Method**: Continual Pre-Training (CPT) using the LMFlow framework
- **Training Details**:
- Learning rate: 2 × 10⁻⁵
- Total batch size: 160
- Maximum token length: 2048
- Warmup ratio: 0.03
- Cosine decay schedule for learning rate reduction
- Training duration: 1 epoch (approximately 2,000 A100 GPU hours)
- **Primary Use**: Next token prediction for astronomy-related text generation and analysis
- **Reference**: [Pan et al. 2024](https://arxiv.org/abs/2409.19750)
## Generating text from a prompt
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
# Load the model and tokenizer
tokenizer = AutoTokenizer.from_pretrained("AstroMLab/astrollama-2-70b-base_aic")
model = AutoModelForCausalLM.from_pretrained("AstroMLab/astrollama-2-70b-base_aic", device_map="auto")
# Create the pipeline with explicit truncation
from transformers import pipeline
generator = pipeline(
"text-generation",
model=model,
tokenizer=tokenizer,
device_map="auto",
truncation=True,
max_length=512
)
# Example prompt from an astronomy paper
prompt = "In this letter, we report the discovery of the highest redshift, " \
"heavily obscured, radio-loud QSO candidate selected using JWST NIRCam/MIRI, " \
"mid-IR, sub-mm, and radio imaging in the COSMOS-Web field. "
# Set seed for reproducibility
torch.manual_seed(42)
# Generate text
generated_text = generator(prompt, do_sample=True)
print(generated_text[0]['generated_text'])
```
## Model Performance and Significance
AstroLLaMA-2-70B-Base_AIC demonstrates notable improvements over its baseline LLaMA-2-70B model, marking a crucial step in specialized astronomical LLMs. Here's a performance comparison chart based upon the astronomical benchmarking Q&A as described in [Ting et al. 2024](https://arxiv.org/abs/2407.11194):
| Model | Score (%) |
|-------|-----------|
| **AstroSage-LLaMA-3.1-8B (AstroMLab)** | **80.9** |
| **<span style="color:green">AstroLLaMA-2-70B-Base (AstroMLab)</span>** | **<span style="color:green">76.0</span>** |
| LLaMA-3.1-8B | 73.7 |
| LLaMA-2-70B | 70.7 |
| Gemma-2-9B | 71.5 |
| Qwen-2.5-7B | 70.4 |
| Yi-1.5-9B | 68.4 |
| InternLM-2.5-7B | 64.5 |
| Mistral-7B-v0.3 | 63.9 |
| ChatGLM3-6B | 50.4 |
It demonstrates that training specialized LLMs can be effective, especially at larger model scales.
## Ethical Considerations
While this model is designed for scientific use, users should be mindful of potential misuse, such as generating misleading scientific content. Always verify model outputs against peer-reviewed sources for critical applications.
## Citation
If you use this model in your research, please cite:
```
@ARTICLE{2024arXiv240919750P,
author = {{Pan}, Rui and {Dung Nguyen}, Tuan and {Arora}, Hardik and {Accomazzi}, Alberto and {Ghosal}, Tirthankar and {Ting}, Yuan-Sen},
title = "{AstroMLab 2: AstroLLaMA-2-70B Model and Benchmarking Specialised LLMs for Astronomy}",
journal = {arXiv e-prints},
keywords = {Astrophysics - Instrumentation and Methods for Astrophysics, Computer Science - Computation and Language},
year = 2024,
month = sep,
eid = {arXiv:2409.19750},
pages = {arXiv:2409.19750},
doi = {10.48550/arXiv.2409.19750},
archivePrefix = {arXiv},
eprint = {2409.19750},
primaryClass = {astro-ph.IM},
adsurl = {https://ui.adsabs.harvard.edu/abs/2024arXiv240919750P},
adsnote = {Provided by the SAO/NASA Astrophysics Data System}
}
``` |