AstroGPT 50M

AstroGPT 50M is an experimental 49.6M-parameter decoder-only language model trained for astronomy and astrophysics text generation.

Model Characteristics

Parameters: 49,583,360
Layers: 8
Attention heads: 8
Embedding dimension: 640
Context length: 256 tokens
Vocabulary size: 8,000
Activation: SwiGLU
Architecture: decoder-only GPT
Training checkpoint: iteration 4,500
Best validation loss: 3.31996

Files

model.pt — model weights
tokenizer.json — tokenizer
gpt.py — model architecture
config.json — model configuration

Intended Use

AstroGPT is intended for experimentation with:

astronomy language modeling
astrophysics text generation
domain-specific LLMs
educational demonstrations
scientific AI research
custom GPT architectures

Evaluation

AstroGPT is evaluated using Astro-Eval, a reusable evaluation framework providing:

benchmark evaluation
generation metrics
scientific terminology scoring
factuality evaluation
error analysis
experiment tracking
model comparison
Markdown reporting

Limitations

This is an experimental relatively small language model. Its outputs may contain factual errors, incomplete explanations, repetition, grammatical errors, or scientifically incorrect statements.

Generated outputs should not be treated as authoritative scientific information.

Checkpoint

This release corresponds to the recovered best checkpoint at training iteration 4,500.

Best validation loss:

3.31996

Related Project

AstroGPT and Astro-Eval were developed together as an experimental project for training and systematically evaluating domain-specific language models.

Downloads last month: 241