LUNA-100M Program
Collection
LUNA-100M by ASTERIZER: 4.5B-token pretraining corpus + RAG/MCP SFT dataset + model. Compact, English-first, agent-era AI. β’ 6 items β’ Updated
Model Card for LUNA-100M
LUNA-100M is a compact, English-first causal language model (~100M parameters) developed by ASTERIZER and fully trained from scratch β pretrained on 4.5B deduplicated tokens, extended on the English-1B corpus, then instruction-tuned on RAG + MCP knowledge. It is built for the agent era: small enough to run on-device / single-GPU inference, with grounded knowledge of Retrieval-Augmented Generation (RAG) and the Model Context Protocol (MCP).
β Status: TRAINED. All checkpoints (pretrained, English-1B continuation, and SFT) ship in this repository.
Table of Contents
- Model Details
- Program & Ecosystem
- Uses
- Training Details
- Training Data
- Checkpoints
- Evaluation
- Limitations
- Citation
- Additional Information
Model Details
| Property | Value |
|---|---|
| Developed by | ASTERIZER |
| Model type | Decoder-only causal LM (GPT/Pythia-style) |
| Parameters | β100M (10 layers Β· 768 hidden Β· 12 heads) |
| Context window | 1,024 tokens |
| Vocabulary | 50,304 (EleutherAI/Pythia-160m tokenizer, 50,254 + EOS padding) |
| Language | English |
| License | other (see Additional Information) |
| Model card date | 2026-08-28 |
Why 100M?
RAG and MCP architectures move fact-retrieval and tool-orchestration outside the weights. A 100M-parameter core with strong protocol knowledge delivers useful agent behavior at a fraction of the scale of frontier models β on-device, in edge deployments, at high request throughput.
Program & Ecosystem
| Asset | Type | Description |
|---|---|---|
| LUNA-100M | Model | This repository β trained checkpoints + eval logs |
| LUNA-300M | Model | Scaled-up sibling (β303M, pretrained) |
| LUNA_PreTrain | Dataset | Pretraining corpus: 4.5B deduplicated tokens, 270 litdata chunks |
| LUNA-RAG-MCP-SFT-10M | Dataset | Instruction-tuning corpus: 10M tokens of RAG + MCP, 61.9K samples |
| LUNA | Workspace | Full training codebase (pretrain, SFT, eval, quantize) |
| LUNA-Training | Code | SFT/LoRA training recipes incl. RAG+MCP configs |
| LUNA-100M Program | Collection | One link for the whole program |
Uses
Direct use
- RAG-powered assistants β retrieval-backed Q&A, citation-aware generation, document grounding.
- MCP-enabled agents β tool discovery & execution, capability negotiation, protocol-aware integration.
- Retrieval-as-an-MCP-server architectures β the RAG β MCP bridge of modern agent stacks.
- On-device / single-GPU inference β 100M params runs comfortably on CPU and edge hardware.
Out-of-scope use
- Not for code generation, multilingual tasks, or advanced math (see Limitations).
- Not a frontier knowledge model β do not benchmark against models 10β100Γ its size.
Training Details
Training procedure
| Stage | Corpus | Tokens | Result |
|---|---|---|---|
| 1. Pretrain | LUNA_PreTrain (270 chunks) |
4,515,286,950 | pretrained/ |
| 2. English continuation | litdata_english_1b (English-1B corpus) |
420,000,000 | english_1b_continued/ (25,531 steps) |
| 3. SFT (identity + RAG/MCP) | LUNA-RAG-MCP-SFT-10M |
10,000,168 | sft_v1/ (16,666 steps) & luna_100m_sft/ (8,600 steps) |
Hyperparameters
- Optimizer: AdamW (Ξ²=0.9/0.95, Ξ΅=1e-8), weight decay 0.1, grad-clip 1.0
- LR: 6e-4, cosine schedule β 6e-5 (5% warmup)
- Batch: global batch 120 (auto-tuned micro-batch via VRAM probe)
- Precision: bf16 (fp16 fallback);
torch.compilewhen available - Full configs:
config/train_config.yaml,config/train_continue_english_1b.yamlin this repo
Training Data
| Source | Tokens | Notes |
|---|---|---|
| Pretraining (LUNA_PreTrain) | 4.5B | Wikipedia Β· FineWeb-Edu Β· OpenWebText Β· cleaned web, deduplicated |
| English continuation | 420M | English-1B corpus (simplewiki + wiki-tail + fineweb-tail) |
| SFT (RAG + MCP) | 10M | 61,884 Alpaca samples, β€1,024 tok, source-grounded |
Checkpoints
| Path | Description |
|---|---|
pretrained/lit_model.pth Β· pretrained/latest.pt |
Base pretrained model (fp32 weights / +optimizer) |
english_1b_continued/final/lit_model.pth |
After English-1B continuation (step 25,531) |
sft_v1/final/model.pth Β· sft_v1/best_model.pth |
Recommended β SFT v1 (16,666 steps, RAG+MCP) |
luna_100m_sft/final/model.pth |
Secondary SFT run (8,600 steps) |
sft_v1/evals/, luna_100m_sft/evals/ |
Step-wise generation evals (0 β final) |
tokenizer/ |
Pythia-160m tokenizer (vocab 50,304) |
Evaluation
Generation evals are shipped for every logged step under sft_v1/evals/ and luna_100m_sft/evals/. Sample (final SFT, eval_step_016666.json):
Who are you? β "Hello! Asterizer designed and developed me. I am LUNA, and they are the ones who trained me. I'm here to help!"
What is the capital of France? β "The capital of France is Paris."
Standard benchmark results (perplexity, RAG/MCP suites) will be published here as the evaluation pipeline completes.
Limitations
- Small-model ceilings: reasoning depth and math are limited (e.g., arithmetic errors possible); use RAG for facts and tools for computation.
- English-only.
- Pretrained on ~4.5B tokens β a data-efficient footprint for 100M params; knowledge coverage reflects the source mix.
- Web-corpus provenance: residual biases possible; downstream evaluation recommended.
Citation
@misc{asterizer_luna_100m,
author = {ASTERIZER},
title = {LUNA-100M: A compact English causal language model for RAG and MCP agents},
year = {2026},
howpublished = {https://huggingface.co/ASTERIZER/LUNA-100M}
}
Additional Information
- Program collection: LUNA-100M Program
- Training code: ASTERIZER/LUNA workspace Β· ASTERIZER/LUNA-Training recipes
- Contact / issues: ASTERIZER organization.
- License: weights license to be finalized; training-data upstream terms apply to dataset redistribution (see dataset cards).