bizmod/Datlas-2.5

Datlas 2.5

Datlas 2.5 is the local model of the Atlas Autopilot Web-RAG agent stack. It is a from-scratch Qwen2-shaped causal LM (random initialisation — no pre-trained weights loaded), trained in a Kaggle notebook on the BlueIsGreen/Edge-Agent-Reasoning-WebSearch-260K corpus.

What is in this repository

Artefact	What it is	Status
model.safetensors (289 MB)	72,410,880 params — 8 layers, hidden 384, FFN 1024, ctx 256, tied embeddings. Trained 300 steps on a Tesla T4 (loss 12.00 → 4.98).	Real, trained, loadable
config_10b.json	The 10.00B Datlas 2.5 target architecture: 43 layers, d=4096, FFN 14336, 32 heads / 8 KV heads (GQA), ctx 4096, vocab 151,936, tied embeddings → 10,001,414,144 parameters.	Spec + config only — no weights yet
train_datlas_10b.py	The exact training engine for that 10B architecture (streaming parquet, prompt-masked loss, OneCycleLR, AdamW, checkpointing).	Runnable on multi-GPU
DATLAS_10B_SPEC.md	Hardware + data budget for the 10B run.	Reference
train_datlas_kaggle.py + training_log.json	The exact 72M training script and its metrics.	Archive

The 10B target — why there are no 10B weights here yet

A 10B from-scratch model is 20.4 GB of bf16 weights before a single optimizer state, i.e. ≈80 GB of AdamW state on top. A Kaggle T4 has 16 GB. A 10B checkpoint therefore cannot be produced in the Kaggle notebook, and this repository will not publish weights that were never trained.

To build it, run train_datlas_10b.py on rented multi-GPU:

torchrun --nproc_per_node=8 train_datlas_10b.py \
  --seq-len 4096 --micro-batch 1 --grad-accum 16 --steps 20000

Minimum: 8×A100-80GB, or 4×H100-80GB with FSDP/ZeRO-3. Budget ≈2–3 T tokens for a Chinchilla-optimal 10B, so the 260K-row corpus (~692M tokens) gives a domain-specialised model, not a general one. If a usable model today matters more than the from-scratch purity, QLoRA a pre-trained ~9–10B base on the same corpus instead.

Usage (the trained 72M checkpoint)

from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("bizmod/Datlas-2.5")
model = AutoModelForCausalLM.from_pretrained("bizmod/Datlas-2.5")
msgs = [{"role": "user", "content": "What should I check before deploying a crawler?"}]
ids = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt")
print(tok.decode(model.generate(ids, max_new_tokens=64)[0][ids.shape[-1]:], skip_special_tokens=True))

Training (72M checkpoint)

• Dataset: 263,098 rows, streamed + shuffled (seed 42); user_prompt / agent_reasoning with role/industry/os metadata; prompt tokens masked from the loss (−100).
• 300 optimisation steps, micro-batch 4 × 256 tokens, AdamW (peak LR 6e-4, OneCycleLR), weight decay 0.1, betas (0.9, 0.95), grad clip 1.0.
• Loss: 12.0018 → 4.9834 (min 2.6563). Device: Tesla T4.

Going to ~5B parameters

Two published routes, both shipped in this repo (see DATLAS_5B_SPEC.md):

• Route A - from scratch: config_5b.json = 4,896,473,088 params (44 layers, d=3072, ffn 8192, 24 heads / 8 KV heads, ctx 4096, tied embeddings). Trains with train_datlas_5b.py on >= 8xA100-80GB.
• Route B - depth up-scaling (recommended): config_5b_dus.json = 4,939,213,824 params. build_datlas_5b_upscale.py takes the pretrained Qwen2.5-3B (36 layers) and duplicates 24 decoder blocks (LLaMA Pro / SOLAR style) to 60 layers, then train_datlas_5b.py --init-from ./datlas-5b-dus continues pretraining. Because the base weights are already trained, a few hundred steps heal the duplicated blocks instead of starting from noise.
• Single 16 GB Kaggle GPU: train_datlas_5b_kaggle.py loads the depth up-scaled 5B in 4-bit NF4 (2.7 GB) with LoRA adapters and continues pretraining on the Edge-Agent-Reasoning corpus.