Philosophy SDF midtrained 3B base

Derived from dlab-spp/t0-mt-3b-base, step-0 commit 8fa2bf935619631864c76e86e172202d94729e09.

One complete epoch over 13,201 decontaminated philosophy documents: 42,928,718 next-token targets, 20,972 packed blocks. LoRA rank 64, alpha 128 on attention and MLP projections; AdamW, learning rate 1e-4, cosine decay, 5% warmup, weight decay 0.01, context 2048. Follows the MSM method in https://arxiv.org/abs/2605.02087.

Root safetensors contain the merged, BF16 model. adapter/ also contains the LoRA adapter for the pinned base. See training_report.json for exact provenance, metrics, and coverage.

Downloads last month
450
Safetensors
Model size
3B params
Tensor type
BF16
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for P0u4a/t0-mt-3b-base-philosophy-sdf

Adapter
(1)
this model

Dataset used to train P0u4a/t0-mt-3b-base-philosophy-sdf

Paper for P0u4a/t0-mt-3b-base-philosophy-sdf