Cronus-150M (Native Ternary SLM)

Cronus-150M is a ~151M parameter general-purpose native ternary small language model developed by OneBit AI. Forward body linear layers use native ternary weights {-1, 0, +1} with int8 activation quantization (BitNet b1.58 style) with fp32 master weights.

Model Details

• Organization: OneBit AI
• Architecture: BitLinear Transformer SLM
• Parameters: 151,100,160
• Hidden Dimension: 768
• Layers: 24
• Attention Heads: 12 (Query) / 4 (KV, Grouped-Query Attention)
• Intermediate Dimension: 2048
• Context Length: 2048
• Normalization: RMSNorm with Sub-Layer Normalization (SubLN)
• Vocabulary Size: 49,152 (SmolLM2-135M tokenizer)
• Precision / Training Mode: int8_tf32

Training Status (Burn / Smoke Run)

• Tokens Trained: ~149.95M tokens (286 updates, real 524,288 tokens/update)
• Training Hardware: NVIDIA GeForce RTX 5070
• Final Training Loss: 5.39 (from 10.99 initial)
• Held-out Probe Loss: 5.53 (from 6.68 initial)

Files in this Repository

• model.pt: Clean model weights (state_dict).
• step_000000286_final.pt: Full resume training checkpoint (model weights, fp32 master weights, optimizer state).
• config.json: Architecture configuration parameters.
• tokenizer.json: Tokenizer vocabulary and BPE merges.