N8Programs/lil-bard

Lil Bard 172M MoE

Lil Bard is a small English story language model pretrained from scratch. It is a base model, not an instruction-tuned or chat model.

The model has 172,052,992 total parameters and 58,806,784 active parameters per token. It uses 16 transformer layers, width 512, 8 feed-forward experts with top-2 routing, and a maximum exported context length of 32,768 tokens.

Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "N8Programs/lil-bard"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    dtype=torch.bfloat16,
    device_map="auto",
)

inputs = tokenizer("Once upon a time", return_tensors="pt").to(model.device)
with torch.inference_mode():
    output = model.generate(
        **inputs,
        max_new_tokens=200,
        do_sample=True,
        temperature=0.8,
        top_p=0.95,
        pad_token_id=tokenizer.pad_token_id,
    )
print(tokenizer.decode(output[0], skip_special_tokens=True))

The tokenizer automatically prepends BOS. Its special-token IDs are EOS 0, BOS 8190, and PAD 8191.

Architecture

Property	Value
Total parameters	172,052,992
Active parameters/token	58,806,784
Layers	16
Hidden size	512
Attention heads / KV heads	4 / 2
Head dimension	128
Experts / selected experts	8 / 2
Dense MLP size	1,536
Expert MLP size	768
Vocabulary	8,192
Maximum exported context	32,768
Published weight dtype	BF16

The checkpoint uses the stock Transformers Qwen3MoeForCausalLM layout. MoE expert weights are stored as per-expert gate_proj, up_proj, and down_proj tensors for compatibility across Transformers releases; loading has been tested with Transformers 4.57.1 and 5.11.0.

Tokenizer

The 8,192-entry tokenizer is a byte-level BPE tokenizer trained on a balanced 1.5-million-document sample of the corpus. It does not use regex, whitespace, or word pretokenization. BOS, EOS, PAD, and UNK are distinct tokens.

Training data

The corpus contained 8,732,634 documents drawn from:

• klusai/ds-tf1-en-3m
• karpathy/tinystories-gpt4-clean
• A deterministic 3-million-row sample of littlelearner/LittleCurriculum

A canonical validation set excluded 1,000 DS-TF1 test rows and 1,000 TinyStories test rows from training.

The model trained for exactly 2,492,032,616 real loss tokens over 25,485 distributed steps on two NVIDIA GB10 systems. Whole-document packing achieved 99.4713% utilization. Training used a local adaptation of N8python/spark-gpt.

The complete training trace is available in the lil_bard_moe_8x2_full W&B run.

Evaluation

Evaluation	Result
Canonical validation loss	1.42228 nats/token
ARC-Easy zero-shot accuracy	32.15%
ARC-Easy zero-shot normalized accuracy	32.79%

ARC-Easy was evaluated on all 2,376 test questions with lm-eval 0.4.12 in BF16, using the base-model prompt format and an explicit BOS token.

Historical checkpoints

To keep ordinary downloads of this repository small, the 25 periodic checkpoints are published separately in N8Programs/lil-bard-checkpts. They span step 1,000 through step 25,000 in increments of 1,000.

Limitations

This model was trained primarily on simple synthetic stories. It has limited world knowledge and reasoning ability, may produce repetitive or incoherent text, and has not been safety-tuned. Do not use it for factual, medical, legal, financial, or other high-stakes decisions.