LACES-BiRWKV-DLM-2.9B-F2 — F2 continued-pretrain release (step 12000)

Bidirectional RWKV-7 masked-diffusion language model (BiRWKV-7, 4.09B params) after the F2 continued-pretraining round: 12.6B tokens on the PT corpora mixture (fineweb-edu / wiki / openwebmath / pg19 blend, pt_mixture_4096_packed), warm-started from the codecpt lineage (codecpt-2p9b/step_00004000 → f-2p9b-full/step_00011000 → this run).

This is the highest MMLU-proxy checkpoint of the BiRWKV lineage to date.

Results (F2 step 12000)

eval	score	baseline (RWKV7-Goose-World3-2.9B)
MMLU-proxy (CoLA protocol, 200 samples, raw frame)	50.00	StateDiffRWKV release 43.50
OBQA (n=500)	60.20	—
RACE (n=500)	52.60	—
pg19 ppl (causal, 512-chunk)	12.58	27.71 (2.2× better)
owt ppl (causal)	12.12	30.39 (2.5× better)
lm1b ppl (causal, OOD-dominated)	4247	5874 (1.38×)
wikitext103 ppl (causal)	10.99	—
lambada ppl (causal)	26.49	—

Knowledge-identification generalizes across tasks (MMLU/OBQA/RACE all at or above the release baseline). Code generation (HumanEval/MBPP) remains at 0.0 — a decode-path property of the architecture at this scale, not a training-data gap; see the tech report.

Full ability-vs-FLOPs curve: results/ability_curve_f2.csv in the GitHub repo (41.50 → 44.50 → … → 50.00 across 11.5B + 12.6B tokens; gains arrive late, after ~7B tokens).

Usage

import torch
from models.birwkv7_diffusion import (
    BiRWKV7ForMaskedDiffusion, iterative_denoise, MASK_TOKEN_ID,
)

model = BiRWKV7ForMaskedDiffusion.from_hf_pretrained(
    "RWKV/RWKV7-Goose-World3-2.9B-HF", dtype=torch.bfloat16
)
state = safetensors.torch.load_file("model.safetensors")
model.load_state_dict(state, strict=False)

ids = torch.tensor([[MASK_TOKEN_ID] * 128])      # all-masked canvas
denoised, commit_step = iterative_denoise(model, ids, torch.ones_like(ids, dtype=torch.bool),
                                          steps=16, temperature=0.0, self_correction=False)

Architecture and training code: Joshua-Zhang-Jiaquan/LACES (dev branch). The checkpoint is a flat bf16 state dict (1952 tensors); strict=False skips the optional conditioning modules (latent FiLM, block-timestep) that this checkpoint predates.

Lineage

RWKV7-Goose-World3-2.9B (frozen HF warm start)
  └─ convert-2p9b-bidir (bidirectional conversion)
      └─ codecpt-2p9b/step_00004000 (code CPT)
          └─ f-2p9b-full/step_00011000 (Phase F, 11.5B tokens)
              └─ f2-2p9b-ptcorpora/step_00012000  ← this release (F2, +12.6B tokens)

Training: 256 seq × 4096 tokens/step global batch, masked CE + λ_c causal replay (0.30→0.10), grouped-commit sampler g=4 at inference. Trainer: train/train_birwkv_diffusion.py in the repo.