RafaelUI/rwkv-layer-probing-checkpoints

RWKV-7 and Transformer (RoPE / NoPE) checkpoints for layer-wise probing

Small language models used in the article Which layers know what? Probing RWKV-7 against Transformers (RoPE and NoPE) at small scale (Alexei Goncharov, November 2026; article repository: RafaelUI/rwkv-layer-probing_ARTICLE on GitHub). 12 runs, seed 0 only, 21 checkpoints per run (step 0 = random initialization, then every 5 % of training).

directory	model
{lang}_{size}_rwkv_s0	RWKV-7
{lang}_{size}_tf_rope_s0	Transformer with rotary position embeddings
{lang}_{size}_tf_nope_s0	Transformer without any position encoding

lang is ru or en. size is M (512 wide, 12 layers, 56.6M parameters, 3 800 steps, ~1.0 B tokens) or L (768 wide, 12 layers, ~115M parameters, 7 600 steps, ~2.0 B tokens). Files: rwkv-0-step{N}.pth or {kind}-step{N}.pth (PyTorch pickles, fp32), val.jsonl and loss_data.jsonl (loss curves), config.json (Transformers), train_log.txt (RWKV).

Not included yet: the second seed (seed 1), the training code, the 16k BPE tokenizers and the data manifests. The models are small research models, not general-purpose language models. Load .pth files only from sources you trust, as pickles can run code.