Kiyo-230M-Preview

Kiyo-230M-Preview is an intermediate training checkpoint of Kiyo-230M, a decoder-only language model being pretrained from scratch on FineWeb-Edu, DCLM-Baseline, FineMath, and Stack-v3-train. It is released as a preview: the training run is still in progress and this checkpoint has seen roughly one fifth of its planned token budget. See Preview Status below before using or comparing it.

The architecture scales the Kiyo-135M design up to ~230M parameters: a Llama-style decoder with grouped query attention, RMSNorm, SwiGLU MLPs, and tied input/output embeddings, following the SmolLM2 architecture family and reusing its 49k-token vocabulary. Compared to Kiyo-135M, it goes deeper and wider — 32 layers at hidden size 768, against 30 layers at 576 — while keeping the same tokenizer, the same data mixture, and the same training code.

Preview Status

This repository holds the weights at optimizer step 81,500 of a run targeting 200B tokens. Three things follow from that, and all three matter when reading the benchmark numbers:

Field	Value
Tokens seen by this checkpoint	42,729,472,000 (42.7B)
Target token budget	200,000,000,000 (200B)
Progress	21.4%
Optimizer step	81,500
Planned total steps	~381,470
Learning rate at this checkpoint	1.2e-3 (peak, stable phase)
LR decay begins at step	~343,320 (final 10% of the run)

The learning rate has not decayed yet. The run uses a warmup-stable-decay schedule whose decay phase covers only the final 10% of steps. This checkpoint sits deep in the stable phase, still at the peak LR of 1.2e-3. Models evaluated mid-plateau consistently score below what the same weights reach after annealing, so the numbers below understate the model rather than describe it.

It has seen fewer tokens than Kiyo-135M. Kiyo-135M was trained on the full 200B tokens. This preview has seen 42.7B. A parameter-count comparison between the two is not a fair comparison of the architectures — it is a comparison of a finished run against a fifth of one.

Weights are not final and will change. Do not treat this checkpoint as the Kiyo-230M release. It exists so the run can be inspected mid-flight; the final model will be published separately.

Model Details

Field	Value
Parameters	229,688,064
Architecture	Llama-style decoder (SmolLM2 architecture family)
Layers	32
Hidden size	768
Intermediate size	1,920
Attention heads	12
KV heads	4
Head dimension	64
Attention type	Grouped query attention
Activation	SwiGLU
Normalization	RMSNorm
Positional encoding	RoPE (theta 10,000)
Vocabulary size	49,152
Context length	2,048
Embeddings	Tied input/output embeddings
Training tokens (this checkpoint)	42,729,472,000
Tokens per optimizer step	524,288
Peak learning rate	1.2e-3
LR schedule	Warmup-stable-decay (1/sqrt), decay not yet reached
Weight format	safetensors

Training Data

Source	Domain	Mixture share
FineWeb-Edu	General web text, education-filtered	45.0%
DCLM-Baseline	General web text, high-quality filtered	30.0%
Stack-v3-train	Source code	17.5%
FineMath	Mathematical reasoning	7.5%

The Stack-v3 portion is sampled with per-language keep probabilities across 32 curated programming languages, giving each language an equal token quota rather than inheriting the corpus's natural skew.

Benchmarks

Self-reported result from the official BananaMind Base Bench 1.1 script, measured on CUDA in float32.

Model	Params	Training tokens	Overall Elo
Kiyo-135M	134.5M	200B	1,126
BananaMind-2-Pro	139.0M	100B	1,124
Rose-Pro	151.3M	—	1,105
Kiyo-230M-Preview	229.7M	42.7B	1,086
GPT-2	124M	10B	990

Figures for BananaMind-2-Pro, Rose-Pro, and GPT-2 are as self-reported on their own model cards, all against the same BananaMind Base Bench 1.1 suite. This preview sits below Kiyo-135M despite having 1.7× the parameters, which is the expected result at 21% of the token budget with the learning rate still at peak — see Preview Status.

Detailed Kiyo-230M-Preview result

Category	Accuracy	z vs. chance	Elo	Significant
Language completion	98.0%	+11.92	1,468	*
Code completion	74.0%	+8.00	1,286	*
World knowledge	72.0%	+7.68	1,079	*
Commonsense	70.0%	+7.35	1,077	*
Context tracking	44.0%	+3.10	931	*
Logical reasoning	44.0%	+3.10	1,017	*
Quantitative	38.0%	+2.12	958	*

* = passes 1.96σ vs. chance; n=50 per category

Every category clears the significance threshold, including Quantitative — the one category where Kiyo-135M does not.

By difficulty

Difficulty	Accuracy
Easy	73.5%
Medium	60.7%
Hard	54.3%

Summary

Metric	Value
Parameters	229,688,064
Overall Elo	1,086
Chance floor	805
Above chance floor	+282
Raw accuracy	62.9%
95% CI on accuracy	57.8% – 67.9%

Scores are self-evaluated and may vary with the benchmark revision, Transformers version, dtype, hardware, and generation settings. With 350 items, the 95% interval spans roughly ±5 accuracy points, so small Elo gaps between models should not be read as decisive.

Usage

pip install -U transformers safetensors torch

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "DedeProGames/Kiyo-230M-Preview"

tokenizer = AutoTokenizer.from_pretrained(model_id)

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16 if torch.cuda.is_available() and torch.cuda.is_bf16_supported() else torch.float16,
).cuda().eval()

prompt = "The meaning of life is "
input_ids = tokenizer(prompt, return_tensors="pt").input_ids.to(model.device)

with torch.no_grad():
    output = model.generate(
        input_ids=input_ids,
        max_new_tokens=64,
        do_sample=False,
        repetition_penalty=1.1,
        pad_token_id=tokenizer.eos_token_id,
        eos_token_id=tokenizer.eos_token_id,
    )

print(tokenizer.decode(output[0], skip_special_tokens=True))

Limitations

This is an unfinished base model, not instruction-tuned — it continues text rather than following instructions, and its weights are an intermediate checkpoint rather than a release. Beyond the preview caveats above, it shares the limits of its scale: fluent and grammatical on language completion, competent on code, but weak on multi-step logic and context tracking, where it stays near chance on the harder items. The 2,048-token context window limits long-document use. It can generate incorrect facts and should not be used for high-stakes decisions without verification. Keep a finite generation limit to avoid repetition or drift on long outputs.

License

Apache 2.0