DedeBckp/BackKiyo-10M

BackKiyo-10M is a compact decoder-only language model from the Kiyo family, created for the backup account of DedeProGames.

The model was pretrained from scratch on 20 billion tokens drawn from FineWeb-Edu, DCLM-Baseline, and FinePhrase.

BackKiyo-10M uses a compact Qwen3-style decoder architecture with grouped query attention, RMSNorm, SwiGLU MLPs, RoPE positional embeddings, and tied input/output embeddings. The model contains just under 10 million parameters and was trained independently from a random initialization.

Model Details

Property	Value
Parameters	9,976,832
Architecture	Qwen3ForCausalLM
Layers	7
Hidden size	384
Attention heads	6
KV heads	2
Head dim	64
Intermediate size	700
Vocabulary size	8192
Context length used in training	2048
Activation	SwiGLU / SiLU
Normalization	RMSNorm
Attention	GQA
Positional encoding	RoPE
Weight tying	Tied input embeddings and LM head
Training tokens	20B
Training precision	bfloat16

Training Data

Source	Domain
FineWeb-Edu	General web text, education-filtered
DCLM-Baseline	General web text, high-quality filtered
FinePhrase	High-quality text and language data

The complete pretraining run consumed approximately 20 billion tokens across this mixture.

Benchmarks

Self-reported results from the BananaMind Base Bench 1.1 suite.

The evaluation contains 350 examples, split across seven categories with 50 examples each. Each item contains four candidate continuations and is scored using conditional mean log-probability.

Overall Result

Model	Params	Overall Elo	Accuracy
BackKiyo-10M	9.98M	920	40.6%

Detailed Results

Category	Accuracy	z vs. chance	Elo	Significant
Language completion	72.0%	+7.68	1,070	*
World knowledge	48.0%	+3.76	920	*
Logical reasoning	40.0%	+2.45	973	*
Commonsense	36.0%	+1.80	860
Quantitative	34.0%	+1.47	915
Context tracking	28.0%	+0.49	815
Code completion	26.0%	+0.16	924

* = passes 1.96σ vs. chance; n=50 per category.

By Difficulty

Difficulty	Accuracy
Easy	46.2%
Medium	40.2%
Hard	35.3%

Benchmark Summary

Metric	Value
Parameters	9,976,832
Overall Elo	920
Chance floor	805
Above chance floor	+115
Raw accuracy	40.6%
95% CI	[35.4%, 45.7%]
z vs. chance	+6.73
Statistical result	Significant
Evaluated examples	350
Scored sequences	1,400
Truncated sequences	0
Truncated continuations	0

The benchmark run reported 0 truncated sequences and 0 truncated continuations.

Scores are self-evaluated and may vary with benchmark revision, Transformers version, dtype, hardware, and evaluation settings.

Usage

Install the required dependencies:

pip install -U transformers safetensors torch

Load the model with Transformers:

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "DedeBckp/BackKiyo-10M"

tokenizer = AutoTokenizer.from_pretrained(model_id)

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=(
        torch.bfloat16
        if torch.cuda.is_available() and torch.cuda.is_bf16_supported()
        else torch.float32
    ),
).eval()

if torch.cuda.is_available():
    model = model.cuda()

prompt = "The meaning of life is "

inputs = tokenizer(
    prompt,
    return_tensors="pt",
)

inputs = {k: v.to(model.device) for k, v in inputs.items()}

with torch.no_grad():
    output = model.generate(
        **inputs,
        max_new_tokens=64,
        do_sample=False,
        repetition_penalty=1.1,
        pad_token_id=tokenizer.eos_token_id,
        eos_token_id=tokenizer.eos_token_id,
    )

print(tokenizer.decode(output[0], skip_special_tokens=True))

Limitations

BackKiyo-10M is a base language model, not an instruction-tuned model. It is designed primarily for text continuation rather than instruction following or conversational use.

At approximately 10 million parameters, the model has very limited capacity compared with larger language models. Benchmark results show its strongest performance in language completion, while world knowledge, reasoning, quantitative tasks, context tracking, and code completion remain substantially more limited.

The model may generate incorrect facts, inconsistent text, repetitive outputs, or lose coherence on difficult or long sequences. It should not be used for high-stakes decisions without independent verification.

License

Apache 2.0