adgw/Grendel-v0.1

Grendel v0.1

Grendel v0.1 is a 9.51 million parameter English language model trained from scratch by adgw, followed by checkpoint averaging and supervised fine-tuning. It explores how much language understanding and practical task performance can fit into a very small decoder.

Training covered more than the three metrics in the Tiny-ML Leaderboard. Alongside grammar, science question answering and language modeling, instruction data included sentiment analysis, multiple-choice answer selection (MCQA), text categorization and rule following. The goal was to develop useful capabilities and broader generalization, rather than optimize exclusively for the public leaderboard. Generalization to unseen tasks and examples remains to be established.

Public benchmark results

Author-reported full-test results, reproduced using the released Hugging Face model files:

Benchmark	Result	Coverage
BLiMP accuracy ↑	75.84%	50,811 / 67,000 pairs
ARC-Easy accuracy ↑	41.04%	975 / 2,376 test questions
WikiText-2 byte perplexity ↓	2.8783	Full raw test text; 1,292,008 UTF-8 bytes

Evaluation uses a 256-token context, FP32 weights with CUDA BF16 autocast, TF32 disabled and batch size 32. BLiMP uses sentence log-likelihood; ARC uses summed answer-continuation log-likelihood without length normalization. No chat template or BOS token is applied. WikiText uses space-joined raw text and disjoint windows of up to 256 target tokens.

Checkpoint selection used repeated public benchmark evaluations, including full-test scores, so these results carry selection bias. See full results, pinned benchmark sources and training provenance.

Local task benchmark

Our authored English suite contains 250 examples in each of seven categories: grammar, advanced grammar, science QA, sentiment, categorization, MCQA and prose language modeling. These are local tasks, separate from BLiMP, ARC and WikiText.

† The local Grendel v0.1 row reports a separate instruction-tuned variant, not the checkpoint downloadable from this repository. The downloadable checkpoint was adapted with local evaluation examples; its seen-example scores are not used in this table. The comparison variant used evaluation-exclusion checks. These tasks were used during development, and whole-lineage training overlap has not been exhaustively audited for all models.

All rows use the same CPU FP32 profile, 256-token context, raw language-model scoring, no BOS and no chat template. The comparison includes all locally evaluated external models with approximately 1–20 million parameters, from seven model authors or labs. Complete results are sorted by Score; partial diagnostics appear at the end.

Model	Parameters	Score ↑	Grammar ↑	Advanced grammar ↑	Science QA ↑	Sentiment ↑	Categorization ↑	MCQA ↑	MCQA generated / format ↑	Prose byte PPL ↓	Mean ↑
Tiny Chick 10M V2.1	9.59M	68.90	95.6%	95.6%	65.6%	26.0%	62.4%	25.6%	0.0% / 0.0%	2.4162	61.8%
Grendel v0.1 †	9.51M	68.18	92.4%	90.8%	50.4%	36.4%	73.2%	23.6%	23.6% / 100.0%	2.4973	61.1%
Tiny Chick 10M V2	9.59M	67.42	96.0%	95.6%	62.8%	22.8%	60.0%	25.6%	0.0% / 0.4%	2.4488	60.5%
Qielbas Tiny 9M M1	9.56M	67.35	96.4%	90.8%	63.2%	22.4%	64.4%	25.2%	1.2% / 4.4%	2.5486	60.4%
Qielbas Tiny 9M M3.1	9.56M	67.28	97.6%	90.0%	64.8%	25.6%	59.2%	24.8%	3.2% / 9.6%	2.5090	60.3%
Qielbas Tiny 9M M2	9.56M	66.76	95.6%	93.2%	61.6%	21.2%	62.0%	25.6%	0.4% / 2.8%	2.5152	59.9%
Tiny Chick 10M V3	9.59M	66.08	94.8%	88.8%	72.0%	21.6%	53.2%	25.2%	0.0% / 0.0%	2.3776	59.3%
Tiny Chick 10M	9.59M	64.00	93.2%	93.6%	53.6%	30.4%	48.4%	25.2%	0.0% / 0.4%	2.4772	57.4%
GoLLeM-v5 16M	17.45M	62.27	91.6%	83.2%	53.6%	32.4%	56.0%	26.0%	6.4% / 28.4%	2.4900	57.1%
qawel	9.59M	59.98	92.4%	80.4%	58.4%	30.0%	36.8%	24.8%	0.0% / 0.0%	2.5511	53.8%
Glint-2	1.06M	53.18	78.4%	70.4%	43.2%	28.8%	17.6%	26.0%	0.0% / 0.0%	3.1928	44.1%
Glint-2.3	1.71M	52.15	80.4%	69.2%	40.4%	29.2%	18.4%	26.0%	0.0% / 0.0%	2.9458	43.9%
GPT-S-5M ‡	5.16M	—	92.8%	87.2%	47.2%	24.0%	47.2%	24.8%	2.0% / 7.2%	—	53.9%
StorySupra-10M ‡	12.59M	—	73.2%	63.2%	32.8%	26.8%	9.6%	27.2%	0.0% / 0.0%	—	38.8%

‡ Partial results: GPT-S-5M and StorySupra-10M use native tokenizers that cannot reconstruct the exact input text. Their six task accuracies and mean are diagnostic; comparable prose byte perplexity is unavailable. They receive no full-ranking Score and are listed after complete evaluations. Parameter counts reflect the loaded models, including when they differ from the model name.

Accuracy is reported for the six classification/selection tasks. Score is their unweighted mean multiplied by the dashboard's fixed parameter-size bonus; it measures parameter-adjusted accuracy, not inference speed or energy efficiency. Prose byte perplexity is reported separately and is not WikiText perplexity. Advanced grammar targets more difficult constructions and vocabulary; it is not a certified CEFR assessment.

Sentiment uses four labels; categorization uses ten. Both apply empty-input calibration to label log-likelihoods. Science QA scores answer continuations; MCQA scores the letters a, b, c, d under an explicit multiple-choice prompt. The six primary task accuracies use likelihood scoring. The separate MCQA generated / format column reports greedy free-answer accuracy and valid-letter format compliance; invalid answers count as incorrect. Exact values, model revisions and scoring details.

Model and training

Property	Value
Parameters	9,509,120
Decoder layers / hidden width	12 / 256
Attention heads / KV heads	8 / 2
MLP	SwiGLU, intermediate width 704
Vocabulary	4,096 byte-level BPE tokens
Normalization / positions	RMSNorm / RoPE
Embeddings	Tied input and output weights
Training and evaluation context	256 tokens
Weights	FP32 safetensors

Pretraining used approximately 15 billion target-token exposures from educational web text, educational PDF text, English Wikipedia and a small instruction component. Supervised adaptation added approximately 78 million target-token exposures from high-quality instructions, reviewed task examples and language-model replay. No reinforcement learning was used. Exact accounting is available in training_provenance.json.

The custom Transformers implementation preserves the original model's arithmetic. Reloaded export logits were verified against the native checkpoint on CPU FP32, CUDA FP32 and CUDA BF16-autocast probes, including KV-cache checks. No pretrained model weights were used.

Usage

Install the versions in requirements.txt. Loading the custom implementation requires trust_remote_code=True.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "adgw/Grendel-v0.1"
tokenizer = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    repo, trust_remote_code=True, dtype=torch.float32
).eval()

inputs = tokenizer("The Earth moves around the Sun. This means that", return_tensors="pt")
with torch.inference_mode():
    output = model.generate(**inputs, max_new_tokens=48, do_sample=False)
print(tokenizer.decode(output[0], skip_special_tokens=True))

Use one unpadded prompt at a time for generation. Keep the prompt plus output within the recommended 256-token context; the architectural limit is 512, but quality beyond 256 has not been validated. Right-padded batches are supported for scoring. Left padding and padded batched generation are unsupported. No chat template is supplied; BOS/EOS are not inserted automatically.

To reproduce the public benchmark profile:

python evaluate_public.py --model adgw/Grendel-v0.1 --device cuda --output results.json

Use --revision <commit> to pin a model revision. The evaluator verifies pinned public benchmark inputs. --device cpu uses FP32 and may produce slightly different public scores. Keep FP32 weights for reproduction; casting all parameters to BF16 changes the numerical configuration.

Limitations and license

At this size, generated explanations can be incorrect or repetitive. Grendel is a research language model, not a validated general-purpose assistant. English is the primary language; multilingual performance is not established. Results on the small local development suite should not be interpreted as accuracy guarantees on new inputs.

Model weights and project code are released under Apache License 2.0; see LICENSE. Training corpora and private task examples are not redistributed. Source-data rights and licenses remain with their respective owners. We acknowledge FineWeb-Edu, FinePDFs-Edu and FineWiki/Wikipedia as training sources, and BLiMP, ARC and WikiText as public evaluation resources.