Klyra-64M

Klyra-64M is a compact language model family developed as an academic research project. The project was initiated and developed by students of Informatics Engineering at Politeknik Negeri Jakarta (State Polytechnic of Jakarta), with a focus on efficient language modeling, staged post-training, reasoning, reinforcement learning, and model distillation at the sub-100M parameter scale.

The project explores how far a ~64M-parameter decoder-only language model can be pushed through staged pretraining, instruction tuning, reasoning-focused training, reinforcement learning, and specialist policy distillation.

Klyra is derived from the MiniMind-compatible architecture/runtime, with a custom training pipeline, data curriculum, post-training stages, evaluation workflow, and model checkpoints.

Current recommended checkpoint: Klyra-64M-Reasoning-Fundamentals
Experimental post-RL checkpoint: Klyra-64M-RL-OPD

---

Highlights

• ~64M trainable parameters
• 8 transformer layers
• 768 hidden size
• 8 attention heads / 4 KV heads
• ~6.4K Byte-Level BPE vocabulary
• 1024-token training context
• ~3B cumulative pretraining + midtraining tokens
• General instruction tuning
• Reasoning-fundamentals stage
• Specialist GRPO branches
• Online Policy Distillation (OPD)
• Evaluated on the same five English benchmark families used in the MiniMind-3 objective evaluation suite

Klyra is designed as a compact research model suitable for low-resource inference, controlled experimentation, and future edge or mobile deployment studies.

---

Model Architecture

Component	Klyra-64M
Architecture	Decoder-only Transformer
Parameters	~63,912,192
Layers	8
Hidden size	768
Attention heads	8
KV heads	4
Vocabulary	~6,400
Context length	1,024
MoE	No
Weight tying	Yes
Training precision	BF16
Tokenizer	MiniMind 6.4K Byte-Level BPE

The model uses tied token embedding / LM-head weights. Parameter counts may therefore differ slightly depending on whether a tool counts tied tensors once or separately.

---

Training Lineage

Random Initialization
        │
        ▼
Foundation Pretraining
~1B tokens
        │
        ▼
Midtraining
+~2B tokens
(~3B cumulative)
        │
        ▼
General Instruction SFT
        │
        ▼
Reasoning Fundamentals SFT
        │
        ├───────────────┬────────────────┬────────────────┐
        │               │                │                │
        ▼               ▼                ▼                ▼
Reasoning RL1      Instruction RL    General RL       Base student
        │
        ▼
Reasoning RL2
        │               │                │
        └───────────────┴────────────────┘
                        │
                        ▼
             Online Policy Distillation
                        │
                        ▼
                Klyra-64M-RL-OPD

---

Available Weights

The public Klyra-64M release currently consists of the following five checkpoints. These are the only model weights currently distributed as part of the public Klyra-64M release.

Project repository: Jahirrrr/Klyra-64M

Weight	Stage	Status	Hugging Face
Klyra-64M-Base	Foundation pretraining	Available	Jahirrrr/Klyra-64M-Base
Klyra-64M-Midtrain	Continued / midtraining	Available	Jahirrrr/Klyra-64M-Midtrain
Klyra-64M-Instruct	General instruction SFT	Available	Jahirrrr/Klyra-64M-Instruct
Klyra-64M-Reasoning-Fundamentals	Reasoning-focused SFT	Available · Recommended	Jahirrrr/Klyra-64M-Reasoning-Fundamentals
Klyra-64M-RL-OPD	RL + Online Policy Distillation	Available · Experimental	Jahirrrr/Klyra-64M-RL-OPD

Recommended checkpoint

For general use and downstream experimentation, the recommended checkpoint is:

Jahirrrr/Klyra-64M-Reasoning-Fundamentals

It currently provides the strongest overall balance between the five-task benchmark mean and manually evaluated reasoning behavior.

Experimental RL checkpoint

The final RL + OPD checkpoint is available at:

Jahirrrr/Klyra-64M-RL-OPD

This checkpoint is retained as an experimental research release. It broadly preserves aggregate benchmark capability, while its behavior differs from the recommended Reasoning-Fundamentals checkpoint on several manual reasoning and instruction-following tests.

Release note: Klyra-64M-Reasoning-RL1, Klyra-64M-Reasoning-RL2, Klyra-64M-IF-RL, and Klyra-64M-General-RL were intermediate specialist checkpoints used during the RL/OPD research pipeline. They are not currently distributed as public Klyra-64M weights.

---

Stage 1 — Foundation Pretraining

The model was initialized from random weights and pretrained on approximately 1B tokens.

Primary corpus:

• openbmb/Ultra-FineWeb-L1
• configuration: CC-MAIN-2025-30

Core settings:

Setting	Value
Sequence length	512
Peak LR	5e-4
Warmup	2%
Scheduler	Cosine
Weight decay	0.1
Precision	BF16
Effective tokens/update	~131K

Foundation validation result:

Metric	Result
Validation loss	2.3649
Perplexity	10.643

---

Stage 2 — Midtraining

Klyra was then continued for approximately 2B additional tokens, giving approximately 3B cumulative pretraining + midtraining tokens.

Training mixture:

Source	Approx. tokens
DCLM-Edu	720M
FineWeb-Edu	480M
Stack-Edu	500M
FineMath	120M
InfiMM-WebMath	80M
Cosmopedia v2	100M
Total additional	~2B

Core settings:

Setting	Value
Sequence length	1024
Peak LR	1.5e-4
Warmup	1%
Minimum LR ratio	0.1
AdamW betas	(0.9, 0.95)
Weight decay	0.1
Precision	BF16

Midtraining validation:

Metric	Result
Validation loss	2.3996
Perplexity	11.018

The midtraining validation set is not directly comparable with the foundation validation set.

---

Stage 3 — General Instruction SFT

Dataset:

• HuggingFaceTB/smol-smoltalk

After assistant-target validation and filtering:

Split	Rows
Train	226,049
Eval	992

Training:

• 2 epochs
• 1024 context
• assistant-only loss
• dynamic padding
• BF16
• peak LR approximately 5e-4
• 10% warmup

Results:

Epoch	Validation loss	Perplexity
1	1.1315	3.100
2	0.9828	2.672

This produced:

Klyra-64M-Instruct

---

Stage 4 — Reasoning Fundamentals

The next stage used a conservative training strategy intended to strengthen fundamental reasoning while limiting degradation of previously acquired capabilities.

Instead of immediately pushing Klyra into difficult competition-style reasoning, the goal was to strengthen:

• arithmetic
• unit-rate reasoning
• short multi-step reasoning
• simple logical deduction
• basic science
• algorithm fundamentals

Training set:

30,000 total examples

18,000 reasoning examples
12,000 general-instruction replay

Evaluation set:

1,000 total examples

600 reasoning
400 general-instruction replay

Training:

Setting	Value
Epochs	1
Peak LR	~6e-5
Context	1024
Loss	Assistant-only
Precision	BF16

The resulting checkpoint is:

Klyra-64M-Reasoning-Fundamentals

This remains the recommended Klyra-64M checkpoint.

---

Stage 5 — Specialist Reinforcement Learning

The Klyra reinforcement-learning experiment started from Klyra-64M-Reasoning-Fundamentals.

Three specialist branches were created.

Reasoning RL

Reasoning RL used a two-stage curriculum.

Primary datasets:

• openai/gsm8k
• ChilleD/SVAMP
• tasksource/ruletaker

Reasoning RL1

Approximate composition:

1,400 GSM8K short/easy
400 SVAMP
700 RuleTaker depth-1
-------------------------
2,500 prompts

Reasoning RL2

Approximate composition:

1,200 GSM8K medium
200 SVAMP
600 RuleTaker depth-2/depth-3
------------------------------
2,000 prompts

RL used a native GRPO-style implementation with:

• grouped generations
• group-relative advantages
• clipped policy ratio
• reference-policy KL regularization
• programmatic/verifiable rewards

---

Instruction-Following RL

The instruction-following branch used approximately 2,000 verified IFEval/RLVR-style prompts.

Source:

sungyub/ifeval-rlvr-verl

The verifier used classic IFEval-compatible rule checks.

The objective emphasized constraints such as:

• exact wording
• keyword inclusion/exclusion
• word/sentence counts
• JSON formatting
• capitalization rules
• punctuation rules
• section/title constraints
• response-format restrictions

Checkpoint:

Klyra-64M-IF-RL

---

General-Capability RL

General RL used train splits only from benchmark-family datasets:

Dataset	Training prompts
ARC-Easy	700
PIQA	900
OpenBookQA	500
CommonsenseQA	900
Total	3,000

Checkpoint:

Klyra-64M-General-RL

Test splits were not used for this RL stage.

---

Stage 6 — Online Policy Distillation

The OPD student starts again from:

Klyra-64M-Reasoning-Fundamentals

Teachers:

Reasoning teacher → Klyra-64M-Reasoning-RL2
IF teacher        → Klyra-64M-IF-RL
General teacher   → Klyra-64M-General-RL

Only teacher trajectories that passed objective verification were accepted for distillation.

Training:

Setting	Value
OPD steps	500
Student batch size	4
Peak LR	8e-6
Weight decay	0.01
Teacher candidates	3
Student initialization	Reasoning-Fundamentals

Output:

Klyra-64M-RL-OPD

This checkpoint is designated as experimental and is not the recommended default checkpoint.

---

Benchmark Results

Evaluation protocol

Klyra is evaluated on the five English benchmark families used in the MiniMind-3 objective evaluation suite:

• ARC-Easy
• PIQA
• OpenBookQA
• HellaSwag
• Social-IQA

The Klyra rows below were produced with the same Klyra evaluation notebook, using conditional-likelihood scoring and the same task family across checkpoints. For the Klyra evaluation, the primary metrics are acc_norm for ARC-Easy, PIQA, OpenBookQA, and HellaSwag, and acc for Social-IQA.

External sub-100M results are included as a reference comparison, not as a universal leaderboard. Different public repositories may use different lm-evaluation-harness versions, model revisions, dtypes, tokenizer implementations, or prompt/chat-template settings. Therefore, cross-project differences should not be interpreted as controlled significance tests unless the models are rerun under one identical evaluation environment.

Klyra five-task benchmark

Model	ARC-Easy	PIQA	OpenBookQA	HellaSwag	Social-IQA	Mean
Klyra-64M-Reasoning-Fundamentals	32.37	58.98	28.60	28.73	35.93	36.92
Klyra-64M-RL-OPD	32.41	59.36	27.80	28.39	35.98	36.79
Klyra-64M-Instruct	32.74	57.94	28.80	28.33	35.31	36.63
MiniMind-3-Exam	35.61	56.26	24.20	28.40	34.19	35.73
MiniMind-3	28.49	50.65	23.60	28.28	34.19	33.04

Klyra observations

• Klyra-64M-Reasoning-Fundamentals currently has the highest five-task Klyra mean at 36.92%.
• Klyra-64M-RL-OPD records the highest Klyra PIQA score at 59.36% and the highest Klyra Social-IQA score at 35.98%.
• The RL-OPD stage broadly preserves aggregate benchmark capability, but does not improve the overall five-task mean over Reasoning-Fundamentals.
• The MiniMind rows are author-reported reference results. The Klyra rows are produced by the Klyra benchmark notebook.

Broader sub-100M comparison

Many sub-100M models do not publish OpenBookQA and Social-IQA results. To include a substantially wider set of public models, the table below uses the three broadly overlapping zero-shot tasks:

ARC-Easy + PIQA + HellaSwag

Mean-3 is the unweighted arithmetic mean of those three scores. The table is sorted by Mean-3. It is intended to provide context for Klyra rather than establish a universal SOTA ranking.

Model	Params	ARC-Easy	PIQA	HellaSwag	Mean-3	Result source
tinctura-v1	96.2M	47.98	65.61	37.96	50.52	Open SLM
Rose-1.5-Medium	98.2M	47.22	64.80	38.09	50.04	Open SLM
Surjo-100m	97.7M	47.64	63.87	35.05	48.85	Open SLM
cRia-LM-75M¹	75.7M	46.97	63.93	35.49	48.80	Open SLM submission
Rose-Medium	97.8M	44.19	62.95	35.29	47.48	Model card
Kiyo-65M	65.0M	46.51	62.30	31.98	46.93	Open SLM
Supra-50M-Base	~51.8M	46.00	62.19	31.72	46.64	Model card
Surjo-50M	53.8M	44.99	63.11	31.37	46.49	Open SLM
Supra-1.5-50M-base-exp	51.8M	48.40	60.01	29.78	46.06	Open SLM
BananaMind-2-Medium	49.6M	43.81	61.86	32.43	46.03	Model card
Veyra2-Apricot-50M-Base	49.3M	42.47	62.13	31.28	45.29	Open SLM
Rose-Mini	49.4M	36.62	58.54	28.95	41.37	Model card
MiniMind-3-Exam	64M	35.61	56.26	28.40	40.09	MiniMind README
Klyra-64M-RL-OPD	64M	32.41	59.36	28.39	40.05	Klyra evaluation
Klyra-64M-Reasoning-Fundamentals	64M	32.37	58.98	28.73	40.03	Klyra evaluation
Klyra-64M-Instruct	64M	32.74	57.94	28.33	39.67	Klyra evaluation
SmolLM2-70M	69.2M	33.00	54.84	26.14	37.99	Model card
Pythia-70M	70.4M	31.65	53.48	27.49	37.54	Open SLM
Stentor3-50M	49.4M	29.67	53.75	27.10	36.84	Open SLM
MiniMind-3	64M	28.49	50.65	28.28	35.81	MiniMind README

¹ The table uses the cRia Open SLM submission values for consistency with the broader comparison. The current cRia model card reports a later checkpoint with higher scores after additional training that includes one epoch over several benchmark-family training splits; those later scores are therefore not substituted into this table.

---

Sub-100M Architecture and Training Comparison

The table below summarizes the architecture and training exposure of the principal sub-100M comparison models above. Pretraining / continued training refers to publicly disclosed token exposure for the cited checkpoint. A dash indicates that the corresponding value was not clearly disclosed in the public model card used for this comparison.

Model	Params	Architecture / depth	Hidden	Q / KV heads	Vocab	Context	Pretraining / continued training
Klyra-64M family	64M	Dense decoder, 8 layers	768	8 / 4	~6.4K	1,024	~1B foundation + ~2B midtraining = ~3B cumulative
tinctura-v1²	96.2M	cagliostro-v3-derived, 18 layers	~640	~10 / 5	~32.8K	~2,048	75B total, from random initialization
Rose-1.5-Medium	98.2M	Rose X1.5, 22 layers	576	9 / 3	32,768	4,096	~100B
Surjo-100m	97.7M	Hybrid GDN-2 + XSA, 18 physical / 34 effective	576	9 / 3	32,768	1,024 train / 2,048 max	20B
cRia-LM-75M	75.7M	Relaxed Recursive Transformer, 13 unique / 24 effective	576	9 / 3	49,152	4,096	10B base + ~1.31B released midtraining + ~15M context extension, followed by benchmark-family train-split adaptation
Rose-Medium	97.8M	Rose X1, 24 layers	512	8 / 2	32,768	2,048	80.06B
Kiyo-65M	65.0M	Llama/SmolLM2-style, 15 layers	512	8 / 4	49,152	2,048	79.999B (~80B)
Supra-50M-Base	~51.8M	Llama-style, 12 layers	512	8 / 4	32,000	1,024	20B
Surjo-50M	53.8M	Hybrid recurrent GDN-2 + XSA, 10 physical / 18 effective	512	8 / 4	32,768	1,024 train / 2,048 max	20B
Supra-1.5-50M-base-exp	51.8M	Supra 1.5 experimental 50M family	—	—	—	—	Public experimental checkpoint; see model card
BananaMind-2-Medium	49.6M	Custom decoder, 12 layers	512	8 / 2	12,288	3,072	49.9997B (~50B)
Veyra2-Apricot-50M-Base	49.3M	Llama-like, 16 layers	512	8 / 2	8,192	1,024	19.99B
Rose-Mini	49.4M	Rose X1, 14 layers	512	8 / 2	—	—	12.207B
SmolLM2-70M	69.2M	Llama, 32 layers	384	6 / 2	49,152	8,192	Upstream SmolLM2-70M base + 30.6B Sutra continued-training exposure
Pythia-70M	70.4M	GPT-NeoX, 6 layers	512	8 / 8 (MHA)	50,304	2,048	299.893B
Stentor3-50M	49.4M	Llama, 15 layers	512	8 / 2	4,096	4,096	~8.3B
MiniMind-3 / MiniMind-3-Exam	64M	Dense decoder, 8 layers	768	8 / 4	6,400	32,768 max position	Public README does not disclose one authoritative total pretraining-token count

² tinctura-v1 is publicly described as a 96.2M-parameter, 18-layer variant of the cagliostro-v3 architecture/recipe trained to 75B tokens. The approximate width/head/vocabulary/context entries shown here correspond to that architecture family; the authoritative checkpoint configuration should be taken from the model repository when reproducing it.

Training-exposure perspective

Klyra's representation-learning exposure is comparatively small:

Model	Disclosed token exposure	Approx. tokens / parameter
Klyra-64M	~3B	~47
Stentor3-50M	~8.3B	~168
cRia-LM-75M³	~11.3B+	~149+
Rose-Mini	~12.2B	~247
Surjo-100m	20B	~205
Surjo-50M	20B	~372
Supra-50M-Base	20B	~386
Veyra2-Apricot-50M	~20B	~405
BananaMind-2-Medium	50B	~1,009
tinctura-v1	75B	~780
Rose-Medium	80.06B	~818
Kiyo-65M	~80B	~1,231
Rose-1.5-Medium	~100B	~1,018
Pythia-70M	~299.9B	~4,258

³ The cRia figure only reflects the disclosed base, released midtraining, and selected context-extension exposure. It excludes the later benchmark-family adaptation epoch because a single aggregate token count for that stage is not reported in the model card.

This comparison is useful for interpreting Klyra's current benchmark position: Klyra reaches its present scores after approximately 3B cumulative foundation and midtraining tokens, whereas many stronger sub-100M peers report tens of billions of training tokens.

External comparison sources

Public comparison values and architecture metadata were collected from the following model cards and public benchmark pages:

• AxiomicLabs Open SLM Leaderboard
• MiniMind-3
• tinctura-v1
• Rose-1.5-Medium
• Rose-Medium
• Rose-Mini
• Surjo-100m
• Surjo-50M
• cRia-LM-75M
• Kiyo-65M
• Supra-50M-Base
• BananaMind-2-Medium
• Veyra2-Apricot-50M-Base
• SmolLM2-70M
• Pythia-70M
• Stentor3-50M

The public sub-100M model ecosystem is larger than this table and changes over time. This section therefore represents a verified comparison set of relevant public peers with overlapping benchmark results, rather than an exhaustive catalog of every model below 100M parameters on the Hugging Face Hub.

---

Manual Reasoning Behavior

Reasoning-Fundamentals showed meaningful improvements on short fundamental reasoning tasks, including:

• integer addition with carry
• subtraction
• unit-rate problems
• basic syllogistic logic
• basic science facts
• binary-search explanation

Example:

Prompt:
What is 17 + 28?

Klyra:
<think>
In the ones place, 7 + 8 = 15; write 5 and carry 1.
In the tens place, 1 + 2 + carry 1 = 4.
So 17 + 28 = 45.
</think>
Final answer: 45.

---

Checkpoint Selection

Recommended general-purpose checkpoint

Use:

Jahirrrr/Klyra-64M-Reasoning-Fundamentals

This is currently the best-balanced public Klyra checkpoint.

General instruction baseline

Use:

Jahirrrr/Klyra-64M-Instruct

This checkpoint is useful as the general instruction-tuned baseline for comparisons with later reasoning and RL stages.

Experimental RL / OPD research

Use:

Jahirrrr/Klyra-64M-RL-OPD

This is the publicly released final checkpoint from the specialist-RL and Online Policy Distillation experiment. The individual specialist-teacher checkpoints are not currently released.

Continued-pretraining experiments

Use:

Jahirrrr/Klyra-64M-Midtrain

For experiments that require an earlier pretrained checkpoint, use:

Jahirrrr/Klyra-64M-Base

---

Tokenizer

Klyra keeps the existing MiniMind-style 6.4K Byte-Level BPE tokenizer.

Important tokens include chat/reasoning control tokens such as:

<|im_start|>
<|im_end|>
<think>
</think>
<tool_call>

The tokenizer was intentionally kept stable across the training lineage so checkpoints remain compatible.

---

Limitations

Klyra contains approximately 64M parameters and was exposed to approximately 3B cumulative pretraining and midtraining tokens prior to post-training.

Known limitations include:

• limited factual knowledge
• hallucination
• weak long-form coherence
• fragile multi-step reasoning
• operation-selection errors
• weak advanced mathematics
• limited coding ability
• repetition on longer generations
• sensitivity to prompt format
• weak exact instruction following in some checkpoints
• no guarantee of factual correctness
• 1024-token training context

The project is intended primarily as a study of capability development in compact language models and should not be interpreted as a substitute for substantially larger production-oriented models.

---

Training-Efficiency Context

Klyra has seen substantially fewer pretraining/midtraining tokens than many competitive sub-100M language models.

Klyra-64M
~3B cumulative pretraining + midtraining tokens
~64M parameters
≈47 representation-learning tokens per parameter

This relatively limited token budget motivates further continued-pretraining experiments.

---

Attribution

Klyra uses a MiniMind-compatible model architecture/runtime.

MiniMind:

https://github.com/jingyaogong/minimind

Klyra training, data curriculum, checkpoint lineage, reasoning stages, RL branches, evaluation workflow, and OPD experiments are separate work developed for the Klyra project.

Please preserve applicable upstream attribution and license notices when redistributing derived code or checkpoints.

---

License

Klyra is released under the Apache License 2.0, subject to the licenses and terms of the upstream code and datasets used during training.

Individual training datasets may have their own licenses and usage requirements. Users are responsible for reviewing those terms before redistribution or commercial use.

---

Citation

If you use Klyra in research, you can cite the repository as:

@misc{klyra64m,
  title        = {Klyra-64M: A Compact Language Model with Staged Reasoning and Reinforcement Learning},
  author       = {Athallah, Zahir Hadi},
  year         = {2026},
  howpublished = {Hugging Face},
  note = {An academic research project developed by Informatics Engineering students at Politeknik Negeri Jakarta}
}

---

Model Family Summary

Klyra-64M-Base                         [Public]
    ↓
Klyra-64M-Midtrain                     [Public]
    ↓
Klyra-64M-Instruct                     [Public]
    ↓
Klyra-64M-Reasoning-Fundamentals       [Public · Recommended]
    │
    ├── Reasoning RL1                  [Intermediate · Not released]
    │       ↓
    │   Reasoning RL2                  [Intermediate · Not released]
    │
    ├── Instruction-Following RL       [Intermediate · Not released]
    │
    └── General-Capability RL          [Intermediate · Not released]
                 │
                 ▼
          Klyra-64M-RL-OPD             [Public · Experimental]

Main project repository: Jahirrrr/Klyra-64M
Recommended checkpoint: Jahirrrr/Klyra-64M-Reasoning-Fundamentals