JunkyAI

A 15.7-million-parameter language model trained from scratch.

JunkyAI is a small English language model built end to end without any pretrained weights: its own byte-level BPE vocabulary, its own transformer implementation, its own training loop, and its own GGUF writer. It speaks simple, fluent English — roughly the register of a children's book — and answers short questions. It does nothing else, by design and by capacity.

It exists to be understood completely rather than to be capable. Every component is a few hundred lines of readable Python with no ML framework above PyTorch tensors: no transformers, no tokenizers, no peft, no gguf.

---

At a glance

Parameters	15,735,168 (~15.7M)
Architecture	Decoder-only transformer, Llama-shaped
Layers / hidden dim	8 / 384
Attention	6 query heads, 2 KV heads (grouped-query)
Feed-forward	SwiGLU, 1024 inner
Normalisation	RMSNorm, pre-norm
Positions	RoPE (interleaved pairs), θ=10000
Context length	512 tokens
Vocabulary	8,192 byte-level BPE, trained on the same corpus
Precision	bfloat16 training, Q8_0 export
File size	20.4 MB (Q8_0 GGUF)
Language	English only

The architecture is deliberately Llama-shaped so the exported GGUF loads in unmodified llama.cpp, Ollama and LM Studio. One detail differs from the usual HuggingFace implementation: RoPE is applied to interleaved pairs rather than a rotate-half split, matching ggml's GGML_ROPE_TYPE_NORM so weights export with no permutation step.

---

Training

Stage 1 — Pretraining

Corpus	TinyStories (roneneldan/TinyStories), 300,000 stories
Tokens	65.5M train / 329k validation
Steps	9,000 (batch 24 × grad-accum 2 × seq 512 = 24,576 tokens/step)
Total tokens seen	221M (~3.4 epochs)
Optimiser	AdamW, lr 3e-4, cosine decay to 10%, 300 warmup, weight decay 0.1
Final validation loss	1.4786 (perplexity 4.4)
Hardware / time	RTX 4060 8GB, ~55 minutes at 67,000 tokens/sec

TinyStories is a corpus of short children's stories written with a small vocabulary. It was chosen because its distribution is close to what a model this size can actually represent — a general web corpus would exceed its capacity and produce worse English, not better.

Stage 2 — Instruction tuning

Data	2,216 single-turn instruction pairs + 179 multi-turn conversations
Source	Synthetic, generated with Groq-hosted openai/gpt-oss-120b
Held-out eval	120 examples on topics absent from the training set
Steps	300 (batch 16, ~2 epochs), lr 1e-4
Final validation loss	2.5164
Time	~2 minutes

Only assistant tokens contribute to the loss; prompt tokens are masked out. Full fine-tuning was used rather than LoRA — at this size the model occupies under 4 GB of VRAM during training, so parameter-efficient methods save nothing and cost quality. LoRA and NF4 QLoRA are implemented in the repository for larger base models.

---

What it can do

• Hold a short, grammatical conversation in simple present and past tense
• Answer "What is X?" for everyday concrete nouns — animals, food, weather, household objects, places
• Greet, thank, say goodbye, and answer questions about itself
• Continue a short story coherently for several sentences
• Decline questions beyond its ability, with varied wording, and offer an alternative

Representative outputs:

How are you?               -> I am good, thank you.
hello                      -> Hi! I am happy to see you.
What is a dog?             -> A dog is a pet. It wags its tail and barks.
Tell me what rain is.      -> Rain is wet and cold.
What is 847 times 293?     -> That is too much for a small model like me.
                              But I can tell you about food or play.
Write me a Python program. -> I cannot work that out. I am sorry.
                              I can talk about simple things instead.

---

What it cannot do

These are capacity limits, not bugs, and no additional training fixes them at this parameter count.

• No arithmetic. It cannot add, multiply or count reliably.
• No code. It cannot read or write any programming language.
• No world knowledge. It knows nothing about real people, places, dates or events. Anything it says about them is invented.
• No reasoning. It cannot follow multi-step logic or answer "why" questions with genuine explanation.
• Long-range logic drifts. Within a few sentences it stays coherent; across a paragraph, causality breaks down. It may describe a character wanting something that belongs to a different character.
• Weak multi-turn. With only 179 conversations in training, it tends to lock onto the first topic and answer later, unrelated questions as if still on it. Single-turn use is markedly more reliable.
• Narrow vocabulary. Words outside the TinyStories range (~1,500–3,000 common English words) have near-random representations. It will produce them as noise.
• English only.

It will sometimes answer confidently and incorrectly. The graceful refusals cover common cases but are not a reliable safety property.

---

Intended use

Educational and experimental. JunkyAI is a complete, legible example of the full language-model pipeline at a scale that trains in an hour on one consumer GPU — useful for understanding tokenisation, transformer internals, pretraining, instruction tuning, quantisation and the GGUF format.

Not suitable for: production systems, factual reference, advice of any kind (medical, legal, financial), decisions affecting people, content moderation, or any setting where an incorrect answer carries cost. It has received no safety tuning or red-teaming beyond the refusal examples described above.

---

Data provenance and licensing

• Pretraining: TinyStories, by Eldan and Li (arXiv:2305.07759), distributed on the Hugging Face Hub as roneneldan/TinyStories under CDLA-Sharing-1.0. That is a share-alike data license: if you publish the data or data derived from it, you must do so under the same terms. Whether model weights count as derived data under CDLA is not settled law, so this card carries the same license as the safer assumption. TinyStories was itself generated by GPT-3.5 and GPT-4, so OpenAI's terms may also apply upstream.
• Instruction data: synthetic, generated through the Groq API using openai/gpt-oss-120b. Model outputs may be subject to the provider's terms; the data inherits whatever biases and errors that model has.
• Code: the training pipeline in this repository is separate from the weights and carries whatever license you choose for it. The license above applies to the released model, because of the corpus it was trained on.

The instruction data was generated and filtered automatically, and was not reviewed example by example.

---

Running it

llama-cli -m junkyai-q8_0.gguf -cnv

The chat template is embedded in the GGUF, so llama.cpp formats turns itself. Ollama, via a Modelfile:

FROM ./junkyai-q8_0.gguf
PARAMETER temperature 0.8
PARAMETER stop "<|eos|>"

Suggested sampling: temperature 0.7–0.8, top-k 40, top-p 0.95, repetition penalty 1.1, and 40–60 new tokens. Longer generations drift.

Ask one question at a time. Clearing the conversation between unrelated questions noticeably improves answers.

---

Reproducing

python scripts/bootstrap_pretrain_data.py     # ~10 min
python scripts/train_tokenizer.py             # ~20 s
python -m junkyai.train --stage pretrain --steps 9000
python scripts/generate_sft_data.py           # needs a Groq API key
python -m junkyai.train --stage sft --steps 300 --init-from checkpoints/junkyai-tiny
python -m junkyai.gguf_export --checkpoint checkpoints/junkyai-tiny-sft --quant q8_0

Verify the exported file against the checkpoint it came from:

python scripts/verify_gguf.py junkyai-q8_0.gguf --checkpoint checkpoints/junkyai-tiny-sft

---

Quantisation

Round-trip error measured against the source weights:

Format	Size	Mean relative error
F16	38 MB	0.018%
Q8_0 (released)	20.4 MB	0.56%
Q4_0	11 MB	9.0%

Q8_0 is the released format. Quantisation error is absorbed by redundant weights, and a 15.7M model has none to spare — Q4_0 saves 9 MB and does sixteen times the damage. Norm tensors are kept at F32 in all formats.