Brought to you via @ContextReq - Built, tested and trained within 24 hours! (excluding all the knowledge I've learnt over ~2 years).

Notio-3.7M-RNN-v1

A deliberately minimal character-level language model (3.7M params, 2-layer GRU) trained on TinyStories, using a curated 109-token vocabulary: 8 boundary/whitespace tags, 94 printable ASCII characters, and 7 smart punctuation marks.

Notio's core idea: the model sees only token ids — a u8 stream, one byte per id — and all human-readable "english/symbols" rendering happens at runtime through a decoder that maps special tags (<spc>, <nwl>, <bos>, <eos>…) to their effects and drops them from display. The stream is byte-sized, not byte-level: it models the 109 curated characters above, not all 256 raw byte values.

Status: v1 Found the floor at 24k steps @ 0.679 val.

Design

• Vocabulary: 109 tokens — 8 specials + 94 printable ASCII + 7 smart punctuation. Every token id fits in one byte, so the entire corpus is a plain u8 stream.
• Boundary tags: each story is wrapped <bos> … <nwl><eos>; whitespace inside the stream is explicit (<spc>, <nwl>, <tab>).
• Runtime decoder: tags map to effects at decode time (<nwl>→newline, <spc>→space, <bos> <eos> <pad> <msk>→dropped). Tags are data in the stream, never shown to humans.

Architecture

component	detail
embedding	wte (109×512) + wpe (1024×512)
core	2 × GRU blocks, d_model=512, pre-LayerNorm + residual
head	LayerNorm + lm_head (109-way), weights tied to wte
total params	3,735,040 (fp32, ~15 MB weights)
context	block_size 1024, stateful truncated BPTT (k1 = k2 = 1024)

The stack is modular: layer 0 (data) → layer 1 (embedding) → 2× layer 2 (GRU block) → layer 11 (head).

Data

• roneneldan/TinyStories (TinyStoriesV2-GPT4-train), cleaned and filtered to stories containing only in-vocab characters: 2,118,989 stories.
• Char-level u8 id stream: 1,799,636,685 train ids / 95,696,324 val ids (95/5, split on a story boundary — train ends <eos>, val starts <bos>, concat hash-verified).
• Pipeline: tag → purge → tokenize; every stage parallelized and byte-exact verified (sha256 manifest, tokenizer round-trip ids identical: True, display identical: True).

Training

hyperparameter	value
optimizer	AdamW, grad-clip 1.0
schedule	lr 4e-4, warmup 500 steps, cosine → 4e-5 over 27k steps
batch	B=32, T=1024 (32,768 tokens/step)
BPTT	gradients truncated per 1024-window, RNN state carried & detached, reset only at sweep end
hardware	single GTX 1660 SUPER (6 GB), ~105k tok/s, ~2.3 h/run
checkpoints	every 1,000 steps: train/val loss, lr, weights, + 2 sampled stories (resumable)

Sample output

val loss @ step 24000: 0.679 --- sample 0 (901 ids) --- Once upon a time, there was a small fish in the woods. He loved to run and play, especially when the guards kept it every day. One day, he noticed a silly water and wanted to help the butterfly. Spot disappeared into the fog, but he just couldn't see it again. So, he had to send a large thing to hang the apples. The duck was getting tired, so he put on his shoe back down. He lay down on the ground and realized could think really wears his lips on them. Buzz was very sad. He'd cheese and no care make him want to leave. He kept doing the water out and stopped on a day of coins. It flew around and knew he could balance them all. The frog was so happy and went off in the small party. But when the creature started to clean, he was patient and laughing. Sam picked up the hose and started singing a play. He was so happy that his friend was so thankful to his friend and agreed to share it with --- sample 1 (901 ids) --- Once upon a time, there was a little girl named Lily. She loved going for milk and put a basket in the sink. One day, Lily's mom asked her to clean the hay and put them on a table. Simpressing brushes, and the leaves, but her mom warned her to just poke him back home. Lily was tired, but she knew his mom would come back again soon. When the clouds in the closet, Lily's grandpa came up and said, "Lily, the puppy is big! It looks like a new adventure." Betweeper, Jenny was very upset. Her friends were surprised and said, "Wow, these butterflies is so big than a palm in the garden. Do you like it?" Lily said "Yes, I'm just taking care of me in you too." From that day on, Lily twirled around and saved the delicious snakes. When sugar, it was just days and candles in the sky. She smiled and said yes. She was so glad that she had the massage and they went home with Max. She promised to weigh

Usage

Checkpoints embed the model config (load with weights_only=False). Repo tools: src/sampler.py (load/generate/decode) and src/complete.py (CLI).

# CLI: prompt in natural text, N completions, temperature/top-k control
python3 complete.py "Once upon a time, there was" \
    --checkpoint notio10k.pt --n 3 --temp 0.9 --top-k 40 --max-tokens 256

import sys
sys.path.insert(0, "path/to/repo")          # repo root: notio.py, complete.py, sampler.py
from sampler import load_model, encode_prompt, generate, ids_to_display

model, device, ck = load_model("notio10k.pt")
ids = generate(model, device, encode_prompt("Once upon a time"),
               max_tokens=256, temperature=0.9, top_k=0)
print(ids_to_display(ids))   # human text; boundary tags dropped/mapped

Verification scoreboard

• Data files: sha256 manifest check (all OK)
• train/val split: concat hash == source hash, boundary-aligned
• Tokenizer: full-corpus round-trip byte-exact (ids + display)
• BPTT: truncation (detached state is a graph leaf), statefulness, and tape continuity formally asserted
• GRU vs LSTM bake-off at equal settings: LSTM −1.5% loss, GRU +14% throughput → GRU chosen

Limitations

• Char-level: occasional misspellings and invented words (e.g. "spaceshimane")
• Small capacity: dream-logic entity swaps persist at this scale ("a small birthday who loved to shiver")
• Occasionally repeats loops near low temperature; sample with temperature/top-k
• Trained on synthetic GPT-4-generated children's stories — style is narrow and young-reader oriented
• English only, 109-token fixed vocabulary