Notio2 โ 8.3M Character-Level Hybrid LM
Version: notio2-8m-curriculum-r1 (in training; card reflects status as of 2026-09-08) Developer: Notio2 project (hobby/research) License (weights): TBD โ see License section
Model Details
Notio2 is a small character-level language model trained to generate children's-style stories. It is the successor to Notio1 (3.79M-param 2xGRU, TinyStories, val 0.679 nats/char).
| Property | Value | NET2NET *2 Layer after 1K steps. |
|---|---|---|
| Parameters | 8,317,184 | 15,559,464 |
| Architecture | hybrid: 5 sliding-window attention blocks + 15 GRU blocks (pattern aggg x5) |
10 swa blocks + 30 GRU blocks |
| d_model | 256 | 256 |
| Property | Value |
|---|---|
| Context (block size) | 4,096 chars |
| Vocabulary | 101 tokens: 7 control glyphs + 94 printable ASCII (pure character level) |
| Attention | 4 heads x 64, RoPE, kv-cache, window 1,024 |
| Recurrent block | GRU (d=256), stateful across BPTT windows |
| Embeddings | wte/wpe learned; lm_head tied to wte |
| Training objective | next-character cross-entropy (stateful truncated BPTT, horizon 512) |
Lineage: Notio1 (3.79M, pure GRU, TinyStories) -> Notio2-22M hybrid (V=8192 subword, retired after degenerate 'soup basin' behavior) -> this model (char-only vocab, deeper, narrower).
Intended Use
- Research and education: tiny-LM behavior, curriculum learning, corpus-difficulty measurement (see 'Mount Corpus-Floor').
- Creative play: sampling short children's-style stories.
Out of scope: production text generation, instruction following (no SFT/RLHF), languages other than English, safety-critical applications. The model has no alignment training and simply models its training distribution.
Training Data
Curriculum-ordered corpus, one pass per epoch, easy-first:
| Component | Tokens | Source / License | Notes |
|---|---|---|---|
| TinyStories (prefix) | ~50M | Eldan & Li 2023, CDLA-Permissive-2.0 | 56,222 deduped stories, prepended |
| Synthetic stories | ~1.5M | generated with Phi-4-mini-instruct (Q4_K_M), local | dedup: exact + MinHash + 5-gram span pass |
| Project Gutenberg children's shelf | ~702M | public domain | 3,490 books, normalize + dedup + content policy |
Corpus total 753.7M chars; 95/5 split: train 716.0M, validation 37.7M Gutenberg-only (val never contains synthetic/TinyStories data, by design).
Preprocessing: encoding normalization (CP1252/mojibake fixes, NFKC, accent folding,
smart-punct -> ASCII), whitespace glyph encoding (ฤ =space ฤ=newline), block wrapping
ฤ...ฤ, exact + fuzzy + span deduplication, and a documented content policy for era
texts (see Ethical Considerations).
Training Procedure
| Hyperparameter | Value |
|---|---|
| Batch / sequence | 40 x 4,096 chars (163,840 chars/step) |
| Optimizer | AdamW, lr 3e-4, warmup 1,000, cosine to 3e-5 |
| Gradient clipping | 1.0 |
| Truncated BPTT | 512 (state carried + detached between chunks) |
| Epoch handling | stateful tape walk; states reset at epoch boundary |
| Early stopping | patience 10 val evals (val every 250 steps) |
| Checkpointing | every 1,000 steps + best-val copy |
Hardware: single NVIDIA GTX 1660 SUPER (6 GB, no tensor cores), Ryzen 5 2600X. ~23k chars/s, ~8.7 h/epoch. Two runs exist:
- gutenberg_baseline: no curriculum prefix (checkpoints archived in
chkpt/gutenberg_baseline/). - curriculum: TinyStories prefix (this card's active run, log at
logs/curriculum_run.log).
Evaluation & Metrics
Metric: next-character cross-entropy (nats/char) on the Gutenberg-only holdout. In-training numbers (not final):
| Step | Baseline val | Curriculum val | NET2NET 2*Layers |
|---|---|---|---|
| 250 | 3.382 | 3.247 | - |
| 500 | 2.990 | 2.707 | - |
| 750 | ~2.7 | 2.562 | - |
| 1000 | ~2.6 | 2.434 | - |
| 1250 | ~2.5 | ~2.35 | TBA |
Mount Corpus-Floor (settled bet, 2026-09-08): Notio1@24k (val 0.679 on TinyStories)
evaluated cross-corpus on this Gutenberg holdout scores 1.836 nats/char (2.05M-token
measurement, Notio/src/eval_cross.py). Conclusion: Gutenberg's character-level floor is
0.9-1.2 nats above TinyStories'. Expected val floor for this model: **1.5-1.7 nats/char**.
Qualitative evaluation: temperature-0.9 samples at 500-step intervals, reviewed for structure (sentence rhythm, punctuation, dialogue) rather than perfect spelling โ the latter is provably below the corpus floor for rare words.
Known Limitations & Failure Modes
- Hedged spelling: rare words will be misspelled/neologized; this is a measured property of the corpus entropy, not a training deficiency.
- Template repetition: the synthetic subset can reuse formulaic openers ('Once upon a time' style); a span-level dedup pass mitigates but does not eliminate it.
- No safety training: outputs are pure next-character modeling. Content reflects the training distribution (see below).
- Context: 4,096-char attention/position budget; generation capped accordingly.
- Degenerate-loop history: earlier subword-vocab versions collapsed into repetition loops ('soup basin'); the char-only vocabulary was adopted specifically to prevent it.
Ethical & Bias Considerations
- Gutenberg content policy Whole-book quarantine of racial-trope/self-harm/heavy-slur categories; clear slurs replaced with era-consistent terms; kept-with-documentation: era-appropriate race/slavery/violence themes, mild oaths, ableist-era language. Residual missed era-biased content may surface in outputs.
- Synthetic data: generated by Phi-4-mini-instruct (children's-story system prompt). An explicit-content scan (403-term list, 2.3M chars) found zero true positives (7 flagged terms, all benign contexts).
- TinyStories: synthetic children's stories, no known harmful categories, formulaic by design.
License
- Weights: TBD (developer decision pending final release).
- TinyStories: CDLA-Permissive-2.0. Project Gutenberg: public domain. Synthetic subset: generated locally with Phi-4-mini-instruct; redistribution of the generated text is at the developer's discretion.
- Code: internal research codebase (Notio2 repository).
Card generated 2026-09-08. Training ongoing; update metrics at the run's conclusion.