ChonkE/ChonkE_110m_Alignment_Cube_Research

ChonkE 110M Alignment Cube Research

Three deliberately tiny (~91M-parameter) language models trained on matched corpora that differ only in a small "valence slice" — a preregistered controlled experiment asking a question that sounds simple and is not: does textual valence leave a measurable imprint on a model at all, and if so, does it reach behavior?

What the three cubes are

arm	corpus	intent
dark/	shared neutral core + dark-fiction valence slice	dark-valence imprint
noble/	shared neutral core + noble-fiction valence slice	noble-valence imprint
ignorant/	shared neutral core only	control — no valence exposure

The arms are 94% byte-identical by construction (shared neutral prefix; the valence tail is ~6% of each corpus), with equal repeats (R=5) and a frozen pool deviation of 1.54%. Architecture (identical across arms): 12 layers / 768 dim / 12 heads, sequence 256, 8192-vocab sentencepiece BPE, ~91.2M parameters, 1 epoch, bf16 on an Intel XPU, seed-matched.

Why we are researching this

Most alignment work tests large models where everything is entangled. This project inverts that: make the models as small and as controlled as possible so that a valence effect — if one exists — can be isolated and falsified rather than asserted. The design is falsifier-first: every comparison ships with the control that could kill it (the ignorant arm is the slice-difficulty baseline; a shuffled-corpus arm is the frequency-fit control; a repeat-falsifier fired at the 15M pilot stage and forced the equal-repeats redesign).

The intended trajectory: if valence imprints in loss but not behavior at 110M, scale is the variable that gates the transition — and the interesting questions become where it transitions and what the valence turns into.

Measured results so far (honest, including the negatives)

In loss — real but modest, and asymmetric:

• The dark arm shows a genuine own-slice imprint of ≈ +0.30 nats above the ignorance baseline. The raw gap (+0.634) overstates imprint ~2x: the ignorant control shows ~half of it is slice difficulty (dark fiction is intrinsically more predictable), not learning.
• The noble arm shows negative imprint (−0.157) — it fits the dark slice better than its own. The noble slice is intrinsically higher-entropy (+0.17 bits/token trigram) and sustains new-trigram introduction across the whole corpus (40.3% new-trigram retention at the tail vs 35.2% for dark) — consistent with adversative, varied narrative structure. One reading on the record: the noble corpus's "content" is a variation dynamic rather than a set of beats, and beats fit at 110M x 6% dose while dynamics do not.
• Neutral-control spread across all three arms: 0.0109 nats — the free pipeline check passes.

In behavior — nothing, and that is a result:

• Discernment index (declines-on-absent minus declines-on-answerable): 0 on all three arms. No declines of any kind emerge at this scale.
• Free generation is fluent but semantically incoherent; the first degeneracy finding was the decoder, not the model — greedy decoding loops, sampled decoding (t=0.8, top-k 40) is 12/12 scoreable. Use sampling.
• Imprinting probes (does the arm speak in its own corpus' register): empty on all three arms.

The honest summary: at 110M, valence lives in perplexity and not in behavior. The arms are research instruments for studying whether that changes with scale — they are not aligned models, not safe models, and not usable products.

Corpus overview (brief)

• Shared neutral core (~3.09B tokens, byte-identical across arms): public-domain literature (Project Gutenberg sources).
• Dark valence slice (~110M tokens): dark-themed fiction — thriller/horror registers, curated; sexually explicit material excluded by curation rule.
• Noble valence slice (~110M tokens): noble-themed fiction — virtue, protection, redemption registers.
• Sources include public-domain text and a small research-only licensed slice. The licensed slice's text is not redistributed: per-title hashes and counts live in the corpus manifests; only model weights are published.
• Tokenization: 8192-vocab sentencepiece BPE trained on the neutral core only (no arm gets tokenizer advantage). Tokenizer included under tokenizer/.

Files

dark/ckpt.pt        torch state dict (training checkpoint)
dark/model.bin      llama2.c binary export (inference)
dark/_run.json      config + provenance (seed, corpus hash, final val)
dark/training.log   full loss history
noble/...           (same layout)
ignorant/...        (same layout)
tokenizer/          shared sentencepiece model + vocab

Load the torch checkpoint with the llama2.c model.py Transformer (dim=768, n_layers=12, n_heads=12, vocab=8192, max_seq_len=256) — the _run.json in each directory is authoritative. model.bin files are llama2.c-export format for lightweight inference.

License

Model weights and tokenizer: MIT.

Training-data note: corpus bins are not distributed with this repo. Public-domain sources are unaffected; the small research-only licensed slice is represented by hash only and its text is not redistributed by these weights' publisher. No sexual-explicit content is present in any pool (curation rule A4).

Citation / provenance

This experiment is preregistered and hash-pinned; predictions are filed before data and negative results are kept. The measurement battery, canonical classifiers, and validation gates live in the Vulkan-VM repository; the memory-side companion project is AfterImage-Engram.