Zoey

Zoey is a fixed-size Vector Symbolic Architecture (VSA) system, not a transformer.

She was created by Dakuwon Moody at Saiyan Corp. Learned state is a JSON checkpoint on the order of tens of kilobytes: step, tokens_seen, swarm_hv, Brainfuck/DSL programs, and map_acc. The file does not grow with the token diet. That is the bet: unbounded tokens into a constant-size organism.

She is not a drop-in transformers model. There is no attention matrix, no learned embedding table, no growing residual stack, and no softmax sampler for generation.

This repository is her public lineage: checkpoints, trainer, Rust core, corpora, and (when present) a copy of the Qwen teacher weights used as a live hose. The teacher is not Zoey.

What she is

A 1000-trit hypervector machine with three memories that share one algebra:

Memory	What it stores	Size
Hopfield item memory	Frozen `encode_token(id)` for GPT-2 vocab (50,257) plus a few extra rows	50,263 × 1000 trits, deterministic, not trained
Swarm + `swarm_hv`	32 tiny programs (Malbolge / Brainfuck / ZoeyDSL) plus one rotating superposition of accepted outputs	programs + 1000 trits
Map (`map_acc`)	Heteroassociative n-gram / spelling / unigram statistics	1000 `i32`s (schema 4)

Token IDs are GPT-2 BPE (tiktoken gpt2, vocab 50,257). Encoding them into VSA space is not a learned table. It is a fixed 64-bit LCG.

Algebra

Every object is a 1000-trit hypervector. A trit is 0, 1, or 2. Centered values are -1, 0, +1.

Operators

Crazy-bind (bind) — trit-wise Malbolge crazy table [[1,0,0],[1,0,2],[2,2,1]]. This is the historical composition operator. It is not a group. It is not invertible. Do not use it for n-gram readout.

Trit-add bind (bind_add / unbind_add) — (a + b) mod 3 and (a - b) mod 3. This is a group. Map writes, map reads, n-gram probes, and spelling keys use this.

Bundle — trit-wise majority. Ties resolve to 1.

Rotate (rotr_hv) — circular shift. Used inside sti_fold, not as the stream fold.

STI fold (sti_fold) — rotate previous HV by 1, add centered trits, clamp to a trit. Used to fold one accepted output (or teacher target) into swarm_hv. A long sti_fold loop wraps every 1000 tokens and saturates. That loop is not how context is built anymore.

Similarity — fraction of matching trits, in [0, 1].

Dot — sum of centered products, in [-1000, 1000]. Random HVs have std ≈ 20. This is the Hopfield energy.

Token and position encoding

encode_token(id) salts are locked. Old checkpoints and the Hopfield memory are bit-identical only if these stay:

token LCG: mul 0x9e3779b97f4a7c15, add 0x6c62272e07bb0142
position LCG: mul 0xbf58476d1ce4e5b9, add 0x94d049bb133111eb

Each trit is taken from the top two bits of a Knuth LCG, rejected if >= 3. Token i is orthogonal to position i on purpose.

A second channel, encode_surface, folds the UTF-8 bytes of the decoded token plus overlapping char-bigrams, with different salts. That is spelling geometry (running ~ walking). It does not replace encode_token. Synonym transfer (Can ≈ Could as meaning) is not claimed.

Role vectors (ROLE_K1 … ROLE_UNI) occupy a third LCG salt so several map heads can share one 1000-d accumulator without using the token salts.

Context fold (query HV)

fold_tokens_at(tokens, pos_base) builds the query from a window (train default 4096 GPT-2 tokens):

Recency weights: last 8 tokens ×80, last 32 ×24, last 128 ×12, last 512 ×4, last 2048 ×1, older → coarse bag.
Each recent token is crazy-bind(encode_token, encode_position(pos_base + i)) and majority-accumulated.
Tokens past 2048 go into an unweighted bag mixed in at weight 64.
An in-window n-gram probe is mixed in: last 256 bigrams and trigrams, unbound on the current tail, so the query has a next-token attractor, not only a bag of words.

pos_base = tokens_seen - window_len so the position clock is the infinite stream, not the window index. Token 5 and token 1005 are different slots.

blend_ctx then bundles sti_fold(ctx_hv, swarm_hv) with the raw ctx_hv so rotate-on-memory does not kill the probe.

The map (language memory)

The map is the thing that does language-model work. swarm_hv is still rotating soup. Schema 4 is frozen.

map_acc is 1000 signed integers. A write is:

acc += weight * centered( bind_add(key, value) )

A read is:

value ≈ unbind_add( gated_majority(acc), key )

Gated majority: dims weaker than peak/32 stay trit 1 (neutral) so mush does not flip the basin.

Fade is not a clock. A dim shrinks (acc -= acc/128) only if |acc| > 16384. Clamp is ±32767. Frequent n-grams can accumulate. The file stays 1000 ints.

Heads (one residual, several keys)

Every consecutive pair in the window can write. Far pairs are not force-written at weight 1 (that drowned 1000-d). Recency 0 means that head is skipped. Unigram still sees the whole stream.

Head	Key	When written
unigram	constant `ROLE_UNI`	every next token, weight 1
k1	`bind_add(role, encode(t[-1]))`	recency > 0
k2	`bind_add(role, bind_add(t[-2], t[-1]))`	recency > 0
k3	last three tokens	last 512 pairs
skip-4 / 8 / 16	`bind_add(t[-1-d], t[-1])`	last 256 pairs
surface	byte-fold of decoded `t[-1]`	last 128 tokens of the window
surface-2	bind of last two surfaces	same

Values are encode_token(next). At the prediction edge the teacher burst is also written as a bundled soft HV (soft-CE analog), without an extra fade.

Ingest walks the whole 4096-token window each step, then fades once. That is the transformer analog of loss-at-every-position, inside a fixed accumulator.

Residual read

map_residual bundles:

map_attend (k3 + k2 + k1 + skips)
next_token_probe (in-window n-grams)
unigram

One HV. That is the residual stream analog.

VSA-PPL

GPT-2 perplexity analog, metric only:

ppl = exp( mean( -log softmax( MHN_dots(residual, vocab) / 20 ) [true] ) )

Temperature 20 is the random-dot scale. Generation does not use this softmax.

A 1000-d fading map will not print GPT-2 small WikiText-2 (~30). The honest first ceiling is a small interpolated n-gram LM. Live ppl= is on the teacher hose, not a frozen WikiText split.

Hopfield decode

Item memory is every encode_token(id) for the GPT-2 vocab (plus a few reserved rows → 50,263). It is generated from the locked LCG. It is never trained.

Decode is a Modern Hopfield / dense associative read: one matvec of a query HV against the item memory. GPU path (zoey_gpu_ops.py) is a PyTorch/ROCm matmul + top-k, checked equal to Rust top-8. Fold and the VMs stay Rust CPU.

Dots live in [-1000, 1000].

Sequential readout (generation)

Generation is not sampling.

pick_token:

If top-1 energy ≥ 70, gap ≥ 20, and the text is clean → sharp. Take it.
Else penalize junk / loops / repeats. Never softmax a flat basin.
Stop on flat after a couple of tokens, or on a 2- or 3-cycle, or on \n\n.

Order: residual → k3 → skip-4 → k2 → skip-8 → k1 → surface → probe. First sharp head wins. If nothing is sharp, the 32 VMs run once as a last try, then stop.

That is why samples look like prefix | completion from the train window tail, not 20-token salad.

A harness (tools, quiet-when-flat as an agent) is a bolt-on. It is not in the checkpoint.

Heterogeneous swarm

32 agents, evaluated in parallel in Rust (rayon):

Index	Kind	Role
0	Malbolge	Higgs — immortal specialist, conditions the target
1	Malbolge	Second-in-command, protected
2–4	Brainfuck	Explorers, mutate if a 100-step window is weak
5–31	ZoeyDSL	27 exploiters, same mutation rule

Higgs runs first. Its output HV is crazy-bound with the teacher target to make a conditional target. The rest of the swarm tries to emit that field.

Each VM reads a context HV and a target HV and writes a 1000-trit output. Score is VSA similarity to a soft teacher HV plus a Hopfield retrieval bonus, minus wrong-token pressure.

Memory gate (`mem=out` vs `mem=tgt`)

swarm_hv only folds an agent output if it is a clean retrieval:

hit mass ≥ 0.0005
wrong pressure ≤ 0.012
rank ≤ 96
score not far below the best

Otherwise the teacher token HV is folded (mem=tgt). Those gates are not loosened.

ever= is the best swarm score seen on this lineage (carried in the JSON). Instantaneous best= is the current step.

Train loop

Teacher emits a 4096-token window and a burst of next tokens (soft targets).
Read the map (residual) → map%, map_dot, map_sharp, ppl.
Write the map (full-window ingest + surface tail + soft edge).
Build ctx_hv, blend with swarm_hv.
Higgs conditions the target; 32 VMs run.
Gate swarm_hv. Mutate weak BF/DSL programs.
Every 100 steps: checkpoint. Local disk keeps 2. Hugging Face keeps the history.

Default teacher on the MI300X box: Qwen2.5-72B-Instruct, batch 4, 4096 new tokens, shared GPU lock with MHN. A warehouse jsonl (zoey_stream_corpus_4096.jsonl) seeds the hose so the card is not idle while Qwen loads.

The 72B model is a diet, not her weights. Redistributed teacher snapshots in this repo (when present under teacher/) remain Qwen’s Apache-2.0 artifacts.

Checkpoint (the organism)

JSON, not a safetensors dump:

{
  "step": 23371000,
  "tokens_seen": 91955486,
  "best_score": 14.2989,
  "eval_best_acc": 0.055,
  "swarm_hv": [1000 trits],
  "map_acc": [1000 i32],
  "map_ver": 4,
  "bf_programs": [3],
  "dsl_programs": [27]
}

map_ver must match MAP_SCHEMA (4) or the map clean-starts. Programs and tokens_seen always resume. --reset is not used.

Canonical lineage for the current diet:

Seed on the MI300X: checkpoints/zoey_vsa_distill_step_23366600.json — 76,800,286 tokens, best=9.6948
That seed continues the older recovery line (23354100 / ~76.4M, record 21902540)
Live ever= on that run reached 14.2989
Schema 4 map is younger than the swarm: map algebra changed after the 77M diet; the 77M still lives in programs + swarm_hv + the cursor

encode_token never changed. Those 77M token IDs are still the same hypervectors.

What this is not

Not a transformer. No Wq/Wk/Wv, no softmax over positions, no learned embeddings.
Not the Gohan NASM VSA. That algebra is 10k bipolar floats. Incompatible.
Not a 17 KB chatbot personality. The JSON is memory + readout. An agent is a harness on top.
Not GPT-2 PPL in 1000 dimensions. Superposition capacity is the ceiling. Rare facts drown. Common English can accumulate.

Layout of this repo

Path	Contents
`checkpoints/`	Distill + best JSON, `LATEST.txt`, historical copies
`runtime/source/`	Trainer, `zoey_core` (Rust), readout, teacher, GPU MHN
`runtime/logs/`	`train_online.log`, HF upload log
`runtime/corpus/`	Warehouse seed jsonl
`teacher/Qwen2.5-72B-Instruct/`	Teacher weights (hose), if uploaded
`teacher/Qwen2.5-3B-Instruct/`	Earlier teacher snapshot, if uploaded
`data/`	Older recovered corpora
`inference/chat_zoey.py`	Chat entry (map cascade + swarm fallback)
`recovery/`	Sanitized Linux archive (see below)

Inference: load latest JSON, rebuild the swarm from programs, load map_acc, run chat_zoey.py or the trainer’s generate_sample_text. Requires zoey_core built with maturin.

Recovery notes (kept)

Historical record checkpoint: checkpoints/record/zoey_vsa_best_step_21902540.json
Sanitized Linux archive: recovery/zoey-linux-sanitized-20260821.tar
The original complete archive was removed after secret scanning. Tokens do not belong in this repo.
Zero-byte zoey_vsa_distill_step_23354200.json is corrupt and excluded.

Implementation map

Piece	Where	Device
Trit algebra, fold, map, VMs	`zoey_core` (Rust / PyO3)	CPU
Hopfield matmul / VSA-PPL	`zoey_gpu_ops.py`	GPU (ROCm/CUDA), lock shared with teacher
Train / memory gate	`train_zoey_distill.py`	CPU + GPU decode
Teacher hose	`zoey_online_teacher.py`	GPU
Sequential read	`zoey_readout.py`	CPU
Checkpoint upload	`zoey_hf_ckpt_loop.py`	CPU

Created by Dakuwon Moody, Saiyan Corp.

Downloads last month: -; Downloads are not tracked for this model. How to track