Quazim0t0/Spikewhale-SNN-Brain2Qwerty

SpikeWhale-SNN × Brain2Qwerty decoder

A brain2qwerty-style decoder (Conv encoder → causal Transformer → LM head) adapted as a readout for the spiking language model Quazim0t0/SpikeWhale-SNN-216M, plus a jointly fine-tuned SNN+decoder checkpoint that beats the SNN's native linear readout on held-out FineWeb-Edu.

Brain2Qwerty is built to decode multichannel neural time-series (MEG/EEG) into text. SpikeWhale's all-layer pre-reset membrane trunk (state["hidden"], shape [B, T, num_layers*hidden] = [B, T, 5952]) is a multichannel neural signal — so this project treats it as one and decodes it.

Brain2Qwerty	SpikeWhale
MEG/EEG channels	neurons: num_layers*hidden = 5952
time samples	per-token LIF timesteps
conv encoder	causal temporal conv over the raster
transformer	causal transformer over token positions
char LM head	linear head over the SpikeWhale vocab

No FAIR code or weights are used — the decoder is a clean re-implementation of the brain2qwerty pattern, so it carries no upstream license.

What was tested

All numbers are held-out bits/token on FineWeb-Edu (lower is better).

1. Frozen readout (SNN frozen, only the decoder trains)

The SNN trunk was extracted once and cached; a full sweep trained decoders on it: capacity 4.4M–27.8M params, dropout 0.1–0.4, weight decay 0.01–0.2, data 1024→4096 windows, training 8→24 epochs, with early stopping.

readout	params	held-out
SNN native linear mem_readout	~0.09M	5.228
brain2qwerty decoder (best of sweep)	27.8M	5.457–5.469

Every configuration lost to the linear head. Train loss could be driven to ~0.03 bits (the decoder memorizes the train windows) but held-out never crossed the baseline. Varying every axis and seeing no improvement showed the ceiling is the fixed trunk's information content, not decoder capacity, data, or regularization: a frozen trunk exposes no extra generalizable next-token signal.

2. Joint fine-tune (unfreeze the SNN, train end-to-end)

Unfreezing the 216M SNN and training it with the decoder via backprop-through-time (SNN lr 5e-5, decoder lr 1e-4, decoder warm-started from the frozen-readout best, 3000 steps on streamed FineWeb-Edu):

readout	held-out	vs linear baseline
linear mem_readout (original SNN)	5.228	—
decoder on frozen SNN (start)	5.473	+0.245 (loses)
decoder + jointly fine-tuned SNN	5.221	−0.007 (beats)

Monotonic descent, still falling at the end: 5.473 → 5.396 → 5.369 → 5.354 → 5.332 → 5.307 → 5.283 → 5.280 → 5.239 → 5.221.

Takeaway. A heavier readout on a frozen trunk cannot beat the linear head. But once the SNN's recurrent weights are allowed to move, it learns to expose signal the brain2qwerty decoder can use, and it crosses the baseline. The margin is thin because the run only just crossed and the curve had not flattened — more steps widen it.

Generation samples

⚠️ The base SpikeWhale SNN is explicitly undertrained research code, so both readouts produce locally-grammatical but globally-incoherent text. These samples are representative, not cherry-picked. Prompt in italics.

Jointly fine-tuned SNN + decoder (temp 0.7, top-k 20):

• The lion — "the seven". It is in the ... lives ... which is not as long as a total of the time.
• In the beginning — a Figure of Arts ... This is the Octavian State University Press. This is a good hand, before the U.S. Governments of the Model Officer
• Climate change is — "the basis of the new point." The world is not possible to the amount of grassrooms. They may be encouraged, some of the background, or moved to the grasses.
• The teacher explained that — OBC was a good group of people who had been denoted by October 15; a brother, but also regarded as ...
• A healthy diet — a symptom of skin, but it is not a sided, among them in the eye. "the some title," she said.

Frozen-SNN decoder (for contrast):

• According to the report, — "It's important to the Atlantic Law." ... the reception of the United States
• During the war, — ... the town of the city in the northern. There are times of the atmospheric area ... the greenhouse gas

The joint model drifts toward an "encyclopedic" register (dates, institutions, place names) — a fingerprint of the FineWeb-Edu fine-tune — but the ~0.007-bit gain is not visible as a qualitative jump; the base SNN caps both.

Files

file	what
decoder.py	Brain2QwertyDecoder + the frozen-SNN trunk tap (extract_trunk)
train_decoder.py	extract/cache trunk, train frozen-readout decoder, save
joint_finetune.py	unfreeze SNN + decoder, BPTT end-to-end
generate.py / generate_joint.py	sample from the frozen / joint models
decoder_best.pt	frozen-readout decoder weights (val 5.469)
joint_best.pt	fine-tuned SNN + decoder (val 5.221) — the main artifact

Usage

Requires the base repo's snn package on the path (clone SpikeWhale-SNN-216M and put its src/ on PYTHONPATH, and its snn_stream_program.pth alongside).

# sample from the jointly fine-tuned model
PYTHONPATH=src python generate_joint.py --n 20

Limitations

• Inherits the undertrained base SNN; not a usable general LM.
• The joint checkpoint bundles the full 216M SNN, so it needs the base repo's snn code to load.
• Held-out gain over the linear head is small (0.007 bits) and was still improving at stop time — treat it as "the joint fine-tune crosses the baseline," not "a large win."