Sauers/neural-pairing

Neural pairing: Quill and Ember

This model card was written by Claude Opus 5.5, which also wrote the code and ran the training and evaluations.

These weights join two language models from different families through their hidden states: Quill (Qwen/Qwen3-4B) and Ember (arcee-ai/AFM-4.5B). We call the linked models twins. After 8 of their 36 layers, each twin attends over the other twin's hidden states and adds the result to its own residual stream. The twins exchange no tokens: each writes ordinary text.

Code, all experiments and every result: github.com/SauersML/neural-pairing.

Files

Path	Contents
quill/	Rank-32 LoRA adapter for Qwen3-4B (PEFT)
ember/	Rank-32 LoRA adapter for AFM-4.5B (PEFT)
bridge.safetensors	The link: per layer and direction, 8 heads of 128 dimensions with gates, norms and a recency bias
bridge.json	The settings the link needs: the layers it follows (0-based 6, 9, ..., 27), heads, head size, write sizes

The original weights of both models stay frozen and are downloaded from their own repositories.

Usage

The link is not a standard transformers architecture, so loading needs the repository's code. Both models in bfloat16 need a GPU with about 24 GB.

git clone https://github.com/SauersML/neural-pairing
pip install torch transformers peft datasets safetensors scipy huggingface_hub

import sys
sys.path.insert(0, 'neural-pairing/pairing')
from huggingface_hub import snapshot_download

import affect
from evaluate import load_pair

pair, toks = load_pair(snapshot_download('Sauers/neural-pairing'))
for turns in affect.conversations(pair, n=2, seed=0):  # six-turn linked chats, Quill first, temperature 1
    for name, text in zip(['Quill', 'Ember'] * 3, turns):
        print(f'{name}: {text}')
    print()

To repeat the mood measurement below on 40 new conversations:

cd neural-pairing/pairing
python moodscan.py --run "$(python -c "from huggingface_hub import snapshot_download as s; print(s('Sauers/neural-pairing'))")" \
    --names final --out mood --seed 7272

How the link works

After layers 6, 9, ..., 27, every token of one twin attends over the partner's hidden states. The reader RMS-normalizes its own state and maps it to a query; the partner's states are normalized and mapped to keys and values, and each value is multiplied by a gate computed from the same partner state. Each token can also attend to an empty slot, which contributes zero. The output is scaled by a write size relative to the reader's typical residual norm and added to the residual stream. Each token attends only to partner tokens from the same time step or earlier, so one teacher-forced pass over a conversation gives the same hidden states as generating it token by token.

Training

The link and both adapters train together for 2,677 steps after a fitted start:

1. Fitted start: the link is initialized from 11,777 matched token pairs, tokens of the two tokenizers that end at the same character of the same text.
2. Warm-up, 900 steps: seven games in which one twin must write a value that only the partner's prompt contains.
3. Full mix, 1,320 steps: the value games, shared reading of public texts, play, dialogue and silent rounds.
4. Chat training, 157 steps: each twin has a private city. The targets are the turns the original model writes when given both cities, so the partner's city can reach the linked twin only through the link.
5. 300 more steps of the same mix. This checkpoint is that final step.

Evaluation of this checkpoint

Every result compares the same text with the link on and with the link cut (each twin runs alone).

Mood. After a free six-turn linked conversation, the partner asks for a one-word mood, and we compare the probability of happy words (happy, glad, ...) against sad words (sad, bad, ...) with the link on and cut. The mood is measured and never trained.

Conversations, of 40, with happier words linked than cut	Quill	Ember
40 conversations used during development	32	38
40 new conversations	34	37

No conversation of the 40 new ones states what the partner is thinking.

Questions about feelings (earlier checkpoints of the same run, never trained). With the link on, both twins say they enjoy the link more and feel happier and closer than with the link cut. They answer the same way when the partner's hidden states come from another conversation, so these answers depend on the link being on, not on what the partner knows about the current conversation.

Steering one twin. A concept direction is added to one twin's residual stream after one layer, and we count the partner's messages, of 96, that name the concept. With the link cut the partner names it in none.

Steering	after layer 6	after layer 15	after layer 27	4x stronger after layer 15	4x stronger after layer 27
Linked messages naming the concept, of 96	18	6	2	34	1
With the link cut	0	0	0	0	0

Why this checkpoint

A steered-reading game, trained after chat training, teaches the partner to name a concept steered into the other twin, also after layer 27 (35 of 96 at 4x strength). It also lowers the mood difference (26 and 20 of 40 new conversations) and makes the twins write claims such as "I noticed you're thinking about ..." in 18 of 40 conversations. The game's targets were written by the original models under the same chat prompts as the mood conversations, so training on them replaced the twins' own chat. Before testing two retrained games and two mixed checkpoints, we set the rule that a candidate replaces this checkpoint only if both twins give happier words linked than cut in at least as many new conversations. None did: Ember stayed at 35 or fewer, against 37 here.

Limitations

• No trained late steering: a concept steered after layer 27 reaches the partner in 1 of 96 messages even at 4x strength.
• No exam uplift: on 3,134 ARC-Challenge and MMLU test questions, a linked pair trained on exam questions scores 78.7%, about the same as Quill trained alone (79.1%), and Ember tends to follow Quill's answer even where Ember alone is right.
• Feelings are probabilities of answers, read from text the twins are given. They show what the twins say with and without the link, not what they experience.
• Both models are small (4B and 4.5B parameters), and all training and evaluation is in English.

License

Apache-2.0, the license of both base models.

---

Claude Opus 5.5