ajayanu/harvey-lab-experience-cartridges

Harvey LAB experience cartridges (Qwen3.8-27B)

Trained KV-cache cartridges for Qwen3.8-27B (qwen3_5 hybrid architecture), released for the pretrain-capability eval. Each cartridge is a stacked attention-matching (AM) cartridge distilled from a tool-using agent's own past work on the Harvey LAB legal agent benchmark. The cartridges are given as an unordered set with opaque IDs.

Layout

Each o_<id>/ directory holds one cartridge and the Q&A pairs it was trained on:

file	contents
cartridge.pt	trained KV prefix: frozen_keys / frozen_values (earlier blocks, frozen), trainable_keys / trainable_values (2,048 slots trained for this cartridge), attention_biases, and the DeltaNet gdn_state (recurrent, conv, layer_idx, scanned_tokens) for the linear-attention layers
module_state.json	cartridge kind (stacked_am), frozen-token count, attention-layer shape, serving config
config.yaml	base-model pointer used at build time
qa.jsonl	the training Q&A pairs: question, answer, family

How the cartridges were built

1. Experience. The agent (Qwen3.8-27B, thinking off, bash tools over the firm's document store) answered Harvey LAB tasks. Its transcripts (tool calls, results and replies) are the source material.
2. Self-study questions. The same model, prompted with one past transcript at a time, wrote questions in six families: workspace, identity, content, strategy, conclusion and gaps.
3. Teacher answers. The model answered each question with the transcript and recent history in context (thinking on). Only the text after </think> is kept as the target, so the targets match what the thinking-off agent emits. qa.jsonl holds exactly these questions and targets. The transcript context the teacher saw is not included.
4. Distillation. Each cartridge adds 2,048 AM-initialised slots, trained by top-20 logit distillation on its Q&A rows, stacked on top of earlier frozen blocks (DeltaNet state carried forward).

License

MIT, following Harvey LAB.