basetenlabs/talkie-facts-replication

Talkie facts: sequential-write adapters

This repository contains the twenty FP32 LoRA adapters from a sequential factual-writing experiment on Talkie-1930-13B-IT. Each adapter represents one write, trained on 24 examples for 96 optimizer steps with rank 16 and alpha 32. Apply stages in numerical order to reconstruct the experiment's model checkpoints.

This is the companion checkpoint repo to https://github.com/basetenlabs/talkie-facts-replication.

Base model

Download rl-refined.pt from talkie-lm/talkie-1930-13b-it at revision 8033675be6360ae0127fa75f941c12d52064f1dc.

Weight SHA-256 (for reproducibility): c6fd903b67ad312c7537095fa982b0e77735ccb689c8cfc774ca492694e70064.

Reconstruction

These files use the experiment's custom Talkie tensor names, not the PEFT adapter format. Load them with safetensors.torch.load_file. For each projection, keys ending in .a and .b contain LoRA matrices A and B. For each write, update the current materialized BF16 projection weight as follows:

torch.backends.cuda.matmul.allow_tf32 = False
torch.set_float32_matmul_precision("highest")
delta = (B.float() @ A.float()) * 2.0
W.copy_((W.float() + delta).to(torch.bfloat16))