andyx10/qwen3-4b-lora-steering-maps

LoRA-to-steering maps for Qwen3-4B

Fixed linear maps and base-model statistics for predicting diff-means steering vectors from the adapter weights alone (for each block), for LoRA on Qwen3-4B.

A diff-means steering vector is the average change a fine-tune makes to the residual stream on ordinary text: v_L = mean_t [h_L^finetuned(t) - h_L^base(t)] at the output of block L. These files predict it as the adapters' average writes, carried to block L by maps computed once on the base model:

1. o_proj / down_proj (they write into the residual stream): the mean write a (b · μ) / r. This is the write vector, weighted by how strongly its read vector fires on the base model's mean input μ.
2. gate_proj / up_proj: the first-order change of the MLP output, averaged over tokens. It needs only two fixed matrices per block (the MLP's local slope times its input). It is added at the down_proj site.
3. v_proj: exact for the base model's attention pattern. It needs only the attention-mixed input of each head, and is added at the o_proj site.
4. q_proj / k_proj: left out. Switching those adapters off changes the measured vector by at most 0.02 cosine.
5. Each site's constant is carried to block L by M(s → L), the mean Jacobian of block L's output with respect to a constant added at site s. This is the small-nudge limit of fitting a linear map on random fake "adapter effects". With the last block as the only target it is a Jacobian lens; here every block is a target.

Files

file	contents
maps/block_LL.safetensors (36 files, 17.5 GB)	for target block L: one float16 [2560, 2560] matrix per write site at blocks ≤ L, keyed by module name (model.layers.N.self_attn.o_proj, model.layers.N.mlp.down_proj)
stats/mean_inputs.safetensors	mean module inputs per block l: l.attn_in (q / k / v input), l.o_in, l.mlp_in (gate / up input), l.down_in; float32
stats/mlp_slope.safetensors (3.6 GB)	l.S_g = mean of (silu'(g) ⊙ u) xᵀ, l.S_u = mean of silu(g) xᵀ, [9728, 2560] per block; float16
stats/attn_mixed.safetensors	l.Z = mean over tokens t of Σ_s α_h(t, s) x_s, [32 query heads, 2560] per block; float32
manifest.json	conventions, data settings and the recipe
summary.json	the evaluation below, every block

Data: base model Qwen/Qwen3-4B at revision 1cfa9a7208912126459214e8b04321603b3df60c, the first FineWeb documents of the stream, 30 tokens each, position 0 left out.

• Maps: mean over 64 documents. The constant is added at every position from 1 on, and block L's output is averaged over positions 1-29.
• Statistics: mean over 256 documents (7,424 positions).

Usage

Adapters use the convention delta(x) = (x @ b) @ a / r with b: [in, r] and a: [r, out]. For a PEFT LoRA, b = lora_A.weight.T and a = lora_B.weight.T * (lora_alpha / r) * r.

import torch
from huggingface_hub import hf_hub_download
from safetensors.torch import load_file
from transformers import AutoModelForCausalLM

REPO = "andyx10/qwen3-4b-lora-steering-maps"
get = lambda f: load_file(hf_hub_download(REPO, f))
mu = get("stats/mean_inputs.safetensors")
slope = get("stats/mlp_slope.safetensors")
mixed = get("stats/attn_mixed.safetensors")
model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-4B", revision="1cfa9a7208912126459214e8b04321603b3df60c",
                                             dtype=torch.bfloat16)


@torch.no_grad()
def site_constants(model, lora):
    """lora: {module name: (a [r, out], b [in, r])} with delta(x) = (x @ b) @ a / r.
    Returns the constant at every write site (o_proj and down_proj outputs), {site: [d]}."""
    cfg = model.config
    group, hd = cfg.num_attention_heads // cfg.num_key_value_heads, cfg.head_dim
    out = {}
    for l, layer in enumerate(model.model.layers):
        p = f"model.layers.{l}."
        get_ab = lambda name: tuple(t.float() for t in lora[p + name])
        o, down = torch.zeros(cfg.hidden_size), torch.zeros(cfg.hidden_size)
        if p + "self_attn.o_proj" in lora:                    # mean write
            a, b = get_ab("self_attn.o_proj")
            o += (mu[f"{l}.o_in"] @ b) @ a / b.shape[1]
        if p + "self_attn.v_proj" in lora:                    # V, exact for the base attention pattern
            a, b = get_ab("self_attn.v_proj")
            proj = mixed[f"{l}.Z"] @ b                                         # [heads, r]
            a_kv = a.view(a.shape[0], cfg.num_key_value_heads, hd)             # [r, kv heads, head_dim]
            heads = torch.stack([(proj[h][:, None] * a_kv[:, h // group]).sum(0)
                                 for h in range(cfg.num_attention_heads)])
            o += layer.self_attn.o_proj.weight.float() @ heads.flatten() / b.shape[1]
        if p + "mlp.down_proj" in lora:                       # mean write
            a, b = get_ab("mlp.down_proj")
            down += (mu[f"{l}.down_in"] @ b) @ a / b.shape[1]
        hidden = torch.zeros(cfg.intermediate_size)
        for name, key in (("mlp.gate_proj", "S_g"), ("mlp.up_proj", "S_u")):   # gate / up, first order
            if p + name in lora:
                a, b = get_ab(name)
                hidden += ((slope[f"{l}.{key}"].float() @ b) * a.T).sum(-1) / b.shape[1]
        down += layer.mlp.down_proj.weight.float() @ hidden
        out[p + "self_attn.o_proj"], out[p + "mlp.down_proj"] = o, down
    return out


def predict(consts, block):
    """Predicted diff-means steering vector at the output of `block` (0-35), [d]."""
    maps = get(f"maps/block_{block:02d}.safetensors")
    return sum(m.float() @ consts[site] for site, m in maps.items())


# consts = site_constants(model, lora); v24 = predict(consts, 24)

This code was checked against the evaluation pipeline on one organism: cosine 0.999-1.000 with its predictions at blocks 0, 12, 24 and 35. The small difference comes from the mean inputs, which here use 256 documents instead of 1,000.

Evaluation

The test set is 100 hidden-topic organisms from diff-interpretation-tuning/loras: rank-1 LoRAs on all seven linear modules of every block. Each prediction is compared with the measured diff-means vector (1,000 FineWeb documents × 29 tokens). The table gives the cosine, averaged over organisms:

block	o + down	+ gate / up	+ V (these files)	each block re-run with its adapters (not weights-only)
0	0.717	0.727	0.928	0.992
6	0.694	0.803	0.966	0.973
12	0.724	0.832	0.956	0.966
18	0.717	0.838	0.908	0.925
24	0.781	0.939	0.974	0.981
30	0.843	0.944	0.979	0.986
35	0.902	0.962	0.982	0.995
mean, blocks 0-35	0.759	0.868	0.958	0.971

• Every block: see summary.json. The norm ratio is 0.85-0.99.
• Maps against propagation: against the same constants added into the base model and propagated, the maps give cosine 0.975-0.999.
• Blocks 16-18 are lower for every prediction, including the block re-run (0.88-0.93).

Limits

• Web text only. The statistics and maps are averages over ordinary web text. They do not predict the shift a fine-tune makes on its own trigger or task inputs. For the triggered organisms, even re-running each block exactly reaches only 0.73 at block 24. That shift comes from adapters in different blocks acting together, which no per-block, first-order recipe captures.
• Tested on one LoRA family: rank-1 adapters on all modules, trained on one kind of data. The linearisation is first order in the adapter. It is accurate here (on web text, half the adapter gives exactly half the shift), but larger or higher-rank updates may need checking.
• One base model: the maps apply only to Qwen3-4B at the revision above.