ArcherL/phi3-medium-jacobian-lens

Jacobian lens for Phi-3-medium-128k-instruct

One [5120, 5120] matrix per layer (layers 0 to 38, target layer 39) that carries a residual-stream vector into the final-layer basis so the model's own unembedding can read it. Fitted with Anthropic's reference implementation, anthropics/jacobian-lens (jlens.fit, commit 581d398613e5), on 500 WikiText-103 passages of 128 tokens, plain mean, no BOS token in the fitting passages (this model is never served with one). Layout follows neuronpedia/jacobian-lens.

Files

File	What
phi3-medium-128k-instruct/jlens/Salesforce-wikitext/Phi-3-medium-128k-instruct_jacobian_lens.pt	the lens, fp16, 2.04 GB, sha256 81dd5c7a40786b9867c4ad8235b4cbe1b9ebdf76978d1b208f2ebe50f4b7081f
.../config.yaml	every fit setting and the exact commands
.../Phi-3-medium-128k-instruct_convergence.csv	per-passage diagnostics from the fit
.../manifest.json	full provenance, cross-half agreement per layer
.../eval-summary.json	readability on the six upstream lens-eval sets, J-lens beside the logit lens

Use

import torch, transformers, jlens
hf = transformers.AutoModelForCausalLM.from_pretrained("microsoft/Phi-3-medium-128k-instruct", dtype=torch.bfloat16).cuda()
tok = transformers.AutoTokenizer.from_pretrained("microsoft/Phi-3-medium-128k-instruct")
model = jlens.from_hf(hf, tok, force_bos=False)
lens = jlens.JacobianLens.from_pretrained("ArcherL/phi3-medium-jacobian-lens", filename="phi3-medium-128k-instruct/jlens/Salesforce-wikitext/Phi-3-medium-128k-instruct_jacobian_lens.pt")
lens_logits, model_logits, _ = lens.apply(model, "Fact: the currency used in Japan is the", positions=[-1])

This model's config.json predates transformers 5; modal/hfload.py in the source repository shows the one RoPE key to copy in memory so it loads on transformers >= 5.5, which jlens requires.

Quality

Cross-half agreement (even vs odd fitting parts, relative difference, lower is better): layer 24 0.125, layer 32 0.063, layer 38 0.016. Readable band on the upstream lens-eval sets: from about layer 26 to 38. Min-over-layers pass@5, J-lens (logit lens in brackets): multihop 0.38 (0.34), multilingual 0.31 (0.32), order-ops 0.45 (0.16), typo 0.18 (0.22), poetry 0.06 (0.04), association 0.02 (0.02). Neuronpedia's Qwen3-14B lens scores in the same range on the same harness. Context depth (0 to 3,300 tokens) does not change the curves.

Caveats

• Per-passage Jacobian norms are heavy-tailed (median 13, three passages above 350, in units of sqrt(d)); nothing was trimmed.
• 73 of 500 passages have no logged diagnostics (their parts resumed from a checkpoint).
• Eval scoring conventions (readout position, synonym sets, first-token scoring of multi-token targets) are documented in the source repository; compare these numbers between lenses on that harness, not to the paper's tables.

License

Lens weights and metadata: MIT. The base model is MIT (Microsoft); the fitting code, anthropics/jacobian-lens, is Apache-2.0.