fittingcorpus: fitted Jacobian maps and results

The artifacts for The Fitting Corpus Is Part of the Measurement: Auditing a Jacobian Lens on Pythia (Tarush Gupta, Interpretability as a Science workshop, NeurIPS 2026).

Code, pre-registrations and documentation: github.com/tarush-ai/fittingcorpus

This repository holds the two things the code repository does not: every stored result, and every fitted map. The Jacobian lens reads a mid-layer hidden state through a map J, the model's derivative of its final hidden state with respect to that layer, averaged over a corpus of prompts. The paper measures how much the corpus chosen for that average changes what the lens reads, on Pythia models from 70M to 2.8B parameters.

Contents

path	what it holds
results/*.json	193 results files, one or more per experiment. Every statistic in the paper is computed from these
results/<experiment>/*.json, *.npz	147 per-cell and per-seed records behind them
results/**/*.pt	382 files, 16.2 GB: 380 fitted Jacobian maps and 2 activation caches. 222 maps also carry a _provenance.json sidecar
corpora/	the corpus manifests: per-file SHA-256, document counts, source datasets and revisions, seed-block ranges. No corpus text

The maps, by the experiment that fitted them (IDs from the code repository's docs/experiments/REGISTRY.md):

directory	maps	experiment
results/lenses/e28/	166	E28: the read ladder over fitting corpus and fitting-set size N
results/lenses/plad/	53	E27: maps fitted on two halves of each corpus (between- against within-corpus distance)
results/lenses/ladder/	15	the model ladder, 70M to 2.8B
results/lenses/misc/	8	smoke tests and one-off fits, including the median and trimmed-mean variants of E16
results/e48/	27	E48 and E52: the 24 maps behind the 8 × 8 factorial (eight fitting corpora × three seeds at 410M), and 3 refits on a deduplicated News 2024 corpus
results/cv6/	30	E75: the 1.4B and 2.8B replication (plus two activation caches)
results/ladder1b_b613/	15	E62: the 1B ladder on band [6,13]
results/r6/	8	E72: within-source resampling
results/r7/, results/r7_verified_snapshot/	47	E74: fitting pools matched on type-token ratio
results/e65_lenses/	11	E65: Pythia training checkpoints

Every map is the released reference implementation's corpus average (anthropics/jacobian-lens at 581d398), fitted in fp32 with TF32 off, except the two robust-average variants in misc/, which are named as such.

Usage

The code repository's scripts fetch from here, with no login needed:

git clone https://github.com/tarush-ai/fittingcorpus && cd fittingcorpus
uv sync && bash repro/01_setup_local.sh
bash repro/04_fetch_results.sh            # results/*.json, about 8 MB
bash repro/RUN_ALL.sh --tier T0           # recompute every statistic in the paper
bash repro/03_fetch_artifacts.sh --all    # the maps, 16 GB, hash-verified on download
bash repro/RUN_ALL.sh --tier T1           # re-score every read from the maps, on CPU

Provenance

Every results file carries a provenance block: the producing script and its SHA-256, the git commit, the command line, and the SHA-256 of every input. payload_sha256 hashes the record with that block excluded, so it is unchanged by a rerun that alters only environment metadata. Rerun a result, compare payload_sha256, and a match means no number moved. A map's sidecar records the map's own SHA-256 and the configuration that fitted it.

The code repository's docs/reproducibility/HF_MIRROR_MANIFEST.tsv lists every file here with its size and hash. tools/hf_mirror_manifest.py --verify --repo Tarush-AI/fittingcorpus checks this repository against it.

Licence

CC BY 4.0. Everything here was produced by this work. The models are EleutherAI's Pythia (Apache-2.0) and the fitter is the released Jacobian-lens implementation (Apache-2.0).

No corpus text is distributed. It is third-party text, and the Wikipedia-derived corpora are CC BY-SA, which cannot be relicensed as CC BY 4.0. The code repository rebuilds the pool from its public sources with bash repro/06_data.sh --build and verifies it against the manifests here. See ATTRIBUTION.md.

Citation

@inproceedings{gupta2026fittingcorpus,
  title     = {The Fitting Corpus Is Part of the Measurement: Auditing a Jacobian Lens on Pythia},
  author    = {Gupta, Tarush},
  booktitle = {Interpretability as a Science Workshop at NeurIPS 2026},
  year      = {2026}
}