AlexWortega/bertingpt

Linear operators between language model families

An empirical test of the strong form of the platonic representation hypothesis (Ziyin & Chuang, arXiv:2507.01098) on real language models: Mistral, Qwen2.5, SmolLM2, OLMo, Pythia and GPT-2.

The paper proves that for linear networks SGD drives two models of different architecture to representations that agree up to a rotation. What survives of that in transformers with different tokenizers and different training corpora is the question here.

This repository holds the code, every result, and the trained operators, which can be applied directly without recomputing anything.

Headline results

One matrix maps representations across all layers at once. For the pair Qwen2.5-1.5B → GPT-2 the universal operator reaches R² = 0.437 against 0.556 for separate per-layer matrices; the random-weight control sits at 0.039.

"Up to a rotation" does not hold. An orthogonal map explains 0.22 of the variance where an unrestricted linear map reaches 0.46.

The mechanism transfers causally. The output of GPT-2's induction head 5.1, predicted linearly from Qwen activations and patched back in, recovers 98.1 % of the head's effect (random weights: 8.8 %).

The embedding is nearly symmetric. Qwen → GPT-2 gives 0.437, GPT-2 → Qwen gives 0.446, so "mutual translation" describes this better than one model being inside the other.

Family matrix

R² of the universal operator, one matrix for all layers:

source \ target	Mistral	Qwen2.5	SmolLM2	OLMo	Pythia	GPT
Mistral-7B-v0.3	—	0.538	0.373	0.150	0.692	0.371
Qwen2.5-1.5B	0.286	—	0.400	0.212	0.360	0.439
SmolLM2-1.7B	0.187	0.548	—	0.169	0.269	0.443
OLMo-1B	0.253	0.533	0.388	—	0.318	0.470
Pythia-1.4B	0.776	0.533	0.385	0.177	—	0.472
GPT-2	0.159	0.466	0.322	0.137	0.232	—

Read it by columns: targets differ more than sources do. GPT-2 is predicted well by everyone, OLMo resists everyone — including Mistral, which is four times wider. The strongest pair is Mistral ↔ Pythia (0.692 / 0.776); we have no explanation for it.

Controls: random source weights give 0.035–0.176; a random source against any target gives between −0.005 and 0.008.

Scaling the carrier

With a fixed number of PCA components residual R² falls with model size, but that is an artefact: 768 components keep almost all of 0.5B's variance (98.6 %) and only three quarters of 7B's (75.7 %). With the retained variance share equalised the trend reverses:

carrier	768 components	equal share (85 %)	random weights	difference
Qwen2.5-0.5B	0.580	0.512	−0.018	0.530
Qwen2.5-1.5B	0.572	0.539	−0.001	0.540
Qwen2.5-3B	0.570	0.554	−0.003	0.558
Qwen2.5-7B	0.547	0.572	−0.067	0.639

Where it stops working

The operators were fit on OpenWebText. Applied without refitting: Wikipedia 0.389, The Pile 0.366, code 0.172, against 0.440 in-domain. Universal across layers, not across domains.

Using the operators

import torch
from huggingface_hub import hf_hub_download

p = hf_hub_download("AlexWortega/bertingpt", "operators/ops_qwen_gpt2.pt")
d = torch.load(p, map_location="cpu")
W, V = d["W"], d["V"]          # source -> target and back
pairs = d["pairs"]             # which layers were matched to which

# layers are normalised with the same statistics used when fitting
def to_target(h_source, i):    # i indexes d["pairs"]
    x = (h_source - d["mean_s"][i]) / d["rms_s"][i]
    y = x.double() @ W
    return y * d["rms_t"][i] + d["mean_t"][i]

operators/ contains:

• ops_pair_<A>_<B>.pt — universal operators for a family pair, both directions;
• ops_rand_*.pt, ops_randt_*.pt — the same for the random-weight controls;
• proj_*.pt, uproj_*.pt — trained linear layers for model stitching;
• W_qwen_full.pt — map from all Qwen layers to GPT-2 layers (for the surrogate).

Figures

figures_tex/ holds 13 report figures, each as PDF (vector, for \includegraphics) and PNG at 300 dpi: the layer correspondence map, residual R², rotation versus linear map, stitching, benchmarks, the universal operator and its spectrum, the causal patch, scaling, the family matrix, out-of-domain transfer, the kinship ladder, and pair asymmetry. Each one is described in figures_tex/README.md.

Typeset in Computer Modern with the cm math set, full frame, inward ticks — the same faces TeX uses, so they sit next to LaTeX body text without a seam. Widths are 3.4″ and 7.0″, for one column and for a full page. Rebuild with python3 scripts/42_figures.py [directory].

Reproducing

python3 scripts/00_smoke.py                       # tokenizer alignment and perplexities
python3 scripts/01_fit_pca.py                     # PCA basis of the source layers
python3 scripts/02_layer_map.py --tag qwen        # layer correspondence map
python3 scripts/03_fit_maps.py --n-pos 5000000 --pca-dim 768 --layer-stride 2 --save-w
python3 scripts/04_stitch.py --mode self --k 6    # must return the model's own ppl
python3 scripts/30_universal.py --tag qwen_gpt2   # universal operator and its inverse
python3 scripts/31_causal.py --repeat-frac 0.5    # causal transfer of the mechanism
python3 scripts/34_family_matrix.py               # the summary matrix

Activations are never stored: a single pass accumulates XᵀX, XᵀY and first moments, and the whole regression is solved from those — the full concatenation of 29 × 1536 over 5M positions would take hundreds of terabytes.

Reading the numbers

residual R², not R². The residual stream of any two models is dominated by the identity of the current token, which both see alike, so raw R² is high even for unrelated models. The feature matrix deliberately includes a one-hot block of frequent tokens: "current token only" is then a regression on a submatrix of the same Gram matrix, and the gain over it is obtained in the same pass.

A stitch's perplexity does not answer "is this the same model". A stronger source can give perplexity better than the target — that is not successful recovery but its absence. KL to the target and top-1 agreement are therefore always reported alongside.

Limitations

• Layers in the family matrix are matched by stretching depth uniformly; a CKA map per pair would give higher numbers.
• Three Mistral pairs have no per-layer ceiling — it did not fit in memory at a source width of 4096.
• The Mistral ↔ Pythia pair is anomalously strong and unexplained.
• All maps were fit on OpenWebText; transfer to code collapses.