geolip-vit-captionbank-coco

The CaptionBert system transplanted to vision: a two-stage, modular design where the core is a reusable 8.66M-parameter ViT encoder distilled into a 5-teacher consensus space, and the alignment bank is a detachable geometric expansion on top of it. Trained on COCO-2017, two seeds, deterministic evals, and shipped with its complete ledgers — including the preregistered negative result on the bank (research honesty is the point of this repo, not a caveat to it).

Sibling systems: the text-side original geolip-captionbert-8192, the constellation predecessor geolip-vit-large-x3, and the loss-matrix baseline clip-vitb-mini-distilled.

Stage A — the core (reusable)

core/student_s{0,1}.pt — ViT (d=240, depth 12, CLS readout, 160px) with a 512-d head into the GPA consensus space of five CLIP towers (B/16-LAION2B, B/32-OpenAI, B/32-LAION2B, B/32-DataComp, B/16-OpenAI; targets from bulk-coco-features, 5-iteration generalized-Procrustes mean shape). Objective: symmetric InfoNCE (τ=0.07) + per-sample MSE, 1:1 — the CaptionBert Stage-A composite with the CV force held out as a readout. 8,000 steps (~17.3 epochs), batch 256, pure Adam 3e-4.

Measured (per seed, deterministic to 4 decimals):

gauge	s0	s1
agreement → consensus	.6717	.6700
zero-shot CIFAR-10 (rotated to CLIP frame)	.2318	.2636
COCO R@1 (rotated)	.0598	.0530
80-class linear-probe mAP (raw 512-d)	.4007	.3943

The agreement trade (this repo's positive finding): adding InfoNCE to a consensus-MSE distillation buys contrastive-class retrieval (the pure-MSE twin sits at R@1 ≈ .004) at the price of ~17% of its consensus agreement (.67 vs .805).

Stage B/C — the alignment bank (expansion)

banks/bank_s{0,1}.pt — per-expert whitened-Procrustes frames (fp64 fits, gated), 512 dense-cosine anchors seeded from consensus rows, and a 538→256→128 projection; enrich() appends the 128-d geometric context → 640-d. cotrained/ holds the Stage-C jointly-tuned pairs. frames.pt has the raw fitted frames + gate record.

Preregistered probe verdict — the bank's geometry training is REFUTED on this substrate, with full controls (2/2 seeds):

probe input	s0 mAP	s1 mAP
raw 512-d	.4007	.3943
fully-random bank (random frames + anchors)	.4159	.4114
real fitted frames, untrained projection	.4149	.4107
real frames, geometry-trained	.4111	.4062

The +1.4–1.6pt enrichment over raw is generic random-feature capacity: random frames match fitted frames, and the 8-term geometry training subtracts slightly. Anyone using the bank should know the enrichment's value here did not come from the geometry — the bank ships because the architecture is reusable and the negative is load-bearing information. (The text-side original reported +0.125 on a 3-class NLI probe; that bed is untested here — this result is about transfer, not their claim.)

Usage

import torch
from PIL import Image
from huggingface_hub import hf_hub_download

repo = "AbstractPhil/geolip-vit-captionbank-coco"
ldr = hf_hub_download(repo, "loader.py")
import importlib.util
spec = importlib.util.spec_from_file_location("loader", ldr)
L = importlib.util.module_from_spec(spec)
spec.loader.exec_module(L)

student = L.load_student(hf_hub_download(repo, "core/student_s0.pt"))
emb = L.embed_images(student, [Image.open("cat.jpg")])   # (1, 512)

bank = L.load_bank(hf_hub_download(repo, "banks/bank_s0.pt"))
enriched = L.enrich(bank, emb)                            # (1, 640)

The 512-d embedding lives in the 5-CLIP consensus frame — for CLIP-text zero-shot, apply a fitted rotation into a CLIP frame first (the ledgers record the rotated numbers; the rotation is a 2,500-pair orthogonal Procrustes fit).

Deviations from the source recipe (disclosed)

Pure Adam constant-lr (source: AdamW+cosine); CV force as readout in Stage A and gradient-excluded in Stage C (source: 0.1 / in-graph); per-sample MSE so the 1:1 weighting is real (the source's per-element MSE is ~1000× smaller than its InfoNCE term). A faithful full-replication arm (source objectives AND optimizer, verbatim) plus an objective-isolate matrix (BCE force, geometric-gradient operator, CV force) is training at time of writing and will be added to this repo with its results.

Files

core/ Stage-A students · banks/ Stage-B banks · cotrained/ Stage-C pairs · frames.pt fitted expert frames + gate record · loader.py standalone two-stage loader (fidelity-gated: reproduces training-side embeddings to 3e-7) · ledgers/ complete JSONL run records including the random-bank control.