clip-vitb-mini-distilled

An 8.66M-parameter ViT-Ti-class student distilled from CLIP ViT-B/16 (LAION-2B) pooled image features on COCO-2017 train only (118,287 images, ~17 epochs) — a controlled, budget-scoped experiment in CLIP distillation objectives, shipped with its full ledger set and a negative headline honestly stated.

This is a research artifact, not a production model. At this data budget the student reaches ~21% of the teacher's zero-shot capability (CIFAR-10 + CIFAR-100 mean). The interesting cargo is the comparative loss study and the frame-ambiguity finding below.

Headline numbers (test-time, deterministic eval)

model	zs CIFAR-10	zs CIFAR-100	COCO R@1 (i→t)	params
teacher CLIP-B/16 LAION-2B	0.946	0.759	0.417	86M (image tower)
this student (InfoNCE, seed 0)	0.267	0.094	0.058	8.66M
random-init floor	0.100	0.007	0.000	8.66M

Zero-shot uses the teacher's text tower (the student lives in the teacher's 512-d projection space). CIFAR images are bicubic-upscaled to the student's 160px input; the teacher is judged at its native 224px in the same harness.

The loss matrix (6 objectives × 2 seeds, 8000 steps, batch 256)

arm	zs C10 (s0/s1)	zs C100	COCO R@1	teacher-agree	eff. rank
feature MSE (mimicry)	.269 / .266	.062	.025 / .026	.714 / .718	122
InfoNCE (ship arm)	.267 / .231	.094 / .085	.058 / .054	.404 / .401	184
SigLIP pairwise	.221 / .253	—	.043 / .040	—	150
affinity KL	.109 / .111	—	.000	−.02 (see below)	175
InfoNCE + geometric terms	.248 / .253	.084 / .072	.057 / .051	.404	185
5-teacher consensus MSE	.154 / .161	—	.004 / .006	.29 (see below)	108

Findings, each 2/2 seeds unless noted:

• The gauge split. Feature-MSE mimicry wins coarse zero-shot and teacher agreement; InfoNCE wins fine-grained zero-shot (+3.2/+2.2 pts CIFAR-100), retrieval (2.2×), and effective rank (184 vs 122). Pick your objective by the gauge you will be judged on.
• Frame ambiguity (the campaign's main finding). The affinity-KL arm scores at floor on every absolute gauge while its embedding cosine to the teacher is −0.02 — orthogonal, not absent. One orthogonal Procrustes rotation fitted on 2,500 held-in pairs recovers agreement to 0.50 and zero-shot from 0.109 to 0.240 — replicated on both seeds with post-rotation agreements matching to 0.004 (ledgers/frame_check.json, ledgers/frame_check_s1.json). The 5-teacher consensus arm tells the same story: rotated agreement 0.713 equals direct mimicry's 0.714. Losses without a cross-space anchor learn the geometry with the global frame free — evaluate them through a fitted rotation or they read as false floors.
• Coupling wins retrieval. Batch-coupled InfoNCE beats the decoupled SigLIP-style pairwise loss on retrieval 2/2 at batch 256 (single batch size tested — disclosed confound).
• Geometric add-on terms were inert here: InfoNCE alone already lands the embedding-spread band the extra terms target.

Usage

import torch
from huggingface_hub import hf_hub_download

repo = "AbstractPhil/clip-vitb-mini-distilled"
ckpt = hf_hub_download(repo, "student_infonce_s0.pt")
ldr = hf_hub_download(repo, "loader.py")

import importlib.util
spec = importlib.util.spec_from_file_location("loader", ldr)
loader = importlib.util.module_from_spec(spec)
spec.loader.exec_module(loader)

model = loader.load_model(ckpt)              # eval mode, cuda if available
x01 = torch.rand(4, 3, 224, 224)             # any [0,1] images
emb = loader.embed_images(model, x01)        # (4, 512) L2-normalized

Outputs are compatible with the teacher's text tower (laion/CLIP-ViT-B-16-laion2B-s34B-b88K) for zero-shot classification and retrieval.

Training recipe

• Student: ViT d=240, depth 12, heads 4, patch 16, input 160px, CLS readout, linear head to the teacher's 512-d projection space.
• Targets: precomputed pooled projection-space features from AbstractPhil/bulk-coco-features (L2-normalized at load), teacher preprocessing verified to cosine 1.00000 against the live tower before any training.
• Objective (ship arm): symmetric InfoNCE student↔teacher, temperature 0.07, in-batch negatives, batch 256.
• Optimizer: pure Adam (no weight decay), fp32, TF32 off, fixed CRC-derived seeds; 8000 steps ≈ 17.3 epochs; peak 4.83 GB on one consumer GPU (~37 min per run).
• Full per-run ledgers in ledgers/ (JSONL; agree_cos is cosine to the teacher target; eval_only rows are deterministic re-evaluations, which reproduce the training-run numbers to 4 decimal places).

Caveats

• COCO-only data budget. The viability bar preregistered for this experiment (≥40% of teacher zero-shot at ≤10% params) was refuted — treat this as a measured data point on the distillation-data curve, not a usable general-purpose encoder.
• Zero-shot numbers come from 160px students judging 32px CIFAR upscales; absolute values are harness-scoped (floor/ceiling in the same harness bracket them).
• Single training configuration per arm (2 seeds); batch-size sensitivity of the contrastive comparisons untested.

Related

• The loss-objective survey this experiment extends: AbstractPhil/loss-manifest
• Teacher feature bank: AbstractPhil/bulk-coco-features

References

• Radford et al., Learning Transferable Visual Models From Natural Language Supervision (CLIP) — https://arxiv.org/abs/2103.00020
• van den Oord et al., Representation Learning with Contrastive Predictive Coding (InfoNCE) — https://arxiv.org/abs/1807.03748
• Yang et al., CLIP-KD: An Empirical Study of CLIP Model Distillation — https://arxiv.org/abs/2307.12732
• Wu et al., TinyCLIP: CLIP Distillation via Affinity Mimicking and Weight Inheritance — https://arxiv.org/abs/2309.12314
• Zhai et al., Sigmoid Loss for Language Image Pre-Training (SigLIP) — https://arxiv.org/abs/2303.15343
• Lin et al., Microsoft COCO: Common Objects in Context — https://arxiv.org/abs/1405.0312
• Cherti et al., Reproducible scaling laws for contrastive language-image learning (OpenCLIP / LAION-2B) — https://arxiv.org/abs/2212.07143