aaronwzl/clip-vit-base-patch32-coco-retrieval

Training

base model	openai/clip-vit-base-patch32
objective	symmetric InfoNCE over (image, caption) pairs, using CLIP's learned temperature — the pre-training objective continued at a low learning rate
data source	sayakpaul/coco-30-val-2014 (streaming split train)
training pairs	4000
epochs	2
optimiser steps	126
batch size	64
optimiser	AdamW, lr 1e-05, weight decay 0.1
schedule	10 % linear warmup, then linear decay to 0
gradient clipping	global norm 1.0
seed	42
device	cuda
wall-clock	345 s
loss	0.4031 → 0.0519

Evaluation-set leakage control

The retrieval evaluation uses specific stream indices of sayakpaul/coco-30-val-2014. The training script streams past those and takes only samples whose index is not in the evaluation set, and asserts the disjointness at run time rather than assuming it.

Train/eval index overlap = 0 (verified, 500 evaluation indices held out against 4000 training indices).

Both index lists are shipped in finetune_config.json, so the disjointness is checkable without rerunning anything.

Effect of fine-tuning

Mean cosine similarity on a held-out sample of evaluation pairs, ground-truth versus foil caption:

	zero-shot	fine-tuned	margin change
MSCOCO	GT 0.3245 / foil 0.1490	GT 0.3165 / foil 0.0976	+0.0434

Fine-tuning widens the ground-truth-versus-foil margin mainly by pushing foil similarity down.

Usage

from transformers import CLIPModel, CLIPProcessor

model = CLIPModel.from_pretrained("aaronwzl/clip-vit-base-patch32-coco-retrieval")
proc  = CLIPProcessor.from_pretrained("aaronwzl/clip-vit-base-patch32-coco-retrieval")

Reproduction

python rebuttal/exp/retrieval/finetune_clip.py \
    --dataset coco --n-train 4000 --epochs 2 \
    --batch-size 64 --lr 1e-05 --seed 42