aaronwzl/clip-vit-base-patch32-flickr-retrieval

Training

base model	openai/clip-vit-base-patch32
objective	symmetric InfoNCE over (image, caption) pairs, using CLIP's learned temperature — the pre-training objective continued at a low learning rate
data source	ThraggBilly/flickr30k_dataset (streaming split train)
training pairs	4000
epochs	2
optimiser steps	126
batch size	64
optimiser	AdamW, lr 1e-05, weight decay 0.1
schedule	10 % linear warmup, then linear decay to 0
gradient clipping	global norm 1.0
seed	42
device	cuda
wall-clock	519 s
loss	0.3571 → 0.0993

Evaluation-set leakage control

The retrieval evaluation uses specific stream indices of ThraggBilly/flickr30k_dataset. The training script streams past those and takes only samples whose index is not in the evaluation set, and asserts the disjointness at run time rather than assuming it.

Train/eval index overlap = 0 (verified, 500 evaluation indices held out against 4000 training indices).

Both index lists are shipped in finetune_config.json, so the disjointness is checkable without rerunning anything.

Effect of fine-tuning

Mean cosine similarity on a held-out sample of evaluation pairs, ground-truth versus foil caption:

	zero-shot	fine-tuned	margin change
Flickr30K	GT 0.3043 / foil 0.1785	GT 0.2949 / foil 0.1215	+0.0477

Fine-tuning widens the ground-truth-versus-foil margin mainly by pushing foil similarity down.

Usage

from transformers import CLIPModel, CLIPProcessor

model = CLIPModel.from_pretrained("aaronwzl/clip-vit-base-patch32-flickr-retrieval")
proc  = CLIPProcessor.from_pretrained("aaronwzl/clip-vit-base-patch32-flickr-retrieval")

Reproduction

python rebuttal/exp/retrieval/finetune_clip.py \
    --dataset flickr --n-train 4000 --epochs 2 \
    --batch-size 64 --lr 1e-05 --seed 42