ChandraPrakashBathula/MINI-CLIP-VLM

Mini-CLIP (Flickr8k)

A small CLIP trained from scratch on Flickr8k: a hand-written ResNet-18 image encoder (ImageNet init) and a 2-block, 4-head pre-LN Transformer text encoder, both projecting into a shared 256-D unit sphere, trained with the symmetric InfoNCE loss (L_i2t + L_t2i)/2 and a learned temperature.

Results (Flickr8k test, 1000 photos × 5000 captions)

	R@1 / R@5 / R@10 (%)	median rank
image → text	23.4 / 50.8 / 64.2	5
text → image	17.3 / 43.1 / 56.9	7

Zero-shot keyword-theme accuracy: 58.1% (chance 11.1%). Temperature 0.07 → 0.036. Parameters: 13.85 M.

Usage

from huggingface_hub import hf_hub_download
import importlib.util
from PIL import Image
path = hf_hub_download("ChandraPrakashBathula/MINI-CLIP-VLM", "modeling_mini_clip.py")
spec = importlib.util.spec_from_file_location("modeling_mini_clip", path); mc = importlib.util.module_from_spec(spec); spec.loader.exec_module(mc)

model, tokenizer = mc.MiniCLIP.from_pretrained("ChandraPrakashBathula/MINI-CLIP-VLM")
cos, probs = model.similarity([Image.open("photo.jpg")], ["a dog running on the beach", "a man riding a bike"], tokenizer)
print(probs)

Training

20 epochs, batch 128, AdamW (lr 0.0003, CNN lr ×0.1, weight decay 0.05), 1-epoch warm-up + cosine decay, AMP, gradient clipping 1.0. Random resized crop (0.6–1.0) + colour jitter, no flip; one random caption of five per photo per epoch.

Limitations

Trained on ~6k photos: a teaching model, not a general-purpose CLIP. Word-level vocabulary (3477 words); unseen words become [UNK]. Flickr8k is for research use.