bmmv-9x2q7/clip100-vit-large-patch14

CLIP ViT-L/14 fine-tuned on 100 image-classification tasks

One fine-tuned vision encoder per task, for model-merging research: <task>/vision_model/ is a transformers.CLIPVisionModel (save_pretrained), and <task>/result.json holds the training record (split sizes, zero-shot and fine-tuned val/test accuracy, recipe).

Recipe

Starting from openai/clip-vit-large-patch14, only the vision tower is trained (text tower, visual_projection and logit_scale frozen). The classifier is the zero-shot head: for each class, the text embeddings of all templates are L2-normalised, averaged and normalised again; logits = logit_scale.exp() * normalize(image_embeds) @ head.T.

setting	value
optimizer	Adam, lr 1e-5, weight decay 0
schedule	CosineAnnealingLR, T_max = 4000, stepped every step
steps / batch	4000 steps, batch 128 (gradient accumulation 4 x 32, same gradient)
precision	fp32 weights, TF32 matmuls
seed	42
augmentation	none (CLIPProcessor only)
data	the task's train split only; val and test are never trained on

This is the default configuration of FusionBench's clip_finetune method (plus TF32 and gradient accumulation).

Tasks

The 20 FusionBench tasks and 80 further tasks (VTAB/ELEVATER, objects and scenes, fine-grained, food, agriculture, medical, remote sensing, characters, documents, people, art, DomainNet). Each task has fixed train / val / test splits; where the source publishes a split that is held out by group (object instance, video, slide, patient, hand model), val is that held-out split, so val accuracy tracks test accuracy. RESULTS_clip-vit-large-patch14.md lists per-task zero-shot and fine-tuned accuracy.

Licenses

These weights are derived from 100 public datasets with different licenses, some of them non-commercial (for example CC BY-NC) and some without a stated license. The license of each source dataset applies to the checkpoint trained on it; check the dataset's own terms before using a checkpoint. The base model is openai/clip-vit-large-patch14.

Loading

from transformers import CLIPModel, CLIPVisionModel
clip = CLIPModel.from_pretrained("openai/clip-vit-large-patch14")
vm = CLIPVisionModel.from_pretrained("bmmv-9x2q7/clip100-vit-large-patch14", subfolder="dtd/vision_model")
clip.vision_model.load_state_dict(vm.vision_model.state_dict())