AutoCLIP: Auto-tuning Zero-Shot Classifiers for Vision-Language Models
Authors: name: Jan Hendrik Metzen, name: Piyapat Saranrittichai, name: Chaithanya Kumar Mummadi

Classifiers built upon vision-language models such as CLIP have shown
remarkable zero-shot performance across a broad range of image classification
tasks. Prior work has studied different ways of automatically creating
descriptor sets for every class based on prompt templates, ranging from
manually engineered templates over templates obtained from a large language
model to templates built from random words and characters. In contrast,
deriving zero-shot classifiers from the respective encoded class descriptors
has remained nearly unchanged, that is: classify to the class that maximizes
the cosine similarity between its averaged encoded class descriptors and the
encoded image. However, weighting all class descriptors equally can be
suboptimal when certain descriptors match visual clues on a given image better
than others. In this work, we propose AutoCLIP, a method for auto-tuning
zero-shot classifiers. AutoCLIP assigns to each prompt template per-image
weights, which are derived from statistics of class descriptor-image
similarities at inference time. AutoCLIP is fully unsupervised, has very low
overhead, and can be easily implemented in few lines of code. We show that for
a broad range of vision-language models, datasets, and prompt templates,
AutoCLIP outperforms baselines consistently and by up to 3 percent point
accuracy.