Title: INFER: Embedding Integration with Feature Refinement for Few-Shot Learning in VLMs
Primary area: foundation or frontier models, including LLMs
Keywords: Foundation models, Attention, Fine-grained classification, few-shot learning

Abstract: On general visual recognition tasks, CLIP has demonstrated remarkable few-shot  performance by aligning image and text modalities. However, CLIP's \textit{sole} reliance on the class (CLS) embedding for image representation limits its capacity to capture spatially fine-grained features, which are crucial for fine-grained classification tasks, where subtle differences (e.g., bird species, car models) depend on small, localized variations rather than just the overall object outline. To address this, we introduce INFER, a feature enhancement strategy for CLIP that enhances image embeddings with spatial information intelligently extracted from all patch embeddings as well as the CLS embedding. To this end, INFER leverages attention heads  to compute attention-weighted features of both the patch and CLS embeddings. The most informative heads for each class, identified by their alignment with class text embeddings, are selected to enrich the patch and CLS features, which are then integrated through a fusion module. In the few-shot learning paradigm, INFER establishes new SOTA performance, highlighting the underutilized potential of CLIP’s internal attention mechanisms and providing a generalizable framework for patch-level enhancement in CLIP.