PictSure: Pretraining Embeddings Matters for In-Context Learning Image
  Classifiers
Authors: name: Lukas Schiesser; name: Cornelius Wolff; name: Sophie Haas; name: Simon Pukrop
Published: 2025-06-16 08:57:03

Abstract: Building image classification models remains cumbersome in data-scarce
domains, where collecting large labeled datasets is impractical. In-context
learning (ICL) has emerged as a promising paradigm for few-shot image
classification (FSIC), enabling models to generalize across domains without
gradient-based adaptation. However, prior work has largely overlooked a
critical component of ICL-based FSIC pipelines: the role of image embeddings.
In this work, we present PictSure, an ICL framework that places the embedding
model -- its architecture, pretraining, and training dynamics -- at the center
of analysis. We systematically examine the effects of different visual encoder
types, pretraining objectives, and fine-tuning strategies on downstream FSIC
performance. Our experiments show that the training success and the
out-of-domain performance are highly dependent on how the embedding models are
pretrained. Consequently, PictSure manages to outperform existing ICL-based
FSIC models on out-of-domain benchmarks that differ significantly from the
training distribution, while maintaining comparable results on in-domain tasks.
Code can be found at https://github.com/PictSure/pictsure-library.