More Context, Less Distraction: Visual Classification by Inferring and
  Conditioning on Contextual Attributes
Authors: name: Bang An, name: Sicheng Zhu, name: Michael-Andrei Panaitescu-Liess, name: Chaithanya Kumar Mummadi, name: Furong Huang

CLIP, as a foundational vision language model, is widely used in zero-shot
image classification due to its ability to understand various visual concepts
and natural language descriptions. However, how to fully leverage CLIP's
unprecedented human-like understanding capabilities to achieve better zero-shot
classification is still an open question. This paper draws inspiration from the
human visual perception process: a modern neuroscience view suggests that in
classifying an object, humans first infer its class-independent attributes
(e.g., background and orientation) which help separate the foreground object
from the background, and then make decisions based on this information.
Inspired by this, we observe that providing CLIP with contextual attributes
improves zero-shot classification and mitigates reliance on spurious features.
We also observe that CLIP itself can reasonably infer the attributes from an
image. With these observations, we propose a training-free, two-step zero-shot
classification method named PerceptionCLIP. Given an image, it first infers
contextual attributes (e.g., background) and then performs object
classification conditioning on them. Our experiments show that PerceptionCLIP
achieves better generalization, group robustness, and better interpretability.
For example, PerceptionCLIP with ViT-L/14 improves the worst group accuracy by
16.5% on the Waterbirds dataset and by 3.5% on CelebA.