Selective Contrastive Learning for Weakly Supervised Affordance
  Grounding
Authors: name: WonJun Moon, name: Hyun Seok Seong, name: Jae-Pil Heo

Facilitating an entity's interaction with objects requires accurately
identifying parts that afford specific actions. Weakly supervised affordance
grounding (WSAG) seeks to imitate human learning from third-person
demonstrations, where humans intuitively grasp functional parts without needing
pixel-level annotations. To achieve this, grounding is typically learned using
a shared classifier across images from different perspectives, along with
distillation strategies incorporating part discovery process. However, since
affordance-relevant parts are not always easily distinguishable, models
primarily rely on classification, often focusing on common class-specific
patterns that are unrelated to affordance. To address this limitation, we move
beyond isolated part-level learning by introducing selective prototypical and
pixel contrastive objectives that adaptively learn affordance-relevant cues at
both the part and object levels, depending on the granularity of the available
information. Initially, we find the action-associated objects in both
egocentric (object-focused) and exocentric (third-person example) images by
leveraging CLIP. Then, by cross-referencing the discovered objects of
complementary views, we excavate the precise part-level affordance clues in
each perspective. By consistently learning to distinguish affordance-relevant
regions from affordance-irrelevant background context, our approach effectively
shifts activation from irrelevant areas toward meaningful affordance cues.
Experimental results demonstrate the effectiveness of our method. Codes are
available at github.com/hynnsk/SelectiveCL.