Color is a Third Wheel in Shape-Texture Bias

Authors: Vatsala Nema, Aditya Pola, Vineeth N. Balasubramanian
Primary area: interpretability and explainable AI
Keywords: CV, Interpretability, Explainability and Transparency

Abstract:
When faced with conflicting shape and texture cues, it is well established that con-
temporary neural networks rely on shape cues less often than humans do. Measur-
ing this gap in preference of shape cues allows us to move towards networks with
an increased shape bias. Complementary to previous works, our work posits that
there is an unexplored yet crucial confounding factor in this debate: color. We
hypothesize that color affects how conflicts between shape and texture cues are
resolved. We test our hypothesis across two dimensions, namely, color variations
and model architectures. To test our hypothesis on an increasing scale of color
variation, we propose color-variants with the following characteristics: a) only
magnitude is preserved, b) discrete colors, c) continuous variation in the color
space, and d) adversarial color variation. To test our hypothesis across the second
dimension, we perform our analysis on two model classes, namely, ViT and CLIP.
This necessitates a new way of measuring shape bias for which we propose a new
metric: Shape Precedence. Our findings extend to the varying nature of shape
bias across model architectures and color distortions at both a global and a local
(patch) level.

Decision: reject

Reviews and discussion:

Metareview: