Tell me why: Visual foundation models as self-explainable classifiers
Authors: name: Hugues Turbé; name: Mina Bjelogrlic; name: Gianmarco Mengaldo; name: Christian Lovis
Published: 2025-02-26 21:40:30

Abstract: Visual foundation models (VFMs) have become increasingly popular due to their
state-of-the-art performance. However, interpretability remains crucial for
critical applications. In this sense, self-explainable models (SEM) aim to
provide interpretable classifiers that decompose predictions into a weighted
sum of interpretable concepts. Despite their promise, recent studies have shown
that these explanations often lack faithfulness. In this work, we combine VFMs
with a novel prototypical architecture and specialized training objectives. By
training only a lightweight head (approximately 1M parameters) on top of frozen
VFMs, our approach (ProtoFM) offers an efficient and interpretable solution.
Evaluations demonstrate that our approach achieves competitive classification
performance while outperforming existing models across a range of
interpretability metrics derived from the literature. Code is available at
https://github.com/hturbe/proto-fm.