ComFe: An Interpretable Head for Vision Transformers
Authors: Evelyn Mannix
Liam Hodgkinson
Howard Bondell
Venue: Accepted by TMLR ()

Abstract: Interpretable computer vision models explain their classifications through comparing the distances between the local embeddings of an image and a set of prototypes that represent the training data. However, these approaches introduce additional hyper-parameters that need to be tuned to apply to new datasets, scale poorly, and are more computationally intensive to train in comparison to black-box approaches. In this work, we introduce Component Features (ComFe), a highly scalable interpretable-by-design image classification head for pretrained Vision Transformers (ViTs) that can obtain competitive performance in comparison to comparable non-interpretable methods. To our knowledge, ComFe is the first interpretable head and unlike other interpretable approaches can be readily applied to large-scale datasets such as ImageNet-1K. Additionally, ComFe provides improved robustness and outperforms previous interpretable approaches on key benchmark datasets while using a consistent set of hyperparameters and without finetuning the pretrained ViT backbone. With only global image labels and no segmentation or part annotations, ComFe can identify consistent component features within an image and determine which of these features are informative in making a prediction. Code is available at github.com/emannix/comfe-component-features.

Published in Transactions on Machine Learning Research (11/2025)
ComFe: An Interpretable Head for Vision Transformers
Evelyn J. Mannix
enelyu..moonir@@uiiinccc......
School of Mothematics and Statistics University of Melbourne
Liam Hodgkinson
thodglinssonmuuiinelb...o
School Mathematics and Statistics
University of Melbourne
Howard Bondell
howard, howard. hww..m/eoddd@@@@eeeeel..e...e wdw.eelbbbedd....o.
School of Mathematics Statistics University of Melbowrne
Reviewed OpenReview: https: oppprev//peerevieuu. net/ forum? id= cI
Abstract
Interpretable computer vision models explain their classifications through comparing the distances between the local emboddings of an image and set of prototypes that represent the training data. However, these approaches introduce additional hyper-parameters that need to be tuned to apply to new datasets, scale poorly, and more computationally intensive to train in comparison to black-bor approaches. In this work, we introduce Component Features (ComFe), highly scalable serpreeable-by-desigeee image classification head for pretrained Vision Transformers (ViTs) that can obtain competitive performance in comparison to comparable non-interpretable methods. To our knowledge, ComFe is the first interpretable head and unlike other interpretable approaches can be readily applied large-scale datasets such as ImageNet-1K, Additionally, ComFe provides improved robustness and outperforms previous interpretable approaches key benchmark datascts while using consistent set of lyperaarameters and without finetuning the pretrained VIT backbone. With only global image labels and no segmentation or part annotations, ComFe can identify consistent component features within an image and determine which of these features are informative in making a prediction. Code is available at github.com cenaunix/comff-copppe.
Component Features
Comparison to Class Prototypes
Prediction: Osteospermunn
Input Image
Ostcospermemm class prototype Sim. Dagg. Bi00
Background class prototype
Sim.
1.0
Figure Illustratonn of ComFe. The image is first clustered into component features, which are then compared to class prototypes. This comparison between component features and class prototypes is used identify the salient parts of the image for predicting the class Osteospermum,, shown by the class confidence heatmap in the final image.

---

Published in Transactions on Machine Learning Research (11/2025)
1 Introduction
Deep learning is applicd in numerous contexts solve challenging computer vision problems, from identifying disease in medical imaging (Zhou al., 2023), species identification (Beloiu et al. 2023) and even in self-driving cars [Lee & Liu 2023 However, standard deep learning approaches are black bores (Rudin| 2019); and it can be challenging to determine whether prediction is being made based on the most relevant features in an image. For example, neural networks can often learn spurious correlations from image backgrounds YYang et al. 2022
Post-hoe interpretability techniques, such as Class Activation Maps (CAM) Zhou et al. 2016), can uncover this behaviour. However, these approaches can be misleading and cannot be used to understand which parts
of the training dataset led to particular classification [Rudin] 2019). Interpretable models that are designed to reason in logical fashion, can achieve this goal by identifying prototypical parts within the training
dataset that provide evidence for particular category (Chen et al. 2019 Nauta et al. 2023
Xue et al.
2024
Nevertheless, interpretable models can have a poor semantic correspondence between the embeddings of protolypes and their visual characteristics (Kim et al. 2022; Hoffmann et al. 2021). Current interpretable approaches are also less accessible in comparison standard black- box approaches: they add new hyper- parameters which need to be tuned for each dataset, and scale poorly to larger networks datasets with large number of classes. Current approaches do not report performance on large scale datasets, such ImageNet-1KK, and doing so would require signisicanl compute and hyperporaaeeeer lunvig to produce meaningful results. Foundation models trained on large datasets using self-supervised learning and a Vision Transformer (ViT) architecture, such as the DINOv2 family (Oquab ct al.] 2024), could help solve these challenges. These models produce embeddings across a range of vision contexts that laithfully reflects semantic similarity (Qquab et al. 2024), and a classifier that can explain predictions using this latent space has the potential to be easy to train, highly scalable and provide good semantic correspondence.
This paper presents Component Features (ComFe), a modular and efficient interpretable-by-deiiee image classification head that addresses these challenges. To the best of our knowledge, ComFe is the first classifier head for downstream prediction that provides interpretability intrinsically, rather than being a post-hoc technique to explain a model's behaviour. ComFe is designed to be used with a pretrained ViT network, and employs a transformer decoder (Vaswani et al., 2017) architecture and hierarchical mixture modelling framework to make explainable predictions. As shown in Fig. ComFe identifics a set of component features within an image and compares them library of class protolypes learned from the training data which can be used to answer why particular prediction is made. This approach is inspired by Detection Transformer (Carion et al. 2020), Fask2Former (Cheng et al.] 2022) and PlainSeg (Hong et al. 2023), in addition to recent advances in semi-supervised learning (Assran et al. 2021 In this work, we:
Mo et al.
2023 Mannix & BBondell 2023
Present ComFe, the first interpretable image classification head, and the first interpretable method that is sufficiently scalable to report results on the ImageNet-1K dataset. ComFe achieves competitive results with comparable non non-interpretable approaches, and provides improved performance range of ImageNet-1K generalisability and robustness benchmarks.
Demonstrate the competitive performance of ComFe on a range of datasets using a consistent set of hyperparameters in comparison to other interpretable approaches that finetune the backbone networks and tunc the hyperparameters for cach datasct.
Highlight how ComFe can consistently detect particular regions of an image (e.g. a bird's head, body and wings), identify informative versus non-informative (i.c. background) regions, and explain why particular predictions are made on the basis of similarity class prototypes that are representative of the training data.
2

---

Published in Transactions on Machine Learning Research (11/2025)
2
Related Work
Interpretable computer vision models. ProtoPNet (Chen et al. 2019) was the first method to show that deep learning based computer vision models could be designed to explain their predictions and obtain performance similar to non-interpretable approaches. They introduced prototypes for interpretability--in representations of concepts from the training data that can be used to explain why a model makes a particular prediction. This allowed for classification to be based on the distance between image patch embeddings and these prototypes within the latent space (Chen et al. 2019). To ensure the interpretability of these prototypes, they constrained them to the embeddings of a patch within the training datasct.
A number of further works, including ProtoTree (Nauta et al.
2021 3 ProtoPShare (Rymarezyk et al. 2021 ProtoPool (Rymarczyk et al., 2022). ProtoPFormer (Xue et al. 2024 and PIP-Net [Nauta et al. 2023 are derived from this approach. ProtoTree incorporates a decision tree into the reasoning process, while the other approaches focus on addressing key weakness of ProtoPNet-the learned prototypes are often background features in the training data, and these need to be pruned to obtain good interpretability. ProtoPShare, PIP-Net and ProtoPool take different approaches to achieve this, through encouraging prototypes to be shared between classes to improving their sparsity. ProtoPFormer improves the application of ProtoPNet to Vision Transformers (ViT) (Dosovitskiy et al., 2021 Touvron et al., 2021 and presents new loss that further encourages learned prototypes to attend to foreground rather than background features.
In contrast, the INTR Paul et al., 2024) approach uses the cross-attention layer from a transformer decoder head to produce explanations of image classifications. While elegant, this moves away from prototypes being associated with the training data, making these models less transparent than the ProtoP family. It is also noted that later works, such as PIP-Net, move away from the inflexibility of the prototypes being determined by the training data, and instead interpret and visualise prototypes by using the closest training example within the latent space, the one with the highest probability under the model (Nauta. et al. 2023
Nevertheless, these methods can be challenging to adapt to new datasets they introduce number of additional hyperparameters to the training process. This is highlighted by most papers in this field only using three benchmarks CUB200 (Wah et al.) 2011), Oxford Pets (Parkhi et al. 2012) and Stanford Cars Krause et al. 2013) with hyperparameter tuning being undertaken for each dataset. They can also be computationally expensive to fit in comparison to non-interpretable approaches and require large amounts of GPU memory.
Self-supervised learning and foundation models. In computer vision, self-supervised approaches train neural networks as functional mappings between the image domain and a represcntation or latent space that captures the semantic information contained within the image with minimal loss (Chen et al. 2020a 2021
2020b). -hate-of-the-art self-supervised approaches can obtain competitive performance downstream tasks such as image classification, segmentation and object detection when compared to fully supervised approaches without any further fine-tuning (Tukra et al.) 2023: Oquab et al. 2024). This is achieved by using techniques such as bootstrapping (Grill et al. 2020), masked autoencoders (He et al., 2022), or contrastive learning (Chen et al. 2020a), which use image augmentations to design losses that encourage a neural network to project similar images into similar regions of the latent space. By combining these approaches on large scale datasets, VIT foundation models such as DINOv2 (Oquab al.) 2024) can be trained that perform well across range of visual contexts. Contrastive languago-image pretraining (CLIP) can also be an effective self-supervised approach, but it requires large image datasets with captions to train models (Radford el. al.
2021
Cherti et al. 2023
3 Methodology
Underlying the reasoning process of the ProtoP family of interpretable approaches is the encoder network. If this network performs poorly, then the erplanation provided by ProtoP approaches will be nonsensical (Hoffmann et al. 2021 Kim et al., 2022). Computer vision foundation models, like the DINOv2 family, provide highly informative patch features that produces an embedding space where the cosine similarity between image patches describes their degree of semantic similarity (Oquab al.] 2024). It has also
Motivation.
3

---

Published in Transactions on Machine Learning Research (11/2025)
ComFe Head
Class Prototypes
Patch Embheddings
Image Protolypes
Class Confidence
Input Image
Clustering Head
Pretrained
Vision Transformer
Standard
Prediction: Searlet tanager
Class Token
Linear
Head
Figure 2: The ComFe framework. A pretrained ViT network is applied to input image X, to produce patch embeddings Z and a class token. The ComFe clustering head is trained to cluster patch embeddings and produce image protolypes P, which are compared to a set of learnt class prototypes C which represent the training data to identify the informative image regions and make a prediction. In constrast, a standard non-interpretable approach will train a linear head to make a prediction using the class token, and does not provide insight into the informative image regions which training images led a particular prediction.
been found that particular directions within the latent space of these models will encode particular concepts [Bhalla et al.] 2024). This work addresses the challenge of leveraging these foundation model features build modular interpretable image classification head that can be applied to pretrained Vision Transformer
(VIT) (Dosvvitskiy et al. 2021 network.
Interpretability with component features. Similarly to the ProtoP methods, we aim to learn class protolypes C, which are directions within this feature space that relate to set of concepts interest (e.g. species of birds). In particular, we want to learn class prototypes in the latent space of the output patch embeddings Z of the network, this will allow the regions of image that contribute to particular classification to be identified. Further, these class prototypes need to be learnt in a way that represents the training data, so they can be used to answer why a particular prediction is made.
This can be posed as a clustering problem, where the goal is to use distributions contred on the class protolypes C to cluster the patch embeddings Z into meaningful groups. Under this framework, the closest. training data patches to the centre of these clusters can function as exemplars, and can be used to visualise each class prototype.
However, the output of a ViT network can have hundreds of patches per image, many of which will be similar, particularly if they make up the same part of particular object scene. To reduce the complexity of solving the class prototype clustering problem, image prototypes P are introduced. These represent component features within an image, which can be used to consider a small set image regions instead of a much larger number of image patches. To learn the image prototypes themselves, a second, nested clustering problem is introduced learn a clustering head go, that generates the image prototypes parametrically on the basis of the patch embeddings Z. This facilitates interpretable inference that is more elficient than considering individual patches, and this approach is illustrated in Fig.
Notation. We refer to an RGB image as X E R xx where w is the image width and h is the height. Throughout this paper, we use bolded characters to refer to matrices, and the notation A refers to the th row of matrix while A refers the 5th column. The respective lowercase letter with two subscripts ai refers to a specific element of the matrix A. We consider a frozen pretrained ViT encoder model f and clustering head go, where 0 refers to the parameters of this network.
4

---

Published in Transactions on Machine Learning Research (11/2025)
Class prototypes C Association matrix
Query Matrix
Transformer
Class Confidence Map:
p(v|Z) Eq. (6
Image X Image label
Class Prodiction:
Pretrained Backbone
Encoded
Image prototypes
ply (2) Eq. 7
H
patches
Training Objective:
Z
P
L(Z, y; Q, C) Eq.
Component Features Map:
p(PIZ) Eq. 19
Figure 3: Summary of the ComFe framework. Given an input image X, the patch embeddings are obtained using a pretrained ViT backbone model, f(X) = Z. These patches are clustered into component. features using
of image prototypes P, that are obtained from transformer decoder clustering head described in Eq- (5)- In producing a classification, the image prototypes P are comparod to the class protolypes C, as described in Eq. (6) and Eq. 7 The variable represents the class prediction of local patch, which is unknown as segmentation labels are assumed to not be available. The component features map (image prototypes) are visualised based on the image prototype P with the greatest likelihood of generating a particular patch in Eq. (19..
Modelling framework. To formalise the model, we consider three key components: (1) the patch embeddings Z == = f(X E &Nxx where d is the dimensionality of the latent space and Nz is the number of patches created created by the pretrained model f; (2) the unknown patch class labels v, described by the class prototypes C E Rede where we have NG class prototypes with a fixed association each class defined by smoothed one-hot encoded matrix & R.Ne with smoothing parameter a; and (3) image prototypes P E Nx which represent component features within image, where Np is the number of desired image prototypes and is much smaller than the number of image patches Nz.
2. of the Prior works CCarnn et al. 2021 Ayzenberg et al. 2024) have found the & normalised form 2, = I ||
outputs provide an effective measure of semantic similarity, through the cosine distance metric. This also facilitates fitting distributions the high dimensional space of these embeddings (Sra 2018). We consider
the following joint probability based on a hierarchical mixture modelling approach MMarin et al. 2005 Zhou
et al. 2020
pZZz, p): :
(1)
where p(v) is the prior distribution for the patch classes in the dataset. This joint probability describes a generative model where the patch class V, parameterised by class prototypes C, describes the distribution of the 5th image prototype P;:, which in turn describes the distribution of the 7th patch embedding Z This defines the image patches Z as independent of the patch class v when conditioned on an image prototype Pj:- We define these distributions in terms of the Mon-Mises Fisher distribution on the unit sphere in E2d (denoted sd) Govvndaraaaa et al. 2023 with density
VMF u. Ca(T) exp(s- p)) x e 3d,
(2)
with mean vector sd, concentration parameter 7, and normalizing constant Ca(T). The model becomes:
p(2..1P,,)) VMF (2:; P;:,))
(3)
No
di VMF.(PFF.)) T2) p(P Σ
(4)
Parameterising the prototypes. The class prototypes C are parameterised as matrix of learnable parameters, while the image prototypes P are fit parametrically following amortised-inspired approach
5

---

Published in Transactions on Machine Learning Research (11/2025)
Margossian & Blei. 2024). To promote the efficiency of ComFe, do not learn the full distribution of each Pj. and instead use point Maximum Likelihood Estimate (MLE).. For each image, this is done by generating ,; deterministically using a transformer decoder /Vaswani et al. 2017), which we refer to as the clustering head
P.: P..(Z,,Qe.) go Q
(5)
where Q E RR-exd is learnable query matrix, that prompts the decoder to calculate the image protolypes by considering the entire patch representation of the image Z. For clarity, we leave the arguments 0, Q and Z (..(.) as implied, and simply write P::
To predict the class that a patch belongs to, Bayes' rule is used and the image prototypes are marginalised out to obtain
Deriving a likelihood.
Z() R Σ Popp C/2 P/T1)
(6)
where o()) is the softmax function and the approximation arises from using a MLE point estimate for PJ:. Uniform categorical prior distributions are assumed, and further details are given in Section A in the supporting information.
The main challenge here that v unknown. We only assume access to image classífication annotations y, which describe the global image label a one-hot encoded class vector. To connect the unknown v class of the image patches to the global annotation y, vector of binary probabilities is constructed using max-pooling
plu 2) = max plu 1/2..),
which describes the maximum probability given by a patch that class I is present in the image. This provides the likelihood
loggp(2.1) -
(8)
where the first term describes how well the image prototypes are clustered, and BCE(y, p(y/ [)) Σiz in (ogg(())2) Σi= (1 m) log(1 p(u (2) denotes the binary cross-entropy loss for multi-label classification problem.
Training objective.
Based the likelihood above, the training objective for ComFe is given by
L(Z, y: 0, Q C)- = Leiscrim (Z. Q.C)
+ feluster (2 Q) + Laux (Z y: C):
which is separated into the discriminative term Liiierrm that uses a binary cross entropy loss and a clustering term Leluster- Additional auxiliary objectives Laux are also specified which ensure the consistency of image prototypes, the uniqueness of class prototypes and improve the fitting process. The discriminative and clustering term are given by
Nx
Np
Celuster Q2
log
VMF P(Am)
(10)
Nz
Cdiserim 0, C) = BCE p(u 2)).
(11)
A more detailed description of the derivation and motivation of the clustering, discriminative and auxiliary terms is provided in Section in the supporting information.

---

Published in Transactions on Machine Learning Research (11/2025)
Class prototype exemplars. The class prototypes C represent the centres of von-Mises Fisher distributions. The image prototypes P from the training dataset with greatest cosine similarity to each class prototype can be used as representatives, to visualise the concept each one describes. These particular image prototypes are called class prototype exemplars, and are used to visualise the class prototypes in Fig. I and Fig. 2 Further examples are also shown in Fig. | and Fig. S5 and Fig. SG in the supporting information. This reflects Eq. where the cosine similarity of an image prototype to each class prototypes is used in the term
(i) = =Ppp: C/T2)
(12)
for classifying an image. This measure in Eq. 12 was used to calculate the similarity values presented in Fig. | and elsewhere within the paper. These exemplars are only used to interpret the class prototypes
post-training, similar to the prototypes in PIP-Net (Nauta et al. 2023 and are not used during the training process.
Background classes. To allow the model to learn which regions of an image are informative, an additional background class is added and it is assumed that each image has one or more background patches. This turns the multi-class image classification problem into a multi-label one, where the background class is added to the label for each image. Further, NN additional background class prototypes are added to C and Φ. This can be done without changing the form of the discriminative loss terms in Eq. (10) as the binary cross-entropy loss naturally extends to multi-label classification problems. Further details are provided in Section A in the supporting information.
Comparison to closely related work. The ComFe head shares a similar architecture to the INTR (Paul et al. 2024) approach, in that they both use transformer decoder on the outputs of a backbone network make a prediction. However, INTR uses the query Q to define classes using one query vector per class and relies cross-attention within the decoder heads to interpret why the model is making a particular prediction. For ComFe, the query has a fixed size that depends on the number of image prototypes Np, and the similarity of the image P to the class C prototypes is used to explain the model outputs. This makes ComFe much more scalable to datascts such as ImageNet, as the number of image prototypes Np is much smaller than the number of classes. This allows for models to be easily trained with large batch sizes on a single GPU.
Recently, Zhu et al. (025)) introduced new ProtoP inspired method, which we refer to ProtoNonParams designed for use with loundation models. ProtoNonParan consists of three key steps; (i) foreground extraction, using a PCA based approach (Oquab et al. 2024) to identify the objects of interest for classification, (ii) non-parametric prototype learning, where the challenge of identifying foreground patches of a known class with part-prototypes is posed as an optimal transport problem and solved using the Sinkhorn-Knopp algorithm,
and (iii) feature space finetuning. Tn this last step, the part-prototype vectors and foundation model weights are frozen, and additional transformer encoder blocks are interleaved through the original layers and finetuned (Zhu et al. 2025). While this approach is able to effectively exclude backgrounds features from becoming part prototypes through the foreground extraction step, this limits the flexibility of the approach in cases where there are multiple possible foreground objects present.
4 Experiments
4.1
Implementation details
The weights of the pretrained VIT network f are frozen during training, and the parameters of the clustering head ge are randomly initialised using Xavier normal distribution (Glorot & Bengio 2010 The class prototypes C are randomly initialised from a uniform distribution on the unit sphere, and the initial queries Q are sampled from a standard normal distribution. Standard choices for training transformers are used, such as the AdamW optimiser {Loshchilov & Hutter 2017b), cosine learning rate decay with linear warmmp [oohchilov & Huttor 2017a; Gotmare et al., (019)) and gradient clipping (Pascanu et al. 2013). For image augmentations, we follow DINOv2 and other works, including random cropping, flipping, color distortion and random greyscale (Chen al. 2020a Oquab al. 2024). For the architecture of the clustering head go, two
7

---

Published in Transactions on Machine Learning Research (11/2025)
transformer decoder layers with eight attention heads are used, and following PlainSeg Hong et al. 2023 the loss is calculated after each decoder layer and averaged.
For each dataset, a total of five image prototypes P giving Q five rows and be class prototypes C are used, where c is the number of classes in the dataset. The first 3c class prototypes are assigned to each class (for three per class), while the remaining 3e are assigned to the background class. Similar temperature parameters are used to previous works (Assran et al., 2021p, with 71 = 0.1, 72 = 0.02 and Te = 0.02. An ablation study on these hyperparameters is undertaken in Section D of the supporting information.
The same set of hyperparameters are used across all of the training runs, with the exception of the batch size which is increased from 64 image to 1024 for only the ImageNet dataset. The number of epochs for ImageNet Russakkvsky el al.] 2015) is also reduced, from 50 epochs per training run to 20 epochs. All results use an input resolution of 224 x 224 pixels, with outputs upsampled using bilinear interpolation to generate pixel-level results from patch-level predictions.
4.2 Datasets
Finegrained image benchmarking datasets including Oxford Pets (37 classes) Parkhi et al. 2012 FGVC
Aircraft (100 classes) (Maji et al. 2013), Stanford Cars (196 classes) (Krause et al.) 2013) and CUB200 (200
classes) (Wah et al.
2011 have all previously been used to benchmark interpretable computer vision models.
These are used to Lest the performance of ComFe, in addition to other datasels including ImageNet-11
(1000
classes) (oussakovsky et al. 2015), CIFAR-10 (10 classes), CIFAR-100 (100 classes) (Krizhevsky et al.
2009
Flowers- 102ts--002000 (102 classes) (Nilsback & Zisscrman 2008) and Food-101 (101 classes) (Bossard et al.
2014
These cover a range of different domains and have been previously used to evaluate the linear fine- Lunit performance of the DINOv2 models (Oquab et al., 2024).
Further test datasets are considered for ImageNet that are designed to mcasure model gencralisability and robustness. TetgeNet--V2 V2 (Recht et al.) 2019) follows the original data distribution but contains harder examples, Sketch (Wang et al. 2019) tests if models generalise to sketches of the ImageNet class and ImageNet-R (Hendrycks et al. (0211a) tests if models generalise to art, cartoons, graphics and other renditions. Finally, the ImageNet-A [Hendrycks et al.) 2021b) test dataset consists of real-world, unmodified and naturally occurring images of the ImageNet classes that are often misclassifidd by ResNet models. While ImageNet- -2 and Sketch contain all of the ImageNet classes, ImageNet-A and ImageNet-R only provide images for 200 class subset.
4.3 Main results
ComFe obtains competitive performance in comparison to previous interpretable approaches. Table [ shows that ComFe heads obtain competitive performance with previous interpretable models of similar sizes. We note that in some cases this is an imperfect comparison, with ComFe using the strong DINOv2 backbones, and prior works using networks initialized with ImageNet weights. We further include the benchmarks published by Zhu et al. (0025), that train interpretable models with the DINOv2 backbone by adding additional transformer blocks, which are finetuned while the original weights are kept frozen. When these additional blocks are finetuned, methods such as TesNet (Wang et al., 2021), EvalProtoPPNet Huang ct al. 2023) and ProtoNonParam can slightly outperform ComFe on CUB200, while ComFe still performs strongly in comparison on Stanford Cars. Without this finetuning step, the ProtoNonParee (Zhu el. al. 2025 approach performs very poorly in comparison to ComFe.
To our knowledge ComFc is the first interprotable head, and while not all benchmarks in Table 1 are directly comparable, we present them to highlight the utility of this approach. We also note that where ComFe uses the same set of hyperparameters for all datasets Table [ other works undertake hyperparameter tuning
for each task (Paul et al.) 2024 Nauta et al. 2023 Xue al.) 2024).
ComFe obtains competitive performance compared to a non-interprctable linear head. Table 2 shows that ComFe is competitive with a linear head, the standard approach of training non-interpretable models with a frozen backbone. ComFe achieves improved performance on number of datasets, including
8

---

Published in Transactions on Machine Learning Research (11/2025)
Table 1: Interpretable performance comparison. Performance (top-1 accuracy) of ComFe and other interpretable image classification approaches on several benchmarking datasets. INTR and ComFe use transformer decoder heads, which adds additional model parameters (#P), while the benchmarks reported by Zhu et al. (2025) introduce K additional transformer blocks to the DINOv2 backbone to support feature space finetuning.
Method
Backbone
Dataset
#P
#P
CUB
Pets
Cars
Airer
Interpretable
Net [Chen et al.
2019
ResNet-34
22M
7U.2
86.1
(Fineluned backiones)
ProtoTree
Naula el al
2021
ResNet-34
22M
82.2
86.6
ProtoPShare
Hymarcz
ResNet-31
22M
71.7
86.5
ProtoPool
Rymarczyk
ResNet-50
26M
85.5
ProtoPForter
CaiT-XXS-24
12M
84.9
90.9
ProtoPFormer Xue
20724
DeiT-S
22M
84.8
91.0
PIP-Net Nauta
2025
CouvNeXt-ting
29M
81..3
92.0
88.2
PIP-Net
Namit et al.
2023
ResNet-50
26M
82.0
88.5
86.5
INTR Paul ei at. 2024
10M
ResNet-50
26M
71.8
90.4
86.8
76.1
ProtoP
[Zhu ct A 2025
DINOv2 VIT-S/14 (K = 5) 21M+-9M
85.5
79.5
TesNet
(Zhu es nl. 21125
DINOv2 VIT-S/14 (K = 21M+9M
89.0
82.6
EvalProtoPNee Zhu et al.
2025
DINOv2 VIT-S/14
5) 21M+9M
88.2
87.1
ProtoNon
Zhu al.
2025
DINO<2 VIT-S/14 (K 21M+99M
88.2
(Frozen backbones)
ComFe
8M DINOv2 VIT-S/14 (1)
21M
87.6
94.6
91.1
77.5
ProtoNouParam
Zhu 2025
DINO VIT-S/14 (6)
21M
78.0
Table Non-interpretable performance comparison. Performance (top-1 accuracy) of ComFe versus a linear head (quub et al. 2024 with frozen features several benchmarking datasets.
Method
Backbone
Dataset
#P
DINOv2
#P
IN-1K
C10
C100
Food
CUB
Pets
Cars
Airer
Flowers
Linear
Oquab et al
2024
<1M
VIT VIT-S/14 (f)
21M
81.1
97.7
87.5
89.1
88.1
95.1
81.6
74.0
99.6
Linear
Oquat et Bl
<1M
VFT-B/14 (f)
86M
84.5
98.7
91.3
92.8
89.6
96.2
88.2
79.4
99.6
Linear
Oquab et al
<1M
VIT-L/14 (1)
300M
86.3
99.3
93.4
94.3
90.5
96.6
90.1
81.5
99.7
ComFF
8M
VIT-S/14 (f)
21M
83.0
98.3
59.2
92.1
87.6
91.6
91.1
77.5
99.0
ComFe
32M
VFT-B/14 (f)
86M
85.6
99.1
92.2
94.2
88.3
95.3
92.6
81.1
99.3
VIT-L/14 (F)
300M
99.4
93.6
94.6
89.2
93.6
99.4
ImageNet, CIFAR-10, CIFAR-100, Food-101, StanfordCars and FGVC Aircraft. For smaller models, we obscrve that ComFe outporforms the linear head by larger margin. For species identification datasets, such as CUB200, Oxford Pets and Flowers-102, ComFe results slightly lower performance compared the linear head. Further investigations into the CUB200 dataset suggest that this may be due to incorrect annotations in the training data, as explored in Section B
Table Generalisation and robustness benchmarks. Performance (top-1 accuracy) of ComFe versus a linear head with frozen features ImageNet-1K generalisation and robustness benchmarks. For IN-R and IN-A accuracy is reported across all classes rather than restricting to the subset of classes included in these datasets. The accuracy only considering these classes is provided in Table S14
Method
Backbone
Test Dataset
#P
DINOsg
#P
IN-V2
Sketch
IN-R
IN-A
Linear
Oquab et
2024
<IM
VII-S/14
21M
70.9
11.2
37.5
18.9
Linear Oquah
2024
<1M
VFT-B/14 (f)
75.1
50.6
47.3
37.3
Linear Oquab
2024
<IM
VIT-L/14
300M
78.0
57.9
52.0
ComPe
VET-S/14
21M
73.4
45.3
26.4
ComFe
32M
VIT-B/14
66M
77.3.
54.8
51.9
43.2
ComFee
57M
VIT-L/14
300M
78.7
59.5
58.8
55.0
ComFe improves generalisability and robustness. Table shows that ComFe is able to improve performance the ImageNet-V2 test set while also improving on generalisability and robustness benchmarks

---

Published in Transactions on Machine Learning Research (11/2025)
Table 4: Training efficiency. Comparison of timings and GPU memory requirements for interpretable approaches trained on the CUB200 dataset using an NVIDIA 80GB A100 GPU and batch size of 64.
Method
Backbone
Epoch time
GPU mem
Epochs
PromoPFormer
VIT-L/16
300M
42s
28GB
200
PIP-Net
ConvNeXt-ting
29M
121s
42GB
60
ComFe
57M
DINO,2 VIT-L/14 (O) 300M
3
5GB
50
in comparison to a linear head. Previous work has shown that while self-supervised learning approaches such as CLIP can improve their performance on ImageNet and ImageNet-V2 by fine-tuning the model backbone, doing so can reduce performance on Sketch, ImageNet-A and ImageNet-R (Radford et al. 2021). This suggests that ComFe may be a viable option to improve model performance with a pretrained VIT network, without compromising on the generalisability and robustness of the model.
ComFe is efficient. Table shows that ComFee is efficient to train in comparison to other interpretable approaches. For the CUB200 dataset, ComFe can train a ViT-L/14 model in thirty minutes on one 80GB NVIDIA A100 GPU, and has a sufficienlly small memory footprint that it could be trained on most mid-range consumer graphics cards.
(a) CUB
(b) CUB
(c) Airer
(d) Airer
(e) Cars
(f) Cars
Soarise Tanager cloos peototype Stm.. 1 1
Artic Tem closs pootoly pe Stmm BEDE, pi50
747-200 clons prototype
DHCC-a-500 clacs prototype Sim. T
Purrai 438 claes prosotype BiB
Fand Renger class prosocypp 85m. 10
Shm.
BIRS 0:98. m0
Stm.
Background class protolype Stm. Bi9
Backaround clasa protolarpe Satm. Bill
Backgrownd peotolype 8tm.
Packground peototype
Backaround class Stm. 0.87
Backgroand classs prototype panet
Background prototype
Background class prototype
Backaround class prototype
Background prototype
Background prosetype
Prediction: Scarlet Tonager
Prediction: Artic Tern
Prediction: 747-200
Prediction: DHC-8-300
Prediction: Ferrari 458
Prediction: Ford Ranger
Figure 4: Visualizing ComFe explanations. Example ComFe predictions showcasing explainability, sampled from the validation images from the FGVC Aircraft, Stanford Cars and CUB200 datasets. The rows show the input images, component features (image prototypes), class prototype similarity and exemplars, and the class confidence heatmap for the final classification.
The image prototypes learnt by ComFe identify visual components across classes In Fig. 4 it can be seen that the image prototypes P partition images containing different. classes in consistent manner.
10

---

Published in Transactions on Machine Learning Research (11/2025)
On the CUB200 dataset, the first image prototype (red) identifies the head of birds, while the second image prototype (yellow) identifies the wings and tail feathers. The third image prototype (green) identifies the body, and the final two image prototypes (blue and cyan) are always associated with the background. Similar patterns are observed across the other datascts, with particular image prototypes identifying the front, side and upper cabin and roof of the car for Stanford Cars, and image prototypes locating the wings, tail and the body of plane for the FGVC Aircraft dataset.
The class prototypes learned by ComFe identify the category which image prototypes belong to. Fig. [ also shows how the image prototypes P are classified as informative (belonging to a specific class) or non-informative by the class prototypes C. We observe that when identifying birds the head appears to be non-informative on the CUB200 datasct, which reflects that a background class prototype has been found by ComFe on the ViT-$ DINOv2 backbone that relates phenomenon occurs in Stanford Cars, where two background class prototypes have been found that describe the upper vehicle cabin and roof of particular types of cars. Further visualisations of these class prototypes are shown in Fig. S5 in the supporting information.
bird's head across different species. A similar
Like other clustering algorithms (Al-Daoud & Roberts 1996), ComFe sensitive to the initialisation of the algorithm, and different features may be found be informative with different seeds. This is further addressed in the discussion and Section C in the supporting information, where the features found to be informative can also be seen to reflect the chosen pretrained backbone Fig. S14
Table Quantifying interprotabilty using FunnyBirds. Performance of mothods trained on FunnyBirch Hesse al. 2023) under accuracy, correctness, completeness, distractability and contrastivity and background independence measures, and example attribution maps.
Method
Backbone
Measure
Acc.
Cor.
Com.
Dist.
Con.
B.I.
ProtoPNet Hesse et al. 2023
ResNet- 50
0.94
0.24
0.92
0.58
0.46
1.00
ComFee
VFT-B/14 w/rreg
0.98
0.74
0.86
0.80
0.61
0.98
(DINOv2)
VIT-S/14 w/reg
0.97
0.67
0.88
0.60
0.54
0.96
VIT-B/14
0.97
0.57
0.01
1.00
0.51
0.98
VIT-S/14
0.96
0.49
0.96
0.73
0.51
0.98
ProtoPNet RosNett50
ComFee ViT-B/14 w/reg
ComFe
ComFe
ComFe
VIT-S/14 w//rg
VIT-B/14
VIT-S/14
ComFe can be used to evaluate the interpretability of a pretrained VIT network. We employ the Funny Birds framework (Hesse et al. 2023 to quantify the interpretability of VIT backbones using a ComFe head. FunnyBirds builds a dataset of cartoonish birds, which are constructed from various discrete parts. This allows the framework define and calculate a number of different measures directly, such as correctness (correlation between model outputs and the reported importance of particular regions), contrastivity (degree to which explanations are different for similar classes) and completeness (degree which explanation describes all aspects of the model's decision), which trades off against distractability (degree to which an explanation ignores irrelevant fcatures in the image) (Hesse et al. 2023). Classification accuracy and background independence (the sensitivity of the output to removal of background objects) are also reported, and all scores range from 0-1 where perfect performance. To provide context, we compare the results of ComFe heads to ProtoPNet, noting that this comparison uses different backbones and metric interfaces. The FunnyBirds measures are more indicative than comprehensive, given the limitations of the artificial dataset, particularly when comparing across different types of explanations (Hesse et al. 2023
11

---

Published in Transactions on Machine Learning Research (11/2025)
Table shows the results for ProtoPNet and ComFe heads using DINOv2 backbones with and without registers. ViT models larger than ViT-S/14 without registers have been observed to learn artefact patches-backkrrong patches that store global information whereas the introduction of registers mitigates this effect and improves performance in larger models (Darcet et al., 2024). ComFe heads on frozen DINOv2 backbones provide stronger results than ProtoPNet on accuracy, correctness, contrastivity and distractabilityy With the exception of the DINOv2 ViT-B/14 model, competitive completeness also achieved. Background independence is also similar between all of the models. While ViT patch embeddings can contain global information, when they are trained suitably, like the small DINOv2 foundation models and variants with registers (Darcet et al., 2024). the kernels that are learnt generally preserve local information. This is supported by the strong performance of a ComFe head with the DINOv2 ViT-B/14 w/reg backbone across these interpretability measures.
However, this is not the case for all ViT networks. The figures below Table shows that the ComFe head for the DINOv2 ViT-B/14 14 network learns to classify using artefact patches background patches contaminated with global information (Darect et al.) 2024). This is highlightod by the abysmal completeness score for this model in Table 5] which by finding no parts to be informative allows for a perfect distractability score. Contrastivity and correctness are more similar between the DINOv2 VIT-S/14 and ViT-B/14 14 models, as all bird parts (eyes, beak, legs, tail and wings) are given mostly equal weight waight-maximum confidence, or very little at all. This is not the case for the DINOv2 variants with registers, where the ComFe heads in the example images both identify that the eyes are not informative.
Other interpretability metrics designed for ProtoP approaches, such as the purity measure introduced with PIP-Net (Nauta et al. 2023), are based on the definition of ProtoP prototypes. This is not directly applicable to ComFe, as the measures would need to be reworked for image and class prototypes, preventing clear comparisons. Further results for ComFFe using DINOv2 backbones with registers and other pretrained ViT models are included in Section C
5 Discussion
Comparison to ProtoP approaches. The ProtoP family considers the relationship between prototypes and classes to be represented by a learnable linear layer, whose weights describe the relationship between prototypes and classes (Chen et al. 2019 Nauta et al. 2023). This layer is key to making the final class prediction, as inference is generally done by max-pooling the prototype activations and passing them through this matrix. The ComFe approach deviates from this by using two sets of protolypes, image protolypes P that describe particular components of an image, and class prototypes C which represent directions in the latent space that describe a particular concept (c.g. type of car). In sense, the class prototypes of ComFe are similar to the protolypes considered by ProtoPNet and related approaches, as they represent key directions within the embedding space that describe particular concepts. However, in ComFe these are preassigned to classes, and the similarity between these class prototypes and the image prototypes for a particular image determines the predicted class of an image.
Limitations. ComFe is designed to be used in conjunction with a high quality foundation model backbone. While these models, like DINOv2 (Oquab et al. 2024), can perform well in many contexts, they can still perform poorly on images that are very different from their training set (Zhang et al. 2024).. As ComFee is not designed to fine-tune the foundation model backbone, it may struggle to perform well in these settings. However, is expected that in future these vision foundation models will continue to improve in performance, as they are usod as visual encoders in multimodal LLMs, such as LLaVA (Liu et al.) 2023) and GPT-4 (OpenAl et al., 20244). For example, it has been recently shown that foundation models can be improved through combining CLIP pre-training objectives with dense self-supervised tasks, such as those used in DINOv2 (Tschannen et al. 2025).
It is further noted that ComFe may identily different features as informative within an image, depending on the algorithm's initialisation. Such variability is common in clustering methods (Al-Daoud & Roberts, 1996) and explored further in Section of the supporting information. In practice, this variability can be addressed by training multiple heads, whose outputs can either be combined in an ensemble or evaluated individually to select the best-performing head [Jain 2010). In our experiments (Fig. [S11 and Fig. S12 the
12

---

Published in Transactions on Machine Learning Research (11/2025)
best-performing head typically yields semantically meaningful confidence map. Ensembling is not explored further, as interpreting the results would be more complex. The features deemed informative also vary across backbones, reflecting differences in their embedding spaces (Fig, S14
ComFe, as presented in this work, is designed for image-level classification, where global labels guide the learning of informative regions. As a result, not all parts of an object may be deemed informative, and some regions can be classified as background, leading incomplete confidence maps. Extending ComFee to semantic segmentation, where supervision is provided by segmentation masks, could address this limitation by encouraging prototypes to capture the full extent of an object. Related approaches, such as ProtoNon (Zhu et al. 2025), employ foreground extraction to obtain more spatially complete confidence maps, and a similar strategy could also be used with ComFee if required.
6 Conclusion
This work presents ComFe, a modular interpretable-by besig image classification head that uses a transformer decoder architecture and a hicrarchical mixture modelling approach build classification models using a pretrained VIT backbone. ComFe learns to identify directions within the latent space defined by this pretrained network that relate to particular concepts. This allows for the model to explain how image is classified, by examining the similarity between component features of the image and these class prototypes. ComFe is fast to train and has low GPU memory requirements, in addition to providing improved performance across a range of benchmarking datasets in comparison to other interpretable methods.
Acknowledgements
We acknowledge the Traditional Custodians of the unceded land on which the research detailed in this paper was undertaken, the Wurundjeri Woi Wurrung and Bunurong peoples of the Kulin nation, and pay our respects their Elders past and present. This research was undertaken using the LIEF HPC-GPGPU Facility hosted at the University of Melbourne. This Facility was established with the assistance of LIEF Grant LE1 70100200. This research was also undertaken with the assistance of resources and services from the National Computational Infrastructure (NCT), which is supported by the Australian Government. Evelyn J. Mannix was supported by an Australian Government Research Training Program Scholarship to complete this work.
References
Moh'd B Al-Daoud and Stuart Roberts. New methods for the initialisation of clusters. Pattern Recognition Letters, 17(5):451-455, 1996.
Mahmoud Assran, Mathilde Caron, Ishan Misra, Piotr Bojanowski, Armand Joulin, Nicolas Ballas, and Michael Rabbat. Semi-supervised lcarning of visual features by non-parametrically predicting view assignments with support samples. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 8443 8452, 2021.
Lev Ayzenberg, Raja Giryes, and Hayit Greenspan. DINOv2 basod self supervised learning for few shot medical image segmentation. In 2024 IEEE International Symposium on Biomedical Imaging (ISBI), pp. 1-5. IEFE, 2024.
Mircla Beloiu, Lucca Heinzmann, Nataliia Rehush, Arthur Gessler, and Verena C Griess. Individual tree-crown detection and species identification in heterogeneous forests using acrial RGB imagery and deep learning. Remole Sensing, 1565) 2023.
Usha Bhalla, Alex Ocsterling, Suraj Srinivas, Flavio Calmon, and Himabindu Lakkaraju. Interproting CLIP with sparse linear concept embeddings (spliCE). In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
13

---

Published in Transactions on Machine Learning Research (11/2025)
Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101-mining discriminative components with random forests. In Computer Vision-ECCC 2014: 13th European, Conference, Zurich, Swilzerland, September 6-12, 2014, Proceedings, Part VI 13, pp. 446 461. Springer, 2014.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European conference on computer vision, pp. 213-229.
Springer, 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised Vision Transformers. In Proceedings of the IEEE/CVE international conference on computer vision, pp. 9650-9660) 2021.
Chaofan Chen, Oscar Li, Daniel Tao, Alina Barnett, Cynthia Rudin, and Jonathan K Su. This looks like that: Deep learning for interpretable image rocognition. Advances in neural information processing systems, 32, 2019.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597-1607. PMLR, 2020a.
Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey E Hinton. Big self-supervised models are strong semi-supervised learners. Advances in neural information processing systems, 33: 22225,, 2020b.
Ting Chen, Calvin Luo, and Lala Li. Intriguing properties of contrastive losses. Advances in Neural Information Processing Systems, 34:1 111118341188/7 2021.
Bowen Cheng, Ishan Misra, Alexander Schwing, Alexander Kirillov, and Rohit Girdhar. Masked-attenntin mask transformer for universal image segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 1290-1299, 2022.
Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive image learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pp. 2818 2829, 2023.
Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision Transformers need registers. In The Twelfth International Conference on Learning Representations, 2024.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.
Xavier Glorot, and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural notworks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp. 249 256. JMLR. Workshop and Conference Procoedings, 2010.
Akhilesh Gotmare, Nitish Shirish Keskar, Caiming Xiong, and Richard Socher. A closer look at deep lcarning heuristics: Learning rate restarts, warmup and distillation. In International Conference Learning Representations, 2019.
Hariprasath Govindarajan, Per Sidén, Jacob Roll, and Fredrik Lindsten. DINO as von Mises-Fishh mixture model. In The Fleventh International Conference Learning Representations, 2023.
Tean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent: A new approach to self-supervised learning. Advances in neural information processing systems, 13:21271-21222 21244, 2020.
14

---

Published in Transactions on Machine Learning Research (11/2025)
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/ /VF conference computer vision and poltern recognition, pp. 16000 16009, 2022.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: A critical analysis of out-of-distribution generalization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 8340-8349) 2021a.
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15262-15271, 2021b.
Robin Hesse, Simone Schaub-Meyer, and Stefan Roth. Funnybirds: A synthetic vision dataset for part-based analysis of explainable Al methods. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3981-3991, 2023.
Adrian Hoffmann, Claudio Fanconi, Rahul Rade, and Jonas Kohler. This looks like that... does it? Short- comings of latent space prototype interpretability in deep networks. arXiv preprint 1XX::2205502963 2021.
Yuanduo Hong, Jue Wang, Weichao Sun, and Huihui Pan. Minimalist and gigh-performance semantic segmentation with plain Vision Transformers. arXiv preprint arXiv:2310.1275: 2023.
Qihan Huang, Menggi Xue, Wenqi Huang, Haofei Zhang, Jie Song, Yongcheng Jing, and Mingli Song. Evaluation and improvement of interpretability for self-cxplainable part-prototype networks. In Proceedings of the IEEE/CVF International Conference on Compuler Vision, pp. 2011 1020, 2023.
Anil K Jain. Data clustering: 50 years beyond k-means. Pattern recognition letters, 31 31()):551-666 2010.
Zi-Hang Jiang, Qibin Hou, Li Yuan, Daquan Zhou, Yujun Shi, Xiaojie Jin, Anran Wang, and Jiashi Feng. All tokens matter: Token labeling for training better Vision Transformers. Advances in neural information processing systems, 14:18590-18609 2021.
Sunnie SY Kim, Nicole Meister, Vikram V Ramaswamy, Ruth Fong, and Olga Russakovsky, HIVE: Evaluating the human interpretability of visual explanations. In European Conference on Computer Vision, pp. 280-298.
Springer, 2022.
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3D object representations for fine-grained categorization. In Proceedings of the IEEE international conference on computer vision workshops, pp. 554-561, 2013.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009. URL https://www.c cs. toronto. wdw/-
centurres-2099 TR. pdf Technical report.
Der-Hau Lee and Jinn-Liang Liu. End-to-end deep learning of lane detection and path prediction for real-time autonomous driving. Signal, Image and Video Processing, 17(1):199-22) 205, 2023.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, Advances in neural information processing systems, 36:34892 34916, 2023.
Tlya Loshchilov and Frank Hutter. SGDR: Stochastic gradient. descent with warm restarts. In International Conference on Learning Representations, 2017a.
Nyy Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2017b.
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine- Eine-grained visual classification of aircraft. Xiv preprint 1306. 2013.
15

---

Published in Transactions on Machine Learning Research (11/2025)
Evelyn Mannix and Howard Bondell. A Mixture of Exemplars approach for efficient out-of-distribution detection with foundation models. or arXiv preprint arXin:231 1. 17093, 2023.
Charles C Margossian and David M Blei. Amortized variational inference: when and why? In Proceedings of the Fortieth Conference on Uncertainty in Artificial Intelligence, pp. 2434 2449, 2024.
Jean-Michel Marin, Kerrie Mengersen, and Christian P Robert. Bayesian modelling and inference on mixtures of distributions. Handbook of statistics, 25:459-500 507, 2005.
Sangwoo Mo, Jong-Chyi Su, Chih- Ma, Mido Assran, Ishan Misra, Licheng Yu, and Sean Bell. RoPAWS: Robust semi-supervised representation learning from uncurated data. In The Eleventh International Conference on Learning Representations, 2023.
Meike Nauta, Ron Van Bree, and Christin Seifert. Neural prototype trees for interpretable fine fine-graine grained image recognition. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 14933, 2021.
Meike Nauta, Jörg Schlötterer, Maurice van Keulen, and Christin Seifert. PIP-Net: Patch-based intuitive prototypes for interpretable image classification. In Proceedings of the IREE/CVF Conference on Computer Vision and Pottern Recognition, pp. 2744 2753, 2023.
Maria-Elena Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sirth Indian conference on computer vision, graphics & image processing, pp. 122-729. IEEE, 2008.
OpenAT, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Tlge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. GPT-4 technical report, 2024. URL https://arxiy: http:///axvvv.og//ee// att///a///230000//
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V. Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel HAZIZA, Francisco Massa, Alaaeldin El-Nouby, Mido Assran, Nicolas Ballas, Wojeiech Galuba, Russell Howes, Po-Yao Huang, Shang- Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Herve Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. DINOv2: Learning robust visual features without supervision. Thansactions on Machine Learning Research, 2024. ISSN 2835-8856.
Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In 2012 IEEE conference on computer vision and pattern recognition, pp. 3498 1508-3505. IEEE, 2012.
Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. On the difficulty of training recurrent neural networks. In Sanjoy Dasgupta and David McAllester (ods.), Proceedings of the 30th International Conference on Machine Learning, volume 28 of Proceedings of Machine Learning Research, pp. 1310-1318, Atlanta, Georgia, USA, 17-19 Jun 2013. PMI.R.
Dipanjyoti Paul, Arpita Chowdhury, Xinqi Xiong, Feng-Ju Chang, David Edward Carlyn, Samuel Stevens, Kaiya L Provost, Anuj Karpatne, Bryan Carstens, Daniel Rubenstein, Charles Stewart, Tanya Berger- Woll, Yu Su, and Wei-Lun Chao. A simple interpretable transformer for fin-grriied image classification and analysis. In The Twelfth International Conference on Learning Representations, 2024.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. Tn International conference on machine learning, pp. 8748 8763. PMTR, 2021.
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do ImageNet classifiers generalize ImageNet? In International conference on machine learning, pp. 5389-5400. PMLR, 2019.
Cynthia Rudin. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature machine intelligence, ((5):206-215 2019.
16

---

Published in Transactions on Machine Learning Research (11/2025)
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet large scale visual recognition challenge. International Journal of Computer Vision (IJCV), 115(3):211 252, 2015.
Dawid Rymarczyk, Lukasz Struski, Jacek Tabor, and Bartosz Zieliński. ProtoPShare: Prototypical parts sharing for similarity discovery in interpretable image classification. In Proceedings of the 27th ACM SIGKDD Conference Knowledge Discovery & Data Mining, pp. 1420-1430, 2021.
Dawid Rymarczyk, Lukasz Struski, Michal Górszczak, Koryna Lewandowska, Jacek Tabor, and Bartosz Zieliáski. Interpretable image classification with differentiable prototypes assignment. In Europeon Conference on Computer Vision, pp. 351 368. Springer, 2022.
Thalles Silva and Adín Ramírez Rivera. Representation learning via consistent assignment of views to clusters. In Proceedings of the 37th ACM/SIGAPP Symposium on Applied Computing, pp. 987 994, 2022.
Suvrit Sra. Directional statistics in machine learning: A brief review. Applied Directional Statistics, pp- 275-20 772-299,110 2018.
Andreas Peter Steiner, Alexander Kolesníkov, Xiaohua Zhai, Ross Wightman, Jakob Uszkoreit, and Lucas Beyer. How to train your ViT? data, augmentation, and regularization Vision Transformers. Transactions on Machine Learning Research, 2022. ISSN 2835-8856.
Hugo Touvron, Matthicu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International conference on machine learning, pp. 10347-10357. PMLR, 2021.
Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muhammad Ferjad Nacem, Ibrahim Alabdulmohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, et al. SigL.P 2: Multilingual vision-language encoders with improved semantic understanding, localization, and dense features. arXiv preprinl Xix:22550 14786, 2025.
Samyakh Tukra, Frederick Hoffman, and Ken Chatfield. Improving visual representation learning through perceptual understanding, In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14486-14495 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The Caltech-UCSD 200ds-200-2011 dataset. Jul 2011. URL htttp:://wnt /twiiviiioo.o.oonnnommmmmm caltech, scuudataaets//cb___a 200
Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust. global representations by penalizing local predictive power. Advances in Neural Information Processing Systems, 32, 2019.
Jiaqi Wang, Huafeng Liu, Xinyue Wang, and Liping Jing. Interpretable image recognition by constructing transparent embodding space. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 895 904, 2021.
Mengqi Xue, Qihan Huang, Haofei Zhang, Jingwen Hu, Jie Song, Mingli Song, and Canghong Jin. ProtoP- Former: Concentrating on prototypical parts in Vision Transformers for interpretable image recognition. In IJCAL, pp. 1516 1524,1 2024.
Yua- Yang, Chi-Ning Chou, and Kamalika Chaudhuri. Understanding rare spurious correlations in neural networks. In ICML 2022: Workshop on Spurious Correlations, Invariance and Stability, 2022.
Yunhua Zhang, Hazel Doughty, and Coes GM Snock. Low-resource vision challenges for foundation models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 21956-21963 2024.
17

---

Published in Transactions on Machine Learning Research (11/2025)
Bingyin Zhao, Zhiding Yu, Shiyi Lan, Yutao Cheng, Anima Anandkumar, Yingjie Lao, and Jose M Alvarez. Fully attentional networks with self-emerging token labeling. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 5585 5595, 2023.
Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. In Proceedings of the IEEE conference computer vision and pattern recognilion, pp. 2921 2929, 2016.
Kevin Zhou, Hayit Greenspan, and Dinggang Shen. Deep learning for medical image analysis. Academic Press, 2023.
Shuo Zhou, Howard Bondell, Antoinette Tordesillas, Benjamin I. P. Rubinstein, and James Bailey. Early identification an impending rockslide location via a spatially-aided Gaussian mixture model. The Annols of Applied Statisties, 14(2):977 992, 2020. doi: 10. 10.1214/20-AAS1332
Zhijie Zhu, Lei Fan, Maurice Pagmueco, and Yang Song. Interpretable image classification via non pa-parrmmtric part prototype learning. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp- 9762-9771, 2025.
18

---

Published in Transactions on Machine Learning Research (11/2025)
Supplementary Material
A Further derivations for the learning objective
Predicting patch class membership. Bayes' rule can be used to obtain the probability that patch belongs to a particular patch class under this model, assuming that the prior distributions for the image prototypes p(j) and the patch classes p(v) are uniform categorical distributions. This assumes, in the first instance, that each patch is equally likely to be generated by each image protolype, and each image prototype is equally likely to be generated by each patch class. First, the probability that an image prototype Py; is generated by a particular patch class can be computed using Bayes rule, where Eq. can be insorted obtain
p(v1P,)) = p(P,,lu)t p(P,:)
(13)
p(P pppp))
(14)
Σi [ip (P pillp(n)
= (P C/T2) ow.
(15)
where a() is the softmax function, and Py: is the L2 normalised form of P Then, consider the probability that patch Zi was generated by specific image prototype Pj., where Eq. 3
can be inserted to obtain
p pZZ, P j:P(PPp:))
p(P,, J: 2.)
(16)
p(Z,:)
p Zz PP j: :.)p(P j:)
(17)
Σ 1- pl (Z, p::)p(
Nr
pl P PP::)
(18)
Σ 1= p(Z P.j:)
Zi P\TT)
(19)
To facilitate the calculation of (P, (.)) to obtain the approximation in Eq. 18
we treat p(P)))) as prior with uniform distribution over the sphere
This gives the probability that patch Z; belongs to a particular patch class v, through
Np
p(v 2x)) =
Σ
p(v (5:)p(5) p(p, 2.) dp,.
(20)
j=1
An
Eg
Np
p(v Σ j=1
p(P) 2.).
(21)
where we approximate the intractable integral by using a point estimate, inserting the py: = Pp dircction that maximises the likelihood of the model, generated using Eq. (5) while optimising for 0. The image prototypes Py: are treated like latent variable relating a patch Z₁ to its patch class v, that are marginalised from the expression.
Deriving the likelihood. We do not have data to directly train a model on the patch classes v, as they are unknown. Instead, we use the global image class labels y as one-hot encoded class vector. The probability
19

---

Published in Transactions on Machine Learning Research (11/2025)
of class being present within image be defined the maximum class probability over all image patches, creating vector of Bernoulli distributions, of which the 7th element is given by
p(un Z max p(v Ze.)
(22)
Using this approach, the negative log likelihood of the model for single image is
- log (2. = - log p(Z) - Σ yi log p(un 2) + (1 - 31) log (1 - p(nn )
(23)
where the discriminative portion of the loss is given by the following binary cross entropy loss
ddiscrim (Z, y/ 0, Q. Q) Σ yn log p(u)) + (1 u) log(1 p(2n p(1v(22))
(24)
and the clustering portion of the loss is given by the marginal distribution
N2
log p(g
1
Σ
log (Z.))
(25)
N2
1=1
1
log
Σ
p(2::10)))())0000)
(26)
Nz
1
log
Z P,:)pp)p(pPP))
(27)
Here the same MLE point approximation P.: = j: is introduced in Eq. 21 to avoid the intractable integral. However, it is found that including the marginal term p(P,,) j:) in this loss function hurts performance, instead the following is used as the clustering loss
Celuster (2;0.)=
log
(Z. P,)D((i
(28)
Nx
N₂
Np
log
VMF VMFa(2... p))
(29)
Nz
3=1
This is due to the marginal term p(P)))) including the class prototypes C. Removing this term from the clustering loss ensures that the prototypes C are primarily learnt to discriminate classes.
Auxilary loss: Image prototype classes. Three additional constraints are added to the model to improve quality using an auxilary loss term T-aux- For the first constraint, we observe that through Eq. (21) there is a relationship between the class prediction the image prototypes and patches
p(yl2) S p()(P))
(30)
where
p(u[P)) = maxp(un|PPp)
(31)
If there are no image protolypes associated with class, the patches will also not be associated with that class. As a result, adding an additional discriminative loss term to ensure that at least one image prototypes reflects the global image label
e y log p(ulP) (1 v) log(1 p(yn (P)
Σ
E1-discrimZZ. y: Q,
(32)
could result in small performance improvement.
20

---

Published in Transactions on Machine Learning Research (11/2025)
Auxilary loss: Image and class prototype diversity. To ensure that the image and class prototypes have no redundancy, second term is added to penalise duplicate image prototype vectors and class protolype vectors. This is achieved by using a contrastive loss term CChen et al. 2020a)
N.
log exp(C,: Ca/Te)
Conntrast (Z;0,Q.C) = = Σ
(33)
Σ j=1 exp(C;; Cj./Te)
i
Nr exp(Pr Pu./Te)
Np اfر
exp(P: Py./Te)
with the temperature parameter Te. We note that this constraint not memory intensive, as the contrastive loss is defined only between the prototypes that belong to particular image.
Image prototype consistency. The third constraint encourages consistency the prototypes assigned to particular patches when the image is augmented. We use loss term inspired by Consistent Assignment for Representation Learning (Nauta et al. 2023 Silva & Rivera 2022) (CARL)
Auxilary loss:
Nz Np log log P, 2 ) (P),1
CCARL. ( = Nz
1
Σ
(34)
where Z and P are the patch embeddings and image protolypes from an augmented view X* of the image X. This loss term is minimised when the softmax vectors given by Eq. 19 from different views are consistent and confident. To create the augmented views we apply the same cropping and flipping augmentations, so that the patches cover the same image region, but different color and blur augmentations.
Final training objective. The total auxiliary loss term given by adding these three component terms.
Laux (Z, y: 0, Q. = C p-ddserrim (Z Q C) + Ccontrast (Z;terratZZZ000 Q) + CCARL (Z:0QQ, Q)
(35)
which fully specifies the complete training objective for a single image, given by
L(Z, y/ 0, Q C) = Leluster (Z; Q) + Laiscrim (Z, y: Q C) + Laux (2, y; Q,
(36)
When fitting the loss over more than one image, we using mini hini-batching approach and use the average over the batch as the final optimisation loss.
Further details on label smoothing and background classes. Label smoothing is utilised for the associations of the class prototypes C to the classes, as described by the matrix &. Formally, we can write this matrix as
if mod -=)=
Ply
(37)
otherwise
where a is the smoothing parameter and e is the number of classes.
When including background classes, the single class label y is extended into a multiclass image label by appending a as the final element, meaning each image assigned both its original class and the background class. The associations matrix extended accordingly using the same label smoothing strategy described above, with additional entries incorporated to handle the expanded class prototypes C. (NNc+Nw) where Ne represents the original number of classes, Nx the number of additional background class prototypes, and d is the feature dimension.
21

---

Published in Transactions on Machine Learning Research (11/2025)
Algorithm Algorithm for training ComFe.
Training set T, NE number of epochs, I backbone model, Aug(.) augmentation strategy that creates two augmentations per image with the same cropping and flipping operations
Input:
Randomly initialise transformer decoder head ge, imput queries Q, class prototypes C and generate class assignment matrix Φ;
while << NE do Randomly split T into B mini-batches; for (Is, yb) E {Ti,..., T....., do
X = Ang(ra)
= Oyn
if using background class prototypes then
== {v, [1, ... 1,.., 1]);
Add the background class to all images.
end if
Z f(X);
P ge Q);
p(u P)) = a
p C/T2) Qwi
p( 2)
2. P/TT)
p(v(2) Σ p(r p:)((PP ():
P(y P) = MaxPool(p(rrg (p(r)P));
P(y Z)) = MaxPool(p(v(2))
Compute aux loss Laux Li-discrim + Lcoutrast + CCARL: Compute final loss L = Leluster + Liiiserim + Lauxi Minimise loss C by updating 0, C and Q;
end for
i=i+1;
end while
22

---

Published in Transactions on Machine Learning Research (11/2025)
B
Visualising class prototypes with exemplars
FGVC Aircraft.
(b) Stanford Cars
(c) CUB200
Figure S5: Class prototype exemplars. Image prototypes from the training data with smallest cosine distance the class prototypes associated with each label the FGCV Aircraft, Stanford Cars and CUB200 datasets.
Visualisation of class prototypes. In Fig. S5 we show the class prototype exemplars for all classes within the FGCV Aircraft, Stanford Cars and CUB200 datasets. The class prototype exemplars are given by the image prototypes from the training dataset with the smallest. cosine distance to the class prototypes for each
23

---

Published in Transactions on Machine Learning Research (11/2025)
label. When visualising the class prototypes in this way, observed that not all of them used by ComFe to make a prediction. For example, for the CUB200 dataset only one class prototype is used to learn most bird species, and the other class prototypes are located in regions with a low cosine similarity to the rest of the training dataset. For the CIFAR-10 dataset, which has a much larger number of images per class, we find that two class protolypes per class are commonly used. When the number of class protolypes are reduced our ablation study in Section D this has minimal impacts on the accuracy of ComFe
(a) Ovenbird
(b) Tennessee warbler
Figure S6: Example visualizations of class prototypes and exemplars. Five exemplars shown for each class prototype.
Evaluating model reasoning. Visualising class prototypes and exemplars illuminates the model's reasoning process. As an example, Fig. shows the Tennessce warbler and Ovenbird class prototypes from the CUB-200 dataset using five exemplars. Although three class protolypes are fitted, two were relevant. for classifying images for these classes, unlike most others which only required one. For the Tennessee warbler class, the first class prototype clearly relates to the chest and back plumage of a Tennessee warbler, while the second class prototype is Cedar warwing. This shows the model is likely to misclassify the latter bird, and this stems from the CUB-200 training dataset containing an image of a Cedar warwing misclassified as a Tennessee warbler. A similar phenomenon is observed for the Ovenbird class, which has a spurious image in the training data of a thllow-throated warbler. Cases where spurious background features are used classify images (Fig. S11 and Fig. S12 are also observable in this way.
24

---

Published in Transactions on Machine Learning Research (11/2025)
C Additional results
(a) Example PIP-Net explanation repro- duced from Nauta et al. 2023).
Por Poweuss
Cls wegit Wesle Skalstty
05707-7
45
+22
32'08 26
E-
(b) Example ProtoPFormer explanation re-
Iaesll banttng cluss protolype
Frihio bunting class prototype
produced from (Xue et al. 2024
Mlash lird
prototype
Stan. scure 8.10, B00
8tm. acone B.RS,
Sim.
D.97.
Why is bird classifieed Indigo Bunting?
Background claze prototyra Sim. ecore
Backgroedd claes prototype Sim, score
Actendie Sodiatty
Hackyround prosotype
Background protatypp Sim.
Global Branch
Teoigo Berting
Backgroend
Backgroand
Backgroond
Background clees protolype
class
protatype 3im. acore
8im.
Stmn.
Sim. score
14.93
Hackyround
Barkyround class peololype Sim. seuce 0:00
Backgrvend claus prototype
prototypa
Local Branch
Todipo Bexting
Stm. scoce Balk
Prediction:
Prediction: Indigo hunting
Prediction:
Prediction: Americon crow
Lazuli bunting
Rol uniggaa Maaol tind
1L.I1
Tela paantt En Snd Banking 12.26 Join Thecklenn Incigs Bentiegr 3nd pulto to Bei Ctccki
Figure S7: Comparison of ComFe explanations versus other work. Example of ComFe classifying a Lazuli bunting and Indigo bunting, alongside examples provided for PIP-Net and ProtoPFormer. It also shown how modifying a red winged black bird changes the predicted class, as expected (Paul et al. 2024
Visual comparisons to previous interpretable frameworks. Fig. S7 compares ComFe explanations to that produced by PIP-Net [Nauta et al., 2023). a key inspiration for the approach. The similarity score that matches prototypes in PIP-Net analogous to the similarity score bet ween image prototypes and class prototypes in ComFe. While PIP-Net learns the association between prototypes and classes through the class weight matrix, ComFe assigns class prototypes to particular classes to simplify the fitting process. ComFe further provides detailed segmentation masks for the image prototypes, class prototype exemplars and prediction outputs, whereas PIP-Net is designed identify informative patches.
We also include an example for ProtoPFormer (Xue et al. 2024) in Fig, S7 The explanation provided by ProtoPFormer uses global and local branches, which are all summed together to provide an overall prediction. With the hyperparameters selected for ComFe, the scale of the ComFe explanations most similarly matches the global branch. However, finer grained explanations may be possible with ComFe by increasing the number of image prototypes or training finer-grained backbone.
In Fig. S7 we further include example considered in INTR PPaul et al. 2024), where an image of a Red winged black bird is altered to remove the red patch on the wings. In this case ComFe behaves as expected, adentifying the red patch as key to predicting the Red winged black hird class, as per the visualised exemplars, and classifies the image as an American crow when the red patch is removed. This was also the behaviour observed under the INTR approach (Paul et al. 2024).
ComFe is able to localise salient image features across range of datasets. Fig. S8 shows that ComFe learns to identify relevant features for the classes of interest across range of datasets, from small low
25

---

Published in Transactions on Machine Learning Research (11/2025)
(a) IN-1K
(b) C10
(c) C100
(d) Flowers
Figure S8: Class confidence heatmaps. Informative regions for ComFe class prodictions across randomly selected validation images from ImageNet, CIPAR-10/100 and Flowers-102 102.
resolution datasets with few classes like CIFAR-10, to large high resolution datasets with a thousand classes such Net.
(b)
(d)
(e)
Input Image
Image Prototypes
Class
Confidence
Figure S9: Faithfulness. Example ComFe predictions using DINOv2 VIT-S/14 backbone from the Oxford Pets dataset that have been altered to remove the informative regions, exclude other regions that potentially might have been informative.
The predictions of salient image features provided by ComFe are faithful under image manip- ulation. Fig. shows that when parts of image are removed that not identified as important to classification, ComFe provides similar output. It also shows than when the regions of image that are used to make a prediction are removed, ComFe has low confidence that any particular class is present and will classify the image as the background class (i.e. predict that no class present). This reflects the training process of the DINOv2 backbones, which uses a self-supervised approach that encourages patches to have similar embeddings even when other parts of the image were removed, and other ViT backbones may less faithful.
Table S6: Background class prototypes. Performance of Coml'e with and without background class prototypes versus a linear head.
Head
Backbone
Background
Test Dataset
Prototypes
IN-IK
IN-V2
Sketch
IN-R
IN-A
Linear Oquab al. 2024
DINOv2 VIT-S/14
81.1
70.9
41.2
37.5
18.9
Comfe
DINOv2 VIT-S/14 (f)
3000
83.0
73.4
45.3
42.1
26.4
Comfe
DINO,2 VIT S/14 (I)
82.6
73.0
44.4
41.6
24.0
26

---

Published in Transactions on Machine Learning Research (11/2025)
Background class prototypes improve the generalisability and robustness of ComFe. Table S6
shows that with and without background classes, ComFe obtains better performance, generalisation and robustness on Image.Net compared to non-interpretable linear head. As shown in previous work (Jiang et al,
2021: Zhao ct al. 2023); the performance and robustness of ViT models can be improved by considering the patch lokens as well as the class lokens. We also find that the use of background class protolypes slightly improves the performance of ComFe, in addition to providing a mechanism for identifying the salient regions of an image when making a prediction.
Table S7: Registers. Performance of ComFe on the DINOv2 backbones with DDrcett et al. 2024 and without (Oquab et al. 2024 registers.
Backbone
Dataset
IN-1K
C10
C100
Food
CUB
Pots
Cars
Airer
Flowers
DINOv2 VIT-S/14 (f)
83.0
98.3
89.2
92.1
87.6
94.6
91.1
77.5
99.0
DINOv2 VIT-S/14 (f) w//reg
82.9
98.2
89.0
91.6
87.8
90.5
76.5
98.8
DINOv2 VIT-L,/14 (f)
86..7
99.4
93.6
94.6
80.2
95.9
93.6
83.9
99.4
DINOv2 VIT-L/14 (I) w/reg
87.2
99.5
94.6
95.6
94.5
85.6
99.6
Including registers in the DINOv2 VIT model can improve the performance of ComFe for large models. Further work on the ViT backbone has found that including register tokens (Darcet el al.j 2024) can prevent embedding artefacts such as patch tokens with large norms, which improves results for bigger models like the ViT-L and larger variants. In Table S7 it is shown that the performance of ComFe is improved for the ViT-L model with registers. Fig. S10 shows that for the ViT-L backbone without registers, the informative patches can be irregular and not contain the relevant object of interest in the image, and the image prototypes can be less natural. However, when registers are used the image prototypes are more descriptive and we were unable to find cases where relevant features from the image were not identified by ComFe.
For some initialisations, ComFe transparently classifies using background features. The features that are found to be informative by ComFe can vary depending on the random seed chosen for the algorithm. As shown in Fig. S11 for the Food-101 dataset, particular initialisation scods result in ComFe finding background features that can be used to accurately classify the training and validation data. However, there appears to be trend across both the Food-101 (Fig. SI1) and Oxford Pets (Fig. S12) datasets that model initialisations resulting in more salient image features being identified have better performance.
We observe that the DINOv2 patch embeddings are influenced by their context. For example, in Fig. S13 we use k-means clustering to visualise the DINOv2 patch embeddings of two images containing a white plate. We find that the surface the plates rest on both fall in the background cluster (yellow), but the plate containing churros is placed in the red cluster while the plate containing a burger is placed in the blue cluster. When ComFe obtains good performance in cases the salient parts of an image are classified as the background, this likely reflects this context leakage from the backbone model.
Prototypes being selected from the background of an image was a challenge in the original ProtoPNet approach, where a pruning process was employed to remove most of the non-informative prototypes from the reasoning process (Chen et al., 2019). and this can still occur in more recent approaches such as PIP-Net (Nauta el al. 2023).
Table S8: Further results for alternate backbones Results for fitting ComFee on the CUB-200 dataset with different frozen ViT-B backbones. Training approaches include supervised on IN-21K (AugReg), self-supervised on IN-1K (DINO, MAE) and foundation models (CLIP, DINOv2 and DINOv2 w/reg).
CLIP
DINO
AugReg
MAE
DINOv2
DINOv2 w/reg
Radford et al. 2021
Caron et al. 2021
Steiner of al.
2022
He
2022
Oquab
2024
Darcet et al. 2024
CUB-200
83.4
80.2
85.6
76.9
85.8
89.4
27

---

Published in Transactions on Machine Learning Research (11/2025)
(a) Class confidence
(b) Image prototypes
(c) Class confidence
(d) Image prototypes registers
w registers
Figure S10: Interpretability with registers. Example ComFe image prototypes and class predictions with the DINOv2 VYT-L/14 backbone trained on ImageNet. The left two columns are obtained using the network without registers, and the right two columns the network that includes registers.
ComFe heads can provide interpretations for a range of different backbones. Table SS shows the performance of ComFe with other pretrained ViT backbones. Higher quality backbones generally perform better, and the image prototypes and features found be informative vary between them (see Fig. S14h. We note that some backbones are more likely to background features as informative (DINO and MAE) while
28

---

Published in Transactions on Machine Learning Research (11/2025)
(a) Seed 92.1%
(b) Seed 2: 92.2%
(c) Seed 4: 92.3%
(d) Seed 92.4%
Figure S11: Food-10P seeds. Class confidence maps using ComFe models initialised with different seeds for an image from the Food-101 validation set..
(a) Seed 1: 94.6%
(b) Seed 4: 94.8%
(c) Seed 3: 94.8%
(d) Seed 2: 95.3%
Figure S12: Oxford Pets seeds. Class predictions using ComFe models initialised with different seeds for image from the Oxford Pets validation set.
(a) Churros
(b) Clustering with DINOv2
Burger
Figure S13: Example image of churros and burger on a white plate from the F01 validation set. The images in the middle are overlayed with a k-means clustering of the combined patch embeddings of the two images.
the results for other backbones highlight artefact patches that may contain global information DDrcett et al. 2024) (AugReg and CLIP).
29

---

Published in Transactions on Machine Learning Research (11/2025)
CLIP RRaford et al.
2021
(b) DINO Caron et al. 2021
(c) AugReg Steiner et al.
2022
(d) MAE He et al.
2022
(e) DINOv2 Oquab et
2024
(f) DINOv2 w /rreg
Darcet et al. 2024
Figure $14: Image prototypes and class confidence heatmaps. Visualizations the outputs of ComFe with different pretrained backbones.
30

---

Published in Transactions on Machine Learning Research (11/2025)
D Ablation studies
In this section we undertake an ablation study on the hyperparameterss of ComPe using the FGVC Aireraft, Oxford Pets and Stanford Cars datasets. For each set of parameters we consider five runs and report the mean and standard deviation of the validation accuracy, The bolded set of parameters reported in the tables is the default one used throughout this paper.
Table S9: Ablation: Transformer decoder layers. Performance of ComFe on the DINOv2 VIT-S/14 backbone with different numbers of transformer decoder layers.
Transformer
Datasct
Layers
Airer
Pets
Cars
1
75.3+1.3
94.5:1044
90.3+0.4
2
77.2+0.9
94.7++0.4
91.1+0.1
4
77.110.6
94.8.1..4
91.610.2
6
77.9
94.4+0.3
91.2+0.2
Tranformer decoder layers. Table 59 shows that for the Stanford Cars and the FGVC Aircraft datasets, including more transformer decoder layers improves the average performance of ComFee across number of seeds. However, for the Oxford Pets dataset there appears to be no significant relationship between accuracy and the size of the transformer decoder.
Table S10: Ablation: Auxilary loss. Performance of ComFe on the DINOv2 VIT-S/14 backbone with elements of the loss removed. In the loss term columns, an denotes when term is removed and a Y denotes when a term is kept.
Loss term
Dataset
L--discrim
LCARL
Lonntrast
Aircr
Pets
Cars
N
N
1.3
94.1+0088
90.3
N
Y
76.4+2.2
94.740.6
90.5++0.7
Y
N
Y
76.8+11.0
94.8++0.6
90.9++0.5
Y
76.7+0.6
94.6:10.3
90.8
Y
Y
Y
76.8+0.7
94.8++0.4
91.0+#0.2
Loss terms. In Table S10 we explore the impact of removing loss terms on the performance of ComFe across the Stanford Cars, FGVC Aircraft and Oxford Pets datasets. We find that that they have a marginal impact on the performance of the ComFe models, but do provide small performance improvement.
Table S11: Ablation: Temperature. Performance of Coml'e on the DINOv2 VIT-S/14 backbone with different 7i and 72 concentration parameters.
Concentration
Datasct
Aircr
Pets
Cars
0.05
0.02
76.8+1.5
94.910.4
90.540.5
0.10
0.01
76.5+0.9
95.0+0.3
90.6+0.00
0.10
0.02
77.0+:0.4
94.910.4
91.0+10.3
0.10
0.05
77.8+
94.9++0.5
91.1+0.66
0.20
0.02
77.411.1
95.110.2
91.240.1
31

---

Published in Transactions on Machine Learning Research (11/2025)
Concentration parameters. In Table S11 explore the impact of the concentration parameters тι and T2 the performance of ComFe. While there is some scope for optimising the choice of these parameters for particular datasets, the performance of ComFe does not appear to be particularly sensitive to these parameters.
Table S12: Ablation: Label smoothing. Performance of ComFe on the DINOv2 ViT-S/14 backbone with different amounts of label smoothing (a).
Label smoothing
Dataset
Airer
Pets
Cars
0.0
77.2
94.8+0.3
91.0+t0.3
0.1
77.3+-0.7
94.8+0.4
91.0++.2
Label smoothing. In Table S12 explore the impact of the label smoothing parameter a the performance of ComFee We find that label smoothing has little impact accuracy, and is not necessary for the success of the method.
Table S13: Ablation: Number of prototypes. Performance of Com on the DINOv2 ViT-S/14 backbone with different numbers of prototypes.
Number of prototypes
Dataset
Np
Nc/c
Aircr
Pets
Cars
5
76.7+0.7
94.6+0.4
91.0+t..4
5
77.6+0.8
94.8+0.2
91.0+0.33
10
76.1+1.2
94.8+0.4
91.3++0.3
Number of prototypes. In Table $13 we explore the impact of the number of image protolypes Np and class prototypes per label Ne/c on the performance of ComFe. While there may be some marginal performance gains for particular datasets for choosing particular set of prototypes, overall these have only small impact on the accuracy of ComFe.
Table $14: Subsetted robustness. Performance of ComFe with and without background class prototypes ImageNet robustness and generalisation benchmarks. We report accuracy with a subset of the logits, restricted to only the classes present in the ImageNet-R (Hendrycks et al. 2021a) and ImageNet-A (Hendrycks)
et al. 2021b) test sets.
Backbone
Background
Test Dataset
Prototypes
IN-R
IN-A
DINOv2 VIT-S/14 (f)
3000
57.7
42.6
DINOY2 VIT-B/14 (f)
3000
67.1
61.7
DINOv2 VIT-L/144 (f)
3000
72.5
71.9
32


Reviews:
[{"review_id": "HfJNoRt7mI", "reviewer": "Reviewer_cAqX", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a novel classifier head that can be attached to frozen, pretrained vision transformers, offering both improved classification accuracy and inherent interpretability. The authors ground their method in statistical theory which expands upon prior Proto approaches and demonstrate the efficacy of their approach with multiple key vision datasets, including ImageNet-1k. Additionally, the authors explore interpretability with compelling examples and corresponding in-depth analysis.\n\nStrengths:\n\n-\tThe proposed method is well motivated and offers a reasonable approach to training dataset specific classifier heads on frozen ViT backbones.\n-\tThe interpretability results are compelling, demonstrating applicability to multiple model types and scales.\n-\tPerformance is rigorously evaluated, for both classification and interpretability, covering multiple benchmarks.\n\nWeaknesses:\n\n-\tSome aspects of the paper were unclear or missing context.\n-\tThe initialization strategy is not well described, which is significant due to the variation observed with training seeds. \n-\tThe approach is brittle, showing sensitivity to initial conditions.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The authors successfully demonstrate matching or improved performance compared with prior approaches, both for classification and interpretability. Evidence is supported by top-1 accuracy, visual confidence maps, and evaluation on FunnyBirds.", "audience_yn": "Yes", "audience_explanation": "The interpretability results are compelling, with many visual examples and in-depth analysis. The authors also discuss how the heads learn to classify class components, which do not end up contributing to the final classification; an interesting result. Furthermore, the authors describe how the interpretability analysis led to the discovery of misclassified samples in the CUB-200 dataset.", "requested_changes": "-\tThe introduction should more clearly state that ComFe is primarily a classifier head exhibits interpretability, rather than a post-training interpretability technique. In some sub-fields, interpretability heads are trained to understand a models behavior without being used for downstream predictions.\n-\tThe head architecture should be better described. The authors state that decoder layers with cross-attention are used, but seeing a more specific diagram would be helpful to understanding the activation flow, and what role Q and C play at a glance. \n-\tThe #P column for the linear methods should be included as they are not part of the DINOv2 backbone.\n-\tThe evaluation metric (i.e. top-1) should be clearly stated in the comparison tables – this is only apparent when considering the broader context.\n-\tThe training protocol should be better described. How are the class exemplars and background images chosen? Are they per batch or static for the training run? Is the seed sensitivity based on the parameter initialization, or is it also susceptible to the dataset shuffling? \n- It would be helpful to understand if ComFe is also applicable to semantic segmentation, especially since certain semantic prototypes did not inform the classification and are instead treated as background, leading to incomplete confidence maps. Semantic segmentation on the other hand would penalize such mapping.", "broader_impact_concerns": "No concern.", "additional_comments": "Questions:\n\n-\tThe interpretability seed sensitivity examples demonstrate similar classification performance with less informative or nonsensical confidence maps. Does this undermine the significance of an interpretable method, or does it lead to clear sample-specific classification errors? \n-\tThere was some discussion in the paper about training multiple heads and picking the best one to address the initialization sensitivity. Can class entries be stitched together from multiple training runs? Otherwise, could there be cases where a more variable dataset such as ImageNet results in some class confidence maps having strong semantic overlap, while others do not?\n-\tThe DINOv2 models with registers show better prediction accuracy, but lower completeness. Is this a desirable aspect for an interpretable model, or largely a consequence of the global classification task? For example, detecting the full foreground object may make more sense to a human observer, but appears to not be necessary for classification.", "extra_scores": {}, "extra_text": {}}, {"review_id": "LsINfTM00l", "reviewer": "Reviewer_aikr", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "It introduces a classifier head that extracts image prototypes on the fly and compares them with learned class prototypes to build a scalable interpretable classifier. The method is then compared against other prototype-based approaches as well as non-interpretable baselines.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The methodology, particularly the use of the vMF distribution for hierarchically modeling the prototypes/patches, feels novel. Since the core idea is scalability, the authors demonstrate that ComFe achieves comparable performance to prototype-based methods while requiring significantly less storage.", "audience_yn": "Yes", "audience_explanation": "Since the classifier head can be added to any existing ViT, this work should be of interest to researchers focused on interpretability and vision transformers.", "requested_changes": "The authors claim that previous methods require hyperparameter tuning across datasets, whereas ComFe does not. However, doesn’t this also apply to ComFe? Both $N_P$ (the number of image prototypes) and $N_c$ (the number of class prototypes per class) appear to be dataset-dependent. Intuitively, more complex datasets would require larger values, while simpler datasets might work with fewer.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "UxhUHe7DGm", "reviewer": "Reviewer_PTeC", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "*Summary:*  \nThis paper proposes ComFe an interpretable classification head for pretrained vision transformers. It represents each image by a few image prototypes and classifies the image by their similarity to learned class prototype patches. The authors show that ComFe is competitive in terms of classification accuracy with non-interpretable methods and other interpretable methods, but scales better. The authors provide a detailed ablation of their proposed method.\n\n*Strengths:*  \n(S1) Interpretability is an important research direction, especially when combined with large, pretrained vision transformers. ComFe addresses this problem.  \n(S2) The empirical results seem convincing.   \n(S3) Through the reduction of the number of image patches to a fixed number of image prototypes, ComFe is memory-lightweight and scales to larger datasets than competitors.\n\n\n*Weaknesses*:  \n(W1) No uncertainties are reported anywhere. Was the method (and the competitors) trained for multiple seeds?  \n(W2) The method presentation is unclear in several places (see below).  \n(W3) The conceptual comparison with competitors is fragmented over three places (beginning of discussion, end of section 3, section 2).   \n(W4) Some qualitative findings should be discussed / explored in more depth (see below).", "claims_and_evidence_yn": "No", "claims_explanation": "While I am not an expert in the field of interpretable classification, to me the main benchmarks look convincing (both in terms of datasets and competitors) and the authors provide several ablations (S2).\n\nHowever, several types of evidence are missing / should be expanded:  \n(E1, see W1) Uncertainties should be reported at least for ComFe in the main benchmarks to assess how large the improvements are over the variability of the method.   \n(E2, see W4) What can background features be used to accurately predict the classes (Fig S10, S11)? How often did this happen? The authors conjecture that a multi-head strategy might solve the issue, but do not evaluate such a strategy.  \n(E3, see W4) Why does the ViT-B/14 model without registers produce background artifacts, but the ViT-S/14 model without registers does not? Do the ComFe models in the main benchmarks (Tab 1, 2) perhaps also produce background artifacts.   \n(E4, see W4) Some explanations of ComFe seem counterintuitive and warrant some more explanation. For instance, I would have assumed that a bird's head is quite informative for its classification, but ComFe assigns the head to the background label. Could the be the case because each class only has three class prototypes and bird heads simply are not among the three most discriminative class features? A related question is why the authors chose to assign as many class prototypes to real classes as to the background. Is it because backgrounds can be varied?  \n\n\n\n(E5, see W2) I had some serious difficulty understanding the method exposition. I am sorry if this is due to a misunderstanding on my side. Equation (5) is impossible to derive without the appendix, because some key assumptions are not mentioned in the main paper, namely that the class and image prototype priors are uniform. Some of the datasets are class balanced, like CIFAR or ImageNet. Others have a strong class imbalance, like Flowers. Why can we assume uniform class prior even in these cases?  \n\n(E6, see W2) I am confused by the treatment of\n\n $\\hat{P}_{j}$ \n\n(I'll omit the hat now as it does not get parsed properly inline) . In Eq 4 it seems as if $P_j$ is a continuous random variable supported on the hypersphere (its distribution is a mixture of von Mises-Fisher distributions). But in Eq (5) when marginalizing over $P_j$ it seems as if one only needs to marginalize over the discrete set {$1, \\ldots, N_{P}$}. Which interpretation is correct? If  $P_j$  is continuous, then why do we not need to integrate over the whole hypersphere in Eq (5)? Moreover, in this case one could compute the product of two vMF distributions in closed form.   \n\n(E7, see W2) Why can we assume a uniform prior over the image prototypes $P_j$? Especially if they are continuous, their distribution is given by the class prior $p(\\nu)$ and Eq (4) and we are not free to simply assume its distribution. Or does this mean that we assume a uniform prior over the set {$1, ..., N_P$}? Similarly, how can we assume the distribution of $P_j$ by Eq (4) but then compute $P_j$ via Eq (7), which most likely will not amount to a mixture of vMFs.   \n(E8, see W2) Even if we assume uniform class priors, I do not understand the derivation from Eq (16) to Eq (17). Each term $p(P_{j:} | \\nu_l)$ depends on $\\phi_{:,\\nu_l}$ in Eq (16), but Eq (17) only depends on $\\phi_\\nu$, not on $\\phi_{\\nu_l}$.  \n(E9, see W2) I did not get Eq (34) and (35) at all.   \n(E10, see W2) How are the class confidence heat maps computed?  \n(E11, see W2) How can one compute the component feature maps like in Fig 1, 2, 3? In particular, where does the spatial information come from? If I understand Eq (7) correctly, the i-th component feature map depends on all patch embeddings of the image $Z$ jointly and not in a spatially localized way.   \n(E12, see W4) Even if the component feature maps (and the class confidence maps) can be computed on a patch level, why do they seem to be on pixel-level in Figs 1, 2, 3? How were the pixel-level maps computed?  \n(E13, see W2) How are the class prototype exemplars computed? If I understand correctly, the class prototypes have the same dimensionality as an image patch embedding. But a class exemplar is a whole image, so it has many embedded patches. Are those images selected for which one patch embedding has high cosine similarity with a class prototype? Or one for which one prototype image has high cosine similarity with the class prototype?  \n\n(E14, see W2): It would be nice to add the auxiliary losses to the main text and to perform an ablation that removes more than one of them at the same time.   \n(E15, see W2): I get that on datasets such as CUBS or Flowers, where each image looks structurally similar, the image prototypes can identify semantic regions such as head, wings, body of a bird and background. But on CIFAR or ImageNet the images are much more varied, yet the five queries are still shared across all images, right? Are the feature components still interpretable?  \n\n\nI am happy to change my evaluation in this category if (E5-E13) get resolved.", "audience_yn": "Yes", "audience_explanation": "Making black-box neural network decision interpretable is an important research direction and crucial for trust- or safety-critical applications (S1).  Moreover, ComFe can leverage pretrained, frozen vision transformer backbones, which have recently gained a lot of attention (S1). Scaling to ImageNet-size datasets is also important (S2). Together, I am confident that the ideas of this paper will be interesting to several people in TMLR's audience.", "requested_changes": "Crucial:  \n- My main concern is with the method description (E5-E13) and I think more detailed explanations / answers to my questions are critical before I can recommend acceptance.  \n- Uncertainties should also be reported (E1).\n\nNice-to-have:  \n- Answers to (E2-E4, E15).  \n- Ablations as in (E14).  \n- Restructuring the conceptual comparisons in on place (W3).", "broader_impact_concerns": "I do not see any ethical implementations of the work that would require adding a Broader Impact Statement.", "additional_comments": "C1: The caption of Tab S14 mentions ComFe without background class prototypes, but the table seems to only show ComFe with background class prototypes.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)