SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Chiyu Ma*
chiyu.ma.gr@dartmouth.edu
Brandon Zhao*
byzhao@caltech.edu
Chaofan Chen
chaofan.chen@maine.edu
We present ProtoConcepts, a method for interpretable image classification combining deep learning and case-based reasoning using prototypical parts. Existing work in prototype-based image classification uses a "this looks like that" reasoning process, which dissects a test image by finding prototypical parts and combining evidence from these prototypes to make a final classification. However, all of the existing prototypical part-based image classifiers provide only one-to-one comparisons, where a single training image patch serves as a prototype to compare with a part of our test image. With these single-image comparisons, it can often be difficult to identify the underlying concept being compared (e.g., "is it comparing the color or the shape?"). Our proposed method modifies the architecture of prototype-based networks to instead learn prototypical concepts which are visualized using multiple image patches. Having multiple visualizations of the same prototype allows us to more easily identify the concept captured by that prototype (e.g., "the test image and the related training patches are all the same shade of blue"), and allows our model to create richer, more interpretable visual explanations. Our experiments show that our "this looks like those" reasoning process can be applied as a modification to a wide range of existing prototypical image classification networks while achieving comparable accuracy on benchmark datasets.
As machine learning models are increasingly adopted in high-impact, high-stakes domains such as healthcare [2, 13], self-driving cars [24], facial recognition [23], etc., the design of human-interpretable models has become essential for ensuring accountability and transparency of their decisions. In particular, a class of models called prototype networks have combined the power of deep learning with the explainability of case-based reasoning to make accurate, human-interpretable decisions on fine-grained image classification tasks [3, 6, 19, 25, 26, 36]. These models make decisions by dissecting an image into informative feature patches, then comparing them to prototypical parts learned during training. Evidence of similarity to each prototypical part is then combined to make a final classification.
When evaluating the interpretability of an image classification network, it is important to consider not only the model's capability to explain its reasoning process, but also the quality of its explanations. The image features analyzed by prototype networks can encode a variety of visual properties, such as color, shape, texture, contrast, or saturation. Because each prototype in the existing prototype network corresponds exactly to a single patch of a single training image, it can be difficult for human users to understand the exact semantic content underlying a prototype's visualization. Although there has been some work in quantifying the importance of each of these feature categories in prototypes [20], the exact semantic content in an image patch can still remain ambiguous (e.g. "Texture is
Figure 1: Image of a Brown Thrasher and how the ProtoPool (left) and ProtoPool-Concepts (ours, right) explain their decisions. Prototype classifications are made by finding patches in the image similar to learned prototypical parts. Single-visualization methods such as ProtoPool can make visually ambiguous decisions when the semantic features underlying a prototype are unclear. Our method provides clearer explanations by providing multiple visualizations of prototypical concepts found in the test image.
important in this prototype, but what kind of texture in particular?"). The ambiguity underlying prototype visualizations has been shown to cause a gap between visual explanations from prototype networks and human understanding of visual similarity [15]. Additionally, a variety of prototype networks have been proposed in which prototypical parts from a single image can serve as evidence for many different classes, thus reducing the total number of prototypes needed for the network to make classifications [19, 26, 25]. The rationale behind this prototype sharing is that certain distinctive features (e.g. "long beak, solid black head, spotted belly") can be found in multiple classes, and thus do not need to be represented as distinct class-specific copies. However, while reducing the number of prototypes can simplify visual explanations by reducing the amount of considered evidence, these methods can often produce seemingly nonsensical explanations when a training patch from an image is used spuriously as evidence for a different class.
Inspired by the difficulty of identifying the meaning behind single-image prototypes, we propose ProtoConcepts, a novel modification to prototype geometry enabling visualizations of prototypical concepts from multiple training images. This allows human users to better understand the conceptual content of a prototype by observing the features shared among its visualizations. An example of a visual explanation produced by our ProtoConcepts is shown in Figure 1 (right). Unlike previous prototype networks where a prototype is visualized using a single training image patch (Figure 1, left), our ProtoConcepts visualizes each prototype with multiple visualizations. Because our model offers many visual examples for each prototype, the semantic meaning of each prototype can be determined with less ambiguity - we can look at the commonality between the visualized training image patches to infer the semantic meaning of each prototype. Our method can be applied to a wide range of prototype-based networks, resulting in better human interpretability while achieving comparable classification accuracy. Our code is available at https://github.com/Henrymachiyu/This-looks-like-those_ProtoConcepts
Attempting to understand deep neural networks, posthoc explanation techniques such as activation maximization [7, 21, 37, 38], image perturbation [8, 12], and saliency visualizations [30-32, 27] fail to explain a network's reasoning process, and their results can be risky and unreliable [18, 1]. On the other hand, prototype-based networks compare learned latent feature representations, called
prototypes, to the latent representations of an image to perform classification. These models are constrained to be intrinsically interpretable. The Prototypical Part Network (ProtoPNet) [3] uses class-specific prototypes, where a pre-determined number of prototypes are assigned to each class. Each prototype is trained to be similar to feature patches from images of its own class and dissimilar to patches from images of other classes. This gives such class-specific models high discriminative power suitable for fine-grained image classification. In the end, prototype similarity scores are aggregated as positive evidence for each class in a "coresheet." The Transparent Embedding Space Network (TesNet) [36] improved on the class-specific ProtoPNet architecture by transforming the latent space geometry into a hypersphere, and representing prototypes as near-orthogonal basis vectors specific to a class subspace in the Grassmannian manifold.
Other prototype-based networks found that abandoning class-specific prototype constraints allowed for a significant reduction in the number of prototypes, making explanations simpler and allowing prototypes to represent similar visual concepts shared in images of different classes. In [25], a ProtoPNet is first trained with enforced class specificity, and then a novel data-based similarity metric is used to "merge" highly similar prototypes across classes into a single prototype representing multiple classes. In Neural Prototype Trees (ProtoTree) [19], the "scoresheet" of the ProtoPNet is replaced by a binary decision tree, in which the scored presence or absence of a prototype in each node determines the traversal path of the tree. Thus, each prototype in the tree represents either positive or negative evidence for the classes at the leaf nodes of the subtree for which the prototype is in the root node. Finally, in the ProtoPool network [26], the soft assignment of prototypes to classes is learned throughout training, and is eventually transformed into a hard assignment at test time.
Our work also relates to works that try to understand the semantic meaning behind a learned prototype of a prototype-based network. For example, in [20], Nauta et al. developed a method to quantify the importance of various visual characteristics relevant to each prototype. Our approach is complementary to, yet differs from theirs in that we illuminate the concept behind each prototype by presenting multiple visualizations of the same prototype, thereby allowing one to better infer the semantic meaning behind each prototype.
Finally, our work is related to posthoc methods in concept visualization such as TCAV [14] and ACE [9]. Posthoc methods do not show any part of the model's actual reasoning process. Worse, their visual explanations may not be faithful to the model's reasoning process. Sometimes, multiple concepts have the same concept vector [as discussed in 5], and concepts may not be clustered in the latent space (since they are not trained to be that way), leading to meaningless concept vectors. Instead, our method is inherently interpretable, and derives its visual concept-based explanations directly from the decision-making process.
We begin with a brief general review of traditional prototype-based models, then propose our method, which represents prototypes as interpretable concepts with multiple visual explanations. We show how our method can be applied as a modification to existing prototype-based networks, and introduce a novel training algorithm to learn meaningful prototypical concepts. Details for the implementation of specific prototype-based classification models are in the Experiments section.
3.1 Prototype Architectures
Existing prototype-based networks [3, 19, 25, 26, 36] can be understood architecturally as the combination of three components. First is a convolutional neural network (CNN)  $f$  parameterized by weights  $w_{\mathrm{conv}}$ , which acts as a latent feature extractor on input images  $\mathbf{x}$ . Next the prototype layer  $p$  considers the convolutional output  $f(\mathbf{x})$  as  $\mathrm{H} \times \mathrm{W}$  "patch" vectors  $z_i \in \mathbb{R}^D$ , each of which corresponds to features from a patch of the original image space. Each patch is compared to  $m$  learned prototype vectors  $\mathbf{P} = \{\mathbf{p}_j\}_{j=1}^m$ , where each prototype  $\mathbf{p}_j \in \mathbb{R}^D$  lies in the same space as the image's latent feature patches. The prototype layer  $p$  calculates a similarity score  $g_{\mathbf{p}_j} \circ f(\mathbf{x})$  for each prototype  $\mathbf{p}_j$  which is monotonically decreasing with respect to the distance between  $\mathbf{p}_j$  and the closest image patch  $\tilde{\mathbf{z}} \in f(\mathbf{x})$  in the model's feature space. Finally, the prototype layer  $g_{\mathbf{p}}$  is followed by a prototype class assignment mechanism  $h$  which assigns evidence logits to each class based on the calculated prototype similarity values  $g_{\mathbf{p}_j} \circ f(\mathbf{x})$  and class assignment weights  $w_h$ . The evidence
Figure 2: Architecture of ProtoConcepts. The convolutional layer includes all types of architectures, including ResNet, VGG and DenseNet. Evidence Layer  $h$  consists of a prototype class assignment mechanism and a fully connected layer. Depending on the specific prototype-based architecture, TesNet and ProtoPNet would have a one-to-one prototype class assignment mechanism, while ProtoPool and ProtoPShare would have a shareable prototype mechanism among the classes.
logits are then normalized by a softmax to yield predicted probabilities for a given image belonging to each class.
In the final model, each prototype vector  $\mathbf{p}_j$  is restricted so that it is equal to a latent patch  $\tilde{\mathbf{z}}\in f(\mathbf{x}_i)$  from some training image  $\mathbf{x}_i$ . Thus, the model's reasoning process can be interpreted visually as an evidence scoresheet of comparisons between test image patches and salient features underlying previously-seen training image patches. We propose to modify the geometry of prototypes  $\mathbf{p}_j$  by instead representing them as a ball in the latent space encompassing multiple training image patches. Mathematically, each prototype is a ball  $\mathbf{p}_j = B(\mathbf{c}_j,r_j)$  around a central vector  $\mathbf{c}_j\in \mathbb{R}^D$  with radius  $r_j\in \mathbb{R}^+$ . This ball representation has the property that the similarity scores of any latent patch vector  $\mathbf{z}_i\in \mathbb{R}^D$  and any vector  $\mathbf{d}_j\in \mathbf{p}_j$  in the prototypical ball are constrained to be the same. Hence, with this design, as long as a prototypical ball  $\mathbf{p}_j$  contains a number of latent training patches, we can visualize the prototype  $\mathbf{p}_j$  by visualizing any of those latent training patches it contains. This set representation allows our prototypes to represent concepts found in all latent training patches contained within a prototypical ball, thereby providing richer semantic explanations than that of visualizing just a single latent training patch. Implementing our concept geometry on a prototype-based model entails minor changes to the architecture and loss; model-specific implementation details are described in Sec. 4.1.1.
3.2 Prototypical Concept Representations
The prototype layer  $g_{\mathbf{P}_j}$  of previous prototype-based classification models uses one of two activation functions for calculating similarity scores between prototypes and image patches: a log-based activation on inverse  $\ell^2$  distance, and a cosine similarity score. In this section, we formalize our ball representation of prototypical concepts with respect to the two corresponding latent spaces.
Log-based Activation: Models such as ProtoPNet [3] and ProtoPool [26] use a log-based function  $g^{\log}$  to compute a similarity score, which is monotonically decreasing with respect to the  $\ell^2$  distance. In particular, for the latent representation  $\mathbf{z} = f(\mathbf{x})$  of an input image  $\mathbf{x}$ , this function computes  $g_{\mathbf{p}_j}^{\log}(\mathbf{z}) = \max_{\tilde{\mathbf{z}} \in \mathrm{patches}(\mathbf{z})} \log((\|\tilde{\mathbf{z}} - p_j\|_2^2 + 1) / (\|\tilde{\mathbf{z}} - p_j\|_2^2 + \epsilon))$  with a small constant  $\epsilon$  to avoid numerical overflow. For prototypical concepts, we propose an analogous similarity function
corresponding to the distance to center  $\mathbf{c}_j$  of prototypical ball  $\mathbf{p}_j = B(\mathbf{c}_j,r_j)$ , bounded below by  $r_j$ :
$$
\tilde {g} _ {\mathbf {p} _ {j}} ^ {\log} (\mathbf {z}) = \max  _ {\tilde {\mathbf {z}} \in \operatorname {p a t c h e s} (\mathbf {z})} \min  \left(\log \left(\frac {\| \tilde {\mathbf {z}} - \mathbf {c} _ {j} \| _ {2} ^ {2} + 1}{\| \tilde {\mathbf {z}} - \mathbf {c} _ {j} \| _ {2} ^ {2} + \epsilon}\right), \log \left(\frac {r _ {j} + 1}{r _ {j} + \epsilon}\right)\right). \tag {1}
$$
Cosine Similarity: Models such as TesNet [36] and Deformable ProtoPNet [6] have found that representing prototypes as unit vectors on a hypersphere can improve the accuracy of prototype-based models. In these models, the prototype layer compares prototypes to latent image patches via the cosine similarity function  $g_{\mathbf{p}_j}^{\cos}(\mathbf{z}) = \max_{\tilde{\mathbf{z}} \in \mathrm{patches}(\mathbf{z})} \tilde{\mathbf{z}}^T \mathbf{p}_j / (\| \tilde{\mathbf{z}} \|_2 \| \mathbf{p}_j \|_2)$ . This cosine similarity can be interpreted as a cosine activation function on the angular distance between normalized vectors on the hypersphere. For models with a hypersphere-based latent space, we replace the cosine similarity function  $g_{\mathbf{p}_j}^{\cos}$  by calculating the cosine similarity to the ball  $B(\mathbf{c}_j, r_j)$  on the hypersphere:
$$
\tilde {g} _ {\mathbf {p} _ {j}} ^ {\cos} (\mathbf {z}) = \max  _ {\tilde {\mathbf {z}} \in \operatorname {p a t c h e s} (\mathbf {z})} \min  \left(\frac {\tilde {\mathbf {z}} ^ {T} \mathbf {c} _ {j}}{\| \tilde {\mathbf {z}} \| _ {2} \| \mathbf {c} _ {j} \| _ {2}}, \cos (r _ {j})\right). \tag {2}
$$
Both Eqs. (1) and (2) ensure that for any patch vector  $\mathbf{z}_i\in \mathbb{R}^D$ , if  $\mathbf{z}_i$  lies outside the ball  $\mathbf{p}_j$ , its distance to  $\mathbf{p}_j$  is the distance to its center  $\mathbf{c}_j$ , and if  $\mathbf{z}_i$  lies inside  $\mathbf{p}_j$ , its distance to  $\mathbf{p}_j$  is kept the same as the radius of the ball. During training, a pass-through estimator of each activation function is used to avoid a zero gradient when training patches lie within the prototypical ball.
3.3 Training Algorithm
The training of prototype-based networks is typically divided into three stages: (1) stochastic gradient descent (SGD) of layers before the last layer; (2) projection of prototypes; (3) optimization of the last layer  $h$ . Because of the ball representation of prototypical concepts, the second stage can be skipped as prototypical balls are visualized by the training patches they contain at the end of the first step.
SGD of layers before last layer: The first training stage prototype-based networks aims to learn a meaningful latent space that clusters important training patches near semantically similar prototypes. To do this, a joint optimization problem is solved to optimize the network parameters using SGD while keeping the last layer weight matrix fixed. To learn semantically meaningful prototypical concepts, we introduce two novel loss terms,  $\mathcal{L}_{\mathrm{Clstk}}$  and  $\mathcal{L}_{\mathrm{Rad}}$ :
$$
\mathcal {L} _ {\mathrm {C l s t k}} = \frac {1}{n} \sum_ {i = 1} ^ {n} \sum_ {j = 1} ^ {k} \min  _ {\substack {\mathbf {p} _ {j} \in \mathbf {P} _ {u _ {i}} \\ \mathbf {p} _ {j} \neq \mathbf {p} _ {1}, \dots , \mathbf {p} _ {j - 1}}} \min  _ {\tilde {\mathbf {z}} \in \text {p a t c h e s} (f (\mathbf {x} _ {i}))} d (\tilde {\mathbf {z}}, \mathbf {p} _ {j}), \mathcal {L} _ {\mathrm {R a d}} = \sum_ {\substack {\mathbf {p} _ {j} \in \mathbf {P} \\ \mathbf {p} _ {j} = B (\mathbf {c} _ {j}, r _ {j})}} r _ {j} ^ {2}. \tag{3}
$$
Here  $d$  is used as the distance function corresponding to the respective latent space geometry of the ProtoConcepts module. The minimization of  $\mathcal{L}_{\mathrm{Clstk}}$  encourages  $k$  prototypical concepts of the correct class to be close to at least one latent patch of each training image. The specific value of  $k$  is treated as a hyperparameter and is chosen by cross-validation. As opposed to the traditional prototype cluster loss [3], which only penalizes the distance of the closest prototype to a latent training patch, we find that our top- $k$  formulation is essential for ensuring that prototypical concept balls contain multiple training patches. The minimization of  $\mathcal{L}_{\mathrm{Rad}}$  encourages prototypical concept balls to be compact. This allows learned prototypical concepts to encapsulate more specific semantic features important for classification.
Prototypical Concept Visualisation: In previous prototype-based networks, the trained prototype vector  $p_j$  is projected (pushed) to the nearest training patch to produce a single patch visualization. However, since we use a set representation for prototypical concepts, it is no longer reasonable to project the prototypical ball or its central vector to the training patches. Instead, we visualize each prototypical concept by the training patches that are contained within each ball at the end of training. Thus, we can produce multiple visualizations from each prototypical concept and skip the projection step altogether.
Pruning and Fine-tuning: Because the number of prototypical concepts is fixed at the beginning of training, it is possible for a trained ProtoConcepts model to have a small number of prototypical
concepts that do not contain any latent training image patches. In other words, these prototypical concepts do not capture any visualizable latent features, and should not be considered when making predictions. To prune the non-visualized prototypical concepts, we design a 0/1 mask on the prototype class assignment mechanism so that the given non-visualized prototypical concepts do not provide evidence for any image classes. After pruning these prototypical concepts, we finally perform a sparse convex optimization on the last layer weight matrix  $w_{h}$  as in previous prototype-based methods [3].
4.1 Case Study 1: Bird Species Identification
To demonstrate and examine the effectiveness of our method, we implement the ProtoConcepts module on the ProtoPNet, ProtoPool, and TesNet networks for the Caltech-UCSD Birds-200-2011 (CUB 200-2011) dataset [35]. The dataset contains 5,994/5,794 images for training and testing across 200 different bird species. We perform offline data augmentation, and crop training and testing images using bounding boxes provided with the dataset.
4.1.1 Implementation
In this section, we describe the implementation changes made to each prototype-based model for our experiments to incorporate prototypical concepts. A full list of exact parameter settings and training schedule for each model can be found in the Appendix.
ProtoPNet: The ProtoPNet model [3] consists of a CNN backbone, a log-based prototype layer  $g^{\log}$ , and a fully connected layer  $h$  [3]. Each of 200 classes is assigned 10 class-specific prototypes, which are constrained to provide evidence only for their corresponding class through the last layer weighting. We train a ProtoPNet-Concepts model by substituting the prototype layer as in Eq. (1), and then training with additional losses  $\mathcal{L}_{\mathrm{Rad}}$  and  $\mathcal{L}_{\mathrm{Clstk}}$  with  $k = 10$ , radius initialization 7.5, and loss weights 0.01 and 0.8, respectively. The training schedule and other hyperparameters are unchanged from the original paper.
TesNet: The TesNet model [36] uses a cosine similarity function in its prototype layer to represent the latent space as a hypersphere and incorporates several hypersphere-based loss functions during training. After modifying the activation function as in Eq. (2), we train TesNet-Concepts with a radius initialization of 8.05, and a top- $k$  cluster loss  $\mathcal{L}_{\mathrm{Clstk}}$  with  $k = 3$ . We set the weight for the radius loss as  $3 \times 10^{-5}$  and the weight for top- $k$  cluster loss as 0.8 with a learning rate of  $1 \times 10^{-4}$  during the warmup stage, and reweight the loss functions described in its original paper with a new training schedule.
ProtoPool: The ProtoPool architecture [26] employs a log-based activation on the prototype layer, and adds a prototype class assignment mechanism that allows sharing prototypes among multiple classes, thereby reducing the number of prototypes. After modifying the prototype layer as in Eq. (1), we train ProtoPool-Concepts using a radius initialization of 4.5 and top- $k$  cluster loss  $\mathcal{L}_{\mathrm{Clstk}}$  with  $k = 10$ . We set a radius loss weight  $3 \times 10^{-3}$ , a top- $k$  cluster weight 0.8, and learning rate for the radius as  $0.5 \times 10^{-3}$ . The training schedule and other hyperparameters are left unchanged from the original paper.
For baseline comparisons, we retrain CNN backbones and previous prototype-based models using publicly available code. As shown in Table 1, for the same CNN architecture, our ProtoConcepts network achieves similar test accuracy to corresponding baseline models.