SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Radius Study: We performed an ablation on ProtoPool-Concepts with a different set of radius initialization and top- $k$  cluster loss with  $k = 10$  under the same training settings described above. Intuitively, a larger radius would encourage more prototypical concepts to capture information and an infinitesimal radius would be equivalent to a single visualization. As shown in Table 2, the number of visualizable prototypical concepts increases with the radius. In addition, radius also influences the accuracy of our ProtoConcepts model after pruning and fine-tuning. If the radius is too small, the number of visualizable prototypical concepts also tends to be small – this means that pruning will remove too many prototypes, and consequently adversely impact performance. On the other hand, if
Table 1: Comparison of ProtoConcepts implementation on ProtoPNet, ProtoPool, TesNet with the corresponding baselines and various backbone architectures. All algorithms on the same architecture perform similarly; the advantage is instead in interpretability.
Arch.	Model	Prototype p #	Acc.[%]
VGG19 [29]	ProtoPNet-Concepts (ours)	1993 ± 3	77.9 ± 0.2
ProtoPNet [3]	2000	78.0 ± 0.2
TesNet-Concepts (ours)	1862 ± 20	78.2 ± 0.8
TesNet [36]	2000	77.9 ± 0.1
Baseline (given in [3])	N/A	75.1 ± 0.4
Densenet-161 [11]	ProtoPNet-Concepts (ours)	1980 ± 2	80.7 ± 0.4
ProtoPNet [3]	2000	80.1 ± 0.3
TesNet-Concepts (ours)	1994 ± 2	81.4 ± 0.3
TesNet [36]	2000	81.5 ± 0.3
ProtoPool-Concepts (ours)	199 ± 1	81.5 ± 0.6
Protopool [26]	202	80.3 ± 0.3
Baseline (given in [3])	N/A	82.2 ± 0.2
Resnet-34 [10]	TesNet-Concepts (ours)	1957 ± 10	79.8 ± 0.4
TesNet [36]	2000	80.7 ± 0.3
ProtoPool-Concepts (ours)	185 ±4	80.4 ± 0.1
ProtoPool [26]	202	80.3 ± 0.2
Resnet-50 (iNat) [10]	ProtoPool-Concepts (ours)	188 ± 2	85.2 ± 0.2
ProtoPool [26]	202	85.5 ± 0.1
ProtoTree [19]	202	82.2 ± 0.7
Resnet-152 [10]	ProtoPNet-Concepts (ours)	1972 ± 10	78.0 ± 0.1
ProtoPNet [3]	2000	78.0 ± 0.3
Baseline (given in [3])	N/A	81.5 ± 0.4
the radius is too large, each prototypical ball would capture too many latent representations whose semantic meanings may be too diverse, thereby making the prototypes less meaningful. Hence, we need to tune the radius to be within a reasonable range.
4.1.2 Interpretability
The reasoning process of a ProtoConcepts network mirrors that of a traditional prototype network, but provides multiple visualizations of the prototypes used in its explanations. Figure 3 shows how our model classifies a testing image of a Baltimore Oriole into the correct class. It is worth noting that because of the shareable prototypes mechanism inherited from ProtoPool, the visualizations of a learned prototypical concept now come from different classes. The most representative features that an ornithologist would use to classify an adult male Baltimore Oriole are the bright orange and black plumage and the slender beak [28]. As shown in Figure 3, our model is able to compare the test image patch containing orange and black plumage with the learned prototypical concept that captures similar features from its own class. Additionally, our model compares the patch containing the wings to the prototypical concept shared among Baltimore Orioles and other similar birds such as Orchard Orioles. Lastly, our model compares the part of image that contains a slender beak along with its black head to the prototypical concept that captures the slender beak and black head of birds such as Orchard Orioles and Eastern Towhees.
Figure 1 shows an example of how a test image of a Brown Thrasher is correctly classified by ProtoPool and ProtoPool-Concepts. Because of the prototype sharing mechanism, it is frequently the case that training image patches from birds of one class are used as evidence of a test image belonging to another class. As a result, the similarities presented by ProtoPool's visual explanations can be unintuitive. As shown in the plot, it is unclear how ProtoPool found the head of a Brown Thrasher to be similar to the blue head of a Barn Swallow. Moreover, it is hard to determine whether the model is comparing the background or the belly of the Brown Thrasher, as shown at the bottom. However, comparisons made by our method are much clearer thanks to multi-visualizations. Although both models attempt to compare the lower part of the bird to the learned prototypes, it is clear that our ProtoPool-Concepts is looking at the pattern on the belly, which is an identifying feature of the Brown Thrasher [28].
Figure 3: The reasoning process of our network in classifying a test image as a Baltimore Oriole. Test image patches are compared to prototypical concepts in latent space, then visual similarity scores are compiled as evidence for each class.
ProtoPNet: Why is this bird classified as a Male Ruby Throated Hummingbird?
ProtoPNet: Why is this bird classified as a Female Ruby Throated Hummingbird?
ProtoPNet-Concepts: Why is this bird classified as a Female Ruby Throated Hummingbird?
ProtoPNet-Concepts: Why is this bird classified as a Male Ruby Throated Hummingbird?
Figure 4: Reasoning of a correctly classified Female Ruby Throated Hummingbird (top) and Male Ruby Throated Hummingbird (bottom) by ProtoPNet (left) and ProtoPNet-Concepts (right).
Figure 4 shows examples of how the two models correctly classify a female and a male ruby-throated hummingbird. In the top row, both of the models found the neck of a female ruby-throated hummingbird from the given test image similar to the white feathers around neck from the training examples. On the other hand, although having a red throat differentiates the male ruby-throated hummingbird from the female [28], the significance of a red-throat to correctly classify a male ruby-throated hummingbird is unclear. As shown in the bottom left of Figure 4, the learned prototype from ProtoPNet is able to compare the prototype that contains a red throat to that of the test image. However, the visualizations from ProtoPNet-Concepts demonstrate that a red throat may not be all that the prototypical concept is representing. It appears to represent broader information about the bird's gray head and the transition to its body, where the additional red or gray color is located on the throat, and the white on the breast.
Table 2: Performance of ProtoPool-Concepts with different radius initializations using ResNet-50(iNat) backbone
Radius	Acc. [%] before pruning	Acc. [%] after pruning & finetuning	Prototype p #
4.0	85.4 ± 0.1	81.5 ± 1.0	61 ± 6
4.5	85.2 ± 0.1	85.2 ± 0.2	188 ± 2
5.0	85.2 ± 0.3	85.0 ± 0.3	202
Table 3: Accuracy comparison of ProtoPool-Concepts to other shared prototype models on the Stanford Cars dataset
Arch.	Model	Prototype p #	Acc.[%]
Resnet 34 [10]	ProtoConcepts (ours)	150 ± 6	89.4 ± 0.5
ProtoPShare [25]	480	86.4
Protopool [26]	195	89.3 ± 0.1
Densenet121 [11]	ProtoConcepts (ours)	194 ± 1	87.1 ± 0.4
Protopool [26]	195	86.4 ± 0.1
4.2 Case Study 2: Car Model Identification
In this case study, we implement our method for the Stanford Cars dataset [17] of 196 car types, using similar procedures and training algorithms as in the CUB-2011-200 experiment. We trained a ProtoPool-Concepts model with radius initialization 4.5, keeping other parameter settings the same as described in Sec. 4.1.1. We compare our method to self-reported accuracies of other shared prototype models ProtoPool [26] and ProtoPShare [25]. Our model achieves similar test accuracy to other shared prototype models, with the advantage of multiple visualizations for prototypical concepts. More details, including visual explanations, can be found in the Appendix.
5 User Study
To show the reduction of ambiguity and resulting improvement in user interpretability, we created a distinction user study similar to HIVE [15] to compare our ProtoConcepts method with ProtoPNet; results are shown in Table 4. We randomly picked ten samples from the test set and calculated the top two predicted classes (i.e., the classes with the highest predicted probabilities according to the model) for each test sample. We then provided visual explanations from the most activated prototypes for these classes by ProtoPNet and ProtoPNet-Concepts. A test-taker was then asked to choose which class the model is actually predicting, looking only at the visual explanations without the class probabilities. Examples of our user study are shown in Figure 7. We released our user study on Amazon Mechanical Turk and collected 50 responses from test takers with a  $98\%$  survey approval rate to ensure the quality of responses, and removed 1 response from both surveys after screening for nonsensical free response answers. We first ran a two-sided  $t$ -test on self-rated ML experience for the test takers from the ProtoPNet and ProtoPNet-Concept. The  $p$ -value is 1, and we are assured that there is no statistically significant difference in machine learning experience between the two groups on average. From the results of our survey, we are able to conclude that users with visual explanations from ProtoConcepts outperform those with visual explanations from ProtoPNet to a statistically significant degree  $(p = 0.003)$ . Moreover, users given visual explanations from ProtoPNet were unable to outperform  $50\%$  random guessing to a statistically significant degree  $(p = 0.289)$ , whereas users given visual explanations from our model did  $(p = 2.85 \times 10^{-5})$ . Our survey results show not only that our model provides a notable improvement in user interpretability, but is able to improve non-expert user performance in a difficult fine-grained classification task whereas the previous ProtoPNet model cannot.
The interpretability of prototype-based models comes from visual explanations of classification decisions. Hence, the exact semantics (e.g., "this bird has a long beak" or "this bird has a spotted belly") underlying these visual explanations depend on the user's visual system to determine. We present a method for clarifying the user's semantic inference by offering multiple visualizations whose
Simulate the model
Prototypes and their source photos are from the specified species.
pedies you think is correct, then click "Next Photo."
Figure 5: User study example questions with architecture ProtoPNet (left), and ProtoConcepts (right). We showed each user 2 possible class explanations for a given test image and asked the user to pick the correct class. We compared the results from two separate surveys, one in which visual explanations are generated by ProtoPNet, and one in which they are generated by ProtoConcepts.
Simulate the model
01.02.03.04
To the bird species you think is correct, then click "Next Photo." Species 2
commonalities can disambiguate our model's exact reasoning. While we find that our concept-based representation is helpful for this process, it is still unable to provide explicit semantics to explain its classifications. With recent progress in large vision-language hybrid models [22, 16, 4, 34], a technique offering explicit semantics for visual explanations could be explored in future work. It is important to note that in some visual domains, concepts may not have natural textual descriptions (e.g., in mammography or other radiology applications where it might look at a patch of a particular type of breast tissue that is difficult to describe and has no associated terminology).
Interpretability is an essential ingredient to trustworthy AI systems [33]. Our work presents a major improvement in interpretability to prototype-based networks, which are one of the leading techniques for interpretable neural networks in computer vision. Thus, our technique can be used for important computer vision applications to discover new knowledge and create better human-AI interfaces for difficult, high-impact applications.
Table 4: Results of the user study comparing ProtoPNet to ProtoConcepts. We report the mean user accuracy for each model with a range of plus/minus 1.96 standard deviations. In addition, we conduct two 1-sided t-tests on the results, with the alternative hypotheses that (1) users with ProtoConcepts' visual explanations score higher on average than with ProtoPNet and (2) Each model, respectively, achieves a higher accuracy than random guessing. We conclude that users presented with visual explanations from ProtoConcepts outperform those with explanations from ProtoPNet to a statistically significant degree.
Model	Mean Acc. [%]±1.96 Std.	p-value (1)	p-value (2)
ProtoConcepts (Ours)	62.1 ± 5.4	0.003	2.85 × 10-5
ProtoPNet [3]	51.5 ± 5.2	0.288
In this work, we present an interpretable method for image classification which incorporates prototypical concepts with multiple visualizations to explain its predictions (this looks like those). Unlike previous works in prototype-based classification which offer only single prototype visualizations, our method allows the user to use commonalities between visualizations of prototypical concepts to better infer the semantic meaning of prototypical concepts. We compare the visual explanations offered by our and previous prototype-based methods and show that our model can achieve comparable accuracy to previous methods.
We gratefully acknowledge support from the National Science Foundation under grants IIS-2130250, RII Track-2 FEC-2218063.
[36] Jiaqi Wang, Huafeng Liu, Xinyue Wang, and Liping Jing. Interpretable image recognition by constructing transparent embedding space. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 895–904, 2021.
[37] Naoya Yoshimura, Takuya Maekawa, and Takahiro Hara. Toward understanding acceleration-based activity recognition neural networks with activation maximization. In 2021 International Joint Conference on Neural Networks (IJCNN), pages 1-8. IEEE, 2021.
[38] Jason Yosinski, Jeff Clune, Anh Nguyen, Thomas Fuchs, and Hod Lipson. Understanding neural networks through deep visualization. In Deep Learning Workshop at the 32nd International Conference on Machine Learning, 2015.
A Ablation Study on Top-  $k$  Cluster Loss
To further demonstrate how top- $k$  cluster loss helps the model performance, we performed an ablation study on ProtoPool-Concepts. We trained ProtoPool-Concepts by adding  $\mathcal{L}_{\mathrm{Clstk}}$  losses with the choices of  $k = 1,3,5,10$  into the model with radius initialization 4.5 using ResNet50 (iNat) backbone. As shown in Table 5, although the classification performance of the model before pruning does not vary much when  $\mathcal{L}_{\mathrm{Clstk}}$  is introduced, the number of visualizable prototypes sharply increases, and the performance after pruning is significantly improved. While, as discussed in the main paper, the  $\mathcal{L}_{\mathrm{Clstk}}$  term ensures the quality of the latent space, encouraging prototypical concepts which enrich the interpretability of the model.
Table 5: Model Performance on ProtoPool-Concepts under ResNet-50 (iNat) Backbone with different choices of losses. Acc.[%] bf. prune refers to the performance of a model before the pruning procedure and Acc.[%] finetuned refers to the performance after finetuning.
Model	Prototype p #	Acc. [%] bf. prune	Acc. [%] finetuned
ProtoPool-Concepts + LClstk, k=1	50 ± 1	85.2 ± 0.3	67.7 ± 2.0
ProtoPool-Concepts + LClstk, k=3	85 ± 8	85.3 ± 0.3	83.5 ± 0.5
ProtoPool-Concepts + LClstk, k=5	108 ± 5	85.5 ± 0.1	84.4 ± 0.3
ProtoPool-Concepts + LClstk, k=10	188 ± 2	85.3 ± 0.3	85.2 ± 0.2
B Training Parameters
In this section we present the exact training hyperparameters, objectives, and schedules used in our experiments for applying the ProtoConcepts module to the ProtoPNet [3], ProtoPool [26], and TesNet [36] prototype networks.
B.1 ProtoPNet
In addition to the original losses ProtoPNet has, we trained ProtoPNet-Concepts with  $\mathcal{L}_{\mathrm{Clstk,k=10}}$  and  $\mathcal{L}_{\mathrm{Rad}}$ . We set the corresponding weight for losses as demonstrated in Table 6. Like ProtoPNet, we have 5 epochs of training as warmup where we trained the add-on layers, prototype layers, and the radius width, as shown in Table 7. Similar to the original ProtoPNet training schedule, We start training the convolution layer and last layer at the joint optimization stage for 10 epochs. Overall, 15 epochs are trained before pruning and fine-tuning. Lastly, after pruning the prototypical balls that have no visualizations, we fine-tune the last layer for 20 epochs.
Table 6: Parameter Settings for ProtoPNet-Concepts
Parameter	Weight
Cross Entropy Weights	1.0
Top-k Cluster Loss	0.8
Separation Loss	-0.08
Radius Loss	0.01
Table 7: Training Schedule for ProtoPNet-Concepts
Training Stage	Model Layers	Learning Rate	Duration
Warmup Stage	add on layers	3 × 10-3
prototype vectors	3 × 10-3	5 epochs
radius	0.5 × 10-4
Joint Stage	convolution f	1 × 10-4
add on layers	3 × 10-3
prototype vectors	3 × 10-3	10 epochs
last layer	1 × 10-4
Finetuning Stage	last layer	1 × 10-4	20 epochs
B.2 ProtoPool
We trained ProtoPool-Concepts, in addition to its original loss terms, with  $\mathcal{L}_{\mathrm{Clstk,k = 10}}$  and  $\mathcal{L}_{\mathrm{Rad}}$ . The corresponding weights for the losses are shown in Table 8. The training schedule is similar to that of the original ProtoPool,
Table 8: Parameter Settings for ProtoPool-Concepts
Parameter	Weight
Cross Entropy Weights	1.0
Top-k Cluster Loss	0.8
Separation Loss	-0.08
Radius Loss	3 × 10-3
Table 9: Training Schedule for ProtoPool-Concepts
Training Stage	Model Layers	Learning Rate	Duration
Warmup Stage	add on layers	1.5 × 10-3
pool layer	1.5 × 10-3	10 epochs
radius	0.5 × 10-4
Joint Stage	convolution f	5 × 10-5
add on layers	1.5 × 10-3	20 epochs
pool Layer	1.5 × 10-3
Finetuning Stage	last layer	1 × 10-4	15 epochs
shown in Table 9. We trained radius width only during the warm-up stage. We set the first 10 epochs as the warm-up stage, and the following 20 epochs as the joint optimization stage. Overall, the model is trained for 30 epochs before pruning and fine-tuning. We further trained additional 15 epochs to fine-tune the last layer weights after pruning the prototypical balls that do not contain any visualizations.
B.3 TesNet
We trained ProtoPool-Concepts, in addition to its original loss terms, with  $\mathcal{L}_{\mathrm{Clstk,k = 3}}$  and  $\mathcal{L}_{\mathrm{Rad}}$ . The corresponding weights for the losses are shown in Table 10. The training schedule is similar to that of the original TesNet, shown in Table 11. We trained radius width during the warm-up stage and joint optimization stage. We set the first 5 epochs as the warm-up stage, and the following 15 epochs as the joint optimization stage. Overall the model is trained for 20 epochs before pruning and fine-tuning. We further trained additional 15 epochs to fine-tune the last layer weights after pruning the prototypical balls that do not contain any visualizations.
Table 10: Parameters Setting for TesNet-Concepts
Parameter	Weight
Cross Entropy Weights	1.0
Top-k Cluster Loss	0.8
Separation Loss	-0.2
Subspace Separation Loss	3 × 10-5
Orthogonality Loss	5 × 10-3
Radius Loss	3 × 10-5
Table 11: Training Schedule for TesNet-Concepts
Training Stage	Model Layers	Learning Rate	Duration
Warmup Stage	add on layers	3 × 10-3	5 epochs
Prototype Vectors layer	3 × 10-3
radius	1 × 10-4
Joint Stage	convolution f	1 × 10-4	15 epochs
add on layers	3 × 10-3
Prototype Vectors Layer	3 × 10-3
radius	1 × 10-5
Finetuning Stage	last layer	1 × 10-4	15 epochs
C More Examples for ProtoPNet-Concepts Visualizations
In this section, we further demonstrate how multi-visualization solves the ambiguity of prototypes learned by ProtoPNet. The visualizations shown are trained with a VGG-19 [29] backbone for both ProtoPNet and
ProtoPNet: Why is this bird classified as a Least Tern?
ProtoPNet-Concepts: Why is this bird classified as a Least Tern?
Why is this bird classified as a Chestnut sided Warbler?
Why is this bird classified as a Chestnut sided Warbler?
Figure 6: Reasoning of a correctly classified Least Tern (top) and Chestnut-sided Warbler (bottom) by ProtoPNet (left) and ProtoPNet-Concepts (right). Some visualization examples of the Prototypical concepts are shown in the yellow bounding boxes, and compared with patches from the test image.
ProtoPNet-Concepts. Figure 6 demonstrates how our method reduces the ambiguity inherent in the ProtoPNet architecture. The top row provides an example of how ProtoPNet and ProtoPNet-Concepts compare and classify a test image of a Least Tern. ProtoPNet highlights the bird's leg on the test image, which one can barely see in the model's learned prototypical features (e.g., is it comparing the wings? Or is it comparing the feet, which are too tiny to see in the image?) On the other hand, it is clear that ProtoPNet-Concepts is comparing the wings, even though the bounding boxes on the test image include the bird's feet. Similarly, the bottom row shows each model's reasoning for a correctly classified test image of a Chestnut-sided Warbler. It is not clear in this case whether ProtoPNet is comparing the background or a barely-captured yellow crown feature. On the other hand, the many examples of the yellow crown from the multi-visualization of ProtoPNet-Concepts clarify that the prototype represents this crucial feature of a Chestnut-sided Warbler [28].
More examples of unclear comparisons can be found in Figure 7. As shown in the plot, the ProtoPNet finds the green wing and spotted breast of the test image of a spotted catbird similar to a green wing with a black spot in the middle. On the other hand, the prototype learned by ProtoPNet-Concepts suggests that the prototypical concept focuses on the texture. In Figure 7, it is also unclear if ProtoPNet finds the feathers or the red feet of the Forster Tern to be similar to its learned prototype visualizations. However, we can clearly see from the ProtoPNet-Concepts multi-visualizations that the prototypical concept is actually the grey wing and white belly instead of the red feet.
Some types of ambiguities are specific to bird types. For instance, it is difficult to determine what the prototypes represent for birds that are black everywhere. For example, as shown in Figure 8, it is hard to see what the prototypes learned by ProtoPNet are capturing except for the dark feathers. Multiple visualizations learned by ProtoPNet-Concepts suggest that color and possibly the shape of the bird's belly are the first concept, whereas the red eye and shape of the head appear to be the second concept.
Ambiguity of the learned prototypes can arise from low quality test images. For example, as shown in Figure 9, visualizations from ProtoPNet have low brightness/contrast on the prototype birds. It is hard to see any distinguishable features captured by those prototypes. On the other hand, although similar visualizations are used by ProtoPNet-Concepts, other visualizations within the ball demonstrate the prototypical concepts (here, blue and white stripes apparently).
Lastly, having multiple visualizations can also help to draw inferences on misclassified cases. Figure 10 and Figure 11 show cases when both ProtoPNet and ProtoPNet-Concepts misclassify the test image. In Figure 10, it is unclear how ProtoPNet found the test image of a Cedar Waxwing to be similar to a Cardinal, except for the color of the fruit it has in its beak. On the other hand, multiple visualizations offer additional information such as the triangular silhouette of the head, dark strip of feathers around the eye, and pointed features at the back of the head. Similarly, in Figure 11, ProtoPNet-Concepts is able to offer more specific information than ProtoPNet, such as the grey and white color contrast, black head and the shape of the body. Here, the algorithms are showing us why it is difficult to tell these birds apart.
ProtoPNet:
Why is this bird classified as a Spotted Catbird?
ProtoPNet-Concepts:
Looks like
Why is this bird classified as a Spotted Catbird?
Comes from
ProtoPNet-Concepts:
Why is this bird classified as a orster Tern?
Why is this bird classified as a Forster Tern?
Comes from
Figure 7: Reasoning of a correctly classified Spotted Catbird (top) and Forster Tern (bottom) by ProtoPNet (left) and ProtoPNet-Concepts (right).
ProtoPNet:
Looks like
Why is this bird classified as a Bronzed Cowbird?
ProtoPNet-Concepts:
Why is this bird classified as a Bronzed Cowbird?
Comes from