SlowGuess/NeurIPS_2023_OCR huggingface dataset:
ike
Figure 8: Reasoning of a correctly classified Bronzed Cowbird by ProtoPNet (left) and ProtoPNet-Concepts (right).
Comes from
ProtoPNet: Why is this bird classified as a Florida Jay?
Figure 9: Reasoning of a correctly classified Florida Jay by ProtoPNet (left) and ProtoPNet-Concepts (right).
ProtoPNet-Concepts: Why is this bird classified as a Florida Jay?
ProtoPNet: Why is this bird falsely classified as a Cardinal?
Figure 10: Reasoning of a Misclassified Cedar Waxwing to Cardinal by ProtoPNet (left) and ProtoPNet-Concepts (right).
ProtoPNet: Why is this bird falsely classified as a Long tailed Jaeger?
ProtoPNet-Concepts: Why is this bird falsely classified as a Long tailed Jaeger?
Figure 11: Reasoning of a Misclassified Black Tern to Long Tailed Jaeger by ProtoPNet (left) and ProtoPNet-Concepts (right).
D More Examples for ProtoPool-Concepts Visualizations
In this section, we present more examples of the reasoning process for ProtoPool-Concepts. The visualizations are obtained from the model with ResNet 50(iNat) [10] backbone. Because prototypes are shared among different classes, the information extracted by ProtoPool can be unintuitive, especially when the prototype does not come from the same class as a given test image. This problem affects ProtoPool as well as ProtoPool-Concepts – while shared prototypes can, in some cases, make a concept clearer by borrowing good examples of that concept from other classes, in our experience it seems to make the concepts less clear. For example, in Figure 12, it is hard to see how the beak of a Cedar Waxwing is similar to the prototype visualization captured by ProtoPool. Moreover, similar to a problem we saw with ProtoPNet, whether the prototype is capturing the feet or the belly of the Cedar Waxwing from the top row is also unclear. On the other hand, ProtoPool-Concepts may be capturing a color contrast of the throat and breast that is common also to the Great Crested Flycatcher and Cedar Waxwing, etc., but since the contrast of the test image is not particularly good, the human might determine not to trust that
ProtoPool: Why is this bird classified as a Cedar Waxwing?
ProtoPool-Concepts: Why is this bird classified as a Cedar Waxwing?
Figure 12: Reasoning of a correctly classified Cedar Waxwing by ProtoPool (left) and ProtoPool-Concepts (right).
ProtoPool:
Why is this bird classified as a Yellow headed Blackbird?
ProtoPool-Concepts:
Why is this bird classified as a Yellow headed Blackbird?
Looks like
Figure 13: Reasoning of a correctly classified yellow-headed blackbird by ProtoPool (left) and ProtoPool-Concepts (right).
ProtoPool:
Why is this bird classified as a Red Bellied Woodpecker?
Looks like
ProtoPool-Concepts:
Why is this bird classified as a Red Bellied Woodpecker?
Looks like
mes from
Figure 14: Reasoning of a correctly classified Red Bellied Woodpecker by ProtoPool (left) and ProtoPool-Concepts (right).
Figure 15: Reasoning of a correctly classified White-breasted Kingfisher by ProtoPool-Concepts
Figure 16: Reasoning of a correctly classified Cedar Waxwing by ProtoPool-Concepts
comparison. ProtoPool-Concepts also appears to capture the yellow and black color pattern on the bird's head. Similarly, in Figure 13, the comparison made at the bottom left for ProtoPool is hard to identify with a single visualization, while ProtoPool-Concepts offers more information such as the yellow head and black body, though the concept is not cohesive, since it includes a bird with red and white stripes. An analyst seeing this concept might determine that this concept may not be trustworthy, which is a useful piece of information. It is worth noting that the prototypical concept of the yellow-headed blackbird shown at the top of Figure 13 is shared with the Cedar Waxwing as shown in the bottom row of Figure 12. Figure 14 shows that ProtoPool-Concepts shares part of a prototype concept with the Cedar Waxwing; this concept appears to indicate a dark-and-light textural pattern that is similar to the wing of the Red Bellied Woodpecker. However, the comparison made by ProtoPool is ambiguous. Figure 15, Figure 16 further demonstrate how the evidence layer and prototype activation are involved in the reasoning process. As explained in the main paper, a weighted score would be computed for each of the prototypical concepts, and a total logit score would be calculated for the final prediction.
E More Examples for TesNet Visualizations for CUB dataset
In this section, we present some examples of the reasoning process of TesNet-Concepts and its comparison with TesNet. The visualizations were obtained from TesNet and TesNet-Concepts trained with a VGG19 backbone. Similar to ProtoPNet and ProtoPool, ambiguity arises from low quality images. In Figure 17, the prototype visualization of TesNet for a Florida Jay shown at the top has low contrast. Whether the prototype is capturing the belly or the feet is unclear. Although a similar prototype visualization is also captured by the prototypical ball for TesNet-Concepts as shown in the top right of the plot, the other visualizations reveal the prototypical concept as the white belly near the blue feathers. Similarly, the multi-visualization shown at the bottom suggests that the prototypical ball likely represents the bird's head and beak. Figure 18 shows the prototype logic for identifying a Cedar Waxwing by the color gradient on the bird's throat. Some of the image patches chosen by TesNet-Concepts look almost identical to the chosen patch of the test image; this is more convincing than the comparisons from TesNet without concepts.
F More Examples for ProtoPool-Concepts Visualizations and Model Performance for Car dataset
In this section, we present some of the visualizations for ProtoPool-Concepts on the Car dataset. The visualizations are from the models trained with a DenseNet121 [11] backbone. Similar to ProtoPool-Concepts for the CUB 200-2011 dataset, the shareable prototypes still make the comparison unintuitive. As shown in Figure 19, ProtoPool seems to compare the front of a BMW to that of an Audi. It appears to find a similarity between the kidney grille of the BMW to the front of the Audi, or it could be the shape of the lights that are similar. On the other hand, the multiple visualizations from ProtoPool-Concepts suggest that the model is actually capturing the kidney grille of the BMW. The other visualizations further show that the ProtoPool-Concept found a part of the car on the bottom, just in front of the tires, is similar to the same part of the other types of SUVs. Figure 20 shows that our model finds the wheel of the Audi TT RS Coupe is similar to the wheels of different types of Audis.
Table 12 shows some additional performance experiments for ProtoPNet-Concepts and ProtoPool-Concepts on the Cars dataset. The training schedule and parameter settings are the same as reported above. The initialized number of prototypes for ProtoPNet-Concepts and ProtoPool-Concepts are 1960 and 195 respectively. The radius initialization for ProtoPNet-Concept is 7.5, and the radius initialization for ProtoPool-Concept is 4.7. As shown in the table, ProtoPNet-Concepts under the DenseNet121 baseline has better performance with fewer visualizable prototypes than ProtoPNet, and its performance is very close to the baseline. On the other hand, ProtoPool-Concepts under a DenseNet161 backbone yields better performance than ProtoPNet with much fewer visualizable prototypes.
Figure 17: Reasoning of a correctly classified Florida Jay by TesNet (left) and TesNet-Concepts (right)
TesNet: Why is this bird falsely classified as a Cedar Waxwing?
TesNet-Concepts: Why is this bird falsely classified as a Cedar Waxwing?
Figure 18: Reasoning of a correctly classified Cedar Waxwing by TesNet (left) and TesNet-Concepts (right)
ProtoPool: Why is this car classified as a BMW X3 SUV 2012?
ProtoPool-Concepts: Why is this car classified as a BMW X3 SUV 2012?
Figure 19: Reasoning of a correctly classified BMW X3 SUV 2012 by ProtoPool (left) and ProtoPool-Concepts (right).
ProtoPool: Why is this car classified as a Audi TT RS Coupe 2012?
ProtoPool-Concepts: Why is this car classified as a Audi TT RS Coupe 2012?
Figure 20: Reasoning of a correctly classified Audi TT RS Coupe 2012 by ProtoPool (left) and ProtoPool-Concepts (right).
Table 12: Accuracy comparison of ProtoPool-Concepts to other shared prototype models on the Stanford Cars dataset
Arch.	Model	Prototype p #	Acc. [%]
DenseNet121 [11]	ProtoPNet-Concepts (ours)	1941 ± 7	88.4 ± 0.1
ProtoPNet [3]	1960	86.8 ± 0.1
Baseline (given in [3])	N/A	89.7 ± 0.1
DenseNet161 [11]	ProtoPool-Concepts (ours)	195	90.4 ± 0.4
ProtoPNet [3]	1960	89.5 ± 0.2
Baseline (given in [36])	N/A	92.5 ± 0.3
G Training software and platform
We implemented our ProtoConcepts on ProtoPNet, ProtoPNet, and ProtoPool using PyTorch. The experiments were run on 1 NVIDIA Tesla V100 or 1 NVIDIA RTX A5000. Using the ProtoConcept module with any base architecture has similar training times to using the base architecture alone.