What Do You See in Common? Learning Hierarchical Prototypes over Tree-of-Life to Discover Evolutionary Traits
Authors: Harish Babu Manogaran
M. Maruf
Arka Daw
Kazi Sajeed Mehrab
Caleb Patrick Charpentier
Josef Uyeda
Wasila M Dahdul
Matthew J Thompson
Elizabeth G Campolongo
Kaiya L Provost
Wei-Lun Chao
Tanya Berger-Wolf
Paula Mabee
Hilmar Lapp
Anuj Karpatne
Venue: ICLR 2025 Poster (2025)
Keywords: deep learning
interpretability
prototype-based neural network
phylogeny
computer vision
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)

Abstract: A grand challenge in biology is to discover evolutionary traits---features of organisms common to a group of species with a shared ancestor in the tree of life (also referred to as phylogenetic tree). With the growing availability of image repositories in biology, there is a tremendous opportunity to discover evolutionary traits directly from images in the form of a hierarchy of prototypes. However, current prototype-based methods are mostly designed to operate over a flat structure of classes and face several challenges in discovering hierarchical prototypes, including the issue of learning over-specific prototypes at internal nodes. To overcome these challenges, we introduce the framework of Hierarchy aligned Commonality through Prototypical Networks (HComP-Net). The key novelties in HComP-Net include a novel over-specificity loss to avoid learning over-specific prototypes, a novel discriminative loss to ensure prototypes at an internal node are absent in the contrasting set of species with different ancestry, and a novel masking module to allow for the exclusion of over-specific prototypes at higher levels of the tree without hampering classification performance.  We empirically show that HComP-Net learns prototypes that are accurate, semantically consistent, and generalizable to unseen species in comparison to baselines. Our code is publicly accessible at Imageomics Institute Github site: https://github.com/Imageomics/HComPNet.

Published as a conference paper at ICLR 2025
WHAT Do YOU SEE IN COMMON? LEARNING HIERARCHICAL PROTOTYPES OVER TREE- OF- TO DISCOVER EVOLUTIONARY TRAITS
Harish Babu Manogaran"*
M. Maruf
Arka Daw
Kazi Sajeed Mehrab-
Caleb Patrick Charpentier
Josef C. Uyeda'
Wasila Dahdul
Matthew J TTommsonn
Elizabeth G Campolongo
Kaiya Provost
Wei-Lan Chao
Tanya Berger- Wolf
Paula M. Mabee
Hilmar Lapp Anuj Karpatne
Virginia Tech Univ. of California, Irvine *Ohio State Univ.
BBattelle Duke Univ.
"Oak Ridge National Laboratory
Phylogenetic
Slack Billed Bilkd
BiledAAri
Nothhm
Black
Sample images of bird species with zoomed-in views of learned prototypes along with their associ- ated score maps. We consider the problem finding evolutionary traits common group of species derived from the same ancestor (blue) that are absent other species from a different ancestor (red), can infer that descendants of the blue node share a common trait: long tail, absent from descendants of the red node.
Figure 1:
ABSTRACT
A grand challenge in biology is discover evolutionary meiis--eatures of or- ganisms common to a group of species with a shared ancestor in the tree of life (also referred as phylogenetic tree). With the growing availability of im- age repositories in biology, there is a tremendous opportunity to discover evo- lutionary traits directly from images in the form of a hierarchy of prototypes. However, current prototype-based methods a flat structure of classes and face several challenges discovering hierarchi- cal prototypes. including the issue of learning over-specific prototypes at internal nodes. To overcome these challenges, we introduce the framework Hierarchy aligned Commonality through Prototypical Networks (HComP-Net). The key novelties in HComP-Net include a novel over-specificity loss to avoid learn- ing over-specific prototypes, novel discriminative loss to ensure prototypes an internal node are absent in the contrasting set of species with different an- cestry, and novel masking module to allow for the exclusion of over-specific prototypes at higher levels of the tree without hampering classification perfor- mance. We empirically show that HComP-Net learns prototypes that are accu- rate, semantically consistent, and generalizable to unseen species in comparison baselines. Our code is publicly accessible Imageomics Institute Github site: https::/githun www..//.ww..a.cicccccococcoo HHcom//eCeee
mostly designed to operate over
"Correspondigg author: Serisshabu@@v..ee

---

Published as a conference paper at ICLR 2025
Over-specific
Over-specific
Common trait
Family, Foiize
Common Phylogenetic Ancestor
Common Phylogenetic Ancestor
Prototype-1
Prototype-2 ISpotted back)
Probotypo-1 (Long tail)
Prototypa-2 2 (Long neck)
Protetypa-1 (Spotted back)
Prototype-2 2 (Sharp beak)
(Mane)
Lion
Boboot
Marrgrove Cuciton
Pled Bilied Grebr
Read bell woslipeceer
Nerthem E Flocer
Mane, Spotted back: common species of Felidae family
Long fall, Long nock: common to both Cuckon and Grebe
Long beak, Spotted back: Common Red
bellidd woodpecker and Northern Flicker
Figure 2:
Examples to illustrate the problem of learning "over-speciffcc prototypes at internal nodes, which only cover one descendant species of the node, instead of learning prototypes common to all descendants.
INTRODUCTION
A central goal in biology is to discover the observable characteristics of organisms, or traits (c.g., beak color, stripe pattern, and fin curvature), that help in discriminating between species and un- derstanding how organisms evolve and adapt to their environment (Houle & Rossoni 2022). For example, discovering traits inherited by a group of species that share a common ancestor on the tree of life (also referred to the phylogenetic tree, see Figure [T is of great interest to biologists to understand how organisms diversify and evolve (O'Leary & Kaufman 2011) (see Appendix A for more details the biological motivation of this problem). However, the measurement of such traits with evolutionary signals, termed evolutionary traits, is not straightforward and often relies on
subjective and labor-intensive human expertise and definitions (Simões al. 2017 Sereno 2007).
hindering rapid scientific advancement (Lürig et al. 2021).
With the growing availability of large-scale image repositories in biology containing tens of thou- sands of species of organisms (Van Horn et al.) 2018 Singer et al. 2018 Wah et al. [01 there an opportunity for machine learning (ML) methods discover evolutionary traits automatically from images, especially those that differentiate visually or genetically similar species (Lürig et al., 2021 [Elhamod et al. (023). This is especially true in light of recent advances in the field of explain- able MI, such as the seminal work of ProtoPNet (Chen et al.) 2019) and its variants (Rymarczyk) al. 2021 2022 Nauta al., 202T) which find representative patches in training images (termed prototypes) capturing discriminatory features for every class. We can thus cast the problem of dis- covering evolutionary traits into asking the following question: what image features or prototypes common across group of species with a shared ancestor in the tree of life that are absent in species with a different shared ancestor?
For example, in Figure [ blue node show the common feature of having "long tails", unlike any of the descendant species of the red node. Learning such common features every internal node hierarchy of prototypes can help biologists generate novel hypotheses of species diversification (e.g., the splitting of blue and red nodes) and accumulation of evolutionary trait changes.
that the four species of birds the left descending from the
Despite the success of ProtoPNet (Chen et al. 2019) and its variants including HPnet (Hase et al. 2019) that first introduced the idea of learning hierarchical prototypes every internal node of the tree, there are three main challenges to be addressed while learning hierarchical prototypes for discovering evolutionary traits. First, existing methods that learn multiple prototypes for every class are prone to learning "over-specific" prototypes at internal nodes of tree, which cover only one (or a few) of its descendant species. Figure shows a few examples to illustrate the concept of over- specific prototypes. Consider the problem of learning prototypes common to descendant species of the Felidae family: Lion and Bobcat. we learn one prototype focusing on the feature the mane (specific only to Lion) and another prototype focusing on the feature of spotted back (specific only to Bobcat), then these two prototypes taken together can classify all images from the Felidae family. However, they do not represent common features shared between Lion and Bobcat and hence not useful for discovering evolutionary traits. Such over-specific prototypes should be instead pushed down be learned at lower levels of the tree (e.g., the species leaf nodes of Lion and Bobcat).
2

---

Published as a conference paper at ICLR 2025
Second, while existing methods such ProtoPShare Rymarczyk et al. 2021), ProtoPool (Rymar- czyk et al. 2022). and ProtoTree (Nauta et al. 2021) allow prototypes to be shared across classes for re-usability and sparsity, in the problem of discovering evolutionary traits, we want to learn proto- types at an internal node n that are not just shared across all íts descendant species but are also absent in the contrasting set of species (i.e., species descending from sibling nodes of v. representing alter- nate paths of diversification). Third, at higher levels of the tree, finding features that are common across a large number of diverse species is challenging (Harmon et al., 2010 Pennell et al. 2015). In such cases, we should be able to abstain from finding common prototypes without hampering accuracy at the leaf nodes feature missing in existing methods.
To address these challenges, we present Hierarchy aligned Commonality through Prototypical Networks (HComP-Net), framework to learn hierarchical prototypes over the tree of life for dis- covering evolutionary traits. Here are the main contributions of our work:
1. HComP- learns common traits shared by all descendant species of an internal node and avoids the learning of over-specific prototypes in contrast to baseline methods using a novel over-specificity loss.
2. HComP-Net uses novel discriminative loss ensure that the prototypes learned at an internal node are absent in the contrasting set of species with different ancestry.
3. HComP-Net includes novel masking module to allow for the exclusion of over-specific prototypes at higher levels of the tree without hampering classification performance.
4. We empirically show that HComP-Net learns prototypes that are accurate, semantically consistent, and generalizable to unseen species compared to baselines on data from 190 species of birds (CUB-200-2011 dataset) (Wah al. 2011]. 38 species of fishes (Elhamod et al., 2023). 30 species of butterflies (Lawrence & Campolongo 2024). 113 species of spiders and 41 species of turtles (Van Horn et al. [002]). We show the ability of HComP-Net
to generate novel hypotheses about evolutionary traits at different levels of the phylogenetic tree of organisms.
2 RELATED WORKS
One of the seminal lines of work in the field of prototype-basee interpretability methods is the framework of ProtoPNet (Chen et al. 2019) that learns set of "prototypical patches" from training images of every class to enable case- reasoning. Following this work, several variants have been developed, such as ProtoPShare (Rymarczyk et al.] 2021). ProtoPool (Rymarczyk et al.) 2022), ProtoTree (Nauta et al. 2021), and HPnet (Hase et al. 2019) suiting to different interpretability requirements. Among all these approaches, our work is closely related to HPnet (Hase et al. 2019),
the hierarchical extension of ProtoPNet that learns a prototype layer for every parent node in the tree. Despite sharing similar motivation, HPnet is not designed to avoid the learning of over-specific prototypes or to abstain from learning common prototypes at higher levels of the tree.
Another related line of work is the framework of PIPNet (Nauta al. 2023), which uses self- supervised learning to reduce the "semantic gap" (Hoffmann et al., 2021 Kim et al. (002)) between the latent space of prototypes and the space of images, such that the prototypes in the latent space correspond to the same visual concept in the image space. In HComP-Net, we build upon the idea of self-supervised learning introduced in PIPNet to learn a semantically consistent hierarchy of prototypes. Our work is also related to ProtoTree (Nauta et al.) 2021). which structures the prototypes as nodes in decision tree to offer more granular interpretability. However, ProtoTree differs from our work in that it learns the tree-based structure of prototypes automatically from data and cannot handle known hierarchy. Moreover, the prototypes learned in ProtoTrre are purely discriminative and allow for negative reasoning, which is not aligned with our objective of finding common traits of descendant species.
Other related works that focus on finding shared features are ProtoPShare (Rymarczyk et al. 2021 and ProtoPool (Rymarczyk et al. 2022). Both approaches aim to find common features among classes, but their primary goal is reduce the prototype count by exploiting similarities among classes, leading to a sparser network. This is different from our goal of finding a hierarchy of prototypes to find evolutionary traits common group of species absent in other species.
3

---

Published as a conference paper at ICLR 2025
X
K
Prototype
Scones
ortt
Prosolpp vectors
Grobal Man- Mao-proling
LCB
Inner
Ex + Lenap Edise
Chidd
dessee
CNN
Node vectors
Product
Chidl
Augmentenn led
Mosking module
Masking module
Node Prototype vectors
Cownowalf score
Mj
Inner Product
Gurbeel Sotmar
desses
Vi
Lmask
Leamabll
Figure Schematic illustration of HComP-Net model architecture.
Outside the realm of prototype- prosotype-based methods, the framework of Phylogeny-guine -guided Neural Net- works (PhyloNN) (Elhamod et al. (2023) shares similar motivation tionary traits by representing biological images in feature spaces structured by tree-based knowledge (i.e., phylogeny). However, PhyloNN primarily focuses the tasks of image generation and trans- lation rather than interpretability. Additionally, PhyloNN can only work with discretized trees with fixed number of ancestor levels per leaf node, unlike our work that does not require any discretization of the tree.
work to discover evolu-
There are also methods for learning hierarchical features in image classification that can clas- sify
categories within coarse-graines datasets (Yan et al. 2015 Hu et al.) 2016 Taherkhani et al. 2019). These methods use semantic hierarchies that separate broad super- categories and sub-categories instead of enforcing multi-level hierarchy among fine-grained im- ages (for example, images of birds only). Other related works include regularization techniques to capture commonalities between similar classes Goo al. (2016), and hierarchical interpretability frameworks Dai et al. (2023) that extend the concept of concept whitening (Chen et al. 2020) to incorporate known hierarchies. However, the interpretability offered by these approaches cannot localize to fine- the-grained image parts. In contrast, our approach enforces a multi-level predefined hierarchy images, and learns prototypes that represent localized attributes common to different nodes of the hierarchy.
PROPOSED METHODOLOGY
3.1 HCOMP-NET MODEL ARCHITECTURE
Given a phylogenetic tree with internal nodes, the goal of HComP-Net is to jointly learn a set of
prototype vectors Pn for every internal node f1....,N N). Our architecture shown in Figure begins with a CNN that acts as a common feature extractor [z; f) for all nodes, where represents the learnable parameters of f. f converts an image into a latent representation Z IR ffx XC where each "patch" at location (h, w) is, Zh, E R Following the feature extractor, for every node we initialize set of K,, prototype vectors P,, prototypes K,, learned at node n varies in proportion to the number of children of node n, with the proportionality constant, i.c., at each node assign 3 prototypes for every child node. To simplify notations, we drop the subscript n in Pn and K,, while discussing the operations occurring in node
), where pı E C.C Here, the number of
We consider the following sequence of operations at every node n. first compute the similarity score between every prototype in and every patch in Z. This results a matrix E R #x where every element represents similarity score between image patches and prototype vectors. We
4

---

Published as a conference paper at ICLR 2025
apply softmax operation across the K channels of 2 such that the vector ža E R at spatial location (h, w) in Z represents the probability that the corresponding patch 2 is similar the K prototypes. Furthermore, the 1t channel of 2 serves as prototype score map for the prototype vector Pi indicating the presence of pi in the image. We perform global max-pooling across the spatial dimensions H x W of Z to obtain vector g E E where the 7th element represents the highest similarity score of the prototype vector pj across the entire image. g is then fed to a linear classification layer with weights é produce the final classification scores for every child node of node n. We restrict the connections the classification layer so that every child node ne connected to distinct set of prototypes, ensure that every prototype uniquely maps child node. & is restricted to be non-negative to ensure that the classification is done solely through positive reasoning, similar the approach used in PIP-Net (Nauta et al., 2023). We borrow the regularization scheme of PIP-Net to induce sparsity in & by computing the logit of child node ne as lgg( + 1). g and @ here are again unique to each node.
3.2 Loss FUNCTIONS USED TO TRAIN HCOMP-NET
Contrastive Losses for Learning Hierarchical Prototypes: PIP-Net (Nauta et al. 2023) intro- duced the idea of using self-supervised contrastive learning to learn semantically meaningful proto- types. We build upon this idea in our work to learn semantically meaningful hierarchical prototypes at every node in the tree as follows. For every input image x, we pass in two augmentations of the image, x' and x" to our framework. The prototype score maps for the two augmentations, Ž' and 2", are then considered as positive pairs. Since w.,w E R represents the probabilities of patch Zp.w being similar to the prototypes from P, we align the probabilities from the two augmentations 2,. and " to be similar using the following alignment loss:
LA=
1
Σ
log(2n, ?n,,w)
HW (b,0)) xW
Since Σx= 2n,w,1 = due to softmax operation, CA is minimum (i.e., =A= 0) when both E and " are identical one-hot encoded vectors. A trivial solution that minimizes LA is when all patches across all images are similar to the same prototype. To avoid such representation collapse, use the following tanh-loss L of PIP-Net (Nauta et al. (023), which serves the same purpose as uniformity losses in Wang & Isola (200)) Silva & Rivera (2022):
Cr
1 K
Σ
log(tanh(
Σ Bb.)))
(2)
where gb.i is the prototype score for prototype with respect to image b of mini-batch. Lr en- courages each prototype pi to be activated at least once in a given mini-batch B images, thereby helping to avoid the possibility of representation collapse. The use of tanh ensures that only the presence of a prototype is taken into account and not its frequency.
To achieve the goal of learning prototypes common to all descendant species of an internal node, we introduce novel loss, termed ever-specificity loss Loesp. that avoids learning over-specific prototypes any node Lousp is formulated as modification of the tanh-loss such that prototype pj is encouraged to be activated at least once in every one of the descendant species d {I....,yy of its corresponding child node in the mini-batch of images fed to the model, as follows:
Over-speyyyyeity Loss:
D. MV
Loosp
1 K
log(tanh( Σ gb.i)),
d=1
Be
where Ba is the subset of images in the mini-batch that belong to species d.
Discriminative loss: In order to ensure that a learned prototype for a child node n. is not activated by any of its contrasting set of species (i.e., species that are descendants of child nodes of n other than nz), we introduce another novel loss function, Laise: defined as follows:
K
Ldise
Σ
Σ
max(gb..); Pe
(4)
K
i= de D.
5

---

Published as a conference paper at ICLR 2025
where D, is the contrasting set of all descendant species of child nodes of other than Rg. This is similar to the separation loss used in other prototype-basee methods such as ProtoPNet (Chen et al., 2019), ProtoTree (Nauta et al. 2021), and TesNet (Wang al. 2021).
Orthogonality loss: We also apply kernel orthogonality as introduced in Wang et al. (2020) to the prototype vectors at every node n, that the learned prototypes are orthogonal and capture diverse
features:
Corth IPP T 112
(5)
where P is the matrix of normalized prototype vectors of size C x K, I is an identity matrix, and I1-11 is the Frobenius norm. Each prototype pi in P is normalized as, Pt PP
Classification loss: Finally, we apply cross-entropy loss for classification at each internal node as follows:
Ler fe Σ yh log(ūh)
where y is ground truth label and ÿ the prediction at every node of the tree.
3.3 MASKING MODULE TO IDENTIFY OVER- SPECIFIC PROTOTYPES
We employ an additional masking module at every node to identify over-specific prototypes with- out hampering their training. The learned mask for prototype Pj simply serves as an indicator of whether pi is over-specific not, enabling our approach to abstain from finding common prototypes there are none, especially at higher levels of the tree. To obtain the mask values, we first calculate the over-specificity score for prototype pi as the product of the maximum prototype scores obtained across all images in the mini-batch belonging to every descendant species d as:
D,
- O
II d=1
max (gb,i) (bERg)
where gb.i is the prototype score for prototype PI with respect to image b of mini-batch and Ba is the subset of images in the mini-batch that belong to descendant species d. Since gb,; takes value between and due to the softmax operation, O ranges from -1 to O, where -1 denotes least over-specificity and 0 denotes the most over-specificity. The multiplication of the prototype scores ensures that even when the score is less with respect to only one descendant species, the prototype will be assigned high over-speciffeity score (close to
As shown in Figure 3 O is then fed into the masking module, which includes a learned mask value Mi for every prototype Pj. We generate M, from Gumbel-softmal distribution (Jang et al.) 2016) so that the values are skewed to be very close to either 0 or 1, i.e., M;- Gumbel-Softmax/ (7, T), where 7/2 are the learnable parameters of the distribution and is temperature. We then compute the masking loss, Lmask, as:
K (Amask stopgrad((.)) + Az, |
maask
(8)
1=1
where Amask and ÅL are trade-off coefficients, ||-I11 is the T., norm added to induce sparsity in the masks, and stopgrad represents the stop gradient operation applied over O, ensure that the gradient of mmakk does not flow back to the learning of prototype vectors and impact their training. Note that the learned masks are not used for praning the prototypes during training, they are only used during inference to determine which of the learned prototypes are over-specific and likely to not represent evolutionary traits. Therefore, even all the prototypes are identified as over-specific by the masking module at an internal node, it will not affect the classification performance at that node.
3.4 TRAINING HCOMP-NET
We first pre-train the prototypes at every internal node in a self-supervised learning manner using alignment and tanh-losses as Las = + Cr. We then fine-tune the model using the follow- ing combined loss: (ACELeE + Lss + Aovsp Adisch + North + Lmakk). where A's are trade-off off parameters. Note that the loss is applied over every node in the tree. We show an ablation of key loss terms used in framework in Table in Appendix

---

Published as a conference paper at ICLR 2025
4 EXPERIMENTAL SETUP
Datasets: In our experiments, we primarily focus the 190 species of birds (Bird) from the CUB- 200-2011 (Wah et al. 2011) dataset, for which the phylogenetic relationship dJetz et al. 2012) is known. The tree is quite large, with total of 184 internal nodes. We removed the background from the images to avoid the possibility of learning prototypes corresponding to background information, such as the bird's habitat, as we are only interested in the traits corresponding to the body of the organism. We also apply our method on fish dataset with 38 species (Fish) (Elhamod et al.) (023)
along with its associated phylogeny (Elhamod et al. 2023), 30 subspecies of Heliconius butterflies (Butterfly) from the Jiggins Heliconius Collection dataset (Lawrence & Campolongo (0024) col- lected from various sources (see Appendix We also consider certain subsets of INar2021 dataset
Van Horn et al. 2021 where the set of species are not too diverse (such elephants and whales from class Mammalia), but exhibit fine-grained, hard-to-quantify hierarchical variation in visually observable traits, such that
find interpretable commonalities between the species. We reiter- ate that our goal is to discover local prototypes as evolutionary traits, c.g., beak color, stripe pattem, and fin curvature. Traits differentiating species at higher levels of the tree often not local, such as global contours, which are beyond the scope of our study. Therefore, we consider 41 species from order Testudines (Turtle) and 113 species from order Araneae (Spider) in the iNat2021 dataset. The phylogeny for Butterfly, Turtle, and Spider were obtained using the rot package (Redelings) al., 2019; Michonneau et al., 2016). Further details of dataset statistics, phylogeny statistics, hyperparameter settings and training strategy are provided in Appendix [F]
Baselines: We compare HComP-Net to ResNet-50 (He et al., 2016), ConvNeXtt-inn(LLi et al. 2022). ProtoPNet (Chen et al. 2019). INTR (Interpretable Transformer) (Paul et al., 2023). PIP-Net (Nauta et al. 2023) and HPnet (Hase et al., 2019) as classification baselines. For HPnet, we used the same hyperparameter settings and training strategy as used by ProtoPNet for the CUB-200-201 dataset. To ensure a fair comparison, we set the number of prototypes per child class to 10 for both HComP-Ne and HPnet on the Bird, Butterfly, and Fish datasets. For the Spider and Lizard datasets, due to the higher diversity of images, we increased the number of prototypes to 20 for both methods. We follow the same training strategy as provided by ProtoPNet for the CUB CUB-200-200 200-2011 dataset.
5 RESULTS
5.1 FINE-GRAINED ACCURACY
Similar HPnet Hase et al. 2019).
Table 1: % Fine- fine-grained Accuracy
we calculate the fine- tite-graied accuracy for each leaf node by calculating the path probability over every image.
Model
Hicrarchy Bird Butterfly Fish Spider Turtle
ResNet-50
74.18
95.76
86.63
74.17 56.23
ConyNeXt-tiny ProtoPNet
84.23
96.28
91.73
84.05
69.56
During inference, the final probabil- ity for leaf class Y given an image X is calculated as, P(Y]X) = P(Y(), (()) y()),.,,, [x)
No
75.80
96.82
91.11
74.17
61.74
INTR
69.22
95.53
86.73
78.91
58.64
PIPNet
83.38
97.87
93.58
83.45
67.24
IT',, PXY where P(Y(())) is the probability of assigning the image X to a node at level I, and L is the
HPnet
Yes
36.18
94.69
77.51
5.85
6.38
HComP-Net
70.01
97.35
90.80
76.19
58.26
depth of the leaf node. Every image assigned to the leaf class with maximum path probability, which is used to compute the fine-grained accuracy. The comparison of the fine-grained accuracy calculated for HComP-Net and the basclines are given in Table [ (best performance bolded and second- bestt performance underlined). Note that despite extensive hyperparameter tuning, HPnet was unable to perform well on Spider and Turtle datasets (See Appendix [F] for details). It is important to note that the primary objective of our work is identifying evolutionary traits through semantically meaningful prototypes (both quantitatively and qualitatively evaluated in Section 5.3 and (ne..../reepcccivlly. and not necessarily achieving high classification accuracy (which may require capturing global features or features prone to dataset bias). Additionally, we observe that the Spider and Turtle datasets from iNat2021 are relatively more challenging since they contain noisy elements, including blurry images, obscured foregrounds and camouflaged backgrounds, that cause models to misclassify these images (examples provided in Figure of Appendix).
7

---

Published as a conference paper at ICLR 2025
biled Ani
Black billed
Black
bilied
Cuskoo
Mengrove
Mangrove
(a) HPnet
(b) HComP-Net
Figure 4: Comparing the part consistency HPnet and HComP-Net for their prototype learned at an inter- nal node in the Bird dataset that corresponds to descendant species (names shown on the rows). For every species, we are visualizing the top-3 images with the highest prototype score for both HPnet and HComP-NV -Net, shown the three columns with zoomed in views of their discovered prototypes. We can that HPnet high- lights varying parts of the bird difficult associate consistent semantic meaning to its learned prototype. In contrast, HComP-Ner consis- tenrly highlights the head region of the bird across all four species and their images.
the 3 species and across multiple images of the
species, making
5.2 GENERALIZING TO UNSEEN SPECIES IN THE PHYLOGENY
We analyze the performance of HComP-Noe in generaliz- ing to unseen species that the model has not seen during training. The biological motivation for this experiment is to evaluate if HComP-Net can situate newly discovered species at its appropriate position in the phylogeny by identifying its common ancestors shared with the known species. An added advantage of our work is that along with identifying the ancestor of unseen species, we also identify the common traits shared by the novel species with known species in the phylogeny.
Table 2: % Fine-graine Accuracy
(on unseen species)
Species Name
HComP-Nel
HPnet
Fish Crow
53.33
10.55
Rock Wren
53.33
10.22
Indigo Bunting
96.67
49.2
Bohemian Waxwing 70.00
44.9
Since unseen species cannot be classified the finest levels (i.e., up the leaf node corresponding to the unseen species), we analyze the ability of HComP-Net to classify unseen species accurately up one level above the leaf level in the hierarchy. With this consideration, the final probability of an unseen species for given image is calculated as, Xun)eeneeen) = P(Y()), ( (f.-1)) (1--1) [X)
T (( X). Note that we leave out the class probability the I.th level, since we do not take into account the class probability of the leaf level. Since such path probability can only be calculated for hierarchical methods, we compare our approach to HPnet which also learns hierarchical proto- types. We leave four species from the Bird training set and calculate their accuracy during inference Table 2
We can see that HComP-Nee is able to generalize better than HPnet for all four species.
5.3 ANALYZING THE SEMANTIC QUALITY OF PROTOTYPES
Following the method introduced in PIPNet
Table 3: Part purity of prototypes on Bird dataset.
Nauta al. 2023), assess the semantic quality of our learned prototypes by evaluat- ing their part purity. prototype with high part purity (close to 1) is one that consis- tently highlights the same image region in the score maps (corresponding consistent local features, such as the eye or wing a bird) across images belonging to the same class.
Model
Covsp Masking Part purity % masked
HPnet
0.14 0.09
HComP-NE
0.68 0.22
HComP-Ne
v 0.75 + 0.17 21.42%
HComP-Nee
0.72 0.19
: 0.77 + 0.16 16.53%
HComP-Net
The part purity is caleulated using the part locations of 15 parts that provided in the CUB dataset. For each prototype, we take the top-10 10 images from each leaf descendant. We consider the 32 x 32 image patch that centered around the max activation location of the prototype from the top-10
8

---

Published as a conference paper at ICLR 2025
images. With these top-10 image patches, calculate how frequently each part is present inside the image patch. For example. a part that is found inside the image patch 8 out of 10 times is given a score of 0.8. In PIP-Net, the highest value among the values calculated for each part is given as the part purity of the prototype. In our approach, since we are dealing with a hierarchy and taking the top-10 from each leaf descendant, particular part, let's say the eye, might have score of 0.5 for one leaf descendant and 0.7 for different leaf descendant. Since we want the prototype to represent the same part for all the leaf descendants, we take the lowest score (the weakest link) among all the leaf descendants as the score of the part. By following this method, for a given prototype we can arrive at a value for each part and finally take the maximum among the values as the purity of the prototype. We take the mean of the part purity across all the prototypes and report the results in Table 3 for different ablations of HComP-Net and also HPnet, which is the only baseline method that can learn hierarchical prototypes.
We can see that HComP-Net, even without the use of ope-sfieyificcty loss, performs much better than HPnet due to the contrastive learning approach we have adopted from PIPNet (Nauta et al. 2023) The addition of over-specifiiity loss improves the part purity because over-specifie prototypes tend to have poor part purity for some of the leaf descendants, which will affect their overall part purity score. Further, for both ablations with and without over-specificity loss, we apply the masking module and remove masked (over-specific) prototypes during the calculation of part purity. We see that the part purity goes higher by applying the masking module, demonstrating its effectiveness in identifying over-specific prototypes. We further compute the purity of masked-out prototypes and notice that the masked-out prototypes have drastically lower part purity (0.29 + 0.17) compared to non-masked prototypes (0.77 + 0.16). We also provide a visual comparison of masked (over- specific) prototype and an unmasked (non-over-specific) prototype in Appendix H An alternative approach to learning the masking module to identify over-specific prototypes using fixed global threshold over O, We show in Table Appendix that given the right choice of such threshold, we can identify over-speeific prototypes. However, selecting the ideal threshold can be non-trivial. On the other hand, our masking module learns the appropriate threshold dynamically as part of the training process.
Figure visualizes the part consistency of prototypes discovered by HComP-Net in comparison to HPnet for the bird dataset. We can see that HComP-Net is finding a consistent region in the image (corresponding to the head region) across all three descendant species and all images of species, in contrast to HPnet. Since HPnett is implemented with a 7 x 7 latent space in comparison to HComP- Net which uses 26 x 26 latent space, we explore HPnet with higher resolution feature map of 28 x 28 and report the qualitative and quantitative results in Figure in Appendix We note that the performance of HPnet does not improve with higher resolution feature maps. Furthermore, thanks to the aligmment loss, in HComP-Net every patch 2n.w is encoded as nearly one-hot encoding with respect to the K prototypes which causes the prototype score maps to be highly localized. The concise and focused nature of the prototype score maps makes the interpretation much more effective compared to baselines.
5.4 ANALYZING EVOLUTIONARY TRAITS DISCOVERED BY HCOMP-NET
We now qualitatively analyze some of the hypothesized evolutionary traits discovered in the hierar- chy of prototypes learned by HComP-Net. Figure 5 shows the hierarchy of prototypes discovered over a small subtree of the phylogeny from Bird (four species) and Fish (three species) dataset. In the visualization of bird prototypes, we can see that the two Pelican species share a consistent re- gion in the learned Prototype labeled 2, which corresponds to the head region of the birds. We can hypothesize this prototype is capturing the white-colored crown common the two species. On the other hand, Prototype 1 finds the shared trait of similar beak morphology (c.g., sharpness of beaks) across the two Cormorant species. We can see that HComP-Net avoids the learning of over-specific prototypes at internal nodes, which are pushed down to individual leaf nodes, as shown in visual- izations of Prototype 3, 4, 5, and 6. Similarly, in the visualization of the fish prototypes, we can see that Prototype I is highlighting a specific fin (dorsal fin) of the Carassius auratus and Notropis hud- sonius species, possibly representing their pigmentation and structure, which is noticeably different compared to the contrasting species of Alosa chrysochloris. Note that while HComP-Ne identifies the common regions corresponding to each prototype (shown as heatmaps), the textual descriptions of the traits provided in Figure are based on human interpretation.
9

---

Published as a conference paper at ICLR 2025
Buak morphology
White colores crown
Strongly pigmonted colored dorsal fin"
Black farnhand"
eye magion Wed faced Cormorant
White forwhand" White Pelican
Brown underbally
Prominent gill cover"
Caudal spol
Head marphology chrysochioris
Pelogic Commorant
Brown Pelican
auratus
Notropts
Figure 5: Visualizing hierarchy prototypes discovered by HComP-Net for Birds Fishes. Note that the textual descriptions of the hypothesized traits shown for every prototype are based on human interpretation. Visualizations for Butterfly, Turtle, and Spider datasets provided in the Appendix
INTR
Attention heads comparable to HComP-Hel protofypes in (NTR (Interpretable Tranaformer)
HComP-Not Prototype
Original Image
Phylogenatic Tree
Level0
Head 3
eyes
Level-2
Head 5
bill
Head
Albatoos
Billed
Marned Grabr
Weatern Grebe
Figure 6: We trace the prototypes learned for Western Grebe at three different levels in the phylogenetic tree (corresponding to different periods of time in evolution). Text in blue is the interpretation of common traits of descendants found by HComP-Net every ancestor node Western Grebe
Figure 6 shows another visualization of the sequence of prototypes learned by HComP-Net for the Western Grebe species at different levels of the phylogeny. We can see that at level 0, we are capturing features closer to the neck region, indicating the likely difference between the length of necks between Grebe species and other species (Cuckoo, Albatross, and Fulmar) that diversify at earlier time in the process of evolution. At level the prototype is focusing the eye region, potentially indicating a difference in the color of red and black patterns around the eyes. At level we are differentiating Western Grebe from Horned Grebe based on the feature of bills. We also validate our prototypes by comparing them with the multi-head cross-attention maps learned by INTR (Paul et al. 2023). We can see that some of the prototypes discovered by HComP-Net can be mapped to equivalent attention heads of INTR. However, while INTR is designed to produce a flat structure of attention maps, we are able place these maps on the tree life. This shows the power of HComP-Nee in generating novel hypotheses about how trait changes may have evolved and accumulated across different branches the phylogeny. Additional visualizations discovered evolutionary traits from all five datasets are provided in the Appendix (Figures 10 29
CONCLUSION
We introduce novel approach for learning hierarchy- prototypes while avoiding the learning of over-specific features at internal nodes of the phylogenetic tree, enabling the discovery of novel evolutionary traits. Our empirical analysis birds, fishes, and butterflies demonstrates the efficacy of HComP-Net over baseline methods. Furthermore, HComP-Net demonstrates a unique ability generate novel hypotheses about evolutionary traits, showcasing its potential in advancing our understanding of evolution. We discuss the limitations of our work in Appendix [ While we focus on the biological problem of discovering evolutionary traits, our work be applied in general to domains involving a hierarchy of classes, which can be explored in future research.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENT
This research is supported by National Science Foundation (NSF) awards for the HDR Ima- geomics Institute (OAC-2118240), We are thankful for the support of computational resources provided by the Advanced Research Computing (ARC) Center at Virginia Tech. This manuscript has been authored by UT-Battelle, LLC, under contract DE-AC05-000R22725 with the US De- partment of Energy (DOE). The US government retains, and the publisher, by accepting the ar- ticle for publication, acknowledges that the US government retains nonexclusive, paid-up, irre- vocable, worldwide license to publish or reproduce the published form of this manuscript or al- low others to do so for US government purposes. DOE will provide public access these re- sults of federally sponsored research in accordance with the DOE Public Access Plan (https:
REPRODUCIBILITY
We have ensured the reproducibility of work by providing detailed set of resources. The source code used for training and evaluation, along with instructions for reproducing the experiments, available via Imageomies Institute GitHub site:
http:///hhhuboooommmgooooo////// which is mentioned in Appendix L Comprehensive implementation details, including hyperparam- eters and compute resources, are described in Appendix IF] Additionally, we have eited all dataset sources and outlined the data processing steps in Appendix F to facilitate accurate replication of our experiments.
REFERENCES
Abien Fred Agarap. Deep learning using rectified arXiv: arXiv:1803.08335. 2018.
linear units
(relu).
arXiv preprint
Chaofan Chen, Oscar Li, Daniel Tao, Alina Barnett, Cynthia Rudin, and Jonathan Su. This looks like that: deep learning for interpretable image recognition. Advances in neural information processing systems, 32, 2019.
Zhi Chen, Yijie Bei, and Cynthia Rudin. Concept whitening for interpretable image recognition. Nature Machine Intelligence, 2(12):772-782, 2020.
Haixing Dai, Lu Zhang, Lin Zhao, Zihao Wu, Zhengliang Liu, David Liu, Xiaowei Yu, Yanjun Lyu, Changying Li, Ninghao Liu, et al. Hierarchical semantic tree concept whitening for interpretable image classification. arXiv preprint 2:X.iv:337704344 2023.
Mohannad Elhamod, Mridul Khurana, Harish Babu Manogaran, Josef C Uyeda, Meghan A Balk, Wasila Dahdul, Yasin Bakis, Henry Bart Jr, Paula M Mabee, Hilmar Lapp, et Discovering novel biological traits from images using sesloognnngguddee neural networks. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 3966-3978, 2023.
R. Farrell. Cub-200-201 I segmentations (1.0) [data set], 2024. URL http:hts/s//a..c.lllo. edd:/recordd//wdd6egee
Wonjoon Goo, Juyong Kim, Gunhee Kim, and Sung Ju Hwang. rraxulomy-regullrrieed semantic deep convolutional neural networks. In Computer Vision-ECCV 2016: 14th European Confer- ence, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part II 14, pp. 86-101. Springer, 2016.
Luke Harmon, Jonathan B Losos, T Jonathan Davies, Rosemary G Gillespic, John L Gittleman, W Bryan Jennings, Kenneth H Kozak, Mark A McPeek, Franck Moreno-Roark, Thomas J Near, et al. Early bursts of body size and shape evolution are rare in comparative data. Evolution, 64 (8):2385-2396, 2010.
Peter Hase, Chaofan Chen, Oscar Li, and Cynthia Rudin. Interpretable image recognition with hierarchical prototypes. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, volume 7, pp. 32-40, 2019.
11

---

Published as a conference paper at ICLR 2025
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Adrian Hoffmann, Claudio Fanconi, Rahul Rade, and Jonas Kohler. This looks like that... does it? shortcomings of latent space prototype interpretability in deep networks. arXiv preprint arXiv:2105. 200002200..0296.. 2021.
David Houle and Daniela M Rossoni. Complexity, evolvability, and the process of adaptation. Anmual Review of Ecology, Evolution, and Systematics, 53, 2022.
Hexiang Hu, Shang-Tong, Zhou, Zhiwei Deng, Zicheng Liao, and Greg Mori. Learning structured inference neural networks with label relations. In Proceedings of the IEEE conference on com- puter vision and pattern recognition, pp. 2960-2968, 2016.
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparamettrrization with gumbel-softmax, arXiv preprint arXiv: 144,141,111111141 2016.
W. Jetz, G. H. Thomas, J. B. Joy, K. Hartmann, and A. O. Mooers. The global diversity of birds in space and time. Nature, 491:444-448, 2012. URL http: hbttt::/bbrrdtree org
Chris Jiggins and Ian Warren. Cambridge butterfly wing collection Chris Jiggins 2001/2 broods batch 1, January 2019a. URL
5281 /eeoooo.ooo... 2549524
Chris Jiggins and Ian Warren. Cambridge butterfly wing collection - Chris Jiggins 2001/2 broods batch 2, January 2019b. URL /KLLtttps///60..cr////./ 5281 2ea.8///zendo..222 2 2550097
Chris Jiggins, Gabriela terfly wing collection batch 3, May 2019. URL
Ian Warren, and Eva Wiltshire. Cambridge but-
5281/ zenodo.
2682458
Anuj Karpatne, Xiaowei Jia, and Vipin Kumar. Knowledge-guided machine learning: Current trends and future prospects. arXiv preprint arXiv:2403.15999, 2024.
Mridul Khurana, Arka Daw, M Maruf, Josef C Uyeda, Wasila Dahdul, Caleb Charpentier, Yasin BakIş, Henry L Bart Jr, Paula M Mabee, Hilmar Lapp, et al. Hierarchical conditioning of diffusion models using tree-of-life for studying species evolution. arXiv preprint arXiv:2408. 001600 2024.
Sunnie SY Kim, Nicole Meister, Vikram V Ramaswamy, Ruth Fong, and Olga Russakovsky. Hive: Evaluating the human interpretability of visual explanations. In European Conference on Com-
puter Vision, pp. 280-298. Springer, 2022.
Christopher Lawrence and Elizabeth G. Campolongo. Heliconius collection (cambridge butterfly). 2024. URL httts://huggingfaae co datasets/ /ttaaatttsiinagmmm...... imageomics/ Bellconius-Colless ort.:caooeeecclleeeeeoooooooooaa
Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang. Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su. Jun Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2003.05499, 2023.
Zhuang Liu, Hanzi Mao, Chao- Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 11976-11986 2022.
Moritz. D Lürig, Seth Donoughe, Erik Svensson, Arthur Porto, and Masahito Tsuboi. Computer vision, machine learning, and the promise of phenomics in ecology and evolutionary biology. Frontiers in Ecology and Evolution, 9:642774, 2021.
Chiyu Ma, Brandon Zhao, Chaofan Chen, and Cynthia Rudin. This looks like those: Illuminating prototypical concepts using multiple visualizations. Advances in Newral Information Processing Systems, 36, 2024.
Anniina Mattila, Chris Jiggins, and Ian Warren. University of Helsinki butterfly collection Anni- ina Mattila bred specimens, February 2019. URL batp::t/////t http///to..ooo/ooo//// wttp:///.....//eed...ooo 2555086
12

---

Published as a conference paper at ICLR 2025
Joana 1. Meier, Patricio Salazar, Gabriela Montejo- Kovaajicc,,,,,,cc,, Ian Warren, and Chris Jggins. Cam- bridge butterfly wing collection Patricio Salazar PhD wild specimens batch 3, October 2020.
URL htttts::://dd http://ooooog/g/.////// www.oo.gg/ooooooooooo... 1tt.:55811//ccce wtt.://0r//....//eeeeeeeeeo
4153502
Jason G Mezey and David Houle. The dimensionality of genetic variation for wing shape in drosophila melanogaster. Evolution, 5965):027-11338, 2005.
Francois Michonneau, Joseph W. Brown, and David J. Winter. rotl: an package to interact with the open tree of life data. Methods in Ecology and Evolution, 7(12):1476-1411, 2016. doi:
10.111//041-2100.
Gabriela Montejo-Kovacevich Chris Jiggins, and Ian Warren. Cambridge butterfly wing collection batch 1- version 2, May 2019a. URL https::/dooi wrw../..r....0 wtw.://22o22/eeooo2232 3082688
Gabriela Montejo-Kovacevvv Chris Jiggins, and Ian Warren. Cambridge butterfly wing collection
batch 2, May 2019b. htt.h:ttp::////o/ org/ 10. 5281/2211// 2281/zenodo.. 2677821
Gabriela Montejo-Kovacevich Chris Jiggins, and Ian Warren. Cambridge butterfly wing collection batch 4, May 2019c. URL
5281/2enodo 2682669
Gabriela Montejo-Kovacevice Chris Jiggins, Ian Warren, Camilo Salazar, Marianne Elias, Imogen Gavins, Eva Wiltshire, Stephen Montgomery, and Owen McMillan. Cambridge and collabora- tors butterfly wing collection batch 10, May 2019d. URL https://aci.orr/111 5281/ zenodo. 2813153
Gabriela Montejo- Kovacevich, Chris Jiggins, Ian Warren, and Eva Wiltshire. Cambridge butter- fly wing collection batch 7, May 2019e. URL
2702457
Gabriela Kontejo-Kovaceviche Chris Jiggins, Ian Warren, and Eva Wiltshire. Cambridge butter- fly wing collection batch 8, May 2019f. URL
2707828
Gabriela Montejo-Kovacevvv Chris Jiggins. Ian Warren, and Eva Wiltshire. Cambridge butter- fly wing collection batch 5, May 2019g. URL
26844906
Gabriela Montejo-Kovacevinn Chris Jiggins, Ian Warren, and Eva Wiltshire. Cambridge butter- fly wing collection batch 6, May 2019h. URL http:://bool.rrrr/00.228////
26867622
Gabriela Montejo-Kovacevice Chris Jiggins, Ian Warren, Eva Wiltshire, and Imogen Gavins. Cam- bridge butterfly wing collection batch 9, May 2019i. URL Atpps://@d..orr/11...22
zerodo. 2714333
Gabriela Montejo-Kovacevidee Eva van der Heijden, and Chris Jiggins. Cambridge butterfly col- lection GMK Broods Ikiam 2018, November 2020a. URL
zerodo.4291099 4291095
Gabriela
Eva van der Heijden, Nicola Nadeau, and Chris Jiggins. Cambridge butterfly wing collection batch 10, November 2020b. URL bttps://aci.org//I 5281/
zzeoooo 4269223
Gabriela Montejo-Kovacevich, Kovacevich, Quentin Paynter, and Amin Ghane. Heliconius erato cyrbia, Cook Islands (New Zealand) 2016, 2019, 2021, September 2021. URL httpttt/////.....oo
151:2555.526257
Samuel G Müller and Frank Hutter. Trivialaugment: tion. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 774-782, 2021.
Tuning-free yet state-of-the-art data augmenta-
Meike Nauta, Ron Van Bree, and Christin Seifert. Neural prototype trees for interpretable fine- grained image recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14933-14943, 2021.
13

---

Published as a conference paper at ICLR 2025
Meike Nauta, Jörg Schlötterer, Maurice van Keulen, and Christin Seifert. Pip-net: Patch-based intuitive prototypes for interpretable image classification. In Proceedings of the IEEE/CVF Con- ference Computer Vision and Pattern Recognition, pp.
2023.
Maureen A O'Leary and Seth Kaufman. Morphobank: phylophenomics in the "cloud". Cladistics, 27(5):529-55 537, 2011.
Dipanjyoti Paul, Arpita Chowdhury, Xinqi Xiong, Feng-Ju Chang, David Carlyn, Samuel Stevens, Kaiya Provost, Anuj Karpatne, Bryan Carstens, Daniel Rubenstein, et al. A simple interpretable transformer for fine-gres tne-grained image classification and analysis. arXiv preprint 1.0157231 2023.
Matthew W Pennell, Richard G FitzJohn, William K Cornwell, and Luke Harmon. Model ade- quacy and the macroevolution of angiosperm functional traits. The American Naturalist, 186(2): E33-E50, 2015.
Erika Pinheiro de Castro, Christopher Jiggins, Karina Lucas da Silva-Brand00ee3, Andre Vic- tor Lucci Freitas, Marcio Zikan Cardoso, Eva Van Der Heijden, Joana Meier, and Ian War- ren. Brazilian Butterflies Collected December 2020 January 2021, February 2022. URL https: petts://oot /ttp:///oi..o
MARTin J RAMirez, Jonathan A Coddington, Wayne P Maddison, Peter E Midford, Lorenzo Pren- dini, Jeremy Miller, Charles E Griswold, Gustavo Hormiga, Petra Sierwald, Nikolaj Scharff, al. Linking of digital images to phylogenetic data matrices using a morphological ontology. System- atic Biology, 56(2):283-22) 294, 2007.
Benjamin Redelings, Luna Luisa Sanchez Reyes, Karen A. Cranston, Jim Allman. Mark Holder, and Emily Jane McTavvih. Open tree of life synthetic tree, 2019. URL 10. 5281/281/// zenodo. 3 3937741
org/
Dawid Rymarezyk, Lukasz Struski, Jacek Tabor, and Bartosz Zieliński. Protopshare: Prototypical parts sharing for similarity discovery in interpretable image classification. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pp. 1420-1430, 2021.
Dawid Rymarczyk, Lukasz Struski, Michat Górszezak, Koryna Lewandowska, Jacek Tabor, and
Bartosz Zielinski. Interpretable image classification with differentiable prototypes assignment. In European Conference on Computer Vision, pp. 351-368. Springer, 2022.
Camilo Salazar, Gabriela Montejo-Kovaceny Chris Jiggins, Ian Warren, and Imogen Gavins. Camilo Salazar and Cambridge butterfly wing collection batch 1, May 2019a. URL https:
Patricio Salazar, Gabriela Montejo-Kovacevich, Warren, and Chris Jiggins. Cambridge butterfly wing collection Patricio Salazar PhD wild and bred specimens batch 1. December 2018. URL
Patricio Salazar, Gabriela Montejo-Kovacevict Ian Warren, and Chris Jiggins. Cambridge butterfly wing collection Patricio Salazar PhD wild and bred specimens batch 2, January 2019b. URL
https:// http:://ooi.orr.... 2tt.::////2222..25/////4
Patricio A. Salazar, Nicola Nadeau, Gabriela
and Chris Jiggins. Sheffield butterfly wing collection Patricio Salazar, Nicola Nadeau, Ikiam broods batch I and 2, November 2020. URL/tttps://daa
orq/10...10... 5281 zenodo. 4288311
Paul Screno. Logical basis for morphological characters in phylogenetics. Cladistics, 23(6): 565-587, 2007.
Thalles Silva and Adín Ramírez Rivera. Representation learning via consistent assignment of views to clusters. In Proceedings of the 37th ACM/SIGAPP Symposium on Applied Computing, pp-
987-994, 2022.
Tiago R Simões, Michael W Caldwell, Alessandro Palci, and Randall L Nydam. Giant taxon- character matrices: quality of character constructions remains critical regardless size. Cladis- fics, 3382):198-21, 2017.
14

---

Published as a conference paper at ICLR 2025
Randal A Singer, Kevin J Love, and Lawrence M Page. A survey of digitized data from us fish collections in the idigbio data aggregator. PloS one, 13(1)xx0007636, 2018.
Fariborz Taherkhani, Hadi Kazemi, Ali Dabouei, Jeremy Dawson, and Nasser M Nasrabadi. A weakly supervised fine label classifier enhanced by coarse supervision. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 6459-6468, 2019.
Grant Van Horn Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. The inaturalist species classification and detection dataset. In Proceedings of the IEEE conference computer vision and paltern recognition, pp. 8769-8778, 2018.
Grant Van Horn, Elijah Cole, Sara Beery, Kimberly Wilber, Serge Belongie, and Oisin Mac Aodha. Benchmarking representation learning for natural world image collections. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 12884-12893, 2021.
Catherine Wah, Steve Branson, Peter Welinder, Pictro Perona, and Serge Belongie. The caltech-ucsd birds-200-2011 dataset. 2011.
Jiaqi Wang, Huafeng Liu, Xinyue Wang, and Liping Jing. Interpretable image recognition by con- structing transparent embedding space. In Proceedings of the IEEE/CVF International Confer- ence on Computer Vision, pp. 895-904, 2021.
Jiayun Wang, Yubei Chen, Rudrasis Chakraborty, and Stella X Yu. Orthogonal convolutional neural networks. In Proceedings the IEEF/CVE conference computer vision and pattern recogni- tion, pp. 11505-11515,, 2020.
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through align- ment and uniformity on the hypersphere. In International Conference on Machine Learning, pp- 9939. PMLR, 2020.
Ian Warren and Chris Jiggins. Miscellancous Heliconius wing photographs (2001-2019) Part 2, February 2019a. URL
www.:/2000....222222 2553501
Ian Warren and Chris Jiggins. Miscellaneous Heliconius wing photographs (2001-2019) Part 1, February 2019b. URL htt.bttts.//d...rgg////// wttp://100...00222m2o22222222230
Ian Warren and Chris Jiggins. Miscellaneous Heliconius wing photographs (2001-2019) Part 3, February 2019c. URL https:://oi..orr///// 00t.:..22122eo//// 2553977
Zhicheng Yan, Hao Zhang, Robinson Piramuthu, Vignesh Jagadeesh, Dennis DeCoste, Wei Di, and Yizhou Yu. Hd-cnn: hierarchical deep convolutional neural networks for large scale visual recognition. In Proceedings of the IEEE international conference computer vision, pp. 2740- 2748, 2015.
15

---

Published as a conference paper at ICLR 2025
APPENDIX
TABLE OF CONTENTS
A Additional Biological Background
16
B Ablation of Over-specificity Loss Trade-off Hyperparameter
17
Ablation of Number of Prototypes
17
Ablation of Individual Losses
17
Consistency of Classification Performance Over Multiple Runs
18
Implementation Details
18
Post-hoe Thresholding to Identify Over-specific Prototypes
20
H Visual Comparison of an Over-specific and Non-over-specifie prototype
20
I Performance of HPnet with high resolution feature maps
21
J Additional Visualizations of the Hierarchical Prototypes Discovered by HComP-Nel
21
Additional Top-K Visualizations of HComP-Net Prototypes
21
Reproducibility
23
M Limitations of Our Work
24
A ADDITIONAL BIOLOGICAL BACKGROUND
One of the first steps any study of evolutionary morphology is character construction - the process of deciding which measurements will be taken of organismal variation that are replicable and mean- ingful for the underlying biology, and how these traits should be represented numerically (Mezey & Houle, 2005). For phylogenetic studies, researchers typically attempt identify synapomorphies versions of the traits that are shared by two or more species, are inherited from their most recent common ancestor, and may have evolved along the phylogeny branch. The difficulty with the tradi- tional character construction process is that humans often measure traits in a way that is inconsis- tent and difficult to reproduce, and can neglect shared features that may represent synapomorphies, but defy easy quantification. To address the problem of human inconsistency, PhyloNN (Elhamod) al. 2023) and Phylo- -iifuu-inf (Khurana et al. 2024) took a wwwwwlldg-guiddddd machine learning (KGML) (Karpatne et al. 2024) approach to character construction, by giving their neural networks knowledge about the biological process they were interested in studying (in their case, phylogenetic history), and specifically optimizing their models to find embedded features (analogous to biolog- ical traits) that are predictive of that process. To address the problem of visual reproducibility, RAMirez et al. (2007) suggested photographing the local structures where the empirical traits vary and linking the images to written descriptions of the traits. In this paper, we take influence from both approaches. We extend the hierarchical prototype approach from Hase et al. (2019) to better reflect phylogeny, similar in theory to the way PhyloNN (Elhamod et al. 2023) and Phylo-Diffusion (Khurana et al. 2024) learned embeddings that reflect phylogeny. Using prototypes, however,
16

---

Published as a conference paper at ICLR 2025
enforce local visual interpretabilityy similar to how researchers may "type-specimens" to define prototypical definitions of particular character states.
Specifically, method is about finding synapomorphies shared derived features unique particular group of species that share a common ancestor in the phylogeny (referred to as clade). While such features may bear similarities to convergent phenotypes in other clades, goal is not to identify features that exhibit convergence. is typical for phylogenetic studies to specifically avoid features that exhibit high levels of convergence, they can lend support for erroneous phylogenetic relationships. Identifying convergence requires additional information such as shared habitat, niche, diet, or behavior, which is not incorporated in our work.
B ABLATION OF EVER-SPECIFICCTY LOss TRADE-OFF HYPERPARAMETER
We have provided an ablation for the over-specificity loss trade-off hyperparameter (Aousp) in Table We can observe that increasing the weight of over-specificity loss reduces the model's classifi- cation performance, as the model struggles to find any commonality, especially at internal nodes where the number of leaf descendant species is large and quite diverse. is natural that species that are diverse and distantly related may share fewer characteristies with each other, in comparison to a set species that diverged more recently from a common ancestor (Harmon et al. 2010 Pennell et al., 2015). Therefore, forcing the model to learn common traits with a strong Lousp constraint can cause the model to perform badly terms of accuracy.
Table Ablation of over-specificity loss trade-off h hyperparameter (Nousp). Done on Bird dataset.
Aouep
Part purity
Part purity with mask applied
% masked
% Accuracy
w/o Cousp
0.68 0.22
0.75 + 0.17
21.42%
58.32
0.05
0.72 0.19
0.77 + 0.16
16.53%
70.01
0.1
0.71 = 0.18
0.74 + 0.16
11.31%
70.97
0.5
0.71 0.19
0.18
4.2%
68.23
1.0
0.70 = 0.19
0.70 0.2
2.13%
62.68
2.0
0.69 0.19
0.69 + 0.19
0.55%
53.16
C ABLATION OF NUMBER OF PROTOTYPES
In Table we vary the number prototypes per child 3 for a node to see the impact on the model's performance, We note that while the accuracy increases marginally with increasing the number of prototypes per child (5) from 10 to 15, it does not affect the performance of the model significantly. Therefore, continue work with 10 for all of our experiments.
Table 5: Ablation of the number of prototypes per child for node (3). Done on Bird dataset.
Number of Prototypes (3)
% Accuracy
10
70.01
15
70.92
20
69.99
D ABLATION OF INDIVIDUAL LOSSES
In Table 6
we perform an ablation of the various loss terms used in our methodology. As it can be observed, the removal of Lousp and Laise degrades performance in terms of both semantic consis- tency (part purity) and accuracy. On the other hand, the removal of self-supervised contrastive loss Las improves accuracy, but the cost of drastically decreasing the semantic consistency.
17

---

Published as a conference paper at ICLR 2025
Table 6: Ablation of individual losses, Done on Bird dataset.
Model
Part purity
Part purity with mask applied
% masked
% Accuracy
HComP-Net
0.72 0.19
0.77 0.16
16.53%
70.01
JCCom--Net w/o Loesp
0.68 + 0.22
0.75 + 0.17
21 11.42%
58.32
HComP-Net w/o Laise
0.69 0.19
0.72 0.17
10.95%
65.99
HComP-Net w/o Lss
0.53 + 0.18
0.57 + 0.15
8.36%
81.62
E CONSISTENCY OF CLASSIFICATION PERFORMANCE OVER MULTIPLE RUNS
We trained the model using five distinct random weight initializations. The results showed that the model's fine-grained accuracy averaged 70.63% with standard deviation of 0. 0.18% the Bird dataset.
F IMPLEMENTATION DETAILS
We have made the code and dataset processing steps for reproducing our results publicly accessible at: https://gitthi. com/ Imageomics/HHommieeee
Model hyperparameters: We build HComP-Net on top of a ConvNeXt-tiny (Liu et al. 2022 ar- chitecture as the backbone feature extractor. We have modified the stride of the max pooling layers of later stages of the backbone from 2 to 1, similar to PIP-Net, such that the backbone produces feature maps of increased height and width, in order to get more fine-grained prototype score maps. We implement and experiment with our method on ConvNeXt-tiny backbones with 26 26 feature maps. The length of prototype vectors C is 768. The weights á at every node of HComP-Net constrained to be non-negative by the of the ReLU activation function (Agarap) 2018 Fur- ther, the prototype activation nodes are connected with non-negative weights only to their respective child classes in W while their weights other classes made and non-trainable. For hy- perparameter tuning HPneet for Spider and Turtle datasets we tried various prototype vector lengths such 128, 512, 1024. We also increased the learning rate by a factor of 10. We also reduced push operation from once every 5 epochs to only done at the end of training. We found prototype length of 128 with original learning and push operation done at the end of training perform the best.
Training details: All models were trained with images resized and appropriately padded to 224 x 224 pixel resolution and augmented using TrivialAugment (Müller & Hutter 2021) I) for contrastive learning. The prototypes are pretrained with self-supervised learning similar to PIP-Net for 10 epochs, following which the model trained with the entire set of loss functions for 60 epochs. We use batch size of 256 for the Bird dataset and 64 for the Butterfly and Fish dataset. The masking module is trained in parallel, and its training continued for 15 additional epochs after the training of the rest of the model is completed. The trade-off hyperparameters for the loss functions are set to
be AGE = 2: AA = 5; AT = 2: Xovep = 0.05; Adiec = 0.1; Aorth = 0.1; Amack = 2.0; AL1 = 0.5. ACE. ÄT and A were borrowed from PIP-Net (Nauta al.) 2023). Ablations to arrive at suitable Aousp are provided in Table 4 Adise and Aorth were chosen empirically and found to work well on all three datasets. Experiment unseen species done by leaving out certain classes from the datasets, so that they are not considered during training.
Dataset and Phylogeny Details: Dataset statistics and phylogeny statistics provided in Table 8 and Table respectively. Bird dataset created by choosing 190 species from CUB-200-201 !!] Wah al., 2011) dataset, which were part of the phylogeny. Background from all images was filtered using the associated segmentation metadata (Farrell 2024). For Fish dataset, we followed the exact same preprocessing steps as outlined in PhyloNN (Elhamod al. (023). We obtain the Spider and Turtle dataset from iNaturalis by filtering for the Araneae order and the Testudines
Licenss:
CC BY
Licenns::
CC BY-NC
I License:
CC BY-NCC
18

---

Published as a conference paper at ICLR 2025
order, respectively. We manually observe that images from iNaturalist noisy and contain unde- sired background artifacts, which may deter our models from learning attributes on the organism GroundingDINO Liu et al. (2023) to detect and crop into 'com-
body. To account for this, plete turtle' and 'complete spider' with a threshold confidence of 50%. We filter images in which GroundingDINO cannot detect the organisms with sufficient confidence. For Butterfly dataset considered each subspecies as an individual class and considered only the subspecies of genus He- liconius from the Heliconius Collection (Cambridge Butterfly (Lawrence Campolongo 2024). There is substantial variation among subspecies of Heliconius species, Furthermore, we balanced the dataset by filtering out the subspecies that did not have 20 or more images. We also sampled subset of 100 images from each subspecies that had more than 100 images.
The models for the Bird dataset were trained on two NVIDIA A100 GPUs with 80GB of RAM each. Butterfly and Fish models were trained a single A100 GPU. As rough estimate, the execution time for the training model on the Bird dataset around 2.5 hours. For Butterfly and Fish datasets, the training is completed in under hour. We used a single A100 GPU during the inference stage for all other analyses.
Compute Resources:
Table High-level statistics of the phylogenies used for different datasets.
Phylogeny
Internal nodes
Max-depth
Min-depth
Bird
184
25
Butterfly
13
5
2
Fish
20
11
Spider
53
17
2
Turtle
38
12
Table Dataset statistics train and validation images).
Dataset
# Classes
Train Validation
Bird
190
5695
5512
Butterfly
30
1418
358
Fish
38
4140
1294
Spider
113
26784
991
Turtle
9951
345
Spider
Turtle
ResNet-50
INTR
#ComP-Net
Figure 7: Sample of images misclassified by ResNet-50, INTR and HCCoP--Nee models on Spider and Turtle dataset. We observe that the dataset is quite noisy with diverse backgrounds along with obscured and blurred images, making challenging
identify and interpret commonalities.
*Note that this dataset is compilation of images from 25 Zenodo records by the Butterfly Genetics Group
Cambridge University, licensed Creativee Commons Attribution 4.0 International (Montejo-Kovee Kovacevich
al.
2020b Salazar al. 2020 Montejo- Kovacevich et al. (2119]]]
Montejo-Kovacevice
et al. 2019 2019000Mooi Montejo-Kovve Kovacevich
et al.
2019c g Warren & Jiggins| 2019b|c Montejo- Kovacevich et al. [2019h] Jiggins & Warren 2019a b.
Meier et al. 2020 Monteeoo-ovaceeeeeee Kovacevich et al. 2019a d /Salazar et al. 20188] | Kovacevich et al. 2019e
Salazar et al. 2019b] P de Castro et al. 2022]
et al. 2019 2020a 2021
Warren
Jiggins 2019a Salazar et al. 2019a| Mattila
2019).
19

---

Published as a conference paper at ICLR 2025
Table 9: Part purity with post-hoc thresholding approach. Done on Bird dataset.
Threshold
Part purity with mask applied
% masked
0.2
0.74 0.28
12.28%
0.3
0.75 + 0.27
13.47%
0.4
0.76 0.26
14.97%
0.5
0.77 = 0.15
16.66%
0.6
0.77 0.26
17.43%
G POST-HOC THRESHOLDING TO IDENTIFY OVER-SPECIFIC PROTOTYPES
alternative approach to learning the masking module is to calculate the over-specificity score for each prototype on the test set after training the model. We calculate the over-specificity scores for the prototypes of a trained model as follows,
D
1op
O
gi)
tOD
For a given prototype, we choose the top, images with the highest prototype scores from each leaf descendant. After taking the mean of the top; prototype score, we multiply the values from each descendant to arrive at the over-specificity score for the particular prototype. Subsequently. choose threshold to determine which prototypes post-hoc thresholding approach that can also be used to identify over-specific prototypes in Table While we can note that this approach can also be effective, validating the threshold, particularly in scenarios where there are no part annotations available (such as part location annotation of CUB- 200-2011) can be an arduous task. In such cases, directly identifying over-specific prototypes as part of the training through the masking module can be the more feasible option.
over-specific. We provide the results of
H VISUAL COMPARISON OF AN OVER-SPECIFIC AND A PROTOTYPE
We do visual comparison of prototype that has been identified over-specific by the masking module and prototype that is not identified over-specific in Figure g As it can be observed in Figure a), the Red-Legged Kittiwake has legs that are shorter in comparison clade Heerman Gull and Western Gull. Therefore, the prototype is identified as over-specific, as long legs are not common all three species. On the other hand, Figure TigureB((b,, the prototype has been identified non-over-specific because all three species share white-colored crowns. Prototype from Figure B(a) has very low activation for Red Legged Kittiwake and also has poor part purity since it does not highlight the same part of the bird in the images of Red-legged Kittiwake.
other species of its
Hoomann Gull
Heeraan Gull
Westem
Western Guil
Gull
legged Kitenkk
legged
Kit
(a) Masked prototype (Over-specific)
(b) Unmasked prototype (Non-over-spcciffe)
Figure 8: Comparison of over- ever-specific (a) vs non-over-peccific (b) prototype identified by masking module at the same internal node. Each corresponds the top-3 closest image the prototype from each species.
20

---

Published as a conference paper at ICLR 2025
1 PERFORMANCE OF HPNET WITH HIGH RESOLUTION FEATURE MAPS
We analyze the performance HPnet with high-resolution feature maps Table TO We modified the backbone by removing the max pooling layers the final stages of the model produce a 28 28 feature map instead of the original X 7 feature map. It can be observed that the accuracy and part purity do improve with high-resolution feature maps. We also make a qualitative comparison between an HPnet and HComP-No Net prototype with higher resolution feature map in Figure 9 showing that part purity does not improve with high-resolution feature maps for HPnet.
Table 10: Performance of HPnet with higher resolution feature map (Feature map dimensions in parentheses)
Model
% Accuracy
Part purity
HComP-Nel (26 x 26)
70.01
0.77 + 0.16
HPnet (7 7)
36.18
0.14 0.09
HPnet (28 x 28)
20.68
0.14 + 0.11
Groove
bilid
biled Ani
Black billed
Black Cuckoc
Cuckao
Mengrove
Cuzkno
(a) HPnet prototype with 28 28 feature map
(b) HComP-Nor -Net prototype with 26 x 26 feature map
Figure 9: Comparison of HPnet (28 28 feature map) (a) and HComP-Net (26 feature map) (b) prototype score maps. Although HPnet with a 28 x 28 feature map highlights a localized region in the image, the prototype highlights varying regions each image. HComP-Net prototype visualization is more localized and is also consistent in the part it highlights.
ADDITIONAL VISUALIZATIONS OF THE HIERARCHICAL PROTOTYPES DISCOVERED BY HCOMP-NET
We provide more visualizations of the hierarchical prototypes discovered by HComP-Net for Butter- fly (Figures 10 and 11), Fish (Figure 12) and Spider and Turtle (Figure 13) datasets in this section. For case of visualization, in each figure we visualize the prototypes learned over a small subtree from the phylogeny. The prototypes the lowest level capture traits that are species-specific, whereas the prototypes at internal nodes capture the commonality between its descendant species. For Fish dataset, we have provided textual descriptions purely based on human interpretation for the traits that are captured by prototypes at different levels. For Butterfly, Spider and Turtle datasets, since the prototypes are capturing different patterns, assigning textual descriptions for them not straightfor- ward. Therefore, refrain from providing any text description for the highlighted regions of the learned prototypes and leave it to the reader's s interpretation.
K ADDITIONAL TOP-K VISUALIZATIONS OF HCOMP-NET PROTOTYPES
We provide additional top-K visualizations the prototypes from Butterfly (Figures 14 to 17 and Fish (Figures 18 to 20 datasets, where every row corresponds to descendant species and the columns corresponds to the top-K images from the species with the largest prototype activation scores. A requirement of semantically meaningful prototype is that it should consistently highlight the same part of the organisms in various images, provided that the part is visible. We can see in the figures that the prototypes learned by HComP-Net consistently highlight the same part across all
21

---

Published as a conference paper at ICLR 2025
Traits common species
Traits common species
Traits specific species
Heliconius telesiphe sotericus
Heliconius erato amalfreda
Heliconius erato lativitta
Heliconlus notabllis
Heliconius sara sara
Heliconius eleuchia primularis
Heliconius eleuchia eleuchia
Figure 10: Visualizing the hierarchy of prototypes discovered by HComP-Net over three levels in the phylogeny of seven species from Butterfly dataset. For each prototype, we visualize one image from each of its leaf descendants. Therefore, for prototypes at the species level (rightmost column), we show only one image, whereas for prototypes at internal nodes, we show multiple images (equal to the number of leaf descendants). For each image, we show the zoomed-in view of the original image as well as the heatmap overlaid image in the region the learned prototype. The prototypes appear to capture different butterfly wing patterns.
top-K images of species, and across all descendant species. We additionally show that HComP- Net can find common traits at internal nodes with a varying number descendant species, including
4 species (Figure 14h, species (Figures 15 and 16), and 10 species (Figure 17) of butterflies, and
5 species (Figure I8). 8 species (Figure 19) and 18 species (Figure 20) for fish. We also provide
several top-k visualizations of prototypes learned for bird species in Figures 21 to 29 This shows
22

---

Published as a conference paper at ICLR 2025
Traits common to species
Traits common to species
Traits specific to species
Hellconlus timareta linaresi
Heliconius cydno alithea
Heliconlus cydno chioneus
Heliconlus cydno cydnides
Heliconius melpomene plesseni
Heliconius melpomene melpomene
Heliconius melpomene rosina
Figure 11: Visualizing the hierarchy of prototypes discovered by HComP-Net over three levels in the phylogeny of seven species from Butterfly dataset.
the ability HComP-Ner to discover common prototypes at internal nodes of the phylogenetic tree that consistently highlight the same regions in the descendant species images even when the number of descendants is large.
L REPRODUCIBILITY
We have ensured the reproducibility our work by providing detailed set of resources. The source code used for training and evaluation, along with instructions for reproducing the experiments, is Com-
available via Imageomics Institute GitHub site: prehensive implementation details, including hyperparameters and compute resources, described
23

---

Published as a conference paper at ICLR 2025
Traits common species
Traits specific to species
Esox americanus
Trait: Dorsal fin morphology"
Gambusia affinis
Trait: Head morphology"
Lepomis gulosus
Trait: Snout shape
Trait: Dorsal fin upper back shape
Figure 12: Visualizing the hierarchy of prototypes discovered by HComP-Ne for subtree with three species from Fish dataset. *Note that the textual descriptions of the hypothesized traits shown for every prototype are based human interpretation.
Turtle
Spider
Gopherus agassizii
Gopherus berlandieri
Gopherus morafkai
Trichonephila clavipes
Nephila pilipes
Neriene radiata
Figure 13:
Visualizing the hierarchy of prototypes discovered by HComP-Net for subtree with three species from Turtle and Spider datasets.
in Appendix Additionally, we have cited all dataset sources and outlined the data processing steps in Appendix to facilitate accurate replication of our experiments.
M LIMITATIONS OF OUR WORK
A fundamental challenge of every prototype-bar based interpretability method (including ours) is the
difficulty in associating a semantic interpretation with the underlying visual concept of a prototype. While some prototypes be interpreted easily based visual inspection of prototype activation
24

---

Published as a conference paper at ICLR 2025
eleuchia eleuchia
Heliconius eleuchia primularis
Hellconius
sara magdalena
Heliconius sara
Figure 14: Top-K visualization of prototype finding commonality between four species of butterfly sharing common ancestor. Each row represents the top 3 images from the respective species. For each image, show the zoomed-in view of the original image well the heatmap overlaid image.
maps, other prototypes are harder to interpret and require additional domain expertise of biolo- gists. To reduce the subjectivity in the analysis of prototypes, visualize the prototype in multiple training images instead single image (c.g., the top-k image patches closest to the prototype) wherever possible, that the semantic meaning of the prototype more apparent. Visualizing pro- totypes with multiple images also helps to overcome challenges introduced due to images taken from non-typical angles. A recent approach called ProtoConcepts (Ma al. 2024) explores modifying the similarity metric to allow multiple image patches to be equally close to given prototype. Such visualizations may help with understanding the underlying semantic concept better. We leave such an extension of our work for future research to explore. Also, while we have considered large phy- logenies as that of the 190 species from the CUB dataset, it may still be representative of all bird species. This limited scope may cause our method to identify apparent homologous evolutionary traits that could differ with the inclusion of more species into the phylogeny. Therefore, our method can be seen system that generates hypotheses about potential evolutionary traits discovered in form hierarchical prototypes.
25

---

Published as a conference paper at ICLR 2025
Heliconius
melpomene amaryllis
Heliconius melpomene bellula
Heliconius melpomene cythera
Heliconius melpomene malleti
Heliconius melpomene melpomene
Heliconlus melpomene nanna
Heliconius melpomenc plesseni
Heliconius melpomene rosina
Heliconius melpomene
Figure 15: Top-K visualization of prototype finding commonality between nine species of butterfly sharing common ancestor. Each row represents the top 3 images from the respective species. For each image, we show the zoomed-in view of the original image as well as the heatmap overlaid image.
26

---

Published as a conference paper at ICLR 2025
Hellconius erato amalfreda
Helioonius erato chestertonii
Heliconius erato cyrbia
Heliconius erato demophoon
Heliconius erato dignus
Heliconius erato
Heliconius erato hydara
Heliconius erato lativitta
Heliconius erato
Heliconius eralo phyllis
Heliconius erato
venus
Heliconius telesiphe sotericus
Figure 16: Top-K visualization of prototype finding commonality between twelve species of but- terfly sharing a common ancestor. Each row represents the top images from the respective species. For each image, we show the zoomed-in view of the original image as well as the heatmap overlaid image.
27

---

Published as a conference paper at ICLR 2025
Heliconius cydno alitha
Heliconius cydno chioneus
Heliconius cydno cydnides
Heliconius timarela linaresi
Figure 17: Top-K visualization of a prototype finding commonality between four species of butterfly sharing common ancestor. Each row represents the top images from the respective species. For each image, we show the zoomed-in view of the original image as well as the heatmap overlaid image.
Notropis buccatus
Notropis dorsaliis
Notropis
stramineus
Notropis
telescopus
Notropis texanus
Figure 18: Top-K visualization of prototype finding commonality between five species of fish sharing common ancestor. Each row represents the top images from the respective species. For each image, show the zoomed-in view of the original image well the heatmap overlaid image.
28

---

Published as a conference paper at ICLR 2025
Lepomis auntus
Lepomis eyanellus
Lepomis gibbosus
Lepomis gulosus
Lepomis humilis
Lepomis
macrochirus
Lepomis
megalotis
Lepomis microlophus
Figure 19: Top-K visualization of a prototype finding commonality between eight species of fish sharing common ancestor. Each row represents the top images from the respective species. For each image, we show the zoomed-in view of the original image as well as the heatmap overlaid image.
29

---

Published as a conference paper at ICLR 2025
surahs
carpo
x otherroldes
Notop'ss &
Notrople bocps
Notrapl bussalus
bushenerr
domalls
fachwankee
Notmope
Notropie is percobronuu
Notrop strmiieeu
Intescopus
tosarig
Notrop is
Phenasabur
Figure 20: Top-K visualization of a prototype finding commonality between eighteen species of fish sharing common ancestor. Each row represents the top images from the respective species. For each image, we show the zoomed-in view of the original image well the heatmap overlaid image.
30

---

Published as a conference paper at ICLR 2025
Bewick
Wren
Wren
Carolina
Wren
House
Wren
Marsh
Wren
Rock
Wren
Winter Wien
Figure 21: Top-K visualization of a prototype finding commonality between seven species of birds sharing common ancestor. Each row represents the top 3 images from the respective species. For each image, we show the zoomed-in view of the original image as well as the heatmap overlaid image.
31

---

Published as a conference paper at ICLR 2025
Brown Creeper
Bewick
Wren
Cactus Wren
Carolina Wren
House
Wren
Marsh
Wren
Rock
Wren
Winter Wren
Figure 22: Top-K visualization of prototype finding commonality between eight species of birds sharing common ancestor. Each row represents the top 3 images from the respective species. For each image, show the zoomed-in view of the original image well the heatmap overlaid image.
32

---

Published as a conference paper at ICLR 2025
Brown Creeper
White breasted Nuthatch
Bewick Wren
Cactus
Carolina Wren
House
Wren
Marsh
Wren
Wren
Winter Wren
Figure 23: Top-K visualization of prototype finding commonality between nine species of birds sharing common ancestor. Each represents the top images from the respective species. For each image, we show the zoomed-in view of the original image as well as the heatmap overlaid image.
33

---

Published as a conference paper at ICLR 2025
Gray Catbird
Brown Creeper
Mockingbird
White breasted Nuthatch
Brown Thrasher
Sage Thrasher
Bewick
Wren
Cactus Wren
Carolina
Wren
Wren
Marsh
Wren
Rock
Wren
Winter Wren
Figure 24: Top-K visualization of a prototype finding commonality between thirteen species of birds sharing common ancestor. Each represents the top 3 images from the respective species. For each image, we show the zoomed-in view of the original image as well as the heatmap overlaid image.
34

---

Published as a conference paper at ICLR 2025
Crested Auklet
Least Auklet
Parakest
Auklet
Rhinoceroe
Auklet
Hored
Puffin
Figure 25: Top-K visualization of a prototype finding commonality between five species of birds sharing common ancestor. Each represents the top images from the respective species. For each image, we show the zoomed-in view of the original image as well as the heatmap overlaid image.
California Gull
Glaucous winged Gull
Herring
Gull
Slaty
backed
Gull
Westem Gull
Figure 26: Top-K visualization of prototype finding commonality between five species of birds sharing common ancestor. Each row represents the top images from the respective species. For each image, show the zoomed-in view of the original image well the heatmap overlaid image.
35

---

Published as a conference paper at ICLR 2025
Califorii
Gull
Glaucous winged Gull
Heermann
Gull
Herring
Gull
Ivory
Gull
Ring billd Gull
Slaty backed Gull
Western
Gull
Red legged Kitlwake
Artic Tern
Black
Tern
Caspian
Tern
Common
Tem
Elegent
Forsters Tern
Least Tern
Figure 27: Top-K visualization of a prototype finding commonality between sixteen species of birds sharing common ancestor. Each row represents the top images from the respective species. For each image, we show the zoomed-in view of the original image as well as the heatmap overlaid image.
36

---

Published as a conference paper at ICLR 2025
Artic Tem
Common Ter
Elegant Tem
Forsters Tem
Figure 28: Top-K visualization of prototype finding commonality between four species of birds sharing common ancestor. Each row represents the top images from the respective species. For each image, show the zoomed-in view of the original image well the heatmap overlaid image.
Artic Tem
Common
Forsters Tern
Figure 29: Top-K visualization of prototype finding commonality between three species of birds sharing common ancestor. Each row represents the top 3 images from the respective species. For each image, we show the zoomed-in view of the original image as well as the heatmap overlaid image.
37


Reviews:
[{"review_id": "C5rsG4RdmB", "reviewer": "Reviewer_ta8u", "summary": "This paper applies deep learning techniques to uncover evolutionary traits in biological data. It leverages contrastive and orthogonality losses to facilitate hierarchical prototype learning. Additionally, the paper introduces over-specificity and discriminative losses to guide and constrain model training. The proposed method demonstrates improved performance over baseline methods across multiple benchmark datasets.", "questions": "Please kindly refer the the weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The motivation is interesting and well-justified. The hierarchical structure of biological data presents significant challenges for distinguishing species and identifying evolutionary traits.\n2. The techniques employed in HComP-Net appear technically feasible. The use of contrastive loss for learning clustered features has proven effective in self-supervised learning, while orthogonality loss helps capture diverse features.\n3. The visualization results are clear and impressive.\n4. The paper is well-structured and easy to follow.", "weaknesses": "1. It seems that the over-specificity and discriminative losses play opposing roles in the direction of model optimization, which raises the question of whether these two losses might interact and lead to abnormal model convergence. It would be beneficial if the authors could provide some theoretical or experimental analysis on this issue.\n2. This is a minor point, but the overall method appears to be a combination of multiple techniques, making the flowchart somewhat complex and redundant.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mo5AzqI0rl", "reviewer": "Reviewer_hRXA", "summary": "The work a novel framework called Hierarchy aligned Commonality through Prototypical Networks (HComP-Net) aimed at discovering evolutionary traits among species by learning hierarchical prototypes over the tree of life. It addresses the challenges of existing prototype-based methods that often produce over-specific prototypes at internal nodes, which can hinder the identification of common traits shared by descendant species. HComP-Net employs a unique over-specificity loss, a discriminative loss to ensure prototypes are absent in contrasting species, and a masking module to maintain classification performance. Through empirical analysis on various datasets, including birds, fishes, turtles, and butterflies, the authors demonstrate that HComP-Net effectively learns accurate, semantically consistent, and generalizable prototypes.", "questions": "Please see the weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "This paper investigates a highly intriguing task: identifying visual features preserved during species evolution. I believe this task is inherently challenging due to the limited and often insufficient quality of training data, making it difficult to obtain stable, semantically interpretable visual features. In this work, the design of the loss function and the associated explanations are intuitive and easy to understand.", "weaknesses": "1. The first comment is related to the choice of using visual data to identify common features in species evolution. Given the scarcity of high-quality biological images, especially in the context of vast evolutionary networks, and the inherent issues of imbalance and interference in such images, I wonder if it might be more precise to analyze common traits directly from textual descriptions or anatomical data. Could you please elaborate on the rationale behind prioritizing visual data for this task?\n\n2. The paper introduces several loss functions aimed at ensuring the diversity and effectiveness of the learned prototypes. I would be very interested to see ablation studies on these loss functions to better understand their individual impact.\n\n3. In Figure 4, when comparing the part consistency between HComP-Net and HPNet, different bird images are used. I am curious to know if this choice is justified and, if so, what the reasoning behind it is. Would it not be more appropriate to use the same images for a clearer comparison?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "R9PTmWoQxU", "reviewer": "Reviewer_hsuk", "summary": "The paper introduces a new framework that can be used for learning biological prototypes within hierarchies while avoiding the learning\nof over-specific features at internal nodes of the genetic tree. The authors perform tests with different datasets including mostly the pictures of birds, fishes and butterflies. The authors focus on the quantitative and qulitative evaluation.", "questions": "- The paper discusses many similarities with ProtoPNet and refers to it often, e.g. “For HPnet, we used the same hyperparameter settings and training strategy as used by ProtoPNet for the CUB-200-2011 dataset”, “We follow the same training strategy as provided by ProtoPNet for the CUB-200-2011 dataset.”. Why ProtoPNet is not used in the comparative experiments (also other non-hierarchical models are used there)?\n- How can the paper help to “advance our understanding of evolution”? E.g. There is a high chance that for the common species (whose pictures are available in large-scale datasets) the same features that were already used to make such a classification will be highlighted. Could the authors present a use case, in which the solution could be successfully used in practice to contribute to the understanding of evolution?\n- How are the representative images (also could be perceived as prototypical images) chosen from the dataset to allow for proper explanations (some images can be taken e.g. from a non-typical angle and do not show the prototypical features well and make the explanations difficult)?\n- Is it possible (and if yes, how) to use this solution in other hierarchical problems (not in the area of biology)?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The idea presented in the paper is interesting and original. The idea is quite simple (which is a plus). I liked that the authors aimed to present a method that can help to facilitate the analysis of different species in biology. \n- The paper is well-written, and also presents a lot of nice and clean graphics and examples that make the content understandable.", "weaknesses": "- The motivation of the paper says that there are many image datasets in the biology, so ML can be used to provide some new visual suggestions for common traits in species belonging to a common group. Nevertheless, such suggestions seem to be more important for some newly discovered species (and not the ones that are already well-known), and for such species there is a possibility of not having so many images. This can decrease the practicality of the method. Statements such as “Furthermore, HComP-Net demonstrates a unique ability to generate novel hypotheses about evolutionary traits, showcasing its potential in advancing our understanding of evolution” are too bold in my opinion.\n- Another issue with a possible practical use of the method is that the proposed solution to provide semantically meaningful information requires human annotation, which can be very subjective. The authors mention this limitation in the appendix, however they do not give any solution for a mitigation. \n- minor: some typos/grammatical mistakes can be found in the paper, e.g. “that are not just shared across all it descendant species but are also” -> “that are not just shared across all ITS descendant species but are also”", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EgynGp289c", "reviewer": "Reviewer_PA9n", "summary": "This work presents an extended approach for hierarchical prototype learning by training a neural network to discover hierarchical prototypes using structured information from a phylogenetic tree. Building upon HPNet (Haze et al., 2019), this approach, termed HComP-Net, contrasts with traditional models that use flat structures by incorporating loss functions such as over-specificity loss and discriminative loss. These functions enable HComP-Net to learn prototypes that align with the hierarchical structure of the tree, enhancing interpretability and consistency. Empirical results highlight HComP-Net’s ability to produce accurate, semantically coherent prototypes transferable to unseen species. Tested on a dataset of 190 bird species and additional organisms, the model performs better than baseline models. Additionally, HComP-Net has been shown to generate visual hypotheses on evolutionary traits, offering insight into traits across various levels of the phylogenetic tree. This research underscores the potential of interpretable representation learning using structured hierarchical prior knowledge.", "questions": "1) The CNN backbone used was ConvNeXt-tiny architecture; why the fine-grained accuracy from this architecture is not included in Table 1? Will it be better/worse when compared to other methods on the Table?\n\n2) In terms of semantically meaningful prototypes, could you discuss the possibility of obtaining similar findings using Explainable AI techniques?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Originality: The research builds on the problem initially formulated in HPNet (Haze et al., 2019), specifically addressing the challenge of over-specific prototypes. The authors introduced over-specificity and discriminative loss functions, enabling HComP-Net to learn prototypes that adhere to the hierarchical structure of a phylogenetic tree. This approach enhances interpretability and brings a fresh perspective to prototype learning.\n\nQuality: Given the identified problem of over-specific prototypes, the authors demonstrate the efficacy of their methods across fine-grained classification tasks, also showing advancements in interpretability. The proposed model performs consistently well in these tasks, validating its quality and effectiveness.\n\nClarity: The paper is well-organized, with clear explanations of the background, literature, methodology, experimental setup, and results. This clarity enhances the readability and accessibility of the research, making its contributions understandable and well-supported.\n\nSignificance: This work highlights the potential of interpretable representation learning driven by structured hierarchical knowledge. By using a phylogenetic tree for guidance, the model provides insights into trait evolution and aligns closely with the biology-inspired data structure, marking a contribution at the intersection of AI and biology.", "weaknesses": "1) On the Need for a Separate Masking Module\nThe fact that an additional masking module is required suggests that the initial contributions, particularly the over-specificity and discriminative loss functions, may not fully address the issue of over-specific prototypes. This need points to a limitation in the proposed loss functions' effectiveness in preventing prototypes from becoming overly tailored to specific species. Ideally, a more robust solution would directly manage prototype specificity through the loss functions alone, reducing reliance on extra modules that could complicate the model and potentially impact interpretability or scalability.\n\n2) On the Role of the Phylogenetic Tree in Classification\nIn section 5.1, the authors suggest that achieving high classification accuracy is not the primary goal. However, providing the model with additional information, like a phylogenetic tree during training and inference, could indeed aid classification performance. The tree may allow the model to leverage hierarchical relationships, which could enhance classification accuracy by using shared traits among related species. Thus, there seems to be a disconnect between the claim of not prioritizing classification accuracy and the model's design, which inherently includes information that could enhance it.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0zIByR1PmP", "reviewer": "Reviewer_z89S", "summary": "This paper addresses a scientific problem: discovering evolutionary traits in biology from images. To tackle this, it proposes the Hierarchy-Aligned Commonality through Prototypical Network (HComP-Net). The primary objective is to reduce over-specific prototypes that lose common features expected to be observed in all species sharing a common ancestor. To achieve this, the approach applies contrastive, orthogonality, and discriminative losses to the prototypes, and introduces over-specificity loss and masking to mitigate over-specific prototypes.", "questions": "Why does the over-specificity loss adopt a specific log-tanh loss form? Doesn’t it simply establish an arbitrary criterion for identifying overly specific prototypes? This choice requires further explanation and discussion.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The method details are well-demonstrated, with a clear overview and examples that make the content easy to follow.\n\nThe study presents a novel approach to improving the interpretability of prototypical networks, allowing for more accurate representation of parent-child relationships.\n\nIt enhances the specialization of deep networks applied in scientific discovery, particularly in phylogenetic analysis.", "weaknesses": "Two limits of experimental demonstration. \n1. The primary contribution lies in incorporating parent-child relationships to reduce over-specific prototypes within the contrastive losses, while the architecture does not appear to include specific structures for establishing a prototype hierarchy.\nIn my understanding, proposing a hierarchical structure is a contribution of HPNet, not HCompPNet. This point should be clarified in title and method description. \n2. The exclusion of certain related works leaves me a question about novelty on practical impact. For instance, PIPNet, a recent and closely related method employing self-supervised learning (2023), is not included in the comparison. It only uses HPNet from 2019. However, semantic gaps has relevance to over-specificity, and authors also mention strong motivation from PIPNet. The reason why the network is excluded needs more detailed explanation or comparison in experiments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)