What Do You See in Common? Learning Hierarchical Prototypes over Tree-of-Life to Discover Evolutionary Traits

Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 18:09:26

Abstract: A grand challenge in biology is to discover evolutionary traits---features of organisms common to a group of species with a shared ancestor in the tree of life (also referred to as phylogenetic tree). With the growing availability of  image repositories in biology, there is a tremendous opportunity to discover evolutionary traits directly from images in the form of a hierarchy of prototypes. However, current prototype-based methods are mostly designed to operate over a flat structure of classes and face several challenges in discovering hierarchical prototypes, including the issue of learning over-specific features at internal nodes. To overcome these challenges, we introduce the framework of Hierarchy aligned Commonality through Prototypical Networks (HComP-Net). We empirically show that HComP-Net learns prototypes that are accurate, semantically consistent, and generalizable to unseen species in comparison to baselines on birds, butterflies, and fishes datasets.

Main paper: 
 introduction: 
 A central goal in biology is to discover the observable characteristics of organisms, or traits (e.g., beak color, stripe pattern, and fin curvature), that help in discriminating between species and understanding Submitted to 38th Conference on Neural Information Processing Systems (NeurIPS 2024). Do not distribute. how organisms evolve and adapt to their environment [1] . For example, discovering traits inherited by a group of species that share a common ancestor on the tree of life (also referred to as the phylogenetic tree, see Figure 1 ) is of great interest to biologists to understand how organisms diversify and evolve [2] . The measurement of such traits with evolutionary signals, termed evolutionary traits, is not straightforward and often relies on subjective and labor-intensive human expertise and definitions [3, 4] , hindering rapid scientific advancement [5] .With the growing availability of large-scale image repositories in biology containing millions of images of organisms [6, 7, 8] , there is an opportunity for machine learning (ML) methods to discover evolutionary traits automatically from images [5, 9] . This is especially true in light of recent advances in the field of explainable ML, such as the seminal work of ProtoPNet [10] and its variants [11, 12, 13] which find representative patches in training images (termed prototypes) capturing discriminatory features for every class. We can thus cast the problem of discovering evolutionary traits into asking the following question: what image features or prototypes are common across a group of species with a shared ancestor in the tree of life that are absent in species with a different shared ancestor? For example, in Figure 1 , we can see that the four species of birds on the left descending from the blue node show the common feature of having "long tails," unlike any of the descendant species of the red node. Learning such common features at every internal node as a hierarchy of prototypes can help biologists generate novel hypotheses of species diversification (e.g., the splitting of blue and red nodes) and accumulation of evolutionary trait changes.Despite the success of ProtoPNet [10] and its variants in learning prototypes over a flat structure of classes, applying them to discover a hierarchy of prototypes is challenging for three main reasons.First, existing methods that learn multiple prototypes for every class are prone to learning "overspecific" prototypes at internal nodes of a tree, which cover only one (or a few) of its descendant species. Figure 2 shows a few examples to illustrate the concept of over-specific prototypes. Consider the problem of learning prototypes common to descendant species of the Felidae family: Lion and Bobcat. If we learn one prototype focusing on the feature of the mane (specific only to Lion) and another prototype focusing on the feature of spotted back (specific only to Bobcat), then these two prototypes taken together can classify all images from the Felidae family. However, they do not represent common features shared between Lion and Bobcat and hence are not useful for discovering evolutionary traits. Such over-specific prototypes should be instead pushed down to be learned at lower levels of the tree (e.g., the species leaf nodes of Lion and Bobcat).Second, while existing methods such as ProtoPShare [11] , ProtoPool [12] , and ProtoTree [13] allow prototypes to be shared across classes for re-usability and sparsity, in the problem of discovering evolutionary traits, we want to learn prototypes at an internal node n that are not just shared across all it descendant species but are also absent in the contrasting set of species (i.e., species descending from sibling nodes of n representing alternate paths of diversification). Third, at higher levels of the tree, finding features that are common across a large number of diverse species is challenging [14, 15] .In such cases, we should be able to abstain from finding common prototypes without hampering accuracy at the leaf nodes-a feature missing in existing methods.To address these challenges, we present Hierarchy aligned Commonality through Prototypical Networks (HComP-Net), a framework to learn hierarchical prototypes over the tree of life for discovering evolutionary traits. Here are the main contributions of our work:1. HComP-Net learns common traits shared by all descendant species of an internal node and avoids the learning of over-specific prototypes in contrast to baseline methods using a novel overspecificity loss.2. HComP-Net uses a novel discriminative loss to ensure that the prototypes learned at an internal node are absent in the contrasting set of species with different ancestry.3. HComP-Net includes a novel masking module to allow for the exclusion of over-specific prototypes at higher levels of the tree without hampering classification performance.4. We empirically show that HComP-Net learns prototypes that are accurate, semantically consistent, and generalizable to unseen species compared to baselines on data from 190 species of birds (CUB-200-2011 dataset) [8] , 38 species of fishes [9] , and 30 species of butterflies [16] . We show the ability of HComP-Net to generate novel hypotheses about evolutionary traits at different levels of the phylogenetic tree of organisms.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes]Justification: We claim that HComP-Net can generate novel hypotheses for potential evolutionary traits (shared traits among species due to common ancestry in the phylogeny) from image by learning prototypes at each internal node in the phylogenetic tree. We show through various visualizations of the prototypes in Figures 5, 6 , and 7 to 16, that the learned prototypes at the internal nodes can identify possible evolutionary traits from images. We also evaluate the improved interpretability of our approach quantitatively in Table 3 by computing part purity metric on Bird dataset.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper.• The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.• The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.• It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
related works: 
 One of the seminal lines of work in the field of prototype-based interpretability methods is the framework of ProtoPNet [10] that learns a set of "prototypical patches" from training images of every class to enable case-based reasoning. Following this work, several variants have been developed such as ProtoPShare [11] , ProtoPool [12] , ProtoTree [13] , and HPnet [17] suiting to different interpretability requirements. Among all these approaches, our work is closely related to HPnet [17] , the hierarchical extension of ProtoPNet that learns a prototype layer for every parent node in the tree. Despite sharing a similar motivation as our work, HPnet is not designed to avoid the learning of over-specific prototypes or to abstain from learning common prototypes at higher levels of the tree.Another related line of work is the framework of PIPNet [18] , which uses self-supervised learning methods to reduce the "semantic gap" [19, 20] between the latent space of prototypes and the space of images, such that the prototypes in latent space correspond to the same visual concept in the image space. In HComP-Net, we build upon the idea of self-supervised learning introduced in PIPNet to learn semantically consistent hiearchy of prototypes. Our work is also related to ProtoTree [13] , which structures the prototypes as nodes in a decision tree to offer more granular interpretability.However, ProtoTree differs from our work in that it learns the tree-based structure of prototypes automatically from data and cannot handle a known hierarchy. Moreover, the prototypes learned in ProtoTree are purely discriminative and allow for negative reasoning, which is not aligned with our objective of finding common traits of descendant species.Other related works that focus on finding shared features are ProtoPShare [11] and ProtoPool [12] .Both approaches aim to find common features among classes, but their primary goal is to reduce the prototype count by exploiting similarities among classes, leading to a sparser network. This is different from our goal of finding a hiearchy of prototypes to find evolutionary traits common to a group of species (that are absent from other species).Outside the realm of prototype-based methods, the framework of Phylogeny-guided Neural Networks (PhyloNN) [9] shares a similar motivation as our work to discover evolutionary traits by representing biological images in feature spaces structured by tree-based knowledge (i.e., phylogeny). However, PhyloNN primarily focuses on the tasks of image generation and translation rather than interpretability.Additionally, PhyloNN can only work with discretized trees with fixed number of ancestor levels per leaf node, unlike our work that does not require any discretization of the tree.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: We discuss the limitations of our work in Supplementary Section I.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.• The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.• The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.• The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.• The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.• If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.• While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. Guidelines:• The answer NA means that paper does not include experiments requiring code.• Please see the NeurIPS code and data submission guidelines (https://nips.cc/ public/guides/CodeSubmissionPolicy) for more details.• While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).• The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https://nips.cc/public/guides/CodeSubmissionPolicy) for more details.• The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.• The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.• At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).• Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
hcomp-net model architecture: 
 Given a phylogenetic tree with N internal nodes, the goal of HComP-Net is to jointly learn a set of prototype vectors P n for every internal node n ∈ {1, . . . , N }. Our architecture as shown in Figure 3 begins with a CNN that acts as a common feature extractor f (x; θ) for all nodes, where θ represents the learnable parameters of f . f converts an image x into a latent representation Z ∈ R H×W ×C , where each "patch" at location (h, w) is, z h,w ∈ R C . Following the feature extractor, for every node n, we initialize a set of K n prototype vectors P n = {p i } Kn i=1 , where p i ∈ R C . Here, the number of prototypes K n learned at node n varies in proportion to the number of children of node n, with β as the proportionality constant, i.e., at each node n we assign β prototypes for every child node. To simplify notations, we drop the subscript n in P n and K n while discussing the operations occurring in node n.We consider the following sequence of operations at every node n. We first compute the similarity score between every prototype in P and every patch in Z. This results in a matrix Ẑ ∈ R H×W ×K , where every element represents a similarity score between image patches and prototype vectors. We apply a softmax operation across the K channels of Ẑ such that the vector ẑh,w ∈ R K at spatial location (h, w) in Ẑ represents the probability that the corresponding patch z h,w is similar to the K prototypes. Furthermore, the i th channel of Ẑ serves as a prototype score map for the prototype vector p i , indicating the presence of p i in the image. We perform global max-pooling across the spatial dimensions H × W of Ẑ to obtain a vector g ∈ R K , where the i th element represents the highest similarity score of the prototype vector p i across the entire image. g is then fed to a linear classification layer with weights ϕ to produce the final classification scores for every child node of node n. We restrict the connections in the classification layer so that every child node n c is connected to a distinct set of β prototypes, to ensure that every prototype uniquely maps to a child node. ϕ is restricted to be non-negative to ensure that the classification is done solely through positive reasoning, similar to the approach used in PIP-Net [18] . We borrow the regularization scheme of PIP-Net to induce sparsity in ϕ by computing the logit of child node n c as log((gϕ) 2 + 1). g and ϕ here are again unique to each node.Contrastive Losses for Learning Hierarchical Prototypes: PIP-Net [18] introduced the idea of using self-supervised contrastive learning to learn semantically meaningful prototypes. We build upon this idea in our work to learn semantically meaningful hierarchical prototypes at every node in the tree as follows. For every input image x, we pass in two augmentations of the image, x ′ and x ′′ to our framework. The prototype score maps for the two augmentations, Ẑ′ and Ẑ′′ , are then considered as positive pairs. Since ẑh,w ∈ R K represents the probabilities of patch z h,w being similar to the prototypes from P, we align the probabilities from the two augmentations ẑ′ h,w and ẑ′′ h,w to be similar using the following alignment loss:EQUATIONSince K i=1 ẑh,w,i = 1 due to softmax operation, L A is minimum (i.e., L A = 0) when both ẑ′ h,w and ẑ′′ h,w are identical one-hot encoded vectors. A trivial solution that minimizes L A is when all patches across all images are similar to the same prototype. To avoid such representation collapse, we use the following tanh-loss L T of PIP-Net [18] , which serves the same purpose as uniformity losses in [21] and [22] :EQUATIONwhere g b,i is the prototype score for prototype i with respect to image b of mini-batch. L T encourages each prototype p i to be activated at least once in a given mini-batch of B images, thereby helping to avoid the possibility of representation collapse. The use of tanh ensures that only the presence of a prototype is taken into account and not its frequency.Over-specificity Loss: To achieve the goal of learning prototypes common to all descendant species of an internal node, we introduce a novel loss, termed over-specificity loss L ovsp that avoids learning over-specific prototypes at any node n. L ovsp is formulated as a modification of the tanh-loss such that prototype p i is encouraged to be activated at least once in every one of the descendant species d ∈ {1, . . . , D i } of its corresponding child node in the mini-batch of images fed to the model, as follows:EQUATIONwhere B d is the subset of images in the mini-batch that belong to species d.Discriminative loss: In order to ensure that a learned prototype for a child node n c is not activated by any of its contrasting set of species (i.e., species that are descendants of child nodes of n other than n c ), we introduce another novel loss function, L disc , defined as follows:EQUATIONwhere D i is the contrasting set of all descendant species of child nodes of n other than n c . This is similar to the seperation loss used in other prototype-based methods such as [10] , [13] , and [23] .Orthogonality loss: We also apply kernel orthogonality as introduced in [24] to the prototype vectors at every node n, so that the learned prototypes are orthogonal and capture diverse features:EQUATION)where P is the matrix of normalized prototype vectors of size C × K, I is an identity matrix, and ∥.∥ 2 F is the Frobenius norm. Each prototype pi in P is normalized as, pi = p i ∥p i ∥ .Classification loss: Finally, we apply cross entropy loss for classification at each internal node as follows:EQUATIONwhere y is ground truth label and ŷ is the prediction at every node of the tree.We employ an additional masking module at every node n to identify over-specific prototypes without hampering their training. The learned mask for prototype p i simply serves as an indicator of whether p i is over-specific or not, enabling our approach to abstain from finding common prototypes if there are none, especially at higher levels of the tree. To obtain the mask values, we first calculate the over-specificity score for prototype p i as the product of the maximum prototype scores obtained across all images in the mini-batch belonging to every descendant species d as:EQUATIONwhere g b,i is the prototype score for prototype p i with respect to image b of mini-batch and B d is the subset of images in the mini-batch that belong to descendant species d. Since g b,i takes a value between 0 to 1 due to the softmax operation, O i ranges from -1 to 0, where -1 denotes least over-specificity and 0 denotes the most over-specificity. The multiplication of the prototype scores ensures that even when the score is less with respect to only one descendant species, the prototype will be assigned a high over-specificity score (close to 0).As shown in Figure 3 , O i is then fed into the masking module, which includes a learned mask value M i for every prototype p i . We generate M i from a Gumbel-softmax distribution [25] so that the values are skewed to be very close to either 0 or 1, i.e., M i = Gumbel-Softmax(γ i , τ ), where γ i are the learnable parameters of the distribution and τ is temperature. We then compute the masking loss, L mask , as:EQUATIONwhere λ mask and λ L1 are trade-off coefficients, ∥.∥ 1 is the L 1 norm added to induce sparsity in the masks, and stopgrad represents the stop gradient operation applied over O i to ensure that the gradient of L mask does not flow back to the learning of prototype vectors and impact their training.Note that the learned masks are not used for pruning the prototypes during training, they are only used during inference to determine which of the learned prototypes are over-specific and likely to not represent evolutionary traits. Therefore, even if all the prototypes are identified as over-specific by the masking module at an internal node, it will not affect the classification performance at that node.We first pre-train the prototypes at every internal node in a self-supervised learning manner using alignment and tanh-losses asL SS = λ A L A +λ T L T .We then fine-tune the model using the following combined loss:(λ CE L CE + L SS + λ ovsp L ovsp + λ disc L disc + λ orth L orth + L mask ), where λ's are trade-off parameters. Note that the loss is applied over every node in the tree. We show an ablation of key loss terms in our framework in Table 6 in the Supplementary Section. 
experimental setup: 
 Dataset: In our experiments, we primarily focus on the 190 species of birds (Bird) from the CUB-200-2011 [8] dataset for which the phylogenetic relationship [26] is known. The tree is quite large with a total of 184 internal nodes. We removed the background from the images to avoid the possibility of learning prototypes corresponding to background information such as the bird's habitat as we are only interested in the traits corresponding to the body of the organism. We also apply our method on a fish dataset with 38 species (Fish) [9] along with its associated phylogeny [9] and 30 subspecies of Heliconius butterflies (Butterfly) from the Jiggins Heliconius Collection dataset [16] collected from various sources1 along with its phylogeny [52, 53] . The qualitative results of Butterfly and Fish datasets are provided in the supplementary materials. The complete details of hyper-parameter settings and training strategy are also provided in the Supplementary Section E.Baselines: We compare HComP-Net to ResNet-50 [54] , INTR (Interpretable Transformer) [55] and HPnet [17] . For HPnet, we used the same hyperparameter settings and training strategy as used byProtoPNet for CUB-200-2011 dataset. For a fair comparison, we also set the number of prototypes for each child in HPnet to be equal to 10 similar to our implementation. We follow the same training strategy as provided by ProtoPNet for CUB-200-2011 dataset. 
fine-grained accuracy: 
 Similar to HPnet [17] , we calculate the fine-grained accuracy for each leaf node by calculating the path probability over every image. During inference, the final probability for leaf class Y given an image X is calculated as, P (Y |X) = P (Y (1) , Y (2) , ...,Y (L) |X) = L l=1 P (Y (l) |X), whereP (Y (l) |X)is the probability of assigning image X to a node at level l, and L is the depth of the leaf node. Every image is assigned to the leaf class with maximum path probability, which is used to compute the fine-grained accuracy. The comparison of the fine-grained accuracy calculated for For every species, we are visualizing the top-3 images with highest prototype score for both HPnet and HComP-Net, shown as the four columns with zoomed in views of their discovered prototypes. We can see that HPnet highlights varying parts of the bird across the 3 species and across multiple images of the same species, making it difficult to associate a consistent semantic meaning to its learned prototype. In contrast, HComP-Net consistently highlights the head region of the bird across all four species and their images.HComP-Net and the baselines are given in Table 1 . We can see that HComP-Net performs better than the other interpretable methods, such as INTR and HPNet, and is also able to nearly match the performance of non-interpretable models, such as ResNet-50, even outperforming it for the Fish and Butterfly dataset. This shows the ability of our proposed framework to achieve competitive classification accuracy along with serving the goal of discovering evolutionary traits. We analyze the performance of HComP-Net in generalizing to unseen species that the model hasn't seen during training. The biological motivation for this experiment is to evaluate if HComP-Net can situate newly discovered species at its appropriate position in the phylogeny by identifying its common ancestors shared with the known species. An added advantage of our work is that along with identifying the ancestor of an unseen species, we can also identify the common traits shared by the novel species with known species in the phylogeny. Since unseen species cannot be classified to the finest levels (i.e., up to the leaf node corresponding to the unseen species), we analyze the ability of HComP-Net to classify unseen species accurately up to one level above the leaf level in the hierarchy.With this consideration, the final probability of an unseen species for a given image is calculated as, P (Y |X unseen ) = P (Y (1) , Y (2) , ..., Y (L-1) |X) =L-1 l=1 P (Y (l) |X). Note that we leave out the class probability at the L th level, since we do not take into account the class probability of the leaf level. We leave four species from the Bird training set and calculate their accuracy during inference in Table 2 . We can see that HComP-Net is able to generalize better than HPnet for all four species.Following the method introduced in PIPNet [18] , we assess the semantic quality of our learned prototypes by evaluating their part purity. A prototype with high part purity (close to 1) is one that consistently highlights the same image region in the score maps (corresponding to consistent local features such as the eye or wing of a bird) across images belonging to the same class. The part For each prototype, we take the top-10 images from each leaf descendant. We consider the 32 × 32 image patch that is centered around the max activation location of the prototype from the top-10 images. With these top-10 image patches, we calculate for each part how frequently the part is present inside the image patch. For example, a part that is found inside the image patch 8 out of 10 times is given a score of 0.8. In PIP-Net, the highest value among the values calculated for each part is given as the part purity of the prototype. In our approach, since we are dealing with a hierarchy and taking the top-10 from each leaf descendant, a particular part, let's say the eye, might have a score of 0.5 for one leaf descendant and 0.7 for a different leaf descendant. Since we want the prototype to represent the same part for all the leaf descendants, we take the lowest score (the weakest link) among all the leaf descendants as the score of the part. By following this method, for a given prototype we can arrive at a value for each part and finally take the maximum among the values as the purity of the prototype. We take the mean of the part purity across all the prototypes and report the results in Table 3 for different ablations of HComP-Net and also HPnet, which is the only baseline method that can learn hierarchical prototypes.We can see that HComP-Net, even without the use of over-specificity loss performs much better than HPnet due to the contrastive learning approach we have adopted from PIPNet [18] . The addition of over-specificity loss improves the part purity because over-specific prototypes tend to have poor part purity for some of the leaf descendants which will affect their overall part purity score. Further, for both ablations with and without over-specificity loss, we apply the masking module and remove masked (over-specific) prototypes during the calculation of part purity. We see that the part purity goes higher by applying the masking module, demonstrating its effectiveness in identifying over-specific prototypes. We further compute the purity of masked-out prototypes and notice that the masked-out prototypes have drastically lower part purity (0.29 ± 0.17) compared to non-masked prototypes (0.77 ± 0. 16 ). An alternative approach to learning the masking module is to identify over-specific prototypes using a fixed global threshold over O i . We show in Table 9 of Supplementary Section F, that given the right choice of such a threshold, we can identify over-specific prototypes. However, selecting the ideal threshold can be non-trivial. On the other hand, our masking module learns the appropriate threshold dynamically as part of the training process. HPnet for the bird dataset. We can see that HComP-Net is finding a consistent region in the image (corresponding to the head region) across all three descendant species and all images of a species, in contrast to HPnet. Futhermore, thanks to the alignment loss, every patch ẑh,w is encoded as nearly a one-hot encoding with respect to the K prototypes which causes the prototype score maps to be highly localized. The concise and focused nature of the prototype score maps makes the interpretation much more effective compared to baselines.We now qualitatively analyze some of the hypothesized evolutionary traits discovered in the hierarchy of prototypes learned by HComP-Net. Figure 5 shows the hierarchy of prototypes discovered over a small subtree of the phylogeny from Bird (four species) and Fish (three species) dataset. In the visualization of bird prototypes, we can see that the two Pelican species share a consistent region in the learned Prototype labeled 2, which corresponds to the head region of the birds. We can hypothesize this prototype to be capturing the white colored crown common to the two species. On the other hand, Prototype 1 finds the shared trait of similar beak morphology (e.g., sharpness of beaks) across the two Cormorant species. We can see that HComP-Net avoids the learning of over-specific prototypes at internal nodes, which are pushed down to individual leaf nodes, as shown in visualizations of Prototype 3, 4, 5, and 6. Similarly, in the visualization of the fish prototypes, we can see that Prototype 1 is highlighting a specific fin (dorsal fin) of the Carassius auratus and Notropis hudsonius species, possibly representing their pigmentation and structure, which is noticeably different compared to the contrasting species of Alosa chrysochloris. Note that while HComP-Net identifies the common Western Grebe species at different levels of the phylogeny. We can see that at level 0, we are capturing features closer to the neck region, indicating the likely difference between the length of necks between Grebe species and other species (Cuckoo, Albatross, and Fulmar) that diversify at an earlier time in the process of evolution. At level 1, the prototype is focusing on the eye region, potentially indicating to difference in the color of red and black patterns around the eyes. At level 2, we are differentiating Western Grebe from Horned Grebe based on the feature of bills. We also validate our prototypes by comparing them with the multi-head cross-attention maps learned by INTR [55] . We can see that some of the prototypes discovered by HComP-Net can be mapped to equivalent attention heads of INTR. However, while INTR is designed to produce a flat structure of attention maps, we are able to place these maps on the tree of life. This shows the power of HComP-Net in generating novel hypotheses about how trait changes may have evolved and accumulated across different branches of the phylogeny. Additional visualizations of discovered evolutionary traits for butterfly species and fish species are provided in the supplementary section in Figures 7 to 16. 
conclusion: 
 We introduce a novel approach for learning hierarchy-aligned prototypes while avoiding the learning of over-specific features at internal nodes of the phylogenetic tree, enabling the discovery of novel evolutionary traits. Our empirical analysis on birds, fishes, and butterflies, demonstrates the efficacy of HComP-Net over baseline methods. Furthermore, HComP-Net demonstrates a unique ability to generate novel hypotheses about evolutionary traits, showcasing its potential in advancing our understanding of evolution. We discuss the limitations of our work in Supplementary Section I. While we focus on the biological problem of discovering evolutionary traits, our work can be applied in general to domains involving a hierarchy of classes, which can be explored in future research.NeurIPS Paper ChecklistQuestion: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: The details of the data splits for each dataset used are provided in Table 8 and overview of phylogeny is given in Table 7 . We also give details of key hyperparameters and the way they were chosen in Supplementary Section E. Full code also provided for reproducibility.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.• The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes] Justification: We have done multiple runs of our model on Bird dataset with different random weight initialization, and report the mean and standard deviation of accuracy in Supplementary Section D Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).• The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)• The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean.• It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.• For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).• If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes] Justification: Details of computer resources used are provided in Supplementary Section E Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.• The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.• The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: The work poses no risk of misuse.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.• Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.• We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 
licenses for existing assets: 
 Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: All datasets used have been cited along with their source wherever necessary.We also mention the license details for each dataset used in Supplementary Section E Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL.• The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.• For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.• If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [Yes]Justification: We provide the full code and the necessary documentation to reproduce the results.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.• The paper should discuss whether and how consent was obtained from people whose asset is used.• At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification: No crowdsourcing or human subject involved in this research.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.• Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. A Ablation of Over-specificity Loss Trade-off HyperparameterWe have provided an ablation for the over-specificity loss trade-off hyperparameter (λ ovsp ) in Table 4 .We can observe that increasing the weight of over-specificity loss reduces the model's classification performance, as the model struggles to find any commonality especially at internal nodes where the number of leaf descendant species are large in number and quite diverse. It is natural that species that are diverse and distantly related may share fewer characteristics with each other, in comparison to a set of species that diverged more recently from a common ancestor [14, 15] . Therefore, forcing the model to learn common traits with a strong L ovsp constraint can cause the model to perform bad in terms of accuracy.  
 
 Appendix: Justification: The work abides by NeurIPS Code of Ethics in every aspect.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.• The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).Justification: There is no negative societal impact of the work performed to the best of our knowledge.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.• The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.• If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).In Table 5 we vary the number of prototypes per child β for a node to see the impact on model's performance. We note that while the accuracy increases marginally with increasing the number of prototypes per child (β) from 10 to 15, it also considerably increases the overall number of prototypes initialized. Therefore we continue to work with β = 10 for all of our experiments. In Table 6 , we perform an ablation of the various loss terms used in our methodology. As it can be observed, removal of L ovsp and L disc degrades performance in terms of both semantic consistency (part purity) and accuracy. On the other hand, removal of self supervised contrastive loss L SS improves accuracy but at the cost of drastically decreasing the semantic consistency. We trained the model using five distinct random weight initializations. The results showed that the model's fine-grained accuracy averaged 70.63% with a standard deviation of 0.18%. EQUATIONFor a given prototype, we choose the top k images with the highest prototype scores from each leaf descendant. After taking mean of the top k prototype score, we multiply the values from each descendant to arrive at the over-specificity score for the particular prototype. Subsequently we choose a threshold to determine which prototypes are over-specific. We provide the results of post-hoc thresholding approach that can also be used to identify overspecific prototypes in Table 9 . While we can note that this approach can also be effective, validating the threshold particularly in scenarious where there is no part annotations available (such as part location annotation of CUB-200-2011) can be an ardous task. In such cases directly identifying over-specific prototypes as part of the training through masking module can be the more feasible option.We provide more visualizations of the hierarchical prototypes discovered by HComP-Net for Butterfly (Figures 7 and 8 ) and Fish (Figure 9 ) datasets in this section. For ease of visualization, in each figure we visualize the prototypes learned over a small sub-tree from the phylogeny. The prototypes at the lowest level capture traits that are species-specific whereas the prototypes at internal nodes capture the commonality between its descendant species. For Fish dataset, we have provided textual descriptions purely based on human interpretation for the traits that are captured by prototypes at different levels.For Butterfly dataset, since the prototypes are capturing different wing patterns, assigning textual description for them is not straightforward. Therefore, we refrain from providing any text description for the highlighted regions of the learned prototypes and leave it to the reader's interpretation.We provide additional top-K visualizations of the prototypes from Butterfly (Figures 10 to 13) and Fish (Figures 14 to 16 ) datasets, where every row corresponds to a descendant species and the columns corresponds to the top-K images from the species with the largest prototype activation scores.A requirement of a semantically meaningful prototype is that it should consistently highlight the same part of the organisms in various images, provided that the part is visible. We can see in the figures that the prototypes learned by HComP-Net consistently highlight the same part across all top-K images of a species, and across all descendant species. We additionally show that HComP-Net can find common traits at internal nodes with varying number of descendant species, including 4 species (Figure 10 ), 5 species (Figures 11 and 12 ), and 10 species (Figure 13 ) of butterflies, and 5 species (Figure 14 ), 8 species (Figure 15 ) and 18 species (Figure 16 ) for fish. We also provide several top-k visualizations of prototypes learned for bird species in Figures 17 to 25 . This shows the ability of HComP-Net to discover common prototypes at internal nodes of the phylogenetic tree that consistently highlight the same regions in the descendant species images even when the number of descendants is large.A fundamental challenge of every prototype-based interpretability method (including ours) is the difficulty in associating a semantic interpretation to the underlying visual concept of a prototype.While some prototypes can be interpreted easily based on visual inspection of prototype activation maps, other prototypes are harder to interpret and require additional domain expertise of biologists.Also, while we have considered large phylogenies as that of the 190 species from CUB dataset, it may still not be representative of all bird species. This limited scope may cause our method to identify apparent homologous evolutionary traits that could differ with the inclusion of more species into the phylogeny. Therefore, our method can be seen as a system that generates potential hypotheses about evolutionary traits discovered in the form of hierarchical prototypes. Sources:[27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51] 


Decision: This paper proposes a method to discover evolutionary traits based on a hierarchical prototypical network, which is able to learn prototypes that are semantically consistent and generalize to unseen species. Specifically, the framework learns a set of hierarchical prototypes that correspond to visual features that correspond to common evolutionary traits that are shared across multiple descendent species. The proposed method is validated on CUB dataset and is shown to be effective, and obtain interpretable prototypes. 

The paper received positive reviews, as the reviewers found the application as well as the proposed framework to learn discriminative yet hierarchical features as interesting, the masking method to be novel, the paper well-written, and the conducted ablation study to be informative. There were concerns on fairness of the comparison over an existing method (HPNet), lack of experimental results on  relevant benchmarks (e.g. iNaturalist).

Although the review scores are positive, in the meta-reviewer's point of view they are mostly uninformative and the paper lacks novelty in both the high-level idea and the specific design of the framework for learning hierarchical prototypes. The idea of using semantic ontologies to identify discriminative features at different semantic granularity has been revisited many times in the last decade in the computer vision community. However, the paper misses out discussions or comparative studies against many relevant works (a few of them listed below) that aim to achieve the same goal of learning hierarchical features.

[Yan et al. 15] HD-CNN: Hierarchical Deep Convolutional Neural Networks for Large Scale Visual Recognition, ICCV 2015
[Hu et al. 16] Learning Structure Inference Neural Networks, CVPR 2016
[Goo et al. 16] Taxonomy-Regularized Semantic Deep Convolutional Networks, ECCV 2016

Reviews: review: paper_summary: paper_summary: The authors investigate the use of prototype-based explainability (as in ProtoPNet) for the visual discovery of evolutionary traits in biology image repositories.
In particular, the authors aim to find traits that apply to group of species in a hierarchical fashion, according ot the tree-of-life hierarchy. 
The authors identify three challenges with state-of-the-art prototype methods such as learning over-specific prototypes that do not apply to all species in a given group, and prototypes that do not descriminate between the group and other groups of species in the hierarchy. Ther main contribution is the design of a loss and a masking mechanism to mitigate those issues.
strength_weakness: strength_weakness: - The results are limited to three relatively small datasets. Why are there no result on the iNaturalist dataset which is at least 57x larger than CUB-200- 2011?
- It was hard to judge the effectiveness of the approach from the provided figures. The images are quite small.
- It was also hard to assess the effectiveness of masking. The figures did not illustrate how it helps.

Minor: I encountered several language issues. Below are ones I noted:
- hiearchy
- seperation
- Futhermore
- scenarious => scenarios 
- overlayed => overlaid
- indicating to difference => to a difference
- hasn’t => has not [avoid abbreviations in a scientific text]
questions: questions: - Are the heatmaps simply activation maps from HComPNet? Would GradCAM-based methods be suited to generate fine-grained visualizations?
limitations: limitations: A fundamental limitation in the application domain of interpretable biological traits is discussed in section I. Beyond a few ablation studies focusing on the introduced losses, I missed a discussion on the limitations of the approach, in particular the effectiveness of masking.
paper_summary: paper_summary: The authors propose a method that automatically learns multiple orthogonal embeddings to act as prototypes. This approach helps the discovery of hierarchical similarities by representing data in a structured space.
strength_weakness: strength_weakness: In the comparison with HPNet, the authors modify HComP-Net by removing the final two max pooling layers, resulting in a more detailed 26x26 feature map. In contrast, HPNet produces only a 7x7 feature map as shown in figure 4(a). Since the architecture and effectiveness of these networks heavily depend on the resolution of feature maps, this discrepancy raises concerns about the fairness of the comparison. To ensure a fair comparison:
  * HPNet should also be adjusted to generate a larger feature map. 
  * This adjustment and its impact on performance should also be included in the ablation study section.

In the generalizing to unseen species section, the evaluation method used by the authors could be extended to include comparisons with non-hierarchical methods. This would provide a more comprehensive evaluation of the method's effectiveness across different types of classification challenges.
questions: questions: as noted in the weaknesses.
limitations: limitations: There are several limitations concerning the comparison with other methods.
paper_summary: paper_summary: The authors propose a novel deep learning based algorithm named HComP-Net that can detect evolutionary traits common to groups of species with shared ancestors. Based on earlier studies, they aim to build a model that can accurately isolate common traits of specific species and reject over-specific features.
strength_weakness: strength_weakness: HComp-Net was tested with only 3 datasets, which is understandable, as proper datasets may not be readily available. Still, a more thorough evaluation is desirable in the future.
questions: questions: I understand each child node is connected to a fixed number of prototypes ($\beta$) in the final classifier. What happens if each child node is connected to all available prototypes? Also, in table 5, can the authors explain why the accuracy is down to 67.93% when $\beta$ is increased to 20?
limitations: limitations: The authors provided the limitations in the appendix.
paper_summary: paper_summary: The paper addresses a major challenge in biology: identifying evolutionary traits, which are features common to a group of species with a shared ancestor in the phylogenetic tree. Compared to the existing works, this submission proposes new architectures and loss to avoid the over-specification problems. In the experiments, the authors demonstrate that the proposed method improves existing works and set up ablation studies to show the impact of different components of the proposed method.
strength_weakness: strength_weakness: [-] More background: For most of the machine learning conference readers, I guess the proposed problem background is required. Therefore, more related work and background sections should be useful. 
[-] I wonder whether the proposed framework can address "Convergent evolution" and other similarity cases. Since these species can have similar features but should not be very close in the evolutionary trees. I suggest the authors to include more details and discussions about the background knowledge.
[-] While the framework has shown promising results on datasets of birds and other animals, I wonder whether the method can show its scalability to larger and more diverse datasets.
questions: questions: Please refer to the weaknesses.
limitations: limitations: I do not think this work has potential negative social impact. The problems sounds very interesting.

Comments: title: Thanks for your feedback
Closing Remarks and Request for Final Reviewer Comments and Score Updates
comment: These new materials help a lot for the background and experiment setting. I will change my score to weak accept.
We sincerely appreciate the insightful comments and valuable feedback provided by our reviewers. As we approach the conclusion of the discussion period, we would like to extend an invitation for any further questions or clarifications. If our responses have adequately addressed your concerns, we kindly request that you consider revising your scores accordingly. Thank you for your time and dedication to this review process.
Thank you very much for your feedback and support.
I appreciate the authors’ clarifications and would like to keep my rating.