Deep Taxonomic Networks for Unsupervised Hierarchical Prototype Discovery
Authors: Zekun Wang
Ethan Haarer
Tianyi Zhu
Zhiyi Dai
Christopher J. MacLellan
Venue: NeurIPS 2025 poster ()
Keywords: unsupervised learning
hierarchical clustering
representation learning
latent variable models
prototype discovery
Primary area: probabilistic_methods
TL;DR: We introduce Deep Taxonomic Networks, a deep latent variable approach that uses a complete binary-tree mixture-of-Gaussians prior in a VAE framework to discover interpretable hierarchical taxonomies and prototype clusters from unlabeled data

Abstract: Inspired by the human ability to learn and organize knowledge into hierarchical taxonomies with prototypes, this paper addresses key limitations in current deep hierarchical clustering methods. 
Existing methods often tie the structure to the number of classes and underutilize the rich prototype information available at intermediate hierarchical levels. 
We introduce deep taxonomic networks, a novel deep latent variable approach designed to bridge these gaps.
Our method optimizes a large latent taxonomic hierarchy, specifically a complete binary tree structured mixture-of-Gaussian prior within a variational inference framework, to automatically discover taxonomic structures and associated prototype clusters directly from unlabeled data without assuming true label sizes.
We analytically show that optimizing the ELBO of our method encourages the discovery of hierarchical relationships among prototypes. 
Empirically, our learned models demonstrate strong hierarchical clustering performance, outperforming baselines across diverse image classification datasets using our novel evaluation mechanism that leverages prototype clusters discovered at all hierarchical levels.
Qualitative results further reveal that deep taxonomic networks discover rich and interpretable hierarchical taxonomies, capturing both coarse-grained semantic categories and fine-grained visual distinctions.

Deep Taxonomic Networks for Unsupervised Hierarchical Prototype Discovery
Zekun Wang', Ethan Haarer', Tianyi Zhu 12, Zhiyi Dai', Christopher MacLellan'
Georgia Institute of Technology UUniversity of Virginia
(zekun, chaarer3, zdai83, cmaclell)@gatech. edu crvsns@virginia.cm
Abstract
Inspired by the human ability to learn and organize knowledge into hierarchical taxonomies with prototypes, this paper addresses key limitations in current deep hierarchical clustering methods. Existing methods often tie the structure the number of classes and underutilize the rich prototype information available at intermediate hierarchical levels. We introduce deep taxonomic networks, a novel deep latent variable approach designed to bridge these gaps. Our method optimizes large latent taxonomic hierarchy, specifically complete binary tree structured mixture-of- of-Gassiin prior within a variational inference framework, to automat- ically discover taxonomic structures and associated prototype clusters directly from unlabeled data without assuming true label sizes. We analytically show that optimizing the ELBO of our method encourages the discovery of hierarchical relationships among prototypes. Empirically, our learned models demonstrate strong hierarchical clustering performance, outperforming baselines across diverse image classification datasets using our novel evaluation mechanism that leverages prototype clusters discovered at all hicrarchical levels. Qualitative results further reveal that deep taxonomic networks discover rich and interpretable hierarchical taxonomics, capturing both coarse-grained semantic categories and fine-grained visual distinctions.
Introduction
The human mind possesses an extraordinary capacity to learn, organize knowledge, and generalize from experience, often constructing rich, abstract hierarchical category structures [42]. This learn- ing journey begins early; even pre-linguistic infants demonstrate an ability to group objects into rudimentary categories based on salient perceptual features like shape and parts, forming the initial
scaffolding of hierarchical taxonomy without the need for explicit semantic symbols 35 g 2) Two
key principles appear fundamental to this organization: the formation hicrarchical taxonomics and the representation of categories via prototypes (3). We naturally structure knowledge in nested levels of abstraction (e.g., collie - dog + mammal - animal) [37]. Within these hierarchies, 'basic level' (e.g., dog, chair) emerges as psychologically privileged 44, representing
optimal trade-off between informativeness and cognitive effort. This prototype serves as a cognitive reference point, allowing for graded membership (e.g., robin is more prototypical bird than a penguin) and facilitating efficient generalization to novel instances based on similarity to the prototype 7 6
Inspired by these powerful human capabilities, early computational approaches, such as Cobweb (3). explicitly attempted to support unsupervised, incremental Icarning of hicrarchical, proba- bilistic prototypes by optimizing category utility-a measure reflecting the trade-off between feature predictability within category and distinctiveness between categories [16] 4 Modern deep learning systems have revisited these themes, developing methods for hierarchical clustering 30
48 29
often
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

integrating hierarchical structures and prototypes within neural networks to enhance performance, interpretability, and robustness. Despite this progress deep learning, two significant gaps persist. Finstly, existing deep hierarchical clustering approaches often tie leaf nodes to fixed class labels and require retraining to handle different classification granularity on the same data. Secondly, by treating leaf clusters as terminal representations, current approaches overlook intermediate prototypes and underutilize rich multi-level abstractions.
(a) Sub- lub-hierarchy of ungulate. Left branch: deer-like silhouettes (including ostriches). Right branch: grazing versus ridden horses. The parent blends both.
(b) Sub-hierarchy of cars. Parent: blends red, white, and blue cars. Left branch: emergency (red fire trucks, police cars); Right branch: red hatchbacks, blue sedans.
Figure 1: Examples of sub-hierarchies discovered by fitting deep taxonomic network to CIFAR-10 data. For each cluster, we sampled nine images from the test set based on likelihood.
To bridge this gap. we propose deep taxonomic networks, novel deep latent variable approach that leverages variational inference [6 18). Our approach optimizes a large latent taxonomic hierarchy structured complete binary-tree mixture-of- -f-asssiinnna prior. This hierarchical prior enables our method to automatically discover abstraction structures and their associated prototypes directly from unlabeled data (see Figure T). We contribute the following: (a) A deep latent variable approach that discovers fine-grained hierarchies from unlabeled data without assuming label counts; (b) A theoretical analysis showing that maximizing the ELBO in our approach results in good hierarchical prototypes that describe the data; (c) A simple training framework that does not require specialized training procedures or pre-training and fine-tuning, and can seamlessly incorporate contrastive learning (3) jointly with the variational inference objective; (d) Our models outperform related hierarchical clustering models on image classification datasets varying complexity and class labels by a large margin using novel evaluation mechanism that leverages rich prototypes discovered at all hierarchical levels; (e) Our approach discovers rich, interpretable hierarchical prototypes different granularity.
2 Related work
Hierarchical clustering Hierarchical clustering organizes data into nested structure of clusters, revealing relationships at multiple granularities [40 45). Traditional agglomerative methods, such Ward's minimum variance approach, iteratively merges the pairs of clusters by minimizing the increase in total within-cluster variance. [45]. Deep learning based approaches learn the hierarchical clusterings in an embedding space, with the advantage of integrating deep representation learning techniques such as contrastive learning 3 in the clustering framework [30 29 48) for more robust performance on high-dimensional data. For example, DeepECT 30 couples autoencoder with projection-based divisive clustering layer to recursively split data into a binary tree in the learned embedding space. Yet, many existing methods are limited by fixed class-based hierarchies and an over-reliance on leaf clusters, ignoring rich intermediate prototypes. Most related to our work is Cobweb 7 (26. M. concept formation system that builds concept hierarchies top-down based on categorical utility and identifies meaningful basic-level categories at intermediate nodes. Cobweb not constrained by label size, and it leverages its entire learned hierarchical structure inference time, including intermediate prototypes [ While Cobweb processes raw inputs and assumes conditional independence of features, deep taxonomie networks employ neural networks to optimize a potentially large, pre-defined taxonomic structure within an embedding space to jointly learn robust data representations and informative hierarchical prototypes all levels of the tree, facilitating the discovery of basic-level clusters.

---

Deep latent variable models Variational autoencoders (VAEs) 36 18 are deep latent variable approaches that use neural networks (encoder networks (o(zix) and decoder networks po(x(z)) to learn data distributions by optimizing the Evidence Lower Bound (EIBO) objective. In this framework, the prior distribution p(z) represents the underlying observation generation process. While standard VAEs use a standard Gaussian prior p(z), the prior can be adjusted to account for specific structures in the data. VaDE [1] proposes a Gaussian mixture prior to jointly learn latent representations and cluster assignments. Other work [O 38 leverages nested Chinese Restaurant Process prior [ to learn hierarchical latent representations. Alternatively, hierarchical VAEs [9 27 44 employ multiple stochastic latent layers to learn multiple approximated posteriors at varying abstraction levels [41 47 For instance, MA-VAE [5] uses VLAE [47] to learn multi- faceted clusterings of data, and Tree (29 constructs ree-like approximated posteriors using LadderVAF [41] for hierarchical clustering. However, these approaches often increase computational cost by optimizing multiple decoder networks and require specialized procedures (5) frequent fine- tunings (29). Contrary to these approaches, deep taxonomic networks utilize a complete binary tree mixture-of-Gaasse prior explicitly support taxonomy within single approximated posterior.
3 Deep Taxonomic Networks
We introduce deep taxonomic networks, a novel deep latent variable approach featuring a complete binary tree Mixture-of- Exxture-ffGGaussinns prior. This method learns hierarchical taxonomy by mapping data to the most prototypical clusters, parameterized by their Gaussian priors. These clusters are optimized for high intra-category similarity (low internal feature entropy) and high information gain about the features from cluster membership. We start by describing the generative process within the VAE framework in Section 3.T Then we describe the variational inference process in Section 3.2 and its connection to prototypicality maximization in Section 3.3 Finally, we introduce a contrastive
learning extension for real-world images in Section 3.4
3.1 Generative process with hierarchical Nexture-of-Gaussian prior
We define the conditional prior distribution over the latent variable using hierarchical struc- ture T. represented as a complete binary tree. Each node c in T, has an associated prior prob- ability p(c) and corresponds to a cluster defined by parameters (Me,a?) such that the condi- tional latent prior p(z c) = N(z le. II... We assume an isotropic Gaussian for simplic- ity, where I denotes a diagonal covariance matrix. Though our current analysis utilizes simplified covariance structure, the underly- ing method is not limited to this configuration and can be extended to other covariance struc- tures. To enforce hierarchical dependency, the parameters of a parent node Cparent are the con- vex combinations of those of its children, Cien and Gright- Specifically, we model the distribution at the parent node as Gaussian approximation to the mixture of its children's 's distributions. By matching the first two moments of the mixture (Mogan 2 Leggn I), where [0, 1] is convex weight, we obtain:
Ps.nj
T
gie [S]
(a) Generation
(b) Inference
Figure 2: networks. (a): solid arrows represent the gener- ative sampling process. The grayed cluster es is selected via the prior distribution p(c). (b): dashed arrows represent the variational inference process. Red: learnable parameters.
The graphic model for deep taxonomic
aN (N 02 1) + (a)--a)Naaaa
Mequiaa -OfMtaen + (1 -xx))eagan
1 + # ||lMass Mepaaa 112 + (1 (1-
(02 +
1 Meaas
2 Equma
where the dimension of the latent space. These constraints ensure that parent clusters represent broader distributions encompassing their children in the latent space. This approach also reduces the number of learnable parameters as the intermediate clusters are inferred from the leaf clusters and the convex weights at each branch (Figure Z). As shown in Figure 2a the overall generative process for
an observation x proceeds as follows: (T) select cluster e in T via a prior p(c); (2) sample a latent
representation z from the chosen cluster's distribution: NN I (3) generate the observation
3

---

conditioned on the latent representation via a decoder network fo: x ~ po z). The decoder defines Gaussian [(x fo(z), I) with unit variance for real-valued x or Bernoulli for binary x.
3.2 Variational inference
Deep taxonomic networks represent the data distribution p(x) using a hierarchical mixture-of- Gaussians prior over T. We achieve this by maximizing the Evidence Lower Bound (ELBO) on logp(x) using an amortized variational posterior go (z, x) = go(z x)go(c x), where we parameterize 9s (z x) = =x Na (x), of (x) by an encoder neural network Io (xx. The overall
inference process is shown in Figure 2h
Formally:
log p(x) log Σ po(x p) ()pe
NET
/ /Eo(z,e golzz (x (x
z z)po(z) c)po(c)
log
dz
( c x)
cET
Px z) z)Po(z c)pa(c)
log Po(x 2
22)oo (z c)po(c)
= -log Bgg (x.cpp)
> E (x.clx)
gz(2, x)
gz(z,
where right-hand side of Equation represents Jensen's inequality and is the evidence lower bound (ELBO) 18 36), (ELBO( f), and can be rewritten as (see full derivation in Appendix A.I)
LELBO(O, 0) = (gs(z/x) [logpo (x z)]
- Exa(c(x) DrC. (go x) po(z c))
(3)
DrI. (ga(c x) || po()))
(4)
The ELBO objective can be interpreted as follows: Equation (2) measures the reconstruction quality
between the encoder Je (x) and the decoder fz(z). Equation (3) is the Kullback-Leible (KL)
divergence between the learned latent distribution of input and the clusters in T. weighted by
(o(c x), which represents the cluster assignment probability of x. Equation (4) regularize the
cluster assignment probability to be close to the prior distribution of the clusters in T As suggested
in [5 5) we can replace {s(c) x) with (a z) = Eder Polc' (p))))))) the cluster assignment
pu(c)pu(xle)
probability of z. with one Monte Carlo sample via reparamcterization trick (18).
3.3 Reinterpretation of ELBO as prototypicality maximization
In hierarchical clustering approaches, the concept of prototypicality quantifies how well a cluster e within the hierarchy T encapsulates and informs about a given sample's latent representation z. Categorical utility (CU) [16] 4 formalizes this notion by providing principled approach from information-these theorr: let Z be the random variable over latent vectors z, CU then defined as the mutual information between 2 and T:
CU T(2:T) H(2) ( TET H(2) is the entropy of the feature distribution. (T= is the conditional entropy of the features given the cluster assignment c. CU quantifies the information gained about the features from knowing favoring clusters in the hierarchical level with low internal entropy H(2 T c) and large reduction from H HZZ), analogous to cognitive basic-level categories [6 4. The prototypical cluster c* = arg max. (2) ( T c)] optimally balances cue validity (rich, predictable internal features) with category validity (relevance as a category for z). Hence, the cluster assignment probability p(c z) now relies cluster prototypicality, and we choose uniform prior p(c) over clusters in T such that Equation 44 becomes a regularizer encouraging utilization of all clusters.
We now demonstrate that optimizing the ELBO for this hierarchical approach encourages the discov- ery of prototypical relationships between the latent representations and each cluster e. Let X' be random variables over x, /1z = (x),,, and o² = (x). we can rewrite Equation 33 as follows:
(elx)) DKL (gs(2) x) (2 c)) = (x.().)))) log (e c)] - Ey (z.c)x)
[log (sl2 2 x)]
/ N(2z Mz, 0 log (iz Me, o² ddz
R Eex(
+ H(2 x))
(5)

---

The term H X) can be approximated with Monte Carlo sampling over minibatches. See
Appendices [ and A.. for full derivations. The entropy for multivariate Gaussian with diagonal
covariance can be written as: (2 T = c) = D log(2 Equation (5) can be expanded as follows:
+ D 2 Ea log o.a As a result,
2
D 1og((2)) +
D
1
+ (Mzd Moa)2
Σ (c(()) CET
log 0
+ + X)
2
2
2
det
Σ co(c/x))
(11(2IT= =
D
D
2
2d
+ (H2d Med)2
+ 11 x))
Fed
CET
D
D
Odd + (Uzd Med)2
E (clxx) (2 T = =c]] + Bgg
+ H2 x))
2 2
pet
(c (- (H T = = + H +
po (- = =c] c]]+ + ( X) +G
(6)
HH (T) + +))
= - -H(2 17) H -)))))(22)))))))))))))) + +2
(2:
where G = Eq (elx)
D 1 Ed 22d Pezd
2 2
0
The 2o(c
x) term in Equation (6)
can be
approximated as (c) by the KL divergence term from Equation
As shown in Equation 7 maximizing ELBO maximizes CU. Additionally, the maximization of
the negative mutual information term - (2; x)) can be understood as optimizing the information
bottleneck between the latent representation and the input x 43.39)
The hierarchical dependency introduced in Section 3.J embeds abstraction and specificity directly into the prior Ps (z c) via the convex weight a. Each prototype's parameters (Ha. thus blend broad parental characteristics with fine-grained child traits. Consequently, maximizing the mutual information ((2; 7) forces z to distinguish these semantically meaningful hierarchies, rather than discovering arbitrary clusters as would possible under a flat prior.
3.4 Integration of transformatiooninvariin feature learning
Learning discriminative representations for taxonomic hierarchies from unlabeled real world images challenging due high intra- and inter-class variance. For example, ship and bird images may share low-level features (e.g., blue sky, central object) yet belong to distinct semantic categories. To address this, we extend deep taxonomic networks with contrastive learning (3). The idea is to learn an image representation that is invariant to different transformations such that only the most descriptive features are preserved. Specifically, each image x is randomly augmented twice, yielding
exp sim(s, hb, )/r)
2NV views, and we minimize the NT-Xent loss: LNT-Xent
log
Σ و expl (sim(,, he))))) where
h is the projection head output on the encoder's features and sim denotes cosine similarity. The projection head absorbs augmentation variance, letting the encoder focus on invariance. We further introduce cluster-level contrastive learning by projecting the cluster assignment distribution p(c z)
and applying the same NT-Xent loss to encourage similar assignments 25 Appendix E for the effects on the two loss terms.
48 29
We refer to
Experiments
Unsupervised clustering accuracy Since deep taxonomic networks construct a hierarchy T without prior knowledge of the true number of classes in the training data, standard unsupervised clustering evaluation methods, such as the Hungarian algorithm [31] which assume one-to-one mapping between clusters and classes, become unsuitable. Instead, we propose a post-hoe annotation strategy. Given the pre-trained taxonomy T derived from the training set Dirain: we first perform forward pass of Derain through the frozen model obtain the cluster prototypicality distributions p(c Ztrain) for each training instance Xtrain Using the ground truth labels E Y associated with
5

---

Dirain. we aggregate these distributions for all data points belonging to the same class. This process yields an annotation matrix A of dimensions [Y] [T]. After normalization, each column of A represents the empirical class distribution P(Y c) for specific cluster e E TT Conceptually, clusters higher in the taxonomy (e.g., the root) tend towards a uniform class distribution over a balanced Dirains as they represent broader collections of data. Conversely, leaf clusters typically exhibit sharper distributions, indicating a higher concentration of specific classes. To evaluate accuracy on a test dataset Drest, we similarly obtain p(c| Zreat) for each test instance Xtest- The predicted class distributions for Xtest is then computed as a weighted sum of the cluster class distributions stored in A, where the weights are given by p(c Zteat): P(y Ztest) = EeEt plc Ztest 2test) = y c). A key advantage of this evaluation approach is its flexibility. Since model parameters updated during this evaluation phase, the deep taxonomic networks can be assessed on datasets with varying sets of classification labels without requiring any retraining or fine-tuning.
Hierarchical clustering metrics In addition to accuracy (ACC) and normalized mutual information (NMI), we also evaluate our approach on hierarchical clustering metrics: leaf purity (LP) and dendrogram purity (DP) (20). However, our approach differs from other hierarchical clustering methods in that we do not assume the lcaf cluster as the final destination of a data point; instead, any cluster can serve Our probabilistic LP measures cluster homogeneity via soft assignments, and our probabilistic DP computes expected purity for same-class data pairs based on their shared likelihood across all potential clusters, resembling a probabilistic version of lowest common ancestors. Detailed formulations for both metries are provided in Appendix B
prototype. We therefore propose probabilistic extensions to both LP and DP.
Datasets and baselines We evaluate the hierarchical clustering performance of deep taxonomic networks on datasets of varying complexities and label sizes: MNIST (3). FashionMNIST (Fashion) [46). CIFAR-10, and CIFAR- 100100 [21]. For CIFAR-100, we evaluate our approach against both the 20 superclasses (CIFAR-20) 20) and the 100 fine-grained classes. To illustrate the ability to discover taxonomic hierarchies, we additionally train our models Omniglot [22]. We provide detailed description of datasets in Appendix C We compare the hierarchical clustering performance of our approach to deep hierarchical clustering methods such as Tre [29 and DeepECT [30]. We further compare our approach to Cobweb [7], which clusters raw pixels, and to Cobweb+VAE, which uses VAE model with our encoder and decoder network architectures to produce latent codes for clustering. On CIFAR variants, contrastive learning is applied to the image inputs (Section (344). whereas for Cobweb+ VAE it is applied only the latent representations. We additionally use the publicly available code to train Tree VAE on CIFAR- with 100 classes using 100 leaf clusters.
DP
LP
ACC
NMI
MNIST
Fashion
CIFAR-10
CIFAR-20
CIFAR-100
0.2
2.0
2.4
0.4.
0.5
0.6
3.2
0.3
0.3
6.3
4
8
10
12
6
8
10
12
4
6
6
10
12
6
8
100
12
4
8
10
12
Figure 3: Hierarchical clustering performance on all evaluated datasets at varying depth of T, X-axis:
depth, Y-axis: performance.
Implementation details architecture from 229 for our approach. Detailed descriptions can be found in Appendix D.I Our approach initializes the Gaussian parameters of leaf clusters in T as well as the convex weights o at each branch such that the rest of clusters in the hicrarchy can be inferred. To determine the number of clusters in T, we vary tree depth on all evaluated datasets. Figure 3 shows all four hicrarchical metrics improve with depth but plateau around depths of 10 for MNIST and Fashion, and 10 12 for CIFAR-10. However, for CIFAR-20 and 100, which feature greater class diversity, all metrics consistently rise with increased depth, indicating our approach's 's scalability with dataset complexity. For a fair comparison across all datasets, we fix a depth of 10 (oyyielding 2047 clusters for all experiments in this paper. For contrastive learning, we use a two-layer MLP (512 + 64) with ReLU as the encoder projection head and single 64-diimensional linear layer for the cluster- projection on p(e z). We set NT-Xeett temperatures to 0.5 (representation) and 0.3 (cluster), with a weighting of 100 to match CELBO following [29] To stabilize the training of large T, we introduce two
For a direct comparison to baselines, we use the same encoder-decoder

---

0.9
0.6
0.3
0.0
lesf
intermediate
intermediate
intermediate
leaf
intermediate
leaf
intermediate
(a) MNIST
(b) Fashion
(c) CIFAR-10
(d) CIFAR-20
(e) CIFAR-100
Figure 4: Prototypicality p(c z) on test data over T. X-axis: Cluster indices of flattened complete
binary tree, ordered left-to-right starting with its 2 leaf clusters. Y-axis: p(c z).
additional entropy regularization terms that penalize biased higher-level parent clusters (i.e., 1) and indistinguishable lower-level clusters where their KL divergences are close. See Appendix D.2 for additional details. We train deep taxonomic networks for 400 epochs on all datasets with Adam [17 at constant learning rate of 1 x 10 and a batch size of 256.
5
Results and discussions
5.1
Prototypicality across the learned taxonomy
Figure 4 shows the prototypicality p(c z) for unlabeled test samples across evaluated datasets. We observe that the most prototypical cluster can occur at any depth in the taxonomy, but is pre- dominantly found in lower-level clusters. This pattern indicates that finer clusters capture the most informative features of each data point 4 whereas higher-level clusters such as root, which reflect more generalized averages, rarely serve as prototypes. Notably, MNIST and Fashion exhibit more intermediate-level prototypes-likely because leaf clusters become too specific (e.g.. unique hand- writing styles) represent general prototype (Figure 5d). By contrast, the greater complexity and variance in 110AR-10, CIFAR-20, and CIFAR- necessitates deeper hierarchies to reach a similar level of feature specificity.
Table 1: Hierarchical clustering performance (%) with standard deviations on 4 evaluated datasets. 1: Results adopted from [9 Contrastive learning is applied during training. Results averaged over 10 random seeds.
Dataset
Models
DP
LP
ACC
NMI
Cohweb
77.4+11
90.7
88.2
78.1 43.4
Cohweb + VAE
72.622.1
87.8+0.9
89.3+1.4
78.642.9
MNIST
DeepECT
74.6
90.7
74.9
76. 7+4.2
Tret
87.9+44.9
96.0+1.9
90.2
90.0+4..
Deep Nex
76.622.3
96.710.2
94.8.1.2
88.110.
Cobweb
57 57.2
78.540.8
75. 75.2+1.7
66.7+2.1
Cobweb + VAE
56.620.8
72.142.0
75.1
66.542.9
Fashion
DeepECTi
44.9
67.8
51.8
57.7.3.7
Tree
54.442.4
71.42..0
63. 63.643.3
64.7+1.
DeepTaxonNer
59.8
81.640.3
81.244.2
72.2
Cohweb + VAE
10.02
18.91 20.27
16.36
2.50
DeepECT
10.01
10.30
10.30
0.1840.11
CIFAR-10
Tree
35.30+1.15
53.85-1.22 12
52.98
41.4441.18
DeepTaxonNer
42.89
54.31 20.63
67.97 20.91
51.83
Cobweb + VAE
5.01
10.91_0.00
9.39
3.30.10.3
DecpECT
5.28
6.97+0.69
6.97
1.71
CIFAR-20*
TreeVAE
10.444p.38
24.16
21.82+0.77
17.80
TeepTaxenNet
17.40
27.87 20.47
40.72.0.30
29.36 29.36.0.47

---

5.2 Hierarchical clustering performance
We evaluate deep taxonomic networks against established baselines on four datasets (Table[). Overall, deep taxonomic networks outperform all baselines in both hierarchical clustering accuracy (ACC. NMI) and hierarchical purity (DP, LP) with the exception on MNIST dataset. We attribute MNIST's lower DP and NMI to its low inter- and intra-class variance, which causes bottom-level clusters to capture handwriting idiosyncrasies rather than digit-level features (Figure[5d). By contrast, on Fashion our approach outperforms all baselines by a large margin. When jointly trained with contrastive learning on CIFAR- -IFAR-10/20, our approach outperforms baselines by learning more consistent hierarchies and achieving higher classification accuracy. Notably, Cobweb models trained on raw pixels perform competitively on MNIST and Fashion due to its feature- intiuurdedddeeeeeeeeee assumption, which can be effective for simpler images where individual pixels alone may suffice to characterize the features 32 24]. However. Cobweb gains little from VAE embeddings, where it inherits feature dependencies from the encoder networks. In contrast, our approach-despeee assuming diagonal covariance jointcy optimizes encoding and clustering end-to-end, implicitly learning feature dependencies and achieving superior performance on both simple and complex, real-world images.
We argue that our approaches benefit from the learned intermediate prototypes. Leaf purity (LP) measures the class entropy at the leaves (the nodes with the most fine-grained semantic classes). Table 1 shows that despite having substantially more leaf nodes (2 nodes) than Tree VAE and DeepECT (10 nodes, corresponding to classes), our approach's 's leaf purity is comparable to the baseline approaches (it has higher LP), suggesting robust, consistent, and well-structured fine-grained semantic classes at the leaf level. In other words, while other approaches have 10 leaves corresponding to the 10 classes, ours can identify more subclasses that are inherent in the data, but are not explicitly called out in the labeling. This is beneficial for classification because our approach enables the model to better disentangle subtle semantic differences that might found similar across labels (e.g., a digit '4' that is similar to a "9' in MNIST, or an 'automobile' that is similar to a 'truck' in CIF.
An interesting observation from the leaf-only approaches (Tre and DeepECT) is that ACC is lower than LP. This is expected as leaf-only approaches assume the same number of leaves as the classes so the classification accuracy depends on both the quality of leaf nodes representing class (L.P) and the routing quality that successfully brings data to a leaf node, and hence ACC should be upper bounded by LP. However, observe substantial increase in ACC in approach (c.g.. CIFARs) compared to VAE, despite having a similar LP. Notably, ACC is not aoper-bounded by LP, as is the case with the leaf-only approach. This result indicates that the additional ACC gain in approach is benefit of utilizing additional intermediate nodes. In other words, by utilizing all levels, our approach correctly capture in intermediate nodes what would otherwise be misclassified in the leaf nodes of leaf-only approaches, results in better classification accuracy.
Adaptation to new classification task without re-training Our approach enables flexible classifi- cation across different label granularities without the need for re-training. Specifically, we used the pre-trained, frozen hicrarchy from CIFAR-20 Table to evaluate the 100 fine- -raeeggaindd classes using the evaluation method described in Section 4 We find that deep taxonomic networks outperform Tree which is re-trained by growing up to 100 cluster nodes, on all metries, with accuracy of 26.36+0.30 compare to 11.98 1..18. This result suggests that our approach is able to adapt to different classification objectives by utilizing the rich hierarchical prototype clusters.
Table 2: CIFAR-100 hierarchical classification results (%) TreeVAE and deep taxonomic network. i The same, frozen model used in Table ] on CIFAR-20. *: Contrastive learning is applied during training VAE Results are averaged over 10 random seeds.
Dataset
Models
DP
LP
ACC
NMI
Tre
3.77
12. 1 15.68_0.39
+o.18
27.57 =0.20
CIFAR-
DeepTaxonnett
8.29
26.36
37.03
Hierarchical classification on pre-trained features While we adopt the same encoder-decoder model architecture as Tree VAE for direct comparison, argue that the performance of our
8

---

approach benefit from models that learn a stronger feature representation. Inspired by L2H [4]. we perform hierarchical clustering on top of unsupervised pre-trained image features from DINOv2 33 Specifically, replace the encoder and decoder with one linear layer each. The encoder maps the DINOv2 feature 128 dimensional hidden representation, and the decoder maps it back to the original feature dimension. We additionally disabled contrastive learning in this setting. Our goal assess whether our methods stay robust with stronger representation while remaining directly comparable to the L2H baseline, which does not use contrastive learning.
Table Hierarchical classification results (%) on CIFAR-10 and CIFAR- 100 using DINOv2 features. *: Results are adopted from 34 Underscore de- notes the second best result.
Dutaset
Models
DP
I.P
ACC
NMI
TEM-TTMII
90.2
95.8
95.6
90.1
CIFAR-10
L2H-Tarde"
98.8
99.5
99.5
98.5
DexpTnxn
88.0
99.6
99.1
97.4
Table 3 shows that when using stronger image features, our approaches match the performance from L2H in flat clustering metries (ACC, NMI) on all evaluated datasets. On hierarchical met-
TEMI
50.2
69.8
68.2
77.8
CIFAR 100
L2H-Tartle"
80.3
89.6
89.6
91.7
DeepTaxonNet
21.0
93.0
88.7
89.4
rics (DP, LP). approaches achieve higher LP despite having much deeper hierarchical clusters and more leaf nodes. While our approaches reach a lower, but comparable DP than L.2H variants, argue that it is because our approaches learn about 100x more intermediate nodes. These results suggest that the performance of our approach is not limited the current model architecture choice, and can benefit from stronger feature representation. while additionally offering much deeper hierarchical clusters, including intermediate clusters, and without requiring prior knowledge of the number of labels.
5.3 Discovery of hierarchical prototypes
(a) A sub- hicrarchy of casual footwear.
(b) A sub-hicrarchy of high-heeled.
6
9
7
9
2
X
g
8
8
8
9
9
9
(c) sub-hierarchy of digits and 9.
(d) sub- sabb-hicarccy based on handwritten fcatures.
2
T1
A
16
U
a o
R
9
T
A& 2
@
t6
P
"TT
(e) sub-hicrarchy of rounded and angular characters.
(f) A sub-hierarchy of characters with dots and lines.
Figure 5: Examples of sub-hierarchy discovered by deep taxonomic networks on MNIST 5cc 5dh
Fashion (5a 5b) and Omniglot (5e 51 Images are sampled from the test set per cluster by likelihood.
For the qualitative results, present examples of sub-hierarchier discovered by deep taxonomic network models trained on CIFAR-10 (Figures Ta and Tb). MNIST (Figures 5e and 5d Fashion
(Figures 5a and 5b). and Omniglot (Figures and 5f

---

On CIFAR- 10, our approach uncovers interpretable hierarchies of both animal and vehicle classes. Figure Ta spliiis far-away deer-like silhouettes from close-up horse shots, then refines by silhouette vs. natural-color context and grazing vs. ridden scenes. Figure Tb divides service vehicles (industrial machinery vs. emergency fleets) from passenger cars, then partitions compact models vs. red, white and blue hatchbacks by form and hue.
Our approach also constructs hierarchies reflecting established categories in Fashion. Figure 5a organizes casual footwear such as sneakers and flat shoes, while Figure 56 distinguishes styles of high-heeled shoes. On MNIST, our model similarly forms hierarchies based on visual criteria. Figure 5c partitions digits clearly by class, separating clusters of "8"s from "9"s. Figure [d captures finer similarities in handwritten styles, grouping visually similar "4"s with "9"s, and certain "3"s with "8"s, s, highlighting the model's 's ability to learn perceptually relevant features beyond labels.
Deep taxonomic networks similarly discover coherent structures from diverse handwritten characters across numerous alphabets in Omniglot. (igure[3elgroup characters by fundamental visual traits, sep- arating predominantly rounded, continuous strokes from more angular, geometric features. Figure 5i further distinguishes characters by their elemental composition and structure, grouping line-based shapes (e.g., f- [-like) apart from those with dots or fragmented strokes, and differentiating simple vertical strokes () from structures like T or II. These examples highlight our approach's 's ability to learn and organize abstract structural properties within complex visual data.
Overall, these qualitative examples show that deep taxonomic networks are able to discover rich, interpretable hierarchical prototypes, capturing both grrrie-graind semantic categories and fine- grained visual distinctions within the data.
6 Conclusion, limitations, and future work
In this paper, propose deep taxonomic networks, novel deep latent variable approach with complete binary tree Soaxuureof-Gaussiiin prior that learns a taxonomic hicrarchy over unlabeled image data by finding the most prototypical clusters. Contrary to previous hierarchical clustering methods, deep taxonomic networks do not reply on the true label size to construct the hierarchy, and treat every cluster the potential prototype of datum. We analytically show that optimizing the learning objective of deep taxonomic networks maximizes the ability to discover hierarchical prototypes of the data. Our empirical results show that our approach outperforms baseline hierarchical clustering methods on datasets of varying complexity and with varying label sizes by a large margin. This is achieved through our novel evaluation method that leverages prototype clusters discovered at all hierarchical levels, and that can use the learned hierarchy to support a new classification objectives on the fly. Finally, we present qualitative results that show examples of subsets of discovered taxonomic hierarchies learned from various datasets, where the hierarchies contain interpretable hierarchical prototypes. Our findings suggest that deep taxonomic networks are powerful new unsupervised hierarchical clustering approach, with the potential to form human concepts.
Limitations and future work While the pre-allocated (up to a compute constraint) complete binary tree prior makes the analysis straightforward, this assumption introduces an inductive bias that dataset should have balanced feature splits. However, this assumption might lead a degraded taxonomic hierarchy if the dataset is dominated by unbalanced data with low inter-class feature entropy. Future work should focus on developing dynamic Soxture-of-Gaussian prior that adapts its structure to the dataset. In addition, while the scope of this work is to study the discovery of taxonomic hierarchy within VAE-based framework, future work should explore the generative capabilities of deep taxonomic networks to produce high quality data at multiple levels of granularity.
Acknowledgments
We would like to thank Douglas Fisher, Kyle Moore, Jesse Roberts, Pat Langley, Nicki Barari, Xin Lian, Zsolt Kira, and Ziqiao Ma for discussions.
10

---

References
[1] Nicki Barari, Xin Lian, and Christopher J. Maclellan. Incremental concept formation over visual images without catastrophic forgetting. 2024.
[2] Marc H. Bornstein and Martha E. Arterberry. The development of object categorization in young children: Hierarchical inclusiveness, age, perceptual attribute, and group versus individual analyses. Developmental Psychology, 46(2).350-365, 2010.
[3] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A Simple Framework for Contrastive Learning of Visual Representations. arXiv, July 2020. arXiv:2002.05709 [es].
[4] James E. Corter and Mark A. Gluck. Explaining basic categories: Feature predictability and information. Psychological Bulletin, 111(2)291-303, March 1992.
[5] Fabian Falck, Haoting Zhang, Matthew Willetts, George Nicholson, Christopher Yau, and Chris Holmes. Multi-facet clustering variational autoencoders, 2021.
[6] DOUGLAS H. FISHER. Conceptual Clustering, Learning from Examples, and Inference, page 38-49, Elsevier, 1987.
[7] Jr. Fisher, Douglas H. Knowledge acquisition via incremental conceptual clustering. PhD thesis, 1987. Copyright Database copyright ProQuest LLC: ProQuest does not claim copyright the individual underlying works; Last updated- 2023-07-26.
[8] Nicholas Frosst and Geoffrey Hinton. Distilling neural network into a soft decision tree, 2017.
[9] Valentina Gliozzi, Julien Mayor, Jon-Fan Hu, and Kim Plunkett. Labels as features (not names) for infant categorization: neurocomputational approach. Cognitive Science, 33(4):709-738, April 2009.
[10] Prasoon Goyal, Zhiting Hu, Xiaodan Liang, Chenyu Wang, and Eric Xing. Nonparametric variational auto-encoders for hierarchical representation learning. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), Oct 2017.
[11] Thomas Griffiths, Michael Jordan, Joshua Tenenbaum, and David Blei. Hierarchical topic models and the nested chinese restaurant process. Advances in neural information processing systems, 16, 2003.
[12] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition, 2015.
[13] Wayne Iba and Pat Langley. Cobweh Models of Categorization and Probabilistic Concept Formation, page 253-273, Cambridge University Press, January 2011.
[14] Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift, 2015.
[15] Zhuxi Jiang, Yin Zheng, Huachun Tan, Bangsheng Tang, and Hanning Zhou. Variational Deep Embedding: An Unsupervised and Generative Approach to Clustering, June 2017. arXiv:1611.0514 [es].
[16] Gregory V. Jones. Identifying basic categorics. Psychological Bulletin, 94(3):423-428 Novem- ber 1983.
[17] Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2017.
[18] Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint 1111v:1312.6114, 2013.
[19] Durk P Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, llya Sutskever, and Max Welling. Improved variational inference with inverse autoregressive flow. Advances neural information processing systems, 29, 2016.

---

[20] Ari Kobren, Nicholas Monath, Akshay Krishnamurthy, and Andrew McCallum. A hierarchical algorithm for extreme clustering. In Proceedings of the 23rd ACM SIGKDD international conference on knowledge discovery and data mining, pages 255-264, 2017.
[21] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[22] Brenden M Lake, Ruslan Salakhutdinov, and Joshua B Tenenbaum. Human-level concept learning through probabilistice program induction. Science, 35066266) 2015.
[23] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, (6(11):2278-2)) 2324, 1998.
[24] Ismael Lemhadri, Feng Ruan, Louis Abraham, and Robert Tibshirani. Lassonet: A neural network with feature sparsity. Journal of Machine Learning Research, 22((17):1-29, 2021.
[25] Yunfan Li, Peng Hu, Zitao Liu, Dezhong Peng, Joey Tianyi Zhou, and Xi Peng. Contrastive clustering, 2020.
[26] Xin Lian, Sashank Varma, and Christopher J. MacL_ellan Cobweb: An incremental and hierarchical model of human-like category learning, 2024.
[27] Lars Maaløe, Marco Fraccaro, Valentin Liévin, and Ole Winther. Biva: A very deep hierarchy of latent variables for generative modeling. Advances in neural information processing systems, 32, 2019.
[28] Christopher J. MacLellan and Harshil Thakur. Convolutional cobweb: A model of incremental learning from 2d images. In Proceedings of the Ninth Annual Conference Advances in Cognitive Systems, 2021.
[29] Laura Manduchi, Moritz Vandenhirtz, Alain Ryser, and Julia Vogt. Tree variational autoencoders, 2023.
[30] Dominik Mautz, Claudia Plant, and Christian Böhm. Deepect: The deep embedded cluster tree. Data Science and Engineering, 5(4):419-432, July 2020.
[31] James Munkres. Algorithms for the assignment and transportation problems. Journal of the society for industrial and applied mathematics, 5(1):32-38, 1957.
[32] Randal S. Olson, Jason H. Moore, and Christoph Adami. Evolution of active categorical image classification via saccadic eye movement, 2016.
[33] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V. Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel HAZIZA, Francisco Massa, Alaaeldin El-Nouby, Mido Assran, Nicolas Ballas, Wojeiech Galuba, Russell Howes, Po- Yao Huang, Shang-Wen Li. Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Herve Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. DINOv2: Learning robust visual features without supervision. Transactions on Machine Learning Research, 2024. Featured Certification.
[34] Emanuele Palumbo, Moritz Vandenhirtz, Alain Ryser, Imant Daunhawer, and Julia E Vogt. From logits to hierarchies: Hierarchical clustering made simple. In Forty-second International Conference on Machine Learning, 2025.
[35] David H. Rakison and George E. Butterworth. Infants' attention to object structure in early categorization. Developmental Psychology, 34(6): 1310-1323,, 1998.
[36] Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In International conference on machine learning, pages 2778-1866 PMLR, 2014.
[37] Eleanor Rosch and Barbara B. Lloyd. Cognition and Categorization. Routledge, 1978.
12

---

[38] Su-Jin Shin, Kyungwoo Song, and II-Chul Moon. Hierarchically clustered representation learning. In Proceedings of the AAAI Conference on Artificial Intelligence. volume 34, pages 9776-5783. AAAI Press, 2020.
[39] Ravid Shwartz-Ziv and Naftali Tishby, Opening the black box of deep neural networks via information, 2017.
[40] H. A. Sneath. The application of computers to taxonomy. Microbiology, 17(1):001-226, August 1957.
[41] Casper Kaae Sønderby, Tapani Raiko, Lars Maalde, Søren Kaac Sønderby, and Ole Winther. Ladder variational autoencoders. Advances in neural information processing systems, 29, 2016.
[42] Joshua B. Tenenbaum, Charles Kemp. Thomas L. Griffiths, and Noah D. Goodman. How to grow mind: Statistics, structure, and abstraction. Science, ((76022):129912225 March 2011.
[43] Naftali Tishby, Fernando C. Pereira, and William Bialek. The information bottleneck method, 2000.
[44] Arash Vahdat and Jan Kautz. Nvae: A deep hierarchical variational autoencoder. Advances in neural information processing systems, 13:11967--19677 2020.
[45] Joe H. Ward. Hierarchical grouping to optimize an objective function. Journal of the American Statistical Association, 58(301):23-244, March 1963.
[46] Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist. mnist: novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXi:: 2017.
[47] Shengjia Zhao, Jiaming Song, and Stefano Ermon. Learning hierarchical features from deep generative models. In International Conference on Machine Learning, pages 4091-4099. PMLR, 2017.
[48] Michal Znaležniak, Przemyslaw Rola, Patryk Kaszuba, Jacek Tabor, and Marek Smieja. Con- trastive hierarchical clustering, 2023.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The paper answers five major claims within the Introduction, enumerated A E. The contributions for each claim can be found as follows: claim is addressed in Sections 3.] to 3.4 and Section 4 claim B is addressed in Sections to 3.3] claim C is
Justification:
addressed in Section 3.4 Section 4 and Section [3] claim D is addressed in Section [.22 and
tables and 2; claim E is addressed in Section 5.1 and Section 5.3 as well as Figures
and
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings. It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Limitations of presented work can be found in Section 6 Jnder the subheading "Limitations and Future Work" and covers topics including bias induced by the binary tree structure.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms
and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory assumptions and proofs
14

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: Section 3.3 provides a proof connecting the maximization of ELBO to the maximization of Categorical Utility prove how our training methods contribute seman- tically meaningful hierarchies. This model also assumes in Section [.1 an isotopic Gaussian distribution for node clusters. All math and proofs are properly numbered and referenced in Additional proofs are in Appendix A
Section 3.1 Section 3.2 and Section 3.3
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proof's in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented
by formal proofs provided in appendix or supplemental material. Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification:
The implementation details can be found in Section along with details
about the Datasets and Metries used to calculate the results. Appendix ffrrherr details the encoder-decoder architecture in Appendix [D.]] with regularizes detailed in Appendix [D.2] Finally, the full set of hyperparameters used to collect results are found in Table
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm.
(b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
15

---

(d)
reproducible, they We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
See Section 4 and appendices D.1 and D.2 and supplemental material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips cc/
cublic/guides httas/guuuddee/eeeeubbi.eeee (CodeSubmissionoool.
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ //nips.ccpputte wttpccpccc/g.uiiidcooeopooc
PonceeSubisssioPPone
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The paper clearly defines the splits and testing details to contextualize results. In Appendix C training splits are detailed for all datasets, in Appendix [.I encoder- decoder for each dataset is defined, along with all hyperparameters in Table 4 Section details other aspects of implementation, including the epochs, batch size, and the Adam Optimizer.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary to appreciate the results and make sense of them. The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: In Section Table and 2 demonstrate results on accuracy, normalized mutual information, dendrogram purity and leaf purity, reporting standard deviations of results, which averaged over 10 random seeds.
16

---

Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi-
dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula,
call to library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: See Appendix D.4
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http:///wwrrp..ccooopcc//cccccaa
Answer: [Yes]
Justification: Yes, the development and writing of this work abides by all rules and regula- tions detailed in the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: This work addresses potential positive impacts in Appendix F.I.T and potential negative impacts and risks in appendix F.1.2 and appendix F2 respectively.
17

---

Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal
impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups). privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The work proposed in this paper does not pose these specific risks that may warrant safeguards.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We provided attribution and licenses for all existing assets (datasets and code) utilized their work, table 8 in appendix Gllearly details the licenses (Creative Commons and MIT licenses) and includes direct URLs to the original sources.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URI
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
18

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, patersswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We will release our models to github upon acceptance.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This paper does involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
19

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core work of this paper does not use LLMs, nor are LLMs used as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does involve LLMs as any important, original, non non-standard components.
Please refer to our LLM policy http:////eearrsssscccoooeeeeeeee for what should or should not be described.
20

---

A Additional Derivations
A.1
Full EBLO Derivation
We begin from
pog = log
Σ px pola e) po(c) dz
2 oET
uo asl2, e x) px(x z) pa(z c) po(c)
= log
Σ cET
dz
a0(2, x)
px z) pa(z pa (c)
log po
2) p)
c) pc(C)
=log Bg (2,clxx)
2 (e x)
(0 x)
2o (2 x)
(8)
We now expand the expectation in Equation B):
LELBOLO. 0) ((,cc)) log 2i) + (x log po c)
+ E (2,c)) logpo(() (2.cl)
log log xi
(9)
Use the factorization go(2, x) (o(c) x) (0(22 x) split the last term:
(x.c))) [log (2, x)] (oo()) Eo x) log col x0) + (oo()) [log Doggo(c [x]( (10)
Plugging Equation 10 into Equation
regroup terms:
1. Reconstruction term
Egg (x/x) [log z)]
(matches Equation 2
2. Cluster-conditioned KL
Ego(clx)) (ggcelee) [log pa (z - log (z x)] -Ega(clx) [DKZ (go(z) (Dz)
x) po(z [))]
matches Equation
3. Cluster-prior KL
E (xx(elx) [ogpo(e) log log x)] DDr (gs(c) x) po(e) (matches Equation (4)
Putting it all together,
LELBO (1) (z)() [oo (o 2)]
(e DR (g (z) x) (z c))]
D (goll x)) (pp(c))
Equation
Equation 5
Equation
21

---

A.2 Derivation of the First Term in Equation 5
B (/), [logpoz>)
Σ cET
20(2, c. x) log po dz
Σ cET
ao(c ((cexx)gx x) q0(z) x) logpo(z dz
Egsle S (x((|x)/ / N(x M2, 02) log.N(z Per o.) dz
CET
Σ Eaolcl2 f N(z M2, o log Hos of) dz cET
Σ aolc Eas(c|xx) x) f N(z Me, 02) [ log(22) logdet - Mc) E- Mc) dz
1 log((T) det E. - Estelunoo) (z no) TE= (z Hol1 Σ [x) xx
m(xxx1+)+l To (m+e -
--E x)
D
log(2
+ log det det Eg +
1
1 (1(EΣ))) + (Ha- pc) T-> - pol)
2
2
For diagonal covariances = diag(o Eg diag(o one has
0 rd
(Rzd pea)
log det E. = Σ log 2
tr(EE=1) = Σ
(11z (Ma=)Me)
TE. (Nx E (Ma) Mc)
02
02
d=1
d=1
d=
Hence the final result:
= -- co x)
D
log(2T) +
Σ
D
log o2
+
1
D
D
(Uzd Med)
(11)
2
2
2
0
Fed
d=1
d
d=1
Σ (clx)
D log(2(r) +
1
Σ
D
2 log cut
1
D
02
zd
+
(Hzd Mod) )2
(12)
2
2
2
0dd
TET
A.3 Derivation of the Second Term Equation
By definition of the conditional (differential) entropy under the variational posterior,
H(2 x)) Ep(x) (astt (log gp(z) x)]]
Because the posterior factorizes
qu(z, x) = q6(c x) gz x)
and (c x) does not depend on 2, the inner expectation simplifies to
(g [log (s | x)] (x|x) [log (z x)]]
Hence,
H(2) (x) = -Ep()) Eys(axo) go xi]]
22

---

We approximate the outer expectation over p(x) by an empirical average over N training samples {x(m) J== and the inner expectation over (z(z xx()) by M Monte Carlo samples {z( L.
mm=1
Thus:
N R ((())())) [log (z )x
1
H(2
(2) ~
N
appeux. by M samplees
N
M
log go (z(mm) (x(1)).
N
M
m==
Equivalently:
N
M
H(2 (2) =
log 9o (z (m.)) |x(
NM
n=1 m==1
where (n) ~ training data,
z (n.m) ~ (0 x()))
via reparameterizatioo trick 18]
B Probabilistic Hierarchical Clustering Metrics
B.1 Probabilistic Dendrogram Purity
We propose a probabilistic extension to Dendrogram Purity (DP) to suit our model where any cluster e can serve as a prototype and data points x (with representations z) have soft assignments p(clz) to all clusters in the hierarchy. Traditional DP relies on the purity of subtrees at the Lowest Common Ancestor (LCA) for pairs of same-class data points. In our probabilistic DP (DPprob), for any two data points x, and x, belonging to the same ground -trtt class GR. we first define a shared cluster likelihood for each cluster с as Se(x,,X,) (Equation 13). The contribution of this pair to DP is then the expected purity over all clusters c, where the purity of individual cluster P(c, G₂) (as defined in Equation 14] is weighted by the normalized likelihood (x, x,) that is shared cluster for the pair (see Equation 15). The final Prrob is the average of these expected purities across all same-class data pairs (as defined in Equation 16).
The mathematical formulation is as follows: Let denote data point and its corresponding repre- sentation. Let e be an arbitrary cluster (node) within the hierarchical structure T. The probabilistic assignment of data point x cluster is given by p(c|z). Let G& denote the k-th ground ttrut class.
The shared cluster likelihood for any cluster c E T for a pair of data points (x,, is defined as:
Sx(x,, xj) - p(clz:)p(ciz;)
(13)
The probabilistic purity of an individual cluster c E T with respect to a ground-truth class Gg is: ExIeg, p(c|z))
G(c,Gr) Eall xm p(c(zm)
(14)
For pair of data points (x, that both belong to the same ground-truth class Gg, their contribution to the Dendrogram Purity, termed the expected purity Ex(x,, Xj, GR), is calculated as:
Ep(x,, x,, G+) ΣOET (S,, x Gx))
(15)
EVer xx)
The overall probabilistic Dendrogram Purity (DPprott is then the average of these expected purities over all distinct pairs of data points belonging to the same ground-truth class:
DP
Σ
Σ X.
Ep(xi, xj, Gk)
(16)
Z
&
ifj
where Z is the total number of such distinct pairs, calculated as - = Ex ((Ca)), and Ep(x, G+) is the expected purity for the pair x,) from class Gp.
23

---

B.2 Probabilistic Leaf Purity
Standard Leaf Purity (LP) evaluates the homogeneity of leaf clusters in hierarchy with respect to ground-truth classes. It typically measures the proportion of data points in leaf clusters that belong to the majority class within each respective leaf. To adapt this metric for our model, where data points x (with representations 2) have soft assignments p(c|z) to clusters in the hierarchy, we define Probabilistic Leaf Purity (Lopr))). This formulation specifically considers the leaf clusters L of the hierarchy and utilizes the probabilistic assignments p(L(z) for L L as fractional counts of data point membership.
The mathematical formulation is as follows: Let L be the set of all leaf clusters in the hierarchy. Let G2 denote the k-th ground-truth class. The probabilistic assignment of data point x (with representation 2) to a specific leaf cluster L L is given by p L[z).
For each leaf cluster L L. we first determine the total probabilistic mass contributed by each ground-trute class G₃::
M(L,Ge)= = E p(T|z.)
(17)
X.EG2
The majority ground-truth class for leaf cluster L, denoted Gj,, is the class that maximizes this probabilistic mass:
G
= arg max M(,, G+)
(18)
Gk
The probabilistic mass of correctly assigned data points within leaf cluster L is therefore ML, GT))
The overall Probabilistic Leaf Purity (LPprot) is then calculated as the ratio of the sum of these correctly assigned probabilistic masses across all leaf clusters to the sum of all probabilistic masses assigned to any leaf cluster by any data point:
LP = ELeL (L, Git Σv'LL Eall, pl p()((z))
(19)
The denominator represents the total probabilistic assignment of all data points to the set of leaf clusters. If, for every data point X,, the sum of its probabilities to leaf clusters Σ Evee p(L (z,) equals 1 (meaning each point's probability mass for Icaf assignment is fully accounted for among the leaves), the denominator simplifies N, the total number of data points.
C Datasets
For our experimental evaluation, we utilize several standard image datasets with varying characteris- ties and complexity. These datasets were chosen to evaluate model's performance across different image types, resolutions, and numbers of classes and samples.
The MNIST dataset 23 is a widely used benchmark consisting total of 70,000 grayscale images of handwritten digits (0-9). The dataset is split into training set of 60,000 images and a testing set of 10,000 images. It comprises 10 distinct classes, with each class containing approximately 7,000 images in total (6,000 for training and 1,000 for testing). Each image has resolution 28 x 28 pixels.
MNIST
Mashion-MNIST Fashion-MNIST [66] is designed as a direct replacement for the original MNIST, offering more challenging benchmark with images of clothing items. This dataset also contains 70,000 grayscale images (28 x 28 pixels), split into 60,000 for training and 10,000 for testing. features 10 distinct classes of apparel, with a similar distribution of approximately 7,000 images per class.
CIFAR-10 The CIFAR- 10 dataset 21 consists of 60,000 color images (32 x 32 pixels) categorized into 10 distinct classes representing real-world objects such as animals, vehicles, and fruits. The dataset is typically divided into 50,000 training images and 10,000 testing images, with 6,000 images per class evenly split between the training and testing sets (5,000 for training, 1,000 for testing).
24

---

CIFAR-100 CIFAR-100 21] is a finer-graine classification dataset containing 60,000 color images (32 32 pixels), typically split into 50,000 for training and 10,000 for testing. It features 100 distinct classes, with each class containing exactly 600 images (500 for training and 100 for testing). These classes can also grouped into 20 broader superclasses. The images depict a wide variety of objects, providing more challenging and granular classification task than CIFAR-10.
Omniglot For exploring the discovery of character hierarchies, we utilized the Omniglot dataset 22] Omniglot is collection of 1,623 different handwritten characters from 50 alphabets. Each character was drawn by 20 different individuals, resulting in 20 samples per class. This dataset is characterized large number of classes and a small number of samples per class, making it suitable for evaluating the model's ability to form hierarchies in a few-shot setting. The images are typically grayscale. For our experiments, we exclusively used the training set of the Omniglot dataset.
D Additional Implementation Details
D.1 Encoder-Decoder Architecture
The encoder architecture varies according to input image type and dimensions:
Grayscale Datasets (28 x 28 pixels). For datasets such as MNIST and FashionMNIST, the encoder
applies three sequential convolutional layers to shrink the spatial dimensions from 28 x 28 down to 3 x while increasing channel capacity:
Cony1: kernel 3 x 3, stride 2, padding 1, maps x 28 x 28
14 x 14, followed by 8 x
Batch Normalization [14] and ReLU.
Conv2: kernel 3 x 3, stride 2. padding 1, maps 8 x 14 X 14
16 X 7 x 7, followed by
Batch Normalization and ReLU.
Conv3: kernel 3 x 3, stride 2. padding 0, maps 16 x 7 X 7 32
3 x 3, followed by
Batch Normalization and ReLU.
The resulting 32 x 3 x 3 tensor serves as the encoded feature representation. For Omniglot, the
encoder comprises six sequential convolutional layers:
Convl: 3, stride 1, padding 1, 1 + 32, preserves 28 x 28, BatchNorm + ReLU.
Conv2: 4 4, stride 2, padding 0, 32 - 32, down to 13 x 13, BatchNorm + ReLU,
Conv3: 3, stride 1, padding I, 32 > 64, maintains 13 x 13, BatchNorm + ReLU.
Conv4: 4 4, stride 2, padding 0, 64 - 64, down to 5 x 5, + BatchNorm ReLU.
Conv5: stride I, padding 1, 64 > 128, maintains 5 x 5, + BatchNorm + ReLU,
Conv6: 4 4, stride 2, padding 0, 128 128, down to x followed by ReLU.
RGB Datasets (32 x 32 pixels). For datasets such as CIFAR-I0 and CIFAR-100, the encoder employs modified ResNet-style blocks [2] used in 229 Each residual block includes a weighted skip connection scaled by 0.1. The number of convolutional filters starts from 32 and doubles with each downsampling stage, culminating in final feature map size of 256 4 x 4.
Latent Space and Decoder. The latent representation z has a dimension of 8 for grayscale datasets and 64 for RGB datasets. The decoder mirrors the encoder architecture, utilizing transposed convolu- tional layers to reconstruct the input images from the latent vector.
D.2 Regularizer Terms
To learn a stable taxonomic hierarchy on a large amount of clusters (i.e., a large T), we propose two regularizers to help stabilize the training. The first regularizer penalizes trivial parent splits such that one of the two children has a very low convex weight a. Following [8], we impose an entropy regularizer at each branch to encourage a uniform split. We decay the entropy regularizer weight exponentially towards the leaf with Aent Formally.
Ront (T)
Σ
depth(() Aame
ar log ae (1 - ac) log(1 ao)
25

---

Additionally, in large taxonomic hierarchy, leaf-level clusters tend to be close to each other, as fewer discriminative features can be used to separate two children down the tree. To encourage discovering as much taxonomic prototypes as possible, the second regularizer penalizes any two leaf clusters from being too close together by a margin set by hyperparameter as measured by their symmetric KI. divergence. We decrease the weighting exponentially bottom-up with Aaa- Formally,
(T)
Σ max 0,
N Nedepth(e) dkI
[DKL (Cian ll Cright) DKL (Cright Clea) }
Can,
where m is the margin and N is the depth of T. In out experiment, we set m to 1.2 and both Aem and Aaa 0.01. We refer to Appendix H.2 for ablation studies on the two regularizer terms.
D.3 Full List of Hyperparameter
We provide the list of hyperparameter used in our experiment in Table 4
Hyperparameter
Value
Training
Learning rate
x 10-3
Batch size
256
Epochs
400
[T]
layers, or 2047 clusters
Adkl
0.01
Aent.
0.01
1.2
ELBO Loss Reconstruction weight
5.0
KL divergence weights
1.0
Contrastive Loss
Embedding contrastive temperature
0.5
Clustering contrastive temperature
0.3
Loss weight
100
Model dim(2 (z)
for grayscale image, 64 for RGB
Table Summary of hyperparameter settings.
D.4 Compute Resources
Experiments are conducted on a single NVIDIA A40 GPU. Estimated training times are clearly provided for reproducibility:
MNIST, FashionMNIST, and Omniglot training typically requires approximately 45 minutes
per run.
CCIARR-O and CIFAR-100 require approximately 3 hours per training run.
E
Ablation Study
E.1
Contrastive Learning
To understand the impact of the two contrastive loss terms, we conduct ablation study presented in T We evaluate four configurations: Embedding only (embedding-level contrastive loss), Clustering only (clustering-level contrastive loss), No contrastive (no contrastive learning), and the
26

---

Full model (both contrastive terms applied). The results show that any form of contrastive learning improves performance over the baseline, with the full model achieving the highest scores across all four hierarchical clustering metrics.
Ablations
DP
LP
ACC
NMI
Full 42.74
54.81
67.13
51.34
Clustering only
39.60
48.84
65,41
50.08
Embedding only
22.34
38.10
40.92
21.91
No contrastive
19.60
37.05
38.59
20.07
Table 5: Ablation study of contrastive loss on CIFAR-10.
E.2 Regularizers
To evaluate the impact of the regularizer terms, we conduct an ablation study on both Fashion and CIFAR-10 datasets, shown in Tables ( and 7 We analyze four settings: the Full model (both regularizers applied), Rent only, Raki only, and No regularizers.
The results indicate that the entropy regularizer (Rant) significantly improves the hierarchical purity metrics-De DP and LP- e-specally LP, across both datasets. Notably, on CIFAR-10, 10, the combination of both regularizers achieves the best performance across all four metrics, suggesting that the two terms are complementary in optimizing the hierarchical structure.
In contrast, on the Fashion dataset, the effect of both regularizers is less pronounced, particularly for the hierarchical clustering accuracy metrics-ACKS and NMI. This difference may stem from the higher complexity and greater intra- and inter-class variance of 10FAR-10, where the regularizers contribute more effectively to refining hierarchical boundaries.
Overall, our results suggest that Rent is crucial for enhancing hierarchical purity, while the combina- tion of Ret and RaK proves especially effective in managing more complex datasets like CIFAR- 10.
Ablations
DP
LP
ACC
NMI
Full
42.74
54.81
67.13
51.34
Rem only
42.13
52.85
66.98
50.70
Rak only
41.43
50.92
66.02
49.98
No regularizers
40.38
50.26
65.39
48.17
Table Ablation study of regularizer terms on CIFAR-10. 10.
Ablations
DP
LP
ACC
NMI
Full
59.12
81.44
81.10
72.29
Rem only
58.91
81.01
81.04
72.29
Rak only
54.69
78.71
80.11
72.12
No regularizers
52.60
78.30
80.62
72.39
Table 7: Ablation study of regularizer terms on Fashion.
Broader Impacts and Risks
F1 Broader Impacts
We introduces Deep Taxonomic Networks for unsupervised hierarchical prototype discovery, method inspired by the human cognitive capacity for learning, organizing knowledge, and forming
27

---

hierarchical conceptual structures 42 35 9 22, particularly the principles of hierarchical taxonomies
and prototype representation 35
F1111 Positive Impacts
By automatically organizing complex, unlabeled data into interpretable hierarchical taxonomies and revealing associated prototypes [3]]] our model can significantly improve data exploration and understanding. It can provide more interpretable representations compared to flat clustering methods 25 (88) This method will be valuable in biology, material science and social science, where
discovering inherent structures in large datasets leads to new insights, hypothesis generation, and innovative discovery.
Inspired by human hierarchical concept formation 42 and the psychological relevance of basic-level categories [4] [16]. this work could potentially contribute developing more intuitive and effective educational tools or interfaces that help users organize and understand complex information by visually representing hierarchical relationships, building on computational models of categorization like Cobweb [
F.1.2 Negative Impacts
As an unsupervised learning method, deep taxonomic networks are susceptible to learning and potentially amplifying biases present in the training data. If the data reflects societal biases (e.g.. representation of certain demographic groups or concepts), the learned taxonomic structure and prototypes could entrench these biases, potentially leading to unfair or discriminatory outcomes if the model used in downstream applications that affect individuals or groups
F.2 Risks
Given that the model is a deep generative latent variable model within the VAE framework (36,, it learns to model the data distribution. If applied to data that could be used to generate or organize misleading information, such grouping images or text in a biased way, the discovered hierarchies could potentially be exploited make fake content appear more structured or credible, contributing to the spread of disinformation. While the model operates on unlabeled data, discovery of fine-grained prototypes and hierarchical clusters across different levels taxonomy could potentially reveal sensitive private information, particularly if the discovered categories are highly specific or linkable individuals.
14
G Licenses
We provide details regarding the licenses of external assets used in this work, presented in Table
Asset
URL
License
Datascts
MNIST 23
Link
Creative Commons Attribation -hare Alike 3.0
FasIion-MNIST 46
Link
MIT License
CIFAR-10/100
Link
MIT License
Omniglot 22
Link
Creative Commons Shaattilk---aaaaAkkkke. 4.0 In- ternational License
Code
Tree VAE Code
Link
MIT License
Table Licenses for assets used in our experiments.
28


Reviews:
[{"review_id": "RjF4RJF7N5", "reviewer": "Reviewer_Lzud", "summary": "The paper proposes a new approach for hierarchical classification, i.e., instead of predicting \"collie\" for an image, a hierarchical classification would predict a chain animal --> mammal --> dog --> collie. The paper proposes to relax the condition of every node in the hierarchy tree to correspond to conventional semantic classes and associate them instead to clusters of samples (images). If I understood correctly, the paper claims to learn both the structure of the hierarchy as well as the classification technique. Each node in the tree is represented by a latent variable that is learned using a variational approach. The paper derives the expected lower bound to maximize for this problem and use encoder-decoder structure to learn the latents and predict classes.\n\nThe proposed method was tested on public datasets CIFAR, MNIST and Fashion datasets.", "questions": "..", "limitations": "..", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. Attempt to mathematically formulate the overall problem will be appreciated in the community. (Pls refer to other reviewers on method novelty).\n2. In several places, the writeup attempts to provide interpretation of mathematical quantities connecting them to the problem setting -- I think this is important for paper with heavy theoretic treatments. I also appreciate the effort on good amount of experimental details, results and analysis performed in the paper.\n\nWeaknesses:\n1. Goal/Contribution: I dont think I understood the merit of the proposed approach. If the tree leaves do not represent any semantic class, I dont understand their meaning, or more importantly, their benefit. Would these more freeform clusters at leaf advantageous for classification? The accuracies on Table 1 does not support that. Clarification on what is the benefit of this structure is needed for understanding the value of this contribution. \nSame concern can extended to internal nodes as well. From the writeup, if appears the clusters pertaining to internal nodes are not exclusive at the same level. What benefit does this form of internal nodes offers us?\n\n2. Method: Does the method learns the hierarchy structure or assumes it is provided? I assumed it does, but not completely sure on it. Lines 188~200 appear to suggest a hierarchy needs to be provided in the beginning but are not clear whether or not this is updated. If it does, this should be clarified in Section 3 how the structure is learned in the variational approach. \n\nThe method section also could elaborate a bit further how the $\\alpha_i$'s (introduced in Line 119) are learned by the encoder-decoder design. Line 169 briefly mentions it, could help to elaborate. In general, would help to include a paragraph how the quantities needed for the hierarchical prediction (e.g., as shown in Figure 2) are learned through encoder-decoder, i.e., connecting the mathematical formulation to implementation.\n\n3. Experimental validation:\n3a. The datasets (CIFAR, MNIST) selected for evaluation are a bit experimental and simpler datasets. In order to convince the reader the robustness of this method, results on more real life datasets, at least ImageNet or OpenImages, are necessary. \n3b. The proposed method seems to be achieving good performance on the simpler MNIST dataset. Given the DP and LP in 40s and 20s for CIFAR, and accuracy be 70%, I cannot help questioning the goal again as to how the mixed clusters at leaves and internal nodes are beneficial .", "was_revised": true, "final_justification": "See my comment below.", "rebuttal": "Thank you for your detailed review and substantive feedback. We were glad to hear that you found our mathematical formulation sound, our writing approachable, and our experiments and analyses substantial.\n\n### **Clarification of the scope:**\n\nBefore responding to your concerns, we would like to first clarify the scope of our work. We frame our work as a **proof-of-concept** experiment to show that our novel variational approach to hierarchical clustering **can discover hierarchical prototypes from data**. To this end, we first *analytically* show that optimizing the lower bound of our approach encourages the discovery of prototypical nodes across the hierarchy given input data (as supported by section 3.3). Then, we *empirically* show that the most prototypical nodes given some inputs indeed happen at *different levels* across the hierarchy, **including the leaf nodes**. \n\n### **For concerns on goal/contribution (“*...Clarification on what is the benefit of this structure is needed for understanding the value of this contribution…*”):**\n\n- **“*...I dont think I understood the merit of the proposed approach. If the tree leaves do not represent any semantic class, I dont understand their meaning, or more importantly, their benefit….*”:**\n    - Our main objective and contribution is that **we assume nodes at *any* level can represent semantic (taxonomic) classes,** including leaf nodes. While the terminology of “leaf” and “intermediate” come from the tree data structure, we do not mean to make a distinction between them with respect to semantic categories. Leaf nodes (lower-level nodes)  represent  “finer-grained” semantic classes than their parents, rather than being  “freeform” or “not represent[ing] any semantic class[es]”.\n- ***“...Would these more freeform clusters at leaf advantageous for classification? The accuracies on Table 1 does not support that…”:***\n    - Leaf Purity (LP) measures the class entropy at the leaves (the nodes with the most fine-grained semantic classes). **Table 1** shows that despite having substantially more leaf nodes ($2^{10}$ nodes) than TreeVAE and DeepECT (10 nodes, corresponding to 10 classes), our approach's leaf purity is comparable to the baseline approaches (it has higher LP), suggesting **robust**, **consistent**, and **well-structured** fine-grained semantic classes at the **leaf level**. In other words, while other approaches have 10 leaves corresponding to the 10 classes, ours can identify more subclasses that are inherent in the data, but are not explicitly called out in the labelings. This is beneficial for classification because our approach enables the model to better disentangle subtle semantic differences that might found similar across labels (e.g., a digit ‘4’ that is similar to a ‘9' in MNIST, or an ‘automobile’ that is similar to a ‘truck’ in CIFAR-10).\n- ***“...What benefit does this form of internal nodes offers us?...”:***\n    - *An interesting observation from the leaf-only approaches (TreeVAE and DeepECT) is that ACC is lower than LP.* This is expected as leaf-only approaches assume the same number of leaves as the classes so the classification accuracy depends on both the quality of leaf nodes representing a class (LP) and the routing quality that successfully brings data to a leaf node and hence *ACC should be upper bounded by LP*. **However, with our approach, we observe a substantial increase in ACC (e.g., on CIFARs) compared to TreeVAE, despite having a similar LP. Notably, ACC is not upper-bounded by LP, as is the case with the leaf-only approach.** This result indicates that the *additional* ACC gain in our approach is a benefit of utilizing additional intermediate nodes. In other words, by utilizing all levels, our approach can correctly capture in intermediate nodes what would otherwise be misclassified in the leaf nodes of leaf-only approaches, results in better classification accuracy.\n\n### **For concerns on method:**\n\n- ***“...Does the method learns the hierarchy structure or assumes it is provided? I assumed it does, but not completely sure on it….”:***\n    - **The method learns the semantics of the hierarchy from the data.**  We assume that the hierarchical structure is a binary tree, but do not provide any hierarchical semantic information. In other words, we pre-allocate a semantically empty structure prior to learning. All semantic categories are learned/discovered through unsupervised training.\n- **“*...Lines 188~200 appear to suggest a hierarchy needs to be provided in the beginning but are not clear whether or not this is updated….*”:**\n    - Lines 188-200 describes the *inference* mechanism employed **after** a hierarchy has been learned (line 191-194). We describe inference first, so the reader can understand how the hierarchy will ultimately be used before describing how it is learned. Prior to inference, we train the hierarchy from unlabeled data—we do not provide any semantic information about the hierarchical categories. During inference (lines 188-200), we perform a forward pass on the *learned* hierarchy to map discovered hierarchical semantic information to the actual class labels of interest for evaluation, as the model is trained without labels, we need to provide labels afterwards to enable classification.\n- ***(From Summary) “...I understood correctly, the paper claims to learn both the structure of the hierarchy as well as the classification technique….”:***\n    - Yes, we *learn* the hierarchical categories and after we map them to the provided  classification labels, we can use them to make predictions.\n- ***“...The method section also could elaborate a bit further how the \\alpha's (introduced in Line 119) are learned by the encoder-decoder design… connecting the mathematical formulation to implementation…”:***\n    - Our approach learns by updating parameters outlined in Figure 2. Parent nodes (as parameterized by $\\mu_{c_\\{parent}}$, and $\\sigma^2_{c_\\{parent}}$) are computed via $\\alpha_c$ using the equation from Section 3.1. During back propagation, we compute the gradient for the $\\alpha_c$ parameters and use these gradients to adjust them accordingly to minimize the overall loss.\n\n### **For concerns on experimental validation:**\n\n- ***“...selected for evaluation are a bit experimental and simpler datasets. In order to convince the reader the robustness of this method, results on more real life datasets…”:***\n    - We use the selected dataset and encoder-decoder architecture following prior literature on variational approach to hierarchical clustering (TreeVAE, NeurIPS 2023) for a direct comparison. We believe the selected datasets vary across a wide range of relevant characteristics (i.e., grayscale vs. RGB images, handwritten vs. natural images, simple vs. complex patterns, and 10 classes vs 100 classes), making them comprehensive for our **proof-of-concept** study.\n    - Nevertheless, we agree that inclusion of results for a more real life dataset could better convince readers of the robustness of our approach. **In alignment with your feedback and that of  reviewer MkCw, we will add the following results for STL-10, a popular subset of ImageNet-1k.** We were able to evaluate our approach and TreeVAE on this dataset during the short rebuttal period given our compute resources**.** Here are the results we plan to add, which highlight our approach’s robust performance on more complex datasets:\n        |  | DP | LP | ACC | NMI |\n        | --- | --- | --- | --- | --- |\n        | TreeVAE | 20.34% | 31.31% | 30.96% | 25.74% |\n        | **DeepTaxonNet** | **29.03%** | **37.93%** | **55.75%** | **40.56%** |\n    - Additionally, **reviewer MkCw** called for a comparison to SCAN which uses a ResNet-18 architecture. And to make our approach comparable (i.e., controlling for the encoder-decoder), we did an evaluation using this other architecture. Interestingly, we found that the new encoder-decoder improved our system in the **CIFAR-10 cases (~77\\% vs. ~68\\% currently).** While we did not evaluate the other datasets using this architecture because we did not have sufficient time during the rebuttal period, **we plan to include an additional ablation study on the effect of stronger encoder-decoder architectures**. These collective results suggest that our theoretical framework indeed benefits from a stronger encoder-decoder and that it is has robust performance, even with more complex patterns (STL-10).\n- **“*...Given the DP and LP in 40s and 20s for CIFAR, and accuracy be 70%, I cannot help questioning the goal again as to how the mixed clusters at leaves and internal nodes are beneficial…*”:**\n    - We want to clarify that our approach's DP is ~43\\%, LP ~54\\%, ACC ~68\\% on CIFAR-10 and DP ~17\\%, LP ~28\\%, ACC ~41\\% on CIFAR-20, because CIFAR-20 contains less samples per class and hence is more challenging.\n    - **Our goal is to learn taxonomies from data, such that a model can discover finer and finer semantic clusterings down the hierarchy, without requiring prior information about a dataset’s semantic labelings.** To achieve this, we proposed a novel variational approach to hierarchical clustering that encodes data at multiple levels of granularity. And given our approach’s ability to learn hierarchical semantic clusterings, our approach can:\n        1. Capture in intermediate nodes what would otherwise be misclassified in the leaf nodes of leaf-only approaches. This lets our approach achieve better classification accuracy.\n        2. Identify subclasses that are inherent in the data (as will be reflected in the lower level (leaf level) of the hierarchy), but are not explicitly called out in the data labelings.\n\nWe thank you again for your thoughtful review and the concerns you raised. We believe the changes based on your feedback substantially strengthen our paper and increase the  contribution and credibility of our work."}, {"review_id": "HWNYTH2fcs", "reviewer": "Reviewer_NMPg", "summary": "This paper presents *Deep Taxonomic Networks* (DTN), a novel deep latent variable framework for unsupervised hierarchical prototype discovery. The method introduces a mixture-of-Gaussians prior over a complete binary tree to model hierarchical abstractions and leverages variational inference jointly with contrastive learning to learn taxonomic structures from unlabeled data.", "questions": "Please refer to Weakness.", "limitations": "Please refer to Weakness.", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths:**\n\n1. **Strong theoretical foundation**: The method section is rigorous, and the derivation of the ELBO objective is comprehensive. In particular, the reinterpretation of ELBO as a prototypicality maximization objective (Section 3.3) provides an insightful information-theoretic justification.\n    \n2. **Well-structured model design**: The use of a binary-tree-structured prior enables the discovery of hierarchical abstractions without assuming the number of ground-truth classes, which is a desirable property for unsupervised clustering.\n    \n3. **Interpretability**: The model discovers visually coherent and semantically meaningful sub-hierarchies, which enhances interpretability beyond flat clustering methods.\n    \n\n**Weaknesses and Suggestions:**\n\n1. **Outdated baselines**: The experimental comparison is mainly limited to older methods such as Cobweb, TreeVAE, and DeepECT. These baselines, while relevant historically, do not fully reflect the current state of unsupervised and self-supervised clustering, especially in the context of vision transformers or modern contrastive learning models. Recent approaches should be included for a more compelling comparison.\n    \n2. **Limited dataset diversity**: The evaluation is conducted on standard benchmarks like MNIST, Fashion-MNIST, CIFAR-10/20. These datasets, while widely used, are relatively outdated and limited in semantic complexity. They may not fully demonstrate the benefits of discovering deep taxonomic structures. I recommend evaluating on more recent and semantically structured datasets.\n    \n3. **Missing ablation analysis**: While the paper introduces several important components, I could not find any dedicated ablation study isolating the contributions of these individual components. As a result, it is difficult to assess which aspects of the method are most responsible for the observed performance gains. I strongly recommend including ablation experiments to disentangle the effects of the hierarchical prior, contrastive learning, and other design choices. Additionally, although the supplementary materials were available for download, I was unable to open them successfully; it is possible that relevant analysis was included there, but I could not verify it.", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "OgrNUjrw5F", "reviewer": "Reviewer_b5Xg", "summary": "This paper proposes a deep hierarchical clustering method - deep taxonomic networks (DeepTaxonNet).  In contrast to existing methods such as DeepECT and TreeVAE, DeepTaxonNet puts more emphasis on the intermediate nodes, allowing clusters associated with every node in the tree to be a potential source for generating an observation.  The tree structure is defined as a complete binary tree with an associated hierarchical mixture of Gaussians prior.  Distributions at parent nodes are constrained to be a convex combination of its children's distributions.  Variational inference is performed by maximizing the Evidence Lower Bound, and a derivation is given that shows that maximizing this bound encourages maximization of Categorical Utility.  Comparison is given on several data sets with the existing methods of Cobweb, DeepECT, and TreeVAE, where the proposed method gives better performance using the proposed metrics in the paper that account for the explicit use of intermediate nodes.  Additionally, DeepTaxonNet allows for adaptation to a new data set without re-training, and is shown to give better results on this task compared with retraining TreeVAE.", "questions": "Please see the weaknesses section above.  With regard to that section, would it be fair to say the main advantage of the proposed approach and leveraging the intermediate nodes is that the level of granularity does not need to be pre-defined, and so the same learned taxonomy can be used for varying tasks without re-training?  Are there additional benefits that you see beyond this?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "strengths:\n\n- evaluation with existing methods on multiple data sets, ablation study given in appendix, demonstration of adaptation to new task without retraining\n\nweaknesses:\n\n- it seems the main contribution/novelty of the proposed method is the more explicit use of non-leaf nodes as having their own generating distributions and serving as potential prototypes.  and this comes at the cost of having a rigid pre-defined binary tree structure.  \n- in that regard then, I would have liked to see more analysis of the benefits of this approach.  in part because the evaluation metrics have been adapted to reflect the use of intermediate nodes, so from a comparison standpoint are not completely fair to the existing methods that are only designed to use the leaf nodes.\n- for instance, new task classification without re-training seems like an advantage, but it would be nice to see more experiments along those lines\n- the qualitative analysis is 5.3 is interesting, but there is not an explicit comparison with existing methods, and in any case more quantitative analysis of the benefits of using the intermediate nodes would be more convincing.", "was_revised": true, "final_justification": "Taking into account the other reviews and the author feedback and updated results, I will maintain my original rating.", "rebuttal": "Thank you for your detailed review. We are glad you found our experiment and ablation study comprehensive and that you are interested in our method’s ability to adapt to new tasks without requiring retraining. Below are point-by-point responses and clarifications to your specific concerns.\n\n- **“*...I would have liked to see more analysis of the benefits of this approach. in part because the evaluation metrics have been adapted to reflect the use of intermediate nodes, so from a comparison standpoint are not completely fair to the existing methods that are only designed to use the leaf nodes…*”:**\n    - Thank you for raising this concern. As **reviewer Lzud** pointed out, we believe more clarification/analysis of our approach’s benefits will strengthen our argument. To the best of our knowledge, **we are the first to use a variational approach to learn hierarchical generative distributions at all levels of the tree**. Given our use of a tree structure and a variational approach, we thought TreeVAE (NeurIPS 2023) was the most recent and relevant baseline for our approach. While it might seem “unfair” to compare to methods that only use the leaf node, **we argue that the use of these intermediate nodes is a key contribution of our approach over prior approaches**. The 4 hierarchical clustering metrics we chose collectively provide a complete analysis at both leaf- and whole tree-level:\n        - Leaf Purity (LP) measures the class entropy at the leaves. In our approach, these nodes represent the most fine-grain semantic classes. Table 1 shows that despite having many more leaf nodes ($2^{10}$ leaves) than TreeVAE and DeepECT (10 leaves, corresponding to 10 classes), our approach's leaf purity is comparable to the others (in a sense that we have even higher LP). This suggests our approach forms **robust**, **consistent**, and **well-structured** fine-grained semantic classes at the **leaf level**.\n        - *An interesting observation from the leaf-only approaches is that ACC is lower than LP*. This is expected as leaf-only approaches assume the same number of leaves as the classes so the classification accuracy depends on both the quality of leaf nodes representing a class (LP) and the routing quality that successfully brings data to a leaf node and hence *ACC should be upper bounded by LP*. **However, with our approach, we observe a substantial increase in ACC (e.g., on CIFAR datasets) compared to TreeVAE, despite having a similar LP. Notably, ACC is not upper-bounded by LP, as is the case with the leaf-only approach.** This result indicates that the *additional* ACC gain in our approach is the benefit of utilizing intermediate nodes. As we mentioned in our response to **Reviewer Ludz, by utilizing all levels, our approach can capture in intermediate nodes what would otherwise be misclassified in the leaf nodes of leaf-only approaches, producing a better classification accuracy**.\n    - Again, we view our approach's ability to utilize all levels of the hierarchy as an **advantage and a novel research contribution** instead of an “unfair comparison to baselines”.\n- ***“...new task classification without re-training seems like an advantage, but it would be nice to see more experiments along those lines…”:***\n    - We appreciate that you found our approach's ability to perform new task classification without re-training advantageous. We are particularly excited about the benefits of this capability in cases where there are large training costs that might prohibit re-training.\n    - One possible direction for future work is to test our framework via large scale training on a real-world dataset, such as applying it to ImageNet-21k data to learn a large hierarchy. We hope to compare the classification performance of our approach on new datasets without re-training against more traditional approaches that use distinct pre-training and fine-tuning stages. Our current work, however, aims to first provide a **proof-of-concept** that our novel variational approach **can successfully learn hierarchical prototypes from data**. Given the theoretical and empirical results presented in the current work, we hypothesize that **one of the practical benefits of our approach will lie in compute- and data-efficient unsupervised and zero-shot classification**.\n    - Because our approach learns generative clusters at the intermediate nodes and subsequently supports classification on a new task without re-training, there is no directly comparable baselines to perform new task classification without re-training in the deep hierarchical clustering literature (that use intermediate nodes) . Most existing approaches involve pre-training and fine-tuning (i.e., SimCLR, MAE) that are first pre-trained on large scale dataset (ImageNet) and then fine-tuning on the new tasks. We believe these other approaches are qualitatively different than the approach we propose as well as the line of prior approaches we use as baselines (i.e., TreeVAE, DeepECT). Our work centered around theoretical inquiries and empirical validation of proposed framework, and we worry that including fine-tuning approaches in our paper will make the proposed framework less clear.\n- ***“...the qualitative analysis is 5.3 is interesting, but there is not an explicit comparison with existing methods…”:***\n    - We also appreciate that you find our qualitative analysis interesting. We want to point out that the purpose of Section 5.3 is to showcase that our approach's learned hierarchy, including intermediate nodes, is both consistent and human interpretable, while the hierarchies learned with previous approaches are worse on both accounts.\n- **“*...and in any case more quantitative analysis of the benefits of using the intermediate nodes would be more convincing…*”:**\n    - Given your suggestion (**and those of the other reviewers**), we plan to add additional results to our paper. Specifically, we plan to further validate our approach’s effectiveness in utilizing intermediate nodes for more complex images. We provide a comparison of our approach and TreeVAE on the STL-10 dataset, a popular subset of the ImageNet:\n        \n        \n        | **STL-10 Results** |  |  |  |  |\n        | --- | --- | --- | --- | --- |\n        |  | DP | LP | ACC | NMI |\n        | TreeVAE | 20.34% | 31.31% | 30.96% | 25.74% |\n        | **DeepTaxonNet** | **29.03%** | **37.93%** | **55.75%** | **40.56%** |\n    - Additionally, **reviewer MkCw** called for a comparison to SCAN which uses a ResNet-18 architecture and to make our approach comparable (i.e., to control for the encoder-decoder), we did an evaluation using this same encoder-decoder architecture within our approach. Interestingly, we found that the new encoder-decoder improved our system in the **CIFAR-10 case (~77\\% vs. ~68\\% currently).** While we did not evaluate the other datasets using this architecture because we did not have sufficient time during the rebuttal period, **we plan to include an additional ablation study on the effect of stronger encoder-decoder architectures**.\n    - These results collectively suggest that our theoretical framework indeed benefits from a stronger encoder-decoder and that even more complex patterns (STL-10) it can achieve robust performance. This provides more evidence to the idea that utilization of intermediate nodes can translate into better predictions, as suggested above.\n\n- ***“…would it be fair to say the main advantage of the proposed approach and leveraging the intermediate nodes is that the level of granularity does not need to be pre-defined, and so the same learned taxonomy can be used for varying tasks without re-training? Are there additional benefits that you see beyond this?…”:***\n    - We agree that using the taxonomy learned from one task, without the level of granularity being pre-defined, to support classification on another without re-training is a key benefit of using intermediate nodes. We also see several additional benefits, such as:\n        - *Utilizing intermediate nodes benefits the classification performance* as our approach can capture in intermediate nodes what would otherwise be misclassified in the leaf nodes of leaf-only approaches (Table 1, section 5.2).\n        - *Utilizing intermediate nodes facilitates the discovery of taxonomic structures within unlabeled datasets (section 5.3).* Our approach can structure, organize, and partition data in-the-wild (in a sense we have no prior knowledge about its semantic categories) for us to analyze and interpret.\n\nWe would like to thank you again for your constructive reviews. We will do our best to incorporate your concerns and suggestions into the final version. We believe the  final paper—updated based on your feedback—will offer a more insightful analysis of the benefits of our approach and strengthen the credibility of our quantitative results."}, {"review_id": "g8D1j17fuf", "reviewer": "Reviewer_MkCw", "summary": "The authors introduce a novel deep hierarchical clustering algorithm, where cluster hierarchies are defined through learned prototypes. This approach is conceptually interesting and demonstrates a degree of novelty. However, the empirical validation is currently limited. In particular, the paper lacks strong baselines and evaluations on complex image datasets, which hinders a thorough assessment of the method's effectiveness [1]. Incorporating additional deep flat clustering methods and testing on more challenging datasets would significantly strengthen the experimental section and better highlight the practical significance and robustness of the proposed approach.\n\n[1] Lu, Yiding, et al. \"A survey on deep clustering: from the prior perspective.\" Vicinagearth 1.1 (2024): 4.", "questions": "Your experimental section includes comparisons with TreeVAE and DeepECT. However, it omits more recent and advanced deep clustering methods, such as SCAN. Could you clarify  the rationale behind excluding these methods from your comparison, and  how your approach performs when applied to more complex patterns, such as those found in  STIL-10 or Image-net?", "limitations": "The central promise and selling (\"deep taxonomic networks discover rich and interpretable hierarchical taxonomies, capturing both coarse-grained semantic categories and fine-grained visual distinctions\") would need more substantiation.\nFurthermore, the evaluation lacks recent and stronger baselines such as SCAN, and does not address performance on complex datasets like  STIL-10 or Image-net, leaving the method’s  competitiveness unclear.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "The technique is somewhat new in the context of deep clustering. It is theoretically/statistically well founded. I believe that the central promise and selling (\"deep taxonomic networks discover rich and interpretable hierarchical taxonomies, capturing both coarse-grained semantic categories and fine-grained visual distinctions\") would need more substantiation.\n\nAlthough the proposed prototype-based hierarchical clustering framework is conceptually novel, but its empirical validation is limited. The absence of strong baselines and the use of only simple datasets hinder a clear assessment of the method's effectiveness. Moreover, the lack of comparison to related approaches such as deep flat clustering further weakens the evaluation. Including more competitive methods and complex datasets would significantly improve the work’s credibility.", "was_revised": true, "final_justification": "The paper is technically solid, and my major concerns regarding the experimental section have been largely resolved.", "rebuttal": "Thank you for your review. We appreciate that you found our hierarchical clustering approach novel and interesting and that our theoretical treatments seem well founded.\n\nWe would like to address your main concerns:\n\n### **Concerns on the lack of evaluation on more complex patterns such as STL-10:**\n\n- ***“...the use of only simple datasets hinder a clear assessment of the method's effectiveness…Including more … complex datasets would significantly improve the work’s credibility…how your approach performs when applied to more complex patterns, such as those found in STIL-10…”***\n    - We chose the  evaluation datasets and the encoder-decoder architecture based on the prior literature on variational approaches for hierarchical clustering (TreeVAE, NeurIPS 2023), so we could make a direct comparison with prior work. We believe the selected datasets cover a wide range of characteristics (i.e., grayscale vs. RGB images, handwritten vs. natural images, simple vs. complicated patterns, and 10 classes vs 100 classes) and are sufficiently comprehensive for our **proof-of-concept** study. **Nevertheless, we agree that inclusion of results on a more complex dataset would be more convincing, and we not that reviewer Lzud also made a similar suggestion.**\n    - **Based on these suggestions, we will add additional results for STL-10 (a popular subset of ImageNet-1k), which we were able to produce  during the short rebuttal period given our available compute resources. The results show that our approach is also robust in the face of more complex patterns:**\n        \n        \n        | **STL-10 Results** |  |  |  |  |\n        | --- | --- | --- | --- | --- |\n        |  | DP | LP | ACC | NMI |\n        | TreeVAE | 20.34% | 31.31% | 30.96% | 25.74% |\n        | **DeepTaxonNet** | **29.03%** | **37.93%** | **55.75%** | **40.56%** |\n\n### **Concerns on the lack of comparison to deep flat clustering methods such as SCAN:**\n\n- ***“...omits more recent and advanced deep clustering methods, such as SCAN…Could you clarify the rationale behind excluding these methods from your comparison…?”:***\n    - We would like to first clarify the scope of our work. We frame our work as a **proof-of-concept** experiment to show that our novel variational approach to hierarchical clustering **can discover hierarchical prototypes from data**. As a result, our analysis is centered around the added value of our approach (utilizing all levels of the hierarchy vs. other hierarchical clustering methods that do not learn intermediate prototypes). We used 4 hierarchical clustering metrics (LP, DP, ACC, NMI) to evaluate multiple hierarchical clustering methods and **our methods' benefit comes from the collective analysis from these metrics.** For instance,\n        - Leaf Purity (LP) measures the class entropy at the leaf nodes, and in our case are the nodes representing the most fine-grain semantic categories. As stated in our response to **reviewer Ludz**, Table 1 shows that despite having many more leaves ($2^{10}$ leaves) than TreeVAE and DeepECT (10 leaves, corresponding to 10 classes), our approach's leaf purity is comparable to other approaches (our approach has even higher LP), suggesting it forms **robust**, **consistent**, and **well-structured** fine-grained semantic classes at the **leaves**.\n        - *An interesting observation from the leaf-only approaches is that ACC is lower than LP*. This is expected as leaf-only approaches assume the same number of leaves as the classes so the classification accuracy depends on both the quality of leaf nodes representing a class (LP) and the routing quality that successfully brings data to a leaf node and hence *ACC should be upper bounded by LP*. **However, with our approach, we observe a substantial increase in ACC (e.g., on CIFAR datasets) compared to TreeVAE, despite having a similar LP. Notably, ACC is not upper-bounded by LP, as is the case with leaf-only approaches.** This result indicates that the *additional* ACC gain in our approach is the benefit of utilizing intermediate nodes. **In other words, by utilizing all levels, our approach can capture in the intermediate nodes what would otherwise be misclassified in the leaf nodes of leaf-only approaches. This translates into a better classification accuracy**.\n    - However, deep flat clustering methods lack hierarchical metrics such as LP and DP, and the only comparable metrics are ACC and NMI. As a result, results on deep flat clustering methods fail to provide a comprehensive analysis to our approach’s added value.\n- ***“...the evaluation lacks recent and stronger baselines such as SCAN,...”:***\n    - While we feel that comparing our new approach to deep flat clustering approaches is beyond the scope of our current work, we agree that showing competitiveness in *classification accuracy* to a stronger approach would significantly improve our work’s credibility. Based on this idea, we ran an analysis of *classification accuracy* between our approach and SCAN on CIFAR-10 for your reference.\n    - Our approach utilized the same encoder-decoder architecture as TreeVAE, so that we could directly compare to this prior approach. However, this encoder-decoder architecture is less powerful than a ResNet-18. To enable a direct comparison to SCAN, we utilized the ResNet-18 encoder-decoder architecture within our approach and tested it on CIFAR-10. For this experiment we kept the same hyperparameters as in our previous experiment. **We found that our approach achieves a much higher classification accuracy on CIFAR-10 when using the ResNet-18 encoder-decoder architecture as compared to the TreeVAE encoder-decoder architecture (\\~77\\% vs. \\~68\\% currently). With this updated architecture, our approach achieves similar performance to SCAN without self-labeling (\\~81.8\\%), even though we have not yet  tuned the hyperparameters for our approach with this new encoder-decoder architecture**.\n    - This result suggests that our theoretical framework indeed benefits from a stronger encoder-decoder architecture and is competitive in classification accuracy to a stronger approach like SCAN. While we did not evaluate our approach with the ResNet-18 encoder-decoder on the other datasets (we did not have sufficient time during the rebuttal period), **we plan to include an additional ablation study on the effect of stronger encoder-decoder architectures as well as *performance comparison* to approach like SCAN.**\n\nWe would like to thank you again for your valuable review and we believe the additional results for STL-10 and for CIFAR-10 with the updated encoder-decoder architecture add more credibility to our experimental validation and substantially strengthen the paper."}]

Author rebuttal:

Meta-review: This paper proposes Deep Taxonomic Networks (DTN), which uses variational inference and contrastive learning to discover flexible taxonomies from unlabeled data. The approach introduces a mixture-of-Gaussians prior structured over a complete binary tree, with both leaf and intermediate nodes representing clusters/prototypes.

The paper received scores of 4, 3, 4, 5. Most reviewers appreciated the theoretical overview and the mathematical formulation of the problem. There were questions on the benefit of intermediate clustering and the limited evaluations. However, through the rebuttal phase the authors have clarified several questions about comparison to baselines and evaluations on more recent benchmarks, which has convinced the reviewers about the validity and usefulness of this method. Thus, we recommend the paper for an Accept.
Decision: Accept (poster)