From Logits to Hierarchies: Hierarchical Clustering made Simple
Authors: Emanuele Palumbo
Moritz Vandenhirtz
Alain Ryser
Imant Daunhawer
Julia E Vogt
Venue: ICML 2025 poster (2025)
Keywords: Hierarchical Clustering
Clustering
Interpretability
Unsupervised Learning
Primary area: general_machine_learning->clustering

Abstract: The hierarchical structure inherent in many real-world datasets makes the modeling of such hierarchies a crucial objective in both unsupervised and supervised machine learning. While recent advancements have introduced deep architectures specifically designed for hierarchical clustering, we adopt a critical perspective on this line of research. Our findings reveal that these methods face significant limitations in scalability and performance when applied to realistic datasets.~Given these findings, we present an alternative approach and introduce a lightweight method that builds on pre-trained non-hierarchical clustering models. Remarkably, our approach outperforms specialized deep models for hierarchical clustering, and it is broadly applicable to any pre-trained clustering model that outputs logits, without requiring any fine-tuning. To highlight the generality of our approach, we extend its application to a supervised setting, demonstrating its ability to recover meaningful hierarchies from a pre-trained ImageNet classifier. Our results establish a practical and effective alternative to existing deep hierarchical clustering methods, with significant advantages in efficiency, scalability and performance.

From Logits to Hierarchies: Hierarchical Clustering made Simple
Emanuele Palumbo 12 Moritz Vandenhirtz Alain Ryser² Imant Daunhawer" Julia E. Vogt"*
Abstract
tering can provide more accurate insights than flat (i.e.. non-hierarchical) clustering methods by introducing mul- tiple levels of granularity and alleviating the need for fixed number of clusters specified priori (Bertsimas et al., 2021; Chami et al., 2020). This aids scientific understand- ing and interpretability by providing a more informative representation (Lipton, 2018; Marcinkeviës & Vogt, 2020). The benefits of modeling hierarchies in the data extend to supervised scenarios. For example, interpretable methods based decision trees (Breiman, 2001; Tanno et al., 2019) hierarchically partition the data so that points in each split are linearly separable into classes. More recent work lever- ages hierarchies in the data to improve supervised methods (Bertinetto et al., 2020; Goren et al., 2024; Karthik et al., 2021) for self-supervision (Long & van Noord, 2023).
The hierarchical structure inherent in many real- world datasets makes the modeling of such hier- archies a crucial objective in both unsupervised and supervised machine learning. While recent advancements have introduced deep architectures specifically designed for hierarchical clustering, we adopt a critical perspective on this line of re- search. Our findings reveal that these methods face significant limitations in scalability and per- formance when applied to realistic datasets. Given these findings, we present an alternative approach and introduce a lightweight method that builds pre-trained non-hierarchical clustering models. Remarkably, our approach outperforms special- ized deep models for hierarchical clustering, and it is broadly applicable to any pre-trained cluster- ing model that outputs logits, without requiring any fine-tuning. To highlight the generality of our approach, we extend its application to super- vised setting, demonstrating its ability to recover meaningful hierarchies from a pre-trained Ima- geNet classifier. Our results establish a practical and effective alternative to existing deep hierar- chical clustering methods, with significant advan- tages in efficiency, scalability and performance.
Among classic algorithms for hierarchical clustering, ag- glomerative methods have been the most widely adopted. These methods compute pairwise distances between data points, often in a lower-dimensional representation space. Starting from the instance level, a hierarchy is then built based on the pairwise distances by recursive agglomeration of similar points or clusters together in a bottom-up fash- ion (Murtagh & Contreras, 2011). More recently, revived interest in hierarchical clustering has sparked novel, sophis- ticated approaches using deep architectures (Mautz et al., 2020; Goyal et al., 2017; Shin et al., 2019; Vikram et al., 2019; Manduchi et al., 2023). These approaches require spe- cialized architectures and complex training schemes. In this work, we uncover two key limitations of these approaches. First, find that these methods struggle handle large- scale datasets, failing to deliver satisfactory performance. This is in part due to their high computational demands and in part due to the difficulty in modeling a large number of clusters. Second, we observe a notable gap in performance at the leaf level compared to non-hierarchical models. This is particularly problematic, as the advantage of introducing hierarchy should not come the expense of the quality of clustering at the leaf-level granularity. These limitations un- derscore the need for more scalable and effective approaches for hierarchical clustering.
1. Introduction
Modeling hierarchical structures in the data is long- standing goal in machine learning research (Bengio et al., 2013; Jordan& & Mitchell, 2015). In many real-world see- marios, data inherently organized in hierarchies, such as phylogenetic trees (Linnæus, 1758; Sneath Sokal, 1962; Penny, 2004), tumor subclasses (Sørlic et al., 2001) and social networks (Ravasz & Barabási, 2003; Crockett et al., 2017). In unsupervised learning, hierarchical clus-
"Shared last authorship 'ETH AI Center, Zurich. DDepartment of Computer Science, ETH Zurich. Correspondence to: Emanucle Palumbo <emanucle, mananuele.pallubbeee palumbo@ inf.ethz.ch>.
Given these findings, we take a critical perspective on re- cent research on deep hierarchical clustering and offer a straightforward yet effective alternative. Rather than de- signing specialized hierarchical clustering models, we de-
Proceedings of the nnnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 20255 by the author(s).

---

From Logits to Hierarchies: Hierarchical Clustering made Simple
velop lightweight method perform hierarchical clus- tering given pre-trained flat model. In particular, we show that a lightweight algorithm implemented on top of noon-hierarchica)) pre-trained models produces hierarchical clustering results that markedly outperform tatt--ffthe-art dedicated models. Notably, our algorithm, which we name Logits to Hierarchies (L.HH), only uses logits and requires no fine-tuning of the pre-trained model. Hence, it generally applies black-box models even without access to internal representations (e.g., API calls to proprietary models) and bypasses the costly computation of pairwise distances be- tween data points. Moreover, it also applies to supervised models, for which the inferred hierarchy of classes can aid model interpretability, e.g., for discovering potential biases such as spurious correlations between classes.
algorithm (Sneath, 1957; Ward, 1963). As examples, single, average, and complete linkage characterize the distance he- tween two clusters the minimum, average, and maximum distance between their data points, respectively. Since these a algorithms can be costly, partieularly in high-dimensional spaces, approximate versions have been developed for faster computation (Abboud et al., 2019; Cochez & Mou, 2015). Notably, linkage methods are still widely applied in many domains as, for instance, in medical research (Nguyen et al., 2024; Senevirathna et al., 2023; Resende et al., 2023). Re- cent work also adopts them to assess how well the repre- sentations from pre-trained encoders generalize to cluster unseen datasets (Lowe et al., 2024).
On the other hand, divisive algorithms start with all objects belonging the same cluster and recursively split them into subclusters. While early approaches are mostly based on heuristics, Dasgupta. (2016) proposed the Dasgupta cost: an objective function for evaluating a hierarchical clustering, with a divisive approach to provide an approximately opti- mal solution. HypHC introduces a continuous relaxation of Dasgupta's s discrete optimization problem with provable guarantees via hyperbolic embeddings that better reflect the geometry of trees compared to Euclidean representations (Chami et al., 2020; Liu et al., 2019). More recently, re- search has focused on developing deep learning approaches for hierarchical clustering with specialized architectures (Mautz et al., 2020; Goyal et al., 2017; Shin et al., 2019; Vikram et al., 2019; Manduchi et al., 2023). Among these, DeepECT learns a hierarchical clustering on top of the em- hedding space of a jointly optimized autoencoder (Mautz et al., 2020). TrE (Manduchi et al., 2023), on the other hand, learns hierarchical clustering in the latent space of a variational autoencoder and provides generative model that adheres to the learned hierarchy, thereby enabling sample generation in a structured manner (Manduchi et al., 2023). However, these approaches have mostly been tested on sim- ple datasets, far from realistic settings. In our experiments in Section 4.1, we show that they present important limitations when deployed on more challenging datasets. We find these limitations to be linked to their high computational demands and their difficulty in modeling hierarchies that consist of a large number of leaf clusters.
In summary, we make the following key contributions:
We reveal significant limitations of recently proposed deep specialized methods for hierarchical clustering, highlighting their weaknesses on large-scale datasets and their subpar performance at the level compared to non-hierarchical approaches.
As an alternative approach, we propose straightfor- ward algorithm for hierarchical clustering that from the logits of pre-trained flat model derives hierar- chical structure of clusters. Our method markedly out- performs specialized hierarchical models and has low computational requirements. With logits as its input, it computes a hierarchical clustering on ImageNet-sized datasets in under a minute on a CPU.
To demonstrate how our method also applies super- vised models, we provide case study on ImageNet, showing how the inferred hierarchy of classes recovers parts of the WordNet hierarchy, and helps to discover potential spurious correlations in the pre-trained model or biases in existing categorizations.
2. Related Work
Hierarchical clustering aims to learn clusters of data points that are organized in hierarchical structure. The methods used can be broadly categorized into agglomerative and di- visive approaches (Nielsen, 2016). The former tackle the problem with a bottom-up approach and iteratively agglom- erate clusters into larger ones until a full hierarchy is built in the form of dendrogram, starting with each datapoint being separate cluster (Murtagh & Contreras, 20. The similarity of data points is measured according to distance function, which for high-dimensional data is often defined on a pnver-dimensiona representation space. Multiple link- age methods have been proposed to compute the distance between clusters of data points formed at a given step of the
Finally, the benefits of modeling a hierarchy in the data are not restricted to the unsupervised setup (Khrulkov al., 2020; Linderman et al., 2023; Sinha et al., 2024). In particu- lar, recent research focuses on leveraging a tree structure in the classes to assess and reduce the severity of misclassifi- cation of supervised models (Karthik et al., 2021). This can lead to safer models in cost-sensitive scenarios (Bertinetto et al., 2020) and allow a classifier to predict at different levels of the hierarchy depending on the required confi- dence (Goren et al., 2024). The visualization of hierarchies also provides global explanations of a model's functionality,
2

---

From Logits to Hierarchies: Hierarchical Clustering made Simple
thereby improving a user's understanding of the model be- havior and fostering trust (Chakraborty al., 2017; Lipton, 2018).
given K-dimensional vector o and set G C {I,.... Kl....,Kk). Kj. This function restricts the softmax operation the elements of at indexes in {I....,,k G. Next, we define
hi (x;; G) = argmax m..sootmxx(ff(e)) (E.......K)
3. Method
In this work, we take a critical perspective on a recent line of research on hierarchical clustering. After uncovering important limitations of recent hierarchical clustering ap- proaches with deep specialized architectures, we propose an alternative strategy. Namely, rather than designing ad- hoc complex methods, we focus on adapting pre-trained flat models to output a hierarchy with minimal overhead. To this end, we introduce a lightweight algorithm to leverage the information contained in the logits of pre-trained flat clustering model to output a hierarchy of clusters. In the following, we describe the proposed procedure and also pro- vide graphical illustration as well as detailed pseudocode.
96 (2; G)
max softmaxx(fo
ke(
where the functions hg and 98 correspond to hy and go but restricting the choice of viable clusters to {1..... K} G. In particular, the function hi computes the cluster assignment for datapoint æ restricting to clusters in {I.... K} \ G. and gō computes the corresponding predicted probability. Lastly, we define
D' = (x E D hg(2)) = c}
i.e., the subset of data points assigned a given cluster {1...., K). Similarly, denote DC De the subset of data points assigned to a group of clusters G c (K).
Let D- = { NN be a dataset consisting of N data points and fo be a nin-hiirarl model trained to parti- tion D into K clusters. We assume that fe outputs unnor- malized logits, from which the cluster assignment k* for datapoint is determined by taking the softmax
Algorithm Logits to Hierarchies (L2H).
Given aggregation function A, and functions ge. 96 defined as above for pre-trained K -llustering model fa.
h = argmax softmax:(fo(z:))
K)
Input: Dataset D.
Output: Hierarchy H.
We define two functions
# Groups initialized as single clusters
hx(x)= argmax (fftmax((fe(x)))
Initialize groups [Gi,...., Gx} where G2 (b:}
(E(.....K)
Hierarchy initialized as empty list
go(x)
max softmax; (fe(z))
H= [
ke
(y......
for step / from 1 to K - do
of which he computes the cluster assignment for a datapoint v, while go computes the predicted probability of the cluster assignment for the datapoint æ.
for group G in G do
# Compute group scores as in Eqn. Compute s(G) : Aue leo (x(x) end for
A key idea behind method is simple yet effective way to determine the relatedness of clusters, groups of clus- ters, while iteratively grouping them together to construct hierarchy. Intuitively, to assess which group of clusters G' is most related to a given group G, we propose the fol- lowing strategy: for data points assigned to clusters in G, we determine which group G' would have the majority of these data points reassigned to, if clusters G were not available. Formally, we define the following functions to compute cluster assignments and corresponding predicted probabilities, restricting only to subset of the total set of clusters.
# Select group with lowest score for merging
Take G* argmingeg
for cluster in {1, {1,..., K} \ G do
# Compute total pred. probability per cluster as in Eqn.2 Compute rp(c) := =
=ED 96 (x; G*)
bgy (m:G")=n
end for
Select G*, most related group to G", for merging Take G argmax [द| ΣeE rp(c)
Update groups
Update & by merging groups G and G*
Update hierarchy
Update H by adding that G and Gf are merged at step f end for
We start with a masked version of the softmax function exp(n;)
We describe our proposed method in Algorithm 1. At the start of the procedure, K groups are initialized as single clusters. 2 At each iteration, two groups are merged into a
if E G if E G
(o; G) Xje exp(
'This passage is primarily for intuition and not strictly accu- rate. To be precise, we do not look at reassignments but rather at predicted probabilities of reassignments (see Equation (2)).
Note that here cluster is used to refer to single cluster found by the pre-trained model, while group refers to set of clusters that are grouped together at a given step of the algorithm.
3

---

From Logits to Hierarchies: Hierarchical Clustering made Simple
Iteration 2
Mostrenssignn
(werighhacood across dusters pront
Iteration
lo...
Mostreanigged
probatilty
Figure I: Illustration the L2H algorithm. The four depicted clusters represent dogs in blue, cats in yellow, horses in red, birds in green respectively. the first iteration (bottom), where groups correspond to single clusters, the dog cluster selected for merging (shaded in grey). When recomputing predicted probabilities for samples in the dogs cluster, restricting the remaining clusters, the cluster of has the highest predicted probability of reassignment. Note how, after merging, these two clusters are considered as single group in the next iteration (top).
single group, constructing tree of clusters up to the root in K - iterations. Each iteration can be split into two stages. In the first stage, score computed for each group. To compute the score for a given group G we aggregate the predicted probabilities for the data points assigned to
the group of clusters G". The most related group to G* is finally selected as G E argmax (G*) [ EeEG rp(c), i.e., by averaging the total reassigned predicted probability across clusters in each group and selecting the group with the highest average.
clusters contained in G
Given that cluster assignments and corresponding predicted probabilities can computed via simple operations on the logits, the whole procedure can be executed given only the logits from pre-trained model fo for the dataset D. For further clarification, we provide graphical illustration of the proposed grouping strategy (Figure 1) and an example Python implementation in Appendix A.
(G)
(0():)
EEG
where chosen aggregation function (c.g., the sum). Then, the lowest-scored group G* is selected for merging at this iteration, which concludes the first stage.
In the second stage, we search for the group G that is most related to G to perform the merging. To do so, as men- tioned above, we look at the subset of data points assigned clusters in G*: for these data points, we recompute cluster assignments and predicted probabilities, this time restricting to clusters not contained in G More formally, the total re- assigned predicted probability to each cluster not contained in G* is computed as
4. Experiments
In this section, present the experimental results of study. In the first part, we empirically demonstrate that ex- isting specialized deep hierarchical clustering models face significant limitations in realistic scenarios. These limita- tions stem from their high computational demands and their difficulty handling a large number clusters. In contrast, our proposed method demonstrates compelling results on challenging vision datasets, achieving substantially better performance compared to these specialized models.
rp(c)=
Σ
96 (x;
Vc (L,,,KK,
DED
hg (e
(2)
Note that this quantity can be interpreted as a measure of relatedness between each cluster c E (1,.., K} \ G", and
While the first part of this section focuses on hierarchi-

---

From Logits to Hierarchies: Hierarchical Clustering made Simple
cal clustering, the second part extends our approach su- pervised setups. We provide case study showeasing the application of the L2H algorithm top of a pre-trained ImageNet classifier. The results illustrate its potential for enhancing model interpretability and uncovering spurious correlations. Moreover, they complement the results for hierarchical clustering, demonstrating that our method can also be applied on top of pre-trained supervised models. Further details datasets, implementations and metrics can be found in Appendix B.
but impacts its scalability large-scale datasets (see Table 2 for more details). Moreover, the comparison in terms of flat clustering metrics highlights that deep hierarchical mod- els produce clusterings at the leaf level that are much less accurate than those obtained with non-hierarchical models.
In contrast to alternative approaches, our proposed method recovers high-quality hierarchies for all three datasets when implemented with both TEMI or TURTLE as the backbone model. The results in the hierarchical metrics demonstrate that our method markedly outperforms existing approaches, with a consistent margin over costly deep learning special- ized models. These findings demonstrate that the L,HH algo- rithm can leverage the information embedded in the logits of a pre-trained flat clustering model to model an accurate hierarchy of the clusters. Most importantly, they show that our approach outperforms sophisticated deep hierarchical models while being more scalable and efficient. Note as well that our method does not require any fine-tuning of the pre-trained model, nor access to the internal representations. By construction, it retains the clustering performance of the pre-trained model at the leaf level, which matches state-of- the-art in our results. Importantly, the efficacy of our method is not hindered by the presence of large number of classes in the dataset, as we witness for other methods. In particular, in Appendix C we show that our method achieves remark- able hierarchical clustering results on datasets as large as ImageNet- Ik (Deng et al., 2009). As well we show in Ap- pendix C that our approach is applicable across different choices for the backbone model, and notably, outperforms deep specialized hierarchical approaches even when the cho- sen backbone model yields weaker clustering performance at the leaf level than TURTLE or TEMI. We also report additional ablations, and in particular we validate the robust- ness of our approach with respect to the hyperparameter K, corresponding to the number of clusters at the leaf level of the hierarchy.
4.1. Hierarchical Clustering
In this section, we evaluate the performance of recent spe- cialized hierarchical clustering approaches on three chal- lenging vision datasets- CIFAR-10, CIFAR-100 (Lake et al.. 2015) and Food-101 101 (Bossard et al., 2014)- com- paring their performance with our proposed method, We show our results in Table I, including metries to evaluate models at the leaf level and metrics to evaluate the quality of the produced hierarchy. To compare model performance at the leaf level, report Normalized Mutual Information (NMI), Adjusted Rand Index (ARI), Accuracy and Leaf Pu- rity (LP). To assess the quality of the hierarchical clustering, we report two metrics: Dendrogram Purity (DP) and Least Hierarchical Distance (LHD). The former was introduced in Kobren et al. (2017) and extends the notion of purity, normally evaluated at the leaf level, to assess the quality of tree clustering: higher purity corresponds to higher quality of the hierarchy. Note that this metric was recently adopted in Manduchi et al. (2023) to benchmark deep hierarchical clustering models. Least Hierarchical Distance, on the other hand, measures the average minimal log-distn deedtsttnce in the hi- erarchy between any pair of data points that have the same true label but different cluster assignments. A better hier- archy corresponds to a lower LHD. More details about our metrics can be found in Appendix B.3. For each dataset, we implement our L2H algorithm on top of two pre- pre-trained flat clustering models, namely TURTLE (Gadetsky et al., 2024) and TEMI (Adaloglou et al., 2023). These are two recent clustering methods (see Appendix B.2 for more details), both of which are not designed to produce hierarchy of clusters but only flat clustering.
In practice, hierarchical clustering results are often used as visualization tool and to analyze the structure of a dataset at different levels of granularity. Hence, to evaluate our proposed approach, we visualize and inspect the hicrarchy obtained with TUR-TU on the CIFAR- dataset in Figure 2. Note that given the absence of leaf labels, we associate class label to each Icaf by looking at the most frequent label among the data points in the given leaf, While an able for the CIFAR- 100 dataset, the authors organize the 100 classes in 20 superclasses. Hence, we color-code the in- ferred leaf labels in the hierarchy by superclasses and check if the hierarchical clustering recovers this global structure. Notably, the global structure of the superclasses is largely
The results in Table 1 uncover the aforementioned limita- tions of recent deep hierarchical clustering methods (Deep- ECT, VAE), which fail to achieve satisfactory perfor- mance even on moderately challenging datasets such as 10. The results prove that these approaches strug- gle at modeling deep hicrarchies, producing overly shallow trees in datasets with a large number of classes such as CIFAR-100 or 101. This is also linked to their high computational complexity. For instance, TreeVAB learns a hierarchical generative model with leaf-specific decoders: this choice helps its performance in a generative scenario
ground-truth hierarchy is not avail-
'In constrast, our approach retains the strong performance of the backbone model (TURTLE or TEMI) at the leaf level.
5

---

From Logits to Hierarchies: Hierarchical Clustering made Simple
Flat
Hicrarchical
Inference on test set
leaves
NMI (1)
ARI (1)
ACC (t)
LP (t)
DP (t)
LHD (4)
CIFAR-10 Agglomerative
0.074
0.038
0.211
0.246
0.121
0.549
10
X
HypHC
0.019
0.009
0.134
0.359
0,104
0,569
10
X
DeepECT
0.006
0.002
0.110
0.110
0.101
0.369
2-3
/
Tre
0,414
0.313
0.497
0.523
0.341
0.410
L2H-TEMI
0,907
0.906
0.956
0.957
0.902
0.348
10
1.2H-Turlle
0.985
0.989
0.995
0.995
0.988
0.277
10
V
CIFAR-100 Agglomerative
0.223
0.020
0.090
0.131
0.019
0.428
100
X
HypHIC
0.072
0.004
0.031
0.560
0.011
0.499
100
X
DeepECT
0.016
0.005
0.070
0.070
0,052
0.121
2-3
Tre
0.199
0.098
0.228
0.242
0.103
0.484
20
L2H-TEM
0.778
0.565
0.682
0.701
0,502
0.298
100
L2H-Turtle
0.917
0.831
0.896
0.897
0.803
0.235
100
Food-101 Agglomerative
0.082
0.004
0.039
0.045
0.011
0.438
101
X
HyplIC
0.035
0.002
0.022
0.630
0.011
0.573
101
X
DeepECT
0.003
0.000
0.011
0.011
0.010
0.333
2-3
TreeVAE
0.114
0.017
0.057
0.058
0.016
0,483
20
L2H-TEMI
0.917
0.841
0.904
0.913
0,801
0.270
101
L2H-Turtle
0.894
0.800
0.876
0.877
0.758
0.297
101
Table 1: Quantitative comparison of hierarchical clustering performance on three datasets (CIFAR-10, CIFAR- 100, Food- 101). We report baseline agglomerative clustering, deep hierarchical specialized models (DeepECT, TreeVAB), and our L2H method applied on top of two state-of-the-art flat models (TEMI, TURTLE). We also indicate the number of leaves in the hiearchy modelled by each approach, and whether a given method can perform inference on a hold-out test set. We bold best results for each metric and underline results that are artifacts of degenerate solutions with shallow hierarchies. Notably the application of L2H does not affect flat clustering performance, retaining the clustering performance of the pre-trained model (TURTLE, TEMI) at the leaf level.
Dataset
CIFAR-10
CIFAR-100
Food-101
ImageNettk K = 1000
K = 10
K = 100
K = 101
Nr 50000
No = 50000
Nr 75750
Nr 1281167
L2H
< 0.01
< 0.01
< 0.01
0.45 + 0.0
Agglomerative
< 0.1
< 0.1
0.8
HypHC
4.0
153.3 19.4
195.3 3.2
DeepECT
24.1 15.1
2.8
67.5 36.6
Tree VAE
364.1 76.8
756.3 178.6
2293.7 211.7
L2H-TURTLE
1.6
1.6 0.0
1.7 0.0
5.25 0.0
Table 2: Training time (in minutes) for our proposed method compared to baselines for hierarchical clustering on CIFAR-10 CIF,R-100, and F01 datasets. At the top, we report the runtime for the L2H algorithm alone. Below, we report the runtime of the TURTLE model plus our L.2H algorithm to produce a hierarchy, compared with the runtime of each baseline model. Results are averaged over three runs and include standard deviations.
reflected in the visualized hicrarchy. Most interesting is that the outliers, for which the color does not coincide with the neighboring leaves, still reflect meaningful semantic asso- ciations. For instance, whale and folphin-despie being aquatic mammals-are grouped with fish species. However, this is not surprising, given their adaptation exclusively to aquatic environments and the presence of similar traits to fishes, like streamlined bodies. On the contrary, mammals such as otter, beaver, and seal, which are only semi-aquatic,
are grouped with other small to medium-sized terrestrial mammals, emphasizing size and communal characteristics like the presence of limbs and fur. Another example the characterization of worm and snake alongside in the hierar- chy. Although snakes are reptiles, their elongated, limbless bodies visually resemble those of non-insect invertebrates like worms. This showcased analysis confirms the efficacy of our method in recovering tree structure that follows meaningful semantic associations. The results indicate that
6

---

From Logits to Hierarchies: Hierarchical Clustering made Simple
ranges
Elecronic
comminees
Vel idee
natund cuttier
man-mrade cutsoor rlings
Treas
Iroects
Kon insect inventtratos
Large orrrivorcs and horthvres
Lange camboress
Small mammale
Mediutt- memmssll
Rower
taple cru
Ren
Lange aquatic mamnala
Rapllis
Figure 2: Visualization of the hierarchical clustering produced by L2H-TURTLE on the CIFAR-100 dataset. The inferred hierarchy is represented as a circular tree. On the lowest level, the leaves are annotated by reporting the most frequent label for the samples in each leaf. Leaves are color-coded according to the superclasses the dataset.
our method produces hicrarchies that enable detailed ex- ploration of the structure in the data varying levels of granularity. Inspecting the hierarchy gives valuable insights for interpretability, revealing underlying associations by the model.
deep hierarchical approaches exhibit a significantly higher computational cost. Moreover, with Tree VAE, increasing dataset size and number of classes markedly increases the computational burden. Finally, it is to note that TURTLE, as well as TEMI, leverages CLIP embeddings for training, Hence, to further validate the superior efficiency and effi- cacy our method compared to alternative deep specialized approaches, in Appendix C we compare our results with the ones obtained by training Tre using CLIP embed- dings. These results again validate that our L2H approach achieves markedly superior performance when compared models have access to foundation model embeddings for training. Once again, this gain in performance comes also with much higher computational efficiency.
We end this section with a comparison in terms of the com- putational cost of our method compared with alternative models, and in particular with specialized deep learning approaches for hierarchical clustering. As the results in Table show, proposed L2H algorithm is extremely lightweight. To compare with the efficiency of alternative methods, we also measure the overall runtime to perform hicrarchical clustering with L2H-TURTLE, which includes the runtime train the TURTLE model, as an example of backbone model. Our approach allows us to perform hierar- chical clustering extremely efficiently, even large-scale datasets such ImageNet-Iee 1k, with total training time of a few minutes. Note that, due to the combined efficiency of our method and state-of-the-art flat clustering models like TURTLE, the overall runtime scales seamlessly with dataset size and number of leaves in the hierarchy. Conversely,
4.2. Case Study: Pre-trained ImageNet Classifier
In this section we complement the results from the previous section by showing that our method can be applied a supervised setup, producing hierarchy given the logits of pre-trained classifier. Specifically, we use the ImageNet- lk dataset (Deng al., 2009), which comprises million

---

From Logits to Hierarchies: Hierarchical Clustering made Simple
Adoo price
1932
Affiien chig
vindpex
pollinac
(a) Complete tree (IK classes)
(b) Subtree of birds (58 classes)
Figure 3: Visualization of inferred hierarchy for the ImageNet-IK dataset. The hierarchy is represented as a circular tree, where the leaf nodes are organized in circle. Figure 3a shows the complete tree colored by the corresponding WordNet hypernyms "artifact" and "organism", which the largest superclasses in the ImageNet dataset. Figure 3b shows the subtree of birds colored by different bird species if they comprise more than one class. The results show that our method recovers significant portion of the global and local hierarchical structure of the ImageNet dataset.
images and thousand distinct classes, with an underlying hierarchical structure organized according to the WordNet hierarchy. We apply the L.HH algorithm on the logits of pre-trained ImageNet classifier model the hierarchy of classes. As pre-trained classifier, model (Wang et al., 2023).
Overall, our results suggest that the inferred hierarchy recov- ers a significant portion of the global and local hierarchical structure the ImageNet dataset given the logits a pre- trained ImageNet classifier trained with oon-hierarchical labels. Yet, the inferred tree also reveals interesting outliers. For example, in Figure 3a, there is distinct subtree for snow-related artifacts (e.g., dogsled, snowmobile, bobsled) within large branch of the tree that comprises organisms. Further investigation shows that this group of artifacts merged with arctic animals (c.g.. malamute, Siberian husky, Eskimo dog), which reveals a spurious correlation between classes and highlights potential biases of the pre-trained model. Likewise, in Figure 3b, see potential outliers such as "bustard" among game birds. Interestingly. this con- stitutes an example where our method reveals an ambiguity in the WordNet hierarchy, which classifies "bustard" as a wading bird, while it is usually defined as a terrestrial game bird, like our inferred categorization.
the InternImage
Figure 3a illustrates the inferred hierarchy for the thou- sand ImageNet classes. The colors indicate whether a leaf node corresponds to the superclass "artifact" or "organism", which are the largest two superclasses that can deter- mined based on the corresponding WordNet hypernyms of each class in ImageNet. Overall, observe distinct separation between the two superclasses in the inferred tree.
In addition, Figure 3b shows subtree of the inferred hi- erarchy that comprises different bird species. Specifically, shows 58 of the classes of birds contained in the Im- ageNet dataset. The leaf nodes are colored by different clades of bird species (based the WordNet hierarchy), showing that the inferred hierarchy groups together related species. For example, the group "aquatic bird" is almost completely represented in one of the two main branches, which further splits into separate cluster for "parrots" and another one for "bird of prey". The other main branch of tree subdivides further into "passerine" and "game bird" forming distinct clusters.
5. Conclusion
In this work, we uncover significant limitations of exist- ing deep hierarchical clustering models, demonstrating that these methods face important challenges in scaling to large-scale datasets, falling short of delivering satisfac- tory performance. As an alterattive approach, we propose
8

---

From Logits to Hierarchies: Hierarchical Clustering made Simple
lightweight yet effective procedure for hierarchical clus- tering based on pre-trained non-hierarchical models. No- tably, our solution proves to be markedly more effective and significantly more computationally efficient than existing methods. Specifically, we demonstrate that our method can successfully handle large datasets with hundreds of classes, taking an important step for the practical applicability of hierarchical clustering methods in realistic settings. More- over, we show that the usefulness of our approach extends to supervised setups by implementing it on top of pre-trained classifier to recover a meaningful hierarchy of classes. A case study on ImageNet shows that our approach provides relevant insights for interpretability, and can reveal potential biases in the pre-trained model or spurious correlation in the data.
References
Abboud, A., Cohen-Addad, V., and Houdrouge, H. Sub- quadratic high-dimensional hierarchical clustering. In Ad- vances in Neural Information Processing Systems, 2019.
Adaloglou, N., Michels, F., Kalisch, H., and Kollmann, M. Exploring the limits of deep image clustering using pretrained models. In British Machine Vision Conference, 2023.
Bengio, Y., Courville, A. C., and Vincent, P. Representation learning: A review and new perspectives. IEEE Trans- actions on Puttern Analysis and Machine Intelligence, 2013.
Bertinetto, L., Mueller, R., Tertikas, K., Samangooci, S., and Lord, N. A. Making better mistakes: Leveraging class hierarchies with deep networks. In Computer Vision and Pattern Recognition. 2020.
Our proposed method is general and may be applied to differ- ent data types beyond vision, which we leave as opportu- nity for future work. Another direction for future work is the investigation of strategies for automatically selecting mean- ingful levels of the inferred hierarchy. Hierarchical clus- tering presents important advantages over non- -on-hierarchical clustering by simultancously capturing the structure in the data at multiple levels of granularity. However, manually inspecting the hierarchy is still necessary to extract valuable insights. Thus, future work could investigate strategies to partly bypass this process, automatically selecting levels of the hierarchy that provide the most meaningful clustering. Our proposed method relies on the assumption that logits from a pre-trained flat model can be used proxy to mea- sure cluster similarities. While empirically supported by our experimental results, this assumption may not always hold, for instance if the pre-trained model is poorly calibrated.
Bertsimas, D., Orfanoudaki, A., and Wiberg, H. Inter- pretable clustering: an optimization approach. Machine Learning, 2021.
Bossard, I.., Guillaumin, M., and Van Gool, L. Food- 101 Mining discriminative components with random forests. In European Conference on Computer Vision, 2014.
Breiman, L. Random forests. Machine Learning, 2001.
Chakraborty, S., Tomsett, R., Raghavendra, R., Harborne, D., Alzantot, M., Cerutti, F., Srivastava, M., Preece, A.. Julier, S., Rao, R. M., et al. Interpretability of deep learning models: A survey of results. In IEEE smart- world, ubiquitous intelligence & computing, advanced & trusted computed, scalable computing & communications, cloud & big data computing, Internet of people and smart city innovation AaaarwotdSCA.CCOMUUAAACCC com/IOP/SCI), 2017.
Acknowledgements
EP is supported by a fellowship from the ETH AI Center, and received funding from the grant #2021-911 of the Strate- gic Focal Area "Personalized Health and Related Technolo- gies (PHRT)" of the ETH Domain (Swiss Federal Institutes of Technology). AR is supported by the Stimul grant #1-007811-44 1-002 and the Vontobel Foundation. The authors are grateful to Laura Manduchi for useful discussions.
Chami, I., Gu, A., Chatziafratis, V., and Ré, C. From trees to continuous embeddings and back: Hyperbolic hier- archical clustering. In Advances in Neural Information Processing Systems, 2020.
Cochez, M. and Mou, H. Twister tries: Approximate hier- archical agglomerative clustering for average distance in linear time. In ACM SIGMOD International Conference on Management of Data, 2015.
Impact Statement
As other ML models, clustering nethods-such the ones explored in this work-are vulnerable to the risk of reflect- ing potential biases and spurious correlations from the data they are trained on. However, hierarchical approaches offer a promising direction for mitigating these risks by enhanc- ing the interpretabiliity and transparency of clustering and classification outcomes.
Crockett, K., Melean, D.,. Latham, A., and Alnajran, N. Cluster analysis of twitter data: A review of algorithms. In International Conference on Agents and Artificial In- telligence, 2017.
Dasgupta., S. A cost function for similarity-based hier- archical clustering. In ACM symposium on Theory of Computing, 2016

---

From Logits to Hierarchies: Hierarchical Clustering made Simple
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L.. Imagenet: A large-scale hierarchical image database. In Computer Vision and Pattern Recognition, 2009.
Long, T. and van Noord, N. Cross-modal scalable hyper- bolic hierarchical clustering. In International Conference Computer Vision (ICCV), 2023.
Gadetsky, A., Jiang, Y., and Brbic, M. Let go of your labels with unsupervised transfer. In International Conference on Machine Learning, 2024.
Lowe, S. C., Haurum, J. B., Oore, S., Moeslund, B., and Taylor, G. W. An empirical study into clustering of unseen datasets with supe-sue encoders. arXiv preprint arXiv:2406.02465, 2024.
Goren, S.,
Galil, I., and EI- El-Yaniv, Hierarchical selective classification, Advances in Neural Information Process- ing, 2024.
Manduchi, L.., Vandenhirtz, M., Ryser, A., and Vogt, J. Tree variational autoencoders. Advances in Neural Information Processing Systems, 2023.
Goyal, P., Hu, Z., Liang, X., Wang, C., and Xing, E. Non- parametric variational auto-encoders for hierarchical rep- resentation learning. In International Conference on Com- puter Vision (ICCV), 2017.
Marcinkevičs, R. and Vogt, J. E. Interpretability and ex- plainability: A machine learning zoo mini-tour. arXiv preprint arXiv:2012.01805, 2020.
Jordan, M. I. and Mitchell, T. M. Machine learning: Trends, perspectives, and prospects. Science, 2015.
Mautz, D., Plant, C., and Böhm, C. DeepECT: The deep embedded cluster tree. Data Science and Engineering, 2020.
Karthik, S., Prabhu, A., Dokania, P. K., and Gandhi, V. No cost likelihood manipulation at test time for making better mistakes in deep networks. In International Conference on Learning Reprexentations, 2021.
Miller, G. A. Wordnet: a lexical database for english. Com- mun. ACM, 1995.
Murtagh, F. and Contreras, P. Algorithms for hierarchi- cal clustering: an overview. WIREs Data Mining and Knowledge Discovery, 2011.
Khrulkov, V., Mirvakhabova, L., Ustinova, E., Oseledets, I., and Lempitsky, V. Hyperbolic image embeddings. In Computer Vision and Pattern Recognition, 2020.
Nguyen, Y., Nocturne, G., Henry, J., Ng. W.-F., Belkhir, R., Desmoulins, F., Bergé, E.,, Morel, J., Perdriger, A., Demiss, E., Devauchelle- V., Sène, D., Dieudé, P., Coudere, M., Fauchais, A.,L., Larroche, C., Vittecoq, O., Salliot, C., Hachulla, E., Le Guern, V., Gottenberg, J.-E., Mariette, X., and Scror, R. Identification of distinct subgroups of Sjögren's 's disease by cluster analysis based clinical and biological manifestations: data from the cross-sectional paris-saclay and the prospective assess cohorts. The Lancet Rheumatology, 2024.
Kobren, A., Monath, N., Krishnamurthy, A., and McCallum,
A. hierarchical algorithm for extreme clustering. In
ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2017.
Lake, B. M., Salakhutdinov, R., and Tenenbaum, J. B. Human-level concept learning through probabilistic pro- gram induction. Science, 2015.
Letunic, I. and Bork, P. Interactive tree of life (iTOL) v6: recent updates to the phylogenetic tree display and anno- tation tool. Nucleic Acids Research, 52, 2024.
Nielsen, F. Hierarchical clustering. Introduction to HPC with MPI for Data Science, 2016
Linderman, R., Zhang, J., Inkawhich, N., Li, H., and Chen, Y. Fine-grain inference on out-of-distribution data with hierarchical classification. In Conference on Lifelong Learning Agents (CoLLAs), 2023.
Pedregosa, F, Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., Vanderplas, J., Passos, A., Cour- napeau, D.,, Brucher, M., Perrot, M., and Duchesnay, Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 2011.
Linnæus, C. Systema natura per regna tria natura, secun- dum classes, ordines, genera, species, cum characteribus, differentüss, synonymis, locis, volume 1. Salvius, Holmiæ (Stockholm), 10th edition, 1758.
Penny, D. Inferring Phylogenies. -Jospph Felsenstein. 2003.
Sinauer Associates, Sunderland, Massachusetts. System- atic Biology, 2004.
Lipton, Z. C. The mythos of model interpretability: In machine learning, the concept of interpretability is both important and slippery. ACM Queue, 2018.
Ravasz, E. and Barabási, A.-L. Hierarchical organization in complex networks. Physical review E, 2003.
Liu, Q., Nickel, M., and Kiela, D. Hyperbolic graph neural networks. Advances in Neural Information Processing Systems, 2019.
Resende, V., Tsilimigras, D. I., Endo, Y., Guglielmi, A., Ratti, F., Aldrighetti, L.., Marques, H. P., Soubrane, O.,
10

---

From Logits to Hierarchies: Hierarchical Clustering made Simple
Lam, V., Poultsides, G. A., Popescu, I,, Alexandreseu, S., Gleisner, A., Martel, G., Hugh, T., Endo, I., Shen, F., and Pawlik, T. M. Machine-based learning hierarchical cluster analysis: Sex-based differences in prognosis following resection hepatocellular carcinoma. World Journal of Surgery, 2023.
large-scale vision foundation models with deformable convolutions. Computer Vision and Pattern Recognition, 2023.
Ward, H. Hierarchical grouping to optimize an objective function. Journal of the American Statistical Association, 1963.
Senevirathna, A. M., Pohl, A. J., Jordan, M. J., Edwards, W. B., and Ferber, R. Differences in kinetic variables between injured and uninjured rearfoot runners: A hierar- chical cluster analysis. Scandinavian Journal of Medicine & Science in Sports, 2023.
Yunfan, L., Mouxing, Y., Dezhong, P., Taihao, L., Jiantao, H., and Xi, Twin contrastive learning for online clus- tering. International Journal of Computer Vision, 2022.
Shin, S.JJ,, Song, K., and Moon, L.-C. Hierarchically clus- tered representation learning. In AAAI Conference on Artificial Intelligence, 2019.
Sinha, A., Zeng, S., Yamada, M., and Zhao, H. Learning structured representations with hyperbolic embeddings. In Advances in Neural Information Processing Systems, 2024.
Sneath, P. H. The application of computers to taxonomy. Microbiology, 1957.
Sneath, A. and Sokal, R. R. Numerical taxonomy. Nature, 1962.
Sørlie, T., Perou, C. M.. Tibshirani, R., Aas, T., Geisler, S., Johnsen, H., Hastie, T., Eisen, M. B., van de Rijn, M., Jeffrey, S. S., Thorsen, T., Quist, H., Matese, J. C., Brown, P. O., Botstein, D., Lønning. P. E., and Bereesen-Dale, A. L. Gene expression patterns of breast carcinomas distinguish tumor subclasses with clinical implications. Proceedings of the National Academy of Sciences of the United States of America, 2001.
Tanno, R., Arulkumaran, K., Alexander, D. C.,, Criminisi, A., and Nori, A. Adaptive neural trees. In International Conference Machine Learning, 2019.
Van Horn, G., Cole, E., Beery, S., Wilber, K., Belongie, S., and Mac Aodha, O. Benchmarking representation learning for natural world image collections. In Computer Vision and Pattern Recognition, 2021.
Vandenhirtz, M., Barkmann, F., Manduchi, L., Vogt, J. E., and Boeva, V. setree: Discovering cellular hierarchies in the presence of batch effects in serna-seq data. arXiv preprint arXiv:2406.1 10..0066.930 19300, 2024.
Vikram, S., Hoffman, D., and Johnson, M. The loracs prior for vaes: Letting the trees speak for the data. International Conference Artificial Intelligence and Statistics, 2019.
Wang, W., Dai, J., Chen, Z., Huang, Z,, Li, Z., Zhu, X., Hu, X., Lu, T., Lu, L.., Li, H., al. Internimage: Exploring

---

From Logits to Hierarchies: Hierarchical Clustering made Simple
Appendix
A. Code Implementation of the I.H Algorithm
In Figure 4, we provide Python implementation of the L2H algorithm proposed in this work using standard scientific computing libraries (NumPy, SciPy). As stated in Section 3, our algorithm only requires the logits as input. It can be executed on the CPU even for large datasets, e.g.. with runtime of less than minute for ImageNet. Note that our procedure can be applied to any pre-trained unsupervised model to perform hierarchical clustering. Further, it can also be applied to logits from a supervised model to infer a hierarchy of classes. We store the hierarchy as a list comprised of groups of clusters that are merged iteratively. The aggregation function for computing the score per group is design choice (as described in Appendix B.2) that can be viewed as a hyperparameter.
import numpy np from scipy. special import zoftmax
def (log ts):
128 Algorithm.
Args:
logits: Logits and & is the number of clusters
where number dstapoints the dataset
Returns:
stepe: Nerging stepa characterizing the hierazchy
***
# Number of clusters is equal to size of last disension in the logits Logits. shape[-1]
initialize groups of clusters single clusters
groupa ((o,) fox
range (8)
Initialize 1ist of steps Lhat characterize hierarchy
steps = []
whole dataset, compate assignments and predicted probabilities soft softmexed_logite softmax (logits, axis--1)
Given the iogits
assignments np. arp.argmxxie togis axis=-l) pred_probe rp. rp.max laaattaaxdd_oogtta.. axiss-1)
for step range (1, Kl:
for for each group (which chosen aggregation function) score_per_gr =
# Compute
group groups:
score_pez__si [groupl sumt (mp.menn (pred_ (assigmments =- c]) for in group]) Set group with the lowest score (lag), will merged this iteration
lag min
key=score_per_grt pe____rgggg))
Set the logits for datapoints assigned to the lowest scoze group
logits_lag logits [ogitt[np.aae where (tt.iiiinassiiiieeeen.nnnnnne Lsin(aasignments, 1sg)) [0]]
Aesssign datapoints in ing to only clusters not in lag,
and ze compute predicted probebilities mam mom_logits_lsg
zep.zers__aia seros_like (1egittt els_not_ir__lg [c
sange (K) if
1sg]
els_in_lag - for
range (K) if e in 189]
mem_logit__isgi; cls_not_in__ss
soEtmax (logitss_ggs cla_not_in__age axie- 11
msm_logit__lsgl;, els_in_lsg]
rezarigmeents = ap.argmax (nsmm_ammmm__oooo inngmaxaim_lggit__llaa axiss-1)
pe_pred_probs ap.max imen_ exis--1)
Compute the tctai reassigned predicted probability per cluster and average across ciusters each group. Then select re_pe_per_grrup =
group the highest avarage.
group: sep.mean (ww.memannn......m [[np. www. prede_eeeed_rrobe (erssssgeeeesssimmeeete -- ell for e in group]) for group in groups if group != lsg
mag max per_g_uppr
Morge "ing" with "mtg" and update "groups".
groups Igr for gr in groupa if ge not in [lag, mbe]] [lag mcg]
Add merging in current itezation to steps steps. (lsg, mtg))
raturn steps
Figure& 4: Python code implementation for the I.H algorithm presented in Section Note that we choose the aggregation function when computing the score per group as described in Appendix B.2.
12

---

From Logits to Hierarchies: Hierarchical Clustering made Simple
B. Experimental Details
B.1. Datasets
In this work, we run experiments on five challenging vision datasets, namely CIFAR-I0 and CIFAR-100 (Lake ct al., 2015), Food-101 (Bossard et al., 2014), ImageNetl (Deng et al., 2009), and as well INaturalist2 (Van Horn al., 2021) introduced in Appendix C. CIFAR-I0 and CIFAR-100 are well-established object classification datasets. The CIFAR-10 dataset consists of 60000 32x32 colored images, divided in 10 classes: airplane, automobile, bind, cat, deer, dog. frog, horse, ship, truck. The train/test splits contain 50000 and 10000 images respectively. Similarly, also the CIFAR-I00 dataset consists of 60000 32x32 colored images. However, they are organized into 100 classes. In addition, the 100 classes are grouped into 20 superclasses. As for CIFAR-10, 10, the train/test splits also contain 50000 and 10000 images respectively. The Food101 dataset is a fine-grained classification dataset of food images, consisting of 101000 images for 101 classes. Images are high-resolution, up to 512 pixels side length. Images are split between 75750 training samples and 25250 test images. The ImageNet- 1k dataset, widely used in computer vision, consists of 1000 classes organized according to the WordNet hierarchy (Miller, 1995), with 1281 167 training and 50000 test samples, respectively. The INaturalist21 dataset (Van Horn et al., 2021) contains 2.7 million images of natural species labelled at different taxonomy levels.
B.2. Implementation Details
For our hierarchical clustering experiments, to train the TURTLE and TEMI models all considered datasets we use the official code provided by the authors with recommended choices for hyperparameters (Gadetsky et al., 2024; Adaloglou et al., 2023). In particular, TEMI employs CLIPViTL/14 representations of the data, while TURTLE employs both CLIPViTL/14 and DINOv2 ViT-g/14 representations. For more details on TURTLE trained using two representation spaces, see the original paper (Gadetsky et al., 2024). We train both TEMI and TURTLE with a number of clusters K equal to the true number of classes in each dataset (unless explicitly stated otherwise, e.g. in the sensitivity analysis reported in Figure 5). For each dataset, we train models on the training set, then report metrics on the test set. Note that the L2H algorithm takes as input logits from the training set infer the hierarchy, while metries that evaluate the quality of the hierarchy are computed on the test set. As the aggregation function A in the L2H algorithm (see Section 3) employ
A g0(x)= D R
90(x))
Dc
BEG
DEDr
cEG
which we find to work well experimentally. However, other choices are possible (see also Table 7). We implement VAE (Manduchi et al., 2023) with their contrastive approach using the provided PyTorch codebase with corresponding defaults. The splitting criterion is set to the number of samples, an inductive bias that benefits this baseline method, since all datasets are balanced (Manduchi et al., 2023; Vandenhirtz et al., 2024). We set the number of clusters to 10 for CIFAR-10 and to 20 for the rest, due to the computational complexity, as seen in Table 2, as well as memory complexity, since every additional leaf adds new decoder. DeepECT (Mautz et al., 2020) is also implemented using their provided codebase with the augmented version. Note that similar to the results shown in Manduchi et al. (2023), for colored datasets, DeepECC fails to grow trees, as they always collapse, indicating that DeepECT fails to find meaningful splits. We implement agglomerative clustering using the scikit-learn library (Pedregosa et al., 201 1), and fit the model using PCA embeddings of the datasets with 50 components and Ward's 's criterion (Ward, 1963) as the linkage method. Using the author's 's original codebase, we further train Hyperbolic Hierarchical Clustering (Liu et al., 2019) on CLIP embeddings of the respective datasets. The authors do not describe how to retrieve cluster assignments using their method, so we follow the agglomerative clustering procedure and assume the leaves of the last K tree nodes created to form cluster, where K corresponds to the chosen number of clusters.
B.3. Metrics
Here we provide more details the metrics reported in our experiments in Section 4.1. In our comparisons, we evaluate models both on flat and hierarchical clustering.
Flat clustering To assess model performance in flat clustering, for each model we take the clustering at the level of the hierarchy where the number of clusters corresponds to the true number of classes K in a given dataset. If the number of leaves at the leaf level of the hicrarchy is smaller than K, as is the case for, e.g., Tree VAE and DeepECT on CIFAR- 100, we consider the clustering at the leaf level. For flat clustering comparisons, we resort to eetl-established metrics, namely
13

---

From Logits to Hierarchies: Hierarchical Clustering made Simple
NMI, ARI, Accuracy, and Purity of the clusters (i.e., Leaf Purity). To compute accuracy and leaf purity, we resort recent implementations in (Gadetsky et al., 2024) and (Manduchi et al., 2023), respectively.
Hierarchical clustering To assess the quality of a learned hierarchy, and compare the results of different models in hierarchical clustering, we resort to two metrics. Dendrogram Purity (DP), introduced in (Kobren et al., 2017), extends the notion of leaf purity to evaluate the purity of hierarchical clusters, and was recently adopted to benchmark hierarchical clustering models (Manduchi et al., 2023). Following the notation of (Kobren et al., 2017), let C* denote the true K-clustering (i.e., true class labeling) of dataset D. Then define
{ (x,, 2y E D, # xz (z;z
Pa =
as the set of pairs of data points that belong to the same true cluster. Dendrogram Purity (DP) is then defined for a hierchical clustering H as
DP(H) = P
Σ
(M((vs((LCA((x;,::)))....
(a:aa))ec
where LCA( x22 computes the least common ancestor node of data points 1 and 22 in H. Ivs(z) returns the set of leaves of the sub-tree rooted at any internal node 2, pur(S1,Sh) = [S n S11/ and C is the set of data points belonging to the true cluster k. One possible caveat of this metric its high correlation with Leaf Purity: with high leaf purity, most pairs of samples sharing the true label will inevitably fall into the same leaf. To address this, we introduce an additional metric for evaluation, namely Least Hierarchical Distance. With a similar notation as above define
ne = f (1,, E D, (xre)) = (x²z) A Mar H) + [xxy; (xx;; 70)
where the function 1(2; H) returns the cluster prediction for datapoint x at the leaf level of H. Hence P is the set of all pairs of points sharing the same true label that are not assigned to the same leaf in H. Least Hierarchical Distance is then defined for a hierarchical clustering H as
logz(td(1(z; 70), (2,; 70))
LHD(H)
P
loga -
(ELP)))E
where td(11,12) 12) computes the number of edges in the shortest path that connects two leaves (1 l2 in the tree defined by H. Different from Dendrogram Purity, Least Hierarchical Distance only takes into consideration pairs of data points with the same true label that do not fall into the same leaf. Hence, it does not exhibit strong correlation with Leaf Purity, being more specific to the quality of the hierarchy rather than influenced by the clustering at the leaf level.
C. Additional Results and Visualizations
Table 3 presents the results of our I.2H algorithm implemented top of the TURTLE backbone model for hierarchical clustering on the ImageNetl K dataset (Deng et al., 2009). These results complement the ones shown in Table 1, demonstrating that our method achieves outstanding performance for hierarchical clustering, even on datasets with large scale and numerous classes. It is worth noting that a direct comparison with alternative approaches (e.g.. DeepECT, TreeVAE) is not feasible in this setting, as these methods lack the scalability to handle datasets of this magnitude and complexity. The quantitative results underscore the effectiveness of our method, successfully uncovering meaningful hierarchical structure a dataset of this size and complexity.
NMI (t)
ARI (1)
ACC (t)
LP (t)
DP (t)
LHD (L)
L2H-TURTLE
0.882
0.621
0.726
0.744
0.560
0.210
Table 3: Hierarchical clustering performance of our L.H method applied on top of the TURTLE pre-trained model on the ImageNetIK dataset.
In Table 4, we present results obtained by applying our proposed approach using an alternative backbone model to TEMI and TURTLE, namely the TCL flat clustering model (Yunfan et al., 2022). Unlike TEMI and TURTLE, TCL. does not
14

---

From Logits to Hierarchies: Hierarchical Clustering made Simple
rely on pre-trained foundation model representations. This ablation highlights the generality of our method, demonstrating that it can be effectively applied with diverse choices of the backbone model. Notably, our approach maintains good hierarchical clustering performance even when the underlying backbone exhibits weaker flat clustering capabilities than TEMI or TURTLE, still outperforming deep specialized hierarchical approaches (see Table 1).
Flat
Hierarchical
NMI (t)
ARI (t)
ACC (f)
LP (t)
DP (1)
LHD (L)
CIFAR-10 L2H--TCL
0.785
0.744
0.868
0.877
0.733
0.398
CIFAR-100
L2H-TCL
0.547
0.215
0.343
0.437
0.218
0.351
Food-101 L2H-TCL|
0.455
0.168
0.279
0.348
0117
0.396
Table 4: Results obtained applying our L2H method with the TCL model used as backbone on CIFAR-10, CIFAR-100, Food-101 datasets.
In this work, we apply our 1.2H algorithm on top of flat models (TURTLE and TEMI) that both leverage CLIP embeddings to cluster the data. Hence, for further comparisons in Table 5 we report the results obtained with agglomerative clustering with Ward's linkage performed on pre-trained CLIP embeddings from CIFAR-10, CIFAR-100, and Food-101 datasets. While leveraging pre-trained CLIP embeddings improves the performance of agglomerative clustering, a direct comparison with the performance of 1.2H-TEMI (see Table 1), which is also solely based on CLIP embeddings, shows that our method still markedly outperforms this baseline.
Flat
Hicrarchical
NMI (t)
ART (t)
ACC (t)
LP (t)
DP (t)
LHD (1)
CIFAR-10 Agg. (CLIP embeddings)
0.799
0.724
0.805
0.826
0.716
0.442
CIFAR-100 Agg. (CLIP embeddings)
0.690
0.386
0.531
0.637
0.341
0.343
Food-101 Agg. (CLIP embeddings)
0.868
0.730
0.837
0.872
0.703
0.299
Table 5: Quantitative results for flat and hierarchical clustering performance of agglomerative clustering performed on CLIP embeddings.
To provide further comparisons that complement the results in Section 4.1, Table 6 presents results on the CIFAR-100 dataset comparing our proposed approach alongside two additional methods. First, we evaluate Tree VAE (Manduchi et al., 2023) trained on CLIP embeddings of CIFAR- 1000, demonstrating that our method achieves substantially better performance. We include this comparison as both TEMI and TURTLE have access to CLIP embeddings for training. Hence this comparison shows that, even in setup where deep specialized hierarchical approaches share access to foundation model embeddings, our proposed approach achieves significantly better performance with much higher efficiency. In particular, training Tree VAE in this setup requires hours GPU, while training L2H- -URTLE requires under two minutes. Furthermore, we include comparison with the recent method introduced by Lowe et al. (2024), which combines UMAP dimensionality reduction with Ward's agglomerative clustering for zero-shot clustering with pre-trained embeddings. Since this approach relies on agglomerative clustering, it presents the drawback of not allowing inference on a separate test set. Hence, we both train and evaluate the this method on the test set, which results an unfair advantage. Despite this, our method still significantly outperforms this additional baseline.
NMI (t)
ARI (1)
ACC (1)
LP (t)
DP (t)
LHD (L)
Tree VAE+CLIP
0.665
0.285
0.255
0.255
0.181
0.285
Lowe et al (2024)
0.753
0.499
0.616
0.664
0.445
0.303
L2H-TEMI
0.778
0.565
0.682
0.701
0.502
0.298
T.2H-TRTLEE 0.917
0.831
0.896
0.897
0.803
0.235
Table Performance comparison on the CIFAR-100 dataset of our L.H approach, using TURTLE and TEMI as backbone models, with the method from Lowe (2024) and Tree VAE trained on CL.IP embeddings. Hierarchical and flat clustering metrics are reported.
15

---

From Logits to Hierarchies: Hierarchical Clustering made Simple
In Table 7, we provide an ablation that reports the results of L2H-TURTLE on the hierarchical clustering experiments from Section 4.1 with different choices for the aggregation function A in the I.2H algorithm. The results indicate that tweaks in the aggregation function alter performance, though without abrupt changes in the metrics. These results also motivate our designated choice of aggregation function-ccoresponeie corresponding to the last row-which works well experimentally.
A
CIFAR-10
CIFAR-100
Food-101
DP (1) LHD (4) DP (1) LHD (L) DP (1) LHD (4)
ECEc EmED: go(xr)
0.988
0.258
0.801
0.244
0.758
0.294
[G| EEEG [ DO (0(xr)
0.988
0.248
0.793
0.283
0.751
0.335
L2H-TURTLE
EnEG [20 Do go (x))
0.988
0.277
0.803
0.235
0.758
0.297
Table 7: Results for hierarchical clustering, in terms of Dendrogram Purity and Least Hierarchical Distance, implementing the L2H algorithm with different choices for the aggregation function A, on top of the TURTLE model.
In Figure 5 we provide ablation to test the sensitivity of our approach to the value of the hyperparameter K, cor- responding to the number of leaves in the hierarchy and the number of clusters set for the pre-trained flat model. In particular, we report the results for L2H-TURTLE on the CIFAR-100 dataset varying the value of K across the range {85, 90, 95, 100, 105, 110, 115). Note that this range is symmetric around the true number of classes/clusters equal to 100. Therefore, we both explore the case of over- and under-estimating the true number of clusters at the leaf level of the hierarchy. We report the results for both flat (NMI, ARI, ACC, LP) and hierarchical (DP, LHD) metries. Best performance across all metrics is achieved when K is set to the true number of clusters, while the performance gracefully degrades when K is set to be an over- or estimaedd value. This demonstrates the robustness and stability of our approach with respect to this hyperparameter, which is particularly important in practical settings where the exact true number of classes is not known a priori. Finally, we find the log-normalized TURTLE model loss to be indicative of the true value of K, with the minimum value achieved when K equals the true number of clusters. In practical settings, one can consider using this metric to select the value of K when a value/proxy for the true number of clusters is not available.
Next, we perform an experiment to test whether, with datasets that are inherently hierarchical, implementing our approach can yield an advantage over flat clustering with the backbone model. To do so, we consider the INaturalist21 dataset (Van Horn et al., 2021), which contains 2.7 million images of natural species labelled at different taxonomy levels (1103 families, 273 orders, 51 classes, 13 phylums, 3 kingdoms), We train TURTLE to model clusters at the more fine-grained family taxonomy level (K family = 1 1103). Then we implement our L2H procedure on top, and use the produced hierarchy to make clustering predictions at coarser taxonomy levels. For instance, Korder - steps from the end of the procedure (see Algorithm 1), we have a fooder--custerring of the data points, and we test its performance on the test set (at the order taxonomy level). Then, we train instances of TURTLE at each coarse taxonomy level, i.e., K E [273, 51, 13,3). and report the corresponding test set performance for comparison. Notably, training TURTLE at the fine-grained level, and using our 1.H approach to construct hierarchy and make predictions more coarse levels, achieves better clustering performance than training separate TURTLE models at each taxonomy level. Note that this performance improvement also comes with much lower compute time, as only a single instance of the TURTLE model is trained (at the finest-graine taxonomy level). We report the results in Table 8.
Model
Family
Order
Class
Phylum
Kingdom
nmi
1p
nmi
acc
Ip
nini
nmi
rmi
ari
Ip
TURILE 0.552 0.052 0.140 0.373 (05122 0,075 0.172 0.562 0.498
0.101
0.203 0.827
0.514 0.244 0.268 0.893
U.461 0.663 0.873
L2H-TURTLE
0.582 0,052 0.140 0.373 00.57 0.097
0.181 0.572
0.497 0.123
0.229
0.797 0.515 0.294 0.385 0,877 0.561 0.562 0.734 0.920
Table 8: Comparison on the INaturalis121 dataset between the clustering performance of our I.2H approach at coarse levels of the hierarchy, and the results obtained by training an instance of the backbone model at each coarse level. Results are averaged across five independent runs, bolding the best results and results that are statistically indistinguishable from the best.
16

---

From Logits to Hierarchies: Hierarchical Clustering made Simple
ARICTI
LP113
0.92
MM
0.325
0.900
ACC
0.91
0.875
0.88
0.86
4 0.775
0.825
0.750
0.82
0.800
0.725
0.80
0.775
100
115
100
105
110
1000
115
100
105
(P(11)
UMD
Log Norn (os
0.80
DP
UND
0.94
Log Norm Loss
0.26
Log Nog-oomm Loss
0.75
0.43
8
0.70
0.02
0.65
0.91
90
130
95 100
05
90
105 110
K
Figure 5: Sensitivity analysis for L2H-TURTLE on the CIFAR- 100 dataset with respect to the K hyperparameter, which corresponds to the number of leaves in the hierarchy and the number of clusters set for training the pre-trained flat model (TURTLE in this case). Note that the true number of clusters is equal to 100. Results for both flat (NMI, ARI, ACC, LP) and hierarchical (DP, LHD) metrics are included, with standard deviations across five independent runs reported as shaded areas around the line indicating mean values. We also include the tog-normalized TURTLE model loss reported in the rightmost plot in the bottom row-that proves to be indicative for model selection with respect to the K hyperparameter.
Finally, we provide ablation where we test slightly different merging strategy for our L2H algorithm. In par- ticular, instead of selecting the lowest-scoring group G E argmingeg=(G) and merging it with the most related group Gl E (egraxceg((-1)) |ि rp(e) (see details in Section 3), we directly consider all possible pairs of groups for merging, and merge the pair G*,G that results the highest value for T Eeerr rp(c), where rp(c) G*) (as also defined in Equation (2)). We report the results with this alternative merging strategy in Table 9, only reporting hierarchical clustering metrics, the merging strategy does not impact the flat level clustering. Results show that considering all pairs of groups for merging at each step, which also results in computational overhead, does not bring benefits in performance.
EnEDAE "_ (==
DP (t)
LHD (4)
1.2H-TURTLE (alt)
0.797
0.246
L2H-TURTLE
0.803
0.235
Table 9: Performance comparison the 100 dataset of our I.2H approach, using TURTLE as backbone model, comparing the merging strategy described in Algorithm with an alternative merging strategy where at each step all pairs of groups of clusters are considered for merging.
In Figures 6, 7, 8, we provide additional visualizations for the hierarchies obtained with our proposed method in our hierarchical clustering experiments, complementing the quantitative and qualitative evidence shown Section 4.1. Note that, to produce these visualizations, as well as the other visualizations of hierarchies in this paper we used the iTOL tool (Letunic & Bork, 2024). Leaves matched the original labels by checking the most frequent label among data points contained in the leaf. In addition to the matched label, we report the purity of each leaf in percentage.
In Figure 9, we provide additional results for the ImageNet case study (Section 4.2) with different colorings for the inferred hierarchy, supplementing our results from Section 4.2. These visualizations show where the subtree of birds (used in Figure 3b) is located within the complete tree and relation to other superclasses, such as mammals, reptiles, dogs, and clothing.
17

---

From Logits to Hierarchies: Hierarchical Clustering made Simple
ship 99.54
airplane 99.32
automobile 99.26
truck 99.46
frog 99.82
deer 99.58
bird 99.64
horse 99.80
dog 99.36
Figure 6: Visualization of the hierarchical clustering produced by L2H-TURTLE for the CIFAR-I0 dataset.
98 36 sudded pownes
Tee autupaape
95 pusoolee
BV sptte
59'00 ned
clock 97.13
tamp 04.00
06 71
bood 17
WMli e5. 77
98:77
lown mooeed 07 og
Hotoraysle 98.02
Gan 59.82
16 ZG Weren
bioycle 81
plots muck 99
BL 09 mip
Tracor 96 78
17.90
bas 79
afrestur U.4
train 95.09
mocest 84
weman 44.55
mountain 81
60 08
584 82.15
otter 32
84,00
26
clout 78.43
aeal 52
plysscaaper 96.52
00
Inidge 91.97
71
castle
porcupine 86.49
forest
52
pine 79.32
possum 83.
villow 76.67
iaccoon
oek tree 99
squireel 94.21
meplo d6
reliott 95.60
hemster 67 60
Sundossw 97.43
sree 94t
kangwoo 98.35
poppy 25
crocoille 07
oxchid 96 77
fiard 47. 53
tulp 74
dinossus 97
ro00 96.90
worm 32
unae 94 50
anai 9t 08
mosroon 07.04
bodish 94
sour 03.55
aoeee 90.81
SONN 03 10.50
lode 80.8
souuettm Ban 0A.23
shaak 86.25
whole 64
delphinn 88.71.
Now N8 98.21
1e100 1988 m000
ineeeee se
oot suuuedur
lgeer 88
wollss 6.0.0
86
lepprd
Figure Visualization of the hierarchical clustering produced by L2H-TURTLE for the CIFAR-100 dataset.
18

---

From Logits to Hierarchies: Hierarchical Clustering made Simple
Maccann cheose 96 D1
Flemm soup 30
90 Ge Auro uneeta
Lazegyu 18.56
Sinaeroo Jantereed didea St at
Escarposs
Chanologg 32 76
Ranee 30 n2
Rrevio 91.92
Bakllsa 82
Bitindago 88-85
Pho 20 07
Spanghon bulegrnn ae.u
Pas thet 98
Samma 60.53
Gyose
Spiing ratn
Belgrast 37 .1
Remato $4.00
Swimp and pre $4.00
Chumos
Cameal 73
Paeto 84
Mecarors $7.30
Frind rce 06.56
Cup calers 92
Mossel 9159
Dersta 18.16
Panni cutta
Cheessa piule 80
Strawbbrrr shartaak $1.81
Guscomabe
Crome crubte 94.38
Hannnus
Applle
tatame 68 12
Bread pushing
Coviine
Canrtt cake 76.10
Susii 91
Red vilvel cake
Sashimi 62
Choostate mousse 46.01
carpacuio R0L41
Titentzu
Bedt sakul 21.28-
Chocolate mousse 30
Greek Salad 98.00--
Frozenn yogun 21.80
Ciuresee salnd 96,
fow cream B8
Caecar salad 69
Lobator r0ll serchwcch 88
Deuted eggs 92 4g
dog 95.57
Hamburger 92 Si
Hueves netcerrs
Drealfest humto 91 5g
Pullted pork sontwich P1
Hacted 82 58
Fish and crips 10
Poutine 53
Colcken quessailla 351
Roveh tven 30
Faletel 51
eakes A8
Chinken wings 90:23
Onion digs 04.2
Takoytli 3M
Filet cateman 00.62
Grlted chenter sandwish 20.22
Tano 02
Optera R0.70
sandleish 50
Conteecaa 91.5
Crosue madiome 91
Eggas bensrtee e8s 24
Bronchhttee
Pizdo 90.76
Weltes 08 80
Golled saloin 92229
Prine ob 25
Bah beck die 9243
Pasinn 03.11
Solltgg 80.33
gis 1478
Ponssies 86
Freecc rookt
Gante
Figure 8: Visualization of the hierarchical clustering produced by L2H-TURTLE for the Food-101 101 dataset.
19

---

From Logits to Hierarchies: Hierarchical Clustering made Simple
(a) Colored by bird, dog, and clothing
(b) Colored by bird, mammal, and reptile
Figure 9: Visualization of the hierarchy produced by our method in the experiment on ImageNet from Section 4.2. We show the complete tree of 1 classes colored by the corresponding WordNet hypernyms "bird", "dog". and "clothing" (Figure 9a) and by "bird", "mammal". and "reptile" (Figure 9b).
20


Reviews:
[{"review_id": "ao8e9lSvvg", "reviewer": "Reviewer_CKwm", "summary": "This paper addresses the issues of traditional hierarchical clustering algorithms, such as high computational costs, and strong model dependency. It proposes a lightweight hierarchical clustering algorithm that directly constructs hierarchical structures using the logits output by pre-trained models, enabling multi-granularity clustering without fine-tuning. Furthermore, extensive experiments demonstrated the effectiveness and generalizability of the proposed method.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the Reviewer for the valuable feedback and suggestions. We appreciate the praise for the efficiency and efficacy of our method, and the highlighting of its value in addressing well-known challenges in hierarchical clustering. We address concerns/questions below.\n> However, increasing the number of datasets and comparison methods [..] \n\nSee comparison with [1] below +  other Rebuttals (e.g. L2H+TCL and TreeVAE+CLIP comparisons, INaturalist results).\n> Given the related work on [...]\n\nWe assume the Reviewer refers to this ICML 2024 workshop paper[1], on zero-shot flat clustering. The proposed strategy (UMAP dim.red. on embeddings + Ward's agglomerative clustering) incurs in high computational cost for large datasets, and more importantly *does not allow for inference on unseen data*. Hence, to test the method from Lowe et al. for comparison we necessarily both train and evaluate it on the test set, thereby giving it an advantage in seeing test data for training. Despite this, this method still underperforms compared to our proposed approach, which further validates and contextualises the effectiveness of our method. Results on CIFAR-100 at https://files.catbox.moe/4nhqck.pdf\n> Theoretically, the design basis of the masked Softmax [..]\n\nWe introduce the masked softmax as a principled variation of the softmax function that allows masking in the indexes. Note that our function definition ensures that a valid probability distribution over the unmasked elements is produced. We do not consider a downstream task in our hierarchical clustering experiments. We would thus appreciate it if the Reviewer could provide clarifications regarding the comment \"the relationship between the hierarchical clustering objective function and the downstream tasks is not clearly established\".\n> In the evaluation metrics, only traditional clustering metrics were used[..]\n\nNote that we use Dendrogram Purity (DP)[2,3] and Least Hierarchical Distance (LHD) - both metrics tailored for hierarchical clustering.\n> No sensitivity analysis of the parameters was conducted, such as the depth of the tree. \n\nWe provide a sensitivity analysis of L2H-TURTLE on the CIFAR-100 dataset, with respect to the depth of the tree. The depth of the tree is controlled by the $K$ hyperparameter corresponding to the number of clusters modelled by the flat model and therefore to the number of leafs in the tree. The analysis demonstrates that across all hierarchical and flat metrics the best performance is achieved when K equals the true number of classes (100). When $K$ deviates from the true number of clusteris, performance degrades gracefully and a meaningful hierarchy is still recovered. This robustness is particularly important in practical setting where the true number of clusters is not known a-priori; in such cases the log-normalized TURTLE model loss can provide useful signal to select the hyperparameter $K$. Results at https://files.catbox.moe/iisaqn.pdf.\n> The summary of contributions and the future work section need further refinement.\n\nWe appreciate the suggestion from the Reviewer, and have refined the manuscript accordingly, summarizing and highlighting more clearly the contributions of our work. As well, we have deepened the section on future work, also in light of some additional results shown in this rebuttal.\n> The majority of the references are from [...] \n\nWe share the Reviewer’s view on the importance of citing recent literature, and have made our best effort to do so throughout the paper. However, as noted in the recent relevant work[2], the number of deep learning-based approaches proposed in the last few years to address hierarchical clustering remains surprisingly limited. This underlines the relevance and timing of our work, where we aim to revive the interest in this underexplored area, building on recent advancements (e.g. in flat clustering models). To that end, we compare with the most recent and relevant baselines[2] and build upon SOTA flat clustering models. Finally note that we have followed the recommendation from the Reviewer contextualizing our approach in comparison with [1] above.\n\n**Replies to \"Questions for the Authors\"**\n\n1.We introduce CIFAR-10, CIFAR-100, Food-101 that we used in section 4.1 and ImageNet1k that we used in Section 4.2 and Table 3.\n\n2.In line 753 we highlight that comparisons with baselines (e.g., DeepECT, TreeVAE) are not feasible on ImageNet1k, as these methods lack the scalability to handle datasets of this magnitude/complexity.\n\n3.We reported Table 5 in the Appendix as it consists of an ablation validating the stability wrt the design choice of the aggregation function. We appreciate the Reviewer's suggestion and have refined the Appendix by removing redundant results.\n\n[1] Lowe et al.(2024)\n[2] Manduchi et al.(2023)\n[3] Kobren et al al.(2017)\n\nWe are happy to answer any additional questions, and would appreciate it if the Reviewer would consider increasing their score to an acceptance.", "claims_and_evidence": "The author conducted several experiments to demonstrate the effectiveness of the proposed method. However, increasing the number of datasets and comparison methods would enhance the credibility of the results.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "The inspection has been carried out, and no issues were found.", "experimental_designs_or_analyses": "The inspection has been carried out, and no issues were found.", "relation_to_broader_scientific_literature": "The paper's key contributions address well-known challenges in hierarchical clustering, such as computational cost and model dependency. They build upon prior work by leveraging pre-trained models without fine-tuning, thereby advancing the state-of-the-art inefficient and semantic-aware clustering methods.", "essential_references_not_discussed": "Given the related work on hierarchical clustering presented at ICML 2024, it is recommended that a performance comparison analysis with the proposed method be conducted. This would provide a more comprehensive evaluation and highlight the new approach's advantages and limitations in the context of recent advancements.", "other_strengths_and_weaknesses": "Strengths:\n1. The proposed method can process ImageNet-scale data in just a few minutes on a CPU, significantly outperforming deep methods such as TreeVAE.\n2. The author conducted experiments to demonstrate the generalizability of the proposed method.\n\nWeaknesses:\n1. Theoretically, the design basis of the masked Softmax lacks rigorous mathematical derivation, and the relationship between the hierarchical clustering objective function and the downstream tasks is not clearly established.\n2. In the evaluation metrics, only traditional clustering metrics were used, and there is a lack of metrics specifically tailored for hierarchical clustering.\n3. No sensitivity analysis of the parameters was conducted, such as the depth of the tree.\n4. The summary of contributions and the future work section need further refinement.\n5. The majority of the references are from five years ago, and there is a need to incorporate more recent and advanced studies. This is important because citing newer literature ensures that the research is aligned with current trends and advancements in the field.", "other_comments_or_suggestions": "Considering the aforementioned Weaknesses.", "questions_for_authors": "1. Does the appendix introduce four datasets, but only three datasets were actually used in the experiments?\n2. Table 3 does not provide comparisons with other methods, making it difficult to determine the effectiveness of the proposed approach.\n3. Should Table 5 be moved to the main text? The appendix should be revised to remove any redundant information.\n4. The remaining issues can be referred to the section on Weaknesses.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BLRuc2WcZ5", "reviewer": "Reviewer_YXaT", "summary": "- Main algorithmic/conceptual ideas: Using logits of flat clustering model, get hierarchical structure on top of them.\n- Main findings/results: deep hierarchical clustering methods have low clustering performance and slow runtime; learned hierarchies from image datasets are interpretable.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the Reviewer for praising performance and efficiency of our method, our metrics/datasets, as well as the clarity of our paper. We appreciate useful the feedback and suggestions, and address the concerns below.\n> line 131[..]\n\nThe number of leafs in the hierarchy obtained with our approach is the number of clusters $K$ modelled by the pre-trained flat model, which is a hyperparameter. Note that we show our approach is robust to changes in $K$, and should an a-priori value for $K$ not be available, the loss of the flat model (e.g. TURTLE) can be useful signal to set it (See rebuttal to Reviewer CKwm).\n> line 199 [..]\n\nThe score we define captures the aggregated probability mass that the model assigns to each cluster/group. When a cluster/group receives the lowest score, depending on the exact aggregation, it indicates one of two scenarios (or both):\n\n- It contains fewer samples, suggesting it is more specialized than others.\n- The model is less confident when assigning samples to it, often because it's highly similar to another cluster/group.\n\nIn either case, the cluster/group is a natural candidate for early merging in a bottom-up hierarchy from fine-grained to coarse. In the first case, it is more specialized than others. The second case is a little more subtle. Consider the case of overclustering, where the number of leaf nodes K is larger than the true number of clusters. In this case, redundant clusters modelling the same true class present low-confidence assignments, as the model can't clearly distinguish between them when assigning samples. This low confidence reduces their score, causing them to be merged early, effectively correcting the overclustering. Thus, our score guides the hierarchy-building process in a principled way, encouraging early merging of either highly specific or redundant clusters, ultimately leading to an accurate hierarchy, as validated empirically.\n> line 144: [..]\n\nWe agree that this assumption underlies our method. While empirically supported, we acknowledge that this assumption may not always hold, for instance in case the flat model is poorly calibrated. We have updated the manuscript to explicitly discuss this assumption and its potential limitations.\n> Is it possible to [..]\n\nAs we state in lines 234 - 236 (right column), by construction, our method retains the clustering performance of the pre-trained model at the leaf level, hence non-hierarchical clustering metrics for TEMI/TURTLE match the L2H-TEMI/L2H-TURTLE results in Table 1. However, our results on the INaturalist21 dataset (see rebuttal to Rev. CaXW) prove that our approach can be used to model multiple granularity levels in datasets with a hierarchical structure. Notably, our approach surpasses the performance obtained by training multiple instances of a flat model (e.g. TURTLE), one at each given granularity. In such a setting, our hierarchical approach surpasses the flat model (e.g. TURTLE) on flat metrics at all but the finest granularity (where the performance matches).\n> Out of curiosity [..]\n\nIt is not necessarily true in principle that all flat clustering methods are faster than hierarchical methods. However, there's been a lot recent research on highly performant and efficient flat clustering (e.g TURTLE), while a comparable research effort has not been witnessed for hierarchical models.\n> Going back to the choice [..]\n\nSee rebuttal to Rev. CaXW.\n> Additionally [..]\n\nNote that in section 4.2 we use InternImage as a backbone mode - not based on CLIP embeddings.\n> But overall [...] \n\nWe appreciate the suggestion from the Reviewer, and implement our method on top of the TCL flat clustering model [1] on the datasets used in Table 1. TCL (i) does not rely on pre-trained embeddings, and (ii) achieves weaker flat results than TEMI/TURTLE. Results at https://files.catbox.moe/cfufvm.pdf. Still, with our approach (L2H-TCL) we outperform deep hierarchical models (DeepECT,TreeVAE) across all flat/hierarhical metrics, which strenghtens our contribution.      \n[1] Li et al. Twin Contrastive Learning for Online Clustering, IJCAI, 2021.\n> Can you explain Figure 1[..]\n\nIn the second step in Fig.1 the pink cluster, selected for merging, is merged with the group containing yellow and blue clusters, as this group has the most reassigned predicted probability mass. The bar contains blue/yellow parts to represent that it aggregates the probability mass reassigned to blue/yellow clusters, that were grouped together at the previous step.\n> Minor: for the benefits of modeling [..]\n\nWe agree with the Reviewer that the benefits of modelling a hierarchy are not restricted to the unsupervised setup, and show results to support this point in section 4.2. We appreciate the useful references and have integrated them in the updated manuscript.\n\nWe are happy to answer any additional questions, and would appreciate it if the Reviewer would consider increasing their score to an acceptance.", "claims_and_evidence": "Yes in general. The whole paper is overall clear.", "methods_and_evaluation_criteria": "Questions for methods:\n- line 131: how do you decide the number of clusters $K$? Is this decided by your input flat clustering method?\n- line 199: why do you choose the lowest score (predicted probabilities) for merging, say when the aggregation function is summation, or others?\n- line 144: Seems like your method is based on the assumption that cluster logits (or probabilities) are a proxy to measure cluster similarities. Why is this assumption reasonable? When can this assumption fail? I encourage the authors to briefly discuss on this point.\n\nMetrics and datasets look good to me, both are reasonable for clustering evaluation.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "Questions: \n- Is it possible to show non-hierarchical clustering metrics for TEMI and TURTLE? My guess is your method will have an absolutely better performance with a stronger flat clustering initialization, especially when I noticed that TEMI and TURTLE are based on CLIP embeddings which is quite powerful (line 765-768). Therefore, it's better to show this increment with your method as an important ablation study. Your results will be stronger if you show there is a significant increase with your method on top of any flat clustering methods.\n- Out of curiosity, why are flat deep clustering methods faster? Is this true for all (maybe SOTA) deep clustering methods?\n- Going back to the choice of flat clustering method, if you use CLIP based embeddings, in Table 2, for the runtime to train the TURTLE model, I would imagine you didn't include the time to train CLIP, which sounds unfair for other methods.\n- Additionally, there's another concern about interpretability arguments in your visualization, because basically CLIP has already seen text embeddings so it could infer some knowledge directly from text, thus probably having an additional advantage to match with wordnet.\n- But overall I don't think this will be a significant issue if the author can either provide reasonable explanation on the choice of clustering method, or showing the result of your method + other weaker flat clusteirng method as initialization. I understand that the final metrics probably won't be as good as what you reported now, but that is still valuable to the community and will make your paper much more convincing.", "relation_to_broader_scientific_literature": "This can be categorized into a bottom-up approach of hierarchical clustering, and is built on trained non-hierarchical clustering methods.", "essential_references_not_discussed": "Minor: for the benefits of modeling a hierarchy in the data are not restricted to the unsupervised setup, there are some works to use hierarchical data for out-of-distribution detection and the hierarchy is used for supervised learning.\n\n[1] Khrulkov, Valentin, et al. \"Hyperbolic image embeddings.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020.\n\n[2] Linderman, Randolph, et al. \"Fine-grain inference on out-of-distribution data with hierarchical classification.\" Conference on Lifelong Learning Agents. PMLR, 2023.\n\n[3] Sinha, Aditya, et al. \"Learning Structured Representations with Hyperbolic Embeddings.\" Advances in Neural Information Processing Systems 37 (2024): 91220-91259.", "other_strengths_and_weaknesses": "- Originality: Hard to judge based on my familiarity of the literature.\n- Significance: The proposed method is lightweight and empirically works well. Although see \"Experimental Designs Or Analyses\" for concerns of the experiment section that may mislead the interpretation of the results. \n- Clarity: overall well-written paper.", "other_comments_or_suggestions": "Can you explain Figure 1 iteration 2? This is not contained in the caption and why is the pink box merged with the first two boxes? Also what does the top bar for iteration 2 mean where it contains both blue and yellow parts?", "questions_for_authors": "For detailed questions see comments above. Here let me summarize major questions:\n- selection of initial clustering methods in your experiments\n- why choose the lowest score for merging\n\nI would like to raise my score if the author can elaborate on these questions.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DxN0Yg8VYD", "reviewer": "Reviewer_pCAf", "summary": "The paper introduces Logits to Hierarchies (L2H), a novel hierarchical clustering method that uses pre-trained non-hierarchical clustering models to build hierarchical structures. L2H is lightweight, doesn't require fine-tuning, and uses logits to generate clusters, outperforming existing deep hierarchical clustering methods in both performance and efficiency. It is also extended to supervised settings, showing its ability to recover meaningful hierarchies from pre-trained classifiers like ImageNet. Empirical results on CIFAR-10, CIFAR-100, Food-101, and ImageNet demonstrate L2H's potential for interpretability and bias detection in supervised models.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "The introduction of the proposed method in the paper is clear and well-structured. The authors provide a detailed explanation of their Logits to Hierarchies (L2H) approach, including its motivation, algorithmic procedure, and mathematical formulation. Additionally, they present pseudocode and visual illustrations that facilitate understanding\n\nHowever, the experimental results reveal a significant performance gap between L2H and the deep hierarchical clustering methods (e.g., DeepECT, TreeVAE). The credibility of the experimental results requires further scrutiny. Specifically, the validity and fairness of the baseline implementations, as well as the generalizability of L2H across different datasets and real-world applications, should be carefully examined.", "methods_and_evaluation_criteria": "The proposed method, L2H, is well-suited for the problem of hierarchical clustering. It builds on pre-trained flat clustering models and uses logits to construct hierarchies, which is a novel approach. The evaluation criteria are appropriate, with metrics that assess both the quality of the flat clustering (e.g., NMI, ARI, Accuracy) and the hierarchical structure (e.g., Dendrogram Purity, Least Hierarchical Distance). The authors also provide a case study on ImageNet, demonstrating the method's applicability to supervised settings, which adds to the method's generality and practical relevance.\n\nHowever, the authors claim that L2H is scalable to large-scale datasets and computationally efficient. The datasets used (CIFAR-10, CIFAR-100, Food-101) may not be sufficient to fully validate its scalability in real-world scenarios. \n\nAdditionally, the comparison with baselines such as DeepECT and TreeVAE requires clarification. Given the large performance gap, it is unclear whether these baselines were optimally configured. More details on hyperparameters, training settings, and computational budgets would ensure a fair comparison.", "theoretical_claims": "The paper does not present any theoretical proofs or claims.", "experimental_designs_or_analyses": "The experimental design primarily focuses on evaluating L2H against deep hierarchical clustering models (DeepECT, TreeVAE) on CIFAR-10, CIFAR-100, and Food-101, with additional results on ImageNet-1K. While the methodology appears reasonable, there are concerns regarding the validity of comparisons:\n\n1.\tThe reported performance gap between L2H and baseline methods is significant, yet the paper lacks details on whether baselines were optimally configured. Ensuring fair hyperparameter tuning and implementation details is essential for a valid comparison.\n\n2.\tWhile the paper claims scalability, the chosen datasets may not fully reflect real-world large-scale clustering challenges. Further analysis on more complex, high-dimensional datasets would strengthen the validation.\n\n3.\tThe authors highlight L2H’s efficiency, but more transparency on runtime conditions (e.g., hardware specifications, dataset size variations) is needed to verify the claimed advantages.", "relation_to_broader_scientific_literature": "The paper is well-situated within the broader literature on hierarchical clustering. The authors discuss the limitations of existing deep hierarchical clustering methods and position their work as a more scalable and efficient alternative. They also connect their work to the growing interest in interpretability and bias detection in machine learning models, particularly in supervised settings. The paper builds on recent advances in pre-trained models and logit-based approaches.", "essential_references_not_discussed": "The paper covers the relevant literature, and no essential references appear to be missing.", "other_strengths_and_weaknesses": "**Strengths:**\n\nThe proposed method is scalable and efficient, making it suitable for large-scale datasets.\n\nThe paper provides some empirical evidence, including visualizations and ablation studies, to support the claims.\n\nThe extension to supervised settings adds practical relevance, particularly for interpretability and bias detection.\n\n**Weaknesses:**\n\nThe method relies on pre-trained models, which may limit its applicability in scenarios where such models are not available.\n\nWhile the method is efficient, the quality of the hierarchy depends on the quality of the pre-trained model's logits, which could be a limitation if the pre-trained model performs poorly.\n\nIn this paper, the validity of the method is verified only by experiment, and the theory is lacking.\n\nThe paper claims the method is lightweight and scalable but lacks extensive experiments on extremely large datasets (e.g., millions of samples or hundreds of thousands of classes). Experiments are limited to smaller datasets like CIFAR and Food-101, leaving scalability claims unverified for the so-called large-scale data.", "other_comments_or_suggestions": "I have no other comment.", "questions_for_authors": "1.\tThe method relies on pre-trained models. Have the authors explored scenarios where pre-trained models are not available, or where the pre-trained model's performance is suboptimal? How does L2H perform in such cases?\n\n2.\tThis paper only uses experiments to verify the effectiveness of the method. Is there any relevant theoretical guarantee to illustrate the effectiveness of the method?\n\n3.\tThe TEMI employs CLIPViTL/14 representations of the data, while TURTLE employs both CLIPViTL/14 and DINOv2 ViT-g/14 representations. Are the compared methods utilize these representations?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1WwO5ZTqwz", "reviewer": "Reviewer_CaXW", "summary": "A method for deriving a hierarchy of clusters from the logits from a flat clustering model, thus allowing the use the accurate leaf clustering from flat models.\n\n##Update After Rebuttal\nMost of my concerns have been addressed and I think it's a good paper. I've upgraded to a 4.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the Reviewer for the positive feedback, in particular for praising the idea, effectiveness, and usefulness of our method. We appreciate the useful feedback/suggestions, and address the concerns below. \n\n> If I understand correctly, the leaf accuracy is unaffected by the application [...] \n\nThe Reviewer is correct that our method by construction retains the leaf-level flat clustering performance of the pre-trained flat model. We make it explicit already in the text (e.g. lines 234-236, right column), but we agree with the Reviewer that in the interest of clarity it can be made more explicit in table 1. In the updated manuscript we add to the caption of Table 1 the final sentence *Notably,the application of L2H does not affect flat clustering performance, retaining the clustering performance of the pre-trained\nmodel (TURTLE, TEMI) at the leaf level.*, using italic to highlight it.\n\nWe also perform an additional experiment on the INaturalist21 (https://github.com/visipedia/inat_comp/tree/master/2021) dataset, which we use a setting to test whether our method can bring an advantage over flat clustering methods, used as backbone, when the nature of a dataset is inherently hierarhical. The INaturalist dataset contains ~2.7million images of species labelled at different taxonomy levels (1103 families, 273 orders, 51 classes, 13 phylums, 3 kingdoms). We perform the following experiment. First we train five instances of TURTLE to model clusters at five different taxonomy levels, i.e. varying K across instances in the range $K \\in \\\\{ 1103, 273, 51, 13, 3 \\\\}$. Then we apply our L2H algorithm on top TURTLE trained at the most fine-grained taxonomy level (i.e. *family*, K=1103), and use the produced hierarchy to make clustering predictions at the more coarse levels. We repeat the experiment across multiple seeds, and compare the performance of the two strategies. The results show that inferring clustering predictions at more coarse levels via the produced hierarchy with L2H leads to better performance compared to re-training TURTLE at the each corresponding level. Results at https://files.catbox.moe/0nt1uq.pdf\n\n> The runtime figures are stated to include the time [...]\n\nWe have refined the manuscript accordingly to clarify this point. The Reviewer is correct, and to confirm that our approach is both more computationally efficient and more performant than deep specialized hierarchical approaches we have trained the best performing baseline (TreeVAE) on the CIFAR-100 dataset CLIP embeddings. The results prove that the performance of TreeVAE improves compared to training in data space, but it is still markedly outperformed by our approach. Note that in this setting TreeVAE takes more than 2 hours on a GPU to train, while e.g. L2H turtle takes under 2 minutes. Since all models in this comparison have access to pre-trained embeddings from foundation models, these results confirm our method is markedly more computationally efficient than alternative deep specialized hierarchical approaches. Results at https://files.catbox.moe/gz0tu8.pdf\n\n> The method is straightforward and effective.[..]\n\nNote that we motivate our choice of merging the cluster/group with the lowest score in the Rebuttal for Reviewer YXaT. We appreciate the interest/suggestion from the Reviewer and will include an ablation on the updated version of the manuscript to test the Reviewer's idea of computing rp(c) for all pairs and merging the highest. It will make an interesting ablation in our work. \n> Other Comments Or Suggestions\n\nWe appreciate the Reviewer signaling these typos, which we will fix in the manuscript.\n\n---\n\nWe are happy to answer any additional questions and would appreciate it if the Reviewer would consider raising their score to full acceptance.", "claims_and_evidence": "The general idea to begin with a flat clustering method and then use the logits to form a hierarchy makes sense and is well-supported by the experiments. I have some minor comments and requests.\n\nIf I understand correctly, the leaf accuracy is unaffected by the application of L2H, so the scores in Table 1 for L2H-TEMI and L2H-TURTLE, respectively. If so, I think this should be made explicit somewhere. Currently, just reading Table 1, it seems as though you are presenting results for a novel flat clustering algorithm. Perhaps they could just be called TEMI and TURTLE in the Table, and then it can be stated that your method is able to use the high leaf accuracy of SOTA flat clustering methods. \n\nThe runtime figures are stated to include the time to train TURTLE. It should also be acknowledged that TURTLE, and hence L2H-TURTLE, requires access to foundation models, which of course have a very long train time, whereas the comparison methods do not.", "methods_and_evaluation_criteria": "The method is straightforward and effective. It is perhaps a bit simple to just take the cluster with the least confident assignments and merge it with another that has the second-highest average logits for that cluster. I wonder how the method compares to other simple operations on logits, such as computing rp(c) for all pairs and merging the highest.", "theoretical_claims": "n/a", "experimental_designs_or_analyses": "Experiments seem correct.", "relation_to_broader_scientific_literature": "This algorithm could be useful as it would reduce the deep hierarchical clustering problem to the deep clustering problem.", "essential_references_not_discussed": "None.", "other_strengths_and_weaknesses": "None.", "other_comments_or_suggestions": "line 37 RHS: \"struggle to handle to\" -> \"struggle to handle\"\nline 236, LHS: \"adjusted Random index\" -> \"adjusted Rand index\"", "questions_for_authors": "See above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes a lightweight method for hierarchical clustering that leverages pre-trained non-hierarchical clustering models, outperforming specialized deep hierarchical clustering models in terms of efficiency, scalability, and performance, and demonstrating its applicability in both unsupervised and supervised settings. After the rebuttal, it receives two weak accept and one accept. Its merits, including the interesting idea, very good efficiency on large dataset ImageNet-1K, and good results, are well recognized by the reviewers. The response well addresses the reviewers' concerns about the additional comparison, detailed analysis, and so on. I think the current manuscript meets the requirement of this top conference and recommend for acceptance. Please incorporate the revision in the updated manuscript.
Decision: Accept (poster)