From Logits to Hierarchies: Hierarchical Clustering made Simple

## Abstract

Abstract The structure of many real-world datasets is intrinsically hierarchical, making the modeling of such hierarchies a critical objective in both unsupervised and supervised machine learning. Recently, novel approaches for hierarchical clustering with deep architectures have been proposed. In this work, we take a critical perspective on this line of research and demonstrate that many approaches exhibit major limitations when applied to realistic datasets, partly due to their high computational complexity. In particular, we show that a lightweight procedure implemented on top of pre-trained non-hierarchical clustering models outperforms models designed specifically for hierarchical clustering. Our proposed approach is computationally efficient and applicable to any pre-trained clustering model that outputs logits, without requiring any fine-tuning. To highlight the generality of our findings, we illustrate how our method can also be applied in a supervised setup, recovering meaningful hierarchies from a pre-trained ImageNet classifier.

## 1 Introduction

Modeling hierarchical structures in the data is a long-standing goal in machine learning research (Bengio et al., ; Jordan & Mitchell, ). In many real-world scenarios, data is inherently organized in hierarchies, such as phylogenetic trees (Linnæus, ; Sneath & Sokal, ; Penny, ), tumor subclasses (Sørlie et al., ) and social networks Ravasz & Barabási (); Crockett et al. (). In unsupervised learning, hierarchical clustering can provide more accurate insights than flat (i.e. non-hierarchical) clustering methods by introducing multiple levels of granularity and alleviating the need for a fixed number of clusters specified a priori (Bertsimas et al., ; Chami et al., ). This aids scientific understanding and interpretability by providing a more informative representation (Lipton, ; Marcinkevičs & Vogt, ). The benefits of modeling a hierarchy in the data extend to supervised scenarios. For example, interpretable methods based on decision trees (Breiman, ; Tanno et al., ) hierarchically partition the data so that points in each split are linearly separable into classes. More recent work leverages hierarchies in the data to improve supervised methods (Bertinetto et al., ; Goren et al., ; Karthik et al., ) or for self-supervision (Long & van Noord, ).

Among classic algorithms for hierarchical clustering, agglomerative methods have been the most widely adopted. These methods compute pairwise distances between data points, often in a lower-dimensional representation space. Starting from the instance level, a hierarchy is then built based on the pairwise distances
by recursive agglomeration of similar points or clusters together in a bottom-up fashion (Murtagh & Contreras, ).
More recently, a revived interest in hierarchical clustering has sparked novel approaches using deep architectures (Mautz et al., ; Goyal et al., ; Shin et al., ; Vikram et al., ; Manduchi et al., ). However, despite their promising methodological contributions, recent approaches require specialized architectures and complex training schemes. Consequently, they cannot be applied to large-scale datasets due to their expensive computational requirements.
Moreover, we find that they often exhibit a lower performance at the leaf level compared to non-hierarchical models.

In this work, we take a critical perspective on recent research on hierarchical clustering and offer a simple alternative. Instead of designing specialized hierarchical clustering models, we develop a lightweight method for hierarchical clustering given a pre-trained flat model. In particular, we show that a lightweight algorithm implemented on top of (non-hierarchical) pre-trained models markedly outperforms specialized models for hierarchical clustering. Notably, our algorithm, which we name Logits to Hierarchies (L2H), only uses logits and requires no fine-tuning of the pre-trained model. Hence, it generally applies to black-box models even without access to internal representations (e.g., API calls to proprietary models) and bypasses the costly computation of pairwise distances between data points. Moreover, it also applies to supervised models, for which the inferred hierarchy of classes can aid model interpretability, e.g., for discovering potential biases such as spurious correlations between classes.

In summary, we make the following key contributions:

- •
In Section 3, we propose a simple algorithm for hierarchical clustering that transforms the logits from a pre-trained model into a hierarchical structure of classes. Our method markedly outperforms specialized hierarchical models and has low computational requirements. With logits as input, it computes a hierarchical clustering on ImageNet-sized datasets in a few minutes on a single CPU core.
- •
In Section 4.1, our experiments reveal significant limitations of recently proposed methods for hierarchical clustering, highlighting their weaknesses on large-scale datasets and subpar performance at the leaf level compared to non-hierarchical approaches.
- •
In Section 4.2, we provide a case study on ImageNet to demonstrate how our method applies to supervised models, showing how the inferred hierarchy of classes recovers parts of the WordNet hierarchy and helps discover potential biases of the pre-trained model and ambiguities in existing categorizations.

## 2 Related work

Hierarchical clustering aims to learn clusters of datapoints that are organized in a hierarchical structure. The methods used can be broadly categorized into agglomerative and divisive approaches (Nielsen, ). The former tackles the problem with a bottom-up approach and iteratively agglomerates clusters into larger ones until a full hierarchy is built in the form of a dendrogram, starting with each datapoint being a separate cluster (Murtagh & Contreras, ). The similarity of datapoints is measured according to a distance function, which for high-dimensional data is often defined on a lower-dimensional representation space. Multiple linkage methods have been proposed to compute the distance between clusters of
datapoints formed at a given step of the algorithm (Sneath, ; Ward, ). As examples, *single*, *average*, and *complete* linkage characterize the distance between two clusters as the minimum, average, and maximum distance between their datapoints, respectively. Since these algorithms can be costly, particularly in high-dimensional
spaces, approximate versions have been developed for faster computation (Abboud et al., ; Cochez & Mou, ). Notably, linkage methods are still widely applied in many domains, for instance, in medical research (Nguyen et al., ; Senevirathna et al., ; Resende et al., ).

On the other hand, divisive algorithms start with all objects belonging to the same cluster and recursively split them into subclusters. While early approaches are mostly based on heuristics, Dasgupta. () proposed the Dasgupta cost: an objective function for evaluating a hierarchical clustering, with a divisive approach to provide an approximately optimal solution. HypHC introduces a continuous relaxation of Dasgupta’s discrete optimization problem with provable guarantees via hyperbolic embeddings that better reflect the geometry of trees compared to Euclidean representations
Chami et al. (); Liu et al. ().
More recently, research has focused on developing deep learning approaches for hierarchical clustering (Mautz et al., ; Goyal et al., ; Shin et al., ; Vikram et al., ; Manduchi et al., ).
Among these, DeepECT learns a hierarchical clustering on top of the embedding space of a jointly optimized autoencoder (Mautz et al., ).
Notably, TreeVAE not only learns a hierarchical clustering in the latent space but also provides a generative model that adheres to the learnt hierarchy, thereby enabling sample generation in a structured manner (Manduchi et al., ). However, all these approaches have mostly been tested on simple datasets, far from realistic settings. As shown in our experiments, they present relevant limitations when deployed on more challenging datasets, mainly due to their high computational complexity.

Finally, the benefits of modeling a hierarchy in the data are not restricted to the unsupervised setup. In particular, a recent line of research focuses on leveraging a tree structure in the classes to assess and reduce the severity of misclassification of supervised models (Karthik et al., ). This can lead to safer models in cost-sensitive classification scenarios (Bertinetto et al., ) and allow a classifier to predict at different levels of the hierarchy depending on the required confidence (Goren et al., ). The visualization of hierarchies also provides global explanations of a model’s functionality, thereby improving a user’s understanding of the model behavior and fostering trust (Chakraborty et al., ; Lipton, ).

## 3 Method

In this work, we take a critical perspective on a recent line of research on hierarchical clustering: as an alternative to designing ad-hoc complex approaches, we focus on adapting pre-trained flat models to output a hierarchy with minimal overhead. To this end, we introduce a lightweight algorithm to leverage the information contained in the logits of a pre-trained flat clustering model to output a hierarchy of clusters.
In the following, we describe the proposed procedure and also provide a graphical illustration as well as detailed pseudocode.

Let $\mathcal{D}=\{\bm{x}_{1},\dots,\bm{x}_{N}\}$ be a dataset consisting of $N$ data points and $f_{\theta}$ be a non-hierarchical model trained to partition $\mathcal{D}$ into $K$ clusters. We assume that $f_{\theta}$ outputs unnormalized logits, i.e. the cluster assignment $k^{*}$ for a datapoint $\bm{x}$ is determined by computing $k^{*}=\operatorname*{arg\,max}_{k\in\{1,\dots,K\}}\text{softmax}_{k}(f_{\theta
}(\bm{x}))$.
We define two functions

$$ $h_{\theta}(\bm{x})=\operatorname*{arg\,max}_{k\in\{1,\dots,K\}}\text{softmax}_ {k}(f_{\theta}(\bm{x}))\qquad\qquad g_{\theta}(\bm{x})=\max_{k\in\{1,\dots,K\} }\text{softmax}_{k}(f_{\theta}(\bm{x}))$ $$

of which $h_{\theta}$ computes the cluster assignment for a datapoint $\bm{x}$, while $g_{\theta}$ computes the predicted probability of the cluster assignment for the datapoint $\bm{x}$.

A key idea behind our method is a simple yet effective way to determine the relatedness of clusters, or groups of clusters, by iteratively grouping them together to construct a hierarchy. Intuitively, to assess which group of clusters $G^{\prime}$ is most related to a given group $G$, we propose the following strategy. For datapoints assigned to clusters in $G$, we determine which group $G^{\prime}$ would have the majority of these datapoints reassigned to if clusters in $G$ were not available.(^1^11This passage is primarily for intuition and not strictly accurate. To be precise, we look not at reassignments but at predicted probabilities of reassignments (see Equation 1).)compute cluster assignments and corresponding predicted probabilities, restricting only to a subset of the total set of clusters.

We start with a masked version of the softmax function

$$ $\text{m_softmax}_{k}(\bm{v};G)=\begin{cases}\frac{\text{exp}({v_{i}})}{\sum_{ j\in\{1,\dots,K\}\setminus G}\text{exp}({v_{j}})}&\text{if }i\notin G\\ 0&\text{if }i\in G\end{cases}$ $$

given a $K$-dimensional vector $\bm{v}$ and a set $G\subset\{1,\dots,K\}$. This function restricts the softmax operation to the elements of $\bm{v}$ at indexes in $\{1,\dots,K\}\setminus G$.
Next, we define functions

$$ $h^{m}_{\theta}(\bm{x};G)=\operatorname*{arg\,max}_{k\in\{1,\dots,K\}}\text{m_ softmax}_{k}(f_{\theta}(\bm{x});G)\qquad g^{m}_{\theta}(\bm{x};G)=\max_{k\in\{ 1,\dots,K\}}\text{m_softmax}_{k}(f_{\theta}(\bm{x});G)$ $$

Note that the $h^{m}_{\theta}$ and $g^{m}_{\theta}$ functions correspond to $h_{\theta}$ and $g_{\theta}$, except restricting the choice of viable clusters to $\{1,\dots,K\}\setminus G$. In particular, the $h^{m}_{\theta}$ function computes the cluster assignment for a datapoint $\bm{x}$ restricting to clusters in $\{1,\dots,K\}\setminus G$, and $g^{m}_{\theta}$ the corresponding predicted probability. Lastly, we define

$$ $\mathcal{D}^{c}:=\{\bm{x}\in\mathcal{D}\mid h_{\theta}(\bm{x})=c\}$ $$

i.e. the subset of datapoints assigned to a given cluster $c\in\{1,\dots,K\}$. Similarly, we denote as $\mathcal{D}^{G}=\cup_{c\in G}D^{c}$ the subset of datapoints assigned to a group of clusters $G\subset\{1,\dots,K\}$.

Figure: Algorithm 1 Logits to Hierarchies (L2H)

We describe our proposed method in Algorithm 1. At the start of the procedure, $K$ groups are initialized as single clusters. At each iteration, two groups are merged into a single group, constructing a tree of clusters up to the root in $K-1$ iterations. Each iteration can be split in two stages. In the first stage, a score is computed for each group. To compute the score for a given group $G$ we aggregate the predicted probabilities for the datapoints assigned to clusters contained in $G$ as

$$ $s(G)=\mathop{\raisebox{-0.2pt}{\scalebox{1.5}{$\Lambda$}}}_{\begin{subarray}{c }\bm{x}\in\mathcal{D}^{c}\\ c\in G\end{subarray}}g_{\theta}(\bm{x})$ $$

where $\Lambda$ is a chosen aggregation function (e.g. sum function).
Then, the lowest scored group $G^{*}$ is selected for merging at this iteration, which concludes the first stage.

In the second stage, we search for the group $G^{\dagger}$ that is most related to $G^{*}$ to perform the merging. To do so, as mentioned above, we look at the subset of datapoints assigned to clusters in $G^{*}$: for these datapoints we recompute cluster assignments and predicted probabilities, this time restricting to clusters not contained in $G^{*}$. More formally, the total reassigned predicted probability to each cluster not contained in $G^{*}$ is computed as

$$ $rp(c):=\sum_{\begin{subarray}{c}\bm{x}\in\mathcal{D}^{G^{*}}\\ h_{\theta}^{m}(\bm{x};G^{*})=c\end{subarray}}g_{\theta}^{m}(\bm{x};G^{*}) \qquad\forall c\in\{1,..,K\}\setminus G^{*}$ (1) $$

Note that this quantity can be interpreted as a measure of relatedness between each cluster $c\in\{1,..,K\}\setminus G^{*}$, and the group of clusters $G^{*}$. The most related group to $G^{*}$ is finally selected as $G^{\dagger}\in\text{argmax}_{G\in\mathcal{G}\setminus\{G^{*}\}}\frac{1}{|G|}
\sum_{c\in G}rp(c)$, i.e. by averaging the total reassigned predicted probability across clusters in each group and selecting the group with the highest average. Given that cluster assignments and corresponding predicted probabilities can be computed via simple operations on the logits, the whole procedure can be executed with as input only the logits for the dataset $\mathcal{D}$ outputted from a pre-trained model $f_{\theta}$. To show this, we report an example Python implementation in Appendix A. Additionally Figure 1 provides an illustration to exemplify the proposed grouping strategy.

Figure: Figure 1: Illustration of the L2H algorithm. The four depicted clusters represent *dogs* in blue, *cats* in yellow, *horses* in red, *birds* in green respectively. In the first iteration (bottom), where groups correspond to single clusters, the dog cluster is selected for merging (shaded in grey). When recomputing predicted probabilities for samples in the dogs cluster, restricting to the remaining clusters, the cluster of cats has the highest predicted probability of reassignment. Note how, after merging, these two clusters are considered as a single group in the next iteration (top).
Refer to caption: extracted/5916361/images/grouping.png

## 4 Experiments

In this section we showcase the experimental results obtained with our proposed approach. In the first part, we focus on the task of hierarchical clustering. We demonstrate that existing specialized models for hierarchical clustering present major limitations when applied in realistic settings. In contrast, our proposed approach achieves convincing results in challenging vision datasets, markedly outperforming alternative methods. In the second part of this section, we present a case study where we discuss the application of the L2H algorithm on top of a pre-trained ImageNet classifier, demonstrating its value for model interpretability and the discovery of spurious correlations.

### 4.1 Hierarchical clustering

In this section, we compare the performance of our proposed method for hierarchical clustering with recent specialized approaches on three challenging vision datasets: namely the CIFAR-10, CIFAR-100 (Lake et al., ) and Food-101 (Bossard et al., ) datasets. We report our results in Table 1.
For each dataset, we implement our algorithm on top of two pre-trained flat clustering models, namely TURTLE (Gadetsky et al., ) and TEMI (Adaloglou et al., ). These are two state-of-the-art clustering methods (see Section B.2 for more details), both of which are *not* designed to produce a hierarchy of clusters. In our evaluation, we report both metrics to evaluate models at the flat level and metrics to evaluate the quality of the produced hierarchy. For comparing models at the flat level, we report NMI, ARI, Accuracy and Leaf Purity (LP).
To assess the quality of the hierarchical clustering, we report two metrics: Dendrogram Purity (DP) and Least Hierarchical Distance (LHD). The former was introduced in Kobren et al. () and extends the notion of purity, normally evaluated at the leaf level, to assess the quality of a tree clustering: higher purity corresponds to higher quality of the hierarchy. Note that this metric was recently adopted in Manduchi et al. () to benchmark deep hierarchical clustering models. Least Hierarchical Distance, on the other hand, measures the average minimal log-distance in the hierarchy between any pair of datapoints that have the same true label but different cluster assignments. A better hierarchy corresponds to a lower LHD. More details about our metrics can be found in the Appendix Section B.3.

The results in Table 1 uncover the aforementioned limitations of recent deep learning methods (DeepECT, TreeVAE).
In particular, these models do not scale well in terms of the depth of the hierarchy, thereby producing overly shallow hierarchies for datasets with a large number of classes (CIFAR-100, Food101). We find this to be linked to their high computational complexity, which we discuss later in Table 2. For instance, TreeVAE learns a hierarchical generative model with leaf-specific decoders: this choice helps its performance in a generative scenario but impacts its scalability to large-scale datasets.
Importantly, the comparison in terms of flat clustering metrics highlights that ad-hoc hierarchical models produce clusterings at the leaf level that are much less accurate than those obtained with non-hierarchical models. The results for DP and LHD highlight the shortcomings in these tasks. Note that we notice the presence of artifacts, underlined in Table 1, for the LHD metric, due to overly shallow hierarchies that degenerate to having only two to three leaves.

**Table 1: Quantitative comparison of hierarchical clustering performance on three datasets (CIFAR-10, CIFAR-100, Food-101). We report as a baseline Agglomerative clustering (Agg), deep hierarchical specialized models (DeepECT, TreeVAE), and our L2H method applied on top of two state-of-the-art flat models (TEMI, TURTLE). We also indicate the number of leaves in the hiearchy modelled by each approach, and whether a given method can perform inference on a hold-out test set. We bold best results for each metric and underline results that are artifacts of degenerate solutions with shallow hierarchies. Results are averaged over five runs.**
|  | Flat | Hierarchical | # leaves | Inference<br>on<br>test set |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | NMI $(\uparrow)$ | ARI $(\uparrow)$ | ACC $(\uparrow)$ | LP $(\uparrow)$ | DP $(\uparrow)$ | LHD $(\downarrow)$ |  |  |
| CIFAR-10 |  |  |  |  |  |  |  |  |
| Agglomerative | 0.074 | 0.038 | 0.211 | 0.246 | 0.121 | 0.549 | 10 | ✗ |
| HypHC | 0.019 | 0.009 | 0.134 | 0.359 | 0.104 | 0.569 | 10 | ✗ |
| DeepECT | 0.006 | 0.002 | 0.110 | 0.110 | 0.101 | 0.369 | 2-3 | ✓ |
| TreeVAE | 0.414 | 0.313 | 0.497 | 0.523 | 0.341 | 0.410 | 10 | ✓ |
| L2H-TEMI | 0.901 | 0.906 | 0.956 | 0.958 | 0.902 | 0.348 | 10 | ✓ |
| L2H-Turtle | 0.985 | 0.989 | 0.995 | 0.995 | 0.988 | 0.277 | 10 | ✓ |
| CIFAR-100 |  |  |  |  |  |  |  |  |
| Agglomerative | 0.223 | 0.020 | 0.090 | 0.131 | 0.019 | 0.428 | 100 | ✗ |
| HypHC | 0.072 | 0.004 | 0.031 | 0.560 | 0.011 | 0.499 | 100 | ✗ |
| DeepECT | 0.016 | 0.005 | 0.070 | 0.070 | 0.052 | 0.121 | 2-3 | ✓ |
| TreeVAE | 0.199 | 0.098 | 0.228 | 0.242 | 0.103 | 0.484 | 20 | ✓ |
| L2H-TEMI | 0.778 | 0.565 | 0.682 | 0.698 | 0.502 | 0.298 | 100 | ✓ |
| L2H-Turtle | 0.917 | 0.831 | 0.896 | 0.896 | 0.803 | 0.235 | 100 | ✓ |
| Food-101 |  |  |  |  |  |  |  |  |
| Agglomerative | 0.082 | 0.004 | 0.039 | 0.045 | 0.011 | 0.438 | 101 | ✗ |
| HypHC | 0.035 | 0.002 | 0.022 | 0.630 | 0.011 | 0.573 | 101 | ✗ |
| DeepECT | 0.003 | 0.000 | 0.011 | 0.011 | 0.010 | 0.333 | 2-3 | ✓ |
| TreeVAE | 0.114 | 0.017 | 0.057 | 0.058 | 0.016 | 0.483 | 20 | ✓ |
| L2H-TEMI | 0.917 | 0.841 | 0.904 | 0.881 | 0.801 | 0.270 | 101 | ✓ |
| L2H-Turtle | 0.894 | 0.800 | 0.876 | 0.843 | 0.758 | 0.297 | 101 | ✓ |

In contrast to alternative approaches, our proposed algorithm recovers high-quality hierarchies for all three datasets, when implemented on top of both TEMI and TURTLE models. Looking at both hierarchical metrics, our method markedly outperforms other approaches, with a consistent margin over costly deep learning specialized methods. These findings demonstrate that L2H can leverage the information embedded in the logits of a pre-trained flat clustering model to model an accurate hierarchy of the clusters, outperforming ad-hoc hierarchical models, even without having access to internal representations. Note as well that our method does not require any fine-tuning of the pre-trained model and, by construction, retains its clustering performance at the leaf level, which matches state-of-the-art in our results. Finally, the efficacy of our proposed procedure is not hindered by the presence of a large number of the classes in the dataset, as we witness for other methods.

Figure: Figure 2: Visualization of the hierarchical clustering produced by L2H-TURTLE on the CIFAR-100 dataset. The inferred hierarchy is represented as a circular tree. On the lowest level, the leaves are annotated by reporting the most frequent label for the samples in each leaf. Leaves are color-colored according to the 20 superclasses in the dataset.
Refer to caption: x1.png

In practice, hierarchical clustering results are often used as a visualization tool, and to analyze the structure of a dataset at different levels of granularity. Hence, to evaluate our proposed approach we visualize and inspect the hierarchy obtained with L2H-TURTLE on the CIFAR-100 dataset in Figure 2. Note that given the absence of leaf labels, we associate a class label to each leaf by looking at the most frequent label among the datapoints in the given leaf. While an off-the-shelf ground-truth hierarchy is not available for the CIFAR-100 dataset, the authors organize the 100 classes in 20 superclasses. Hence, we color-code the inferred leaf labels in the hierarchy by superclasses and check if the hierarchical clustering recovers this global structure. Notably, the global structure of the superclasses is largely reflected in the visualized hierarchy. Most interesting is that the outliers, for which the color does not coincide with the neighboring leaves, still reflect meaningful semantic associations.
For instance, *whale* and *dolphin*—despite being aquatic mammals—are grouped with fish species. However, this is not surprising, given their adaptation exclusively to aquatic environments and the presence of similar traits to fishes, like streamlined bodies. On the contrary, mammals such as *otter*, *beaver*, and *seal*, which are only semi-aquatic, are grouped with other small to medium-sized terrestrial mammals, emphasizing size and communal characteristics like the presence of limbs and fur. Another example is the characterization of *worm* and *snake* alongside in the hierarchy. Although snakes are reptiles, their elongated, limbless bodies visually resemble those of non-insect invertebrates like worms. This showcased analysis confirms the efficacy of our method in recovering a tree structure that follows meaningful semantic associations. The results indicate that our method produces hierarchies that enable detailed exploration of the structure in the data at varying levels of granularity. Inspecting the hierarchy gives valuable insights for interpretability, revealing underlying associations by the model.

Finally, we end this section with a comparison in terms of the computational cost of our method compared with alternative models, and in particular with specialized deep learning approaches for hierarchical clustering. As the results in Table 2 show, our proposed L2H algorithm is extremely lightweight. For completeness, we also measure the overall runtime to perform hierarchical clustering with our method, including the runtime to train the TURTLE model, as an example model used to compute logits as input. Our approach allows to perform hierarchical clustering extremely efficiently even on large-scale datasets such as ImageNet-1K, with a total training time of a few minutes. Note that, due to the combined efficiency of our method and state-of-the-art flat clustering models, the overall runtime scales seamlessly with dataset size and number of leaves in the hierarchy. Conversely, alternative hierarchical deep learning approaches exhibit a significantly higher computational cost. Moreover, a large dataset size and number of classes markedly increase the computational burden.

**Table 2: Training time (in minutes) for our proposed method compared to baselines for hierarchical clustering on CIFAR-10, CIFAR-100, and Food-101 datasets. At the top, we report the runtime for the L2H algorithm alone. Below, we report the runtime of the TURTLE model plus our L2H algorithm to produce a hierarchy, compared with the runtime of each baseline model. Results are averaged over three runs.**
|  | Dataset |  |  |  |
| --- | --- | --- | --- | --- |
|  | CIFAR-10<br>$K=10$<br>$N_{\text{tr}}=50000$ | CIFAR-100<br>$K=100$<br>$N_{\text{tr}}=50000$ | Food-101<br>$K=101$<br>$N_{\text{tr}}=75750$ | ImageNet-1K<br>$K=1000$<br>$N_{\text{tr}}=1281167$ |
| L2H | $<0.01$ | $<0.01$ | $<0.01$ | $0.47$ |
| Agglomerative | $0.1$ | $0.1$ | $0.9$ | - |
| HypHC | $163.7$ | $153.3$ | $195.3$ | - |
| DeepECT | $24.1$ | $26.2$ | $67.5$ | - |
| TreeVAE | $364.1$ | $756.3$ | $2293.7$ | - |
| L2H-TURTLE | $1.6$ | $1.6$ | $1.7$ | $5.27$ |

### 4.2 Case study: Pre-trained ImageNet Classifier

Next, we demonstrate how our method can be used in a supervised setup to produce a hierarchical clustering given the logits of a pre-trained classifier. Specifically, we use the ImageNet-1K dataset Deng et al. (), which comprises over a million images and a thousand distinct classes with an underlying hierarchical structure. We apply L2H on the logits of a pre-trained ImageNet classifier to determine the hierarchy of classes. As a pre-trained classifier, we use the InternImage model (Wang et al., ). The resulting hierarchical clustering is visualized in Figure 3.

Figure 3(a) shows the inferred tree for the thousand ImageNet classes. The colors indicate whether a leaf node corresponds to the superclass “artifact” or “organism”, which we determine based on the corresponding WordNet hypernyms of each class. Overall, we observe a distinct separation between the two superclasses in the inferred tree.
Figure 3(b) zooms into a subtree of the inferred tree that comprises different bird species. Specifically, it shows 58 of the 60 classes of birds found in the ImageNet dataset. The leaf nodes are colored by different clades of bird species (based on the WordNet hierarchy), showing that the inferred hierarchy groups together related species. For example, the group “aquatic bird” is almost completely represented in one of the two main branches, which further splits into a separate cluster for “parrots” and another one for “bird of prey”. The other main branch of the tree subdivides further into “passerine” and “game bird” forming distinct clusters.

Overall, our results suggest that the inferred hierarchy recovers a significant portion of the global and local hierarchical structure of the ImageNet dataset given the logits of a pre-trained ImageNet classifier trained with non-hierarchical labels. Yet, the inferred tree also reveals interesting outliers. For example, in Figure 3(a), there is a distinct subtree for snow-related artifacts (e.g., dogsled, snowmobile, bobsled) within a large branch of the tree that comprises organisms. Further investigation shows that this group of artifacts is merged with arctic animals (e.g., malamute, Siberian husky, Eskimo dog), which reveals an intuitive correlation between classes and highlights potential biases of the pre-trained model. Likewise, in Figure 3(b), we see potential outliers such as “peacock” among the group of parrots or “bustard” among game birds.
While the WordNet hierarchy classifies “bustard” as a wading bird, hence among aquatic birds, the inferred hierarchy places it among game birds. Interestingly, common definitions of bustards as terrestrial game birds support the inferred categorization.

Figure: (a) Complete tree (1K classes)
Refer to caption: x2.png

## 5 Conclusion

In this work, we propose a lightweight yet effective procedure for hierarchical clustering based on pre-trained non-hierarchical models.
Notably, our solution proves to be markedly more effective and significantly more computationally efficient than alternative methods. Different from existing models for hierarchical clustering, our method can successfully handle large datasets of many classes, taking an important step in deploying hierarchical clustering methods in challenging settings. Moreover, we show that the usefulness of our approach extends to supervised setups, by implementing it on top of a pre-trained classifier to recover a meaningful hierarchy of classes. A case study on ImageNet shows that this approach provides relevant insights for interpretability, and can reveal potential biases of the pre-trained model and ambiguities in existing categorizations.

While we provide extensive results on image datasets, in this work we do not explore other data modalities. However, our procedure is general and may be applied to different data types, which we leave for future work. Hierarchical clustering presents important advantages over non-hierarchical clustering by simultaneously capturing the structure in the data at multiple levels of granularity. However, inspecting the hierarchy is still necessary to extract valuable insights. An important focus for future work is to investigate strategies to partly bypass this process, automatically selecting levels of the hierarchies that provide the most meaningful clustering.

## Reproducibility Statement

We share a Python implementation of our L2H algorithm both in Appendix A and in the supplementary material. To ensure the reproducibility of our experimental results, in Appendix B we provide detailed insights on datasets and metrics, as well as on the implementation of our method and the compared baselines.

## Appendix

## Appendix A Code implementation of the L2H algorithm

In Figure 4, we provide a Python implementation of the L2H algorithm proposed in this work using standard scientific computing libraries (NumPy, SciPy). As stated in Section 3, our algorithm only requires the logits as input. It can be executed on the CPU even for large datasets, e.g., with a runtime of less than a minute for ImageNet-1K. Note that our procedure can be applied to any pre-trained unsupervised model to perform hierarchical clustering. Further, it can also be applied to logits from a supervised model to infer a hierarchy of classes. The output of the algorithm is a list, that contains the two groups that are merged at each step, hence characterizing the hierarchy. The aggregation function for computing the score per group is a design choice (as described in Section B.2) that can be viewed as a hyperparameter.

Figure: Figure 4: Python code implementation for the L2H algorithm presented in Section 3. Note that we choose the aggregation function when computing the score per group as described in Section B.2.

## Appendix B Experimental details

### B.1 Datasets

In this work, we run experiments on four challenging vision datasets, namely CIFAR-10 and CIFAR-100 (Lake et al., ), Food-101 (Bossard et al., ) and ImageNet-1K (Deng et al., ). CIFAR-10 and CIFAR-100 are well-established object classification datasets. The CIFAR-10 dataset consists of 60000 32x32 colored images divided in 10 classes: *airplane*, *automobile*, *bird*, *cat*, *deer*, *dog*, *frog*, *horse*, *ship*, *truck*. The train/test splits contain 50000 and 10000 images respectively. Similarly, also the CIFAR-100 dataset consists of 60000 32x32 colored images. However, they are organized into 100 classes. In addition, the 100 classes are grouped into 20 superclasses. As for CIFAR-10, the train/test splits also contain 50000 and 10000 images respectively. The Food101 dataset is a fine-grained classification dataset of food images, consisting of 101000 images for 101 classes. Images are high-resolution, up to 512 pixels side length. Images are split between 75750 training samples and 25250 test images. The ImageNet-1K dataset, widely used in computer vision, consists of 1000 classes organized according to the WordNet hierarchy (Miller, ), with 1281167 training and 50000 test samples, respectively.

### B.2 Implementation details

For our hierarchical clustering experiments, to train the TURTLE and TEMI models on all considered datasets, we use the official code provided by the authors with recommended choices for hyperparameters (Gadetsky et al., ; Adaloglou et al., ). In particular, TEMI employs CLIPViTL/14 representations of the data, while TURTLE employs both CLIPViTL/14 and DINOv2 ViT-g/14 representations. For more details on TURTLE trained using two representation spaces, see the original paper (Gadetsky et al., ). We train both TEMI and TURTLE with a number of clusters $K$ equal to the true number of classes in each dataset. For each dataset, we train models on the training set, then report metrics on the test set. Note that the L2H algorithm takes as input logits from the training set to infer the hierarchy, while metrics that evaluate the quality of the hierarchy are computed on the test set. As the aggregation function $\Lambda$ in the L2H algorithm (see Section 3) to score each group we employ

$$ $s(G)=\sum_{c\in G}\frac{1}{|\mathcal{D}^{c}|}\sum_{\bm{x}\in\mathcal{D}^{c}}g_ {\theta}(\bm{x})$ $$

which we find to work well in practice. However, other choices are possiblle. We implement TreeVAE (Manduchi et al., ) with their contrastive approach using the provided PyTorch codebase with corresponding defaults. The splitting criterion is set to the number of samples, an inductive bias that benefits this baseline method, since all datasets are balanced (Manduchi et al., ; Vandenhirtz et al., ). We set the number of clusters to $10$ for CIFAR-10 and to $20$ for the rest, due to the computational complexity, as seen in Table 2, as well as memory complexity, since every additional leaf adds a new decoder. DeepECT (Mautz et al., ) is also implemented using their provided codebase with the augmented version. Note that similar to the results shown in Manduchi et al. (), for colored datasets, DeepECT fails to grow trees, as they always collapse, indicating that DeepECT fails to find meaningful splits.
We implement agglomerative clustering using the scikit-learn library (Pedregosa et al., ), and fit the model using PCA embeddings of the datasets with $50$ components and wards criterion (Ward, ) as the linkage method.
Using the author’s original codebase, we further train Hyperbolic Hierarchical Clustering (Liu et al., ) on CLIP embeddings of the respective datasets.
The authors do not describe how to retrieve cluster assignments using their method, so we follow the agglomerative clustering procedure and assume the leaves of the last $K$ tree nodes created to form a cluster, where $K$ corresponds to the chosen number of clusters.

### B.3 Metrics

Here we provide more details on the metrics reported in our experiments in Section 4.1. In our comparisons, we evaluate models both on flat and hierarchical clustering.

Flat clustering To assess model performance in flat clustering, for each model we take the clustering at the level of the hierarchy where the number of clusters corresponds to the true number of classes $K$ in a given dataset. If the number of leaves at the leaf level of the hierarchy is smaller than $K$, as is the case for, e.g., TreeVAE and DeepECT on CIFAR-100, we consider the clustering at the leaf level. For flat clustering comparisons, we resort to well-established metrics, namely NMI, ARI, Accuracy, and Purity of the clusters (i.e., Leaf Purity). To compute accuracy and leaf purity, we resort to recent implementations in Gadetsky et al. () and Manduchi et al. (), respectively.

Hierarchical clustering To assess the quality of a learned hierarchy, and compare the results of different models in hierarchical clustering, we resort to two metrics. Dendrogram Purity (DP), introduced in Kobren et al. (), extends the notion of leaf purity to evaluate the purity of hierarchical clusters, and was recently adopted to benchmark hierarchical clustering models (Manduchi et al., ). Following the notation of Kobren et al. (), let $\mathcal{C}^{*}$ denote the true $K$-clustering (i.e., true class labeling) of a dataset $\mathcal{D}$. Then define

$$ $\mathcal{P}^{*}=\Bigl{\{}(x_{i},x_{j})\forall x_{i},x_{j}\in\mathcal{D},x_{j} \neq x_{j}\mid C^{*}(x_{i})=C^{*}(x_{j})\Bigr{\}}$ $$

as the set of pairs of data points that belong to the same true cluster. Also let $C^{*}_{k}$ be the set of datapoints in cluster $k$.
Dendrogram Purity (DP) is then defined for a hierchical clustering $\mathcal{H}$ as

$$ $DP(\mathcal{H})=\frac{1}{|\mathcal{P}^{*}|}\sum_{k=1}^{K}\sum_{(x_{i},x_{j}) \in\mathcal{C}_{k}^{*}}\text{pur}(\text{lvs}(\text{LCA}(x_{i},x_{j})),\mathcal {C}_{k}^{*}),$ $$

where $\text{LCA}(x_{1},x_{2})$ computes the least common ancestor node of datapoints $x_{1}$ and $x_{2}$ in $\mathcal{H}$, $\text{lvs}(z)$ returns the set of leaves of the sub-tree rooted at any internal node $z$ and $\text{pur}(S_{1},S_{2})=|S_{1}\cap S_{2}|/|S_{1}|$. One possible caveat of this metric is its high correlation with Leaf Purity: with a high leaf purity, most pairs of samples sharing the true label will inevitably fall into the same leaf. To address this, we introduce an additional metric for evaluation, namely Least Hierarchical Distance. With a similar notation as above we define

$$ $\bar{\mathcal{P}}^{*}=\Bigl{\{}(x_{i},x_{j})\forall x_{i},x_{j}\in\mathcal{D}, x_{j}\neq x_{j}\mid C^{*}(x_{i})=C^{*}(x_{j})\wedge l(x_{i};\mathcal{H})\neq l (x_{j};\mathcal{H})\Bigr{\}}$ $$

where the function $l(x;\mathcal{H})$ returns the cluster prediction for datapoint $x$ at the leaf level of $\mathcal{H}$. Hence $\bar{\mathcal{P}}^{*}$ is the set of all pairs of points sharing the same true label that are *not* assigned to the same leaf in $\mathcal{H}$. Least Hierarchical Distance is then defined for a hierarchical clustering $\mathcal{H}$ as

$$ $LHD(\mathcal{H})=\frac{1}{|\bar{\mathcal{P}}^{*}|}\sum_{(x_{i},x_{j})\in\bar{ \mathcal{P}}^{*}}\frac{\log_{2}(td(l(x_{i};\mathcal{H}),l(x_{j};\mathcal{H}))) -1}{\log_{2}(K)-1}$ $$

where $td(l_{1},l_{2})$ computes the number of edges in the shortest path that connects two leaves $l_{1},l_{2}$ in the tree defined by $\mathcal{H}$. Different from Dendrogram Purity, Least Hierarchical Distance only takes into consideration pairs of data points with the same true label that do not fall into the same leaf. Hence, it does not exhibit strong correlation with Leaf Purity, being more specific to the quality of the hiearchy, rather than influenced by the clustering at the leaf level.

## Appendix C Additional visualizations

In Figures 7 we provide additional visualizations for the hierarchies obtained with our proposed method in our hierarchical clustering experiments, complementing the quantitative and qualitative evidence shown in Section 4.1. Leafs are matched to the original labels by checking the most frequent label among datapoints contained in the leaf. In addition to the matched label, we report purity of each leaf, in percent.

In Figure 8, we provide additional results for ImageNet with different colorings for the inferred hierarchy, supplementing our results from Section 4.2. These visualizations show where the subtree of birds (used in Figure 3(b)) is located within the complete tree and in relation to other superclasses, such as mammals, reptiles, dogs, and clothing.

Figure: Figure 5: Visualization of the hierarchical clustering produced by L2H-TURTLE for the CIFAR-10 dataset.
Refer to caption: x4.png

Figure: Figure 6: Visualization of the hierarchical clustering produced by L2H-TURTLE for the CIFAR-100 dataset.
Refer to caption: x5.png

Figure: Figure 7: Visualization of the hierarchical clustering produced by L2H-TURTLE for the Food-101 dataset.
Refer to caption: x6.png

Figure: (a) Colored by bird, dog, and clothing
Refer to caption: x7.png

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: Q1. Why is IN-1k not included in Table 1?

Q2. What is the input to the agglomerative clustering baseline? Is it the raw pixel values of the images?

Q3. Is the performance of L2H-TEMI on flat clustering metrics identical to the performance of TEMI? Similarly for (L2H-)TURTLE.

Q4. Did you really train the auto-encoder for DeepECT from scratch on CIFAR-10/100 in 25 minutes on a single CPU core...? This seems rather unlikely, but it does appear to be what the paper claims.

Q5. In Fig 2, why is there a space between woman and otter but not between palm tree and sunflower? The two pairs are equally far from each other.
soundness: 2
strengths: It is valuable to have strong baselines for complex methods, even when those methods are simple.

The paper is generally well written.
summary: The paper presents a method to map from the logit space output from a pretrained clustering network to convert its output into a hierarchical structure. They demonstrate that by building on existing foundation models and non-hierarchical clustering methods, this simple method can outperform bespoke deep hierarchical clustering methods.
weaknesses: 1. Experiments are too restricted. In the introduction, the authors state that much real-world data is hierarchical, such as taxonomic data. However, they barely run experiments on datasets that possess hierarchical labels. CIFAR-10 does not have hierarchical labels, and so it is unclear how the authors evaluated it for dendritic purity, etc. As far as I am aware, Food101 doesn't either. CIFAR-100 only has one intermediate level in its hierarchy. They refer to the datasets used as being challenging, but I don't think it is appropriate to describe CIFAR-10 as such when we have models achieving >99% accuracy on it. They refer to CIFAR-100 and Food101 as having a large number of classes, but given that models are routinely trained on IN-1k (1,000 classes), IN-21k (21,000 classes), iNaturalist (10,000 classes), and 100 classes is quite small in comparison to these datasets. For work that is benchmarking a relatively simple method, I think it is important that the authors benchmark on more complex and more real-world datasets such as iNaturalist or BIOSCAN-5M from the taxonomic labelling perspective, or even non-image modalities for [more comprehensive analysis]. For a method that trains in 5 minutes on IN-1k, I do not think the authors can justify not exploring larger datasets and demonstrating the methodology works well at scale, both in terms of its performance and execution time. Even worse though, quantitative IN-1k results are not shown (i.e. in Table 1).

2. Baselines are inadequate. What is being proposed in the paper is a method for mapping from logits to a hierarchy, harnessing a couple of methods (TEMI and TURTLE) which can map from the embeddings of pretrained encoder (CLIP and DINOv2) to logits for clusters.
(2a.) The authors introduce the flexibility in their algorithm to consider multiple aggregation operations, but only consider summation ($Λ=\Sigma$). This leaves the question of what is the best aggregation operation unclear.
(2b.) Moreover, the paper leaves open the question of whether this is a *good* methodology of mapping from the TEMI or TURTLE outputs to a hierarchy. They present one method for this mapping, but there are no comparators given for it. For example, what if you used agglomerative clustering to cluster the TEMI or TURTLE logits? Does the L2H method beat this obvious baseline?
(2c.) The L2H-TEMI/TURTLE methods presented have the benefit of building on pretrained foundation models, whereas the competing methods are trained from scratch. Hence it would perhaps be more appropriate to compare against alternative methods for producing clusters from the embeddings of pretrained models, for instance [Lowe et. al. (2024)](https://arxiv.org/abs/2406.02465) investigate methodologies for doing this. [That paper concludes that the best way to cluster embeddings from a pretrained model is UMAP with >5 dims for dimensionality reduction, followed by Agglomerative clustering. So although they don't analyze hierarchical embeddings, the methodology they recommend is actually hierarchical and should serve as a useful baseline for this paper.]

3. Algorithm 1 (which is in a sense the main output of the work) is not adequately well written.
    - The same variable, $s$, is used for both the step number and the score.
    - Some variables are defined in the text of the paper, and used in the algorithm without definitions in the algorithm itself. The algorithm should stand alone without needing to read the rest of the paper to understand it. $f_θ$ is defined and never used, whilst $g_θ$, $g^m_θ$, $K$, $Λ$ are used and never defined.
    - argmin yields a single index, not a set, so it is unclear why the authors use an $\in$ symbol in $G^*\in \text{argmin}_{G\in \mathrm{G}} s(G)$, etc. Perhaps this is to cover the edge case of ties..? But this unnecessarily adds confusion to the notation.
    - $K$ is not updated in the outer loop, so the inner loop at L173 refers to cluster indices that no longer correspond to unmerged clusters.

**Typos and minor points**
- L311 Agglomerative clustering is abbreviated as Agg, but as far as I can see, this abbreviation is never used.
- The methodology for several things is only given in the appendix (e.g. agglomerative clustering), but nowhere in the main text does it let the reader know this or where in the appendix one might find these details (which a reader otherwise may well assume are omitted from the text entirely).
- There are a couple of places where a word is repeated (e.g. L271 "Appendix Appendix")
- L730 The text says it is giving the aggregation function, $Λ$, but the equation is the score function
- L734 "possiblle"
- L341 British English "modelled" contrasts with the American English used in the rest of the text
- Some references are cased incorrectly, e.g. key reference "Deepect" -> DeepECT
- Some references don't include publication details, e.g. Bengio (2014)
- Some references cite arXiv versions of papers where peer-reviewed versions are available (which should generally be preferred for citations) e.g. Karthik (2021b)
- Most references don't have links to the paper being cited, which should ideally be present for the convenience of the reader in the modern, digital era. DOIs can be made clickable simply by importing the doi package in main.tex (and don't need a URL field also present in the bibtex)
- One reference has mojibake: Nguyen (2024) "Identification of distinct subgroups of sj&#xf6;gren’s disease"

**References**
- iNaturalist-21 [arXiv:2103.16483](https://arxiv.org/abs/2103.16483)
- BIOSCAN-5M: [arXiv:2406.12723](https://arxiv.org/abs/2406.12723)
- Lowe et. al. (2024): [arXiv:2406.02465](https://arxiv.org/abs/2406.02465)

[N.B I realized a sentence was half-formed after the reviews were released, and added it later in an update to the review.]

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: Do other methods assume a starting division to K clusters?
soundness: 3
strengths: The methods elegantly bypass the standard O(N^2) complexity required for standard agglomerative clustering

The model outperforms the other baselines both in terms of runtime and clustering performance.

The paper is well written and easy to follow.
summary: This work proposes a new algorithm for generating hierarchies based on logits of pre-trained models. The new method outperforms other deep clustering techniques both in terms of performance and runtime.
weaknesses: I'm concerned that the suggested models received a better starting point. If I understand correctly, the pre-trained models know better how to cluster to the starting 10/100/101 clusters. It would benefit this work if there would be an ablation study that demonstrated the good results caused by the suggested algorithm and not by the pre-trained models.

The suggested algorithm is generic, how does it perform on non-vision-based clustering tasks?
How does it perform compared to other (more) deep hierarchical models?
I hope to see a broader comparison, with more than two models for deep custring and on various clustering tasks aside from vision.

The runtime comparison is not 100% fair, in my opinion, since L2H utilizes pre-trained models. 
The pre-trained models already know how to cluster the leafs, while other models need to learn it.

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 3
questions: It is necessary to validate the approach on representation space constructed by unsupervised models.
soundness: 3
strengths: The paper is clearly written, the experiments are properly conducted and shows that the method outperforms other hierarchical approaches. The idea is nice as it uses as a backbone a non-hierarchial clustering method.
summary: The authors provide a simple and fast hierarchical clustering method, which works well in the case of many-class problems. It works on the pretrained representation in the logits space. The method is a version of aglomerative clustering which can be based on a chosen standard clustering backbone. The paper is well written. The idea is simple and can be easily applied even to large datasets.
weaknesses: The novelty of the approach is rather limited, the method is in fact a shallow clustering approach applied in the representation space. Consequently, since in training the representation the knowledge of classes was used, it is hard to say how the method would deal in the case when the model did not know the proper classes beforehand. In particular a test of themodel on a few common unsupervised representations on ImageNet is necessary (for example given by SimClr or MaskedAutoEncoders).

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: Did you consider using the logits of standard image classification models (e.g. a ResNet trained to classify ImageNet)?
soundness: 2
strengths: - The paper is well-written and easy to read.
- The authors compare their method with three related methods. 
- The authors provide the pseudo code of their algorithm.
summary: The authors propose a new hierarchical clustering algorithm that leverages a pre-trained flat clustering models.
The key observation is that the logits of those model encode class similarity.
weaknesses: The contribution is rather limited and incremental. It is well known that the logits encode class similarity, and have been frequently used to reconstruct classification hierarchy. Examples for previous work with similar observation: 
- https://arxiv.org/abs/2007.06068 
- https://dl.acm.org/doi/abs/10.1145/3491102.3501823

The evaluation is based on two pre-trained image clustering models, TURTLE and TEMI.
This limits the generalizability of the results, as these are not the same backbones used for the other baselines.

Minor presentation remarks:
- Provide full names of the acronyms NMI and ARI
- datapoints => data points

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 1
questions: Please refer to the weakness section.
soundness: 2
strengths: The proposed method is simple.
The proposed method looks effective from the experimental results.
summary: This paper proposes a simple method for hierarchical clustering based on pre-trained non-hierarchical models. Specifically, the method performs on top of (non-hierarchical) pre-trained models, which uses the logits output by the pre-trained models to iteratively compute clusters in a fine-to-coarse manner. The authors verify the effectiveness of the method on four datasets, including CIFAR-10, CIFAR-100, Food-101 and ImageNet1K.
weaknesses: + The paper presents some challenges in clarity, particularly in the methods section, which may hinder reviewers from fully understanding how the proposed method operates. For example,
  - The terms "clusters" and "groups" seem to be used somewhat interchangeably, which can lead to confusion regarding their meanings. It would be helpful for the authors to clarify the distinctions between these terms in the context of the paper.
  - it should number the equations and reference specific formulas in the algorithm table to illustrate the execution process of the proposed method more effectively.

+ The authors assert that the proposed method does not require fine-tuning of the pre-trained model. However, it would be helpful to clarify how the logits are generated for different datasets. For example, in a network trained on ImageNet1K, the output is 1000-dimensional. If we are now clustering on CIFAR-10, does this imply that its logits are also 1000-dimensional? In other words, does this mean that the number of clusters at the finest-level hierarchy is 1,000?

+ In the context of hierarchical clustering, determining the number of hierarchies is crucial. I would appreciate further explanation on how this is accomplished within the proposed method.

+ Regarding line 190, where $s(G)$ represents the sum of the probabilities of samples belonging to $G$, it seems that if $G$ has more samples, its $s(G)$ value would naturally be larger. This raises the question of whether the relatedness between clusters is sensitive to the number of samples, which may not align with our intuitive understanding.

Meta-review: This works critically examines deep learning approaches to hierarchical clustering, and proposes to instead infer a hierarchical clustering from pre-trained representations and flat clusterings. The main strengths are the simplicity and generality of the method, which can be applied to any input clustering that provides logits, the contrast of the method with existing divisive/top-down deep clustering methods, and improved results along with results on both unsupervised and supervised clustering (z6t7, AeLC, LaeG). The main weaknesses are the insufficient experiments (z6t7) and baselines (z6t7, AeLC, k4QD) and a minor weakness of showing results only for vision (AeLC) while clustering could apply to data of any modality. The submission is missing controlled comparisons between the proposed method and existing methods on the same and standard representations (z6t7, LaeG) and missing a baseline for the extension from flat to hierarchical clustering, such as agglomerative clustering (z6t7). The authors discuss these points, especially the choice of datasets and comparisons, but do not convince the reviewers (see the additional comments on the discussion).

Explanation of Decision:

Five reviewers with expertise on deep learning, clustering, meta-learning, and hierarchical modeling agree on rejection (z6t7: 3, LaeG: 3, k4QD: 3, AeLC: 5, abpg: 5). The meta-reviewer does not find cause to overrule the expert reviewers. On the balance, the weaknesses—especially the limited comparisons and datasets—outweigh the strengths of this work at this time. It is important to provide baselines and comparisons of overall performance, as done here, as well as controlled experiments that match representations/architectures/aggregation choices. Without controlled experiments, it is difficult to understand the significance of the proposed method and its results, so for this reason and in agreement with the overall evaluation by the reviewers the meta-reviewer sides with rejection. The authors are nevertheless encouraged to incorporate the precise feedback, for instance the suggestions from z6t7 on informative comparisons, and resubmit.

Decision: Reject