Title: Hierarchical clustering with dot products recovers hidden tree structure
Venue: NeurIPS 2023
Accepted: yes
Keywords: agglomerative clustering

generative model

graphical model

hierarchical clustering

high-dimensional data
Abstract: In this paper we offer a new perspective on the well established agglomerative clustering algorithm, focusing on recovery of hierarchical structure. We recommend a simple variant of the standard algorithm, in which clusters are merged by maximum average dot product and not, for example, by minimum distance or within-cluster variance. We demonstrate that the tree output by this algorithm provides a bona fide estimate of generative hierarchical structure in data, under a generic probabilistic graphical model. The key technical innovations are to understand how hierarchical information in this model translates into tree geometry which can be recovered from data, and to characterise the benefits of simultaneously growing sample size and data dimension. We demonstrate superior tree recovery performance with real data over existing approaches such as UPGMA, Ward's method, and HDBSCAN.

Hierarchical clustering with dot products recovers hidden tree structure

Abstract

In this paper we offer a new perspective on the well established agglomerative clustering algorithm, focusing on recovery of hierarchical structure. We recommend a simple variant of the standard algorithm, in which clusters are merged by maximum average dot product and not, for example, by minimum distance or within-cluster variance. We demonstrate that the tree output by this algorithm provides a bona fide estimate of generative hierarchical structure in data, under a generic probabilistic graphical model. The key technical innovations are to understand how hierarchical information in this model translates into tree geometry which can be recovered from data, and to characterise the benefits of simultaneously growing sample size and data dimension. We demonstrate superior tree recovery performance with real data over existing approaches such as UPGMA, Ward's method, and HDBSCAN.

1 Introduction

Hierarchical structure is known to occur in many natural and man-made systems [32], and the problem considered in this paper is how to recover this structure from data. Hierarchical clustering algorithms, [25, 36, 37, 41] are very popular techniques which organise data into nested clusters, used routinely by data scientists and machine learning researchers, and are easily accessible through open source software packages such as scikit-learn [38]. We focus on perhaps the most popular family of such techniques: agglomerative clustering [17], [24, Ch.3], in which clusters of data points are merged recursively.

Agglomerative clustering methods are not model-based procedures, but rather simple algorithms. Nevertheless, in this work we uncover a new perspective on agglomerative clustering by introducing a general form of generative statistical model for the data, but without assuming specific parametric families of distributions (e.g., Gaussian). In our model (section 2.1), hierarchy takes the form of a tree defining the conditional independence structure of latent variables, using elementary concepts from probabilistic graphical modelling [30, 28]. In a key innovation, we then augment this conditional independence tree to form what we call a dendrogram , whose geometry is related to population statistics of the data. The new insight which enables tree recovery in our setting (made precise and explained in section 3) is that dot products between data vectors reveal heights of most recent common ancestors in the dendrogram .

We suggest an agglomerative algorithm which merges clusters according to highest sample average dot product (section 2.2). This is in contrast to many existing approaches which quantify dissimilarity

Alexander Modell

Nick Whiteley

between data vectors using Euclidean distance. We also consider the case where data are preprocessed by reducing dimension using PCA. We mathematically analyse the performance of our dot product clustering algorithm and establish that under our model, with sample size n and data dimension p growing simultaneously at appropriate rates, the merge distortion [20, 27] between the algorithm output and underlying tree vanishes. In numerical examples with real data (section 4), where we have access to labels providing a notion of true hierarchy, we compare performance of our algorithm against existing methods in terms of a Kendall τ b correlation performance measure, which quantifies association between ground-truth and estimated tree structure. We examine statistical performance with and without dimension reduction by PCA, and illustrate how dot products versus Euclidean distances relate to semantic structure in ground-truth hierarchy.

Related work Agglomerative clustering methods combine some dissimilarity measure with a 'linkage' function, determining which clusters are combined. Popular special cases include UPGMA [44] and Ward's method [48], against which we make numerical comparisons (section 4). Owing to the simple observation that, in general, finding the largest dot product between data vectors is not equivalent to finding the smallest Euclidean distance, we can explain why these existing methods may not correctly recover tree structure under our model (appendix E). Popular density-based clustering methods methods include CURE [22], OPTICS [7], BIRCH [52] and HDBSCAN [9]. In section 4 we discuss the extent to which these methods can and cannot be compared against ours in terms of tree recovery performance.

Existing theoretical treatments of hierarchical clustering involve different mathematical problem formulations and assumptions to ours. One common setup is to assume an underlying ultrametric space whose geometry specifies the unknown tree, and/or to study tree recovery as n → ∞ with respect to a cost function, e.g, [10, 16, 43, 11, 14, 15, 31, 12]. An alternative problem formulation addresses recovery of the cluster tree of the probability density from which it is assumed data are sampled [42, 20, 27]. The unknown tree in our problem formulation specifies conditional independence structure, and so it has a very different interpretation to the trees in all these cited works. Moreover, our data are vectors in R p , and p →∞ is a crucial aspect of our convergence arguments, but in the above cited works p plays no role or is fixed. Our definition of dendrogram is different to that in e.g. [10]: we do not require all leaf vertices to be equidistant from the root - a condition which arises as the 'fixed molecular clock' hypothesis [18, 34] in phylogenetics. We also allow data to be sampled from non-leaf vertices. There is an enormous body of work on tree reconstruction methods in phylogenetics, e.g. listed at [4], but these are mostly not general-purpose solutions to the problem of inferring hierarchy. Associated theoretical convergence results are limited in scope, e.g, the famous work [18] is limited to a fixed molecular clock, five taxa and does not allow observation error.

2 Model and algorithm

2.1 Statistical model, tree and dendrogram

Where possible, we use conventional terminology from the field of probabilistic graphical models, e.g., [28] to define our objects and concepts. Our model is built around an unobserved tree T = ( V , E ) , that is a directed acyclic graph with vertex and edge sets V and E , with two properties: T is connected (ignoring directions of edges), and each vertex has at most one parent, where we say u is a parent of v if there is an edge from u to v . We observe data vectors Y i ∈ R p , i = 1 , . . . , n , which we model as:

comprising three independent sources of randomness:

Z 1 , . . . , Z n are i.i.d., discrete random variables, with distribution supported on a subset of vertices Z ⊆ V , |Z| < ∞ ;

X ( v ) := [ X 1 ( v ) · · · X p ( v )] ⊤ is an R p -valued random vector for each vertex v ∈ V ;

E 1 , . . . , E n are i.i.d, R p -valued random vectors. The elements of E i are i.i.d., zero mean and unit variance. For each z ∈ Z , S ( z ) ∈ R p × p , is a deterministic matrix.

For each v ∈ Z , one can think of the vector X ( v ) as the random centre of a "cluster", with correlation structure of the cluster determined by the matrix S ( z ) . The latent variable Z i indicates which cluster

the i th data vector Y i is associated with. The vectors X ( v ) , v ∈ V \ Z , correspond to unobserved vertices in the underlying tree. As an example, Z could be the set of leaf vertices of T , but neither our methods nor theory require this to be the case. Throughout this paper, we assume that the tree T determines two distributional properties of X . Firstly, we assume T is the conditional independence graph of the collection of random variables X j := { X j ( v ); v ∈ V} for each j , that is,

A1. for all j = 1 , . . . , p , the marginal probability density or mass function of X j factorises as:

where Pa v denotes the parent of vertex v .

However we do not necessarily require that X 1 , . . . , X p are independent or identically distributed. Secondly, we assume

A2. for each j = 1 . . . , p , the following martingale-like property holds:

for all vertices v ∈ V except the root.

The conditions A1 and A2 induce an additive hierarchical structure in X . For any distinct vertices u, v ∈ V and w an ancestor of both, A1 and A2 imply that given X j ( w ) , the increments X j ( u ) -X j ( w ) and X j ( v ) -X j ( w ) are conditionally independent and both conditionally mean zero.

To explain our algorithm we need to introduce the definition of a dendrogram , D = ( T , h ) , where h : V → R + is a function which assigns a height to each vertex of T , such that h ( v ) ≥ h ( Pa v ) for any vertex v ∈ V other than the root. The term 'dendrogram' is derived from the ancient Greek for 'tree' and 'drawing', and indeed the numerical values h ( v ) , v ∈ V , can be used to construct a drawing of T where height is measured with respect to some arbitrary baseline on the page, an example is shown in figure 1(a). With ⟨· , ·⟩ denoting the usual dot product between vectors, the function

will act as a measure of affinity underlying our algorithm. The specific height function we consider is h ( v ) := α ( v, v ) . The martingale property A2 ensures that this height function satisfies h ( v ) ≥ h ( Pa v ) as required, see lemma 2 in appendix C. In appendix E we also consider cosine similarity as an affinity measure, and analyse its performance under a multiplicative noise model, cf. the additive model (1).

2.2 Algorithm

̸

Combining the model (1) with (2) we have α ( Z i , Z j ) = p -1 E [ ⟨ Y i , Y j ⟩| Z i , Z j ] for i = j ∈ [ n ] , [ n ] := { 1 , . . . , n } . The input to algorithm 1 is an estimate ˆ α ( · , · ) of all the pairwise affinities α ( Z i , Z j ) , i = j ∈ [ n ] . We consider two approaches to estimating α ( Z i , Z j ) , with and without dimension reduction by uncentered PCA. For some r ≤ min { p, n } , let V ∈ R p × r denote the matrix whose columns are orthonormal eigenvectors of ∑ n i =1 Y i Y ⊤ i associated with its r largest eigenvalues. Then ζ i := V ⊤ Y i is r -dimensional vector of principal component scores for the i th data vector. The two possibilities for ˆ α ( · , · ) we consider are:

In the case of ˆ α pca the dimension r must be chosen. Our theory in section 3.2 assumes that r is chosen as the rank of the matrix with entries α ( u, v ) , u, v ∈ Z , which is at most |Z| . In practice r usually must be chosen based on the data, we discuss this in appendix A.

Algorithm 1 returns a dendrogram ˆ D = ( ˆ T , ˆ h ) , comprising a tree ˆ T = ( ˆ V , ˆ E ) and height function ˆ h . Each vertex in ˆ V is a subset of [ n ] , thus indexing a subset of the data vectors Y 1 , . . . , Y n . The leaf vertices are the singleton sets { i } , i ∈ [ n ] , corresponding to the data vectors themselves. As algorithm 1 proceeds, vertices are appended to ˆ V , edges are appended to ˆ E , and the domain of the function ˆ α ( · , · ) is extended as affinities between elements of ˆ V are computed. Throughout the paper we simplify notation by writing ˆ α ( i, j ) as shorthand for ˆ α ( { i } , { j } ) for i, j ∈ [ n ] , noting that each argument of ˆ α ( · , · ) is in fact a subset of [ n ] .

̸

(a)

(b)

(c)

Algorithm 1 Dot product hierarchical clustering

Input: pairwise affinities ˆ α ( · , · ) between n data points

1: Initialise partition P 0 := {{ 1 } , . . . , { n }} , vertex set ˆ V := P 0 and edge set ˆ E to the empty set.

2: for m ∈ { 1 , . . . , n -1 } do

3: Find distinct pair u, v ∈ P m -1 with largest affinity

4: Update P m -1 to P m by merging u, v to form w := u ∪ v

5: Append vertex w to ˆ V and directed edges w GLYPH<1> u and w GLYPH<1> v to ˆ E 6: Define affinity between w and other members of P m as

7: end for

Output: Dendrogram ˆ D := ( ˆ T , ˆ h ) , comprising tree ˆ T = ( ˆ V , ˆ E ) and heights ˆ h ( v ) := ˆ α ( v, v ) for v ∈ ˆ V \ P 0 , and ˆ h ( v ) := max { ˆ h ( Pa v ) , ˆ α ( v, v ) } for v ∈ P 0 .

Implementation using scikit-learn The presentation of algorithm 1 has been chosen to simplify its theoretical analysis, but alternative formulations of the same method may be much more computationally efficient in practice. In appendix B we outline how algorithm 1 can easily be implemented using the AgglomerativeClustering class in scikit-learn [38].

3 Performance Analysis

3.1 Merge distortion is upper bounded by affinity estimation error

In order to explain the performance of algorithm 1 we introduce the merge height functions:

m ( u, v ) := h ( most recent common ancestor of u and v ) , u, v ∈ V ,

ˆ m ( u, v ) := ˆ h ( most recent common ancestor of u and v ) , u, v ∈ ˆ V .

To simplify notation we write ˆ m ( i, j ) as shorthand for ˆ m ( { i } , { j } ) , for i, j ∈ [ n ] . The discrepancy between any two dendrograms whose vertices are in correspondence can be quantified by merge distortion [20] - the maximum absolute difference in merge height across all corresponding pairs of

̸

̸

vertices. [20] advocated merge distortion as a performance measure for cluster-tree recovery, which is different to our model-based formulation, but merge distortion turns out to be a useful and tractable performance measure in our setting too. As a preface to our main theoretical results, lemma 1 explains how the geometry of the dendrogram D , in terms of merge heights, is related to population statistics of our model. Defining d ( u, v ) := h ( u ) -h ( w ) + h ( v ) -h ( w ) for u = v ∈ V , where w is the most recent common ancestor of u and v , we see d ( u, v ) is the vertical distance on the dendrogram from u down to w then back up to v , as illustrated in figure 1(a).

Lemma 1. For any two vertices u, v ∈ V ,

The proof is in appendix C. Considering the first two equalities in (4), it is natural to ask if the estimated affinities ˆ α ( · , · ) being close to the true affinities α ( · , · ) implies a small merge distortion between ˆ m ( · , · ) and m ( · , · ) . This is the subject of our first main result, theorem 1 below. In appendix E we use the third equality in (4) to explain why popular agglomerative techniques such as UPGMA [44] and Ward's method [48] which merge clusters based on proximity in Euclidean distance may enjoy limited success under our model, but in general do not correctly recover tree structure.

Let b denote the minimum branch length of D , that is, b = min { h ( v ) -h ( Pa v ) } , where the minimum is taken over all vertices in V except the root.

Theorem 1. Let the function ˆ α ( · , · ) given as input to algorithm 1 be real-valued and symmetric but otherwise arbitrary. For any z 1 , . . . , z n ∈ Z , if

̸

then the dendrogram returned by algorithm 1 satisfies

̸

The proof is in appendix C.

3.2 Affinity estimation error vanishes with increasing dimension and sample size

Our second main result, theorem 2 below, concerns the accuracy of estimating the affinities α ( · , · ) using ˆ α data or ˆ α pca as defined in (3). We shall consider the following technical assumptions.

A3 (Mixing across dimensions) . For mixing coefficients φ satisfying ∑ k ≥ 1 φ 1 / 2 ( k ) < ∞ and all u, v ∈ Z , the sequence { ( X j ( u ) , X j ( v )); j ≥ 1 } is φ -mixing.

A4 (Bounded moments) . For some q ≥ 2 , sup j ≥ 1 max v ∈Z E [ | X j ( v ) | 2 q ] < ∞ and E [ | E 11 | 2 q ] < ∞ , where E 11 is the first element of the vector E 1 .

A5 (Disturbance control) . max v ∈Z ∥ S ( v ) ∥ op ∈ O (1) as p →∞ , where ∥ · ∥ op is the spectral norm. A6 (PCA rank) . The dimension r chosen in definition of ˆ α pca , see (3) , is equal to the rank of the matrix with entries α ( u, v ) , u, v ∈ Z .

The concept of φ -mixing is a classical weak-dependence condition, e.g. [19, 39]. A3 implies that for each j ≥ 1 , ( X j ( u ) , X j ( v )) and ( X j + δ ( u ) , X j + δ ( v )) are asymptotically independent as δ →∞ . However, it is important to note that ˆ α data and ˆ α pca in (3), and hence the operation of algorithm 1 with these inputs, are invariant to permutation of the data dimensions j = 1 , . . . , p . Thus our analysis under A3 only requires there is some permutation of dimensions under which φ -mixing holds. A4 is a fairly mild integrability condition. A5 allows control of magnitudes of the 'disturbance' vectors Y i -X ( Z i ) = S ( Z i ) E i . Further background and discussion of assumptions is given in appendix C.2.

Theorem 2. Assume that A1 -A5 hold and let q be as in A4 . Then

̸

If additionally A3 is strengthened from φ -mixing to independence, S ( v ) = σ I p for some constant σ ≥ 0 and all v ∈ Z (in which case A5 holds), and A6 holds, then

̸

The proof of theorem 2 is in appendix C.2. We give an original and self-contained proof of (6). To prove (7) we use a recent uniform-concentration result for principal component scores from [49]. Overall, theorem 2 says that affinity estimation error vanishes if the dimension p grows faster enough relative to n (and r in the case of ˆ α pca , noting that under A6 , r ≤ |Z| , so it is sensible to assume r is much smaller than n and p ). The argument of O P ( · ) is the convergence rate; if ( X p,n ) is some collection of random variables indexed by p and n , X p,n ∈ O P ( n 2 /q / √ p ) means that for any ϵ > 0 , there exists δ and M such that n 2 /q / √ p ≥ M implies P ( | X p,n | > δ ) < ϵ . We note the convergence rate in (6) is decreasing in q where as rate in (7) is not. It is an open mathematical question whether (7) can be improved in this regard; sharpening the results of Whiteley et al. [49] used in the proof of (7) seems very challenging. However, when q = 2 , (6) gives O P ( n/ √ p ) compared to O P ( √ nr/p + √ r/n ) in (7), i.e. an improvement from n to √ nr in the first term. We explore empirical performance of ˆ α data versus ˆ α pca in section 4.2.

3.3 Interpretation

By combining theorems 1 and 2, we find that when b > 0 is constant and ˆ α is either ˆ α data or ˆ α pca , the merge distortion

̸

converges to zero at rates given by the r.h.s of (6) and (7). To gain intuition into what (8) tells us about the resemblance between ˆ D and D , it is useful to consider an intermediate dendrogram illustrated in figure 1(b) which conveys the realized values of Z 1 , . . . , Z n . This dendrogram is constructed from D by adding a leaf vertex corresponding to each observation Y i , with parent Z i and height p -1 E [ ∥ Y i ∥ 2 | Z 1 , . . . , Z n ] = h ( Z i ) + p -1 tr[ S ( Z i ) ⊤ S ( Z i )] , and deleting any v ∈ Z such that Z i = v for all i ∈ [ n ] (e.g., vertex c in figure 1(b)). The resulting merge height between the vertices corresponding to Y i and Y j is m ( Z i , Z j ) . (8) being small implies this must be close to ˆ m ( i, j ) in ˆ D as in figure 1(c). Moreover, in the case ˆ α = ˆ α data, the height ˆ h ( { i } ) of leaf vertex { i } in ˆ D is upper bounded by p -1 ∥ Y i ∥ 2 , which under our statistical assumptions is concentrated around p -1 E [ ∥ Y i ∥ 2 | Z 1 , . . . , Z n ] , i.e., the heights of the leaves in figure 1(c) approximate those of the corresponding leaves in figure 1(b).

̸

Overall we see that ˆ D in figure 1(c) approximates the dendrogram in figure 1(b), and in turn D . However even if m ( Z i , Z j ) = ˆ m ( i, j ) for all i, j , the tree output from algorithm 1, ˆ T , may not be isomorphic (i.e., equivalent up to relabelling of vertices) to the tree in figure 1(b); ˆ T is always binary and has 2 n -1 vertices, whereas the tree in figure 1(b) may not be binary, depending on the underlying T and the realization of Z 1 , . . . , Z n . This reflects the fact that merge distortion, in general, is a pseudometric on dendrograms. However, if one restricts attention to specific classes of true dendrograms D , for instance binary trees with non-zero branch lengths, then asymptotically algorithm 1 can recover them exactly. We explain this point further in appendix C.3.

4 Numerical experiments

We explore the numerical performance of algorithm 1 in the setting of five data sets summarised below. The real datasets used are open source, and full details of data preparation and sources are given in appendix D.

Simulated data. A simple tree structure with vertices V = { 1 , 2 , 3 , 4 , 5 , 6 , 7 , 8 } , edge set E = { 6 GLYPH<1> 1 , 6 GLYPH<1> 2 , 6 GLYPH<1> 3 , 7 GLYPH<1> 4 , 7 GLYPH<1> 5 , 8 GLYPH<1> 6 , 8 GLYPH<1> 7 } and Z = { 1 , 2 , 3 , 4 , 5 } (the leaf vertices). Z 1 , . . . , Z n are drawn from the uniform distribution on Z . The X j ( v ) are Gaussian random variables, independent across j . Full details of how these variables are sampled are in appendix D. The elements of E i are standard Gaussian, and S ( v ) = σ I p with σ = 1 .

20 Newsgroups. We used a random subsample of n = 5000 documents from the well-known 20 Newsgroups data set [29]. Each data vector corresponds to one document, capturing its p = 12818 Term Frequency Inverse Document Frequency features. The value of n was chosen to put us in the regime p ≥ n , to which our theory is relevant - see section 3.2. Some ground-truth labelling of documents is known: each document is associated with 1 of 20 newsgroup topics, organized at two hierarchical levels.

Zebrafish gene counts. These data comprise gene counts in zebrafish embryo cells taken from their first day of development [47]. As embryos develop, cells differentiate into various types with specialised, distinct functions, so the data are expected to exhibit tree-like structure mapping these changes. We used a subsample such that n = 5079 and p = 5498 to put us in the p ≥ n regime. Each cell has two labels: the tissue that the cell is from and a subcategory of this.

Amazon reviews. This dataset contains customer reviews on Amazon products [1]. A random sample of n = 5000 is taken and each data vector corresponds to one review with p = 5594 Term Frequency Inverse Document Frequency features. Each product reviewed has labels which make up a three-level hierarchy of product types.

S&P 500 stock returns. The data are p = 1259 daily returns between for n = 368 stocks which were constituents of the S&P 500 market index [2] between 2013 to 2018 . The two-level hierarchy of stock sectors by industries and sub-industries follows the Global Industry Classification Standard [3].

4.1 Comparing algorithm 1 to existing methods

We numerically compare algorithm 1 against three very popular variants of agglomerative clustering: UPGMA with Euclidean distance, Ward's method, and UPGMA with cosine distance. These are natural comparators because they work by iteratively merging clusters in a manner similar to algorithm 1, but using different criteria for choosing which clusters to merge. In appendix E we complement our numerical results with mathematical insights into how these methods perform under our modelling assumptions. Numerical results for other linkage functions and distances are given in appendix D. Several popular density-based clustering methods use some hierarchical structure, such as CURE [22], OPTICS [7] and BIRCH [52] but these have limitations which prevent direct comparisons: they aren't equipped with a way to simplify the structure into a tree, which it is our aim to recover, and only suggest extracting a flat partition based on a density threshold. HDBSCAN [9] is a density-based method that doesn't have these limitations, and we report numerical comparisons against it.

Kendall τ b ranking correlation. For real data some ground-truth hierarchical labelling may be available but ground-truth merge heights usually are not. We need a performance measure to quantitatively compare methods operating on such data. Commonly used clustering performance measures such as the Rand index [40] and others [23, 21] allow pairwise comparisons between partitions, but do not capture information about hierarchical structure. The cophenetic correlation coefficient [45] is commonly used to compare dendrograms, but relies on an assumption that points close in Euclidean distance should be considered similar which is incompatible with our notion of dot product affinity. To overcome these obstacles we formulate a performance measure as follows. For each of n data points, we rank the other n -1 data points according to the order in which they merge with it in the ground-truth hierarchy. We then compare these ground truth rankings to those obtained from a given hierarchical clustering algorithm using the Kendall τ b correlation coefficient [26]. This outputs a value in the interval [ -1 , 1] , with -1 , 1 and 0 corresponding to negative, positive and lack of association between the ground-truth and algorithm-derived rankings. We report the mean association value across all n data points as the overall performance measure. Table 1 shows results with raw data vectors Y 1: n or PC scores ζ 1: n taken as input to the various algorithms. For all the data sets except S&P 500, algorithm 1 is found to recover hierarchy more accurately than other methods. We include the results for the S&P 500 data to give a balanced scientific view, and in appendix E we discuss why our modelling assumptions may not be appropriate for these data, thus explaining the limitations of algorithm 1.

4.2 Simulation study of dot product estimation with and without PCA dimension reduction

For high-dimensional data, reducing dimension with PCA prior to clustering may reduce overall computational cost. Assuming ζ 1: n are obtained from, e.g., a partial SVD, in time O ( npr ) , the time complexity of evaluating ˆ α pca is O ( npr + n 2 r ) , versus O ( n 2 p ) for ˆ α data, although this ignores the cost of choosing r . In table 1 we see for algorithm 1, the results for input Y 1: n are very similar to those for ζ 1: n . To examine this more closely and connect our findings to theorem 2, we now compare ˆ α data and ˆ α pca as estimates of α through simulation. The model is as described at the start of section 4. In figure 2(a)-(b), we see that when p is growing with n , and when p is constant, the ˆ α pca error is very slightly smaller than the ˆ α data error. By contrast, in figure 2(c), when n = 10 is

fixed, we see that the ˆ α pca error is larger than that for ˆ α data. This inferior performance of ˆ α pca for very small and fixed n is explained by n appearing in the denominator of the second term in the rate O P ( √ nr/p + √ r/n ) for ˆ α pca in theorem 2 versus n appearing only in the numerator of O P ( n 2 /q / √ p ) for ˆ α data. Since it is Gaussian, this simulation model has finite exponential-of-quadratic moments, which is a much stronger condition than A4 ; we conjecture the convergence rate in this Gaussian case is O P ( √ log n/p ) for ˆ α data, which would be consistent with figure 2(a). These numerical results seem to suggest the rate for ˆ α pca is similar, thus the second result of theorem 2 may not be sharp.

200

400

600

800

1000

n

10 0

10 1

maximum error

p

log

n

p

n

1/2

p

n

/log

n

p

n

p

growing with

n

200

400

600

800

1000

n

10 0

10 1

p

=10 1

p

=10 2

p

=10 3

p

fixed

200

400

600

800

1000

p

10 0

6×10

2×10

3×10

n

=10 1

n

=10 2

n

=10 3

n

fixed

data

pca

(a)

(b)

(c)

̸

4.3 Comparing dot product affinities and Euclidean distances for the 20 Newsgroups data

In this section we expand on the results in table 1 for the 20 Newsgroups data, by exploring how inter-topic and intra-topic dot product affinities and Euclidean distances relate to ground-truth labels. Most existing agglomerative clustering techniques quantify dissimilarity using Euclidean distance. To compare dot products and Euclidean distances, figures 3(a)-(b) show, for each topic, the top five topics with the largest average dot product and smallest average Euclidean distance respectively. We see that clustering of semantically similar topic classes is apparent when using dot products but not when using Euclidean distance. Note that the average dot product affinity between comp.windows.x and itself is not shown in figure 3(b), but is shown in 3(a), by the highest dark blue square in the comp.windows.x column. In appendix D we provide additional numerical results illustrating that with n fixed, performance in terms of τ b correlation coefficient increases with p .

comp.graphics

comp.os.ms-windows.misc

comp.sys.ibm.pc.hardware

comp.sys.mac.hardware

comp.windows.x

misc.forsale

rec.autos

rec.motorcycles

rec.sport.baseball

rec.sport.hockey

sci.crypt

sci.electronics

sci.med

sci.space

talk.politics.guns

talk.politics.mideast

talk.politics.misc

alt.atheism

soc.religion.christian

1.0

1.5

2.0

2.5

3.0

1e

6

(a)

Average dot products

between documents of each topic

comp

comp.sys

misc.forsale

rec

rec.sport

religion

sci

talk.politics

comp.graphics

comp.os.ms-windows.misc

comp.sys.ibm.pc.hardware

comp.sys.mac.hardware

comp.windows.x

misc.forsale

rec.autos

rec.motorcycles

rec.sport.baseball

rec.sport.hockey

sci.crypt

sci.electronics

sci.med

sci.space

talk.politics.guns

talk.politics.mideast

talk.politics.misc

alt.atheism

soc.religion.christian

0.0022

0.0024

0.0026

0.0028

0.0030

0.0032

0.0034

(b)

Average Euclidean distances

between documents of each topic

comp.os.ms-windows.misc

comp.graphics

sci.crypt

comp.sys.ibm.pc.hardware

comp.sys.mac.hardware

sci.electronics

sci.space

misc.forsale

talk.politics.guns

alt.atheism

talk.politics.misc

rec.autos

sci.med

rec.motorcycles

soc.religion.christian

rec.sport.baseball

talk.politics.mideast

rec.sport.hockey

0.5

0.6

0.7

0.8

0.9

1.0

1.1

1.2

Average dot products

1e

6

(c)

Dot products between windows.x

and other tech subjects are large

Average dot product affinity between

windows.x documents and other topics

rec.motorcycles

rec.autos

sci.electronics

sci.space

rec.sport.baseball

comp.graphics

sci.med

misc.forsale

talk.politics.misc

rec.sport.hockey

soc.religion.christian

comp.sys.mac.hardware

talk.politics.guns

comp.os.ms-windows.misc

alt.atheism

comp.sys.ibm.pc.hardware

sci.crypt

talk.politics.mideast

0.0029

0.0030

0.0031

0.0032

0.0033

0.0034

0.0035

Average Euclidean distances

(d)

Euclidean distances between

windows.x and other tech

subjects are less informative

Average Euclidean distance between

windows.x documents and other topics

graphics

windows.misc

windows.x

electronics

forsale

pc.hardware

mac.hardware

baseball

hockey

autos

motorcycles

space

med

crypt

guns

politics.misc

mideast

atheism

christian

religion.misc

(e)

Dendrogram of topics using dot products

crypt

mideast

baseball

hockey

atheism

christian

religion.misc

med

guns

politics.misc

space

windows.misc

graphics

windows.x

forsale

pc.hardware

mac.hardware

electronics

autos

motorcycles

(f)

Dendrogram of topics using UPGMA

For one topic ('comp.windows.x') the results are expanded in figures 3(c)-(d) to show the average dot products and average Euclidean distances to all other topics. Four out of the five topics with the largest dot product affinity belong to the same 'comp' topic class and other one is a semantically similar 'sci.crypt' topic. Whereas, the other topics in the same 'comp' class are considered dissimilar in terms of Euclidean distance.

In order to display visually compact estimated dendrograms, we applied algorithm 1 and UPGMA in a semi-supervised setting where each topic is assigned its own PC score, taken to be the average of the PC scores of the documents in that topic, and then the algorithms are applied to cluster the topics. The results are shown in figures 3(e)-(f) (for ease of presentation, leaf vertex 'heights' are fixed to be equal).

5 Limitations and opportunities

Our algorithm is motivated by modelling assumptions. If these assumptions are not appropriate for the data at hand, then the algorithm cannot be expected to perform well. A notable limitation of our model is that α ( u, v ) ≥ 0 for all u, v ∈ V (see lemma 3 in appendix C). This is an inappropriate assumption when there are strong negative cross-correlations between some pairs of data vectors, and may explain why our algorithm has inferior performance on the S&P 500 data in table 1. Further discussion is given in appendix E. A criticism of agglomerative clustering algorithms in their basic form is that their computational cost scales faster than O ( n 2 ) . Approximations to standard agglomerative methods which improve computational scalability have been proposed [33, 5, 35]. Future research could investigate analogous approximations and speed-up of our method. Fairness in hierarchical clustering has been recently studied in cost function-based settings by [6] and in greedy algorithm settings by [13]. Future work could investigate versions of our algorithm which incorporate fairness measures.

References

Amazon product reviews dataset. https://www.kaggle.com/datasets/kashnitsky/ hierarchical-text-classification , . Accessed: 2023-05-16.

S&P500 stock data. https://www.kaggle.com/datasets/camnugent/sandp500 , . Accessed: 2023-05-16.

S&P500 hierarchy. https://en.wikipedia.org/wiki/List_of_S%26P_500_ companies , . Accessed: 2023-05-16.

Wikipedia list of phylogenetics software. https://en.m.wikipedia.org/wiki/List_of_ phylogenetics_software . Accessed: 2023-05-16.

Amir Abboud, Vincent Cohen-Addad, and Hussein Houdrougé. Subquadratic high-dimensional hierarchical clustering. Advances in Neural Information Processing Systems , 32, 2019.

Sara Ahmadian, Alessandro Epasto, Marina Knittel, Ravi Kumar, Mohammad Mahdian, Benjamin Moseley, Philip Pham, Sergei Vassilvitskii, and Yuyan Wang. Fair hierarchical clustering. Advances in Neural Information Processing Systems , 33:21050-21060, 2020.

Mihael Ankerst, Markus M Breunig, Hans-Peter Kriegel, and Jörg Sander. Optics: Ordering points to identify the clustering structure. ACM Sigmod record , 28(2):49-60, 1999.

Matteo Barigozzi, Haeran Cho, and Dom Owens. Fnets: Factor-adjusted network estimation and forecasting for high-dimensional time series. arXiv preprint arXiv:2201.06110 , 2022.

Ricardo JGB Campello, Davoud Moulavi, and Jörg Sander. Density-based clustering based on hierarchical density estimates. In Advances in Knowledge Discovery and Data Mining: 17th Pacific-Asia Conference, PAKDD 2013, Gold Coast, Australia, April 14-17, 2013, Proceedings, Part II 17 , pages 160-172. Springer, 2013.

Gunnar E Carlsson, Facundo Mémoli, et al. Characterization, stability and convergence of hierarchical clustering methods. J. Mach. Learn. Res. , 11(Apr):1425-1470, 2010.

Moses Charikar and Vaggos Chatziafratis. Approximate hierarchical clustering via sparsest cut and spreading metrics. In Proceedings of the Twenty-Eighth Annual ACM-SIAM Symposium on Discrete Algorithms , pages 841-854. SIAM, 2017.

Vaggos Chatziafratis, Mohammad Mahdian, and Sara Ahmadian. Maximizing agreements for ranking, clustering and hierarchical clustering via max-cut. In International Conference on Artificial Intelligence and Statistics , pages 1657-1665. PMLR, 2021.

Anshuman Chhabra and Prasant Mohapatra. Fair algorithms for hierarchical agglomerative clustering. In 2022 21st IEEE International Conference on Machine Learning and Applications (ICMLA) , pages 206-211. IEEE, 2022.

Vincent Cohen-Addad, Varun Kanade, and Frederik Mallmann-Trenn. Hierarchical clustering beyond the worst-case. Advances in Neural Information Processing Systems , 30, 2017.

Vincent Cohen-Addad, Varun Kanade, Frederik Mallmann-Trenn, and Claire Mathieu. Hierarchical clustering: Objective functions and algorithms. Journal of the ACM (JACM) , 66(4):1-42, 2019.

Sanjoy Dasgupta. A cost function for similarity-based hierarchical clustering. In Proceedings of the forty-eighth annual ACM symposium on Theory of Computing , pages 118-127, 2016.

William HE Day and Herbert Edelsbrunner. Efficient algorithms for agglomerative hierarchical clustering methods. Journal of classification , 1(1):7-24, 1984.

Ronald W DeBry. The consistency of several phylogeny-inference methods under varying evolutionary rates. Molecular Biology and Evolution , 9(3):537-551, 1992.

Roland L Dobrushin. Central limit theorem for nonstationary markov chains. i. Theory of Probability & Its Applications , 1(1):65-80, 1956.

Justin Eldridge, Mikhail Belkin, and Yusu Wang. Beyond hartigan consistency: Merge distortion metric for hierarchical clustering. In Conference on Learning Theory , pages 588-606. PMLR, 2015.

Edward B Fowlkes and Colin L Mallows. A method for comparing two hierarchical clusterings. Journal of the American statistical association , 78(383):553-569, 1983.

Sudipto Guha, Rajeev Rastogi, and Kyuseok Shim. Cure: An efficient clustering algorithm for large databases. ACM Sigmod record , 27(2):73-84, 1998.

Lawrence Hubert and Phipps Arabie. Comparing partitions. Journal of classification , 2:193-218, 1985.

Anil K Jain and Richard C Dubes. Algorithms for clustering data . Prentice-Hall, Inc., 1988.

Stephen C Johnson. Hierarchical clustering schemes. Psychometrika , 32(3):241-254, 1967.

Maurice George Kendall. Rank correlation methods. Griffin, 1948.

Jisu Kim, Yen-Chi Chen, Sivaraman Balakrishnan, Alessandro Rinaldo, and Larry Wasserman. Statistical inference for cluster trees. Advances in Neural Information Processing Systems , 29, 2016.

Daphne Koller and Nir Friedman. Probabilistic graphical models: principles and techniques . MIT press, 2009.

Ken Lang. Newsweeder: Learning to filter netnews. In Machine learning proceedings 1995 , pages 331-339. Elsevier, 1995.

Steffen L Lauritzen. Graphical models , volume 17. Clarendon Press, 1996.

Bogdan-Adrian Manghiuc and He Sun. Hierarchical clustering: o (1) -approximation for wellclustered graphs. Advances in Neural Information Processing Systems , 34:9278-9289, 2021.

Henok Mengistu, Joost Huizinga, Jean-Baptiste Mouret, and Jeff Clune. The evolutionary origins of hierarchy. PLoS computational biology , 12(6):e1004829, 2016.

Nicholas Monath, Ari Kobren, Akshay Krishnamurthy, Michael R Glass, and Andrew McCallum. Scalable hierarchical clustering with tree grafting. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining , pages 1438-1448, 2019.

Priya Moorjani, Carlos Eduardo G Amorim, Peter F Arndt, and Molly Przeworski. Variation in the molecular clock of primates. Proceedings of the National Academy of Sciences , 113(38): 10607-10612, 2016.

Benjamin Moseley, Sergei Vassilvtiskii, and Yuyan Wang. Hierarchical clustering in general metric spaces using approximate nearest neighbors. In International Conference on Artificial Intelligence and Statistics , pages 2440-2448. PMLR, 2021.

Fionn Murtagh. A survey of recent advances in hierarchical clustering algorithms. The computer journal , 26(4):354-359, 1983.

Fionn Murtagh and Pedro Contreras. Algorithms for hierarchical clustering: an overview. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery , 2(1):86-97, 2012.

Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. Scikitlearn: Machine learning in python. the Journal of machine Learning research , 12:2825-2830, 2011.

Magda Peligrad. An invariance principle for φ -mixing sequences. The Annals of Probability , pages 1304-1313, 1985.

William M Rand. Objective criteria for the evaluation of clustering methods. Journal of the American Statistical association , 66(336):846-850, 1971.

Chandan K Reddy and Bhanukiran Vinzamuri. A survey of partitional and hierarchical clustering algorithms. In Data clustering , pages 87-110. Chapman and Hall/CRC, 2018.

Alessandro Rinaldo, Aarti Singh, Rebecca Nugent, and Larry Wasserman. Stability of densitybased clustering. Journal of Machine Learning Research , 13:905, 2012.

Aurko Roy and Sebastian Pokutta. Hierarchical clustering via spreading metrics. Advances in Neural Information Processing Systems , 29, 2016.

R. R. Sokal and C. D. Michener. A statistical method for evaluating systematic relationships. University of Kansas Science Bulletin , 38:1409-1438, 1958.

Robert R Sokal and F James Rohlf. The comparison of dendrograms by objective methods. Taxon , pages 33-40, 1962.

Baris Sumengen, Anand Rajagopalan, Gui Citovsky, David Simcha, Olivier Bachem, Pradipta Mitra, Sam Blasiak, Mason Liang, and Sanjiv Kumar. Scaling hierarchical agglomerative clustering to billion-sized datasets. arXiv preprint arXiv:2105.11653 , 2021.

Daniel E Wagner, Caleb Weinreb, Zach M Collins, James A Briggs, Sean G Megason, and Allon M Klein. Single-cell mapping of gene expression landscapes and lineage in the zebrafish embryo. Science , 360(6392):981-987, 2018.

Joe H Ward Jr. Hierarchical grouping to optimize an objective function. Journal of the American statistical association , 58(301):236-244, 1963.

Nick Whiteley, Annie Gray, and Patrick Rubin-Delanchy. Discovering latent topology and geometry in data: a law of large dimension. arXiv preprint arXiv:2208.11665 , 2022.

F Alexander Wolf, Philipp Angerer, and Fabian J Theis. Scanpy: large-scale single-cell gene expression data analysis. Genome biology , 19:1-5, 2018.

Wang Xuejun, Hu Shuhe, Yang Wenzhi, and Shen Yan. On complete convergence for weighted sums of-mixing random variables. Journal of Inequalities and Applications , 2010:1-13, 2010.

Tian Zhang, Raghu Ramakrishnan, and Miron Livny. Birch: an efficient data clustering method for very large databases. ACM sigmod record , 25(2):103-114, 1996.

Appendices

A Wasserstein dimension selection

In order to compute the principal component scores ζ 1 , . . . , ζ n , one must choose the dimension r . Traditionally and somewhat heuristically, this is done by finding the 'elbow' in the scree plot of eigenvalues. The bias-variance tradeoff associated with choosing r was explored by [49], who suggested a data-splitting method of dimension selection for high-dimensional data using Wasserstein distances. Whilst this method may be more costly than the traditional 'elbow' approach, it was empirically demonstrated in [49] to have superior performance.

Algorithm 2 Wasserstein PCA dimension selection [49]

Input:

data vectors Y 1 , . . . , Y n ∈ R p .

1: for r ∈ { 1 , ..., min( n, p ) } do

2: Let V ∈ R p × r denote the matrix whose columns are orthonormal eigenvectors associated with the r largest eigenvalues of ∑ ⌈ n/ 2 ⌉ i =1 Y i Y ⊤ i

3: Orthogonally project Y 1 , . . . , Y ⌈ n/ 2 ⌉ onto the column space of V , ˆ Y i := VV ⊤ Y i

4: Compute Wasserstein distance d r between ˆ Y i , . . . , ˆ Y ⌈ n/ 2 ⌉ and Y ⌈ n/ 2 ⌉ +1 , . . . , Y n (as point sets in R p )

5: end for

Output: selected dimension ˆ r = argmin { d }

r .

We note that in practice, the eigenvectors appearing in this procedure could be computed sequentially as r grows, and to limit computational cost one might consider r only up to some r max < min( n, p ) .

B Implementation using scikit-learn

Algorithm 1 can be implemented using the Python module scikit-learn [38] via their AgglomerativeClustering class, using the standard 'average' linkage criterion and a custom metric function to compute the desired affinities. However, AgglomerativeClustering merges clusters based on minimum distance metric, whereas algorithm 1 merges according to maximum dot product. Therefore, the custom metric function we used in our implementation calculates all the pairwise dot products and subtracts them from the maximum. This transformation needs to then be rectified if accessing the merge heights.

All code released as part of this paper is under the MIT License and can be found at https: //github.com/anniegray52/dot_product_hierarchical

C Proofs and supporting theoretical results

Lemma 2. The height function h ( v ) := α ( v, v ) satisfies h ( v ) ≥ h (Pa v ) for all v ∈ V except the root.

Proof.

where A2 combined with the tower property and linearity of conditional expectation implies E [ ⟨ X ( v ) -X ( Pa v ) , X ( Pa v ) ⟩ ] = 0 .

Lemma 3. For all u, v ∈ V , α ( u, v ) ≥ 0 .

̸

Proof. If u = v , α ( u, v ) ≥ 0 holds immediately from the definition of α in (2). For u = v with most recent common ancestor w ,

where the second equality uses A1 together with standard conditional independence aguments, and the third equality uses A2 .

Proof of lemma 1. Let w be the most recent common ancestor of u and v . For each j = 1 , . . . , p , the property A1 together with standard conditional independence arguments imply that X j ( u ) and X j ( v ) are conditionally independent given X j ( w ) , and the property A2 implies that E [ X j ( u ) | X j ( w )] = E [ X j ( v ) | X j ( w )] = X j ( w ) . Therefore, by the tower property of conditional expectation,

Hence, using the definitions of the merge height m , the height h and the affinity α ,

which proves the first equality in the statement. The second equality is the definition of α For the third equality in the statement, we have

.

where the first equality uses the definition of d , and the second equality uses the definition of h and h ( w ) = m ( u, v ) = α ( u, v ) .

C.1 Proof of Theorem 1

The following lemma establishes an identity concerning the affinities computed in algorithm 1 which will be used in the proof of theorem 1.

Lemma 4. Let P m , m ≥ 0 , be the sequence of partitions of [ n ] constructed in algorithm 1 . Then for any m ≥ 0 ,

Proof. The proof is by induction on m . With m = 0 , (9) holds immediately since P 0 = {{ 1 } , . . . , { n }} . Now suppose (9) holds at step m . Then for any distinct pair w,w ′ ∈ P m +1 ,

either w or w ′ is the result of merging two elements of P m , or w and w ′ are both elements of P m . In the latter case the induction hypothesis immediately implies:

In the case that w or w ′ is the result of a merge, suppose w.l.o.g. that w = u ∪ v for some u, v ∈ P m and w ′ ∈ P m . Then by definition of ˆ α in algorithm 1 ,

where the final equality uses w = u ∪ v . The induction hypothesis thus holds at step m +1 .

The following proposition establishes the validity of the height function constructed in algorithm 1. Some of the arguments used in this proof are qualitatively similar to those used to study reducible linkage functions by, e.g., Sumengen et al. [46], see also historical references therein.

Proposition 1. With ˆ V the vertex set and ˆ h the height function constructed in algorithm 1 with any symmetric, real-valued input ˆ α ( · , · ) , it holds that ˆ h ( v ) ≥ ˆ h (Pa v ) for all vertices v ∈ ˆ V except the root.

Proof. The required inequality ˆ h ( v ) ≥ ˆ h ( Pa v ) holds immediately for all the leaf vertices v ∈ P 0 = {{ 1 } , . . . , { n }} by the definition of ˆ h in algorithm 1. All the remaining vertices in the output tree, i.e., those in ˆ V\ P 0 , are formed by merges over the course of the algorithm. For m ≥ 0 let w m = u m ∪ v m denote the vertex formed by merging some u m , v m ∈ P m . Then w m = Pa u m and w m = Pa v m . Each u m is either a member of P 0 or equal to w m ′ for some m ′ < m . The same is true of each v m . It therefore suffices to show that ˆ h ( w m ) ≥ ˆ h ( w m +1 ) for m ≥ 0 , where by definition in the algorithm, ˆ h ( w m ) = ˆ α ( u m , v m ) . Also by definition in the algorithm, ˆ h ( w m +1 ) is the largest pairwise affinity between elements of P m +1 . Our objective therefore is to upper-bound this largest affinity and compare it to ˆ h ( w m ) = ˆ α ( u m , v m ) .

The affinity between w m = u m ∪ v m and any other element w ′ of P m +1 (which must also be an element of P m ) is, by definition in the algorithm,

where the last inequality holds because u m , v m , by definition, have the largest affinity amongst all elements of P m . For the same reason, the affinity between any two distinct elements of P m +1 neither of which is w m (and therefore both of which are elements of P m ) is upper-bounded by ˆ α ( u m , v m ) . We have therefore established ˆ h ( w m ) = ˆ α ( u m , v m ) ≥ ˆ h ( w m +1 ) as required, and this completes the proof.

Proof of theorem 1. Let us introduce some definitions used throughout the proof.

̸

̸

We arbitrarily chose and then fix i, j ∈ [ n ] with i = j , and define

Let u denote the most recent common ancestor of the leaf vertices { i } and { j } in ˆ D and let m ≥ 1 denote the step of the algorithm at which u is created by a merge, that is m = min { m ′ ≥ 1 : u ∈ P m ′ } . We note that by construction, u is equal to the union of all leaf vertices with ancestor u , and by definition of ˆ h in algorithm 1 ˆ h ( u ) = ˆ H .

Let v denote the most recent common ancestor of z i and z j in D , which has height h ( v ) = H .

Lower bound on m ( z i , z j ) -ˆ m ( i, j ) . There is no partition of u into two non-empty sets A,B ⊆ [ n ] such that ˆ α ( k, l ) < ˆ H for all k ∈ A and l ∈ B . We prove this by contradiction. Suppose that such a partition exists. There must be a step m ′ ≤ m at which some A ′ ⊆ A is merged some B ′ ⊆ B . The vertex w formed by this merge would have height

where the first equality is the definition of ˆ h ( w ) in the algorithm and the second equality holds by lemma 4. However, in this construction u is an ancestor of w , and ˆ h ( w ) < ˆ h ( u ) therefore contradicts the result of proposition 1.

As a device to be used in the next step of the proof, consider an undirected graph with vertex set u , in which there is an edge between two vertices k and l if and only if ˆ α ( k, l ) ≥ ˆ H . Then, because there is no partition as established above, this graph must be connected. Now consider a second undirected graph, also with vertex set u , in which there is any edge between two vertices k and l if and only if α ( z k , z l ) ≥ ˆ H -M . Due to the definition of M in (10), any edge in the first graph is an edge in the second, so the second graph is connected too. Let k , l , and ℓ be any distinct members of u . Using the fact established in lemma 1 that α ( z k , z l ) and α ( z l , z ℓ ) are respectively the merge heights in D between z k and z l , and z l and z ℓ , it can be seen that if there are edges between k and l and between l and ℓ in the second graph, there must also be an edge in that graph between k and ℓ . Combined with the connectedness, this implies that the second graph is complete, so that α ( z k , z l ) ≥ ˆ H -M for all distinct k, l ∈ u . In particular α ( z i , z j ) ≥ ˆ H -M , and since m ( z i , z j ) = α ( z i , z j ) , we find

Upper bound on m ( z i , z j ) -ˆ m ( i, j ) . Let S v = { i ∈ [ n ] : z i = v or z i has ancestor v in D} . For k, l ∈ S v , lemma 1 tells us α ( z k , z l ) is the merge height between z k and z l , so α ( z k , z l ) ≥ H . Using (10), we therefore have

It follows from the definition of ˆ h in the algorithm that if S v = [ n ] , the heights of all vertices in ˆ D are greater than or equal to H -M . This implies ˆ H ≥ H -M . In summary, we have shown that when S v = [ n ] ,

It remains to consider the case S v = [ n ] . The proof of the same upper bound (14) in this case is more involved. In summary, we need to establish that the most recent common ancestor of { i } and { j } in ˆ D has height at least H -M . The main idea of the proof is to consider the latest step of the algorithm at which a vertex with height at least H -M is formed by a merge, and show the partition formed by this merge contains the most recent common ancestor of { i } and { j } , or an ancestor thereof.

̸

To this end let m ∗ denote the latest step in algorithm 1 at which the vertex formed, w ∗ , has height greater than or equal to H -M . To see that m ∗ must exist, notice

̸

by definition of M in (10). Combined with the definition of ˆ h in algorithm 1, the vertex formed by the merge at step 1 of the algorithm therefore has height greater than or equal to H -M . Therefore m ∗ is indeed well-defined.

Our next objective is to show that the partition P m ∗ formed at step m ⋆ contains an element which itself contains both i and j . We proceed by establishing some facts about S v and P m ∗ .

Let ¯ S v := [ n ] \ S v . For k ∈ S v , l ∈ ¯ S v , v cannot be an ancestor of z l , by lemma 1 α ( z k , z l ) is the merge height of z k and z l , and b is the minimum branch length in D , so we have α ( z k , z l ) ≤ H -b . From (10) we then find

We claim that no element of P m ∗ can contain both an element of S v and an element of ¯ S v . We prove this claim by contradiction. If such an element of P m ∗ did exist, there would be a step m ′ ≤ m ∗ at which some A ′ ⊆ S v is merged with some B ′ ⊆ ¯ S v . But the vertex w ′ formed by this merge would be assigned height ˆ h ( w ′ ) = ˆ α ( A ′ , B ′ ) ≤ H -b + M < H -M , where the first inequality uses lemma 4 and (16), and the second inequality uses the assumption of the theorem that M < b/ 2 . Recalling the definition of w ⋆ we have ˆ h ( w ∗ ) ≥ H -M . We therefore see that w ∗ is an ancestor of w ′ with ˆ h ( w ∗ ) > ˆ h ( w ′ ) , contradicting the result of proposition 1.

̸

Consider the elements of P m ∗ , denoted A and B , which contain i and j respectively. We claim that A = B . We prove this claim by contradiction. Suppose A = B . As established in the previous paragraph, neither A nor B can contain an element of ¯ S v . Therefore, using lemma 4 and (13),

̸

Again using the established fact that no element of P m ∗ can contain both an element of S v and an element of ¯ S v , m ∗ cannot be the final step of the algorithm, since that would require P m ⋆ = { [ n ] } . Therefore ˆ α ( A,B ) is one of the affinities which algorithm 1 would maximise over at step m ∗ +1 , so the height of the vertex formed by a merge at step m ∗ +1 would be greater than or equal to H -M , which contradicts the definition of m ∗ . Thus we have proved there exists an element of P m ∗ which contains both i and j . This element must be the most recent common ancestor of { i } and { j } , or an ancestor thereof. Also, this element must have been formed by a merge at a step less than or equal to m ⋆ and so must have height greater than or equal to H -M . Invoking proposition 1 we have thus established ˆ H ≥ H -M . In summary, in the case S v = [ n ] , we have shown

Combining the lower bound (12) with the upper bounds (14), (17) and the fact that i, j were chosen arbitrarily, completes the proof.

C.2 Supporting material and proof for Theorem 2

Definitions and interpretation for assumptions A3 and A5

We recall the definition of φ -mixing from, e.g., [39]. For a sequence of random variables { ξ j ; j ≥ 1 } , define:

where F j i is the σ -algebra generated by ξ i , . . . , ξ j . Then { ξ j ; j ≥ 1 } is said to be φ -mixing if φ ( k ) ↘ 0 as k →∞ .

To interpret assumption A5 notice

where the first inequality uses the independence of E i and Z i and the second inequality uses the fact that the elements of the vectors E i are i.i.d. Since Y i -X ( Z i ) = S ( Z i ) E i , A5 thus implies E [ ∥ Y i -X ( Z i ) ∥ 2 ] ∈ O ( p ) as p →∞ , which can be viewed as a natural growth rate since p is the dimension of the disturbance vector Y i -X ( Z i ) . In the proof of proposition 2 below, A5 is used in a similar manner to control dot products of the form ⟨ Y i -X i , X j ⟩ and ⟨ Y i -X i , Y j -X j ⟩ .

Proof of Theorem 2. For the first claim of the theorem, proposition 2 combined with the tower property of conditional expectation imply that for any δ > 0 ,

from which (6) follows.

̸

The second claim of the theorem is in essence a corollary to [49][Thm 1]. A little work is needed to map the setting of the present work on to the setting of Whiteley et al. [49][Thm 1]. To see the connection, we endow the finite set Z in the present work with the discrete metric: d Z ( u, v ) := 0 for u = v , and d Z ( v, v ) = 0 . Then ( Z , d Z ) is a compact metric space, and in the setting specified in the statement of theorem 2 where A3 is strengthened to independence, s = p and S ( v ) = σ I p for all v ∈ Z , the variables Y 1 . . . , Y n ; { X ( v ) , v ∈ Z} ; E 1 , . . . , E n exactly follow the Latent Metric Model of Whiteley et al. [49].

Moreover, according to the description in section 2.1, the variables Z 1 , . . . , Z n are i.i.d. according to a probability distribution supported on Z . As in [49], by Mercer's theorem there exists a feature map ϕ : Z → R r associated with this probability distribution, such that ⟨ ϕ ( u ) , ϕ ( v ) ⟩ = α ( u, v ) , for u, v ∈ Z . Here r , as in A6 , is the rank of the matrix with elements α ( u, v ) , which is at most Z .

Theorem 1 of [49] in this context implies there exists a random orthogonal matrix Q ∈ R r × r such that

Consider the bound:

where orthogonality of Q has been used, and the final inequality uses Cauchy-Schwarz and the triangle inequality for the ∥ · ∥ norm. Combining the above estimate with (19), the bound:

and A4 completes the proof of the second claim of the theorem.

Proposition 2. Assume the model in section 2.1 satisfies assumptions A3 -A5 , and let φ and q be as in A3 and A4 . Then there exists a constant C ( q, φ ) depending only on q and φ such that for any δ > 0 ,

where

Proof. Fix any i, j such that 1 ≤ i < j ≤ n . Consider the decomposition:

The proof proceeds by bounding E [ | ∆ k | q | Z 1 , . . . , Z n ] for k = 1 , . . . , 4 . Writing ∆ 1 as

we see that ∆ 1 is a sum p random variables each of which is conditionally mean zero given Z 1 , . . . , Z n . Noting that the two collections of random variables { Z 1 , . . . , Z n } and { X ( v ); v ∈ V} are independent (as per the description of the model in section 2.1), under assumption A3 we may apply a moment inequality for φ -mixing random variables [51][Lemma 1.7] to show that there exists a constant C 1 ( q, φ ) depending only on q, φ such that

where second inequality holds by two applications of Jensen's inequality and q ≥ 2 , and the final inequality uses the fact that for a, b ≥ 0 , ( a + b ) q ≤ 2 q -1 ( a q + b q ) , the Cauchy-Schwartz inequality, and the independence of { Z 1 , . . . , Z n } and { X ( v ); v ∈ V} .

For ∆ 2 , we have

where [ · ] k denotes the k th element of a vector. Since the three collections of random variables, { Z 1 , . . . , Z n } , { X ( v ); v ∈ Z} and { E 1 , . . . , E n } are mutually independent, and the elements of each vector E j ∈ R p are mean zero and independent, we see that given { Z 1 , . . . , Z n } and { X ( v ); v ∈ V} , ∆ 2 is a simple average of conditionally independent and conditionally mean-zero where

random variables. Applying the Marcinkiewicz-Zygmund inequality we find there exists a constant C 2 ( q ) depending only on q such that

Noting that q ≥ 2 and applying Minkowski's inequality to the r.h.s. of (23), then using the independence of { Z 1 , . . . , Z n } , { X ( v ); v ∈ Z} and { E 1 , . . . , E n } and the i.i.d. nature of the elements of the vector E j ,

Substituting into (23) and using the tower property of conditional expectation we obtain:

where the second inequality holds by Jensen's inequality (recall q ≥ 2 ). Since the r.h.s. of (24) does not depend on i or j , the same bound holds with ∆ 2 on the l.h.s. replaced by ∆ 3 .

Turning to ∆ 4 , we have

̸

Noting that i = j , and that the elements of E i and E j are independent, identically distributed, and mean zero, we see that ∆ 4 is a sum of p 2 random variables which are all conditionally mean zero and conditionally independent given Z 1 , . . . , Z n . The Marcinkiewicz-Zygmund inequality gives:

Applying Minkowski's inequality to the r.h.s. of (25),

where the final inequality holds by Jensen's inequality. Substituting back into (25) and using ∥ S ( u ) ⊤ S ( v ) ∥ F ≤ p 1 / 2 ∥ S ( u ) ⊤ S ( v ) ∥ op ≤ p 1 / 2 ∥ S ( u ) ∥ op ∥ S ( v ) ∥ op , we obtain:

Combining (22), (24) and (26) using the fact that for a, b ≥ 0 , ( a + b ) q ≤ 2 q -1 ( a q + b q ) , we find that there exists a constant C ( q, φ ) depending only on q and φ such that

where M ( q, X , E , S ) is defined in the statement of the proposition and is finite by assumptions A4 and A5 . By Markov's inequality, for any δ ≥ 0 ,

and the proof is completed by a union bound:

C.3 Interpretation of merge heights and exact tree recovery

Here we expand on the discussion in section 3.3 and provide further interpretation of merge heights and algorithm 1. In particular our aim is to clarify in what circumstances algorithm 1 will asymptotically correctly recover underlying tree structure. For ease of exposition throughout section C.3 we assume that Z are the leaf vertices of T .

As a preliminary we note the following corollary to theorem 1: assuming b > 0 , if one takes as input to algorithm 1 the true merge heights, i.e. (up to bijective relabelling of leaf vertices)

ˆ α ( · , · ) := m ( · , · ) = α ( · , · ) , where n = |Z| , then theorem 1 implies that algorithm 1 outputs a dendrogram D whose merge heights ˆ m ( · , · ) are equal to m ( · , · ) (up to bijective relabeling over vertices). This clarifies that with knowledge of m ( · , · ) , algorithm 1) constructs a dendrogram which has m ( · , · ) as its merge heights.

We now ask for more: if once again m ( · , · ) is taken as input to algorithm 1, under what conditions is the output tree ˆ T equal to T (upto bijective relabelling of vertices)? We claim this holds when T is a binary tree and that all its non-leaf nodes have different heights. We provide a sketch proof of this claim, since a complete proof involves many tedious and notationally cumbersome details.

To remove the need for repeated considerations of relabelling, suppose T = ( V , E ) is given, then w.l.o.g. relabel the leaf vertices of T as { 1 } , . . . , {|Z|} and relabel each non-leaf vertex to be the union of its children. Thus each vertex is some subset of [ |Z| ] .

Now assume that T is a binary tree and that all its non-leaf nodes have different heights. Note that |V| = 2 |Z| 1 , i.e., there are |Z| 1 non-leaf vertices. The tree T is uniquely characterized by a sequence of partitions ˜ P 0 , . . . , ˜ P |Z|1 where ˜ P 0 := {{ 1 } , . . . , {|Z|}} , and for m = 1 , . . . , |Z| 1 , ˜ P m is constructed from ˜ P m -1 by merging the two elements of ˜ P m -1 whose most recent common ancestor is the m th highest non-leaf vertex (which is uniquely defined since we are assuming no two non-leaf vertices have equal heights).

To see that in this situation algorithm 1, with ˆ α ( · , · ) := m ( · , · ) and n = |Z| as input, performs exact recovery of the tree, i.e., ˆ T = T , it suffices to notice that the sequence of partitions P 0 , . . . , P |Z|1 constructed by algorithm 1 uniquely characterizes ˆ T , and moreover ( ˜ P 0 , . . . , ˜ P |Z|1 ) = ( P 0 , . . . , P |Z|1 ) . The details of this last equality involve simple but tedious substitutions of m ( · , · ) in place of ˆ α ( · , · ) in algorithm 1, so are omitted.

D Further details of numerical experiments and data preparation

All real datasets used are publicly available under the CC0: Public domain license. Further, all experiments were run locally on a laptop with an integrated GPU (Intel UHD Graphics 620).

D.1 Simulated data

For each v ∈ V , X 1 ( v ) , . . . , X p ( v ) are independent and identically distributed Gaussian random variables with:

D.2 20 Newsgroups

The dataset originates from [29], however, the version used is the one available in the Python package 'scikit-learn' [38]. Each document is pre-processed in the following way: generic stopwords and e-mail addresses are removed, and words are lemmatised. The processed documents are then converted into a matrix of TF-IDF features. Labels can be found on the 20 Newsgroups website http://qwone.com/~jason/20Newsgroups/ , but are mainly intuitive from the title of labels, with full stops separating levels of hierarchy. When using PCA a dimension of r = 34 was selected by the method described in appendix A.

The following numerical results complement those in the main part of the paper.

0

2000

4000

6000

8000

10000

12000

p

0.05

0.10

0.15

0.20

0.25

Kendall

b

correlation measure

Y

1:

n

1:

n

D.3 Zebrafish gene counts

These data were collected over a 24-hour period (timestamps available in the data), however, the temporal aspect of the data was ignored when selecting a sub-sample. To process the data, we followed the steps in [47] which are the standard steps in the popular SCANPY [50] - a Python package for analysing single-cell gene-expression data to process the data. This involves filtering out genes that are present in less than 3 cells or are highly variable, taking the logarithm, scaling and regressing out the effects of the total counts per cell. When using PCA a dimension of r = 29 was selected by the method described in appendix A.

D.4 Amazon reviews

The pre-processing of this dataset is similar to that of the Newsgroup data except e-mail addresses are no longer removed. Some data points had a label of 'unknown' in the third level of the hierarchy, these were removed from the dataset. In addition, reviews that are two words or less are not included. When using PCA a dimension of r = 22 was selected by the method described in appendix A.

D.5 S&P 500 stock data

This dataset was found through the paper [8] with the authors code used to process the data. The labels follow the Global Industry Classification Standard and can be found here: [3]. The return on day i of each stock is calculated by

where p op i and p cl i is the respective opening and closing price on day i . When using PCA a dimension of r = 10 was used as selected by the method described in appendix A.

D.6 Additional method comparison

Table 3 reports additional method comparison results, complementing those in table 1 which concerned only the average linkage function (noting UPGMA is equivalent to using the average linkage function with Euclidean distances). In table 3 we also compare to Euclidean and cosine distances paired with complete and single linkage functions. To aid comparison, the first column (average linkage with the dot product) is the same as in table 1. In general, using complete or single linkage performs worse for both Euclidean and cosine distances. The only notable exception being a slight improvement on the simulated dataset.

E Understanding agglomerative clustering with Euclidean or cosine distances in our framework

E.1 Quantifying dissimilarity using Euclidean distance

The first step of many standard variants of agglomerative clustering such as UPGMA and Ward's method is to find and merge the pair of data vectors which are closest to each other in Euclidean distance. From the elementary identity:

we see that, in general, this is not equivalent to finding the pair with largest dot product, because of the presence of the terms ∥ Y i ∥ 2 and ∥ Y j ∥ 2 . For some further insight in to how this relates to our

(a)

(b)

modelling and theoretical analysis framework, it is revealing to the consider the idealised case of choosing to merge by maximising merge height m ( · , · ) versus minimising d ( · , · ) (recall the identities for m and d established in lemma 1). Figure 5 shows two simple scenarios in which geometry of the dendrogram has an impact on whether or not maximising merge height m ( · , · ) is equivalent to minimising d ( · , · ) . From this example we see that, in situations where some branch lengths are disproportionately large, minimising d ( · , · ) will have different results to maximising m ( · , · ) .

UPGMA and Ward's method differ in their linkage functions, and so differ in the clusters they create in practice after their respective first algorithmic steps. UPGMA uses average linkage to combine Euclidean distances, and there does not seem to be a mathematically simple connection between this and algorithm 1, except to say that in general it will return different results. Ward's method merges the pair of clusters which results in the minimum increase in within-cluster variance. When clusters contain equal numbers of samples, this increase is equal to the squared Euclidean distance between the clusters' respective centroids.

E.2 Agglomerative clustering with cosine distance is equivalent to an instance of algorithm 1

The cosine 'distance' between Y i and Y j is:

In table 1 we show results for standard agglomerative clustering using d cos as a measure of dissimilarity, combined with average linkage. At each iteration, this algorithm works by merging the pair of data-vectors/clusters which are closest with respect to d cos ( · , · ) , say u and v merged to form w , with dissimilarities between w and the existing data-vectors/clusters computed according to the average linkage function:

This procedure can be seen to be equivalent to algorithm 1 with input ˆ α ( · , · ) = 1 -d cos ( · , · ) . Indeed maximising 1 -d cos ( · , · ) is clearly equivalent to minimizing d cos ( · , · ) , and with ˆ α ( · , · ) = 1 -d cos ( · , · ) the affinity computation at line 6 of algorithm 1 is:

where on the r.h.s. of the final equality we recognise (28).

E.3 Using cosine similarity as an affinity measure removes multiplicative noise

Building from the algorithmic equivalence identified in section E.2, we now address the theoretical performance of agglomerative clustering with cosine distance in our modelling framework.

Intuitively, cosine distance is used in situations where the magnitudes of data vectors are thought not to convey useful information about dissimilarity. To formalise this idea, we consider a variation of our statistical model from section 2.1 in which:

Z are the leaf vertices of T , |Z| = n , and we take these vertices to be labelled Z = { 1 , . . . , n }

X is as in section 2.1, properties A1 and A2 hold, and it is assumed that for all v ∈ Z , p -1 E [ ∥ X ( v ) ∥ 2 ] = 1 .

the additive model (1) is replaced by a multiplicative noise model:

where γ i > 0 are all strictly positive random scalars, independent of other variables, but otherwise arbitrary.

The interpretation of this model is that the expected square magnitude of the data vector Y i is entirely determined by γ i , indeed we have

where h is as in section 2.1. We note that in this multiplicative model, the random vectors E i and matrices S ( v ) from section 2.1 play no role, and one can view the random variables Z 1 , . . . , Z n as being replaced by constants Z i = i , rather than being i.i.d.

Now define:

Theorem 3. Assume that the model specified in section E.3 satisfies A3 and for some q ≥ 2 , sup j ≥ 1 max v ∈Z E [ | X j ( v ) | 2 q ] < ∞ . Then

̸

Proof. The main ideas of the proof are that the multiplicative factors γ i , γ j in the numerator ˆ α cos ( i, j ) cancel out with those in the denominator, and combined with the condition p -1 E [ ∥ X ( v ) ∥ 2 ] = 1 we may then establish that α ( i, j ) and ˆ α cos ( i, j ) are probabilistically close using similar arguments to those in the proof of proposition 2.

Consider the decomposition:

Applying the Cauchy-Schwartz inequality, and adding and subtracting p -1 / 2 ∥ X ( j ) ∥ and 1 in the final term of this decomposition leads to:

̸

̸

The proof proceeds by arguing that the term (32) is in O P ( n 2 /q / √ p ) , and the terms (33) and (34) are in O P ( n 1 /q / √ p ) , where we note that this asymptotic concerns the limit as n 2 /q / √ p → 0 .

̸

In order to analyse the term (32), let ˜ P denote the probability law of the additive model for Y 1 , . . . , Y n in equation (1) in section 2.1, in the case that S ( v ) = 0 for all v ∈ Z . Let ˜ E denote the associated expectation. Then for any δ > 0 and i, j ∈ [ n ] , i = j ,

The conditional probability on the r.h.s of the final equality can be upper bounded using the inequality (27) in the proof of proposition 2, and combined with the same union bound argument used immediately after (27), this establishes (32) is in O P ( n 2 /q / √ p ) as required.

To show that (33) and (34) are in O P ( n 1 /q / √ p ) , it suffices to show that max i ∈ [ n ] ∣ ∣ p -1 / 2 ∥ X ( i ) ∥ -1 ∣ ∣ is in O P ( n 1 /q / √ p ) . By re-arranging the equality: ( | a | -1)( | a | +1) = | a | 2 -1 , we have

where ∆ j ( i ) := | X j ( i ) | 2 -1 . Thus under the model from section E.3, p -1 ∑ p j =1 ∆ j ( i ) is an average of p mean-zero random variables, and by the same arguments as in the proof of proposition 2 under the mixing assumption A3 and the assumption of the theorem that sup j ≥ 1 max v ∈Z E [ | X j ( v ) | 2 q ] < ∞ , combined with a union bound, we have

Together with (35) this implies max i ∈ [ n ] ∣ ∣ p -1 / 2 ∥ X ( i ) ∥ -1 ∣ ∣ is in O P ( n 1 /q / √ p ) as required, and that completes the proof.

E.4 Limitations of our modelling assumptions and failings of dot-product affinities

As noted in section 5, algorithm 1 is motivated and theoretically justified by our modelling assumptions, laid out in sections 2-3 and E.3. If these assumptions are not well matched to data in practice, then algorithm 1 may not perform well.

The proof of lemma 3 shows that, as a consequence of the conditional independence and martingalelike assumptions, A1 and A2 , α ( u, v ) ≥ 0 for all u, v . By theorems 2 and 3, ˆ α data , ˆ α pca approximate α (at the vertices Z ) under the additive model from section 2.1, and ˆ α cos approximates α under the multiplicative model from section E.3. Therefore if in practice ˆ α data ( i, j ) , ˆ α pca ( i, j ) or ˆ α cos ( i, j ) are found to take non-negligible negative values for some pairs i, j , that is an indication that our modelling assumptions may not be appropriate.

Even if the values taken by ˆ α data ( i, j ) , ˆ α pca ( i, j ) or ˆ α cos ( i, j ) are all positive in practice, there could be other failings of our modelling assumptions. As an academic but revealing example, if instead of (1) or (29), the data were to actually follow a combined additive and multiplicative noise model, i.e.,

then in general neither ˆ α data , ˆ α pca nor ˆ α cos would approximate α as p →∞ .

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nFor agglomerative clustering, this paper proposes a method where the affinity of clusters at each hierarchy is naturally visualized as their height. Section 2 describes its statistical model (a nonparametric model based on first order moments) and the clustering algorithm derived from it (Algorithm 1) is presented. Algorithm 1 is extremely simple (needless to say, this simplicity is a virtue of the proposed method), and is only a slight modification of many of the standard agglomerative clustering methods. However, despite its simplicity, the deep insights of the algorithm are explained in Section 3. Remarkably, very roughly speaking, the affinity of each hierarchical cluster is expressed as the height of its dendrogram, which, under certain assumptions, asymptotically approaches its true value. Finally, Section 4 describes the effectiveness of the proposed algorithm on real data, as well as its limitations, and provides code for follow-up studies. \n\nNote: I was not able to understand the details (techniques in the supplementary material) of some of the theoretical analysis in this paper within the initial review period. For Theorem 1, I was able to follow the outline of its key mathematical induction. For Theorem 2, I have not been able to grasp the details. However, the statements in the text are very reliable, and the authors have made an effort to add intuitive explanations to these theoretical results.\n\nSTRENGTHS:\n- This paper is an excellent combination of a simple methodology and deep insights behind it. Needless to say, methodological simplicity is a virtue of the proposed method at this time.\n- The phenomenon of affinities between clusters at each level of hierarchy appearing visualized as heights on the dendrogram will be the attractive results for a wide range of readers, both theoretically and in terms of application.\n- It is excellent that the theoretical results (Theorem 1 and Theorem 2) are carefully interpreted in Section 3.3 as detailed intuitions.\n- Authors also share the code (Jupyter Notebook) for re-experimentation with the community, which is a great contribution to subsequent researchers.\n- In Section 5, the paper explicitly mentions the slightly stronger assumption of the proposed model and clarifies for what cases (what data) it has a negative impact. This is an essential finding in the development of science and technology.\n\nWEAKNESSES:\nI have not been able to find any convincing weaknesses for this paper in the initial peer review. However, I have some concern about whether the statistical model assumed by the proposed method is causing model speciﬁcation for data with so-called multiple clusters. I have inquired about this concern with the author in the following question. If the concern is resolved, I have not found any major weakness in this paper.\n\nQUESTIONS:\nI am very grateful to authors for sharing very insightful and interesting ideas. I have some questions because this paper contains simple methodology but very surprising (and perhaps intuitively non-trivial) results. These questions are the reason I am not giving this paper a higher score at this time for its overall rating.\nThese may simply be my lack of insight, but perhaps a brief mention in the text or in the supplementary material would make the advantages of the proposed method clearer to some readers.\nI would be very grateful if a response from the authors could clear up some of my questions.\n\n(1) Another aspect of the model assumptions (Section 2.1).\n\nIt is very interesting to note that the only model mentioned in this paper is the chain rule by Equation 3, and everything else can be expressed nonparametrically. However, does Equation 3 imply that \"all data are i.i.d. samples from a single cluster (a single nonparametric distribution that assumes only first order moments)\"? In other words, is there model misspecification occurring for data sets with multiple clusters (e.g., a 2-cluster Gaussian mixture model)? If we use the chaining rule in Equation 3 to marginalize the random variables at each node in turn from the terminal node to the root node of the tree structure, this may be a model in which all data are sampled i.i.d. from some nonparametric distribution with identical mean values (only the first order moments are specified). Is this intuition wrong? If this intuition is correct, then surely the \"PCA does not do anything wrong\" in the second half of Theorem 2 (Equation 9) makes a lot of sense. Conversely, if the proposed statistical model is such that it has multiple clusters (like a Gaussian mixture model, where each cluster tends to gather data at its center), then it is highly counterintuitive that \"still PCA does not do anything bad\". In summary, my question comes down to whether the proposed model assumes a one-cluster i.i.d. model.\n\n(2) The validity of greedily searching for the closest affinity pair in row 3 of Algorithm 1. \n\nFrom the viewpoint of an agglomerative algorithm, it seems quite reasonable to greedily look for the closest affinity pair in line 3 of the algorithm. On the other hand, from the perspective of a statistical model, a probabilistic selection (i.e., Markov chain selection), such as finding a single pair from a categorical distribution weighted by affinity proportions, also seems natural. Does the fact that this row 3 is a greedy selection play an important role in the theoretical analysis in section 3? The actual intent of this question can be attributed to something that is also closely related to question (1), as follows.\nFor example, if the true data distribution is like a Gaussian mixture model with multiple clusters, is it possible to recover the true data distribution (mixture model structure) by, for example, truncating the tree at some hierarchy (resolution) of the tree structure obtained with the proposed model? In other words, would there not be a misspecification problem as a statistical model? My rough intuition tells me that it is not easy to recover the structure of a statistical model in the case of a greedy merge.\nI also know that such a topic is a bit off the author's actual intent for this paper. However, the question arises because I am very curious as to why the statistical model behind the proposed method leads to very good properties (Theorem 1 and Theorem 2) in Algorithm 1.\n\n(3) (Very minor.) Reason why DAG is first introduced. \n\nIs there a reason why when the tree model is introduced in lines 68-69, it is broadly defined as directed acyclic graphs (DAGs) rather than restricted to binary trees?\nDAGs are often used as the structure of Bayesian networks. Are there any indications that the theoretical results of this study for binary trees may be extended to DAGs or Bayesian networks?\n\nFinally, once again, I appreciate your sharing of some very interesting ideas.\n\nLIMITATIONS:\nThe paper carefully questions the weaknesses of the proposed method in Section 5. It is well explained what kind of data and in what cases the proposed method is unlikely to show its true value.\n\nOn another point, I am slightly concerned whether the statistical model in the proposed method implicitly assumes i.i.d. data with a single cluster, as I have asked the authors in my question. However, this concern may be addressed by a response from the authors.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents new analysis and perspectives on one of the most widely used clustering algorithms, hierarchical agglomerative clustering (HAC). In particular, the authors consider the relationship between a particular generative process of data and a dot-product based linkage of HAC. Empirical and theoretical results are presented.\n\nSTRENGTHS:\nThis paper presents an interesting perspective on a widely studied clustering algorithm. \n\nI don't think that the intention of the paper is to yield surprise that for certain kinds of data there might be one linkage function (dot product) that works better than others. Rather I think the intention of the paper is to draw connections between clustering models and HAC.\n\nIn particular, I see the strengths as: \n\n* Connections between the model for latent tree structures in Eq. 1 and Theorem 2 and dot-product based average linkage\n* Connections between models for trees with heights and HAC.\n* Empirical analysis of some of the theoretical scaling properties: Figure 2.\n\nWEAKNESSES:\nI think that the paper could be improved in the following ways: \n\n* W1. I think that more explicit treatment of Eq. (2) and (3) would improve the presentation; e.g., showing where/why these hold under the model in Eq. (1). \n* W2. My understanding is that Lemma 4 & Prop 1 looks quite similar to standard HAC proofs about reducibility? I think it would greatly improvement the treatment of the result to explain the differences / distinctions. Apologies if I have missed something.\n* W3. While the evaluation metric seems quite reasonable, might also be interesting to show results across other metrics such as dendrogram purity or similar metrics. I see this as a minor point though.\n* W4. In the advent of distributed (typically relatively low) dimensional representations from deep neural encoder models. I wonder if the authors could provide perspective on \"theorem 2 says that affinity estimation error vanishes if the dimension p grows faster enough relative to n\". E.g., should we think of this method as appropriate in such circumstances where p is likely in range of 128-1024 or so and n might be in the millions or billions?\n\nQUESTIONS:\n* Q1. How could the presentation be modified to address W1?\n* Q2. (W2) How does reducibility related to the proofs of Lemma 4 / Prop 1?\n* Q3. (W3) did you consider using any other metrics for evaluation? If you were to add another metric to provide a new slice of information, which metrics/measurements would you want to add?\n* Q4. Please see W4\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors discuss a phylogenetic reconstruction problem (I'm not 100% clear on the exact problem, though) and suggest to use the dot product as a measure of similarity (or affinity). In particular, they seem to use the UPGMA algorithm (Alg. 1) where similarity is defined by dot product (scaled by $1/p$).\n\nThe proposed method is shown to recover the underlying tree structure with guaranteed accuracy under reasonable theoretical assumptions. Empirical results indicate good performance in a number of cases where the ground-truth hierarchy is known.\n\nSTRENGTHS:\n- the theoretical guarantees appear very strong\n\nWEAKNESSES:\n- presentation should be improved so that the problem under study is clearer\n- it seems that methodologically, there is no novelty beyond proposing to use the dot product as a similarity measure in the UPGMA method\n- limited set of benchmark methods in the empirical part\n\nQUESTIONS:\nI am familiar with phylogenetics and probabilistic graphical models, but despite reasonable efforts, I fail to be able to comprehend the modeling setup. The authors should clearly explain what the model in Eq. (1) means. In particular, the meaning of the latent variables $Z_i$, and the two mappings, $X(Z_i)$ and $S(Z_i)$, that generate the observed data from the latent variables and the random noise, $E_i$, should be explained.\n\nAnother general thing I was left wondering is the logic behind the dot product as an affinity measure. It is common to use the cosine similarity $\\cos \\theta = \\frac{\\langle x, y\\rangle}{||x|| ||y||}$, where $\\theta$ is the angle between the vectors $x$ and $y$, and $\\langle x, y\\rangle$ is the dot product between them. In other words, the cosine similarity is the dot product scaled by the product of the Euclidean norms of the two vectors. Such scaling can be motivated by noting that without normalization, the dot product has the peculiar property that, for instance, $\\langle Cx, x\\rangle = C \\langle x, x\\rangle$, i.e., for $C>1$, the vectors $Cx$ and $x$ are more similar (in terms of the dot product) to each other than the vectors $x$ and $x$ (so the vector $x$ to itself). For normalized vectors, the dot product and the cosine similarity are identical. (Further, for normalized vectors, the Euclidean distance equals $\\sqrt{2 - 2{\\langle x, y\\rangle}}$ so it too is a monotonic function of the dot product.)  I would appreciate some comments on the motivation of using the dot product as opposed to commonly used metrics such as the Euclidean distance or the cosine similarity, and on when it is likely to be appropriate (and when not).\n\ndetailed comments:\n- I suppose some would prefer to use the $x \\cdot y$ notation for dot product since $\\langle x,y \\rangle$ often denotes the (more general) inner product\n- p. 2: \"distributional properties $\\mathbf{X}$\": should this be \"... of $\\mathbf{X}$\"?\n- p. 7: When you say you compare against the UPGMA method, you should say what distance metric you use in UPGMA. In fact, it would be interesting to see the results of UPGMA (and other methods) based on various distance metrics (dot product, cosine, Euclidean, ...). And indeed, why not include more phylogenetic methods such as Neighbor-Joining (NJ), etc?\n- appendix C, proof of Lemma 2: the second line of the displayed equation should have a factor 2 in front of the dot product term (as in $||x+y||^2 = ||x||^2 + 2 \\langle x,y \\rangle + ||y||^2$)\n\nLIMITATIONS:\nSee above", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies hierarchical agglomerative clustering under a specified generative process for the underlying data vectors. The main focus of the paper is similarity based clustering that deals with inner products between vectors rather than computing pairwise distances that has been studied before.\n\nThe goal of the paper is to provide recovery guarantees for the underlying tree structure. The paper provides such guarantees under their specified generative model and they bound the maximum merge distortion  by the affinity estimation error (see Theorem 1). Moreover, they provide tradeoffs for the estimation error based on the dimension of the data vectors and the sample size. Finally, the authors test their method in terms of runtime and quality against several other methods from the literature.\n\nSTRENGTHS:\n+hierarchical tree recovery is not a well-understood problem so the overall research direction is interesting\n+dot products are used a lot in real-world applications so having analyses based on them is important\n+the flavor of the guarantees seem to be towards the right direction for what we would mean \"hierarchical tree recovery\"\n+the technical aspects of the proofs\n\nWEAKNESSES:\n-The generative model should be better explained and compared to other previously studied models for hierarchical clustering. In particular, it was not clear to me how to think of your model given that Hierarchical Stochastic Block Models and well-clusterable graphs are defined and seem more natural to me, see e.g. papers below:\n\nCohen-Addad, Kanade, Mallmann-Trenn, Mathieu: \"Hierarchical clustering: Objective functions and algorithms\" \nChatziafratis, Mahdian, Ahmadian: \"Maximizing Agreements for Ranking, Clustering and Hierarchical Clustering via MAX-CUT\"\nManghiuc, Sun: \"HierarchicalClustering: O(1)-Approximation for Well-Clustered Graphs\" \n\nSo overall, having a better exposition and examples for why this particular model you proposed is a natural one, would help the readers a lot.\n\n-The difference from other papers that focus on distance-based hierarchical clustering is important for the mathematics, but conceptually the algorithm is very related. So here the innovation in terms of conceptual contribution is slightly weakened. However, from technical viewpoint there are interesting ideas the paper introduces.\n\n-In the statement of the Theorems 1 and 2, I missed some interpretaion of the results and especially some comparison to related work on distance-based methods. Do we learn something interesting by this dot-product analysis? How are the dimension/sample/estimation tradeoffs different that those other papers?\n\n-The numerical experiments: I am not sure they convey the benefits of your algorithm in terms of estimation error and recovery. In Table 1, I would clarify what are the most significant advantages of your algorithm as currently it is hard to grasp.\n\nQUESTIONS:\nFrom above: \nQ1: Do we learn something interesting by this dot-product analysis and how are the dimension/sample/estimation tradeoffs different that those other papers that did distance-based analyses?\nQ2: Perhaps easy, but giving more evidence to motivate your particular hierarchical tree model would be appreciated.\nQ3: Technical question: Perhaps to help illustrate some of your techniques/ideas, it would be helpful to instantiate your model to a small-depth hierarchy. Say there are only 2 or 3 levels like you had in some of your experimental datasets (S&P500, 20Newsgroups etc). Can your theorems be more interpretable in this scenario?\n\nLIMITATIONS:\nI like the paper and I find that the authors addressing the concerns mentioned above will significantly improve their paper.\nI don't find any major limitations, apart from better motivating their model, comparing it with the existing ones, and better illustrating their Theorems and what we learn from them (in comparison to distance-based methods).", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper discusses a new perspective on hierarchical clustering using the dot product as similarity measure instead of some distance. Under mild conditions on the probabilistic graphical model that generates the data, the proposed algorithm is shown to faithfully recover the underlying tree geometry. Surprisingly, the theoretical results show that the performance of the approach does not suffer, but instead benefits from high-dimensional data.\n\nSTRENGTHS:\nThe paper is well written and all theoretical results and definitions are accompanied by intuitive explanations (e.g., lines 84-90, lines 159-166, Fig. 1, or Sec. 3.3). This makes the paper accessible to an audience not versed in graphical models, for example. The results seem quite promising; it is noteworthy that the authors also included \"negative\" results for the S&P500 dataset, and that they made an effort in investigating the limitations of the proposed method.\n\nWEAKNESSES:\nI see very little weaknesses in the paper (see below for a list of typos). There are only few points unclear (see Questions below). The only concern I have is that the results claim that the performance of the method improves if the dimensionality of the data increases. This is probably caused by the mixing condition, which ensures that more dimensions deliver more relevant information. This assumption is quite questionable in my opinion, as usually more dimensions do not reveal substantially more information about the underlying distribution. As an example, consider natural images: going from low resolutions to higher resolutions may at first improve clustering performance, but at some point increasing the resolution will bring only diminishing returns. The reason is that, as the resolution increases, also the correlation between dimensions increases, which is in contrast to A1. I would appreciate seeing results based on, e.g., lower numbers of TF-IDF features to understand how this affects clustering performance.\n\nAnother concern is that Fig. 3a and 3c seem to be inconsistent. In 3a, at the position of comp.windows.x, there are three dark blue squares, one light blue square, and (maybe?) a hidden pink x. In 3c, the top five topics contain two dark blue and two light blue squares. Please check if there was a mistake in generating the figure.\n\n## Minor:\n- line 56: \"interpretation to in all...\"\n- line 78: \"distributional properties OF $\\mathbf{X}$\"\n\nQUESTIONS:\n- In line 102, it is not clear why $\\mathbb{E}[< Y_i, Y_j>|Z_1,\\dots,Z_n]$ is conditioned on all $Z_n$\n- What is the meaning of the operator $O_{\\mathbb{P}}$?\n- In Table 1, the results for simulated data do not depend on whether PCA was applied or not. Is this caused by the fact that the dimensions are independent?\n- Since the PCA version of the algorithm operates on fixed dimension $r$, it is not clear to my why in (9) an increase of $p$ should bring any benefit. Could you expand on that?\n\nLIMITATIONS:\nThe authors clearly outline the limitations of their study, which is highly appreciated. A possible further limitation of the method is that, apparently, good performance guarantees can only be given for high-dimensional data, which is slightly counterintuitive. Societal implications are not to be expected.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-19"}