SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Annie Gray
School of Mathematics
University of Bristol, UK
annie(gray@bristol.ac.uk
Alexander Modell
Imperial College London, UK
a.modell@imperial.ac.uk
Patrick Rubin-Delanchy
patrick.rubin-delanchy@bristol.ac.uk
Nick Whiteley
nick.whiteley@bristol.ac.uk
In this paper we offer a new perspective on the well established agglomerative clustering algorithm, focusing on recovery of hierarchical structure. We recommend a simple variant of the standard algorithm, in which clusters are merged by maximum average dot product and not, for example, by minimum distance or within-cluster variance. We demonstrate that the tree output by this algorithm provides a bona fide estimate of generative hierarchical structure in data, under a generic probabilistic graphical model. The key technical innovations are to understand how hierarchical information in this model translates into tree geometry which can be recovered from data, and to characterise the benefits of simultaneously growing sample size and data dimension. We demonstrate superior tree recovery performance with real data over existing approaches such as UPGMA, Ward's method, and HDBSCAN.
Hierarchical structure is known to occur in many natural and man-made systems [32], and the problem considered in this paper is how to recover this structure from data. Hierarchical clustering algorithms, [25, 36, 37, 41] are very popular techniques which organise data into nested clusters, used routinely by data scientists and machine learning researchers, and are easily accessible through open source software packages such as scikit-learn [38]. We focus on perhaps the most popular family of such techniques: agglomerative clustering [17], [24, Ch.3], in which clusters of data points are merged recursively.
Agglomerative clustering methods are not model-based procedures, but rather simple algorithms. Nevertheless, in this work we uncover a new perspective on agglomerative clustering by introducing a general form of generative statistical model for the data, but without assuming specific parametric families of distributions (e.g., Gaussian). In our model (section 2.1), hierarchy takes the form of a tree defining the conditional independence structure of latent variables, using elementary concepts from probabilistic graphical modelling [30, 28]. In a key innovation, we then augment this conditional independence tree to form what we call a dendrogram, whose geometry is related to population statistics of the data. The new insight which enables tree recovery in our setting (made precise and explained in section 3) is that dot products between data vectors reveal heights of most recent common ancestors in the dendrogram.
We suggest an agglomerative algorithm which merges clusters according to highest sample average dot product (section 2.2). This is in contrast to many existing approaches which quantify dissimilarity
between data vectors using Euclidean distance. We also consider the case where data are preprocessed by reducing dimension using PCA. We mathematically analyse the performance of our dot product clustering algorithm and establish that under our model, with sample size  $n$  and data dimension  $p$  growing simultaneously at appropriate rates, the merge distortion [20, 27] between the algorithm output and underlying tree vanishes. In numerical examples with real data (section 4), where we have access to labels providing a notion of true hierarchy, we compare performance of our algorithm against existing methods in terms of a Kendall  $\tau_{b}$  correlation performance measure, which quantifies association between ground-truth and estimated tree structure. We examine statistical performance with and without dimension reduction by PCA, and illustrate how dot products versus Euclidean distances relate to semantic structure in ground-truth hierarchy.
Related work Agglomerative clustering methods combine some dissimilarity measure with a 'linkage' function, determining which clusters are combined. Popular special cases include UPGMA [44] and Ward's method [48], against which we make numerical comparisons (section 4). Owing to the simple observation that, in general, finding the largest dot product between data vectors is not equivalent to finding the smallest Euclidean distance, we can explain why these existing methods may not correctly recover tree structure under our model (appendix E). Popular density-based clustering methods include CURE [22], OPTICS [7], BIRCH [52] and HDBSCAN [9]. In section 4 we discuss the extent to which these methods can and cannot be compared against ours in terms of tree recovery performance.
Existing theoretical treatments of hierarchical clustering involve different mathematical problem formulations and assumptions to ours. One common setup is to assume an underlying ultrametric space whose geometry specifies the unknown tree, and/or to study tree recovery as  $n \to \infty$  with respect to a cost function, e.g., [10, 16, 43, 11, 14, 15, 31, 12]. An alternative problem formulation addresses recovery of the cluster tree of the probability density from which it is assumed data are sampled [42, 20, 27]. The unknown tree in our problem formulation specifies conditional independence structure, and so it has a very different interpretation to the trees in all these cited works. Moreover, our data are vectors in  $\mathbb{R}^p$ , and  $p \to \infty$  is a crucial aspect of our convergence arguments, but in the above cited works  $p$  plays no role or is fixed. Our definition of dendrogram is different to that in e.g. [10]: we do not require all leaf vertices to be equidistant from the root - a condition which arises as the "fixed molecular clock" hypothesis [18, 34] in phylogenetics. We also allow data to be sampled from non-leaf vertices. There is an enormous body of work on tree reconstruction methods in phylogenetics, e.g. listed at [4], but these are mostly not general-purpose solutions to the problem of inferring hierarchy. Associated theoretical convergence results are limited in scope, e.g., the famous work [18] is limited to a fixed molecular clock, five taxa and does not allow observation error.
2 Model and algorithm
2.1 Statistical model, tree and dendrogram
Where possible, we use conventional terminology from the field of probabilistic graphical models, e.g., [28] to define our objects and concepts. Our model is built around an unobserved tree  $\mathcal{T} = (\mathcal{V},\mathcal{E})$  that is a directed acyclic graph with vertex and edge sets  $\mathcal{V}$  and  $\mathcal{E}$ , with two properties:  $\mathcal{T}$  is connected (ignoring directions of edges), and each vertex has at most one parent, where we say  $u$  is a parent of  $v$  if there is an edge from  $u$  to  $v$ . We observe data vectors  $\mathbf{Y}_i\in \mathbb{R}^p,i = 1,\dots ,n$ , which we model as:
$$
\mathbf {Y} _ {i} = \mathbf {X} \left(Z _ {i}\right) + \mathbf {S} \left(Z _ {i}\right) \mathbf {E} _ {i}, \tag {1}
$$
comprising three independent sources of randomness:
-  $Z_{1}, \ldots, Z_{n}$  are i.i.d., discrete random variables, with distribution supported on a subset of vertices  $\mathcal{Z} \subseteq \mathcal{V}$ ,  $|\mathcal{Z}| < \infty$ ;
-  $\mathbf{X}(v) \coloneqq [X_1(v) \cdots X_p(v)]^\top$  is an  $\mathbb{R}^p$ -valued random vector for each vertex  $v \in \mathcal{V}$ ;
-  $\mathbf{E}_1, \ldots, \mathbf{E}_n$  are i.i.d,  $\mathbb{R}^p$ -valued random vectors. The elements of  $\mathbf{E}_i$  are i.i.d., zero mean and unit variance. For each  $z \in \mathcal{Z}$ ,  $\mathbf{S}(z) \in \mathbb{R}^{p \times p}$ , is a deterministic matrix.
For each  $v \in \mathcal{Z}$ , one can think of the vector  $\mathbf{X}(v)$  as the random centre of a "cluster", with correlation structure of the cluster determined by the matrix  $\mathbf{S}(z)$ . The latent variable  $Z_{i}$  indicates which cluster
the  $i$ th data vector  $\mathbf{Y}_i$  is associated with. The vectors  $\mathbf{X}(v), v \in \mathcal{V} \setminus \mathcal{Z}$ , correspond to unobserved vertices in the underlying tree. As an example,  $\mathcal{Z}$  could be the set of leaf vertices of  $\mathcal{T}$ , but neither our methods nor theory require this to be the case. Throughout this paper, we assume that the tree  $\mathcal{T}$  determines two distributional properties of  $\mathbf{X}$ . Firstly, we assume  $\mathcal{T}$  is the conditional independence graph of the collection of random variables  $X_j := \{X_j(v); v \in \mathcal{V}\}$  for each  $j$ , that is,
A1. for all  $j = 1, \dots, p$ , the marginal probability density or mass function of  $X_{j}$  factorises as:
$$
p (x _ {j}) = \prod_ {v \in \mathcal {V}} p \left(x _ {j} (v) | x _ {j} (\mathrm {P a} _ {v})\right),
$$
where  $\mathrm{P}\mathfrak{a}_v$  denotes the parent of vertex  $v$ .
However we do not necessarily require that  $X_{1},\ldots ,X_{p}$  are independent or identically distributed. Secondly, we assume
A2. for each  $j = 1 \ldots, p$ , the following martingale-like property holds:
$$
\mathbb {E} \left[ X _ {j} (v) \mid X _ {j} \left(\mathrm {P a} _ {v}\right) \right] = X _ {j} \left(\mathrm {P a} _ {v}\right),
$$
for all vertices  $v\in \mathcal{V}$  except the root.
The conditions A1 and A2 induce an additive hierarchical structure in  $\mathbf{X}$ . For any distinct vertices  $u, v \in \mathcal{V}$  and  $w$  an ancestor of both, A1 and A2 imply that given  $X_{j}(w)$ , the increments  $X_{j}(u) - X_{j}(w)$  and  $X_{j}(v) - X_{j}(w)$  are conditionally independent and both conditionally mean zero.
To explain our algorithm we need to introduce the definition of a dendrogram,  $\mathcal{D} = (\mathcal{T},h)$ , where  $h:\mathcal{V}\to \mathbb{R}_+$  is a function which assigns a height to each vertex of  $\mathcal{T}$ , such that  $h(v)\geq h(\mathrm{Pa}_v)$  for any vertex  $v\in \mathcal{V}$  other than the root. The term "dendrogram" is derived from the ancient Greek for "tree" and "drawing", and indeed the numerical values  $h(v)$ ,  $v\in \mathcal{V}$ , can be used to construct a drawing of  $\mathcal{T}$  where height is measured with respect to some arbitrary baseline on the page, an example is shown in figure 1(a). With  $\langle \cdot ,\cdot \rangle$  denoting the usual dot product between vectors, the function
$$
\alpha (u, v) := \frac {1}{p} \mathbb {E} \left[ \langle \mathbf {X} (u), \mathbf {X} (v) \rangle \right], \quad u, v \in \mathcal {V}, \tag {2}
$$
will act as a measure of affinity underlying our algorithm. The specific height function we consider is  $h(v) \coloneqq \alpha(v, v)$ . The martingale property A2 ensures that this height function satisfies  $h(v) \geq h(\mathrm{Pa}_v)$  as required, see lemma 2 in appendix C. In appendix E we also consider cosine similarity as an affinity measure, and analyse its performance under a multiplicative noise model, cf. the additive model (1).
Combining the model (1) with (2) we have  $\alpha(Z_i, Z_j) = p^{-1} \mathbb{E}[\langle \mathbf{Y}_i, \mathbf{Y}_j \rangle | Z_i, Z_j]$  for  $i \neq j \in [n]$ ,  $[n] := \{1, \ldots, n\}$ . The input to algorithm 1 is an estimate  $\hat{\alpha}(\cdot, \cdot)$  of all the pairwise affinities  $\alpha(Z_i, Z_j)$ ,  $i \neq j \in [n]$ . We consider two approaches to estimating  $\alpha(Z_i, Z_j)$ , with and without dimension reduction by uncentered PCA. For some  $r \leq \min\{p, n\}$ , let  $\mathbf{V} \in \mathbb{R}^{p \times r}$  denote the matrix whose columns are orthonormal eigenvectors of  $\sum_{i=1}^{n} \mathbf{Y}_i \mathbf{Y}_i^\top$  associated with its  $r$  largest eigenvalues. Then  $\zeta_i := \mathbf{V}^\top \mathbf{Y}_i$  is  $r$ -dimensional vector of principal component scores for the  $i$ th data vector. The two possibilities for  $\hat{\alpha}(\cdot, \cdot)$  we consider are:
$$
\hat {\alpha} _ {\mathrm {d a t a}} (i, j) := \frac {1}{p} \langle \mathbf {Y} _ {i}, \mathbf {Y} _ {j} \rangle , \quad \hat {\alpha} _ {\mathrm {p c a}} (i, j) := \frac {1}{p} \langle \zeta_ {i}, \zeta_ {j} \rangle . \tag {3}
$$
In the case of  $\hat{\alpha}_{\mathrm{pca}}$  the dimension  $r$  must be chosen. Our theory in section 3.2 assumes that  $r$  is chosen as the rank of the matrix with entries  $\alpha(u, v), u, v \in \mathcal{Z}$ , which is at most  $|\mathcal{Z}|$ . In practice  $r$  usually must be chosen based on the data, we discuss this in appendix A.
Algorithm 1 returns a dendrogram  $\hat{\mathcal{D}} = (\hat{T},\hat{h})$ , comprising a tree  $\hat{\mathcal{T}} = (\hat{\mathcal{V}},\hat{\mathcal{E}})$  and height function  $\hat{h}$ . Each vertex in  $\hat{\mathcal{V}}$  is a subset of  $[n]$ , thus indexing a subset of the data vectors  $\mathbf{Y}_1,\ldots ,\mathbf{Y}_n$ . The leaf vertices are the singleton sets  $\{i\}$ ,  $i\in [n]$ , corresponding to the data vectors themselves. As algorithm 1 proceeds, vertices are appended to  $\hat{\mathcal{V}}$ , edges are appended to  $\hat{\mathcal{E}}$ , and the domain of the function  $\hat{\alpha} (\cdot ,\cdot)$  is extended as affinities between elements of  $\hat{\mathcal{V}}$  are computed. Throughout the paper we simplify notation by writing  $\hat{\alpha} (i,j)$  as shorthand for  $\hat{\alpha} (\{i\} ,\{j\})$  for  $i,j\in [n]$ , noting that each argument of  $\hat{\alpha} (\cdot ,\cdot)$  is in fact a subset of  $[n]$ .
(c)
Figure 1: (a) An example of the dendrogram  $\mathcal{D}$  with  $\mathcal{V} = \{a, b, c, d, e, f, g\}$ , see lemma 1 for interpretation of the merge height  $m(\cdot, \cdot)$  and distance  $d(\cdot, \cdot)$ . Horizontal distances in this diagram are chosen arbitrarily. (b)  $\mathcal{D}$  augmented according to  $\mathcal{Z} = \{a, b, c, d\}$  and the realization:  $Z_1, Z_2, Z_3 = a, Z_4 = b, Z_5 = Z_6 = d$ , see section 3.3 for discussion. (c) The dendrogram  $\hat{\mathcal{D}}$  output from algorithm 1 in the case  $\hat{\alpha} = \hat{\alpha}_{\mathrm{data}}$ .  $\hat{\mathcal{D}}$  can be seen to approximate the dendrogram in (b) and hence  $\mathcal{D}$ .
Algorithm 1 Dot product hierarchical clustering
Input: pairwise affinities  $\hat{\alpha} (\cdot ,\cdot)$  between  $n$  data points
1: Initialise partition  $P_0 \coloneqq \{\{1\}, \dots, \{n\}\}$ , vertex set  $\hat{\mathcal{V}} \coloneqq P_0$  and edge set  $\hat{\mathcal{E}}$  to the empty set.
2: for  $m \in \{1, \dots, n - 1\}$  do
3: Find distinct pair  $u, v \in P_{m-1}$  with largest affinity
4: Update  $P_{m - 1}$  to  $P_{m}$  by merging  $u,v$  to form  $w\coloneqq u\cup v$
5: Append vertex  $w$  to  $\hat{\mathcal{V}}$  and directed edges  $w \to u$  and  $w \to v$  to  $\hat{\mathcal{E}}$
6: Define affinity between  $w$  and other members of  $P_{m}$  as
$$
\hat {\alpha} (w, \cdot) := \frac {| u |}{| w |} \hat {\alpha} (u, \cdot) + \frac {| v |}{| w |} \hat {\alpha} (v, \cdot),
$$
and  $\hat{\alpha} (w,w)\coloneqq \hat{\alpha} (u,v)$
Output: Dendrogram  $\hat{\mathcal{D}} \coloneqq (\hat{\mathcal{T}}, \hat{h})$ , comprising tree  $\hat{\mathcal{T}} = (\hat{\mathcal{V}}, \hat{\mathcal{E}})$  and heights  $\hat{h}(v) \coloneqq \hat{\alpha}(v, v)$  for  $v \in \hat{\mathcal{V}} \setminus P_0$ , and  $\hat{h}(v) \coloneqq \max \{\hat{h}(\mathrm{Pa}_v), \hat{\alpha}(v, v)\}$  for  $v \in P_0$ .
Implementation using scikit-learn The presentation of algorithm 1 has been chosen to simplify its theoretical analysis, but alternative formulations of the same method may be much more computationally efficient in practice. In appendix B we outline how algorithm 1 can easily be implemented using the AgglomerativeClustering class in scikit-learn [38].
3 Performance Analysis
3.1 Merge distortion is upper bounded by affinity estimation error
In order to explain the performance of algorithm 1 we introduce the merge height functions:
$$
m (u, v) := h (\text {m o s t r e c e n t c o m m o n a n c e s t o r o f} u \text {a n d} v), \quad u, v \in \mathcal {V},
$$
$$
\hat {m} (u, v) := \hat {h} (\text {m o s t r e c e n t c o m m o n a n c e s t o r o f u a n d v}), \quad u, v \in \hat {\mathcal {V}}.
$$
To simplify notation we write  $\hat{m}(i,j)$  as shorthand for  $\hat{m}(\{i\}, \{j\})$ , for  $i,j \in [n]$ . The discrepancy between any two dendrograms whose vertices are in correspondence can be quantified by merge distortion [20] – the maximum absolute difference in merge height across all corresponding pairs of
vertices. [20] advocated merge distortion as a performance measure for cluster-tree recovery, which is different to our model-based formulation, but merge distortion turns out to be a useful and tractable performance measure in our setting too. As a preface to our main theoretical results, lemma 1 explains how the geometry of the dendrogram  $\mathcal{D}$ , in terms of merge heights, is related to population statistics of our model. Defining  $d(u,v) \coloneqq h(u) - h(w) + h(v) - h(w)$  for  $u \neq v \in \mathcal{V}$ , where  $w$  is the most recent common ancestor of  $u$  and  $v$ , we see  $d(u,v)$  is the vertical distance on the dendrogram from  $u$  down to  $w$  then back up to  $v$ , as illustrated in figure 1(a).
Lemma 1. For any two vertices  $u, v \in \mathcal{V}$ ,
$$
m (u, v) = \frac {1}{p} \mathbb {E} [ \langle \mathbf {X} (u), \mathbf {X} (v) \rangle ] = \alpha (u, v), \quad d (u, v) = \frac {1}{p} \mathbb {E} [ \| \mathbf {X} (u) - \mathbf {X} (v) \| ^ {2} ]. \tag {4}
$$
The proof is in appendix C. Considering the first two equalities in (4), it is natural to ask if the estimated affinities  $\hat{\alpha}(\cdot, \cdot)$  being close to the true affinities  $\alpha(\cdot, \cdot)$  implies a small merge distortion between  $\hat{m}(\cdot, \cdot)$  and  $m(\cdot, \cdot)$ . This is the subject of our first main result, theorem 1 below. In appendix E we use the third equality in (4) to explain why popular agglomerative techniques such as UPGMA [44] and Ward's method [48] which merge clusters based on proximity in Euclidean distance may enjoy limited success under our model, but in general do not correctly recover tree structure.
Let  $b$  denote the minimum branch length of  $\mathcal{D}$ , that is,  $b = \min \{h(v) - h(\mathrm{Pa}_v)\}$ , where the minimum is taken over all vertices in  $\mathcal{V}$  except the root.
Theorem 1. Let the function  $\hat{\alpha}(\cdot, \cdot)$  given as input to algorithm 1 be real-valued and symmetric but otherwise arbitrary. For any  $z_1, \ldots, z_n \in \mathcal{Z}$ , if
$$
\max_{\substack{i,j\in [n],i\neq j}}|\alpha (z_{i},z_{j}) - \hat{\alpha} (i,j)| <   b / 2,
$$
then the dendrogram returned by algorithm 1 satisfies
$$
\max  _ {i, j \in [ n ], i \neq j} | m (z _ {i}, z _ {j}) - \hat {m} (i, j) | \leq \max  _ {i, j \in [ n ], i \neq j} | \alpha (z _ {i}, z _ {j}) - \hat {\alpha} (i, j) |. \tag {5}
$$
The proof is in appendix C.
3.2 Affinity estimation error vanishes with increasing dimension and sample size
Our second main result, theorem 2 below, concerns the accuracy of estimating the affinities  $\alpha (\cdot ,\cdot)$  using  $\hat{\alpha}_{\mathrm{data}}$  or  $\hat{\alpha}_{\mathrm{pca}}$  as defined in (3). We shall consider the following technical assumptions.
A3 (Mixing across dimensions). For mixing coefficients  $\varphi$  satisfying  $\sum_{k\geq 1}\varphi^{1 / 2}(k) < \infty$  and all  $u,v\in \mathcal{Z}$ , the sequence  $\{(X_j(u),X_j(v));j\geq 1\}$  is  $\varphi$ -mixing.
A4 (Bounded moments). For some  $q \geq 2$ ,  $\sup_{j \geq 1} \max_{v \in \mathcal{Z}} \mathbb{E}[|X_j(v)|^{2q}] < \infty$  and  $\mathbb{E}[|\mathbf{E}_{11}|^{2q}] < \infty$ , where  $\mathbf{E}_{11}$  is the first element of the vector  $\mathbf{E}_1$ .
A5 (Disturbance control).  $\max_{v\in \mathcal{Z}}\| \mathbf{S}(v)\|_{\mathrm{op}}\in O(1)$  as  $p\to \infty$  , where  $\| \cdot \|_{\mathrm{op}}$  is the spectral norm.
A6 (PCA rank). The dimension  $r$  chosen in definition of  $\hat{\alpha}_{\mathrm{pca}}$ , see (3), is equal to the rank of the matrix with entries  $\alpha(u, v)$ ,  $u, v \in \mathcal{Z}$ .
The concept of  $\varphi$ -mixing is a classical weak-dependence condition, e.g. [19, 39]. A3 implies that for each  $j \geq 1$ ,  $(X_{j}(u), X_{j}(v))$  and  $(X_{j + \delta}(u), X_{j + \delta}(v))$  are asymptotically independent as  $\delta \to \infty$ . However, it is important to note that  $\hat{\alpha}_{\mathrm{data}}$  and  $\hat{\alpha}_{\mathrm{pca}}$  in (3), and hence the operation of algorithm 1 with these inputs, are invariant to permutation of the data dimensions  $j = 1, \dots, p$ . Thus our analysis under A3 only requires there is some permutation of dimensions under which  $\varphi$ -mixing holds. A4 is a fairly mild integrability condition. A5 allows control of magnitudes of the "disturbance" vectors  $\mathbf{Y}_i - \mathbf{X}(Z_i) = \mathbf{S}(Z_i)\mathbf{E}_i$ . Further background and discussion of assumptions is given in appendix C.2.
Theorem 2. Assume that A1-A5 hold and let  $q$  be as in A4. Then
$$
\max  _ {i, j \in [ n ], i \neq j} | \alpha (Z _ {i}, Z _ {j}) - \hat {\alpha} _ {\text {d a t a}} (i, j) | \in O _ {\mathbb {P}} \left(\frac {n ^ {2 / q}}{\sqrt {p}}\right). \tag {6}
$$
If additionally A3 is strengthened from  $\varphi$ -mixing to independence,  $\mathbf{S}(v) = \sigma \mathbf{I}_p$  for some constant  $\sigma \geq 0$  and all  $v \in \mathcal{Z}$  (in which case A5 holds), and A6 holds, then
$$
\max  _ {i, j \in [ n ], i \neq j} | \alpha \left(Z _ {i}, Z _ {j}\right) - \hat {\alpha} _ {\mathrm {p c a}} (i, j) | \in O _ {\mathbb {P}} \left(\sqrt {\frac {n r}{p}} + \sqrt {\frac {r}{n}}\right). \tag {7}
$$
The proof of theorem 2 is in appendix C.2. We give an original and self-contained proof of (6). To prove (7) we use a recent uniform-concentration result for principal component scores from [49]. Overall, theorem 2 says that affinity estimation error vanishes if the dimension  $p$  grows faster enough relative to  $n$  (and  $r$  in the case of  $\hat{\alpha}_{\mathrm{PCA}}$ , noting that under A6,  $r \leq |\mathcal{Z}|$ , so it is sensible to assume  $r$  is much smaller than  $n$  and  $p$ ). The argument of  $O_{\mathbb{P}}(\cdot)$  is the convergence rate; if  $(X_{p,n})$  is some collection of random variables indexed by  $p$  and  $n$ ,  $X_{p,n} \in O_{\mathbb{P}}(n^{2/q}/\sqrt{p})$  means that for any  $\epsilon > 0$ , there exists  $\delta$  and  $M$  such that  $n^{2/q}/\sqrt{p} \geq M$  implies  $\mathbb{P}(|X_{p,n}| > \delta) < \epsilon$ . We note the convergence rate in (6) is decreasing in  $q$  where as rate in (7) is not. It is an open mathematical question whether (7) can be improved in this regard; sharpening the results of Whiteley et al. [49] used in the proof of (7) seems very challenging. However, when  $q = 2$ , (6) gives  $O_{\mathbb{P}}(n/\sqrt{p})$  compared to  $O_{\mathbb{P}}\left(\sqrt{nr/p} + \sqrt{r/n}\right)$  in (7), i.e., an improvement from  $n$  to  $\sqrt{nr}$  in the first term. We explore empirical performance of  $\hat{\alpha}_{\mathrm{data}}$  versus  $\hat{\alpha}_{\mathrm{PCA}}$  in section 4.2.
3.3 Interpretation
By combining theorems 1 and 2, we find that when  $b > 0$  is constant and  $\hat{\alpha}$  is either  $\hat{\alpha}_{\mathrm{data}}$  or  $\hat{\alpha}_{\mathrm{pca}}$ , the merge distortion
$$
\max  _ {i, j \in [ n ], i \neq j} | m \left(Z _ {i}, Z _ {j}\right) - \hat {m} (i, j) | \tag {8}
$$