SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In the case that  $w$  or  $w'$  is the result of a merge, suppose w.l.o.g. that  $w = u \cup v$  for some  $u, v \in P_m$  and  $w' \in P_m$ . Then by definition of  $\hat{\alpha}$  in algorithm 1,
$$
\begin{array}{l} \hat {\alpha} (w, w ^ {\prime}) = \frac {| u |}{| w |} \hat {\alpha} (u, w ^ {\prime}) + \frac {| v |}{| w |} \hat {\alpha} (v, w ^ {\prime}) \\ = \frac {| u |}{| w |} \frac {1}{| u \| w ^ {\prime} |} \sum_ {i \in u, j \in w ^ {\prime}} \hat {\alpha} (i, j) + \frac {| v |}{| w |} \frac {1}{| v \| w ^ {\prime} |} \sum_ {i \in v, j \in w ^ {\prime}} \hat {\alpha} (i, j) \\ = \frac {1}{| w | | w ^ {\prime} |} \sum_ {i \in w, j \in w ^ {\prime}} \hat {\alpha} (i, j), \\ \end{array}
$$
where the final equality uses  $w = u \cup v$ . The induction hypothesis thus holds at step  $m + 1$ .
The following proposition establishes the validity of the height function constructed in algorithm 1. Some of the arguments used in this proof are qualitatively similar to those used to study reducible linkage functions by, e.g., Sumengen et al. [46], see also historical references therein.
Proposition 1. With  $\hat{\mathcal{V}}$  the vertex set and  $\hat{h}$  the height function constructed in algorithm 1 with any symmetric, real-valued input  $\hat{\alpha}(\cdot, \cdot)$ , it holds that  $\hat{h}(v) \geq \hat{h}(\mathrm{Pa}_v)$  for all vertices  $v \in \hat{\mathcal{V}}$  except the root.
Proof. The required inequality  $\hat{h}(v) \geq \hat{h}(\mathrm{Pa}_v)$  holds immediately for all the leaf vertices  $v \in P_0 = \{\{1\}, \ldots, \{n\}\}$  by the definition of  $\hat{h}$  in algorithm 1. All the remaining vertices in the output tree, i.e., those in  $\hat{\mathcal{V}} \setminus P_0$ , are formed by merges over the course of the algorithm. For  $m \geq 0$  let  $w_m = u_m \cup v_m$  denote the vertex formed by merging some  $u_m, v_m \in P_m$ . Then  $w_m = \mathrm{Pa}_{u_m}$  and  $w_m = \mathrm{Pa}_{v_m}$ . Each  $u_m$  is either a member of  $P_0$  or equal to  $w_{m'}$  for some  $m' < m$ . The same is true of each  $v_m$ . It therefore suffices to show that  $\hat{h}(w_m) \geq \hat{h}(w_{m+1})$  for  $m \geq 0$ , where by definition in the algorithm,  $\hat{h}(w_m) = \hat{\alpha}(u_m, v_m)$ . Also by definition in the algorithm,  $\hat{h}(w_{m+1})$  is the largest pairwise affinity between elements of  $P_{m+1}$ . Our objective therefore is to upper-bound this largest affinity and compare it to  $\hat{h}(w_m) = \hat{\alpha}(u_m, v_m)$ .
The affinity between  $w_{m} = u_{m} \cup v_{m}$  and any other element  $w'$  of  $P_{m+1}$  (which must also be an element of  $P_{m}$ ) is, by definition in the algorithm,
$$
\begin{array}{l} \hat {\alpha} (w _ {m}, w ^ {\prime}) = \frac {| u _ {m} |}{| u _ {m} | + | v _ {m} |} \hat {\alpha} (u _ {m}, w ^ {\prime}) + \frac {| v _ {m} |}{| u _ {m} | + | v _ {m} |} \hat {\alpha} (v _ {m}, w ^ {\prime}) \\ \leq \max  \left\{\hat {\alpha} \left(u _ {m}, w ^ {\prime}\right), \hat {\alpha} \left(v _ {m}, w ^ {\prime}\right) \right\} \\ \leq \hat {\alpha} (u _ {m}, v _ {m}), \\ \end{array}
$$
where the last inequality holds because  $u_{m}, v_{m}$ , by definition, have the largest affinity amongst all elements of  $P_{m}$ . For the same reason, the affinity between any two distinct elements of  $P_{m+1}$  neither of which is  $w_{m}$  (and therefore both of which are elements of  $P_{m}$ ) is upper-bounded by  $\hat{\alpha}(u_{m}, v_{m})$ . We have therefore established  $\hat{h}(w_{m}) = \hat{\alpha}(u_{m}, v_{m}) \geq \hat{h}(w_{m+1})$  as required, and this completes the proof.
Proof of theorem 1. Let us introduce some definitions used throughout the proof.
$$
M := \max  _ {i \neq j} | \hat {\alpha} (i, j) - \alpha \left(z _ {i}, z _ {j}\right) |. \tag {10}
$$
We arbitrarily chose and then fix  $i,j\in [n]$  with  $i\neq j$  , and define
$$
H := m \left(z _ {i}, z _ {j}\right), \quad \hat {H} := \hat {m} (i, j). \tag {11}
$$
Let  $u$  denote the most recent common ancestor of the leaf vertices  $\{i\}$  and  $\{j\}$  in  $\hat{\mathcal{D}}$  and let  $m \geq 1$  denote the step of the algorithm at which  $u$  is created by a merge, that is  $m = \min \{m' \geq 1 : u \in P_{m'}\}$ . We note that by construction,  $u$  is equal to the union of all leaf vertices with ancestor  $u$ , and by definition of  $\hat{h}$  in algorithm  $1 \hat{h}(u) = \hat{H}$ .
Let  $v$  denote the most recent common ancestor of  $z_{i}$  and  $z_{j}$  in  $\mathcal{D}$ , which has height  $h(v) = H$ .
Lower bound on  $m(z_i, z_j) - \hat{m}(i, j)$ . There is no partition of  $u$  into two non-empty sets  $A, B \subseteq [n]$  such that  $\hat{\alpha}(k, l) < \hat{H}$  for all  $k \in A$  and  $l \in B$ . We prove this by contradiction. Suppose that such a partition exists. There must be a step  $m' \leq m$  at which some  $A' \subseteq A$  is merged some  $B' \subseteq B$ . The vertex  $w$  formed by this merge would have height
$$
\begin{array}{l} \hat {h} (w) = \hat {\alpha} \left(A ^ {\prime}, B ^ {\prime}\right) \\ = \frac {1}{\left| A ^ {\prime} \right| \left| B ^ {\prime} \right|} \sum_ {k \in A ^ {\prime}, l \in B ^ {\prime}} \hat {\alpha} (k, l) <   \hat {H} = \hat {h} (u), \\ \end{array}
$$
where the first equality is the definition of  $\hat{h}(w)$  in the algorithm and the second equality holds by lemma 4. However, in this construction  $u$  is an ancestor of  $w$ , and  $\hat{h}(w) < \hat{h}(u)$  therefore contradicts the result of proposition 1.
As a device to be used in the next step of the proof, consider an undirected graph with vertex set  $u$ , in which there is an edge between two vertices  $k$  and  $l$  if and only if  $\hat{\alpha}(k,l) \geq \hat{H}$ . Then, because there is no partition as established above, this graph must be connected. Now consider a second undirected graph, also with vertex set  $u$ , in which there is any edge between two vertices  $k$  and  $l$  if and only if  $\alpha(z_k,z_l) \geq \hat{H} - M$ . Due to the definition of  $M$  in (10), any edge in the first graph is an edge in the second, so the second graph is connected too. Let  $k,l$ , and  $\ell$  be any distinct members of  $u$ . Using the fact established in lemma 1 that  $\alpha(z_k,z_l)$  and  $\alpha(z_l,z_\ell)$  are respectively the merge heights in  $\mathcal{D}$  between  $z_k$  and  $z_l$ , and  $z_l$  and  $z_\ell$ , it can be seen that if there are edges between  $k$  and  $l$  and between  $l$  and  $\ell$  in the second graph, there must also be an edge in that graph between  $k$  and  $\ell$ . Combined with the connectedness, this implies that the second graph is complete, so that  $\alpha(z_k,z_l) \geq \hat{H} - M$  for all distinct  $k,l \in u$ . In particular  $\alpha(z_i,z_j) \geq \hat{H} - M$ , and since  $m(z_i,z_j) = \alpha(z_i,z_j)$ , we find
$$
m \left(z _ {i}, z _ {j}\right) - \hat {m} (i, j) \geq - M. \tag {12}
$$
Upper bound on  $m(z_{i},z_{j}) - \hat{m} (i,j)$ . Let  $S_{v} = \{i\in [n]:z_{i} = v$  or  $z_{i}$  has ancestor  $v$  in  $\mathcal{D} \}$ . For  $k,l\in S_v$ , lemma 1 tells us  $\alpha (z_k,z_l)$  is the merge height between  $z_{k}$  and  $z_{l}$ , so  $\alpha (z_k,z_l)\geq H$ . Using (10), we therefore have
$$
\hat {\alpha} (k, l) \geq H - M, \quad \forall k, l \in S _ {v}. \tag {13}
$$
It follows from the definition of  $\hat{h}$  in the algorithm that if  $S_{v} = [n]$ , the heights of all vertices in  $\hat{\mathcal{D}}$  are greater than or equal to  $H - M$ . This implies  $\hat{H} \geq H - M$ . In summary, we have shown that when  $S_{v} = [n]$ ,
$$
m \left(z _ {i}, z _ {j}\right) - \hat {m} (i, j) \leq M. \tag {14}
$$
It remains to consider the case  $S_v \neq [n]$ . The proof of the same upper bound (14) in this case is more involved. In summary, we need to establish that the most recent common ancestor of  $\{i\}$  and  $\{j\}$  in  $\hat{\mathcal{D}}$  has height at least  $H - M$ . The main idea of the proof is to consider the latest step of the algorithm at which a vertex with height at least  $H - M$  is formed by a merge, and show the partition formed by this merge contains the most recent common ancestor of  $\{i\}$  and  $\{j\}$ , or an ancestor thereof.
To this end let  $m^*$  denote the latest step in algorithm 1 at which the vertex formed,  $w^*$ , has height greater than or equal to  $H - M$ . To see that  $m^*$  must exist, notice
$$
\max  _ {k \neq l \in [ n ]} \hat {\alpha} (k, l) \geq \alpha \left(z _ {i}, z _ {j}\right) - M, \tag {15}
$$
by definition of  $M$  in (10). Combined with the definition of  $\hat{h}$  in algorithm 1, the vertex formed by the merge at step 1 of the algorithm therefore has height greater than or equal to  $H - M$ . Therefore  $m^*$  is indeed well-defined.
Our next objective is to show that the partition  $P_{m^*}$  formed at step  $m^*$  contains an element which itself contains both  $i$  and  $j$ . We proceed by establishing some facts about  $S_v$  and  $P_{m^*}$ .
Let  $\bar{S}_v\coloneqq [n]\setminus S_v$ . For  $k\in S_v$ ,  $l\in \bar{S}_v$ ,  $v$  cannot be an ancestor of  $z_{l}$ , by lemma 1  $\alpha (z_k,z_l)$  is the merge height of  $z_{k}$  and  $z_{l}$ , and  $b$  is the minimum branch length in  $\mathcal{D}$ , so we have  $\alpha (z_k,z_l)\leq H - b$ . From (10) we then find
$$
\hat {\alpha} (k, l) \leq H - b + M, \quad \forall k \in S _ {v}, l \in \bar {S} _ {v}. \tag {16}
$$
We claim that no element of  $P_{m^*}$  can contain both an element of  $S_v$  and an element of  $\bar{S}_v$ . We prove this claim by contradiction. If such an element of  $P_{m^*}$  did exist, there would be a step  $m' \leq m^*$  at which some  $A' \subseteq S_v$  is merged with some  $B' \subseteq \bar{S}_v$ . But the vertex  $w'$  formed by this merge would be assigned height  $\hat{h}(w') = \hat{\alpha}(A', B') \leq H - b + M < H - M$ , where the first inequality uses lemma 4 and (16), and the second inequality uses the assumption of the theorem that  $M < b/2$ . Recalling the definition of  $w^*$  we have  $\hat{h}(w^*) \geq H - M$ . We therefore see that  $w^*$  is an ancestor of  $w'$  with  $\hat{h}(w^*) > \hat{h}(w')$ , contradicting the result of proposition 1.
Consider the elements of  $P_{m^*}$ , denoted  $A$  and  $B$ , which contain  $i$  and  $j$  respectively. We claim that  $A = B$ . We prove this claim by contradiction. Suppose  $A \neq B$ . As established in the previous paragraph, neither  $A$  nor  $B$  can contain an element of  $S_v$ . Therefore, using lemma 4 and (13),
$$
\hat {\alpha} (A, B) = \frac {1}{| A | | B |} \sum_ {k \in A, l \in B} \hat {\alpha} (k, l) \geq H - M.
$$
Again using the established fact that no element of  $P_{m^*}$  can contain both an element of  $S_v$  and an element of  $\bar{S}_v$ ,  $m^*$  cannot be the final step of the algorithm, since that would require  $P_{m^*} = \{[n]\}$ . Therefore  $\hat{\alpha}(A, B)$  is one of the affinities which algorithm 1 would maximise over at step  $m^* + 1$ , so the height of the vertex formed by a merge at step  $m^* + 1$  would be greater than or equal to  $H - M$ , which contradicts the definition of  $m^*$ . Thus we have proved there exists an element of  $P_{m^*}$  which contains both  $i$  and  $j$ . This element must be the most recent common ancestor of  $\{i\}$  and  $\{j\}$ , or an ancestor thereof. Also, this element must have been formed by a merge at a step less than or equal to  $m^*$  and so must have height greater than or equal to  $H - M$ . Invoking proposition 1 we have thus established  $\hat{H} \geq H - M$ . In summary, in the case  $S_v \neq [n]$ , we have shown
$$
m \left(z _ {i}, z _ {j}\right) - \hat {m} (i, j) \leq M. \tag {17}
$$
Combining the lower bound (12) with the upper bounds (14), (17) and the fact that  $i, j$  were chosen arbitrarily, completes the proof.
C.2 Supporting material and proof for Theorem 2
Definitions and interpretation for assumptions A3 and A5
We recall the definition of  $\varphi$ -mixing from, e.g., [39]. For a sequence of random variables  $\{\xi_j; j \geq 1\}$ , define:
$$
\varphi (k):= \sup_{j\geq 1}\sup_{A\in \mathcal{F}_{1}^{j},B\in \mathcal{F}_{j + k}^{\infty},\mathbb{P}(A) > 0}|\mathbb{P}(B|A) - \mathbb{P}(B)|  .
$$
where  $\mathcal{F}_i^j$  is the  $\sigma$ -algebra generated by  $\xi_i, \ldots, \xi_j$ . Then  $\{\xi_j; j \geq 1\}$  is said to be  $\varphi$ -mixing if  $\varphi(k) \searrow 0$  as  $k \to \infty$ .
To interpret assumption A5 notice
$$
\mathbb {E} [ \| \mathbf {S} (Z _ {i}) \mathbf {E} _ {i} \| ^ {2} | Z _ {1}, \ldots , Z _ {n} ] \leq \| \mathbf {S} (Z _ {i}) \| _ {\mathrm {o p}} ^ {2} \mathbb {E} [ \| \mathbf {E} _ {i} \| ^ {2} ] \leq \max _ {v \in \mathcal {Z}} \| \mathbf {S} (v) \| _ {\mathrm {o p}} ^ {2} p \mathbb {E} [ | \mathbf {E} _ {1 1} | ^ {2} ],
$$
where the first inequality uses the independence of  $\mathbf{E}_i$  and  $Z_{i}$  and the second inequality uses the fact that the elements of the vectors  $\mathbf{E}_i$  are i.i.d. Since  $\mathbf{Y}_i - \mathbf{X}(Z_i) = \mathbf{S}(Z_i)\mathbf{E}_i$ , A5 thus implies  $\mathbb{E}[\| \mathbf{Y}_i - \mathbf{X}(Z_i)\| ^2 ]\in O(p)$  as  $p\to \infty$ , which can be viewed as a natural growth rate since  $p$  is the dimension of the disturbance vector  $\mathbf{Y}_i - \mathbf{X}(Z_i)$ . In the proof of proposition 2 below, A5 is used in a similar manner to control dot products of the form  $\langle \mathbf{Y}_i - \mathbf{X}_i,\mathbf{X}_j\rangle$  and  $\langle \mathbf{Y}_i - \mathbf{X}_i,\mathbf{Y}_j - \mathbf{X}_j\rangle$ .
Proof of Theorem 2. For the first claim of the theorem, proposition 2 combined with the tower property of conditional expectation imply that for any  $\delta >0$
$$
\begin{array}{l} \mathbb {P} \left(\max  _ {1 \leq i <   j \leq n} \left| p ^ {- 1} \langle \mathbf {Y} _ {i}, \mathbf {Y} _ {j} \rangle - \alpha (Z _ {i}, Z _ {j}) \right| \geq \delta\right) \\ \leq \frac {1}{\delta^ {q}} \frac {1}{p ^ {q / 2}} \frac {n (n - 1)}{2} C (q, \varphi) M (q, \mathbf {X}, \mathbf {E}, \mathbf {S}), \tag {18} \\ \end{array}
$$
from which (6) follows.
The second claim of the theorem is in essence a corollary to [49][Thm 1]. A little work is needed to map the setting of the present work on to the setting of Whiteley et al. [49][Thm 1]. To see the connection, we endow the finite set  $\mathcal{Z}$  in the present work with the discrete metric:  $d_{\mathcal{Z}}(u,v)\coloneqq 0$  for  $u\neq v$ , and  $d_{\mathcal{Z}}(v,v) = 0$ . Then  $(\mathcal{Z},d_{\mathcal{Z}})$  is a compact metric space, and in the setting specified in the statement of theorem 2 where  $\mathbf{A3}$  is strengthened to independence,  $s = p$  and  $\mathbf{S}(v) = \sigma \mathbf{I}_p$  for all  $v\in \mathcal{Z}$ , the variables  $\mathbf{Y}_1,\ldots ,\mathbf{Y}_n;\{\mathbf{X}(v),v\in \mathcal{Z}\} ;\mathbf{E}_1,\ldots ,\mathbf{E}_n$  exactly follow the Latent Metric Model of Whiteley et al. [49].
Moreover, according to the description in section 2.1, the variables  $Z_{1},\ldots ,Z_{n}$  are i.i.d. according to a probability distribution supported on  $\mathcal{Z}$ . As in [49], by Mercer's theorem there exists a feature map  $\phi :\mathcal{Z}\to \mathbb{R}^r$  associated with this probability distribution, such that  $\langle \phi (u),\phi (v)\rangle = \alpha (u,v)$ , for  $u,v\in \mathcal{Z}$ . Here  $r$ , as in A6, is the rank of the matrix with elements  $\alpha (u,v)$ , which is at most  $\mathcal{Z}$ .
Theorem 1 of [49] in this context implies there exists a random orthogonal matrix  $\mathbf{Q} \in \mathbb{R}^{r \times r}$  such that
$$
\left. \max  _ {i \in [ n ]} \left\| p ^ {- 1 / 2} \mathbf {Q} \zeta_ {i} - \phi (Z _ {i}) \right\| \in O _ {\mathbb {P}} \left(\sqrt {\frac {n r}{p}} + \sqrt {\frac {r}{n}}\right). \right. \tag {19}
$$
Consider the bound:
$$
\begin{array}{l} \left| \hat {\alpha} _ {\mathrm {p c a}} (i, j) - \alpha (Z _ {i}, Z _ {j}) \right| = \left| \frac {1}{p} \langle \zeta_ {i}, \zeta_ {j} \rangle - \langle \phi (Z _ {i}), \phi (Z _ {j}) \rangle \right| \\ \leq \left| \left\langle p ^ {- 1 / 2} \mathbf {Q} \zeta_ {i} - \phi (Z _ {i}), p ^ {- 1 / 2} \mathbf {Q} \zeta_ {j} \right\rangle \right| \\ + \left| \left\langle \phi (Z _ {i}), p ^ {- 1 / 2} \mathbf {Q} \zeta_ {j} - \phi (Z _ {j}) \right\rangle \right| \\ \leq \left\| p ^ {- 1 / 2} \mathbf {Q} \zeta_ {i} - \phi (Z _ {i}) \right\| \left(\left\| p ^ {- 1 / 2} \mathbf {Q} \zeta_ {j} - \phi (Z _ {j}) \right\| + \| \phi (Z _ {j}) \|\right) \\ + \| \phi (Z _ {i}) \| \left\| p ^ {- 1 / 2} \mathbf {Q} \zeta_ {i} - \phi (Z _ {i}) \right\|, \\ \end{array}
$$
where orthogonality of  $\mathbf{Q}$  has been used, and the final inequality uses Cauchy-Schwarz and the triangle inequality for the  $\| \cdot \|$  norm. Combining the above estimate with (19), the bound:
$$
\begin{array}{l} \max _ {i \in [ n ]} \| \phi (Z _ {i}) \| ^ {2} \leq \max _ {v \in \mathcal {Z}} \| \phi (v) \| ^ {2} = \max _ {v \in \mathcal {Z}} \alpha (v, v) \\ \leq \sup  _ {j \geq 1} \max  _ {v \in \mathcal {Z}} \mathbb {E} [ | X _ {j} (v) | ^ {2} ] \leq \sup  _ {j \geq 1} \max  _ {v \in \mathcal {Z}} \mathbb {E} [ | X _ {j} (v) | ^ {2 q} ] ^ {1 / q} \tag {20} \\ \end{array}
$$
and A4 completes the proof of the second claim of the theorem.
Proposition 2. Assume the model in section 2.1 satisfies assumptions A3-A5, and let  $\varphi$  and  $q$  be as in A3 and A4. Then there exists a constant  $C(q,\varphi)$  depending only on  $q$  and  $\varphi$  such that for any  $\delta >0$ ,
$$
\begin{array}{l} \mathbb {P} \left(\max  _ {1 \leq i <   j \leq n} \left| p ^ {- 1} \langle \mathbf {Y} _ {i}, \mathbf {Y} _ {j} \rangle - \alpha (Z _ {i}, Z _ {j}) \right| \geq \delta \Bigg | Z _ {1}, \dots , Z _ {n}\right) \\ \leq \frac {1}{\delta^ {q}} \frac {1}{p ^ {q / 2}} \frac {n (n - 1)}{2} C (q, \varphi) M (q, \mathbf {X}, \mathbf {E}, \mathbf {S}) \tag {21} \\ \end{array}
$$
$$
\begin{array}{l} M(q,\mathbf{X},\mathbf{E},\mathbf{S}):= \sup_{k\geq 1}\max_{v\in \mathcal{Z}}\mathbb{E}\left[|X_{k}(v)|^{2q}\right] \\ + \mathbb {E} \left[ | \mathbf {E} _ {1 1} | ^ {q} \right] \left(\sup  _ {p \geq 1} \max  _ {v \in \mathcal {Z}} \| \mathbf {S} (v) \| _ {\mathrm {o p}} ^ {q}\right) \sup  _ {k \geq 1} \max  _ {v \in \mathcal {Z}} \mathbb {E} \left[ | X _ {k} (v) | ^ {q} \right] \\ + \mathbb{E}\left[|\mathbf{E}_{11}|^{2q}\right]\sup_{p\geq 1}\max_{v\in \mathcal{Z}}\| \mathbf{S}(v)\|_{\mathrm{op}}^{2q}. \\ \end{array}
$$
Proof. Fix any  $i,j$  such that  $1\leq i < j\leq n$ . Consider the decomposition:
$$
p ^ {- 1} \left\langle \mathbf {Y} _ {i}, \mathbf {Y} _ {j} \right\rangle - \alpha \left(Z _ {i}, Z _ {j}\right) = \sum_ {k = 1} ^ {4} \Delta_ {k}
$$
$$
\Delta_ {1} := p ^ {- 1} \left\langle \mathbf {X} \left(Z _ {i}\right), \mathbf {X} \left(Z _ {j}\right) \right\rangle - \alpha \left(Z _ {i}, Z _ {j}\right)
$$
$$
\Delta_ {2} := p ^ {- 1} \langle \mathbf {X} (Z _ {i}), \mathbf {S} (Z _ {j}) \mathbf {E} _ {j} \rangle
$$
$$
\Delta_ {3} := p ^ {- 1} \left\langle \mathbf {X} \left(Z _ {j}\right), \mathbf {S} \left(Z _ {i}\right) \mathbf {E} _ {i} \right\rangle
$$
$$
\Delta_ {4} := p ^ {- 1} \langle \mathbf {S} (Z _ {i}) \mathbf {E} _ {i}, \mathbf {S} (Z _ {j}) \mathbf {E} _ {j} \rangle
$$
The proof proceeds by bounding  $\mathbb{E}[|\Delta_k|^q |Z_1,\ldots ,Z_n]$  for  $k = 1,\dots ,4$ . Writing  $\Delta_{1}$  as
$$
\Delta_ {1} = \frac {1}{p} \sum_ {k = 1} ^ {p} \Delta_ {1, k}, \qquad \Delta_ {1, k} := X _ {k} (Z _ {i}) X _ {k} (Z _ {j}) - \mathbb {E} \left[ X _ {k} (Z _ {i}) X _ {k} (Z _ {j}) | Z _ {1}, \ldots , Z _ {n} \right].
$$
we see that  $\Delta_{1}$  is a sum  $p$  random variables each of which is conditionally mean zero given  $Z_{1},\ldots ,Z_{n}$ . Noting that the two collections of random variables  $\{Z_1,\dots,Z_n\}$  and  $\{\mathbf{X}(v);v\in \mathcal{V}\}$  are independent (as per the description of the model in section 2.1), under assumption A3 we may apply a moment inequality for  $\varphi$ -mixing random variables [51][Lemma 1.7] to show that there exists a constant  $C_1(q,\varphi)$  depending only on  $q,\varphi$  such that
$$
\begin{array}{l} \mathbb {E} \left[ | \Delta_ {1} | ^ {q} \mid Z _ {1}, \dots , Z _ {n} \right] \\ \leq C _ {1} (q, \varphi) \left\{\frac {1}{p ^ {q}} \sum_ {k = 1} ^ {p} \mathbb {E} \left[ | \Delta_ {1, k} | ^ {q} \mid Z _ {1}, \dots , Z _ {n} \right] + \left(\frac {1}{p ^ {2}} \sum_ {k = 1} ^ {p} \mathbb {E} \left[ | \Delta_ {1, k} | ^ {2} \mid Z _ {1}, \dots , Z _ {n} \right]\right) ^ {q / 2} \right\} \\ \leq C _ {1} (q, \varphi) \left\{\frac {1}{p ^ {q}} \sum_ {k = 1} ^ {p} \mathbb {E} \left[ | \Delta_ {1, k} | ^ {q} \mid Z _ {1}, \dots , Z _ {n} \right] + \frac {1}{p ^ {q / 2}} \frac {1}{p} \sum_ {k = 1} ^ {p} \mathbb {E} \left[ | \Delta_ {1, k} | ^ {q} \mid Z _ {1}, \dots , Z _ {n} \right] \right\} \\ \leq 2 C _ {1} (q, \varphi) \frac {1}{p ^ {q / 2}} \sup  _ {k \geq 1} \mathbb {E} \left[ | \Delta_ {1, k} | ^ {q} \mid Z _ {1}, \dots , Z _ {n} \right] \\ \leq 2 ^ {q + 1} C _ {1} (q, \varphi) \frac {1}{p ^ {q / 2}} \sup  _ {k \geq 1} \max  _ {v \in \mathcal {Z}} \mathbb {E} \left[ | X _ {k} (v) | ^ {2 q} \right], \tag {22} \\ \end{array}
$$
where second inequality holds by two applications of Jensen's inequality and  $q \geq 2$ , and the final inequality uses the fact that for  $a, b \geq 0$ ,  $(a + b)^q \leq 2^{q - 1}(a^q + b^q)$ , the Cauchy-Schwartz inequality, and the independence of  $\{Z_1, \ldots, Z_n\}$  and  $\{\mathbf{X}(v); v \in \mathcal{V}\}$ .
For  $\Delta_2$ , we have
$$
\Delta_ {2} := \frac {1}{p} \sum_ {k = 1} ^ {p} \Delta_ {2, k}, \qquad \Delta_ {2, k} := [ \mathbf {S} (Z _ {j}) ^ {\top} \mathbf {X} (Z _ {i}) ] _ {k} \mathbf {E} _ {j k},
$$
where  $\left[\cdot\right]_k$  denotes the  $k$ th element of a vector. Since the three collections of random variables,  $\{Z_{1},\ldots,Z_{n}\}$ ,  $\{\mathbf{X}(v);v\in\mathcal{Z}\}$  and  $\{\mathbf{E}_1,\dots,\mathbf{E}_n\}$  are mutually independent, and the elements of each vector  $\mathbf{E}_j\in \mathbb{R}^p$  are mean zero and independent, we see that given  $\{Z_1,\dots,Z_n\}$  and  $\{\mathbf{X}(v);v\in\mathcal{V}\}$ ,  $\Delta_2$  is a simple average of conditionally independent and conditionally mean-zero
random variables. Applying the Marcinkiewicz-Zygmund inequality we find there exists a constant  $C_2(q)$  depending only on  $q$  such that
$$
\begin{array}{l} \mathbb {E} \left[ | \Delta_ {2} | ^ {q} | Z _ {1}, \dots , Z _ {n}, \mathbf {X} (v); v \in \mathcal {Z} \right] \\ \leq C _ {2} (q) \mathbb {E} \left[ \left| \frac {1}{p ^ {2}} \sum_ {k = 1} ^ {p} \left| \Delta_ {2, k} \right| ^ {2} \right| ^ {q / 2} \mid Z _ {1}, \dots , Z _ {n}, \mathbf {X} (v); v \in \mathcal {Z} \right]. \tag {23} \\ \end{array}
$$
Noting that  $q \geq 2$  and applying Minkowski's inequality to the r.h.s. of (23), then using the independence of  $\{Z_1, \ldots, Z_n\}$ ,  $\{\mathbf{X}(v); v \in \mathcal{Z}\}$  and  $\{\mathbf{E}_1, \ldots, \mathbf{E}_n\}$  and the i.i.d. nature of the elements of the vector  $\mathbf{E}_j$ ,
$$
\begin{array}{l} \mathbb {E} \left[ \left| \frac {1}{p ^ {2}} \sum_ {k = 1} ^ {p} | \Delta_ {2, k} | ^ {2} \right| ^ {q / 2} \middle | Z _ {1}, \dots , Z _ {n}, \mathbf {X} (v); v \in \mathcal {Z} \right] ^ {2 / q} \\ \leq \frac {1}{p ^ {2}} \sum_ {k = 1} ^ {p} \mathbb {E} \left[ | \Delta_ {2, k} | ^ {q} \mid Z _ {1}, \dots , Z _ {n}, \mathbf {X} (v); v \in \mathcal {Z} \right] ^ {2 / q} \\ = \frac {1}{p ^ {2}} \sum_ {k = 1} ^ {p} \mathbb {E} \left[ \left| [ \mathbf {S} (Z _ {j}) ^ {\top} \mathbf {X} (Z _ {i}) ] _ {k} \right| ^ {q} | \mathbf {E} _ {j k} | ^ {q} \right\rvert   Z _ {1}, \dots , Z _ {n}, \mathbf {X} (v); v \in \mathcal {Z} \bigg ] ^ {2 / q} \\ = \frac {1}{p ^ {2}} \mathbb {E} \left[ | \mathbf {E} _ {1 1} | ^ {q} \right] ^ {2 / q} \sum_ {k = 1} ^ {p} \left| [ \mathbf {S} (Z _ {j}) ^ {\top} \mathbf {X} (Z _ {i}) ] _ {k} \right| ^ {2} \\ = \frac {1}{p ^ {2}} \mathbb {E} \left[ \left| \mathbf {E} _ {1 1} \right| ^ {q} \right] ^ {2 / q} \left\| \mathbf {S} \left(Z _ {j}\right) ^ {\top} \mathbf {X} \left(Z _ {i}\right) \right\| ^ {2} \\ \leq \frac {1}{p ^ {2}} \mathbb {E} \left[ | \mathbf {E} _ {1 1} | ^ {q} \right] ^ {2 / q} \max  _ {v \in \mathcal {Z}} \| \mathbf {S} (v) \| _ {\mathrm {o p}} ^ {2} \| \mathbf {X} (Z _ {i}) \| ^ {2}. \\ \end{array}
$$
Substituting into (23) and using the tower property of conditional expectation we obtain: