SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Similarly, we prove the error in general metric by first analyzing the error in 2-HST metric. Then the result follows from Lemma 3.4. Let  $\text{cost}_k^T(D)$ ,  $\text{cost}_k^{T'}(D)$  and  $\text{OPT}_k^T(D)$  be defined analogously to (3), (4) and (5), where " $y \in U$ " in the summation is changed into " $y \in D$ " since  $D$  is the demand set. That is,
$$
\operatorname {c o s t} _ {k} ^ {T} (D) = \sum_ {y \in D} \min  _ {x \in C _ {0}} \rho^ {T} (x, y), \tag {8}
$$
$$
\operatorname {c o s t} _ {k} ^ {T ^ {\prime}} (D, C _ {1}) = \min  _ {| F \cap T (v) | = 1, \forall v \in C _ {1}} \sum_ {y \in D} \min  _ {x \in F} \rho^ {T} (x, y), \tag {9}
$$
$$
O P T _ {k} ^ {T} (D) = \min  _ {F \subset D, | F | = k} \sum_ {y \in D} \min  _ {x \in F} \rho^ {T} (x, y) \equiv \min  _ {C _ {1} ^ {\prime}} c o s t _ {k} ^ {T \prime} (D, C _ {1} ^ {\prime}). \tag {10}
$$
We have the following.
Lemma B.2.  $\text{cost}_k^T(D) \leq 10OPT_k^T(D) + 10ck\epsilon^{-1}\triangle \log n$  with probability  $1 - 4k/n^c$ .
Proof. The result follows by combining the following Lemma B.4, Lemma B.5, and applying union bound.  $\square$
Lemma B.3. For any node  $v$  in  $T$ , with probability  $1 - 1/n^c$ ,  $|\hat{N}_v \cdot 2^{h_v} - N_v \cdot 2^{h_v}| \leq c\epsilon^{-1}\triangle \log n$ .
Proof. Since  $\hat{N}_v = N_v + \text{Lap}(2^{(L - h_v) / 2} / \epsilon)$ , we have
$$
P r \left[ \left| \hat {N} _ {v} - N _ {v} \right| \geq x / \epsilon \right] = e x p \left(- x / 2 ^ {\left(L - h _ {v}\right)}\right).
$$
As  $L = \log \triangle$ , we have
$$
P r \left[ | \hat {N} _ {v} - N _ {v} | \geq x \triangle / \left(2 ^ {h _ {v}} \epsilon\right) \right] \leq e x p (- x).
$$
Hence, for some constant  $c > 0$
$$
P r \left[ \left| \hat {N} _ {v} \cdot 2 ^ {h _ {v}} - N _ {v} \cdot 2 ^ {h _ {v}} \right| \leq c \epsilon^ {- 1} \triangle \log n \right] \geq 1 - e x p (- c \log n) = 1 - 1 / n ^ {c}.
$$
Lemma B.4 (DP Subtree Search). With probability  $1 - 2k / n^{c}$ ,  $\text{cost}_{k}^{T'}(D) \leq 5OPT_{k}^{T}(D) + 4ck\epsilon^{-1}\triangle \log n$ .
Proof. The proof is similar to that of Lemma 3.5. Consider the intermediate output of Algorithm 4,  $C_1 = \{v_1, v_2, \dots, v_k\}$ , which is the set of roots of the minimal disjoint subtrees each containing exactly one output center  $C_0$ . Suppose one of the optimal "root set" that minimizes (4) is  $C_1^* = \{v_1', v_2', \dots, v_k'\}$ . Assume  $C_1 \neq C_1^*$ . By the same argument as the proof of Lemma 3.5, we consider for some  $i, j$  such that  $v_i \neq v_j'$ , where  $v_i$  is not a descendant of  $v_j'$  and  $v_j'$  is either a descendant of  $v_i$ . By the construction of  $C_1$ , we know that  $score(v_j') \leq \min \{score(v_i), i = 1, \dots, k\}$  when  $v_j' \in C_1^* \setminus C_1$ . Consider the swap between  $C_1$  and  $C_1^*$ . By the definition of tree distance, we have  $OPT_k^T(U) \geq \sum_{v_i \in C_1 \setminus C_1^*} N_{v_i} 2^{h_{v_i}}$ , since  $\{T(v_i), v_i \in C_1 \setminus C_1^*\}$  does not contain any center of the optimal solution determined by  $C_1^*$  (which is also the optimal "root set" for  $OPT_k^T$ ).
Let us consider the optimal clustering with center set be  $C^* = \{c_1^*, c_2^*, \dots, c_k^*\}$  (each center  $c_j^*$  is a leaf of subtree whose root be  $c_j'$ ), and  $S_j'$  be the leaves assigned to  $c_j^*$ . Let  $S_j$  denote the set of leaves in  $S_j'$  whose distance to  $c_j^*$  is strictly smaller than its distance to any centers in  $C_1$ . Let  $P_j$  denote the union of paths between leaves of  $S_j$  to its closest center in  $C_1$ . Let  $v_j''$  be the nodes in  $P_j$  with highest level satisfying  $T(v_j'') \cap C_1 = \emptyset$ . The score of  $v_j''$  is  $2^{h_{v_j''}} N(v_j'')$ . That means the swap with a center  $v_j'$  into  $C_1$  can only reduce  $4 \cdot 2^{h_{v_j''}} N(v_j'')$  to  $cost_k^{T'}(U)$  (the tree distance between any leaf in  $S_j$  and its closest center in  $C_1$  is at most  $4 \cdot 2^{h_{v_j''}}$ ). We just use  $v_j'$  to represent  $v_j''$  for later part of this proof for simplicity. Summing all the swaps over  $C_1^* \setminus C_1$ , we obtain
$$
\operatorname{cost}_{k}^{T^{\prime}}(U) - OPT_{k}^{T}(U)\leq 4\sum_{v^{\prime}_{j}\in C_{1}^{*}\backslash C_{1}}N_{v^{\prime}_{j}}2^{h_{v^{\prime}_{j}}},
$$
Applying union bound with Lemma B.3, with probability  $1 - 2 / n^{c}$ , we have
$$
N _ {v _ {j} ^ {\prime}} 2 ^ {h _ {v _ {j} ^ {\prime}}} - N _ {v _ {i}} 2 ^ {h _ {v _ {i}}} \leq 2 c \epsilon^ {- 1} \triangle \log n.
$$
Consequently, we have with probability,  $1 - 2k / n^{c}$
$$
\begin{array}{l} \operatorname {c o s t} _ {k} ^ {T \prime} (D) \leq 5 O P T _ {k} ^ {T} (D) + 4 c | C _ {1} \backslash C _ {1} ^ {*} | \epsilon^ {- 1} \triangle \log n \\ \leq 5 O P T _ {k} ^ {T} (D) + 4 c k \epsilon^ {- 1} \triangle \log n, \\ \end{array}
$$
Lemma B.5 (DP Leaf Search). With probability  $1 - 2k / n^{c}$ , Algorithm 6 produces initial centers with  $\mathrm{cost}_k^T (D)\leq 2\mathrm{cost}_k^{T'}(D) + 2ck\epsilon^{-1}\triangle \log n$
Proof. The proof strategy follows Lemma 3.6. We first consider one subtree with root  $v$ . Let  $cost_1^{T'}(v, U)$  denote the optimal  $k$ -median cost within the point set  $T(v)$  with one center in 2-HST:
$$
\operatorname {c o s t} _ {1} ^ {T ^ {\prime}} (v, D) = \min  _ {x \in T (v)} \sum_ {y \in T (v) \cap D} \rho^ {T} (x, y). \tag {11}
$$
Suppose  $v$  has more than one children  $u, w, \ldots$ , and the optimal solution of  $cost_1^{T'}(v, U)$  chooses a leaf node in  $T(u)$ , and our HST initialization algorithm picks a leaf of  $T(w)$ . If  $u = w$ , then HST chooses the optimal one where the argument holds trivially. Thus, we consider  $u \neq w$ . We have the following two observations:
As  $h_u = h_w$ , leveraging Lemma B.3, with probability  $1 - 2 / n^c$
$$
\begin{array}{l} 2 ^ {h _ {u}} \cdot \left(N _ {u} - N _ {w}\right) \leq 2 ^ {h _ {u}} \left(\hat {N} _ {u} - \hat {N} _ {w}\right) + 2 c \epsilon^ {- 1} \triangle \log n \\ \leq 2 c \epsilon^ {- 1} \triangle \log n. \\ \end{array}
$$
since our algorithm picks subtree roots with highest scores. Then we have  $cost_1^T(v, D) \leq cost_k^{T'}(v, D) + N_w \cdot 2^{h_u} + 2c\epsilon^{-1}\triangle \log n \leq 2cost_k^{T'}(v, D) + 2c\epsilon^{-1}\triangle \log n$  with high probability. Lastly, applying union bound over the disjoint  $k$  subtrees gives the desired result.
B.5 Proof of Theorem 4.3
Proof. The privacy analysis is straightforward, by using the composition theorem (Theorem B.1). Since the sensitivity of  $cost(\cdot)$  is  $\triangle$ , in each swap iteration the privacy budget is  $\epsilon / 2(T + 1)$ . Also, we spend another  $\epsilon / 2(T + 1)$  privacy for picking a output. Hence, the total privacy is  $\epsilon / 2$  for local search. Algorithm 6 takes  $\epsilon / 2$  DP budget for initialization, so the total privacy is  $\epsilon$ .
The analysis of the approximation error follows from Gupta et al. (2010), where the initial cost is reduced by our private HST method. We need the following two lemmas.
Lemma B.6 (Gupta et al. (2010)). Assume the solution to the optimal utility is unique. For any output  $o \in O$  of  $2\triangle \epsilon - DP$  exponential mechanism on dataset  $D$ , it holds for  $\forall t > 0$  that
$$
P r [ q (D, o) \leq \max _ {o \in O} q (D, o) - (\ln | O | + t) / \epsilon ] \leq e ^ {- t},
$$
where  $|O|$  is the size of the output set.
Lemma B.7 (Arya et al. (2004)). For any set  $F \subseteq D$  with  $|F| = k$ , there exists some swap  $(x, y)$  such that the local search method admits
$$
c o s t _ {k} (F, D) - c o s t _ {k} (F - \{x \} + \{y \}, D) \geq \frac {c o s t _ {k} (F , D) - 5 O P T (D)}{k}.
$$
From Lemma B.7, we know that when  $\text{cost}_k(F_i, D) > 6OPT(D)$ , there exists a swap  $(x, y)$  s.t.
$$
\operatorname {c o s t} _ {k} \left(F _ {i} - \{x \} + \{y \}, D\right) \leq \left(1 - \frac {1}{6 k}\right) \operatorname {c o s t} _ {k} \left(F _ {i}, D\right).
$$
At each iteration, there are at most  $n^2$  possible outputs (i.e., possible swaps), i.e.,  $|O| = n^2$ . Using Lemma B.6 with  $t = 2\log n$ , for  $\forall i$ ,
$$
P r [ c o s t _ {k} (F _ {i + 1}, D) \geq c o s t _ {k} (F _ {i + 1} ^ {*}, D) + 4 \frac {\log n}{\epsilon^ {\prime}} ] \geq 1 - 1 / n ^ {2},
$$
where  $\text{cost}_k(F_{i+1}^*, D)$  is the minimum cost among iteration 1, 2, ...,  $t + 1$ . Hence, we have that as long as  $\text{cost}(F_i, D) > 6OPT(D) + \frac{24k\log n}{\epsilon'}$ , the improvement in cost is at least by a factor of  $(1 - \frac{1}{6k})$ . By Theorem 4.2, we have  $\text{cost}_k(F_1, D) \leq C(\log n)(6OPT(D) + 6k\triangle\log n/\epsilon)$  for some constant  $C > 0$ . Let  $T = 6Ck\log\log n$ . We have that
$$
\begin{array}{l} E [ \operatorname {c o s t} (F _ {i}, D) ] \leq (6 O P T (D) + 6 k \epsilon^ {- 1} \triangle \log n) C (\log n) (1 - 1 / 6 k) ^ {6 C k \log \log n} \\ \leq 6 O P T (D) + 6 k \epsilon^ {- 1} \triangle \log n \leq 6 O P T (D) + \frac {2 4 k \log n}{\epsilon^ {\prime}}. \\ \end{array}
$$
Therefore, with probability at least  $\left(1 - T / n^{2}\right)$ , there exists an  $i \leq T$  s.t.  $cost(F_{i},D) \leq 6OPT(D) + \frac{24k\log n}{\epsilon'}$ . Then by using the Lemma B.7, one will pick an  $F_{j}$  with additional additive error  $4\ln n / \epsilon'$  to the min  $\{cost(F_j,D), j = 1,2,\dots,T\}$  with probability  $1 - 1 / n^2$ . Consequently, we know that the expected additive error is
$$
2 4 k \triangle \log n / \epsilon^ {\prime} + 4 \log n / \epsilon^ {\prime} = O \left(\epsilon^ {- 1} k ^ {2} \triangle (\log \log n) \log n\right),
$$
with probability  $1 - 1/poly(n)$ .
C Extend HST Initialization to  $k$ -Means
Naturally, our HST method can also be applied to  $k$ -means clustering problem. In this section, we extend the HST to  $k$ -means and provide some brief analysis similar to  $k$ -median. We present the analysis in the non-private case, which can then be easily adapted to the private case. Define the following costs for  $k$ -means.
$$
\operatorname {c o s t} _ {k m} ^ {T} (U) = \sum_ {y \in U} \min  _ {x \in C _ {0}} \rho^ {T} (x, y) ^ {2}, \tag {12}
$$
$$
\operatorname {c o s t} _ {k m} ^ {T} ^ {\prime} (U, C _ {1}) = \min  _ {| F \cap T (v) | = 1, \forall v \in C _ {1}} \sum_ {y \in U} \min  _ {x \in F} \rho^ {T} (x, y) ^ {2}, \tag {13}
$$
$$
O P T _ {k m} ^ {T} (U) = \min  _ {F \subset U, | F | = k} \sum_ {y \in U} \min  _ {x \in F} \rho^ {T} (x, y) ^ {2} \equiv \min  _ {C _ {1} ^ {\prime}} c o s t _ {k m} ^ {T} ^ {\prime} (U, C _ {1} ^ {\prime}). \tag {14}
$$
For simplicity, we will use  $cost_{km}^{T}'(U)$  to denote  $cost_{km}^{T}(U, C_{1})$  if everything is clear from context. Here,  $OPT_{km}^{T}$  (14) is the cost of the global optimal solution with 2-HST metric.
Lemma C.1 (Subtree search).  $cost_{km}^{T}'(U) \leq 17OPT_{km}^{T}(U)$ .
Proof. The analysis is similar with the proof of Lemma 3.5. Thus, we mainly highlight the difference. Let us just use some notations the same as in Lemma 3.5 here. Let us consider the clustering with center set be  $C^* = \{c_1^*, c_2^*, \ldots, c_k^*\}$  (each center  $c_j^*$  is a leaf of subtree whose root be  $c_j'$ ), and  $S_j'$  be the leaves assigned to  $c_j^*$  in optimal k-means clustering in tree metric. Let  $S_j$  denote the set of leaves in  $S_j'$  whose distance to  $c_j^*$  is strictly smaller than its distance to any centers in  $C_1$ . Let  $P_j$  denote the union of paths between leaves of  $S_j$  to its closest center in  $C_1$ . Let  $v_j''$  be the nodes in  $P_j$  with highest level satisfying  $T(v_j'') \cap C_1 = \emptyset$ . The score of  $v_j''$  is  $2^{h_{v_j''}} N(v_j'')$ . That means the swap with a center  $v_j'$  into  $C_1$  can only reduce  $(4 \cdot 2^{h_{v_j''}})^2 N(v_j'')$  to  $cost_{km}'(U)$ . We just use  $v_j'$  to represent  $v_j''$  for later part of this proof for simplicity. By our reasoning, summing all the swaps over  $C_1^* \setminus C_1$  gives
$$
c o s t _ {k m} ^ {T} ^ {\prime} (U) - O P T _ {k m} ^ {T} (U) \leq \sum_ {v _ {j} ^ {\prime} \in C _ {1} ^ {*} \backslash C _ {1}} N _ {v _ {j} ^ {\prime}} \cdot (4 \cdot 2 ^ {h _ {v _ {j} ^ {\prime}}}) ^ {2},
$$
$$
O P T _ {k m} ^ {T} (U) \geq \sum_ {v _ {i} \in C _ {1} \backslash C _ {1} ^ {*}} N _ {v _ {i}} \left(2 ^ {h _ {v _ {i}}}\right) ^ {2}.
$$
Summing them together, we have  $\text{cost}_{km}^T'(U) \leq 17\text{OPT}_{km}^T(U)$ .
Next, we show that the greedy leaf search strategy (Algorithm 5) only leads to an extra multiplicative error of 2.
Lemma C.2 (Leaf search).  $cost_{km}^{T}(U) \leq 2cost_{km}^{T}'(U)$ .
Proof. Since the subtrees in  $C_1$  are disjoint, it suffices to consider one subtree with root  $v$ . With a little abuse of notation, let  $cost_1^{T'}(v, U)$  denote the optimal  $k$ -means cost within the point set  $T(v)$  with one center in 2-HST:
$$
\operatorname {c o s t} _ {1} ^ {T ^ {\prime}} (v, U) = \min  _ {x \in T (v)} \sum_ {y \in T (v)} \rho^ {T} (x, y) ^ {2}, \tag {15}
$$
- Since one needs to pick a leaf of  $T(u)$  to minimize  $cost_1^{T'}(v, U)$ , we have  $cost_1^{T'}(v, U) \geq \sum_{x \in ch(v), x \neq u} N_x \cdot (2^{h_x})^2$  where  $ch(u)$  denotes the children nodes of  $u$ .
- By our greedy strategy,  $\text{cost}_1^T(v, U) \leq \sum_{x \in \text{ch}(u)} N_x \cdot (2^{h_x})^2 \leq \text{cost}_1^{T'}(v, U) + N_u \cdot (2^{h_u})^2$ .
since our algorithm picks subtree roots with highest scores. Then we have  $cost_1^T(v, U) \leq cost_1^{T'}(v, U) + N_w \cdot (2^{h_w})^2 \leq 2cost_1^{T'}(v, U)$ . Since the subtrees in  $C_1$  are disjoint, the union of centers for  $OPT_1^T(v, U)$ ,  $v \in C_1$  forms the optimal centers with size  $k$ . Note that, for any data point  $p \in U \setminus C_1$ , the tree distance  $\rho^T(p, f)$  for  $\forall f$  that is a leaf node of  $T(v)$ ,  $v \in C_1$  is the same. That is, the choice of leaf in  $T(v)$  as the center does not affect the  $k$ -median cost under 2-HST metric. Therefore, union bound over  $k$  subtree costs completes the proof.
We are ready to state the error bound for our proposed HST initialization (Algorithm 4), which is a natural combination of Lemma C.1 and Lemma C.2.
Theorem C.3 (HST initialization).  $cost_{km}^{T}(U) \leq 34OPT_{km}^{T}(U)$ .
Theorem C.4. In a general metric space,
$$
E \left[ \operatorname {c o s t} _ {k m} (U) \right] = O \left(\min  \left\{\log n, \log \bigtriangleup \right\}\right) ^ {2} O P T _ {k m} (U).
$$