SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Thus  $\Lambda$  is a locus cover. Next, we will show that it is the smallest such set. Let  $l\in \Lambda$  be any leaf node, and define  $\Lambda^{\prime} = \Lambda \setminus \{l\}$ . We must show that  $\Lambda^\prime$  is not a locus cover. Assume for contradiction that  $\Lambda^{\prime}$  is a locus cover. This means that given some tuple  $\pmb{y}^{\prime}$  whose entries are the elements of  $\Lambda^{\prime}$ , we have  $\Pi (\Lambda^{\prime}) = \mathcal{V}$ . This implies that the path between any two nodes in  $\Lambda$  is also contained in  $\Pi (\Lambda^{\prime})$ . Since the leaves form a locus cover and any  $y\in \mathcal{V}$  can be constructed by the appropriate choice of  $\pmb{w}$  along with a path between two leaf nodes,  $l$  must either be one of the entries of  $\Lambda^{\prime}$  (one of the endpoints of the path) or it must be an internal node. Both cases are contradictions- $\Lambda^{\prime}$  cannot include  $l$  by assumption, and  $l$  is assumed to be a leaf node. It follows that  $\Lambda$  is a minimum locus cover.
A.3 Proof of Lemma A.5 (tree pairwise decomposability)
Lemma A.5 (Tree pairwise decomposability). Let  $T = (\mathcal{V}, \mathcal{E})$  be a tree and  $\Lambda \subseteq \mathcal{V}$ . Then  $\Pi(\Lambda)$  is pairwise decomposable.
Proof. Assume for contradiction that  $\exists y^{*}\in \mathcal{V}$  where  $y^{*}\notin \Pi (\{\lambda_{i},\lambda_{j}\})$ $\forall \lambda_i,\lambda_j\in \Lambda$  , but  $y^{*}\in \Pi (\Lambda)$  Then,  $y^{*}\in m_{\Lambda}(\mathbf{w})$  for some w. Note that  $y^{*}\notin \Pi (\{\lambda_{i},\lambda_{j}\})$  implies  $y^{*}\notin \Gamma (\lambda_{i},\lambda_{j}),\forall \lambda_{i},\lambda_{j}\in \Lambda$  because  $\Pi (\{\lambda_i,\lambda_j\}) = \Gamma (\lambda_i,\lambda_j)$  due to the uniqueness of paths in trees. As such,  $\cap_{\lambda \in \Lambda}\Gamma (y^*,\lambda)$  must contain an immediate relative  $y^\prime$  of  $y^{*}$  where
$$
\begin{array}{l} \sum_ {i = 1} ^ {| \Lambda |} \mathbf {w} _ {i} d ^ {2} \left(y ^ {*}, \lambda_ {i}\right) = \sum_ {i = 1} ^ {| \Lambda |} \mathbf {w} _ {i} \left(d \left(y ^ {\prime}, \lambda_ {i}\right) + 1\right) ^ {2} \\ > \sum_ {i = 1} ^ {| \Lambda |} \mathbf {w} _ {i} d ^ {2} \left(y ^ {\prime}, \lambda_ {i}\right). \\ \end{array}
$$
Therefore,  $y^{*} \notin \Pi(\Lambda)$  because  $y^{*}$  is not a minimizer of  $\sum_{i=1}^{|\Lambda|} \mathbf{w}_i d^2(y, \lambda_i)$ . So, it must be that  $y^{*} \in \Pi(\{\lambda_i, \lambda_j\})$  for some  $\lambda_i, \lambda_j \in \Lambda$  when  $y \in \Pi(\Lambda)$ .
A.4 Proof of Theorem A.6 (Algorithm B.1 correctness)
Theorem A.6 (Algorithm B.1 correctness). Algorithm B.1 returns a locus cover for phylogenetic trees.
Proof. We first prove that Algorithm B.1 will halt, then according to the stopping criterion, it is guaranteed that Algorithm B.1 returns a locus cover. Suppose the algorithm keeps running until it reaches the case in which all of the leaf nodes are included in  $\Lambda$  (i.e.  $\Lambda = \mathcal{V}$ ), this results in the trivial locus cover of the phylogenetic tree and the algorithm halts.
A.5 Proof of Lemma A.7 (phylogenetic tree pairwise decomposability)
Lemma A.7 (Phylogenetic tree pairwise decomposability). Let  $T = (\mathcal{V}, \mathcal{E})$  be a tree with  $\mathcal{V} = \text{Leaves}(T)$  and  $\Lambda \subseteq \mathcal{V}$ . Then  $\Pi(\Lambda)$  is pairwise decomposable.
Proof. Suppose to reach a contradiction that  $\exists y^{*}\in \mathcal{V}$  where  $y^{*}\in \Pi (\Lambda)$ , but  $y^{*}\notin \Pi (\{\lambda_{i},\lambda_{j}\})$ $\forall \lambda_{i},\lambda_{j}\in \Lambda$ . In other words,  $y^{*}$  is a Fréchet mean for some weighting on  $\Lambda$ , but  $y^{*}$  is not a Fréchet mean for any weighting for any two vertices within  $\Lambda$ .
For arbitrary  $\lambda_i, \lambda_j \in \Lambda$ , define the closest vertex to  $y^*$  on the path between  $\lambda_i$  and  $\lambda_j$  as
$$
v^{\prime}(\lambda_{i},\lambda_{j}) = \operatorname *{arg  min}_{v\in \Gamma (\lambda_{i},\lambda_{j})}d(y^{*},v).
$$
$$
\{\lambda_{1},\lambda_{2}\} = \operatorname *{arg  min}_{\lambda_{i},\lambda_{j}\in \Lambda}d(y^{*},v^{\prime}(\lambda_{i},\lambda_{j}))
$$
the pair of vertices in  $\Lambda$  whose path  $\Gamma(\lambda_1, \lambda_2)$  passes closest to  $y^*$ . To avoid notational clutter, define  $v' = v'(\lambda_1, \lambda_2)$ . Thus,  $v'$  represents the closest vertex to  $y^*$  lying on the path between any two  $\lambda_i, \lambda_j \in \Lambda$ .
Because  $v'$  lies on the path  $\Gamma(\lambda_1, \lambda_2)$ , there exists some weighting  $\mathbf{w}$  for which
$$
v^{\prime} = \operatorname *{arg  min}_{v\in \Gamma (\lambda_{1},\lambda_{2})}\bigl(\mathbf{w}_{1}d^{2}(v,\lambda_{1}) + \mathbf{w}_{2}d^{2}(v,\lambda_{2})\bigr).
$$
For this  $\mathbf{w}$ , we have that  $m_{\{\lambda_1,\lambda_2\}}(\mathbf{w}) = \arg \min_{y\in \mathcal{V}}d(y,v')$ , the set of leaf nodes of smallest distance from  $v^{\prime}$ . By the initial assumption that  $y^{*}\notin \Pi (\{\lambda_{1},\lambda_{2}\})$ , we have that  $y^{*}\notin m_{\{\lambda_{1},\lambda_{2}\}}(\mathbf{w})$ . Let  $y^\prime$  be a vertex of  $m_{\{\lambda_i,\lambda_j\}}(\mathbf{w})$  that is closest to  $y^{*}$ . Clearly,  $d(y^{\prime},v^{\prime})^{2} < d(y^{*},v^{\prime})^{2}$ .
For any  $v \in \mathcal{V}$ , respectively define the sets of vertices (1) shared in common on the two paths  $\Gamma(v, y^*)$  and  $\Gamma(v, y')$ , (2) on the path  $\Gamma(v, y^*)$  that are not on  $\Gamma(v, y')$ , and (3) on the path  $\Gamma(v, y')$  that are not on  $\Gamma(v, y^*)$  as
$$
d _ {C} (v) = \Gamma (v, y ^ {*}) \cap \Gamma (v, y ^ {\prime}),
$$
$$
d _ {y ^ {*}} (v) = \Gamma (v, y ^ {*}) \backslash \Gamma (v, y ^ {\prime}), \text {a n d}
$$
$$
d _ {y ^ {\prime}} (v) = \Gamma (v, y ^ {\prime}) \backslash \Gamma (v, y ^ {*}).
$$
We have that  $|d_{y^*}(v)| + |d_{y^*}(v)| + 1 = |\Gamma(y^*, y')|$  (the difference by 1 represents the final vertex shared in common by both paths) and also that  $|d_{y'}(v')| < |d_{y^*}(v')|$  since  $y'$  is closer to  $v'$  than  $y^*$  is.
Suppose  $\exists \lambda_3, \lambda_4 \in \Lambda$  yielding
$$
u ^ {\prime} = v ^ {\prime} \left(\lambda_ {3}, \lambda_ {4}\right) \in \Gamma \left(y ^ {*}, y ^ {\prime}\right) \text {s u c h t h a t} d ^ {2} \left(y ^ {*}, u ^ {\prime}\right) \leq d ^ {2} \left(y ^ {\prime}, u ^ {\prime}\right).
$$
This is to say that  $u'$  is defined analogously to  $v'$ , with the additional restrictions that  $u'$  lie on the path  $\Gamma(y^*, y')$  and  $u'$  be at least as close as close to  $y^*$  as to  $y'$ . Then,  $|d_{y^*}(u')| < |d_{y^*}(v')|$  implying  $d^2(y^*, u') < d^2(y^*, v')$ , which contradicts the definition of  $v'$ .
Suppose now that  $\exists \lambda_3, \lambda_4 \in \Lambda$  yielding
$$
u ^ {\prime} = v ^ {\prime} \left(\lambda_ {3}, \lambda_ {4}\right) \notin \Gamma \left(y ^ {*}, y ^ {\prime}\right) \text {s u c h t h a t} d ^ {2} \left(y ^ {*}, u ^ {\prime}\right) \leq d ^ {2} \left(y ^ {\prime}, u ^ {\prime}\right).
$$
This is to say that  $u'$  is again defined analogously to  $v'$ , but  $u'$  is not on  $\Gamma(y^*, y')$  and is at least as close to  $y^*$  as to  $y'$ . Then, either  $\cap_{\lambda \in \Lambda} d_C(\lambda) \neq \emptyset$  (i.e. all lambdas lie in the same branch off of  $\Gamma(y', y^*)$ ), in which case the path between any two lambdas will have the same closest node  $v'$  to  $y^*$ , or it is possible to choose a  $\lambda_5 \in \Lambda$  such that  $d_C(\lambda_3) \cap d_C(\lambda_5) = \emptyset$ . In this situation, it must be that  $\Gamma(\lambda_3, \lambda_5) \cap \Gamma(y', y^*) \neq \emptyset$ , which implies there exists a vertex  $t \in \Gamma(\lambda_3, \lambda_5)$  such that  $d^2(y^*, t) < d^2(y^*, u')$  which violates the definition of  $u'$ .
Altogether, for all  $v \in \Gamma(\lambda_i, \lambda_j)$  with arbitrary  $\lambda_i, \lambda_j \in \Lambda$ , we have that  $d^2(y', v) < d^2(y^*, v)$ . This implies that for all  $\lambda \in \Lambda$ , there exists a  $y' \in \mathcal{V}$  such that  $d^2(y', \lambda) < d^2(y^*, \lambda)$  and therefore that  $y^* \notin \Pi(\Lambda)$  which contradicts that  $y^* \in \Pi(\Lambda)$ . It must be that  $\forall y^* \in \mathcal{V}$  where  $y^* \in \Pi(\Lambda)$ , we have  $y^* \in \Pi(\{\lambda_i, \lambda_j\}) \forall \lambda_i, \lambda_j \in \Lambda$ .
A.6 Proof of Theorem A.8 (minimum locus cover for grid graphs)
Theorem A.8 (Minimum locus cover for the grid graphs). The minimum locus cover for a grid graph is the pair of vertices in the furthest opposite corners.
Proof. We would like to show two things:
1. any vertex can be resolved by construction of  $\pmb{w}$  using the furthest pair of two vertices (i.e. opposite corners) and
2. there does not exist a locus cover of size one for grid graphs with more than one vertex.
Let  $G = (\mathcal{V}, \mathcal{E})$  be a grid graph and let  $\Lambda = \{\lambda_1, \lambda_2\}$  be a set of two vertices who are on opposite corners of  $G$  (i.e., peripheral vertices achieving the diameter of  $G$ ). For notational convenience, set  $\Lambda = (\lambda_1, \lambda_2)$  We can reach any interior vertex by an appropriate setting the weight vector  $\boldsymbol{w} = [\boldsymbol{w}_1, \boldsymbol{w}_2]$ . Then set  $\boldsymbol{w} = \left(\frac{d(v, \lambda_2)}{d(\lambda_1, \lambda_2)}, \frac{d(v, \lambda_1)}{d(\lambda_1, \lambda_2)}\right)$ . Finally, the Fréchet mean is given by
$$
\begin{array}{l} m_{\boldsymbol{\Lambda}}(\boldsymbol {w}) = \operatorname *{arg  min}_{y\in \mathcal{Y}}\frac{d(v,\lambda_{2})d^{2}(y,\lambda_{1}) + d(v,\lambda_{1})d^{2}(y,\lambda_{2})}{d(\lambda_{1},\lambda_{2})} \\ = \operatorname * {a r g   m i n} _ {y \in \mathcal {Y}} d (v, \lambda_ {2}) d ^ {2} (y, \lambda_ {1}) + d (v, \lambda_ {1}) d ^ {2} (y, \lambda_ {2}) \\ \ni v. \\ \end{array}
$$
Hence  $\Lambda$  is a locus cover. We can clearly see that  $\Lambda$  is a minimum locus cover—the only way to obtain a smaller set  $\Lambda'$  is for it to include only a single vertex. However, if  $\Lambda' = \{v'\}$  contains only a single vertex, it cannot be a locus cover so long as  $G$  contains more than one vertex— $v'$  is always guaranteed to be a unique minimizer of the Fréchet mean under  $\Lambda'$  which misses all other vertices in  $G$ . Thus  $\Lambda$  is a minimum locus cover.
A.7 Proof of Lemma A.9 (loci of grid subspaces)
Lemma A.9 (Locus of grid subspaces). Given any pair of vertices in  $\Lambda$ , we can find a subset  $G'$  of the original grid graph  $G = (\mathcal{V}, \mathcal{E})$  which takes the given pair as two corners.  $\Pi(\Lambda)$  equals to all the points inside  $G'$ .
Proof. The result follows from application of Theorem A.8 to the choice of metric subspace.
A.8 Proof of Lemma A.10 (grid pairwise decomposability)
Lemma A.10 (Grid pairwise decomposability). Let  $G = (\mathcal{V}, \mathcal{E})$  be a grid graph and  $\Lambda \subseteq \mathcal{V}$ . Then  $\Pi(\Lambda)$  is pairwise decomposable.
Proof. Suppose we have a grid graph  $G = (\mathcal{V}, \mathcal{E})$  and a vertex  $\hat{y} \in \Pi(\Lambda)$  with  $\Lambda \subseteq \mathcal{V}$ . Due to the fact that  $\hat{y} \in \Pi(\Lambda)$  and the fact that  $G$  is a grid graph, we have that  $\hat{y} \in \Gamma(\lambda_{\alpha}, \lambda_{\beta})$  for some  $\lambda_{\alpha}, \lambda_{\beta} \in \Lambda$ . Then by Lemma A.10,
$$
\hat {y} \in \Pi (\left\{\lambda_ {\alpha}, \lambda_ {\beta} \right\}) \subseteq \cup_ {\lambda_ {i}, \lambda_ {j} \in \Lambda} \Pi (\left\{\lambda_ {i}, \lambda_ {j} \right\}).
$$
Therefore loci on grid graphs are pairwise decomposable.
A.9 Proof of Theorem A.11 (no nontrivial locus covers for complete graphs)
Theorem A.11 (Trivial locus cover for the complete graph). There is no non-trivial locus cover for the complete graph.
Proof. We show that there is no nontrivial locus cover for complete graphs by showing that removing any vertex from the trivial locus cover renders that vertex unreachable. We proceed by strong induction on the number of vertices,  $n$ .
Base case We first set  $n = 3$ . Let  $K_{3} = (\mathcal{V},\mathcal{V}^{2})$  be the complete graph with three vertices:  $\mathcal{V} = \{v_1,v_2,v_3\}$ , and without loss of generality, let  $\Lambda = \{v_{2},v_{3}\}$  be our set of observed classes. There are two cases on the weight vector  $\boldsymbol {w} = [w_{2},w_{3}]$ :
Case 1: Suppose  $\pmb{w} \notin \mathrm{int}\Delta^2$ . This means that either  $w_{2} = 1$  or  $w_{3} = 1$  which leads to the Fréchet mean being either  $v_{2}$  or  $v_{3}$ , respectively. Neither of these instances correspond to  $v_{1}$  being a minimizer.
Case 2: Suppose  $\pmb{w} \in \mathrm{int}\Delta^2$ . Then the Fréchet mean is given by
$$
m_{\boldsymbol{\lambda}}(\boldsymbol {w}) = \operatorname *{arg  min}_{y\in \mathcal{Y}}w_{2}d^{2}(y,v_{2}) + w_{3}d^{2}(y,v_{3})
$$
Assume for contradiction that  $v_{1} \in \Pi(\Lambda)$ :
$$
\begin{array}{l} w _ {2} d ^ {2} (v _ {1}, v _ {2}) + w _ {3} d ^ {2} (v _ {1}, v _ {3}) = w _ {2} + w _ {3} \\ > w _ {3} \quad \text {b e c a u s e} \boldsymbol {w} \in \operatorname {i n t} \Delta^ {2} \\ = w _ {2} d ^ {2} \left(v _ {2}, v _ {2}\right) + w _ {3} d ^ {2} \left(v _ {2}, v _ {3}\right). \\ \end{array}
$$
Therefore,  $v_{1} \notin \Pi(\Lambda)$ . This is a contradiction. Thus there is no nontrivial locus cover for  $K_{3}$ .
Inductive step: Let  $\pmb{K}_{n-1} = (\mathcal{V}, \mathcal{V}^2)$  be the complete graph with  $n-1$  vertices. Assume that there is no nontrivial locus cover for  $\pmb{K}_{n-1}$ . We will show that there is no nontrivial locus cover for  $\pmb{K}_n$ . Let the weight vector be  $\pmb{w} = [w_1, \dots, w_{n-1}]$  corresponding to vertices  $v_1, \dots, v_{n-1} \in \mathcal{V}$  with  $\Lambda = \{v_1, \dots, v_{n-1}\}$ . We want to show that  $v_n \notin \Pi(\Lambda)$ . We proceed by cases on  $\pmb{w}$ .
Case 1: Suppose  $\pmb{w} \notin \mathrm{int}\Delta^{n - 2}$  where  $m$  entries of  $\pmb{w}$  are zero, then by strong induction we know that  $v_{n} \notin \Pi(\Lambda) = \{v_{i}\}_{i=1}^{n-m}$ , i.e., there is no nontrivial locus cover for  $K_{n-m}$ .
Case 2: Suppose  $\pmb{w} \in \mathrm{int}\Delta^{n - 2}$ .
Assume for contradiction that  $v_{n} \in \Pi(\Lambda)$ . Using this assumption, we obtain the following
$$
\begin{array}{l} \sum_ {i = 1} ^ {n - 1} w _ {i} d ^ {2} (v _ {n}, v _ {i}) = \sum_ {i = 1} ^ {n - 1} w _ {i} \\ > \sum_ {i = 1} ^ {n - 2} w _ {i} \quad \text {b e c a u s e} \boldsymbol {w} \in \operatorname {i n t} \Delta^ {n - 2} \\ = \sum_ {i = 1} ^ {n - 1} w _ {i} d ^ {2} (v _ {n - 1}, v _ {i}). \\ \end{array}
$$
Therefore,  $v_{n}$  is not a minimizer, so  $v_{n} \notin \Pi(\Lambda)$ . This is a contradiction, hence  $K_{n}$  has no nontrivial locus cover.
A.10 Proof of Theorem 4.8 (active next-class selection for trees)
Proof. We will prove the result by showing that the two optimization problems are equivalent. Let  $v$  be a solution to the following optimization problem:
$$
\begin{array}{l l} \arg \max  & d (y, b) \\ y \in \mathcal {Y} \setminus \Pi (\Lambda) \\ \text {s . t .} & b \in \partial_ {\mathrm {i n}} T ^ {\prime}, \\ & \Gamma (b, y) \setminus \{b \} \subseteq \mathcal {Y} \setminus \Pi (\Lambda), \end{array}
$$
where  $\partial_{\mathrm{in}}T^{\prime}$  is the inner boundary of  $T^{\prime}$ , the subgraph of  $T$  whose vertices are  $\Pi (\Lambda)$ . This optimization problem can be equivalently rewritten as
$$
\begin{array}{l l} \arg \max  & | \Gamma (y, b) | \\ y \in \mathcal {Y} \setminus \Pi (\Lambda) \\ \text {s . t .} & b \in \partial_ {\mathrm {i n}} T ^ {\prime}, \\ & \Gamma (b, y) \setminus \{b \} \subseteq \mathcal {Y} \setminus \Pi (\Lambda), \end{array}
$$
and we can furthermore introduce additional terms that do not change the maximizer
$$
\begin{array}{l l} \arg \max  _ {y \in \mathcal {Y} \setminus \Pi (\Lambda)} & | \cup_ {\lambda_ {i}, \lambda_ {j} \in \Lambda} \Gamma (\lambda_ {i}, \lambda_ {j}) \cup \Gamma (b, y) | \\ \text {s . t .} & b \in \partial_ {\mathrm {i n}} T ^ {\prime}, \\ & \Gamma (b, y) \setminus \{b \} \subseteq \mathcal {Y} \setminus \Pi (\Lambda). \end{array}
$$
Equivalently, we can also connect  $b$  to one of the elements of  $\Lambda$
$$
\begin{array}{l l} \arg \max  _ {y \in \mathcal {Y} \setminus \Pi (\Lambda)} & | \cup_ {\lambda_ {i}, \lambda_ {j} \in \Lambda} \Gamma (\lambda_ {i}, \lambda_ {j}) \cup \Gamma (\lambda_ {i}, b) \cup \Gamma (b, y) | \\ \text {s . t .} & b \in \partial_ {\mathrm {i n}} T ^ {\prime}, \\ & \Gamma (b, y) \setminus \{b \} \subseteq \mathcal {Y} \setminus \Pi (\Lambda). \end{array}
$$
Due to the uniqueness of paths in trees, this optimization problem also has the following equivalent form without any dependence on  $b$ :
$$
\begin{array}{l} v \in \operatorname * {a r g   m a x} _ {y \in \mathcal {Y} \backslash \Lambda} | \cup_ {\lambda_ {i}, \lambda_ {j} \in \Lambda} \Gamma (\lambda_ {i}, \lambda_ {j}) \cup \Gamma (\lambda_ {i}, y) | = \operatorname * {a r g   m a x} _ {y \in \mathcal {Y} \backslash \Lambda} | \cup_ {\lambda_ {i}, \lambda_ {j} \in \Lambda} \Pi (\{\lambda_ {i}, \lambda_ {j} \}) \cup \Pi (\{\lambda_ {i}, y \}) | \\ = \underset {y \in \mathcal {Y} \backslash \Lambda} {\arg \max } | \cup_ {\lambda_ {i}, \lambda_ {j} \in \Lambda \cup \{y \}} \Pi (\{\lambda_ {i}, \lambda_ {j} \}) | \\ = \underset {y \in \mathcal {Y} \backslash \Lambda} {\arg \max } | \Pi (\Lambda \cup \{y \}) | \\ \end{array}
$$
using Lemma A.5.
Therefore  $v$  is a maximizer of  $\left|\Pi (\Lambda \cup \{v\})\right|$ , as required.
B Algorithms and Time Complexity Analyses
We provide time complexity analyses for Algorithms B.1, B.2, and B.3.
B.1 Analysis of Algorithm B.1 (locus cover for phylogenetic trees)
We provide Algorithm B.1 with comments corresponding to the time complexity of each step.
Algorithm 1 Locus cover for phylogenetic trees
Require: phylogenetic tree  $T = (\mathcal{V},\mathcal{E}),\mathcal{V} = \text{Leaves}(T)$ $N\gets |\mathcal{V}|$ $P\gets$  sortbylength  $\left[\left[\Gamma (y_{i},y_{j})\right]_{i,j\in [N]}\right)$ $P\gets$  reverse  $(P)$ $\Lambda \gets \emptyset$   
for  $\Gamma (y_i,y_j)$  in  $P$  do  
if  $\Pi (\Lambda) = \mathcal{V}$  then  
return  $\Lambda$   
else  
 $\Lambda \gets \Lambda \cup \{y_i,y_j\}$   
end if  
end for
Combining these, we obtain the following time complexity:
$$
O (N | \mathcal {E} | + N ^ {2} \log N + N ^ {2} + N ^ {2} K ^ {2} D \max  \{N | \mathcal {E} |, N ^ {2} \log N \}) = O (N ^ {2} K ^ {2} D \max  \{N | \mathcal {E} |, N ^ {2} \log N \}).
$$
B.2 Analysis of Algorithm B.2 (computing a pairwise decomposable locus)
We first provide Algorithm B.2 here, with comments corresponding to the time complexity of each step.
Algorithm 2 Computing a pairwise decomposable locus
Require:  $\Lambda ,\mathcal{V},G = (\mathcal{V},\mathcal{E})$ $\Pi \leftarrow \emptyset$ $D\gets \mathrm{diam}(G)$  ▷  $O(N|\mathcal{E}| + N^2\log N)$    
for  $\lambda_{i},\lambda_{j}\in \Lambda$  do ▷  $O(K^{2})$    
for  $w_{1}$  in  $\{\frac{0}{D},\frac{1}{D},\dots,\frac{D}{D}\}$  do ▷  $O(D)$ $\mathbf{w}\gets [w_1,1 - w_1]$ $\Pi \leftarrow \Pi \cup m_{\{\lambda_i,\lambda_j\}}(w)$  ▷  $O(N|\mathcal{E}| + N^2\log N)$    
end for   
end for   
return II
We first compute the diameter of the graph  $G = (\mathcal{V}, \mathcal{E})$  with  $N = |\mathcal{V}|$ , which is done in  $O(N|\mathcal{E}| + N^2\log N)$  time using Dijkstra's algorithm to compute the shortest paths between all pairs of vertices. We then iterate over all pairs of elements in  $\Lambda$  with  $K = |\Lambda|$ , which amounts to  $O(K^2)$  iterations. Within this, we perform  $O(D)$  computations of the Fréchet mean, for which each iteration requires  $O(N|\mathcal{E}| + N^2\log N)$  arithmetic operations or comparisons. Combining these, the final time complexity is
$$
O (N | \mathcal {E} | + N ^ {2} \log N + K ^ {2} D (N | \mathcal {E} | + N ^ {2} \log N)) = O (K ^ {2} D \max \{N | \mathcal {E} |, N ^ {2} \log N \}).
$$
B.3 Analysis of Algorithm B.3 (computing a generic locus)
We provide Algorithm B.3 with comments corresponding to the time complexity of each step.
Following a similar argument from the analysis of Algorithm B.2, the time complexity is
$$
O (N | \mathcal {E} | + N ^ {2} \log N + D ^ {K} (N | \mathcal {E} | + N ^ {2} \log N)) = O (D ^ {K}),
$$
Algorithm 3 Computing a generic locus
Require:  $\Lambda, \mathcal{Y}, G = (\mathcal{V}, \mathcal{E})$
$$
\Pi \gets \emptyset
$$
$$
D \leftarrow \operatorname {d i a m} (G)
$$
for w in  $\left\{\frac{0}{D},\frac{1}{D},\dots,\frac{D}{D}\right\}^{|\Lambda|}$  do
$$
\Pi \leftarrow \Pi \cup m _ {\mathbf {A}} (w)
$$
return II
$$
\triangleright O (N | \mathcal {E} | + N ^ {2} \log N)
$$
$$
\triangleright O (D ^ {K})
$$
Figure 4: CLIP on CIFAR-100 with the WordNet hierarchy. (Left) Reliability diagrams across a range of Softmax temperatures, highlighting the CLIP default temperature, the optimal temperature for LOKI, and the minimizer of the Expected Calibration Error (ECE). All three are well-calibrated. (Center) Tradeoff between optimizing for ECE and the expected squared distance. As with the reliability diagrams, the CLIP default temperature, the LOKI-optimal temperature, and the ECE-optimal temperature are similar. (Right) Tradeoff between optimizing for zero-one error and the expected squared distance. Depending on the metric space, temperature scaling can improve accuracy.
where  $K$  is the number of observed classes.
C.1 Additional calibration experiments
In the main text, we evaluated the effect of calibrating the Softmax outputs on the performance of LOKI using a metric space based on the internal features of the CLIP model. Here, we perform the same analysis using two external metric spaces.
Setup We perform our calibration analysis using the default and WordNet tree.
Results Our results are shown in Figure 4. Like our experiment using an internally-derived metric, we find that the optimal Softmax temperature is close to the CLIP default and the optimal temperature for calibration. For the default tree, we again found that temperature scaling can be used to improve accuracy using LOKI.
C.2 Ablation of LOKI formulation
LOKI is based on the Fréchet mean, which is defined as  $\arg \min_{y\in \mathcal{Y}}\sum_{i = 1}^{K}\mathbf{P}_{\lambda_i|x}d^2 (y,\lambda_i)$ . However, this is not the only approach that can be considered. For example, the Fréchet median, often used in robust statistics, is defined as  $\arg \min_{y\in \mathcal{Y}}\sum_{i = 1}^{K}\mathbf{P}_{\lambda_i|x}d(y,\lambda_i)$ , without squaring the distances. More generally, we define  $\arg \min_{y\in \mathcal{Y}}\sum_{i = 1}^{K}\mathbf{P}_{\lambda_i|x}d^\beta (y,\lambda_i)$  and evaluate different choices of  $\beta$ .
Setup We conduct this experiment on ImageNet using SimCLR as our pretrained classifier with  $K = 250, 500$ , and 750 randomly selected classes.
Results From this analysis, we conclude that using the Fréchet mean is the optimal formulation for Loki, as it achieved the lowest mean squared distance for all settings of  $K$ .
Table 4: Expected squared distances of SimCLR+Loki alternatives on ImageNet. We ablate over the choice of distance exponent in LOKI (where  $\beta = 2$ , corresponding to the Fréchet mean), including the Fréchet median ( $\beta = 1$ ). That is, we tune  $\beta : \hat{y} \in \arg \min_{y \in \mathcal{Y}} \sum_{i=1}^{K} \mathbf{P}_{\lambda_i|x} d^\beta(y, \lambda_i)$  and find that the optimal setting is  $\beta = 2$ , corresponding to LOKI.
β =	K = 250	K = 500	K = 750
0.5	61.28	46.57	36.31
1	52.98	41.06	33.14
2 (Loki)	46.78	37.76	29.88
4	47.99	39.58	34.65
8	65.29	58.42	54.90
C.3 Comparison across metric spaces