SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Expected squared distances cannot be directly compared across metric spaces, as they may be on different scales. Our solution is to use a form of normalization: we divide the expected squared distance by the square of the graph diameter. This brings all of the values to the 0-1 range, and since  $\mathbb{E}[d^2 (y,\hat{y})] / \mathrm{diam}(G)^2$  indeed also generalizes the 0-1 error, this enables comparison between 0-1 errors and those from different metric spaces. We provide these results in Table 1, again for our CLIP experiments on CIFAR-100. This evaluation metric enables us to determine which metric spaces have geometry best 'fit' to our pretrained models.
Setup Using  $\mathbb{E}[d^2 (y,\hat{y})] / \mathrm{diam}(G)^2$ , we re-evaluate our CLIP results on CIFAR-100 shown in Table 1 for the ResNet-50 architecture.
Results For CIFAR-100, we observe that the WordNet metric space resulted in the lowest error and therefore has the best geometry.
Table 5: Comparison across metric spaces for CLIP on CIFAR-100 by normalizing by the squared diameter of the metric space:  $\mathbb{E}[d^2 (y,\hat{y})] / \mathrm{diam}(G)^2$ .
Metric space G	diam(G)2	E[d2(y,haty)]/diam(G)2
Complete graph	1	0.5941 (0-1 error)
Default tree	16	0.4493
WordNet tree	169	0.1157
CLIP features	0.9726	0.2686
In this section, we provide additional details about our experimental setup.
D.1 CLIP experiments
All experiments are carried out using CLIP frozen weights. There are no training and hyperparameters involved in experiments involving CLIP, except for the Softmax temperature in the calibration analysis. We evaluated using the default CIFAR-100 test set. The label prompt we use is "a photo of a [class_name]."
D.2 ImageNet experiments
To construct our datasets, we randomly sample 50 images for each class from ImageNet as our training dataset then use the validation dataset in ImageNet to evaluate LOKI's performance. We extract the image embeddings by using SimCLRv1 $^4$  and train a baseline one-vs-rest classifier. We use WordNet phylogenetic tree as the metric space. The structure of phylogenetic tree can be found at here $^5$ . We test different numbers of observed classes,  $K$ , from 1000 classes. Observed classes are sampled in two ways, uniformly and Gibbs distribution with the concentration parameter 0.5.
D.3 LSHTC experiments
We generate a summary graph of the LSHTC class graph (resulting in supernodes representing many classes) by iteratively:
1. randomly selecting a node or supernode
2. merging its neighbors into the node to create a supernode
until the graph contains at most 10,000 supernodes. In the LSHTC dataset, each datapoint is assigned to multiple classes. We push each class to its supernode, then apply majority vote to determine the final class. We test different numbers of observed classes,  $K$ , from 10,000 classes. We collect datapoints which are in the observed classes. Then split half of dataset as training dataset and make the rest as testing dataset, including those datapoints which are not in the observed classes. We train a baseline classifier using a 5-NN model and compare its performance with LOKI.
E Broader Impacts and Limitations
As a simple adaptor of existing classifiers, it is possible for our method to inherit biases and failure modes that might be present in existing pretrained models. On the other hand, if the possibility of harmful mis-classifications are known a priori, information to mitigate these harms can be baked into the metric space used by LOKI. Finally, while we have found that LOKI often works well in practice, it is possible for the per-class probabilities that are output by a pretrained model to be sufficiently mis-specified relative to the metric over the label space, or for the metric itself to be mis-specified. Nonetheless, we have found that off-the-shelf or self-derived metric spaces to work well in practice.
F Random Locus Visualizations
In Figures 1, 5, 6, 7, 8, and 9, we provide visualizations of classification regions of the probability simplex when using LOKI with only three observed classes out of 100 total classes, and different types of random metric spaces. The first example in Figure 1 shows the prediction regions on the probability simplex when using standard arg max prediction—the three regions correspond to predicting one of the three classes (0, 39, and 99) and no regions corresponding to any of the other classes \{1, ..., 38, 40, ..., 98\}. We compute these regions using a version of Algorithm B.3, and while
$^{3}$ Code implementing all of our experiments is available here: https://github.com/SprocketLab/loki.  $^{4}$ Embeddings are extracted from the checkpoints stored at: https://github.com/tonylins/simblr-converter
<sup>5</sup>https://github.com/cvjena/semantic-embeddings/tree/master/ILSVRC
it does have exponential time complexity, the exponent is only three in this case since we consider only three observed classes.
On the other hand, the other three examples in Figure 1 and Figures 5, 6, 7, 8, and 9 all show the prediction regions on the probability simplex when using LOKI. Figure 5 shows this for random tree graphs. Here, the prediction regions are striped or contain a single triangle-shaped region in the center—these correspond, respectively, to intermediate classes along branches of the tree leading up from the observed class and the prediction region formed by the common-most parent node. Figure 6 shows similar regions, although these are more complex and are thus more difficult to interpret, as phylogenetic trees are metric subspaces equipped with the induced metric from trees. Furthermore, in order to generate phylogenetic trees with 100 leaves, we needed to create much larger trees than the ones used for Figure 5, which led to narrower prediction regions due to the higher graph diameter. Finally, Figures 7, 8, and 9 each show the prediction regions using LOKI when the metric space is a random graph produced using Watts-Strogatz, Erdős-Rényi, and Barabási-Albert, models respectively. These prediction regions are more complex and represent complex relationships between the classes.
Figure 5: Classification regions in the probability simplex of 3-class classifiers faced with a 100-class problem where the classes are related by random trees graphs.
Figure 6: Classification regions in the probability simplex of 3-class classifiers faced with a 100-class problem where the classes are related by random phylogenetic trees graphs.
Figure 7: Classification regions in the probability simplex of 3-class classifiers faced with a 100-class problem where the classes are related by random small-world graphs generated by a Watts-Strogatz model.
Figure 8: Classification regions in the probability simplex of 3-class classifiers faced with a 100-class problem where the classes are related by random Erdős-Rényi graphs.
Figure 9: Classification regions in the probability simplex of 3-class classifiers faced with a 100-class problem where the classes are related by random Barabási-Albert graphs.