SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Trees Many label spaces are endowed with a tree metric in practice: hierarchical classification, in which the label space includes both classes and superclasses, partial labeling problems in which internal nodes can represent the prediction of a set of classes, and the approximation of complex or intractable metrics using a minimum spanning tree. We show that for our purposes, trees have certain desirable properties that make them easy to use with LOKI—namely that we can easily identify a locus cover that satisfies both Definition 4.5 and Definition 4.6. Conveniently, we also show that any locus in any tree satisfies Definition 4.7.
We first note that the leaves of any tree yield the minimum locus cover. This is a convenient property-- any label from any label space endowed with a tree metric can be predicted using LOKI using only classifiers trained using labels corresponding to the leaves of the metric space. This can be especially useful if the tree has long branches and few leaves. Additionally, for tree metric spaces, the minimum locus cover (Definition 4.5) is also an identifying locus cover (Definition 4.6). This follows from the construction of the weights in the proof of Theorem A.4 (shown in the Appendix) and the property that all paths in trees are unique. Finally, we note that any locus in any tree is pairwise decomposable--the proof of this is given in the Appendix (Lemma A.5). We will see later that this property yields an efficient algorithm for computing the locus.
Phylogenetic Trees Image classification datasets often have a hierarchical tree structure, where only the leaves are actual classes, and internal nodes are designated as superclasses—examples include the ImageNet [6] and CIFAR-100 datasets [20]. Tree graphs in which only the leaf nodes are labeled are referred to as phylogenetic trees [3]. Often, these graphs are weighted, but unless otherwise mentioned, we assume that the graph is unweighted.
For any arbitrary tree  $T = (\mathcal{V}, \mathcal{E})$ , the set of labels induced by phylogenetic tree graph is  $\mathcal{V} = \text{Leaves}(T)$ . We provide a heuristic algorithm for obtaining locus covers for arbitrary phylogenetic trees in Algorithm B.1 (see Appendix). We prioritize adding endpoints of long paths to  $\Lambda$ , and continue adding nodes in this way until  $\Pi(\Lambda) = \mathcal{V}$ . Similarly to tree metric spaces, any phylogenetic tree metric space is pairwise decomposable. We prove the correctness of Algorithm B.1 and pairwise
decomposability of phylogenetic trees in the Appendix (Theorem A.6 and Lemma A.7). Later, we give algorithms for computing the set of nodes in an arbitrary locus in arbitrary graphs—if the locus is pairwise decomposable, the algorithm for doing so is efficient, and if not, it has time complexity exponential in  $K$ . Due to the pairwise decomposability of phylogenetic trees, this polynomial-time algorithm to compute  $\Pi(\Lambda)$  applies.
Grid Graphs Classes often have a spatial relationship. For example, classification on maps or the discretization of a manifold both have spatial relationships—grid graphs are well suited to these types of spatial relationships. We obtain minimum locus covers for grid graphs satisfying Definition 4.5, but we find that these are not generally identifying locus covers. On the other hand, we give an example of a simple identifying locus cover satisfying Definition 4.6. Again, we find that grid graphs are in general pairwise decomposable and hence follow Definition 4.7.
We find that the pair of vertices on furthest opposite corners yields the minimum locus cover. While the set of vertices given by Theorem A.8 (found in the Appendix) satisfies Definition 4.5, this set does not in general satisfy Definition 4.6. This is because the path between any two vertices is not unique, so each minimum path of the same length between the pair of vertices can have an equivalent minimizer. On contrast, the following example set of vertices satisfies Definition 4.6 but it clearly does not satisfy Definition 4.5. Example: Given a grid graph, the set of all corners is an identifying locus cover. On the other hand, the vertices given by Theorem A.8 can be useful for other purposes. Lemma A.9 (provided in the Appendix) shows that subspaces of grid graphs can be formed by the loci of pairs of vertices in  $\Lambda$ . This in turn helps to show that loci in grid graphs are pairwise decomposable in general (see Lemma A.10 in the Appendix).
The Complete Graph The standard classification setting does not use relational information between classes. As before, we model this setting using the complete graph, and we show the expected result that in the absence of useful relational information, LOKI cannot help, and the problem once again becomes standard multiclass classification among observed classes. To do so, we show that there is no nontrivial locus cover for the complete graph (Theorem A.11 in the Appendix).
4.3 Label Subspaces in Practice
While it is desirable for the set of observed classes to form a minimum or identifying locus cover, it is often not possible to choose the initial set of observed classes a priori—these are often random. In this section, we describe the more realistic cases in which a random set of classes are observed and an active learning-based strategy to choose the next observed class. The aim of our active learning approach is, instead of randomly selecting the next observed class, to actively select the next class so as to maximize the total size of the locus—i.e., the number of possible classes that can be output using LOKI. Before maximizing the locus via active learning, we must first address a much more basic question: can we even efficiently compute the locus?
Computing the Locus We provide algorithms for obtaining the set of all classes in the locus, given a set of classes  $\Lambda$ . We show that when the locus is pairwise decomposable (Definition 4.7), we can compute the locus efficiently using a polynomial time algorithm. When the locus is not pairwise decomposable, we provide a general algorithm that has time complexity exponential in  $|\Lambda|$ —we are not aware of a more efficient algorithm. We note that any locus for every type of graph that we consider in Section 4.2 is pairwise decomposable, so our polynomial time algorithm applies. Algorithms B.2 and B.3 along with their time complexity analyses can be found in the Appendix.
Large Locus via Active Next-Class Selection We now turn to actively selecting the next class to observe in order to maximize the size of the locus. For this analysis, we focus on the active learning setting when the class structure is a tree graph, as tree graphs are generic enough to apply to a wide variety of cases—including approximating other graphs using the minimum spanning tree. Assume the initial set of  $K$  observed classes are sampled at random from some distribution. We would like to actively select the  $K + 1$ st class such that  $|\Pi(\Lambda)|$  with  $\Lambda = \{\lambda\}_{i=1}^{K+1}$  is as large as possible.
Theorem 4.8. Let  $T = (\mathcal{V}, \mathcal{E})$  be a tree graph and let  $\Lambda \subseteq \mathcal{V}$  with  $K = |\Lambda|$ . Let  $T'$  be the subgraph of the locus  $\Pi(\Lambda)$ . The vertex  $v \in \mathcal{V} \setminus \Lambda$  that maximizes  $|\Pi(\Lambda \cup \{v\})|$  is the solution to the following optimization problem: arg  $\max_{y \in \mathcal{V} \setminus \Pi(\Lambda)} d(y, b)$  s.t.  $b \in \partial_{in} T'$  and  $\Gamma(y, b) \setminus \{b\} \subseteq \mathcal{V} \setminus \Pi(\Lambda)$ . where  $\partial_{in} T'$  is the inner boundary of  $T'$  (all vertices in  $T'$  that share an edge with vertices not in  $T'$ ).
Table 1: CIFAR-100. Improving CLIP predictions using LOKI. Results are reported as  $\mathbb{E}[d^2 (y,\hat{y})]$  in the respective metric space. CLIP-like zero-shot models can be improved using LOKI even without access to an external metric, and internal class embedding distances are used. When an external metric is available, LOKI outperforms CLIP using the default CIFAR-100 hierarchy and WordNet.
Model	Metric Space	arg max	LOKI	Relative Improvement
CLIP-RN50 [32]	Internal	0.2922	0.2613	10.57%
CLIP-ViT-L-14 [32]	Internal	0.1588	0.1562	1.63%
ALIGN [16]	Internal	0.1475	0.1430	3.02%
CLIP-RN50 [32]	K100	0.5941*	0.5941*	0.0%*
CLIP-RN50 [32]	Default tree	7.3528	7.1888	2.23%
CLIP-RN50 [32]	WordNet tree	24.3017	19.5549	19.53%
* methods equivalent under the complete graph as LOK1 reduces to arg max prediction.
This procedure can be computed in polynomial time—solving the optimization problem in Theorem 4.8 simply requires searching over pairs of vertices. Hence we have provided an efficient active learning-based strategy to maximize the size of the locus for trees.
In this section, we provide experimental results to validate the following claims:
1. LOKI improves performance of zero-shot foundation models even with no external metric.
2. LOKI adapts to label spaces with a large number of unobserved classes.
3. The active approach given in Theorem 4.8 yields a larger locus than the passive baseline.
4. The same active approach yields better performance on ImageNet.
5. With LOKI, calibration can improve accuracy, even with no external metric.
5.1 LOKI Improves Zero-Shot Models
We evaluate the capability of LOKI to improve upon zero-shot models where all classes are observed.
Setup Our experiment compares the zero-shot prediction performance of CLIP [32] on CIFAR-100 [20] to CLIP logits used with LOKI. First, we consider the setting in which no external metric relating the labels is available, and instead derive internal metric information from Euclidean distances between text embeddings from the models using their respective text encoders. Second, we consider three external metric spaces for use with LOKI: the complete graph  $K_{100}$  and two phylogenetic trees: the default CIFAR-100 superclasses [20] and WordNet [2].
Results The results of this experiment are given in Table 1. When no external metric information is available, LOKI still outperforms CLIP-like models that use the standard prediction rule—in other words, LOKI seems to unconditionally improve CLIP. As expected, under the complete graph, our method becomes equivalent to the standard prediction mechanism used by CLIP. On the other hand, LOKI outperforms CLIP when using the default CIFAR-100 tree hierarchy and even more so when using the WordNet geometry, with a  $19.53\%$  relative improvement in mean squared distance over the CLIP baseline. We postulate that the strong performance using WordNet is due to the richer geometric structure compared to that of the default CIFAR-100 hierarchy.
5.2 LOKI on Partially-Observed Label Spaces
To validate our approach on partially observed label spaces, we evaluate the performance of adapting a logistic classifier trained on SimCLR embeddings of ImageNet [6], 5-NN models trained on a 9,419 class subset of the PubMed dataset, $^{2}$  and the 325,056-class LSHTC dataset [30].
$^{2}$ https://www.kaggle.com/datasets/owaiskhan9654/pubmed-multilabel-text-classification
Figure 2: (Top left) ImageNet. Mean squared distances under uniform class sampling and under the Gibbs distribution—LOKI improves upon the baseline SimCLR one-vs-rest classifier. (Top right) Synthetic. Active class selection consistently leads to larger loci compared to uniform sampling. (Bottom) Active selection on ImageNet. Active class selection improves performance on ImageNet.
Setup For ImageNet, we use the WordNet phylogenetic tree as the metric space [2]. In this setting, we sample random subsets of size  $K$  of the 1000 ImageNet classes and compare a baseline one-vs-rest classifier to the same classifier but using LOKI to adapt predictions to classes beyond the original  $K$ . We conduct two sets of experiments. In the first, we sample  $K$  classes uniformly, while in the second, we adopt a more realistic sampler—we sample from a Gibbs distribution:  $P(\lambda | \theta) = \frac{1}{Z} \exp(-\theta d(\lambda, \lambda^c))$ , where  $\lambda^c = m_{\mathcal{Y}}(\mathbf{1}_N)$  is the centroid of the metric space,  $\theta$  is the concentration parameter around the centroid, and  $Z$  is the normalizer. While the Gibbs distribution sampler is more realistic, it is also the more challenging setting—classes which have low probability according to this distribution are less likely to appear in the locus. For PubMed, we derive our metric from Euclidean distances between SimCSE class embeddings [11]. Finally for LSHTC, we summarize the default graph by randomly selecting nodes and merging them with their neighbors until we obtain a graph with 10,000 supernodes representing sets of classes.
Results Figure 2 shows the mean squared distances compared to the baseline one-vs-rest classifiers, across various settings of  $K$ . We find that LOK1 always significantly outperforms the baseline, even in the more challenging setting of sampling
Table 2: PubMed. LOKI improves baseline for all settings of  $K$ . The metric space is Euclidean distances applied to SimCSE embeddings.
K	5-NN	5-NN + LOKI
100	1.68666	1.42591
250	1.52374	1.47801
500	1.64074	1.45921
Table 3: LSHTC. LOKI improves baseline for all settings of  $K$ . We summarize the metric space graph by generating 10,000 supernodes.
K	5-NN	5-NN + LOKI
50	1.2519	0.2896
100	1.4476	0.3467
150	1.7225	0.3969
200	1.7092	0.3171
250	1.6465	0.3995
300	1.6465	0.4004
according to a Gibbs distribution. Tables 2 and 3 show our improvements when using LOKI over the baseline 5-NN models. While LOKI consistently yields an improvement on PubMed and LSHTC, the improvement is more dramatic on LSHTC.
5.3 Large Loci via Active Learning
We evaluate our active next class selection approach on increasing the locus size. We expect that compared to passive (random) selection, our active approach will lead to larger loci, and in practice, a larger set of possible classes that can be predicted using LOKI while observing fewer classes during training.
Setup We compare with a baseline that randomly selects the next class. We first generate a synthetic random tree with size  $N$  and fix an initial  $K$ . In active selection, we use the approach described in Theorem 4.8 to select the next node. As a passive baseline, we randomly sample (without replacement) the remaining nodes that are not in  $\Lambda$ .
Figure 3: CLIP on CIFAR-100 with no external metric. (Left) Reliability diagrams across a range of Softmax temperatures, highlighting the CLIP default temperature, the optimal temperature for LOKI, and the minimizer of the Expected Calibration Error (ECE). All three are well-calibrated. (Center) Tradeoff between optimizing for ECE and the expected squared distance. As with the reliability diagrams, the CLIP default temperature, the LOKI-optimal temperature, and the ECE-optimal temperature are similar. (Right) Tradeoff between optimizing for zero-one error and the expected squared distance. Temperature can be tuned to improve accuracy when using LOKI.
Results The results are shown in Figure 2. We set  $N = 100$  and the initial  $K = 3$ , then we average the result over 10 independent trials. The active approach consistently outperforms the random baseline as it attains a larger locus with fewer nodes selected.
5.4 Improving Performance via Active Learning
Next, we evaluate the our active next class selection approach on improving error on ImageNet. While we found that the active approach indeed leads to an increased locus size compared to the passive baseline, we expect that this increased locus size will lead to improved performance.
Setup We randomly sample 500 ImageNet classes, and using the WordNet metric space, we use our active approach to iteratively sample 50 more classes. We compare this to a passive baseline in which the 50 classes are sampled randomly. We repeat this experiment over 10 independent trials.
Results Figure 2 shows that our active approach yields improved error over the passive baseline. The gap between the active approach and the passive baseline widens with more selection rounds.
5.5 Improving Accuracy via Calibration
Finally, we evaluate the effect of calibrating the Softmax outputs on the performance of LOK1.
Setup We calibrate via Softmax temperature scaling [13] using CLIP on CIFAR-100. We do not use an external metric space, and instead use Euclidean distance applied to the CLIP text encoder.
Results The reliability diagram in Figure 3 shows that the optimal Softmax temperature for LOKI is both close to the default temperature used by CLIP and to the optimally-calibrated temperature. In Figure 3 (right), we find that appropriate tuning of the temperature parameter can lead to improved accuracy with CLIP, even when no external metric space is available.
In this work, we proposed LOKI—a simple adaptor for pretrained models to enable the prediction of additional classes that are unobserved during training by using metric space information. We comprehensively answered the space of new questions that arise under LOKI in terms of learning, optimal metric space settings, and a practical active selection strategy. Experimentally, we showed that LOKI can be used to improve CLIP even without external metric space information, can be used to predict a large number of unobserved classes, and a validation of our active selection strategy.
We are grateful for the support of the NSF under CCF2106707 (Program Synthesis for Weak Supervision) and the Wisconsin Alumni Research Foundation (WARF).
[1] Gokhan Bakir, Thomas Hofmann, Bernhard Schölkopf, Alexander J. Smola, Ben Taskar, and S.V.N Vishwanathan. Predicting Structured Data. The MIT Press, July 2007.
[2] Björn Barz and Joachim Denzler. Hierarchy-based image embeddings for semantic image retrieval. In 2019 IEEE Winter Conference on Applications of Computer Vision (WACV), pages 638-647. IEEE, 2019.
[3] Louis J. Billera, Susan P. Holmes, and Karen Vogtmann. Geometry of the space of phylogenetic trees. Advances in Applied Mathematics, 27(4):733-767, 2001.
[4] Yeshwanth Cherapanamjeri, Nicolas Flammarion, and Peter L. Bartlett. Fast mean estimation with sub-gaussian rates. In Alina Beygelzimer and Daniel Hsu, editors, Proceedings of the Thirty-Second Conference on Learning Theory, volume 99 of Proceedings of Machine Learning Research, pages 786–806. PMLR, 25–28 Jun 2019.
[5] Carlo Ciliberto, Lorenzo Rosasco, and Alessandro Rudi. A consistent regularization approach for structured prediction. In Advances in Neural Information Processing Systems 30 (NIPS 2016), volume 30, 2016.
[6] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248-255, 2009.
[7] Ali Farhadi, Ian Endres, Derek Hoiem, and David Forsyth. Describing objects by their attributes. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 1778-1785, 2009.
[8] Maurice R. Fréchet. Les éléments aléatoires de nature quelconque dans un espace distancié. Annales de l'instit Henri Poincaré, 1948.
[9] Andrea Frome, Greg S Corrado, Jon Shlens, Samy Bengio, Jeff Dean, Marc' Aurelio Ranzato, and Tomas Mikolov. Devise: A deep visual-semantic embedding model. In C.J. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems, volume 26. Curran Associates, Inc., 2013.
[10] Junyu Gao, Tianzhu Zhang, and Changsheng Xu. I Know the Relationships: Zero-Shot Action Recognition via Two-Stream Graph Convolutional Networks and Knowledge Graphs. Proceedings of the AAAI Conference on Artificial Intelligence, 33(01):8303-8311, July 2019.
[11] Tianyu Gao, Xingcheng Yao, and Danqi Chen. SimCSE: Simple contrastive learning of sentence embeddings. In Empirical Methods in Natural Language Processing (EMNLP), 2021.
[12] Colin Graber and Alexander Schwing. Graph structured prediction energy networks. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
[13] Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. In Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 1321–1330. PMLR, 06–11 Aug 2017.
[14] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770-778, 2016.
[32] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages 8748-8763. PMLR, 2021.
[33] Tim Rocktäschel, Sameer Singh, and Sebastian Riedel. Injecting logical background knowledge into embeddings for relation extraction. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1119-1129, Denver, Colorado, May-June 2015. Association for Computational Linguistics.
[34] Abhinaba Roy, Deepanway Ghosal, Erik Cambria, Navonil Majumder, Rada Mihalcea, and Soujanya Poria. Improving Zero-Shot Learning Baselines with Commonsense Knowledge. Cognitive Computation, 14(6):2212-2222, November 2022.
[35] Alessandro Rudi, Carlo Ciliberto, GianMaria Marconi, and Lorenzo Rosasco. Manifold structured prediction. In Advances in Neural Information Processing Systems 32 (NeurIPS 2018), volume 32, 2018.
[36] Alessandro Rudi, Carlo Ciliberto, GianMaria Marconi, and Lorenzo Rosasco. Manifold structured prediction. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.
[37] Changho Shin, Winfred Li, Harit Vishwakarma, Nicholas Carl Roberts, and Frederic Sala. Universalizing weak supervision. In International Conference on Learning Representations, 2022.
[38] Richard Socher, Milind Ganjoo, Christopher D Manning, and Andrew Ng. Zero-shot learning through cross-modal transfer. In C.J. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems, volume 26. Curran Associates, Inc., 2013.
[39] Russell Stewart and Stefano Ermon. Label-free supervision of neural networks with physics and domain knowledge. In Thirty-First AAAI Conference on Artificial Intelligence, 2017.
[40] Roman Vershynin. Introduction to the non-asymptotic analysis of random matrices. In Compressed Sensing, 2010.
[41] Harit Vishwakarma and Frederic Sala. Lifting weak supervision to structured prediction. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
[42] X. Wang, Y. Ye, and A. Gupta. Zero-shot recognition via semantic embeddings and knowledge graphs. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6857-6866, Los Alamitos, CA, USA, jun 2018. IEEE Computer Society.
The appendix is organized as follows: in Appendix A, we provide proofs of the claims made in the main text. Next, in Appendix B, we provide algorithms and time complexity analyses for the algorithms referenced in the main text. Appendix C contains additional experimental results. In Appendix D, we provide additional details about our experimental setup. Then in Appendix E, we discuss the broader impacts and limitations of LOKI. Finally, in Appendix F, we provide more details and examples of the locus visualizations shown in Figure 1.
A.1 Proof of Theorem 4.1 (LOKI sample complexity)
In this section, we provide a formal statement and proof of our sample complexity result for LOKI, including additional required definitions and assumptions.
First, we define the required tools to prove the sample complexity bound for LOKI. For the purposes of this proof, we define the Fréchet variance as the function over which the Fréchet mean returns the minimizer.