Title: Geometry-Aware Adaptation for Pretrained Models
Venue: NeurIPS 2023
Accepted: yes
Keywords: structured prediction

learning on graphs

partially observed label spaces

high cardinality label spaces
Abstract: Machine learning models---including prominent zero-shot models---are often trained on datasets whose labels are only a small proportion of a larger label space. Such spaces are commonly equipped with a metric that relates the labels via distances between them. We propose a simple approach to exploit this information to adapt the trained model to reliably predict new classes---or, in the case of zero-shot prediction, to improve its performance---without any additional training. Our technique is a drop-in replacement of the standard prediction rule, swapping $\text{argmax}$ with the Fréchet mean. We provide a comprehensive theoretical analysis for this approach, studying (i) learning-theoretic results trading off label space diameter, sample complexity, and model dimension, (ii) characterizations of the full range of scenarios in which it is possible to predict any unobserved class, and (iii) an optimal active learning-like next class selection procedure to obtain optimal training classes for when it is not possible to predict the entire range of unobserved classes. Empirically, using easily-available external metrics, our proposed approach, Loki, gains up to 29.7% relative improvement over SimCLR on ImageNet and scales to hundreds of thousands of classes. When no such metric is available, Loki can use self-derived metrics from class embeddings and obtains a 10.5% improvement on pretrained zero-shot models such as CLIP.

Geometry-Aware Adaptation for Pretrained Models

Abstract

Machine learning models-including prominent zero-shot models-are often trained on datasets whose labels are only a small proportion of a larger label space. Such spaces are commonly equipped with a metric that relates the labels via distances between them. We propose a simple approach to exploit this information to adapt the trained model to reliably predict new classes-or, in the case of zero-shot prediction, to improve its performance-without any additional training. Our technique is a drop-in replacement of the standard prediction rule, swapping arg max with the Fréchet mean. We provide a comprehensive theoretical analysis for this approach, studying (i) learning-theoretic results trading off label space diameter, sample complexity, and model dimension, (ii) characterizations of the full range of scenarios in which it is possible to predict any unobserved class, and (iii) an optimal active learning-like next class selection procedure to obtain optimal training classes for when it is not possible to predict the entire range of unobserved classes. Empirically, using easily-available external metrics, our proposed approach, LOKI, gains up to 29.7% relative improvement over SimCLR on ImageNet and scales to hundreds of thousands of classes. When no such metric is available, LOKI can use self-derived metrics from class embeddings and obtains a 10.5% improvement on pretrained zero-shot models such as CLIP.

1 Introduction

The use of pretrained models is perhaps the most significant recent shift in machine learning. Such models can be used out-of-the-box, either to predict classes observed during pretraining (as in ImageNet-trained ResNets [14]) or to perform zero-shot classification on any set of classes [32]. While this is exciting, label spaces are often so huge that models are unlikely to have seen even a single point with a particular label. Without additional modification, pretrained models will simply fail to reliably predict such classes. Instead, users turn to fine-tuning-which requires additional labeled data and training cycles and so sacrifices much of the promise of zero-shot usage.

How can we adapt pretrained models to predict new classes, without fine-tuning or retraining? At first glance, this is challenging: predictive signal comes from labeled training data. However, relational information between the classes can be exploited to enable predicting a class even when there are no examples with this label in the training set. Such relational data is commonly available, or can be constructed with the help of knowledge bases, ontologies, or powerful foundation models [39].

How to best exploit relational structure remains unclear, with a number of key challenges: We might wish to know what particular subset of classes is rich enough to enable predicting many (or all) remaining labels. This is crucial in determining whether a training set is usable or, even with the aid of structure, insufficient. It is also unclear how approaches that use relational information interact with the statistical properties of learning, such as training sample complexity. Finally, performing adaptation requires an efficient and scalable algorithm.

39

39

39

39

80

80

80

80

60

60

60

60

40

40

40

40

20

20

20

20

99

argmax

0

99

Tree Graph

0

99

Grid Graph

D

0

99

Small-world Graph

0

This work addresses these challenges. It proposes a simple and practical approach to learning in structured label spaces, with theoretical guarantees. First, we offer a simple way to translate the soft outputs (i.e., probability vectors) produced by any supervised learning model into a more general model that can exploit geometric information for label structure. In other words, our approach, called LOKI, 1 is a simple adaptor for pretrained models. LOKI can be applied via a fixed linear transformation of the model outputs. LOKI's simplicity makes it applicable to a broad range of settings while enabling very high-cardinality predictions subject to a potentially small model output budget-we provide a visualization of this key idea in Figure 1.

Theoretically, we provide a rich set of results for the metric-based adaptation setting. First, we introduce a learning-theoretic result in terms of the sample complexity of the pretrained model. It captures a key tradeoff between the number of classes and metric structure, the problem dimensionality, and the number of samples used to train the model prior to adaptation. Next we exhaustively study the properties of training sets, determining for a wide class of relational structures the minimum number (and structure) of subsets that enable reliable prediction. Finally we show how to exploit this result in an active learning-like approach to selecting points that will improve deficient training datasets.

Experimentally, we show that using structural information improves prediction in high-cardinality settings. We demonstrate the strength of the active learning-based approach to dataset expansion over random baselines. Finally, and most excitingly, we show that even in zero-shot models like CLIP, where it is possible to produce probability vectors over any possible class, the use of our adaptor leads to a 19.53% relative improvement.

2 Background

First, we introduce the problem setting, notation, and mathematical tools that we will use. Afterward, we discuss how LOKI relates to prior work.

Problem Setting As in conventional supervised learning, we have a dataset ( x 1 , y 1 ) , . . . , ( x n , y n ) drawn from a distribution on X × Y , where X and Y are the input and label spaces. In our setting, N := |Y| is finite but large; often N ≫ n -so that many labels will simply not be found in the training dataset. We let Λ ⊆ Y with | Λ | = K be the set of observed labels. For convenience of notation, we also define λ := [ λ i ] K i =1 , y := [ y i ] N j =1 to be the vectors of elements in Λ and Y .

In addition to our dataset, we have access to a relational structure on Y . We assume that Y is a metric space with metric (distance) d : Y 2 → R ; d encodes the relational structure of the label space. Specifically, we model this metric space using a graph, G = ( Y , E ) where E ⊆ Y 2 × R + is a set of edges relating the labels and the standard shortest-path distance d : Y 2 → R ≥ 0 . In addition to its use in prediction, the metric d can be used for evaluating a model by measuring, for example, 1 n ∑ n i =1 d 2 ( f ( x i ) , y ) -the analogue of the empirical square loss.

Fréchet Mean Estimator Drawing on ideas from structured prediction [5, 35], we use a simple predictor that exploits the label metric. It relies on computing the Fréchet mean [8], given by

where w ∈ R K ≥ 0 is a set of weights. The Fréchet mean generalizes the barycenter to metric spaces and is often used in geometric machine learning [26].

Locus of the Fréchet mean. The locus of the Fréchet mean is the set of all Fréchet means under different weights [29]. We write it as Π( y ) := ∪ w ∈ ∆ K -1 m y ( w ) .

Π( y ) can be thought of the set of all labels in Y that are reachable by the Fréchet mean given { y i } K i =1 ⊆ Y and different choices of its parameter w . Intuitively, we can think of the locus for a given dataset as describing how usable it is for predicting beyond the observed labels. Trivially, if { y i } K i =1 = Y , then Π( y ) = Y . We are primarily interested in the case in which { y i } K i =1 ⊂ Y yet we still have Π( y ) = Y , or that | Π( y ) | is at least sufficiently large.

2.1 Relation to Prior work

LOKI is primarily related to two areas: zero-shot learning and structured prediction.

Zero-Shot Learning Like zero-shot learning (ZSL), LOKI is capable of predicting unobserved classes. Our framework is most closely related to generalized ZSL, which uses side information to predict both observed and unobserved classes. Many types of external knowledge are used in ZSL, including text [28, 38, 9], attributes [22, 23, 7], knowledge graphs [42, 34, 10], ontologies [25], and logical rules [33]. Our work is most closely related to ZSL approaches that rely on knowledge graph information. Often, ZSL methods that use knowledge graph information rely on the use of graph neural network architectures [42, 17, 18]. However, we note that these architectures can be heavyweight and can be challenging to scale up to extremely large graphs, whereas LOKI does not have architectural requirements and scales linearly in the size of the graph when K is small.

Structured Prediction Structured prediction (SP) operates in label spaces that are endowed with algebraic or geometric structure [1, 21]. This includes problems such as predicting permutations [19], non-Euclidean and manifold regression [31, 36], and learning on graphs [12]. LOKI is the most immediately related to the latter, however, any finite metric space can be represented as a graph, which further lends to the flexibility of our approach. Even in discrete structured prediction settings, the cardinality of the label space may be combinatorially large. As such, LOKI can be viewed as a simple method for adapting classifiers to structured prediction.

The Fréchet mean has been used in structured prediction-but in approaches requiring training. In [5], ˆ f ( x ) = arg min y ∈Y 1 n ∑ n i =1 α i ( x ) d 2 ( y, y i ) , where α ( x ) = ( K + nνI ) -1 K x . K is the kernel matrix for a kernel k : X ×X → R , so that K i,j = k ( x i , x j ) , ( K x ) i = k ( x, x i ) . ν is a regularization parameter. In other words, the weight w i corresponding to y i is the average produced by solving a kernel regression problem at all points x k in the dataset where y k = y i . It has also used in weak supervision (WS) for metric-equipped label spaces [41, 37], where the goal is to produce labeled data for training structured prediction models.

3 Framework

We introduce our framework-LOKI. We show how LOKI can be used to adapt any supervised classifier over a set of K classes to a much richer set of possible class predictions. It does so by weighting the Fréchet mean by the classifier's per-class prediction probabilities or logits, allowing it to predict any class in the locus of the Fréchet mean-potentially far more classes than the initial K . Next, we show how LOKI can be expressed as a fixed linear transformation of a model's outputs. Finally, we show that LOKI relates to standard classification.

3.1 LOKI: Adapting Pretrained Models

We describe our approach to adapting pretrained classifiers-trained on a set of classes Λ -to the metric geometry of the label space Y , enabling the prediction of unobserved classes.

We model unobserved classes Y \ Λ using the Fréchet mean among observed classes weighted by their prediction probabilities P ( y = λ i | x and y ∈ Λ) . We denote the vector of model outputs as

P y | x := [ P λ i | x ] K i =1 = [ P ( y = λ i | x and y ∈ Λ)] K i =1 . Then predictions using LOKI are given by

3.2 LOKI as a linear transformation of model outputs

Most standard classifiers output a vector of prediction probabilities, P y | x , whose entries correspond to the confidence of predicting a specific class. Predictions are typically given by ˆ y ∈ arg max i ∈ [ K ] ( P y | x ) i . LOKI generalizes this prediction rule when viewed as a linear transformation of P y | x . Consider the LOKI prediction rule ˆ y ∈ m λ ( P y | x ) = arg min y ∈Y ∑ K i =1 P λ i | x d 2 ( y, λ i ) = arg max j ∈ [ N ] ( DP y | x ) j , where D j,i = [ -d 2 ( y j , λ i ) ] ; D ∈ R N × K is the matrix of negative squared distances between the observed classes and the rest of the label space. Thus LOKI can be used within standard classification pipelines when the model output P y | x is multiplied by the fixed matrix D .

3.3 Generalizing Standard Classification

We provide a simple intuition for our approach. The fact that LOKI reduces to standard classification among the observed classes has several implications. This includes the idea that under our framework, forms of few-shot, zero-shot, hierarchical, and partial label learning all reduce to standard classification when additional metric information is introduced.

Generalizing the arg max prediction rule In the absence of this metric information-a situation that we model using the complete graph and setting Λ = Y -our framework also recovers standard classification. Indeed, both in terms of error modeling and in terms of inter-class similarity, the intuition of standard classification and of the 0-1 loss are captured well by the unweighted complete graph-simply treat all differences equally. This graph is given as K N := ( Y , Y 2 ×{ 1 } ) -i.e., every label is equidistant from every other label. Plugging this into LOKI, we obtain the following:

̸

which is exactly the standard classification prediction rule.

̸

Generalizing the 0-1 loss via the expected squared distance The expected squared distance E [ d 2 ( y, ˆ y )] is the standard loss function for many structured prediction problems . Note that accuracy fails in such settings -since it cannot distinguish between small and large errors. This is most clearly seen in the extreme case of regression, where test accuracy will virtually always be zero no matter how good a trained model is. At the other extreme, the complete graph, this loss function becomes the standard 0-1 loss: E [ d 2 ( y, ˆ y )] = E [ 1 { y = ˆ y } ] . As an adaptor for structured label spaces, we use the empirical version of this loss to evaluate LOKI. Note that the expected squared distance subsumes other metrics as well. For example, when Y = R , we can derive the standard MSE by setting d ( y, ˆ y ) = | y -ˆ y | , which is just the standard L1 distance metric. Other scores such as recall at top-k can be similarly obtained at the cost of d ( · , · ) being a true metric. In other words, E [ d 2 ( y, ˆ y )] is an very general metric that supports any metric space, and we use it throughout this work.

4 Theoretical Results

Challenges and Opportunities The arg max of per-class model probabilities is a ubiquitous component of classification pipelines in machine learning. In order to predict unobserved classes using metric space information, LOKI replaces this standard component. As a simple but significant change to standard pipelines, LOKI opens up a new area for fundamental questions. There are three main flavors of theoretical questions that arise in this setting:

How does the performance of LOKI change as a function of the number of samples?

What minimal sets of observed classes are required to predict any class in the metric space?

How can we acquire new classes that maximize the total number of classes we can predict?

Excitingly, we provide a comprehensive answer to these questions for the most common metric spaces used in practice. First, we provide a general error bound in terms of the number of samples, observed classes, problem dimension, and the diameter of the metric space, that holds for any finite metric space. Second, we characterize the sets of observed classes that are required to enable prediction of any class, and show how this set differs for various types of metric spaces of interest. Finally, we provide an active learning algorithm for selecting additional classes to observe so as to maximize the number of classes that can be predicted and we characterize the types of metric spaces for which the locus can be computed efficiently. These results provide a strong theoretical grounding for LOKI.

4.1 Sample Complexity

What is the interplay between predicting unobserved classes based on metric space information and standard learning-theoretic notions like the sample complexity needed to train a model? Our first result illustrates this tradeoff, and relates it to the squared distance loss. Suppose that we only observe K ≤ N of the classes at training time, and that we fit a K -class Gaussian mixture model. We use LOKI to adapt our pretrained classifier to enable classification of any of the N classes. We have that (a formal statement and interpretation are in the Appendix),

Theorem 4.1 ( Informal LOKI sample complexity) . Let Y be a set of classes represented by d dimensional vectors under the Euclidean distance, and let Λ ⊆ Y be the set of K observed classes. Assume that n training examples are generated by an identity covariance Gaussian mixture model over classes Λ , and that test examples are generated over all classes Y . Assume that we estimate a Gaussian mixture model on the training set and obtain probability estimates ˆ P ( y i | x ) for i ∈ [ K ] for a sample ( x, y ∗ ) from the test distribution. Then with high probability, under the following model,

the sample complexity of estimating target y ∗ from the test distribution D test with prediction ˆ y ∗ is:

where α is related to the sensitivity of the Fréchet variance to different node choices.

4.2 Optimal Label Subspaces

Next we characterize the subset of distinct labels required to predict any label using our label model with respect to various types of metric spaces.

We first consider label spaces whose metric is a tree graph-such metrics are, for example, related to performing hierarchical classification and weak supervision (where only partial labels are available). Weconsider a special type of tree called a phylogenetic tree, in which only the leaves can be designated as labels-phylogenetic trees are commonly used to relate the labels of image classification datasets. Afterwards we perform a similar analysis for grid graphs, which are important for label spaces that encode spatial information. Finally, we discuss the case in which no useful metric information is available, i.e., the complete graph.

Our goal in this section is to characterize the properties and size of { λ i } K i =1 in each of these metric spaces such that we still have Π( Λ ) = Y . We characterize 'optimal' subsets of classes in each of the spaces under a certain notions of optimality. We provide several relevant definitions pertaining to this concept, starting with a notion of being able to predict any possible class using observed classes.

Definition 4.2 (Locus cover) . Given a set Λ ⊆ Y for which we construct a tuple of its elements Λ , if it holds that Π( Λ ) = Y , then Λ is a locus cover.

Definition 4.2 captures the main idea of LOKI-using some set of observed classes for which we can train classifiers, we would like to be able to predict additional unobserved classes using the geometry that relates the observed and unobserved classes. Namely, elements of Π( Λ ) are 'reachable' using LOKI. We refine this Definition to describe the trivial case that defaults to standard classification and the nontrivial case for which LOKI moves beyond standard classification.

Definition 4.3 (Trivial locus cover) . If Λ = Y , then Λ is the trivial locus cover.

This Definition captures the notion of observing all of the classes in the label space. Here, all of the elements of Y are trivially reachable using LOKI.

̸

Definition 4.4 (Nontrivial locus cover) . A locus cover Λ is nontrivial if Λ = Y .

LOKI is more useful and interesting when faced with a nontrivial locus cover-under Definition 4.4, we can use some subset of classes Λ to predict any label in Y .

Definition 4.5 (Minimum locus cover) . Given a set Λ ⊆ Y , if Λ is the smallest set that is still a locus cover, then it is a minimum locus cover.

In cases involving an extremely large number of classes, it is desirable to use LOKI on the smallest possible set of observed classes Λ such that all labels in Y can still be predicted. Definition 4.5 characterizes these situations-later, we obtain the minimum locus covers for all trees and grid graphs. It is worth noting that the minimum locus cover need not be unique for a fixed graph.

Definition 4.6 (Identifying locus cover) . Given a set Λ ⊆ Y , if Λ is a locus cover where ∀ y ∈ Y , ∃ w ∈ ∆ | Λ |-1 such that m Λ ( w ) = { y } , then Λ is an identifying locus cover.

The Fréchet mean need not be unique-as an arg min , it returns a set of minimizers. In certain metric spaces, the minimum locus cover can yield large sets of minimizers-this is undesirable, as it makes predicting a single class challenging. Definition 4.6 appeals to the idea of finding some set of classes for which the Fréchet mean always returns a unique minimizer-this is desirable in practice, and in some cases, moreso than Definition 4.5.

Definition 4.7 (Pairwise decomposable) . Given Λ ⊆ Y , Π(Λ) is called pairwise decomposable when it holds that Π(Λ) = ∪ λ 1 ,λ 2 ∈ Λ Π( { λ 1 , λ 2 } ) .

In many cases, the locus can be written in a more convenient form-the union of the locus of pairs of nodes. We refer to this definition as pairwise decomposability. Later, we shall see that pairwise decomposability is useful in computing the locus in polynomial time.

Trees Many label spaces are endowed with a tree metric in practice: hierarchical classification, in which the label space includes both classes and superclasses, partial labeling problems in which internal nodes can represent the prediction of a set of classes, and the approximation of complex or intractable metrics using a minimum spanning tree. We show that for our purposes, trees have certain desirable properties that make them easy to use with LOKI-namely that we can easily identify a locus cover that satisfies both Definition 4.5 and Definition 4.6. Conveniently, we also show that any locus in any tree satisfies Definition 4.7.

We first note that the leaves of any tree yield the minimum locus cover. This is a convenient propertyany label from any label space endowed with a tree metric can be predicted using LOKI using only classifiers trained using labels corresponding to the leaves of the metric space. This can be especially useful if the tree has long branches and few leaves. Additionally, for tree metric spaces, the minimum locus cover (Definition 4.5) is also an identifying locus cover (Definition 4.6). This follows from the construction of the weights in the proof of Theorem A.4 (shown in the Appendix) and the property that all paths in trees are unique. Finally, we note that any locus in any tree is pairwise decomposable-the proof of this is given in the Appendix (Lemma A.5). We will see later that this property yields an efficient algorithm for computing the locus.

Phylogenetic Trees Image classification datasets often have a hierarchical tree structure, where only the leaves are actual classes, and internal nodes are designated as superclasses-examples include the ImageNet [6] and CIFAR-100 datasets [20]. Tree graphs in which only the leaf nodes are labeled are referred to as phylogenetic trees [3]. Often, these graphs are weighted, but unless otherwise mentioned, we assume that the graph is unweighted.

For any arbitrary tree T = ( V , E ) , the set of labels induced by phylogenetic tree graph is Y = Leaves ( T ) . We provide a heuristic algorithm for obtaining locus covers for arbitrary phylogenetic trees in Algorithm B.1 (see Appendix). We prioritize adding endpoints of long paths to Λ , and continue adding nodes in this way until Π(Λ) = Y . Similarly to tree metric spaces, any phylogenetic tree metric space is pairwise decomposable. We prove the correctness of Algorithm B.1 and pairwise

decomposability of phylogenetic trees in the Appendix (Theorem A.6 and Lemma A.7). Later, we give algorithms for computing the set of nodes in an arbitrary locus in arbitrary graphs-if the locus is pairwise decomposable, the algorithm for doing so is efficient, and if not, it has time complexity exponential in K . Due to the pairwise decomposability of phylogenetic trees, this polynomial-time algorithm to compute Π(Λ) applies.

Grid Graphs Classes often have a spatial relationship. For example, classification on maps or the discretization of a manifold both have spatial relationships-grid graphs are well suited to these types of spatial relationships. We obtain minimum locus covers for grid graphs satisfying Definition 4.5, but we find that these are not generally identifying locus covers. On the other hand, we give an example of a simple identifying locus cover satisfying Definition 4.6. Again, we find that grid graphs are in general pairwise decomposable and hence follow Definition 4.7.

We find that the pair of vertices on furthest opposite corners yields the minimum locus cover. While the set of vertices given by Theorem A.8 (found in the Appendix) satisfies Definition 4.5, this set does not in general satisfy Definition 4.6. This is because the path between any two vertices is not unique, so each minimum path of the same length between the pair of vertices can have an equivalent minimizer. On contrast, the following example set of vertices satisfies Definition 4.6 but it clearly does not satisfy Definition 4.5. Example : Given a grid graph, the set of all corners is an identifying locus cover. On the other hand, the vertices given by Theorem A.8 can be useful for other purposes. Lemma A.9 (provided in the Appendix) shows that subspaces of grid graphs can be formed by the loci of pairs of vertices in Λ . This in turn helps to show that loci in grid graphs are pairwise decomposable in general (see Lemma A.10 in the Appendix).

The Complete Graph The standard classification setting does not use relational information between classes. As before, we model this setting using the complete graph, and we show the expected result that in the absence of useful relational information, LOKI cannot help, and the problem once again becomes standard multiclass classification among observed classes. To do so, we show that there is no nontrivial locus cover for the complete graph (Theorem A.11 in the Appendix).

4.3 Label Subspaces in Practice

While it is desirable for the set of observed classes to form a minimum or identifying locus cover, it is often not possible to choose the initial set of observed classes a priori-these are often random. In this section, we describe the more realistic cases in which a random set of classes are observed and an active learning-based strategy to choose the next observed class. The aim of our active learning approach is, instead of randomly selecting the next observed class, to actively select the next class so as to maximize the total size of the locus-i.e., the number of possible classes that can be output using LOKI. Before maximizing the locus via active learning, we must first address a much more basic question: can we even efficiently compute the locus?

Computing the Locus We provide algorithms for obtaining the set of all classes in the locus, given a set of classes Λ . We show that when the locus is pairwise decomposable (Definition 4.7), we can compute the locus efficiently using a polynomial time algorithm. When the locus is not pairwise decomposable, we provide a general algorithm that has time complexity exponential in | Λ | -we are not aware of a more efficient algorithm. We note that any locus for every type of graph that we consider in Section 4.2 is pairwise decomposable, so our polynomial time algorithm applies. Algorithms B.2 and B.3 along with their time complexity analyses can be found in the Appendix.

Large Locus via Active Next-Class Selection We now turn to actively selecting the next class to observe in order to maximize the size of the locus. For this analysis, we focus on the active learning setting when the class structure is a tree graph, as tree graphs are generic enough to apply to a wide variety of cases-including approximating other graphs using the minimum spanning tree. Assume the initial set of K observed classes are sampled at random from some distribution. We would like to actively select the K +1 st class such that | Π(Λ) | with Λ = { λ } K +1 i =1 is as large as possible.

Theorem 4.8. Let T = ( Y , E ) be a tree graph and let Λ ⊆ Y with K = | Λ | . Let T ′ be the subgraph of the locus Π(Λ) . The vertex v ∈ Y \ Λ that maximizes | Π(Λ ∪{ v } ) | is the solution to the following optimization problem: arg max y ∈Y\ Π(Λ) d ( y, b ) s.t. b ∈ ∂ in T ′ and Γ( y, b ) \ { b } ⊆ Y \ Π(Λ) . where ∂ in T ′ is the inner boundary of T ′ (all vertices in T ′ that share an edge with vertices not in T ′ ).

This procedure can be computed in polynomial time-solving the optimization problem in Theorem 4.8 simply requires searching over pairs of vertices. Hence we have provided an efficient active learning-based strategy to maximize the size of the locus for trees.

5 Experimental Results

In this section, we provide experimental results to validate the following claims:

LOKI improves performance of zero-shot foundation models even with no external metric.

LOKI adapts to label spaces with a large number of unobserved classes.

The active approach given in Theorem 4.8 yields a larger locus than the passive baseline.

The same active approach yields better performance on ImageNet.

With LOKI, calibration can improve accuracy , even with no external metric.

5.1 LOKI Improves Zero-Shot Models

We evaluate the capability of LOKI to improve upon zero-shot models where all classes are observed.

Setup Our experiment compares the zero-shot prediction performance of CLIP [32] on CIFAR-100 [20] to CLIP logits used with LOKI. First, we consider the setting in which no external metric relating the labels is available, and instead derive internal metric information from Euclidean distances between text embeddings from the models using their respective text encoders. Second, we consider three external metric spaces for use with LOKI: the complete graph K 100 and two phylogenetic trees: the default CIFAR-100 superclasses [20] and WordNet [2].

Results The results of this experiment are given in Table 1. When no external metric information is available, LOKI still outperforms CLIP-like models that use the standard prediction rule-in other words, LOKI seems to unconditionally improve CLIP. As expected, under the complete graph, our method becomes equivalent to the standard prediction mechanism used by CLIP. On the other hand, LOKI outperforms CLIP when using the default CIFAR-100 tree hierarchy and even more so when using the WordNet geometry, with a 19 . 53 % relative improvement in mean squared distance over the CLIP baseline. We postulate that the strong performance using WordNet is due to the richer geometric structure compared to that of the default CIFAR-100 hierarchy.

5.2 LOKI on Partially-Observed Label Spaces

To validate our approach on partially observed label spaces, we evaluate the performance of adapting a logistic classifier trained on SimCLR embeddings of ImageNet [6], 5-NN models trained on a 9,419 class subset of the PubMed dataset, 2 and the 325,056-class LSHTC dataset [30].

K

50

100

150

[

d

2

(

y

,

y

)]

Class sampling under uniform distribution

SimCLR

SimCLR + Loki

0

200

400

600

800

1000

K

50

100

150

[

d

2

(

y

,

y

)]

Class sampling under Gibbs distribution

SimCLR

SimCLR + Loki

10 0

10 1

10 2

# of selections

20

40

60

80

100

Locus size

Active next class selection

Active learning

Random

500

510

520

530

540

550

34

36

[

d

2

(

y

,

y

)]

Active next-class selection on ImageNet using SimCLR + Loki

SimCLR + Loki (passive)

SimCLR + Loki (active)

K

Setup For ImageNet, we use the WordNet phylogenetic tree as the metric space [2]. In this setting, we sample random subsets of size K of the 1000 ImageNet classes and compare a baseline one-vs-rest classifier to the same classifier but using LOKI to adapt predictions to classes beyond the original K . We conduct two sets of experiments. In the first, we sample K classes uniformly, while in the second, we adopt a more realistic sampler-we sample from a Gibbs distribution: P ( λ | θ ) = 1 Z exp( -θd ( λ, λ c )) , where λ c = m Y ( 1 N ) is the centroid of the metric space, θ is the concentration parameter around the centroid, and Z is the normalizer. While the Gibbs distribution sampler is more realistic, it is also the more challenging setting-classes which have low probability according to this distribution are less likely to appear in the locus. For PubMed, we derive our metric from Euclidean distances between SimCSE class embeddings [11]. Finally for LSHTC, we summarize the default graph by randomly selecting nodes and merging them with their neighbors until we obtain a graph with 10,000 supernodes representing sets of classes.

Results Figure 2 shows the mean squared distances compared to the baseline one-vs-rest classifiers, across various settings of K . We find that LOKI always significantly outperforms the baseline, even in the more challenging setting of sampling

according to a Gibbs distribution. Tables 2 and 3 show our improvements when using LOKI over the baseline 5-NN models. While LOKI consistently yields an improvement on PubMed and LSHTC, the improvement is more dramatic on LSHTC.

5.3 Large Loci via Active Learning

We evaluate our active next class selection approach on increasing the locus size. We expect that compared to passive (random) selection, our active approach will lead to larger loci, and in practice, a larger set of possible classes that can be predicted using LOKI while observing fewer classes during training.

Setup We compare with a baseline that randomly selects the next class. We first generate a synthetic random tree with size N and fix an initial K . In active selection, we use the approach described in Theorem 4.8 to select the next node. As a passive baseline, we randomly sample (without replacement) the remaining nodes that are not in Λ .

0.2

0.4

0.6

0.8

1.0

Confidence

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

Reliability

ECE optimal

Loki optimal

CLIP default

Perfect calibration

0.26

0.28

0.30

0.32

0.34

0.36

[

d

2

(

y

,

y

)]

0.1

0.2

0.3

0.4

0.5

ECE

ECE vs. Loki perf.

ECE optimal

CLIP default

Loki optimal

0.005

0.010

0.015

0.020

0.025

Softmax temperature

0.26

0.27

0.28

0.29

Expected squared distance

0.60

0.62

0.64

0.66

0.68

0.70

0.72

0.74

Zero-one error

CIFAR-100 CLIP internal metric

error tradeoff

Argmax prediction

CLIP default

0.002

0.004

0.006

0.008

0.010

0.012

0.014

Softmax temperature

Results The results are shown in Figure 2. We set N = 100 and the initial K = 3 , then we average the result over 10 independent trials. The active approach consistently outperforms the random baseline as it attains a larger locus with fewer nodes selected.

5.4 Improving Performance via Active Learning

Next, we evaluate the our active next class selection approach on improving error on ImageNet. While we found that the the active approach indeed leads to an increased locus size compared to the passive baseline, we expect that this increased locus size will lead to improved performance.

Setup We randomly sample 500 ImageNet classes, and using the WordNet metric space, we use our active approach to iteratively sample 50 more classes. We compare this to a passive baseline in which the 50 classes are sampled randomly. We repeat this experiment over 10 independent trials.

Results Figure 2 shows that our active approach yields improved error over the passive baseline. The gap between the active approach and the passive baseline widens with more selection rounds.

5.5 Improving Accuracy via Calibration

Finally, we evaluate the effect of calibrating the Softmax outputs on the performance of LOKI.

Setup We calibrate via Softmax temperature scaling [13] using CLIP on CIFAR-100. We do not use an external metric space, and instead use Euclidean distance applied to the CLIP text encoder.

Results The reliability diagram in Figure 3 shows that the optimal Softmax temperature for LOKI is both close to the default temperature used by CLIP and to the optimally-calibrated temperature. In Figure 3 (right), we find that appropriate tuning of the temperature parameter can lead to improved accuracy with CLIP , even when no external metric space is available.

6 Conclusion

In this work, we proposed LOKI-a simple adaptor for pretrained models to enable the prediction of additional classes that are unobserved during training by using metric space information. We comprehensively answered the space of new questions that arise under LOKI in terms of learning, optimal metric space settings, and a practical active selection strategy. Experimentally, we showed that LOKI can be used to improve CLIP even without external metric space information, can be used to predict a large number of unobserved classes, and a validation of our active selection strategy.

References

Gökhan Bakir, Thomas Hofmann, Bernhard Schölkopf, Alexander J. Smola, Ben Taskar, and S.V.N Vishwanathan. Predicting Structured Data . The MIT Press, July 2007.

Björn Barz and Joachim Denzler. Hierarchy-based image embeddings for semantic image retrieval. In 2019 IEEE Winter Conference on Applications of Computer Vision (WACV) , pages 638-647. IEEE, 2019.

Louis J. Billera, Susan P. Holmes, and Karen Vogtmann. Geometry of the space of phylogenetic trees. Advances in Applied Mathematics , 27(4):733-767, 2001.

Yeshwanth Cherapanamjeri, Nicolas Flammarion, and Peter L. Bartlett. Fast mean estimation with sub-gaussian rates. In Alina Beygelzimer and Daniel Hsu, editors, Proceedings of the Thirty-Second Conference on Learning Theory , volume 99 of Proceedings of Machine Learning Research , pages 786-806. PMLR, 25-28 Jun 2019.

Carlo Ciliberto, Lorenzo Rosasco, and Alessandro Rudi. A consistent regularization approach for structured prediction. In Advances in Neural Information Processing Systems 30 (NIPS 2016) , volume 30, 2016.

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A largescale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition , pages 248-255, 2009.

Ali Farhadi, Ian Endres, Derek Hoiem, and David Forsyth. Describing objects by their attributes. In 2009 IEEE Conference on Computer Vision and Pattern Recognition , pages 1778-1785, 2009.

Maurice R. Fréchet. Les éléments aléatoires de nature quelconque dans un espace distancié. Annales de l'institut Henri Poincaré , 1948.

Andrea Frome, Greg S Corrado, Jon Shlens, Samy Bengio, Jeff Dean, Marc' Aurelio Ranzato, and Tomas Mikolov. Devise: A deep visual-semantic embedding model. In C.J. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems , volume 26. Curran Associates, Inc., 2013.

Junyu Gao, Tianzhu Zhang, and Changsheng Xu. I Know the Relationships: Zero-Shot Action Recognition via Two-Stream Graph Convolutional Networks and Knowledge Graphs. Proceedings of the AAAI Conference on Artificial Intelligence , 33(01):8303-8311, July 2019.

Tianyu Gao, Xingcheng Yao, and Danqi Chen. SimCSE: Simple contrastive learning of sentence embeddings. In Empirical Methods in Natural Language Processing (EMNLP) , 2021.

Colin Graber and Alexander Schwing. Graph structured prediction energy networks. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc., 2019.

Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. In Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference on Machine Learning , volume 70 of Proceedings of Machine Learning Research , pages 1321-1330. PMLR, 06-11 Aug 2017.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 770-778, 2016.

Daniel Hsu and Sivan Sabato. Loss minimization and parameter estimation with heavy tails. Journal of Machine Learning Research , 17(18):1-40, 2016.

Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning , volume 139 of Proceedings of Machine Learning Research , pages 4904-4916. PMLR, 18-24 Jul 2021.

Michael C. Kampffmeyer, Yinbo Chen, Xiaodan Liang, Hao Wang, Yujia Zhang, and Eric P. Xing. Rethinking knowledge graph propagation for zero-shot learning. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 11479-11488, 2018.

Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In International Conference on Learning Representations , 2017.

Anna Korba, Alexandre Garcia, and Florence d'Alché-Buc. A structured prediction approach for label ranking. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 31. Curran Associates, Inc., 2018.

Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.

Volodymyr Kuleshov and Percy S Liang. Calibrated structured prediction. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 28. Curran Associates, Inc., 2015.

Christoph H. Lampert, Hannes Nickisch, and Stefan Harmeling. Learning to detect unseen object classes by between-class attribute transfer. In 2009 IEEE Conference on Computer Vision and Pattern Recognition , pages 951-958, 2009.

Christoph H. Lampert, Hannes Nickisch, and Stefan Harmeling. Attribute-based classification for zero-shot visual object categorization. IEEE Transactions on Pattern Analysis and Machine Intelligence , 36(3):453-465, 2014.

M. Lerasle and R. I. Oliveira. Robust empirical mean estimators, 2011.

Juan Li, Ruoxu Wang, Ningyu Zhang, Wen Zhang, Fan Yang, and Huajun Chen. Logic-guided semantic representation learning for zero-shot relation classification. In Proceedings of the 28th International Conference on Computational Linguistics , pages 2967-2978, Barcelona, Spain (Online), December 2020. International Committee on Computational Linguistics.

Aaron Lou, Isay Katsman, Qingxuan Jiang, Serge Belongie, Ser-Nam Lim, and Christopher De Sa. Differentiating through the fréchet mean. In Proceedings of the 37th International Conference on Machine Learning , pages 6393-6403, 2020.

Stanislav Minsker. Geometric median and robust estimation in Banach spaces. Bernoulli , 21(4):2308 - 2335, 2015.

Mohammad Norouzi, Tomas Mikolov, Samy Bengio, Yoram Singer, Jonathon Shlens, Andrea Frome, Greg Corrado, and Jeffrey Dean. Zero-shot learning by convex combination of semantic embeddings. In Yoshua Bengio and Yann LeCun, editors, ICLR , 2014.

Tom M W Nye, Xiaoxian Tang, Grady Weyenberg, and Ruriko Yoshida. Principal component analysis and the locus of the fréchet mean in the space of phylogenetic trees. Biometrika , 104(4):901-922, Dec 2017.

Ioannis Partalas, Aris Kosmopoulos, Nicolas Baskiotis, Thierry Artières, Georgios Paliouras, Éric Gaussier, Ion Androutsopoulos, Massih-Reza Amini, and Patrick Gallinari. Lshtc: A benchmark for large-scale text classification. ArXiv , abs/1503.08581, 2015.

Alexander Petersen and Hans-Georg Muller. Fréchet regression for random objects with euclidean predictors. The Annals of Statistics , 2016.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning , pages 8748-8763. PMLR, 2021.

Tim Rocktäschel, Sameer Singh, and Sebastian Riedel. Injecting logical background knowledge into embeddings for relation extraction. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pages 1119-1129, Denver, Colorado, May-June 2015. Association for Computational Linguistics.

Abhinaba Roy, Deepanway Ghosal, Erik Cambria, Navonil Majumder, Rada Mihalcea, and Soujanya Poria. Improving Zero-Shot Learning Baselines with Commonsense Knowledge. Cognitive Computation , 14(6):2212-2222, November 2022.

Alessandro Rudi, Carlo Ciliberto, GianMaria Marconi, and Lorenzo Rosasco. Manifold structured prediction. In Advances in Neural Information Processing Systems 32 (NeurIPS 2018) , volume 32, 2018.

Alessandro Rudi, Carlo Ciliberto, GianMaria Marconi, and Lorenzo Rosasco. Manifold structured prediction. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 31. Curran Associates, Inc., 2018.

Changho Shin, Winfred Li, Harit Vishwakarma, Nicholas Carl Roberts, and Frederic Sala. Universalizing weak supervision. In International Conference on Learning Representations , 2022.

Richard Socher, Milind Ganjoo, Christopher D Manning, and Andrew Ng. Zero-shot learning through cross-modal transfer. In C.J. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems , volume 26. Curran Associates, Inc., 2013.

Russell Stewart and Stefano Ermon. Label-free supervision of neural networks with physics and domain knowledge. In Thirty-First AAAI Conference on Artificial Intelligence , 2017.

Roman Vershynin. Introduction to the non-asymptotic analysis of random matrices. In Compressed Sensing , 2010.

Harit Vishwakarma and Frederic Sala. Lifting weak supervision to structured prediction. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems , 2022.

X. Wang, Y. Ye, and A. Gupta. Zero-shot recognition via semantic embeddings and knowledge graphs. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 6857-6866, Los Alamitos, CA, USA, jun 2018. IEEE Computer Society.

The appendix is organized as follows: in Appendix A, we provide proofs of the claims made in the main text. Next, in Appendix B, we provide algorithms and time complexity analyses for the algorithms referenced in the main text. Appendix C contains additional experimental results. In Appendix D, we provide additional details about our experimental setup. Then in Appendix E, we discuss the broader impacts and limitations of LOKI. Finally, in Appendix F, we provide more details and examples of the locus visualizations shown in Figure 1.

A Deferred Proofs

A.1 Proof of Theorem 4.1 (LOKI sample complexity)

In this section, we provide a formal statement and proof of our sample complexity result for LOKI, including additional required definitions and assumptions.

First, we define the required tools to prove the sample complexity bound for LOKI. For the purposes of this proof, we define the Fréchet variance as the function over which the Fréchet mean returns the minimizer.

Definition A.1 (Fréchet variance) . The Fréchet variance is defined as

Additionally, we will require a technical assumption related to the sensitivity of the Fréchet variance to different node choices.

Assumption A.2 (Fréchet variance is 1 α -bi-Lipschitz) . For a metric space defined by a graph G = ( V , E ) , the Fréchet variance is K -bi-Lipschitz if there exists a K ≥ 1 such that

for all V, ˜ V ∈ V , and a fixed w ∈ ∆ K -1 . For our purposes, such a K always exists: consider setting K = diam ( G ) 2 max V 1 ,V 2 ∈V | Ψ w ( V 1 ) -Ψ w ( V 2 ) | . However, this is a very conservative bound that holds for all graphs that we consider. Instead, we assume access to the largest α = 1 K ≤ 1 such that

which may be problem dependent.

Theorem A.3 (LOKI sample complexity) . Let Y be a set of points on the d dimensional 2-norm ball of radius R , and let Λ ⊆ Y be the set of K observed classes. Assume that Λ forms a 2 R/ ( √ dK -1) -net under the Euclidean distance. Assume that training examples are generated by drawing n samples from the following process: draw x ∼ N ( y, I ) where y ∼ Unif (Λ) , and at test time, draw x ∼ N ( y, I ) where y ∼ Unif ( Y ) . Assume that we estimate a Gaussian mixture model with K components (each having identity covariance) on the training set and obtain probability estimates ˆ P ( y i | x ) for i ∈ [ K ] for a sample ( x, y ∗ ) from the test distribution. Then with high probability, under the following model,

the sample complexity of estimating target y ∗ from the test distribution D test with prediction ˆ y ∗ is:

where d is the dimensionality of the input and α is our parameter under Assumption A.2.

Proof. We begin by detailing the data-generating process.

At training time, our underlying data-generating process, D train, is as follows:

We begin with a 2 R √ dK -1 -net, Y , on the d dimensional 2-norm ball of radius R under the Euclidean distance, and let Λ ⊆ Y

Draw y ∼ Unif ( Y ) .

Discard draws if y ̸∈ Λ .

Draw x ∼ N ( y, I ) .

At test time, we do not discard draws and allow for classes not in Λ and use the following data generating process, D test :

We begin with a 2 R √ dK -1 -net, Y , on the d dimensional 2-norm ball of radius R under the Euclidean distance, and let Λ ⊆ Y

Draw y ∼ Unif ( Y ) .

Draw x ∼ N ( y, I ) .

Given a labeled training dataset D = { ( x i , y i ) } n i =1 containing n points drawn from D train with | Λ | = K distinct classes, we would like to fit a K -component Gaussian mixture model with identity covariance. We first perform mean estimation of each of the classes separately using the median-ofmeans estimator [15, 27, 24]. Using the estimator yields the following parameter estimation bound [27, 4]:

with probability 1 -δ .

Next, we consider the relationships between four quantities: Ψ P ( y ∗ ) , Ψ P (ˆ y ∗ ) , Ψ ̂ P ( y ∗ ) , Ψ ̂ P (ˆ y ∗ ) , where P is the vector of probabilities from the true Gaussian mixture, ̂ P is the vector of probabilities from the estimated model, y ∗ ∈ m Λ ( P ) is the target class, and ˆ y ∗ ∈ m Λ ( ̂ P ) is the predicted class. While it is problem-dependent as to whether Ψ P ( y ∗ ) ≤ Ψ ̂ P (ˆ y ∗ ) or Ψ P ( y ∗ ) > Ψ ̂ P (ˆ y ∗ ) , a similar argument holds for both cases. So without loss of generality, we assume that Ψ P ( y ∗ ) ≤ Ψ ̂ P (ˆ y ∗ ) . Then by the definition of the Fréchet mean, the following inequalities hold:

We proceed by obtaining upper and lower bounds of the existing bounds in Equation 2. First, we will obtain an upper bound on Ψ ̂ P ( y ∗ ) -Ψ P ( y ∗ ) .

and consequently,

For notational convenience, we define the following:

Then (4) becomes:

Now we define the following: L := || x -y z || 2 2 with z ∈ arg min j || x -y j || 2 2 is the smallest distance to a class mean, and L + E i := || x -y i || 2 2 with E i > 0 . Similarly, define ˆ L := || x -ˆ y z || 2 2 with z ∈ arg min j || x -ˆ y j || 2 2 is the smallest distance to an estimated class mean, and ˆ L + ˆ E i := || x -ˆ y i || 2 2 with ˆ E i > 0 . Finally, we define T := √ ˆ L + √ ˆ L + ˆ E i + √ L + √ L + E i .

Then we can bound the parts separately:

̸

For i = z , we have

̸

and in the case of i = z , we bound a z b ≤ 1 . So overall, we have a i b ≤ 1 exp { 1 i = z 1 2 E i } for all i ∈ [ K ] .

̸

̸

Plugging (6), (7), (9), and (10) into (5), we obtain

(8)

Next, we must control E i -ˆ E i in order to obtain the bound for ˆ a i ˆ b .

Using (6), we obtain

̸

̸

̸

̸

Next, recalling the fact that our class means form an ε -net on the radiusR ball, we use Lemma 5.2 from [40] to bound L as L ≤ 2 R √ dK -1 .

̸

Next, we will consider cases on E i . We will consider the cases in which E i < log R 2 and E i ≥ log R 2 .

We will begin with the case in which E i < log R 2 , then Equation 11 becomes:

Next, the case in which E i ≥ log R 2 , then Equation 11 becomes:

Setting M ≤ K to be the number of terms for which E i < log R 2 holds and by combining the two bounds, we obtain the following:

Ultimately, our bound is

Next, we will obtain a lower bound on Ψ ̂ P (ˆ y ∗ ) -Ψ P ( y ∗ ) .

Combining both of these bounds with Equation 2, we obtain

Next, we obtain a bound on M . Since M ≤ K is the number of terms for which E i < log R 2 , we have that M = V K , where V ∈ [0 , 1] is the ratio of the volumes of the ball for which E i < log R 2 , and the ball containing the entire set of classes Y . We have

which is an upper bound of the radius of the ball for which E i < log R 2 is true. Now we construct V :

Combining this with our error bound, we have

However, we can choose K to weaken our dependence on R .

Finally, we have

which completes the proof.

Discussion of Theorem A.3. The above bound contains several standard quantities, including the dimensionality of the inputs d , the radius R of the ball from which labels are drawn, the number of classes and samples used to estimate the model K and n respectively, and δ , which is used to control the probability that the bound holds. Naturally, the bound scales up with higher d and K , and improves with an increased number of samples n . The dependence on R is also related to its dependence on d -so long as we have that K ≥ 2( √ d ( R 1 -2 d -log R/R )) -1 +1 and for d ≥ 2 , R increases, which improves the bound. The bound includes a metric space dependent quantity α , which is related to the amount that the Fréchet variance can change subject to changes in its argument, i.e., which node is considered.

A.2 Proof of Theorem A.4 (minimum locus cover for trees)

Theorem A.4 (Minimum locus cover for trees) . The minimum locus cover for a tree graph T is Λ = Leaves ( T ) .

Proof. We would like to show to show two things:

any node can be resolved by an appropriate construction of w using only the leaves and

removing any leaf makes that leaf node unreachable, no matter what the other nodes are in y -i.e., we must use at least all of the leaf nodes.

If both of these conditions hold (i.e., that the leaves form a locus cover and that they are the smallest such set), then the the leaves of any tree form a minimum locus cover. To set up the problem, we begin with some undirected tree, T , whose nodes are our set of labels: T = ( Y , E ) where Λ = Leaves ( T ) ⊆ Y is the set of leaf nodes. Moreover, let Λ be a tuple of the leaf nodes. We will start by proving that Λ is a locus cover. We will show this by cases on v ∈ Y , the node that we would like to resolve:

If v ∈ Λ , i.e. v is a leaf node, then setting w i = 1 { Λ i = v } yields

If v ̸∈ Λ , we have a bit more work to do. Since v is an internal node, consider any pair of leaves, l 1 = l 2 such that v is along the (unique) path between l 1 and l 2 : v ∈ Γ( l 1 , l 2 ) . Then

̸

Thus Λ is a locus cover. Next, we will show that it is the smallest such set. Let l ∈ Λ be any leaf node, and define Λ ′ = Λ \ { l } . We must show that Λ ′ is not a locus cover. Assume for contradiction that Λ ′ is a locus cover. This means that given some tuple y ′ whose entries are the elements of Λ ′ , we have Π( Λ ′ ) = Y . This implies that the path between any two nodes in Λ is also contained in Π( Λ ′ ) . Since the leaves form a locus cover and any y ∈ Y can be constructed by the appropriate choice of w along with a path between two leaf nodes, l must either be one of the entries of Λ ′ (one of the endpoints of the path) or it must be an internal node. Both cases are contradictionsΛ ′ cannot include l by assumption, and l is assumed to be a leaf node. It follows that Λ is a minimum locus cover.

A.3 Proof of Lemma A.5 (tree pairwise decomposability)

Lemma A.5 (Tree pairwise decomposability) . Let T = ( Y , E ) be a tree and Λ ⊆ Y . Then Π(Λ) is pairwise decomposable.

Proof. Assume for contradiction that ∃ y ∗ ∈ Y where y ∗ / ∈ Π( { λ i , λ j } ) , ∀ λ i , λ j ∈ Λ , but y ∗ ∈ Π(Λ) . Then, y ∗ ∈ m Λ ( w ) for some w . Note that y ∗ / ∈ Π( { λ i , λ j } ) implies y ∗ / ∈ Γ( λ i , λ j ) , ∀ λ i , λ j ∈ Λ , because Π( { λ i , λ j } ) = Γ( λ i , λ j ) due to the uniqueness of paths in trees. As such, ∩ λ ∈ Λ Γ( y ∗ , λ ) must contain an immediate relative y ′ of y ∗ where

Therefore, y ∗ / ∈ Π(Λ) because y ∗ is not a minimizer of ∑ | Λ | i =1 w i d 2 ( y, λ i ) . So, it must be that y ∗ ∈ Π( { λ i , λ j } ) for some λ i , λ j ∈ Λ when y ∈ Π(Λ) .

A.4 Proof of Theorem A.6 (Algorithm B.1 correctness)

Theorem A.6 (Algorithm B.1 correctness) . Algorithm B.1 returns a locus cover for phylogenetic trees.

Proof. We first prove that Algorithm B.1 will halt, then according to the stopping criterion, it is guaranteed that Algorithm B.1 returns a locus cover. Suppose the algorithm keeps running until it reaches the case in which all of the leaf nodes are included in Λ (i.e. Λ = Y ), this results in the trivial locus cover of the phylogenetic tree and the algorithm halts.

A.5 Proof of Lemma A.7 (phylogenetic tree pairwise decomposability)

Lemma A.7 (Phylogenetic tree pairwise decomposability) . Let T = ( V , E ) be a tree with Y = Leaves ( T ) and Λ ⊆ Y . Then Π(Λ) is pairwise decomposable.

Proof. Suppose to reach a contradiction that ∃ y ∗ ∈ Y where y ∗ ∈ Π(Λ) , but y ∗ / ∈ Π( { λ i , λ j } ) ∀ λ i , λ j ∈ Λ . In other words, y ∗ is a Fréchet mean for some weighting on Λ , but y ∗ is not a Fréchet mean for any weighting for any two vertices within Λ .

For arbitrary λ i , λ j ∈ Λ , define the closest vertex to y ∗ on the path between λ i and λ j as

the pair of vertices in Λ whose path Γ( λ 1 , λ 2 ) passes closest to y ∗ . To avoid notational clutter, define v ′ = v ′ ( λ 1 , λ 2 ) . Thus, v ′ represents the closest vertex to y ∗ lying on the path between any two λ i , λ j ∈ Λ .

Because v ′ lies on the path Γ( λ 1 , λ 2 ) , there exists some weighting w for which

For this w , we have that m { λ 1 ,λ 2 } ( w ) = arg min y ∈Y d ( y, v ′ ) , the set of leaf nodes of smallest distance from v ′ . By the initial assumption that y ∗ ̸∈ Π( { λ 1 , λ 2 } ) , we have that y ∗ ̸∈ m { λ 1 ,λ 2 } ( w ) . Let y ′ be a vertex of m { λ i ,λ j } ( w ) that is closest to y ∗ . Clearly, d ( y ′ , v ′ ) 2 < d ( y ∗ , v ′ ) 2 .

Also, let

For any v ∈ V , respectively define the sets of vertices (1) shared in common on the two paths Γ( v, y ∗ ) and Γ( v, y ′ ) , (2) on the path Γ( v, y ∗ ) that are not on Γ( v, y ′ ) , and (3) on the path Γ( v, y ′ ) that are not on Γ( v, y ∗ ) as

We have that | d y ∗ ( v ) | + | d y ∗ ( v ) | +1 = | Γ( y ∗ , y ′ ) | (the difference by 1 represents the final vertex shared in common by both paths) and also that | d y ′ ( v ′ ) | < | d y ∗ ( v ′ ) | since y ′ is closer to v ′ than y ∗ is.

Suppose ∃ λ 3 , λ 4 ∈ Λ yielding

This is to say that u ′ is defined analogously to v ′ , with the additional restrictions that u ′ lie on the path Γ( y ∗ , y ′ ) and u ′ be at least as close to y ∗ as to y ′ . Then, | d y ∗ ( u ′ ) | < | d y ∗ ( v ′ ) | implying d 2 ( y ∗ , u ′ ) < d 2 ( y ∗ , v ′ ) , which contradicts the definition of v ′ .

Suppose now that ∃ λ 3 , λ 4 ∈ Λ yielding

̸

This is to say that u ′ is again defined analogously to v ′ , but u ′ is not on Γ( y ∗ , y ′ ) and is at least as close to y ∗ as to y ′ . Then, either ∩ λ ∈ Λ d C ( λ ) = ∅ (i.e. all lambdas lie in the same branch off of Γ( y ′ , y ∗ ) ), in which case the path between any two lambdas will have the same closest node v ′ to y ∗ , or it is possible to choose a λ 5 ∈ Λ such that d C ( λ 3 ) ∩ d C ( λ 5 ) = ∅ . In this situation, it must be that Γ( λ 3 , λ 5 ) ∩ Γ( y ′ , y ∗ ) = ∅ , which implies there exists a vertex t ∈ Γ( λ 3 , λ 5 ) such that d 2 ( y ∗ , t ) < d 2 ( y ∗ , u ′ ) which violates the definition of u ′ .

̸

Altogether, for all v ∈ Γ( λ i , λ j ) with arbitrary λ i , λ j ∈ Λ , we have that d 2 ( y ′ , v ) < d 2 ( y ∗ , v ) . This implies that for all λ ∈ Λ , there exists a y ′ ∈ Y such that d 2 ( y ′ , λ ) < d 2 ( y ∗ , λ ) and therefore that y ∗ / ∈ Π(Λ) which contradicts that y ∗ ∈ Π(Λ) . It must be that ∀ y ∗ ∈ Y where y ∗ ∈ Π(Λ) , we have y ∗ ∈ Π( { λ i , λ j } ) ∀ λ i , λ j ∈ Λ .

A.6 Proof of Theorem A.8 (minimum locus cover for grid graphs)

Theorem A.8 (Minimum locus cover for the grid graphs) . The minimum locus cover for a grid graph is the pair of vertices in the furthest opposite corners.

Proof. We would like to show two things:

any vertex can be resolved by construction of w using the furthest pair of two vertices (i.e. opposite corners) and

there does not exist a locus cover of size one for grid graphs with more than one vertex.

Let G = ( V , E ) be a grid graph and let Λ = { λ 1 , λ 2 } be a set of two vertices who are on opposite corners of G (i.e., perhipheral vertices achieving the diameter of G ). For notational convenience, set Λ = ( λ 1 , λ 2 ) We can reach any interior vertex by an appropriate setting the weight vector w = [ w 1 , w 2 ] . Then set w = ( d ( v,λ 2 ) d ( λ 1 ,λ 2 ) , d ( v,λ 1 ) d ( λ 1 ,λ 2 ) ) . Finally, the Fréchet mean is given by

Hence Λ is a locus cover. We can clearly see that Λ is a minimum locus cover-the only way to obtain a smaller set Λ ′ is for it to include only a single vertex. However, if Λ ′ = { v ′ } contains only a single vertex, it cannot be a locus cover so long as G contains more than one vertexv ′ is always guaranteed to be a unique minimizer of the Fréchet mean under Λ ′ which misses all other vertices in G . Thus Λ is a minimum locus cover.

A.7 Proof of Lemma A.9 (loci of grid subspaces)

Lemma A.9 (Locus of grid subspaces) . Given any pair of vertices in Λ , we can find a subset G ′ of the original grid graph G = ( Y , E ) which takes the given pair as two corner. Π( Λ ) equals to all the points inside G ′ .

Proof. The result follows from application of Theorem A.8 to the choice of metric subspace.

A.8 Proof of Lemma A.10 (grid pairwise decomposability)

Lemma A.10 (Grid pairwise decomposability) . Let G = ( Y , E ) be a grid graph and Λ ⊆ Y . Then Π(Λ) is pairwise decomposable.

Proof. Suppose we have a grid graph G = ( Y , E ) and a vertex ˆ y ∈ Π(Λ) with Λ ⊆ Y . Due to the fact that ˆ y ∈ Π(Λ) and the fact that G is a grid graph, we have that ˆ y ∈ Γ( λ α , λ β ) for some λ α , λ β ∈ Λ . Then by Lemma A.10,

Therefore loci on grid graphs are pairwise decomposable.

A.9 Proof of Theorem A.11 (no nontrivial locus covers for complete graphs)

Theorem A.11 (Trivial locus cover for the complete graph) . There is no non-trivial locus cover for the complete graph.

Proof. We show that there is no nontrivial locus cover for complete graphs by showing that removing any vertex from the trivial locus cover renders that vertex unreachable. We proceed by strong induction on the number of vertices, n .

Base case We first set n = 3 . Let K 3 = ( V , V 2 ) be the complete graph with three vertices: V = { v 1 , v 2 , v 3 } , and without loss of generality, let Λ = { v 2 , v 3 } be our set of observed classes. There are two cases on the weight vector w = [ w 2 , w 3 ] :

Case 1: Suppose w ̸∈ int ∆ 2 . This means that either w 2 = 1 or w 3 = 1 -which leads to the Fréchet mean being either v 2 or v 3 , respectively. Neither of these instances correspond to v 1 being a minimizer.

Case 2: Suppose w ∈ int ∆ 2 . Then the Fréchet mean is given by

Assume for contradiction that v 1 ∈ Π( Λ ) :

Therefore, v 1 / ∈ Π( Λ ) . This is a contradiction. Thus there is no nontrivial locus cover for K 3 .

Inductive step : Let K n -1 = ( V , V 2 ) be the complete graph with n -1 vertices. Assume that there is no nontrivial locus cover for K n -1 . We will show that there is no nontrivial locus cover for K n . Let the weight vector be w = [ w 1 , ...w n -1 ] corresponding to vertices v 1 , ..., v n -1 ∈ V with Λ = { v 1 , ..., v n -1 } . We want to show that v n ̸∈ Π( Λ ) . We proceed by cases on w .

Case 1: Suppose w ̸∈ int ∆ n -2 where m entries of w are zero, then by strong induction we know that v n ̸∈ Π( Λ ) = { v i } n -m i =1 , i.e., there is no nontrivial locus cover for K n -m . Case 2: Suppose w ∈ int ∆ n -2 .

Assume for contradiction that v n ∈ Π( Λ ) . Using this assumption, we obtain the following

Therefore, v n is not a minimizer, so v n / ∈ Π(Λ) . This is a contradiction, hence K n has no nontrivial locus cover.

A.10 Proof of Theorem 4.8 (active next-class selection for trees)

Proof. We will prove the result by showing that the two optimization problems are equivalent. Let v be a solution to the following optimization problem:

where ∂ in T ′ is the inner boundary of T ′ , the subgraph of T whose vertices are Π(Λ) . This optimization problem can be equivalently rewritten as

and we can furthermore introduce additional terms that do not change the maximizer

Equivalently, we can also connect b to one of the elements of Λ

Due to the uniqueness of paths in trees, this optimization problem also has the following equivalent form without any dependence on b :

Therefore v is a maximizer of | Π(Λ ∪ { v } ) | , as required.

B Algorithms and Time Complexity Analyses

We provide time complexity analyses for Algorithms B.1, B.2, and B.3.

B.1 Analysis of Algorithm B.1 (locus cover for phylogenetic trees)

We provide Algorithm B.1 with comments corresponding to the time complexity of each step.

B.2 Analysis of Algorithm B.2 (computing a pairwise decomposable locus)

We first provide Algorithm B.2 here, with comments corresponding to the time complexity of each step.

We first compute the diameter of the graph G = ( Y , E ) with N = |Y| , which is done in O ( N |E| + N 2 log N ) time using Dijkstra's algorithm to compute the shortest paths between all pairs of vertices. We then iterate over all pairs of elements in Λ with K = | Λ | , which amounts to O ( K 2 ) iterations. Within this, we perform O ( D ) computations of the Fréchet mean, for which each iteration requires O ( N |E| + N 2 log N ) arithmetic operations or comparisons. Combining these, the final time complexity is

B.3 Analysis of Algorithm B.3 (computing a generic locus)

We provide Algorithm B.3 with comments corresponding to the time complexity of each step. Following a similar argument from the analysis of Algorithm B.2, the time complexity is

0.2

0.4

0.6

0.8

1.0

Confidence

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

Reliability

ECE optimal

CLIP default

Loki optimal

Perfect calibration

7.20

7.25

7.30

7.35

[

d

2

(

y

,

y

)]

0.1

0.2

0.3

0.4

0.5

ECE

ECE vs. Loki perf.

ECE optimal

CLIP default

Loki optimal

0.005

0.010

0.015

0.020

0.025

Softmax temperature

7.20

7.25

7.30

7.35

Expected squared distance

0.592

0.593

0.594

0.595

0.596

0.597

0.598

Zero-one error

CIFAR-100 default tree

error tradeoff

Argmax prediction

CLIP default

0.002

0.004

0.006

0.008

0.010

0.012

0.014

Softmax temperature

0.2

0.4

0.6

0.8

1.0

Confidence

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

Reliability

ECE optimal

Loki optimal

CLIP default

Perfect calibration

20.0

22.5

25.0

27.5

30.0

[

d

2

(

y

,

y

)]

0.1

0.2

0.3

0.4

0.5

ECE

ECE vs. Loki perf.

ECE optimal

CLIP default

Loki optimal

0.005

0.010

0.015

0.020

0.025

Softmax temperature

20

22

24

Expected squared distance

0.60

0.65

0.70

0.75

0.80

Zero-one error

CIFAR-100 WordNet tree

error tradeoff

Argmax prediction

CLIP default

0.002

0.004

0.006

0.008

0.010

0.012

0.014

Softmax temperature

where K is the number of observed classes.

C Additional Experiments

C.1 Additional calibration experiments

In the main text, we evaluated the effect of calibrating the Softmax outputs on the performance of LOKI using a metric space based on the internal features of the CLIP model. Here, we perform the same analysis using two external metric spaces.

Setup We perform our calibration analysis using the default and WordNet tree.

Results Our results are shown in Figure 4. Like our experiment using an internally-derived metric, we find that the optimal Softmax temperature is close to the CLIP default and the optimal temperature for calibration. For the default tree, we again found that temperature scaling can be used to improve accuracy using LOKI.

Setup We calibrate via Softmax temperature scaling [13] using CLIP on CIFAR-100. We do not use an external metric space, and instead use Euclidean distance applied to the CLIP text encoder.

Results The reliability diagram in Figure 3 shows that the optimal Softmax temperature for LOKI is both close to the default temperature used by CLIP and to the optimally-calibrated temperature. In Figure 3 (right), we find that appropriate tuning of the temperature parameter can lead to improved accuracy with CLIP , even when no external metric space is available.

C.2 Ablation of LOKI formulation

LOKI is based on the Fréchet mean, which is defined as arg min y ∈Y ∑ K i =1 P λ i | x d 2 ( y, λ i ) . However, this is not the only approach that can be considered. For example, the Fréchet median , often used in robust statistics, is defined as arg min y ∈Y ∑ K i =1 P λ i | x d ( y, λ i ) , without squaring the distances. More generally, we define arg min y ∈Y ∑ K i =1 P λ i | x d β ( y, λ i ) and evaluate different choices of β .

Setup We conduct this experiment on ImageNet using SimCLR as our pretrained classifier with K = 250, 500, and 750 randomly selected classes.

Results From this analysis, we conclude that using the Fréchet mean is the optimal formulation for Loki, as it achieved the lowest mean squared distance for all settings of K .

Table 4: Expected squared distances of SimCLR+Loki alternatives on ImageNet. We ablate over the choice of distance exponent in LOKI (where β = 2 , corresponding to the Fréchet mean), including the Fréchet median ( β = 1 ). That is, we tune β : ˆ y ∈ arg min y ∈Y ∑ K i =1 P λ i | x d β ( y, λ i ) and find that the optimal setting is β = 2 , corresponding to LOKI.

C.3 Comparison across metric spaces

Expected squared distances cannot be directly compared across metric spaces, as they may be on different scales. Our solution is to use a form of normalization: we divide the expected squared distance by the square of the graph diameter. This brings all of the values to the 0-1 range, and since E [ d 2 ( y, ˆ y )] / diam ( G ) 2 indeed also generalizes the 0-1 error, this enables comparison between 0-1 errors and those from different metric spaces. We provide these results in Table 1, again for our CLIP experiments on CIFAR-100. This evaluation metric enables us to determine which metric spaces have geometry best 'fit' to our pretrained models.

Setup Using E [ d 2 ( y, ˆ y )] / diam ( G ) 2 , we re-evaluate our CLIP results on CIFAR-100 shown in Table 1 for the ResNet-50 architecture.

Results For CIFAR-100, we observe that the WordNet metric space resulted in the lowest error and therefore has the best geometry.

D Experimental Details

In this section, we provide additional details about our experimental setup. 3

D.1 CLIP experiments

All experiments are carried out using CLIP frozen weights. There are no training and hyperparameters involved in experiments involving CLIP, except for the Softmax temperature in the calibration analysis. We evaluted using the default CIFAR-100 test set. The label prompt we use is "a photo of a [class_name]."

D.2 ImageNet experiments

To construct our datasets, we randomly sample 50 images for each class from ImageNet as our training dataset then use the validation dataset in ImageNet to evaluate LOKI's performance. We extract the image embeddings by using SimCLRv1 4 and train a baseline one-vs-rest classifier. We use WordNet phylogenetic tree as the metric space. The structure of phylogenetic tree can be found at here 5 . We test different numbers of observed classes, K , from 1000 classes. Observed classes are sampled in two ways, uniformly and Gibbs distribution with the concentration parameter 0.5.

D.3 LSHTC experiments

We generate a summary graph of the LSHTC class graph (resulting in supernodes representing many classes) by iteratively:

randomly selecting a node or supernode

merging its neighbors into the node to create a supernode

until the graph contains at most 10,000 supernodes. In the LSHTC dataset, each datapoint is assigned to multiple classes. We push each class to its supernode, then apply majority vote to determine the final class. We test different numbers of observed classes, K , from 10,000 classes. We collect datapoints which are in the observed classes. Then split half of dataset as training dataset and make the rest as testing dataset, including those datapoints which are not in the observed classes. We train a baseline classifier using a 5-NN model and compare its performance with LOKI.

E Broader Impacts and Limitations

As a simple adaptor of existing classifiers, it is possible for our method to inherit biases and failure modes that might be present in existing pretrained models. On the other hand, if the possibility of harmful mis-classifications are known a priori, information to mitigate these harms can be baked into the metric space used by LOKI. Finally, while we have found that LOKI often works well in practice, it is possible for the per-class probabilities that are output by a pretrained model to be sufficiently mis-specified relative to the metric over the label space, or for the metric itself to be mis-specified. Nonetheless, we have found that off-the-shelf or self-derived metric spaces to work well in practice.

F Random Locus Visualizations

In Figures 1, 5, 6, 7, 8, and 9, we provide visualizations of classification regions of the probability simplex when using LOKI with only three observed classes out of 100 total classes, and different types of random metric spaces. The first example in Figure 1 shows the prediction regions on the probability simplex when using standard arg max prediction-the three regions correspond to predicting one of the three classes (0, 39, and 99) and no regions corresponding to any of the other classes { 1 , ..., 38 , 40 , ..., 98 } . We compute these regions using a version of Algorithm B.3, and while

it does have exponential time complexity, the exponent is only three in this case since we consider only three observed classes.

On the other hand, the other three examples in Figure 1 and Figures 5, 6, 7, 8, and 9 all show the prediction regions on the probability simplex when using LOKI. Figure 5 shows this for random tree graphs. Here, the prediction regions are striped or contain a single triangle-shaped region in the center-these correspond, respectively, to intermediate classes along branches of the tree leading up from the observed class and the prediction region formed by the common-most parent node. Figure 6 shows similar regions, although these are more complex and are thus more difficult to interpret, as phylogenetic trees are metric subspaces equipped with the induced metric from trees. Furthermore, in order to generate phylogenetic trees with 100 leaves, we needed to create much larger trees than the ones used for Figure 5, which led to narrower prediction regions due to the higher graph diameter. Finally, Figures 7, 8, and 9 each show the prediction regions using LOKI when the metric space is a random graph produced using Watts-Strogatz, Erd˝ os-Rényi, and Barabási-Albert, models respectively. These prediction regions are more complex and represent complex relationships between the classes.

39

6

39

6

39

39

6

75

75

75

75

75

75

75

75

50

50

50

50

50

50

50

25

25

25

25

25

25

25

25

99

Tree Graph

66

Tree Graph

o

66

Tree Graph

0

66

Tree Graph

66

Tree Graph

66

Tree Graph

99

Tree Graph

99

Tree Graph

6

39

39

39

39

39

39

39

75

75

75

75

75

75

50

50

50

50

50

99

Tree Graph

0

25

99

Tree Graph

99

Tree Graph

0

25

99

Tree Graph

66

Tree Graph

66

Tree Graph

0

99

Tree Graph

25

99°

Tree Graph

25

39

39

39

39

39

39

39

39

75

75

75

75

75

75

75

75

50

50

50

50

50

25

25

25

25

25

25

66

Tree Graph

b

o

99

Tree Graph

o

99

Tree Graph

b

o

99

Tree Graph

b

1o

99

Tree Graph

D

o

66

Tree Graph

99

Tree Graph

Q

99°

Tree Graph

o

39

39

39

39

39

39

39

39

1

75

75

75

75

75

50

50

50

50

50

25

25

25

25

25

25

25

99°

Tree Graph

39

0

99

Tree Graph

39

99

Tree Graph

39

99

Tree Graph

39

o

66

Tree Graph

39

66

Tree Graph

39

o

99

Tree Graph

6

99°

Tree Graph

39

0

75

75

75

75

75

75

75

50

50

50

50

50

50

25

25

25

25

25

25

66

o

66

99

o

99°

99

o

66

o

66

Tree Graph

b

99

o

Tree Graph

Tree Graph

Tree Graph

Tree Graph

Tree Graph

Tree Graph

Tree Graph

25

6

6

6

6

6

39

39

75

75

75

75

75

75

75

75

50

50

50

50

50

50

25

25

25

25

25

25

25

25

o

o

o

o

01

6

39

39

39

39

6

6

39

75

75

75

50

50

50

50

25

25

25

25

66

0

。

0

Phylo. Tree Graph”

o

66

01

66

o

39

39

39

39

39

39

39

39

75

75

75

75

75

75

75

75

50

50

50

50

50

25

25

25

25

99

99

99

Q

o

99

0

1o

66

Phylo. Tree Graph

0

o

66

0

o

99

0

01

66

Phylo. Tree Graph

0

o

39

1

39

39

39

39

39

39

39

75

75

75

50

50

50

25

25

25

25

25

25

0

0

99

Phylo.Tree Graph

o

0

o

0

39

39

39

39

6

6

6

39

75

75

75

75

75

75

75

50

50

50

50

50

50

50

25

25

25

25

25

25

25

99

o

99

o

9g

66

o

66

66

Phylo. Tree Graph

o

66

99

6

6

39

6E

39

39

6

39

75

75

75

75

75

75

75

75

50

50

50

50

50

50

50

50

25

25

25

25

25

25

25

25

0

。

。

o

。

o

39

39

39

39

39

39

39

39

75

75

75

75

50

50

50

66

Small World Graph°

o

25

66

Small World Graph

66

Small World Graph

o

o

99

Small World Graph

o

99

Small World Graph

o

25

99

Small World Graph

U

10

6

39

6

39

6

39

6

39

75

75

75

75

75

75

75

50

50

50

50

50

66

Small World Graph

o

25

s

o

25

66

Small World Graph

o

25

Io

25

66

Small World Graph

o

U

o

25

U

o

25

39

39

39

39

6

39

6

6E

75

75

75

75

75

75

75

50

50

50

50

50

25

25

25

25

25

25

0

0

0

39

39

39

39

39

39

39

39

75

75

75

75

75

75

75

75

50

50

50

50

50

50

50

25

25

25

25

25

0

99

Small World Graph°

o

66

Small World Graph

o

66

Small World Graph

o

66

Small World Graph

o

66

Small World Graph

66

Small World Graph

o

6

39

39

6E

6

6

6

6

75

75

75

75

75

75

75

75

50

50

50

50

50

50

25

25

25

25

25

25

25

25

1o

o

o

1o

o

o

o

39

39

39

39

39

39

39

39

75

75

75

75

75

75

50

50

50

50

50

50

25

25

25

25

25

U

o

o

o

o

0

o

39

39

39

39

39

39

39

39

75

75

75

75

75

75

50

50

50

50

25

25

25

25

25

25

0

0

0

o

o

D

o

0

o

o

0

6

39

6

39

6

6

6

39

75

75

75

75

50

50

50

25

25

一

25

25

25

25

o

o

o

o

o

。

39

39

39

39

39

39

6

39

75

75

75

75

75

75

50

50

50

50

50

50

50

25

25

25

25

25

25

25

o

o

o

o

o

o

01

39

6

39

6

6

6

39

6

75

75

75

75

75

75

75

75

50

50

50

50

50

50

50

50

25

25

25

25

25

25

25

25

Barabasi-Albert Graph

66

0.

6

39

39

39

39

6

39

39

75

75

75

75

75

75

75

50

50

50

50

25

25

25

25

25

25

25

Barabasi-Albert Graph

66

Barabasi-Albert Graph

66

Barabasi-Albert Graph o

66

Barabasi-Albert Graph o

66

Barabasi-Albert Graph

66

0.

Barabasi-Albert Graph o

66

Barabasi-Albert Grapho

66

Barabasi-Albert Graph

66

39

39

39

39

39

39

39

6

75

75

75

75

75

+

75

75

75

50

50

50

50

50

Barabasi-Albert Graph 0

66

25

Barabasi-Albert Graph 0

66

Barabasi-Albert Graph 0

66

25

Barabasi-Albert Graph0

66

25

Barabasi-Albert Graph0

66

25

Barabasi-Albert Graph0

66

■

25

Barabasi-Albert Graph0

66

25

Barabasi-Albert Graph o

66

0.

25

39

39

39

39

39

39

39

39

75

75

75

75

75

50

50

50

50

25

25

25

25

25

25

25

Barabasi-Albert Graph

99

0.

Barabasi-Albert Graph

99

Barabasi-Albert Graph

66

Barabasi-Albert Grapho

66

0.

Barabasi-Albert Graph

66

0.

Barabasi-Albert Graph

66

0.

Barabasi-Albert Graph

66

0.

Barabasi-Albert Graph

,66

0.

39

39

39

39

39

39

6

6

75

75

75

75

75

75

75

50

50

50

50

50

50

50

50

25

25

25

25

25

25

25

25

Barabasi-Albert Grapho

66

Barabasi-Albert Graph o

66

Barabasi-Albert Graph 0

66

Barabasi-Albert Grapho

66

Barabasi-Albert Graph o

66

Barabasi-Albert Graph

66

Barabasi-Albert Grapho

99

Barabasi-Albert Graph0

66

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors explore the adaptation of pretrained models from a geometric perspective. Specifically, this paper leverages the label information as a metric space and proposes a simple approach to predict new classes based on the pretrained models’ zero-shot prediction without any further training. This paper also provides some theoretical analysis w.r.t learning-theoretical results, locus cover and active next-class selection. In the empirical results, this paper conducts experiments on several datasets and verify the effectiveness of the proposed method.\n\nSTRENGTHS:\n- The paper is generally well-written and easy to follow.\n- The topic of adaptation on the pretrained models from the geometric perspective is interesting.\n- The proposed method is simple and efficient.\n- The main claims are generally supported by theoretical results.\n\nWEAKNESSES:\n- The proposed method may require some additional information such as label relationship, which may not be applicable in some real-world situations.\n- It lacks theoretical analysis about the superiority of the proposed LOKI.\n- The empirical results are not adequate to support the main claims.\n\nQUESTIONS:\n- As discussed in the Weaknesses, it is not clear whether the proposed LOKI is the optimal geometric adaptation method for enabling the prediction of unobserved classes. Are there any alternatives for Fréchet mean estimator to yielding more reliable predictions? (An example) It would benefit a lot if some empirical or theoretical comparison are added to demonstrate the effectiveness of the proposed LOKI.\n- I appreciate this paper that most claims are supported by theoretical results. It would further improve the paper if more empirical results were added to support these claims. For example, some toy experiments can be provided to verify Theorem 4.8.\n- In Table 1, the effectiveness of LOKI is explored on CIFAR-100. It would be more convincing to conduct experiments on more real-world datasets.\n- The experiments in Table 1 seems not comprehensive. From the results, we can see that LOKI presents limited improvements on ViT-L-14 backbone. It would be better to add more discussions and more comprehensive empirical results.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper explores the concept of geometry-aware adaptation for label spaces. The paper introduces a method called \"LOKI\" that allows pretrained models to make predictions for classes that were not observed during training. LOKI utilizes metric space information to adapt the model's predictions to unobserved classes.\n\nThe paper discusses the theoretical foundations of LOKI, including the definition of loci and identifying locus covers in graphs. It presents algorithms for efficiently computing the locus and describes an active learning-based strategy to select the next class for observation in order to maximize the size of the locus.\n\nExperimental results demonstrate that LOKI improves the performance of zero-shot models even without external metric space information. It adapts to label spaces with a large number of unobserved classes and outperforms baseline models in terms of mean squared distances. The paper also validates the effectiveness of the active next class selection approach, showing that it leads to larger loci compared to random selection.\n\nSTRENGTHS:\n1. Originality: The paper exhibits a high degree of originality in several aspects. Firstly, it introduces the concept of geometry-aware adaptation for label spaces, which is a novel approach to address the challenge of predicting unobserved classes in pretrained models. The formulation of LOKl and its integration with metric space information is a unique contribution. Additionally, the paper presents novel definitions, algorithms, and strategies related to loci, identifying locus covers, and active learning-based class selection. These original contributions set the paper apart and make it a valuable addition to the field.\n\n2. Clarity: The paper excels in terms of clarity, making the research accessible to a wide range of readers. The authors provide clear explanations of concepts, definitions, and algorithms, ensuring that readers can understand the technical aspects of LOKI. The paper is well-structured, with logical flow and section organization. \n\n3. Quality: The paper demonstrates a high level of quality in terms of its theoretical foundations, experimental methodology, and presentation of results. The authors provide a thorough and rigorous analysis of the problem, offering formal definitions and proving relevant theorems. \n\n4. Significance: The paper holds significant importance in the field of machine learning and predictive modeling. It addresses a challenging problem of predicting unobserved classes in pretrained models, which has practical implications in various real-world scenarios. The introduction of LOKI and its ability to adapt predictions using metric space information expands the capabilities of pretrained models, enabling them to make more accurate predictions for classes that were not observed during training. The experimental results demonstrate the effectiveness of LOKI and its superiority over baseline models, further highlighting its significance in advancing the field.\n\nWEAKNESSES:\n1. Lack of Comparative Analysis: While the paper presents LOKI as a novel approach, it would benefit from a more comprehensive comparative analysis against existing methods or related works. Providing such a comparison would strengthen the paper's argument for the originality and effectiveness of LOKI.\n\n2. Scalability Analysis: The paper briefly mentions the computational complexity of LOKI, but a more detailed analysis of its scalability would be valuable.\n\nQUESTIONS:\n1. While the paper briefly mentions the computational complexity of LOKI in term of label spaces, will time consumption of this method be also related with higher-dimensional metric spaces? When compared with original method without LOKI, how much is extra consumption of memory and runtime with LOKI? \n\n2. When multiple external metrics are provided, can we decide which metric is most suitable in advance?\n\n3. Is the property pairwise decomposable necessary for the improvement of LOKI? Does it only work on the dataset satisfying this property?\n\nLIMITATIONS:\nNo potential negative societal impact of their work. Suggestions are provided in the weakness and questions.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors consider the problem of predicting examples from unseen but \"known\" classes. Taking inspiration from structured prediction, the paper intends to exploit the knowledge of structure in the full label space. The authors propose an alternative to the popular \"argmax-over-logits\" prediction, called Loki, by computing the Frechet mean with an appropriate distance metric in label space. A theoretical result on a simple logistic data model shows the sample complexity bound with Loki. Also considered is the important question of what training labels are necessary and sufficient to be able to span the whole label space during prediction. On the experimental effectiveness, the authors demonstrate reasonable percentage point improvement over baseline \"argmax\" standard prediction.\n\nSTRENGTHS:\nStrengths:\n1. My knowledge in structured prediction is limited. However, it seems to me that the results in this paper are significant and interesting\n2. I found the method Loki to be a neat approach to predicting unseen classes, when the distance metric for th label space is known. It is a scalable method since the computation involved is a fixed linear transformation\n3. The authors have stated and and given proofs for the minimal locus cover for two non-trivial label space structures with clear motivation - phylogenetic trees and grid graphs\n4. An algorithm of a \"greedy\" nature through an active-learning framework is then given, which is polynomial time. \n5. Experimental results: it seems that Loki achieves strong results for CLIP\n6. While I have not checked the proofs in detail, the methodology appears sound and rigorous.\n\nWEAKNESSES:\nWeaknesses:\n1. It is not clear to me if there can be guarantees provided when the true label space metric is not available or only approximately known\n2. Theorem 1 is for the case of logistic regression. It is not clear whether this is a powerful enough model for practice. \n3. I have either missed this part or there needs to be a detailed discussion of the implications of Theorem 1\n4. I suggest providing \"error bars\" or significance level for the numbers reported in Table 1\n5. More experimental details would be helpful for reproducing the results \n6. It is unclear to me if there are other competitive structured prediction techniques applied to pretrained models and what gains Loki makes over them, if any.\n\nQUESTIONS:\nQuestions:\n1. Line 210: \"In cases involving an extremely large number of classes, it is desirable to use LOKI on the smallest 211 possible set of observed classes\" - is this because sample complexity of in Theorem 1 scales as K^2? Are there other considerations?\n2. I would prefer to get more explanation for the proposed use-case of grid graphs. While phylogenetic trees is more intuitive, grid graphs might benefit from a simple examples\n3. Does a counterpart of Theorem 4.8 hold for grid graphs?\n4. Sec 5.2: Is one-vs-rest the most competitive baseline?\n5. For the result on computing loci for partially oserved label spaces, are PubMed and LSHTC the standard datasets to consider? Please provide any reasoning behind this choice. Any references will be useful\n6. Fig 1 (and related ones in Appendix)- it is not immediately clear what the authors intend to demonstrate here.\n7. I am curious about the impact of Loki on calibration.\n\nLIMITATIONS:\nLimitations:\nI did not see an explicit discussion on the limitations and scope for improvement", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

comments: SUMMARY:
This paper considers the zero-shot model adaptation to testing tasks that include classes not seen during training. The authors propose a post-hoc method called LOKI, which applies a class-graph-based transformation to make predictions on these unseen classes. The authors also provide the theoretical analysis and experimental results to support the proposal.

STRENGTHS:
The problem considered in this paper is significant in the machine learning community, and the proposed solution is technologically reasonable.  
The theoretical results are interesting and provide a solid justification for the proposed solution.   
The paper is well-written and easy to follow.

WEAKNESSES:
The graph of LOKI seems heavily rely on the prior knowledge of the classes. 
In the Sections 5.1 and 5.2 of this paper, experiments were conducted on CIFAR100 and ImageNet, respectively. Are there any additional results on other datasets?

QUESTIONS:
How can a reliable graph be obtained in real-world applications? The experiments implement it using hierarchical trees and WordNet. Does this limit the practical application of the proposal?


--- 
Thanks for the detailed clarifications, which have addressed my concerns. I would like to keep my score.

LIMITATIONS:
The authors do not provide a discussion about the limitations and potential negative societal impact.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a method capable of adapting a pre-trained model to a larger label space. To achieve this, it leverages the information of geometric distances between labels within the target larger space and replaces the common argmax operation with the Frechet mean. The paper also includes several theoretical analyses on sample complexity and optimal label subspaces.\n\nBased on the proposed locus definition, the paper presents an active learning paradigm with the objective of expanding the maximum label coverage by introducing new labels. The empirical results demonstrate that the proposed method outperforms the vanilla baseline method in tasks of zero-shot classification and partially-observed label spaces.\n\nSTRENGTHS:\n- the usage of Frechet mean is simple and intuitive for solving partial label classification tasks\n- the theoretical analyses are thorough and address several key questions in terms of sample complexity and optimal label subspace\n\nWEAKNESSES:\n- Evaluation Fairness: The empirical results from Table 1 and 2 are reported as $E[d^2(y, \\hat{y})]$ in the respective metric space that Loki is optimized for. It is not too surprising that Loki outperforms the argmax/one-vs-rest baseline trained without such information. The paper lacks more fair metrics like accuracy or a held-out metric space to demonstrate the overall effectiveness of Loki.\n- Stronger Baselines: Both zero-shot classification and partially-observed classification experiments only include vanilla baselines (e.g., CLIP argmax, SimCLR with one-vs-rest classifier). Considering the proposed method is pretty simple, the paper should compare against stronger baselines [1][2][3][4] in terms of performance and efficiency. Additionally, the claim that graph neural network architectures are heavyweight and challenging to scale up to extremely large graphs, where Loki performs better, needs quantitative measurements to support it (e.g., running efficiency, model size, etc.).\n- Uncalibrated Uncertainty in Loki Formulation: Often, the probability of outputs from classification is uncalibrated [5], and the paper lacks an ablation study examining how the correctness of uncertainty $P_{\\lambda}$ affects the final performance.\n- Effectiveness of Active Learning: While the final experiment indicates that the proposed active learning paradigm successfully leads to larger loci, there is no guarantee or proof on how well the paradigm affects the final performance in classification tasks.\n- Clarity: Section 4.2 is too wordy; consider emphasizing key statements and bringing several key algorithms from B.2, B.3 to the main text for better clarity.\n\n[1] X. Wang, Y. Ye, and A. Gupta. Zero-shot recognition via semantic embeddings and knowledge graphs. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6857–6866, Los Alamitos, CA, USA, jun 2018. IEEE Computer Society.\n[2] Abhinaba Roy, Deepanway Ghosal, Erik Cambria, Navonil Majumder, Rada Mihalcea, and Soujanya Poria. Improving Zero-Shot Learning Baselines with Commonsense Knowledge. Cognitive Computation, 14(6):2212–2222, November 2022.\n[3] Christoph H. Lampert, Hannes Nickisch, and Stefan Harmeling. Learning to detect unseen object classes by between-class attribute transfer. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 951–958, 2009.\n[4] Junyu Gao, Tianzhu Zhang, and Changsheng Xu. I Know the Relationships: Zero-Shot Action Recognition via Two-Stream Graph Convolutional Networks and Knowledge Graphs. Proceedings of the AAAI Conference on Artificial Intelligence, 33(01):8303–8311, July 2019.\n[5] Nixon, Jeremy, Michael W. Dusenberry, Linchuan Zhang, Ghassen Jerfel, and Dustin Tran. \"Measuring Calibration in Deep Learning.\" In CVPR workshops, vol. 2, no. 7. 2019.\n\nQUESTIONS:\nAs mentioned in the weakness section, here are few key questions left to answer:\n- How does Loki perform in terms of accuracy?\n- How does Loki perform compared to stronger baselines from recent literatures?\n- How does uncalibrated probability affect the final performance?\n- How useful is the proposed active learning in terms of final classification performance?\n\nLIMITATIONS:\nThe authors have adequately discussed several limitations and negative societal impact in the appendix.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-27"}