Exploiting a Zoo of Checkpoints for Unseen Tasks
Authors: Jiaji Huang
Qiang Qiu
Kenneth Church
Venue: NeurIPS 2021 Poster (2021)
Keywords: task space
generalization
Gaussian process
meta learning

Abstract: There are so many models in the literature that it is difficult for practitioners to decide which combinations are likely to be effective for a new task. This paper attempts to address this question by capturing relationships among checkpoints published on the web. We model the space of tasks as a Gaussian process. The covariance can be estimated from checkpoints and unlabeled probing data. With the Gaussian process, we can identify representative checkpoints by a maximum mutual information criterion. This objective is submodular. A greedy method identifies representatives that are likely to "cover'' the task space. These representatives generalize to new tasks with superior performance. Empirical evidence is provided for applications from both computational linguistics as well as computer vision.

Exploiting a Zoo of Checkpoints for Unseen Tasks
Jiaji Huang Baidu Research Sunnyvale, CA, 94089 huangjiay
Qiang Qiu
School of Electrical and Computer Engineering Purdue University, West Lafayette, IN, 47907
qqiuêpurdue.
Kenneth Chureh
Baidu Research Sunnyvale, CA, 94089
kemmethchurchhaaidd.comm
Abstract
There are so many models in the literature that it is difficult for practitioners to decide which combinations are likely to be effective for task. This paper attempts to address this question by capturing relationships among checkpoints published the web. We model the space of tasks Gaussian process. The covariance can be estimated from checkpoints and unlabeled probing data. With the Gaussian process, we can identify representative checkpoints by maximum mutual information criterion. This objective is submodular. A greedy method identifies representatives that likely "cover" the task space. These representatives generalize to new tasks with superior performance, Empirical evidence is provided for applications from both computational linguistics as well as computer visior
Introduction
There are many model checkpoints published. For example, upon acceptance of this paper, the Huggingface repository"inchhe includes over 18,000 checkpoints trained for dozens of tasks (The tasks are tagged very coarsely. There will be many more we tag in a fine-grained way). On the other hand, novel tasks can always arise, exciting tremendous interest meta learning learning to learn I 3 4 This paper does not propose a new learning-to-lean method. Rather, we are interested the question. "Can we use the checkpoint 2oo to build something that can adapt to unseen tasks?"
To address the question, have understand how tasks relevant to each other. A paraphrase of task relevance is task transferability [5]. Intuitively, transferring to new task is easier we warm start from a very relevant Earlier work like Taskonomy 5 explicitly uses performance metrics (e.g., test accuracy) of transfer learning. Since then, multiple works have developed task descriptors to avoid running training and evaluation as Taskonomy. A few prominent examples are Task2vec [6), Attribution Map (A-Map) (7]. and LEEP g of these methods has its own pros and cons. However, these studies mainly aim at interpreting the performance of transfer learning 9 10 They mainly attack the question, "For a target task, which source task is the most relevant?". - It is different from the question we are interested in.
Efforts in this space originates from computer vision applications.
Very recently, Task2vec has also been extended computational linguistics 9 Each
Understanding the relation among tasks is crucial for meta learning and transfer learning. is known that dissimilar tasks can impose significant challenges for these learning frameworks [1] 12 Aware
results can reproduced using code sulhttps://gitieee wtt.c/oo//cbccee haatmm/aaidu--ae resaiddurreeearccaaak..eeee http://thh....oooooaeereereeeaaa https://huggingface co/models
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

of this fact, several recent works [33 14 develop generalization error bounds for learning new tasks. In particular, they address the question, "how many tasks has be seen, so that meta- trained model can generalize to new tasks?". These works often assume that there is shared function, mapping data to representations for all tasks. Then they propose their measure of task diversity, which further employed to bound the generalization error for new task. These bounds suggest that the seen tasks must be sufficiently diversified to "cover" all components that may be used for a future task. However, it is not obvious how to compute these diversity measures in practice.
Back to our problem, zoo of checkpoints solve a collection of seen tasks. We are interested in combining/fusing a subset of them. Motivated by the theoretical works [13] 14] we want the selected checkpoints (essentially the tasks they solve) to be diversified. In perhaps tangential problem [15]. the authors consider where to place sensors in a room such that they are the most informative about the undeployed locations. They show that the placement should be diversified and well represent the landscape. To this end, we could draw an analogy. Consider the task space as a room, and checkpoints as sensors. The unseen tasks are locations where no sensor placed but we wish to infer their information. In later sections, we shall see how this analogy helps us design strategy for selecting checkpoints.
A relevant concept to our work is model ensemble, which has a long history [16), and still actively being studied recently [7]. However, model ensemble usually assumes a shared output space for all component models, therefore no notion of multiple (seen or unseen) tasks. Another relevant topic is feature selection [88. Indeed, if we consider each checkpoint a feature extractor, picking checkpoints amount to identifying useful features. However, feature selection methods usually aim at finding the most relevant features for a target variable. In other words, they only target at a single seen task.
2 Motivating the Problem
Consider zoo of n checkpoints, where the i-th checkpoint is trained for the i-th task. In this paper, we define the i-th task as joint distribution (x, y) to be fit to, where x E X' is input and y E 34 is target output. The i-th checkpoint is trained on a dataset Dr drawn from pt. D; is often proprietary. Or due to non-trivial preprocessing, it is very difficult to obtain the data exactly as it was. The tasks are assumed to share the same input space, but differ in their target spaces. This is a natural assumption. Consider a zoo of convolutional networks that handle various computer vision tasks (5), or a collection of huggingface transformers for multiple linguistic tasks. The X' is space of images for the former case, and text for the later.
Suppose there is universe of (possibly infinitely many) tasks, drawn from "fyper-distributionn" P. That is (Pi..... Pa
~ P. Modeling a family of tasks as "distribution over distributions" [] 19 has long history in learning-to-lean information from .I,... Pn. and we are interested in building a "stronger" model from them that can easily generalize unseen tasks (Pn-1....). It should be emphasized that these new tasks are not revealed to us before we build the "stronger" model.
literature. The n checkpoints in the zoo have distilled
To be more specific, let the i-th checkpoint be in the (commonly seen) form, h, o f. : X + V. where f; is a feature extractor. h; is a task-specific "head" for the i-th task, and o denotes function composition. h; is often "simple" function, e.g., softmax classifier. Note that h; may be available. This is true for many checkpoints in huggingface repository. We are interested in selecting subset S of checkpoints from the zoo before seeing any new task Pn+1s..., and building "stronger" feature extractor by concatenating the selected f/'s, f A Biesf, The f should be quickly adaptable to new tasks Pn+1...- In other words, given some small training data for a new task, we want a task-specific "head" on top of the f to be "casily" learned and work well on test data.
necessarily
Of course, this approach should have some budget constraints. Indeed, when n is big, concatenating all the f,'s 's may be costly in computation, and result in overfitting, At this point, one may propose to use feature selection methods like Henn--Rduddncy--aa Nar-MMax---eeeee Relevance (mRMR) [8]. That is, to pick the f, 's that are the most informative for new task, meanwhile least redundant mutually. However, in our case, no new task is revealed to us before selecting. Thus we cannot simply apply mRMR.
Having ruled out feature selection methods, we start to re-think the problem by modeling the space of tasks. To this end, we may make an analogy to optimal sensor placement [15]. In that application, one is supposed to pick the best locations to place sensors, so that their readouts

---

the most informative about the unsensed locations. In information theoretic language, this would be max I (sensed location; unsensed location), where .:.)) is mutual information. our problem. each checkpoint handles its specific task. It can be considered as a "sensor" placed at task. Then, the checkpoint zoo is "pilot deployment" that places many "sensors" on grids of the task space, which can reveal how the tasks are related. Selecting checkpoints is essentially picking a few seen tasks that may be the most informative about the unseen ones. In the following sections, we concretize the above idea by modeling and exploiting the task space.
Modeling the Task Space
Theoretical works I 1 13]
often do not specify a form for the hyper-distribution P. In this work, we assume P a Gaussian process. So a next step is to define the covariance function, k(p,, Py), for P. In other words, we want to come up with a similarity measure between the two distributions p. and Pj. Moreover, by definition, the k has to be Positive Semi-Definite (PSD). In the following, we first rule out several options that are less applicable. Then propose approach.
3.1 Taxonomy Task Similarities
If we had access to data samples D,, and Dj. a natural idea would be comparing p; and pj directly. The difficulty is that the output spaces Di and y, are not directly comparable. For example, 3/ may be the input images' class labels, whereas V/ may be the foreground to be segmented. Therefore, straightforward measures like KL divergence cannot be applied. To address this issue, Geometric Dataset Distance (GDD) [20] proposes to estimate Wasserstein distance between p; and Pj. by carefully designing the "transport cost" between }' and )' using D, and Dj- However, D, and D) are often proprietary. For example, the data to train huggingface transformers are often preprocessed with various tools. It is very hard (if not impossible) recover the exact same D;,
Other approaches treat s(p,, Pý) as the transferability from the i-th task to the j-th. Taskonomy [5] empirically computes that by running training and testing for this i-to-j transfer learning problem. Doing that for all (i.j) pairs is very costly. Besides, non non-trivial normalizations have to be applied as performance metrics can differ for different tasks. To circumvent this brute-force process, people have come up with vector representations of tasks, reducing tasks similarities to distances in this vector space. For example, Task2vec 66 argues that Fisher information matrix (essentially gradient w.r.t. parameters the checkpoint) is an indicative representation for the corresponding task. However, assumes that all checkpoints have the same model configuration, which is very strong assumption and need not be satisfied in practice. Another work, A-Map (7) is based on the intuition that similar tasks should have similar saliency map, which is the gradient of (pooled) feature w.r.t. input. Therefore, it only requires some unlabelled probing data, and model architectures can be different. However, the kack-propagatton can be more costly. Further more, compare two saliency maps, their inputs have to be the same. This constraint can be violated for NLP applications, as the same input text often mapped to two different sequences of embedding vectors using two checkpoints.
Recently, NCE 21 and LEEP g measure task transferability by estimating an approximation of conditional entropy, H (yy,ly = H I(y:;,) Ignoring (y,)) and focusing on the mutual information term, we can understand these two methods as measuring the dependency between two target spaces )/ and y/. However, to estimate H H(y,ly:). data labels must be available.
For clarity, we list the discussed methods Tab. T Also, the last row of Tab. D highlights some properties desired method should have. We can see various reasons why these existing methods are not applicable. Nevertheless, LEEP (8) provides us with a principled way of understanding the relationship between two tasks, i.e., measuring dependency between their target spaces. We next derive another approximation to this dependency, which a desired method.
3.2 Proposed Approach
Following the setup in NCE 21 and LEEP g we also assume m input samples X & [x ...,..., x..,, with target outputs Y, A wy.....,??" " We wish to measure the dependency between Y, and Y,, One approach is Kernel Alignment
the i-th task, and Y, A [y].... y in the j-th task.
3

---

Table 1: Candidate methods to measure task relevance. Those requiring access to Dj or Dj not applicable in our setting. Those requiring proppappaaatioon are costly, K: - also rules out some candidates. In contrast, the last row highlights desired method.
task
task j
Probing data (unlabelled)
Method
back-prop
ビ 0
checkpoint
data D;
checkpoint, j data Dj
GDD [20]
x
X
Taskonomy [
Task2vec [6
V
V
7
x
x
A-Map 7
x
x
4
/
/
LEEP 8
desired method
X
X
X
?
(KA)
22
23
In the case of linear kernlls, KA can be calculated as YY, Y YJYY,) KA(Y,, Y,) Y,,Y, Y,
YJJ,, To give some intuitions of the above, let columns of Y; and Y, be one-hot. Then Y, Y; (same for YJ) is a binary matrix who assigns 1 for two inputs with the same label, and otherwise. KA measures the dependency between the two tasks by cosine similarity of their assignment patterns.
However, we do not have access to the data labels. Moreover, sometimes the task-specific heads are unavailable, so even a prediction of the labels (denoted as Y, and Y)) cannot be obtained. We tackle this difficulty by resorting to unsupervised "probing" data [xX ..... need not be the inputs that train the checkpoint(s). Regardless of the availability of (.).) and h-)-), we can extract feature representations by
x"".. ..) Note that this [x!..... x"]
F, A [x()),, (x.").... [x(")) and F, A [[ (.."),.... ,.... [x"x)" (xx"),,(((")))
For notation easiness, in the following, we may write f. fx(x") and fz = fx²) ) for any input x".
Then we use
F. F))
KA(F,, F2)
F.F.F.F (FJE,,F
as approximation KA(Y, Y,, Y,).
natural question to ask is how good this approximation is. We assure ourselves by the following reasoning. Let
)} and Y, = [8) sT ( h,(,)"),, be the predictions made for each task (Although Y, may not be obtained h; not available. Same for 5). If each checkpoint works sufficiently well for its own task, then Y, very "close" to Y., so Y, to Y, Thus KA(Y,, Y,, well approximates KA(Y,, Yz)-
Y, [V] ST) (h, h, (f,")))
It remains to validate if KA(F,, e. F2) well approximates KA(Y,, K,). The only gap between F, and Y, is a task-specific head (-), likewise, (-).) for F, and Y, We argue that many task-specific architectures are "well conditioned" (24 25 which guarantees that pairwise inner products are not distorted too much. This property further results in KA(F,, F2) R KA(Ý,, Y,). To be more concrete, consider a simple special case, where h;(f) = M,f and (f( = M, are linear mappings.
By the conditocnd assumption, have M. M, cI and M, M, dI for some c, > 0.
Then it is easy to show that KA(F,, F,) = KA(Y,, Y,).
Last but not least, we have the following guarantee positive definiteness. Proposition Let k(i.j) KA(F,,F) F,), then k > 0.
We are not the first to use kernel alignment to understand deep nets. (26 proposes to use linear and RBF Kernel alignment to measure similarity between representations. Although the methodologies appear similar, the goals are quite different.
(26) attempts to understand representation similarity, an open-ended problem. In contrast, we are explicitly interested in measuring the dependency between two tasks' output spaces. And turns out that KA between their respective features a sensible surrogate. Moreover, Proposition I enables us to construct Gaussian process on the task space.

---

221
0.6
00
detbbrrt
Cenall
dil les uncease
0.00
0.25
0.50
0.75
1.00
1.25
150
1.75
2.00
Figure 1: Left: matrix computed using 34 checkpoints from huggingface. Note that by definition, the diagonal is all-one. Right: hierarchical clustering on K. Colors denote clusters. Key observations: 1) Similar names usually share colors, e.g., bert-* checkpoints are orange and t5-* 's are green, as they respectively handle the same task. 2) A checkpoint fine-tuned on another task can become very different, e.g., roberta-large-mnli against roberta-large.
3.3 Example: Characterize Linguistic Task Space using Huggingface Checkpoints
We showcase a concrete example using 34 checkpoints from huggingface. They trained for various tasks (or their combinations). For example, the bert-* (27 checkpoints are trained for masked language modeling and next sentence prediction. 15-+'s are for text-to-text generation. Complete details of these checkpoints and their respective tasks can be found in supplementary material. Note also that multiple checkpoints may target at the same task. These checkpoints are usually named with the same prefix. For example, the 15-s's only differ by their architecture configurations (c.g.. number of layers). In analogous to sensor placement, this amounts places multiple checkpoints at the same location in the task space. It is somewhat redundant to do so. However, they grant us a chance do sanity check on the estimation of That is, the checkpoints corresponding to the same task should have large x(i,j) j) values.
"pilot deployment" that
We input training of wikitext2 as probing data, and extract the contextualized word embeddings after penultimate layer. So task specific layers like softmax classifiers are ignored. Note that these checkpoints employ different tokenizers, often resulting in the same word being split into different multiple sub-words. We adopt the strategy in (28). taking the representation of word as that of its last sub-word's Now columns F; are contextualized word representations from the i-th checkpoint. Then we can compute (A(F,, ;, F2) for each pair of checkpoints to get the matrix. To visualize the relationship between these checkpoints (thus the tasks they target), we apply hierarchical clustering the matrix and visualize a dendrogram in fig.
In fig. [ we observe that checkpoints with the same prefix are usually close, e.g.. the t5-*'s. "s. This is less surprising, since obviously they trained for the same task. A more surprising discovery is that longformer-base [29] is very similar to roberta-base (88 A closer check of their documentation informs that they both trained for masked language modeling, although longformer-base innovated a novel attention mechanism. On the other hand, adapting checkpoint to another task can result significant difference, e.g., roberta-large-min against roberta-large. Therefore, the indeed reflects our intuitions on tasks.
Finally, there are also few counter counter-intuitive cases. For example, gpt is very different from gpt2, but they are both trained for causal language modeling. We conjecture it their different tokenizers that drive (A(F,, F,) be small. A full understanding of this observation is deferred to future study.
We use scipy functions: linkage with method==ward". and dendrogram with color threshold=0.9.
5

---

3,4 Robustness of the Estimated Covariance
Estimating & relies on probing data. Therefore, it is natural ask, how robust K is against size of the probing data. In addition, we have implicitly simplified and assumed that the probing data, as well as the inaccessible training inputs for all tasks, have the same distribution on X. However, in practice, the input distri- bution may have domain shift from task to task, and from task to prob- ing data as well. Hence, we also want to check whether is stable against different genre of probing text.
wicltextZ-rrain
1.0
10-10K
0.9
10. 1000
cosivecix")
0.7
0.6
wikitext--triin
We denote the covariance estimated in the previous section as x², Now we estimate K using various subsets of the wikitext2 training set. It guaranteed that any smaller subset is included in a bigger subset. We want to see how quickly these k's approach x*, size of probing data increases. The same experiment is also repeated on another corpora, 10K sentences taken from billion-word benchmark, dubbed IB- This benchmark is created from news crawl, a different genre from wiki.
18-10K
10a
10a
10a
10a
10
10
word counts probing data
word counts probing data
(a) KL divergence
(b) correlation
Figure 2: Measure the difference between A (estimated from a subset of probing data) and x² (estimated from full probing data). x converges quickly to k=
To compare against k", we measure the KL-divergence between two Gaussian distributions with 'º and K as their respective covariances. Another simple measure is the cosine similarity between vectorized vec(k) and vec(x*). We plot <TT.(>*|[k) and cos(vec(k" ), vec(x:) against the number of words in the probing text, shown in fig. 22 (2) respectiivl. For both plots, the left most part of the x-axis stands for zero probing data. In this case, it is natural to assume x an identity matrix, ie., all checkpoints (thus the tasks they were trained for) are independent.
In fig. 2 we observe that as the size of probing data increases, KL divergences quickly converge to O, while the cosines quickly converge I. In addition, let wiiki be the covariance estimated from full wikitext2 training set, and Kjß the covariance from full 1B-10K. It turns out that K' [. [ 0.76, not large value considering the range of KL--divergences in fig. 2a On the other hand, their cosine similarity is as high as 0.98. To conclude, the estimate of covariance function reasonably robust against size and genre of probing data.
4 Exploiting the Task Space
We have characterized the task space using a zoo of checkpoints. Now we are ready to selecting checkpoints. The idea is to pick seen tasks that are the most informative about the the task space. Again, remember that this is conducted before any new task is revealed to us.
Suppose have budget of using K checkpoints. Let Z (with cardinally n) be the zoo of all checkpoints, and the set of selected checkpoints. We want to maximize the mutual information between the tasks solved by S, and the remaining of the task space. With some abuse of notation, we denote this mutual information as 7(S 2 S). Here S should be understood as the tasks corresponding to the checkpoints in S. We optimize the following constrained objective,
max SC2:8S|KK
((S:=-S)
The above a combinatorial optimization problem. Enumerating all cases is infeasible. Instead, heuristic is to use greedy search. Starting from an empty S, each time we include a checkpoint i that brings the largest gain to the objective. In specific, denote S = Z S, S = S (i) and
S, = 2 - u (i)). The gain d; can be calculated as,
5. = I(S,; - ((S
= (H(S)) H(S.)S.) - (S()) H(S())
(3)
-(,(S,) (1()) + H(S)) - - ((2) + (S)]
-H(U)()) - -(()))))

---

The first term implies that i should surprise the current S the most, thus incorporating more informa- tion. Meanwhile, the second term suggests that should be representative of the remaining of the task space, thus avoiding outliers.
Since assume Gaussian process the task space, any subset of Z is Gaussian distributed. And the two terms in Eq. (3) can be easily calculated. Specifically, let x(S, S) denote the covariance for subset S, x(2, S) the row vector of cross-covariance between {i} and 8, and k(S, i) its transpose. Then,
1 ((i,,))( x(j, (.) S.)-1x(S.;
5, H({)))) ((()))))
(4)
The greedy selection process is summarized in Algorithm | Denote S(K) as the set of selected checkpoints for K value. Algorithm guarantees that for any K1 < K2, S(K)) & S(K2).
Algorithm Maximum Mutual Information (MMI) based Selection of Checkpoints
Input: Checkpoint zoo Z with cardinality covariance matrix E | number of checkpoints to pick, K
Output: set S of selected checkpoints, where |S] K
1: S + Ø
2: for k = 1, .....: K do
S do Compute information gain 6; using Eq. 5: end for
3: for
4:
6: i* arg max
7: S S-SU
8:
end for
4.1 Discussion Quality of the Greedy Approximation
It is natural to ask how good the greedy method is. At this point, we need to review the definition of submodular functions. set function f(S) is submodular if for any sets C S' S Z and E Z S- f(S U (i)) - f(S)= 2 f(S' U (i)) f(S). It is not hard to show that the following holds. Proposition 2. I(S; Z - 8) is submodular function of S.
Another useful concept is monotonic function. A function f(S) is monotonic if for any i E 2, J(S U {i)) > S(S). For monotonic submodular functions, it is known that greedy method guaranteed to achieve objective no smaller than 1/e of the optimal one [30]. In our case, T(S; 2 S) is submodular but not monotonic. T [(2; 2 2-223 S) 0, indicating that when set sufficiently large, further enlarging S will reduce mutual information.
thus becomes less clear if the (1 - I/e) guarantee still holds. In optimal senor placement [5] the authors show that if the covariance is estimated for sufficiently many fine fine-graines grained grids in the room, then the mutual information is monotonic up to a reasonably big K. This may suggest that our has to be estimated for sufficiently many tasks. In other words, the checkpoint zoo must be rich enough. Nevertheless,
hard to come up with rigorous guarantee like [15], as assumptions made 2D physical space does not necessarily transfer to abstract task space. However, in experiments, we have found that Algorithm Saachhmmawwwwe aawways works a monotonic regime.
Experiments
In the previous sections, we have presented two key components, estimation of к and MMI based selection of checkpoints. In this section, we experiment with the two components combined. First, we apply algorithm to the x estimated in section 3.3, and show its effectiveness on multiple linguistic tasks. The baseline we compare against is random selection of checkpoints, and single commonly adopted checkpoint, e.g., bert-base-uncased. Then we extend to image classification tasks. Again we observe constant improvements over random picks, and other straightforward alternative.
7

---

0.2
0.64
0:80
0.
0.92
0.85
0.85
teet- Beri--teaecrrreeed
0..70
0.34
0.00
0.75
bert largla--ee-edd cased mansom
0.92
0.68
0.70
0.
0.90
0.86
Checkssentssoo
K-checkports used
checspaints ueed
<cheepponts uned
(a) Chunking
(b) NER
(c) POS Tagging
(d) Semantic tagging
0.96
0.7
permon
0.75
0.70
2.7
0.61
cas
0.60
Kcrecpponts used
K Kcteeppoints used
ccheoipannss sced
chcheepssnts used
(c) Event factuality
(1)
Tagging parent
in (g) Tagging grandparent in (h) Tagging great grandpar- phrase-structume tree
phrase-structure tree
ent in phrase-structure tree
Figure Performance on a suite of probing tasks: MMI achieves near optimal performance, compared with random search (34 trials at K = 1 and 20 for K > 2). In most of these tasks, the top layer of bert-base-uneased and bert-base-cased are inferior to first pick by MMI. Surprisingly, in many cases, the top layer of bert-base-cased is worse than half of the 34 checkpoints.
5.1 Linguistic Tasks Using Huggingface Checkpoints
We run algorithm on the matrix estimated in section 3.3 As K increases from 1 to 5, the S set incrementally includes roberta-base, destibbert-bas-unnasse These checkpoints solve multiple tasks: masked language modeling, text-to-text generation, next sentence prediction, and text denoising. Referring to the visualization of и matrix (left pannel of fig. D. find that the selected checkpoints all belong to the big cluster in the lower right block. This matches our intuition, as the checkpoints outside this cluster are very isolated.
15-base, bert-base-cased and bart-large.
Then, we apply the selected checkpoints to suite of probing tasks collected in 28 To be self- contained, we briefly describe the tasks and datasets.
(a) Chunking examines spans and boundaries for each word. We use CoNLL2000 shared task dataset (31)
(b) NER predicts the entity type for each word. We use CoNLL2003 shared task dataset 32
(c) POS Tagging (POS) checks if word representations can capture basic syntax. We use PTB
dataset (33].
(d) Semantic Tagging (ST) is interested in identifying the semantic role in context. The original
source of dataset can be found in [34].
(e) Event Factuality(EF) aims at determining if an event is factual. We use ItHappened-v2 dataset 33 Events given scores between -3 (non-factual) and 3 (factual). For instance, in "I decided not to go.", the word "decided" has a score of 3 whereas "go" has -3. We use Pearson correlation compare predicted scores against groundtruths.
(f-h) Syntactic constituency ancestor tagging: A set of tasks that aim at predicting the constituent label for (f) parent: (g) Grand parent (Gparent); and (h) Great Grand parent (GGparent) of word in the phrase structure tree. The structure derived from PTB.
Following 228, train softmax top of the combined word representations (&esff) for each task. The gradients are not back pack-propagated through the checkpoints. There are two reasons why we choose to do so. First, for an easier comparison with (28). Second, there cases where checkpoints are published as blackboxes, and back propagation is not allowed. One prominent example is GPT- 3 [36]. Although should respect this fact. Another design choice is that f, is taken to be the feature at top layer of the checkpoint. We do
do not have access GPT-3 in this paper, we think that our experiment design
feature intermediate layers, again respect the fact of blackboxes.
The baseline to compare against is random selection. specific, let R C be a set of randomly picked checkpoints, where [R| K. The task specific head is trained on top of &iERf, For K 1,

---

Table Compare MMI's selection (Algorithm 1 K 5) against a strong baseline [28], which uses the best layer of bert-large-cased for each task. Note that the best layer changes from task to task. It has to be found by brute-force training and evaluation for all 24 layers.
chunking
NER
POS
ST
EF
parent
Gparent
GGparent
MMI selected (K' = 5)
93.88
88.64
97.57
94.35
78.35
96.64
85.85
73.32
bert- wert-large-cased best layer [28)
93.64
84.44
96.73
93.83
76.25
96.10
82.46
67.90
R ranges over each single checkpoint in the total 34. For K 2 2, we run 20 random picks using different random seeds. Then, the metrics from these random runs can visualized by violin plots (fig. 3p. In each violin plot, the bars on the top and middle indicate best and median performances respectively. We observe an increasing trend in the metrics as K increases from MMI achieves near optimal performance compared with random search, especially when K > 1. Note also that random selection results large variance in performance. Another interesting baseline is a single checkpoint that are commonly adopted. We plot bert-base-uncasee and bert-large-cased (at is inferior to half of the 34 checkpoints. This echos some discoveries made in [28) That higher layers of bert model is less transferable to new tasks. Instead, stronger transferability is found in middle-layer features.
Importantly,
K 1). Surprisingly, for most tasks, bert-large-
Following the above discussion, we collect metrics reported in (28 for the best layer of bert-large- cased. Since layers capture different levels of information [37. the best layer is often task dependent. It has to be found by brute-force training and evaluation for each task. Tab. lists these metrics. Also reported are metries achieved by &ie at K = 5. We observe that MMI outperforms of bert-large-cased for most of the tasks, and often wins by notable margin. This is remarkable, as checkpoints are selected before seeing any new task, and fixed throughout all tasks.
best layer
5.2 Image Classification Tasks on Cifar100
We are motivated by applications computational linguistics. However, we find that the proposed framework extends to computer vision tasks as well. In this section, we simulate an example using c0f dataset.
We create n = 100 "seen" tasks. Each task is a 30-way classification problem. The 30 classes are randomly sampled from the total 100 classes. Each task has 480 (=500-20) training samples per class. There are 20 training samples held out for each class. The reason will be clear soon. A resnet-50 is trained for each of these "seen" tasks, stored as checkpoint. Then, create 20 "unseen" tasks the same way, but each of them has only 10 training samples (part of the aforementioned holdouts) per class. This small training data is manifest the transferability of the selected checkpoints. Finally, the remaining 10 holdouts per-class are used as probing data to estimate k. We augment this small set of probing images by applying rotations of +5p, 100 and +15 degrees.
1.0
80
gain over over random (%4
20
peek
70
random
40
0.8
sacc
60
80
0.6
random
0.0
50
0
10
15
20
15 checkpolnts used
10
20
40
60
60
k checkpoints used
Figure 4: Covariance x for the Figure 5: accuracy averaged Figure 6: Average (over 20 new 100 seen tasks
the 20 unseen tasks
tasks) improvements over ran- dom picks
Figure shows the estimated for the 100 seen tasks. Compared with the a in fig. less structured. That is, tasks tend to be equally distant from each other. This probably suggests that the

---

space of vision tasks fundamentally differs from that of linguistic tasks. We then apply algorithm | on the x to select up to K 20 checkpoints. For each task t, classification head is learned using its 10 x 30 training data. The performance of this task is measured by accuracy on the standard validation data (excluding classes not handled in this task), denoted as a00, As before, baseline is random selection. In addition, suppose we could peek at the first unseen task. We may identify the top-K K best checkpoints for this task, by running training and evaluation. Then we can keep using these top-K K checkpoints for all new tasks. This gives another method to compare against, dubbed "peek".
In fig. 55 we plot, E,ace,, the average validation accuracy over tasks against K. The shaded area are accuracies of runs using different random seed. Overall, as expected, the accuracies increase and saturate at larger K. To have a better view, we plot the gains of MMI and "peck" against random. That is, the MMI (and "peek") curve subtracting the average curve of the shaded area in fig. 5 The gains are shown in fig- 6 MMI steadily outperforms the random baseline. Whereas "peek" fluctuates and can be inferior than random at multiple K's. This may be caused by "overfitting" to the first task that it peeked, which prevents generalization to other tasks. In analogous to senor placement, this amounts to optimizing the placement for one location, which could harm the accuracies for other locations. In contrast, MMI does not exhibit this issue.
6 Conclusion
In this paper, we motivate a problem that addresses two recent facts: 1) huge amount of published checkpoints; 2) emerging new tasks. By drawing analogy from optimal sensor placement, we propose a checkpoint selection framework without seeing any new task. The key steps are: 1) Estimating task relevance with proprictary constraints (e.g., data, task-specife heads); 2) selection by maximizing mutual information, assuming a Gaussian Process on the tasks. Effectiveness is validated on tasks from computational linguistics as well as computer vision. However, it should reminded that there are a few assumptions (section (3.2) made for step 1). We partially validated them on text data (section 3.3 and 3.4]. A more comprehensive validation on other data (e.g.. images) is left for future work. Aside from that, it is also interesting to explore more fine-grained models of the task space (e.g.. hierarchical) and experiment with more tasks.
Acknowledgement
The authors thank Nelson Liu for helping data of linguistic experiments. The first author thanks Yifan Wei for her company and support during the pandemic lockdown.
References
[1] Baxter, "A model of inductive bias learning," artificial intelligence research, 2000.
[2] O. Vinyals, C. Blundell, T. Lillicrap, K. Kavukcuoglu, and D. Wierstra, "Matching networks for one shot learning," in Advances in Neural Information Processing Systems, 2016.
[3] C. Finn, P. Abbeel, and S. Levine, "Model-agnostic meta-learning for fast adaptation of deep networks," in International Conference on Machine Learning, 2017.
[4] J. Snell, K. Swersky, and R. S. Zemel, "Prototypical networks for few-shot learning," in International Conference on Neural Information Processing Systems, 2017.
[5] A. R. Zamir, A. Sax, W. Shen, L. J. Guibas, J. Malik, and S. Savarese, "Taskonomy: Disentan- gling task transfer learning." in IEEE conference on computer vision and pattern recognition, 2018.
[6] A. Achille, M. Lam, R. Tewari, A. Ravichandran, S. Maji, C. C. Fowlkes, S. Soatto, and P. Per- ona, "Task2vec: Task embedding for meta-learning," in IEEE/CVE International Conference on Computer Vision, 2019.
[7] Song, Y. Chen, X. Wang, C. Shen, and Song, "Deep model transferability from attribution maps," in Advances in Neural Information Processing Systems, 2020.
10

---

[8] C. V. Nguyen, T. Hassner, M. Seeger, and C. Archambeau. "Leep: A new measure to evaluate transferability of learned representations"" in International Conference on Machine Learning. 2020.
[9] T. Vu, T. Wang, T. Munkhdalai, A. Sordoni, A. Trischler, A. Pettarella-Micke S. Maji, and M. lyyer, "Exploring and predicting transferability across nip tasks," " in 2020 Conference on Empirical Methods in Natural Language Processing, 2020.
[10] Y. Li. X. Jia, R. Sang, Y. Zbu, B. Green, L. Wang, and B. Gong, "Ranking neural checkpoints." arXiv preprint arXiv:201 I. 1200, 2020.
[11] G. Jerfel, E. Grant, T. L. Griffiths, and K. Heller, "Reconciling meta-learning and continual learning with online mixtures of tasks," in Neural Information Processing Systems, 2019.
[12] Z. Wang, Z. Dai, B. Póczos, and J. Carbonell, "Characterizing and avoiding negative transfer," in IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019.
[13] N. Tripuraneni, M. I. Jordan, and C. Jin, "On the theory of transfer learning: The importance of task diversity," in Advances in Neural Information Processing Systems, 2020.
[14] S. Du, W. Hu, S. M. Kakade, J. D. Lee, and Q. Lei, "Few-shot learning via learning the representation, provably," in International Conference on Learning Representations, 2021.
[15] A. Krause, A. Singh, and C. Guestrin, "Near-optimal sensor placements in gaussian processes: Theory, efficient algorithms and empirical studies," Journal of Machine Learning Research, vol. 9, pp. 235-284, 2008.
[16] D. Opitz and R. Maclin, "Popular ensemble methods: An empirical study," Journal of Artificial Intelligence Research, vol. 11, pp. 169-198, 1999.
[17] S. Zhang, M. Liu, and J. Yan, "The diversified ensemble neural network," " in 34th Conference on Neural Information Processing Systems, 2020.
[18) H. Peng, F. Long, and C. Ding, "Feature selection based mutual information: Criteria of max-dependency, max-relevance, and min-redundancy," IEEE Transactions on pattern analysis and machine intelligence, vol. 27, no. 8, pp. 1226-123, 2005.
[19] R. Amit and R. Meir, "Meta-learning by adjusting priors based extended pac-bayes theory," in International Conference on Machine Learning, 2018.
[20] D. Alvarez-Melis and N. Fusi, "Geometric dataset distances via optimal transport," in In Conference Neural Information Processing Systems, 2020.
[21] A. T. Tran, C. V. Nguyen, and T. Hassner, "Transferability and hardness of supervised classifi- cation tasks," in International Conference on Computer Vision, 2019.
[22] N. Cristianini, J. Shawe-Taylor, A. Elisseeff, and J. Kandola, "On kernel-target alignment," in Advances in Neural Information Processing Systems, 2001.
[23] C. Cortes, M. Mohri, and A. Rostamizadeh, "Algorithms for learning kernels based on centered alignment," Journal of Machine Learning Research, 2012.
[24] E. Vorontsov, C. Trabelsi, S. Kadoury, and C. Pal, "On orthogonality and learning recurrent networks with long term dependencies," in International Conference on Machine Learning, 2017.
[25] N. Bansal, X. Chen, and Z. Wang, "Can we gain more from orthogonality regularizations in training deep cnns?," in 32nd Conference on Neural Information Processing Systems, 2018.
[26] S. Kornblith, M. Norouzi, H. Lee, and G. Hinton, "Similarity of neural network representations revisited," International Conference on Muchine Learning, 2019.
[27] Devlin, M. Chang, K. Lee, and K. Toutanova, "Bert: Pre-training of deep bidirectional
transformers for language understanding." arXiv preprint arXiv: w:Xiv:1810.04805 2018
[28] N. F. Liu, M. Gardner, Y. Belinkov, M. E. Peters, and N. A. Smith, "Linguistic knowledge and transferability of contextual representations," in Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2019.
[29] I. Beltagy, M. F. Peters, and A. Cohan, "Longformer: The long-document transformer," arXiv preprint 2055500004.051150 2020.
11

---

[30] G. Nemhauser, L. Wolsey, and M. Fisher, "An analysis of the approximations for maximizing submodular set functions," Mathematicall Programming, no. 265-294, 1978.
[31] E. F. Sang and S. Buchholz, "Introduction the conll-2000 shared task: Chunking," in CoNLL, 2000.
[32] E. F. Sang and F. D. Meulder, "Introduction to the conll-2003 shared task: Language-independee named entity recognition," in CoNLL, 2003.
[33] M. P. Marcus, M. A. Marcinkiewicz, and B. Santorini, "Building large annotated corpus of english: The penn treebank," " Computational linguistics, vol. 19, no. 2, pp. 313-330, 1993.
[34] Bjerva, B. Plank, and J. Bos, "Semantic tagging with deep residual networks," in COLING, 2016.
[35] R. Rudinger, A. S. White, and B. V. Durme, "Neural models of factuality," in NAACL, 2018.
[36] B. Brown, Mann, N. Ryder, et al., "Language models are few-shot learners," arXiv preprint arXiv:2005. 14165, 2020.
[37] J. Vig and Y. Belinkov, "Analyzing the structure of attention in a transformer language model," arXiv preprini arXiv: 1906: 04280, 2019.
12


Reviews:
[{"review_id": "zgzziX58hjV", "reviewer": "Reviewer_YBj2", "summary": "\nThe paper considers the question of how to select an ensemble of existing pretrained models to be used as feature extractors, when the task for which they are to be used is not known a priori. It seems to aim less to generalize to a given fixed target task distribution, but rather at generalizing over an expectation over target task distributions.\n\nTo achieve this, the authors propose essentially two steps:\n\nFirstly, considering the feature representation of each model from the zoo, and computing a within-model similarity matrix between the features extracted for one fixed model over a set of samples. Then to use the cosine angle between similarity matrices of different models as a similarity between models.\n\nSecondly, to select a fixed number of models (corresponding to a fixed budget) based on mutual information estimates between the set of models and the remaining models. This is intuitively appealing.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: the approach seems to be novel. The reviewer finds the amount of originality good enough for NeurIPS.\n\nQuality: \n\nthere are limitations in the experiments:\n\n1 lack of comparison against existing baselines for ensemble selection, in particular A-Map seems to be within the scope of the question.\n\nThey compare only against random selection for NLP and random and top-k performance selection for CIFAR-100. At least for CIFAR computing gradients for A-MAP as another baseline should be not an issue. \n\npossibly they could also use \n\n2a evaluation using a small number of single point tasks in 5.1. While they seem to optimize without having a fixed task in mind, they evaluate on four standard tasks, for which one may suspect that some of the NLP models were implicitly optimized for and thus possibly leaking onto the 4 tasks. \n\n2b As a consequence, when evaluating against 4 fixed tasks as in 5.1, one should also show how the best single model selected for one task using a task-dependent selection algorithm as baseline performs ... on all 4 tasks (including also the three tasks for which it was not optimized).\n\nThe reviewer disagrees on the statement that backprop is expensive when discussing A-MAP. Each model was trained using backprop, thus it seems to be feasible to use backprop also after training, at least until the feature representation if the computations below were a black box.\n\n\nSignificance: In terms of setup there is one doubt: While theoretically interesting, what is the practical applicability of optimizing for an expectation over tasks and not for a given single target task ? When this can be helpful?\n\nQuestion: 5.2 - how often this experiment has been repeated?\n\n\nClarity, the paper is written such that it can be read easily.\nHowever, a few things can be improved:\n\n3.2 the interpretation of the kernel alignment in terms of cosine-angle between similarities matrices within a model can be added. This interpretation also holds without any assumptions on the tasks\n\nSec 2: generalizing to unseen tasks seems to amount to optimizing over an expectation of a statistic over tasks. This can be worked out somewhat. \n\nIs there a connection between optimizing MI over feature similarities between the chosen set and the complement set of tasks, and under certain assumptions, between the expected prediction performance over many tasks ? It might be helpful to assume that each model and its feature representation is optimal for the task it was trained for, among the set of all models used for MI optimization. \nThe reviewer would be surprised if such a link could not be discussed (whereas the submodularity does not give a clear result, this would provide some theoretical link to the set of a priori unknown target tasks).\n\n\ntypos:\n\nThis objective is submoludar. \n\nhow many tasks has to be seen\n--> have\n\n picking checkpoints amount to identifying useful \n--> amounts\n\nIn optimal senor placement [15]\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "As stated in the previous box:\n\nWhile theoretically interesting, what is the practical applicability of optimizing for an expectation over tasks and not for a given single target task ? When this can be helpful (as opposed to optimizing for a given target task which one usually has in practice)?\nThe reviewer sees this as some level of weakness and feels that motivating this would make the paper stronger.\n\nThe discussions on submodularity yielded no conclusive result and are maybe not the most important.\n\nThe reviewer sees no issues regarding societal impact. Reusing checkpoints is positive, as it does  waste less GPU training time and energy.", "review_text": "", "extra_scores": {"ethical_concerns": "The reviewer sees no issues regarding ethics.", "needs_ethics_review": "No", "ethics_review_area": ["I don’t know"], "time_spent_reviewing": "3.5 hours"}, "extra_text": {}}, {"review_id": "RqoqSJCdepL", "reviewer": "Reviewer_zpV4", "summary": "When no task specific heads are available sample labels cannot be predicted and the dependency between two tasks and thus task transferability cannot me measured. The paper uses a Gaussian process to model the task space and comes up with a closed solution for the mutual information gain. Proposes a greedy selection algorithm for selecting checkpoints to maximize the information gain. Experiments with text and image datasets are performed to show that selecting checkpoint based on this criterion outperforms random selection.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n-\tUsing unsupervised probing to get around the problem of missing true and predicted labels and showing that kernel alignment can be achieved reasonably well at the level of feature extractors without having access to task specific heads. \n-\tGood leveraging of the analogy between sensor replacement and checkpoint selection for a more generalizable task transfer from seen tasks to unseen ones.\nWeaknesses: \n-\tImaging experiments are less compelling\n\n\nEach checkpoint is added in a greedy way to maximize the gain, where the gain is formulated as the information gained by adding the current checkpoint to the current set of checkpoints minus the information lost by removing the current checkpoint from the remaining part of the checkpoint zoo. A good checkpoint generalizable to unseen tasks is the one that is representative of the remaining part of the checkpoint zoo (the information loss minimum when removed) but at the same that surprises the current set of checkpoints the most. \n\nIn the CIFAR 100 experiments classes within seen and unseen tasks can overlap as 30 classes per task are chosen randomly. What would happen if unseen tasks do not contain any classes used in seen tasks?\n\nImaging experiments select n=100 seen tasks. How does the accuracy change with respect to n?\n\nIt is not clear why choosing too many checkpoints hurts the performance in such a significant way? Overfitting explanation is not very compelling\n\nTypos:\nFrom the abstract: “This objective is submoludar”. Is this supposed to be submodular?\nAn extra “the” in line 225\nIs missing in line 232.\nSenor placement lince 340\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"ethical_concerns": "None.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5 hours"}, "extra_text": {}}, {"review_id": "93LBb02RKai", "reviewer": "Reviewer_8waw", "summary": "provide a method of how to (adaptably) combine  existing task models for new unseen task with aiming to relieve practitioners'  selection difficulty.  Through analogizing sensor placement problem, the authors give the above method and conduct effective verification on language and CV tasks.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The idea of the submission is interesting and it uses the sensor placement as checkpoints to formulate the problem in hand for intuitive understanding. Concretely, using GP to model task space to capture the task relation and while using checkpoints and unlabeled probing data to estimate covariance of the GP involved, then using Max mutual information criterion to identify the representative checkpoints for new tasks. To my best knowledge, this is relatively new. the experiments also show effectiveness of the proposed method. My comments are\n1. a task corresponds to its data distribition, then what differences are between the distribution/task-oriented inductive learning and the proposed method? \n2. when a task is viewed a domain, again what differences are between the domain generalization and the proposed method? \n3. the checkpoint selection here is quite similar to the coreset selection, thus in some sense this point is not new.\n4. if the selected checkpoint diversity is needed, or can the diversity be useful? ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Considering both data biases and task biases, the proposed method inherits naturally these biases to some extent.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": ">4hr"}, "extra_text": {}}]

Author rebuttal:

Meta-review: PRELIMINARY: The reviewers generally agree that the paper is interesting and can be accepted for publication at NeurIPS.
Decision: Accept (Poster)