Pre-Trained Model Reusability Evaluation for Small-Data Transfer Learning
Authors: Yao-Xiang Ding
Xi-Zhu Wu
Kun Zhou
Zhi-Hua Zhou
Venue: NeurIPS 2022 Accept (2022)
Keywords: transfer learning
metric learning
meta-learning

Abstract: We study {\it model reusability evaluation} (MRE) for source pre-trained models: evaluating their transfer learning performance to new target tasks. In special, we focus on the setting under which the target training datasets are small, making it difficult to produce reliable MRE scores using them. Under this situation, we propose {\it synergistic learning} for building the task-model metric, which can be realized by collecting a set of pre-trained models and asking a group of data providers to participate. We provide theoretical guarantees to show that the learned task-model metric distances can serve as trustworthy MRE scores, and propose synergistic learning algorithms and models for general learning tasks. Experiments show that the MRE models learned by synergistic learning can generate significantly more reliable MRE scores than existing approaches for small-data transfer learning.

Pre- Pre-Trained Model Reusability Evaluation for Small-Data Transfer Learning
Yao-Xiang Ding State Key Lab for CAD&CG Zhejiang University yxdingezju.ede edu.. edu. cn
Xi-Zhu Wu
National Key Lab for Novel Software Technology
Nanjing University wuxz@lamda .nju edu. cn
Kun Zhou State Key Lab for CAD&CG
Zhi-Hua Zhou
National Key Lab for Novel Software Technology Nanjing University
Zhejiang University kunzhou@acm.c org
zhouzh@nju. werechnnju.uuu.... edu.edu...
Abstract
We study model reusability evaluation (MRE) for source pre-trained models: eval- uating their transfer learning performance new target tasks. In special, we focus on the setting under which the target training datasets are small, making it diffi- cult to produce reliable MRE scores using them. Under this situation, we propose synergistic learning for building the task-model metric, which can be realized by collecting set of pre-trained models and asking group of data providers to par- ticipate. We provide theoretical guarantees to show that the learned task-model metric distances serve as trustworthy MRE scores, and propose synergistic learning algorithms and models for general learning tasks. Experiments show that the MRE models learned by synergistic learning can generate significantly more reliable MRE scores than existing approaches for small-data transfer learning.
Introduction
Reusing pre-trained models have played essential roles in modern learning pipelines for decreasing training cost, alleviating the requirement of big datasets, and reducing the danger of catastrophic forgetting. The growing number of pre-trained models promotes the birth of large pre-trained model zoos, making closer towards the future learnware market [Khou model from these model zoos for doing model transfer, one has to do model reusability evaluation (MRE) first: evaluating the transfer learning performance of the models the target task and iden- tifying the best model. The role of MRE is crucial since no matter how good the transfer learning strategy is, incorrect MRE would still lead to the danger of negative transfer. MRE has received growing attention in recent years Achulle et all 2013, Tran el all 20T Nguven alj 2120 WY
Tan, 2022 When selecting
srall 2020, Ding and Zhou, 2020 You all 20Z11]. But most existing studies focus on large-data MRE, under which the target dataset is sufficiently large.
In this work, we focus on small-data MRE, under which the target training datasets are small. Reusing pre-trained models is essential under the small-data scenario since learning from scratch difficult. Unfortunately, large-data MRE approaches are usually invalid for small data since they usually focus more on simplicity and efficiency, but not generalization and robustness, which are essential for small-dat MRE. small-data scenario due to the fundamental burden set by the laws of statistics.
indeed challenging to obtain reliable MRE results under the
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

for Synarbbltcc
HeyuEssen
II
Mestc Learingg
Hayak
Pre Trakned blodd bets for Serget Teals
for Srripangttt looning
Veated
Aquectal si) HI
Loarneg
Sy
Data Proider
S
Date
ps
Pre-tuiiigg
Torget Taate fo- =(Sp.h)
Figure 1: The illustration of the proposed approach. The left figure illustrates the overall procedure. in which the left part illustrates the synergistic learning stage and the right part illustrate the testing MRE stage. The right figure illustrates Algorithm The arrows started from the data providers denote the queries, in which the solid purple ones are SQUERY and the dashed orange one is MQUERY.
Our solution is based on two observations. First, model reusability usually depends on the task- model relationship other than the task-independent property of any specific small sample. We could grasp this relationship by meta-learning an MRE function among tasks and models. Second, this learning process is often realizable in practice, in special for large model zoo platforms where suffi- cient pre-trained models and data providers are available. Based on these observations, we conduct the following studies in this paper:
Problem formuation. We provide the formulation of small-data MRE and synergistic learning to learn the MRE function by metric learning. Synergisite learning works for general learning scenar- ios beyond classification, on which most previous MRE approaches focus.
Theoretical analysis. We propose access risk analysis showing that the MRE model learned by syn- ergistic learning guarantees to generate reliable MRE scores. The theory not only provides guarantee even for using non-convex deep networks the MRE model, but also motivates feature pre-training in synergistic learning.
Algorithm and model design. We propose synergistic learning algorithms and MRE model struc- tures for general learning tasks, which have the auxiliary advantage of protecting data privacy. Mean- while, we propose a more elaborate MRE model strucuture for classification.
Experimental verification. Experimental results show that synergistic learning can generate signif- icantly more reliable MRE scores for small-data transfer learning than existing MRE approaches.
2 Problem Setup
In this section, we provide the formal definition of small-data MRE and synergistic learning.
2.1 Small-Data MRE
Denote by 2 = X' x y the observation space, in which X the input and output spaces respectively. A target learning task f can be represented by a probability measure" μ(æ, y) over some Z, Z. Furthermore, we assume that all μ are drawn from the task environment T, probability measure over the space of all target tasks.
In small-data MRE, target tasks f are drawn from T. For each μt, the objective is to learn a prediction rule fa such that for any observation (z, y) ~ μ4, fu()) close to y. The learner of μ will be given target training dataset Sp, set of pre-trained models H, model transfer strategy and an MRE function U,. S, {zv (IK, ) is sample drawn from μ such that the sample size K a small number. H is the pre-trained model set which each model h E Hl is drawn from the model environment M, a probability measure over the space of all pre-trained models. We assume that the models in H are sampled independently from j while different H can be given for different μ. (SS h) is transfer learning strategy which can be used to transfer pre-trained model into f using Sy- We do not restrict the choice of म, but assume that и is fixed for any target task. The MRE function vz (S,,, h) is real-valued function taking S, and h as the inputs
Wee will use je denote both task and its probability measure below when there ambiguity.

---

and outputing the transferability score. Below we use instead of Un to simplify notations. Without loss of generality, among all possible v. we define v" as the optimal MRE function such that for any n. Su h1, /2, v² (S,,, h1) < v² (S,,, h2) if and only if h has better transfer learning performance with S₁ than h22 Thus we assume that the MRE score is monotonically decreasing w.r.t transfer performance. If no special properties exist for v², estimating v" would be extremely difficult for small-data MRE. Fortunately, in general, we could assume that model reusability usually depends on the task-model relationship other than the -eseeendeenddnt property of any specific small sample, as discussed in Section M. Therefore, we assume that the ground-trute model transferability depends only the target task y but does not depend on the random draw of S from μ., i... vSp Sy μ. v² (S,) v" (S").. Thus we use the notation v" (H. h) instead of u² (Sp, h) for any Sys h. How this assumption can be relaxed is discussed in Section
2.2 Synergistic Learning
Synergistic learning is the preparation stage for learning an MRE function i that accurately approxi- mate v" before any MRE problem defined in Section starts. synergistic learning process works with the following three prerequisites:
A set of pre-trained models HsynLearn {he T drawn from M are given. Note that the models in this set could be different from models given in any future MRE problems;
A set of data providers D = [Zd]a=] participate in learning, such that each data provider represents an observation subspace Za C Z. Furthermore, UZd 2. We define the closure of any task μr ~ T as 2,, { Z Zd. s.t. 2 S 2/2
A validation MRE function Goalid is provided for synergistic learning. Uvalid is unbiased between any task and modef.. Yu,, h, Uratia((s, h)) Es (Vvatia h)] = v² (n, h).
The basic idea of synergistic learning is to establish a metric i(S h) between the target training datusets and the models so that the metric distance could be used as the MRE score. Section we will show that learning e only requires the data providers to answer two kinds of queries:
SQUERY Za, 11, M): a single data provider Za is queried to sample M observations from a given probability measure p over 2a, and return the output of th. (f2 )==1): J), a func- tion of the observations.
M
MQUERY (Za, 12 (n
): multiple data providers {Za, |=j are queried
jointly. Each data provider 2d, sample M; observations from a given probability measure Me over Zd,- Aggregating the sampled observations from all data providers, the output of (1 1==1- { {22,5 1=11..... { MMa) a function of all observations, is returned. Furthermore, 4m cannot be realized by aggregating multiple SQUERY.
The main difference between these two queries lies in the scope of involved data providers, which is important when the data privacy is sensitive. SQUERY is answered by single data provider. How this kind of queries can be answered in privacy-guarantedd ways has received many studies [Kinkevren eeall 2OIT, Konečný all, 201G). In contrast, MQUERY needs be answered by multiple data providers jointly. Protecting data privacy is much harder in this situation since their data need to be aggregated. Therefore. we set an auxiliary target of reducing the number of times to use MQUERY.
Finally, for learning é, we assume that is formed by three parts: (1) gu (Sw), the task feature backbone; (2) go (h), the combination of the model specification generator and the model feature backbone; (3) da (Crus Ca), the metric module, in which cp is the output of 9₁ (S,). Cs. is the output of gr. (h), and 9 is the learnable parameter of dø(Cu, Cx)- Ip and Oh transfer target training dataset Sy and pre- trainra model h into their representations Cp. and C₁. The metric module then calculates the metric distance between e₁ and сy as the MRE score. The details are introduced in Section !
EEen though Vontud is unbiased, it may have high variance when S₁ is small. Thus Peated can not be used directly for dmall-data MRE. Pleesse refer to Section W for details about model specifications.
3

---

3 Theoretical Analysis
In this section, we discuss the theoretical foundation of synergistic learning. Readers who are inter- ested more on algorithmic ideas can skip this section without affecting understanding significantly. In the analysis, we assume that gy and ga are fixed and focus on learning the metric module. Thus the learnable parameter for i is 0.
For metric-based synergistic learning, we assume that Nm pre-trained models { pled from M. N₂ tasks {/r ) sampled from T and Ns target training datasets {Spc.d ) Ns sampled from each task are used for training. We also assume that for any pair of his and Suykk the ground-truth MRE score e" "(4, h2) are given. Define r-- (s
Ni sam-
j-
as I[A, (1. h') < (S, h,, h') 2 0] in which A.- (1s, n², h.) = v" h) o" (ul, H), (S hp). = ((S,.h) h) S(Srr, (1Sprh'), h'), and I is the indicator function. Since o* the ground-truth MRE function, a desirable i should minimize
R(O)= = Br (
where the expectation is taken over S n. Spr H., pl' T. ~ M.. While for the convenience of optimization and analysis, we define our objective using the triplet surrogate loss.
Define r (Spu Sy² h, h/; 8) as I[A,, (4., h²) < OJ,As(S,, h, + h 7]+ in which [x]+ max (r. 0} and y > 0. When > 1, r.- upper bounds ru-, Meanwhile, r is consistent with To-
since r (,, Spr, h, hf; 6) 0 indicates that To- (S Sw 0); = 0. Therefore, define our
goal as minimizing the following expected risk
(6) [ (S,,, Sw., h 6)]
possible way to achieve this goal to minimize the following empirical risk
(0) (NrNsNm)²
Σ (S Swi, he, hys 6)]
where the summation is taken over all models, tasks and datasets used for training. We want to know whether minimizing the empirical risk Per (6) indeed leads to the minimization of the expected risk (8). Denote by θ the parameter space for 0 and igg denote the i with learnable parameter 0. We provide an access risk bound showing the effectiveness of minimizing the above empirical risk. Use GS Sye, h h') to denote [A (s.. Spr, h, h²) + 7]- Additionally, we assume that (S Spr, h h') 0 when Go(S Spr, h, h²) O. Furthermore, let
v(R(GN)) (N.Ns.,..)
v h,, h']l
in which the summation is taken over all tasks, datasets and models used for training and we have V (Sys Swr, h h²] I[A,, (H, h²) < (G Sp h, h')]. Meanwhile, we denote
V [ Es,,, ~n, (S }}. Now we ready state the following theorem whose proof is provided in the appendix (Section AT).
Theorem 3.1. Under Assumption
there exist optimal parameter 0* 2 and constant B >> O.
v0 E O, the following event
R²(()) - Rd' (00-) S 3 [V[R(G + A(N,, Nm. Ns)]
holds with high probability, in which A(N., Nm, Ns) = 0(1 1(1//N/ 1/VNm, 1/NNs
Theorem shows several interesting insights. First, V[n(G()) is the gradient norm of the metric distance gaps from the training data. The bound shows that the access risk will be small when the gradient norm tends to zero, even when the global minimum of the empirical risk has not been reached. This makes the bound informative when non-convex models, such as DNNs, are used. Second, the bound has the normal O(1/ (N.) O(1/VNm)) sample complexity dependence for both tasks and models. This is in agreement with our intuition such that lacking any of the tasks and models would lead to the failure of learning. Finally, the most interesting take-away is the sample complexity for target datasets in each task, which has a dependence of the feature variance V. If V is small, the order becomes (1/NNs) instead of O(1/ /s significant drop for the number of the training datasets. In Section B. we show that this result inspires us to include feature variance reduction in the synergistic learning process, which would significantly reduce using MQUERY.

---

Algorithm 1 Synergistic Learning
1: Given: model set HSyLearru- data providers D.
2: if Feature Pre-Training then
3: repeat
4: Liecco f- SQUERY 1......... UPDATE(09 Ed=1 decom (28)):
5: until reach end.
6: end if
7: repeat
8: Task & Model Sampling: P1, ................................ 14 T, h1, h22 bo................. hm M:
9:
Reusability Validation: 9/44, 2d, h2) + SQUERY 2d),
v" "(uihh, ) < COMBINE| (1) Za, h2s 1 |2mch
iE [t], k E [m]] d E 12
10: Targel Dataset Generation: Sp, + MQUERY 2,,,),
I,...,4;
11: UPDATE ú; (S,. Yi=1 (hx Mm k== {0" (14, h4 12.mm (i to minimize Equation D:
12:
until reach end.
Learning Method
In this section, we introduce the synergistic learning algorithm and the MRE model.
4.1 Synergistic Learning Algorithm
The realization of synergistic learning is illustrated in Algorithm M. In this section, besides the metric learning step of minimizing Equation D (Line 11), we discuss other crucial steps below in general. We consider two general synergistic learning settings. One is named isolated closure setting, which indicates that any task closure includes a single data provider. Otherwise, we name it grouped closure setting, which indicates that there exist multi-data- provider task closures.
Task and model sampling (Line 8). To generate the training data for synergistic learning, tasks and models are needed be sampled. At this stage, no interaction to the data providers is needed.
Reusability validation (Line 9). The objective for reusability validation is to acquire value of ot between any pairs of task and model that are sampled. In general, this can be done by sending models to the data providers and estimate v" using Evalid- Under the isolated closure setting, only SQUERY is needed obviously. We could ask the data providers to sufficiently large data, making Voalid an accurate estimator. But the key challenge appears under the grouped closure setting since the statisties used for calculating Ueatid should be obtained from multiple providers the same time. To achieve this by SQUERY, we require Gvalid to have a special structure. Specifically, for task fr, let S,, (Za) be sample generated from the marginal distribution of μ over Za. member of
its task closure. We require Veatia(u, h) COMBINE ( Za, h) [ in which ri Zu, h) (z,,(,))) (S, (Zz), h)]. In this construction, ( h) is a function over single data providers and COMBINE is aggregation function decided by specific MRE function, thus only SQUERY needed for calculation.
Target dataset generation (Line 10). For isolated closure setting, this step can be done by sampling from a data provider using SQUERY. However, for grouped closure setting, MQUERY necessary for this step since the target datasets must include the raw data. This issue can be relieved by the insight brought from Theorem I For each task, the number of the target data needed closely related to the data variance. A feature pre-training stage can be introduced to reduce using MQUERY.
Feature pre-training (Line 2-6). Feature pre-training the preparation stage for synergistic learn- ing. It is participated by all data providers, aiming at learning a feature extractor 9μ, which could output low-variance features for the raw data. Once 9μ is learned, would be integrated into the MRE model. As result, in synergstic learning, the data providers only need to use much fewer data for answering MQUERY. On the other hand, the feature pre-training stage itself should be done by using only SQUERY. We argue that this can be realized by using learning objectives which are de- composable w.r.t. the data providers. More specifically, a decomposable objective the summation of individual objectives, such that each individual objective Ld Liccom only takes the data from single
5

---

211
Nicaa Speci
Turofom
Setradee Fandion
Model Envemberr
futance Wine
FC
R
Product
Tesk Turoforn
Neddi Trenafern
C Cortaancs Wise
Instanc-
Wrighted Enramble
Tranform
(a) Basic Model
(b) Task TM
(c) Model TM
(d) Model Ensembler
Figure 2: The models proposed for synergistic learning. (a) illustrates the basic model. (b) and (c) illustrate the task and model transform modules (TMs) used in classification. (d) illustrates the model ensembler. The components within the dashed box of the same color are the same.
data provider Za its input. Then the purpose of the queries reduces to calculating each L which can be done by SQUERY as expected.
Due to space limitation, we provide more details on how to implement the above synergistic learning steps to different learning scenarios in Section E!
4.2
MRE Model
First, we introduce the basic structure of the MRE model defined in Section B. which is illustrated in (a) of Figure In this model, the target data first processed by the task feature backbone to generate the feature Ca. Afterwards, a task transform module is responsible to generate the task representation o₂ which aggregates all the information from the target dataset. For isolated closure setting, the task transform module can be simply an instance-wise averaging operation. In corre- spondence, before the model feature backbone, model specification generator is introduced which can be any function that transforms model into vector. The model feature ca is then transformed into the model representation oe using the model transform module. For isolated closure setting, the model transform module can usually be omitted, i.e. CA Φh-
For many learning scenarios, using more elaborate MRE model structure is useful, such classifi- cation. Below we consider small-data transfer learning for classification with supervised pre-trained models. A target task has LT classes {wi task class universe Vr. Meanwhile, for pre-trained La- classification model h, we denote by (iv ) the classes that the models are trained on. We assume that all the model classes belong to the model class universe YMs which could be different from Dr. Similar to [[ran et all 2010. Nguyen et all 2024 You all, 202II] we assume that any transfer strategy for this scenario, such as global fine-tuning and head re- re-training, can be used. While according to our experience, head re- training. in which the feature backbones of the pre-trained models are frozen and only the prediction heads are re-trained, is more proper than global fine-tuning for small-data transfer learning.
L to distinguish, in which all the classes belong to the
The overall structure of the MRE model for classification follows the basic model, but the task and model transform modules have more complex structures. For the task transform module ((b) of Figure D), take care of modeling the relationship among task classes. On one hand, instead of directly generating the feature representation of the whole task 6y. the module generates the features
(6)) for all task classes first, and then aggregate them to form Φn. Any set-to-set transform module, such as Deep Sets [Zaheer ef all, M and Transformer [[aswan ef all PUTAA all 2020), can be used. What is essential here is to generate the feature for one task class based on the context information from all other task classes.
More importantly, we propose an attention breent model transform module. This module is moti- vated by our observations during studying MRE: there is close connection between p(y (ir) and model reusability for classification, which is also pointed out by recent studies [Tran all 2010 Nguven et all 2.20 The module is illustrated in (c) of Figure First, the module takes both the model and task class features {ch He 11 [ as its inputs and transform them into the model- LLx
attention-aware task class features (o) 1/r Subsequently, {Ø) Her [e are aggregated to form the
[

---

model feature On. The core component of this module is the model ensembler ((d) of Figure DD, For a pair of model and task classes yt, Dr. the model ensembler calculates the attention weight of yt to yv. which is wh,r' with the inner-product attention. And then, the model class features are linearly combined by the attention weights to form oi, which represents the selected model class information on yt. We require Wi,r to be closely related to p(y,, 9v) to make it represent the correlation between yt and fr. Therefore, besides the metric loss defined in Equation D we introduce additional atten- tion supervision loss to supervise the learning of attention weights. For wt,"", we treat as the output probability of a binary classifier. The training labels are generated from p(u,, Dv). To be specific, we set the label to one p(ul(r)) > 71, Pin (y)) > 72 and zero otherwise, in which 71, 72 are two thresholds. And then, the attention supervision loss is calculated from the logistic loss. Finally, the overall training loss is formulated as
Lall = Lmnetric + Watt
in which Lmsttri is the small-batch version of the metric loss defined in Equation D. Lats is the attention supervision loss defined above, and Watt is the weight for Lutt.
5 Experiments
In the experiments, we first do metric visualization to verify whether synergistic learning can learn meaningful metric space. Furthermore, we conduct experiments for both in-dataset and cross-dataset MRE to verify the performance of synergistic learning. All experiments are conducted on servers with NVIDIA Tesla V100 GPUs. The code2 is implemented with TensorFlow [Abadi al 201]] (Apache 2.0 License). More details of the experimental setups discussed in Section and more experimental results are included in Section D
5.1 Metric Visualization
We adopt two ten-class datasets MNIST [LeCnn
[198]] and CIFAR-10 KKizhevsky PMR3] to
visualize the task-model metric learned by synergistic learning. For each dataset, we randomly gen- crated 20 five-class pre-trained models, as well as 20 data providers with the same class assignments. for synergistic learning, and another 20 models for testing. We treat each model as five detectors for the corresponding classes and try to use synergistic learning to obtain the detector-data metric. We consider two settings of synergistic learning. The first is learning with full data: all training sets are used for metric learning and there no feature pre-training stage. The second is learning with part data: 10% of the training set for each class is used for metric learning, and there is feature pre-training stage involved using the full training set. Figure illustrates the t-SNE Maaten and Hmiton [XDS visualizations of the learned metric, which is calculated from the testing models and instances. We can see that meaningful metric distance spaces are learned for both learning with full and part data: synergistic learning makes the instances and detectors with the same classes closer. These results provide preliminary proof' for the effectiveness of synergistic learning.
5.2 In-Dataset MRF
Next, we use the dSprites dataset |Mtthhy et all of images generated from six latent factors. We select three factors, shape, scale, orientation, to form the task domains. Another two real-valued factors, position X and Y, are used the prediction targets. There are 720 domains and 1024 instances under each domain in total. For each domain, we randomly select 800 instances as the training set which are used obtain 720 pre-trained models. The remaining 224 instances are used as the testing set. We randomly select 503 domains as the in-distribution domains and the remaining 217 domains as the out-distribution domains. Under this setting, we treat each in-distribution domain as a data provider, thus there are 503 data providers in total. For synergistic learning, we only use the training sets and models from the in-distribution domains. (po h) is set according to the mean squared error (MSE). We set training K = 10, but resting K = 1 for verifying the performance under the extremely challenging situation. During testing, we generate testing tasks using the testing sets from either the in-distribution domains or the out-distribution domains. For in-distribution domains, the models used for testing consist of only
[0 for testing in-dataset MRE. dSprites consists
*The code is available on https: thub http:///ggtttbbooooooo////a/////
7

---

"duar" "Tene"
"arglans"
7
Neeg"
(a) MNIST (full data)
(b) MNIST (part data)
(c) CIFAR-IO (full data) (d) CIFAR- 10 (part data)
Figure 3: 1-SNE visualization of the learned metric MNIST and CIFAR-10. (a) and (c) show the results learned with full data. (b) and (d) show the results learned with feature pre-tra pre-training and part data. The class names for the data clusters (dot markers) are annotated. The detection models (cross markers) have the same color to their corresponding classes (better view in color mode).
9, 0,dy
0, 2,
400
CAND MALL
HILL Bank
MAE Back
MBC Bast
MRt Pank
MAS Bank
222
bart
(a) In-distribution domain
(b) In-distribution domain 2
(c) In-distribution domain 3
1=,,, m, Syntearn
9. 0.0y 0. +3 Syskeam
0 1,0, Syntear
100
50
100 158
200
100 101
122
100
50
1200 2000 Rann
MR0 100
50 223 109
(d) Out-distributiin domain
(c) Out-distribution domain 2
Out-distribution domain
Figure 4: Performance comparison on dSprites: the MRE score rank vs. the true performance rank for MSE (left subfigure) and synergistic learning (right subfigure). (a-c) show in in-distribution results on three random domains, and (d-f) show out- out-distribution results accordingly. The subfigure titles are the indexes of the domain factors. Please refer to Section for more results.
in-distribution models. The out-distribution domains follow the similar rule. Figure shows the ranking performance comparison between using MSE directly the target datasets and synergistic learning (SynLearn). It can be observed that the synergistic learning ourperforms the naive MSE prediction significantly: under the extreme situation where testing K 1, MSE almost fails totally in generating useful rankings, while synergistic learning can still perform desirably.
5.3 Cross-Dataset MRE
Finally, we CIFAR- 100 [Knzhevsky, 200 and MinilmageNet [Vinyals al MIG for cross- dataset MRE experiments. Results on more datasets are included in Section D. We consider two settings for experiments: Reuse CIFAR- pre-trained models MinilmageNeet target tasks and the opposite. For each setting, we pre-train 200 20-class models on the source dataset, in which 100 models used for synergistic learning and the other 100 for testing. We use head re-training the transfer strategy. For the target dataset, its pre-defined training set is used for synergistic learning and its pre-defined testing set used for testing. All training and testing tasks are fixed to be five- way classification. For synergistic learning, we randomly generate 100 data providers, each of which holds the data of five target dataset classes. Full training is used for metrie pre-training and 10% of the training set is used for metric learning. For each testing task, 50 instances are sampled from each class to test accuracy. For performance evaluation, we use Kendall's r-coefficient [ to measure the rank correlation between the MRE scores and the testing accuracy. For emphasizing performance on top-performed models, we also employ the weighted version of Kendall's т- coefficient, Tar, for which exchange between elements with rank and (starting from zero) has weight 1 1/(r+1) +1/s))+1++++ 11(++1) [Vigna, 201). We compare synergistic learning (SynL-earn) with three
state-of-the-art MRE methods: NCE Tran et all
2010
LEEP Nguven et all 2020 and LogME

---

Table 1: Results for the classification experiments. For CIFAR-100 -> MinilmageNet, the input shape is 32 x 32 x 3. For MinilmageNett + CIFAR-100, the input shape is 84 x 84 x 3. T and Tw indicate the Kendall's 7-coefficient and its weighted version calculated from 100 randomly generated five-class target tasks. The results are +-5% confidence interval calculated from five random seeds. K indicates the number of training instances per class for each of the testing tasks.
Setting
Method
K
K 10
K 15
K 20
CIFAR-100
LogME
0.139L00000
1.1
0,30010,0799
0.24300:06
0420-10217
0,255 10042
04181002700
NCE
0.177=3030
0.393=0.13
0.500-000115
0.568-00097
0335=00015
06644-00766
00397=0000
0.611
Nin
LEEP
0:28:10000
0.30:20000
0.967100015
0669200004
0412:00011
0.00010051
0.4 0011
0.203200016
SynLeunn
0.499
0.714:00.03
0.882:00.018
0.735:00016
0,497
0.740:100016
0.502:0.011
0.7501..0221
MinilnngeNet
LegMF
NCE
0 0.16620044
0.173.10.009
0.312 176
0,382LL0,100 105
0:344-20.03
0.25710.0200
0.886100054
0.13-20 164
0.288:00087
0,30610,011
04782:0.163
0.556:10.011
0.345 10.011
0.310=0008
0.479200 189
0.585.10.076
CIAR 100
Synl.comm
LEEP
0.235-40.009
0.630:20.122
0.055
0,426:10.827
0.523-00:225
0642:::.022
0.550=3.066
0.428::0.019
0.362=00019
0438100976
0.619#00263
0.394=0.122
0.43::60022
0.699100007
0.631
(on MIzII The results are illustrated in Table M1 It can be observed that synergistic learning significantly outperforms other methods for small-data MRE. Note that for results in Table M we fix K 10 during synergistic learning, while we observe significantly more robust performance of SynLearn over other approaches when the testing K varies. We conduct ablation studies to verify the effectiveness of the attention supervision, the feature pre- pre-training, and the choice of the training K. The results are provided in Section D
6 Related Work
Direct approaches for MRE. The direct approaches are based on the statisties calculated on the target training datasets as the MRE scores. The direct approaches involve only simple statistics cal- culation, meanwhile auxiliary information is used, thus are usually quite simple and efficient. The
representative studies of the direct approaches are [[ran all 2010 Nguven et all, 2020]. In [Dem
aLal 2019] the negative conditional entropy (NCE) score is proposed, which is an information- theoretic quantity measuring the entropy for p(ylū). In [Nguyen et all. 2020], the log expected empirical prediction (LEEP) score is proposed, which can be regarded as an improvement of NCE, The LEEP score also closely related to p(y 0). But it uses the soft prediction probability in calcu- lation, to take the place of the hard label assignment calculation in NCE. Thus LEEP uses more information of prediction uncertainty. However, as shown in our experiments, both NCE and LEEP suffer from significant performance degeneration for small-data MRE. This is not surprising since the statisties calculated from small data usually have higher variance.
Learning approaches for MRE. The learning approaches conducts learning for MRE. Similar to the existing specification-bast based approaches, the testing-stage models and tasks are used for learning. In comparison, no testing tasks and models are necessary for synergistic learning. The representative
approaches are [Yon cf all 2021) Achill el all 2019). In [You et all 202II. the logarithm of maximum evidence (LogME) approach is proposed. Different from NCE and LEEP which aim at finding the correlation between the source model predictions and the target outputs, LogME builds the correlation between source model features and the target outputs. Training using target data is necessary for LogME. The advantage of LogME is its wide applicability for different learning problems. While its performance degenerates more significantly than NCE and LEEP under the small-data scenario. This is likely caused by the necessity of learning on the target data which would lead to stronger over- over-fitting. In [Achllle et all pmn the Model2Vec approach is proposed. Model2Vec uses metric learning to build task- metric, which is similar to synergistic learning. But Model2Vec focuses on generating the metric distances for fixed set of models, thus is learning approach for MRE. In comparison, synergistic learning generates the task-model metric for future MRE problems in which the models are unseen during synergistic learning.
Few-shot learning. In few-shot learning (FSL) [Vinyals et all BUGG ef
W01 Snell at
2017. ei all 2017) a meta-model is learned to solve future small-data learning tasks. Usually, FSL does not assume to use pre-trained models except for few cases [Chowdhury et al. [02]] For MRE, its relationship to FSL is similar to that to transfer learning: MRE can serve as a good preparation step for any FSL task that is allowed to use a pool of pre-trained models.

---

7 Limitations and Future Work
In this work, we proposed the synergistic learning approach for the small-data model reusability evaluation (MRE) problem. In this section, we discuss the limitations of synergistic learning, as well as possible future research directions.
The task-independent assumption. Synergistic learning is based on the assumption that the model transferability has little relationship with the task- inseeendeenddntt properties of the small sample. But in real problems, bad samples indeed exist whose task indeppenddnnt properties affect transfer perfor- mance. We believe that fundamental limits exist for dealing with these samples, thus the relaxation of this assumption is challenging. Note that Theorem I does not rely on this assumption, thus synergistic learning can be done even when it is not held. While lacking this special property could possibly degenerate the performance. Exploring how to tackle this challenge would be an important future research topic.
Effectiveness on more learning scenarios. Due to the resource limitation, we only conduct exper- iments for supervised pre-trained models. In recent years, reusing unsupervised pre-trained models has been increasingly popular. MRE for unsupervised pre-trained models is very meaningful topic, on which the research is still missing far we know. Note that synergistic learning does not restrict the type of pre-trained models: it can be used whenever a ground-truth MRE function is defined. Thus it can also work for unsupervised pre-trained models in principle. We plan ify this point in our future researches. Furthermore, in this paper, we only use image datasets for experiments. Applying synergistic learning application in other modalities is also interesting.
MRE in learnware. Learnware [Zhoou and Tan, pre-trained models accommodated in learnware market, which holds various machine learning models submitted by developers all over the world, to enable future user, who knows nothing about the models in the learnware market in advance, need to build their machine learning models from scratch. One of the key ingredient of learnware is model specification, which enable the models to be efficiently and adequately identified and reused, given the constraint that neither the training data of model developers nor that of users are leaked to the market. Once a set of potentially helpful learnwares have been identified for user, there be various ways to reuse them to help address users own task. This paper can be viewed as providing a new way to reuse the identified learnwares.
2022 is a growing research topic about reusing
Acknowledgements
This research is supported by NSFC (61921006, 62206245) and the Collaborative Innovation Center of Novel Software Technology and Industrialization. We would like to thank the anonymous review- for their contructive suggestions, well as Han-Jia Ye, Ming Pang, and Lu Wang for helpful discussions and their thoughtful comments.
References
Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. [TensorFlow): A system for (Large-Scale) machine learning. In USENIX symposium on operating systems design and imple- mentation (OSDI), pages 265-283, 2016.
Alessandro Achille, Michael Lam, Rahul Tewari, Avinash Ravichandran, Subhransu Maji, Char- less C Fowlkes, Stefano Soatto, and Pietro Perona. Task2vec: Task embedding for meta-learning. In Proceedings of the IEEE/CVF international conference on computer vision (ICCV), pages 6430-6439, 2019.
Nicolo Cesa- Cesa-Bianchi and Gábor Lugosi. Prediction, learning, and games. Cambridge university press, 2006.
Arkabandhu Chowdhury, Mingchao Jiang, Swarat Chaudhuri, and Chris Jermaine. Few-shot im- age classification: Just use library of pre-trained feature extractors and simple classifier. In Proceedings of the IEFF/CVF international conference on computer vision (ICCV), pages 9445- 9454, 2021.
10

---

Stéphan Clémençon, Gábor Lugosi, Nicolas Vayatis, et al. Ranking and empirical minimization of u-statistics. The annals of statistics, 36(2):844-877 2008.
Yao-Xiang Ding and Zhi-Hua Zhou. Boosting-based reliable model reuse. In Proceedings of the asian conference on machine learning (ACML), pages 145-160, 2020.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-leaming, for fast adaptation of deep networks. In Proceedings of the international conference on machine learning (ICML), pages 126-1135, 2017.
Dylan J Foster, Ayush Sekhari, and Karthik Sridharan. Uniform convergence of gradients for non- convex learning and optimization. In Advances in neural information processing systems 31 (NeurlPS), pages 8745-8756, 2018.
Xiand Gao, Xiaobo Li, and Shuzhong Zhang. Online learning with non-convex losses and non- stationary regret. In International conference on artificial intelligence and statistics (AISTATS), pages 235-243, 2018.
Gregory Griffin, Alex Holub, and Pietro Perona. Caltech-256 object category dataset. 2007.
David Ha, Andrew Dai, and Quoc V Le. Hypernetworks. In International conference on learning representations (ICLR), 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recogni- tion. In Proceedings of the IEEE conference computer vision and pattern recognition (CVPR), pages 770-778, 2016.
Jakub Konečný, H Brendan McMahan, Felix X Yu, Peter Richtárik, Ananda Theertha Suresh, and Dave Bacon. Federated learning: Strategies for improving communication efficiency. arXiv preprint ar arXiv:1610.05490 2016.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-bard based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278-2222 1998.
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, (NNov):2579-25 260992266502220 2008.
Loic Matthey, Irina Higgins, Demis Hassabis, and Alexander Lerchner. dsprites: Disentanglement testing sprites dataset, 2017.
Colin McDiarmid. On the method of bounded differences. Surveys combinatorics, 141(1):148- 188, 1989.
Yurii Nesterov and Boris T Polyak. Cubic regularization of newton method and its global perfor- mance. Mathematical programming. (08(1):177-203) 2006.
Cuong Nguyen, Tal Hassner, Matthias Seeger, and Cedric Archambeau. Leep: A new measure to evaluate transferability of learned representations. In Proceedings of the international conference on machine learning (ICML), pages 7294-7305, 2020.
Boris Oreshkin, Pau Rodríguez López, and Alexandre Lacoste. Tadam: Task dependent adaptive metric for improved few-shot learning. Advances in neural information processing systems 31 (NeurIPS), pages 719-729, 2018.
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In Advances in neural information processing systems 30 (NIPS), pages 4080-4090, 2017.
Anh T Tran, Cuong V Nguyen, and Tal Hassner. Transferability and hardness of supervised clas- sification tasks. In Proceedings of the IEEE/CVE international conference on computer vision (ICCV), pages 1395-1405, 2019.
11

---

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances neural information processing systems 30 (NIPS), pages 6000-6010, 2017.
Sebastiano Vigna. A weighted correlation index for rankings with ties. In Proceedings of the 24th international conference on World Wide Web, pages 1166-1176, 2015.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. In Advances in neural information processing systems 29 (NIPS), pages 3630-3638, 2016.
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-ucsd birds-200-2011 dataset. 2011.
Xi-Zhu Wu, Wenkai Xu, Song Liu, and Zhi-Hua Zhou. Model reuse with reduced kernel mean embedding specification. arXiv preprint arXiv:2001.07/33 007155, 2020.
Han-Jia Ye, Hexiang Hu, De-Chuan Zhan, and Fei Sha. Few-shot learning via embedding adaptation with set-to-set functions. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR), pages 1808-8811 2020.
Kaichao You, Yong Liu, Jianmin Wang, and Mingsheng Long. Logme: Practical assessment of pre- trained models for transfer learning. In Proceedings of the international conference on machine learning (ICML), pages 12133-12143, 2021.
Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Russ R Salakhutdinov, and Alexander J Smola. Deep sets. In Advances in neural information processing systems 30 (NIPS), pages 3391-3401, 2017.
Zhi-Hua Zhou and Zhi-Hao Tan. Learnware: Small models do big. arXiv preprint arXiv:2210.036677 2022.
Martin Zinkevich, Markus Weimer, Lihong Li, and Alex J Smola. Parallelized stochastic gradient descent. In Advances in neural information processing systems 23 (NIPS), pages 2595-2603, 2010.
12


Reviews:
[{"review_id": "nI21DRVKmk-", "reviewer": "Reviewer_kp6n", "summary": "This paper focuses on the reusability of pre-trained models with many source data. In particular, this paper aims to predict the transfer learning performance in advance when the target dataset is small. To evaluate model reusability, this paper proposes synergistic learning (metric distance converted to MRE score) as one of the metric learning methods. The proposed method experimentally shows that it is a more useful method to predict the transfer learning performance than baseline methods.", "questions": "1. This reviewer intuitively believes that the proposed method is to perform metric learning, which consumes quite a computational cost. Isn't it more computationally expensive than the transfer learning method (fine-tuning the head of networks) performed in this paper?\n\n2. The authors assume that the GT of model transferability depends only on the target task $u$ but does not depend on the random draw of $S_u$ from $u$ but this is not proven theoretically. This reviewer thinks this assumption is hard to realize typically.\n\nRegarding the above questions, please correct me if I am wrong or misunderstood.", "limitations": "This reviewer thinks that the authors have properly stated the limitation of this paper.", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength\n\nThe proposed method can robustly calculate MRE scores even in situations where there is little learning data. The proposed method is also a generic method applicable to most problems.\n\nClarity\n\nFrom the reader's point of view, the notation of this paper is somewhat confusing because it is different from that of other general transfer learning papers. It is also unclear what circumstances SQUERY and MQUERY assumed. Since there is a lot of space left in Figure 1, it is recommended to explain the situation of SQUERY and MQUERY in pictures.\nFinally, some grammatical errors are seen. Therefore, this reviewer recommends that you organize your thesis well.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "IxX0wbW6xGZ", "reviewer": "Reviewer_wt4R", "summary": "This work investigates the estimation of model reusability when the size of target data is small. Specifically, features are extracted for tasks and pre-trained models, respectively. Then, the metric distance between them is optimized as MRE score. The theoretical analysis and the empirical study confirm the effectiveness of the proposed method. ", "questions": "My major concern is about the application for unsupervised pre-trained models, which demonstrate better transfer performance than supervised counterparts.", "limitations": "Yes.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strong.  \n1.\tWith the development of pre-training, evaluating the reusability of existing pre-trained models appropriately is important for applications.  \n2.\tThe framework is elaborated with the example of classification.  \n3.\tThe empirical study shows that the proposed method can learn better MRE score than existing methods.  \n\nWeak.  \n1.\tSome notations are not clear. For example, the loss in Line 108 contains a term of $v(\\mu’, h’)$ without any explanation about how to obtain it. Besides, it is better to elaborate the feature generator for models, i.e., $c_h$ and the validation MRE function $v_{valid}$ with examples to help illustrate.  \n2.\tFor classification, the proposed method requires the label space for pre-trained models which is unavailable for models learned without labels. It may limit the application of the proposed method, which is only applicable for supervised pre-trained models.  \n3.\tThe current experiments only contain models pre-trained on target data. It is better to include existing pre-trained models as in LogME to demonstrate the effectiveness for a diverse model zoo.  \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "IljEZtw48TH", "reviewer": "Reviewer_xY6u", "summary": "This paper proposes synergistic learning, a new metric for Model Reusability Evaluation (MRE) of pre-trained models to downstream tasks with small data. This task-model metric is learned using a set of pre-trained models and one or more data providers. Given a validation MRE function, synergistic learning establishes a metric between data providers and pre-trained models so that this metric distance can be used as an MRE score. The authors propose four different types of synergistic models: 1) basic model, 2) task transform model, 3) model transform model, 4) ensembler model. Experiments are run on benchmark MNIST, CIFAR-10, dSprites, CIFAR-100 and MiniImagenet datasets, and show that the proposed method is competitive as a MRE metric compared to SOTA methods.", "questions": "Q1: In the experiments, can you clarify how many data providers are you actually using? \n\nQ2: How do the experimental results of the proposed MRE model relate to the bounds presented in Theorem 3.1, especially in terms of the number of the training datasets required when the feature variance is small?\n", "limitations": "Limitations are clearly stated.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The paper tackles the important problem of learning to select pre-trained models for downstream tasks with small training data. \n- The paper is technically sound and the claims are accompanied by theoretical and experimental analyses.\n- The experiments show that the proposed method is successful as a MRE metric, compared to counterpars.\n\nWeaknesses:\n- The main weakness that I see is related to clarity and organization of the paper, especially in terms of definitions and assumptions. For example, it seems that feature extractors are an important part of the proposed method, however this assumption is not clearly stated until later in the paper. Also, important assumptions for the small data transfer scenario such as re-training of heads only (lines 187-195) are provided later in the paper. In general, the presence of definitions and assumptions in several parts of the paper make it difficult to read, and a big effort is required for understanding the ideas, assumptions and limitations of the proposed approach. I would suggest the authors to reorganize the paper in such a way that assumptions and definitions of the general approach are provided first, then an explanation of the proposed method (the theory and the algorithm). \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept