Towards Enabling Meta-Learning from Target Models
Authors: Su Lu
Han-Jia Ye
Le Gan
De-Chuan Zhan
Venue: NeurIPS 2021 Poster (2021)
Keywords: Meta-Learning
Few-Shot Learning
Model Matching
Knowledge Distillation

Abstract: Meta-learning can extract an inductive bias from previous learning experience and assist the training of new tasks. It is often realized through optimizing a meta-model with the evaluation loss of task-specific solvers. Most existing algorithms sample non-overlapping $\mathit{support}$ sets and $\mathit{query}$ sets to train and evaluate the solvers respectively due to simplicity ($\mathcal{S}$/$\mathcal{Q}$ protocol). Different from $\mathcal{S}$/$\mathcal{Q}$ protocol, we can also evaluate a task-specific solver by comparing it to a target model $\mathcal{T}$, which is the optimal model for this task or a model that behaves well enough on this task ($\mathcal{S}$/$\mathcal{T}$ protocol). Although being short of research, $\mathcal{S}$/$\mathcal{T}$ protocol has unique advantages such as offering more informative supervision, but it is computationally expensive. This paper looks into this special evaluation method and takes a step towards putting it into practice. We find that with a small ratio of tasks armed with target models, classic meta-learning algorithms can be improved a lot without consuming many resources. We empirically verify the effectiveness of $\mathcal{S}$/$\mathcal{T}$ protocol in a typical application of meta-learning, $\mathit{i.e.}$, few-shot learning. In detail, after constructing target models by fine-tuning the pre-trained network on those hard tasks, we match the task-specific solvers and target models via knowledge distillation.

Towards Enabling Meta-Learning from Target Models*
Su Lu
Han-Jia Ye
Le Gan De-Chuan Zhan
State Key Laboratory for Novel Software Technology Nanjing University, Nanjing, 210023, China {lus, (lus.yehj)0u weweeeellalleeee amda. nju. edu. cn, {ganl, (haadde@jjuu edu. cn
Abstract
Meta-learning can extract an inductive bias from previous learning experience and assist the training of new tasks. often realized through optimizing a meta- model with the evaluation loss of task-specific solvers. Most existing algorithms sample non-overlapping support sets and query sets to train and evaluate the solvers respectively due to simplicity (S/Q protocol). Different from S/Q protocol, we can also evaluate task-specific solver by comparing to a target model T. which is the optimal model for this task model that behaves well enough this task (S/T protocol). Although being short of research, S/T protocol has unique advantages such as offering more informative supervision, but it is computationally expensive. This paper looks into this special evaluation method and takes a step towards putting it into practice. We find that with small ratio of tasks armed with target models, classic meta-learning algorithms can be improved a lot without consuming many resources. We empirically verify the effectiveness of S/T protocol in a typical application of meta-learning, i.e., few-shot learning. In detail, after constructing target models by fine-tuning the pre-trained network those hard tasks, match the susc-fpe solvers and target models via knowledge distillation.
Introduction
Meta-leaming means improving performance measures family of tasks by their training experience [22] It has been researched in various fields such as image classification 11 16] and reinforcement learning 66 [4]. By reusing transferable knoal-nnoweedge extracted from previous tasks, we can learn new tasks with a higher efficiency or shortage of data.
A typical meta-learning algorithm can be decomposed into two iterative phases. In the first phase, we train a solver of a task on its training set with assistance of meta-model. In the second phase, we optimize the solver's 's performance to update meta-model. One key factor in this procedure is the way to evaluate the solver because the evaluation result acts as the supervision signal for meta- model. directly the solver's training loss its performance metric, and optimize this metric over a distribution tasks. Obviously, inner-task over-fitting may happen during the training of task task-specific solvers, resulting in inaccurate supervision signal for the meta-model. This drawback is even more amplified in applications where the training set each task limited such few-shot learning and noisy learning.
Early meta-learning algorithms [19 23
Intuitively, assessment of solvers should be independent of their training sets. This principle draws forth two important meta-learning algorithms in 2016 24 28), which respectively export solver evaluation from the perspective of "data" and "model". In this paper, we call these two methodologies S/Q protocol and S/T protocol. In S/Q protocol, S means support set and Q means query set. They
DDe- Che-Chuan Zhan is the corresponding author.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

contain ovn-overia instances sampled from a same distribution. By training the solver on S and evaluating it on 2, we are able to obtain an approximate generalization eтror of the solver and eventually provide the meta-model with reliable supervision signal. Another choice is to compare the trained solver with an ideal target model 'T. Assuming that 'T works well on a task, we can minimize the discrepancy between the trained solver and 7 to pull the solver closer to T. Here TT can be Bayesian optimal solution to a task or a model trained on a sufficiently informative dataset. Figure D gives an illustration of both S/Q protocol and S/T protocol.
Although appeared in the same year, S/Q pro- tocol is more widely accepted by meta-learning society 4 g 13 [0] while the research about how to leverage target models remains imma- ture. The main reason is the simplicity of SIQ and the computational hardness of S/T. How- ever, S/T protocol has some unique advantages. Firstly, it does not depend on possibly biased and noisy query sets. Secondly, by viewing sup- port sets and their corresponding target mod- els as (feature, label) samples, meta-learning is reduced supervised learning and can transfer insights from supervised learning to im- prove meta-learning (2] Thirdly, can treat the target model as a teacher and incorporate a teacher-student framework like knowledge dis- tillation [7] and curriculum learning [ meta- learning. Thus, it is necessary and meaningful to study S/T protocol in meta-learning.
Support Set
Salver
Updlte Meta ModH
Query Loss
Query Set.
(a) S/Q protocol for meta- meta-learning.
Suppport Set
Solver
Uppiate Meda Modi
Modi
Discrepancy
Targo Model
Derision
Boundary
(b) S/T protocol for meta-learning.
This paper looks into S/T protocol and takes step towards enabling meta-learning from target models. We mainly answer two questions: (1) we already have access to target models, how to learn from them? What are the benefits of learning from them? (2) In a real-world applica- tion, how obtain target models efficiently and make S/T protocol computationally tractable? For the first question, we propose match the task-specific solver to the target model in output space. Learning from target models brings us more robust solvers. For the second question, we focus on a typical application scenario of meeannca i.e., few-shot learning. We construct target models by fine-tuning the globally pre-trained network on those hard tasks to maintain efficiency.
Figure 1: Comparison between S/Q protocol and S/T protocol. (a) In S/Q protocol, each task contains sup- port set S and a query set Q. We train a solver on S and evaluate it on Q. and query loss is used to optimize meta- model. (b) In S/T protocol, each task contains support set and target model TT. After training solver on S, we directly minimize the discrepancy between it and T.
2 Related Work
Meta-Learning. Meta-learning aims extracting task-level experience (so-called meta-knowledge) from seen tasks, while generalizing the learned knowledee to unseen tasks efficiently. Re- searchers have studied several kinds of meta-knowledge like model initialization
25 embedding
network [2]] 12 9 20 4. external memory [19 5). optimization strategy [[5 18 and data aug- Despite their diversity in meta- knowledge most existing models are trained under S/Q protocol, and rely randomly sampled and possibly biased query set. Actually, most and both S/Q protocol and S/T protocol can applied to them. Thus, our work on S/T is general, and it has wide application field.
mentation strategy
algorithms are
Learning from Target Models. The idea of learning from target models in meta-learning first proposed by (28). In (28. the authors constructed model regression network that explicitly regresses between snalle classifiers and target models in parameter space. Here both solvers and target models are limited to low-dimensional linear classifier, making it feasible to regress between them. From our perspective, matching two models' parameters is not practical when the dimension of parameters is too high. Thus, we match two models in output space in this paper. Similarly, there

---

other papers focusing on meta- meta-learnigg from target models [27 311. The most similar work to us [3], which constructs target models with abundant instances and matches task-specific solvers and target models. However, they all assume that every single task has target model, increasing both space and time complexity of S/T protocol. To summarize, we claim that one key point in putting S/T protocol into practice is reducing the requirement for target models. In this paper, we focus on those hard tasks, and find that by learning from a small ratio of informative target models, classic meta-learning algorithms can be improved.
3 Preliminary
Meta-learning extracts high-level knowledge by meta-model from meta-training tasks sampled from a task distribution P(T) and reuses the learned meta-model on new tasks belonging to the same distribution. Each task r has a task-specific support set S = {x,, y:)) +x)x,::)) solver g X - Y parameterized by 7/-- Without loss of generality. meta-model can be defined as f + G parameterized by o that receives a support set as input and outputs solver. Here & is the space support sets and C is the space of solvers. In other words, encodes the training process of on S under the supervision of meta-knowledde Oy. Taking two well-known meta-learning algorithms, as examples, we have the following concrete forms of f:
and we can train on S
MAML 3 and ProtoNet
MAML meta-leams model initialization 0, and fine-tunes it on each S with one gradient descent step to obtain a sask-fpecific solver can be written as Equ (T). is step size and 8: V Y + R+ is some loss function.
f(S: =9 - Σ (X.Y))ES
(g(xx)
7=
ProtoNet meta-learns an embedding function de, parameterized by O, and generates a lazy solver which classifies instance the category of its nearest class center. Here g is implicitly parameterized by both 0, and embedded support instances.
f(S:0,) = g (-; ( (x;) E ())
S/Q Protocol. How to evaluate the solver trained on S? The answer to this question differs conventional SIQ protocol [24] from S/T protocol. In S/Q protocol, we sample another query
Q = {, } apart from for each task. Instances in and Q are i.i.d. distributed and have a same label set, and we evaluate g by its loss on Q. Since S and Q contain non-overlapping instances, loss Q is more reliable supervision signal. SIQ protocol can be formulated as Equ (3). Here Dir is the meta-training training set and we can sample meta-tret veta-training tasks T"r from it.
min Σ
Σ (x).,))
(3)
OFIED UHE
SIT Protocol. Any sampled query set Q can be biased and noisy, which may cause an inaccurate evaluation of the solver. An alternative directly matching the task-specific solver g= f(S) and target model T that works well on the corresponding task. By computing the distance from the solver to target model, we obtain a more robust training signal to update meta-model. By replacing the solver evaluation part in Equ (3), we have the following S/T protocol Equ 4). Here L: G x G + R. some loss function to measure the discrepancy between g and target model T.
min
Σ
C(f(S",TT)
(4)
("=
4 Effect of Target Model
We have introduced some basic concepts in meta-learning, and formulate S/Q protocol and S/T protocol in Section 3 In this section, assume that target models are available. and study how
3

---

to utilize them to assist meta-learning Firstly, we propose a model matching framework based on output comparison. Secondly, we verify the effectiveness of our proposal in synthetic experiment. Moreover, we try to decrease the ratio of tasks that have target models, and show that it is possible to reduce the resource consumption of S/T protocol.
4.1 Model Matching
In SIT protocol, one key point is how to match the solver g and its target model T. In other words, we need to specify the concrete formulation of C(g, T). Gener- ally, methods to match to T can be classi- fied into two categories. Firstly, we can di- rectly match two models' parameters or use another model regress between two mod- els' parameters [28]. For example, let Ys and Yr be the parameters of g and T, we can set L(g, T) = [xx,y))ss (()(x.), + Al1y, 7r 1l. Here À is a balancing hyper- parameter. This method may work well for low-dimensional parameters, but not suitable for complex models like deep neural net- works. A better alternative is to match two models in their output space, i.e., L(g, 7) E(x,))ES [1 ())()( + (D)((x)),xxx))) Here D(:..) is a function that measures the discrepancy between T (x;) and g(x:). If we instantiate D(-,.) as KL divergence (L(-II-) for classification problem, the aforementioned loss function is equivalent that of knowledge distillation. Figure 2 is an illustration of approaches to matching a solver to a target model.
Parameter Space
Output Space
Partncti
Krunledy
Requesuu
Tisillatan
Figure 2: Two approaches to matching 9 and target
model T. Left: matching them in parameter space. Right: matching them output space.
4.2 Empirical Study: Sinusoid Regression
In this part, assume that target models are available, and evaluate the effectiveness of our proposed matching approach. We construct a synthetic regression problem, and try to answer the following questions: (1) Can S/T protocol outperform S/Q protocol when target models are available? (2) Is possible to improve meta-learning with only a few target models?
Setting. Consider regression tasks T(x)) a sin c) where b, and c are uniformly sampled from [0.1,5], 5). [0.5, 2). and [0.5, 2n) respectively. For each task, we generate 10 support instances by uniformly sampling in range [-5, 5]. For S/Q protocol,
additionally sample 30 query instances for each task. We then set = T(x) + e where e ~ (0, 0.5) is a Gaussian noise. 10000 tasks are used for both meta-training and meta-testing. 500 tasks are used for meta-validation.
Algorithms. We consider two classic meta-learning algorithms, MAML 3 and ProtoNet [21. MAML can be directly applied to regression task, but ProtoNet is originally designed for clas- sification. this part, we modify ProtoNet slightly to fit regression problem. In detail, we try to meta-learn an embedding function & IR + R1000 with assistance of which the similarity- based regression model g(- (((x;))(Z,,, 34) E S}) works well across all tasks. Here for any expl (xp(( d(x)))
instance (I. y). ú = g(x) = Es...ie WiYh and w; = E Exp( {(x)) A same embed- ding network is used in two algorithms. We train MAML and ProtoNet under S/Q protocol and S/T protocol. When using S/Q protocol, we minimize MSE loss on 30 query instances to optimize d. For S/T protocol,
match the solver and the target model in output space, and set ((T(z)), g(x))) = [TT();; - g(x;)
Thus, the loss function under S/T protocol is C(g, 7) = E(arw)ess [[1 - (1]-)]l, (x)))( - 1 + ) - [13)) À hyper-parameter. More implementation details can be found in the supplementary material.
Superiority of SIT Protocol. Table shows the MSE of four models on meta-testing tasks. We can see that models trained under S/T protocol consistently outperform models trained under S/Q protocol. In Figure 3 we visualize randomly chosen mela-testing task. Different colors are used for different meta-learning algorithms, and dotted lines and dashed lines are used for S/Q protocol and SIT protocol respectively. We can that models trained under S/T protocol fit the target sinusoid

---

curve better. It is meaningful to discuss why target models improve meta-learning algorithms. In
this empirical study, distillation from target models can be interpreted as label denoising. detail,
we can prove] that meta-learning loss under SIT protocol (1 ())gg())) yll2 + X)] -T(x) 1|
an upper bound of (1)())) (y Xe )e). which is the standard MSE loss between the output of
solver 9 and cleaner label y de (raw label y equals to TTx)) + c). Therefore, the larger À is, the
cleaner training labels are. Table 2 is an ablation study on hyper-parameter X. As expected, both
algorithms trained under S/T achieve better performance with larger X. These results demonstrate the superiority of S/T protocol when target models are available.
Reducing the Requirement for Target Mod- els. Despite the satisfying results in the empir- ical study, it does not mean that we can apply SIT protocol in real-world applications and nec- essarily obtain higher performance. Up till now, we have assumed that every single meta-training task has a target model. This assumption is too strong from two aspects. Firstly, we usually don't I have ready-made target models, and con- structing target models is not trivial. Secondly, even though we have designed a method to con- struct target models, it will cost too much time to construct target model for every single meta- training task. Existing researches that focus on meta-learning from target models often bypass this dilemma by restricting the complexity of solvers and target models (28) building one global target model. In this paper, we study a more general methodology reducing the num- ber of required target models. If we randomly choose small subset of meta- meta-training tasks, and only provide these tasks' target models, how will the model performance change? To answer the question, we first randomly sample subsets of tasks that have target models, and abandon target models for other tasks. In this case, the meta-learning loss of tasks without target mod- els degenerates to S/Q loss. By ranging the size of this subset, we can plot the performance curve of MAML and ProtoNet in Figure Then, we heuristically select the hardest tasks from all meta-traine meta-training tasks and only deploy target mod- els for these tasks. In this regression problem, c), and
Table 1: Average test MSE of two meta-learning algo- rithms. Models trained under S/T protocol outperform those trained under S/Q protocol.
MAML S/Q
ProtoNet SIQ
Method
SIT
SIT
MSE D
4.933
3.621
4.706
3.332
Table 2: Average test MSE of two meta-learning algo- rithms with different À values. Larger À offers cleaner labels, resulting better models.
A
0.8
0.5
0.2
MSE: MAML(S/T)
3.220
3,419
3,621
3.833
MSE: ProtoNet(S/TT
3.137
3.304
3.332
3.550
cargor MANLINO MAMLAS?) PrateNegSS
inatonce
Figure 3: Visualization of a randomly sampled meta- testing task. Dotted lines are used for S/Q protocol while dashed lines are used for S/T protocol. We can see that models trained under S/T protocol can fit the target sinusoid curve better.
MrMLL/random
ProtoNctignnddee
sinusoid curve is defined as a sin larger a or smaller b induce steeper curves. We simply consider these steep curves as hard tasks, and sort the hardness of all meta-training tasks according to b. Another two performance curves using this heuristic are also plotted in Figure
0
0
3.5
300
%00
1800
2000
5008
166168
of tanks with targut models
4 We can see that when using this naive heuristic, we can obtain an evident performance gain with only 500(5%) target models. This finding inspires us to analyse the hardness of tasks in meta-learning, and confirms the possi- bility of learning from a few target models.
Figure 4: Change of MSE loss over number of meta- training tasks that have target models. By selecting hard tasks heuristically, we are able to obtain an evident performance gain with small number of target models.
We leave the proof to the supplementary material.
5

---

5 Application Case: Few-Shot Learning
Few-shot learning is typical application of meta-learming. It aims at recognizing new categories with only few labelled instances. In few-shot learning, we have two datasets that contain non-overlapping classes, f.e., D and Dis Du is composed of seen classes while D contains unseen classes. We can sample N-way K-s K-shof) meta-training tasks from D to train the meta-model, and expect that the trained meta-model will also work well on D
5.1 Task Hardness
Following the idea of constructing target models for hard tasks, we firstly investigate which tasks hard in few-shot learning. We consider the relationship between classes as key factor that determines the hardness of classification task. Assuming that there are C classes in Dir, we first compute a similarity matrix F E xC" whose element F uv equals the similarity be- tween the w-th class centre and the e-th class center. In few-shot learning, pre-tran ure-training the backbone network on DD' has become a com- mon practice [26 30 and we can compute these class centres based on the pre-trained model oftt as Equ (5) and Equ (6). Equ (5). K2 is the number of instances of class u in 2 and with
Table 3: Average accuracy on auxiliary dataset D Fine-tuned target models outperform a single pre-trained target model on randomly sampled tasks.
Target Model
pre-train
fine-tune
Accuracy on D
98.24
99.37
0.2
98
ratio of task
0.1
96
a bit abuse notation, we use y; = u to select instances belonging to the u-th class.
pre-ttaimed :
fine-tuned
0.0
14
15
16
17
18
Cu
Σ
TPI (x:),
u E [C"]
hardness
K
(x (.
Figure 5: Grouping of 1000 tasks according to their hardness. Both 6e and de achieve lower accuracy on harder tasks, verifying the reasonability our proposed hardness metric. Fine-tuned target models obtain a re- markable performance gain hard tasks.
(5)
Cu Cy
u v E [C"]
(6)
uv |c,,ll ||c,ll'
With similarity matrix F, we can take out the sub-similarity matrix of task by slicing the rows and columns corresponding to classes contained т. The hardness of task T is defined as the sum of its sub- sub-similarity matrix. The more similar classes in 7 are, the more difficult to differ them from each other. The hardness of every meta-training task can be evaluated with similarity matrix F, and we compute only once.
5.2 Target Model Construction
As mentioned in last part, pre-training the backbone network on seen classes is a widely used technology in few-shot learning. The pre-trained network di is optimized using cross-entropy loss on the whole meta-truining set, and can classify all classes in D Since there are C classes in Du, the output of d" is C -dieensional vector. Given a specific N-way task T. naive approach to obtain a target model is taking out N corresponding dimensions of the pre-trained model's output. However, using a single pre-trained model assist the meta-learning of all tasks is sub-optimal. We claim that fine-tume tine-tuning the pre-trained model on the subset of Drr that contains classes in can give us a better target model for T.
Evaluation on Auxiliary Dataset. To verify the reasonability of the heuristic task hardness metric and the effectiveness of the fine-tuning approach, we need another auxiliary dataset D 2= contains same classes as D and on Ded, We conduct an experiment on minilmageNet (44 to check whether fine-tuned target models
evaluate the accuracy of constructed target models (trained on D
An N-way K shot task is classification task with N classes and K instances in each class.

---

d frer clam center
trus dlas curter Instance 8
truc clais conter Instance la dianc conter
Insiaace
03
(a) Bayesian optimal classifier.
(b) ProtoNct trained under S/Q.
(c) ProtoNet trained under S/T.
Figure 6: Decision boundaries of three different models in raw 2-d space. Different point colors represent different classes, and different background colors represent different classification regions. 5% tasks have target models. A 5-way 10-shot task visualized. (a) Bayesian optimal model constructed with parameters {un)) and {E, ) Although having the lowest misclassification error in expectation, it is not robust to noises since the decision boundary very steep. (b) ProtoNet trained under S/Q protocol. Decision boundary still not regular. (c) ProtoNet trained under S/T protocol. Decision boundary is very smooth due to the regularization effect of knowledge distillation. Models trained under S/T protocol are more robust to noisy of biased instances.
better than pre-trained target models. Firstly, we pre-train ResNet-12 with linear layer on the meta-training split of minilmageNet. After that, we randomly sample 1000 5-way tasks from Du, and fine-tune the pre-trained backbone obtain 1000 target models. For each task т, we take out all instances in D. that belong to classes in T to evaluate 6f and df. Table shows the average accuracy on auxiliary dataset Dt We can see that fine-tuned target models achieve higher accuracy because they are task-specific, but the performance gain is marginal. The pre-trained model already works well enough on these seen classes. This means not cost-effective to fine-tune a target model for every single meaa-trainin task. In Figure [] we divide these 1000 tasks into 10 bins according to their hardness. In each bin, we compute the average accuracy of nd and Φ". Now we can draw two conclusions. Firstly, both diPl and @ achieve lower accuracy on harder tasks, and this verifies the reasonability of our proposed hardness metric. Secondly, the performance gain of fine-tuned target models are most remarkable on hard tasks, and this means fine-tuning target models for hard tasks can simultaneously save computing resources and improve the pre-trained target model.
Now we can summarize our S/T protocol for few-shot learning. Firstly, we pre-train del on D and then sample meta-training tasks from seen classes. Secondly, we sort the meta-training tasks according to their hardness, and fine-tune the pre-trained network to obtain local target models for small ratio of hardest tasks. Denote by DIr the set of tasks that have target models and DI the set of tasks that do not have target models. For tasks in D we train task-specific solvers on their support sets, and then evaluate these solvers under S/T protocol. For tasks in Dj, we simply use S/Q to compute query loss, as shown in Equ (7.
min Σ
Σ
[(1 X)(((((T)())))))) + (KL(("(())I(((())())))
(E (x.Y)JES)
Σ
Σ (()()))))))))))))
(8% OD E,
Different from S/Q protocol, SIT protocol does not rely on randomly sampled query sets, and target models usually offer more information than instances. Distillation term plays the role of regularization, enforcing the solvers for hard tasks to be smooth (see next subsection). Although the idea of S/T protocol is proposed 2016, it is not widely used due to its computational intractability. However, in this paper we propose an efficient method to construct target models, and only deploy target models for small ratio of hard tasks. This opens the door for future research of S/T protocol, and uncarth the potential of existing meta-learning algorithms.
7

---

5.3 Empirical Study: Gaussian Classification
In this part, we test our proposed method on synthetic classification dataset. The purposes of this empirical study are two-fold: (1) check whether S/T protocol with only few target models can improve classic meta- veta-learning algorithm; (2) study why distillation from target models can help.
Setting. In this experiment, randomly generate 100 2-d Gaussian distributions. There 64 classes for meta-training trata-traiiig..... 16 classes for meta-validation, and 20 classes for meta-testing We sample 100 instances for each class to form the whole dataset. For each class, sample its mean vector ~ [100, 10) E R² and covariance matrix Σ = Σ' E' where E' ~ 42x2 ( 2, 2] E R2xx Here U means uniform distribution. We then sample 10000 5-way 10-shot tasks for both meta-training and meta-testing. After every 500 episodes, we sample 500 tasks for metaavalidation.
Algorithms. In this part, we use ProtoNet 21 trained under S/Q protocol as our baseline. It
meta-learns a shared embedding function Φ R² - 10:00 across tasks, and classifies an instance into the category of its nearest support class center. To be specific, let Cn K (xx.y.)
o(x,) be the support class center of the n-th clas then for instance x, the model will predict its N-
dimensional label ý as In = Σ expl (xx) , E [M]. As comparison, we also train a ProtoNet under S/T protocol. Here the target model is constructed by fine-tuning the pre-trained global embedding network on specific tasks. To check whether S/T protocol can work with only a few target models, we set the ratio the tasks that have target models to 5% and 10%. As presented in last part, we sort all weea--rainnng tasks according to their hardness and fine-tune the pre-trained backbone on those hardest tasks. Refer to supplementary material for more details.
(ep
Results and Discussions. Firstly, we report the meta-testing accuracy of different models in Table Methods under S/T protocol outperform vanilla ProtoNet by large mar- gin. Even with only 5% target models, we can obtain remarkable accuracy improve- ment. Then, we study why SIT protocol can help ProtoNet learn better. In Figure 6 we visualize a 5-way 10-shot task and the decision regions of 3 models in raw 2-d space. Figure 6aa is the Bayesian optimal classifier T. ie.., for an instance x, p(y n|x) & |7,1 exp (- (o) (x-
Table 4: Average accuracy on meta-testing set. Models trained under S/T protocol outperform those trained un- der S/Q protocol even though there are only few target models. 6e means directly using the pre-trained network to solve thet-ttssing tasks without meta- training phase. The second row and the third row represent biased sam- pling, where only sample instances that have low likelihoods. When instances are biased, the superiority of S/T protocal is more evident because target models make task task-specific solvers more robust.
Protocol
SIQ
SIT-5%
S/T-
ACC
82.33
87.90
90.32
92.87
where ,t, and E,, are the mean vector and covariance matrix of class n. Because different classes have different covariance matrices, the decision boundary of Bayesian classifier is very steep. Figure (6b and Figure (6c are results of ProtoNet trained under S/Q protocol and S/T protocol respectively. In Figure 6c) the decision boundary is smooth and regular, which is different from the previous two models. This result offers a natural interpretation of S/T protocol's benefit: target models impose regularization on task-specific solvers, making them more robust to noisy and biased instances. In fact, [32) also gives a similar conclusion: knowledge distillation can be seen special label smoothing and it regularize model training. In order to more clearly verify this property, we sample biased tasks only containing instances that have low likelihoods (< 0.3 or 0.1), and test different models them. In the second row and third row of Table 4 we can see that S/T protocol can defend biased sampling to the maximum extent because of the strong supervision offered by target models.
ACC(< 0.3)
77.41
81.25
87.66
90.14
ACC(< 0.1)
65.57
70.10
79.22
84.02
'With bit abuse of notation, we y: select instances belonging the r-th class.
8

---

Table 5: Average test accuracy with 95% confidence intervals on mela-testing tasks of minilmageNet and migredl All the methods use ResNet-12 as backbone network except MAML with x mark. The row with mark uses 4- 4-layer ConvNet as backbone network, which is shallower than ResNet-12. 12. Blue values are cited from existing papers while red values are reproduced by us. Best results in bold. We can that MAML and ProtoNet trained under S/T protocol outperform models trained under S/Q protocol even with few target models, Specifically, ProtoNet trained under S/T protocol achieves state-of-the- performance most cases.
minilmageNet
fieredImageNet
Method
5-way 1-shot
5-way 5-shot
5-way 1-shot
5-way 5-shot
DcepEMD 33
65.91 + 0.82
82.41 + 0.56
71.16 + 0.87
86.03 + 0.58
FEAT 30
66.78 0.20
82.05 + 0.14
70.80 + 0.23
84.79 0.16
FRN
66.45 + 0.19
82.83 + 0.13
72.06 + 0.22
86.89 + 0.14
MAML (S/Q)*
8
48.70 1.84
63.11 + 0.92
MAML (S/Q, rc- (e-implement)
58.84 + 0.25
74.62 = 0.38
63.02 0.30
67.26 + 0.32
MAML (5/7-5%)
59.14 + 0.33
75.77 + 0.29
68.39 + 0.34
MAML (S/T-10%)
60.06 + 0.35
76.34 0.42
65.23 0.45
70.02 - 0.33
ProtoNet (S/Q) 21
60.37 + 0.83
78.02 0.57
65.65 0.92
83,40 + 0.65
ProtoNet (S/Q, re-implement)
65.30 0.30
79.93 + 0.39
70.34
84.68 + 0.55
ProtoNet (S/T-5%)
67.35 0.49
81.67 = 0.62
71.25 0.37
85,80 0.31
ProtoNet (S/T-10%)
68.03 + 0.52
82.53 + 0.47
72.41 + 0.39
86.91 + 0.47
Table 6: Ablation study. "Random" means selecting tasks randomly rather than according their hardness.
"6 means using pre-trained network as target model for all tasks. Best results are bold. can see that our proposed heuristic hardness metric and the fine- tuning strategy improve model performance.
minilmageNet
fieredmageNeet
Model
5-way 1-shot
5-way 5-shot
5-way 1-shot
5-way 5-shot
MAML
58.84
74.62
63.02
67.26
MAML -5%--o00%))))mmm
59.66
74.90
65.11
68.63
MAML (S/T-10%-6")
59.35
75.88
64.78
69.26
MAML (S97--10%-bardnesss-)))
60.06
76.34
65.23
70.02
ProtoNet (S/Q)
65.30
79.93
70.34
84.68
ProtoNet (S7T-10%
66.72
81.05
71.22
85.37
ProtoNet (5/T-10%-oP)
67.47
81.70
71.55
86.04
ProtoNet
68.03
82.53
72.41
86.91
5,4 Empirical Study: Benchmark Evaluation
In this part, we evaluate our S/T protocol on two benchmark datasets, i.e. minilmageNNet 24 and tieredImageNet [17]. Refer to supplementary material for dataset details We try to answer four questions: (1) Can we achieve SOTA performance with a classic meta- peaaninn model trained under S/T protocol? (2) How does each component influence model's performance? (3) How does the hyper-parameter influence model's performance? (4) How much time does S/T protocol cost?
Algorithms. We implement two classic meta-learning algorithms, MAML and ProtoNet, under SIT protocol. We use ResNet-13 12 as the backbone network, which is pre- mre-trained on the meta-training set. For a fair comparison, we only include other algorithms that also use ResNet-12 12 backbone network in Table [] More implementation details can be found in the supplementary material.
Competitive Results against SOTA. We show in 5hat MAML or ProtoNet can be improved a lot when trained under S/T protocol with only 5% or 10% target models. Note that vanilla ProtoNet does not use pre-training trick, and we re- re-implement with pre-training. ProtoNet is proposed in 2017, but we can obtain SOTA performance by retraining it under SIT protocol with only a few target models. This verifies the superiority of S/T protocol. In fact, SIT protocol a generic training protocol that can be applied to any meta-learning algorithm, and we apply S/T protocol to more meta-learning algorithms in the supplementary material to show the effectiveness of our method.
*Our code is available at htttttts://gtthu htttp:///gthhb...o/////lu////// com/njulus////TT

---

Table 7: Time consumption fine-tuning target models on minilmageNet. Number of Target Models
Time Consumption (min)
500 (5%)
224.3
1000 (10%)
420.6
2000 (20%)
851.7
Ablation Study. In this part, check the ef- fectiveness of each component. shoows that our proposed hardness metric and fine- tuning strategy help to improve performance. Randomly sampling 10% tasks and constructing target models for these tasks improves model performance. The third row and the seventh row in T verify that learning from target mod- els is beneficial even though the target models are not optimal. With only 10% locally fine- tuned target models and our heuristic hardness metric, can achieve nearly state-of-the-an performance by ProtoNet.
Lvway Loiket
Sway babel
L-ban sccrae 0g
S-abot
Figure 7: Performance change over A. We can see that larger tends to benefit model accuracy. We set À to 0.8, a relatively large value.
We check the influence of hyper-parameter À in Equ 7 We sample 5-way tasks from minilmageNet, and try different values. Figure shows that larger tends to benefit model performance. We set À to 0.8, relatively large value, most of experiments.
Time Consumption. In S/T protocol for few-shot learning, need construct target models through fine-tuning the globally pre-trained network. This will cost extra time to train a model. In this part, we try answer the following question: how much time does S/T protocol cost in few-shot learning? We range the ratio tasks that have target models in f5%. 10%, 20%}. and report the time consumption of fine-tuning target models minilmageNet.. Results are shown in Table 7 We the experiment on an Nvidia GeForce RTX 2080ti GPÜ and Intel(R) Xeon(R) Silver 11 CPU. We that about 4 hours are needed to fine-tune target models for 5% meta-training tasks, and time consumption for fine-tuning 2000 target models is still acceptable.
6 Conclusion
In this paper, study S/T meta-learning protocol that evaluates task-specific solver by comparing to a target model. S/T protocol offers a more informative supervision signal for meta-learning, but difficult to use in practice owing to its high computational cost. We find that by only deploying target models for those hardest tasks, we can improve existing meta-learning algorithms while maintaining efficiency. We propose heuristic task hardness metric and convenient target model construction method for few-shot learning. Experiments on synthetic datasets and benchmark datasets demonstrate the superiority of S/T protocol and effectiveness of our proposed method.
Acknowledgements
This work is supported by National Key R&D Program of China (2020AAA0109400)),, NSFC (41901270, 61773198, 12006112), NSF of Jiangsu Province (BK20190296. BK20200313),, and CCF-Baidu Open Fund PP.
References
[1] Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th International Conference Machine Learning, pages 41-48, 2009.
10

---

[2]
Wei-Lun Chao, Han-Jia Ye, De-Chuan Zhan, Mark Campbell, and Kilian Q Weinberger. Revis- iting meta-learning as supervised learning. CoRR, abs/2002.00573. 2020.
[3] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adap- tation of deep networks. In Proceedings of the 34th International Conference on Machine Learning. pages 1126-1133 113,, 2017.
[4] Micah Goldblum, Steven Reich, Liam Fowl, Renkun Ni, Valeriia Cherepanova, and Tom Goldstein. Unraveling meta-learning
Understanding feature representations for few-shot tasks, In Proceedings of the 37th International Conference on Machine Learning, pages 3607-3616,
2020.
[5] Alex Graves, Greg Wayne, and Ivo Danihelka. Neural turing machines. CoRR, abs/1410.5401.. 2014.
[6] Abhishek Gupta, Russell Mendonca, YuXuan Liu, Pieter Abbeel, and Sergey Levine. Meta- reinforcement learning of structured exploration strategies. In Advances in Neural Information Processing Systems 31, pages 5302-5311. 1,, 2018.
[7] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network.
CoRR, abs//503.02531 2015.
[8] Ekaterina lakovleva, Jakob Verbeek, and Karteek Alahari. Meta-learning with shared amortized variational inference. In Proceedings of the 37th International Conference on Machine Learning. pages 4572 4582, 2020.
[9] Gregory Koch, Richard Zemel, and Ruslan Salakhutdinov. Siamese neural networks for one- shot image recognition. In 32nd International Conference on Machine Learning Workshop. volume 2, 2015.
[10] Kwonjoon Lee, Subhransu Maji, Avinash Ravichandran, and Stefano Soatto. Meta-learning with differentiable convex optimization. In Proceedings of the 32nd IEEE Conference on Computer Vision and Pattern Recognition, pages 10657-10666 2019.
[11] Su Lu. Han-Jia Ye, and De-Chuan Zhan. Tailoring embedding function to heterogeneous few-shot tasks by global and local feature adaptors. In Proceedings of the 35th AAAI Conference on Artificial Intelligence, pages 1776-8783, 2021.
[12] Boris Oreshkin. Pau Rodríguez López, and Alexandre Lacoste. Tadam: Task dependent adaptive metric for improved few-shot learning. In Advances in Neural Information Processing Systems
31, pages 721-731. 2018.
[13] Seong-Jin Park, Seungju Han, Won Back, Insoo Kim, Juhwan Song, Hae Beom Lee, Jae-Joon Han, and Sung Ju Hwang. Meta variance transfer: Learning to augment from the others. In Proceedings of the 37th International Conference on Machine Learning, pages 7510-7520, 2020.
[14] Kate Rakelly, Aurick Zhou, Chelsea Finn, Sergey Levine, and Deirdre Quillen. Efficient off- policy rentforeinfr learning via probabilistic context variables. In Proceedings of the 36th International Conference on Machine Learning, pages 5331-5340, 2019.
[15] Sachin Ravi and Hugo Larochelle. Optimization as a model for few-shot learning. In Proceed- ings of the 5th International Conference Learning Representations, 2017.
[16] Avinash Ravichandran, Rahul Bhotika, and Stefano Soatto. Few-shot learning with embedded class models and shot-free meta training. In Proceedings of the 17th International Conference
on Computer Vision, pages 331-339, 2019.
[17] Mengye Ren, Eleni Triantafillou, Sachin Ravi, Jake Snell, Kevin Swersky, Joshua B. Tenen- baum, Hugo Larochelle, and WRichard S. Zemel. Meta-learning for semi-supervised few-shot classification. In Proceedings of the 6th International Conference on Learning Representations,
2018.
[18] Andrei A. Rusu, Dushyant Rao, Jakub Sygnowski, Oriol Vinyals, Razvan Pascanu, Simon Osindero, and Raia Hadsell. Meta-learning with latent embedding optimization. In Proceedings the 7th International Conference on Learning Representations, 2019.
[19] Adam Santoro, Sergey Bartunov, Matthew Botvinick. Daan Wierstra, and Timothy Lillicrap. Meta-learning with memory-augmented neural networks. In Proceedings of the 33rd Interna- tional Conference on Machine Learning, pages 1842-1850) 1850, 2016
[20] Christian Simon, Piotr Koniusz, Richard Nock, and Mehrtash Harandi. Adaptive subspaces for few-shot learning. In Proceedings of the 33rd IEEE Conference on Computer Vision and Pattern Recognition, pages 4136-4145 2020.
[21] Jake Snell, Kevin Swersky, and Richard S. Zemel. Prototypical networks for few-shot learning.
In Advances in Neural Information Processing Systems 30, pages 4077-4087. 2017.
[22] Sebastian Thrun and Lorien Pratt. Learning to Learn. Springer Science & Business Media,
2012.
11

---

[23]
Ricardo Vilalta and Youssef Drissi. A perspective view and survey of meta-learning. Artificial Intelligence Review, 18(2):7-995, 2002.
[24]
Oriol Vinyals, Charles Blundell. Timothy Lillicrap, Koray Kavukcuoglu, and Daan Wierstra. Matching networks for shot learning. In Advances in Neural Information Processing
Systems 29, pages 3630-3638. 2016.
[25]
Risto Vuorio, Shao-Hua Sun, Hexiang Hu, and Joseph J Lim. Multimodal model-agnostic meta- learning via task- modulation. In Advances in Neural Information Processing Systems 32,
pages 1-12. 2019.
[26]
Yan Wang, Wei-Lun Chao, Kilian Q Weinberger, and Laurens van der Maaten. Simpleshot: Revisiting nearest-neighhor classification for few-shot learning. CoRR, abs/1911.04660 2019.
[27] Yu-Xiong Wang. Adrien Bardes, Ruslan Salakhutdinov, and Martial Hebert. Progressive
knowledge distillation for generative modeling. 2019.
[28]
Yu-Xiong Wang and Martial Hebert. Learning to learn: Model regression networks for casy small sample learning. In Proceedings of the 14th European Conference Computer Vision, pages 616-634, 2016.
[29] Davis Wertheimer, Luming Tang, and Bharath Hariharan. Fine-grained few-shot classification with feature map reconstruction networks. CoRR, abs/2012.01 506, 2020.
[30] Han-Jia Ye, Hcxiang Hu, De-Chuan Zhan, and Fei Sha. Few-shot learning via embedding
adaptation with set-to-set functions. In Proceedings of the 33rd IEEE Conference Computer
Vision and Pattern Recognition, pages 8808 8817, 2020.
[31] Han-Jia Ye. Lu Ming. De-Chuan Zhan, and Wei-Lun Chao. Few-shot learning with strong teacher. CoRR, abs/2107 .00197, 2021.
[32] Li Yuan, Francis EH Tay, Guilin Li, Tao Wang, and Jiashi Feng. Revisiting knowledge distillation via label smoothing regularization. In Proceedings of the 33rd IEEE Conference Computer Vision and Pattern Recognition, pages 3903-3911, 2020.
[33] Chi Zhang, Yujun Cai, Guosheng Lin, and Chunhua Shen. Deepemd: Few-shot image classifi- cation with differentiable earth mover's 's distance and structured classifiers. In Proceedings of the 33rd IEEE Conference on Computer Vision and Pattern Recognition, pages I 12203-1220 12213, 2020.
12


Reviews:
[{"review_id": "soBZJn2er4t", "reviewer": "Reviewer_V6R5", "summary": "This paper investigated the S/T meta-learning framework which is not widely used due to the huge computational cost. The proposed method estimates the hardness of tasks and selects the target models of a small ratio of hard tasks. Then, the meta-model is learned with the S/Q meta-loss and the knowledge distillation loss between the selected target models and solvers. Experiments show that the proposed method outperforms the two classic meta-learning methods with the cost of constructing target models. ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1.\tGenerally speaking, this paper is easy to read and understand. In the introduction, it will be clearer to match the solver, target model and meta-model to the corresponding modules or neural networks. Maybe the authors can give brief idea of how to construct the target models in the introduction. The description about S/Q and S/T redundantly appears several times in the paper and can be shorten. \n2.\tThe equivalence between the S/T meta-loss and the simplified one (Line 179-180) is interesting. Does it mean that adding random noise to training data is the key to meta-learning? Do the two losses lead to the same results? Can the authors empirically verify it?\n3.\tThe study and analysis on the target models are sufficient and solid. \n4.\tThe experiments on miniImageNet and tieredImageNet show that the proposed method achieves obvious improvements over the two classic meta-learning methods with the cost of constructing target models.\n5.\tOverall, I think this paper can to some extent advance the meta-learning research.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "It would be better to explicitly discuss the limitation in the main pages. Societal impact is N/A.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "UQi7C3ktSkG", "reviewer": "Reviewer_f3eh", "summary": "The authors propose a novel algorithm for meta-learning, in which instead of just optimizing for the query loss, they also optimize the inner loop produced model to do as well as some larger ‘teacher’ model that is an actual expert on the task at hand. The data used to train those experts are the same data we normally use during the meta-training phase in a particular benchmark, therefore their process does not require more data, but its efficiency can either be better or worse than a model trained with the standard S/Q loss depending on how many ‘expert’ models are pretrained. \n", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Abstract: \n\nOverall a very good abstract. However, you never defined the term ‘Target Model’ and it is one of the key terms necessary to understand what is going on. I’d recommend an explicit definition somewhere in the abstract, to ensure the reader understands.\n\nWriting Quality:\n\nOverall the paper reads very well, with the occasional typo, but overall it is precise, concise and very clear. \n\nA few points I’d like to mention is, firstly, Figure 1 is not readable without zooming, and the text colour matches the surrounding box which makes it even harder to read. I recommend making the figure a bit larger, and the text kept dark.\n\nTable 5 states the word Black in what appears to be bold, therefore confusing the reader, with the actual bold results being the best results. I recommend adapting the colours/formatting here to make it more intuitive.\n\nConclusion appears to be weirdly formatted for some reason. I assume the figure has some role to play there -- I recommend reformatting this bit to make it cleaner.\n\nNovelty:\n\nVery novel, and surprising work. Using the same data, but by producing such expert models and trying to basically distill their capabilities into the resulting adapted models in each task seems to be very powerful, generalization wise. This could be the beginning of a good meta learning penalty term! :)\n\nI find the results very convincing as to the claims made, and I appreciate that multiple independent runs were done for each data point in the tables. \n\nOverall quite good, assuming the abstract issue is fixed, I recommend acceptance.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The impact is the same as any other few-shot learning work undetaken in the context of deep learning. Such concerns are better described in full papers in the ethics and deep learning subfield. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1"}, "extra_text": {}}, {"review_id": "IKBxD1LPfX", "reviewer": "Reviewer_mVfo", "summary": "This paper proposes an efficient S/T protocol meta-learning algorithm. Specifically, in order to reduce the number of required target models and the high computational cost, only the target models on those hardest tasks are constructed by fine-tunning the pre-trained network, then the knowledge distillation is used to match the task-specific solvers to target models in the output space. Experiment results verify the effectiveness of the proposed algorithms.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper proposes an efficient S/T protocol meta-learning algorithm. Specifically, in order to reduce the number of required target models and the high computational cost, only the target models on those hardest tasks are constructed by fine-tunning the pre-trained network, then the knowledge distillation is used to match the task-specific solvers to target models in the output space. Experiment results verify the effectiveness of the proposed algorithms.\n\nStrong points:\n1.The motivation is clear.\n\n2.An efficient S/T protocol meta learning algorithms is proposed by fine-tuning globally pre-trained model on several hardest tasks.\n\n3.Experiment results illustrate that the classic meta learning algorithms trained via the proposed S/T protocol can achieve SOTA performance.\n\nWeak points: \n1.The main contribution should be emphasized.  In this manuscript, the two contributions are introduced and verified respectively. It would be better to emphasize on the intro section.\n\n2.The proposed algorithm is still under the condition of same task distribution, because the target models is always trained on a sufficiently informative dataset. However, in many real-world scenarios the training distribution differs from the test distribution. How the out of distribution will influence the algorithm performance should be further investigated in several benchmark datasets, such as [1].\n\n3.Meta learning algorithms aim to achieve fast adaptive ability during training phase. Thus, when encountered new tasks, even those tasks have new label space, the learnt model can still achieve better performance with only few data. If the label space of new tasks is different from the training task one, will the proposed approach still effective?\n\n4.The recognition of task hardness depends on the base model \\phi^{pt}, which is pre-trained on training task dataset. Will the per-trained model influence the final performance should be further investigated.\n5.It would be better to give in-depth analysis on why this algorithm can generalize better.\n\n\nOverall, the topic is interesting, and experiments verified the proposed approach outperform than other algorithms. However, the contribution is still marginal, and this research lacks theoretical analysis. Authors didn’t provide insight into how the proposed method can achieve better generalizability.\n\nIf providing satisfying answers, I will accept the paper. \n\n\n\n[1] Koh, P. W., Sagawa, S., Marklund, H., Xie, S. M., Zhang, M., Balsubramani, A., ... & Liang, P. (2020). Wilds: A benchmark of in-the-wild distribution shifts. arXiv preprint arXiv:2012.07421.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No potential negative societal impact of their work", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "at least 12 hours"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers agree that the paper has interesting ideas, and represents an exciting direction in meta learning domain. Hopefully, reviewer comments should help preparing the next draft. 
Decision: Accept (Poster)