On the Role of Pre-training for Meta Few-Shot Learning
Authors: Chia-You Chen
Hsuan-Tien Lin
Gang Niu
Masashi Sugiyama
Venue:  (2021)
Keywords: Meta-Learning
Episodic Training
Pre-training
Disentanglement

Abstract: Few-shot learning aims to classify unknown classes of examples with a few new examples per class. There are two key routes for few-shot learning. One is to (pre-)train a classifier with examples from known classes, and then transfer the pre-trained classifier to unknown classes using the new examples. The other, called meta few-shot learning, is to couple pre-training with episodic training, which contains episodes of few-shot learning tasks simulated from the known classes. Pre-training is known to play a crucial role for the transfer route, but the role of pre-training for the episodic route is less clear. In this work, we study the role of pre-training for the episodic route. We find that pre-training serves a major role of disentangling representations of known classes, which makes the resulting learning tasks easier for episodic training. The finding allows us to shift the huge simulation burden of episodic learning to a simpler pre-training stage. We justify such a benefit of shift by designing a new disentanglement-based pre-training model, which helps episodic learning achieve competitive performance more efficiently. 

Under review as a conference paper at ICLR 2021
ON THE ROLE OF PRE-TRAINING FOR META FEW- SHOT LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Few-shot learning aims to classify unknown classes of examples with a few new examples per class. There are two key routes for few-shot learning. One (pre- )train a classifier with examples from known classes, and then transfer the pre- trained classifier to unknown classes using the new examples. The other, called meta few-shot learning. is to couple pre-training with episodic training, which contains episodes of few-shot learning tasks simulated from the known classes. Pre-training is known to play crucial role for the transfer route, but the role of pre-training for the episodic route less clear. this work, we study the role of pre-training for the episodic route. We find that pre-training serves major role of disentangling representations of known classes, which makes the resulting learning tasks easier for episodic training. The finding allows to shift the huge simulation burden of episodic training to simpler pre-training stage. We justify such a benefit of shift by designing new disentanglement based pre- training model, which helps episodic training achieve competitive performance more efficiently.
INTRODUCTION
In recent years, deep learning methods have outperformed most of the traditional methods in super- vised learning, especially in image classification. However, deep learning methods generally require lots of labeled data achieve decent performance. Some applications, however, do not have the lux- ury to obtain lots of labeled data. For instance, for bird classification, an ornithologist typically can only obtain few pictures per bird species to update the classifier. Such needs of building classifiers from limited labeled data inspire some different research problems, including the few-shot learning problem (Fnn et al. 2017: Snell et al. 2017 Rajeswaran et al. 2019) Oreshkin et al. 2018 Vinyals) al. 2016 Lee et al. 2019). In particular, few-shot learning starts with a training dataset that con- sists of data points for "seen" classes, and required to classify "unseen" ones in the testing phase accurately based on limited labeled data points from unseen classes.
Currently, there are two main frameworks, meta- weeta-lerrnng (Finn et al.] 2017; (nell et al. 2017| Chen al. 2019) and transfer learning (Dhillon et al. 2020), that deal with the few-shot learning problem. For transfer learning, the main idea is to train a traditional classifier on the meta-train dataset. In the testing phase, these methods finetune the model on the limited datapoints for the labeled novel classes. For meta-learning frameworks, their main concept is episodic training (Vinyals et al. 2016). For the testing phase of few-shot learning, the learning method is given N novel classes, each containing K labeled data for fine-tuning and Q query data for evaluation. Unlike transfer learning algorithms, episodic training tries simulate the testing literature in the training phase by sampling episodes in training dataset.
In these two years, some cunsfer-learning methods (Dhillon al. 2020 with sophisticated design in the finetuning part have competitive performance to the meta-learning approaches. Moreover, researchers (Lee et al., 2019 Sun et al. 2019) Chen et al. 2019 Oreshkin et al. 2018) have pointed out that combining both the global classifier (pre-training part) in the transfer learning framework and the episodic training concept for the meta-learning framework could lead to better performance. the episodic training part (Vinyals al.) 2016 Finn
Yet, currently most of the attentions jet al. (2177; (Snell et al. [017]] Oreshkin et al. 2018 Sun et al. 2019 Lee et al., 2011 and the role of pre-training is still vague.

---

Under review as a conference paper at ICLR 2021
Meta-learning and pre-training has both improved lot in the past few years. However, most of the works focus on the accuracy instead of the efficiency. For meta-learning, to make the progress more efficient, intuitive way is to reduce the number of episodes. Currently, there only limited researches (Sun et al. 2019) working on reducing the number of episodes. One of the methods (Chen al. 2019: Lee al. 2019) is apply better weight initialization method, the one from pre-training, instead of the random initialization. Another method (Sun et al. (2019) is to mimic how people learn. For example, when we are learning dynamic programming. given knapsack problem with simple constraint and the one with strong constraint, we will learn much more when we solve the problem with strong constraint. [Sun et al. (2019) followed the latter idea and crafted the hard episode to decrease amount of necessary episodes.
In this work, we study the role of pre-training in meta few-shot learning. We study the pre-training from the disentanglement of the representations. Disentanglement is the property that whether the datapoints within different classes has been mixed together. [Frosst et al. (2019) pointed out that in- stead of the last layer of the model all representations after other layers were entangled. The last layer does the classifier and the rest captures some globally shared information. By analyzing the disen- tanglement property of episodic training, though the pre- training gives better representation that benefits the episodic training, the representation becomes more disentangled after episodic training. That is say, episodic training has spent some effort on making the representation more disentan- gled. Benefited from the understanding, we design a sophisticated pre-training method that is more disentangled and helps episodic training achieve competitive performance more efficiently. With our pre-traie vee-training loss, the classical meta-lcarning algorithm, ProtoNet (Snell et al. (017), achieves competitive performance other methods. Our study not only benefits the episodic training but also points out another direction to sharpen and speed-up up episodic training.
To sum up. there are three main contributions in this work:
1. A brief study of the role of pre-training in episodic training.
2. A simple regularization loss that sharpens the classical meta-learning algorithms.
3. A new aspect for reducing the necessary episodic training episodes.
RELATED WORK
Few-shot learning tries to mimic the human ability to generalize to novel classes with limited data- points. In the following, we briefly introduce the recent progress of the transfer-learning framework and two categories of the meta-learning framework. Afterward, we give a brief introduction of the not well studied episode efficiency problem.
2.1 TTANSFER-LEARNIGG FRAMEWORK
In the training phase, the transfer-learning framework trains classifier the general classification task across all base classes instead of utilizing episodic training. And for the testing phase, transfer- learning methods finetune the model with the limited labeled data. There are several kinds of tricks.
Qi et al. 2018 proposed method to append the mean of the embedding with a given class as final layer of the classifier. Qiao et al. (2018) used the parameter of the last activation output to predict the classifier for novel classes d dynamically. Gidaris & Komodakis (0118 proposed a similar concept with Qiao al. (2018). They also embedded the weight of base classes during the novel class prediction. Moreover, they introduced an attention mechanism instead of directly averaging among the parameters of each shot. Besides embedding base classes weight to the final classifier, Dhillon et al.) (2020) utilized label propagation by the uncertainty on single prediction to prevent overfitting in the finetune stage, which is quite similar to the classical classification tasks.
2.2 META-LEARNING FRAMEWORK
For meta-learning like framework, the main concepts are learning to learn and episodic training VVinyals al. 2016). Learning learn refers to learn from lot of tasks to benefit the task learning. To prevent confusion, the original train and test phase are regarded as "meta-train" and "meta-test". The term "train" and "test" would be referred to the one in each small task. Episodic
2

---

Under review as a conference paper at ICLR 2021
training is the process of mimicking the task structure in meta-test during training. If the meta-test phase consists of K support examples and Q query examples from N classes, then we will sample lots of tasks that also have K support examples and Q query examples from N classes in the meta- train phase. Meta-learning algorithms have developed rapidly in recent years. We briefly categorize them into two categories, aptimization-basee methods, metric-based methods.
Optimization-basee Method Optimization-basee methods try to get an embedding that could easily fit subtasks by adding some extra layers. Finn et al. (2017) proposed MAML (Model-Agnostic Meta-Learning), which got network that was the closest to all the best model in the low-way low- shot tasks. However, MAML might be quite inefficient due to the computation of Hessian matrix. To leverage the issue, iMAML (Rajeswaran et al. 2019) and foMAML (Finn et al., 2017) provides different approximation to avoid the heavy computation. However, MAML still suffers from the high dimensional overfitting issue. LEO model (Rusu al. 2019) solves the overfitting issue by learning a low dimensional latent space. Instead of aiming to get an embedding that could benefit the latter fully connected layer, MetaOptNet (Lee al. 2019) aims get an embedding that could benefit the latter differentiable support vector machine.
Metric-based Method Instead of learning an embedding that could benefit the latter additional layer, metric-based methods aim to get an embedding that could easily classify the classes by simple metrics. Matching Networks (Vinyals et al., 2016) conducts a cosine similarity metric with Full Context Encoding module. Prototypical Networks (Snell et al.) (017)) replaces the cosine similarity metric with the squared Euclidean distance and computes the mean of the embedding in the support- set as the prototype. Relation Network (Sung et al., 2018) embeds a relation module in the learning metric. Instead of using consistent metric in the task, TADAM (Oreshkin et al. 2018) designs dese-depnndent metric that could dynamically fit new class combinations.
2.3 MIXED FRAMEWORK
Some recent works have found that using a global pre-trained classifier as the initialization weight could lead to better meta meta-learning result. [Sun et al. (2019) used the pre-trained classifier weight as initialization weight and launched simple gradient- gradient-based with restricting the learning process as shift and scale. Meta-Baseline (Chen al. (2020) also follows the initialization literature and applies cosine similarity metric for the following learning process. Chen al. (2019) changed the original pre-trained network structure into weight imprinting taste and simple gradient-based method for the episodic training part. TTriantafilouu et al. (2020) also utilized the pre-trained initialization and derived combination between MAML Finn et al. (2017) and ProtoNet (Snell et al. 2017).
2.4 EPISODE REDUCTION METHODS
Recent researchers have found that pre-trained classifier leads to better meta-learning results. On the other hand, we could reduce the amount of episodes by using a pre-trained classifier. Besides utilizing the pre-training weight to reduce the number of episodes, Meta Transfer Learning (Sun jtt al. 2019) proposes the concept of hard episode. For each normal episode, MTL adds the class with the worst performance in pool. After collecting for while, MTL creates hard episodes by sampling from the pool.
Instead of crafting hard episodes, our approach tries to utilize more in the pre-training phase. We propose a simple regularization that could reduce the difference between the embeddings the pre- trained classifier and the episodic training one. It has significantly reduced the number of episodes and achieves similar (even better) performance for the original algorithms. Moreover, for shallow and deep backbones, it increases the final accuracy.
METHODOLOGY
No matter in the route of transferring the classifier or the route of episodic training, pre-training serves crucial role. And in meta few-shot learning, pre-training provides an initialization weight for further episodic training. In recent episodic- based methods, the pre-training model is split into two parts, backbone and classifier. The last linear layer serves as the classifier and maps the embedding
3

---

Under review as a conference paper at ICLR 2021
to logits. Others work the backbone and transform the photo into embedding. After pre- training. the classifier part is directly dropped, since the target classes may be unseen or the order may have changed. Though the split is quite naive, the afterward episodic learning converges faster and better based on the initialization. Thus, previous works conclude that pre-training provides a better representation. However, what makes it better is not clear. What the role of pre-training in meta few-shot learning? More specifically, what is the character of backbone in pre-training and episodic training?
In ths section, we choose prototypical network (Snell et al. 2017) as the representative. Following the analysis of general learning literature by Frosst et al. (2019). we utilize the similar loss to mea- sure the disentanglement and entanglement property of the backbone in pre-training and episodic training. Benefited from our observation, we give an attempt to transfer the computation burden of episodic training by adding an sophisticated loss in pre-training.
3.1 NOTATION
In a N-way K-shot few-shot classification task, we given small support set of NK labeled data = (1)1.//).... (INK, ]N where I2 R and yi E {1,... N). S, denotes the set of examples labeled with class
3.2 PROTOTYPICAL NETWORK
Prototypical Network (Snell al. 2017) is one of the classical metric-based meta few-shot learning algorithms. First, computes prototypes C; which are M-diimensional representations C; E gM by averaging the output of the embedding function fo RDD R from the same class set S₁.
C;
Σ (2.y)ES,
fo(x)
S
Then, the prototypical network caleulates probability distribution for given data æ by softmax over the Euclidean distance between each prototype and the embedding fe(x).
(ep((d((o(z)) cn))
po = n1(x)) n' x'E(U....)))
Σ
expl - d(Fo(r), cw)))
3.3 SOFT-NEAREST NEIGHBOR LOsS
"Soft-Nearest Neighbour loss [...] is proposed by Frosst et al. (2019)" Frosst et al. credits Salakhut- dinov and Hinton (2007) for the soft nearest neighbour loss, which itself draws inspiration from Goldberger et al. (2005)'s Neighbourhood Component Analysis.
Soft-Nearest Neighbor Loss (SNN-Loss, Eq is first proposed by Salakhutdinov & Hinton (007).
Then, Frosst et al. (2019) applies the loss to analyze disentanglement of layer. The loss measures the disentanglement property of a representation. Higher loss means more entangled representation. In the original study, general supervised learning, which is the pre-training part in meta few-shot learning, has an entanglement trend in all the layers instead of the last layer. On the other hand, in meta fho-shot learning, especially metric-based methods, after the pre-training phase, the last layer is replaced with a metric. This leads to an interesting question. What should the afterward last layer be, entangled or disentangled? By the experiment in Sec we find that with sufficient amount of pre-training the representation is less entangled after episodic learning. In other words, the metric prefers a more disentangled representation.
Σ j
eap(-l11:; 1z. =ry113
Soft-Nearest-NiihhoorL..sss
1
Σ
Σ
cxp(-1las In xmnl²))
iE iE(L...)
nE(1.b)).z3
4

---

Under review as a conference paper at ICLR 2021
Algorithm 1 Regularized Pre-trained Prototypical Network (RP-Proto)
Require: Training set ID { V11)....., (æNK K. YN ), where each Ynk n.
1: M: metric for judging distance between two vector:
2: fe: stands for the feature extractor which may output dimension vector.
3: g: maps the output of ʃa to dimension.
4: b: batch sizes for pretraining.
5: p. pretrained iteration number.
6: a: weight vector for the regularization loss.
7: L: cross entropy loss
8: Initialize W; with dimension for E {l.... K}
9: for i E range(p) do
10:
(X,Y) <- (ample(D.b);
11:
Le - (
12:
heg +
Σ M(folz),W, Wy)
hotal le (xx,))
13:
14:
f0.9 + backprop(hootat (fo.g))
15:
end for
16: fo < ProtoNet(fe- D)
return fe
3.4 REGULARIZED PRE-TRAINED PROTOTYPICAL NETWORK (RP-PROTO)
In the previous section, we conclude that the last layer in the backbone would be more disentangled after episodic training. As result, we wonder whether providing a more disentangled representa- tion of the penultimate layer in the pre-training part could speed up the later episodic training. A naive way to provide a more disentangled representation may be transferring the loss in the episodic training part to pre-training. However, the naive method slows down the pre-training part due to the episode construction. Also, it may require an additional sampler to craft the episode task. To prevent from additional sampler, we could directly compute the episodic training loss among each batch during pre-trate pre-training. However, the naive method suffers from three critical issues. First, if the batch size is not the scale the number pre-training classes, some class may not exist in the batch. Second, the amount of each class may be non-uniform. Third, the computation of the gradient of a single instance may mix up with other instance in the batch, which makes parallelization harder.
To deal with the above issues, introduce surrogate loss, which aims for embedding with more disentanglement. We capture the disentanglement representation from another aspect, instances within the same class should gather together in the embedding space. The surrogate loss brey is based on the distance between fo(r) and Wy, where W, is an M-dimensional learnable parameter.
(r) = d(fo(z),
For the pre-training phase on the ordinary classification task, fa corresponds to the network right before mapping to final K K-dimension embedding. Then, we add lzeg to the ordinary classification loss with multiplying weight of a.
ltotal ielassificatiio (z. y) + X (2, (a 10
(5)
Wy is the surrogate mean of each class. Instead of directly summing up all the embeddings, W, could be calculated by backpropagation. Moreover, when W,, equals to zero, it could be considered as an L2 regularization on the feature space, which scales the embedding. When W,, is learnable, makes each instance in the same class closer which satisfies the our goal of disentangled represen- tation. We have described the detail in A
Regarding the flaws of the naive approach, in this auxiliary loss schema, a data point doesn't 't need to consider the distance of data points in other classes, As a result, it could avoid the large computation effort of the cross-classes datapoints distance and the non-uniform amounts of datapoints issue.
5

---

Under review as a conference paper at ICLR 2021
4 EXPERIMENTS
4.1 DATASET DETAILS
Minilmagenet Proposed by Vinyals et al. (2016) is a widely used benchmark for few-shot learn- ing. It is subset of ILSVRC-2015 (Deng et al. 2009) with 100 classes, 64 for meta-train, 16 for meta-validation and 20 for meta-test. Each class contains 600 images of size 84x84.
CIFAR-FS FC100 Proposed by Bertinetto et al. (2019) and Oreshkin et al. (2018). both are splits between the original classes of CIFAR 100 (Krizhevsky al.). They also follow the mini- Imagenet structure with 64 base classes, 16 validation classes, and 20 novel classes. The pictures in CIFAR100 are 32 32 low resolution. The main difference is that CIFAR-FS randomly splits among the classes, but FC100 splits the classes carefully with less semantic overlap between the base classes and the novel classes. In general, FC100 is more difficult than CIFAR-FS.
4.2 EXPERIMENT SETUP
First, we design an experiment study the role pre-training in meta-fewshot learning. We train eral classifiers with sufficient epochs and launch the episodic training with the prototypical network methods. The disentanglement of the penultimate layer pre-training and the last layer in episodic training are compared. The experiment is evaluated in 5-way 5-shot settings.
Second, to evaluate the power of the auxiliary loss and validate our conjecture. We compare the episodic training performance based on the backbones with and without our auxiliary loss in pre- training phase. The experiment is evaluated in both 5-way 5-shot and 5-way 1-shot settings. more detail, we search the auxiliary loss weighted coefficient a = 10 from C = -1 to -3.5 for each backbone on each dataset.
4.3 IMPLEMENTATION DETAILS
Model Architecture For the backbone of ResNet10 and ResNet18, we follow the preprocessing from Chen et al. (2019). We use image sizes with 224x224 for both pre-trained prototypical net- work and regularization pre-trained prototypical network. For ResNet12 backbone, we follow the implementation from Sun et al. (2019) and use image sizes with 80x80. For the Conv4 backbone, we follow the implementation in prototypical network (Snell et al. 2017) with image size 84x84. The detail parameters are in the appendix.
Pre-training For pre-training the backbone, follow the settings in Chen et al. (202)). We use a stochastic gradient descent optimizer with learning rate as 0.001 for all the backbones and the batch size is 128.
Episodic training For the episodic training part, we also follow the settings in Chen et al. (2020) We use stochastic gradient descent optimizer with learning rate as 0.001 and turn off the data aug- mentation part in the preprocess pipeline to reach more stable result.
Evaluation
For the performance evaluation of episodic training, we turn off all the data augmenta- tion and record the mean of the performance and the 95% confidence intervals among 600 randomly sampled episodes.
4.4 LAYER DISENTANGLEMENT WITH METRIC
In this part, we design experiments to figure out the role of pre-training. We utilize the soft-nearest neighbor loss Eq to measure the disentanglement of the last layer in episodic training among episodes. Fig 1 is the soft-nearest loss in episodic training. No matter in the shallow backbone Conv4 or the deep backbone ResNet10, ResNet18 the loss has a decreased trend while the episode increases. For lower noft-nearet-nnighhbr loss the representation becomes to say, in episodic learning, the metric prefers less entangled representation. This makes sense in the higher level. Since in episodic training, especially metric-hased methods, the metric may
disentangled. That

---

Under review as a conference paper at ICLR 2021
Table 1: 5-way 5-shot and 5-way 1-shot performance on minilmagenet. The weighted coefficient
regularzaation loss 10 to C.
BACKBONE
SHOT
SHOT
Matching Networks
Conv4
43.56 + 0.84
55.31 + 0.73
Prototypical Networks
Conv4
48.70 + 1.84
63.11 + 0.92
LEO
WRN-28-10
61.76 + 0.08
77.59 = 0.12
Chen
ResNet18
51.87 0.77
75.68 0.63
SNAIL.
ResNet12
55.71 + 0.99
68.88 + 0.92
AdaResNet
ResNet12
56.88 0.62
71.94 0.57
TADAM
ResNet12
58.50
76.70 + 0.30
MTL
ResNet12
6
75.50 + 0.80
MetaOptNet
ResNet12
62.64 0.61
78.63 + 0.46
Prototypical Networks (Pre)
Conv4
42.08 0.75
67.77 + 0.68
Prototypical Networks (Pre)
ResNet10
55.34 + 0.81
76.97 + 0.67
Prototypical Networks (Pre)
ResNet12
56,83 + 0.83
76.45 = 0.64
Prototypical Networks (Pre)
ResNet18
57.80 0.84
77.71 - 0.63
RP-Proto (euclidean) (C: -1.5)
Conv4
50.38 + 0.80
69.56 0.71
RP-Proto (cuclidcan) (C: -1.5)
ResNet10
56.92 0.84
77.45 + 0.65
RP-Proto (euclidean) (I-shot C: -2.5) (5-shot C: -3.5.)
ResNet12
57.84 0.82
77.08 = 0.70
RP-Proto (euclidean) (I-shot C: -1.0) (5-shot C: -3.5)
ResNet18
57.88 0.86
77.43 0.63
be geometric related, e-g. the distance the center of each class. the representation is highly entangled, it is non-intuitive for the metric to figure the class of each instance.
4.5 PERFORMANCE EVALUATION
11.58
In the minilmagenet dataset result (Table our method shows competitive performance to other methods. We find that adding the auxiliary loss could make the shal- low network has similar performance with the large one.
$2.00
Moreover, when the backbone is shallow, we have outper- formed all other methods with the same backbone. For experiments on 10100 and cifar-fs, the results are similar minilmagenet. For detailed results, please check the appendix.
Conuh
Figure 1: Comparison for the SNN-
Loss during the meta-learning process 5-way 5-shot minilmagenet.
For major conjecture, disentangled representations could benefit the following episodic training, we do a one by one comparison between general pre-trained Prototypical Networks and the regularized pre- trained one by recording the meta-test performance after every 100 episodes in Fig 2 Instead of ResNet18, all the regularized pre-trained backbones lead to a faster convergent speed by providing better initial accuracy. This could support our conjecture that disentangled representations could boost the speed episodic training.
4.6 REGULARIZATION AND DISENTANGLEMENT
Our goal of the regularization loss is to control the disentanglement of the embedding space. In the previous experiments, our methods indeed benefit the speed of episodic training. However, how the regularization affects the disentanglement and the performance is still not clear. There two potential reasons for interaction. First, the regularized pre- pre-training leads the model to start in better initialization point with higher disentanglement and higher accuracy. Second, the regularized term helps the model to start in an initialization space with better gradient property to achieve higher disentanglement and optimal weight.
Fig shows the experiment in 5-way 5-shot minilmagenet. The SNN-loss with regularization is always lower than the original However, it doesn't support the first possible reason, since the

---

Under review as a conference paper at ICLR 2021
EN
CM
3.
0.10
0350
0.10
53 20
200
10.
1301 220
Ad
140
200
ean mpneees
mpad
(a) Conv4 (5-shot)
(b) Res (5-shot)
(c) ResNet12 (5-shot)
(d) ResNet18 (5-shot)
045
2
650
Pretrained
Pistrained Pisto
Prerrsined Proto
Fretrained
Seguila Reetrained Proto
Pretminndd Pooto
Reguuaiized Peermsiied Proto
209
750
330
199
200
290
103
150
230
35
x0
130
150
250
x0
spaader
1ga0
(e) Conv4 (1-shot)
(1) ResNeetlO (1-shot)
(g) ResNett2 (1-shot)
(h) ResNet18 (I-shot)
Figure 2: Comparison for the converge speed between standard pre-trained and regularized pre- trained on 5-way 5-shot and 5-way 1-shot minilmagenet.
31.50
0.200
33.40
0.625
23.30
0.90
11.30
0.75
21.55
0.50
0-15
0:33
11:20
02%
2.22
1100 astidie)
mooch 1000 aedddedd
Neach eeiooes
(a) SNN-Loss
(b) Accuracy
Figure 3: SNN-loss ablation Figure 4: Accuracy and SNN-loss with different weighted parameter in parison with regularization.
episodic training
initial SNN-loss is quite similar to the one without regularization. However, after several episodes, the loss severely decreases. This makes the second claim more convineing.
In Fig 4a larger weighted parameter leads to higher disentanglement. And in Fig 4b there is a large gap for the initialization accuracy for different weighted parameters. However, better initialization accuracy doesn't t imply better final accuracy. For instance, C -1.5 has a worse initialization accuracy but converges to the best weight in the end. With suitable weighted parameters, the rep- resentation is disentangled properly, the final accuracy has great improvement. It is similar to the general regularization idea that suitable regularization could lead to performance improvement but extreme regularization may harm the performance.
CONCLUSION
The disentanglement analysis provides a new aspect to view the role of pre-training in meta few- shot learning- Furthermore, the relationship between the backbone and the classifier is discussed. Though in higher concept, the metric should replace the role of pre-training classifier in the episodic training phase. However, the backbone needs to share the role of the classifier, so the backbone has disentangled trend in the classifier. Benefited from the observation, to transfer the computation burden of episodic training to the pre- preitrain With the help of the loss, episodic training could achieve competitive performance with fewer episodes.
designed auxiliary loss
REFERENCES
Luca Bertinetto, Joao F. Henriques, Philip Torr, and Andrea Vedaldi. Meta-learning with differen- tiable closed-form solvers. In International Conference on Learning Representations, 2019. URL http://pww......eeeeeo//o/ooooaa
httpps://openss net feet:/ooommidddeee
8

---

Under review as a conference paper at ICLR 2021
Wei- Yu Chen, Yen-Chent Liu, Zsolt Kira, Yu-Chiang Frank Wang, and Jia-Bin Huang. A closer look at few-shot classification. In International Conference on Learning Representations, 2019.
URL /RR//tttps///peeeeeeeeee net/ wtr.mo 1orumm2iddllxxLXAAcc
Yinbo Chen, Xiaolong Wang, Zhuang Liu, Huijuan Xu, and Trevor Darrell. A New Meta-Baseline for Few-Shot Learning. arXiv e-prints, art. arXiv:2003.0453 March 2020.
Deng, W. Dong, R. Socher, L. Li, Kai Li, and Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pallern Recognition, pp- 248-255, 2009.
Guneet Singh Dhillon, Pratik Chaudhari, Avinash Ravichandran, and Stefano Soatto. A baseline for few-shot image classification. International Conference on Learning Representations, 2020.
URL htttp:://ppeneevtttiet/ net sinet/oorum?ddrry XBBrYDS
Chelsea Finn, Pieter Abbeel, and Sergey Levine. agdel-agnostic meta-learning for fast adaptation of deep networks. In Doina Precup and Yee Whye Teh (eds.). Proceedings of the 34th International Conference on Machine Learning. volume 70 of Proceedings of Machine Learning Research, pp- 1126-1135, International Convention Centre, Sydney, Australia, 06-11 Aug 2017. PMLR. URL
Nicholas Frosst, Nicolas Papernot, and Geoffrey E. Hinton. Analyzing and improving represen- tations with the soft nearest neighbor loss. In ICML, pp. 2012-2020 2019. URL http:
//proceedings mmr http.//ppeeeee///foostttttte html
Spyros Gidaris and Nikos Komodakis. Dynamic few-shot visual learning without forgetting, In The IEEE Conference Computer Vision and Pattern Recognition (CVPR), June 2018
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. cifar100. URL http: http::::wwwwwcs..oorro..
Sttt/"kriz/cifaat html
Kwonjoon Lee, Subhransu Maji, Avinash Ravichandran, and Stefano Soatto. Meta-learning with differentiable convex optimization. In CVPR, 2019.
Boris Oreshkin, Pau Rodríguez López. and Alexandre Lacoste. Tadam:
Task dependent adaptive metric for improved few-shot learning. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa- MisacBianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems 31, pp. 721-731. 2018.
Hang Qi. Matthew Brown, and David G. Lowe. Low-shot learning with imprinted weights. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018
Siyuan Qiao, Chenxi Liu, Wei Shen, and Alan L. Yuille. Few-shot image recognition by predicting parameters from activations. In The IEEE Conference on Computer Vision and Pattern Recogni- fion (CVPR), June 2018.
Aravind Rajeswaran, Chelsea Finn, Sham Kakade, and Sergey Levine. Meta-learning with implicit gradients. In H. Wallach, H. Larochelle, A. Beygelzimer, F. dAlché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems 32, pp. 124. Curran Associates, Inc., 2019. URL http:: nips cc/paper////pcpc 8306 1166-meta--1 htt.a-alaaaannnaaattiiiiniiiiiee
Andrei A. Rusu, Dushyant Rao, Jakub Sygnowski, Oriol Vinyals, Razvan Pascanu, Simon Osindero, and Raia Hadsell. Meta-learning with latent embedding optimization. In International Confer- on Learning Representations, 2019. URL Ttttps://openreett net www.:nnun//odiidee..ooo BJck1hAcK7
Ruslan Salakhutdinov and Geoff Hinton. Learning nonlinear embedding by preserving class neigh- bourhood structure. In Marina Meila and Xiaotong Shen (eds.). Proceedings of the Eleventh In- ternational Conference Artificial Intelligence and Statistics, volume of Proceedings of Ma- chine Learning Research, pp. 419, San Juan, Puerto Rico, 21-24 Mar 2007. PMLR. URL http:://proceedings... wtw.://ssss.p....ss..e.m. wrw.s v2. www.t// akwutdi 2 html

---

Under review as a conference paper at ICLR 2021
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learn- ing. In Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vish- wanathan, and R. Gamett (eds.), Advances in Neural Information Processing Systems 30, pp- 4077-4087. Curran Associates, Inc., 2017. URL http:://apper::e nips. cc cc/paper/ 6996-protatypical wnwpr:oosss---sttts------------- for fww-fewwwstee
Qianru Sun, Yaoyao Liu, Tat-Seng Chua, and Bernt Schiele. Meta-transfer learning for few-shot learning. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
Flood Sung, Yongxin Yang. Zhang, Tao Xiang, Philip H.S. Torr, and Timothy M. Hospedales. Learning compare: Relation network for few-shot learning, In The IEEE Conference Com- puter Vision and Pattern Recognition (CVPR). June 2018.
Eleni Triantafilluu, Tyler Zhu, Vincent Dumoulin, Pascal Lamblin, Utku Evci, Kelvin Xu, Ross Goroshin, Carles Gelada, Kevin Swersky, Pierre-Antoine Manzagol, and Hugo Larochelle. Meta- dataset: A dataset of datasets for learning to learn from few examples. In International Confer- ence on Learning Representations, 2020. URL hettps://openzewee net forum?id=
rkcAGAVKPT
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, koray kavukcuoglu, and Daan Wierstra. Matching networks for one shot learning. D. D. Lee, M. Sugiyama, U. V. Luxburg, Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems 29, pp. 3630-3638. Curran Associates, Inc., 2016. URL http::/papers.... .nw.snn.pp.. wiw.s cc/paper/ 11r8smmmttcccciinaeetttt-------- leaehottlerriingg....
A APPENDIX
A.I PREPROCESS PIPELINE
Table Preprocess pipeline
FUNCTION
PARAMETER
RandomSizedCrop RandomHorizzmtalFiii
image size
Normalize Mean
(0.485, 0,456, 0.406) (0.229, 0.224, 0.225)
Normalize Std
A.2 PEFORMANCE EVALUATION ON FC AND CIFAR-FS
10

---

Under review as a conference paper at ICLR 2021
Table 3: 5-way 5-shot and 5-way 1-shot performance on CIFAR-FS.
BACKBONE
1 SHOT
SHOT
MAML
Conv4
58.9 + 1.9
71.5 + 1.0
Prototypical Networks
Conv4
55.5
72.0
Relation Networks
64-96- 128-256
55.0 1.0
69.3
R2D2
96-192-384-512
6.2
0.14
MetaOptNet
ResNet12
72.8
85.0
Prototypical Networks (Pre)
Conv4
0.8
74.19 0.73
Prototypical Networks (Pre)
ResNet10
63.45
81.62 0.65
Prototypical Networks (Pre)
ResNet12
67.58
83.25 0.64
Prototypical Networks (Pre)
ResNet18
65.65 0.92
82.80 0.62
RP-Proto (euclidean) (C: -1)
Conv4
54.66
75.16 0.75
RP-Proto (cuclidean) (C: -))
ResNet10
66.20 0.93
83.77 0.69
RP-Proto (euclidean) (C: -3)
ResNet12
67.52 0.93
83.30 0.63
RP Pr-Proto (cuclidean) (C: 33)
ResNet18
65.34 + 0.90
82.98 + 0.68
Table 4: 5-way 5-shot and 5-way 1-shot performance on fc100.
BACKBONE
1 SHOT
5 SHOT
Prototypical Networks
Conv4
35.3 4: 0.6
48.6 + 0.6
TADAM
ResNet12
40.1 0.4
56.1 + 0.4
MetaOptNet
ResNet12
47.2 0.6
62.5 + 0.6
Prototypical Networks (Pre)
Conv4
35.63 0.72
51.80 + 0.73
Prototypical Networks (Pre)
ResNet10
38.55 + 0.70
57.71 + 0.71
Prototypical Networks (Pre)
ResNet12
39.48 0.66
$7.1&40.77
Prototypical Networks (Pre)
ResNet18
39.02 + 0.69
56.57 + 0.73
RP-Proto (cuclidean) (C: (1.5)
Conv4
37.36 0.68
53.91 + 0.70
RP- RP-Proto (euclidean) (1-shot C: -1- (5-shot C: -2)
ResNet10
39.03 0.69
57.96 0.71
RP- RP-Proto (euclidean) (C:-3)
ResNet12
39.73 + 0.69
56.11
- 0.72
RP- PPoProto (euclidean) (C: -3)
ResNet18
38.88 0.67
56.93 0.73
11


Reviews:
[{"review_id": "t04Q6lueCUt", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies how a regularization loss that measures the distance between feature $x$ and learnable class embedding $W_y$ helps meta-learning. The authors show that the SNN loss is lower when the regularization loss is in use in the meta-training phase, which supports the authors' conjecture that this regularization helps disentanglement of the penultimate layer in the backbone network, thus enhancing episodic learning. \n\nPros:\n1. The method is very neat and easy to implement\n2. The experiment results are mostly in favor of the authors' claims\n\nQuestions:\n\n1. Not sure if the term \"pre-training\" is proper here. The main focus of this paper, is the regularization term $l_{reg}$. Why don't tell the audience about this in a more straightforward way? \n2. Should it be $N$-way $K$-shot, why $1 \\leq y_i\\leq \\mathbf{K}$? I believe there's a typo. \n3. In Figure 1, all of the three lines are of some sort of V-shaped, i.e. they grew up a bit after reaching minima in the middle. Do you think might be the reason for this phenomenon? ", "extra_scores": {}, "extra_text": {}}, {"review_id": "p1yz5JpsH3Z", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Paper summary\n\nIn this paper the authors provide a summary of the role of pre-training in meta-learning. They investigate the performance implications of pretraining for a common pretraining method, prototypical networks, and propose an additional regularization loss to improve the generalization of pre-training. They evaluate the proposed method on miniimagenet and cifar100\n\n---------------------------------------------------------------------------------------------------------------------------\nPositives and negatives\n\n+ The summary of the pretraining role in meta-learning is accurate, and the proposed idea of using soft-nearest neighbor as a regularization term in pretraining is novel.\n+ The method is simple to understand and implement.\n+ The experiments section is well documented.\n+ I like the algorithm explanation of the method.\n- The paper should mention that pre-training in meta-learning is not at all a novel idea, many papers from couple of years ago use pretraining to initialize the weights of their convolutional backbone (see for example LEO, which is cited by the authors). Therefore the authors should make it clear that the only novel idea here is the use of Soft nearest neighbor loss as a regularization.\nGiven the above and if we look at table 1, we can see that the soft nearest neighbor loss does not actually improve the accuracy compared to simple pretraining for larger backbones. Therefore I would consider the significance of this result very small, and therefore further study should be done to see if it helps under different conditions (e.g. different datasets, more limited data regimes, etc).\n- Given the small size of the datasets under consideration, I am surprised the experiments section is so weak. I would expect to see experiments for more datasets and more meta-learning methods, e.g. maml, or matching networks, or even simple linear fit on the embeddings). For an example of a more thorough set of experiments, refer to https://arxiv.org/abs/1910.01319, which also looks at pretraining in the context of meta-learning but examines many more different backbone architectures, meta-learning algorithms, and datasets.\n\n---------------------------------------------------------------------------------------------------------------------------\nRecommendation\n\nAs it stands this paper requires a significant rewrite to meet the standards of this conference. Therefore I recommend this paper be rejected. The following aspects should be improved: the paper should be clear that the novel contribution is the soft-nearest neighbor regularization, and therefore the major focus of the experiments section should be on ablating this loss and testing it extensively under different conditions (e.g. more datasets, different meta-learning methods, low data regime, etc). Under the tested conditions, the proposed soft-nearest neighbor regularization does not seem to help much. The paper also should be thoroughly reviewed and rewritten as it was very hard to follow and had several sentences which are not possible to understand (a few examples below).\n\n---------------------------------------------------------------------------------------------------------------------------\nQuestions\n\n* Figure 1, what is the scale of SNN loss? From the y-scale it looks like it varies from 11.5 to 11.44. I have no idea whether this is basically just noise. Would be good to explain either in the caption or figure what is an SNN value for fully disentangled and fully entangled data.\n* I didn’t understand what the authors mean by the “parallelism property”. Section 2.4. Could you elaborate further on it? The episodes in episodic training are also trained in a batched SDG setting so it would consider them “parallel” in a sense.\n* In the last sentence of 2.4 the authors say “for shallow and deep backbones, it increases performance”, however from table 1 I read that it only increases performance for shallow backbones. Could the authors justify this sentence?\n---------------------------------------------------------------------------------------------------------------------------\nFeedback (not related to the score)\n\n* Table 1, would be good to have an explanation of what is the hyperparameter C. The only mention of it is in Eq. 5 as the exponent of alpha, so it can be understood as the regularization but it would be nice to explain in both in the main text under eq. 5 as well as in the caption for Table 1.\n* In general I would recommend the authors write more in the figure captions. For example figure 3 and figure 4 are very hard to understand, and a few sentences explaining the point of the figure would really help the reader.\n* To improve the strength of the paper, I would suggest focusing on the disentanglement properties of the SNN regularization and, as proposed above, explore it under a lot more settings. In addition, I would look at how this SNN regularization affects domain transfer as I suspect it could actually be more useful in that case (e.g. pretrain on miniimagenet, test on CIFAR or even MNIST)\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "MRSCzo5gcTW", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\n========\nThis paper investigates the role of pre-training as an initialization for meta-learning for few-shot classification. In particular, they look at the extent to which the pre-trained representations are disentangled with respect to the class labels. They hypothesize that this disentanglement property of those representations is responsible for their utility as the starting point for meta-learning. Motivated by this, they design a regularizer to be used during the pre-training phase to encourage this disentanglement to be even more prominent with the hope that this pre-trained solution is now closer to the optimal one, thus requiring less additional episodic training which is time-consuming. They show experimentally that their modified pre-training phase sometimes leads to better results as an initialization for Prototypical Networks compared to the standard pre-trained solution, and sometimes converges faster.\n\nPros\n====\nThe topic of study of this paper is very interesting. I definitely agree that the role of each of the pre-training and meta-learning phases are not yet well-understood, and making progress on understanding this will shed light on the most promising directions for few-shot classification.\n\nAlso, using the Soft-Nearest-Neighbor-Loss is an interesting property to measure (and to try and reinforce) in the pre-trained representations.\n\nCons\n====\n[A] The biggest weakness of this work, in my opinion, is the lack of connection with previous work that is very similar. Specifically, the property that is referred to in this paper as ‘disentanglement’ is very related to previous notions that have been studied in this context of pre-training for few-shot learning. Specifically, [2] argued that the success of the cosine classifier during pre-training (compared to a standard classifier) is due to explicitly minimizing the intra-class variance of each class (which leads to better clustering, and to better “‘disentanglement” in the way that the term is used in this paper).\n\nPushing that direction further, [1] proposed regularizers whose purpose is to directly encourage the pre-training phase to have this property of better clustering: minimizing the intra-class variance and maximizing the inter-class variance. This paper should be discussed as related work and should be compared to experimentally since their approach is very similar to the one proposed here.\n\n[B] Unfortunately I also found the writing to be of poor quality. Some minor grammatical or wording errors did not distract me too much from understanding the intended meaning, but there were certain statements which I found hard to understand, or disagreed with. Some examples are below:\n\n“Due to episodic training, meta-learning methods generalize better than traditional transfer-like methods for the novel classes”. The jury is actually still out on this, so I don’t think it’s appropriate to make this claim. Better generalization was the motivation of episodic models, indeed, but in practice non-episodic approaches have been shown to perform quite well, as pointed out in this paper too.\n\n“Episodic sampling is time-consuming”. Can you explain why that is? I don’t disagree (based on my experience too) but I don’t believe it is obvious, and it would be useful to explain this.\n\n“In the previous section, we conclude that the last layer in the backbone would be more disentangled after episodic training”. It’s unclear to me how that conclusion follows.\n\n[C] Another weakness is that the proposed method does not perform too strongly compared to the baselines / previous methods. It seems that the gain is larger for smaller architectures, which is in line with the observation in [2] that minimizing intra-task variance is most beneficial for small backbones. On the other hand, for larger architectures, RP-Proto is not better than plain proto (notice the overlap in the confidence intervals in the respective entries of Table 1). \n\n[D] I also found the experiments to be weak in terms of the analysis of disentanglement during pre-training and episodic training. Contrary to the authors’ observation, it doesn’t really seem to me that the disentanglement loss is going down too much during episodic training by looking at Figure 1. The conv and resnet18 curves are mostly flat. The resnet10 one does go down noticeably but then starts going up again. I’m also not sure what causes the large discrepancy in the behavior of the resnet10 curve compared to the other two? My initial thought was network capacity, but resnet 10’s capacity is in between that of the other two networks if I understand correctly, so it’s hard to draw a conclusion there.\n\nOverall\n======\nI vote for rejection of this paper in its current form, mostly due to the missing comparison with the very related method mentioned above, the quality of the writing and the weakness of the experimental results, as described above.\n\nSuggestion for additional experiments\n============================\nTable 1 shows that MetaOptNet outperforms RP-Proto (I’m looking at the entry of RP-Proto with the ResNet12 backbone for an apples-to-apples comparison with MetaOptNet). I would be curious to see an RP-MetaOptNet variant too. More generally, does the proposed regularizer also lead to improvements in episodic approaches that are closer to state-of-the-art compared to Prototypical Networks?\n\nFurther, as an additional data point, it would be useful to also report the performance of the pre-trained network itself on the few-shot test tasks (without a meta-learning phase at all). For an apples-to-apples comparison with the reported Prototypical Network variants, Prototypical Networks can be used to solve each test task still, but operating directly on top of the representation learned from pre-training, instead of the representation produced by the episodic phase.\n\nAdditional comments for fixing minor issues and improving clarity\n===============================================\nBelow are a few more recommendations and singled-out sentences that I think should be re-written to improve clarity.\n\nIn the “Mixed Framework” section, [3] should also be cited among the papers that used a pre-trained solution as the initialization for the meta-learning stage as this is how the meta-learners in that paper were trained as well.\n\nThe provided reference for fo-MAML is incorrect. fo-MAML was actually introduced in the original MAML paper (Finn et al, 2017). The provided (Nichol et al., 2018) reference introduced Reptile, which is similar but not the same as fo-MAML.\n\n“Episodic training is key to make meta-learning prominent”. I’m not sure what this sentence means. Are there other ways of meta-learning in this context without episodic training?\n\nThe description of optimization-based meta-learning: “[...] try to get an embedding that could easily fit subtasks by adding some extra layers”. This is not entirely accurate. MAML, for instance, does not add any extra layers per task. Instead, the entire network is rapidly fine-tuned within each task as well as meta-learned across tasks.\n\nReferences\n=========\n[1] Unraveling Meta-Learning: Understanding Feature Representations for Few-Shot Tasks. Goldblum et al. ICML 2020.\n[2] A Closer Look at Few-shot Classification. Chen et al. ICLR 2019.\n[3] Meta-Dataset: A Dataset of Datasets for Learning to Learn from Few Examples. Triantafillou et al. ICLR 2020.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "1lbfco9UIL", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "#### Summary\n\nThe submission attempts to understand the role of episodic fine-tuning in a few-shot classification context.\n\nUsing Prototypical Networks as a case study, the authors measure the entanglement of class representations (using a soft nearest neighbour measure as was done by Frosst et al. (2019) in the supervised learning case) during episodic fine-tuning for several backbone architectures fine-tuned on 5-way 5-shot mini-ImageNet episodes. Based on these observations, the paper concludes that episodic fine-tuning tends to decrease entanglement in the penultimate layer and proposes to add a regularization term to the supervised pre-training phase which encourages disentanglement in that layer.\n\nResults are presented on the mini-ImageNet benchmark and the proposed approach is claimed to perform on-par with or better than competing approaches. Accuracy curves are also shown to support the claim that the proposed approach requires training on less episodes.\n\n#### Strengths and weaknesses\n\n* **+** The paper’s topic is very relevant to questions being raised in the recent literature regarding the differences between supervised and episodic training.\n* **+** The premise of studying the role of episodic fine-tuning from the perspective of feature (dis)entanglement is an interesting application of Frosst et al. (2019)’s work to the few-shot classification setting.\n* **-** Vague or inconsistent use of terminology.\n* **-** Poor writing and presentation.\n* **-** Performance of the proposed approach is not competitive with competing approaches, contrary to what's claimed in the submission.\n\n#### Recommendation\n\nI recommend rejection. While the paper’s premise is interesting, the submission suffers from poor writing and presentation quality, and I’m not entirely convinced by the claimed causal relationship between representation (dis)entanglement and episodic fine-tuning.\n\n#### Detailed justification\n\nMy main concerns have to do with the presentation of the results and the interpretation of episodic fine-tuning as a way to decrease the representation’s entanglement.\n\nThe list of competing approaches in Table 1 is incomplete and outdated. For instance, Tian et al. (2020)’s \"Rethinking Few-Shot Image Classification: a Good Embedding Is All You Need?\" obtains around 64.8% on mini-ImageNet 5-way 1-shot using a ResNet-12 architecture. Tian et al. (2020) also lists several approaches using a Conv4 backbone that achieve a mini-ImageNet 5-way 1-shot performance greater than 50.4%. I therefore disagree with the assertion that \"our method shows competitive performance to other methods\" and that \"when the backbone is shallow, we have outperformed all other methods with the same backbone\".\n\nThe paper hints at the fact that the Euclidean metric used by Prototypical Networks doesn’t work well with highly entangled representations. Another possible interpretation is that the backbone is pre-trained using a linear output layer, which computes the inner-product between the representation and class weight vectors, and that the squared distance computed by the Euclidean metric is not well suited to the resulting representation. According to that interpretation, part of what episodic fine-tuning does is correct for this mismatch in metrics between (meta-)training and (meta-)testing. I think the paper’s interpretation would be more convincing if the pre-trained backbone used a quadratic output layer (i.e. the logits are computed as the negative squared distance between the representations and class weight vectors) and therefore controlled for metric mismatch.\n\nI have additional issues with the way in which results are presented:\n\n* Section 3.3 mentions a result presented in Section 4.4, then Section 3.4 begins by stating that the previous section concludes that the last layer in the backbone is more disentangled after episodic training. This means that in order to have the proper context to understand Section 3.4, the reader needs to jump forward and read Section 4.4. I recommend changing the order of the presentation so that it is more linear.\n* In Table 1, 95% confidence intervals are provided, but the absence of identification of the best-performing approach(es) in each setting makes it hard to draw high-level conclusions at a glance. I would suggest bolding the best accuracy in each column along with all other entries for which a 95% confidence interval test on the difference between the means is inconclusive in determining that the difference is significant.\n\nOverall, the submission could benefit from another round of careful proofreading. It contains several grammar mistakes which, while they do not significantly compromise clarity, make reading the paper harder than it should be. Examples include:\n\n* \"[...] and crafted the hard episode to make necessary episodes fewer.\"\n* \"Disentanglement is the property whether the data-points [...]\"\n* \"Benefited by the understanding, [...]\"\n\nI also noted a few false or unsupported statements:\n\n* \"Due to episodic training, meta-learning methods generalize better than traditional transfer-like methods for the novel classes.\" Can the authors expand on this? What do they mean by \"generalize\"? Do they refer to test classes from the same domain (e.g. mini-ImageNet test classes), or test classes from different domains (i.e. cross-domain generalization)? I don’t see the statement as generally accepted, especially given the many recent papers that show strong performance with well-tuned transfer learning baselines.\n* \"Soft-Nearest Neighbour loss [...] is proposed by Frosst et al. (2019)\" Frosst et al. credits Salakhutdinov and Hinton (2007) for the soft nearest neighbour loss, which itself draws inspiration from Goldberger et al. (2005)’s Neighbourhood Component Analysis.\n* \"Though the split is quite naive, the afterward episodic learning shows promising improvement.\" Can the authors point to work that provides empirical evidence for this statement?\n\nFinally, the submission’s use of terminology is vague or inconsistent at times:\n\n* The categorization of approaches as either \"supervised pre-training\" or \"meta few-shot learning\" feels incomplete to me. While some approaches (most recently Meta-Baseline and Meta-Dataset’s few-shot learners) do perform supervised pre-training followed by episodic fine-tuning, most well-known approaches such as Matching Networks, Prototypical Networks, MAML, etc. do not prescribe a supervised pre-training phase.\n* The term \"meta few-shot learning\" is not widely used in the literature and appears to be introduced in this paper as far as I can tell. It’s defined in the introduction to be the combination of supervised pre-training and episodic fine-tuning, but it’s also used in Section 3.2 to categorize Prototypical Networks, whose formulation does *not* prescribe a supervised pre-training phase.\n\n#### Questions\n\n1. What do the authors mean when they state that \"[due] to the parallelism property, normal training literature is much faster than episodic training\"? Isn’t the forward propagation through the embedding function in a few-shot learner such as Prototypical Networks just as parallelizable as the forward propagation in a supervised classifier?\n1. The submission claims that the proposed approach alleviates the burden of training episodically on a large number of episodes. Given the performance of well-tuned supervised baselines, why should we perform episodic fine-tuning? Shouldn’t it be sufficient to use the pre-trained backbone as-is?\n1. The submission follows Chen et al. (2020) for the pre-training and episodic fine-tuning procedure but doesn’t compare against it in Table 1. How come?\n\n#### Additional feedback\n\n1. This is arguably inconsequential, but the paper motivates few-shot learning using bird classification as an example, stating that \"an ornithologist typically can only obtain a few pictures per bird species\". I don’t know if I agree that this is typically the case: looking at the iNaturalist online database, thousands of pictures can be found for a large number of bird species.\n1. The related work section appears to contain mostly pre-2020 references and does not mention recent work such as (Simple) CNAPs, SUR, CrossTransformers, etc.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: There is value in analyzing pre-training for few-shot learning, and the observation that improved disentanglement might lead to better initialization schemes for few-shot learners is worth exploring. However, in its current state, the reviewers do not think the paper is ready for publication. Specifically, work needs to be done to improve the clarity, comparison to related work, and experimental analysis.

Decision: Reject