A Study on Representation Transfer for Few-Shot Learning
Authors: Chun-Nam Yu
Yi Xie
Venue: ICLR 2022 Submitted (2022)
Keywords: few-shot learning
transfer learning

Abstract: Few-shot classification aims to learn to classify new object categories well using only a few labeled examples. Transfering feature representations from other models is a popular approach for solving few-shot classification problems.In this work we perform a systematic study of various feature representations for few-shot classification, including representations learned from MAML, supervised classification, and several common self-supervised tasks. We find that learning from more complex tasks tend to give better representations for few-shot classification, and thus we propose the use of representations learned from multiple tasks for few-shot classification. Coupled with new tricks on feature selection and voting to handle the issue of small sample size, our direct transfer learning method offers performance comparable to state-of-art on several benchmark datasets.  


Under review as a conference paper at ICLR 2022
A STUDY ON REPRESENTATION TRANSFER FOR FEW-SHOT LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Few-shot classification aims to learn classify new object categories well using only a few labeled examples. Transfering feature representations from other mod- els is a popular approach for solving few-shot classification problems. In this work we perform a systematic study of various feature representations for few-shot clas- sification, including representations learned from MAML, supervised classifica- tion, and several common self--upprvised tasks. We find that learning from more complex tasks tend to give better representations for few-shot classification, and thus we propose the use of representations learned from multiple tasks for few- shot classification. Coupled with new tricks on feature selection and voting to handle the issue of small sample size, our direct transfer learning method offers performance comparable to state-of-art on several benchmark datasets.
INTRODUCTION
Few-shot lear)ing(FSI.) aims train a prediction model with very few labeled examples. It is motivated both by our interest in the ability of humans to learn new concepts with very few exam- ples (Carey & Bartleti, (01I), and practical need build machine learning models without the expensive process of collecting a large amount of labeled training data. In typical few-shot learning settings we assume we have access a large amount of data (labeled or unlabeled) from a source domain as prior knowledge, and small amount of labeled data for a target task in a different domain.
Popular approaches for the few-shot learning problem include meta-learning and transfer learning. Meta-learning creates learning tasks that are similar to the target task from the source domain data, and tries to learn a model that can quickly adapt to new task (Finn et al. 2017 Ren et al., 2018 Wang et al. 2019). Transfer learning, on the other hand, transfers models learned from the source domain data to the target task through different adaptation techniques. In this work we focus on transfer learning through representations, which aims to learn good feature representation from source data so that when learning the target task we can focus on training a much simpler task model (e.g. a linear classifier) instead of having to learn feature extraction at the same time. In computer vision and natural language processing, pre-training with large amount of data has become the dominant method for building state-of-art models (Iglovikov & Shvets 2018 Li et al. 2018: Cuil eet al., 22188; Ruder et al., 2019). For few-shot classification, recent works (Tian et al. (0200) show that even tre-training with simple classification task in the source domain gives good feature extractor for training few-shot models.
In this work we attempt systematically study different representations for few-shot learning to identify the good ones, including representations from common supervised and self-supervised pre- training tasks (Gidaris al.) 2018 Doersch et al. 2015 He al. 2020 Chen et al.] (220a)), and also representations from the meta-learning method MAML et al. 2017) by treating it feature extractor. Compared to typical evaluations of self-supervised representation learning which usually involve the same data distribution (train and test on same input data but for different tasks), few-shot learning differ two important aspects. The first one is data shift: the training and test data have different distribution and usually contain different object categories. The second one is extreme small sample size compared to typical evaluations of transfer learning. These differences make a proper study of these different representations worthwhile.
In addition to learning good representations from source data, we believe there should also be em- phasis how to use them properly in few-shot settings. Representations learned from large

---

Under review as a conference paper at ICLR 2022
amount of pre-training data usually high- dighessiensi,nl,,, and for some few-shot learning task (e.g,, 5-way 1-shot classification) it is difficult to figure out which features are directly related to the target concept given only handful of examples. In this work also investigate ways to perform feature selection and noise reduction when adapting to the target task, given a good representation learned.
In summary, our contributions in this work include: 1) We study systematically different represen- tations for few-shot classification, to understand why some representations work better than others. 2) We investigate the use of multi-task feature representations in few-shot learning and show their utility. 3) We introduce feature selection via training with auxiliary classes, and a voting scheme that follows naturally from multi-tash prediction to improve few-shot learning.
RELATED WORKS
Motivated by bridging the gap between Al and human learning ability, we have seen an ever in- creasing attention in the area of few-shot learning in machine learning community, Model Agnostic Meta-Learning (MAML) was proposed by Finn et al.] (2017); aims to search for proper ini- tialization of the neural network that can be rapidly adapted to variety of new tasks with few optimization steps. Many improvements have been proposed based on MAML including first-order approximations (Nichol et al. (0188) incorporating task-specific information (Lee & Choil 2018). etc. Inspired by the concept of "learning to learn", many metric-learning methods are also developed via meta-learning. By simulating the seo-shot task setting while training the backbone model, these
metrie-lcarning models (i.e Prototypical Networks, Relation Networks) (Snell et al.) 2017: Sung
al. 2018) aim learn the best feature representation which can be generalized well to unseen novel tasks.
Our work is motivated by the recent studies Chen al. (2019) and Tian al. 20200) which show that pre-training a good feature extractor (e.g., using supervised classification) gives very good per- formance on few-shot classification. We continue along this line of study investigate which pre- training tasks give the best feature extractor for few-shot classification. Our study of MAML as a feature extractor is motivated by the work in Raghu al. (2019). which show that MAML in few-shot learning benefits more from the learned features rather than fast model adaptation.
Gidaris al. (2019) and [Su et al. (2020) study the use of self-supervised tasks including rota- & Favaro (016) to improve few-shot learning. They use the self- celf-supevvised tasks as feature regularizers on top of meta-learning algorithms such as Pro- toNet(Snell et al. 2017) and MAML et al. (017)) which is different from our direct joint representation learning followed by transfer. Our approach also closely related Doersch & Zisserman (2077) which consider jointly using multiple self-supervised tasks to learn visual repre- sentations for classification. Zhhi et al. (2019) has done a large scale study of the performance of representation learning large set of downstream tasks, while focus few-shot classification in this work.
Gidaris et al.) 2018) and
3 REPRESENTATION TRANSFER FOR FEW-SHOT LEARNING
In all the following experiments we adopt the following protocol to investigate transfer learning. We first train feature representation using
multiple tasks the source domain data, and use the last layer of activations as our feature representation. During adaptation stage to the new few- shot domain, we train a linear logistic regression classifier top of the fixed feature representation to adapt the model. This is a common protocol in many evaluations of representation learning methods. In this work we focus on two backbones: ConvNet4 and ResNet12. ConvNet4 is a 4-layer convolutional neural network with hidden layer size of 64 in each layer, where each layer is a 3x3 convolution followed by ReLU, batchnorm, and max- max-pooling poolppooing of stride ResNet12 network (He et al.] 2016) with 4 residual blocks of width 64- 160-320---40) We also consider wider version of ResNet12 2 which we denote as ResNetW112 by increasing the width to 128-320-640-640 to better accommodate multi-task learning. The feature representation size is fixed at 640. Our code is implemented in PyTorch (Paszke et al.) 2019). We focus on the two most commonly used datasets in studying few-shot learning, Mini-ImageNNer ImageNet Vinyals et al., 2016)) and Tierd- ImageNet(Ren al.) 2018). in the work below. We found that there are two ways to construct the 84x84 inputs of these datasets from ImageNet (see Appendix A.T). We use the approach of directly taking random
residual
2

---

Under review as a conference paper at ICLR 2022
resized crops of 84x84 on the ImageNet images, as it allows to experiment more diverse set of representation learning methods.
3.1 REPRESENTATIONS FROM META-LEARNING
We begin our study of representation transfer for few-shot learning with feature representations produced by MAML. Previous studies Raghu et al. (2019) show that the benefits of MAML come mainly from the feature representation learned in the last layer of the neural network instead of the ability to do fast adaptation. In this section we want to go further and consider the question: from a representation transfer point of view, what source tasks are the best for target distribution of few-shot learning tasks? Is learning from a set of 5-way 1-shot source tasks always the best if we want to perform 5-way 1-shot classification the target domain?
We use modified version of MAML algorithm(Algoritmm) D that only adapts the last layer of the neural network on the support set, as adapting the last layer only performs very close to adapting the whole network (the almosi-no-inner-loopl (ANNAA) algorithm in (Raghu et al.) 2019)). We split the parameters of the neural network into weights for the linear layer & and the rest of the earlier feature extraction layers 8, and write (x(f((z)) for the output of network with input We use (X:5. Y X². y,?) denoting the support set inputs, support set labels, query set inputs, query set labels to represent the sampled few-shot classification task T;. Note that only the linear layer param- eters e are updated in the inner loop using the support set, reflecting that we only update the linear layer during adaptation. Parameters for feature representation @ are only updated in the outer loop.
Algorithm Modified MAML for few-shot classification that adapts only the last layer (ANIL)
Given: inner and outer step size hyperparameters a, B, task distribution Randomly initialize 0, Φ while not done do Sample batch tasks T P(T)
for all T, = (X Y X r, do Update o - Ø- a a ol y,s)
end for
Update 0 - 0 V ET, (Jo y,)) σ - BV Et. (go x.)
end while
We train 5-way I-shot(5Wlss, 5-way 5-shot(5w.ss) 10-way 1-shot(10w1s) and 10-way 5- shot( 10w5s) tasks on Mini- Image-t and fiered-Imageni using both the ConvNet4 and ResNet12 architecture. To evaluate the representations learned, we extract the last layer features from these n-way k-shot networks, and train a logistic regression model on top of it during the adaptation stage on the support set. Each accuracy number reported in Tables I and 2 are the median of 5 trials, each of which is the average of 600 random draws of n-way k-shot tasks.
The MAML algorithm is implemented using the higher package (Grefenstette al. 2019 We use an outer loop step size of 0.001 and inner step size of 0.01 for 5-way tasks and 0.05 for 10-way tasks. There are 5 inner loop steps during training and 10 inner loop steps during evaluations. We use task batch size of 4 and run for 400 epochs. Due to the occassional overfitting of ResNet12 models, we select the model that performs best in terms of loss on the validation set provided in Mini-ImageNNet and Tiered-Imagent
and report the results on the test set.
From the results on ConvNet4 Table we can see that contrary to intuition, training and testing on the same type of n-way k-shot tasks do not result in the best performance. In general, representations learned from 10-way tasks perform better than representations learned from 5-way tasks, irrespective of the number of ways in the target tasks. And learning with more shots are also usually better. The results for ResNet12 are similar (Table 2) except for the 5w1s outlier in Mini ImageNe.. [neel] et al. (217)) has previously noted the effect of models trained on larger number of ways performing better than models trained on number of ways matching the target task, while Cao et al. (2019) and Triantafillou al. (2020) have studied the effect of number of shots in few-shot learning.
3

---

Under review as a conference paper at ICLR 2022
Table 1: Few-shot classification accuracies of different MAML source representations based on
ConvNet4 on Mini-ImageNet and Tiered-Imagel
Mini ImageNet
Tiered- Imageeet
source rep./target task
5w1s
5w5s
10w1s
10w5s
5w1s
5w5s
10w1s
10w5s
49.25
62.30
32.90
46.20
48,95
62.71
33.71
47.23
ConvNet4-5w1s
(0.76)
(0.67)
(0.43)
(0.42)
(0.89)
(0.77)
(0.56)
(0.52)
51.07
64,88
34.85
49.20
51.91
66.29
36,52
51.12
ConvNet4-5w5s
(0.79)
(0.72)
(0.47)
(0.44)
(0.84)
(0.76)
(0.58)
(0.56)
51.68
64.04
35.12
47.94
50.98
65.38
35.55
49.69
ConwNet4-1Ow1s
(0.82)
(0.73)
(0.45)
(0.43)
(0.89)
(0.77)
(0.54)
(0.52)
49.71
62.57
33.57
48.66
52.54
67.02
37.11
52.21
ConvNet4-10w.5s
(0.82)
(0.73)
(0.45)
(0.43)
(0.86)
(0.78)
(0.58)
(0.55)
Table 2: Few-shot classification accuracies of different MAML source representations based on ResNet12 on Mini-ImageNer and Tiered-ImageNet.
Mini- ImaieemageNet
source rep./target task
5w1s
5w5s
10w1s
10w5s
5w1s
5w5s
10w1s
10w5s
57.26
69.51
39.47
52.83
60.77
72.87
44.33
58.58
ResWet12-5w1s
(0.84)
(0.66)
(0.51)
(0.46)
(0.95)
(0.84)
(0.65)
(0.57)
53.78
67.28
36.63
51.31
61.93
75.18
46.00
60.89
Res5et12-5w.5
(0.88)
(0.70)
(0.51)
(0.45)
(0.98)
(0.75)
(0.67)
(0.58)
54.67
68.39
37.70
52.46
62.56
75.52
46.71
61.27
ResNet12-10w1s
(0.81)
(0.69)
(0.54)
(0.47)
(0.97)
(0.75)
(0.64)
(0.59)
53,86
67.61
36.83
51.92
63.98
77.47
48.78
64.62
ResNet12-10w5s
(0.84)
(0.67)
(0.52)
(0.44)
(0.97)
(0.78)
(0.68)
(0.57)
From an adaptation point of view, 5-way models should transfer better to 5-way tasks, since they come from a more 'similar' task distribution. However, from feature learning perspective, this is not surprising. More difficult pairing of classes like cat VS tiger are more common for higher way tasks in a random sampling setting, thus they force the models learn more discriminative features that might transfer better to new domains.
Note that our adaptation here is based on logistic regression, which is different from fast adaptation by performing few gradient steps in MAML using the linear weights learned in the last layer. The logistic regression results sometimes can be 1-2% lower than adaptation using MAML, since the feature representations are trained using MAML and the last layer linear weights also encode some prior information on what the adapted weights should be close to. However as we use logistic regression for all model adaptations the comparison should be fair, and the results indicate increasing the number of ways allow better representations to be learned for transfer.
3.2 REPRESENTATION FROM SUPERVISED AND SELF-SUPERVISED TASKS
In the last section we study the feature representations learned by MAML on few-shot learning tasks, by considering MAML. feature learning algorithm. In computer vision it is common to pre-train on ImageNet classification, and then adapt the representation learned to downstream tasks such as image segmentation or object detection (Iglovikov & Shvets 2018 Li al. 2018). In recent years there are also many works on using self-supervised tasks to pre-train computer vision and NLP models (Chen al. 2020a Baevski al. 2019: Lan et al.) 2019). The paper Tian al. (2020) showed that by just pre- training on the classification task using the source training set and adapting using logistic regression, they can obtain results in few-shot learning that are competitive with state- of-art algorithms. They also studied the use of distillation to improve the pre-training. In this section
we follow this line of investigation and study systematically the few-shot learning performance of different supervised and self-supervised tasks.
We train these classification models using a learning rate 0.05 for Mini-ImageNett and 0.01 for Tiered-ImageNe for 100 epochs using batch size of 64. The learning rates are decayed by factor of 10 at the 60th and 80th epochs. From the training partition of these two datasets, we further split the partition into training set (consisting of 90% of training partition) for training the classification models, and a validation (remaining 10% of partition) for evaluation and model selection. As we can from both Mini-ImageNe and Imaree- in Table 3 adapting from classification repre-
4

---

Under review as a conference paper at ICLR 2022
Table 3: Few-shot classification accuracies of classification representation compared MAMI..
Mini-ImageNet
Tiered- -maee-ee
source rep./target task
5w1s
5w5s
10w1s
10w5s
5w
5w5s
10w1s
10w5s
45.41
62.38
31.52
47.35
49.27
66.28
35.07
52.07
ConvNet4-class
(0.75)
(0.74)
(0.45)
(0.43)
(0.85)
(0.77)
(0.51)
(0.56)
48.72
64.89 84
35.87
47.26
48.77
65.65
35.19
51.30
ConvNet4-maml
(0.79)
(0.71)
(0.43)
(0.42)
(0.85)
(0.76)
(0.51)
(0.53)
61.75
78.98
47.09
66.93
71.36
85.56
58.15
75.77
ResNetW12-olass
(0.79)
(0.60)
(0.53)
(0.44)
(0.91)
(0.64)
(0.66)
(0.54)
57.74
65.97
37.79
51.94
60.47
74.69
45.08
63.49
ResNet12-maml
(0.76)
(0.67)
(0.47)
(0.44)
(0.84)
(0.81)
(0.52)
(0.52)
Table 4:
Few-shot classification accuracies of different supervuee source representations. Tiered-ImageNe
Mini-ImageNet
source rep./target task
5w1s
5w5s
10w Is
10w5s
5w1s
5w5s
10w1s
10w5s
34.61
48.60
22.10
34.05
35,40
46.87
22.10
31.24
ResNetW12-ro
(0.64)
(0.66)
(0.36)
(0.40)
(0.69)
(0.69)
(0.36)
(0.40)
32.86
44.93
19.92
29.40
26.17
33.39
14.75
20.66
ResNetW11-loc
(0.63)
(0.65)
(0.35)
(0.38)
(0.51)
(0.59)
(0.26)
(0.34)
46.63
64.83
33,56
50.75
52.63
72.00
39,26
59.37
ResNetW12-comtras
(0.75)
(0.66)
(0.48)
(0.47)
(0.86)
(0.72)
(0.58)
(0.58)
sentation using logistic regression does not beat training with MAML when we use the simpler CNN model ConvNet4. The main reason for this is the ConvNet4 model does not have sufficient capac- ity for the classification task (only 51% top-1 accuracy on Mini-Imagenet and 26% top-1 accuracy on ImageNN)), and thus the representation learned is not good enough for transfer. When we switch to the higher capacity ResNet12 model (81% top-1 for mini mini-ImageNNe and 70% top-1 for tiered-Imman Merred--maeeee),,, the transfer learning approach works much better than MAML, with few-shot learning accuracies close to many state-of-art methods (see comparison methods in Table IT) This consistent with the observations in Tian al. (2020), which we are reproducing here for compar- isons with other feature representations that follow. We believe the classification tasks (64 classes for Mini- Meni-lmaeeNet and 351 for (ereed-maaeeNet force the CNN models to learn richer and more stable feature representations than the 5-way 10-way MAML classification tasks. It is uncommon to sample more difficult pairs of classes (e.g. cat VS tigers) in 5-way or 10-way MAML setup based on random sampling, and these difficult pairs make the models learn more distinguishing features.
In recent years there have been many studies on using self-supervision to pre-train classification models to reduce the requirement of labeled data (Gidaris et al. 2018 Doersch et al.. (015): He et al., 2020) Chen et al. 2020a). Below we also consider several common self-supervision pre- training tasks on FSL, including rotation prediction, location prediction (related jigsaw), and contrastive learning, to see how well these self-supervised representations transfer to FSL tasks.
For rotation prediction (Gidaris et al.) 2018), we randomly rotate the 84x84 input image 0, 90, 180, or 270 degrees and use these rotation angles as class labels. For location prediction (Sun et al., 2019) we sample the input image at higher 168x168 resolution, and split the image into equal parts (top-left, top-right, lower-left, lower-right), and use the location of the split as class labels. For contrastive prediction, we follow the same implementation as in the simCLR paper (Chen al, 2020a), using an output embedding size of 1 128, and a batch size of 128. We tried larger batch sizes like 256 and 512 but this did not improve the results. These models are trained using step size of 0.05 and batch size 64 (apart from the contrastive models).
From Table we can see that the self-supervised tasks provide fairly good feature representations for few-shot Iew-shotcearning on lini-Nm and Tiered- ImageNet. The relative strength of each feature representation is consistent with previous results on self-supervised learning with ImageNet (He) et al. 2020), with rotation slightly better than location prediction and the more recent contrastive learning better than both of them. However, they are still not as good as direct transfer using the supervised classification training 4
To evaluate the properties of these sulf-supervised tasks further, we cross cvos-evalluate the representa- tions learned from one task the other, using the 10% validation set from the training partition
5

---

Under review as a conference paper at ICLR 2022
Table 5: Prediction accuracies for holdout-evaluation of different self-supervised representations.
Mini-ImageNet- holdout evaluation by task classification
train on/evalute on
rotation
location
classification
81.33
47.14
53.27
rotation location contrastive
30.16
84.64
59.00
22.79
43.13
79.80
49.59
50.42
53.38
(which has the same distribution these models trained on). We just take the representations from these three models, and the re-train the last layer of the model using logistic regression using the 90% training data from the training partition (exact same data on which the representations are trained on), and test on the remaining 10%. We omit the contrastive representation because the contrastive loss is dependent on batch size and not easily comparable. We can see from Table that the representations trained on one task usually perform reasonably on the other two, but is not competitive with models trained for that particular task. For example, the representation trained for classification does not perform particularly well for rotation (47.14%) and location (53.27%), both of which are 4-way classification problems. This motivates us to consider the question of whether train representations that excel in all these tasks, and whether such representations will be useful in few-shot learning.
4 MULTI-TASK REPRESENTATIONS
When training for feature representations separately, we minimize over the training data from the source domain the following objective functions:
Lasss (1re; 0, Webs), 1)
Leot (f (Trot (I:;; yi ); A, Wrot); yft)),
Liue (f (1x; y (oc); 0, wloc) y loc).
where Trut and The are the rotation and location-bann -based cropping transform, and Leks Lrot> Lioe the corresponding cross-entropy loss for supervised classification, rotation prediction, and location prediction respectively. The neural networks are represented by f(; w) with backbone parameters 0 and linear(head) layer weights w.
To combine these different learning tasks into shared neural network for joint training, major difficulty is the differences their Doersch & Zisserman (2017). A rotated or cropped input image can affect the learning of the classification task if the classification task is jointly trained with the rotation or location prediction task. Nevertheless, we consider jointly training the classification task with rotation or location prediction tasks with the transformed inputs:
" 1=1 (Lebs (f (T:: V. ); Wets), v1 Arot (f ( vi 1. Wrot), ), ()o)))
En (f (or); A, webs), v( + Atoc (f yo 0, Wloc), ()))),
where Arot and Aioc control the relative weights of the tasks. The backbone parameters are shared among different tasks, while the weights Welss Wrot, Whoe are task-specific heads.
We also consider jointly training all three tasks, based on the composed transformation Tioctrots cropping based location followed by random rotation.
Σn. (Irttessaa' Wels), vi Aroe Torok (f 0, Wrot), vi 10:0)) + Mioe Taoe (f 0, woo). (oo)).)o))) 100))
where 2. = TTocc-rot rot y!
We simply fix Arot and Aloe at 1 (all three tasks have equal weights). We find that training with all versions of rotation location crop in the same mini-batch works slightly better than domly sampling one of them, although this can reduce the effective batch size for the supervised classification task. For Imag-Ntt adopt this approach of taking all 4 copies of rotation or

---

Under review as a conference paper at ICLR 2022
Table 6: Prediction accuracies for holdout-evaluation of different multi-task representations.
Mini-ImageNett holdout evaluation by task classification
train on/evalute on
rotation
location
CeeNNetW12-clsttee
81.59
86.35
60.75
DesSNeWW12-csstlese
75.78
51.82
80.03
ResNetW112-lsttnes ResNeaW12clssrottlloss
80.86
52.37
75.21
78.31
85.94
72.74
Table 7: Few-shot classification accuracies of different multi-task trained representations.
Mini-ImageNNet
Ticred-ImageNet
source rep//arget task
5wls
5w5s
10wls
10w5s
5w1s
5w5s
10wls
10w5s
ResNetWW12
61.75
78.98
47.09
66.93
71.36
85.56
58.15
75.77
-cls
(0.79)
(0.60)
(0.53)
(0.44)
(0.91)
(0.64)
(0.66)
(0.54)
ResNetW12
63.66
81.68
49.38
70.48
72.65
86.17
59.88
77.25
-cls+rot
(0.80)
(0.57)
(0.54)
(0.42)
(0.86)
(0.61)
(0.65)
(0.51)
ResNetW12
61.67
79.19
46.89
67.33
70.55
85.33
57.33
75.43
-cls+loc5
(0.80)
(0.59)
(0.52)
(0.41)
(0.92)
(0.59)
(0.65)
(0.53)
ResNetW12
62.56
80.35
48.43
68.63
70.11
85.14
57.51
75,47
-cls+rottloc5
(0.77)
(0.59)
(0.50)
(0.44)
(0.91)
(0.63)
(0.65)
(0.51)
location crop in the same mini-batch, while for the larger Tiered-ImageNet we the sampling version instead to reduce training time.
We again consider the holdout accuracy on different tasks for these joint models. From Table 6 we can see that by jointly training with the other tasks, the performance on the corresponding task improves. For the joint model with location prediction (cls+loc4), accuracy on the holdout set drops by more than 5%, because for that model the training inputs are the 4 corners of an image and the model has never seen whole pictures of the object categories be learned. We can see how the learning tasks can interfere with each other in this case. We therefore create a 5-class version of the location prediction task, by including the rescaled version of the whole picture as an extra class, in addition to the original 4 corners. The resulting model(cls+loc5) has improved classification accuracy. We can also observe that there is a tradcoff among accuracies of different tasks due to finite learning capacity of the models, and this is particularly evident in the joint model cls+rot+loc5.
find that the classification
From Table diction(cls+rot) significantly improves the few-shot learning accuracy the representation. On the other hand there no significant improvement with the model with joint location training. For the joint model with all 3 tasks (cls+rot+loc)), it is better than the base supervised representation but slightly worse than the joint model with rotation prediction only, indicating that there might be capacity problem with fitting all three tasks.
see that for Mini-ImageNee ImageNet, training a multi-task model with rotation pre-
For Fiered-Imagele prediction (7. For the joint model with location and joint model with all 3 tasks (cls+rot+loc5) there are declines compared to the baseline model. The capacity constraint is more severe with Tiered- ImageNe as it is more complex dataset with larger number of classes. The classification and location prediction accuracies are 60% and 89% on the holdout set for the cls+loe5 model, but once we add the rotation task the holdout accuracies for classification, location, and rotation prediction drop to 56%, 79% and 87% respectively.
again see improvement with the multi-task representation with rotation
5 MAKING USE OF AUXILIARY CLASSES AND VOTING AT ADAPTATION
5.1 IRRELEVANT FEATURE ELIMINATION WITH AUXILIARY CLASSES
In previous sections, we aim to investigate which feature representation can serve the few-shot learn- ing task with more accurate features to reach a better predictive accuracy. All these feature repre- sentations are learned from upstream tasks with sufficient training classes and samples. However, find that these extra data are not only important during the feature extraction phase, they can
7

---

Under review as a conference paper at ICLR 2022
update
Auxillary classes
Cross-entropy
Loss
Train the linear classifier auxiliary classes 3 unseen classes
Pre-trained feature extractor
3-way 1-shot
Figure 1: Adapting with auxiliary classes downweigh irrelevant features
Table 8: Few-shot classification accuracies with auxiliary classes.
Mini -ani-lmaeeNet
Tiered Imaee-et
source rep./taget task
5w1s
5w5s
10w Is
10w5s
5wls
5w5s
10w1s
10w5s
61.75
78.98
47.09
66.93
71.36
85.56
58.15
75.77
ResNetW12-cls
(0.79)
(0.60)
(0.53)
(0.44)
(0.91)
(0.64)
(0.66)
(0.54)
63.70
79.42
49.29
67.40
71.92
85.66
58.59
76.25
ResNerW12-clstaux
(0.79)
(0.57)
(0.53)
(0.43)
(1.03)
(0.64)
(0.68)
(0.55)
63,66
81.68
49.38
70.48
72.65
86.17
59.88
77.25
ResNe:W12-clsrrot
(0.80)
(0.57)
(0.54)
(0.42)
(0.86)
(0.61)
(0.65)
(0.51)
66.28
81.92
51.08
70.80
73.69
86.82
59.50
77.50
ResNet112-ls+tottaxx
(0.78)
(0.55)
(0.52)
(0.41)
(0.91)
(0.60)
(0.67)
(0.53)
61.67
79.19
46,89
67.33
71.02
85.96
58.04
75,92
ResNetW12-cs+toc5
(0.80)
(0.59)
(0.52)
(0.41)
(0.95)
(0.63)
(0.68)
(0.53)
64.10
79.97
47.89
67.29
71.78
85.83
58.09
76.10
ResNeeW12-cbs+oo5-aar
(0.75)
(0.56)
(0.52)
(0.42)
(0.93)
(0.62)
(0.69)
(0.52)
also help improve the performance of the final linear classifier in the adapting stage. Consider the example of a 3-way I-shot classification of animals (sheep, cat, pig), the classifier cannot tell if "sky" is discriminative feature for sheep only one of the three images contain blue sky. The classifier can confuse the features of the animal that it should learn with background features that are merely correlated with the subject.
To solve this problem, as shown in Figure 1 after determining the well-trained backbone model, we obtain the embedding representation for the few-shot support data (e.g. from a 5-way 1-shot task) as well as samples from auxiliary classes (e.g. the 64 training categories from Mini-ImageNet), Next, we train the logistic regression classifier using all these data together. this case, the output size of the classification logits is extended (5+64). We just randomly sample 1000 examples from the train- ing set as our auxiliary classes data. By utilizing data in auxiliary classes, our final linear classifier learns downgrade and eliminate the contribution of those spurious features while emphasizing the unique task-related features. In our example the classifier learns to NOT associate "sky" with sheep, because the sky feature is also likely appear in auxiliary classes examples, and such association can lead to misclassification of the auxiliary classes examples. In the inference phase, extra connections and output logits corresponding to the auxiliary data are discarded. And the class in the current few-shot task with highest probability will be the final prediction.
The experiment result using auxiliary classes shown in Table g using the best multi task represen- tations from the previous sections. We can see that for most models there are good improvements for 5w1s and 10wls tasks confirming our intuition, while the improvements for 5w5s or 10w5s are smaller since there are 5 examples per class, making it less likely for the model to rely on spurious features. The improvements are also smaller for Tiered- NmageNe..
5.2 VOTING WITH AUXILIARY TASK INSTANCES
In Section we experimented with multi-task representations. For these models, they are not just capable of predicting the target class normal images, but also rotated images and cropped corners images since they are trained on these inputs. This offers yet another opportunity in improving the prediction performance via voting. We can generate set of rotated or cropped copies
8

---

Under review as a conference paper at ICLR 2022
Table 9: Few-shot classification accuracies with Docation-based voting.
Mini- -mageNet
Tiered- -maee-et
souree rep./target task
5w1s
5w5s
10w1s
10w5s
5w
5w5s
10w1s
10w5s
61.67
79.19
46.89
67.33
71,02
85,96
58,04
75.92
ResNeeW12-cls+oc5
(0.80)
(0.59)
(0.52)
(0.41)
(0.95)
(0.63)
(0.68)
(0.53)
63.38
81.87
48.85
70.54
72.02
86.41
59.15
77.82
ResNetW\12-css+ooc5--ooo
(0.77)
(0.57)
(0.54)
(0.44)
(0.93)
(0.60)
(0.69)
(0.51)
62.56
80.35
48.43
68.63
70.91
85.81
58.11
75.92
ResNaaW\1-ddssrrttllensss
(0.77)
(0.59)
(0.50)
(0.44)
(0.94)
(0.59)
(0.65)
(0.51)
63.53
82.27
49.11
71.45
71.66
86.59
58.92
77.52
ResNewW\2-cls-rottoooosss
(0.82)
(0.57)
(0.55)
(0.42)
(0.95)
(0.61)
(0.66)
(0.51)
Table 10: Few-shot classification with rotation-based voting.
Mini- Mite-emageNet
Ticred- Ieteeet
source rep./target task
5w1s
5w5s
10w1s
10w5s
5w
5w5s
10w1s
10w5s
63.66
81.68
49.38
70.48
72.65
86.17
59.88
77.25
ResNetW12-cls-rot
(0.80)
(0.57)
(0.54)
(0.42)
(0.86)
(0.61)
(0.65)
(0.51)
62.89
81.32
48,91
70.07
73.02
86.70
59.76
77.84
ResNetW12-cls+totee
(0.81)
(0.57)
(0.53)
(0.42)
(0.91)
(0.62)
(0.66)
(0.52)
of the input image, and run the target classifier on all of them, and take the majority prediction the final prediction. In the following we consider two voting schemes, one based on generating 4 copies of rotated input image for voting, and another one based on generating the copies cropped corners and one rescaled input) for voting, corresponding to the rotation prediction and location prediction task we used.
We can see from Tables Saales/andd and 10 results of voting. Voting based on the cropped images from the location prediction problem is effective in improving the prediction accuracies across all the various few-shot learning tasks. The accuracies improve average 2%, which is quite large for these problems. On the other hand, voting based on rotated versions the same image does not seem to help, presumably because unlike cropping the rotated images all contain the same information about the class. degrades performance slightly on Mini-Imagen ImageNet but slightly improves performance on frered-ImageNee but both are within standard error. We also tried voting based rotated and cropped copies using the base classification representation. But this degrades the performance since the classification model was not trained on classifying cropped or rotated images, indicating the importance of multi-task training.
We finally compare our multi-task representations with the above feature selection and voting heuris- ties against some of the state-of-art methods based on inductive learning published in the past two years, and the results are shown in Table II The results on 5wls and 5w5s learning are shown as they are the most commonly published numbers. We use the cls+rot+loc5 representation with aux- iliary classes and location voting as our best model for Mini-ImageNNet and cls+rot with auxiliary classes our best model for Wvered-ImageNee Our approach is competitive with these state-of-art models. Our best results are obtained with the original input image, as it supports the cropping of
Table 11: Comparison of our approach against some state-of-art methods.
Mini ImageNet
Geered-ImageNer
Method
Backbone
5wls
5w5s
5wls
5w5s
DeepEMD ZZang et al. 2020 FEAT Ye et al. (2020)
ResNet12
65.91(0.82)
82.41(0.56)
71.16(0.87)
86.03(0.58)
ResNet12
66.78(0.20)
12.05(0.14)
70.80(0.23)
84.38(0.16)
MetaOptNer Lee el 2019
ResNet12
62.64(0.61)
78.63(0.46)
65.99(0.72)
81.56(0.53)
STn el al. 2020
ResNet12
61.20(1.80)
75.50(0.80)
65.60(1.80)
80.80(0.80)
CAN et al.
2019
ResNet-12
63.85(0.48)
79.44(0.34)
69.89(0.51)
84.23(0.37)
Distillation Tian al. (2020)
ResNet-12
64.82(0.60)
82.14(0.43)
71.52(0.69)
86.03(0.49)
Boosting(Gidans Gidans et (2019)
WRN-28-10
63.77(0.45)
80.70(0.33)
70.53(0.51)
84.98(0.36)
Fine-tuning Dhillon | (2019)
WRN-28-10
57.73(0.62)
78.17(0.49)
66.58(0.70)
85.55(0.48)
Centroid align(A ATrastyabi el al. 22020))
WRN-28-10
65.92(0.60)
82.85(0.55)
74.40(0.68)
86.61(0.59)
Ours (best model, crop original)
ResNerW12
65.23(0.81)
73.69(0.91) 83.37(0.56)
86.82(0.60)
Ours (best model, 84x84 resized)
ResNetW12
63.05(0.79)
80.11(0.57)
69.78(0.88)
84.86(0.65)

---

Under review as a conference paper at ICLR 2022
184x184 images without upscaling for the location prediction task. We also show results the 84x84 resized image, using clstrot models for both Mini-Imagete and Tiered- MrreedlmmgeNett since we cannot include the location prediction task with this resolution. There is considerable gap be- tween these two input preprocessing approaches, which we discuss in Appendix A.1 We believe the biggest benefit of our approach is the ability deal with two concerns in few-shot learning separately: transfering to a new domain and dealing with the small number of examples given in that new domain. We handle the first one by studying representations for few-shot learning transfer, and this problem can benefit from the numerous works done in representation learning in computer vision and NLP in recent years. We handle the second problem by imposing extra constraints when adapting a linear classifier through prior knowledge. This modular approach can allow progress on solving either problems translate to improvements in few-shot learning.
6 FUTURE WORK AND CONCLUSIONS
We have presented in this work systematic study of different learned feature representations for few-shot learning. Coupled with some new heuristics for feature selection, we find that this trans- fer learning approach with multi-task representations is competitive with state-of-art methods. For future work we would like to study if deeper networks would allow us to learn these multi-task representations better, and also obtain better understanding of how features are shared compete against each other in these multiple tasks. We are also interested in exploring what other types of structures constraints can be exploited during the classifier adaptation stage.
REFERENCES
Arman Afrasiyabi, Jean-François Lalonde, and Christian Gagné. Associative alignment for few-shot image classification. In European Conference on Computer Vision, pp. 18-35. Springer, 2020.
Alexei Baevski, Michael Auli, and Abdelrahman Mohamed. Effectiveness of self-supervised pre- training for speech recognition. arXiv preprint arXiv:1911.03912. 2019.
Tianshi Cao, Mare Law, and Sanja Fidler. theoretical analysis of the number of shots in few-shot learning. arXiv preprint 1909. 2019.
Susan Carey and Elsa Bartlett. Acquiring single new word, u: Papers and reports on child language development, 15: 17-29, 2011.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597-1607. PMLR, 2020a.
Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey Hinton. Big self- supervised models are strong semi-supervised learners. arXiv preprint arXiv:2006. 10029, 2020b.
Wei-Yu Yu Chen, Yen-Cheng Liu, 7solt Kira, Yu-Chiang Frank Wang, and Jia-Bin Huang. closer look few-shot classification. arXiv preprint 9rxiv:190.004222, 2019.
Yin Cui, Yang Song, Chen Sun, Andrew Howard, and Serge Belongie. Large scale fine-grained categorization and domain-specifee transfer learning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4109-4118, 2018.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. leee, 2009.
Guneet S Dhillon, Pratik Chaudhari, Avinash Ravichandran, and Stefano Soatto. baseline for few-shot image classification. arXiv preprint arXiv: 190.002999. 2019.
Carl Doersch and Andrew Zisserman. Multi-task sulevvuper visual learning. In Proceedings of the IEEE International Conference on Computer Vision, pp. 2051-2060, 2017.
10

---

Under review as a conference paper at ICLR 2022
Carl Doersch, Abhinav Gupta, and Alexei A Efros. Unsupervised visual representation learning by context prediction. In Proceedings of the IEEE international conference on computer vision, pp- 1422-1430, 2015.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-leaming for fast adaptation of deep networks. In International Conference on Machine Learning, pp. 1126-1135. PMLR, 2017.
Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. arXiv preprint
2018.
Spyros Gidaris, Andrei Bursuc, Nikos Komodakis, Patrick Pérez, and Matthieu Cord. Boosting few-shot visual learning with self-supervision. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 8059-8066 2019.
Edward Grefenstette, Brandon Amos, Denis Yarats, Phu Mon Htut, Artem Molchanov, Franziska Meier, Douwe Kiela, Kyunghyun Cho, and Soumith Chintala. Generalized inner loop meta- learning, arXiv preprint arX1v:1910.01727, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 9729-9738, 2020.
Ruibing Hou, Hong Chang, Bingpeng Ma. Shiguang Shan, and Xilin Chen. Cross attention network for few-shot classification. arXiv preprint 191007791. 2019.
Vladimir Iglovikov and Alexey Shvets. Ternausnet: U-net with vggll encoder pre-trained on ima- genet for image segmentation. arXiv preprint arXiiv1801.057444 2018.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Sori- cut. Albert: A lite bert for self-supervised learning of language representations. arXiv preprint arXiv: 11922, 2019.
Kwonjoon Lee, Subhransu Maji, Avinash Ravichandran, and Stefano Soatto. Meta-learning with differentiable convex optimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10657-10665, 2019.
Yoonho Lee and Seungjin Choi. Gradient- meta-learning with learned layerwise metric and subspace. In International Conference on Machine Learning, pp. 2927-2936. PMLR, 2018.
Zeming Li, Chao Peng, Gang Yu, Xiangyu Zhang, Yangdong Deng, and Jian Sun. Detnet: A
backbone network for object detection. arXiv preprint arXiv: 1804.06215 2018.
Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta-learning algorithms. arXiv preprint 1=XXiv:803.02999 2018
Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. In European conference on computer vision, pp. 69-84. Springer, 2016.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32: 8026,8037, 2019.
Aniruddh Raghu, Maithra Raghu, Samy Bengio, and Oriol Vinyals. Rapid learning or feature reuse? towards understanding the effectiveness of maml. arXiv preprint arXiv:1909.099 091799999999 2019.
Mengye Ren, Eleni Triantafillou, Sachin Ravi, Jake Snell, Kevin Swersky, Joshua B Tenenbaum, Hugo Larochelle, and Richard S Zemel. Meta-learing for semi semi-supervised sew-shot classifica- tion. arXiv preprint arXiv: 1803 00676, 2018

---

Under review as a conference paper at ICLR 2022
Sebastian Ruder, Matthew E Peters, Swabha Swayamdipta, and Thomas Wolf. Transfer learning in natural language processing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Tutorials, pp. 18-18, 2019.
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for sew-shot learning. arXiv preprint arXiv: 1703.07 0777000..55... 2017.
Jong-Chyi Su, Subhransu Maji, and Bharath Hariharan. When does self-supervision improve few-
shot learning? In European Conference on Computer Vision, pp. 645-666. Springer, 2020.
Qianru Sun. Yaoyao Liu, Zhaozheng Chen, Tat-Seng Chua, and Bernt Schiele. Meta-transfer learn- ing through hard tasks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020.
Yu Sun, Eric Tzeng, Trevor Darrell, and Alexei A Efros. Unsupervised domain adaptation through self-supervision. arXiv preprint arXiv: 1825, 2019.
Flood Sung, Yongxin Yang, Li Zhang, Tao Xiang, Philip HS Torr, and Timothy M Hospedales. Learning compare: Relation network for few-shot learning. In Proceedings of the IEEE con- ference on computer vision and pattern recognition, pp. 1199-1208, 2018.
Yonglong Tian, Yue Wang, Dilip Krishnan, Joshua B Tenenbaum, and Phillip Isola. Rethinking few-shot image classification: a good embedding is all you need? In Computer Vision-EBe ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XIV 16, pp. 266-282. Springer, 2020.
Eleni Triantafillou, Vincent Dumoulin, Hugo Larochelle, and Richard Zemel. Learning flexible classifiers with shot-conditional episodic (scone) training. 2020.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. Advances in neural information processing systems, 29:3630-3638, 2016.
Yu- Yu-Xiong Wang, Deva Ramanan, and Martial Hebert. Meta-learning detect rare objects. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9925-9934 2019.
Han-Jia Ye, Hexiang Hu, De-Chuan Zhan, and Fei Sha. Few-shot learning via embedding adaptation with set-to- set functions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8808-8817, 2020.
Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, Andre Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, et al. A large-scale study of representation learning with the visual task adaptation benchmark. arXiv preprint arXiv: 1xtv:19100008677 2019.
Chi Zhang, Yujun Cai, Guosheng Lin, and Chunhua Shen. Deepemd: Few-shot image classifica- tion with differentiable earth mover's is distance and structured classifiers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 12203-1221, 2020.
A APPENDIX
A.1 EFFECT OF INPUT IMAGE SIZE ON FEW-SHOT CLASSIFICATION ACCURACY
In the literature the datasets Mini- Imag-em and Viered-ImageNye are usually constructed from the standard ImageNet data(Deng et al. 2009), but depending on the interpretation of the input image size of 84x84 from the original papers( Vinyals et al.) 2016 Ren et al. 2018), there could be two different versions of Mini-ImageNet or Tiered- Berred-ImageNet constructed. One approach directly resizes the input image file to 84x84, while the other approach performs random resized crops of size 84x84 analogous to the common data augmentation approach in ImageNet training (crops of 224x224 instead). We have seen implementations data loaders that read from resized 84x84 python pickle files, and also data loaders that directly perform random resized crops from the input jpeg image.
12

---

Under review as a conference paper at ICLR 2022
Table 12: Few-shot classification different input image resolution.
-mai-lmageNet
Tiered-ImageNet
source rep./target task
5w
5w5s
10w1s
10w5s
5w1s
5w5s
10w1s
10w5s
60.53
78.44
45.77
65.87
68.38
83,25
54.83
73.02
(e5NeeW\1-cds(84x844 resized)
(0.80)
(0.61)
(0.53)
(0.43)
(0.94)
(0.67)
(0.66)
(0.53)
61.75
78.98
47.09
66.93
71.36
85.56
58.15
75.77
ResNewW12-cls84x84 random crop)
(0.79)
(0.60)
(0.53)
(0.44)
(0.91)
(0.64)
(0.66)
(0.54)
Table 13: Few-shot classification using different number of augmented examples from training set.
Mini-ImageNee
source rep./target task
5wls
5w5s
10wls
10w5s
ResNerWW12-cs
61.75(0.79)
78.98(0.60)
47.09(0.53)
66,93(0.44)
ResNetW ResNsaW11-cls-aux 12-cls-aux
63.62(0..7)
79.53(0.58)
47.97(0.50)
67.06(0.44)
ResNetW12-cls -2-
63.96(0.81)
79.39(0.58)
48.17(0.53)
67.16(0.43)
DesNetWi1-cls-ause
63.69(0.83)
79.43(0.60)
47.90(0.55)
67.31(0.43)
But these two versions of the data contain different amount information as images in the second approach is of higher resolution, especially when data augmentation is used. Data augmentation used on the support set for both logistic regression and MAMI.. Table [2] shows the few-shot transfer accuracy of the base classification model on Mini-ImageNeet and Tiered- fiered-ImageNee In gen- eral using the original ImageNet input image compared to the downsampled 84x84 version gives 1-2% advantage for Mini-ImageNNt,, and 2-3% advantage for fiered-ImageNe considered in this paper. In this work we focus on the original input image, because representation learning methods like location prediction needs larger 184x 184 input image to avoid effect of in- terpolation in upscaling, and constrative learning benefits from having larger set of random crops from a higher resolution image. This is consistent with the usual training of representation learning methods ImageNet because the input images of input transformations. We discovered this issue of input resolution when we downloaded two different versions of the data and noticed there are considerable differences in their accuracies.
for the models we
not resized to 224x224 to allow for types
A.2 EFFECT OF NUMBER OF AUGMENTED AUXILIARY EXAMPLES ON FEW-SHOT CLASSIFICATION ACCURACY
We also consider more systematic scheme for augmenting the support set with auxiliary classes from the training data apart from randomly sampling 1000 examples from the training set. In Ta- ble 13 we consider augmenting 1, and 10 examples per training class(64 classes) the support set Mini-ImageNet. We can that the improvement is similar for different number of exam- ples augmented. One might be able to obtain better results by increasing the number of augmented examples but at the same time control their total sample weight relative examples in the support set, but we do not explore this further in this work.
13


Reviews:
[{"review_id": "yZ0ccqkU0eP", "reviewer": "Reviewer_Tk8a", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Studying the utility of different representations for few-shot learning tasks is an interesting and important topic, due to the success and wide adoption of the transfer learning paradigm. Jointly training for different objectives is also interesting to study, as well as the proposed voting and auxiliary class downstream training methods proposed here. These have the potential to produce impactful results. The paper is also clearly written for the most part and easy to follow. However, the study is limited to simple datasets, and the proposed methods do not yield a consistent gain, especially on tiered-ImageNet. Some additional concerns and feedback below:\n\nCorrectness\n- “The paper Tian et al. (2020) show that by just pre-training … and adapting using logistic regression, they can obtain results … that are competitive with state-of-the-art algorithms” - This isn’t exactly true. While the approach of Tian et al indeed doesn’t use meta-learning, their approach is more complicated than what this sentence makes it sound, as it involves a distillation phase aside from just standard pre-training.\n\n- ‘... and train a logistic regression model on top of it during the adaptation stage on the query set’ - this training should take place on the support set, not the query set. I assume this is just a typo but it would be great if the authors could clarify.\n\n- ‘Transfer learning, on the other hand, … , a much simpler task model (e.g. a linear classifier) instead of having to learn feature extraction at test time’. This doesn’t seem correct - meta-learning approaches sometimes learn simple models too, e.g. Prototypical Networks is a form of a linear classifier, and transfer learning often involves fine-tuning the entire network at test time [1] (some examples in the context of few-shot classification: [3, 4]). The fact that just training a linear classifier on top of pre-trained features works so well for mini- and tiered- imagenet might just be a symptom of the simplicity of these datasets and resulting transfer problems, but this is not true of transfer learning more generally. \n\n- ‘it is very difficult to figure out which feature is relevant given only a handful of examples’ - is there some evidence to support this claim? Doesn’t the success of simple linear classification on top of pre-trained features contradict this claim to some degree?\n\nRelationship to previous work\n- The observation that using a larger ‘way’ during training than testing has already been made in prior literature [4]. The authors there also hypothesized that this is due to harder training tasks being beneficial for representation learning. Though in this case the experiments are done with MAML instead of Prototypical Networks, this connection should still be discussed.\n\n- In addition, the effect of the number of shots used at training time on test performance on different configurations of tasks has been explored both theoretically and empirically in the past [5,6].\n\n- The approach used here is reminiscent of the ‘input harmonization’ in [7]. Can we get an additional performance boost from also incorporating the lasso architecture proposed there? This allows different tasks to have slightly different weights instead of forcing them all to use exactly the same feature extractor.\n\n- [8] (and approaches cited within) also study transfer learning with several different types of pre-trained representations, on several different data-limited downstream tasks. They explore both supervised and self-supervised representation learning approaches. It is worth discussing the connections to that work and how the goals of this work differ from previous studies.\n\nAdditional experiments\n- What happens if a more naive approach is used for multi-task training, where each task receives different inputs (e.g. the classification loss is computed on the original images instead of the rotated or cropped ones). The assumption made here is that this would perform worse, but it would be useful to actually report these results and verify this hypothesis. It seems, for instance, that [9] performed joint training between a supervised a self-supervised objective without making any modifications, and still observed improvement in performance.\n- Can location-based voting and rotation-based voting be combined? More generally, how does this compare to voting with random augmentations (e.g. those typically used for data augmentation, like color jittering, horizontal flipping, etc)? This would help understand if this voting is beneficial only when the augmentations chosen are the same ones used for training.\n- The authors hypothesize that the reason auxiliary class training helps is to alleviate the issue of learning spurious features. Is there some evidence to support this hypothesis? Perhaps looking at the gap between the support and query accuracy in both cases (with and without the auxiliary classes) would be helpful, since if the classifier is learning spurious features perhaps this would lead to overfitting on the support set and not generalizing to the query set?\n\nClarity\n- It’s not entirely clear how logistic regression differs from fast adaptation of MAML, when only adapting the final layer of MAML. I’m assuming the main difference is that a new random readout layer is initialized instead of re-using MAML’s meta-learned classification layer. Is there also a difference in the number of steps, e.g. logistic regression takes more steps than fast adaptation? It would be good to clarify exactly what the differences are.\n\n- ‘To further simplify the re-initialization step’, citing the Reptile paper. This sentence is confusing; it’s not clear what re-initialization step means and how this paper simplifies it. It seems more appropriate to mention Reptile in the context of first-order approximations for MAML.\n\n- The term ‘domain’ is used in an inappropriate way throughout the paper. I wouldn’t say that the test set of mini-ImageNet comprises a different domain than the training set of mini-ImageNet. Referring to it as ‘held-out class split’, for instance, would be more appropriate.\n\n- Why not just refer to ‘modified MAML’ by its name, ANIL (from Raghu et al, as cited).\n\nReferences\n- [1] Do Better ImageNet Models Transfer Better? Kornblith et al. CVPR 2019.\n- [2] Meta-Dataset: A Dataset of Datasets for Learning to Learn from Few Examples. Triantafillou et al. ICLR 2020.\n- [3] Comparing Transfer and Meta Learning Approaches on a Unified Few-Shot Classification Benchmark. Dumoulin et al. NeurIPS 2021.\n- [4] Prototypical Networks for Few-shot Learning. Snell et al. NeurIPS 2017.\n- [5] A Theoretical Analysis of the Number of Shots in Few-shot Learning. Cao et al. ICLR 2020.\n- [6] Learning Flexible Classifiers with Shot-CONditional Episodic (SCONE) Training. Triantafillou et al. Meta-learn workshop, NeurIPS 2020.\n- [7] Multi-task Self-Supervised Visual Learning. Doersch et al. ICCV 2017.\n- [8] A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmark. Zhai et al. 2020.\n- [9] Boosting Few-Shot Visual Learning with Self-Supervision. Gidaris et al. 2019.\n\n", "summary_of_the_paper": "This paper conducts an empirical study on the usefulness of different representations for downstream few-shot learning tasks, using the transfer strategy of training a new linear layer on top of the frozen pre-trained network. They also propose to utilize auxiliary data during this downstream learning phase: instead of learning an N-way readout layer for an N-way classification task, they learn a (N+T)-way one, where T is the number of training classes, using a subset of the training dataset for this. To perform inference on the query set, though, they only consider the N classes appearing in the given task. They hypothesize that this approach prevents the linear classifier from paying attention to spurious features, and it sometimes yields a gain in performance. They also propose to combine different training objectives, including the supervised classification objective and different self-supervised ones in a multi-task manner, and propose to do so by keeping the input the same for all objectives involved, meaning that to train the supervised loss together with the rotation, the former will also be computed on rotated images. Finally, they propose a voting scheme, where the predictions for different augmentations of each input image are aggregated to form the final prediction. All experiments are conducted on mini-ImageNet and tiered-ImageNet, and gains are reported mostly on the former.\n", "summary_of_the_review": "Although the paper studies an interesting topic, I have some concerns about correctness of various statements in the paper, the relationship with prior work that has explored similar questions, and the need for additional experiments and analyses to aid in better understanding where the observed performance gains come from. Finally, a weakness of this work is that the study is limited to simple datasets and a linear transfer learning adaptation technique, which makes me question whether the conclusions will generalize to other scenarios.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "wNedRwIqxp", "reviewer": "Reviewer_EEwV", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "My main concern of this paper is its novelty. In my opinion, it is already a kind of common sense that training with more complex data results in better feature representation. (that's why the race to develop a giant pre-learning model with big data is on)  \nUsing multiple source tasks is also a straightforward direction to go. For example, it is well-studied in the following paper.   \n  - Zamir et al., Taskonomy: Disentangling Task Transfer Learning, CVPR 2018.  \n \nI understand that the focus of this paper is different (i.e., few-shot, self-supervised tasks),  but at high level, the findings and conclusions in the paper seem rather obvious to me. Or, is this result really surprising for few-shot learning? If so, please clarify it as the reviewer doesn't understand the point.  \n \nAs for the feature selection (auxiliary class) and voting methods, I can find no novelty in them either. In my understanding, they are nothing but multi-task learning and test-time augmentation, respectively. ", "summary_of_the_paper": "This paper studies few-shot learning from the viewpoint of transferring feature representations. The authors investigate few-shot performance with varying complexities of source tasks together with a few empirical tricks for improvements. The main finding in the paper is that transferring from more complex source tasks tend to result in better performance. Accordingly, using multiple source tasks is also found to be useful.\n\nStrengths:  \n(+) The paper is clearly motivated and well-written.  \n(+) Reported scores could be useful to server as a baseline in this direction of research.  \n\nWeaknesses:  \n(-) No technical novelty.  \n(-) Results and conclusions seem rather obvious.  ", "summary_of_the_review": "Overall, I find this paper presents an interesting case study of feature transfer focusing on few-shot learning, it lacks enough novel insights and technical contributions to warrant publication at ICLR. \n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "VdUtPlkb9gh", "reviewer": "Reviewer_G5x6", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is generally well written with a clear explanations and research direction.\nThe topic is relevant and of interest to the community.\n\nAfter initial description I was saddened that the analysis of feature representation of few-shot methods involved only MAML. It is in fact significant method that influenced many other, but its performance was quickly surpassed by more advanced methods which should be analyzed instead. At least some more methods should be analyzed as then instead of studying few-shot methods the paper studies MAML only (which might mean that conclusions might not generalize). The self-supervised methods studied (rotation, location and contrastive) are also very simple, and although their analysis is good, the proposed way of combining them is difficult to see as a strong technical novelty. The same goes for the voting system built on top of the proposed training scheme. The results confirm the importance of both, however the technical contribution itself seems weak.\n\nIn Table 11, only the authors method is using modified ResNetW12 -- this raises the question how significant that is? If the authors' method requires it, then other methods should be compared fairly with same backbone. Regardless of this, the results do not build a strong case for the proposed method (since only for a 5w5s on miniImageNet it outperforms existing methods). Comparable results would be impressive if the proposed method would provide some additional benefit, however none were mentioned.", "summary_of_the_paper": "The authors study impact of self-supervised methods on feature representation in few-shot learning task. They analyze the way features are transferred for MAML method and 3 self-supervised methods. They introduced a method combining self-supervised representation training methods with voting system in order to improve performance.", "summary_of_the_review": "Interesting case study of feature representations for few-shot learning. However, it lacks depth (only MAML analyzed as few-shot method) and the proposed solution lacks technical novelty and convincing results.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "L_ml8_nEnCj", "reviewer": "Reviewer_28ox", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "# Main review \nThe main part of the paper is dedicated to the study on the various training approaches for the transferred backbone. This study confirm the observations already published in the literature (here work of Tian et al is especially cited). Author could also have cited work of Mangla et al that also addressed this point and provide some insights on the benefits of using additional self-supervised task in the context of Few Shot Learning. This study part is quite long and some experiments could be omitted (e.g. training the backbone only on one self-supervised task) and does not really bring some new learnings with respects to previous works. In previous works it is already clear that self-supervised tasks should be considered as additional tasks. \n\nNote also that contrastive loss (from Khosla et al) has also been proposed recently in the context of Few Shot Learning by Ma et al, where 63.76% accuracy is reported on 5w1s (see table 3 of the paper)  which is rather pointing in favour of using such loss in the context of Few Shot Learning. This is contradictory to the comment made on table 4. This is not to point something to be cited, but rather to highlight the weakness of the observations and conclusions made in this part.\n\nAlthough being cited, no comparison result to Tian et al is provided in the benchmarks.\n\nSection 5 presents some really interesting tricks to improve the classifier. \nFrom table 8, a significant systematic gain is observed when using auxiliary classes. I find it as being the main contribution of this paper. Additional studies would be interesting to be performed in order to see what is the impact of the number of additional samples used as well as more detailed explanation on how additional samples are used. \n- It is not clear here if all of the 1000 additional samples are always used here  (1000 is not a multiple of 64)\n- does the 64 additional classes are always used?\n- what is the impact if using less auxiliary classes with less samples per class?\n\nIn the final comparison, a combination of the various approaches is compared to SOTA solutions. However it is not clear from where the benefits comes from in the selected combinations. Especially since we can observe that the combination is different for mini-ImageNet and tiered-ImageNet. Could we have more insights why the voting solution does not work for tiered-ImageNet?\n\nTable 11 reports comparative results with other SOTA approaches. However since backbone architecture is different and with the use of auxiliary classes and voting mechanism it is not clear where the benefit come from. Ablation studies is required to see what are the relative benefits of the varying tricks. Especially when we can observe that the settings between mini-ImageNet and tiered-ImageNet are different (location voting used only for mini-ImageNet)\n\n\n# Minor remarks\n- page 3, it is mentioned from results in Tables 1 and 2, that it is better to train on 10-ways tasks. But from those tables 5w5s and and 5w1s are still competitive for imagenet. conclusions here are not that straightforward.\n- table 3 is not cited in the paper. It should be cited in section 3.2 when discussing about MAML based training vs supervised training.\n\n\n# Additional references\n- Mangla et al, \"Charting the right manifold: Manifold mixup for few shot learning\"; WACV 2020.\n- Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna,Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, andDilip  Krishnan.Supervised  contrastive  learning.arXivpreprint arXiv:2004.11362, 2020\n- Jiawei Ma, Hanchen Xie, Guangxing Han, Shih-Fu Chang, Aram Galstyan, Wael Abd-Almageed, Partner-Assisted Learning for Few-Shot Image Classification. arxiv:2109.07607, 2021", "summary_of_the_paper": "The paper provides a study on different approaches to train a backbone network used in a transfer fashion for Few Shot Learning. \nRepresentation learned from MAML, supervised classification, and self-supervised tasks are considered here and their performance are compared. From this analysis it is confirmed that the use of multi-task trained representation (based on supervised classification and self-supervised auxilary tasks) leads to the most performing backbone.\n\nFurthermore it is proposed to improve the classifier part using two additional tricks: \n- use of auxiliary classes to help eliminate irrelevant features\n- use of voting with image transformations (rotation, location splitting)\nThis additionnal trick allows to boost performance over traditionnal simple classifier. \n\nFinal results show performance on par with various state of the art solutions.", "summary_of_the_review": "From the contributions reported by the authors:\n- contribution 1 and 2 are not really novel and do not really provide some new insight with respect to previously published works and current trends in training a backbone for Few Shot Learning\n- contribution 3 (feature selection, voting) is quite novel and provide some interesting gains, but more experiments would be needed to assess the relative benefits of those proposals.\n\nConsidering that the focus in this paper is more put on contribution 1 and 2, a significant reorganization of the paper as well as more experimental results should be done in order to focus more on contribution 3. That's why my vote is to reject this paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "JEMz-45aOvw", "reviewer": "Reviewer_63j7", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "* Although the ideas of using the MAML algorithm to adapt the last layer of the neural network which performs very close to adapting the whole network (Raghu et al., 2019) and simple baselines perform close to SOTA from learning a good representation through a proxy task (Tian et al., 2020) already exist. The authors of the paper acknowledge it and conducted experiments on how simple these baselines (ConvNet, ResNet) can be. And which proxy tasks (rotation prediction, location prediction) are helpful.\n* In-depth analysis and experimentation with various techniques, architectures like ResNet, ConvNet, and sizes of the network help point out the limitations of those techniques and lead to the critical finding that complex tasks (combining classification, rotation prediction, and location prediction) tend to give better representations for few-shot classification. \n* The use of transfer learning with multi-task representations and novel tricks like “eliminating irrelevant features using auxiliary classes” improved the performance on 5w1s, 5w5s, 10w1s by emphasizing the unique task-related features. The other trick is “voting with auxiliary task instances” where a set of rotated or cropped copies of the input image are given with the input image may not have any impact on the learned feature representations but it helped to improve the accuracy even closer to SOTA.\n* The description of various hyperparameters and the reasoning for constraints used in the experiments helped to better understand and compare the results to other methods.\n\n#### __Concerns:__\n\nI have some queries about the studies presented in the paper.\n\n* In Section 3.2, Imagenet pre-training, It’s unclear how much data is used? \n    * As miniImagenet, TieredImageNet are subsets of ImageNet, Is data of common classes removed from the pre-training data at the time of pre-training?.\n    * To what extent does the amount of data and the diversity of data used in the pre-training affect the representations and performance of the various models.\n* The authors have chosen the representation size to be 640, may I know the reason behind it? Will the performance of the various models used in the studies vary with size?\n* Although the paper provides various experiments on the Initialization based method (MAML), Self Supervised Methods, will the proposed tricks improve Metric Based Models like Prototypical Networks, Matching Networks?\n\n#### __Other suggestions:__\n It is difficult to follow section 3.2 due to the missing reference to Table 3.\n\n", "summary_of_the_paper": "This paper presents an interesting study comparing the various feature representations for few-shot classification tasks (in Computer Vision) learned from the supervised classification with/without MAML, from multi-task prediction, and self-supervised tasks like rotation prediction and location prediction, and contrastive learning. \n\nThe paper also introduces two tricks “irrelevant feature elimination using auxiliary classes”  and “voting with auxiliary task instances” which made the ImageNet pre-trained model’s performance on downstream tasks of 5 way 1 shot & 5 way 5 shot close to SOTA (State Of The Art) while handling the issue of a small sample size of 5 examples per class.", "summary_of_the_review": "The idea of improving existing baselines to be almost comparable to SOTA with some techniques may not be a novel one but this paper stands out in experimenting and in-depth analysis of the various techniques and also combining them with heuristic tricks like feature selection and voting to handle the issue of small sample size. Hopefully, the authors can address my queries/concerns (mentioned above) in the rebuttal period.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work studies a number of feature representations for few-shot classification, including representations learned from MAML, supervised classification, and some self-supervised tasks. The main conclusion of the study is that learning from more complex tasks result in better representations for few-shot classification. As a practical solution, then, the authors suggest using representations learned from multiple tasks for few-shot classification.

The paper studies an important problem in machine learning, and reviewers all appreciate that. However, they raised concerns about the draft in its current state. Authors replied to these comments, and while reviewers acknowledged and appreciated the responses and the revision of the draft, unfortunately that did not convince the reviewers. Several major concerns remained unresolved at the end. Specifically, EEwV believes that the paper is a case study, which though useful, does not bring deep new insights or findings. 63j7 believes that even after revisions made to the paper, there are additional experiments required to understand and examine the claims. Tk8a finds the submission unready for publication due to weak experimental analysis and suggests running additional experiments to examine the hypotheses made by the authors (e.g, relating to spurious features, the need of input harmonization, the benefit of voting) and better tie in the findings of this work to related work. Tk8a provides a list of concrete suggestion along these lines. Similar to EEwV, 28ox also thinks that the paper lack novelty or does not really bring new insights on the way to train the backbone.

Based on these comments, and the ratings, I encourage authors to address these issues and resubmit.
Decision: Reject