IEPT: Instance-Level and Episode-Level Pretext Tasks for Few-Shot Learning
Authors: Manli Zhang
Jianhong Zhang
Zhiwu Lu
Tao Xiang
Mingyu Ding
Songfang Huang
Venue: ICLR 2021 Poster (2021)
Keywords: few-shot learning
self-supervised learning
episode-level pretext task

Abstract: The need of collecting large quantities of labeled training data for each new task has limited the usefulness of deep neural networks. Given data from a set of source tasks, this limitation can be overcome using two transfer learning approaches: few-shot learning (FSL) and self-supervised learning (SSL). The former aims to learn `how to learn' by designing learning episodes using source tasks to simulate the challenge of solving the target new task with few labeled samples. In contrast, the latter exploits an annotation-free pretext task across all source tasks in order to learn generalizable feature representations. In this work, we propose a novel Instance-level and Episode-level Pretext Task (IEPT) framework that seamlessly integrates SSL into FSL. Specifically, given an FSL episode, we first apply geometric transformations to each instance to generate extended episodes. At the instance-level, transformation recognition is performed as per standard SSL. Importantly, at the episode-level, two SSL-FSL hybrid learning objectives are devised: (1) The consistency across the predictions of an FSL classifier from different extended episodes is maximized as an episode-level pretext task. (2) The features extracted from each instance across different episodes are integrated to construct a single FSL classifier for meta-learning. Extensive experiments show that our proposed model (i.e., FSL with IEPT) achieves the new state-of-the-art. 

Published as a conference paper at ICLR 2021
IEPT: INSTANCE-LEVEL AND EPISODE-LEVEL PRE- TEXT TASKS FOR FEW-SHOT LEARNING
Manli Zhang, Jianhong Zhang & Zhiwu Lu
Gaoling School of Artificial Intelligence, Renmin University of China, Beijing. China Beijing Key Laboratory of Big Data Management and Analysis Methods, Beijing, China [manl i zhang, j ianhong, luzhi wu wrw.eeeu..u.....
Tao Xiang
Mingyu Ding
University of Surrey, Guildford, Surrey, UK t. txxiang@surrey.co aac..aac..... ac.uk
The University of Hong Kong, Hong Kong mingyuding@hkui haku.hhhh... hk
Songfang Huang
Alibaba DAMO Academy, Hangzhou, China songfang, he @ @all ibaba inn..ni... com
ABSTRACT
The need of collecting large quantities of labeled training data for each new task has limited the usefulness deep neural networks. Given data from a set source tasks, this limitation few-shot learning (FSL) and self seff-supervise learning (SSL). The former aims to learn "how to learn' by designing learning episodes using source tasks to simulate the challenge of solving the target new task with few labeled samples. In contrast, the latter exploits an annotation-free pretext task across all source tasks in order to learn generalizable feature representations. In this work, we propose a novel Instance-level and Episode-level Pretext Task (IEPT) framework that seamlessly tegrates SSL into FSL.. Specifically, given an FSL. episode, we first apply geometric transformations to each instance generate extended episodes. At the instance- level, transformation recognition is performed per standard SSL. Importantly, at the episode-level, two SSL-FSL hybrid learning objectives are devised: (1) The consistency across the predictions of FSL classifier from different extended episodes is maximized as an episode-level pretext task. (2) The features extracted from each instance across different episodes integrated construct single FSL classifier for meta-learning. Extensive experiments show that our proposed model (i.e., FSL with IEPT) achieves the new state-of-the-art.
be overcome using two transfer learning approaches:
INTRODUCTION
Deep convolutional neural networks (CNNs) (Krizhessky et al. 2012 He et al. 2016b Huang et al. 2017) have seen tremendous successes in wide range of application fields, especially in visual recognition. However, the powerful learning ability of CNNs depends on large amount of manually labeled training data. In practice, for many visual recognition tasks, sufficient manual annotation is either too costly to collect or not feasible (e.g.. for rare object classes). This has severely limited the usefulness of CNNs for real-worid application scenarios. Attempts have been made recently to mitigate such a limitation from two distinct perspectives, resulting in two popular research lines, both of which aim to transfer knowledge learned from the data of set of source tasks to a new target one: few-shot learning (FSL.) and self-supervise learning (SSL).
FSL (Fei-Fei et al., 2006 Vinyals al. 2016 Finn et al.) [017]] Snell et al.) 2017 Sung al., 2018) typically takes 'learning to learn' or meta-learning paradigm. That is, it aims to learn algorithm for learning from few labeled samples, which generalizes well across any tasks. that end, adopts
"Corresponding author.

---

Published as a conference paper at ICLR 2021
optiodes
Rocation Gassifier
rstarce-ios S2L lou Caw
Eplicee-lee ST, loss arang cpoole:
Clasiifier (nuntler))
CWV
Supervised PSL icet ovey dessods
Transformer (cplaade melggaatoon
FSL Classfier
Superdied Coop for splinee
Figure 1: Schematic of our approach to FSL. Given a training episode, we apply 2D rotations by O, 90, 180, and 270 degrees to each instance to generate four extended episodes. After going through feature extraction CNN, four losses over three branches are designed: (1) In the top branch, we employ self-supervised rotation classifier with the instance-level SSL loss Linst- (2) In the middle branch, an FSL classifier is exploited to predict the FSL classification probabilities for each episode. We maximize the classification consistency among the extended episodes by forcing the four probability distributions to be consistent using Lepie. The average supervised FSL loss Lauz is also computed. (3) In the bottom branch, we utilize an integration transformer module to fuse the features extracted from each instance with different rotation transformations; they are then used to compute an integrated FSI. loss Linteg- Among the four losses, Linst and Lepis are the self-supervised losses, and Lauz and Linteg are the supervised losses.
an episodic training strategy - the source tasks are arranged into learning episodes, each of which contains n classes and k labeled samples per class to simulate the setting for the target task. Part of the CNN model (e.g., feature extraction subnet, classification layers, or parameter initialization) is then meta-leamed for rapid adaptation to tasks.
In contrast, SSL (Doersch et al. 2015 Noroozi & Favaro 2016
lizuka et al. 2016 Doersch & Zisserman 2017 Noroozi al. 2018) does not require the source data to be annotated. Instead,
exploits an annotation-frea pretext task on the source task data in the hope that a task-generalizable feature representation can be learned from the source tasks for easy adoption adaptation in target task. Such a pretext task gets its self-supervised signal at the per-instance level. Examples include rotation and context prediction (Gidaris al., 2018 Doersch al. 2015), jigsaw solving (Noroozi & Favaro, 2016). and colorization (lizuka et al. 2016 Larsson et al. 2016). Since these pretext tasks are class aggnosst,, solving them leads to the learning of transferable knowledge.
Since both FSL and SSL aim to reduce the need of collecting a large amount of labeled training data for a target task by transferring knowledge from a set of source tasks, is natural to consider combining them in single framework. Indeed, two recent works (Gidaris et al. 2019 Su et al.) 2020) proposed integrate SSL into FSL by adding an auxiliary SSL pretext task in an FSL model. It showed that the SSL learning objective is complementary to that of FSL and combining them leads to improved FSL performance. However, in (Gidaris et al., 2019 Su et al.) 2020), SSL is combined with FSL in a superficial way: it is only taken separate auxiliary task for each single training instance and has no effect on the episodic training pipeline of the FSL model. Importantly, by ignoring the class labels of samples, the instance-level SSL learning objective is weak its own. Since meta-learning across episodes is the essence of most contemporary FSL models, we argue that adding instance-level SSL pretext tasks alone fails to exploit fully the complementarity of the aforementioned FSL and SSL, for which a closer and deeper integration is needed.
To that end, in this paper we propose a novel Instance-level and Episode-level Pretext Task (IEPT) framework for few-shot recognition. Apart from adding an instance-level pretext SSL task as in Gidaris al., 20199] Su et al. 2020), we introduce two episode-level SSL-FSL hybrid learning objectives for seamless FSL- integration. Concretely, as illustrated in Figure I our full model has three additional learning objectives (besides the standard FSL one): (1) Different rotation transformations are applied to each original few-shot episode to generate set of extended episodes, where each image has a rotation label for the instance-level pretext task (i.e., to predict the rotation label). (2) The consistency across the predictions of an FSL classifier from different extended episodes maximized as an episode-leve pretext task. For each training image, the rotation transformation does not change its semantic content and hence its class label; the FSL classifier predictions across different extended episodes thus should be consistent, hence the consistency regularization objective. (3) The correlation of features across instances from these extended episodes is modeled by
2

---

Published as a conference paper at ICLR 2021
transformer- attention module, optimizing the fusion of the features of each instance/image and its various rotation-trann traiiooomrrnsformeed versions mainly for task adaptation during meta-testing Importantl, with these three new learning objectives introduced in IEPT, any meta-learning based FSL model can now benefit more from SSL. by fully exploiting their complementarity.
Our main contributions are: (1) For the first time, we propose both instance-level and episode-level pretext tasks (IEPT) for integrating SSL into FSL. The episode-level pretext task enables episodic training of SSL and hence closer integration of SSL with FSL. (2) In addition these pretext tasks, FSL further benefits from SSL by integrating features extracted from various rotation-transformee versions of the original training instances. The optimal way of feature integration is learned by a transformer-baa based attention module, which is mainly designed for task adaptation during meta- meta-testing, (3) Extensive experiments show that the proposed model achieves the new state-of-the-are
RELATED WORK
Few-Shot Learning. The recent FSL studies are dominated by meta-learning based methods. They
can be divided into three groups: (1) Metric-based methods (Vinyals et al. 2016) [neel et al. 2017]
Sung al.
2018: Allen al. 2019 Xing al., 2019 Li et al. 2019a by Wu et al. 2019 Ye
ct al. 2020 Afrasiyabi et al. 2020 Liu et al. 2020 Zhang et al. 2020) aim to learn the distance
metric between feature embeddings. The focus of these methods often on meta-learning of
feature-extraction CNN, whilst the classifiers used are of simple form such as a nearest-neighhor
classifier. (2) Optimization- based methods (Finn et al.) [2017; Ravi & Larochelle [017] Rusu et al.) 2019 Lee et al., 2019) learn to optimize the model rapidly given few labeled samples per class in the new task. (3) Model-based methods (Santoro et al. 2016 Munkhdalai & Yu 2017; Mishra et al., 2018) focus designing either specific model structures parameters capable of rapid updating.
Apart from these three groups of methods, other FSL methods have attempted feature hallucination
(Schwartz et al. 2018 Hariharan & Girshick 2017 Gao et al. 2018 Wang et al. 2018 Zhang et al.)
[T et al. (219)) which generates additional samples from the given few shots for network
finetuning, and parameter predicting (Qiao et al., 2018 et al. 2018 Gidaris & Komodakis 2019 2018) which learns to predict part of the parameters of network given few samples of new classes for quick adaptation. In this work, we adopt the metric-based Prototypical Network (ProtoNet) (Snell eet al.) 2017) as the basic FSL classifier for the main instantiation of our IEPT framework due to its simplicity and popularity. However, we show that any meta-learning based FSL method can be combined with our IEPT (see results in (iguree())
Self-Supervised Learning. In SSL,, it is assumed that the source task data label-free and a pretext task is designed to provide seff-supervision signals at the instance-level. Existing SSL approaches differ mainly in the pretext task design. These include predicting the rotation angle (Gidaris et al., 2018) and the context of image patch (Doersch et al.] 2015 Nathan Mundhenk et al., 2018). jigsaw solving (Noroozi & Favaro 2016; Noroozi et al.| 2018) (i.e, shuffling and then reordering image patch), and performing images reversion (lizuka et al. 2016: Pathak et al.) 2016 Larsson al.) 2016). SSL has been shown to be beneficial to various down-steam tasks such as semantic object matching (Novotny al. 2018), object segmentation et al. 2019) and object detection (Doersch & /iisserman 2017) by learning transferable feature presentations for these tasks.
Integrating Self-Supervised Learning into Few-Shot Learning. To the best of our knowledge, only two recent works (Gidaris et al.. 2019 [Su et al. 2020) have attempted combining SSL with FSL. However, the integration of SSL into FSL often shallow: the original FSL training pipeline is intact; in the meantime, an additional loss on each image w.r.t. a self-supervised signal like the rotation angle or relative patch location is introduced. With pretext tasks solely at the instance level, combining the two approaches (i.c., SSL and FSL) can only be superficial without fully exploiting the episodic training pipeline unique to FSL. Different from (Gidaris et al. 2019 Su et al. 2020). we introduce an episode-lewe level pretext task to integrate SSL into the episodic training in FSL. fully. Specifically, the consistency across the predictions of an FSL classifier from different extended episodes maximized to reflect the fact that various rotation transformations should not alter the class-label prediction. Moreover, features of each instance and its various rotation transformed versions are now fused for FSL classification, to integrate SSL with FSL for the supervised classification task. Our experimental results show that thanks to the closer integration of SSL and FSL, our IEPT clearly outperforms (Gidaris et al. 2019 Su et al. 2020) (see Table
3

---

Published as a conference paper at ICLR 2021
3 METHODOLOGY
3.1 PRELIMINARY
Problem Setting. Given n-way k-shot FSL task sampled from test set D., to imitate the test setting, an FSL model is typically trained in an episodic way. That is, n-way k-shot episodes are randomly sampled from a training set De. where the class label space of D, has no overlap with that of D. Each episode E. contains a support set Se and a query set Q.. Concretely, we first randomly sample set of classes Ce from the training set, and then generate S. and 2. by sampling support samples and query samples from each class in Ce. respectively. Formally, we have {(2,, y:)ly; E Ce. I,...,? n X q). where S. n = 0. For simplicity, we denote 4 = n X k and l = n x q. In the meta-training stage, the training process has an inner and an outer loop in each episode: in the inner loop, the model is updated using Se; its performance is then evaluated on the query set Q. in the outer loop to update the model parameters or algorithm that one wants to meta-learn.
Se (1 y1) E Ce, 1, ..... n k} and 2.
Basic FSL Classifier.
We employ ProtoNet (Snell et al. 2017) as the basic FSL model. This model has ceature-extractiinn CNN and a simple non-paran con-pammetrric classifier. The parameter of the feature extractor is to be meta-learned Concretely, in the inner loop of an episode, ProtoNet fixes the feature extractor and computes the mean feature embedding for each class as follows:
he
Σ (E..B))SS
fa(a;) T(ye c),
(1)
K
where class e E Ces So feature extractor with learnable parameters o, and the indicator function. By computing the distance between the feature embedding of each query sample and that of the corresponding class, the loss function used to meta-learn & in the outer loop is defined as:
(Ser 2x)
1
Σ
llog E.Ec. exp( diS (11),
exp(- d(f (3:)) hy.))
JER.
where .(.,)) denotes distance function (e.g., the l2 distance).
3.2 PRETEXT TASKS IN IEPT
The schematic of our IEPT is illustrated in Figure I We first define set of 2D-rotttion operators g = {gr = o.... R 1), where Gr means the operator of rotating the image by *90 degrees and the total number of rotations (R = 4 in our implementation). Given an original episode Eg {Se, 2.} as described in Sec. 3.1 we utilize the 2D-rotation operators from G in turn to transform each image in Ee. This results in a set of R extended episodes (including the original one) E {(St 2-Hr = O,..., R 1), where S {(r;, Vs, r)lys Ce, 1,..., 4 and O. = { (Zt, (...))) D1 E Cc 1,.,, 1g). Now each episode is denoted as E = ((x, r).r) E d -I,... la, la + 1,.., la + 4)). where the first 4. samples are from S. and the rest from O Note that (S0, 20] is the original episode (Se, (c). With the rotation transformations, each sample (x,, ye, r1)) in E carries a class label y, for supervised learning (from the inherent class) and label r; from the rotation operator for self-supervised learning. After generating the set of extended episodes E, the feature extractor fo is applied to each image æ; in E. On these episodes, we design two self-supervised pretext tasks, one at the instance-level and the other episode-level.
Instance- Pretext Task. The instance-level task is to recognize different rotation transforma- tions. The idea is that if the model to be meta-learned here (i.e., fo) can be used distinguish different transformations, it must understand the canonical poses of objects (e.g., animals have legs touching the ground and trees have leaves on top), a vital part of class-agnostic and thus transferable knowledge. With the self-supervised rotation label r,, we consider the mapping: fe... : I; + Ta for each instance (x,, Ve, rt)) E E, where fe... a rotation classifier with learnable parameters 0rot- Given the input pair (I; rz), the total instance-level rotation loss is a cross-entropy loss:
Linst R(4
1
Σ
Σ
log exp(Ifomm. (fo(z:))lr)
(3)
(n)
ΣP= exp([fo (fo(x)))le))
P=u (E....::)E
where (So... (fo(z:)) R is the rotation scoring vector and [[l means taking the r-th element.
4

---

Published as a conference paper at ICLR 2021
Episode-Level Pretext Task. We design the episode-level task based on simple principle: although different extended episodes contain images with different rotation transformations, these transforma- tions do not change their class labels. Consequently, the FSL classifier should produce consistent probability distributions for each instance across different extended episodes. Such consistency can be measured using the Kullback-Leible (KL) divergence. Formally, for each extended episode {S 2r) in E, we first define the probability distribution of FSL classification over the query set Q as P = lpfi- :P R8x' where p. R² is the probability distribution for 2; in 2 with its
e-th element [p (c = 1,... n2) being:
WM exp(- ((())
E. exp( d(S (z)).
The above probability is computed as in Sec. 331 and the class embedding hy is obtained from S The mean probability distribution of the R extended episodes is thus given by:
R-1
P
1
Σ P
(5)
R
r=0
The total episode-level consistency regularization loss is computed with the KL divergence loss:
R-1
4
Cepis
Σ
mean (p, (log p. log p:)).
(6)
Rlg
r=0
where mean(-) (-) is an element-wise averaging function.
3.3 INTEGRATED FSL TASK
The two tasks introduced so far are self-supervisse tasks without using the class labels in the query set. Now we describe how the supervised classification task, the extended episodes can be used.
Given the set of extended episodes F, we denote the feature set of E as Feeb, where Femb = (fo(z.) Ve: r) E E,, = 0.... R- 1.... 4. + +1 Note that each extended episode in E corresponds to one specific rotation transformation of the same set of images from the original episode Eg. Therefore, in order to capture the correlation among instances with different transformations and learn how best combine them to form the class mean for meta-learning, is deployed w.r.t. each image in E, (i.e., all images are assumed to be independent). Specifically, based on Femb, we construct the feature tensor F E (((x+ where is the feature dimension. We then adopt transformer to obtain the integrated representation for FSL classification. The transformer architecture is based on a self-attention mechanism, as in (Vaswani et al., 2017). It receives the triplet input (F, F, F) as (Q, K, V) (Query, Key, and Value, respectively). With F()) being the i-th row of F (w.rt. the i-th image in Ec), the attentive module is defined as: F F())
instance attention module
(F()
(F FO Fv),
(F(O)
F(0)
F 9 + softmax
600
(8)
vdk
where dx d, and Wo, WK, Wy represent the parameters of three fully-connected layers respec- tively (the parameters of the integration transformer are collected as 0..r). Note that the key and value are computed from each image and its augmented versions, i.c., they are computed independently without using inter-image correlation. With the attentive feature Fatt (R(+++()x Rxd, the integrated representation Finteg [FS; FQ] E Rlle+l)xxRd (FS and F are respectively for the support set and query set) is given by:
Finteg flatten(Fatt).
(9)
where flatten(-) denotes flattening Fatt along the last two dimensions, i.e., concatenating the attentive features from different extended episodes for the corresponding images. The integrated representation is then inputted to the FSL classifier to define the FSL classification loss:
1
ln
exp(-d(F.)) n())))
Linteg
log
(10)
T
Exec, exp( d(F nE))
im1
where the class embedding h- = the integrated FSI. task actually acts an alternative to prediction averaging.
FS Ily = c) is computed on the support set. Note that
5

---

Published as a conference paper at ICLR 2021
3.4 TOTAL Loss
The total training loss for our full model consists of the self-supervised losses from the pretext tasks and the supervised losses from the FSL tasks. In this work, in addition Linteg Eq. (TO), another supervised FSL loss Loaz is also used (see Figure D Louz is the average FSL classification loss over the extended episodes. Formally, it can written as:
R-1 R
Lau
( 2
(11)
r=
Therefore, the total loss Ciotat for training our full model is given as follows:
snaliance
episode-level
Liotal
Linst +
Lepis
+
ws Louz Linteg
(12)
self-supervised
supervised
where W1, wy are the loss weight hyperparameters.
3.5 INFERENCE
During the test stage, we only exploit the integrated representation Finteg for the final FSL predictt. The predicted class label for 2i Qa can be computed with Eq. (TO) as:
exp(-d(F, hg))
y pred argmaxx yEC. Eoec. exp(-d( h!:))
(13)
3.6 FULL IEPT ALGORITHM
For easy reproduction, we present the full algorithm for FSL with IEPT in Algorithm| | Once learned, with the learned d. can perform the inference over the test episodes with Eq. (13)
Algorithm 1 FSL. with IEPT
Input: The training D., rotation operator set Ç
The loss weight hyperparameters Wi, W2, wa
Output: The learned sb
I: Randomly initialize all learnable parameters {sb, Prot, Oine)
2: for iteration ... Maxlteration do
Randomly sample episode E, from 2>,
4:
Generate the set of extended episodes E' from E, using G
5:
Compute the SSL loss Linst for the instance-level pretext task with Eq.
6:
Compute the SSL loss Lepis for the episode-level pretext task with Eq. 66
Compute the supervised FSL loss Laus over the extended episodes with Eq. Compute the supervised FSL loss Linteg for the integrated episode with Eq.
10
9:
Liotal
Linat + W2 Lepis + Wy Laux + Linteg Update vi based on V Liotat
10:
11:
end for
return v/.
4 EXPERIMENTS
4.1 EXPERIMENTAL SETUP
Datasets. Two widely-used FSL datasets are selected: minilmageNet (Vinyals et al.) 2016) and tieredImageNet (Ren et al. 2018). The first dataset consists of a total number of 100 classes (600 images per class) and the trannvaliationntest split is set to 64/16/20 classes as (Ravi Larochelle 2017). The second dataset is larger dataset including 608 classes totally (nearly 1,200 images per class), which is split into 35 351/97/160 classes for train/validation/tss.. Both datasets are subsets sampled from ImageNet (Russakovsky al. 2015)

---

Published as a conference paper at ICLR 2021
Table I: Comparative results for 5-way 1/5-shot FSL., The mean classification accuracies (top-1, %)
with the 95% confidence intervals are reported. f indicates the result is reproduced by ourselves.
minilmageNet
Iagee
Method
Backbone
I-shot
5-shot
1-shot
5-shot
Matching (Vinyals et al. 2016
Conv4-64
43.56 + 0.84
55.31 + 0.73
ProtoNer! (nell et al. 2017
Conv4-64
52.61 + 0.52
71.33
+ 0.41
53.33 + 0.50
72.10 + 0.41
MAML (Finn ef al. 2017
Conv4-64
48.70 + 1.84
63.10
+ 0.92
51.67
+ 1.81
70.30 + 0.08
Relation Net (Sung ol al
2018
Conv4-64
50.40 + 0.80
65.30
1 0.70
54.48
+ 0.93
71.32 1 0.78
IMP Allen et al. 2019
Conv4-64
52.91 + 0.49
71.37
0.42
53.63
+ 0.51
71.89 + 0.44
DN4 ( ct al. 2119
Conv4-64
51.24 0.74
71.02 0.64
DN PARN (Wü at.
2019
Conv4-64
55.22 + 0.84
71.55 + 0.66
PN+rot Gidaris
2019
Conv4-64
53.63 0.43
71.70 + 0.36
CC+rot Gidaris
2019
Conv4-64
$4.83
71.86 1 0.33
DSN-MR (Simon et a..
2020
Conv4-64
55.88 0.90
70.50 1 0.68
Centroid (frrssyybb:
2020
C4
53.14 1.06
71.45 1 0.72
Neg-Cosine (Liu et 2020
C4
70.41 0.66
(ours)
Conv4-64
56.26
1:26:0000 0.45
73.91
= 0.34
58.25
0.48
75.63 0.46
ProtoNet Snell al.
2017
Conv4-512
53.25
0.44
73.15 + 0.35
57.88
0.50
76.82
0.40
MAMI Finn a. 17
Conv4-512
49.33
0.60
65.17 0.49
52.84
0.56
70.91 + 0.46
Relation Net (Sung
2018
Conv4-512
50.86
0.57
67.32 0.44
54.69
+ 0.59
72.71 + 0.43
PN+rot Gidans al.
Conv4-512
56.02
74.00
0.35
CC+rot Gidams
2019
Conv4-512
56.27 =
74.30
+ 0.33
IEPT (ours)
Conv4-512
58.43
0.46
75.07 +: 0.33
60.91
0.59
79.61 + 0.45
ProtoNer (nell et al.
2017
ResNet-12 12
62.39
0.51
80.53 + 0.42
68.23
+ 0.50
84.03 1 0.41
TADAM Oreshiin et al.
2018
ResNet-12
58.50
0.30
76.70 + 0.38
MetaOptNet (Lee el al. 2019
R2
78.63 1: 0.46
65.99 0.72
81.56 + 0.63
MTL Sun et al.] 2019
ResNet-12
61.20
75.50 + 0.80
65.62 1.80
80.61 + 0.90
CAN (Hou et at. 2019
ResNet-12
63.85
79.44 0.34
69.80
84.23 + 0.37
AM3 (Xing ct al.. [0019]
ResNet-12
65.21
75.20 0.36
67.23
78.95 + 0.22
Shot-Firee Kavichannran et al.
2019
ResNet-12
0.44
77.61
66.87
82.61 0.43
Neg-Cosint (L1u al al.] 2020
12
63.8
81.57
Distill (Tian et |0,00
ResNet-12
6..8
0.414
0.69
86.03 + 0.49
DSN-MK (Simon el al. 2020
ResNet-12 60.7
79.51
67.39
82.85 + 0.56
DeepEMD' (Zhang el al. 2020
ResNet-12 0.82
82.41
0.87
86.03 + 0.58
FEAT (Ye ct 2020
ResNet-12 0.20
82.05
70.80 0.23
84.79 + 0.16
ProtoNat+Kolation (Su et al. 2020
ResNet-18
76.00 + 0.60
78.90 + 0.70
IEPT (ours)
ResNet-12
67. 0.44 82.90 1 0.30
72.24
0.50 86.73 1 0.34
Feature Extractors. For fair comparison with published results, our IEPT adopts three widely-used feature extractors: Conv4-64 (Vinyals et al. 2016). Conv4-512, and ResNet- (He et al., 2016a). Particularly, Conv4-512 is almost the same as Conv4-64 except having different channel size of the last convolution layer. To speed up the training process, as in many previous works YYe et al. 2020 Zhang et al. 2020 Simon et al. 2020), we pretrain all the feature extractors on the training split of each dataset for our IEPT. Following (He et al.) 20)6 we use the temperature scaling skill during the training phase. On both datasets, the input image size is 84 84. The output feature dimensions of Conv4-64, C12v4-512, and ResNet-12 are 64, 512, and 640, respectively.
Evaluation Metries. We take the 5-way 5-shot (or 1-shot) FSL evaluation setting, in previous works. We randomly sample 2,000 episodes from the test split and report the mean classification accuracy (top-1, %) as well as the 95% confidence interval. Since the integration transformer copes with each sample independently, we take strict tonntransdective setting during evaluation.
Implementation Details. PyTorch is used for our implementation. We utilize the Adam optimizer (Kingma & Ba 2015) for Conv4-64 & Conv4-512 and the SGD optimizer for ResNet- 12 to train our IEPT model. The hyperparameters of our IEPT model are selected according the performance on the validation split. We will release the code soon.
4.2 MAIN RESULTS
Comparison to State-of-the-Arts. We compare our IEPT with two groups of baselines: (1) Recent
SSL-based FSL methods (Gidaris et al. 2019 Su al. 2020 (2) Representative/lates FSL methods
(w/o SSL) (Snell et al. Finn et al. 201
Lee et al. 2019 Ravichandran et al., 2019 Simon
et al. 2020 Zhang al. 2020 Ye et al. 2020 Liu et al. 2020
The comparative results for 5-way
7

---

Published as a conference paper at ICLR 2021
Table 2: Ablation study results for our full IEPT model over minilmageNet and rieredlmageNet. Our full model includes two self-supervised losses (i.e. Lepis and Linst) and two supervised losses (i.c. Lauz and Linteg). Conv4-64 is used as the feature extractor.
mini mageNet
tiered tieredImageNt mageNet
Linteg
Linst
Lepis
Laux
1-shot
5-shot
1-shot
5-shot
55.04
0.52
72.01
0.41
56.98
0.47
74.15
0.51
55.49 + 0.56
72.54 0.46
57.41
0.51
74.65
55.88 + 0.43
72.97 + 0.40
57.76
= 0.45
75.06 - 0.40
55.97 + 0.57
73.28
+ 0.39 0.39
57.83 = 0.55
75.22 - 0.48
56.26 + 0.45
73.91 + 0.34
58.25 0.48
75.63
0.46
75
70
LAT EP1
60
hoourbcy
50
40
way way
Swway
Figure 2: (a) Comparison among different combination methods over episodes for FSL with self- supervision. (b) Illustration of the effect of different choices of R on the performance of our model (R denotes the number of extended episodes used for SSL.). (c) Comparative results obtained by our IEPT using different basic FSL classifiers (i.e. ProtoNet, FEAT, and IMP). It can be clearly that integrated episode-based fusion leads to more separation between classes. All figures present 5-way 1-shot/5-shot results minilmageNet, using Conv4-64 as the feature extractor.
1/5-shot FSL are shown Table T We have the following observations: (1) When compared with the representative/lltest FSL methods (w/o SSL), our IEPT achieves the best performance on all datasets and under all settings, validating the effectiveness of SSL. with IEPT for FSL. (2) Our IEPT also clearly outperforms the two SSL-based FSL methods (Gidaris et al. 2019 Su et al. 2020) which only use instance-level pretext tasks, demonstrating the importance closer/episode-leve integration of SSL into FSL. (3) The improvements achieved by IEPT over ProtoNet range from 2% 5%. Since our IEPT takes ProtoNet as the baseline, the obtained margins provide direct evidence that SSL brings significant benefits to FSL. Note that our IEPT is also shown to be effective under both the fine-grained FSL and cross-domain FSL settings in Sec. 4.3 (see Table
Ablation Study. Our full IEPT model is trained with four losses (see Eq. (22), including two self-supervised losses and two supervised losses: the episode-level SSL. loss Lepis- the instance-level SSL loss Cinat, the auxiliary FSL loss Laux and the integrated FSL loss Linteg- To demonstrate the contribution of each loss, we present the ablation study results for our full IEPT model in Table 2 where Conv4-64 used as the backbone. We start with Linteg and then add the additional three losses successively. It can be observed that the performance of our model continuously increases when more losses are used, indicating that each loss contributes to final performance.
4.3 FURTHER EVALUATIONS
Different Combination Methods over Episodes. We have introduced a transformer-basee attention module fuse the features of each instance from all extended episodes (and an integrated episode can be obtained) for the supervised classification task (see Sec. 3.3p. In this experiment, we compare with two alternative ways of across-episode integration: (1) Averaging extended episodes: the extended episodes are directly fused for FSL classification; (2) Averaging all episodes: the extended episodes as well as the integrated episode are fused for FSL classification. We present the comparative results on minilmageNet in Figure 2 a). For comprehensive comparison, the results of FSL with each single extended episode are also reported. We can observe that: (1) The performance of 'Episode 0%* the highest among the four baselines (i.e., FSL with single extended episode), perhaps because the feature extractor is pretrained on the original images without rotation transformations, (2) FSL by averaging extended episodes (i.e., 'Averaging extended episodes") indeed improves each of the four
8

---

Published as a conference paper at ICLR 2021
Table 3: Comparative results for the fine-grained FSL CUB (Wah at.
2011 and the cross-domain
FSL on minilmageNett + CUB.
CUB
minilmageNet
:CUB
Method
Backhone
1-shot
5-shot
1-shot
5-shot
NatchingNet (Vinvals et al.
2016
Conv4-64
61.16 + 0.89
72.86 + 0.70
42.62 + 0.55
56.53 + 0.44
ProtoNet (Snell et al. 2017
Conv4-64
63.72 + 0.22
81.50 = U.15
50.51 = 0.56
69.28 = 0.40
MAML Finn el al. 2017
Conv4-64
55.92 1 0.95
72.09 1 0.76
43.59 1 0.54
54.18 1 0.41
Relation Net Sung el al. 2018
Conv4-64
62.45 1 0.98
76.11 - 0.69
49.84 1 0.54
68.98 - 0.42
FFAT (Ye et at. 2020)
Conv4-64
68.87
0.22
82.90
0.15
51.52
+ 0.54
70.16
+ 0.40
IEPT (ours)
Conv4-64
69.97 0.49
84.33
0.33
52.68
0.56
72.98 0.40
baselines. (3) FSL with integrated episode (i.c., 'Integrated episode") is superior to FSL by simply averaging extended episodes. (4) Comparing 'Integrated episode' with 'Averaging all episodes' the performance of FSL with integrated episode is more stable across different settings, furthering validating the usefulness of our across-episode integration. Overall, the episode-integration module indeed effective in FSL with self-supervision. This is also supported by the visualization results
presented in Appendices A.3
A.4
Different Number of Extended Episodes. all the above experiments, the number of the extended episodes R is set to 4 (rotation by 09, 90% 180", 270%). Figure 2bb) shows the impact of the value of R. Note that when Rt = I, our IEPT model is equivalent to ProtoNet which is without supervupevviioon It can be seen that the performance of model consistently grows when R. increases from to 4. Additionally, the study on exploiting other pretext tasks for our IEPT is presented in Appendix [..]
Different Basic FSL Classifiers. As mentioned in See. 3.1 adopt ProtoNet as the basic FSL classifier due to its scalability and simplicity. To further show the effectiveness of our IEPT when other basic FSL classifiers are used, provide the results obtained by IEPT using ProtoNet, FEAT, and IMP for FSL in Figure C).). It can be clearly observed that our IEPT leads to an improvement of about 1-4% over each basic FSL method (ProtoNet, FEAT, or IMP), indicating that our IEPT can be applied to improve a variety of popular FSL methods.
To evaluate our IEPT algorithm under the fine-gres one-grained FSL and cross-domain FSL settings, we conduct experiments on CUB (Wah et al., ( and minilmageNet + CUB, respectively. For tine-grained FSL on CUB, following Ye et al. 2020), we randomly split the dataset into 100 training classes, 50 validation classes, and test classes. For cross-domain FSL on minilmageNet -> CUB, the 100 training classes from minilmageNet; the 50 validation classes and 50 test classes (using the aforementioned split for fine- -eeeegraindd FSL) are from CUB. Under both settings, we use Conv4-64 as the feature extractor. The 5- 5-way 1/5-shot FSL results are shown in Table 3 Our IEPT clearly achieves the best results, yielding 1-3% improvements over the second- best FEAT. This shows the effectiveness our IEPT under both iine-grained and cross-domain settings.
Comparative Results for Fine-Grained FSL and Cross-Domain FSL.
CONCLUSION
We have proposed a novel Instance-level and Episode-level Pretext Task (IEPT) framework for integrating SSL into FSL. For the first time, have introduced episode-level pretext task for FSI. with self-supervision, in addition to the conventional instance-level pretext task. Moreover, we have also developed an episode extension-inteeration framework by introducing an integration transformer module to fully exploit the extended episodes for FSL. Extensive experiments on two benchmarks demonstrate that the proposed model (i.e., FSL with IEPT) achieves the new state-of-the- art. Our ongoing research directions include: exploring other episode-leve pretext tasks for FSL with self-supervision, and applying FSL with celf-supervision to other vision problems.
ACKNOWLEDGMENTS
This work was supported in part by National Natural Science Foundation of China (61976220 and 61832017), Beijing Outstanding Young Scientist Program (BJJWZYJHO) 1009000020)9)) Open Project Program Foundation of Key Laboratory of Electrocsssssss Information Processing, Chinese Academy of Sciences (0FIP-0-)02006), and Alibaba Innovative Research (AIR) Program.

---

Published as a conference paper at ICLR 2021
REFERENCES
Arman Afrasiyabi, Jean-Framme Françoiss Lalonde, and Christian Gagné. Associative alignment for few-shot image classification. In European Conference on Computer Vision (ECCV), 2020.
Kelsey R. Allen, Evan Shelhamer, Hanul Shin, and Joshua Tenenbaum. Infinite mixture prototypes for few-shot learning. In International Conference on Machine Learning (ICML), pp. 232 222-241, 2019.
Ting Chen, Simon Komblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. International Conference on Learning Representations (ICLR), 2020.
Carl Doersch and Andrew Zisserman. Multi-task self-supervised visual learning. International Conference on Computer Vision (ICCV), pp- 2051-2060, 2017.
Carl Doersch, Abhinav Gupta, and Alexei Efros. Unsupervised visual representation learning by context prediction. In International Conference on Computer Vision (ICCV), pp.
2015.
Li Fei-Fei, Rob Fergus, and Pietro Perona. One-shot learning of object categories. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), pp. 594-611, 2006.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model aadel-agnostic meta- meta-lcarning for fast adaptation of deep networks. In International Conference Machine Learning (ICML). pp. 1126-1135, 2017.
Hang Gao, Zheng Shou, Alireza Zareian, Hanwang Zhang, and Shih-Fu Fu Chang. Low-shot learning via covariance- preserving adversarial augmentation networks. In Advances in Neural Information Processing Systems, pp. 975 175-985,222 2018.
Spyros Gidaris and Nikos Komodakis. Dynamic few-shot visual learning without forgetting. In IEEE Conference on Computer Vision and Puttern Recognition (CVPR), pp. 4367-4375 2018
Spyros Gidaris and Nikos Komodakis. Generating classification weights with gnn denoising autoencoders for few-shot learning. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 21-30, 2019.
Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised vesupervisedeeppeseeaatttiii learning by predicting image rotations. In International Conference on Learning Representations (ICLR), 2018.
Spyros Gidaris, Andrei Bursuc, Nikos Komodakis, Patrick Perez, and Matthieu Cord. Boosting few-shot visual learning with self supervupevision. In International Conference on Computer Vision (ICCV), pp. 8059 8068, 2019.
Bharath Hariharan and Ross Girshick. Low-shot visual recognition by shrinking and hallucinating features. In International Conference on Computer Vision (ICCV), pp. 3037-3046, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp- 770-778, 2016a.
Kaiming Hc, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In IEEE Conference on Computer Vision and Puttern Recognition (CVPR), pp. 770-778, 2016b.
Ruibing Hou, Hong Chang, MA Bingpeng. Shiguang Shan, and Xilin Chen. Cross attention network for few-shot classification. Advances in Neural Information Processing Systems, pp. 4003-4014, 2019.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In IEEE Conference on Computer Vision and Puttern Recognition (CVPR), pp. 4700-4708, 2017.
Satoshi lizuka, Edgar Simo-Serra, and Hiroshi Ishikawa. there be color! joint end-to-end learning global and local image priors for automatic image colorization with simultancous classification. ACM Transactions on Graphics (ToG), pp. 1-11, 2016.
Xu Ji. João Henriques, and Andrea Vedaldi. Invariant information clustering for unsupervised image clas- sification and segmentation. In International Conference on Computer Vision (ICCV), pp. 9865-996899 2019.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR), 2015.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey F. Hinton. Imagenet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems, pp. 1097-1105, 2012.
10

---

Published as a conference paper at ICLR 2021
Gustav Larsson, Michael Maire, and Gregory Shakhnarovich. Learning representations for automatic colorization. In European Conference on Computer Vision (ECCV), pp. 577-593, 2016.
Kwonjoon Lee, Subhransu Maji, Avinash Ravichandran, and Stefano Soatto. Meta-lcarning with differentiable convex optimization. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10657- 10665, 2019.
Hongyang Li, David Eigen, Samuel Dodge, Matthew Zeiler, and Xiaogang Wang. Finding task-relevant features for few-shot learning by category traversal. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10, 2019a.
Wenbin Li, Lei Wang, Jinglin Xu, Jing Huo, Yang Gao, and Jieho Luo. Revisiting local descriptor based image- to-class measure for few-shot learnig. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7260-7268, 2019b.
Bin Liu, Yue Cao, Yutong Lin, Qi Li, Zheng Zhang, Mingsheng Long, Han Hu. Negative margin matters: Understanding margin in few-shot classification. In European Conference on Computer Vision (ECCV), 2020.
Leland MeInnes, John Healy, and James Melville. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint arXiv: (802:0380033 2018.
Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, and Pieter Abbeel. A simple neural attentive meta-learner. In International Conference on Learning Representations (ICLR), 2018.
Tsendsuren Munkhdalai and Hong Yu. Meta networks. In International Conference on Machine Learning (ICML), pp.
2017.
T Nathan Mundhenk, Daniel Ho, and Barry Y Chen. Improvements to context based self-supervised learning. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp- 9339-9348, 2018.
Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. In
European Conference on Computer Vision (ECCV), pp. 69-84, 2016
Mehdi Noroozi, Ananth Vinjimoor, Paolo Favaro, and Hamed Pirsiavash. Boosting self supfrvupe learning via knowledge transfer. IEFE Conference on Computer Vision and Pattern Recognition (CVPR), pp- 9359-9367, 2018.
David Novotny, Samuel Albanie, Diane Larlus, and Andrea Vedaldi, Self-supervised learning of geometrically stable features through probabilistic introspection. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3637-3645, 2018.
Boris Oreshkin, Pau Rodríguez López, and Alexandre Lacoste. Tadam: Task dependent adaptive metric for improved few-shot learning. In Advances Neural Information Processing Systems, pp. 721-731, 2018
2018.
Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei Efros. Context encoders: Feature learning by inpainting. In JEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2536-2544, 2016.
Hang Qi. Matthew Brown, and David G Lowe. Low-shot learning with imprinted weights. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5822-5830, 2018
Siyuan Qiao, Chenxi Liu, Wei Shen, and Alan L Yuille. Few-shot image recognition by predicting parameters from activations. In IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp. 7229-7238, 2018.
Sachin Ravi and Hugo Larochelle. Optimization model for few-shot learning. In International Conference on Learning Representations (ICLR), 2017.
Avinash Ravichandran, Rahul Bhotika, and Stefano Soatto. Few-shot learning with embedded class models and shot-free meta training. In International Conference on Computer Vision (ICCV), pp. 331-339, 2019.
Mengye Ren, Eleni Triantafillou, Sachin Ravi, Jake Snell, Kevin Swersky, Joshua Tenenbaum, Hugo Larochelle, and Richard S. Zemel. Meta-learning for semi-supervised few-shot classification. In International Conference on Learning Representations (ICLR), 2018
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh. Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bemstein, Alexander C. Berg, and Li Fei-Fei. Imagenet large scale visual recognition challenge. In International Journal of Computer Vision (IJCV), pp. 211-252, 2015.
11

---

Published as a conference paper at ICLR 2021
Andrei A Rusu, Dushyant Rao, Jakub Sygnowski, Oriol Vinyals. Razvan Pascamu, Simon Osindero, and Raia Hadsell. Meta-leaming with latent embedding optimization. In International Conference on Learning Representations (ICLR), 2019.
Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, and Timotyy Lillicrap. Meta-learning with memory-augmentee neural networks. In International Conference on Machine Learning (ICML), pp- 1842-1850, 2016.
Eli Schwartz, Leonid Karlinsky, Joseph Shtok, Sivan Harary, Mattias Marder, Abhishek Kumar, Rogerio Feris, Raja Giryes, and Alex Bronstein. Delta- encoder: an effective sample synthesis method for few-shot object recognition. In Advances in Neural Information Processing Systems, pp. 2850-2860, 2018.
Christian Simon, Piotr Koniusz, Richard Nock, and Mchrtash Harandi. Adaptive subspaces for fow-shot learning. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 11336-1145 2020.
Jake Snell, Kevin Swersky. and Richard Zemel. Prototypical networks for few-shot learning. In Advances in Neural Information Processing Systems, pp. 4077-4087, 2017.
Jong-Chyy Su, Subhransu Maji, and Bharath Hariharan. When does self superviinn improve few-shot lcarning? In European Conference Computer Vision (ECCV), 2020.
Qianru Sun, Yaoyao Liu, Tat-Seng Chua, and Bernt Schiele. Meta-transfer learning for few-shot learning. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 403-412, 2019.
Flood Sung, Yongxin Yang, Li Zhang, Tao Xiang, Philip HS Torr, and Timothy M Hospedales. Learning to compare: Relation network for few-shot learning. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1199-1208, 2018
Yonglong Tian, Yue Wang, Dilip Krishnan, Joshua B Tenenbaum, and Phillip Isola. Rethinking few-shot image classification: good embedding all you need? In European Conference on Computer Vision (ECCV), 2020.
Satoshi Tsutsui, Yanwei Fu, and David Crandall. Meta-reinforeed synthetic data for onc-shot fine-grained visual recognition. In Advances in Neural Information Processing Systems, pp. 3057-3066, 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones. Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in Neural Information Processing Systems, pp- 5998-6008, 2017.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, koray kavukcuoglu, and Daan Wicrstra. Matching networks for one shot learning. In Advances in Neural Information Processing Systems, 9630-3638, 2016.
C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongic. The caltech-uesd birds-200-2011 dataset. Technical Report CNS-TR-200 1-01,, California Institute of Technology, 2011. 1.
Yu Yu-Xiong Wang, Ross Girshick, Martial Hebert, and Bharath Hariharan. Low-shot learning from imaginary data. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp.
2018
Ziyang Wu, Yuwei Li, Lihua Guo, and Kui Jia, Parn: Position aware relation networks for few-shot learning. In International Conference on Computer Vision (ICCV), pp. 6659-6667, 2019.
Chen Xing, Negar Rostamzadeh, Boris Oreshkin, and Pedro O O. Pinheiro. Adaptive cross- modal few-shot learning. In Advances in Neural Information Processing Systems. pp. 4847-4857, 2019.
Han-Jia Ye, Hexiang Hu, De-Chuan Zhan, and Fei Sha. Fhw-shot learning via embedding adaptation with set-to- set functions. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8808 8817, 2020.
Chi Zhang, Yujun Cai, Guosheng Lin, and Chunhua Shen. Deepemd: Few-shot image classification with differentiable earth mover's distance and structured classifiers. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 12203-122022 12211, 2020.
Hongguang Zhang. Jing Zhang, and Piotr Koniusz. Fhw-shot learning via saliency-guider hallucination of samples. In IEFF Conference on Computer Vision and Pattern Recognition (CVPR). pp- 2770-2779, 2019.
12

---

Published as a conference paper at ICLR 2021
A APPENDIX
A.1 COMPARISON AMONG DIFFERENT SSL STRATEGIES
To generate the extended episodes in IEPT, we apply four rotation transformations (i.e. rotation by 0% 909, 180", 270%) to each image. It makes sense to explore whether other self-supervised strategies are also effective for our IEPT. To this end, we exploit shuffling image patches (see Figure 3 for self-supervised learning (SSL). Specifically, we divide each image into 2*2 patches and reorganize the patch orders to obtain a shuffling label. To compare with the rotation strategy fairly, we choose only four shuffling orders: (1. 2, 3, 4), (2. 3, 4. 1). (3, 4, 1, 2) and (4, 2, 3). Note that the (1, 2, 3, 4) shuffling order equals to the original image. Similar to the rotation strategy, a fully- connected layer is utilized to recognize the shuffling order. The comparative results are shown in Table 4 We can see that both IEPT with shuffling and IEPT with rotation achieve better performance than the original ProtoNet. Particularly, IEPT with shuffling yields 1-3% and 3-4% improvements under 5-shot and 1-shot, respectively. This clearly shows the effectiveness of our IEPT for FSL even when different SSL strategies are used to define the pretext tasks.
Table 4: FSL results obtained by our IEPT using two SSL strategies (i.e. rotation and shuffling image patches). Conv4-64 used the feature extractor.
minilmageNNe
IiaredImageNee
Method
Backbone
1-shot
5-shol
1-shot
5-shol
ProtoNet
Conv4-64
52.61
0.52
71.33 + 0.41
53.33
0.50
72.10
0.41
IEPT (rotation)
Conv4-64 64
56.26 0.45
73.91
0.34
58.25
0.48
75.63 = 0.46
IEPT (shuffling)
Conv4-64
55.57 0.60
72.84
0.54
57.81 0.48
74.92
0.50
Figure 3: Illustraiion of the self-supervised strategy by shuffling image patches.
A.2 COMPARISON AMONG DIFFERENT INTEGRATION APPROACHES
We employ the integration transformer to find the intrinsic correlation of various rotation-transformmd instances. The transformer architecture is based on a self-attention mechanism. Concretely, receives the feature sets of extended episodes input Q. K and V. Further, it matches each query in Q with a list of keys in K and returns the weighted sum of corresponding values. To show the importance of the transformer module, we compare it with two other integration approaches (i.e. concatenating and averaging) to integrate the fcatures of extended episodes. The comparative results in Table 5 demonstrate that the integration transformer consistently performs better than the simply concatenatingl concatenating/averaiin approaches. This suggests that the attention-based integration transformer is a better choice for designing the integration module.
Table 5: Comparative results obtained by three different approaches integrating the features from the extended episodes, with Conv4-64 being the feature extractor.
miniImageNet
lieee Net
Method
Backbone
1-shot
5-shot
1-shot
5-shot
concatenating
Conv4-64
51.52 + 0.60
73.36 1 0.49
50.78 1 0.68
72.79 1 01.7
averaging
Conv4-64
51.58 + 0.62
70.97
0.54
53.91 + 0.69
72.52 + 01.99
transformer (ours)
Cony4-64
56,26 + 0.45
73.91
0.34
58.25 + 0.48
75.63 + 0.46
13

---

Published as a conference paper at ICLR 2021
Figure 4: Feature visualizations of number of test episodes using the UMAP algorithm (Melnnes et al. 2018). Each row indicates a group of test extended episodes (the first four columns, rotation by 0" 90% 180% 270") and their integrated episode (the last column). The 5-way 5-shot FSL (with Conv4-64) is adopted minilmageNet.
A.3 FEATURE VISUALIZATIONS OF TEST EPISODES
We provide the feature visualizations of test episodes in Figure It can be seen that an integrated episode (the last one in each row) clearly has better cluster data structure than the corresponding four extended episodes (the first four ones in each row). This indicates that our transformer-basee across-episode integration is indeed effective for few-shot classification with self-supervision.
A.4 ATTENTION VISUALIZATION OF TEST EPISODES
We present attention map visualization of two test episodes (left and right) in Figure 5 Each average attention map is computed by averaging the attention map of all instances of a certain class. We can observe that: (1) The average attention maps from different classes vary significantly, showing that the diverse semantics of different classes can be reflected by our attention- based integration transformer. (2) When the classes of two episodes overlap (e.g., 'trifle" and 'dalmatian') ) the average attention maps of an overlapped class from two episodes are similar, illustrating that our attention based integration transformer well capture the semantics of classes across episodes.
14

---

Published as a conference paper at ICLR 2021
Attention
Altintion
thaoder
trile
samilldd Colimtiin
tittle
bookshbpp
brfle
Viee
damnstion
Figure 5: Feature and attention visualization of two test episodes (left and right). Both figures present 5-way 5-shot results on minilmageNet, using Conv4-64 as the feature extractor.
A.5 COMPARISON WITH SIMPLE BASELINE FOR SSL+FSL
We compare our IEPT with a simple baseline that trains the model with Laua Linst and then makes inference by just averaging the outputs of different extended episodes. The results shown in Table 6 It can be observed that the performance of our IEPT is much more effective than that of simple integration, due to the extra use of Linteg + Lepie for FSL.
Table 6: Comparison with the simple baseline that trains the model with Laue + Linst and then makes inference by just averaging the outputs different extended episodes.
minilmageNNe
Imageee
Method
Backbone
1-shot
5-shot
1-shot
5-shot
Lawe + Linet
Conv4-64
53.25 1 0.46
71.50 = 0.42
55.06
0.44
72.87
0.42
IEPT (ours)
Cony4-64
56.26 + 0.45
73.91
0.34
58.25 + 0.48
75.63 + 0.46
A.6 DIFFERENT ALTERNATIVES OF SELF-SUPERVISED LOSSES
In Table we provide further ablation study regarding different alternatives of Lepis and Linst- For the episode-level self-supervised loss Lepis. we compare our implementation (using the KI. loss between each distribution and the mean distribution) with that using pairwise KL loss (i.c., the KL loss between each pair of distributions). For the instance-level self-supervised loss Linsts we compare our implementation (using the rotation prediction loss) with the recent self-supervised learning technique (Chen et al., 2020). We observe that our implementation achieves slight performance improvements over those using the pairwise KL loss or the contrastive learning loss.
Table 7: Ablation study results regarding different alternatives of Lepis and Linst- minilmageNNtt
lieredImageNet
Method
Backbone
1-shot
5-shot
1-shot
5-shot
IEPT (CinarSimCLR)
Conv4-64
56.04 0.44
73.67 = 0.41
58.24
+ 0.43
75.59 + 0.41
IEPT (Legis- Pairwisc)
Conv4-64
55.95 + 0.47
73.72 - 0.40
57.91
= 0.45
75.28 + 0.40
IEPT (ours)
Conv4-64
56.26 0.45
73.91 0.34
58.25 0.48
75.63 + 0.46
A.7 APPLICATION OF IEPT TO OPTIMIZATION
METHOD MAML
In Table 8 we show the results obtained by applying our IEPT to the optimization based model MAML (Finn al. 2017). We use Conv4-64 as the feature extractor. We can see that our IEPT brings 0.7%-2.11 improvements to MAML. This further shows the flexibility (as well as effectiveness) of IEPT for FSL..
15

---

Published as a conference paper at ICLR 2021
Table 8: Comparative results by applying IEPT to the optimization-baseed method MAMI..
minilmageNet
tieredImageNett
Method
Backbone
1-shot
5-shot
1-shot
5-shot
MAML
Conv4-64
48.70
1.84
63.10
0.92
51.67
1.81
70.30 + 0.80
MAML+TEPT
Conv4-64
49.68
0.50
65.22 + 0.48
52.85 + 0.52
71.04 + 0.49
Acqusayy (86
Accuracy 196)
52
90
68
0.1
1.0
5.0
10.0
0.1
0.5
1.0
5.0
10.0
Value
Value
Figure 6: Visualization of our hyper-parameter analysis under 5-way 1-shot (left) and 5-shot (right)
on mönilmageNet. Conv4-64 is used as the feature extractor.
A.8 HYPER-PARAMETER SENSITIVITY TEST
We select the hyper- parameters W1, W2 and Wy from the candidate set (0.1,0.5. 5.0, 10.0) and show the apper-parameter analysis results in Figure 6 We find that the performance relatively stable. Concretely, the performance of our TEPT is not sensitive to wi and w2 with proper values, but too large wi (i.e. wh = 10.0) tends to cause obvious degradation, perhaps because the FSL. task is biased by the rotation prediction loss.
IEPT is
16


Reviews:
[{"review_id": "u1sQ-9IPiMY", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes both Instance-level and episode-level pretext task. In comparison to existing works (Gidaris et al., 2019; Su et al., 2020), the main novelty is to design the episode-level pretext task, which enforces consistent predictions for images with different rotations. \n\nThe paper is clearly written with experiments supporting the effectiveness. \nHowever, the novelty is limited. It is more like existing works (Gidaris et al., 2019; Su et al., 2020) plus the the regularization of consistency for images with different augmentations. However, the latter is also not new. Indeed, it has been used in [1-2], but the authors neglect them. \n\n\n[1] Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averaged consis- tency targets improve semi-supervised deep learning results. In NeurIPS, 2017.\n[2] Samuli Laine and Timo Aila. Temporal ensembling for semi-supervised learning. arXiv, 2016.\n[3] Mehdi Sajjadi, Mehran Javanmardi, and Tolga Tasdizen. Regularization with stochastic transformations and perturbations for deep semi-supervised learning. In NeurIPS, 2016.\n\n\n======\nComments after rebuttal: \nI know the authors develop two components for FSL, my concern is that these components are incremental and have limited novelty. \nHowever, I admit this paper is a high quality paper in presenting its idea, organization and empirical evaluation. Hence I increase my score to accept now. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "ppmjBbA2u_", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In Instance-Level and Episode-Level Pretext Tasks for FSL the authors present a novel method to take advantage of auxiliary prediction tasks and consistency regularization tasks which have had large success in Self-Supervised Learning settings to improve upon FSL approaches. Furthermore, the authors incorporated a transformer-based predictor to improve upon to be used with multiple augmentations of an instance to improve upon naive averaging of multiple predictions.\n\nEmpirically, the authors demonstrate the benefits of incorporating both instance and episode-level tasks by showing significant improvements over the ProtoNet approach upon which this work builds and also achieving state-of-the-art results on several tasks with different architectural backbones. Through ablations this work demonstrates the benefit of each proposed additional loss and shows robustness to choices in pretext transformation and architectural backbone. \n\n\nThis work is appropriately justified and explained and with satisfactory experimentation.\n\nFor section 3.3, it may help to explain that the integrated FSL task is presented as an alternative to prediction averaging. As written, the rationale for this approach is only apparent in subsequent sections.", "extra_scores": {}, "extra_text": {}}, {"review_id": "WFioOP7iq0e", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper addresses the problem of few-shot classification by incorporating self-supervised learning into the standard episode-based meta learning. Specifically, it adopts the pretext task of rotation prediction into the episode design. For each sampled episode, additional episodes are constructed by using rotated examples in the original support set and query set. Two self-supervised losses are designed based on the augmented episode sets – (1) recognizing different rotation transformations as an instance-level pretext task; (2) ensuring consistent predictions of class labels across different episodes as an episode-level pretext task. Two few-shot losses are also designed – (1) predicting class labels for each individual episode; (2) predicting class labels for the fused episode set based on attention. Experimental evaluation is conducted on two standard few-shot classification benchmarks, namely miniImageNet and tieredImageNet, and shows improved performance.\n\nStrengths:\n\n++ Self-supervised learning and few-shot learning are two important techniques to transfer knowledge for addressing new tasks, but their combination is less explored.\n\n++ The proposed approach that integrates self-supervised learning into the episode design is interesting.\n\nSuggestions and questions:\n\t\n-- Compared with the state-of-the-art methods, the performance improvements of the proposed approach are marginal. For example, while the proposed approach is more complicated, it is comparable to much simpler approach like Tian et al.\n\n-- The proposed way of integrating self-supervised learning into episode learning is claimed to be general. In addition to the metric-learning based approach ProtoNet, it would be interesting to show if the proposed approach can be applied to other types of meta-learning techniques, such as optimization based approach MAML.\n\n-- Following the previous comment, it would be interesting to show if the proposed approach can be applied to other self-supervised learning techniques, such as the recent MoCo or simCLR models.\n\n-- In the design of a set of extended episodes, examples within the same episode belong to the same rotation transformation. How if this is not guaranteed? That is, the examples within the episode are randomly sampled from the rotation transformations.\n\n-- In Eq 6, how if using a pairwise KL loss without computing the mean distribution in Eq 5?\n\n-- Why further introducing an auxiliary loss is helpful, given that the attention mechanism already fuses the information from all the episodes?\n\n-- How is the hyperparameter sensitivity regarding different ws in Eq 12?\n\n-- Table 2 did not provide extensive ablations regarding different combinations of the loss functions.\n\nPost-comments to the author's response:\n\nAfter reading the other reviewers’ comments and the authors’ rebuttal, I am still concerned with the novelty of the approach, experimental evaluation, and performance improvements over previous work. These issues are pointed out by the other reviewers as well. Hence, I will go with my original decision of rejecting the paper.", "extra_scores": {}, "extra_text": {}}, {"review_id": "WASybGbsmAL", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a method for combining self-supervised learning (SSL) (in the form of predicting the rotation applied to an image) with few-shot learning (FSL) in the domain of image classification. Compared to prior work, this paper introduces -- (i) a consistency loss which ensures FSL episodes with different rotations agree in their class predictions; and (ii) an integration method which derives the label of an image from a fused representation of all its rotations. These lead to an improvement over 3 FSL benchmarks.\n\nStrengths:\n- The paper is well-written, with extensive discussion of prior and contemporary work. The technical details are presented in clear precise terms. Despite not being an expert in this area, I had no difficulty understanding the FSL setup and the new contributions of this paper.\n\n- The experiments are also quite thorough with convincing ablation studies and several additional details in the Appendix. Overall this is solid empirical work.\n\n- The paper presents relatively simple ideas which lead to significant improvements. Hence, it is likely to be impactful for future work looking to build on these results.\n\nWeaknesses:\n- Though effective, the novel loss terms introduced seem rather ad-hoc. There is not much understanding gained from reading the paper about why these extensions help. A large part of the improvement over the baseline ProtoNet seems to come from the integration method (based on the ablation study). This is very interesting, but could have been explored in more depth. E.g. does this approach also work when we have many training instances?\n\n- There is not much discussion of which hyperparameters were tuned, neither how sensitive the results are to this tuning.\n\nOther comments:\n- It is not clear what the visualization in Figure 3 represents, or how it \"supports the effectiveness of episode-integration module\" (section 4.3). Are the figures comparable to each other?\n\n- Why do we need the extra mean function in Eq. 6, when there is already a summation over i and r?\n\n- Please spell out the conference names in the references.", "extra_scores": {}, "extra_text": {}}, {"review_id": "OOgQIBg-19", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper solves the problem of few-shot learning. The recent success of SSL and FSL proves that they can handle situations that few label data are provided. Motivated by this, the author proposed a novel framework IEPT that seamlessly integrates self-supervised learning methods to few-shot learning. Unlike other trivial combination of SSL and FSL methods, this paper proposed instance-level and episode-level pretext tasks to bring on closer integration. Further, this paper proposed to use transformer to integrate features from different images and augmentations. Experiments show the model achieves new SOTA. \n\n1. In the inference phase, you use transformer to integrate embedding from both support set and query set, which seemingly makes a transductive method. Therefore, you should compare your method to other transductive FSL methods.\n2. Your ablation experiments are not complete. You are supposed to give results of training with L_{integ} (Eq.(10)) and L_{epis}  (Eq.(6)).\n3. There is at least a baseline method you should compare. That is you train with L_{aux} (Eq.(11)) and L_{inst} (Eq.(3)), and inference by averaging outputs (ensemble).\n4. You are using multiple augmentations on each image at test time. It seems unfair to most previous tasks. It is not clear if the success is due to the ensemble effect.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The submission proposes instance-level and episode-level pretext tasks as an unsupervised data augmentation mechanism for few-shot learning. Furthermore, transformer are proposed to integrate features from different images and augmentations. The paper received one clear accept, one accept, one borderline accept and two borderline reject recommendations. The main concerns of the R5 and R2 were weak ablation study and the lack of a clear advantage of the method in terms of results compared to the prior state of the art. In the rebuttal, the authors provided more ablation studies. Similarly, the reviewers were concerned about the novelty of the paper being incremental compared to the prior works. Based on the majority vote, the meta reviewer recommends acceptance.

Decision: Accept (Poster)