IEPT: Instance-Level and Episode-Level Pretext Tasks for Few-Shot Learning

1 INTRODUCTION . Deep convolutional neural networks ( CNNs ) ( Krizhevsky et al. , 2012 ; He et al. , 2016b ; Huang et al. , 2017 ) have seen tremendous successes in a wide range of application fields , especially in visual recognition . However , the powerful learning ability of CNNs depends on a large amount of manually labeled training data . In practice , for many visual recognition tasks , sufficient manual annotation is either too costly to collect or not feasible ( e.g. , for rare object classes ) . This has severely limited the usefulness of CNNs for real-world application scenarios . Attempts have been made recently to mitigate such a limitation from two distinct perspectives , resulting in two popular research lines , both of which aim to transfer knowledge learned from the data of a set of source tasks to a new target one : few-shot learning ( FSL ) and self-supervised learning ( SSL ) . FSL ( Fei-Fei et al. , 2006 ; Vinyals et al. , 2016 ; Finn et al. , 2017 ; Snell et al. , 2017 ; Sung et al. , 2018 ) typically takes a ‘ learning to learn ’ or meta-learning paradigm . That is , it aims to learn an algorithm for learning from few labeled samples , which generalizes well across any tasks . To that end , it adopts ∗Corresponding author . an episodic training strategy – the source tasks are arranged into learning episodes , each of which contains n classes and k labeled samples per class to simulate the setting for the target task . Part of the CNN model ( e.g. , feature extraction subnet , classification layers , or parameter initialization ) is then meta-learned for rapid adaptation to new tasks . In contrast , SSL ( Doersch et al. , 2015 ; Noroozi & Favaro , 2016 ; Iizuka et al. , 2016 ; Doersch & Zisserman , 2017 ; Noroozi et al. , 2018 ) does not require the source data to be annotated . Instead , it exploits an annotation-free pretext task on the source task data in the hope that a task-generalizable feature representation can be learned from the source tasks for easy adoption or adaptation in a target task . Such a pretext task gets its self-supervised signal at the per-instance level . Examples include rotation and context prediction ( Gidaris et al. , 2018 ; Doersch et al. , 2015 ) , jigsaw solving ( Noroozi & Favaro , 2016 ) , and colorization ( Iizuka et al. , 2016 ; Larsson et al. , 2016 ) . Since these pretext tasks are class-agnostic , solving them leads to the learning of transferable knowledge . Since both FSL and SSL aim to reduce the need of collecting a large amount of labeled training data for a target task by transferring knowledge from a set of source tasks , it is natural to consider combining them in a single framework . Indeed , two recent works ( Gidaris et al. , 2019 ; Su et al. , 2020 ) proposed to integrate SSL into FSL by adding an auxiliary SSL pretext task in an FSL model . It showed that the SSL learning objective is complementary to that of FSL and combining them leads to improved FSL performance . However , in ( Gidaris et al. , 2019 ; Su et al. , 2020 ) , SSL is combined with FSL in a superficial way : it is only taken as a separate auxiliary task for each single training instance and has no effect on the episodic training pipeline of the FSL model . Importantly , by ignoring the class labels of samples , the instance-level SSL learning objective is weak on its own . Since meta-learning across episodes is the essence of most contemporary FSL models , we argue that adding instance-level SSL pretext tasks alone fails to exploit fully the complementarity of the aforementioned FSL and SSL , for which a closer and deeper integration is needed . To that end , in this paper we propose a novel Instance-level and Episode-level Pretext Task ( IEPT ) framework for few-shot recognition . Apart from adding an instance-level pretext SSL task as in ( Gidaris et al. , 2019 ; Su et al. , 2020 ) , we introduce two episode-level SSL-FSL hybrid learning objectives for seamless SSL-FSL integration . Concretely , as illustrated in Figure 1 , our full model has three additional learning objectives ( besides the standard FSL one ) : ( 1 ) Different rotation transformations are applied to each original few-shot episode to generate a set of extended episodes , where each image has a rotation label for the instance-level pretext task ( i.e. , to predict the rotation label ) . ( 2 ) The consistency across the predictions of an FSL classifier from different extended episodes is maximized as an episode-level pretext task . For each training image , the rotation transformation does not change its semantic content and hence its class label ; the FSL classifier predictions across different extended episodes thus should be consistent , hence the consistency regularization objective . ( 3 ) The correlation of features across instances from these extended episodes is modeled by a transformer-based attention module , optimizing the fusion of the features of each instance/image and its various rotation-transformed versions mainly for task adaptation during meta-testing . Importantly , with these three new learning objectives introduced in IEPT , any meta-learning based FSL model can now benefit more from SSL by fully exploiting their complementarity . Our main contributions are : ( 1 ) For the first time , we propose both instance-level and episode-level pretext tasks ( IEPT ) for integrating SSL into FSL . The episode-level pretext task enables episodic training of SSL and hence closer integration of SSL with FSL . ( 2 ) In addition to these pretext tasks , FSL further benefits from SSL by integrating features extracted from various rotation-transformed versions of the original training instances . The optimal way of feature integration is learned by a transformer-based attention module , which is mainly designed for task adaptation during meta-testing . ( 3 ) Extensive experiments show that the proposed model achieves the new state-of-the-art . 2 RELATED WORK . Few-Shot Learning . The recent FSL studies are dominated by meta-learning based methods . They can be divided into three groups : ( 1 ) Metric-based methods ( Vinyals et al. , 2016 ; Snell et al. , 2017 ; Sung et al. , 2018 ; Allen et al. , 2019 ; Xing et al. , 2019 ; Li et al. , 2019a ; b ; Wu et al. , 2019 ; Ye et al. , 2020 ; Afrasiyabi et al. , 2020 ; Liu et al. , 2020 ; Zhang et al. , 2020 ) aim to learn the distance metric between feature embeddings . The focus of these methods is often on meta-learning of a feature-extraction CNN , whilst the classifiers used are of simple form such as a nearest-neighbor classifier . ( 2 ) Optimization-based methods ( Finn et al. , 2017 ; Ravi & Larochelle , 2017 ; Rusu et al. , 2019 ; Lee et al. , 2019 ) learn to optimize the model rapidly given a few labeled samples per class in the new task . ( 3 ) Model-based methods ( Santoro et al. , 2016 ; Munkhdalai & Yu , 2017 ; Mishra et al. , 2018 ) focus on designing either specific model structures or parameters capable of rapid updating . Apart from these three groups of methods , other FSL methods have attempted feature hallucination ( Schwartz et al. , 2018 ; Hariharan & Girshick , 2017 ; Gao et al. , 2018 ; Wang et al. , 2018 ; Zhang et al. , 2019 ; Tsutsui et al. , 2019 ) which generates additional samples from the given few shots for network finetuning , and parameter predicting ( Qiao et al. , 2018 ; Qi et al. , 2018 ; Gidaris & Komodakis , 2019 ; 2018 ) which learns to predict part of the parameters of a network given few samples of new classes for quick adaptation . In this work , we adopt the metric-based Prototypical Network ( ProtoNet ) ( Snell et al. , 2017 ) as the basic FSL classifier for the main instantiation of our IEPT framework due to its simplicity and popularity . However , we show that any meta-learning based FSL method can be combined with our IEPT ( see results in Figure 2 ( c ) ) . Self-Supervised Learning . In SSL , it is assumed that the source task data is label-free and a pretext task is designed to provide self-supervision signals at the instance-level . Existing SSL approaches differ mainly in the pretext task design . These include predicting the rotation angle ( Gidaris et al. , 2018 ) and the context of image patch ( Doersch et al. , 2015 ; Nathan Mundhenk et al. , 2018 ) , jigsaw solving ( Noroozi & Favaro , 2016 ; Noroozi et al. , 2018 ) ( i.e . shuffling and then reordering image patch ) , and performing images reversion ( Iizuka et al. , 2016 ; Pathak et al. , 2016 ; Larsson et al. , 2016 ) . SSL has been shown to be beneficial to various down-steam tasks such as semantic object matching ( Novotny et al. , 2018 ) , object segmentation ( Ji et al. , 2019 ) and object detection ( Doersch & Zisserman , 2017 ) by learning transferable feature presentations for these tasks . Integrating Self-Supervised Learning into Few-Shot Learning . To the best of our knowledge , only two recent works ( Gidaris et al. , 2019 ; Su et al. , 2020 ) have attempted combining SSL with FSL . However , the integration of SSL into FSL is often shallow : the original FSL training pipeline is intact ; in the meantime , an additional loss on each image w.r.t . a self-supervised signal like the rotation angle or relative patch location is introduced . With pretext tasks solely at the instance level , combining the two approaches ( i.e. , SSL and FSL ) can only be superficial without fully exploiting the episodic training pipeline unique to FSL . Different from ( Gidaris et al. , 2019 ; Su et al. , 2020 ) , we introduce an episode-level pretext task to integrate SSL into the episodic training in FSL fully . Specifically , the consistency across the predictions of an FSL classifier from different extended episodes is maximized to reflect the fact that various rotation transformations should not alter the class-label prediction . Moreover , features of each instance and its various rotation-transformed versions are now fused for FSL classification , to integrate SSL with FSL for the supervised classification task . Our experimental results show that thanks to the closer integration of SSL and FSL , our IEPT clearly outperforms ( Gidaris et al. , 2019 ; Su et al. , 2020 ) ( see Table 1 ) . 3 METHODOLOGY . 3.1 PRELIMINARY . Problem Setting . Given an n-way k-shot FSL task sampled from a test set Dt , to imitate the test setting , an FSL model is typically trained in an episodic way . That is , n-way k-shot episodes are randomly sampled from a training set Ds , where the class label space of Ds has no overlap with that of Dt . Each episode Ee contains a support set Se and a query set Qe . Concretely , we first randomly sample a set of n classes Ce from the training set , and then generate Se andQe by sampling k support samples and q query samples from each class in Ce , respectively . Formally , we have Se = { ( xi , yi ) |yi ∈ Ce , i = 1 , ... , n × k } and Qe = { ( xi , yi ) |yi ∈ Ce , i = 1 , ... , n × q } , where Se ⋂ Qe = ∅ . For simplicity , we denote lk = n × k and lq = n × q . In the meta-training stage , the training process has an inner and an outer loop in each episode : in the inner loop , the model is updated using Se ; its performance is then evaluated on the query set Qe in the outer loop to update the model parameters or algorithm that one wants to meta-learn . Basic FSL Classifier . We employ ProtoNet ( Snell et al. , 2017 ) as the basic FSL model . This model has a feature-extraction CNN and a simple non-parametric classifier . The parameter of the feature extractor is to be meta-learned . Concretely , in the inner loop of an episode , ProtoNet fixes the feature extractor and computes the mean feature embedding for each class as follows : hc = 1 k · ∑ ( xi , yi ) ∈Se fφ ( xi ) · I ( yi = c ) , ( 1 ) where class c ∈ Ce , fφ is a feature extractor with learnable parameters φ , and I is the indicator function . By computing the distance between the feature embedding of each query sample and that of the corresponding class , the loss function used to meta-learn φ in the outer loop is defined as : Lfsl ( Se , Qe ) = 1 |Qe| ∑ ( xi , yi ) ∈Qe − log exp ( −d ( fφ ( xi ) , hyi ) ) ∑ c∈Ce exp ( −d ( fφ ( xi ) , hc ) ) , ( 2 ) where d ( · , · ) denotes a distance function ( e.g. , the l2 distance ) .

Summary: This paper addresses the problem of few-shot classification by incorporating self-supervised learning into the standard episode-based meta learning. Specifically, it adopts the pretext task of rotation prediction into the episode design. For each sampled episode, additional episodes are constructed by using rotated examples in the original support set and query set. Two self-supervised losses are designed based on the augmented episode sets – (1) recognizing different rotation transformations as an instance-level pretext task; (2) ensuring consistent predictions of class labels across different episodes as an episode-level pretext task. Two few-shot losses are also designed – (1) predicting class labels for each individual episode; (2) predicting class labels for the fused episode set based on attention. Experimental evaluation is conducted on two standard few-shot classification benchmarks, namely miniImageNet and tieredImageNet, and shows improved performance.