IEPT: Instance-Level and Episode-Level Pretext Tasks for Few-Shot Learning

1 INTRODUCTION . Deep convolutional neural networks ( CNNs ) ( Krizhevsky et al. , 2012 ; He et al. , 2016b ; Huang et al. , 2017 ) have seen tremendous successes in a wide range of application fields , especially in visual recognition . However , the powerful learning ability of CNNs depends on a large amount of manually labeled training data . In practice , for many visual recognition tasks , sufficient manual annotation is either too costly to collect or not feasible ( e.g. , for rare object classes ) . This has severely limited the usefulness of CNNs for real-world application scenarios . Attempts have been made recently to mitigate such a limitation from two distinct perspectives , resulting in two popular research lines , both of which aim to transfer knowledge learned from the data of a set of source tasks to a new target one : few-shot learning ( FSL ) and self-supervised learning ( SSL ) . FSL ( Fei-Fei et al. , 2006 ; Vinyals et al. , 2016 ; Finn et al. , 2017 ; Snell et al. , 2017 ; Sung et al. , 2018 ) typically takes a ‘ learning to learn ’ or meta-learning paradigm . That is , it aims to learn an algorithm for learning from few labeled samples , which generalizes well across any tasks . To that end , it adopts ∗Corresponding author . an episodic training strategy – the source tasks are arranged into learning episodes , each of which contains n classes and k labeled samples per class to simulate the setting for the target task . Part of the CNN model ( e.g. , feature extraction subnet , classification layers , or parameter initialization ) is then meta-learned for rapid adaptation to new tasks . In contrast , SSL ( Doersch et al. , 2015 ; Noroozi & Favaro , 2016 ; Iizuka et al. , 2016 ; Doersch & Zisserman , 2017 ; Noroozi et al. , 2018 ) does not require the source data to be annotated . Instead , it exploits an annotation-free pretext task on the source task data in the hope that a task-generalizable feature representation can be learned from the source tasks for easy adoption or adaptation in a target task . Such a pretext task gets its self-supervised signal at the per-instance level . Examples include rotation and context prediction ( Gidaris et al. , 2018 ; Doersch et al. , 2015 ) , jigsaw solving ( Noroozi & Favaro , 2016 ) , and colorization ( Iizuka et al. , 2016 ; Larsson et al. , 2016 ) . Since these pretext tasks are class-agnostic , solving them leads to the learning of transferable knowledge . Since both FSL and SSL aim to reduce the need of collecting a large amount of labeled training data for a target task by transferring knowledge from a set of source tasks , it is natural to consider combining them in a single framework . Indeed , two recent works ( Gidaris et al. , 2019 ; Su et al. , 2020 ) proposed to integrate SSL into FSL by adding an auxiliary SSL pretext task in an FSL model . It showed that the SSL learning objective is complementary to that of FSL and combining them leads to improved FSL performance . However , in ( Gidaris et al. , 2019 ; Su et al. , 2020 ) , SSL is combined with FSL in a superficial way : it is only taken as a separate auxiliary task for each single training instance and has no effect on the episodic training pipeline of the FSL model . Importantly , by ignoring the class labels of samples , the instance-level SSL learning objective is weak on its own . Since meta-learning across episodes is the essence of most contemporary FSL models , we argue that adding instance-level SSL pretext tasks alone fails to exploit fully the complementarity of the aforementioned FSL and SSL , for which a closer and deeper integration is needed . To that end , in this paper we propose a novel Instance-level and Episode-level Pretext Task ( IEPT ) framework for few-shot recognition . Apart from adding an instance-level pretext SSL task as in ( Gidaris et al. , 2019 ; Su et al. , 2020 ) , we introduce two episode-level SSL-FSL hybrid learning objectives for seamless SSL-FSL integration . Concretely , as illustrated in Figure 1 , our full model has three additional learning objectives ( besides the standard FSL one ) : ( 1 ) Different rotation transformations are applied to each original few-shot episode to generate a set of extended episodes , where each image has a rotation label for the instance-level pretext task ( i.e. , to predict the rotation label ) . ( 2 ) The consistency across the predictions of an FSL classifier from different extended episodes is maximized as an episode-level pretext task . For each training image , the rotation transformation does not change its semantic content and hence its class label ; the FSL classifier predictions across different extended episodes thus should be consistent , hence the consistency regularization objective . ( 3 ) The correlation of features across instances from these extended episodes is modeled by a transformer-based attention module , optimizing the fusion of the features of each instance/image and its various rotation-transformed versions mainly for task adaptation during meta-testing . Importantly , with these three new learning objectives introduced in IEPT , any meta-learning based FSL model can now benefit more from SSL by fully exploiting their complementarity . Our main contributions are : ( 1 ) For the first time , we propose both instance-level and episode-level pretext tasks ( IEPT ) for integrating SSL into FSL . The episode-level pretext task enables episodic training of SSL and hence closer integration of SSL with FSL . ( 2 ) In addition to these pretext tasks , FSL further benefits from SSL by integrating features extracted from various rotation-transformed versions of the original training instances . The optimal way of feature integration is learned by a transformer-based attention module , which is mainly designed for task adaptation during meta-testing . ( 3 ) Extensive experiments show that the proposed model achieves the new state-of-the-art . 2 RELATED WORK . Few-Shot Learning . The recent FSL studies are dominated by meta-learning based methods . They can be divided into three groups : ( 1 ) Metric-based methods ( Vinyals et al. , 2016 ; Snell et al. , 2017 ; Sung et al. , 2018 ; Allen et al. , 2019 ; Xing et al. , 2019 ; Li et al. , 2019a ; b ; Wu et al. , 2019 ; Ye et al. , 2020 ; Afrasiyabi et al. , 2020 ; Liu et al. , 2020 ; Zhang et al. , 2020 ) aim to learn the distance metric between feature embeddings . The focus of these methods is often on meta-learning of a feature-extraction CNN , whilst the classifiers used are of simple form such as a nearest-neighbor classifier . ( 2 ) Optimization-based methods ( Finn et al. , 2017 ; Ravi & Larochelle , 2017 ; Rusu et al. , 2019 ; Lee et al. , 2019 ) learn to optimize the model rapidly given a few labeled samples per class in the new task . ( 3 ) Model-based methods ( Santoro et al. , 2016 ; Munkhdalai & Yu , 2017 ; Mishra et al. , 2018 ) focus on designing either specific model structures or parameters capable of rapid updating . Apart from these three groups of methods , other FSL methods have attempted feature hallucination ( Schwartz et al. , 2018 ; Hariharan & Girshick , 2017 ; Gao et al. , 2018 ; Wang et al. , 2018 ; Zhang et al. , 2019 ; Tsutsui et al. , 2019 ) which generates additional samples from the given few shots for network finetuning , and parameter predicting ( Qiao et al. , 2018 ; Qi et al. , 2018 ; Gidaris & Komodakis , 2019 ; 2018 ) which learns to predict part of the parameters of a network given few samples of new classes for quick adaptation . In this work , we adopt the metric-based Prototypical Network ( ProtoNet ) ( Snell et al. , 2017 ) as the basic FSL classifier for the main instantiation of our IEPT framework due to its simplicity and popularity . However , we show that any meta-learning based FSL method can be combined with our IEPT ( see results in Figure 2 ( c ) ) . Self-Supervised Learning . In SSL , it is assumed that the source task data is label-free and a pretext task is designed to provide self-supervision signals at the instance-level . Existing SSL approaches differ mainly in the pretext task design . These include predicting the rotation angle ( Gidaris et al. , 2018 ) and the context of image patch ( Doersch et al. , 2015 ; Nathan Mundhenk et al. , 2018 ) , jigsaw solving ( Noroozi & Favaro , 2016 ; Noroozi et al. , 2018 ) ( i.e . shuffling and then reordering image patch ) , and performing images reversion ( Iizuka et al. , 2016 ; Pathak et al. , 2016 ; Larsson et al. , 2016 ) . SSL has been shown to be beneficial to various down-steam tasks such as semantic object matching ( Novotny et al. , 2018 ) , object segmentation ( Ji et al. , 2019 ) and object detection ( Doersch & Zisserman , 2017 ) by learning transferable feature presentations for these tasks . Integrating Self-Supervised Learning into Few-Shot Learning . To the best of our knowledge , only two recent works ( Gidaris et al. , 2019 ; Su et al. , 2020 ) have attempted combining SSL with FSL . However , the integration of SSL into FSL is often shallow : the original FSL training pipeline is intact ; in the meantime , an additional loss on each image w.r.t . a self-supervised signal like the rotation angle or relative patch location is introduced . With pretext tasks solely at the instance level , combining the two approaches ( i.e. , SSL and FSL ) can only be superficial without fully exploiting the episodic training pipeline unique to FSL . Different from ( Gidaris et al. , 2019 ; Su et al. , 2020 ) , we introduce an episode-level pretext task to integrate SSL into the episodic training in FSL fully . Specifically , the consistency across the predictions of an FSL classifier from different extended episodes is maximized to reflect the fact that various rotation transformations should not alter the class-label prediction . Moreover , features of each instance and its various rotation-transformed versions are now fused for FSL classification , to integrate SSL with FSL for the supervised classification task . Our experimental results show that thanks to the closer integration of SSL and FSL , our IEPT clearly outperforms ( Gidaris et al. , 2019 ; Su et al. , 2020 ) ( see Table 1 ) . 3 METHODOLOGY . 3.1 PRELIMINARY . Problem Setting . Given an n-way k-shot FSL task sampled from a test set Dt , to imitate the test setting , an FSL model is typically trained in an episodic way . That is , n-way k-shot episodes are randomly sampled from a training set Ds , where the class label space of Ds has no overlap with that of Dt . Each episode Ee contains a support set Se and a query set Qe . Concretely , we first randomly sample a set of n classes Ce from the training set , and then generate Se andQe by sampling k support samples and q query samples from each class in Ce , respectively . Formally , we have Se = { ( xi , yi ) |yi ∈ Ce , i = 1 , ... , n × k } and Qe = { ( xi , yi ) |yi ∈ Ce , i = 1 , ... , n × q } , where Se ⋂ Qe = ∅ . For simplicity , we denote lk = n × k and lq = n × q . In the meta-training stage , the training process has an inner and an outer loop in each episode : in the inner loop , the model is updated using Se ; its performance is then evaluated on the query set Qe in the outer loop to update the model parameters or algorithm that one wants to meta-learn . Basic FSL Classifier . We employ ProtoNet ( Snell et al. , 2017 ) as the basic FSL model . This model has a feature-extraction CNN and a simple non-parametric classifier . The parameter of the feature extractor is to be meta-learned . Concretely , in the inner loop of an episode , ProtoNet fixes the feature extractor and computes the mean feature embedding for each class as follows : hc = 1 k · ∑ ( xi , yi ) ∈Se fφ ( xi ) · I ( yi = c ) , ( 1 ) where class c ∈ Ce , fφ is a feature extractor with learnable parameters φ , and I is the indicator function . By computing the distance between the feature embedding of each query sample and that of the corresponding class , the loss function used to meta-learn φ in the outer loop is defined as : Lfsl ( Se , Qe ) = 1 |Qe| ∑ ( xi , yi ) ∈Qe − log exp ( −d ( fφ ( xi ) , hyi ) ) ∑ c∈Ce exp ( −d ( fφ ( xi ) , hc ) ) , ( 2 ) where d ( · , · ) denotes a distance function ( e.g. , the l2 distance ) .

Summary: This paper presents a method for combining self-supervised learning (SSL) (in the form of predicting the rotation applied to an image) with few-shot learning (FSL) in the domain of image classification. Compared to prior work, this paper introduces -- (i) a consistency loss which ensures FSL episodes with different rotations agree in their class predictions; and (ii) an integration method which derives the label of an image from a fused representation of all its rotations. These lead to an improvement over 3 FSL benchmarks.