Self-Supervised Prime-Dual Networks for Few-Shot Image Classification

1 INTRODUCTION . Few-shot image classification aims to classify images from novel categories ( query samples ) based on very few labeled samples from each class ( support images ) ( Hong et al. , 2020a ; Sun et al. , 2021 ) . During the training stage , the few-shot learning ( FSL ) is given a set of support-query set pairs with class labels . Once successfully trained , the model needs to be tested on unseen classes . The major challenge here is that the number of available support samples N is very small , often N ≤ 5 . In an extreme case , N = 1 where it is called one-shot learning . In order to achieve this so-called learnto-learn capability , the FSL needs to capture the inherent visual or semantic relationship between the support samples and query samples , and more importantly , this learned relationship or prediction should be able to generalize well onto unseen classes ( Liu et al. , 2020d ) . A fundamental challenge in prediction is that : if we know entity A and are trying to predict entity B , how do we know if the prediction of B , denoted by Φ ( B ) , is accurate or not ? Is there any way that we can verify the accuracy of the prediction Φ ( B ) ? As we know , this is impossible since B has no ground-truth for us to evaluate or verify its prediction accuracy . If we can come up an indirect approach to effectively evaluate the prediction accuracy , it is expected that the learning and prediction performance can be significantly improved . In this work , we propose to explore a prime-dual commutative network design for effective prediction , specifically for few-shot image classification . As illustrated in Figure 1 , the prime network Φ is the original network that learns the forward prediction from A to B̂ = Φ ( A ) . The dual network Γ performs the reverse prediction from B to Â = Γ ( B ) . If we cascade these two networks together which establishes a prediction loop from A to B and then back to A , we have Â = Γ ( B̂ ) = Γ ( Φ ( A ) ) . ( 1 ) Since A is given , which has the ground-truth value , the difference between A and its prime-dual loop prediction result Â forms a self-supervision loss LS = d ( A , Â ) = d ( A , Γ ( Φ ( A ) ) ) , ( 2 ) where d is a distance metric function . This self-supervision loss LS can be used to improve the training performance based on the coupling between the prime and dual networks . Furthermore , it can used to verify and adjust the prediction result by minimizing the self-supervision loss . In this work , we propose to study this prime-dual network design with self-supervision for few-shot learning by exploiting the commutative relationship between the support set ( entityA ) and the query set ( entity B ) . Specifically , the prime network learns to predict the labels of query samples using the support set with ground-truth labels as training samples . Meanwhile , the dual network learns to predict the labels of the support samples using the query set with ground-truth labels as training samples . For example , in 5-way 1-shot learning , the support set consists of 5 images from 5 classes with only one image per class . The query set also has 5 images from 5 classes . When training the prime and dual networks , the support set and the query set are switched for training samples . This forward and reserve prediction process with commutative support and query sets forms a label prediction loop and establishes a self-supervision constraint between the ground-truth labels and their predicted values . The prime-dual networks are jointly trained with the help from the self-supervision loss . This loss is also used during the testing stage to adjust and optimize the prediction results . Our extensive experimental results demonstrate that the proposed self-supervised commutative learning and optimization method outperforms existing state-of-the-art few-shot learning methods by a large margin on various benchmark datasets . 2 RELATED WORK AND UNIQUE CONTRIBUTIONS . Few-shot learning ( FSL ) aims to recognize instances from unseen categories with few labeled samples . There are three major categories of methods that have been developed for FSL . ( 1 ) Data Augmentation is the most direct method for few-shot learning , which explores different approaches to synthesize images to address the issue of few training samples . For example , self-training jigsaw augmentation ( Chen et al. , 2019 ) is able to synthesize new images by segmenting and reorganizing labeled and unlabeled gallery images . Mangla et al . ( 2020 ) apply self-supervision algorithms augmented with manifold mixup ( Verma et al. , 2019 ) for few-shot classification tasks . The F2GAN ( Hong et al. , 2020b ) and MatchingGAN methods ( Hong et al. , 2020a ) use generative adversarial networks ( GANs ) to construct high-quality samples for new image categories . ( 2 ) Optimization-based methods aim to learn a good initial network model for the classifier . This learned model can be then quickly adapted to novel classes using a few labeled samples . MAML ( Finn et al. , 2017 ) proposes to train a set of initialization models based on second-order gradients and meta-optimization . TAML ( Jamal & Qi , 2019 ) reduces the bias introduced by the MAML algorithm to enforce equity between the tasks . In the Latent Embedding Optimization ( LEO ) method ( Rusu et al. , 2018 ) , gradient-based optimization is performed in a low-dimensional latent space instead of the original high-dimensional parameter space . ( 3 ) Metric-based methods aim to learn a good metric space so that samples from novel categories can be effectively distinguished and correctly classified . For example , MatchingNet ( Vinyals et al. , 2016 ) applies a recurrent network to calculate the cosine similarity between samples . ProtoNet ( Snell et al. , 2017 ) compares features between samples in the Euclidean space . RelationNet ( Sung et al. , 2018 ) uses a CNN model and ( Garcia & Bruna , 2017 ) uses the graph convolution network ( GNN ) to learn the metric relationship . In this work , we also consider cross-domain FSL . For the cross-domain classification task , the model needs to generalize well from the source domain to a new or unseen target domain without accessing samples from the unseen domain during the training stage . Sun et al . ( 2021 ) propose a modelagnostic explanation-guided training method that dynamically finds and emphasizes the features which are important for the predictions . This improves the model generalization capability . To characterize the variation of image feature distribution across different domains , the LFT method ( Tseng et al. , 2020 ) learns the noise distribution by adding feature-wise transformation layers to the image encoder . To avoid over-fitting on the source domain and increase the generalization capability to the target domain , the batch spectral regularization ( BSR ) method ( Liu et al. , 2020b ) attempts to suppress all singular values of the batch feature matrices during pre-training . Another set of methods ( Shankar et al. , 2018 ; Volpi et al. , 2018 ) learn to augment the input data with adversarial learning ( Yang et al. , 2020b ) in order to generalize the task from the source domain to the unseen target domain . In this work , we propose a commutative prime-dual network design for few-shot learning . In the literature , the mutual dependency and reciprocal relationship between multiple modules have been explored to achieve better performance . For example , ( Xu et al. , 2020 ) has developed a reciprocal cross-task architecture for image segmentation , which improves the learning efficiency and generation accuracy by exploiting the commonalities and differences across tasks . Sun et al . ( 2020 ) design a reciprocal learning network for human trajectory prediction , which consists of forward and backward prediction neural networks . The reciprocal learning enforces consistency between the forward and backward trajectory prediction , which helps each other to improve the learning performance and achieve higher accuracy . Zhu et al . ( 2017 ) design the CycleGAN contains two GANs forming a cycle network that can translate the images of the two domains into each other to achieve style transfer . Liu et al . ( 2021 ) develop a Temporal Reciprocal Learning ( TRL ) approach to fully explore the discriminative information from the disentangled features . Zhang et al . ( 2021b ) design a support-query mutual guidance architecture for few-shot object detection . Unique Contributions . Compared to existing work in the literature , the major contributions of this work include : ( 1 ) We propose a new prime-dual network design to explore the commutative relationship between support and query sets and establish a unique self-supervision constraint for few-shot learning . ( 2 ) We incorporate the self-supervision loss into the coupled prime-dual network training to improve the few-shot learning performance . ( 3 ) During the test stage , using the dual network to map the prediction results back to the support set domain and using the self-supervision constraint as an objective function , we develop an optimization-based scheme to verify and optimize the performance few-shot learning . ( 4 ) Our proposed method has significantly advanced the stateof-the-art performance of few-shot image classification . 3 METHOD . In this section , we present our method of self-supervised prime-dual network ( SPDN ) learning and optimization for few-shot image classification . 3.1 SELF-SUPERVISED COMMUTATIVE LEARNING . Figure 2 provides an overview of our proposed method of self-supervised commutative learning and optimization for few-shot image classification . In a typical setting of K-way N -shot learning , N labeled image samples from each of the K classes form the support set . For example , in a 5-way 1-shot learning , K = 5 and N = 1 . Given a very small support set S = { Skn|1 ≤ k ≤ K , 1 ≤ n ≤ N } , the objective of the FSL is to predict the labels of the query images Q = { Qkm|1 ≤ k ≤ K , 1 ≤ m ≤ M } from the same K classes in M batches During the training stage , the labels of both support and query samples are available . The prime network ΦS→Q for few-shot classification is trained on these support-query sets , aiming to learn and represent the inherent visual or semantic relationship between the support and query images . Once successfully learned , we will apply this network to unseen classes . Specifically , in the test stage , given a labeled support set S′ = { S′kn|1 ≤ k ≤ K , 1 ≤ n ≤ N } from these K unseen classes , we need to predict the labels for the query set Q′ = { Q′km|1 ≤ k ≤ K , 1 ≤ m ≤M } also from these unseen classes . Therefore , the fundamental challenge of FSL is to characterize and learn the inherent relationship between the support set S and the query set Q . Once learned , we can then shift or transfer this relationship to S′ and Q′ of unseen classes to infer the labels of Q′ . In this work , as discussed in the following section , we propose to establish a graph neural network ( GNN ) to characterize and learn this relationship . We recognize that , within the framework of few-shot learning , the support set and the query set are in an equal and symmetric position to each other . More specifically , if we can learn to predict the labels of query set Q from support set S , certainly , we can switch their order , predicting the labels of the support set S from the query set Q using the same network architecture . This observation leads to an interesting commutative prime-dual network design for few-shot learning . As illustrated in Figure 2 , we introduce a dual network ΓQ→S , which performs the reverse label prediction of the support set S from the query set Q . Let L ( S ) and L ( Q ) be the label vectors of S and Q , respectively . Let L̂ ( S ) and L̂ ( Q ) be the predicted labels . The forward prediction by the prime network can be written as L̂ ( Q ) = ΦS→Q [ L ( S ) ] , ( 3 ) while the reverse prediction by the dual network can be written as L̂ ( S ) = ΓQ→S [ L ( Q ) ] , ( 4 ) If both networks Φ and Γ are well trained , and if we pass the label prediction output of the prime network as input to the dual network , then , we expect that the predicted labels for the support set should be close to its ground-truth . This leads to the following self-supervision loss LSS = ||L ( S ) − L̂ ( S ) ||2 = ||L ( S ) − ΓQ→S [ L̂ ( Q ) ] ||2 ( 5 ) = ||L ( S ) − ΓQ→S [ ΦS→Q [ L ( S ) ] ] ||2 . This self-supervision constraint can be established on both support set and query set , resulting in a coupled prime-dual network training . Figure 3 ( a ) and ( b ) shows the training processes for the prime network and the dual network , respectively . Specifically , from the support set S , the prime network learns to predict the labels of the query set Q . As in existing few-shot learning , we have the loss LPQ = ||L̂ ( Q ) −L ( Q ) ||2 between the predicted query labels and their ground-truth values . We then use the query samples and their predicted labels as input to the dual network ΓQ→S , we can predict the labels of the support set L̂ ( S ) and compute the self-supervision loss LPS = ||L̂ ( S ) − L ( S ) ||2 . These two losses are combined to form the loss function for training the prime network LP = ||L̂Φ ( Q ) − L ( Q ) ||2 + α · ||L̂Φ , Γ ( S ) − L ( S ) ||2 . ( 6 ) α is a weighting parameter whose default value is set to be 0.5 in our experiments . Similarly , for the training of the dual network , as shown in Figure 3 ( b ) , its loss function is given by LD = ||L̂Γ ( S ) − L ( S ) ||2 + α · ||L̂Γ , Φ ( Q ) − L ( Q ) ||2 . ( 7 )

Summary: The authors design a prime-dual architecture for few-shot learning to characterize the inherent relationship between the support set and the query set and introduce a self-supervision constraint for performance improvement. In particular, it proposes to correct query sample labels between the prime network and dual network and design an optimized prediction for the query samples with a selection mechanism. Extensive experiments are conducted on datasets with the ablation analysis, and the method achieves favorable results over existing algorithms.