Self-Supervised Prime-Dual Networks for Few-Shot Image Classification

1 INTRODUCTION . Few-shot image classification aims to classify images from novel categories ( query samples ) based on very few labeled samples from each class ( support images ) ( Hong et al. , 2020a ; Sun et al. , 2021 ) . During the training stage , the few-shot learning ( FSL ) is given a set of support-query set pairs with class labels . Once successfully trained , the model needs to be tested on unseen classes . The major challenge here is that the number of available support samples N is very small , often N ≤ 5 . In an extreme case , N = 1 where it is called one-shot learning . In order to achieve this so-called learnto-learn capability , the FSL needs to capture the inherent visual or semantic relationship between the support samples and query samples , and more importantly , this learned relationship or prediction should be able to generalize well onto unseen classes ( Liu et al. , 2020d ) . A fundamental challenge in prediction is that : if we know entity A and are trying to predict entity B , how do we know if the prediction of B , denoted by Φ ( B ) , is accurate or not ? Is there any way that we can verify the accuracy of the prediction Φ ( B ) ? As we know , this is impossible since B has no ground-truth for us to evaluate or verify its prediction accuracy . If we can come up an indirect approach to effectively evaluate the prediction accuracy , it is expected that the learning and prediction performance can be significantly improved . In this work , we propose to explore a prime-dual commutative network design for effective prediction , specifically for few-shot image classification . As illustrated in Figure 1 , the prime network Φ is the original network that learns the forward prediction from A to B̂ = Φ ( A ) . The dual network Γ performs the reverse prediction from B to Â = Γ ( B ) . If we cascade these two networks together which establishes a prediction loop from A to B and then back to A , we have Â = Γ ( B̂ ) = Γ ( Φ ( A ) ) . ( 1 ) Since A is given , which has the ground-truth value , the difference between A and its prime-dual loop prediction result Â forms a self-supervision loss LS = d ( A , Â ) = d ( A , Γ ( Φ ( A ) ) ) , ( 2 ) where d is a distance metric function . This self-supervision loss LS can be used to improve the training performance based on the coupling between the prime and dual networks . Furthermore , it can used to verify and adjust the prediction result by minimizing the self-supervision loss . In this work , we propose to study this prime-dual network design with self-supervision for few-shot learning by exploiting the commutative relationship between the support set ( entityA ) and the query set ( entity B ) . Specifically , the prime network learns to predict the labels of query samples using the support set with ground-truth labels as training samples . Meanwhile , the dual network learns to predict the labels of the support samples using the query set with ground-truth labels as training samples . For example , in 5-way 1-shot learning , the support set consists of 5 images from 5 classes with only one image per class . The query set also has 5 images from 5 classes . When training the prime and dual networks , the support set and the query set are switched for training samples . This forward and reserve prediction process with commutative support and query sets forms a label prediction loop and establishes a self-supervision constraint between the ground-truth labels and their predicted values . The prime-dual networks are jointly trained with the help from the self-supervision loss . This loss is also used during the testing stage to adjust and optimize the prediction results . Our extensive experimental results demonstrate that the proposed self-supervised commutative learning and optimization method outperforms existing state-of-the-art few-shot learning methods by a large margin on various benchmark datasets . 2 RELATED WORK AND UNIQUE CONTRIBUTIONS . Few-shot learning ( FSL ) aims to recognize instances from unseen categories with few labeled samples . There are three major categories of methods that have been developed for FSL . ( 1 ) Data Augmentation is the most direct method for few-shot learning , which explores different approaches to synthesize images to address the issue of few training samples . For example , self-training jigsaw augmentation ( Chen et al. , 2019 ) is able to synthesize new images by segmenting and reorganizing labeled and unlabeled gallery images . Mangla et al . ( 2020 ) apply self-supervision algorithms augmented with manifold mixup ( Verma et al. , 2019 ) for few-shot classification tasks . The F2GAN ( Hong et al. , 2020b ) and MatchingGAN methods ( Hong et al. , 2020a ) use generative adversarial networks ( GANs ) to construct high-quality samples for new image categories . ( 2 ) Optimization-based methods aim to learn a good initial network model for the classifier . This learned model can be then quickly adapted to novel classes using a few labeled samples . MAML ( Finn et al. , 2017 ) proposes to train a set of initialization models based on second-order gradients and meta-optimization . TAML ( Jamal & Qi , 2019 ) reduces the bias introduced by the MAML algorithm to enforce equity between the tasks . In the Latent Embedding Optimization ( LEO ) method ( Rusu et al. , 2018 ) , gradient-based optimization is performed in a low-dimensional latent space instead of the original high-dimensional parameter space . ( 3 ) Metric-based methods aim to learn a good metric space so that samples from novel categories can be effectively distinguished and correctly classified . For example , MatchingNet ( Vinyals et al. , 2016 ) applies a recurrent network to calculate the cosine similarity between samples . ProtoNet ( Snell et al. , 2017 ) compares features between samples in the Euclidean space . RelationNet ( Sung et al. , 2018 ) uses a CNN model and ( Garcia & Bruna , 2017 ) uses the graph convolution network ( GNN ) to learn the metric relationship . In this work , we also consider cross-domain FSL . For the cross-domain classification task , the model needs to generalize well from the source domain to a new or unseen target domain without accessing samples from the unseen domain during the training stage . Sun et al . ( 2021 ) propose a modelagnostic explanation-guided training method that dynamically finds and emphasizes the features which are important for the predictions . This improves the model generalization capability . To characterize the variation of image feature distribution across different domains , the LFT method ( Tseng et al. , 2020 ) learns the noise distribution by adding feature-wise transformation layers to the image encoder . To avoid over-fitting on the source domain and increase the generalization capability to the target domain , the batch spectral regularization ( BSR ) method ( Liu et al. , 2020b ) attempts to suppress all singular values of the batch feature matrices during pre-training . Another set of methods ( Shankar et al. , 2018 ; Volpi et al. , 2018 ) learn to augment the input data with adversarial learning ( Yang et al. , 2020b ) in order to generalize the task from the source domain to the unseen target domain . In this work , we propose a commutative prime-dual network design for few-shot learning . In the literature , the mutual dependency and reciprocal relationship between multiple modules have been explored to achieve better performance . For example , ( Xu et al. , 2020 ) has developed a reciprocal cross-task architecture for image segmentation , which improves the learning efficiency and generation accuracy by exploiting the commonalities and differences across tasks . Sun et al . ( 2020 ) design a reciprocal learning network for human trajectory prediction , which consists of forward and backward prediction neural networks . The reciprocal learning enforces consistency between the forward and backward trajectory prediction , which helps each other to improve the learning performance and achieve higher accuracy . Zhu et al . ( 2017 ) design the CycleGAN contains two GANs forming a cycle network that can translate the images of the two domains into each other to achieve style transfer . Liu et al . ( 2021 ) develop a Temporal Reciprocal Learning ( TRL ) approach to fully explore the discriminative information from the disentangled features . Zhang et al . ( 2021b ) design a support-query mutual guidance architecture for few-shot object detection . Unique Contributions . Compared to existing work in the literature , the major contributions of this work include : ( 1 ) We propose a new prime-dual network design to explore the commutative relationship between support and query sets and establish a unique self-supervision constraint for few-shot learning . ( 2 ) We incorporate the self-supervision loss into the coupled prime-dual network training to improve the few-shot learning performance . ( 3 ) During the test stage , using the dual network to map the prediction results back to the support set domain and using the self-supervision constraint as an objective function , we develop an optimization-based scheme to verify and optimize the performance few-shot learning . ( 4 ) Our proposed method has significantly advanced the stateof-the-art performance of few-shot image classification . 3 METHOD . In this section , we present our method of self-supervised prime-dual network ( SPDN ) learning and optimization for few-shot image classification . 3.1 SELF-SUPERVISED COMMUTATIVE LEARNING . Figure 2 provides an overview of our proposed method of self-supervised commutative learning and optimization for few-shot image classification . In a typical setting of K-way N -shot learning , N labeled image samples from each of the K classes form the support set . For example , in a 5-way 1-shot learning , K = 5 and N = 1 . Given a very small support set S = { Skn|1 ≤ k ≤ K , 1 ≤ n ≤ N } , the objective of the FSL is to predict the labels of the query images Q = { Qkm|1 ≤ k ≤ K , 1 ≤ m ≤ M } from the same K classes in M batches During the training stage , the labels of both support and query samples are available . The prime network ΦS→Q for few-shot classification is trained on these support-query sets , aiming to learn and represent the inherent visual or semantic relationship between the support and query images . Once successfully learned , we will apply this network to unseen classes . Specifically , in the test stage , given a labeled support set S′ = { S′kn|1 ≤ k ≤ K , 1 ≤ n ≤ N } from these K unseen classes , we need to predict the labels for the query set Q′ = { Q′km|1 ≤ k ≤ K , 1 ≤ m ≤M } also from these unseen classes . Therefore , the fundamental challenge of FSL is to characterize and learn the inherent relationship between the support set S and the query set Q . Once learned , we can then shift or transfer this relationship to S′ and Q′ of unseen classes to infer the labels of Q′ . In this work , as discussed in the following section , we propose to establish a graph neural network ( GNN ) to characterize and learn this relationship . We recognize that , within the framework of few-shot learning , the support set and the query set are in an equal and symmetric position to each other . More specifically , if we can learn to predict the labels of query set Q from support set S , certainly , we can switch their order , predicting the labels of the support set S from the query set Q using the same network architecture . This observation leads to an interesting commutative prime-dual network design for few-shot learning . As illustrated in Figure 2 , we introduce a dual network ΓQ→S , which performs the reverse label prediction of the support set S from the query set Q . Let L ( S ) and L ( Q ) be the label vectors of S and Q , respectively . Let L̂ ( S ) and L̂ ( Q ) be the predicted labels . The forward prediction by the prime network can be written as L̂ ( Q ) = ΦS→Q [ L ( S ) ] , ( 3 ) while the reverse prediction by the dual network can be written as L̂ ( S ) = ΓQ→S [ L ( Q ) ] , ( 4 ) If both networks Φ and Γ are well trained , and if we pass the label prediction output of the prime network as input to the dual network , then , we expect that the predicted labels for the support set should be close to its ground-truth . This leads to the following self-supervision loss LSS = ||L ( S ) − L̂ ( S ) ||2 = ||L ( S ) − ΓQ→S [ L̂ ( Q ) ] ||2 ( 5 ) = ||L ( S ) − ΓQ→S [ ΦS→Q [ L ( S ) ] ] ||2 . This self-supervision constraint can be established on both support set and query set , resulting in a coupled prime-dual network training . Figure 3 ( a ) and ( b ) shows the training processes for the prime network and the dual network , respectively . Specifically , from the support set S , the prime network learns to predict the labels of the query set Q . As in existing few-shot learning , we have the loss LPQ = ||L̂ ( Q ) −L ( Q ) ||2 between the predicted query labels and their ground-truth values . We then use the query samples and their predicted labels as input to the dual network ΓQ→S , we can predict the labels of the support set L̂ ( S ) and compute the self-supervision loss LPS = ||L̂ ( S ) − L ( S ) ||2 . These two losses are combined to form the loss function for training the prime network LP = ||L̂Φ ( Q ) − L ( Q ) ||2 + α · ||L̂Φ , Γ ( S ) − L ( S ) ||2 . ( 6 ) α is a weighting parameter whose default value is set to be 0.5 in our experiments . Similarly , for the training of the dual network , as shown in Figure 3 ( b ) , its loss function is given by LD = ||L̂Γ ( S ) − L ( S ) ||2 + α · ||L̂Γ , Φ ( Q ) − L ( Q ) ||2 . ( 7 )

Summary: This paper proposes a prime-dual network to extract discriminative relationships between support and query examples of the multi classes with few examples. While the prime network predicts labels of a query out of the support set, the dual network reverses the process and infer the category labels of the support set using query examples. For training, a weight self-supervised loss contains the sum of dual and prime networks and is evaluated on inner-domain and cross-domain cases.