Year: 2021

Abstract:
We construct a prime-dual network structure for few-shot learning which establishes a commutative relationship between the support set and the query set, as well as a new self-supervision constraint for highly effective few-shot learning. Specifically, the prime network performs the forward label prediction of the query set from the support set, while the dual network performs the reverse label prediction of the support set from the query set. This forward and reserve prediction process with commutated support and query sets forms a label prediction loop and establishes a self-supervision constraint between the ground-truth labels and their predicted values. This unique constraint can be used to significantly improve the training performance of few-shot learning through coupled prime and dual network training. It can be also used as an objective function for optimization during the testing stage to refine the query label prediction results. Our extensive experimental results demonstrate that the proposed self-supervised commutative learning and optimization outperforms existing state-of-the-art few-shot learning methods by large margins on various benchmark datasets.

heading: 1 INTRODUCTION
text: Few-shot image classification aims to classify images from novel categories (query samples) based on very few labeled samples from each class (support images) (Hong et al., 2020a; Sun et al., 2021). During the training stage, the few-shot learning (FSL) is given a set of support-query set pairs with class labels. Once successfully trained, the model needs to be tested on unseen classes. The major challenge here is that the number of available support samples N is very small, often N ≤ 5. In an extreme case, N = 1 where it is called one-shot learning. In order to achieve this so-called learnto-learn capability, the FSL needs to capture the inherent visual or semantic relationship between the support samples and query samples, and more importantly, this learned relationship or prediction should be able to generalize well onto unseen classes (Liu et al., 2020d).
A fundamental challenge in prediction is that: if we know entity A and are trying to predict entity B, how do we know if the prediction of B, denoted by Φ(B), is accurate or not? Is there any way that we can verify the accuracy of the prediction Φ(B)? As we know, this is impossible since B has no ground-truth for us to evaluate or verify its prediction accuracy. If we can come up an indirect approach to effectively evaluate the prediction accuracy, it is expected that the learning and prediction performance can be significantly improved.
In this work, we propose to explore a prime-dual commutative network design for effective prediction, specifically for few-shot image classification. As illustrated in Figure 1, the prime network Φ is the original network that learns the forward prediction from A to B̂ = Φ(A). The dual network Γ performs the reverse prediction from B to Â = Γ(B). If we cascade these two networks together which establishes a prediction loop from A to B and then back to A, we have
Â = Γ(B̂) = Γ(Φ(A)). (1)
Since A is given, which has the ground-truth value, the difference between A and its prime-dual loop prediction result Â forms a self-supervision loss
LS = d(A, Â) = d(A,Γ(Φ(A))), (2)
where d is a distance metric function. This self-supervision loss LS can be used to improve the training performance based on the coupling between the prime and dual networks. Furthermore, it can used to verify and adjust the prediction result by minimizing the self-supervision loss.
In this work, we propose to study this prime-dual network design with self-supervision for few-shot learning by exploiting the commutative relationship between the support set (entityA) and the query set (entity B). Specifically, the prime network learns to predict the labels of query samples using the support set with ground-truth labels as training samples. Meanwhile, the dual network learns to predict the labels of the support samples using the query set with ground-truth labels as training samples. For example, in 5-way 1-shot learning, the support set consists of 5 images from 5 classes with only one image per class. The query set also has 5 images from 5 classes. When training the prime and dual networks, the support set and the query set are switched for training samples. This forward and reserve prediction process with commutative support and query sets forms a label prediction loop and establishes a self-supervision constraint between the ground-truth labels and their predicted values. The prime-dual networks are jointly trained with the help from the self-supervision loss. This loss is also used during the testing stage to adjust and optimize the prediction results. Our extensive experimental results demonstrate that the proposed self-supervised commutative learning and optimization method outperforms existing state-of-the-art few-shot learning methods by a large margin on various benchmark datasets.

heading: 2 RELATED WORK AND UNIQUE CONTRIBUTIONS
text: Few-shot learning (FSL) aims to recognize instances from unseen categories with few labeled samples. There are three major categories of methods that have been developed for FSL. (1) Data Augmentation is the most direct method for few-shot learning, which explores different approaches to synthesize images to address the issue of few training samples. For example, self-training jigsaw augmentation (Chen et al., 2019) is able to synthesize new images by segmenting and reorganizing labeled and unlabeled gallery images. Mangla et al. (2020) apply self-supervision algorithms augmented with manifold mixup (Verma et al., 2019) for few-shot classification tasks. The F2GAN (Hong et al., 2020b) and MatchingGAN methods (Hong et al., 2020a) use generative adversarial networks (GANs) to construct high-quality samples for new image categories. (2) Optimization-based methods aim to learn a good initial network model for the classifier. This learned model can be then quickly adapted to novel classes using a few labeled samples. MAML (Finn et al., 2017) proposes to train a set of initialization models based on second-order gradients and meta-optimization. TAML (Jamal & Qi, 2019) reduces the bias introduced by the MAML algorithm to enforce equity between the tasks. In the Latent Embedding Optimization (LEO) method (Rusu et al., 2018), gradient-based optimization is performed in a low-dimensional latent space instead of the original high-dimensional parameter space. (3) Metric-based methods aim to learn a good metric space so that samples from novel categories can be effectively distinguished and correctly classified. For example, MatchingNet (Vinyals et al., 2016) applies a recurrent network to calculate the cosine similarity between samples. ProtoNet (Snell et al., 2017) compares features between samples in the Euclidean space. RelationNet (Sung et al., 2018) uses a CNN model and (Garcia & Bruna, 2017) uses the graph convolution network (GNN) to learn the metric relationship.
In this work, we also consider cross-domain FSL. For the cross-domain classification task, the model needs to generalize well from the source domain to a new or unseen target domain without accessing samples from the unseen domain during the training stage. Sun et al. (2021) propose a modelagnostic explanation-guided training method that dynamically finds and emphasizes the features which are important for the predictions. This improves the model generalization capability. To characterize the variation of image feature distribution across different domains, the LFT method (Tseng et al., 2020) learns the noise distribution by adding feature-wise transformation layers to the
image encoder. To avoid over-fitting on the source domain and increase the generalization capability to the target domain, the batch spectral regularization (BSR) method (Liu et al., 2020b) attempts to suppress all singular values of the batch feature matrices during pre-training. Another set of methods (Shankar et al., 2018; Volpi et al., 2018) learn to augment the input data with adversarial learning (Yang et al., 2020b) in order to generalize the task from the source domain to the unseen target domain.
In this work, we propose a commutative prime-dual network design for few-shot learning. In the literature, the mutual dependency and reciprocal relationship between multiple modules have been explored to achieve better performance. For example, (Xu et al., 2020) has developed a reciprocal cross-task architecture for image segmentation, which improves the learning efficiency and generation accuracy by exploiting the commonalities and differences across tasks. Sun et al. (2020) design a reciprocal learning network for human trajectory prediction, which consists of forward and backward prediction neural networks. The reciprocal learning enforces consistency between the forward and backward trajectory prediction, which helps each other to improve the learning performance and achieve higher accuracy. Zhu et al. (2017) design the CycleGAN contains two GANs forming a cycle network that can translate the images of the two domains into each other to achieve style transfer. Liu et al. (2021) develop a Temporal Reciprocal Learning (TRL) approach to fully explore the discriminative information from the disentangled features. Zhang et al. (2021b) design a support-query mutual guidance architecture for few-shot object detection.
Unique Contributions. Compared to existing work in the literature, the major contributions of this work include: (1) We propose a new prime-dual network design to explore the commutative relationship between support and query sets and establish a unique self-supervision constraint for few-shot learning. (2) We incorporate the self-supervision loss into the coupled prime-dual network training to improve the few-shot learning performance. (3) During the test stage, using the dual network to map the prediction results back to the support set domain and using the self-supervision constraint as an objective function, we develop an optimization-based scheme to verify and optimize the performance few-shot learning. (4) Our proposed method has significantly advanced the stateof-the-art performance of few-shot image classification.

heading: 3 METHOD
text: In this section, we present our method of self-supervised prime-dual network (SPDN) learning and optimization for few-shot image classification.

heading: 3.1 SELF-SUPERVISED COMMUTATIVE LEARNING
text: Figure 2 provides an overview of our proposed method of self-supervised commutative learning and optimization for few-shot image classification. In a typical setting of K-way N -shot learning, N labeled image samples from each of the K classes form the support set. For example, in a 5-way 1-shot learning, K = 5 and N = 1. Given a very small support set S = {Skn|1 ≤ k ≤ K, 1 ≤ n ≤ N}, the objective of the FSL is to predict the labels of the query images Q = {Qkm|1 ≤ k ≤ K, 1 ≤ m ≤ M} from the same K classes in M batches During the training stage, the labels of both support and query samples are available. The prime network ΦS→Q for few-shot classification is trained on these support-query sets, aiming to learn and represent the inherent visual
or semantic relationship between the support and query images. Once successfully learned, we will apply this network to unseen classes. Specifically, in the test stage, given a labeled support set S′ = {S′kn|1 ≤ k ≤ K, 1 ≤ n ≤ N} from these K unseen classes, we need to predict the labels for the query set Q′ = {Q′km|1 ≤ k ≤ K, 1 ≤ m ≤M} also from these unseen classes. Therefore, the fundamental challenge of FSL is to characterize and learn the inherent relationship between the support set S and the query set Q. Once learned, we can then shift or transfer this relationship to S′ and Q′ of unseen classes to infer the labels of Q′. In this work, as discussed in the following section, we propose to establish a graph neural network (GNN) to characterize and learn this relationship.
We recognize that, within the framework of few-shot learning, the support set and the query set are in an equal and symmetric position to each other. More specifically, if we can learn to predict the labels of query set Q from support set S, certainly, we can switch their order, predicting the labels of the support set S from the query set Q using the same network architecture. This observation leads to an interesting commutative prime-dual network design for few-shot learning. As illustrated in Figure 2, we introduce a dual network ΓQ→S, which performs the reverse label prediction of the support set S from the query set Q. Let L(S) and L(Q) be the label vectors of S and Q, respectively. Let L̂(S) and L̂(Q) be the predicted labels. The forward prediction by the prime network can be written as
L̂(Q) = ΦS→Q[L(S)], (3)
while the reverse prediction by the dual network can be written as
L̂(S) = ΓQ→S[L(Q)], (4)
If both networks Φ and Γ are well trained, and if we pass the label prediction output of the prime network as input to the dual network, then, we expect that the predicted labels for the support set should be close to its ground-truth. This leads to the following self-supervision loss
LSS = ||L(S)− L̂(S)||2 = ||L(S)− ΓQ→S[L̂(Q)] ||2 (5) = ||L(S)− ΓQ→S[ΦS→Q[L(S)]] ||2.
This self-supervision constraint can be established on both support set and query set, resulting in a coupled prime-dual network training. Figure 3 (a) and (b) shows the training processes for the prime network and the dual network, respectively. Specifically, from the support set S, the prime network learns to predict the labels of the query set Q. As in existing few-shot learning, we have the loss LPQ = ||L̂(Q)−L(Q)||2 between the predicted query labels and their ground-truth values. We then use the query samples and their predicted labels as input to the dual network ΓQ→S, we can predict the labels of the support set L̂(S) and compute the self-supervision loss LPS = ||L̂(S) − L(S)||2. These two losses are combined to form the loss function for training the prime network
LP = ||L̂Φ(Q)− L(Q)||2 + α · ||L̂Φ,Γ(S)− L(S)||2. (6)
α is a weighting parameter whose default value is set to be 0.5 in our experiments. Similarly, for the training of the dual network, as shown in Figure 3(b), its loss function is given by
LD = ||L̂Γ(S)− L(S)||2 + α · ||L̂Γ,Φ(Q)− L(Q)||2. (7)

heading: 3.2 GRAPH NEURAL NETWORK FOR FEW-SHOT IMAGE CLASSIFICATION
text: The proposed prime and dual networks share the same network design, which will be discussed in this section. The only difference between these two networks is that their support and query samples are switched. In the following, we use the prime network as an example to explain its design.
The central task of few-shot learning is to characterize the inherent relationship between the query and support samples, based on which we can infer the labels of the query samples from the support samples (Tseng et al., 2020; Liu et al., 2020b). In this work, we propose to use a graph neural network (GNN) to model and analyze this relationship. InK-wayN -shot learning, givenK classes, each with N support samples {Skn}, we need to learn the prime network to predict the labels for K query samples {Qk}. This implies, in each of the total M training batch, we have K × (N + 1) support samples and query samples. As illustrated in Figure 4(a), we use a backbone network, for example, Resnet-10 or Resnet-12, to extract feature for each of these support and query samples. We denote their features by S = {stkn} and Q = {qtk} where t represents the update iteration index in the GNN. Initially, t = 0. These support-query sample features form the nodes for the GNN, denoted by {xtj |1 ≤ j ≤ J}, J = K × (N + 1), for the simplicity of notations. The edge between two graph nodes represents the correlation ψ(xti,x t j) between nodes x t i and x t j . Note that our GNN has two groups of nodes: support sample nodes and query sample nodes. The support samples nodes have labels while the labels of the query samples need to be predicted by the prime network. If xti and xtj are both support nodes, we have
ψ(xti,x t j) =
{ 1 if L(xti) = L(x t j),
0 if L(xti) 6= L(xtj). (8)
Here, L(·) represents the label of the corresponding support sample. Since the labels for the query nodes are unknown, the correlation for edges linked to these query nodes need to be learned by the GNN. Initially, we set them to be random values between 0 and 1.
Each node of the GNN combines features from these neighboring nodes with the corresponding correlation as weights and updates its own feature by learning a multi-layer perceptron (MLP) network Go[·] as follows
xt+1j = Go
[ J∑
i=1
xtj · ψ(xti,xtj)
] . (9)
At each edge, another MLP network Ge[·, ·] is learned to predict the correlation between two graph nodes,
ψ(xti,x t j) = Ge[xti,xtj ], (10)
whose ground-truth values are obtained using the scheme discussed in the above. The feature generated by the prime GNN is then passed to a classification network to predict the query labels. Both the prime and dual GNNs are jointly trained with their final classification networks.

heading: 3.3 SELF-SUPERVISED OPTIMIZATION OF FEW-SHOT IMAGE CLASSIFICATION
text: Besides improving the training performance through mutual enforcement, the proposed selfsupervised prime-dual network design can be also used in the testing stage to optimize the label prediction of query samples. Specifically, we can use the dual network to refine and optimize the label prediction results obtained by the prime network. As illustrated in Figure 4(b), given a support
set S and a query set Q, the support set has class labels L(S). Let L̂(Q) be the prediction result, the output of the softmax layer of the classification network. In existing approaches of few-shot learning or other network prediction scenarios, we are not able to verify if the prediction is accurate or not since the ground-truth is not available for test samples. However, in this work, with the dual network ΓQ→S being successfully trained, we can use the prediction result L̂(Q) as input to the dual network to predict the class labels of the original support samples
L̂(S) = ΓQ→S[L̂(Q)]. (11)
Note that these support samples DO have ground-truth labels L(S). Define the label prediction error by
El(S) = ||L(S)− L̂(S)||2. (12) We assume that the correct query sample labels L∗(Q) is within the neighborhood of the prediction result L̂(Q). Let Ω be the set of candidate assignments of query labels which are within the neighborhood of L̂(Q). For example,
Ω = {L̃(Q) : ||L̃(Q)− L̂(Q)||2 ≤ ∆}, (13)
where ∆ is a given threshold for the label vector distance. We then search the candidate query labels L̃(Q) within the neighborhood set Ω to minimize the support label prediction error El(S) in (12). The optimized prediction of the query samples is given by
L∗(Q) = arg min L̃(Q)∈Ω || L(S)− ΓQ→S[L̃(Q)] ||2. (14)
From the experimental results, we will see that this unique self-supervised optimization of the query label prediction is able to significantly improve the few-shot image classification performance.

heading: 4 EXPERIMENTAL RESULTS
text: In this section, we provide experimental results on various benchmark datasets to demonstrate the performance of our proposed SPDN method for few-shot learning.

heading: 4.1 IMPLEMENTATION DETAILS
text: We use ResNet-10 as the backbone of our feature encoder. The input images are resized to 224×224 and the output feature vector size is 1 × 1 × 512. We choose the Adam optimizer with a learning rate of 0.01 and a batch size of 64 for training of 400 epochs. In the episodic meta-training stage, we use the graph neural network (GNN) discussed in the above section to generate the feature embedding for query samples. The prime network ΦS→Q and the dual network ΓQ→S are jointly trained. These two networks are both trained for 400 epochs with 100 episodes per epoch. In each episode, we randomly select K categories (K=5, 5-way) from the training set. Then, we randomly select N samples (N=1 or 5 for 1-shot or 5-shot) from each category to compose support set and query set, respectively. In the test stage, we use the average of 1000 trials as the final result for all the experiments. For each trial, we randomly select K categories from the test set. Similar to the training stage, N (1 or 5) samples are randomly selected as the support set and 15 samples as the query set from each category.

heading: 4.2 DATASETS
text: Five benchmark datasets are used for performance evaluation and comparison with other methods in the literature, Mini-ImageNet (Ravi & Larochelle, 2016), CUB (Wah et al., 2011), Cars (Krause et al., 2013), Places (Zhou et al., 2017) and Plantae (Van Horn et al., 2018). More details about dataset settings are presented in Appendix A.1.

heading: 4.3 RESULTS
text: To demonstrate the performance of our SPDN method, we conduct a series of experiments under different few-shot classification settings. In the literature, there are two major scenarios for testing the FSL methods: (1) intra-domain learning where the training classes and test classes are from the same object domain, for example, both from the Mini-ImageNet classess, and (2) cross-domain learning where the FSL is trained on one dataset (e.g., Mini-ImageNet) and the testing is performed on another dataset (e.g., CUB). Certainly, the cross-domain scenario is more challenging.

heading: 4.3.1 INTRA-DOMAIN FSL RESULTS.
text: First, we conduct intra-domain FSL experiments on the Mini-ImageNet. Table 1 summarizes the performance comparison with state-of-the-art FSL methods mainly developed in the past two years. We also list the backbone network used for extracting the features for the input images. We can see that, for the 5-way 1-shot image classification task, our method (with ResNet-10 backbone) outperform the current best method (with ResNet-12 backbone) from (Zhang et al., 2021a) by 5.42%. Another method which uses the same ResNet-10 backbone is the GNN+FT method (Tseng et al., 2020). Our method outperforms this method by 12.23%. For the 5-way 5-shot classification task, our method outperforms the current best by more than 5%, which is quite significant.
Second, we evaluate our method on intra-domain fine-grained image classification tasks on the CUB dataset. In this case, the FSL needs to learn subtle features to distinguish objects from close categories. Table 2 summarizes the performance results on 5-way 1-shot and 5-way 5-shot classification tasks. We can see that, for the one-shot classification task, our method outperforms the current best method, FRN (Wertheimer et al., 2021) by 6.72%. For the 5-shot classification task, our method improves the classification accuracy by 2.80%.

heading: 4.3.2 CROSS-DOMAIN FSL RESULTS.
text: The cross-domain few-shot learning is more challenging. Following existing methods, we train the model on the Mini-ImageNet object domain and test the trained model on other domains, including the CUB, Cars, Places and Plantae datasets. Table 3 summarizes the results for 5-way 1-shot classification (top) and 5-way 5-shot classification (bottom). We can see that our SPDN method has dramatically improved the classification accuracy on these cross-domain FSL tasks. For example, on the Cars dataset, our method outperforms the current best TPN+ATA (Wang & Deng, 2021) by 4.15%. On the Plantae dataset, the performance gain is 5.59%, which is quite significant. For the 5-way 5-shot classification task, the performance gains on these datasets are also very significant,
between 0.37-8.68%. This demonstrates that our SPDN method is able to learn the inherent visual relationship between the support and query samples and can generalize very well onto unseen classes in new object domains.

heading: 4.4 ABLATION STUDIES
text: In this section, we conduct ablation studies to further understand the proposed SPDN method and analyze the contributions of major algorithm components.
From algorithm design perspective, our SPDN method has two major components: self-supervised learning (SSL) of the prime and dual networks, and the self-supervised optimization (SSO) of the predicted query labels. We adopt the single GNN-based model (Tseng et al., 2020) as the baseline of our method and the SSL and SSO algorithm components are added onto this baseline method. To understand the performance of these two algorithm components, in the following experiment, we train the SPDN method using training samples from the Mini-ImageNet. We conduct intra-domain few-shot image classification on the Mini-ImageNet and cross-domain few-shot image classification on the CUB, Cars, Places, and Plantae datasets. Table 4 summarizes the results for 5-way 1-shot and 5-way 5-shot image classification. The second column shows the intra-domain few-shot image classification results on the Mini-ImageNet. The rest columns show the results for the cross-domain classification results. We can see that the self-supervised prime-dual network training is able to improve the classification accuracy by up to 1.8%. The performance gain achieved by the selfsupervised optimization of the predicted query labels is much more significant, ranging from 7-10%. This dramatic performance gain is a surprise to us. In the following, we will provide additional ablation studies to further understand this SSO algorithm module. Compared to the SSO module, the performance improvement by the first SSL module is relatively small. This is because the major new contribution of the SSL module is the self-supervised loss which aims to further improve the learning on the baseline GNN. However, it has successfully trained a dual network, which plays a very important role in the second SSO module. It is used to search and optimize the predicted labels of the query samples, resulting in major performance gain. We discuss the specific optimization results of our self-supervised optimization (SSO) module through an experiment in Appendix A.3.
In the following experiments, we attempt to further understand the behavior and performance of the SSO algorithm module. First, we conduct an experiment to understand the search and optimization process of SSO. Suppose L(Q) is the true label of the query samples. Let
L̃(Q) = L(Q) + λ ·∆L, (15)
be a label vector within the neighborhood of L(Q). Here, ∆L is a pre-generated random vector and λ is a disturbance coefficient to control the amount of variation. With the label vector L̃(Q) and the query samples, we can predict the labels of the support vector using the dual network. Then, we can compute the prediction error El(S) as in (12). Figure 5(a) shows the label prediction error El(S) as a function of λ. This experiment was performed on 5-way 1-shot image classification on the CUB dataset. We can see that the minimum error is achieved at λ = 0. This implies the groundtruth labels of the query samples have the minimum self-supervised label error El(S). This is a very important property of our SSO method. It suggests that, when the predicted query labels are not correct, and the ground-truth labels are within its neighborhood, we can use the SSO method to search for these ground-truth labels using the minimum self-supervised support label error criteria.
During our self-supervised optimization of the predicted query labels, we choose a small neighborhood Ω within the neighborhood of the predicted query labels L̂(Q) with a maximum distance ∆. This ∆controls the number of search positions in the label space. If we search more positions or candidate query labels, we can obtain smaller self-supervised label errors of the support samples El(S). Figure 5(b) plots the value of El(S) as a function of the number of search positions. This experiment was performed on 5-way 1-shot image classification on the CUB dataset. We can see that the error drops significantly with the number of searched positions. We recognize that, for each search position, we need to run the dual network once. This does introduce extra computational complexity. But, the amount of performance gain is very appealing. In our experiments, we limited the number of search positions to the 5, i.e., the nearest 5 label vectors (integer vectors) to the predicted query label.

heading: 5 CONCLUSION
text: In this work, we have successfully developed a novel prime-dual network structure for few-shot learning which explores the commutative relationship between the support set and the query set. The prime network performs the forward label prediction from the support set to the query set, while the dual network performs the reverse label prediction from the query set to the support set. This forward and reserve prediction process with commutative support and query sets forms a label prediction loop and establishes a self-supervision constraint between the ground-truth labels and their predicted values. We have established a self-supervised support error metric and used the learned dual network to optimize the predicted query labels during the testing stage. Our extensive experimental results on both intra-domain and cross-domain few-shot image classificaiton have demonstrated that the proposed self-supervised prime-dual network learning and optimization have significantly improved the performance of few-shot learning, especially for cross-domain few-shot learning tasks. We have also conducted detailed ablation studies to provide in-depth understanding of the significant performance gain achieved by the self-supervised optimization process. The self-supervised primedual network design is general and can be naturally incorporated into other prediction and learning methods.

heading: A APPENDIX
text: In this appendix, we provide more details of experimental settings and additional results to further understand the performance of our proposed method.
A.1 DATASET
In our experiments, the following 5 datasets are used for performance evaluations and comparisons.
(1) Mini-ImageNet has randomly selected 100 categories from the ImageNet (Deng et al., 2009) and each category has 600 samples of size 84 × 84. The 100 categories are divided into a training set with 64 categories, a validation set with 16 categories, and a testing set with 20 categories. (2) CUB is a fine-grained dataset with 200 bird species mainly living in North America (Wah et al., 2011). We randomly split the dataset into 100, 50, 50 classes for training, validation and testing, respectively. (3) Cars contains 16,185 images of 207 fine-grained car types, which consist of 10 BMW models and 197 other car types (Krause et al., 2013). We randomly selected 196 categories include 98 training, 49 validation and 49 testing for the experiment. (4) Places is a dataset of scene images (Zhou et al., 2017), containing 73,000 training images from 365 scene categories, which are divided into 183 categories for training, 91 for validation and 91 for testing. (5) Plantae is a sub-set of the iNat2017 dataset (Van Horn et al., 2018), which contains 200 types of plants and a total of 47,242 images. We split them into 100 classes for training, 50 for validation, and 50 for testing.
The Mini-ImageNet is the most popular benchmark for few-shot classification. It is usually used as a baseline dataset for model training. The CUB dataset is more frequently used for few-shot fine-grained classification tasks. The Cars, Places and Plantae datasets are used for model testing in cross-domain few-shot classification tasks.
A.2 THE VISUALIZATION OF FEATURE IN SELF-SUPERVISED LEARNING.
The proposed SPDN method incorporates the self-supervised constraint into the training process, aiming to improve the quality of learned features and the generalization capability of the few-shot learning. Figure. 6 shows the tSNE visualization of the learned features of 100 samples from the mini-ImageNet dataset for each class in a 5-way 5-shot setting. We can see that, with the selfsupervised learning, the features of each class are more concentrated into clusters.
A.3 SELF-SUPERVISED OPTIMIZATION (SSO) MODULES
The proposed self-supervised optimization (SSO) modules aim to correct the predicted query labels. In the following experiment, we are trying to understand how many incorrect label prediction of the query labels have been successfully corrected by the SSO module. Table 5 shows the results from the 5-way 1-shot on the CUB dataset. We keep track of 75 randomly selected query samples. If we predict the query labels only using the prime network without using the SSO (before SSO), the number of query samples with incorrect labels is 57, and the number of correct ones is 18, which
is very low. After we apply the SSO, the number of query samples with incorrect labels is reduced to 45, the number of correct ones increases to 30. We can examine this correction process in more detail. The SSO module has corrected the labels for 15 samples, as shown in the third row (Incorrect → Correct Label) of the table. However, it has also mis-corrected the labels for 3 samples, as shown in the last row (Correct → Incorrect Label) of the table. In our experiments, we have observed that the SSO module is able to correct the labels for much more query samples than those miscorrected one. This implies that the dual network and the self-supervision constraint are working very well for few-shot learning. This explains the significant performance achieved by the proposed self-supervised prime-dual network method.
A.4 EXTENSION TO N -SHOT IMAGE CLASSIFICATION
In the main paper, we have used the 5-way 1-shot image classification as an example to present our method of self-supervised prime-dual network (SPDN) and optimization for few-shot image classification. This method can be naturally extended to genericK-wayN -shot image classification. Figure 7 illustrates an example of extension to 5-way 5-shot. In this case, each class, in both training and test stages, has 5 support samples and one query sample. In the prime network, we use these 5 support samples to predict the label of the query sample. To ensure that the dual network shares the same network structure as the prime network, for the reverse prediction, we randomly select one sample (denoted by s0) from the support set and switch it with the query sample q0. During the training and inference of the dual network, this updated support set is used to predict the label of s0, which is then compared to its ground-truth label to compute the self-supervised loss. This loss is used for joint prime-dual network training, as well as the self-supervised optimization of the label prediction for the query sample q0.
A.5 FURTHER UNDERSTANDING OF THE SELF-SUPERVISED OPTIMIZATION OF QUERY LABEL PREDICTION
In our proposed SPDN method, the self-supervised optimization of the query label prediction plays an important role and improves the performance significantly. In this section, we provide more experimental results to demonstrate and further under the performance of this algorithm module. Figure 8 shows 6 examples of 5-way 1-shot image classification. Initially, the predicted label for these query samples are incorrect. Then, we perform self-supervised search of the query labels within the neighborhood of the predicted label. We use this predicted labels as input to the dual network to predict the labels of the support samples. The label prediction error of the support
samples is used as the optimization objective. In Figure 8, under each query sample, we show the decreasing of the optimization objective (support label error) with the number of searched candidate query labels. These results show that it is sufficient to search 5-8 candidate query label vectors.
It should be noted that the self-supervised optimization query label prediction can correct the incorrect label prediction, adjusting incorrect label prediction into correct ones. Certainly, it will make mistake or mis-correct the query label prediction, adjusting correct label predictions into incorrect ones. However, the probability of the mis-correction is much lower. For example, Table 6 shows percentages of correct adjustment and incorrect adjustment by the optimization module on the Cars dataset. Specifically, the percentage of correct adjustment from incorrect query labels into correct ones is 21.6%. In the meantime, the percentage of incorrect adjustment is 5.7%. This result in a performance improvement of 15.8% in the overall few-shot image classification, from 32.8% to 48.6%, which is quite significant.
A.6 FURTHER DISCUSSION OF THE PROPOSED METHOD
The key idea and motivation behind our dual network design is as follows: one central challenge in network prediction is that we have no ways to check if the prediction is accurate or not, since we do not have the ground truth. To address this issue, we develop the prime-dual network structure, where the successfully learned dual network is used as a verification module to verify if the prediction results are good enough or not. It maps the prediction results back to the current known data. We establish the self-supervised loss defined on the current known data, use it as the objective function to perform local search and refinement of the prediction results. This process is unique and contributes significantly to the overall performance. The prime network is the baseline GNN+FT network using support samples to predict query samples. The dual network is another GNN+FT network (in opposite direction) using query samples to predict support samples. These two networks form a prediction loop and a self-supervised loss is then derived. We implement this new idea on the the GNN+FT few-shot learning method to demonstrate its performance. The proposed idea is generic and can be applied to other methods, even in other prediction and learning problems, which will be studied in our future work. Our proposed idea is new. However, it does introduce additional complexity. According to our estimation, it will add about 40-60% extra complexity on top of the existing baseline since a majority of computation, such as feature extraction, does not need to recomputed during the search process. In our future work, we plan to develop schemes to reduce the complexity of the self-supervised optimization, for example by merging multiple search steps into one execution cycle.

Review summaries:
[1] This paper follows the line of metric-based few-shot learning methods with GNNs. The novelty of this paper is to design dual GNN graphs to capture the consistency of label prediction from the support set to the query set and its reversible task (from the query set to the support set).  [2] The proposed method achieves considerable performance gains compared with other state-of-the-art methods. 

The paper proposed a self-supervised learning framework and a dual network to improve the performance of few-shot learning. The dual network is based on GNN architectures and learn the relationship between the features from the support set and query set. Experiment shows improvement over the baselines on several few-shot learning benchmarks. 

The authors design a prime-dual architecture for few-shot learning to characterize the inherent relationship between the support set and the query set and introduce a self-supervision constraint for performance improvement. In particular, it proposes to correct query sample labels between the prime network and dual network and design an optimized prediction for the query samples with a selection mechanism. Extensive experiments are conducted on datasets with the ablation analysis, and the method achieves favorable results over existing algorithms. 

This paper proposes a prime-dual network to extract discriminative relationships between support and query examples of the multi classes with few examples. While the prime network predicts labels of a query out of the support set, the dual network reverses the process and infer the category labels of the support set using query examples. For training, a weight self-supervised loss contains the sum of dual and prime networks and is evaluated on inner-domain and cross-domain cases.