Self-Supervised Prime-Dual Networks for Few-Shot Image Classification

Abstract: We construct a prime-dual network structure for few-shot learning which establishes a commutative relationship between the support set and the query set, as well as a new self- supervision constraint for highly effective few-shot learning. Specifically, the prime network performs the forward label prediction of the query set from the support set, while the dual network performs the reverse label prediction of the support set from the query set. This forward and reserve prediction process with commutated support and query sets forms a label prediction loop and establishes a self-supervision constraint between the ground-truth labels and their predicted values. This unique constraint can be used to significantly improve the training performance of few-shot learning through coupled prime and dual network training. It can be also used as an objective function for optimization during the testing stage to refine the query label prediction results. Our extensive experimental results demonstrate that the proposed self-supervised commutative learning and optimization outperforms existing state-of the-art few-shot learning methods by large margins on various benchmark datasets.

Reviews and author responses:

 I appreciate the authors responding to my concerns. Overall, I'm somewhat convinced about the first two questions, where the additional explanations make sense to a certain level.

However, I'm still concerned about the third question regarding fair comparisons. I was not talking about integration with other methods, but rather whether they were fairly compared. That is, if other methods don't require optimization on unseen testing classes while the proposed method relies on it, it is NOT fair to directly compare them. Whether integrating with other methods or not, the comparisons should be made in equivalent settings, so that the effectiveness of the proposed components can be properly tested.

I keep my rating as is, but the point mentioned above still remains unaddressed.

 While appreciating the author's replies, I still have concerns about the motivation of the work and the evaluation of the idea.  Therefore, I am keeping my score as "marginally below the acceptance threshold" for the ICLR submission of the work.  

[1] This paper follows the line of metric-based few-shot learning methods with GNNs. The novelty of this paper is to design dual GNN graphs to capture the consistency of label prediction from the support set to the query set and its reversible task (from the query set to the support set).

[2] The proposed method achieves considerable performance gains compared with other state-of-the-art methods.
 Pros:

[1] This paper extends the usage of Graph Neural Networks in few-shot image classification tasks, which is to design dual graphs and regularize the consistencies of prediction from the support set to the query set and from the query set to the support set.

[2] The experiments are comprehensive. The author conducted experiments on intra-domain classification and inter-domain classification tasks.

======================

Cons:

[1] **About motivation.** The novelty of this paper is to design a dual network to make bi-directional predictions. However, the motivation is not clearly stated in the paper. In the testing stage, the authors only use the support samples to make label predictions in the query set, so why the consistency loss can improve the performance in the few-shot learning? I suggest the author visualize the feature distributions of samples in the support set and the query set to explain how the consistency loss influences the feature distributions.

[2] **About the model design.** Do prime networks and dual networks share the same parameters? Or individual training? When testing, how does the author use the dual networks?

[3] **About the relation between this work and EGNN[A], DPGN[B], and Mutual CRF-GNN[C].** These methods follow the line of the graph-based methods and also focus on the relation between the query samples and support samples. The author should carefully compare their method with these papers. (i) These papers also design losses to supervise the prediction of support samples after message passing and also use complex graph network structures to capture the relation in edges, distributions, and label predictions. Does this paper share similar ideas? I would like to remind that when message passing in GNN, the information also transforms from the query set to the suppert set. (ii) This paper should add semi-supervised learning settings in EGNN and DPGN to show their effectiveness. 

A. Edge-labeling Graph Neural Network for Few-shot Learning, CVPR19

B. Distribution propagation graph network for few-shot learning, CVPR20

C. Mutual crf-rnn for few-shot learning, CVPR21

[4] It seems that this work is more effective for cross-domain few-shot classification than intra-domain few-shot classification. The author should explain why their methods are effective as I fail to find any specific design for cross-domain classification.

[5] As the author recognized, the dual network will be forwarded for each neighborhood search, resulting in much extra computation overhead. I think it is a bit unfair to compare their methods with other methods in Table 1. Although the paper achieves promising results, I have some concerns about motivation, model design, and experimental comparisons with other GNN-based methods, as listed in Main Review.

 Thanks for the authors' clarification. I am convinced by the second point. I agree to upgrade the score to 6 considering the significant performance gain, and good readability. I would like to suggest that the authors justify a bit more on the motivation in the final version.

 Thank you very much!

 Thanks for your comments!

Let it be clear that our idea is totally different from the GNN process, as explained in the following.

1) In the GNN process used by the GNN+FT method (ICLR2020), which is our baseline, the query features and the support features are fused together based on their correlation coefficients. The fused the features are then concatenated to the old features of support and query features for another round GNN process. So, this is a feature fusion process, not a cyclic regularization. 

2) The main difference of our paper is: we use the baseline GNN+FT as the prime network, we then learn a dual network which is able to verify if the prediction results of the prime network are accurate or not. Based on the self-supervised loss, it can also optimize these prediction results using local search. This has significantly improved the overall performance. 

Clearly, this idea and process are different from fusion process inside the GNN+FT between support and query features within each episode of few-shot learning. 

Hope this clarifies the details. 

Thank you very much!



 We have tried the single model approach. In this case,  the process becomes data augmentation. There was no performnace improvement on the baseline. The reason is that a network cannot correct prediction errors made by itself. 

In our method, the prime network is the baseline method, GNN+FT. We constructed a dual network. This network aims to verify if the prediction result of the prime network is accurate or not. Using the self-supervision loss, the dual network can  optimize the prediction result based on local search. This step has significantly improved the overall performance. 

That is why we need two networks. A single network does not work. 

As for your comment on the GNN+FT, we have double checked the code. It is using 224*224 input image size. We followed the same experimental configuration as the baseline GNN+FT method.

Hope this clarifies the details.
Thank you very much!

 Thanks the authors for the response. I keep my score conditioned on the promises of will revise/refine/provide in the final paper.

 Thank you for the clarification on the symmetric property of the dual and prime networks.  

Unfortunately, I still have some difficulties understanding the motivation of having two networks. Without the extra specified constraints, I think the overall process can be easily completed by a single model. For example, suppose we have a single network that takes two different views of input, then we would have the same prime-dual property. 

Besides, if I have not missed some details, note that GNN+FT (ICLR 2020) uses 84x84 input size for convolution backbone (see  https://github.com/hytseng0509/CrossDomainFewShot/blob/57cc8410a1adcf61d75dfe984f0fdfeea4fb083d/train.py#L82).


 Thanks for the authors' reply, which addressed my first concern.

I still consider the author did not answer the question [2]. My focus is neither the performance gain, nor its message passing process, but its regularization after message passing. As I said, in these GNN based methods, the query features will also be passed to the support features, and the objective also regularizes on support samples that are influenced by query samples as I said. In this sense, I consider it is also a cyclic regularization. Apart from showing the performance gain, I would like to know the real difference. If you want to answer my question directly, I think you should tell me whether these papers and your method share the same idea in this aspect, and the strength of your method compared to theirs if you consider there are some similarities. If you consider there are no similarities, please tell me my misunderstanding above, instead of just saying "Our prime-dual network and the reciprocal constraint has nothing to do with the GNN message passing." and repeating your designs and performance gain again.

 As your suggested, we have added performance comparison with two extra papers that you mentioned: EGNN and DPGN methods. As shown in Table 1, our method is able to outperform the DPGN (CVPR20) by 10.78% on 1-shot learning, and outperform the EGNN (CVPR19) by 14.73% on 5-shot learning. This is very significant and demonstrates the efffectiveness of our new method.

 Dear Reviewer, we sincerely thank you for your prompt reply to our review responses! We really appreciate this opportunity for discussion with you to further explain our idea and method to you.

As you suggested, we just included a tSNR feature visualization in the Supplemental Materials (please see Section A.2). It clearly shows that the proposed method is able to better aggregate the learned features for each class than the baseline method without using the proposed self-supervised prime-dual network method. 

As explained in our responses, the idea in our method is to develop a dual network which is able to check if the prediction is accurate or not. It addresses an important issue is network prediction where the inference is forward only, with no mechanism to check if the prediction is accurate or not since the ground truth is not available during test.

Now, using the dual network, we can map the prediction result for the test data (query samples) back to the current available data (support samples). It forms a self-supervision loss. This loss function can be used as an objective function to check if the prediction results for the query samples are accurate or not. In additional, we can use it as an objective function to adjust and optimize the prediction result. This step is called self-supervised optimization in the paper. 

We implement this new idea on the the GNN+FT few-shot learning method published in ICLR 2020. Our prime-dual network and the reciprocal constraint has nothing to do with the GNN message passing. As explained in the above, we constructed another GNN as the dual network to predict the labels of support samples from query samples, where the support and query samples are switched. We did not change anything in the GNN+FT. So, the new and unique contribution is the prime-dual network design and self-supervised contraint.

Specifically, the prime network is the baseline GNN+FT network using support samples to predict query samples. The dual network is another GNN+FT network (in opposite direction) using query samples to predict support samples. These two networks form a prediction loop and a self-supervised loss is then derived.

We have demonstrated that this new method is able to improve the accuracy up to 11%, which is quite significant. Our ablation study also shows that this unique self-supervision constraint/loss works in both training (self-supervised learning) and testing (self-supervised optimization). 

The proposed idea is generic and can be applied to other methods, even in other prediction and learning problems, which will be studied in our future work. The impressive results in this paper can already justify the contribution of this method.

We hope this can better explain our idea to you and answer your concerns! If you need further information, we are more than happy to provide in our continuing discussion. 





 We really appreciate your positive review of our paper and your valuable comments! We will carefully follow your comments to revise our paper. Here are our detailed responses to your comments:

$\textbf{ 1. For the comment of cycle consistency loss:}$  Thanks for pointing out this! We shall include this cycle consistency paper in our related work. They do have similar expressions in formulating the loss function. However, our method is uniquely different from this cycle consistency method: (A) our method aims to address the problem of prediction, namely, how to characterize and optimize the prediction results. Specifically, our method establishes self-supervision on the predicted labels, while the cycle consistency applies constraints on the images. (B) Once successfully trained, our dual network is used in the self-supervised optimization stage to refine the prediction results, which was not possible in the cycle consistency method. 

$\textbf{2. For the comment on ablation study:}$ Thanks for this insightful comment! The SSO (self-supervised optimization during inference) does perform much better than the SSL (self-supervised learning during training). It is due to the difference between the characteristics of training set and test set. However, as the major motivation of this work, we aim to learn a dual network to characterize the forward prediction performance. This is a very important problem in learning and prediction: there is NO mechanism for us to check if the prediction is accurate or not since we do not have ground truth. However, with the learned dual network, we can use the dual network to establish a self-supervised criteria as an objective function to performance local refinement of the prediction results. 

$\textbf{3. For the comment on other methods:}$ Thanks for this very helpful comment! Yes, we do agree that the proposed self-supervised prime-dual network is quite general. It can be potentially applied to other methods in the literature to further improve their performance on few-shot learning. 

Thanks again for your review and valuable comments!



 Thanks for the authors' rebuttal. However, I retain my score to borderline reject now mainly because of its partially satisfactory rebuttal and seemingly not clear motivations.

[1] I also share the same question with Gmcb (Q2, Q3) about motivation and why this method can work. I think the author still does not present clearly when the gain comes from without showing the performance comparison that they plan to do ["It can be potentially applied to other methods in the literature to further improve their performance on few-shot learning. In the final paper, we will include results to demonstrate this capability."]. In addition, we suggest the author do a tsne-visualization of features learned. Without these visualizations, I can not understand why a dual network influences the feature distribution intuitively, for example, does it minimize the feature distance between support set and support set for solving their semantic difference?

[2] I am still confused about the real difference between SSO and the original GNN based few-shot learning methods. As proposed in my comments[3], with message passing processes, the information will also pass from the queries samples to the support samples, and the objective functions also regularize the label prediction of the support samples. In this sense, It also a cyclic self-supervised regularization. My question is that whether your paper shares the same idea with them? Why does your paper outperform these methods, in your point of view? The authors seem not to answer these questions.

 We sincerely thank you for your review of our paper and for your valuable comments! We will carefully follow your comments to revise our paper. Here are our responses to your comments. We hope that our responses be able to address your concerns. 

$\textbf{1. For comments on strength.} $ Thanks for your comments: "The idea of the paper looks interesting", "The paper is well written and the classification improvements seem to be significant", as also agreed by previous three reviewers. 

$\textbf{2. For comments on two network structures.} $ Thanks for pointing out this! Yes, the forward and backward prediction are totally symmetric, with only the support and query sets being switched. So, the prime and dual networks share the same network structure. However, since the support and query samples are still different, so, the weights of these two networks are slightly different. Therefore, it is not efficient to copy them over, we tried this in our experiments. Actually, during training, the input to the dual network is the prediction results of the prime network. They are jointly trained based on the self-supervision constraints. In the final paper, we will include this discussion.

$\textbf{3. For comments on the performance gain by extra learning module.} $  Thanks for this valuable comment! The key idea and motivation behind our dual network design is as follows: one central challenge in network prediction is that we have no ways to check if the prediction is accurate or not, since we do not have the ground truth. To address this issue, we develop the prime-dual network structure, where the successfully learned dual network is used as a verification module to verify if the prediction results are good enough or not. It maps the prediction results back to the current known data. We establish the self-supervised loss defined on the current known data, use it as the objective function to perform local search and refinement of the prediction results. This process is unique and contributes significantly to the overall performance. Therefore, the dual network is not just an extra learning module since baseline network, such as ResNet, already has sufficient learning power. We will include this discussion in the final paper.

$\textbf{4. For comments on the image size of 224x224 v.s. 84x84.} $ Thanks for pointing this out! We notice that, in the literature, some papers are using the size of 224x224, for example, the GNN+FT and Cross-Transformer methods, and some are using the size 84x84. In this paper, we use the GNN+FT (ICLR 2020) as our baseline which uses the 224x224 size. We have demonstrated very significant performance improvement on top of this method. In the final paper, we will include ablation studies on different input image sizes. 

Thanks again for your kind review and valuable comments! We hope the above responses be able to address your concerns. If you need further information, we will be more than happy to provide and discuss with you. 



 We really appreciate your time and efforts in reviewing our paper and for your valuable comments! We will carefully follow your comments to revise and improve our paper. Here are our detailed responses to your comments. 

$\textbf{1. For comment about the motivation.}$ Sorry that the motivation is not clear to you. In this paper, we have developed a new idea called self-supervised prime-dual network for few-shot prediction/classification. One major challenge in prediction is that there is no mechanism for us to check if the prediction result is accurate or not since the ground-truth data is not available. To address this issue, we develop the prime-dual network. The prime network performs the traditional forward prediction, the dual network performs the backward prediction. This prediction loop forms a self-supervision constraint. They are jointly learned during the training stage. Once successfully learned, the dual network can be then used as a verification module during testing to optimize the prediction result using local search. Our experiments demonstrate that this new method has dramatically improved the few-shot classification performance. 

As you suggested, we will provide the tSNR figure to show the improvement of feature embeddings. We will further revise the paper to make the motivation more clear.

$\textbf{2. For comment on model design.}$ Yes, the prime and dual network share the same network structure, since the forward and backward predictions are totally symmetric, with only the support and query samples are being switched. However, they do have slightly different weights, since  their input images are slightly different since the support and query images are being switched. We will include this discussion in the final paper.

During  testing, we use the dual network as a verification module to optimize the forward prediction result with local search, which is described in the Section of Self-Supervised Optimization. 

$\textbf{3. For comment on difference from existing graph neural networks.}$ Thanks for pointing out this! As discussed in the above, the main contribution of this paper is developing a self-supervised prime-dual network for few-shot learning. We use the GNN+FT method as the baseline to demonstrate the significant performance improvement achieved by our new idea. After checking these new reference papers, we have found that our method also outperforms these methods. We will review these reference papers in the final paper and include them in the performance comparison. We also expect that our method can be implemented on top of these methods to improve their performance. 

$\textbf{4. For comment on cross-domain.}$ Thanks for this valuable comment! Yes, unlike traditional prediction method which learns the network from the training data and applies the model directly to the test samples using purely forward inference, without any feedback from the test samples, since there is not feedback mechanism for it to check if the prediction result is accurate or not. 

Our self-supervised prime-dual network method has successfully addressed this problem. The dual network is learned and used to verify the performance of the prediction and optimize the prediction result using local search with the self-supervision loss as the objective function. This optimization-based inference method considers the feedback from the test image, certainly having much better generalization power than existing methods. This is why our methods performs even better in cross-domain tasks.

We will include this discussion in the final paper.

$\textbf{5. For comment on complexity.}$ Thanks for pointing our this! In terms of accuracy and few-shot classification performance, the comparison is totally fair since we follow the same evaluation protocol as in other papers. Our proposed idea is new. However, it does introduce additional complexity. In the final paper, we will add the discussion on computational complexity. According to our estimation, it will add about 40-60% extra complexity on top of the existing baseline since a majority of computation, such as feature extraction, does not need to re-computed during the search process. In our future work, we plan to develop schemes to reduce the complexity of the self-supervised optimization, for example by merging multiple search steps into one execution cycle. 

Thanks again for all your valuable comments! We hope that our responses have addressed your concerns. If you need further information,  please let us know and we will be more than happy to provide and discuss with you.




 We sincerely thank you for the positive review of our paper and for your valuable comments! We will carefully follow your comment to revise our paper. Here are our detailed responses to your comments:

$\textbf{1. For the comment on strength}:$ Thank you very much for positive recommendation of our method!

$\textbf{2. For the comment on bipartite graph}:$ Sorry for the error in presentation. It should be a simple GNN, instead of a bipartite graph. We used the GNN method from Ref. ((Tseng et al., 2020)  as our baseline and demonstrated that our proposed idea is able to significantly improve the performance of few-shot learning and outperformed existing methods. 

$\textbf{3. For the comment on prime and dual network}:$ Thanks for the valuable comment! In our method, the prime and dual network did share the same network structure. This is a natural design since the forward and backward predictions are symmetric to each other. However, they do have switched training samples. Therefore, they have slightly different network weights. In our future studies, we shall explore how different architectures of the dual network affect the overall performance. We expect that this difference  be relatively small. 

$\textbf{4. For the comment on Figure 7}:$ Thanks for pointing out this! In Figure 7, we demonstrate that we can perform local search on very few candidates to reach to the target performance, typically between 5 and 10. Therefore, in our method, we simply search the closest 20 candidate positions, which is sufficient. 

$\textbf{5. For the comment on cross-domain experiments}:$ Thanks for pointing out this interesting point! In the Table, we showed the performance gains on top of the current best methods for 1-shot and 5-shot. For different experiments, the current best methods are different. The difference that your pointed out is caused by the performance variations of the current best methods. If we look at the performance improvement upon our baseline, it is quite consistent. We will include this discussion in the final paper!

Thanks again for your positive review and valuable comments!



 Thank you very much for your positive review of our paper and your insightful comments! We will carefully follow your comments to revise and improve our paper! Here are our detailed response to your comments.

1) $\textbf{For the notations in (6)(7)(11)-(14)}$: thanks for pointing this out! In the final paper, we will revise these notations for the labeling function to avoid confusion with the loss functions. 

2) $\textbf{For the comment on Figure 2}$: Sorry for the confusion of the figure. In the final paper, we will refine this figure to better represent support and query samples, with and without labels.

3) $\textbf{For the ablation study comment}$: In this work, we have developed the idea of self-supervised prime-dual network to learn the forward and backward predictions between support and query samples. This self-supervised contraints can improve the performance of training as illustrated in Figure 3. This step is called self-supervised learning. It can also improve the performance of inference during testing since we can use the dual network as an optimization function to perform local refinement of the predicted labels of query samples. This step is called self-supervised optimization. This is quite unique in our method design since tradtional network prediction can only perform forward prediction and there is not mechanism to check if the prediction is accurate or not. Our ablation studies show that both self-supervised learning and self-supervised optimization have significant contributions.

4) $\textbf{For the feature visualization comment}$: Thanks for this very helpful comment! In the final paper, we will provide tSNE visualization of learned features with and without the proposed self-supervision method.

Thanks again for your positive review of our paper and your valuable comments!






The paper proposed a self-supervised learning framework and a dual network to improve the performance of few-shot learning. The dual network is based on GNN architectures and learn the relationship between the features from the support set and query set. Experiment shows improvement over the baselines on several few-shot learning benchmarks.  Although predicting based a GNN structure is not novel in few-shot learning. The idea of self-supervision is interesting. 

The notations need improvement. For example, the prediction L in equation (6) - (7) are not consistent equation (11) - (14). And i would suggest use other notation because of the confusion of loss function. 

For figure 2, what is the difference between two types of samples with labels? what is the difference between two types of samples without labels? Although I can understand through the text but the figure can be improved a bit.

The ablation study, the authors divide it into two parts, i.e., self-supervised optimization and self-supervised learning. I do not understand why these two can be divided? Are there any details for how to do that?

To my understanding, the essence of the prediction from the GNN is still based on feature similarity. In this way, is there any feature representation visualization for the learned model please? I am interested in how the representation would change by using the self-supervised learning.
 The idea is interesting and seems to perform well. The writting needs improvement.

The paper proposes a prime-dual network for few shot image classification, where the support set and query set are used in a flip manner so that the labeled samples in support set can be compared with themselves to determine a loss for network update. STREHGTH
The proposed method is easy to follow and understand, and the experiment results are promising when directly compared with state-of-that-art methods.

WEAKNESS
1. This method reminds me of the cycle consistency loss in the well-known CycleGAN approach, but since CycleGAN is not referenced and discussed in the paper, I was wondering if I misunderstood that or is there a reason for that? Although it's a distantly related work in terms of applications, the formulation of problem and loss functions seem quite relevant. If that's the case, it would be desired to discuss the connection to prior work for better justification.

2. In ablation study, it appears that simply doing SSL is not helping much compared with baseline approach. Does it mean the discrepancy between training classes and unseen testing classes is still the dominant reason for generally low accuracy in few-shot learning task? If so, is it indicating that the SSL stage is not the major concern in such setting, and one should focus more on addressing the query and support sets in testing classes?

3. Following 2, if SSO is more essential for the performance gain, is it fair comparison with other SOTA results? In other words, did other methods go through an optimization stage on unseen testing classes, or do some of them in fact not rely on updates during testing? The overall approach makes sense to me, but I'm not entirely sure how does the proposed approach compare with existing work, in terms of methodology as well as experiment settings, so I'll need the questions above to be addressed before making a most informed decision.

The authors design a prime-dual architecture for few-shot learning to characterize the inherent relationship between the support set and the query set and introduce a self-supervision constraint for performance improvement. In particular, it proposes to correct query sample labels between the prime network and dual network and design an optimized prediction for the query samples with a selection mechanism. Extensive experiments are conducted on datasets with the ablation analysis, and the method achieves favorable results over existing algorithms.  Strengths: 1. The paper tackles one of the important issues of meta/few-shot learning: ground-truth is not available for test samples. The issue is important and very practical in my opinion. 2. The proposed method utilizes the trained dual network to predict the class labels of support samples, which is kind of the concept of the autoencoder technique, simple and promising. The idea of constructing and learning the relationship between support set and query set is reasonable and interesting. 3. In this paper, the authors provide comprehensive experiments, including the comparison of the state-of-the-art methods in public benchmarks, the ablation analysis, and how the parameters affect the performance such as the searched position number, etc. For example, Table 6 shows the number of the proposed SSO that can help mis-correct the query labels. The analysis of ablation and SSO modules provide evidence and make it convincing. 

Weaknesses: 1. In Section 3.2, the bipartite graph was mentioned, but I didn't see the authors use the property of the bipartite graph since there are edges constructed in the support set and also in the query set (Figure 4-a). This may not utilize the property from a bipartite graph. 2. As the paper mentioned, the prime and dual networks share the same network design, but I'd like to know whether the authors have used different network designs for prime and dual. What would be the pros and cons with the same/different designs? 3. In Figure 7, the 6 examples are shown for the 5-way 1-shot image classification experiment, I'd like to know why we show the position up to 20? Will the maximum be calculated by K(N + 1) as mentioned in the paper? 3. For the cross-domain result, the discussion may need to add more. For example, why the proposed method has less gain in Cars compared to 1-shot and 5-shot, and why does only CUB increase the performance gain when it's 5-way 5-shot?   Overall the paper is clear to understand, and it's a simple idea but effective approach. In this paper, most of the experiments provide the discussion and are convincing, except for a more detailed discussion in the cross-domain FSL result. I believe this paper will provide different insights into this field. 

This paper proposes a prime-dual network to extract discriminative relationships between support and query examples of the multi classes with few examples. While the prime network predicts labels of a query out of the support set, the dual network reverses the process and infer the category labels of the support set using query examples. For training, a weight self-supervised loss contains the sum of dual and prime networks and is evaluated on inner-domain and cross-domain cases.   Strengths: 

The paper is well written and the classification improvements seem to be significant. 


Weekness: 
- While the training stage is a stochastic process and dual network is the reverse of the prime of in mapping support to the query, I was wondering how both of the networks wouldn’t be the copy of each other at the end of the training? 

- What happens if we attach the used auxiliary learning module on top of ResNet and ablate for example prototypical network? How we can be sure that the accuracy gain of the model is not obtained by an extra learning module. 

- I was wondering why 224x224 images are used while miniImageNet dataset contains 84x84 images?
 The idea of the paper looks interesting, but I have some concerns about the motivation of the method and some of the evaluations of the work are missing.  

Meta-review: This work proposes a so-called self-supervised approach for few-shot learning. The self-supervision doesn't refer to the lack of use of any labels as in regular self-supervised embedding learning methods (here support sets are labelled), but refer to the fact that the query set's labels aren't used in their proposed objective. Instead the query labels are predicted by a primary network, which then uses these predicted labels to predict the ground truth labels on the support set. The support set label predictions can thus be used to derive a learning signal for the model. Some results are presented that suggest the method is competitive with respect to the state of the art.

Reviewers are quite split on this work. Even the reviewers who are technically leaning toward accepting this work (rating of 6) mention concerns that are worrying, e.g. reviewer Gmcb and wTnW both mention concerns related to the fairness of the evaluation. 

I too share similar concerns. First, the method in question is effectively a transductive method (as opposed to inductive, as are many of the baselines this work compares to), a distinction that the paper does not make explicit or address directly. This distinction is important, as it is well known that transductive methods have an advantage over inductive methods. I did try to look for some published transductive baselines. One is the method of Zhang et al. (2021a), which the authors do beat on mini-ImageNet, but not on CUB (and in fact, the paper only reports the results from Zhang et al. on mini-ImageNet, even though the original paper actually reports results on CUB, which I find odd). On CUB, for 1-shot, Zhang et al. (2021a) outperforms SPDN, while for 5-shot SPDN does only very slightly better. The paper is not clear as to whether the compared baselines are transductive or not in the cross-domain experiments either. 

Second, by introducing a dual network that is separate from the primal network, the proposed model effectively is increasing the capacity of their model, relative to using only a primal network. This capacity is mostly used when performing the self-supervised optimization of the query labels, which would explain why this aspect of the proposed method is what yields the largest improvements. Given that capacity has a large effect on the performance of methods on few-shot learning benchmarks, I'm quite concerned that this is the more likely explanation for the (sometimes surprisingly large) improved performance.

That said, I don't find the paper entirely without merit. The label optimization procedure is neat, and is probably the most interesting innovation of the paper. The use of a primal-dual architecture on the other hand is more incremental, e.g. relative to architectures used in semi-supervised few-shot learning.

Overall, at this point, given the lukewarm evaluation by reviewers and the lingering concerns (or at a minimum, lack of clarity) on the fairness of the evaluation, I'm afraid I'm not comfortable to recommend accepting this work as it currently stands.

Decision: Reject