Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: Pros:

1) The paper is well-motivated, clearly written and neatly organized. I like the writing style of this paper: explaining very clearly what the improvements are and how they perform with respect to the counterpart methods. 

2) The ablation study validates the effectiveness of the improved adaptation of the feature extractor using transductive task-encoding, and that of the soft k-means iterative estimation of means and covariances for classification.

3) Experiments are solid and extensive. The results on Meta-Dataset, mini/tiered-ImageNet are competitive, as opposed to the state-of-the-art methods.

Cons:

1) The originality of the proposed method is incremental. Compared to simple CNAPS (Betni et al, CVPR 2020), the task encoder is extended to incorporate both a support-set embedding and a query-set embedding through Long-Short Term memory (LSTM) network, and the classifier is extended to include the unlabeled examples in the query set as well. As far as I know, encoding both support examples and query ones using LSTM has been used in Matching Network (Vinyals et al., 2016) while the idea of using query examples to refine class porotypes via soft k-means clustering were proposed by Ren et al. ,2018.

2)  Regarding the soft-kmean iterative estimation, why can it be only used at test time while the performance decreases significantly if used during training as well? How many iterations are performed? Will various number of iterations matter? In particular, I am not satisfied with the limited explanation on this method provided in Section 5.2 (last paragraph on page 7), and I wish the authors give more analysis and insight on this problem.

3)  Some minor points:
In Table 2, what does "BN” mean?
How to set the value of $\beta$ in Equation (6)?
Some typos: In Section 3.2, "metalearning framework "  --> "meta learning framework "; in Section 4.2, "ou-of-domain" --> "out-of-domain".

review_title: This paper proposes a transductive few-shot classification method on the basis of the simple Conditional Neural Adaptive Processes (CNAPS) introduced by Bateni et al. The proposed method, called transductive CNAPS, extends the simple CNAPS by exploiting the query set both in the feature adaptation stage and classification stage. Extensive experiments are conducted on Meta-Dataset, mini-ImageNet and tiered-ImageNet, and very competitive results are reported.   

final_score: rating: 6: Marginally above acceptance threshold
review_content: In order to improve few shot visual classification, the authors propose a transductive meta-learning method using unlabelled examples. The authors have introduced a two-step transductive encoder as well as soft k-means clustering procedure on the existing simple CNAPS architecture.

Pros:
The paper is well written.
The authors have improved upon the existing simple CNAPS model by making two changes:
i) Simple CNAPS extracted feature vector by passing support example through a CNN to get support task representation $g_\theta(S)$ which is given as an input to FiLM layers to learn scale and shift parameters. This produces an adapted feature extractor to map both support/query onto adapted feature space. While in transductive SNAPS the support task representation is used to compute support-set embedding ($e_s$) and query-set embedding ($e_q$). Both $e_s$ and $e_q$ are processed by a two-step LSTM to generate the final transductive task-embedding used for adaptation.
ii) Introduce an iterative algorithm to assign soft-label assignment of query image which in turn performs iterative estimation of class means and covariances. 

Cons:
i) The performance on Meta-Dataset in Table 1 seems to show that the average rank is better only in the case for the Out-of-Domain test case.
ii) Table 2 shows when Transductive CNAPS compared with other transductive approaches the performance is not that great. However, when it is compared with other SNAPS architecture the performance is good. The missing experiment of excluding the overlapping classes would provide better insight into it.
iii) As already mentioned by the authors, this work resembles Gaussian Mixture Models.

Overall the model's novelty over the existing Simple SNAPS architecture is limited. Even the quantitate scores reported are not superior enough; for example in the case of Meta-Dataset the model suffers in In-Domain accuracy and does not achieve SOTA in case of mini/tiered-ImageNet. 

Final review
The authors have addressed some of my concerns so I am updating the rating. I still feel that the mode's novelty is limited and thus my highest rating will be 6. 
review_title: Interesting work but lacks novelty and performance improvements in some cases are limited

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper proposes a transductive few-shot learning method, Transductive CNAPS, by using the unlabeled examples. Experimental results on the Meta-Dataset and mini/tiered-ImageNet benchmarks are reported. 

The proposed approach is a simple extension of the simple CNAPS method. It extends the simple CNAPS by using the unlabeled query instances to update the class centroids \mu_k and variance Q_k with the predicted class probabilities. The level of technical contribution and novelty is very incremental and low.

The experiments are not very convincing. First, for both experiments on Meta-Dataset and min/tiered-ImageNet, the feature extractor is pretrained on some ImageNet subset. This is not appropriate as it borrows significant information from the pretraining dataset. The authors also mentioned this issue. 
Second, the transductive few-shot learning methods need to be compared with on the Meta-Dataset in Table 1.  Even with the current comparison methods, the performance gains of the proposed approach are not very notable. 

Some additional questions: 
1.	In Table 1, are all the results obtained using exactly the same sets of training/testing tasks for all comparision methods? What are the source domains for each of the out-of-domain task?
2.	Do all the comparision methods perform the same pre-training? 

review_title: incremental contribution

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper proposes to extend the simple CNAPS few-shot learning method to the transductive setting. Specifically, the proposed method introduces transductive task encoding and soft k-means iterative estimation to improve CNAPS. The proposed method is able to achieve SOTA results on the transductive benchmarks.

+ The proposed method can effectively improve the CNAPS as it achieves SOTA or comparable results on Meta-dataset, mini-ImageNet and tiered-ImageNet. 
+ The writing is clear and easy to follow.

- The proposed method lacks enough novelty. The proposed method is actually a transductive extension of CNAPS. The authors introduce two novel components: transductive task encoding and soft k-means clustering. The former is a natural and straight-forward solution while the latter is also similar to previous work [1]. This weakens the paper's overall contribution. 
- The experiments could be improved. As stated in the experiment section, some results are not well-prepared before the submission, e.g. Overlapping-excluded ImageNet results.
- According to Fig.4, the authors could further analyze why transductive updates can not remain improvements as the shot increases. Is there a way to design an adaptive scheme of deciding whether to use or not?
[1] Mengye Ren, Eleni Triantafillou, Sachin Ravi, Jake Snell, Kevin Swersky, Joshua B. Tenenbaum,
Hugo Larochelle, and Richard S. Zemel. Meta-learning for semi-supervised few-shot classification 

review_title: Please find below for the detailed comments

Meta-review: The submission proposes a transductive few-shot classification method on the basis of the simple Conditional Neural Adaptive Processes (CNAPS) introduced by Bateni et al. The paper received two borderline accept and two borderline reject reviews, indicating that the paper may not be yet ready for a publication. The meta reviewer recommends rejection based on the observations below.

All reviewers indicated that the paper is well-motivated, clearly written and neatly organized. However, all four reviewers agree that the novelty of the paper compared to the CNAPS paper is limited. The main novelty of the method being transductive CNAPS extends the task encoder of CNAPS to incorporate both a support-set embedding and a query-set embedding through Long-Short Term memory (LSTM) network. Similarly, the classifier in CNAPS has been modified to operate in the transduction setting, i.e. it is extended to include the unlabeled examples in the query set.  The reviewers indicate that extension of the task encoder via LSTM may not be enough technical novelty for such a competitive venue. Additionally, in terms of experimental evaluation, although R1 found the experimental evaluation adequate, R3 indicated some concerns about the unexpected behaviour of the method and R4&R2 found the benchmark evaluations limited.

Decision: Reject