Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper introduces a Primal Dual Minimum Description Length (PADDLE) formulation, which balances data fitting accuracy and model complexity for a given few-shot task. It extends the current benchmarks, so that the query-set classes of a given task are unknown and do not match exactly the support-set classes, but just belong to a much larger set of possible classes. With a drop in performance on some of the best performing SOTA transductive few-shot methods, the paper introduces a hyper-parameter free MDL-inference formulation which balances data-fitting accuracy and model complexity for a given few-shot task, subject to supervision constraints from the support set.
strengths_and_weaknesses: **Strengths**

- Block coordinate descent algorithm guarantees a convergence
- No hyperparameters need of tuning them
- The formulation can be used on top of any base class

**Weaknesses**

- The paper is hard to follow and is very dense. It would be great to add more empirical results to make it stronger.
questions: I think a few important baselines are missing. Please find then below:

1. Li C, Kothawade S, Chen F, Iyer R. PLATINUM: Semi-Supervised Model Agnostic Meta-Learning using Submodular Mutual Information. arXiv preprint arXiv:2201.12928. 2022 Jan 30.

2. Li, X., Sun, Q., Liu, Y., Zhou, Q., Zheng, S., Chua, T.-S., and Schiele, B. Learning to self-train for semi-supervised few-shot classification. Advances in Neural Information Processing Systems, 32:10276–10286, 2019.
limitations: See sections above.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes an algorithm called PADDLE to perform transductive few-shot classification where the classes in the query set may not entirely overlap with the classes in the support set. PADDLE consists of a partially supervised clustering optimization that balances partition complexity and the fit of prototypes to the data. An efficient block coordinate descent algorithm to solve this optimization is proposed and analyzed. Experiments on several few-shot classification benchmark datasets in the proposed transductive setting show that PADDLE often exhibits strong gains in accuracy relative to transductive baseline methods.
strengths_and_weaknesses: Strengths
- The few-shot setting investigated in this work is timely and realistic due to removing the assumption of perfect class overlap between support and query sets.
- The proposed algorithm is highly effective in the proposed transductive scenario. Performance is often quite significantly better than baselines.
- Experiments are thorough and informative. Sensible ablations demonstrate the utility of the partition complexity term and the proposed block coordinate descent algorithm.
- Analysis of the complexity of the algorithm and its convergence properties are provided.
- The writing is clear and insightful, drawing connections to classical algorithms.

Weaknesses
- It is unclear how the proposed method relates to previously proposed transductive few-shot algorithms and to existing MDL-based clustering algorithms. A related work section should be added.
- Experiments show the effectiveness of PADDLE in the proposed imbalanced transductive scenario but they do not show the limitations of PADDLE a more balanced setting.
- The task generation procedure is not described in sufficient detail.
questions: - How does the current work differ from other transductive few-shot approaches such as those proposed by Ziko et al. (2020) and Veilleux et al. (2021)?
- Please describe the task generation procedure in more detail. For example, how were the classes in the query set sampled? Are they guaranteed to partially overlap with the support set in some way?
- In figure 1, is there any insight as to why the accuracy of some baseline methods actually increases with $K_{eff}$?
- Is there any empirical evidence that setting $\lambda = |\mathbb{Q}|$ works better than alternatives?
limitations: Limitations could be improved by discussing the following:
- How can $K$ be set in a real-world scenario where the number of novel classes at test time is unknown?
- When is the transductive setting likely to not hold? Is there any recourse when there exists only a single query example?
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper introduced a PrimAl Dual Minimum Description LEngth (PADDLE) formulation, which balances data-fitting accuracy and model complexity for a given few-shot task, under supervision constraints from the support set.It is hyper-parameter free, and could be applied on top of any base-class training. Furthermore, this paper derive a fast block coordinate descent algorithm for optimizing our objective, with convergence guarantee, and a linear computational complexity at each iteration. Comprehensive experiments show highly competitive performances of this method, more so when the numbers of possible classes in the tasks increases.
strengths_and_weaknesses: Strengths:

1.The formulation proposed by this method is hyper-parameter free, and could be applied on top of any base-class training. This is very rare and significant.

2.The writing of the article is concise and clear. The formula is rigorous and the logic is clear.

3.Experiments are sufficient and strongly confirm the validity of the proposed method. 

4.The proposed few-shot setting is practical and will have a positive impact on many related work.

Weaknesses:

1. There seems to be a lack of a framework to give a visual overview of the proposed approach.

2. On mini-ImageNet dataset and set to 5-shot, the proposed method is slightly inferior to TIM [1]. What is the reason for this? Why didn't this happen on the tiered-ImageNet dataset?

3. Experimental Settings are not clearly explained, including software and hardware conditions, the settings of each hyper-parameter, etc.

Reference

[1] M. Boudiaf, I. M. Ziko, J. Rony, J. Dolz, P. Piantanida, and I. Ben Ayed, “Transductive information maximization for few-shot learning,” in Neural Information Processing Systems (NeurIPS), 2020. 
questions: 1. Based on their paper, I find that support is not very useful for few-shot learning compared with query set. I wonder if the randomly generated support sets would bring about good performance.
limitations: They have  adequately addressed the limitations and potential negative societal impact of their work.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper considers a more practice problem about the classes between the support and query sets, i.e., imbalanced K-way classification. A PADDLE method is proposed to solve the problem, which balances data-fitting accuracy and model complexity for a given few-shot task. Experiments show that the method achieves competitive performances in the standard few-shot datasets.  
strengths_and_weaknesses: Strengths:

This paper considers a very interesting problem, i.e., the query-set classes are unknown and do not match exactly the support-set classes, but just belong to a much larger set of possible classes.

Weaknesses

The clarity for this paper should be improved and related work should be discussed.
 


questions: 1. Important related work is missing from this paper, for example: transductive few-shot learning and few-shot image classification.
2. Some transductive few-shot learning methods should be compared in table 1, such as FEAT [1], SIB [2], DPGN [3].  
3. The equations (2) and (C) do not match, u_{n} does not appear in the (2). Is the class prototypes W a learned parameter? if not, why minimize the W? 
4. The optimization objective of transductive few-shot learning methods has a cross-entropy loss and entropy-based loss, but equation (2) has data-fitting accuracy and partition complexity.  What is the relationship between them？  


[1] Few-Shot Learning via Embedding Adaptation with Set-to-Set Functions

[2] Empirical Bayes Transductive Meta-Learning with Synthetic Gradients

[3] DPGN: Distribution Propagation Graph Network for Few-shot Learning
limitations: Yes
review_title: null

Meta-review: metareview: The reviewers have raised some concerns on the baselines used in the empirical evaluation. However, the additional experiments by the authors seem to have convinced the reviewers. Please make sure to add these additional experiments in the camera-ready version of the paper.
confidence: Less certain
Decision: Accept