Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This paper studies transductive evaluation in few-shot learning, which allows the model to use statistics of all the unlabeled query set examples when making class predictions. Models evaluated in transductive fashion typically attain higher metrics by using this extra information. Most work on transductive evaluation, however, assumes that the query set is evenly distributed across the possible classes. This is of course an unrealistic assumption for the real world and this paper specifically studies this aspect: (1) They evaluate in a more realistic transductive evaluation scenario where the class counts within the query set are sampled according to a Dirichlet distribution and show that most transductive evaluation-based methods perform worse in this setup; (2) They propose a modification of an existing model for transductive evaluation to work better in this scenario where the class counts can be varied in the query set.
main_review: Originality: the paper considers an under-studied but important aspect of transductive evaluation that to my knowledge hasn't been considered in previous work. Additionally the authors propose a modification to an existing transductive evaluation method [1] to deal with varied class makeup in the query set. The extension is simple but well-motivated as there was careful analysis to show why it would be expected to work better.

Quality: the submission seems very technically sound. The proposal for their extension is backed up by strong reasoning and the experiments seem to be conducted well.

Clarity: the paper was very well-written. It motivates the problem and why it is important well. The setup of the new sampling scheme for query set class distribution was also described well. Lastly, the description of previous methods for transductive shot-learning and the proposed extension were also stated clearly and were easy to follow.

Significance: the paper presents some interesting results: (1) It shows how most of the transductive evaluation papers are benefitting from the uniform class distribution assumption in the query set. In fact, the authors observe that more than half of the transductive methods studied perform worse than inductive baselines in this more realistic setting. This definitely brings to question some of the value of this type of work as they are taking advantage of an artificial scenario. (2) The proposed transductive method has strong performance, as its the leading method for mini-ImageNet and tiered-ImageNet in most cases and sometimes gives benefit up to 2-3% compared to the leading transductive method.

[1] Boudiaf et al. Transductive information maximization for few-shot learning. NeurIPS 2020.
limitations_and_societal_impact: Though not discussed, the societal impact of this work is probably similar to that of any other few-shot method.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The researchers use transductive inference and introduce the effect of arbitrary class distributions within the query sets of few-shot tasks, rather than using class-balanced tasks, by removing the class-balance artefact. They do this by modeling the marginal probabilities of the
classes as Dirichlet-distributed random variables, rather than from a known and fixed uniform distribution.
They assess their model by comparing transductive methods over 3 datasets, and show their experiment setting’s performance drops compared to inductive methods. They also propose a generalization of the mutual-information loss based on alpha-divergences, which is an extension of the Shannon mutual information and tolerates class-distribution variations more effectively.
main_review: The quality of the paper was fairly good. It is significant as it builds upon the current few-shots learning landscape, which utilizes the transductive and inductive methods. They propose the Dirichlet distribution to sample query sets as it creates a realistic setting, instead of using the current artificially balanced uniform distribution setting,

The paper illustrates that transductive few-shot methods claiming significant gains over inductive tasks may perform worse when evaluated with their realistic setting, based on their results. The paper demonstrates the conclusion with thorough experiments. 

The paper also proposes alpha TIM formulation, with better generalizability n when training a few-shot scenario and sheds light on the use of alpha-divergences in such applications. The experiments also demonstrate that the alpha TIM consistently performs better with the same budget of hyper-parameter optimization as the standard TIM.

The hardware requirement is low, requiring only a single 1080 Ti GPU.

Furthermore, in terms of the clarity of the paper, it flowed well. The paper should be reproducible.
Typo on the caption of Table 1: "Comparaisons"
limitations_and_societal_impact: The authors have not addressed the limitations of their work.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper evaluates several transductive few-shot learning methods [15,28,20,21,23,17] on class-imbalanced FSL tasks. The class distribution in the query set follows a Dirichlet distribution. The support set is balanced. This work proposes a novel method based on $\alpha$-divergences addressing the imbalance problem. The novel method $\alpha$-TIM achieves competitive performance compared to several recent baselines.
main_review: \+ means strength, \- means weakness

**Originality**

\+ *Novel setting.* This paper presents a novel, more realistic setting for the evaluation of transductive learning methods. Unlike previous work on few-shot imbalance [32, 33, 34, 35], this paper focuses on the imbalance present in the query set.

\+ *Novel algorithm.* This paper presents an augmentation of the TIM algorithm [23] that combines with $\alpha$-divergences [37, 38, 39, 40, 41]. This algorithm achieves state-of-the-art performances compared to several recent state-of-the-art transductive few-shot methods [15,28,20,21,23,17]. Unlike this previous work, the authors apply a unique mechanism to deal with class imbalance.

**Quality**

\+ *The paper compares a range of recent methods, datasets and backbones.* Several recent transductive [15,28,20,21,23,17] and non-transductive [4, 29, 30] algorithms are evaluated in this novel setting. The datasets included Mini-ImageNet, tiered-ImageNet, and CUB. And backbones included WRN and ResNet-18. 

\- *Evaluation on only one distribution is limiting.* The authors use one alpha value for the Dirichlet distribution (lines 257-264) to model query set imbalance during evaluation. However, it is not clear why this distribution was picked over others. The class imbalance community commonly a range of distributions, for example, step imbalance and linear imbalance (Buda et al., 2018), long-tail imbalance (Wang et al., 2018), and in few-shot previous work sampled uniformly at random [32,33,34,35], or a combination of these distributions [35]. It might be appropriate to evaluate several distributions or vary the $a$ in the Dirichlet distribution. 

\- *The optimal $\alpha$ in $\alpha$-TIM might depend on the Dirichlet distribution and the shot setting.* Figure 3 shows that in the 1-shot setting, the optimal $\alpha$ value for the test dataset is around the value of the $\alpha=2$ similar to that concentration parameter used in the Dirichlet distribution ($a=2$). It could be interesting to show the correlation between $\alpha$ and $a$ in a graph. The authors could also examine a wider $\alpha$ range as the values for the 5-shot settings sometimes do not appear optimal. What are the optimal $\alpha$'s for higher shot settings (e.g. 10-shot, 20-shot)? Is there a correlation between the $\alpha$ and shots? 

\- *Missing values in Table 1.* What is $alpha$-TIM performance on the balanced setting? For completeness and clarity, for methods [4, 29, 30], it might be worth adding a symbol (similar to the red/blue arrow) to indicate no change in performance. Also, there is no performance for [15] in 10-shot and 20-shot settings with mini-ImageNet and tiered ImageNet. Was that intentional? 

**Clarity**

\+/\- *The paper is generally well organised but some clarifications and adjustments would help.* Some adjustments would make this work easier to follow. For instance, the bullet points for the contributions section. The paper should emphasise earlier that imbalance is studied only at the query set level, and support sets are assumed to be balanced. Some sentences could be improved, split or condensed, e.g. lines 81-83. 

\- *Clarification for eq. (4).* Why do the authors multiply the $\mathcal{D}_\alpha (p||u_K)$ by the term $K^{1-\alpha}$ in eq. (4)? The term does not appear in the RHS of eq. (2). In addition, it might be worth adding a derivation in the appendix. 

\- *Clarification about control and imbalance of the query set (Q).* The authors claim (in lines 121-122) that recent studies [32, 33, 34, 35] "rely on" keeping the query set balanced. However, I can't entirely agree, and I would argue that for *non-transductive* few-shot classification methods such as those studied in [32, 33, 34, 35], the distribution of Q does not matter during meta-testing. During the meta-training (assuming full access to the base dataset), Q can be controlled arbitrarily. Keeping a balanced Q is usually done for convenience but this does not restrict "non-transductive" methods. However, I do agree that for *transductive* methods assuming that the query set is balanced and accessible all at once is unrealistic, and will directly affect performance (as evidenced in this paper) so studying this area seems important. 

\- *No $\lambda$ in mutual information tradeoff?*. Eq. (5) does not contain the $\lambda$ in eq. (1). What is the reasoning behind this? What is the performance when $\lambda$ is included and tuned with $\alpha$?

**Significance**

\+ *The paper presents a more realistic evaluation of transductive methods.* This analysis shows that several relevant and recent transductive few-shot methods underperform under query set imbalance. These results would be both interesting and valuable to the community. 

\+ *The paper presents a novel and interesting approach to the problem.* The paper presents an interesting take on the class imbalance problem through the $\alpha$-divergences. Making the method $\alpha$-TIM significantly different from TIM and having the most significant improvements in the higher shot settings. 

\- *However, the paper feels like work in progress and could be improved significantly* (as outlined here). I am inclined towards a borderline reject at the moment, but I look forward to hearing back from the authors and seeing how this work progresses in the future. 

**Some spelling mistakes**

\- line 133, "priori" --> "a prior"

\- line 283, "outperforms" --> "outperform"

\- line 304, "results points" --> "result points" / "results point"

**References**

Buda et al., 2018, "A systematic study of the class imbalance problem in convolutional neural networks." Neural Networks 106 (2018): 249-259.

Wang et al., 2017, "Learning to Model the Tail." NeurIPS 2017


_____
### POST-REBUTTAL
_____
The authors have answered all of my questions above expectations through additional experiments and clear explanations. I will now recommend this paper for the conference with a score of 7 and increased confidence (conditioned on the inclusion of the additional experiments in the main paper / supplementary materials).

limitations_and_societal_impact: Some limitations were listed, but the paper would benefit from a separate limitation section.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper studies transductive few-shot learning in the more realistic setting with arbitrary class distributions within the query sets. In the proposed setting, authors use Dirichlet distribution to model the marginal probabilities of the classes in the query sets as random variables and generate random samples within the simplex. Focusing on two recent transductive methods PT-MAP and TIM, they first show that class-balance prior is encoded in these two methods and then extend TIM by generalizing mutual-information loss to \alpha divergences to more effectively deal with arbitrary class distributions. The experiments validate proposed \alpha-divergence approach on three standard benchmark datasets (mini-ImageNet, tiered-ImageNet and CUB) and show that the method outperforms inductive and transductive few-shot learning methods in the proposed setting.
main_review: Pros:
- The paper is well-motivated, clearly written and deals with a more realistic evaluation of the transductive few-shot learning methods.
- Proposed alpha-divergence approach is a simple and effective extension of TIM for dealing with the arbitrary class distributions of the query
- Experimental results show improvements over existing methods

Cons:
- The paper focuses on only two transductive methods PT-MAP and TIM to show that the class-balance prior is encoded in these methods. What about other transductive methods? For example, from the experiments, BD-CSPN does not seem to be so sensitive (especially under assumption that some performance drop is expected). Why is the paper focused only on transfer learning and not on meta-learning approaches?  The current study is not comprehensive.
- Experimental results are not convincing. First, there is a lack of stronger baselines in Table 1. Authors claim to outperform state-of-the-art inductive methods but do not compare to recent approaches such as MetaOptNet [1], CTM [2] and DeepEMD [3]. The difference between inductive and transductive methods in this setting seems small and with stronger baselines claims might change. 
- Why is Dirichlet parameter a set to 2? How does the performance change under different imbalance distributions? Where is the point where existing methods start performing better? Extremely, how does the method compare under a standard balanced scenario? More experiments are needed to answer these questions.
- In Table 1, how is the \alpha parameter tuned? Which values of \alpha are reported in the table? Similarly for TIM, how is \lambda tuned and which values are reported?

[1] Meta-Learning with Differentiable Convex Optimization. CVPR ‘19

[2] Finding Task-Relevant Features for Few-Shot Learning by Category Traversal. CVPR ‘19

[3] DeepEMD: Differentiable Earth Mover’s Distance for Few-Shot Learning. CVPR ‘20

limitations_and_societal_impact: Limitations of the work are not mentioned. For example, is it better to use existing methods with balanced class distributions (see cons)?
review_title: null

Meta-review: The submission investigates the effect of a non-uniform marginal label distribution on transductive approaches to few-shot classification. When the marginal label distribution of test episodes is sampled from a Dirichlet distribution, the paper finds that state-of-the-art transductive approaches suffer a substantial performance drop. The submission also introduces a generalization of the mutual information loss which better handles class imbalance and is shown to outperform competing approaches in that setting.

Reviewers found that the paper is well-written and sheds light on an underexplored and important aspect of evaluating transductive approaches to few-shot classification. Overall they found that the proposed generalization of the mutual information loss is technically sound and convincingly backed up by experiments.

Some reviewers expressed concerns about the experimental design (the value of alpha and the number of values tried, its optimal value as a function of the shot setting, the choice of baselines to compare against) which were addressed to their satisfaction by the authors.

Ultimately, all reviewers agree that the paper should be accepted. I therefore recommend acceptance.
Decision: Accept (Poster)