Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper analyses the effect of class imbalance on few-shot learning problems. It draws a number of interesting (but kind of expected) conclusions e.g., the support set imbalance has a larger influence on the FSL performance compared to base class imbalance, a high impact of imbalance on gradient-based meta-learning methods compared to metric learning approaches. The paper is overall  

Pros: 

+ The paper is nicely written with a clear structure and exposition of ideas. 
+ An extensive number of FSL methods have been tested with three imbalance settings (linear, step, and random imbalance) on multiple datasets (Meta-Dataset and Mini-ImageNet) across various backbones. 
+ The paper considers class imbalance in both the base training and finetuning on the support set. 
+ Overall, the paper presents a thorough and detailed analysis of the class imbalance problem in FSL. 

Cons: 
- An approach to deal with the imbalance in FSL settings could have made the paper even more stronger. 
- Specifically, two very simple rebalancing methods are studied in the paper i.e., Random over-sampling and Random shot meta-training. An algorithmic approach for appropriate rebalancing in the loss function (e.g., [a,b,c]) would be intreresting to analyze. 

[a] Ren et al., Learning to Reweight Examples for Robust Deep Learning

[b] Khan et al., Cost-sensitive learning of deep feature representations from imbalanced data 

[c] Cui et al., Class-Balanced Loss Based on Effective Number of Samples



review_title: An extensive collection of results on imbalanced FSL

final_score: rating: 4: Ok but not good enough - rejection
review_content: Summary 

This paper introduces a new benchmark for imbalanced few-shot learning where the number of samples per class is different. The authors extensively evaluate 10 SOTA few-shot methods on this benchmark and show consistent performance drop in this challenging setting. They also show that simple over-sampling techniques can alleviate the imbalanced issue in few-shot learning. 

Pros

-This paper introduces a new benchmark to evaluate the imbalance problem in few-shot learning.  
-The evaluation is quite extensive and includes 10 SOTA method under different imbalanced settings

Cons

-The authors ignore the long-tail recognition literature [1, 2, 3] which is highly relevant and in my opinion, more important than the proposed imbalanced few-shot learning problem. [1] introduced a long-tail recognition benchmark where the ImageNet classes are divided into many-shot, medium-shot and few-shot classes based on the number of training examples. This setting is more realistic because the statistics of real-world datasets also follow a long-tail. I do agree that the imbalanced problem is very important. But I am not convinced that the proposed imbalanced few-shot learning (or meta-learning) evaluation protocol is appealing to the imbalanced problem community. 

[1] Large-Scale Long-Tailed Recognition in an Open World. Liu et al., CVPR'19

[2] Learning imbalanced datasets with label-distribution-aware margin loss. Cao et al., NeurIPS'19

[3] Decoupling representation and classifier for long-tailed recognition. Kang et al., ICLR'20

-The authors also ignore the generalized few-shot learning literature [4, 5, 5] which is closely related to imbalanced problems and few-shot learning. In particular, [4] introduced a benchmark that evaluates the performance on both base and novel classes where base classes have many samples and novel classes have only few shot examples.  

[4] Low-shot visual recognition by shrinking and hallucinating features. Hariharan et al., ICCV'18

[5] Low-shot learning with imprinted weights. Qi et al., CVPR'18

[6] Low-shot learning from imaginary data. Wang et al., CVPR'18

-There is no novelty except the proposed benchmark. The over-sampling techniques are standard and expected to improve the performance. 

Justification of the rating

As an evaluation paper, the authors ignore a large group of highly relevant works in long-tail recognition and generalized few-shot learning. I think the proposed benchmark is somewhat incremental to the existing long-tail recognition benchmark and recommand a rejection.  

----------------------------------------------------
Post-rebuttal

I have read the rebuttal and other reviews. The first version of this submission fails to cite any highly relevant works in long-tailed recognition and generalized few-shot learning. I believe that addressing this issue would require a major revision. The authors argue that their proposed few-shot imbalanced setting is different from the long-tailed recognition problem and generalized few-shot learning setting.  But this is only partially true. It is unclear whether previous approaches for long-tailed and generalized few-shot learning can be already applied to address the few-shot imbalanced problem. Moreover, I am not convinced that the proposed setting is more appealing than those two existing imbalanced settings because the 5-way classification problem seems to be an artificial setting that rarely happens in practice. Finally, I realize that the proposed setting is actually not new. [Lee et al., ICLR 2020] have explored a very similar setting. Thus, I would keep my original review and recommend rejection.

review_title: Official Blind Review #1

final_score: rating: 5: Marginally below acceptance threshold
review_content: The authors present a detailed study of few-shot class-imbalance along three axes: dataset vs. support set imbalance, effect of different imbalance distributions (linear, step, random), and effect of rebalancing techniques. The authors extensively compare over 10 state-of-the-art few-shot learning methods using backbones of different depths on multiple datasets. The analysis reveals that 1) compared to the balanced task, the performances of their class-imbalance counterparts always drop, by up to 18.0% for optimization-based methods, although feature-transfer and metric-based methods generally suffer less, 2) strategies used to mitigate imbalance in supervised learning can be adapted to the few-shot case resulting in better performances, 3) the effects of imbalance at the dataset level are less significant than the effects at the support set level. 


Pros: 
1) the paper covers the state-of-the-art few-shot learning methods, over 10 methods are compared in the paper;
2) the work reveals some interesting insights in few-shot learning, such as the three analysis summarized in Abstract. 
3) the experiments are reasonable. There are a number of comparisons between different methods on different data sets. The codes to reproduce the experiments is released under an open-source license. 

Cons: 
1) the paper does not provide a new model and the contribution is marginal. 
2) the experiments does not introduce new datasets as benchmark, all the datasets are heavily manipulated during testing. Is there any new data sets provides to test the assumptions of class-imbalance few-shot learning?
3) the paper does not fully discuss new possible research directions in the field of class imbalance few learning. Although the authors discuss some insight into the previously unaddressed CI problem in the (meta-) training dataset and conclude that the effects of imbalance at the dataset level are less significant than the effects at the support set level, the future work along this direction seems still unclear.
review_title: This paper systematically investigated the class imabalance problem in few-shot learning from multiple aspects.  

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper conducts extensive comparison experiments to study the effect of class-imbalance for many few-shot approaches. A detailed study of few-shot class-imbalance along three axes: dataset vs. support set imbalance, effect of different imbalance distributions (linear, step, random), and effect of rebalancing techniques, are presented. Also, this paper is clearly written and easy to understand. 

1. Though eleven few-shot approaches are considered, some strong baselines are missing, such as [1];
2. In the contribution part, this paper declares "compare over 10 state-of-the-art few-shot learning methods using backbones of different depths on multiple datasets", however, "backbones of different depths" is commonly used in few-shot learning literature. Therefore it cannot reflect much contribution of this paper;
3. Some related work is not discussed [2,3]. For instnace, prior work [2] discusses the effect of different value of $k$ in meta-training and meta-testing, which is pretty much similar to the concept "imbalance" studied in this paper;
4. Overall, the contribution of this paper is somewhat limited. Apart from conducting extensive experiments, more informative observations and conclusions should be made.

[1] A Baseline for Few-Shot Image Classification. ICLR 2020.
[2] A Theoretical Analysis of the Number of Shots in Few-Shot Learning. ICLR 2020.
[3] Learning to Stop While Learning to Predict. ICML 2020.


review_title: This paper conducts extensive comparison experiments to study the effect of class-imbalance for many few-shot approaches.

Meta-review: The paper studies the effectiveness of few-shot learning techniques in settings where the training labels are imbalanced. While addressing an interesting practical problem, reviewers raised concerns about the paper's technical depth, insufficient distinction to existing techniques for coping with label imbalance, and limited qualitative conclusions from the results. The authors incorporated some of these comments in their revision, but a more comprehensive update on the latter two points appears appropriate.
Decision: Reject