MDFL: A UNIFIED FRAMEWORK WITH META-DROPOUT FOR FEW-SHOT LEARNING

Abstract: Conventional training of deep neural networks usually requires a substantial amount of data with expensive human annotations. In this paper, we utilize the idea of meta-learning to integrate two very different streams of few-shot learning, i.e., the episodic meta-learning-based and pre-train finetune-based few-shot learning, and form a unified meta-learning framework. In order to improve the generalization power of our framework, we propose a simple yet effective strategy named meta-dropout, which is applied to the transferable knowledge generalized from base categories to novel categories. The proposed strategy can effectively prevent neural units from co-adapting excessively in the meta-training stage. Extensive experiments on the few-shot object detection and few-shot image classification datasets, i.e., Pascal VOC, MS COCO, CUB, and mini-ImageNet, validate the effectiveness of our method.


Reviews and author responses:

 Thank you authors for the feedback. I don't see significant contributions in this current version. It is actually a good direction to explore the better generalization of meta-models using regularization methods. Hope to see an improved version in the future.

 Thanks the authors for providing the response. Unfortunately, I still have concerns about the contributions of this submission as also mentioned by other reviewers. Thus, I keep my score.

The paper presents a new framework to unify two streams of few-shot learning methods, the episodicc meta-learning and pre-train finetune-based few-shot learning. Besides, a new meta-dropout strategy is proposed to improve the generalization ability. Extensive results on several benchmark like PascalVOC, COCO, CUB and mini-ImageNet validate the proposed approach.  Pros:
1. The idea to unify the episodic meta-learning and pre-train finetune-based few-shot learning is meaningful for the improvment of the few-shot learning performance.
2. The proposed algorithm is extensively validated on the popular benchmarks like PascalVOC, COCO, CUB and mini-ImageNet.


Cons:
1. The performance gain of meta-dropout is of large variance. For example, Based on Table 1, different splitting setting may have large variations. In split1, the gain of meta-dropout on Meta-R-CNN is around 10 for 1shot for split1 but only 3.3 for split2.  Also, how about the performance gain for meta-dropout on other approaches besides from Meta R-CNN and TFA. 
2. The comparison in the experiments miss several state-of-art paper results. For example, in Table 1 on the VOC2007 benchmark, there are a few published paper with better performance as listed. To name a few:
[R1] Wu, J.; Liu, S.; Huang, D.; and Wang, Y. 2020. Multi-Scale Positive Sample Refinement for Few-Shot Object Detection.In ECCV 2020
[R2] Li, B.; Yang, B.; Liu, C.; Liu, F.; Ji, R.; and Ye, Q. 2021. Beyond Max-Margin: Class Margin Equilibrium for Fewshot Object Detection. In CVPR 2021
3. The contribution of the paper is not significant enough to match the level of ICLR.  The paper targets at the interesting problem of few-shot learning and presents a unified framework and a meta-dropout method. Although the experiments are widely validated on several benchmarks, the main concern is the comparison of the experiments miss several State-of-the-art paper results. 

-----------------------------------------------------------------------------
Update after the rebuttal:
The rebuttal does not well address the concerns mentioned in the previous round of review. For example, the experiments can list the setup using a fixed selection of novel samples for a fair comparison with state-of-the-arts besides from the multiple rounds of results listed in the paper. 

 Thanks for your review.

1. The episode-based and pre-train finetune-based methods are two existing mainstream methods in few-shot learning. For example, Meta R-CNN is an episode-based method and TFA belongs to the category of pre-train finetune-based methods. Based on the current categories, we present and explain the common mathematical principles of these two streams of methods. By using our proposed formulation, these two different streams of methods become similar in the view of the optimization process.

2. The differences between the episode-based and pre-train finetune-based methods include the training pipeline (normal training or episodic training) and the distribution of datasets during training (One overall task or multiple serial tasks). The motivation is that our unified framework can list several common and important components in two different methods and identify which part of a model is the transferable knowledge, so the proposed meta-dropout can be easily applied to existing few-shot models. 

3. The novelty of our framework includes: (1) Based on the original meta-learning framework, we consider the distribution of source tasks (T) and the distribution of data in each task (Dt) that can present the common characteristics of the above two methods and explain two different training pipelines in theory. (2) Our framework can identify which part of a model is the transferable knowledge, so the proposed meta-dropout can be easily applied to existing few-shot models. The formulations in the sections 3.2.3 and 3.2.4 are re-formulated based on that of 3.2.1. The components of these formulations are similar. The re-formulated functions are used to explain the important and common components among different streams of methods in one framework. 

4. It is true that we apply the dropout mechanism to meta-learning models. However, it is important to know that we apply it to the meta-knowledge of few-shot models, thanks to our unified framework pointing out which part of models is meta-knowledge in existing few-shot models. The insight from our framework we think is a strong contribution. In order to prove the correctness of our meta-dropout, we do experiments to show the performance of applying meta-dropout, normal dropout, and both in Table.8 and Table.14 in the appendix. The experimental results show that only using meta-dropout can achieve the best performance.

5. The goal of our paper is to prove the generalization power of the proposed framework. We do extensive experiments on the detection and classification tasks. In detection, the results of comparison between our method with the state-of-the-art papers are shown in Table.7 in the section of the appendix. In Table.7, our method FSCE* outperforms the baseline by a large margin and achieves better performance than other state-of-the-art methods. In classification, we choose a representative classification method (Baseline++) for proving the validate the generalization power of our method with the commonly used datasets, such as CUB and mini-ImageNet [1,2].

[1] ICLR2019, A CLOSER LOOK AT FEW-SHOT CLASSIFICATION

[2] ICJAI2021, Conditional Self-Supervised Learning for Few-Shot Classification


 Thank you for your review.

1. Our framework, which explains the common mathematical principles of these two streams of methods, is different from the methods that combine pre-training and episodic training in methodology to get a fused detector or classifier. Our framework can list several common and important components in two different methods and identify which part of a model is the transferable knowledge, so the proposed meta-dropout can be easily applied to existing few-shot models, such as Meta R-CNN, TFA, Baseline++, and FSCE.

2. In Figure.1, we use the trainable backbone as an example of the meta-knowledge and the last fully connected head as the task-specific knowledge for better understanding. In the different approaches, meta-knowledge can be different types, such as the frozen feature in TFA and the initialization point of backbone in FSCE, etc.

3. Our framework is built upon the optimization theory of meta-learning and distinguishes meta-knowledge (applying meta-dropout) from task-knowledge (applying normal dropout). In order to optimize few-shot models better, meta-knowledge should be more generalized for providing a good start point. In order to prove the correctness of our meta-dropout, we do experiments to show the performance of applying meta-dropout, normal dropout, and both in Table.8 and Table.14 in the appendix. That is why our strategy is using the idea of dropout on meta-knowledge, not target-knowledge.


 Thank you for your review.

1. Our framework, which explains the common mathematical principles of these two streams of methods, is different from the methods that combine pre-training and episodic training in methodology to get a fused detector or classifier. Our framework can list several common and important components in two different methods and identify which part of a model is the transferable knowledge, so the proposed meta-dropout can be easily applied to existing few-shot models, such as Meta R-CNN, TFA, Baseline++, and FSCE.

2. We add the missing references and modify our statement in the related works.

3. The comparisons of our method with TFA are presented in Table.1, Table.2 and Table.4. The results show that our method can achieve higher average precision for novel classes in almost all settings. Following most few-shot methods [1,2,3,4,5], our model cares more about the performance of models on novel classes. However, according to our experiments, applying meta-dropout does not hurt the accuracy of base classes. We add the results of TFA w/o meta-dropout on base and novel classes in Table.12 in the appendix. 

4. In Table.2, we can’t reproduce the precision of TFA in their original paper. In order to make the comparison become fair, our method is compared to our re-implemented TFA. We think it can show the effectiveness of our meta-dropout. In addition, the improvement of meta-dropout on Meta R-CNN, TFA, FSCE, and Baseline++ can prove the generalization of our method further.

5. We just follow the Meta R-CNN, TFA, and other recent methods [5,6,7] to report AP50.

6. In the ablation studies, all comparisons are fair. In Table.3, meta-dropout on Meta R-CNN obtains about 1% improvement on average. In Table.4, meta-dropout on TFA can get 4% improvement on average. In Table.6, meta-dropout on Baseline++ can bring 2%-3% improvement. In Table.7, meta-dropout on FSCE brings 2%-3% improvement on average. All comparison results show that our meta-dropout can improve the baselines in a fair setting. 

7. We follow most of the few-shot methods [5,6,8,9,10] to do ablation studies with VOC 1 split. 

8. The results of comparison between our method with the state-of-the-art papers are shown in Table.7 in the section of the appendix. In Table.7, we build our model based on FSCE [6], and our method FSCE* outperforms the baseline by a large margin and achieves better performance than other state-of-the-art methods. These recent papers use the fixed novel samples to run the experiment only once and we use different novel samples to run multiple experiments to get more stable results as shown in Table.1,2 and 5.

9. The goal of our paper is to prove the generalization power of our meta-dropout. We do extensive experiments on the detection and classification tasks and follow the commonly used datasets, such as CUB and mini-ImageNet in few-shot classification [1,2,11]. 

[1] ICLR2021, FREE LUNCH FOR FEW-SHOT LEARNING: DISTRIBUTION CALIBRATION

[2] ICLR2019, A CLOSER LOOK AT FEW-SHOT CLASSIFICATION

[3] ICCV2019, Few-shot Object Detection via Feature Reweighting

[4] CVPR2020, Few-Shot Object Detection with Attention-RPN and Multi-Relation Detector

[5] CVPR2021, Beyond Max-Margin: Class Margin Equilibrium for Few-shot Object Detection

[6] CVPR2021, FSCE: Few-Shot Object Detection via Contrastive Proposal Encoding

[7] CVPR2021, Generalized Few-Shot Object Detection without Forgetting 

[8] ECCV2020, Multi-Scale Positive Sample Refinement for Few-Shot Object Detection

[9] CVPR2021, Transformation Invariant Few-Shot Object Detection

[10] ICCV2019, Meta R-CNN: Towards General Solver for Instance-level Low-shot Learning

[11] ICJAI2021, Conditional Self-Supervised Learning for Few-Shot Classification

 Thank you for your review. 

1. For the first question in Cons, the performance gain of meta-dropout is related to the number of novel samples and the intrinsic characteristic of novel instances. However, meta-dropout significantly improves the baselines of few-shot detection (Meta R-CNN and TFA) and classification (Baseline++) in almost all settings (different split or shot). In addition to Meta R-CNN and TFA, we also provide the result of our meta-dropout on FSCE (CVPR2021) in Table.7 in the section of the appendix which shows the generalization of our method further.

2. Your main concern is the results of comparison between our method with the state-of-the-art papers, which are shown in Table.7. In Table.7, we build our model based on FSCE, and our method FSCE* outperforms the baseline by a large margin and achieves better performance than other state-of-the-art methods. These recent papers [1,2,3,4,5,6] use a single fixed selection of novel samples to run the experiment only once. But the approaches in Table.1/2/5 use several different selections of novel samples to run multiple experiments to report more stable results. That is the reason why we didn’t list these papers in Table.1.

3. Contribution: (1) Our framework, which presents the important and common mathematical principles of existing few-shot models, is different from the methods that combine pre-training and episodic training in methodology. (2) The proposed framework is useful for guiding the improvement of existing few-shot models, such as our meta-dropout on meta-knowledge can be generalized to different types of models and tasks. That is why we do extensive experiments on detection and classification tasks with different kinds of baselines.

[1] CVPR2021, Beyond Max-Margin: Class Margin Equilibrium for Few-shot Object Detection

[2] ECCV2020, Multi-Scale Positive Sample Refinement for Few-Shot Object Detection

[3] CVPR2021, Few-Shot Object Detection via Classification Refinement and Distractor Retreatment

[4] CVPR2021, Generalized Few-Shot Object Detection without Forgetting

[5] CVPR2021, Hallucination Improves Few-Shot Object Detection

[6] CVPR2021, FSCE: Few-Shot Object Detection via Contrastive Proposal Encoding

The authors propose a unified framework for meta-learning based algorithm and pretrain finetune based algorithm. Drop out is applied for few-shot learning and achieve better generalization ability. Experiments on two tasks validate the effectiveness of the proposed algorithm. Strengths: validate the proposed algorithm on both few-shot object detection and few-shot classification tasks.

Weakness: unclear contributions

The paper claims to have three contributions: 1. utilize the idea of meta-learning to integrate two very different streams of few-shot learning, i.e., the episodic meta-learning-based and pre-train ﬁnetune-based few-shot learning, and form a uniﬁed meta-learning framework. 2. propose a simple yet effective strategy, named meta-dropout, to improve the generalization power of meta-knowledge in our framework. 3. Experiments of baselines from different streams evaluate the effectiveness of our approach on the few-shot object detection and image classiﬁcation datasets, i.e., Pascal VOC, MS COCO, CUB, and mini-ImageNet.

However, after reading this paper, I felt the contribution claimed is not supported well and thus, the contribution of this paper is not very clearly stated. The detailed reasons are as follows:

In section 3, the authors formulated episode-based meta-learning and pretrain finetune based meta-learning. From section 3.1, it seems the difference only lies in if the backbone is trainable or not. I do not see any motivation of dividing into two categories and why a unified framework is a contribution here. Also, 3.2.2 and 3.2.3 states the unified framework, and i do not see any novelty for these formulations here.

Section 3.3 shows how to apply the dropout mechanism to improve the generalization of meta-learning models. I do not see any further improvements for drop out but only applying drop out to meta-learning models. Thus, i do not feel this is a strong contribution for this paper.

As for the experiment results, i appreciate the authors do experiment on two tasks. However, the baselines are too old (baselines in 2019). The datasets are also not large-scaled as tiered-Imagenet or Meta-dataset. Thus, it does not have sufficient proof on the state-of-the-art performance of the algorithm.

 The pros of this submission is to validate the algorithm on different few-shot tasks but the contribution is unclear.

The paper addresses few-shot learning in a combination of episodic meta-learning-based and pre-train finetune-based approach. It proposes meta-dropout, that consists in using dropout on training the base classes during meta-training, but not when tuning on novel classes. It presents results for both classification and object detection tasks. The paper is hard to read and the novelty and impact of the proposed contributions are not clear. Some questions that would help to clarify the formulation:
1)  The paper claims to have a novel benefit from combining pre-training with episodic training. In that direction, references on previous works combining pre-training and episodic training are missing. Just to mention a few combining pre-training and episodic training: https://arxiv.org/pdf/2003.04390.pdf
https://arxiv.org/abs/1903.03096.
2) Related works claim "Pre-train finetune-based approaches are basic yet ignored in few-shot learning due to the excellent
performance of episode-based methods". But this statement ignores recent literature showing superior performance of 
finetune-based approaches for few shot classification (not referenced in the paper) and also the findings of Wang et al. 2020  for object detection (referenced in the paper). Some references: 
https://openreview.net/forum?id=Hq8rqCDkETi
https://arxiv.org/abs/1904.04232
https://openreview.net/pdf?id=Q0hm0_G1mpH
3) It is not clear the improvement or novelty of the proposed approach when compared with existent approaches. As TFA seams to be the closer reference, a clear comparison with it is missing (Wang et al. 2020). The results presented are not comparable as they dont present base classes performance. TFA main property is to preserve base classes performance while at the same time it obtains good performance on novel classes. As table containing results only show novel classes performance, it is not clear if this property is maintained or if results shown on novel may be associated with a decrease in base classes performance.
4) There are no partial results on the element that compose the formulation that would help to clarify the importance of the individual contributions. The results on table 2 showing the  benefits of meta-dropout are inferior to TFA original performance. TFA original results are not presented in the table. 
5) The fact that AP is improved but not AP50 should be better justified as it can be seen as a proxy of localization precision performance.
6) The change in batch size alone adopted in the paper can induce some non intuitive side effects that make comparison with other results unclear.
7) It is not clear why results of VOC where show on split 1 only.
8) References and comparison with SOTA few shot detection approaches and results are missing.
9) Results on classification on top of CUB and Mini-Imagenet seems to benefit from a strong baseline. The evaluation of the proposal in a more diversified few shot dataset such as Meta-Dataset and/or VTAB would clarify the relevance of the proposal in respect to few shot classification SOTA results. The paper is hard to read and the novelty and impact of the proposed contributions are not clear. Results lack a clear comparison with the recent literature. 

This paper aims to improve the generalization power of meta-learning. It claims to propose an effective strategy named meta-dropout, which is applied to the transferable knowledge generalized from base categories to novel categories. The proposed strategy aims to prevent neural units from co-adapting excessively in the meta-training stage (where somehow I did not get the idea of what it is). I understand from the abstract and introduction that the paper aims to combine two different solutions of meta-learning and build a unified one that could potentially improve the task generalization ability of the meta-model. However, it is not easy for me to find any proof of new ideas to achieve that target in the method sections. More importantly, it is not new in terms of the "idea" of combining "the episodic meta-learning-based and pre-train finetune-based few-shot learning". For example in the CVPR19 paper "Meta-transfer learning for few-shot learning" or in the ICLR19 paper "Meta-Learning with Latent Embedding Optimization", they all use the idea of pre-trained models in solving few-shot tasks and train models with episodic tasks. Other detailed comments are in the following: It is not clear in Fig1 what is the target knowledge or concrete format of the knowledge to learn or transfer to testing tasks; It is not clear why using a dropout in the meta-learning process can be called meta-dropout. Any new design of its optimization in meta-level?
 This paper is not easy to follow and lacks novelty in the method. 

Meta-review: This work proposes an approach to unify pre-training-based and meta-learning-based few-shot learning, inspired by dropout.

None of the reviewers support the acceptance of this work, despite the authors' detailed rebuttals, with the majority of reviewers confirming their preference for rejection following the author response. 

I unfortunately could not find a good reason to dissent from the reviewers majority opinion, and therefore also recommend rejection at this time.

Decision: Reject