The Unreasonable Effectiveness of the Class-reversed Sampling in Tail Sample Memorization

Abstract: Long-tailed visual recognition poses significant challenges to traditional machine learning and emerging deep networks due to its inherent class imbalance. A common belief is that tail classes with few samples cannot exhibit enough regularity for pattern extraction. What makes things worse, the limited cardinality may lead to low exposure of tail classes in the training stage. Re-sampling methods, especially those who naively enlarge the exposure frequency, eventually fail with head classes under-represented and tail classes overfitted.
      Arguing that long-tailed learning involves a trade-off between head class pattern extraction and tail class memorizing, we first empirically identify the regularity of classes under long-tailed distributions and find that regularity of the same training samples will be sharply decreased with the reduction of class cardinality. Motivated by the recent success of a series works on the memorization-generalization mechanism, we propose a simple yet effective training strategy by switching from instance-balanced sampling to class-reversed sampling to memorize tail classes without seriously damaging the representation of head classes. Closely after- wards, we give the theoretical generalization error upper bound to prove that class- reversed sampling is better than instance-balanced sampling during the last train- ing stage. In our experiments, the proposed method can reach the state-of-the-art performance more efficiently than current methods, on several datasets. Further experiments also validate the superior performance of the proposed sampling strategy, implying that the long-tailed learning trade-off could be effectively tackled only in the memorization stage with a small learning rate and over-exposure of tail samples.

Reviews and author responses:

This paper investigates long-tailed visual recognition from a memorization-generalization point of view.  A simple yet effective combinational sampling method is proposed to tackle long-tailed learning trade-off via overexposure of tail samples in the late stage of training. Specifically, switching instance-balanced sampler to class-reversed sampler for the last several epochs of training can help networks learn tail classes of low-regularity. Experiments show this method can reach sota performance more efficiently compared to baselines. 
 
Pros: 

+ This method is simple yet effective, and can be plugged into almost any existing methods to improve the robustness of low-regularity classes.

+ Compared to previous sota methods (BBN & Decouple) which adopted a 2-stage training strategy, this method is trained end-to-end.

+ This paper empirically identifies that low regularity of tail classes is the primary hurdle for long-tailed learning and can be alleviated by appropriate memorization mechanisms.

+ This paper provides complete experiments and ablation studies to demonstrate the effectiveness of their methods.

+ The overall writing is clear, and the method is easy to understand.
 
Cons: 

- Though effective according to the experiments, this method seems incremental and to some extent lacks novelty. The idea of feeding more tail-classes shares similar spirits with BBN, which uses class-reversed sampling to re-balance the feature extractor. Although authors of this paper may adopt a simple architecture design compared to previous methods.

- Regarding the experimental results, there is a 2% improvement on Long-tailed CIFAR-100 with imbalance ratio of 100 and 50, but considering other settings, especially long-tailed CIFAR-10, the gain is far from “unreasonable effective” as indicated by the author. Meanwhile, this new sampling-switching method may perform even or worse compared to baseline in some settings.

- This paper relates the idea of adopting CR sampling in later periods of training to recent research on memorization mechanisms, but the authors didn’t provide some theoretical explanations that can more densely link his proposal into memorization. A deeper analysis of the proposed method would have been nice, such as a theoretical lower bound of decision boundaries of tail classes after adopting the reverse class sampler compared to the instance sampler. In general, the authors need to provide more insights into this problem as well as his proposal. This method seems somewhat incremental now.

- Hyperparameters such as S seem to be sensitive.

Minor comments: 

* More comparisons against recent research on long-tailed problems could be added, as well as the references.

**Post-Rebuttal**

I appreciate the authors' efforts to make this submission much more solid. The investigation of the tail sample memorization phenomenon is insightful and would be beneficial to the community. However, the experimental results on standard benchmarks (e.g. ImageNet) are still not convincing enough. Thus, I will only increase my score to 5 (marginally below acceptance threshold). I definitely encourage the authors to further polish the manuscript and submit to future venues.



Summary:

This paper aims to address the problem of long-tail classification by revising the data resampling strategy. The main contribution is that it considers the representation trade-off between the head and tail classes. To this end, it combines instance-balanced sampling and class-reversed sampling, by first conducting training using instance-balanced sampler for several epochs and then switching to class-reversed sampler to continue training for the last several epochs. The approach is tested on three long-tail classification benchmarks, including CIFAR-10, CIFAR-100, and iNatualist 2018, and compared with state-of-the-art results.

Strengths:

1. The authors tackle an important and challenging problem of long-tail classification.
2. The proposed approach is simple and interesting.
3. Experimental evaluations demonstrate the effect by introducing the mixed strategy of instance-balanced and class-reversed sampling.

Weakness:
	
1. While the authors claimed that they challenged the hypothesis by Kang et al. that the learning of feature representation and classifier should be completely decoupled in long-tail classification, from my perspective this paper is a nature extension of Kang et al. Similar to Kang et al., this paper further demonstrates the importance of progressive learning to address long-tailed distribution, which first mainly focuses on head classes (first stage) and then on tail classes (second stage). Different from Kang et al., a) this paper uses more aggressive sampling strategy at the second stage, by replacing class-balanced sampler with class-reversed sampler, and b) this paper shows that the feature can be further fine-tuned at the second stage.

2. This paper mainly reported the overall performance. It would be more convincing to provide the per-class performance to show the improvement on the tail classes.

3. This paper mainly focused on the comparison with Kang et al. Since Kang et al. extensively conducted experiments on the long-tailed ImageNet and Places datasets, how does the proposed approach perform on these two datasets?

4. While the time to switch from instance-balanced sampling to class-reversed sampling is a hyper-parameter as the authors mentioned, I was wondering if there is any principle to guide this design choice. Also, if this is dataset/distribution dependent or sensitive.

5. Consider an even smoother transition, from instance-balanced sampling to class-balanced sampling and finally to class-reversed sampling. Will this combination outperform the strategy in the paper?

6. In the discussion section, the authors claimed that the near-optimality of their method and that little room left for the improvement of the successive resampling strategy. This seems a very strong argument. I was wondering if there is any formal theoretical guarantee on this.

Post Rebuttal:

I do appreciate the efforts and additional experiments and theoretical analysis that the authors made in the rebuttal. While this paper proposed an interesting approach to long-tail recognition, some connections, distinctions, and comparisons with related work and thorough experimental analysis were missing in the original manuscript, as mentioned by other reviewers as well. Some of these concerns were addressed in the rebuttal, but not fully clarified. For example, the new comparison on the more challenging ImageNet-LT dataset (Table 3) shows that the proposed approach does not outperform or is even worse than Decouple [Kang et al.] for the overall performance. Also, Table 5 shows the trade-off between head and tail categories. But similar trade-off has not been fully investigated for the baselines; for example, by changing the hyper-parameters in Decouple [Kang et al.], Decouple [Kang et al.] could also significantly improve the tail accuracy while slightly decreasing the head accuracy. This makes the paper not ready for this ICLR. I encourage the authors to continue this line of work for future submission.

Thank very much for your valuable comments and suggestions, we have made the following updates to address your concerns:
1.	We have provided the experiments on ImageNet-LT with different backbones. (see Table.3 and Table.5 in this version)

Please let us know if there is anything still not clear or any other suggestions.


Thank very much for your valuable comments and suggestions, we have made the following updates to address your concerns:
1.	We have provided the theoretical analysis of different sampling strategies and proved that class-reversed sampling is better than instance-balanced sampling during the last training process. (see Section 3.3 and Appendix A.1 in this version)
2.	We have provided fine-grained level results on CIFAR-100 and ImageNet-LT, including many-shot, medium-shot, few-shot and overall performance. (see Table.4 and Table.5 in this version)
3.	We have moved the ablation studies of value of S from appendix to the main context for better understanding. (see Table.6 in this version)

Please let us know if there is anything still not clear or any other suggestions.

Thank very much for your valuable comments and suggestions, we have made the following updates to address your concerns:
1.	We have provided fine-grained level results on CIFAR-100 and ImageNet-LT, including many-shot, medium-shot, few-shot and overall performance. (see Table.4 and Table.5 in this version)
2.	We have provided the experiments on ImageNet-LT with different backbones. (see Table.3 and Table.5 in this version)

Please let us know if there is anything still not clear or any other suggestions.

Thank very much for your valuable comments and suggestions, we have made the following updates to address your concerns:
1.	We have provided fine-grained level results on CIFAR-100 and ImageNet-LT, including many-shot, medium-shot, few-shot and overall performance. (see Table.4 and Table.5 in this version)
2.	We have provided the experiments on ImageNet-LT with different backbones. (see Table.3 and Table.5 in this version)
3.	We have moved the ablation studies of value of S and smoother transition from appendix to the main context for better understanding. (see Table.6 and Table.7 in this version)
4.	We have provided the theoretical analysis of different sampling strategies and proved that class-reversed sampling is better than instance-balanced sampling during the last training process. (see Section 3.3 and Appendix A.1 in this version)

Please let us know if there is anything still not clear or any other suggestions.


Thank all reviewers very much for the valuable comments and suggestions, we have made the following updates to address the reviewer’s concerns (all changes are presented in blue font in the latest manuscript):
1.	Provide the theoretical analysis of different sampling strategies and prove that class-reversed sampling is better than instance-balanced sampling during the last training process. (see Section 3.3 and Appendix A.1 in this version)
2.	Provide the experimental results on ImageNet-LT with different backbones. (see Table.3 in this version)
3.	Provide fine-grained level results on CIFAR-100 and ImageNet-LT, including many-shot, medium-shot, few-shot and overall performance. (see Table.4 and Table.5 in this version)
4.	Move the ablation studies of value of S and smoother transition from appendix to the main context. (see Table.6 and Table.7 in this version)
5.	Move the total error experiment from the main context to appendix. (see Table.13 in this version)
6.	Remove the “any generalization bound?” part of Section 5 due to we have provided the theoretical analysis.
7.	Improve some expression of our paper.

We have revised our paper according to all the valuable comments and please let us know if there is anything still not clear or any other suggestions.


We sincerely appreciate the reviewer for recognizing the contribution of our method and we’re delighted that most parts of the manuscript are valued, except several misunderstanding on the method detail as well as performance. Your concerns are addressed in the following point-by-point response:

1. From the perspective of both architecture and training process, our method differs from BBN a lot. BBN utilizes class-balanced sampling and class-reversed sampling simultaneously and fuses features in “mix-up” manner in every epoch. Conversely, our method only makes a slight change to the standard training process and utilizes only one sampling strategy in each stage, which is exactly not how the bilateral-branch network works. In addition, it is worth to note that with quite a simple setting, our method even able to achieve comparable or better performance within less training epochs (see Figure 1).
2. The meaning of “unreasonable effectiveness” is that using a quite simple switching method with fewer training epochs, we can achieve comparable or better performance than previous SOTA. It is worth to note that our method is overall better than Decoupling, while only 0.1% worse than BBN on the CIFAR-10-LT with imbalance ratio 100 (see Table 1). Considering that the training time of each epoch of our method is only 61.5% of that of BBN (we will release the efficiency analysis in the future version), we think the proposed method is meaningful.
3. Our method of memorizing tail class samples is motivated by the paper (Exploring the Memorization-Generalization Continuum in Deep Learning, CoRR abs/2002.03206), where the authors prove that network is able to learn low-regularity samples based on the strongest domain regularity learned first. Motivated by this, we identify the relations between regularity of training samples and long-tailed problem and find that regularity can be sharply decreased with the reduction of class cardinality  (see Appendix A.3). Therefore, we emphasize to learn the high-regularity head class samples first and memorize the low-regularity tail class samples during the small learning rate stage. We have done comprehensive experiments to show that our switching method is effective in solving the trade-off between head classes and tail classes (see Appendix A.4 and A.5).
4. Indeed, we have proved that the hyper-parameter of switching time is not dataset/distribution dependent or sensitive (see Table.7). It is interesting that the model could achieve performance gains as soon as there is a class-reversed sampling, no matter what the specific value of S is. Given that this part is very important for understanding the contribution of our method, we will move the relevant experiments from the appendix to the main context.
5. Thanks for the suggestions on the theoretical research. We are working closely to provide the generalization bound of the proposed method, as well as a numerical example for the change of theoretical generalization bounds after adopting the reverse class sampler compared to the instance balanced sampler. This theoretical analysis would be updated in the revision of the manuscript.
6. In the manuscript we’ve compared the proposed method with some important pieces of recent work like BBN and Decouple. In the revision we would like to better present the empirical evidences at a fine-grained level, e.g. provide Many-shot, Medium-shot, Few-shot, and overall performance on ImageNet-LT, as advised by AnonReviewer4. Moreover, we’ll make sure that all the references have been properly listed.


We sincerely appreciate the reviewer for recognizing the contribution of our method. Moreover, we would like thank for your constructive thinking towards improving our manuscript. However, we hold views that our method differs from Decouple a lot in terms of the training process, the spirit as well as the motivation.

We would like to address your concerns in the following point-by-point response:
1. From the perspective of training process, our method differs from Decouple a lot. As mentioned in the manuscript, our method actually violates the spirits of the Decouple, though owning a two-stage training process which looks similar to the Decouple at the first glance. In Decouple, feature extractor is trained in a common way in the first stage. Afterwards, only the classifier is re-trained with class-balanced sampling. However, in our method, the feature extractor and classifier are trained together in both stages. At the first stage when the learning rate is large, instance balanced sampling is used; while class-reversed sampling is utilized at the second stage when learning rate is small.
Furthermore, our motivation is significant different from Decouple. They just find that the learning of feature extractor and classifier should be different, and then design strategy to achieve decoupling training, without explaining why we should decouple the training procedure. However, we first empirically identify the low-regularity of tail classes during long-tailed distributions (see Appendix A.3) and indicate that the reason why the training procedure should be different is that the learning of low-regularity tail class samples will disrupt the learning of high-regularity head class samples. In order to memorize tail samples while maintain the representative ability of head samples, we propose our switching training strategy.
2. Thanks for your constructive advice. In the revision of the manuscript, we would like to better present the empirical evidences at a fine-grained level, e.g. provide Many-shot, Medium-shot, Few-shot, and overall performance on ImageNet-LT, as advised by AnonReviewer4, too.
3. Due to the limited time and resources, we did not manage to conduct experiments on benchmarks like ImageNet-LT. Now we are working closely to provide more convincing evidences and analysis in the next version before the end of the stage 2 discussion.
4. Indeed, we have proved that the hyper-parameter of switching time is not dataset/distribution dependent or sensitive (see Table.7). It is interesting that the model could achieve performance gains as soon as there is a class-reversed sampling, no matter what the specific value of S is. Given that this part is very important for understanding the contribution of our method, we will move the relevant experiments from the appendix to the main context.
5. Indeed, we have proved that the smoother transition switching is comparable or worse compared with our switching method (see Table.8). Given that this part is very important for understanding the contribution of our method, we will move the relevant experiments from the appendix to the main context.
6. Thanks for the suggestions on the theoretical research. We are working closely to provide the generalization bound of the proposed method, as well as a numerical example for the change of theoretical generalization bounds after adopting the reverse class sampler compared to the instance balanced sampler. This theoretical analysis would be updated in the revision of the manuscript.


We sincerely appreciate the reviewer for recognizing the contribution of our method. We hold views that our method differs from Decouple a lot in terms of the training process, even the spirits of these two methods are contradictory. Thus the novelty of the proposed method could not be overlooked.

We would like to address your concerns in the following point-by-point response:
1. Our method is indeed very simple, but it does not mean no novelty. From the perspective of training process, our method differs from Decouple a lot. As mentioned in the manuscript, our method actually violates the spirits of the Decouple, though owning a two-stage training process which looks similar to the Decouple at the first glance. In Decouple, feature extractor is trained in a common way in the first stage. Afterwards, only the classifier is re-trained with class-balanced sampling. However, in our method, the feature extractor and classifier are trained together in both stages. At the first stage when the learning rate is large, instance-balanced sampling is used; while class-reversed sampling is utilized at the second stage when learning rate is small.
1.Due to the limited time and resources, we did not manage to conduct experiments on large-scale benchmarks like ImageNet-LT as well as stronger networks like. Now we are working closely to provide more convincing evidences and analysis in the next version before the end of the stage 2 discussion, including following issues with your concerns addressed:
(1) Provide Many-shot, Medium-shot, Few-shot, and overall performance on ImageNet-LT, which we’re working towards.
(2) Provide empirical evidences utilizing ResNext as the backbone architecture if we still got time.


Thanks again for your constructive advices. Due to the limited time and resources, we did not manage to conduct experiments on benchmarks like ImageNet-LT. Now we are working closely to provide more convincing evidences and analysis in the next version before the end of the stage 2 discussion, including following issues with your concerns addressed:
1. Provide Many-shot, Medium-shot, Few-shot, and overall performance on Long-tailed CIFAR-100, which we have already recorded in previous experiments.
2. Provide Many-shot, Medium-shot, Few-shot, and overall performance on ImageNet-LT, which we’re working towards.

Thank you for explaining the differences between your method and BBN. I apologize for my misunderstanding of your implementations, but as I mentioned in my Comments to your Clarification, I understand that you did lots of analysis on the trade-off between head and tail, but directly showing the head & tail accuracies is still very important. My suggestion is that since you are working on the trade-off between head and tail, you should either report ImageNet-LT using Many-shot, Medium-shot, Few-shot, and overall settings, or you can conduct similar experiments on your Long-Tailed CIFAR datasets. Otherwise, I don't think your experiments can support your ideas.

We sincerely appreciate the reviewer for recognizing the contribution of our method. However, we disagree with the posted negative judgement. Instead, we hold the views that our method differs from BBN a lot. Given the facts that the previous methods, like BBN or Decoupling, have **not** thoroughly discussed the trade-off between instance-balanced sampler and class-reversed sampler, we have had chances to find this simple yet effective method, which may have potential influence on both theoretical study and practical application for successive researchers.

We would like to answer your concerns in the following point-by-point response:
1. From the perspective of both architecture and training process, our method differs from BBN a lot. BBN utilizes instance-balanced sampling and class-reversed sampling simultaneously and fuses features in “mix-up” manner **in every epoch**. Conversely, our method only makes a slight change to the standard training process and utilizes only one sampling strategy in each stage, which is exactly **not** how the bilateral-branch network works.
2. From our points of view, however, it seems the discussions in BBN and Decoupling left much space for us. If these two methods have already thoroughly discussed the stated trade-off, why don't they adopt this more elegant and efficient way to solve the long-tailed recognition problem? 
In our paper, we discuss the relationships between sample class regularity and long-tailed learning to better handle this trade-off. For example, our empirical study indicates that regularity can be sharply decreased with the reduction of class cardinality, which is not covered within BBN and Decoupling. Moreover, with such an extremely simple method achieved impressive performance, we believe there is still rooms left for later researchers not only explore the potential theoretical explanations for long-tailed learning, but also to utilize our method widely in practical long-tailed recognition applications with notably lower costs.
3. Thanks for the writing advice, we will improve it in the next edition.

My previous comments might not be all correct, but you still didn't convince me in your response. Since you mentioned that your paper aims to deal with the trade-off between learning of high-regularity head classes and low-regularity tail classes, why you didn't report your experimental results in the form of how the Decouple reported ImageNet-LT, i.e., reporting Many-shot, Medium-shot, Few-shot, and overall separately？so we can better understand how your method improves the trade-off. Indeed, since all the previous methods using Long-Tailed CIFAR only reported overall accuracies under different Imbalance ratios, you may just follow their settings, but you can still use ImageNet-LT, right?

I understand that you did lots of analysis on how the classes perform under different frequencies, but directly showing the head & tail accuracies is still very important. My suggestion is that since you are working on the trade-off between head and tail, you should either report ImageNet-LT using Many-shot, Medium-shot, Few-shot, and overall settings, or you can conduct similar experiments on your Long-Tailed CIFAR datasets. Otherwise, I don't think your experiments can support your motivation. A good example of studying the trade-off between head and tail would be [1], but I didn't mean to compare your method with this paper, because you are concurrent work. I just think their experiments and analysis are more convincing with respect to the story of solving the trade-off between head and tail.

[1] https://arxiv.org/pdf/2010.01809.pdf

We appreciate the time and the effort of all the reviewers. We note that all reviewers have raised questions about our differences between our method and other methods, therefore we have prioritized clarifying them with following notions:

1. ```The motivation of our method is to memorize low-regularity tail class samples without seriously impacting the high-regularity head class representation, which is significantly different from existing methods. We first empirically identify the low-regularity of tail classes during long-tailed distributions (see Appendix A.3) and find that regularity of the same training samples can be sharply decreased with the reduction of class cardinality. In addition, inappropriate learning low-regularity samples may hurt the learned representation ability of high-regularity samples (see Exploring the Memorization-Generalization Continuum in Deep Learning, CoRR abs/2002.03206). Therefore, how to deal with the trade-off between learning of high-regularity head classes and low-regularity tail classes is indeed the primary hurdle for long-tailed learning, which is not within the discussion of Decouple and BBN. 
2. Our method is quite different from BBN. **Reviewer 4** mentioned “this paper is just a special case and a simple version of BBN”. However, this judgment is reckless and opinionated, without any supporting evidences. Our method achieves ideal performance (see Table 1 and Table 3) by only making a slight change to the standard training process, which is easy to follow and to be plugged into other existing methods. Conversely, BBN designs a more complicated bilateral-branch architecture. During its training, two samplers simultaneously sample data from different distributions, extracting features respectively and fusing the two features and losses like “mix-up” method, which is of higher computational and time complexity (see Figure 1). In our experiments, these complicated settings seem to be redundant.
3. Our method provides a new perspective to treat the long-tailed problem. **Reviewer 4** mentioned “I don’t think this paper provides any new ideas for the later researchers”, which is inappropriate. The most valuable contribution is that we can treat the long-tailed problem as dealing with the trade-off between high-regularity head classes and low-regularity tail classes and we solve it through a memorization-generalization perspective. The idea of memorizing low-regularity samples and the strategy of learning low-regularity tail class samples during small learning rate stage, which have been proved to be effective, can throw some light on the long-tailed recognition research field.  

To sum up, the motivation and solution of our method are different from existing methods. Being aware of that such a simple method achieves such impressive performance, the empirical study on the mechanism behind becomes particularly important.


##########################################################################

Summary:

In short, the main contribution of this paper is to jointly combine the instance-balanced sampler and the class-reversed sampling in the training, where the latter is only applied to the last few epochs. It also provides a viewpoint of the long-tailed problems from the memorization-generalization mechanism.

##########################################################################

Pros:

+ The proposed method is indeed very simple.

##########################################################################

Cons:

- From my point of view, this paper is just a special case and a simple version of BBN [BBN: bilateral-branch network with cumulative learning for long-tailed visual recognition, CVPR, 2020]. The main contribution of this method is jointly combining the instance-balanced sampler and the class-reversed sampling in training, which is exactly how the bilateral-branch network works.
- Since BBN and Decoupling have already thoroughly discussed the trade-off between instance-balanced sampler and class-reversed sampling, I don't think this paper provides any new ideas for the later researchers.


##########################################################################

Questions during the rebuttal period:

One piece of advice: try to make the introduction more concise in your future work. The exact same sentence "We only switch from instance-balanced sampler to class-reversed sampler for the last several epochs of training." shows three times in the introduction, which makes the audience question the limited contribution of this paper.

##########################################################################

Reasons for scores:

Since the novelty is limited, and the points made by this paper have already been well discussed by previous work BBN CVPR 2020 and Decoupling ICLR 2020, I decide to reject this submission.

##########################################################################

This paper proposes that decoupling feature representation and classification is not the optimal solution for the long-tailed recognition problem. They propose to train the model with instance-balanced sampling first, and switch to reverse sampling at the last epochs with a small learning rate, so that the tail classes could be better represented than decoupling approaches (Kang et al. 2020) while the whole framework is much simpler than BBN (Zhou et al. 2020). The results are promising on CIFAR and iNaturalist.

#### Pros:
1. This paper is well written and easy to follow. It presents the ideas clearly.
2. The proposed approach is simple and effective on CIFAR and iNaturalist, and the ablation study and analysis is very comprehensive.
3. It brings some insights into the long-tail problem.

#### I have some concerns:
1. The novelty is not enough for me. This paper seems to be a soft version of the decoupled approach (Kang et al. 2020), by combining the reverse sampling strategy proposed by BBN( Zhou et al. (2020)). Since it first trains the network normally, and then switch the sampling strategy and trains the whole network with a small learning rate, actually, the feature representation part is almost done in the first instance-balanced sampling stage, and is just fine-tuned a little with reverse sampling.
2. About the experiments, 
(1) I wonder why the authors did not conduct experiments on ImageNet-LT, which is considered to be one of the most challenging datasets. It will be more convinced if they show that the performance boosts on ImageNet-LT.
(2) All the experiments are based on ResNet-50. Will we still get improvement on stronger networks? Or the improvement will be absorbed by the larger backbone? Please let me know if I take it in the wrong way.


Meta-review: Reviewers were concerned about the technical novelty because the two-stage sampling strategy is similar to BBN and the decoupling of features and classifiers. Rebuttal addressed some concerns about the experiments, but Reviewers' major concerns remained. 

Decision: withdrawn-rejected-submissions