Training A Multi-stage Deep Classifier with Feedback Signals
Authors: Chao Xu
Yu Yang
Rongzhao Wang
Guan Wang
Bojia Lin
Venue: Submitted to ICLR 2023 (2023)
Keywords: multi-stage classification
training framework

Abstract: Multi-Stage Classifier (MSC) - several classifiers working sequentially in an arranged order and classification decision is partially made at each step - is widely used in industrial applications for various resource limitation reasons. The classifiers of a multi-stage process are usually Neural Network (NN) models trained independently or in their inference order without considering the signals from the latter stages. Aimed at two-stage binary classification process, the most common type of MSC, we propose a novel training framework, named Feedback Training. The classifiers are trained in an order reverse to their actual working order, and the classifier at the later stage is used to guide the training of initial-stage classifier via a sample weighting method. We experimentally show the efficacy of our proposed approach, and its great superiority under the scenario of few-shot training. 

Under review as a conference paper at ICLR 2023
TRAINING A MULTI-STAGE DEEP CLASSIFIER WITH FEEDBACK SIGNALS
Anonymous authors Paper under double-blind review
ABSTRACT
Multi-Stage Classifier (MSC)- several classifiers work sequentially in an arranged order and classification decision partially made at each step is widely used in industrial applications for various resource limitation reasons. The classifiers of multi-stage process are usually Neural Network (NN) models trained indepen- dently or in their inference order without considering the signals from the latter stages. Aimed at two-stage binary classification process, the most common type of MSC, we propose novel training framework, named Feedback Training. The classifiers are trained in an order reverse to their actual working order, and the classifier at the later stage is used to guide the training of initial-stage classifier via sample weighting method. We experimentally show the efficacy of our pro- posed approach, and its great superiority under the scenario of few-shot training.
INTRODUCTION
The -tate-of-the-art deep neural networks have equipped a various of applications with much better quality, especially the emergence of Ber[Devlin al.] (2018), a Transformer Vaswani al. (2017)-
based pre-training language model, and series of its derivatives [rronn et al. (2020): : Lan et al. (2019). Their great success is mainly due its scalability to encode large-scale data and to maneuver billions of model parameters. However, it is rather difficult to deploy them to real-time products such as Fraud Detection Senator et al. (1995); Kirkland et al. (1999), Search and Recommendation systems Covington et al. (2016); Ren et al. (2021). and many mobile applications, not only because of the high computational complexity but also the large memory requirements.
Several techniques developed to make the trade-off between performance and model scale. Knowledge Distillation (KD) Hinton et al. (2015); Sanh et al.] (2019) is the most empirically suc- cessful approach used to transfer the knowledge learnt from heavy Teacher to more light-weight and faster Student. Besides, Pruning Han et al.] (015); Frannkle & Carbin (2018) and Quantization Han et al. (2015); Chen et al., (2020) further compress deep models even smaller. In many prac- tical situations, however, we need super tiny models to meet the demanding memory and latency requirements, which would inevitably suffer serious performance degradation.
From another perspective, multi- classification system Trapeznikov et al. (2012) is widely used to reduce the opportunity of calling the deep and cumbersome models by filtering out some or even most of the input samples using simpler and faster models trained with limited data and easier features. In multi-stage system, light- weight models such as SVM, Logistic Regression or k- Nearest Neighbors are used as earlier stage classifiers, classifying the samples (usually relatively easier negative ones) based on simple or easily accessible features, and leaving indeterminate ones for later. Models of later stages need to be heavier to deal with harder samples as well as more complex and costly features. A two-stage working mechanism is simply shown Figure a).
In several practical multi-stage applications, shown in Figure bb) and (c), classifiers in different stages are trained independently or sequentially without considering the relationships among them Isler et al. (2019): Kruthika et al. (2019).
To build tighter connections between classifiers in a multi-stage system for better collaboration, most exist methods Mendes et al. (2020): Qi et al. (2019); Sabokrou al. (2017); (Zng et al. (2013) jointly optimize the multi-staee classifiers in a way like cascade, allowing the contextual information to transfer from earlier stages later. However, most of them primarily consider classification

---

Under review as a conference paper at ICLR 2023
accuracy rather than latency, and therefore would not make the classification decisions until the final stage.
In this paper, we consider to further explore how to forge closer connections between classifiers in a two-stage classification problem. We propose a novel training framework, Feedback Training, where the whole decision-making pipeline is consisted of two classifiers, extremely lightweight Pre-classifier followed by relatively heavier Main-classifier. Different from existing methods, these two models are trained in the reverse order of inference, that is, the first-stage model would be trained under the guidance of the second-stage one through a sample weighting method. The capacity of Pre-classifier is more effectively explored by considering the learning results of Main-classifees
Our contributions can be summarized threefold:
1) We propose novel training framework for two-stage classification applications.
2) We discuss a sample weighting method that assists Pre-classifier to learn according to its prefer- ence.
3) We verify our approach on two data sets that it outperforms baseline models significantly and shows greater superiority under few-shot scenarios.
Training Samples
Pre-classifier
X
Pre-classifier
Pre-classifeer
Pre-classifier
Main classifier
Main-classifier
Main-classifier
Main-classifier
(8) Working Process
(b) Independent Training
(c) Sequential Training
(d) Feedback Training
Figure 1: Working process and different training strategies for two-stage classifiers. We the terms Pre-classifier and Main-classifier to denote the classifiers working at 1st and 2nd stages. (a) The working process of two-stage classifier: only samples passed Pre-classifier would be fed into heavier Mlan-classifier for final decision. otherwise would be judged as negative without calling Main-classifier. (b) Independent Training: all classifiers are trained independently without con- sidering the training results of each other. (c) Sequential Training: Classifiers are trained in their working order. Only samples passed Pre-classifier would fed into Main-classifier for training. (d) Feedback Training: Classifiers are trained in their reverse order of inference. Pre-classifier assigns different attention to different samples based on the training result of Main-classifier and the pro- posed sample weighting approach.
PRELIMINARIES
We consider binary classification problem with the training set D
(Ins Fn) vn)),
where x; denotes tth observed training sample paired with its label y/s E (0, 1}. In a m-stage process, ...,....,,) working in the given order. The jth predictive function is parameterized by 0j- The classification decision partially made in each step. One popular design is to filter out negative samples as many possible in earlier stages and leaving the positive ones to the end for final decision. When classifiers are trained independently without considering the others, each one is trained by optimizing the basic Cross-Entrope loss in [
there are series of predictive functions, F [fe, (-)l-
C(fe, (-)) = - Σ
Lee (yt, fo, (xx))
(1)
1=1
where Lee (ce(v.0) y logg (1 v) -ogg( 9)
(2)
2

---

Under review as a conference paper at ICLR 2023
We also define set of thresholds T {thyli 1.2,... m. for each classifier. During infer- ence, samples whose predictive scores are lower than corresponding thresholds would be judged as negative, and would not be sent to the next stage.
In sequential training, only the samples that pass all the previous classifiers are fed into the next stage for training. We formally define an indicator function I = 1 1[Ve<e-Score" > thy} to denote whether I; would be used to train the jth classifier. Score is the predictive score h.th classifier for r;. Hence the total samples used to train fo, (-) is n Em= D. Then we optimize Eq
((f = R2j
Σ H Ce fo, (x.))
(3)
3 FEEDBACK LEARNING
3.1 BACKGROUND AND MOTIVATION
Classification problems are becoming casier, achieving high performance simply by fine-tuning those pre-made large-scale language models with enough high quality observed samples. However, serving those heavy models practical applications is quite challenging due to their demanding requirements for memory and latency. Therefore, design muti-stage classifier for faster infer- ence.
Aimed binary classification problems, decision system consists of two networks combined, as demonstrated in Figure [a). Pre-classifier a light-weight Logistic Regression model running on all input and tries its best identify positive samples and meanwhile filter tive ones as possible. The Main-classifier is Transformer-bass heavier network. Samples passed first-stage are potentially positive ones and would be fed into Main-classifier for final judgement. Similar workflows are widely used practically. One common learning strategy is to straightforwardly train the Main clans-classifier only with samples passed Pre-classifier as shown Figure However, this method arises a dual funnel Mendes et al. (2020) issue where a great number of samples are filtered out by the initial stage and the Main-classifier trained on this bias data set would lose its generalization ability.
many nega-
Figure 2: Scores predicted by Main-classific and Pre-classifier.
If classifiers are trained independently (see Figure [b)), there would be an inconsistency problem shown in Figure 2 Some positive cases passed Pre-classifier may be rejected by Main-classifier (useless recalls of cre-ciassif,er), but some other positive cases rejected by Pre-classifier while they highly scored by Main-classife preference these two kinds of cases, the overall performance would be improved. The same is true for negative samples. Therefore, modeling this preference is undoubtedly useful for the overall
(potentially valuable recalls). If Pre-classifie rotates the scoring
3

---

Under review as a conference paper at ICLR 2023
performance, without improving the accuracy of the classifiers. To this end, propose the Feedback Training, a reversed order training framework, as described in the next section.
3.2 FEEDBACK TRAINING FRAMEWORK
Main-classifier In order to maintain the generalization performance of the Main-classifier, we train independently the first place on full training dataset with Cross-Entropy loss as shown in Eq
Pre-classifier To better collaborate with the Main-classifier a combined pipeline with its limited capacity, the Pre-classifier is trained with the following learning preferences:
Pass positive samples and pay more attention to the samples could be identified by Main- classifier.
Filter out negative samples and those could not be identified by Main-classifier (whether they are positive or negative).
Modeling the bbove-mentionee learning preferences of Pre-classifier is not easy it doesn't 't have a formal mathematical definition, but Machine Learning is essentially an optimization process that requires definite objective. To address this problem, sample weighted loss function as 4
Clu,, (x)) - w(ss) Lee (2:))
(4)
where w(s;) is the sample weight of x; based on the prediction Scoremaii (s; for simplification in the formula) of Main-classifier
3.3 WEIGHTED SAMPLING
To embody the ahove-mentioned learning preferences, following points are considered for the design of our sample weighting approach:
For Positive samples, Pre-classifier expected to focus more on the samples with high Score for the reason that samples with low Scoremain supposed to be rejected by Main-Classifier even they pass the Pre-classifier.
For negative samples, it's more important to reject the ones with high Score main as it might be accepted by Main-classifier.
If one sample's s main is hugely high, it should weighted higher, but not unlimited high.
The weights for negative samples should have lower bound case too many of them are weighted near 0 as most of them are supposed to be scored very low by Main-classifier
These principles be formulated as in Eq 5 and curved in Figure
w(s) minn(a(s - 0.5, Epos, (pos), Wmax)
=
(5)
minnWWngg.man min + a(s 0.5, bnngs (neg); Wmar)
0
02,4,a) + perp(-7
(6)
where t is the temperature parameter, controlling the steep degree of the weighting function, Wmoz and Wneg_min denote the upper bound of all samples and the lower bound of negative samples. Besides, the attention intensity, controlling the theoretic fluctuation range of weighting method. We force the model to pay more attention to the positive samples by setting apos = 2 and aneg =

---

Under review as a conference paper at ICLR 2023
1-Hhgg
weiggtt negeltiv samples
weight possiive semples
score Main classifier
score of classifee
Figure 3: Sample weighting curves for negative and positive samples.
EXPERIMENTS AND RESULTS
4.1 EXPERIMENTAL SETUP
In this section, we describe the settings of our experiments.
Datasets
The proposed approach is empirically verified on two data sets.
1) Task Extraction to judge whether one input sentence a Commitment you made or Request from others. We have training samples in 4 languages (PT, IT, FR, DE), each consists of 50K sentences machine translated from English. Evaluation samples are organic sentences from Enron emails Klimt & Yang (2004) and bad cases from user feedback, both annotated manually, 20K per language.
2) Yahoo! Answers Xiang Zhang & Lecun (2015) topic classification dataset has 10 main cate- gories, each class contains 140K training samples and 6K testing samples. To take use of them in binary classification scenario, we only consider class "Society & Culture" positive while all others as negative. Only the "Answer" column is retained as the input of models. Besides, samples less than 5 tokens are filtered out. There 1.46M (146K X 10) samples in total and they are split into 8:1:1 1 for training, validation and evaluation.
Experiment Setup
Baselines: we consider the Independent and Sequential Training as our baseline. Both frameworks share the same model architectures and settings with Feedback Training.
Generally, the direct output of a binary classification model is a score, and we choose threshold to decide whether it is positive or negative. In our two-stage Classifier, We define the term PassRate to denote the proportion of test samples passing through the Pre-classifier. For the sake of fairness, the PassRate for Pre-classifiers Baseline and Treatments are set to be the same. In other words, they call the same times of Main-classifier and consumed the same computation resource.
Comparison Method:
Implementation
Pre-classifier: a Logistic Regression model trained by scikit-leamn0.24.11 1), using 30k token and Bi- gram features. The 30k features are selected using chi2-score from all tokens and Bi-gram features. Besides, balaneed class weighting method King & Zeng (200T) adopted to mitigate unbalanced distribution of class labels.
Main-classiffie: 12-layer Transformer-based model plus a softmax classification head, which has been pre-trained based InfoXLM Chi et al. (2021), XLM-Roberta Conneau et al. (2020) equivalent mulli-lingune models. The fine-tuning is conducted by minimizing Cross-Entropy loss with Adam Kingma & Ba (2014) (Ir 0.00005 and batch-size 64). It is multi-lingual model which shared by all languages when multiple languages are involved. The threshold of Main- classifier is 0.5.
5

---

Under review as a conference paper at ICLR 2023
-0.62
-0.64
-0.66
10
12
-0.64
-0.65
10
12
iterations
Figure 4: The optimization process of Feedback Training using Genetic Algorithm on Task Extrac- tion Italian dataset.
Lang.
Ryre
P Pmoón
Rmain
FLe2e
0.6720
0.6988
0.8909
0.6449
DE
0.6720
0.7010
0.9004
0.6495
0.6943
0.7214
0.9266
0.6801
0.5946
0.5830
0.8977
0.5573
FR
0.5946
0.5681
0.8920
0.5292
0.6047
0.5962
0.8994
0,5789
0.6154
0.6538
0.9296
0.6102
IT
0.6154
0.6609
0.8984
0.6021
0.6971
0.6585
0.9310
0.6537
0.7168
0.6976
0.9259
0.6802
PT
0.7168
0.6958
0.9320
0.6817
0.7301
0.7218
0.9272
0.7143
Table 1: Results Task Extraction for different languages. In each group, the three lines denote the results of Independent, Sequential and Feedback Training. P and denote precision and call. Metrics footnoted with 'pre", "main" and 'e2e" denote the performance of Pre-classifier, Main- classifier and the whole workflow.
Genetic Algorithm (GA): As the numerical optimization method is highly dependent on initializa- tion and the underivable objective function is easy be caught in abnormal solution. The (pos 'negg Wneg_män- Wmar are searched by Genetic Algorithm (using scikit-opt library) with the objective to maximize the Fleee on validation set. Figure shows example of this optimization process for language Italian on Task Extraction dataset.
Evaluation Metrics
We compare the performance of the multi-stage Classifiers in terms of F1, the geometric average of Precision (P) and Recall (R). The precision of the whole pipeline equals that of the Main-classifier in the final stage while the pipeline recall equals the production of that of all classifiers.
4.2 RESULTS AND ANALYSIS
Task Extraction We summarize the experimental results on the test set in 4 languages in Table T comparing the proposed Feedback Training framework with Independent and Sequential Training. For each language, the PassRate of Pre-classifier is the same for the control and treatment models. With Feedback Training, the e2e performances are consistently better than these of baseline in all languages, demonstrating the proposed training strategy allows the whole pipeline to collaborate better than other training strategies. Sequential Training performs Training for the reason that most of training samples are filtered out by Pre-classifier and the Main- model tends to overfit over the small number of the survived samples.
worse than Independent

---

Under review as a conference paper at ICLR 2023
PassRate
Rpre
P imoin
Rmain
Fle2e
20%
0.6161
0.7059
0.5159
0,4383
0.6194
0.7244
0.5477
0.4462
30%
0.7242
0.7009
0.4487
0.4440
0.7393
0.7167
0.4543
0.4574
40%
0,8034
0.6987
0.4095
0.4473
0.7914
0.6923
0.4210
0.4499
50%
0,8706
0.6952
0.3823
0.4502
0.8857
0.6932
0.3825
0.4551
Table 2: Experiments on Yahoo! Answer with different PassRates of Pre-classifier. In each group, the two lines denote the results Independent and Feedback Training. The same goes for all the tables below.
DataPortion
P Pare
Ryre
Parain
RMain
Ruze
Fle2e
1%
0.1604
0.4797
0.7128
0.4549
0,2182
0.3342
0.1666
0.6416
0.7174
0.4742
0.3042
0.4272 (+27.84%)
2%
0.1731
0.5158
0.7355
0.4742
0.2446
0,3671
0.1771
0.6668
0.7117
0.4856
0.3237
0.4450 (+21.23%)
5%
0,1904
0.5675
0.7210
0.4686
0.2661
0,3886
0.2244
0.6686
0.7075
0.4693
0.3138
0.4348 (+11.8999)
10%
0.2017
0.6011
0.7191
0.4607
0.2861
0.4094
0.2088
0.7179
0.7019
0.4665
0.3349
0.4535 (+10.77%)
Table 3: Results of using different proportion of data train Pre-classifer Yahoo Answer dataset. The Pre-classifiers are trained with 1% to 10% training data while Main-classifiers is trained with full training set. PassRate of Pre-classifier 30%.
Yahoo! Answer We further compare the proposed approach with Independent Training on this open-source dataset. Table sshoss Feedback Training outperforms baseline in case of all PassRate of Pre-classifier. More notably, the smaller the PassRate, the greater the advantage. Examining the experiments with PassRate equals 20% 40%, although the recall of Pre-classifier remains changed under Feedback framework, the Main-classifier gets significant improvement. This demon- strates that Feedback method indeed learns to pass more samples that the Main-classific can better deal with.
Experimental results under fewshot scenario are summarized in Table 3 Pre-classifiers are trained with 1% to 10% training data while Mlin-classifiers are trained with full training set. The Feedback Training shows superiority over the baseline, especially when less than 5% data used to train Pre-classifier. The e2e gain of Feedback Training mainly derives from the much higher recall of Pre-classifier. Instead of focusing on the classification accuracy, the Pre-classifier is forced pay more attention to recall more positive samples and meanwhile to avoid opposite forces from some hard negative samples that even can not be distinguished by Mais-classifier.
DataPortion
Ppre
Rore
PMain
RMain
Rz20
Fle2e
5%
0.1904
0.5676
0.6676
0.4990
0.2832
0.3977
0.2007
0.7032
0.6560
0.5065
0.3562
0.4617 (+16.10%)
10%
0.2017
0.6011
0.6721
0.4854
0.2918
0.4069
0.2110
0.7148
0.6574
0.5011
0.3582
0.4637 (+13.96%)
25%
0.2191
0,6530
0.7290
0.4476
0,2923
0.4173
0.2162
0.7352
0.7175
0.4433
0.3259
0.4482 (+7.41%)
50%
0.2173
0.7180
0.7627
0.3984
0.2861
0.4161
0.2165
0.7348
0.760
0.3959
0.2909
0.4208 (+ 113%))
Table Results of using different proportion of data to train both Pre-classifer and Main-classifier on Yahoo Answer dataset.
More thorough fewshot experiments are conducted by using different proportion of data to train both Pre-classifer and Main-classifier, the results are shown in Table 4 The proposed approach still

---

Under review as a conference paper at ICLR 2023
shows its advantages for the following reasons: (1) Main-classifier trained with fewer samples tends to score higher to the test data (experimentally, as the imput samples for Main-classffier are those passed Pre-classifier, and the Main-classifier here is worse than the one trained with full dataset. Hence can not detect negative samples well and tend to give higher scores overall), thus acquires higher recall; (2) As the Main-classifier has higher recall, and the target of GA is maximize the Fle2e, therefore the learned Pre-classifier also prefers to acquire higher recall with the sacrifice of precision more intensively. With the combined effects of these two reasons, the power of Feedback Training is amplified. This result indicates the great value of Feedback Training when the training data scarce.
5
CONCLUSIONS
This paper proposes novel learning framework, Feedback Training. to improve the end-to-end performance of two-stage Classifier, the most common type MSC. Experiments on both Task Extraction and Yahoo! Answer demonstrate the superior performance of the proposed approach and it becomes more advantageous when it comes to few-shot training scenario, reducing the require- ments the number of training samples for Pre-classifier in great extent when Main-classifier is given. The proposed Feedback Training method works only on two-stage binary claaf-classification sys- tem, further work should be carried out extend this approach to multi-stage multi- cultiiclassification scenario.
REFERENCES
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 13:1877-1901,209
Ting Chen, Lala Li, and Yizhou Sun. Differentiable product quantization for end-to-end embedding compression. In International Conference on Machine Learning, pp. 1617-1626. PMLR, 2020.
Zewen Chi, Li Dong, Furu Wei, Nan Yang, Saksham Singhal, Wenhui Wang. Xia Song, Xian-Ling Mao, Heyan Huang, and Ming Zhou. InfoXLM: An information-thhoreee framework for cross- lingual language model pre-training. In Proceedings of the 2021 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 3576-3588, Online, June 2021, Association for Computational Linguistics. doi: 10. 18653/vll 2all-naccl-min.220 URL httpt:ttts... : /acl http:///aaaaaaaaaaaecceec
anthologee org 2001 naacl - ma i n. 280
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. Un- supervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 8440-8451, Online, July 2020. Association for Computational Linguistics. doi: 10. 2011865//1/2220.ad// main. 747. UtLhttps: //aclanthology. wwt.:y..or//// wwtp://...ooooooooooooooo ac1-malnlm 747
Paul Covington, Jay Adams, and Emre Sargin. Deep neural networks for youtube recommendations.
In Proceedings of the 10th ACM conference on recommender systems, pp. 191-198, 2016.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint / 1810.04805, 2018.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprint arXiv: /803.03300 2018.
Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint arXiv: 15000:151 2015.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint arXiv: / 00..50533...2333 2015.
8

---

Under review as a conference paper at ICLR 2023
Yalein Isler, Ali Narin, Mahmut Ozer, and Matjaž Pere. stat classification of congestive heart failure based on short-term heart rate variability. Chaos, Solitons & Fractals, 118:145-151, 151, 2019.
Gary King and Langche Zeng. Logistic regression in rare events data. Political Analysis, 9:137-163, Spring 2001.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. International Conference on Learning Representations, 12 2014.
J Dale Kirkland, Ted Senator, James Hayden, Tomasz Dybala, Henry G Goldberg, and Ping Shyr. The nasd regulation advanced-detection system (ads). Al Magazine, 20(1):55-55, 1999.
Bryan Klimt and Yiming Yang. The enron corpus: new dataset for email classification research. In Machine Learning: ECML 2004, pp. 217-226, Berlin, Heidelberg, 2004. Springer Berlin Hei- delberg.
KR Kruthika, HD Maheshappa. Alzheimer's Disease Neuroimaging Initiative, et Multistage classifier-based approach for alzheimer's disease prediction and retrieval. Informatics in Medicine Unlocked, 14:34 42, 2019.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Sori- cut. Albert: A lite bert for supervidd learning of language representations. arXiv preprint arXiv: 1909. 1 1942, 2019.
Andre Mendes, Julian Togelius, and Leandro dos Santos Coelho. Multi-stage transfer learning with an application to selection process. arXiv preprint arXiv:2006.01277 2020.
Tangquan Qi, Yong Xu, and Haibin Ling. Tourism scene classification based on multi-stage transfer learning model. Neural computing and applications, 31(8):4341-435 2019.
Ruiyang Ren, Yingqi Qu, Jing Liu, Wayne Xin Zhao, Qiaoqiao She, Hua Wu, Haifeng Wang, and Ji-Rong Wen. Rocketqav2: A joint training method for dense passage retrieval and passage re- ranking. arXiv preprint arXiv:21 10.07367, 2021.
Mohammad Sabokrou, Mohsen Fayyaz, Mahmood Fathy, and Reinhard Klette. Deep-cascade: Cas- cading 3d deep neural networks for fast anomaly detection and localization in crowded scenes. IEEE Transactions on Image Processing, 26(4):1992-200 2004, 2017.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108 2019.
Ted E Senator, Henry G Goldberg, Jerry Wooton, Matthew A Cottini, AF Umar Khan, Christina D Klinger, Winston M Llamas, Michael P Marrone, Raphael WH Wong, et al. The fincen artificial intelligence system: Identifying potential money laundering from reports of large cash transac- tions. In IAAL, pp. 156-170, 1995.
Kirill Trapeznikov, Venkatesh Saligrama, and David Castañón. Multi-stage classifier design. In Asian conference on machine learning, pp. 459-474. PMLR, 2012.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 6008, 2017.
Junbo Zhao Xiang Zhang and Yann Lecun. Character-level convolutional networks for text classifi- cation. Advances in Neural Information Processing Systems, 6495 2015.
Xingyu Zeng, Wanli Ouyang, and Xiaogang Wang. Multi-stage contextual deep learning for pedes- trian detection. In Proceedings of the IEEE International Conference on Computer Vision, pp- 121-128, 2013.


Reviews:
[{"review_id": "Cw2hXfBMOs", "reviewer": "Reviewer_ND4V", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a novel training framework, named Feedback Training. The classifiers are trained in an reverse order, the main-classifier then followed by pre-classifier. And the pre-classifier at the later stage is trained with sample weighting method, which are based on the training result of Main-classifier. The experiments show the efficacy of the proposed approach, and its great advantage under the scenario of few-shot training.", "summary_of_the_review": "Overall it is a good paper, but not ready for publication.", "strength_and_weaknesses": "Quality/Clarity: the paper is well written and the techniques presented are ok to follow. The authors design a muti-stage classifier including main-classifier and the pre-classifier with the later is trained with weighted samples.\n\nOriginality/significance: the idea is incremental, there have been multiple existed approaches such cascaded classifier to speed up inference, adboost algorithm with weighted samples, etc. \n\nExperiments: the method should be compared with some baselines such as adboost.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "BpDZSU4NbV", "reviewer": "Reviewer_7ycA", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a training framework, named Feedback training, to improve the performance of two-stage binary classifier, a common type of multi-stage classification. Experiments on both task extraction and Yahoo Answer demonstrate the superior performance of the proposed method. ", "summary_of_the_review": "Given the comments about the weakness of the paper, the paper cannot meet the quality of ICLR. ", "strength_and_weaknesses": "Strength:\n\nDeveloping an effective algorithm to trade-off between performance and model scale is an interesting research topic and practical for resource-constrained applications. \n\n\nWeakness:\n\n1. Figure 2 is not intuitive to explain the problem. And the motivation for proposing a reversed-order training framework is not well explained. \n\n2. The logic of the proposed method is hard to follow. It’s not clear why the sample weighted loss function is able to address the challenge of modeling the learning preference as described in Sec. 3.2.\n\n3. The weighted sampling strategy in Sec. 3.3 seems to be ad-hoc without any theoretical justification. \n\n4. The experimental evaluation is weak without the comparison with some state-of-the-art comparisons.\n\n5. Since the proposed method is only for the binary classifier, its practical usage is limited. It would be good to extend the method to multi-class classification. \n\n6. There are works studying the efficiency of Committee-based models (ensembles or cascades), e.g. [a]. This paper lacks discussion and comparison of such related works. \n[a] WISDOM OF COMMITTEES: AN OVERLOOKED APPROACH TO FASTER AND MORE ACCURATE MODELS, ICLR 2022\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "9hRDzWr3ACd", "reviewer": "Reviewer_WfBE", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In view of the problems caused by Sequential Training and Independent Training in multi-stage classification, this paper proposes a new learning framework called feedback training, which outperforms the baseline methods in the two-stage binary-classification settings.", "summary_of_the_review": "My main concerns are the writing, reproducibility, and significance of experiments.", "strength_and_weaknesses": "Strengths.\nThis paper proposes a feedback training strategy, i.e., first train the main classifier, then determine the classification “difficulty” of each sample based on the trained main classifier, and finally adjust the weight of each sample during the training of the pre-classifier. \nIn general, the author’s motivation is clear, and proposes a simple yet effective (according to the experimental results presented in the paper) solution to solve the mentioned problems of previous Sequential Training and Independent Training methods.\n\nWeaknesses\n1) Although the paper is generally well organized and presented, there are also some obvious irregularities that may hinder understanding. For example, the meanings of “pre_score” and “main_score” in Figure 2 should be clearly expressed in the caption. For another example, for all tables in the experiment part, the corresponding training methods of each line should be indicated in the table rather than in the caption, and it is better to bold the best results for all tables for better clarity.\n2) The reproducibility of the paper is doubted, since some details of the experiments are not specified. For example, for the few-shot experiment, 1%~10% of the training data is selected. However, how the data is selected and which data is selected (manually selected or randomly) are not clearly explained to ensure reproducibility. In addition, the specific value of PassRate for Table 1 is not specified.\n3) Experiments can be more abundant and comprehensive to prove the effectiveness of the method. For example, more models can be adopted as the pre-classifier and the main-classifier. In addition, Figure 2 shows the inconsistency problem of independent training. I'd like to see a similar figure for feedback training, which can show more intuitively whether the pre-classifier achieves the claimed filtering effect. Further, for Yahoo! Answers dataset, the paper only splits \"society&culture\" into positive samples, and other classes are splitted as negative samples to meet the binary classification setting. I'd like to see the results of more splittings of classes to confirm that the results of baselines are consistently improved under different splittings.\n4) Some minor grammatical errors need to be corrected. For example, the last paragraph of the first page: \"exist methods\" should be changed into \"existing methods\". The caption of Figure 1: \"only samples passed\" should be changed into \"only samples that passed\". The second paragraph of Section 3.1: \"Transformer-base\" should be changed into \"Transformer-based\", and \"bias data set\" into \"biased data set\". The second sentence of the first paragraph below Figure 2 (\"but some other...\") lacks a predicate. Experience Setup: Whether the first letter after the colon is in uppercase or lowercase should be determined.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject