Learning Top-k Classification with Label Ranking
Authors: Bin Cao
Kai Wang
JING FAN
Jianwei Yin
Venue: Submitted to ICLR 2023 (2023)

Abstract: Class confusability and multi-label nature of examples inevitably arise in classification tasks with the increasing number of classes, which poses a huge challenge to classification. To mitigate this problem, top-$k$ classification is proposed, where the classifier is allowed to predict $k$ label candidates and the prediction result is considered correct as long as the ground truth label is included in the $k$ labels. However, existing top-k classification methods neglect the ranking of the ground truth label among the predicted $k$ labels, which has high application value. In this paper, we propose a novel three-stage approach to learn top-$k$ classification with label ranking. We first propose an ensemble based relabeling method and relabel the training data with $k$ labels, which is used to train the top-$k$ classifier. We then propose a novel top-$k$ classification loss function that aims to improve the ranking of the ground truth label. Finally, we have conducted extensive experiments on four text datasets and four image datasets, and the experimental results show that our method could significantly improve the performance of existing methods.

Under review as a conference paper at ICLR 2023
LEARNING TOP-K CLASSIFICATION WITH LABEL RANKING
Anonymous authors Paper under double-blind review
ABSTRACT
Class confusability and multi-label nature of examples inevitably arise in classifi- cation tasks with the increasing number of classes, which poses a huge challenge to classification. To mitigate this problem, top-k classification is proposed, where the classifier allowed to predict k label candidates and the prediction result is considered correct long as the ground truth label is included in the labels. However, existing top-k classification methods neglect the ranking of the ground truth label among the predicted k labels, which has high application value. In this paper, we propose novel three-stage approach to learn top-k classification with label ranking. We first propose an ensemble based relabeling method and relabel the training data with k labels, which is used to train the top-k classifier. We then propose a novel top-k classification loss function that aims to improve the ranking of the ground truth label. Finally, we have conducted extensive experiments on four text datasets and four image datasets, and the experimental results show that method could significantly improve the performance of existing methods.
INTRODUCTION
Multi-class classification aims classify examples into one of than classes, and as the number of classes increases to large extent, e.g., thousands of classes, training a multi-class clas- sifier will become extremely challenging due to the presence of multi-label nature of the examples and class confusability (Gupta et al. 2014; Lapin et al. [2015] Chang et al. (2017). To mitigate this problem, the task of top-k classification where the classifier is allowed to predict & label candidates and the prediction result is considered correct long as the ground truth label is included in the k: labels. This evaluation measure is com- monly referred to as the top-k error (Lapin et al. 20066) i.e., the loss function will not penalize k: - mistakes. Though state-of-the-ar models directly trained with cross-entropy can also yield remark- able results in terms of top-k error, the data used for training must be both large and clean (Berrada ctt al. 22088), which cannot be guaranteed in real scenarios. Moreover, traditional top-1 error loss function like cross-entropy may have over-fitting problem when noisy label exists (Berrada al., 2018 Hence, loss functions tailored for top-k error minimization are needed.
proposed (Berrada et al. 2018 Petersen et al. 20022)
However, existing top-k classification loss functions (Lapin al. 2015 Chang al., 2017 Berrada et al. 2018) only consider whether the ground truth label is reported in the predicted i labels and neglect the ranking of the ground truth label within the top-k candidates. In fact, ranking is crucial for tasks of top-k classification. For example, in classic human-in-the-low annotation scenario, classification model trained with small amount of labeled data is used to predict for each unlabeled example, and humans are required to check the prediction result of classification model and relabel those with low confidence. However, manually selecting the correct label from large label set is time-consuming and inefficient. In this case, the model is allowed to predict the & most likely labels so that humans could easily find the ground truth label from those k labels, i.e., top-k classification. Meanwhile, improving the ranking of the ground truth label in the & labels will allow humans to get the ground truth label at the first time, which could effectively improve the efficiency of humans checking. The ranking motivation behind above scenario actually aligns with applications like recommendation system and search engine (Oosterhuis & de Rijke, 2020).
(Zanzotto) 2019) data

---

Under review as a conference paper at ICLR 2023
Thiniing sut Validation set: single-Sabel examples
Subset
Classifiee
05
Average
Top-lk-
Modified training sob kolabel esamples
1l Training with up
Pauy
Top-k Classifer
Scbact zz
Classifiur
Predict
92 04
11
Random
Training with C2059 eatropy
Training single-label exomples
Retsining the ground truth label
Training hase classifiers
Relaheling training data
Training top-k classifior
Figure The proposed three-stage approach for top-k classification.
Therefore, in this paper, aim to design top-k classification method that predict ko most likely labels where the ground truth label is not only included but also its ranking should be as high possible. Driven by the idea of multi-label classification (MLC) (Tsoumakas & Katakis (007). we propose a novel three-stage approach for top-k classification. As shown in Figure in the first stage, we use the existing training and validation sets train the base classifier which will be used to predict k-labels for each training example in next stage. Considering that the correct ls labels are crucial for top-k classification in our problem setting, we use the idea of ensemble learning (Sagi & Rokach 2018 to train m base classifiers. Each base classifier is trained with classic cross-entropy loss function on the subset that randomly selected from training and validation sets. In the second stage, we relabel each single-label example to k-label example by using one ground truth label and & - most likely labels. More specifically, we predict m probability distributions p for each sample by the base classifiers, and then average all probability distributions to get the average probability distribution Pang- Finally, we output the most likely k: 6-1 labels for each training sample besides its ground truth label according the Paug- We refer to these k 1 most likely labels as pseudo-labels.
In the third stage, based on the transformed k-label examples, we train a multi-label classifier which predict exactly k labels for a test example. Then, the trained multi-label classifier can be viewed as a top-k classifier. It is important to note that, we propose new top-k loss function with Jabel ranking (TkLR) for training in this stage. For the example with k labels, TkLR aims to maximize the difference between the scores of these labels and the scores of other labels. To improve the ranking of the ground truth label, we embed an additional rank loss in TkLR, which aims to maximize the difference between the scores of the ground truth label and the scores of pseudo-labels.
Finally, we conduct sufficient experiments on four text datasets and four image datasets with BERT (entoon & Toutanova| 2019) and Swin Transformer (Liu et al. [02I] as the backbone model respec- tively. Then we evaluate the experimental results with top-k accuracy and Normalized Discounted Cumulated Gains at top (NDCG@K) (Wang et al. 2013). The experimental results demon- strate that method could significantly improve the performance of existing top-k classification methods.
In brief, the main contributions of this paper are summarized as follows:
We propose to consider the ranking of ground truth label in top-k classification, which has high application value but hasn't 't been well addressed so far.
We propose a novel three-stage approach to learn top-k classification with label ranking, which can be easily deployed with different classification models.
We propose an ensemble based relabeling method to obtain the most likely i labels for each training example which will benefit the final top-k classification.
We propose novel top-k loss function that takes the ranking of the ground truth label into account.
The extensive experiments over different text and image datasets show that our method greatly outperforms existing baselines in terms of top-k accuracy and NDCG@K metrics.
The remainder of this paper is structured follows. In Section 2, present the related work. Section 3 describes our approach. We describe our experiments in Section 4. Section 5 concludes the paper.
2

---

Under review as a conference paper at ICLR 2023
2
RELATED WORK
In this section, we will present our related work in terms of top-k classification and multi-label
classification.
Top-k classification. Lanpin al. (Lapin et al. 2015) proposed top-k multiclass SVM based tight convex upper bound of the top-k hinge loss. Then Chu et al. al.:Chu et al. 2018) propose an opti- mized top-k multiclass SVM algorithm, which employs semismooth Newton algorithm for the key building block to improve the training speed. Chang et al. (Chang et al. 2017 propose generic, robust multiclass SVM formulation that directly aims at minimizing weighted and truncated com- bination of the ordered prediction scores. Recently Berrada et al. (Berrada et al. 2018) propose Smooth loss function which new top-k classification loss function for deep neural networks. The Smooth loss function creates margin between the correct top-k predictions and the incorrect ones. Petersen et al. (Petersen et al. 2022)) propose a family of differentiable top-k cross-entropy classification losses and relax the assumption of fixed k. However, all the above methods do not take into account the ranking of ground truth labels among the predicted & labels.
Multi-label classification. Boutell et al. (Boutell et al. 2004) decomposed the multi-label clas- sification into several independent binary classification problems. Read et al. et al. 2011) transformed the multi-label classification problem into chain of binary classification problems. For text dataset, Yang et al. (Yang et al. 2018) proposed sequence-to-sequene sification model learn the correlation among labels. Huang al. (Huang et al., 2021) introduce the application of balancing loss functions for multi-label text classification. For image dataset, Chen al. (Chen et al. 2019) propose a multi-label classification model based on Graph Convo- lutional Network (GCN) to capture and explore label dependencies. Lanchantin et al. (Lanchantin al. proppos the Classification Transformer (C-Tran), a general framework for labti-label image classification that leverages Transformers to exploit the complex dependencies among visual features and labels.
multi-label clas-
Though our method is inspired from multi-label classification, the difference is obvious: 1) The input is different. The number of ground truth label per training sample for multi-label classification is indeterminate, while the training sample of our method only has one ground truth label. 2) The output result is different. The number of labels predicted by multi-label classification is also uncer- tain, while the number of labels predicted by our method is fixed at k. 3) The evaluation measure is different. The all predicted labels multi-label classification are required to be ground truth labels, while our method only requires the ground truth label included in the predicted k labels.
3 THREE-STAGE APPROACH
3.1 OVERVIEW
As can be seen from the Figure] three stages of our approach are: training base classifiers, relabel- ing training data and training top-k classifier.
Stage Training base classifiers. Base classifiers are used predict the i most likely labels for each training example in stage 2. Considering that the correct & labels are crucial for top-k classification in our problem setting, we use the idea of ensemble learning to train m base classifiers. Specifically, given the single-label dataset D which is composed of training set and validation set. we randomly select m different subsets from dataset D to train the m base classifiers, where the proportion of subsets in dataset D is a. The loss function of training process is cross-entropy loss function. The settings of hyperparameters a and m will be discussed in Appendix 4.2
Stage 2: Relabeling training data. The purpose of this step is to relabel each single-label training example with its most likely labels. We consider the ground truth label to be the most likely label, so we only need to find the other k: 1 most likely labels. Specifically, given the sanele-label training set ((2; H=) where x; are the input features of each example i and yi is the ground truth label. We aim to relabel each example (I;, ys) to (x,, Yt), where Y, E R is composed of the ground truth label y/4 and the I - 1 other labels that most related to IG, To this end,
use the m base classifiers to predict m different probability distribution p of all labels for each sample in the training set, where p E RRC, is the number of total labels. Then we average the m probability
3

---

Under review as a conference paper at ICLR 2023
distributions and get the average probability distribution Parg. According to the Pavg, output the - labels with the highest probability except the ground truth label, and we use these k - labels as the pseudo-labels of the sample. At this point, relabeling is complete, and each example in the training set has one ground truth label and k - pseudo-labels. Note that, the training set this stage is used for deriving probability distribution training set.
different labels and obtaining for modified
Stage 3: Training top-k classifier. We take the output of the second stage training data and use our proposed top-k loss function to train new top-k classification model. The loss function is described in Section 3.2
3.2 TOP-K K LOss FUNCTION WITH LABEL RANKING
In this section, we will introduce our rank top-k loss function. In order to better understand the top-k classification, we start with the simple case of k=1, then the top-1 classification is an ordinary multi- class classification. Given a training sample (xi, Vi), i = 1, ...,,,,,, where y/ y : (1...., c). the classifier will output the score of the sample on each label, i.e., the score vector := (s1,.... 8c)- The multi-class classification aims to maximize the probability Py of ground truth label y. where Py is calculated by Softmax. Then multi-class classification aims at minimizing the following Softmax cross-entropy loss:
c.v
Lk=1 log(
Σj=1
where sy is the score of the ground truth label. From Eq it can be seen that reducing the loss is equivalent increasing Sy. Taking 8y as the object, Eq can be transferred as follows:
Cx-1 = - logl(
log
2j=1 e
2j=1
e -8
(2)
log
log
1+
j=1
デ=1.
The goal of Eq 2 is to minimize (sy- 15--s) sy), that is, is hoped that the score the ground truth label should be much larger than the scores of other labels. We define the ground truth label as positive label and the rest as negative labels, then Eq can be rewritten as follows:
Ck=1 = log 1 Σ
where neg is the set of the negative labels, 8n is the score of the negative label, Sp is the score of the positive label. Since & = 1, 3qg3ha has 1 positive label and e - 1 negative labels. Obviously, Eq 3 takes maximizing the score of positive label the optimization goal, and directly outputs the label with the largest score as the prediction result in the prediction stage.
When A 1, each training sample has labels, like the case of k = L, the loss function expects that the scores of the & labels are larger than the rest labels. Specifically, given training sample (æ,, Y)) which is the top-k relabeling result of (æ,, yi), where E & is composed of the ground truth label and the k - 1 other pseudo-labels that are most related to the loss function aims to maximize the scores of the Y, and minimize the scores of the rest labels. Similarly, we define Y, as positive labels and the rest as negative labels, then loss function can be rewritten as follows:
C log(1
Σ
(4)
where pos is the set of the l positive labels and neg is the set of the k negative labels. Obviously, Eq hopes to maximize the score of positive labels, and the classifier only needs to output the labels with the largest scores the predicted results.
4

---

Under review as a conference paper at ICLR 2023
Table Statistics of the datasets
Dotaset
Train
Validate
Test
Labels
Domain
Ohsumed
3357
2026
2017
23
Text
WOS
28139
9451
9395
134
Text
20Ng
11270
3692
3820
20
Text
TREC
4882
570
500
50
Text
CIFARIO0
40000
10000
10000
100
Image
Aircraft
3333
Image
CUB-200-2011
7099
2315
2374
200
Image
Indour67
9398
3069
3153
67
Image
However, the loss function in E does not consider the ranking of the ground truth label. Another goal of our top-k classification is to expect that the ground truth label has a higher ranking in the predicted & labels, which means that the score of the ground truth label should be large as possible. More specifically, the score of the ground truth label sy should be larger than that of other positive labels, i.c. maximize (sy sp), + y. Then the final loss function can be written follows:
C log(1
Σ
cn
Σ exv exx-)y)
nEneg.pEpox
pEpos
(5)
= =log(1
Σ
e*v)
n.nE
PEpox
pEpos
where pos' by is the set of positive labels other than the ground truth label y. We name the third part e Epeporty exp in the loss function of Eq 5 as rank loss.
Adding rank loss to the final loss function has two benefits. The first is to improve the ranking of the ground truth label in the predicted labels mentioned above. Second, the top-k classification expects the ground truth label to appear in the k predicted labels, and the rank loss could improve the score of the ground truth label, which is helpful for the ground truth label to be predicted the k labels more likely. Therefore, rank loss also helps improve the accuracy of top-k classification.
4 EXPERIMENT
In this section, we perform extensive experiments validate the effectiveness of our method. First, we describe the experimental setup in detail. Second, we compare our method with different ap- proaches. We then conduct experiments both text datasets and image datasets, simultaneously evaluating our experimental results with two evaluation metrics. The source code will available httpggiiwww..o./aeee/9999///////
4.1 EXPERIMENTAL SETUP
Dataset and model selection. our experiment, we select 4 single-label text datasets and
single-label image datasets: Ohsumed (Joachims) 1998), 20NG (Johnson & Zhang 2016). WOS
(Kowsari et al. 2017). TREC (Li & Roth 2002), CIFAR100 (Krizhevsky et al. |2099) Aircraft (Lu
al.] 2021). CUB-200-2011 (Wah et al. 2011), Indoor67 QQattoni & Torralba 22009). Ohsumed
includes the medical abstracts of MEDLINE database and describes different cardiovascular dis-
eases. 20 Newsgroups (20NG) a collection of newsgroup documents posted on 20 different top-
ics. WOS-46985 (WOS) collects the abstracts of papers published in Web Of Science. The TREC dataset is dataset for question classification consisting open-domain, fact-based questions divided into broad semantic categories. CIFAR100 is a subset of the Tiny Images dataset and consists of 60000 32x32 color images. CUB-200-2011 is a challenging dataset of 200 bird species. Aircraft contains different aircraft model variants, most of which are airplanes. Indoor67 contains 67 indoor scene categories, and total of 15620 images. Table shows the statistics method is general top-k classification method that be applied to various models. In experi- ment, we choose BERT (Kenton & Toutanova (019) as the backbone model for the text dataset and Swin Transformer (Liu et al. 2021) the backbone model for image dataset.
all datasets. Our
5

---

Under review as a conference paper at ICLR 2023
Comparison methods. For text dataset, compare method with four approaches: (1) BERT + CE, a simple multi-class classification method that uses cross-entropy (CE) as the loss function of BERT. (2) BERT+ +opp- SVM,, which obtains the features from the fine-tuned BERT and then imput the features into top-k SVM (Lapin et al. 2015) for top-k classification. (3) BERT+TOPpH Entropy,
which trains the top-k classifier by top-k Entropy (Lapin al. 2015) and the features obtained from the fine-tuned BERT. (4) BERT+Top-k SVMsemismooth, which use Top-k SVMSemiimmooh (Chu al. 2018) and the features obtained from the fine-tuned BERT train the top-k classifier. (5) BERT + Smooth, which uses the recent top-k loss function Smooth loss function (Berrada et al. 2018 as the loss function of the BERT. (6) BERT + Relabel, a multi-label classification method of BERT that uses the training set after relabeling process and uses the binary cross-entropy loss function. (7) BalancedL.os
Relabel, the state-of-the-art multi-label text classification model Balancedl (Huang et al. 2021) trained by the training set after relabeling process.
Similarly, we compare our method with the following approaches on image dataset: (1) Swin CE, multi-class classification method that uses cross-entropy (CE) as the loss function of Swin Trans- former. (2) Swin+Topp- SVM, which obtains the features from the fine-tuned Swin and then input the features into top-k SVM (Lapin et al. 2015) for top-k classification. (3) Swin+Top-k Entropy, which trains the top-k classifier by top-k Entropy (Lapin et al.) 2015) and the features obtained from the fine-tuned Swin. (4) Swin+ Top-k SVM SVM Semismoath which use Top-k SVM Semismonth (Chu ett al. 2018) and the features obtained from the fine-tuned Swin to train the top-k classifier. (5) Swin Smooth, which uses the Smooth loss function as the loss function of the Swin Transformer (6) Swin + Relabel, multi-label classification method for Swin Transformer that uses the training set after relabeling process and uses the binary cross-entropy loss function. (7) C-Tran Relabel, the
state-of-the-alt multi-label image classification model C-Tran (Lanchantin et al. 2021 trained by the training set after relabeling process.
Ablation experiment. We simply name our method as BERT (Swin) + Relabel + TkLR, and to
demonstrate the effectiveness of our proposed rank loss, we remove the runk loss from our method and conducted experiment, namely BERT (Swin) Relabel + TkLR--o-Rank.
the base classifiers training process, we choose BERT as the backbone network of the classifier for text dataset and Swin-small the backbone network of the classifier for image dataset. In the top-k classifier training process, for text dataset, we set epoch to 20, maximum sequence length to 512, batch size to 10, and learning rate to 2e-5. We train model with Adam W (Loshchilov & Hutter (017) optimizer with weight decay = 0.01. For image classification, we set epoch to 100, batch size 64, learning rate 5e-5 and adopt AdamW optimizer with weight decay 5e-2. Training process and convergence analyses of the loss function are discussed in Appendix A.2
Experimental setting.
Evaluation metrics. Considering the ranking of the ground true label among the predicted & la- bels, we use top-k accuracy (Yan al., 2018) and Normalized Discounted Cumulated Gains at top K (NDCG@)) (Wang et al. 2013) as our evaluation metrics. The specific calculation methods are as follows.
Top-k accuracy is defined as:
T2
flag:
If ground truth label in the predicted labels Otherwise
Acc
flag.
(6)
0
n
where n is the number of samples.
NDCG@k is defined as:
Suppose that l is the total number of the labels, NDCG@K is defined according to the predicted score vector i E IR and the ground truth label vector y E 1), as follows:
DCG@k=
Σ
Vj log(pos(j) + 1)
NDCG@K
DCG@k
(E)ankx()
log
where rank)()) (v) is the label indexes of the top-k highest scores of the current prediction result, pos(j) is the position of j. i.c., pos(j) = 1,2, 3... I|y/lo counts the number of labels in the ground truth label vector y-

---

Under review as a conference paper at ICLR 2023
Chsuneed
Cift/000
Chsumed
Ci0s110
0..66
634
0.95
0.35
1op-k Ao0annc
0.94
Top-k securay
0.92
Top-k prcsraaa
0.94
Top accuray
1192
0:50
0.93
0.50
0.92
0.88
0.88
k-5
0.91
k-d
036
0.92
010205 00120000000A000OOO
(a)
(b)
Figure 2: Setting of hyperparameters ar and m
Table 2: Top-k accuracy results on text datasets
Ohsumed Top 5
20NG
WOS
TREC
Methods
3
Top 7
Top 3
Top 5
Top 7
Top 3
Top 5
Top 7
Top 3
Top 5
Top
BERT+CE
0.8567 0.8984 0.9207 0,9623 0,9812 0.9812 0.8923 0,9187 0.9327
0,9680 0.9780 0.9800
BERT+Top-K SVM
0,8691 0.9048 0.9296 0.9702 0.9851 0.9898 0.8879 0.9139 0.9308
0.9500
0.9620 0.9780
BERT+Top-k Entropy
0,8820 0.9103 0.9291 0.9715 0.9851 0.9906
0.8956
0.9167 0.9280
0,9360
0.9440 0.9260
BERT+Top-k
0.8840 0.9182 0.9390 0.9699 0.9859 0.9908
0.9046
0.9275 0.9386
0.9500
0.9720 0.9780
SVM Semismanth
BERT+Smooth
0.8587 0.8855 0.9103 0,9573 0.9712 0.9838
0.8978
0.9137 0,9218
0.9620
0.9660 0.9720
ReRT+Relabel
0.8954 0,9316 0.9504 0.9673 0.9835 0.9898
0.9095
0.9348 0.9494
0.9760
0,9800 0.9860
BalancedI oss-Relabel
0.8780 0.9236 0.9484 0.9631 0.9822 0.9890
0,9088
0.9343 0.9485 0.9680
0.9820 0.9820
BERT+Relabel
+TkLR-No-Raae Rank
0.8984
0.9266 0.9514 0.9699 0.9817 0.9885
0.91 18 0,9350 0.9485
0.9760
0.9820 0.9800
RERT-RRelbee-TKK.RR
0.9147 0.9415 0.9579 0.9764 0.9882 0.9927 0.9191 0.9402 0.9564 0.9780 0.9820 0.9860
All experiments were implemented with Python 3.8 and PyTorch 1.8 on a Linux server. Our experi- ments for text classification on the RTX 3080Ti and for image classification on the RTX 3090.
4.2 HYPERPARAMETER SETTING
In our experiments, we employ Bagging (Zoou Bagging method randomly selects data with a proportion of from the total training data for training each time. We conduct experiments both on text and image datasets to determine the number of base classifiers m and proportion a. We first randomly initialize a, and determine the optimal m with the fixed a. Then determine with the optimal m. The experimental results are shown in Figure Figure 2a and Figure 2b show that in general, the larger m is, the higher the top-k accuracy is. On the Ohsumed dataset, when
2012)
method to train m base classifiers. The
8, the overall results are good, and the performance is no longer significant when m continues to increase. On the Cifar 100 dataset, the result also good when m = 5, and the effect of increasing m is not obvious. Figure 2c and Figure 2d show the performance of different a with the optimal m. The results show that when a = 0.8, top-k classifier can achieve the best performance on the Ohsumed dataset and the 10f dataset. We found similar experimental results on other datasets. Therefore, for hyperparameter m, we set the m to 8 for the text dataset and to 5 for the image dataset. For hyperparameter o, set to 0.8 for both the text dataset and the image dataset. The theoretically analysis of hyperparameters is present in A.I
4.3 EXPERIMENTAL RESULT ON TEXT DATASET
SablePland Table show the top-k accuracy and the NCG values for different & of all methods on text dataset, respectively. It is clear that our method BERT+Relabel+TH achieves the best results in both top-k accuracy and N DCG metrics, which demonstrates the effectiveness of our method to solve the top-k classification problem.
As can be seen from the top-k accuracy in Table 2] our method is much better than the method BERT+CF. Especially on Ohsumed dataset, when set different k; the accuracy of method is at
7

---

Under review as a conference paper at ICLR 2023
Table N DDG results on text datasets
Ohsumed
20NG
Methods
NDCG@3
NDCG@5
NDCG@7
NDCG@3
NDCG@5
NDCG@7
BERT+CE BERT+Top-k SVM
0.8016
0.8188
0.8265
0,9230
0,9307
0,9332
0.8101
0,8229
0.8269
0,9239
0.9118
0,9083
BERT+Top-k Entropy
0.8249
0,8339
0.8376
0,9260
0.9279
0.9280
BERT+Top-k SVMaemimmook
0.8274
0,8407
0.8458
0,9283
0.9336
0,9342
BFRT+Smooth
0,8021
0.8064
0.8037
0.8948
0.8879
0,8897
BERT+Relabel
0.7731
0.7536
0.8196
0.7891
0.6393
BulancedLoss
0.7700
0.7794
0.7787
0.8742
0.8315
0,8456
BFIRI-Relahel+Tk... R-- RR-No-Rank
0.7773
0.7739
0.7283
0.8658
0.7254
0.6661
MHRTTRRelabl+TKL..
0.8401
0.8508
0.8528
0.9342
0.9393
0.9372
TREC
Methods
NDCG@3
NDCG@5
NDCG@7
NDCG@3
NDCG@5
NDCG@7
BERT+CE BERT+Top-k SVM
0,8579
0.8688
0.8737
0,9465
0.9507
0,9514
0.8569
0.8665
0.8709
0.9347
0.9332
0,9334
BURT+Top-k Entropy
0.8645
0,8700
0.8725
0.91.32
0,9201
0.8972
BERT+Top-k SVM Somemmitto
0,8705
0.8798
0.8836
0,9331
0.9413
0,9439
BERT+Smooth
0.8550
0.8513
0.8487
0.9396
0.9326
0,9359
BeRT+Relabel
0.7793
0,7833
0,81155
0.8984
0,8192
0.7370
Balanced
0.8285
0.8348
0.8323
0.8942
0.8962
0.6870
BFKK+*Rlahcd+TM Ri-o---NRRRan
0.7953
0.7792
0.7935
0.9024
0.8376
0.7452
HERT-Relabel-TKKK.
0.8768
0.8809
0.8840
0.9474
0.9517
0.9523
Table 4: Top-k accuracy results on image datasets
Cifar100
Aircraft
CUB-200-2011
Indoor67
Methods
Top 3
Top 5
TTop7 7
Top 3
Top 5 Top
Top 3
Top 5
Top 7
Top 3
Top 5
Top
Swin+CE
0.8694 0.9117 0.9330
0.8797 0,9142 0,9316 0.9385 0,9613 0.9726 00,4833 0.9696 0,9822
Swin+Top-k SVM
0.8592
0.9050 0.9277
0.8974 0.9307 0.9499
0,9364 0.9650 0.9730 0.9353 0.9670 0.9772
Swin+Top-k Entropy
0.8570
0.9029 0.9259
0.8695
0.8986 0.9100
0.8614
0.9254 0.9465
0.9198
0.9312 0.9543
Swin+Top-k
0.8596 0.9067 0.9289
0.8968
0.9298 0.9499
0.9398
0.9667 0.9743
0.9423
0.9667 0.9791
S.V Seneimmoohh
Swin+Smooth
0.8741 0.9133 0.9326
0.8905
0.9316 0.9424
0,9495
0,9676 0,9760
0,9413
0,9619 0,9708
Swin+Relabel
0.8559
0.9077 0.9335
0.8644
0.9178 0.9391
0.9196
0.9549 0.9747
0.9340
0.9689 0.9784
C-Tran+Relabel
0.7810 0.8564 0.8877
0.8470
0.8962 0.9259
0.9073
0.9423 0.9604
0.8979
0.9407 0.9629
Swin+Relabel
0.8612 0.9097 0.9345
0.8773
0.9181 0.9388
0,9410
0.9638 0,9752
0.9445
0,9705 0.9826
-TkLR-NN-Rank
Swin-Relabee-TIL...R
0,8830 0.9265 0,9442
0.9040
0,9358
0.9511
0.9528
0.9718
0.9832
0.9531
0,9759 0.9838
least 3.7% higher than that of BERT+CE method. At the same time, cross entropy aims to maximize the probability the ground truth label, hence BERT+CE will perform well on the N DCG metric in Table 3 Nevertheless, the results of method NDCG metric are still better than that of BERT+CE. which illustrates the effectiveness of our rank loss. The top-k accuracy results of BERT+Relabel are similar to the results of BalancedLoss<Relateee are better than BERT+CE, BERT+Smooth,
Besides, in the most cases, the
results of BERT+Relabel and Balanced BERT+Top-k SVM, BERT+Top-k Entropy and BERT+Top-K SVMSemismooth overall all text datasets, which illustrate the relabeling process could improve the top-k classification accuracy and the idea of predicting the most likely & labels is effective. However, the results of RelbeRelabel and Balancedl Relabel on DDG metric are bad, because they do not consider the ranking of the ground truth label, on the contrary, our loss function does.
Compared with BERT+Relabel, our method BEKTT-RelabeI+TKLA achieves better results in top- accuracy and NDG metric on all datasets, which shows that our proposed loss function is very effective. Though the top-7 accuracy of BERT+Relabel is equal to BERT-RelabelITKLA are extremely better than BERT+Relabel.
the result on NDG metric of BERT+Relabel+ which shows the strength of rank loss. Comparing the results of BERT BERT-RRllablI-RRLRRR No-mkk and BERT+RelabcI+TKLLR shows that adding rank loss can not only improve the top-k accuracy, but also improve the N DCG metric.
8

---

Under review as a conference paper at ICLR 2023
Table 5: NDCG@k results on image datasets
Cifar100
Aircraft
Methods
NDCG@3
NDCG@5
NDCG@7
NDCG@3
NDCG@5
NDCG@7
Swin+CE
0.8029
0.8204
0.8278
0.8111
0.8253
0.8313
Swin+Top-k SVM
0.7960
0.8149
0.8232
0.8332
0.8458
0.8507
Swin+Top-k Entropy
0.7932
0.8136
0.8228
0.7541
0.8125
0.8121
Swin+Top-k SVMgemiamooth
0.7965
0.8163
0.8240
0.8336
0.8456
0.8506
Swin+Smooth
0.7949
0.7958
0.7925
0.8176
0.8351
0.8346
Swin+Relahel
0,7511
0.7647
0.7643
0.7727
0.7941
0.8087
C-Tran+Relabel
0.6918
0,7172
0,7215
0,7463
0.7686
0.7772
Swin+RRlahcl-TlLLR -o-Roo--kk
0.7679
0.7747
0.7761
0.7872
0.8000
0.8054
Twin-RRelabll-kLLR
0.8143
0.8249
0.8296
0,8399
0.8468
0.8545
CUB-200-2011
Indoor67
Methods
NDCG@3
NDCG@5
NDCG@7
NDCG@3
NDCG@5
NDCG@7
Swin+CE
0,9025
0.9120
0,9159
0.8987
0,9076
0,9119
Swin+Top-k SVM
SVM
0,8860
0.8987
0.8991
0.8749
0.8841
0.8859
Swin+Top-k Entropy
0.7170
0.8294
0.8550
0,8518
0.8352
0.8318
Swin+Top-k SVMgemiammont
0.8937
0.9018
0.9022
0.8868
0.8939
0.8923
Swin+Smooth
0.9011
0.9056
0.9071
0.8811
0.8853
0.8909
Swin+Relahel
0.8463
0.8602
0.8754
0.8617
0.8799
0.8817
C-Tran+Relabel
0.8247
0.8307
0,8460
0.8177
0,8315
0.8292
Swin+RRlahel--KLLR
0.8704
0.8704
0.8831
0.8734
0.8731
0.8796
Swin-Rellacll-KkLRR
0.9110
0,9151
0.9152
0.9016
0,9100
0,9137
4.4 EXPERIMENTAL RESULT ON IMAGE DATASET
Table 4 and Table show the top-k accuracy and the NDCG values for different & of all methods on image dataset, respectively. The experimental results on the image dataset are similar those on the text dataset, our method achieves the best results overall on both top-k accuracy and N DDG metries, which illustrates the generality of our method.
Compared with the Swin+CE method, our method Swin+Relahel+ accuracy, especially on the dataset with low top-k accuracy such as Cifar100 and Aircraft. Although Swin+CE performs well on the N DCG metric due to the cross entropy, and gets slightly better results than Swin+Relabel+ the best results the NDCG metric all other cases. This shows that the rank loss is still very effective on image datasets. The results between Swin+CE, Swin+Top-k SVM, Swin+Top-k almost similar and better than Swin+Top-k Entropy. But
has improved the top-k
at DDG@@7 on CUB-200 200-2011, Swin+Relabel+ still gets
SVMsenismont and Swin+Smooth the performance of Swin+CE on the NDCG metric is better, because other methods neglect the ranking of ground truth label. The experimental performance of C-Tran+Relabel is poor on two metrics, especially on datasets Cifar100, Compared with the method Rank, Swin+Relabel+TkL.R obviously has a better performance both on top-k accuracy and N DCG metric, which shows the rank loss could also improve the top-k accuracy and the N DCG metric on image dataset.
5 CONCLUSION
In this paper, we propose the importance of ground truth label ranking in top-k classification, which none of the previous top-k classification methods take into account. Inspired by the multi-label classification, we transform the top-k classification problem into problem of predicting the most likely labels and we propose a novel three-stage approach for top-k classification, which can be easily deployed with different classification models. We then propose an ensemble based relabeling method and relabel the training data with k labels. Besides, we propose novel top-k loss function that takes the ranking of the ground truth label into account. Finally, conduct extensive experi- ments on both text datasets and image datasets, and the experimental results show that our method achieves the best results.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Leonard Berrada, Andrew Zisserman, and M Pawan Kumar. Smooth loss functions for deep top-k classification. In ICLR, 2018.
Matthew Boutell, Jiebo Luo, Xipeng Shen, and Christopher Brown. Learning multi-label scene classification. Pattern Recognition, 37()):1777-1771, 2004.
Xiaojun Chang, Yao-Liang Yu, and Yi Yang. Robust top-k multiclass svm for visual category recog- nition. SIGKDD, pp. 75-83, 2017.
Zhao-Min Chen, Xiu-Shen Wei, Peng Wang, and Yanwen Guo. Multi-label image recognition with graph convolutional networks. In CVPR, pp. 5177-5186, 2019.
Dejun Chu, Rui Lu, Jin Li, Xintong Yu, Changshui Zhang. and Qing Tao. Optimizing top-k multi- class svm via semismooth newton algorithm. TNNLS, 29(12))624-6677,, 2018.
Maya R Gupta, Samy Bengio, and Jason Weston. Training highly multiclass classifiers. JMLR, 15 (1):1461-1492, 2014.
Yi Huang, Buse Giledereli, Abdullatif Koksal, Arzucan Ozgur, and Elif Ozkirimli. Balancing meth- ods for multi-label text classification with long-tailed class distribution. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2021.
Thorsten Joachims. Text categorization with support vector machines: Learning with many relevant features. In ECML, pp. 137-142. Springer, 1998.
Rie Johnson and Tong Zhang. Supervised and semi-supervised text categorization using Istm for region embeddings. In ICML, pp. 526-534. PMLR, 2016.
Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT, pp. 4171 11866 2019.
Kamran Kowsari, Donald E Brown, Mojtaba Heidarysafa, Kiana Jafari Meimandi, Matthew S Ger- ber, and Laura Barnes, Hdltex: Hierarchical deep learning for text classification. ICMLA, pp. 364-371. IEEE, 2017.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Jack Lanchantin, Tianlu Wang, Vicente Ordonez, and Yanjun Qi. General multi-label image classi- fication with transformers. In CVPR, pp. 16478-16488, 2021.
Maksim Lapin, Matthias Hein, and Bernt Schiele. Top-k multiclass svm. NIPS, 28, 2015.
Maksim Lapin, Matthias Hein, and Bert Schiele. Loss functions for top-k emor: Analysis and insights. In CVPR, pp. 1477-147, 2016.
Xin Li and Dan Roth. Learning question classifiers. In COLING, 2002.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, pp. 10012- 10022, 2021.
Loshchilov and Hutter. Fixing weight decay regularization in adam, corr, abs/1711.05101. In ICLR, volume 30, 2017.
Zhichao Lu, Gautam Sreekumar, Erik Goodman, Wolfgang Banzhaf, Kalyanmoy Deb, and Vishnu Naresh Boddeti. Neural architecture transfer. TPAMI, 43(9):2971-2988)
2021. 2021.
Harrie Oosterhuis and Maarten de Rijke. Policy-aware unbiased learning to rank for top-k rankings. In SIGIR, pp. 489-498, 2020.
Felix Petersen, Hilde Kuchne, Christian Borgelt, and Oliver Deussen. Differentiable top-k classifi- cation learning. In ICMI., pp. 17656-17668. PMLR, 2022.
10

---

Under review as a conference paper at ICLR 2023
Ariadna Quattoni and Antonio Torralba. Recognizing indoor scenes. In CVPR, pp. 413-420. IEEE, 2009.
Jesse Read, Bernhard Pfahringer, Geoff Holmes, and Eibe Frank. Classifier chains for multi-label classification. Machine Learning, 85(3):333, 2011.
Omer Sagi and Lior Rokach. Ensemble learning: survey, Wiley Intendisciplinary Reviews: Data Mining and Knowledge Discovery, 8(4): 2018.
Grigorios Tsoumakas and loannis Katakis. Multi-label classification: An overview. IJDWM, 3(3): 1-13, 2007.
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-uesd birds-200-2011 dataset. 2011.
Yining Wang, Liwei Wang, Yuanzhi Li, Di He, Wei Chen, and Tie- Yan Liu. theoretical analysis of ndeg ranking measures. In COLT 2013, volume 8, pp. 6. Citeseer, 2013.
Caixia Yan, Minnan Luo, Huan Liu, Zhihui Li, and Qinghua Zheng. Top-k multi-class svm using multiple features. Information Sciences, 432:479-494, 2018.
Pengcheng Yang, Xu Sun, Wei Li, Shuming Ma, Wei Wu, and Houfeng Wang. Sgm: sequence generation model for multi-label classification. In COLING, pp. 3915-3926, 2018.
Fabio Massimo Zanzotto. Human-in-the-loop artificial intelligence. JAIR, 64:243-252, 2019.
Zhi-Hua Zhou. Ensemble methods: foundations and algorithms. CRC press, 2012.
A APPENDIX
A.1 THEORETICALLY ANALYSIS OF HYPERPARAMETERS FOR ENSEMBLE LEARNING
In statistics and machine learning, ensemble learning uses multiple learning algorithms to obtain better predictive performance than could be obtained from any of the constituent learning algorithms alone. Take the easy binary classification as an example. Given E [-1,+1] and the ground truth function f. Suppose that each base classifier h, has an independent generalization error i.e.,
P (h,(x:) * f())
After combining m number of such base classifiers according to
m Σ
H(x) sign
h(((
the ensemble makes an error only when least half of its base classifiers make errors. Therefore, by Hoeffding inequality, the generalization error of the ensemble is
[m/2]
P(H(x) + f(x)) = Σ
(m) (1- k
m
k S exx (2 - 1)2
The above formula shows that when the number m of classifiers the ensemble increases, the error rate of the ensemble decreases exponentially. That is to say, higher number of classifiers can make the prediction results of the ensemble model more accurate.
In our method, the basic classifier in stage 1 is used to predict the & most likely labels for samples, the prediction results is extremely important for subsequent training. Using only one basic classifier cannot guarantee the stability and accuracy of the prediction results, so we use the idea of ensemble learning to train multiple classifiers in stage 1. However, the number of ensemble models is not as good possible, too many models will consume lot of resources and time. At the same time, when the number of models reaches a certain level, increasing the number of models has little effect on the prediction results, At the same time, in order to ensure the diversity of each basic classifier,
11

---

Under review as a conference paper at ICLR 2023
Tich
035
0.8
2.
Cor Evtoyy
016
27
Opach
Spo
Epoch
(a) Training loss of different loss (b) Training loss of TkLR for dif- (c) Validation accuracy of different functions
ferent l
loss functions
Figure 3: Training process
the training data of each classifier should be different, so cannot be 1. However, if a is too small, a better basic classifier cannot be trained. Therefore, in practice, the number m of ensemble models and the proportion are usually determined through experiments such Fig
In our experiments, we employ Bagging method to train multiple base classifiers. The Bagging method randomly selects data with proportion of from the total training data for training each time. If a is too small, the classifier will underfit, and if a is too large, the classifiers will be too similar, both of which will make the prediction performance of the ensemble classifiers worse. Thus, is usually obtained through experimental verification. We conduct experiments both on text and image datasets determine the number of base classifiers m and proportion a. We first randomly initialize a, and determine the optimal m with the fixed a. Then determine with the optimal m. The experimental results are shown in Figure 2 Figure 22 and Figure show that in general, the larger m is, the higher the top-k accuracy is. On the Ohsumed dataset, when
8, the overall results are good, and the performance is no longer significant when m continues to increase. is not
5, and the effect of increasing obvious. Figure 2c and Figure 2d show the performance of different a with the optimal m. The results show that when dataset and the difar100 dataset. We found similar experimental results on other datasets. Therefore, for hyperparameter m, set the to 8 for the text dataset and to 5 for the image dataset. For hyperparameter o, we set a to 0.8 for both the text dataset and the image dataset.
the C0 dataset, the result is also good when
0.8, top-k classifier
achieve the best performance
the Ohsumed
A.2 CHARACTERISTICS INVESTIGATION OF THE Loss FUNCTION
Time complexity. We will discuss the time complexity of our loss function in this part. We first ignore the label ranking part and start with & = 1. then our loss function is define as:
C = log
Σ e en--a
NEneg
which is equal to cross-entropy. Obviously, the time complexity of the loss function O(1). Sup- pose the total number of categories is when k > 1, the complexity of Eq depends
Sp
E PEpos
where pps] and Inegl = The total number of calculations is kic k). Due to the number of categories e is much larger than h in practice, the complexity of Fq is O(e * k.). At the same time, since the complexity of the label ranking part O(1), the time complexity of the loss function is O(c* * k).
Training process. To illustrate the performance of our loss function, we show the training process of our loss function. Fig shows the training process. First of all, it can be seen from Fig 3a our loss function converges slower than other loss functions, and the initial loss is larger. Comparing TkLR and AELR-NO-RAND at the same time, it is obvious that ME-R-NO-RANK converges faster, which shows that label ranking has an impact on the function. At the same time, T converge to loss value close to O like other functions, which shows that proposed loss
12

---

Under review as a conference paper at ICLR 2023
0.3
0.20
-0.2
-0.0
-0.5
-1.
255
-2
-35
-55
00
25
50
75
(a) Epoch =
(b) Epoch =1
(c) Epoch = 20
Figure 4: Visualization of top-k classification on test set. Top-k classification maximizes Sp 8n- Fig (a) and (b) are the unconverged state, and Fig (c) is the converged state.
0.30
10
0.0
0.20
30
.0.7
35
0.0
10
50
25
160 125
5/9
(a) Epoch =
(b) Epoch=
(c) Epoch = 20
Figure 5: Visualization of label ranking on test set. Label ranking maximizes 8y Sp 18, where ky
means + Fig (a) and (b) are the unconverged state, and Fig (c) is the converged state.
function is effective. Fig3b) shows the training process for different kt can be seen from the figure that the larger & is, the slower the convergence is. Due the complexity of the loss function is Occ* x k),, the larger & is. the greater the loss is. Fig 3e shows the top-k accuracy on the validation during training with different loss functions. It can be seen from the figure that our loss function not only achieves better results, but also leads other loss functions at the beginning of training. The results of the multi-labe methods are significantly better than the single-label methods, which shows the effectiveness of our idea of using multi-label to solve top-k problem.
Analysis of the convergence. We illustrate the state of convergence by analyzing the objectives of the loss function. The first objective is top-k classification which aims to maximize (sp 8x)- sio/sshoww the state of 8, and 8, from never converged to converged. Fig [gg44ashonn shows the distribution of 5p and 5n when epoch=0 (training is not started). From the figure, it can be seen that sp and sn are distributed near (0,0). When epoch=1, Fig 4b shows that the distribution slowly moves to the left and up, which indicates that the objective converging. When epoch=20, Fig 4c) shows that the distribution of sp is concentrated in [0, 15], while the distribution of s, is concentrated in [-5, 2.5), which is in line with the goal of maximizing (s,, 8.). At the same time, compared with Fig|4a and Fig (bb the distribution of Fig 4c is more concentrated, which shows that the convergence of the objective function completed.
Another objective label ranking which aims to maximize (s,, sp).P P Fig also shows the state of sy and 8p from never converged converged. Fig 5al shows that the distribution of s, and Sp W is concentrated around (0,0) when the initial state epoch=0. Fig shows that the distribution of sy and sp vy shifts upward when epoch=1, which shows that the optimization objective (sy sp) is converging. Fc shows that 8y is distributed between [8. 15]. and sp by is distributed between [0, 10]. Compared with 5a and 55g the distribution in Fig 5c is more concentrated, and the objective is greatly optimized, so the convergence is complete.
13


Reviews:
[{"review_id": "llZrLAginl", "reviewer": "Reviewer_ALCH", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes an approach that performs top-k classification with label ranking. The proposed approach has three stages. The first stage trains the base classifier. The second stage relabels training data using one ground truth label and k-1 most likely labels. The third stage trains the top-k classification model by introducing a new top-k loss function. Using text and image datasets, the paper evaluated the effectiveness of the proposed approach.", "summary_of_the_review": "The proposed approach is technically somewhat shallow.\nTheoretical analysis for hyperparameters should be added to the paper.\nTo show the generality of the proposed approach, more machine-learning approaches should be tested in the experiment.", "strength_and_weaknesses": "Strength:\n- Simple model to be easily implemented.\n- General approach to accommodate various machine learning approaches.\n- Extensive experiments are performed in the paper. \n\nWeakness:\n- Technical depth of the proposed approach is not so deep.\n- More machine learning approaches should be tested to evaluate the effectiveness of the proposed approach used in the proposed approach.\n- The impact of hyperparameters should be theoretically analyzed.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "i8XNRrjcqsx", "reviewer": "Reviewer_6iPJ", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a three-stage approach to learning top-k classification with label ranking concerned during modeling. ", "summary_of_the_review": "This work has not touched the core problem of top-k classification (e.g., non-differentiable) and some related (e.g., Sparsemax+top-K truncation) works are absent.\n\nSolving top-k classification from the view of multi-label perspective is interesting to some audiences but perhaps needs more technical supports.\n", "strength_and_weaknesses": "Pros:\n1. The proposed method is simple and easy to follow.\n2. The experiments show that the proposed method outperforms other baselines.\n\nCons:\n1. My first concern is that training base classifiers to obtain the prediction uncertainty is time-consuming which is however not discussed in the paper. \n2. The three-stage approach which relabels the training data with multiple labels and enhances the ground-truth one is reasonable but not novel. The formulated objective, i.e., Eq. (5), is correct but is not very inspiring to me.\n3. Following 1&2, one can alternatively do dropout at stage 1 to have the uncertainty measure.\n4. The noisy label was mentioned in the introduction as a motivation but was not discussed later.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "_KPzv2BIS6", "reviewer": "Reviewer_Pewp", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies the top-k classification problem. Different from the existing top-k classification setting, in this paper the labels (including the ground truth label) are ranked. It proposes a three-stage approach to this problem: \n1. base classifier construction, \n2. training data construction with relabeling, and \n3. training top-k classifier with a revised order-sensitive loss function. The proposed approach is tested on text and image datasets. ", "summary_of_the_review": "- The motivating example in the introduction doesn't properly highlight the setting of the paper. Indeed that in the human-in-the-loop scenario,  we would like to present the annotators the most relevant data for annotation; But in this scenario, isn't the setting the same as retrieval or, even more so, label ranking, where only top-k ranked items are shown? I think coming up with an appealing motivating example is important, especially as the experiments are based on syntactic datasets. (By which I mean the datasets are augmented to study the setting.)\n\n- The setting considered in this paper is in fact exactly the same as the one in *label ranking*, when the evaluation focuses on top-ranked label(s). The authors should consider them in the related work. \n\n- Between Eq. (4) and (5): \"should **be** as large as possible\".\n\n- I was expecting more discussion about the modified loss function in Eq. (5). For example, when you put it on top of BERT, what does the training process look like? Does it converge fast?\n\n- Experiments. The results of LSAN and C-Tran are so underwhelming, that one would ask why they were included and why the authors called them state of the art for the respective tasks. No better baseline could be found?", "strength_and_weaknesses": "Strength:\n- The writing of this paper is clear. The proposed idea is presented in a way that even for readers not in the field could follow easily. \n- The empirical study has considered both text and image datasets, with different characteristics. And the proposed approach has produced reasonable results. \n\nWeaknesses:\n- I think the proposed setting, order-aware top-k classification, should be better motivated. \n- One of the key technical contributions is the loss function used to train the top-k classifier. More discussion about this loss function is desired.\n- I think the related work can also be strengthened. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "BoVn9P_p3Bi", "reviewer": "Reviewer_iYfY", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper studies top-k classification problem. Observing existing top-k classification methods neglect the ranking of the ground truth label among the predicted k labels, the authors proposed to take label ranking into consideration. Specifically, a new 3-stage approach was proposed to tackle this problem. In the first stage, an ensemble method is to train several base classifiers. In the second stage, these base classifiers are used to predict class probabilities, which are then averaged to create top-k most likely labels (including the ground truth label) for each example. In the third stage, a novel loss function that considers the ranking of the ground truth label is used to train a top-k classifier. The experimental results show that the performances of the proposed method are significantly better than those of existing methods on several text and image datasets.", "summary_of_the_review": "I vote for a weak accept since I think the merits outweigh the weak points.\n\n**Questions.**\n\n1. How many trials were conducted in the experiments?\n\n**Comments.**\n\n1. In Eq. 7, \"pso\" -> \"pos\".", "strength_and_weaknesses": "**Strong points.**\n\nThe consideration of the ranking of the ground truth label is well-motivated and is of significance for top-k classification problems. The proposed 3-stage method is relatively intuitive, and was clearly described. The experimental results show the method is very effective.\n\n**Weak points.**\n\nThe proposed 3-stage method is more complex and requires more computations. It is not clear from the experiments whether the better performances were due to the more complex model and more computations. Comparing with other ensemble methods might help make this point clearer. Nevertheless, it seems that the new loss function in the third stage did help improve performances. More theoretical understanding of the proposed loss function is needed.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject