Modeling label correlations implicitly through latent label encodings for multi-label text classification
Authors: Zhizhong Zeng
Yufen Liu
Wenpeng Gao
Baihong Li
Ting Zhang
Xinguo Yu
Zongkai Yang
Venue: ICLR 2022 Submitted (2022)
Keywords: multi-label text classification
multi-label classification
label correlation
label embedding

Abstract: Multi-label text classification (MLTC) aims to assign a set of labels to each given document. Unlike single-label text classification methods that often focus on document representation learning, MLTC faces a key challenge of modeling label correlations due to complex label dependencies. Previous state-of-the-art works model label correlations explicitly. It lacks flexibility and is prone to introduce inductive bias that may not always hold, such as label-correlation simplification, sequencing label sets, and label-correlation overload. To address this issue, this paper uses latent label representations to model label correlations implicitly. Specifically, the proposed method concatenates a set of latent labels (instead of actual labels) to the text tokens, inputs them to BERT, then maps the contextual encodings of these latent labels to actual labels cooperatively. The correlations between labels, and between labels and the text are modeled indirectly through these latent-label  encodings and their correlations. Such latent and distributed correlation modeling can impose less a priori limits and provide more flexibility. The method is conceptually simple but quite effective. It improves the state-of-the-art results on two widely used benchmark datasets by a large margin. Further experiments demonstrate that its effectiveness lies in label-correlation utilization rather than document representation. Feature study reveals the importance of using latent label embeddings. It also reveals that contrary to the other token embeddings, the embeddings of these latent labels are sensitive to tasks; sometimes pretraining them can lead to significant performance loss rather than promotion. This result suggests that they are more related to task information (i.e., the actual labels) than the other tokens.

Under review as a conference paper at ICLR 2022
MODELING LABEL CORRELATONS IMPLICITLY THROUGH LATENT LABEL ENCODINGS FOR MULTI- LABEL TEXT CLASSIFICATION
Anonymous authors Paper under double-bline review
ABSTRACT
Multi-label text classification (MLTC) aims assign a of labels to each given document. Unlike single-label text classification methods that often focus on doc- ument representation learning, MLTC faces a key challenge of modeling label correlations due to complex label dependencies. Previous state-of-the-ane works model label correlations explicitly. lacks flexibility and prone to introduce inductive bias that may not always hold, such as label-correlation simplifica- tion, sequencing label sets, and label-correlaton overload. To address this issue, this paper uses latent label representations to model label correlations implicitly. Specifically, the proposed method concatenates a set of latent labels (instead actual labels) the text tokens, inputs them to BERT, then maps the contextual encodings of these latent labels to actual labels cooperatively. The correlations be- tween labels, and between labels and the text are modeled indirectly through these latent-label encodings and their correlations. Such latent and distributed correla- tion modeling can impose less a priori limits and provide more flexibility. The method is conceptually simple but quite effective. It improves the state-of-the-ar results on two widely used benchmark datasets by large margin. Further exper- iments demonstrate that its effectiveness lies in label-correlation utilization rather than document representation. Feature study reveals the importance of using la- tent label embeddings. It also reveals that contrary to the other token embeddings, the embeddings these latent labels are sensitive to tasks; sometimes pretraining them can lead significant performance loss rather than promotion. This result suggests that they are more related to task information (i.c., the actual labels) than the other tokens.
INTRODUCTION
Multi-label text classification (MLTC) aims to associate each text sample with set of labels. It can be applied wide spectrum of scenarios, including text categorization (Schapire & Singer 2000),
information retrieval (Gopal & Yang 2010) and web mining (Zhang & Zhou [013 Liu et al. 2020 Unlike single-label text classification, MLTC text usually associated with more than one label. Typically, these labels correlate with each other. Utilizing their correlations effectively
benefit
their predictions significantly (Tsoumakas et al.) 2009 Gibaja & Ventura (015).. However, due to the high-order nature of these correlations, it is challenging task (Zhang & Zhou 2013).
There are different ways to deal with this challenge. relatively earlier line of works models the correlations independently of text context (Prabhu & Varma, 2014 Jernite et al.) 2017 Peng al.) 2018 (Sngh et al.) 2018). A representative class in this line is the tree-structure methods. Typically, they build a hierarchical structure for the label space statistically or according taxonomy. During predictions, they usually utilize the modeled label correlations through some regularization methods (Peng et al. 22018) Singh al. 2018). Since the structure of labels that heavily limits the correlations is defined in advance, they have introduced a strong inductive bias that suggests that labels correlate with each other following tree structure unrelated to text context. But in real tasks, the labels and the text often correlate with each other closely in complicated way. The correlations between
'The source cade is provided in the supplementary file.

---

Under review as a conference paper at ICLR 2022
labels can be quite different depending on text contexts. It may not be appropriate to simplify their correlations a priori too much.
Another representative class of works in this line is the label embedding methods. In computer vi- sion, researchers have used label embeddings to capture label correlations to enhance performances on zero-shot learning and other tasks (Palatucci et al.) 2009 Yogatama al. 2015: Ma al. 2016). Inspired by them, |Wang et al. (2018) introduce joint label embeddings to model label relations and label-text relations to enhance text classification performances. Their work extended by Xiao al. (2019). In these works, the correlations between labels are modeled by the similarities between label embeddings; they are learned adaptively. Such correlation modeling is more flexible and expressive compared with the tree structure methods. And they do not only model label correlations but also model label-text relations. However, they still model label correlations independently of context.
A relatively recent line of works not only embeds labels but also encodes them contextually through mechanisms. Such mechanisms allow them to jointly model the correlations between labels, and between labels and the text context with less a priori limits. This line of works can be further divided into three major classes: sequence-to-
attention-base based encoding (or encoding-decodiin
sequence (Nam et al., 2017, Yang et al., 2018)), sequence-to-set (Yang et al. 2019 Tsai & Lee 2020) and joint encoding (Zhang et al. (2221).
The sequence-to-sqquemee methods are first in the line. They are based on the well-known sequence- to-sequence framework. Generally, they encode text by deep neural networks and then decode them to targeted labels one by one. They lead breakthrough in this area by eliminating the inductive bias that label correlations are independent of text context. But due to the nature of the classic seq2seq framework, they also introduce another inductive bias: there is an order among the labels associated with each text. However, it is difficult to define such orders appropriately. And this issue critical to these methods (Vinyals et al. 2015).
To overcome the difficulties in defining appropriate label orders, the sequence-to-set methods (Yang al. 2019 Tsai & Lee, 2020) are proposed. Different from seq2seq methods that calculate loss each time label is generated, they use reinforcement learning to delay loss calculation to the time when the whole label sequence is generated. Since the label sequences are constructed adaptively during training by reinforcement learning, it can obtain them more reasonably. However, the labels are essentially set rather than a sequence. Constructing the label sequence adaptively still regards the labels as sequence rather than a set. It does not eliminate the label-sequence inductive bias completely.
Very recently, new method (Zhang et al. 202 (it is the most related work to ours) encodes the text and the labels jointly within a unified encoding process using BERT (Devlin et al. 2018). performs extra tasks on the label encodings to further model label correlations. By doing so, it removes the label-sequence inductive bias and obtains state-of-the-ar results. But their work still suffers from the following three issues. (1) Their extra tasks are designed based the following consideration: in the context of a given text, the co-occurred labels should have stronger mutual correlations. Since unrelated but common labels can co-occur frequently, we think this inductive bias may still be excessive. We call this issue label-correlation overload. (2) Their extra task plcp models label-correlations pairwisely rather than globally: it may also be an oversimplification. (3) Although jointly encoding actual labels and texts has addressed the label-sequence inductive bias elegantly, can be less effective and even impossible when the number of actual labels is very large. This issue limits its applications heavily.
All the above methods share a common fundamental characteristic: they model label correlations explicitly. As one may observe from the above discussions, due to the high-order, contextual and non-sequential nature of label correlations, it difficult to model them explicitly without introduc- ing improper inductive bias. To overcome this difficulty, this paper uses latent label encodings to model actual labels and their correlations implicitly. Specifically, it pads a set of latent labels rather than actual labels in front of the inputs to BERT, and then maps the contextual encodings of these latent labels to actual class labels cooperatively. The label-label and label-text correlations regarding the actual labels are modeled indirectly through these latent-label encodings and their correlations. Since the correlations between these latent-label encodings, and their relations with actual labels and the text context are all trained adaptively, it imposes fewer priori limits and provides more flexibility.
2

---

Under review as a conference paper at ICLR 2022
Furthermore, by encoding latent labels rather than actual labels, our method can well address the forementioned three issues of the most related work to ours (Zhang et al., ) (1) Our label- correlations are learned adaptively with few priori limits, this avoid label-correlation overload introduced by empirical extra tasks; (2) By extensively utilizing latent-label encodings and their relations in each label's prediction, we address the oversimplification of pairwise label-correlation modeling. (3) The number of latent-label is an adjustable
that can be adjusted to accommodate specific tasks, usually is much smaller than the number of actual labels. This feature addresses the issue of lassi well.
Gereral-purpose text classification algorithms are not specially designed for MLTC, such as TexICNN (Kim 2014) and BERT (Devlin et al., 2018). They do not consider label correlations. In a sense, they can also be regarded as modeling label correlations implicitly. We think that the advantage of our algorithm over them in capturing label correlations lies in the following aspects: compared with their latent neurons, our latent label encodings are structural and high-orderly cor- related. These features make it more expressive in modeling label correlations, just like the actual label encodings do.
We summarize the major contributions of this paper follows:
(1) To the best of our knowledge, the previous state-of-the-art works for MLTC regard label correlations as key challenge. They model label correlations explicitly, which lacks flexibility and hence is prone to suffer from inappropriate inductive bias. Differently, we use latent label encodings to model label correlations implicitly. The difference between work and the previous state-of- the-art works in modeling label correlations is fundamental, it provides a new line of thought.
(2) Our algorithm outperforms the state-of sate-of-thee-re the--rt-rttt results two widely used benchmark datasets by a large margin. Feature study shows that significantly outperforms using actual label encodings for classification. Error analysis indicates that the advantage of algorithm lies in label-correlation utilization rather than document representation. The inspiring results demonstrate the great potential of modeling label correlations implicitly.
(3) Feature study further reveals that contrary to the other token embeddings, sometimes pretraining latent label embeddings can lead to significant performance loss instead of promotion. This suggests that they are more related to task information (i.e., the actual labels) than the other tokens.
2
METHODOLOGY
Given a text T containing m sequential word tokens Wo. W1 Wm My E V 300sism-11 a MLTC task aims to determine label set L.S = (To, 41 .1........ 1p-1..,,,p.... 11.1, E L(O 1 n). m is the length of the text, LS is the set of labels associated with the text, p is the number of labels in ISS, V is the vocabulary of word tokens, is the complete set of all the candidate labels, n is the cardinality of IL. Our method to solve this problem is straightforward: it pads k latent labels = (Ug. Ill................... .., .,,l1) in front of the input BERT and maps the encodings of these latent labels to actual labels through a dense neural network. Figure T shows its framework. In the following, we describe them detail.
Input
As illustrated in Figure [1 the input is a concatenation of latent labels and text tokens. Specifically, let the latent labels be 7l (Mo, .l....................... 14-11), the tokenized text be (wo, Wi...., Wm-1), the input then is (Mo, .l....................... -1,-1, 1, wo, wi,... Wm-1, [SEP]). These latent labels do not exist in the vocabulary of BERT; their initial embeddings are generated randomly. The number & of latent labels a hyperparameter. Since all these latent labels share the same position embedding 0, they are a set rather than a sequence. Usually, the number of latent labels is much smaller than that of the actual labels. It makes the input to BERT more concise and casier to train compared with using actual label embeddings. Also, the condensed representations of latent labels makes it easier for them to correlate with each other closely.
BERT within-task further pretraining (optional)
The setting of the BERT part is the same as traditional BERT. is well known that in in-domain further pretraining BERT can enhance its adaptability to the specific contexts. So, employ optional
3

---

Under review as a conference paper at ICLR 2022
D) Plya 1
Output
Flatten
BERT
Input
H1
11
11
Wo
W
mm-/
[SEP]
Latent labels (If)
Tokenized text (sc)
Unpretrained
Pretrained
Figure I: The framework of LLEM.
within-task further pretraining process that follows the within-task pretraining procedure introduced in (Sun et al. 2019). Since the latent labels are task-targeted, their embeddings are not pretrained during this process as other tokens do. As shown by our experiment, pretraining them can decrease performance significantly (see (.2)). Note that this is only an optional process; our algorithm can still outperform the state-of-the-as results by a large margin without this process.
Output and Loss
As illustrated in Figure 1 let the contextual encodings of these latent labels be Hi [ha,, hy, ..... h I. They are first flattened and then fed to a single layer dense network to obtain the predictions:
ÿ == sigmoid W flatten bo)
W. E kddxK and b, E R" are the trainable weights and bias of the dense neural network re- spectively, d is the dimension of the encodings, is the number of actual labels. For the recent state-of-the-art LACO algorithm (Zhang et al. 202T), its final prediction on each label is determined only by the encoding of the label itself. On the contrary, our prediction each label is determined by all the encodings of these latent labels cooperatively. The advantage of our method is twofold. Firstly, utilizes only the label encodings but also their correlations. Secondly, the encodings of these latent labels are shared among all the predictions; compared with predicting each label separately, this general sharing can enhance robustness.
Like previous works, we use Binary Cross Entropy as the loss function for the multi-label text classification tasks: Ciiite - la Inp, (1 q:) In p.)l
(2)
where P(ul D) is the probability of label i predicted by the model (ys E L), and gi E {0,1)
indicates whether y LS.
EXPERIMENT
3.1 DATASETS
We evaluate our Latent Label Encoding Method (LLEM) on two widely used multi-label text classi- fication datasets: AAPD (Yang al. 2018) and RCV1-V2 (Lewis al. 2004). The AAPD (Arxiv

---

Under review as a conference paper at ICLR 2022
Academic Paper) dataset includes 55,840 abstracts of papers involving 54 topics in computer sci- ence. RCV1-V2 (Reuters Corpus Volume I) consists of 804,414 manually categorized newswire stories involving 103 topics. We followed the same data-splitting as Yang et al. (2018).
3.2 EXPERIMENT SETTING
We implement our model using Pytorch and run it 1 NVIDIA Tesla V100 (32G). The model is based on the English casssud versions of BERT, just as other BERT-based MLTC models do (Cai et al. 2020, Zhang et al. 2021). The BERT-base model can be further within- task pretrained on the training set to enhance its adaptability to specific contexts (optional) (Sun et al. 2019).
The syyper-parametet of the optional within-task pretraining of the BERT-based model are set ac- cording to Sun et al. (2019). Specifically, the batch size is 32, the max sequence-length is 128, the warm-up steps are 10,000, the learning rate is 5e-5, and the train steps 100,000.
The training hhper--praaann parameters of our method are tuned by monitoring Micro-FI score on the valida- tion set. They are set as follows. The number of latent label is 15 and 20 for AAPD and RCV1-V2 respectively, the batch size is 24, the dropout probability is 0.1, and the max sequence-lengtt is 512. We use Adam method, and set 81 0.9, (32 0.999 respectively. For sequences longer than 512 (including latent labels), we use head+tail truncation proposed by |Sun et al. (2019) with trunc medium 128. Following (Hwwrdd & Ruder (2018) and [unn al. (2019), we utilize layer-learning rate decay strategy with decay factor = 0.95. We also use slanted triangular (Howard & Ruder 2018) with base learning rate set as 5e-5, and the warm-up proportion as 0.1. The total training epochs are set as 100 and 8 for AAPD and RCV1-V2 respectively. We save the best model on the validation set for testing-
3.3 EVALUATION METRICS
For MLTC tasks, a text associated with more than one label. The objective not to find such "a" label correctly, but to find them "all" correctly. Therefore, the recall metric is also important in addition to precision. So instead of precision the comprehensive metries Micro-Fi and Hamming Loss are the most important metrics for MLTC tasks (Chen et al.) | Yang et al.) 2018). Micro-F1 is the weighted average of Fl scores each class label, while Hamming Loss represents the fraction of misclassified instance-label pairs. They are good metrics to evaluate the overall performance of an MLTC algorithm. Macro-F1 is more sensitive to performance on low-frequence labels. Due to the effective usage
deficiency of data, the performance low-frequee frequency labels is highly dependent of label correlations. Therefore, Macro-Fl is a good tool to analyze the capacity of an algorithm in utilizing label correlations. Besides these metrics, Micro/Macro-Prectte Precisson and Micro/Macro-Ress are also reported for reference.
3.4 COMPARING ALGORITHMS
We compare our algorithm with a wide spectrum of competitive algorithms, including the well- known pereral-purpose text-classification algorithms: Binary Relevance (BR) (Boutell al. 2004). TexICNN (Kim 2014) and BERT (Devlin et al. 2018);; label embedding algorithms: LEAM (Wang al. 2018 and LSAN (iiao al., 2019); the sequence generation model (SGM) (Yang et al. 2018 ; the deep reinforced sequence-to-se to-set model (Yang et al. 2019); the OCD framework (Tsai & Lee. 2020); the ML-Reasoner (Wang al. 2021); b;; and two recent state-of-the-arr BERT based
algorithms: the hybrid-BERT (Cai et al. 2020) and LACO (+plcp, +clcp) (Zhang et al. 2021). To the best of our knowledge, these algorithms have provided all the previous state-of tfethharrt results on the two datasets regarding major metrics.
Among these algorithms, LACO (+plep, +clep) is most related to our Latent Label Encoding Method (LLEM). Both our works encode texts and labels jointly and are based on the BERT-base pretraining model. The key differences are as follows: (1) LACO encodes the actual labels in the same space of text tokens to capture label-label and label-text correlations. On the contrary, our method encodes the latent labels rather than the actual labels, the correlations regarding actual la- bels are modeled indirectly through these latent-label encodings and their relations. (2) In their work, the encodings of labels are used for prediction (and extra tasks) separately; differently, in work correlations between the latent label encodings utilized globally and shared by all the
5

---

Under review as a conference paper at ICLR 2022
Table 1: Performance of different algorithms the datasets. Evaluation metries include: Ham- ming Loss (HL), Micro (Mi-) and Marco (Ma-) average Precision (P), Recall (R), F1-Score (FI). Symbol i denotes higher better, and vice versa.
AAPD dataset
RCVI-V2 dataset
Algorithm
HL
Mi- Mi-P/R/FIT
Ma- P/R/FI1
HL
Mi- Mi- P/ F//1/
Ma- P/ P/R/FI1
BR 2004
0.0316
64.4/648/645
0.0086
90.4/81../
85.8
Tex Kim 2014
0.0256
84.9/5455/6.4
0.0089
92.21 79.8 85.5
DeVT al 2018
0.0224
68.7738.773.44
68.7/52.1 57.2
0.0073
92.7 83.2/877
77.3 61.9/ 66.7
LEAM Wang
2018
0.0261
76.5/ 59.6/67.0
52.47
40.3 45.6
0.0090
87.1 84.1 85.6
69.5 / 65.8 67.6
LSAN Xina el al.
2019
0.0242
77.7 64.6/
67.6 47.2 53.5
0.0075
91.3 84.1 /7.5
74.9 65.0/ 68.4
SGM Yang
2018
0.0251
65.9/
0.0081
86.7 85.0/ 86.9
Seq2Set Tibk et 201
0.0247
73.9/ 70.5
0.0073
90.0 85.8 87.9
TsDi Lec
72.0
58.5
ML-R Wang et Bl
112
0,0248
72.6/7188 72.2
0.0079
89.0 85.2 87.1
HBLA caeee 2020
0.0223
76.8 /7,4
0.0063
90.6 89.2 89.9
LACO Zhang
A021
0.0213
80.2 / /9.6/744
70.4 04.0/44...0. 59.1
0.0072
90.8 / 85.6 / 88.1
75.9
6 69.2
LACO PhoppZZhangee al.
2021
0.0212
79.5 70.8 74.9
68.47 55.8
59.9
0.0070
90.8 86.2 88.4
76. 66.55
66.5 69.2
LACO+cleg Zhing
0.0215
78.9 / 70.8/ 74.7
71.9 56.6 611.2
0.0070
86.4 88.5
77.6/71.5/73
ILEM
0.0207
80.1 7 71.67 75.6
66..77 57.67 61.5
0,00556
92.8 89.37 91.0
78.3775.37
LLEM+epre
0.0203
80.9 79.9/717/766
71.8/ 63.0
0.0056
93.7 / 88.2 90.8
79.2 1122/11//73..
predictions; (3) Our work has not performed any extra tasks to further capture label correlations as Zhang et al. (2021); instead, our work has performed an optional unsupervised within- further pretraining.
RESULTS AND ANALYSIS
In this section, provide the experimental results, and then perform in-depth analysis them and our method.
4.1 EXPERIMENTAL RESULTS
We list the experimental results of our algorithms and the compared baselines on the two datasets in Table To the best of our knowledge, the results of the compared baselines include all the previous state-of-the-arr results on the two datasets regarding major metrics. Our algorithms for comparison are follows. (1) LLEM; (2) LLEM+ LLEM that have all the token embeddings (except for the latent label embeddings) within-task pretrained.
From the results, one can observe that our algorithms are quite competitive. It outperforms the previ- ous state-of-the art results on both datasets by a large margin regarding the two major metrics Micro- Fl and Hamming Loss. Specifically, for the AAPD dataset, LLEM + wpre outperforms the previous best Micro-F1 (provided by LACO+plep) by 1.47%, previous best Hamming Loss (also provided by LACO+plcp) by 4.25%; for the RCV1-V2 dataset, LLEM+ outperforms the previous best Micro-F1 (provided by hybrid BERT) by 1.00%. previous best Hamming Loss (also provided by hybrid BERT) by 11.11% 1.11%. 1.. Even without within-task pretraining, our base LLEM still outperforms AAPD by 0.93% and 2.42%, the corresponding state- of-the-art results on RCV1-V2 by 1. 1.22% and 11.1 1%, respectively. This indicates that our method can achieve substantial improvements by modeling label correlations implicitly through latent label encodings.
the corresponding state-of-the-art results
4.2 FEATURE STUDY
The objective of the feature study is to evaluate whether using latent labels is value-added compared with using actual labels, as well as investigate the effect of pretraining on latent label embeddings. To this end, we compare label embeddings instead of latent label embeddings and predict each label based on its encoding; we name it as Actual Label Encoding Model (ALEM+ The second baseline is LLEM+ wpre_all ie., LLEM that have all the token (including the latent label) embeddings within-task pretrained. Besides the mentioned difference, all the settings for these compared algorithms the same. We
algorithm LpEM+ with two baselines. The first baseline using actual

---

Under review as a conference paper at ICLR 2022
Table 2: Feature study results. Comparing L.EM+ with two baselines A wpre and LLEM+wpreead aall to study the value of latent label encoding, and the effect of pretraining on latent label embeddings.
AAPD
RCVI-V2
Model
HL 4
Mi-F1 f
Ma-F1
HL 4
Mi-F1 1
Ma-F1
+LEM+wpre
0.0203
76.01
63.05
0.0056
90.84
73.84
ALEM wpre
0.0227
73.84
59.53
0.0058
90.51
71.93
L
0.0214
74.83
59.48
0.0057
90.59
73.21
compare these algorithms on the two datasets based on the major metrics, i.e., Hamming Loss, Micro-Fl and Macro-F1. The results are listed in Table
The results show that our algorithm significantly outperforms w It indicates that it is value-added by modeling label correlations implicitly. Compared with Micro-F1, the gap between the two algorithms is even bigger on the Macro-F1 metric, this means the advantage of latent label encoding more prominent for less frequent labels, sugesting that it is more superior in label- correlation utilization. The advantage of our algorithm is also more notable on the AAPD dataset. We think capturing label correlations is more difficult for this dataset due its much smaller size and more professional text type. This makes the capacity of our method in label-correlation utilization more valuable and hence can lead to more prominent results.
One can also observe that LLEM+ outperforms w aall significantly on the more challenging dataset AAPD. We think it is also attributed the difficulties in capturing label- correlations in AAPD. In such difficult cases, the latent labels should relate to actual labels more closely to better capture their correlations and contribute their predictions. But pretraining the latent label embeddings without knowing the actual labels cannot contribute to this purpose at all; it can even misguide the latent-label embeddings and make the connection between latent labels and actual labels more difficult. This result suggests that compared with the other token embeddings, the latent label embeddings are sensitive to specific tasks; they are more related to task information (i.e., the actual labels) rather than texts in certain cases.
4.3 ERROR ANALYSIS
LACO (and its extensions LACO+plep, LACO+clep) is the latest tfte-off-hh-ar baseline. the most related work to our method. LACO+plep also has provided the previous best results on AAPD, which is more challenging in label-correlation utilization due to its much smaller size and more pro- fessional text type. In this subsection, we further perform two error analyses on AAPD to provide more in-depth comparisons between LLEM +wre and LACO+plcp on their label-correlation uti- lization. We use Macro-Fl Fl the comparing metric since it is more sensitive to label-correlation utilization, as discussed in 3.3. These two error-analy analyses compare the two algorithms on labels with different frequencies (F), as well as on samples with different number of labels (L.). The design in- tuition of these two analyses are as follows. Due to the deficiency of data, the performance on low frequency labels are always much lower, their performances are highly dependent on effective usage of label correlations (Yang et al. 2019 Menon et al. 2020)). And for intensive-labe samples, it is difficult predict all their labels correctly and completely. But the co-existing labels can contribute to each other's 's predictions through their correlations. Therefore, an algorithm that can make effec- tive usage of tabel-correlations should be capable of providing good results on low-frequency labels, as well as providing good results on intensive-label samples.
The first error analysis compares our LLEM+wpre with LACO+plep on labels with different fre- quencies. We divide the labels into four groups according to their frequencies, just as iZhang] ctt al. 202II: the first group: F > 4500; the second group: 1700 < F < 4500; the third group: 870 < F < 1700; and the last group: F < 870. The results are listed in Fig 2(a) As the frequency decreases, the performances of the two algorithms both decrease notably, indicating the difficulty in predicting fow-freyyency labels. The two algorithms provide similar results nigher-frequene la- bels (F > 1700), indicating that they have similar capacities in document representation. But for the lower-frequency labels (F 1700), algorithm provides notably higher performance, suggest-
7

---

Under review as a conference paper at ICLR 2022
ing that it is superior in label-correlation utilization. Taken together, these results suggest that the advantage our algorithm lies in label correlation utilization rather than document representation.
The second enror analysis compares LLEM+ wpre with LACO+plep samples with different num- bers of labels. We divide the samples into three groups according to their number of labels (L): the first group: L I 2; the second group: L = 3; and the last group: L 2 4. The results listed in Fig 2b) As the number of labels (L) increases, both performances decrease notably, indicating that it difficult to handle intensive-label samples. The algorithms provide close results on the sparsest-label samples (L < 2). For such sparsest-label samples, it is very close to single-label text classification. This further indicates that the two algorithms have similar capacities document representation. But for the more intensive-label samples (L 2 3). our algorithm again shows sig- nificantly higher performance, indicating that more effective in label-correlation utilization. To sum up, these results further demonstrate that the advantage of our algorithm lies in label correlation utilization rather than document representation.
08
0.8
LLEM-Mppr LACO-plcp
LLEM+ wpre LAC0+plcp
Macro-Fi F1
0.6
Macro-F1
0.6
0.4
0.4
0.2
0.2
450000
=700=F=4500 870 <T's 1700
F70
LS3
L-3
L24
(a) Error analysis on labels with different frequen- (b) Error analysis samples with different number of labels (L)
cies (F)
Figure 2: Comparative error analysis of our method (LLEM+wpre) and the state-of-the of-the-art (LACO+plep) the more challenging dataset AAPD.
RELATED WORKS
The research on multi-label text classification follows two directions: enhancing document repre- sentation and modeling label correlations.
Document representation. Document representation is common for all NLP tasks. Basically, the developments document representation for MLTC are inspired by other areas NLP. These in- clude RNN based models (Liu et al. combining CNN and RNN in models (Chen et al. [017 Lai et al. al. (0155).
2016). CNN based models (Kurata et al. 2016 Liu et al. (017),
2019 Adhikari et al. 2019), and menttionn-echaiismm based
models Yang al. 2016 You et al.
recently the powerful
pretrained BERT (Devlin et al. 2018) based methods (Cai et al. 2020 Zhang et al. 2021).
Modeling label-correlations. Different from single-label text classification (SLTC), multi-label text classification (MLTC) faces key challenge of modeling label correlations. There are two ma- jor lines of works this end. The first line of works model label-correlations separately to document representation. These include the methods transforming multi-label text classification to multi or bi- nary class text classifications (Tsoumakas & Katakis 2007 Read al. 2011). the tree-structure methods (Prabhu & Varmal [014]] Jernite et al.] 2017 Peng et al. 2018 Singh et al.) (018), the label embedding methods (Wang et al. 2018] Xiao et al. 2019), and the correlation neural network method (Xun et al. (220)). Since labels and the text often correlate with each other complicatedly, can be over-simplified to separate these two processes. The other line works integrates label- correlation modeling with document representation by encoding labels contextually with attention mechanisms. These methods include sequencc-to-ssqqueeee (Nam et al. 2017 Yang et al. 2018), sequence-to-se (Yang al.) 2019; Tsai & Lee, 2020), and most recently the joint encoding with multi-tasks (Zhang et al. 2021). Although they have overcame previous over- werr-immlificattoon induc- tive bias, they suffer from issues such as label-sequence and label-correlation overload.
8

---

Under review as a conference paper at ICLR 2022
6 CONCLUSIONS
How to model label correlations effectively is a specific and important challenge to multi-label text classification. Previous state-of-the-an works for MLTC model label correlations explicitly. It lacks flexibility and hence is prone to introduce inductive bias that may not always hold. Differently, this paper proposes method to model label correlations implicitly through latent label encodings. The difference between our work and previous state-of-the-art works in modeling label-correlations is fundamental; it provides a new line of thought. The method outperforms the state-of-the- art results on two widely used benchmarks by large margin. Analyses also show that the effecttivnness of our method lies in label-correlation utilization rather than document representation learning. The inspiring results demonstrate the great potential of modeling label correlations implicitly.
The other contributions of this paper include but are not limited to: the concepts of sparse- (SL) samples and intensive-label (IL) samples for analyzing label-correlation utilization; revealing the singular nature of latent label embeddings compared with the other token embeddings regarding pretraining.
In future works, we think it is intresting for us and the other researchers to investigate how the latent-label correlate with each other, with the actual labels and the context, and how can they be interpreted. They certainly can shed light on new methods of their design and training.
REFERENCES
Ashutosh Adhikari, Achyudh Ram, Raphael Tang, and Jimmy Lin. Docbert: Bert for document classification. arXiv preprint arXiv:1904.083900 2019.
Matthew Boutell, Jiebo Luo, Xipeng Shen, and Christopher M Brown. Learning multi-label scene classification. Patlern recognition, 370))1757-1771, 2004.
Linkun Cai, Yu Song, Tao Liu, and Kunli Zhang. A hybrid bert model that incorporates label se- manties via adjustive attention for multi-label text classification. leee Access, 8: 8:152183-152192, 2020.
Guibin Chen, Deheng Ye, Zhenchang Xing, Jieshan Chen, and Erik Cambria. Ensemble applica- tion of convolutional and recurrent neural networks for multi-label text categorization, 2017 international joint conference on neural networks (IJCN), pp. 2377-2383. IEEE, 2017.
Jacob Devlin, Ming- Wei Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.
Eva Gibaja and Sebastián Ventura. A tutorial on multilabel learning. ACM Computing Surveys (CSUR), 47(3):1-38 2015.
Siddharth Gopal and Yiming Yang. Multilabel classification with meta-level features. In Proceed- ings of the 33rd international ACM SIGIR conference on Research and development in informa- fion retrieval, pp. 2010.
Jeremy Howard and Sebastian Ruder. Universal language model fine-tuning for text classification. arXiv preprint arXiv: 1801v18001.611 16616,0.6 2018
Yacine Jernite, Anna Choromanska, and David Sontag. Simultaneous learning of trees and represen- tations for extreme classification and density estimation. In International Conference on Machine Learning. pp. 1665-1674, PMLR, 2017.
Yoon Kim. Convolutional neural networks for sentence classification. Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, pp. 1746-1751, 2014.
Gakuto Kurata, Bing Xiang, and Bowen Zhou. Improved neural network- multi-label clas- sification with better initialization leveraging label co-occurrence. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 521-526, 2016.

---

Under review as a conference paper at ICLR 2022
Siwei Lai, Liheng Xu, Kang Liu, and Jun Zhao. Recurrent convolutional neural networks for text classification. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 29, 2015.
David D Lewis. Yiming Yang, Tony Russell-Rose, and Fan Li. Revl: A new benchmark collection for text categorization research. Journal of machine learning research, 5(App):361 2004.
Jingzhou Liu, Wei-Cheng Chang, Yuexin Wu, and Yiming Yang. Deep learning for extreme multi- label text classification. In Proceedings of the 40th international ACM SIGIR conference on research and development in information retrieval, pp. 115-124, 2017.
Pengfei Liu, Xipeng Qiu, and Xuanjing Huang. Recurrent neural network for text classification with multi-task learning. arXiv preprint arXiv:160S.0m 05101, 2016.
Weiwei Liu, Xiaobo Shen, Haobo Wang, and Ivor W Tsang. The emerging trends of multi-label learning. arXiv preprint arXiv:201..1119, 2020.
Yukun Ma. Erik Cambria, and Sa Gao. Label embedding for zero-shot fine-grained named entity typing. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pp. 171-180, 2016.
Aditya Krishna Menon, Sadcep Jayasumana, Ankit Singh Rawat, Himanshu Jain, Andreas Veit, and Sanjiv Kumar. Long-tail learning via logit adjustment. arXiv preprint vrXiv:2007.07314, 2020.
Jinseok Nam, Eneldo Loza Mencía, Hyunwoo
Kim, and Johannes Furnkranz. Maximizing subset accuracy with recurrent neural networks in multi-label classification. In Advances in Neural Information Processing Systems, pp. 5413-5423, 2017.
Mark Palatucci, Dean Pomerleau, Geoffrey Hinton, and Tom M Mitchell. Zero-shot learning with semantic output codes. In Advances in Neural Information Processing Systems 22: 23rd Annual Conference on Neural Information Processing Systems 2009. Proceedings of a meeting held 7-10 December 2009, Vancouver, British Columbia, Canada., 2009.
Hao Peng, Jianxin Li, Yu He, Yaopeng Liu, Mengjiao Bao, Lihong Wang, Yangqiu Song, and Qiang Yang. Large-scale hierarchical text classification with recursively regularized deep graph-cnn. In Proceedings of the 2018 world wide web conference, pp. 1063-1072, 2018.
Yashoteja Prabhu and Manik Varma. Fastxml: A fast, accurate and stable tree-classifier for treme multi-label learning. In Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 263-272, 2014.
Jesse Read, Bernhard Pfahringer, Geoff Holmes, and Eibe Frank. Classifier chains for multi-label classification. Machine learning, 83(3)333--39, 2011.
Robert E Schapire and Yoram Singer. Boostexter: A boosting-bast based system for text categorization. Machine learning, 135-1663 2000.
Gaurav Singh, James Thomas, Iain J Marshall, John Shawe-Taylor, and Byron C Wallace. Struc- tured multi-label biomedical text tagging via attentive neural tree decoding. arXiv preprint arXiv:1810.01 01680, 2018
Chi Sun, Xipeng Qiu, Yige Xu, and Xuanjing Huang. How to fine-tune bert for text classification? In China National Conference on Chinese Computational Linguistics, pp. 194-206. Springer, 2019.
Che-Ping Tsai and Hung-Yr Yi Lee. Order-free learning alleviating exposure bias in multi-label clas- sification. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp- 2020.
Grigorios Tsoumakas and loannis Katakis. Multi-label classification: An overview. International Journal of Data Warehousing and Mining (IJDWM), 3(3):1-13, 2007.
Grigorios Tsoumakas, Ioannis Katakis, and Ioannis Vlahavas. Mining multi-label data. In Data mining and knowledge discovery handbook, pp. 667-685. Springer, 2009.
Oriol Vinyals, Samy Bengio, and Manjunath Kudlur. Order matters: Sequence to sequence for sets. arXiv preprint arXiv: 15l:::51119999 2015.
10

---

Under review as a conference paper at ICLR 2022
Guoyin Wang, Chunyuan Li, Wenlin Wang, Yizhe Zhang, Dinghan Shen, Xinyuan Zhang, Ricardo Henao, and Lawrence Carin. Joint embedding of words and labels for text classification. arXiv preprint arX1v:1805.041 2018.
Ran Wang, Robert Ridley, Weiguang Qu, Xinyu Dai, al. novel reasoning mechanism for multi- label text classification. Information Processing & Management, 58(2):102441, 2021.
Lin Xiao, Xin Huang, Boli Chen, and Liping Jing. Label-specific document representation for multi-label text classification. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 466-475, 2019.
Guangxu Xun, Kishlay Jha, Jianhui Sun, and Aidong Zhang. Correlation networks for extreme multi-label text classification. In Proceedings the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 1074-1082, 2020.
Pengcheng Yang, Xu Sun, Wei Li, Shuming Ma, Wei Wu, and Houfeng Wang. Sgm: sequence generation model for multi-label classification. arXiv preprint /806. 2018.
Pengcheng Yang, Fuli Luo, Shuming Ma, Junyang Lin, and Xu Sun. deep reinforced sequence- to-set model for multi-label classification. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 5252-5258, 2019.
Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alex Smola, and Eduard Hovy. Hierarchical attention networks for document classification. In Proceedings of the 2016 conference the North American chapter of the association for computational linguistics: human language technologies, pp. 1480-1489, 2016.
Dani Yogatama, Dan Gillick, and Nevena Lazic Embedding methods for fine grained entity type classification. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pp. 291-299 2015.
Ronghui You, Zihan Zhang, Ziye Wang, Suyang Dai, Hiroshi Mamitsuka, and Shanfeng Zhu. Atten- tionxml: Label tree-based attention-aware deep model for high-performance extreme multi-label text classification. Advances in Neural Information Processing Systems, 32:5820-5830, 2019.
Min-Ling Zhang and Zhi-Hua Zhou. A review on multi-label learning algorithms. IEEE transactions on knowledge and data engineering,
2013.
Ximing Zhang, Qian-Wen Zhang, Zhao Yan, Ruifang Liu, and Yunbo Cao. Enhancing label correlation feedback in multi-label text classification via multi-tase task learning. arXiv preprint arXiv:2106.03103, 2021.


Reviews:
[{"review_id": "iX8rgelBtu", "reviewer": "Reviewer_BUhV", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Recent interest for multi-label classification lies in how to model the label correlations. Instead of modeling the label correlations explicitly, this paper suggests that it is more effective to implicitly model the label correlations via latent labels. \n\nThe reported experimental results show that the proposed LLEM outperforms the other baselines. And the empirical study shows that pretraining latent label embeddings cannot improve the results. If I understand correctly, the latent labels can be considered as the expanded [CLS] tokens prepended to the sentences/tokenized text. Since the [CLS] token is intended to represent the sentence, pretraining the latent label embeddings may not capture the label correlations and lead to performance loss.\n\nWhat concerns me most is how the classification benefits from the latent labels. Modeling the label correlations explicitly is intuitive and explainable, such as the label embedding methods. It would be interesting to have more empirical analysis, such as how the number of latent labels influence the classification results. \n\nIn addition, the paper mentions the label-correlation overload, but lack of further discussions. More detailed theoretical and empirical study could be given, such as how LLEM learns the label correlations in an adaptive manner with few a priori limits.\n\n**Typo**\n\n*Given a text T containing m sequential word tokens w_0, w_1, . . . , w_{m−1}, w_i*, the subscript of the last token should be *m*", "summary_of_the_paper": "This paper addresses the task of multi-label text classification by modeling the label correlations implicitly. Different from the previous works that explicitly model the label correlations, such as the label embedding methods, this paper proposes modeling the label correlations via latent labels. The proposed method outperforms the baselines on two multi-label text classification benchmarks in the reported experimental results.", "summary_of_the_review": "This papers proposes a simple method to model the label correlations implicitly via latent labels. The reported experimental results show the proposed method is effective. Whereas the method lacks of theoretical and empirical analysis. In summary, this paper is not good enough for ICLR.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "YiejFalUhtM", "reviewer": "Reviewer_82Sn", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Strengths:\n- encouraging experiment results that show marginal gains over baselines on two small scale XMTC datasets\n\n## Weakness\n- lack of complexity analysys. What is the time complexity with respect to number of labels during training and inference?\n- How scalable is the proposed method? Can it be used in extreme multi-label classificiation problems where number of labels is million or more? Such experiments are missing.\n- The rationale of latent labels are not convincing. What does the latent labels mean? Does it serve as a semantic cluster that consists of multiple labels? How do you find such latent labels? Otherwise, how can you claim you are modeling the correlation between labels?\n- The experiment section should also mention the difference in model parameters. Are the proposed methods using more number of parameters?\n\n", "summary_of_the_paper": "This paper consider multi-label text classification problem and propose a cross attention Transformer encoder to model the correlation between latent labels and input text sequence. The hidden states of those latent labels are concatnate as input to layers of MLP for the classification head. The experiment results show marginal gain over the baselines.", "summary_of_the_review": "This paper propose a cross-attention Transfomrer architecture for the multi-label text classification problem. Such modeling architecture is not new in MLTC literature. In addition, the rational of latent labels are not techincal sound and justified. Time complexity with respect to number of labels is also missing. Finally, the author should consider experiment on large-scale multi-label datasets such as Wiki-500K and Amazon-3M. Given those reasoning, I would vote this paper being marginal below the acceptance threshold.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "WGfgoUSLms", "reviewer": "Reviewer_fS9V", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Strengths:\n* The model has been tested and compared against the LACO algorithm [1] that sets the SOTA on two widely-used multi-label text classification datasets: AAPD and RCV1-V2, and outperforms LACO using Hamming Loss and Micro-F1. \n* Especially, the proposed method has even better performance than the baseline LACO algorithm, on the low-frequency labels and intensive-label samples.\n\n## Weaknesses:\n* The paper's overall contribution and impact seem quite limited. The idea  is very similar to LACO [1], and the paper’s contribution is that it encodes the latent label rather than the actual label.\n* Baseline models: There are many recent papers on XMC (extreme multi-label classification), investigating how to utilize the label correlations. Although these papers focus on the MLC cases where there are thousands or millions of labels. These methods are also applicable to general MLC task. ECLARE [2] and vanilla BERT-based AttentionXML [3], without the tree-based hierarchical structure, might be very strong baseline models. I am curious to know the performance of these models. I might have misunderstood something here. The author can clarify the choice of baseline models or clarify my misunderstanding in the rebuttal.\n* Some questions on the algorithm itself:\n   * Label semantics is not utilized in the model design since the embeddings of the latent labels are randomly initialized.\n   * How to interpret the latent labels?\n   * K, the number of latent labels, is a hyperparameter. But how to choose K? There is no ablation study on tuning k in the experiments section.\n   * The authors mentioned LACO can be less effective and even impossible when the number of actual labels are very large. However, the authors only show the experimental results of AAPD and RCV1-V2 datasets, which have 54 and 103 labels respectively. There are popular datasets [4] in the XMC community, for example RCV1-2K, EURLex-4K and AmazonCat-13K. If the authors can show the proposed method is applicable to these XMC datasets as well. The advantage over LACO can be justified and the model can have even bigger impact in the XMC community.\n\n1. Enhancing Label Correlation Feedback in Multi-Label Text Classification via Multi-Task Learning\n2. ECLARE: Extreme Classification with Label Graph Correlations\n3. AttentionXML: Label Tree-based Attention-Aware Deep Model for High-Performance Extreme Multi-Label Text Classification\n4. http://manikvarma.org/downloads/XC/XMLRepository.html\n", "summary_of_the_paper": "The paper presents a method that uses latent label representations to model label correlations implicitly, for the multi-label text classification (MLTC) task. The method concatenates a set of randomly generated latent labels to input text tokens. Then the method uses this as the input to the BERT model. At last, the contextual encodings of these latent labels are used to generate predictions for the actual labels.\nThe model has been tested and compared against the LACO algorithm [1] that sets the SOTA on the AAPD and RCV1-V2 datasets, and outperforms LACO using Hamming Loss and Micro-F1. Especially, the proposed method has even better performance than the baseline LACO algorithm, on the low-frequency labels and intensive-label samples.\n\n1. Enhancing Label Correlation Feedback in Multi-Label Text Classification via Multi-Task Learning", "summary_of_the_review": "I would like to reject this paper since overall contribution and impact seem quite limited. Also, the authors neglect recent advances in related fields.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "IzrISpYs7Hv", "reviewer": "Reviewer_8DdA", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "- The main strength of this paper lies in its conceptually simple but very effective idea of latent label encodings, which is quite different from all previous methods using actual label encodings. The new LLEM method is simple and straightforward, and it naturally eliminates some ad-hoc designed tasks in the previous state-of-the-art method LACO, but surprisingly achieves better performance.\n\n- The latent label encoding reminds me of the P-tuning for acquiring a best-performing prompt (https://arxiv.org/pdf/2103.10385.pdf). Since neural networks are inherently continuous, discrete prompts can be sub-optimal. The P-tuning also leverages trainable continuous prompt embeddings to serve as prompts fed as the input to the pre-trained language models, and then optimizes the continuous prompts using gradient descent as an alternative to discrete prompt searching. They are two completely different NLP tasks, but seem to share some common insight, and they might provide a new line of thought for some interesting future research.\n\n- Figure 2 shows the LLEM outperforms LACO in every single group of data (divided by label frequencies or numbers), but corresponding discussions just present the observations of experimental results, and the better performance does not seem to easily lead to the conclusion that LLEM is superior in label-correlation utilization. As mentioned by the authors, some further work are still required to \"investigate how the latent-label correlate with each other, with the actual labels and the context, and how can they be interpreted\". \n\n- The paper was well written and organized, and there is an extensive survey on recent work of multi-label text classification.\n\n- Minor issues\n    - typos: \"intresting\" on Page 9 \n", "summary_of_the_paper": "This paper proposes a novel multi-label text classification method named LLEM that jointly encodes a document and latent labels (with smaller number than actual labels), and tries to better model label correlations implicitly and impose less a priori limits compared with previous state-of-the-art works. The method is conceptually simple but outperforms the state-of-the-art results on two widely used benchmarks.", "summary_of_the_review": "- It is an interesting and useful study on multi-label text classification.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "CqRru6rTCd0", "reviewer": "Reviewer_FMut", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "- Technically, the idea of appending multiple \"latent tokens\" to the beginning of each document is not new. As far as I know, it was first proposed in [1] (the BERT-XML model) for multi-label text classification and later in [2]. Note that in both papers, this is just a small trick instead of their key novelty, so there may be even earlier work already proposing this trick. That being said, the novelty of this submission is rather limited.\n\n- It is quite confusing why these \"latent tokens\" can be claimed as \"labels\" and can implicitly model label correlations. Essentially, they can be viewed as multiple [CLS] tokens or \"probes\" to text semantics. In [1] and [2], their explanation is that the label space is quite large (e.g., >10K), so one [CLS] token (i.e., a 768-dimension vector) may not be informative enough to predict the relevant labels. In your paper, I cannot see intuitive explanations or experiments supporting your claim of \"implicit label correlations\".\n\n- Some important baselines are missing, including XML-CNN [3] and AttentionXML [4].\n\n- Significance tests are missing. It is unclear whether your improvement in Tables 1 and 2 is statistically significant or not. Please run each experiment multiple times and report standard deviation or p-values.\n\n- In Section 4.2, it is confusing to append true labels to the input text sequence during training. No doubt this cannot perform well because the encoder already knows the labels and will not focus on the remaining part. However, during inference, you no longer have such label information.\n\n[1] Xun et al., Correlation Networks for Extreme Multi-label Text Classification. KDD 2020.\n\n[2] Zhang et al., MATCH: Metadata-Aware Text Classification in A Large Hierarchy. WWW 2021.\n\n[3] Liu et al., Deep learning for extreme multilabel text classification. SIGIR 2017.\n\n[4] You et al., Attentionxml: Label tree-based attention-aware deep model for high-performance extreme multi-label text classification. NeurIPS 2019.", "summary_of_the_paper": "This paper proposes to implicitly model label correlations in multi-label text classification. Different from previous studies (e.g., tree-based models) that describe label correlations explicitly, this paper appends \"latent labels\" to the beginning of each document and feeds it into a BERT classifier. These \"latent labels\" are randomly initialized, and the concatenation of their output is used for classification.\n\nThe authors conduct experiments on two benchmark datasets, AAPD and RCV1. Experimental results show that their proposed method outperforms several baselines for multi-label text classification. Ablation studies further show that using \"latent labels\" is better than using \"actual labels\" in their framework.", "summary_of_the_review": "The key idea has been proposed in previous studies, so the novelty is rather limited. The intuition is not well explained.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes an approach for multi-label text classification. The method constitutes appending few "label" tokens to the beginning of the text input instead of the traditional single <CLS> token. The paper shows improvements over a competitive baseline on two datasets.

Reviewers agree that the novelty and contribution of the paper are marginal. The method of appending extra "fake" tokens has been used in other works as a "trick". It is also unclear how adding a few extra tokens allow for the model to represent label dependencies better.

The authors did not respond to the reviews, so there was no further dicussion.
Decision: Reject