Towards Better Representations for Multi-Label Text Classification with Multi-granularity Information
Authors: FangFang Li
PuZhen Su
Junwen Duan
Weidong Xiao
Venue: EMNLP 2023 Findings (2023)
Keywords: Multi-label text classification
Text representation
Contrastive learning
Multi-granularity information
TL;DR: We present a multi-granularity information enhancement framework to improve text representation for multi-label text classification and transform the anisotropy problem in pre-trained language models.

Abstract: Multi-label text classification (MLTC) aims to assign multiple labels to a given text. Previous works have focused on text representation learning and label correlations modeling using pre-trained language models (PLMs). 
However, studies have shown that PLMs generate word frequency-oriented text representations, causing texts with different labels to be closely distributed in a narrow region, which is difficult to classify.
To address this, we present a novel framework $\textbf{CL}$($\underline{C}$ontrastive $\underline{L}$earning)-$\textbf{MIL}$ ($\underline{M}$ulti-granularity $\underline{I}$nformation $\underline{L}$earning) to refine the text representation for MLTC task. We first use contrastive learning to generate uniform  initial text representation and incorporate label frequency implicitly. Then, we design a multi-task learning module to integrate multi-granularity (diverse text-labels correlations, label-label relations and label frequency) information into text representations, enhancing their discriminative ability. Experimental results demonstrate the complementarity of the modules in CL-MIL, improving the quality of text representations and yielding stable and competitive improvements for MLTC.

Towards Better Representations for Multi-Label Text Classification with Multi-grann -lattlgranullrrtty Information
Fangfang Li', Puzhen Su', Junwen Duan'; Weidong Xiao²
School of Computer Science and Engineering, Central South University
Science and Technology on Information Systems Engincering Laboratory, National University of Defense Technology (lifangfang, 2ing,,21410 21471 1091, Jwduan]ecsu wwt.:/ee.uuuuuuccommemooo cn, wdxiao@nudt. edu. cn
Abstract
(coarse-grained), For example, L4 is jointly corre- lated with T2. Tn and Tn
Multi-label text classification (MLTC) aims to assign multiple labels to a given text. Previ- works have focused text representation learning and label correlations modeling using pre-trained language models (PLMs). How- ever, studies have shown that PLMs generate word frequency-oriented text representations, causing texts with different labels be closely distributed in narrow region, which is diffi- cult to classify. To address this, we present novel framework, CL(Contrastive Learning)- MIL (Multi-granulart granularity Information Learning), to refine the text representation for MLTC task. We first contrastive learning generate uniform initial text representation and incorpo- rate label frequency implicitly. Then, we de- sign multi-task learning module integrate multi- graaularimuu (diverse text-labels correla- tions, label-label relations and label frequency) information into text representations, enhanc- ing their discriminative ability. Experimental results demonstrate the complementarity of the modules in CL.-MIL, improving the quality of text representations, and yielding stable and competitive improvements for MLTC.
Previous methods solve MLTC task in diverse perspectives. Among them, some methods focus on fine-grained (local semantic features) (Kurata et al., 2016) and coarse coarse-grained (global semantic features) textual information (Liu et al., 2016), re- spectively. Besides, (Wang et al., 2018; Xiao et al., 2019) utilize joint embedding and label-wise at- tention to enhance the text-labels correlations, and other methods learn the correlations between la- bels (Nam et al., 2017; Yang al., 2018). These methods separately utilize information from dif- ferent granularities to improve the classification performance. However, learning information on single granularity is insufficient to derive effective text representations compared to keeping a proper balance of multi-granularity information.
Recently, PLMs (Devlin et al., 2019; Peters et al., 2018) have been proved powerful to generate high-quality semantic representations. Competi- tive and strong PLMs based methods (Zhang et al., 2021; Su et al., 2022) are proposed to explore la- bels correlations and knowledge between samples. Yet, (Gao al., 2019; Zhou et al., 2022: Etha- yarajh, 2019a) reveal that PLMs appear anisotropy problem after fine-tuned, leading text representa- tions collapse into a narrow cone space. Texts consisting of high-frequency words are gathered closely, even correlated to extremely low-frequency labels. When samples with low-frequency labels concentrate densely, it will become more difficult to classify and also aggravate the long-tail problem (Menon et al., 2021) due the lack of samples.
1 Introduction
Multi-Label Text Classification (MLTC) is a fun- damental task in the natural language processing (NLP) field which refers to assigning multiple rel- evant labels to each text. MLTC has been widely applied to various real-world scenarios like recom- mendation system (Guo al., 2016), sentiment analysis (Wang al., 2016) and information re- trieval (Yang and Gopal, 2012) and others (Jain et al., 2016; Papanikolaou et al., 2014; Zhang et al., 2014). In MLTC, there are diverse text-labels cor- relations. As shown in Figure 1, labels I,1, L22 are directly correlated to specific type of keywords and phrases (fine (fine-graine) grained) T2, Ta, whereas latent cor- relations are determined by multiple information
To address these, we propose multi-granulard information enhancement framework to improve the quality of representations and relieve the anisotropy problem. We first perform contrastive learning to implicitly introduce label frequency into the initial representations and refine the uniformity of it. It only focuses on target samples (correlated
Cooresppnndigg author

---

low-frequency labels), and push away source samples that share low label co-occurrence with labels in uarget sample according to the label distri- bution. Moreover, we adopt the Multi-g grauularryl Information Learning (MIL) module via multi-task learning to enhance the expressiveness of represen- tations. Specificallyy, MIL module explores direct and latent text-labels correlations through Dual- grained Interactive Learning (DIL.) task and mod- els relations (strong relate, weak relate and contra- dict) between labels by Constraint Label Relations Learning (CLRL) task.
Correlated Labels
Direct Latent
Tuo
Text Seuqence
Figure I: Illustration of different text-labels correlations. The direct correlations are determined by one specific type of key-words, while the latent correlations require a mix of multiple features.
Both tasks share the same encoder that jointly embeds text and labels to directly introduce la- bel frequency into representations, which provides MIL an access to transform the orientation of two differ-
label correlations, (Tsoumakas and Katakis, 2007) and (Read et al., 2011) used unique label combina- tions and classification chain. Seq2seq- -ased meth- ods (Nam et al., 2017; Yang et al., 2018) transform MLTC into label sequence generation tasks. These methods tackle MLTC task via different granulari- tics information, which strikes us the fcasibility of balancing multi-gra multi-granularity information to improve the expressiveness of text representations.
anisotropy problem. In DIL,, ent interactive learning tasks obtain both direct and latent text-labels correlations. CLRL, utilize label frequency and labels co-occurrence to explore the constraint relations, since the cor- relations between labels exists not only in single instance, predicting the label correlations from one instance may contradict with the the others.
Aniootropy problem Recently, pre-trained lan- guage models (PLMs) (Devlin et al., 2019; Liu et al., 2019) have become paradigms for various NLP tasks. PLMs based methods (Adhikari et al.. 2019; Zhang et al., 2021; Su et al., 2022) further achieve remarkable performance MLTC. How- ever, Ethayarajh (2019h); Su al. (2021) have demonstrated the anisotropy problem limits the ex- pressiveness of text representations. The represen- tations generated by PLMs are constrained within a narrow cone-shaped space after fine-tuned. Li et al. (2020) reveals that texts with high-frequency words concentrate densely in the narrowest part of the cone, while texts with low-frequency words dis- tribute sparsely. The anisotropy problem in MLTC, owing to high word frequency, leads samples with low-frequency labels to be closely gathered with other samples of different label frequencies, which can corrupt semantic information of representation and hinder the classification performance of MLTC. Therefore, we use contrastive learning to improve the uniformity of representations and introduce la- bel frequency information into representations to alleviate the anisotropy problem.
We illustrate that our method outperforms se- ries of competitive baseline AAPD and RCV1- V2 datasets by experimental results. We summarize our contributions as follows:
1. We propose cunlti-granularity information enhancement framework that explores diverse text-label correlations and label-label relations to improve the quality of text representations.
2. To alleviate the anisotropy problem, we intro- duce label frequency into text representations using contrastive learning and the MIL mod- ule.
3. Experimental results on two MLTC datasets demonstrate the effectiveness and competitive- ness of our method.
2 Related Work
Multi-label Text Classification In MLTC, early works like CNN (Kim, 2014; Kurata et al., 2016; Poria et al., 2017) and RNN (Liu et al., 2016) were proposed learn fine-grained and coarse-grained text information. Besides, Wang et al. (2018); Xiao et al. (2019); Chen et al. (2021) adopted joint em- bedding and text-labels fusion strategy to explore the correlations between text and labels. To capture
Proposed Method
Our framework. illustrated in Figure consists of two modules: a contrastive learning (CL.) mod- ule and a multi- gralui-grnnlarity information learning (MIL) module. The CL module refines initial rep-

---

Consirater Lahal Reintions Leoraing
Contrantivv Learning MLTC
Duul-grainel Inteructive Leurning
Text Repss
V1
Yi
Vel & Rdw
Moz
Meahe
MErC
Seurce
Target
Mas
TepetSangs
lyforeatee
Fio
C
Labels Reps
anturrii Aype
Teat-Labels Correlations
Sorees Robir Weak Robe
Cestrata
Daul-grainz! Information
Information
Labef
LubeleZabell
Preguoncy
Conreatioos
Infirmation
Information
Text Reps
Labels Reps
Appees Trassformee
Mau
Hows
Hay
Text- Text
Tex-Labelk Encoder
Labels-Lbels
Encoy
Eyrny
Eyas
[CLS]
Tyr
[SEP]
Ty:
(SEP)
Text
Labels
Figure 2: The framework of our proposed approach.
resentations for the MIL module, which introduces guati-granullriit information to enhance overall expressive ability. Besides, label frequency is also introduced by both CL and MIL module to alleviate the anisotropy problem.
sequently, we generate positive (f,, t and nega- tive (4i, 5j) pairs for CL. To create a positive pair, we select a sample t; from the target dataset and apply the dropout function to obtain 4t. For neg- ative pairs, we randomly select samples sj from the source dataset, ensuring that fo and 8j share negligible
3.1 Problem Formulation
label co-occurrence. In the spe- cific scenario of CL data mini-batch, where the batch size is denoted N 1, our method yields N negative pairs and single positive pair. The ultimate objective of CL training is articulated as follows:
Given MLTC training D [(r,, 1:)
N} consisting of N samples, each a; text and corresponding to label sequence Y; {v1, 1/, 3n Wh E (0, 1}} consisting of labels. The aim MLTC is to learn a mapping function from the text F; to the corresponding label set Y,
eimitt.at)/
LCL = log
3.2 Contrastive Learning for MLTC
Σ
Ej=1 (L 1/T + esim(tss;)/
We propose contrastive learning (CL) objective for the MITC task to enhance the effectiveness of the MIL module during the fine-tuning stage. This involves providing more uniform and effective ini- tial representations and implicitly introducing label frequency information into it. Besides, considering randomly sampled negatives may bias the repre- sentations of samples with similar labels. Our CL. objective is based on the labels co-occurrence and mainly focuses www-frequecy labels.
(1) where sim(a, 3) denotes the cosine similarity, cal- culated as a ||]|| = t
represents the augmented representation derived from applying dropout to f4 and 20 constitutes the temperature hyper- parameter.
3.3 Multi-Granularity Information Learning (MIL)
The primary objective of the MIL module is facilitate the learning of multi-granularity informa- tion, thereby enriching the representations from diverse perspectives. To address the anisotropy problems and to explore diverse text-label corre-
Initially, delineate two CL datasets: the get dataset, comprising samples associated with low-freque wwwffreuencyy labels, and the source dataset, en- compassing the entirety the training data. Sub-

---

lations, we propose dual- granularity interactive learning (DIL) task. Concurrently, a constraint label relations learning (CLRL) task is proposed to model the relations between labels and deepen text-labels interactive learning. CLRL achieves auxiliary augmentation by optimizing label repre- sentations and implicitly passing label relations to the DIL task.
global perspective in interactive information explor- ing is imperative. For example, in AAPD dataset, label machine learning (ML) is typically correlated to no specific key-phrases thus the classification of MI requires comprehensively integration of the global semantic feature.
In this task, we extract and learn the repre- sentation Hio) = {Hz Hzz. Har} through coarse-grained information extraction layer, which is consisted of 15 layers of Dilated Gated CNN (DGCNN). Within the DGCNN architecture, the dilated CNN acts to expand the receptive field of the overall structure, thereby enabling to cap- ture more extensive information. Concurrently, the gated framework facilitates transmitting the coarse- grained textual label interactions across multiple channels.
3.3.1 Dual-granularity Interactive Learning
DIL, we design two different granularities in- teractive leaning tasks to capture the varied text- labels correlations. Fine- grained interactive learn- ing (FIL) task achieves interaction based on the output representations of the encoder, and coarse- grained interactive learning (CIL) task interacts text with labels through 12 layers of transformer encoder.
The first layer is calculated by:
Fine-grained Interactive Leaning In instances where direct correlations between text and labels are assessed, such as the AAPD dataset, the label computer vision (CV) is commonly correlated to specific keyphrases (image, vision and pixel) in the given text. To address this, the FIL task interacts text with labels at the token-level to perform fine- grained interactive learning.
H() oo (1 0) + Cono1D (H(()) & σ =iggmiid(Cono1112(H
(6)
To improve the effectiveness of classifying repre- sentations, we use CNN and Multi-head Attention with residual structure, and generate the final representation Hen. by performing GlobalMax- Pooling:
We first normalize both the word-level text rep- resentation and labels representation by 1.2 normal- ization:
Haemp #MuliitcallAtiinninn((
H.n.. CCoooool(HHaamppHHHia
(7)
Hword ((HWord) HLabel LaHee))))
(2)
HoI1 =GlobalMaxPolinng(HH..
e([x1.". ;n,))
21
Fn
Multi-Label Prediction For both the HeIL and HFIL we take a fully connected layer with sig- moid activation as the multi-label classifier, and use Binary Cross Entropy as the loss function:
222
(3)
where Hword E Rddc Label E Rlxh Then, we transpose Habeel and interact with Hword by dot product to get text-labels correlations matrix C:
LBCE
viy Log((;))+++ v.y) log(1 (v)) (8)
C= C=Hword HT.
where N is the number of training samples, and is the length of labels sequence. Vij E (0, 1} and
where C E Rdxl for Cij E C indicates the inter- action between Ha, and L In order to learn the correlations more explicitly, we use CNN with Max-Pooling under the ReLU activation, and obtain correlations score S E ddx1 by tanh function. The final representation HFIL can be calculated by:
ūij E [o, 1] are the true value and the predicted value of the yj. yj E Y, = {30. V1, -..-.-,. :
3.3.2 Constraint Label Relations Learning
The objective of CLRL is to assist the DIL task to perform MLTC by learning similar and constraint relations between labels. As the text and labels are embedded in the same semantic space, the label frequency information is integrated into the DIL task through optimizing label relations. Introduc- ing label frequency thus alleviates the anisotropy problem caused by word frequency bias.
HFr. = GllbaalMaXPoolingSS x Hword) (5)
Coarse-grained Interactive Leaning Given that latent correlations between text and labels are influenced by multiple discrete text segments, a

---

Label Set
Definition
Herein, HA and Ha epitomize the representa- tions of matching label A and B, respectively. To explore the relations between labels, we utilize a fully connected layer with softmax activation as the label relations classifier. The adopted loss func- tion is the Categorical Cross Entropy, which can be articulated as:
VI
For any imput sample (D;,Y.) ki),1. Y, (sn. 3 and yj =.
For any input sample (D,, Y.), Vs E EY
Y7
(yn, ye) and y5 0, yu E Y
My, <Mayyve Mi Youufre
For any input sample (12, Y.). EY
Y,
{yr. (33) and yj 0, Y
Mys >Mayore E g
tow freq,
LOCE = Σ yk log(ūk) k=1
(12)
Table I: Definition of constraint label
where is the category of the CLRL label, whilst Yk and ÿk signify the actual and predicted values of category l within the CLRL label, respectively, In the scenario where Matching Label B aligns with Y; Y," Y, the CLRL labels are respec- tively designated as directly related [1,0,0], indi- rectly related [0,1,0], or irrelevant [0,0,1].
In CLRL, we first analyze the distribution of the training labels and then count the fre- quency of co-occurrence between each single la- bel and the rest labels. For a given labels set Y = (y1, ".))) we calculate the label re- lations matrix M Latel by:
3.3.3 Training Objectives
MLadel
Count(y:.
1m =15=1
The loss functions for the multi-label text classi- (9) fier (FIL & CIL) are defined as LFIL and LCIL. utilizing Binary Cross Entropy LECE- The loss function for the label relations classifier is denoted as LCLRL, employing Categorical Cross Entropy Look. The total loss of MIL be calculate by:
Mi.1
M1,2
...
Mia
Ma.1
M2,2
Maa
Ma.
M1a
Mr
where Mij indicates the frequency of the co- occurrence between label yı and label yj in the training dataset.
CMIL aLCIL + +BFFIL + ACCLRL (13)
We then calculate the average frequency co- occurrence of each label by:
where both a and B are hyper-parameters (0.2) and is hyper-paramett
(0.1) we set a = 1. 8 = 1, A = 0.35 as the final setting. In addition, the final predictions of DIL are the summed by the output of FIL and CIL weighted respectively by and 1 ry during inference stage, where set 0.5.
1 M, 1-1
Σ j=1ifi
Mij
(10)
and then obtain the low-freeue www-freqenccy labels sequence y. yloom/reg " : according Mi, where Mi,a; M1, S M,
Experimental Settings
In light of the foregoing, we randomly select one text-correlated label, denoted as Y, Subsequent to this, we partition the remaining labels into three distinct subsets: y,+. Y,", and Y,, based on the relations of labels and the low-frequency labels sequence. The definition of these label subsets can be found in Tab 1.
In this section, we evaluate the main experimental results of the baseline models and our method on two datasets. Implementation details of our method can be found in Appendix A.
4.1 Datasets
To validate method, conduct experiments two benchmark datasets AAPD (Yang et al., 2018) and RCV1-V2 (Lewis et al., 2004). Tab 2 shows statisties of both datasets.
To elucidate further, we first randomly select y/ from the text correlated labels as matching label A, and partition the input labels into three constraint label sets according to matching label A. Then, we randomly select one label from the rest of the labels as matching label B. We obtain the relations embedding Hrelations to perform the CLRL by:
4.2 Evaluation Metrics
Following previous works (Yang al., 2018), we adopt Hamming Loss (HL) and Micro-FI (F1) scores as our main evaluation metrics. To further
Hrelations = (HA. Ho, HA - Hal) (11)

---

Dataset
.V
C
NV
AAPD
55,840
54
2.4
163.4
RCVI-V2
804,414
103
3.2
123.9
Table Statisties of datasets. .V and L denote the total number of texts and labels. Z is the average number of labels associated with the text. N means the average length of all texts.
(a) Initial Representation
(b) Initial Representation with CL
evaluate our method, we also take Micro-Precisiin (Pre) and Micro-Recall (Rec) as auxiliary analysis metrics.
Text representations of test generated by initial models in 3D PCA. Where in [h.m.]), h = denotes that the current sample contains high-freueeccy label, while middle and low-frequency label respectively.
Figure 3:
and / denote the occurrence of
4.3 Baseline Methods
We compare our method with two sets of competi- live MLTC methods including: text representation learning and label correlation exploring methods.
The first set of methods involves (1) CNN (Kim, 2014) fine-grained text representation learning method that utilizes multiple convolutional kernels to extract text representations. (2) BERT (Devlin al., 2019), a coarse-grained text representation learning method which takes the [CLS] token as a global classification representation. (3) LEAM (Wang et al., 2018) proposes joint embedding of text and labels to obtain text representation. (4) LSAN (Xiao et al., 2019) proposes an adaptive attention fusion strategy and classifies each docu- ment by building label-specific representation.
01.
(a) BERT
(b) CL-BERT
The second set of methods involves (5) SGM (Yang al., 2018) applies the sequence genera- tion model to transform the MLTC problem into sequence generation problem. and proposes global embedding mechanism to capture label correla- tions. (6) Seq2set (Yang et al., 2019) adds Set decoder the basis of the SGM and exploit of the disorder of Set reduce the impact of incor- label sorting. (7) LACO (Zhang al., 2021) utilizes multi-task framework with two auxiliary label correlation learning tasks and one MLTC task.
(c) DIL
(d) CL-DIL
BAD
Mil.git
(f) CL-MIL
(8) CL-BERT+ANN (Su et al., 2022) proposes a KNN mechanism according to the relevant labels between samples and designs a multi-label con- trastive learning objective enhance the effect of ANN mechanism on MLTC.
Figure 4: Text representations of test set generated by fine-tuned models in 3D PCA. present different see- narios of applying CL to the backbone models, and illustrate the specific conditions for performance im- provement applying CL.
5
Results and Analysis
evaluation metrics such Rec (72.5/87.2), HL (0. (0.0208/0.0068), and F1 (75.8/88.9).
5.1 Main Results
Table 3 displays the results of compared meth- ods on two datasets. It is noted that CL-MIL outperforms all baselines considerably on major
For text representation learning, when compared to coarse-grained learning methods like BERT, which uses the [CLS] token for global representa-

---

Table and Figure demonstrate the perfor- mance changes of applying CL to backbone models with varying degrees of label information: BERT (only embeddings). DIL (Interactive Learning), and MIL.11/ (label relations). For BERT, which learns no label information, applying CL. may be counter- productive and greatly hinder performance. Simi- larly, CL offers no improvement to DIL, which only learns text-label correlations. However, implement- ing CL in MIl which learns constraint label relations, can effectively improve the performance on main evaluation metrics. We believe the fun- damental reason for this is that the training target of CL for MLTC is to push non-neighbor samples apart based label co-occurrence, which aligns precisely with the objective of CLRL. This obser- AAPD dataset Pre(+)
RCV1-V2 dataset Pre(+)
Model
HLL(-)
Ree(+)
FI(+)
HIL--)
Rec(+)
Fi(+)
CNN (Kurata al., 2016)
0.0256
84.9
54.5
66.4
0.0089
92.2
79.8
85.5
BERT
(Devlin et al., 2019)
0.0230
76.7
69.8
73.1
0.0078
89.1
85.6
87.3
LEAM
(Wang et al., 2018)
0.0261
59.6
67.0
0.0090
87.1
84.1
85.6
LSAN
(Xiao et al., 2019)
0.0242
77.7
64.6
70.6
0.0075
91.3
84.1
87.5
LACO
(Zhang al., 2021)
0.0213
80.2
69.6
74.5
0.0072
90.8
85.6
88.1
SGM (Yang al., 2018)
0.0251
74.6
65.9
69.9
0.0081
88.7
85.0
86.9
Seq2set' (Yang et al., 2019)
0.0247
67.4
70.5
0.0073
85.8
LACO plop (Zhang ct al., 2021)
0.0212
79.5
70.8
74.9
0.0070
90.8
86.2
88.4
LACO dep (Zhang et al., 2021)
0.0215
78.9
70.8
74.7
0.0070
90.6
86.4
88.5
CL-BERT kYN (Su et al., 2022)
0.0216
75.1
0.0071
88.3
CL-MMIL (Ours)
0.0208
79.3
72.5
75.8
0.0068
90.6
87.2
88.9
Table 3: Predictive performance of each comparing algorithm two datasets. Hamming Loss (HL.), Micro Precision (Pre), Recall (Rec), FI-Score (FI) are used as evaluation metrics. The (-) represents the lower score the better performance, and the (+) is the opposite. Models with i denote for its results are quoted from previous papers.
tion in classification, CL-MIL improves the F1 by increasing the Pre. Compared to the-ggaieed learn- ing methods like CNN and LACO, which focus on more detailed information, CL-MIL increases the Rec and improves the F1. This suggests a need for an appropriate balance of different granularities in- formation to improve the overall ability of learning text-labels correlations, and more specific analysis are in Section 5.3.
shown in Figure 3, in initial stage, the representa- tions after CL distribute more uniformly in geom- etry space. Besides, samples that only correlated high-frequency labels preliminary pushed away from the central space of samples of low-frequency middle Middle-frequency labels. The refined representa- tions will more suitable for MIL. module.
Among label correlations learning methods, LACOptop- LACOdccp. and CL-BERT provide more significant improvements than other base- line methods. Comparing to these three strong methods, CL-MIL reduces the HL by 1.9%-3.7% 129%%-4.2% %, and improves the F1 by 0.9% 1.5% 0.55 10.%%-0.7% on two datasets.
AAPD
Model
HLL-)
Pre(+)
Rec(+)
F1(+)
BERT
0.0230
76.7
69.8
73.1
+CL.
0.0298
63.8
78.1
70.1
DIL (Ours)
0.0214
78.3
72.2
75.1
+CL
0.0217
77.6
72.5
74.9
MIL (Ours)
0.0211
78.8
72.4
75.5
+CL
0.0208
79.3
72.5
75.8
vation suggests that CL for MLTC has high-order relevance to CLRL.
Table Ablation over BERT, DIL and MIL with or without CL on AAPD dataset.
In conclusion, CL brings a more uniform and task-adapted initial representation, and further im- proves the auxiliary effectiveness of CLRL by ex- ploiting the contrastive relations among low co- occurrence frequency labels.
5.2 The Impact of Contrastive Learning
For contrastive learning (CL), we utilize BERT (with joint embedding), DIL and MIL the backbone models verify the effect of CL. and visualize the text representations of test set gen- erated by these models in 3 dimension PCA. As
5.3 Analysis of Dual-grained Interactive Learning
We conduct an ablation study to validate the relia- bility and effect of DIL. Table 5 shows the compari-

---

Picco--rrcisioon
Micro-Rocall
Micre-F1
15.
80-
730
14.8
Precision
Recall
71.5
23
(025.125)
11,25,775)
700-
733
74
1.0
oid
a'a
738
0'4
Plyper-parameters
parametess Y
Y
Figure 5: Hyper-parameters analysis of DIL on AAPD dataset
AAPD dataset
RCV1-V2 dataset
Model
HI(()
Pre(+)
Rec(+)
F1(+)
HL(-)
Pre(+)
Rec(+)
FI(+)
CIL
0.0229
76.8
70.2
73.5
0.0075
88.5
87.3
87.9
FIL
0.0298
76.2
66.1
71.1
0.0074
89.3
86.6
87.9
DIL
0.0214
78.3
72.2
75.1
0.0073
89.1
87.3
88.2
Table Analysis coarse-grained interactive learning (CIL.) fine-grained interactive learning (FIL.) and on AAPD and CCV1-VV datasets.
Nilla BERLE.
5.4 Analysis of Constraint Label Relations Learning
Miero-F F
DIL
BERT
According to Table 3, we can notice that label cor- relations learning methods obtain higher Rec than text representation learning methods. It indicates that these label correlations learning methods im- proving F1 without considering constraint relations are based increasing Rec at the of decreas- ing Pre. A potential reason is that learning label correlations explores whether the labels share co- occurrence current sample, which may lead to certain degree of label bias (high co-occurrence frequency but not correlated in current sample). It will hinder the performance since the labels have been already embedded together with the text in text.
group
Mid-group
Tuil-group
Figure Micro-FI performance four label frequency distributions.
son of DIL with coarse-grained interactive learning (CIL) and fine-grained interactive learning (FIL) respectively. The result reveals that it is an appro- priate integration of CIL and FIL that DIL achieves effective performance rather than two dispersed modules.
space and sufficiently interacted with the
Differ from these methods, our MIL takes constraint label relations into consideration, which judges the relations not only by the co-occurrence of current sample but also by the labels occurrence frequency. By this means, the label bias can be alleviated, and MII can better en- hance the FIL module to obtain more considerable with both Pre and Rec improved.
We also conduct perer-pprameters analysis DIL to further illustrate the effect different granularities on classification performance. Figure 5 shows the influence of training loss weights & and 3 on Pre, Rec and F1 score under different proportion of predictions The curves prove that coarse-grained information is related to Rec while fine-grained information correlate to Pre. Besides, the curves also illustrate that an appropriate ratio of training loss weights determines the lower bound of classification performance, while an apt proportion predictions determines the upper.
To further illustrate the effectiveness of CLRL, analyze the label frequency of the AAPD test and divide the labels into three groups! according
'The head label group (head-group), the middle-frequency group (mid-group) and the tail lahel group (tail-group) accord- ing to the label distribution of training set.

---

the definition shown in Table 6. As shown in Fig- ure MlLdr outperforms DIL in the mid-group and tail-group, while BERT only improves per- formance in the head-group group compared to BERT. Additionally, when CLRL is applied to BERT with- out interactive text-label correlation learning, per- formance decreases in the mid-group and tail-group labels, demonstrating that CLRL can better explore label relations and improve low-frequency perfor- mance under the reinforcement of DIL.
Ethics Statement
Our method is used to address multi-label text clas- sification. Therefore, ethical considerations of text classification models generally apply to our method. We encourage users to assess potential biases be- fore deploying text classification models.
A Acknowledgements
This research is supported by the National Nat- ural Science Foundation of China [62172449, 72374070, 62006251], Hunan Provincial Natu- ral Science Foundation China [2021JJ30870,
Label Groups
Definition
For any label L, in head-group.
head-group
Frequency (L,) 2 3000.
20221/3021, 20211J4072273 Changsha Municipal Natural Science Foundation [kq2202300J. Training Program for Excellent Young Innovators of Chang- sha [kq21070041, The science and technology inno- vation Program of Hunan Province [2022RC1 105] Innovation Fund And
For any label L, in mid-group, 3000 >
mid-group
Frequency (1.,) 2 1000.
tail-group
For any label L; in tail-group, 1000 Frequency (L.).
Table 6: Definition of label frequency groups
and of Chinese this research was supported in part by the High Performance Computing Center of Central South University.
6 Conclusion
This paper proposed multii-rannuarrty informa- tion enhanced MLTC framework. Through exten- sive experiments, we demonstrated the competi- tiveness and stability of our method and confirmed the effectiveness of incorporating multi-granularity information to enhance representations in MLTC. Additionally, our extended analysis highlighted the complementarity of the modules in our framework and emphasizes the necessity of each module.
References
Ashutosh Adhikari, Achyudh Ram, Raphael Tang, and Jimmy Lin. 2019. Docbert: Bert for document classi- fication. ArXiv preprint, abs/1904.0$$
Benjamin Bengfort and Rebecca Bilbro. 2019. Yel- lowbrick: Visualizing the Scikit-Learn Model Selec- tion Process. The Journal of Open Source Software, 4(35).
Limitations
Haibin Chen, Qianli Ma. Zhenxi Lin, and Jiangyue Yan. 2021. Hierarchy-aware label semantics matching net- work for hierarchical text classification. In Proceed- ings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joini Conference on Natural Language Processing (Volume 1: Long Papers), pages Association for Computational Linguisties.
While our method has yielded promising results by utilizing BERT as the encoder of our framework, it comes with certain limitations worth consider- ing. Firstly, the foremost limitation is the restricted input length of the BERT encoder set at 512. In extreme label scenario, it's not feasible to embed all the labels, limiting our approach cases with a normal number labels. In future work, we could explore a hierarchical label embedding strategy or migrate to other pre-trained language models with unlimited embedding lengths. Secondly, although our framework explores the multi- multi-granularity infor- mation after joint embedding of text and labels, it would be engaging to investigate the effect of la- bel embedding order. Hence, trying different label embedding orders will provide potential insights for future work and enhance the performance of MLTC tasks.
Online.
Jacob Devlin, Ming-Was Wei Chang. Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisticx: Humun Language Tech- nologies, Volume (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Kawin Ethayarajh. 2019a. How contextual are con- textualized word representations? Comparing the geometry BERT, ELMo, and GPT-2 embeddings. In Proceedings of the 2019 Conference on Empirical

---

Methods in Natural Language Processing and the 9th International Joint Conference on Natural Lan- guage Processing DEMNI.P-IJCNI.P pages 55-65, Hong Kong, China, Association for Computational Linguisties.
Pengfei Liu, Xipeng Qiu, and Xuanjing Huang. 2016. Recurrent neural network for text classification with multi-task learning. In Proceedings of the Twenty- Fifth International Joint Conference on Artificial In- telligence, IJCAI 2016, New York, NY, USA, 9-15 July 2016, pages
ICCTAAAAI Press.
Kawin Ethayarajh. 2019b. How contextual are con- textualized word representations? Comparing the geometry of BERT, Mo, and GPT-2 embeddings. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Lan- guage Processing (EMNLP-IJCNE) pages 55-65, Hong Kong, China, Association for Computational Linguistics.
Yinhan Liu, Myle Oit, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis. Luke Zettlemoyer. and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach. ArXiv preprint, abs/1907.11692.
Aditya Krishna Menon, Sadeep Jayasumana, Ankit Singh Rawat, Himanshu Jain, Andreas Veit, and Sanjiv Kumar. 2021. Long-tail learning via logit adjustment. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net.
Jun Gao, Di He, Xu Tan, Tao Qin, Liwei Wang, and Tie- Yan Liu. 2019. Representation degeneration problem in training natural language generation models. In 7th International Conference on Learning Represen- lations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net.
Jinseok Nam, Eneldo Loza Mencía, Hyunwoo J. Kim, and Johannes Fürnkranz. 2017. Maximizing subset accuracy with recurrent neural networks multi- label classification. In Advances in Neural Informa- fion Processing Systems 30: Annual Conference Neural Information Processing Systems 2017, De- cember 4-9, 2017, Long Beach, CA, USA, pages 5413 1413.5423.
Guo, Bo Jin, Ruiyun Yu, Cuili Yao, Chonglin Sun, and Degen Huang. 2016. Multi-label classification methods for green computing and application for mobile medical recommendations. IEEE ACCESS, 4:3201-3209.
Himanshu Jain, Yashoteja Prabhu, and Manik Varma. 2016. Extreme multi-label loss functions for recom- mendation, tagging, ranking & other missing label ap- plications. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco. CA, USA, August 13-17, 2016, pages 935-944. ACM.
Y. Papanikolaou, D. Dimitriadis, G. Tsoumakas, M. Laliotis, and I. P. Vlahavas. 2014. Ensemble approaches for large-scale multi-label classification and question answering in biomedicine. In CLEF 2014.
Matthew F. Peters, Mark Neumann, Mohit Tyyer, Matt Gardner, Christopher Clark. Kenton Lee, and Luke Zeutleooer. 2018. Deep contextualized word repre- sentations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume I (Long Papers), pages 2227-2237, New Orleans, Louisiana. Association for Computa- tional Linguisties.
Yoon Kim. 2014. Convolutional neural networks for sentence classification. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1746-175 1751, Doha. Qatar. Association for Computational Linguis- tics.
Gakuto Kurata, Bing Xiang, and Bowen Zhou. 2016. Improved neural betsork-based multi- label classifi- cation with better initialization leveraging label co- occurrence. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 521-526, San Diego. Califoria. Association for Computational Linguisties.
Soujanya Poria, Haiyun Peng, Amir Hussain, Newton Howard, and Erik Cambria. 2017. Ensemble appli- cation of convolutional neural networks and multiple kernel learning for multimodal sentiment analysis. Neurocomputingg page 217-230.
Jesse Read, Bernhard Pfahringer, Geoff Holmes, and Eibe Frank. 2011. Classifier chains for multi-label classification. Machine learning, 85:333-359.
David D Lewis, Yiming Yang, Tony Russell-Ra Rose, and Fan Li. 2004. Revl: A new benchmark collection for text categorization research. Journal of machine learning research, 5(Apr):361-39,
Jianlin Su, Jiarun Cao, Weijie Liu, and Yangyiwen Ou. 2021. Whitening sentence representations for bet- ter semantics and faster retrieval. ArXiv preprint, abs/2103.15316.
Bohan Li, Hao Zhou, Junxian He, Mingxuan Wang. Yiming Yang, and Lei Li. 2020. On the sentence embeddings from pre-trained language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9119-9130. Online, Association for Computa- tional Linguistics.
Xi'ao ao Su, Ran Wang, and Xinyu Dai. 2022. Contrastive learning-enhanced nearest neighbor mechanism for multi-label text classification. In Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics (Volume 2: Short Papers), pages

---

672-679, Dublin, Ireland. Association for Computa- tional Linguistics.
Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6120- 6130, Dublin, Ireland. Association for Computational Linguisties.
Grigorios Tsoumakas and loannis Katakis. 2007. Multi- label classification: An overview. International Journal of Data Warehousing and Mining (LJDWM),
A Implementation Details
3(3):1-13.
We implement our model based on Bert4keras and run NVIDIA Tesla V100s. The English BERT version is base-uncased BERT with 12 layers of encoder and 768 of hidden size. On contrastive learning stage, we use maximum input sequence length of 230 and a batch size of 32 for AAPD, and use a maximum input sequence length of 200 and a batch size of 32 for RCV1-V2. On MLTC stage, we use maximum input sequence length of 320 and batch size of 32 for both datasets. We train all the models on both stages and both datasets up 20 epochs with early stop of 2 patience and take the Adam as our optimizer with learning rate of xx003
Guoyin Wang, Chunyuan Li, Wenlin Wang. Yizhe Zhang, Dinghan Shen, Xinyuan Zhang. Ricardo Henao, and Lawrence Carin. 2018. Joint embed- ding of words and labels for text classification. In Proceedings of the 56th Annual Meeting of the As- sociation for Computational Linguistics ( (Volume I: Long Papers), pages 2321 2321-2331, Melbourne, Aus- tralia. Association for Computational Linguistics.
Yaqi Wang, Shi Feng, Daling Wang. Ge Yu, and Yifci Zhang. 2016. Multi-label chinese microblog emotion classification via convolutional neural network. In Asia-Pacific Web Conference.
Lin Xiao, Xin Huang, Boli Chen, and Liping Jing. 2019. Label-specific document representation for multi-label text classification. In Proceedings of the 2019 Conference on Empirical Methods in Natu- ral Language Processing and the 9th International Joint Conference on Natural Language Process- ing (EMNLP-IJCNLP) pages 466-475, Hong Kong, China. Association for Computational Linguistics.
B The PCA Visualization Details
In Figure 3 and Figure 4, we utilize 3 dimensional PCA visualization tool (Bengfort and Bilbro, 2019) to illustrate the text representations (to be fed into MIL module) distribution in geometry space. The text representations are divided into seven cate- gories according to the frequency distribution of their corresponding labels in the training set.
Pengcheng Yang. Fuli Luo, Shuming Ma, Junyang Lin, and Xu Sun. 2019. deep reinforced sequence-to-se model for multi-label classification. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5252-5258, Flo- rence, Italy. Association for Computational Linguis-
tics.
Pengcheng Yang, Xu Sun, Wei Li, Shuming Ma, Wei Wu, and Houfeng Wang. 2018 SGM: Sequence generation model for multi-label classification. In Proceedings of the 27th International Conference on Computational Linguistics, pages 3915-392,, Santa Fe, New Mexico, USA. Association for Computa- tional Linguistics.
Y. Yang and S. Gopal. 2012. Multilabel classification with meta-level features in a learning-to-rank frame- work. Machine Learning, 88(1-2):47-6.
Zhang, M., Zhou, and Z. 2014. A review on multi- label learning algorithms. IEEE Transactions on Knowledge and Data Engineering, 1818))11191188
Ximing Zhang, Qian- Wen Zhang, Zhao Yan, Ruifang Liu, and Yunbo Cao. 2021. Enhanciig label corre- lation feedback in multi-label text classification via multi-task learning. In Findings the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 1190-1200, Online. Association for Computa- tional Linguistics.
Kun Zhou, Beichen Zhang, Xin Zhao, and Ji-Rong Wen. 2022. Debiased contrastive learning of unsupervised sentence representations. In Proceedings of the 60th


Reviews:
[{"review_id": "apggkn85Ml", "reviewer": "Reviewer_HWyb", "summary": "", "questions": "A: At line 306, the author mentioned about the $lowfreq$ labels, what's the threshold for the frequency of $lowfreq$ labels? Also, what's the threshold for head-group, mid-group and tail group labels as mentioned at lines 502-504?\n\nB: In previous works [2, 3] in representation learning for multi-label text classification, researchers usually do 2d plot on the learned representation, via t-SNE mainly but PCA can also be an option. However, this paper used 3d plot to visualize the first 3 principal component (PC) values, could the author share the variance represented by each of the $PC$ ( $PC_{1}, PC_{2}, PC_{3}$)? If $Var(PC_{1}) + Var(PC_{2}) >95 $ %, then the information carried by $PC_{3}$ bring negligible information about the representation on latent space.\n\nC: Do the author have evaluation results (i.e. F1 or Hamming loss) when either the contrastive learning objective or Constraint Label Relations Learning objective is dropped?\n\n[2] Zihan Wang, Peiyi Wang, Lianzhe Huang, Xin Sun, and Houfeng Wang. 2022. Incorporating Hierarchy into Text Encoder: a Contrastive Learning Approach for Hierarchical Text Classification. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7109–7119, Dublin, Ireland. Association for Computational Linguistics.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,  \nWe would like to extend our heartfelt appreciation to the reviewer for dedicating the time to review our submission. We are grateful for the valuable feedback provided, and we have conscientiously considered each comment to make the necessary revisions and address the raised concerns. We have diligently added the missing reference, which serves as a relevant and significant contribution to the revised manuscript. Furthermore, we will address the concerns regarding typographical errors, grammar, style, and presentation to enhance the overall quality of the paper. Here, we will provide a detailed explanation addressing each concern you have raised.\n\n$\\textbf{Question-1:}$ Lacks sufficient detail and poor writing styles.   \n$\\textbf{Answer-1:}$ We sincerely apologize for this oversight. In response, we have meticulously revised the methodology section to offer enhanced clarity and detail regarding our proposed method. Additionally, we have made improvements to the overall writing style in our revised manuscript to ensure the readability for our readers.\n\n$\\textbf{Question-1.1:}$ Missing the definition for the frequency at which labels are categorized as head-group, mid-group, and tail-group.  \n$\\textbf{Answer-1.1:}$ For any label $L_i$ in the head-group, the frequency should be greater than or equal to 3000. For any label $L_i$ in the mid-group, the frequency should be between 3000 and 1000. And for any label $L_i$ in the tail-group, the frequency should be less than 1000. In our revised manuscript, we have now included explicit definitions for the frequency thresholds that categorize labels into head-group, mid-group, and tail-group. \n\n$\\textbf{Question-1.2:}$ It's unclear how transition from $\\mathcal{H}\\_{relations}$ in Eq. (11) to $\\mathcal{L}\\_{CLRL}$ in Eq. (13), via $\\mathcal{L}\\_{CCE}$ in Eq. (12).   \n$\\textbf{Answer-1.2:}$ We obtain the representation of label relations, $\\mathcal{H}\\_{relations}$, by concatenating the individual representations $\\mathcal{H}\\_A$, $\\mathcal{H}\\_B$, and $\\mathcal{H}\\_{A-B}$. We then apply a fully connected layer with a $softmax$ activation as the label relations classifier and use Categorical Cross Entropy, denoted as $\\mathcal{L}\\_{CCE}$, as the corresponding loss function. In order to improve the clarity of this transition, we will provide a more comprehensive explanation in our revised manuscript.\n\n$\\textbf{Question-1.3:}$ Certain equations lack clarity, and there are missing definitions for specific terms. For instance, in Eq. 13, none of the three loss symbols ($\\mathcal{L}$) have been previously introduced or formally defined.   \n$\\textbf{Answer-1.3:}$ The loss functions for the multi-label text classifier are defined as: $\\mathcal{L}\\_{CIL}$ and $\\mathcal{L}\\_{FIL}$ utilize Binary Cross Entropy, while $\\mathcal{L}\\_{CLRL}$ corresponds to the loss function of the label relations classifier, employing Categorical Cross Entropy. We will introduce and formally define these three loss symbols ($\\mathcal{L}$) in our revised manuscript. Additionally, we will provide clear explanations of each equation and symbol throughout the manuscript to enhance readability.\n\n$\\textbf{Question-2:}$ The primary contribution to performance seems to be the Dual-granularity Interactive Learning (DIL) method, as highlighted in Table 5. This method aims to extract joint information from text-labels, utilizing both label and document semantics via multi-headed attention (Coarse-grained) and dot product (Fine-grained). However, this concept is not novel and has been previously explored in works such as [1, 3].  \n$\\textbf{Answer-2:}$ \n* The DIL module significantly enhances the performance of our approach. However, it is worth noting that DIL's effectiveness is limited when applied to labels with mid-group and tail-group distribution. Hence, our research emphasizes not only on DIL but also incorporates additional modules to establish a comprehensive multi-granularity enhancement framework. Together, these modules alleviate the impact of label frequency, improve the performance of low-frequency labels, and enhance the quality of text representation. We firmly believe that the synergy of these components is what makes our method competitive.\n* While we acknowledge that the concept of leveraging multi-grained information has been explored in previous studies (e.g., [3]), those primarily focused on the label perspective, modeling the correlation between multi-granularity labels and text. On the contrary, our method places greater emphasis on the text itself, facilitating the learning of interactive information between multi-grained text features and labels. This primarily aims to refine text representations and capture intricate correlations between text and labels.\n\n$\\textbf{Question-3:}$ Training Complexity with Multiple Objectives: The training process of the proposed model involves optimizing multiple objectives. This multi-objective optimization can lead to increased training complexity and may present challenges in finding a balance between the different objectives. The paper should elaborate on the training process, addressing potential challenges and demonstrating how each component contributes to the final performance.   \n$\\textbf{Answer-3:}$ In our ablation study, we thoroughly evaluate the contribution of each component to the final performance. \n* In Tab. (4), we provide a detailed analysis when the contrastive learning objective is dropped (referred to as $MIL_{clrl}$). We also investigate the scenario when the CLRL objective is dropped ($DIL+CL$) and the case when both the contrastive learning and CLRL objectives are dropped ($DIL$). \n* Fig. (5) and Tab. (5) further explore and confirm the learning balance of DIL across different granularities during the training process.\n\nIn the revised manuscript, we will provide a more comprehensive description of the training process. Additionally, we will outline a detailed procedure for optimizing multiple objectives, thereby effectively addressing the challenges encountered and highlighting the impact of each objective on the overall performance.\n\n$\\textbf{Question-4(A):}$ At line 306, the author mentioned about the $lowfreq$ labels, what's the threshold for the frequency of $lowfreq$ labels? Also, what's the threshold for head-group, mid-group and tail group labels as mentioned at lines 502-504?  \n$\\textbf{Answer-4(A):}$ We apologize for any lack of clarity in explaining the threshold for categorizing labels as $lowfreq$ at line 306. The threshold for determining the $lowfreq$ labels is based on the average frequency co-occurrence of each label, calculated using Eq.(10): $\\overline{\\mathcal{M}}\\_i = \\frac{1}{l-1}\\sum\\_{j=1,j\\neq i}^l\\mathcal{M}\\_{i,j}$. The $lowfreq$ labels $Y_i^{lowfreq}$={ $y\\_a,\\cdot\\cdot\\cdot,y\\_b$} are determined according to the value of $\\overline{\\mathcal{M}}\\_i$, where $\\mathcal{M}\\_{i,a} , \\mathcal{M}\\_{i,b}\\leq \\overline{\\mathcal{M}}\\_i$. \nFor the thresholds of the head-group, mid-group, and tail-group labels:\n* For any label $L_i$ in head-group, $Frequency\\left(L_i\\right)\\geq3000$;\n* For any label $L_i$ in mid-group, $3000>Frequency\\left(L_i\\right)\\geq1000$;\n* For any label $L_i$ in tail-group,  $1000>Frequency\\left(L_i\\right)$.\n\n$\\textbf{Question-5(B):}$ In previous works [2, 3] in representation learning for multi-label text classification, researchers usually do 2d plot on the learned representation, via t-SNE mainly but PCA can also be an option. However, this paper used 3d plot to visualize the first 3 principal component (PC) values, could the author share the variance represented by each of the $PC (PC_1, PC_2, PC_3)$? If $Var\\left(PC_1\\right)+Var\\left(PC_2\\right)>95%$, then the information carried by $PC_3$ bring negligible information about the representation on latent space.  \n$\\textbf{Answer-5(B):}$ We appreciate the reviewer's suggestion to provide variance information for the first three principal components ($PC_1$, $PC_2$, and $PC_3$) in the 3d plot visualization. When we performed 3d reduction on the 768-dimensional vector representation, we found that:\n* $Var\\left(PC_1\\right)=0.5630604$;\n* $Var\\left(PC_2\\right)=0.24930695$;\n* $Var\\left(PC_3\\right)=0.18763264$,\n\nwhich proved the feasibility of using 3d visualization. On the other hand, our choice of 3d visualization serves to prove the effectiveness of our method more clearly.\n\n$\\textbf{Question-6(C):}$ Do the author have evaluation results (i.e., F1 or Hamming loss) when either the contrastive learning objective or Constraint Label Relations Learning objective is dropped?  \n$\\textbf{Answer-6(C):}$ We apologize for the unclear description of the ablation study. We have conducted experiments to evaluate the performance in Tab. (4) under three conditions:\n1. the contrastive learning objective is dropped ($MIL_{clrl}$); \n2. the constraint label relations learning objective is dropped ($DIL+CL$); \n3. both contrastive learning and constraint label relations learning objectives are dropped ($DIL$).\n\n| **Model** | **HL(-)** | **Pre(+)** | **Rec(+)** | **F1(+)** |\n| --- | --- | --- | --- | --- |\n| **BERT** | 0.0230 | 76.7 | 69.8 | 73.1 |\n| **+CL** | 0.0298 | 63.8 | 78.1 | 70.1 |\n| **DIL (Ours)** | 0.0214 | 78.3 | 72.2 | 75.1 |\n| **+CL** | 0.0217 | 77.6 | 72.5 | 74.9 |\n| **MIL$_{clrl}$ (Ours)** | 0.0211 | 78.8 | 72.4 | 75.5 |\n| **+CL** | 0.0208 | 79.3 | 72.5 | 75.8 |\n\n*Table4: Ablation over BERT, DIL and MIL$_{clrl}$ with or without CL on AAPD dataset.*", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- A comprehensive diagram detailing the model architecture.\n\n- Clear visualizations of the learned representations, accompanied by experiments and an ablation study that support the proposed approach for enhanced learning of representations in HTC.\n\n- Notable improvements in classification performance on both RCV-1 and AAPD datasets, which are commonly utilized for multi-label classification.\"", "reasons_to_reject": "- The methodology lacks sufficient detail and poor writing styles, making it challenging for readers to follow.\n  - The definition for the frequency at which labels are categorized as head-group, mid-group, and tail-group is missing. This omission hinders the reviewer's understanding on the effectiveness for improvement of labels with varying frequency.\n  - It's unclear how one transitions from $\\mathcal{H_{\\mathrm{relations}}}$ in Eq. (11) to $\\mathcal{L_{\\mathrm{CLRL}}}$ in Eq. (13), via $\\mathcal{L_{\\mathrm{CCE}}}$ in Eq.(12).\n\n  - Certain equations lack clarity, and there are missing definitions for specific terms. For instance, in Eq. 13, none of the three loss symbols ($\\mathcal{L}$) have been previously introduced or formally defined. Based on the reviewer's understanding, $ \\mathcal{L_{\\mathrm{CIL}}} $ and $\\mathcal{L_{\\mathrm{FIL}}}$ refer to the BCE mentioned in Eq. 8; and $\\mathcal{L_{\\mathrm{CLRL}}}$ refer to the Eq. 12. \n\n- The work builds on incremental research. The primary contribution to performance seems to be the Dual-granularity Interactive Learning (DIL) method, as highlighted in Table 5. This method aims to extract joint information from text-labels, utilizing both label and document semantics via multi-headed attention (Coarse-grained) and dot product (Fine-grained). However, this concept is not novel and has been previously explored in works such as [1, 3].\n\n- Training Complexity with Multiple Objectives: The training process of the proposed model involves optimizing multiple objectives, including the contrastive loss function, Fine-grained Interactive Learning, Coarse-grained Interactive Learning, and Constraint Label Relations Learning. This multi-objective optimization can lead to increased training complexity and may present challenges in finding a balance between the different objectives. The paper should elaborate on the training process, addressing potential challenges and demonstrating how each component contributes to the final performance. (See question C)\n\n[1] Lin Xiao, Xin Huang, Boli Chen, and Liping Jing. 2019. Label-Specific Document Representation for Multi-Label Text Classification. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 466–475, Hong Kong, China. Association for Computational Linguistics.", "questions_to_authors": "A: At line 306, the author mentioned about the $lowfreq$ labels, what's the threshold for the frequency of $lowfreq$ labels? Also, what's the threshold for head-group, mid-group and tail group labels as mentioned at lines 502-504?\n\nB: In previous works [2, 3] in representation learning for multi-label text classification, researchers usually do 2d plot on the learned representation, via t-SNE mainly but PCA can also be an option. However, this paper used 3d plot to visualize the first 3 principal component (PC) values, could the author share the variance represented by each of the $PC$ ( $PC_{1}, PC_{2}, PC_{3}$)? If $Var(PC_{1}) + Var(PC_{2}) >95 $ %, then the information carried by $PC_{3}$ bring negligible information about the representation on latent space.\n\nC: Do the author have evaluation results (i.e. F1 or Hamming loss) when either the contrastive learning objective or Constraint Label Relations Learning objective is dropped?\n\n[2] Zihan Wang, Peiyi Wang, Lianzhe Huang, Xin Sun, and Houfeng Wang. 2022. Incorporating Hierarchy into Text Encoder: a Contrastive Learning Approach for Hierarchical Text Classification. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7109–7119, Dublin, Ireland. Association for Computational Linguistics.", "paper_topic_and_main_contributions": "The paper focuses on the topic of multi-label text classification and proposes a multi-granularity enhancement framework, which utilizes both fine-grained and coarse-grained information between labels. The author first proposes fine-tuning the BERT classifier with a contrastive learning objective. Then, the author proposes three different loss functions: Fine-grained Interactive Learning, Coarse-grained Interactive Learning, and Constraint Label Relations Learning. These functions aim to better learn the text-label correlation information and enhance the performance on low-frequency labels. Experimental results and ablation studies on two MLTC datasets demonstrate the effectiveness of the approach and lead to a better understanding of text representation.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- Line 126-127, 130-131, 145, 149-150, : Change the citation styles to remove parenthesis, use \\citet{CITATION} instead\n\n- Unusual upper-case within sentence: Line 183: \"Fine-tuning\" -> \" fine-tuning\"\n\n- At line 333, it should be $k$\n\n- The function symbol $\\mathcal{H}$ is being overloaded and kept being used ($\\mathcal{H_{\\mathrm{word}}}, \\mathcal{H_{\\mathrm{label}}^{\\mathrm{Norm}}}, \\mathcal{H_{\\mathrm{FIL}}}, \\mathcal{H_{\\mathrm{relations}}}$ etc.), with different super- and subscript. This make the formalism of equations hard to follow.", "Missing_References": "[3] Haibin Chen, Qianli Ma, Zhenxi Lin, and Jiangyue Yan. 2021. Hierarchy-aware Label Semantics Matching Network for Hierarchical Text Classification. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 4370–4379, Online. Association for Computational Linguistics."}}, {"review_id": "rCel9lqKUO", "reviewer": "Reviewer_xJF7", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,\n\nWe thank the reviewer for your positive and constructive feedback. And we apologize for typos and grammar mistakes. They will be corrected so that the overall writing style meets EMNLP standard. We have carefully considered your concerns and suggestions and have made the necessary revisions to address the issues you raised.\n\n$\\textbf{Question-1:}$ The paper is not very well written. It required quite an effort to fully understand the proposed approach.  \n$\\textbf{Answer-1:}$ Regarding the concern about the overall clarity and readability, we apologize for any confusion caused by the initial manuscript. We have thoroughly revised the manuscript to improve its structure and enhance the clarity of the proposed approach to make it more readable.\n\n$\\textbf{Question-2:}$Lines 182, 270, 442: The authors misuse the sense of \"efficiency\". \"Efficiency\" is usually connected with the runtime, not with the classification performance.  \n$\\textbf{Answer-2:}$ We agree that \"efficiency\" is commonly associated with runtime and not with classification performance. We have rectified this error by using “effectiveness” (lines 182, 270), “effectively” (line 442) to describe the aspects of classification performance.\n\n$\\textbf{Question-3:}$ Equation 11: Shouldn't it be $\\left|H_A-H_B\\right|$ or am I missing something?  \n$\\textbf{Answer-3:}$ We highly agree with your enlightening suggestion after we reviewing the Equation 11 and relevant section. In revised manuscript, we will correct the $\\left|H_{A-B}\\right|$ to a more appropriate form ($\\left|H_A-H_B\\right|$).\n\n$\\textbf{Question-4:}$ There are a few typos here and there. I suggest the authors have a careful proofreading of their manuscript.  \n$\\textbf{Answer-4:}$ We have carefully proofread the manuscript and addressed the typos and grammatical errors to ensure its clarity and readability.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The idea is interesting and the proposed method seems to be effective.\n* The ablations tests and the representations demonstration support the claims of the authors.\n* The hyper-parameter analysis is interesting.", "reasons_to_reject": "* The paper is not very well written. It required quite an effort to fully understand the proposed approach.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes a multi-granularity information enhancement framework to improve text representations and in effect multi-label text classification (MLTC). To achieve that the authors use contrastive learning to implicitly enhance the initial representations of the texts with the label frequencies. Given a source sample containing a low-frequency label, the authors create a positive pair using dropout and negative pairs using texts that contain frequent labels and their labels do not co-occur frequently with the labels of the source sample.\n\nThen the authors learn representations that encode fine-grained interactions using a CNN over the similarity matrix of the representations of the labels tokens and the text tokens, and coarse-grained interactions using a 15-layerDilated CNN and a Multi-head Attention over the text token representations. The fine-grained representations are passed through an MLP that performs MLTC. Similarly, the coarse-grained representations are passed through another MLP that performs MLTC.\n\nThe authors also employ Constrained Label Relations Learning. Given two labels, A and B, the authors use a Logistic Regression classifier to determine whether A is directly, indirectly, or not related to B.\n\nAll three modules are combined using multi-task learning by operating on representations extracted by encoding all the possible pairs of texts and labels using a BERT-base model, i.e., the encoder is shared and updated by each module jointly.\n\nThe authors compare with several baselines and report improved F1 as a result of increasing recall at the expense of reduced precision. They also perform ablation tests to demonstrate the effectiveness of each component.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "* Lines 182, 270, 442: The authors misuse the sense of \"efficiency\". \"Efficiency\" is usually connected with the runtime, not with the classification performance.\n* Equation 11: Shouldn't it be $|\\mathcal{H}_A - \\mathcal{H}_B|$ or am I missing something?\n* There are a few typos here and there. I suggest the authors have a careful proofreading of their manuscript.\n"}}, {"review_id": "WpeyD1YYxY", "reviewer": "Reviewer_WQiA", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,  \nWe would like to express our gratitude to the reviewer for the careful reading and valuable comments. And we have carefully considered each comment and made necessary revisions to address the concerns raised. For presentation suggestion, in the revised manuscript, we will provide additional explanations and clarify each equation and symbols throughout the manuscript to reduce the confusions and improve the overall readability. Here we will explain the concerns point by point.\n\n$\\textbf{Question-1:}$ F1 score improvement is mainly due to the balance between precision and recall (increased recall while losing precision.) So, it is unsure if the proposed method is really competitive since some of the existing methods could easily increase F1 with the precision/recall balancing.  \n$\\textbf{Answer-1:}$ We agree that this balance is crucial in assessing the competitiveness of our method. However, we would like to emphasize that our method offers additional advantages beyond mere F1 score improvement. To support the competitiveness of our method, we conducted several ablation studies.\n* We demonstrated the enhancement of text representation quality achieved by our method from a novel perspective using 3D PCA visualization. This perspective allows us to evaluate whether the word/label bias issue gets sufficient alleviation, which also further reinforces the effectiveness of our approach. \n* We also evaluated the performance of our method across different label frequency groups. The results demonstrated an overall improvement among all label groups, further validating the competitiveness of our approach. \n* Our method achieved competitive performance in terms of Hamming Loss (another evaluation metric). The performance on F1 and Hamming Loss strengthens our claim of competitiveness.\n\n$\\textbf{Question-2:}$ There are too many hyperparameters to optimize and certain model decisions (e.g., using DGCNN) are not clearly described considering their complexities.  \n$\\textbf{Answer-2:}$  To capture more comprehensive information, we use DGCNN as the core of the coarse-grained information extraction layer. In DGCNN, \n* The dilated convolution is used to expand the receptive field of the overall structure so that it can capture more distant information.\n* The gated structure is used to transmit the coarse-grained text label interactive information in multiple channels.\n* The multi-head attention structure completes the integration of the entire sequence information.\n\nIn the revised manuscript, we will provide a detailed description of hyperparameters settings and a clear explanation of the DGCNN architecture and its advantages in capturing interactive information in the representation sequence. These clarifications will help readers better understand the complexities involved and provide a clear understanding of the optimization process.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "With the proposed approach, they achieved strong hamming loss and F1 score, showing the word/label bias issue is mitigated, helping multi-label classification with text labels.\n", "reasons_to_reject": "1. F1 score improvement is mainly due to the balance between precision and recall (increased recall while losing precision.) So, it is unsure if the proposed method is really competitive since some of the existing methods could easily increase F1 with the precision/recall balancing.\n\n2. There are too many hyperparameters to optimize and certain model decisions (e.g., using DGCNN) are not clearly described considering their complexities.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper addresses multi-label classification with text labels by utilizing contrastive learning as the base learning method and utilizes dual-graded interactive learning and constraint label relations learning to improve the performance considering the label frequency and correlation information.\n\nWith those approaches, they achieved strong hamming loss and F1 score, showing the word/label bias issue is mitigated, helping multi-label classification with text labels.\n\nHowever, F1 score improvement is mainly due to the balance between precision and recall (increased recall while losing precision.) So, it is unsure if the proposed method is really competitive since some of the existing methods could easily increase F1 with the precision/recall balancing.\nAlso, there are too many hyperparameters to optimize and certain model decisions (e.g., using DGCNN) are not clearly described considering their complexities.\n\n", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "L_CIL, L_FIL, and  L_CLRL should be explicitly defined to reduce confusions."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposed an interesting idea, an effective proposed method, supporting ablation tests and representation demonstrations, and insightful hyper-parameter analysis. However, this paper has limitations in poor writing quality, unclear transitions in equations, a lack of detail in methodology, and potential concerns about the competitive advantage of the proposed method in terms of F1 score improvement.
Decision: Accept-Findings