Instances and Labels: Hierarchy-aware Joint Supervised Contrastive Learning for Hierarchical Multi-Label Text Classification
Authors: Simon Chi Lok Yu
Jie He
Victor Gutierrez Basulto
Jeff Z. Pan
Venue: EMNLP 2023 Findings (2023)
Keywords: Natural language processing
Multi-label Classification
Contrastive Learning in NLP
TL;DR: Bridges the gap between Supervised Contrastive Learning and Hierarchical Multi-Label Text Classification

Abstract: Hierarchical multi-label text classification (HMTC) aims at utilizing a label hierarchy in multi-label classification. Recent approaches to HMTC deal with the problem of imposing an overconstrained premise on the output space by using contrastive learning on generated samples in a semi-supervised manner to bring text and label embeddings closer. However, the generation of samples tends to introduce noise as it ignores the correlation between similar samples in the same batch. One solution to this issue is supervised contrastive learning, but it remains an underexplored topic in HMTC due to its complex structured labels. To overcome this challenge, we propose **HJCL**, a **H**ierarchy-aware **J**oint Supervised **C**ontrastive **L**earning method that bridges the gap between supervised contrastive learning and HMTC. Specifically, we employ both instance-wise and label-wise contrastive learning techniques and carefully construct batches to fulfill the contrastive learning objective. Extensive experiments on four multi-path HMTC datasets demonstrate that HJCLachieves promising results and the effectiveness of Contrastive Learning on HMTC. Code and data are  available at https://github.com/simonucl/HJCL.

Instances and Labels: Hierarchy-aware Joint Supervised Contrastive
Learning for Hierarchical Multi-Label Text Classification
Simon Chi Lok U Jie Hel*
Víctor Gutiérrez-Basulto" Jeff Z. Pan
*University of Edinburgh
CCrddiff University
c. c. ed ac. .k, j.he@ed. ac. uk ac.uk
gutierrreebsuultoeccrrrfff ac. ac.uk, j
Abstract
Britan Covernmantionnn rasie ration's base lendry cate qui pomt
vestondiy 1a percent. Engand isss cemmerce banis,
Hierarchical multi-label text classification (HMTC) aims at utilizing a label hierarchy in multi-label classification. Recent approaches to HMTC deal with the problem of impos- ing an overconstrained premise on the output space by using contrastive learning on gener- ated samples in a semi-superviser manner to bring text and label embeddings closer. How- ever, the generation of samples tends to intro- duce noise as ignores the correlation between similar samples the same batch. One solu- tion to this issue is supervised contrastive learn- ing. but it remains an underexplored topic in HMTC due its complex structured labels. To overcome this challenge, we propose HJCI., Hierarchy-awan Joint Supervised Contrastive Learning method that bridges the gap between supervised contrastive learning and HMTC. Specifically, we employ both instance-wise and label-wise contrastive learning techniques and carefully construct batches to fulfill the contrastive learning objective. Extensive ex- periments on four multi-path HMTC datasets demonstrate that HJCL achieves promising re- sults and the effectiveness of Contrastive Learn- ing HMTC. Code and data are available at https//gitubb.co//immnnncc com/simonull//HILL..
persent about 288 parsent nearnss New York
undervine wntoten
Nev
Clawsifeds
Opiiion
Worlt
Maroetig, Bunking
Frans
Kinglan
Figure I: EExmmle of an input sample and its annotated labels from the New York Times dataset (Sandhaus, 2008). The Label Taxonomy a subgraph of the actual hierarchy.
task of assigning multiple hierarchically structured categories to documents is known as hierarchical multi-label text classification (HMTC).
A major challenge for HMTC is how to semanti- cally relate the input sentence and the labels in the taxonomy perform classification based on the hierarchy. Recent approaches to HMTC handle the hierarchy in global way by using graph neural net- works to incorporate the hierarchical information into the input text to pull together related input em- beddings and label embeddings in the same latent space (Zhou et al., 2020; Deng et al., 2021; Chen al., 2021: Wang al., 20226; Jiang et al., 2022). At the inference stage, most global methods reduce the learned representation into level-wise embed- dings and perform prediction in a top-down fashion to retain hierarchical consistency. However, these methods ignore the correlation between labels at different paths (with varying lengths) and different levels of abstraction.
1 Introduction
Text classification is a fundamental problem in nat- ural language processing (NLP), which aims to assign one multiple categories to given doc- ument based on its content. The task is essential in many NLP applications, e.g. discourse re- lation recognition (Chan et al., 2023), scientific document classification (Sadat and Caragea, 2022), c-commerce product categorization (Shen et al., 2021). In practice, documents might be tagged with multiple categories that can be organized in concept hierarchy, such taxonomy of knowl- edge graph (Pan al., 2017b,a), cf. Figure 1. The
To overcome these challenges, we develop a method based on contrastive learning (CL.) (Chen et al., 2020). So far, the application of contrastive learning in hierarchical multi-label classification
"The two authors contributed equally. Corresponding author.

---

has received very little attention. This is because it is difficult to create meaningful positive and neg- ative pairs: given the dependency of labels on the hierarchical structure, each sample could be char- acterized with multiple labels, which makes it hard to find samples with the exact same labels (Zheng et al., 2021). Previous endeavors in text classi- fication with hierarchically structured labels em- ploy data augmentation methods to construct posi- tive pairs (Wang al., 2022a; Long and Webber, 2022). However, these approaches primarily focus on pushing apart inter-class labels within the same sample but do not fully utilize the intra-class labels across samples. A notable exception is the work by Zhang et al. (2022a) in which CL is performed across hierarchical samples, leading to consider- able performance improvements. However, this method is restricted by the assumption of a fixed depth in the hierarchy i.c., assumes all paths in the hierarchy have the same length.
representations and task output, i.e., the hierarchi- cal relation between labels. Alignment refers the encoder being able to assign similar features to closely related samples/labels. We also emphasize that unlike previous methods (Zhang et al., 2022a), our approach has no assumption on the depth of the hierarchy.
Our main contributions are as follows:
We propose HJCL, representation learning approach that bridges the gap between super- vised contrastive learning and Hierarchical Multi-label Text Classification.
We propose a novel supervised contrastive loss on hierarchical structure labels that weigh based on both hierarchy and sample similar- ity, which resolves the difficulty of applying vanilla contrastive in HMTC and fully utilizes the label information between samples.
We evaluate HJCL on four multi-path datasets. Experimental results show its effectiveness. We also carry out extensive ablation studies.
To tackle the above challenges, introduce a supervised contrastive learning method, HJCL, based on utilising in-batch sample information for establishing the label correlations between samples while retaining the hierarchical structure. Techni- cally, HJCL aims at achieving two main goals: 1) For instance pairs, the representations of intra-class should obtain higher similarity scores than inter- class pairs, meanwhile intra-class pairs at deeper levels obtain more weight than pairs higher lev- els. 2) For label pairs, their representations should be pulled close if their original samples are similar. This requires careful choices between positive and negative samples to adjust the contrastive learning based on the hierarchical structure and label simi- larity. To achieve these goals, we first adopt text encoder and a label encoder to map the embeddings and hierarchy labels into shared representation space. Then, we utilize a multi-head mechanism to capture different aspects of the semanties to label information and acquire label-specific embeddings, Finally, we introduce two contrastive learning ob- jectives that operate at the instance level and the label level. These two losses allow HICL to learn good semantic representations by fully exploiting information from in-batch instances and labels. We note that the proposed contrastive learning objec- tives are aligned with two key properties related to CL: uniformity and alignment (Wang and Isola, 2020). Uniformity favors feature distribution that preserves maximal mutual information between the
Related Work
Hierarchical Multi-label Text Classification Existing HMTC methods can be divided into two groups based on how they utilize the label hi- erarchy: local or global approaches. The local approach (Kowsari et al.. 2017: Banerjee al., 2019) reuses the idea of flat multi-label classifi- cation tasks and trains unique models for each level of the hierarchy. In contrast, global meth- ods treat the hierarehy as a whole and train a single model for classification. The main objective is to exploit the semantic relationship between the input and the hierarchical labels. Exiistng meth- ods commonly use reinforcement learning (Mao et al., 2019), meta-learning (Wu et al., 2019), atten- tion mechanisms (Zhou al., 2020), information maximization (Deng et al., 2021), and matching networks (Chen et al., 2021). However, these meth- ods learn the input text and label representations separately. Recent works have chosen to incorpo- rate stronger graph encoders (Wang et al., 2022a), modify the hierarchy into different representations, e.g. text sequences (Yu et al., 2022), or directly in- corporate the hierarchy into the text encoder (Jiang et al., 2022; Wang et al., 2022b). To the best of our knowledge, HJCL is the first work to utilize supervised contrastive learning for the HMTC task. Contrastive Learning In HMTC, there are two major constraints that make challenging for su-

---

pervised contrastive learning (SCL) (Gunel et al., 2020) to be effective: multi-label and hierarchical labels. Indeed, SCL was originally proposed for samples with single labels, and determining posi- tive and negative sets becomes difficult. Previous methods resolved this issue mainly by reweight- ing the contrastive loss based on the similarity to positive and negative samples (Suresh and Ong, 2021; Zheng al., 2021). Note that the pres- ence of hierarchy exacerbates this problem. Con- trastiveIDRR (Long and Wehber, 2022) performed semi-supervised contrastive learning on hierarchy- structured labels by contrasting the set of all other samples from pairs generated via data augmenta- tion. Su et al. (2022b) addressed the sampling issue using ANN strategy on the trained samples. In contrast to previous methods, HJCL makes fur- ther progress by directly performing supervised contrastive learning on in-batch samples. In re- cent study in computer vision, HiMulConE (Zhang et al., 2022a) proposed a method similar to ours that focused on hierarchical multi-label classifica- tion with hierarchy of fix depth. However, HJCL does not impose constraints on the depth of the hier- archy and achieves this by utilizing attention mechanism.
projects the query Q, key K and value V onto h different heads and an attention function is applied individually to these projections. The output is a linear transformation of the concatenation of all attention outputs: The multi-headed attention is defined as follows (Lee et al., 2018):
Multihead K, V) = w [O.1|
(1) 3' where Oj = Attention (QW),, KW VWJ), and W3, Wh E R4x wy E Bdd and wo e Redd xd are learnable parameters in the multi-head attention. || represents the concatenation operation, di = dg/h and di du/h.
Supervised Contrastive Learning Given with samples and labels, we define the set of label embeddings as Z {zy E R i [1, m]. C [1, n]} and the set of ground-truth la- bels as Y = {Mn E [0, i E [1. E n]. Each label embedding can be seen as an inde- pendent instance and can be associated to a la- bel (Rty)Bry) We further define I = {zy E Z My 1} as the gold label set. Given an- chor sample 2ij from I, we define its positive set Pij {2kj E I ykj = Dij 1} and its negative set as Ni / M{zzi) U Pty- The supervised contrastive learning loss (SupCon) (Khosla et al., 2020) is formulated as follows:
3 Background
Task Formulation Let y [u1,,.. ... 9n be a set of labels. A hierarchy H = (T, r) is a labelled tree with T - EV) a tree and r V + y a labelling function. For simplicity, we will not dis- tinguish between the node and its label, i.c. label I will also denote the corresponding node. Given an input text X' {x1...., }m3 and a hierarchy H. the hierarchical multi-label text classification (HMTC) problem aims at categorizing the input text into a set of labels Y S V. i.e., at finding a function F such that given a hicrarchy, it maps a document x; to a label set Y C y. Note that, as shown in Figure 1, a label set Y could contain ele- ments from different paths in the hierarchy. We say that label set Y is multi-path if we can partition Y (modulo the root) into sets Y :...,, yk k 2, such that each Y is a path in H.
Coon
Σ Ziyel
Σ EpEP
log E. L exp(zi5 (a/T)
expl2ss . sp/T)
Pig
(2)
Methodology
The overall architecture of HJCL is shown in Fig. 2. In nutshell, HJCL first extracts label-aware embedding for each label and the tokens from the input text in the embedding space. HJCL combines two distinct types of supervised contrastive learn- ing to jointly leverage the hicrarchical informa- tion and the label information from in-batch sam- ples.: (i) Instance-level Contrastive Learning and (ii) Hierarchy-aware Label-enhanced Contrastive Learning (HiLeCon).
4.1 Label-Aware Embedding
Vaswani et al. (2017) extended the standard attention mechanism (Luong al., 2015) to allow the model to jointly attend to information from different representation sub- spaces at different positions. Instead of comput- ing a single attention function, this method first
Multi-headed Attention
In the context of HMTC, a major challenge is that different parts of the text could contain informa- tion related to different paths in the hierarchy. To overcome this problem, we first design and extract label- aware embeddings from input texts, with the

---

Audaner Sintrasth can (
GndT ShiNaN Castroltia 2
Erednosing
Aepresentationn Space
Clazsyfir
Graph Fusion
(b) Linat.
(c) Enccativn
Objection
(a)
(b)
(c)
Figure 2: The model architecture for HICL. The model is split into three parts: (a) shows the multi-headed attention and the extraction of aabeee embeddings; parts (b) and (c) show the instance- wise and label-wise contrastive learning. The legend on the lower left of (a) shows the labels corresponding each color. We use different colors identify the strength of contrast: the lighter the color, the less pushing/pulling between two instances/labels.
objective of learning the unique label embeddings between lahels and sentences in the input text.
tions and the learned embeddings strengthen the label information in the embeddings used by con- trastive learning, a; = (Billy] E R22d. An attention mechanism is then applied to the final represen- tation 24 = a. II where 2; E Rd, ag E Rmx1 ag = softmax( (W a.; + b.)) (Wa E Idd2ad and b E Rd are trainable parameters)
Following previous work (Wang et al., 2022a; Jiang et al., 2022), we use BERT (Devlin et al.. 2019) as text encoder, which maps the input tokens into the embedding space: H [hi...., hm)) ) where h2 is the hidden representation for each input token I; and E R For the label embeddings, we initialise them with the average of the BEERT-embedding of their text description, Y' ="= fy' .'), ".),' Y' E Raxdd_ To learn the hi- erarchical information, graph attention network (GAT) (Velickovic et al., 2018) is used to propagate the hierarchical information between nodes in Y'.
Instance-level Contrastive Learning For instance-wise contrastive learning, the objective is simple: the anchor instances should be closer to the instances with similar label-structure than to the instances with unrelated labels, ef. Fig. 2. Moreover, the anchor nodes should be closer to positive instance pairs at deeper levels in the hierarchy than to positive instance pairs at higher levels. Following this objective, we define a distance inequality: distt, pos < distf pos < dist" where 1 < & < E1 S L and dist, now is the distance between the anchor instance X₃ and X', which have the same labels at level l.
Afte mapping them into the same representa- tion space, we perform multi-head attention as defined at Eq. I, by setting the jt label embed- ding Y/ as the query Q, and the input tokens rep- resentation H as both the key and value. The label-aware embedding 9i is defined as follows: gi Multihead(y H, (I,II II), where i e [1, and g: C TRd. Each gi is computed by the attention weight between the label y/: and each input token in H, then multiplied by the input tokens in H to get the label-aware embeddings. The label-aware em- bedding gı can be seen as the pooled representation of the input tokens in H weighted by its semantic relatedness to the label yi-
Given a mini-batch for instances ((Z Yi))
where Z, = {zy j E [0, n]}: Z5 E Raxd con- tains the label-aware embeddings for sample i, we define their subsets at level E as Z = {zij 2ij E Zi, depth(wn) < f. Y (wij) depth(us) I f).
exp(x X5/-)
Clavel (Zf, ZT) = =log
Σ ExeeNhy exp(X; X
where X average(Z)) (Z)) and x Rd is the mean pooling representation of z E I xd.
4.2 Integrating with Contrastive Learning
Following the general paradigm for contrastive learning (Khosla et al., 2020; Wang et al., 2022a), the learned embedding 9i has be projected into a new subspace, in which contrastive learning takes
Line. +
clod ff (/..2)).. expl
P
[
2
place. Taking inspiration from Wang et al. (2018)
where L {1..... {x } is the set of levels in the taxonomy, [L] is the maximum depth and the term
and Liu et al. (2022), we fuse the label representa-

---

exp( 7)) is penalty applied to pairs constructed from deeper levels in the hierarchy, forcing them to be closer than pairs constructed from shallow levels.
Big = p(y; Yj) C
Tik = p(v,, 1x)
(3)
where p(,, 1,) is used to normalize the di values. HiLeCon is then defined as
Label-level Contrastive Learning We will also introduce label-wise contrastive learning. This is possible due extraction of label-aware beddings in Section 4.1, which allows us learn each label embedding independently. Although Equation 2 performs well in multi-class classifi- cation (Zhang et al., 2022b), it is not the case for multi-label classification with hierarchy. (1) It ignores the semantic relation from their origi- nal sample (&), (2) Nij contains the label embeddings from the same samples but with differ- ent classes, Zik- Pushing apart labels that are con- nected the hierarchy could damage the classifica- tion performance. To bridge this gap, we propose a Hierarchy-awand Gabel-Enhancee Contrastive Loss Function (Hil.eCon), which carefully weighs the contrastive strength based on the relatedness of the positive and negative labels with the anchor labels. The basic idea is to weigh the degree of con- trast between two label embeddings 2₁. 2j by their samples' label similarity, Yi, Yj E {0, 13". In par- ticular, in supervised contrastive learning, the gold labels for the samples from where the label pairs come can be used for their similarity measurement. We will use a variant of the Hamming metric that treats differently labels occurring at different levels of the hierarchy, such that pairs of labels at higher level should have larger semantic difference than pairs of labels at deeper levels. Our metric between Y and Y, defined follows:
LislaCim N
Σ
(4)
Pi
ApEPss
[log L LecePiy ouf( Fa) +> (au ms)
where n is the number of labels and ((:-) is the exponential cosine similarity measure between two embeddings. Intuitively, in LHiLeCon the label em- beddings with similar gold label sets should be close each other in the latent space, and the mag- nitude of the similarity is determined based on how similar their gold labels are. Conversely, for dis- similar labels.
4.3 Classification and Objective Function
At the inference stage, we flatten the label-awar embeddings and pass them through a linear layer to get the logits s; for label i:
S = (W. g2 (g.)) bb,)
(5)
where W, C Raxnd bg R R S
na and { Instead of Binary Cross-entropy. we use the novel loss function "Zero-bounded Log- sum-exp exp & Pairwise Rank-based" (ZLPR) (Su et al., 2022a), which captures label correlation in multi-label classification:
p(Y,,Y,)
Σ
dist(yrke Bykk)
CZLER log
1+
Σ
c
-5
+ log
1+
Σ e
LE
JEA
R=O
[L]]
l. +1
Yik + Uikk Otherwise
dist(yik. Ujs))
where 8i, 8j E IR are the logits output from the Equation (5). The final prediction is follows:
where is the level of the i-th label in the hier- archy. For example, the distance between News and Classifields in Figure 1 is 4, while the distance between United Kingdom and France is only 1. In- tuitively, this is the case because United Kingdom and France are both under Countries, and samples with these two labels could still share similar con- texts relating the World News.
y Ypred = {ui > 0}
(6)
Finally, we define our overall training loss function:
L = λ Linst. + Ag LHILeCon
(7)
where \ι and Az are the weighting factors for the Instance-wise Contrastive loss and HiLeCon.
We can now use our metric to set the weight between positive pairs Zij E Pit and negative pairs Zik E NGj in Eq. 2:
'The maximum value for p. i.c. the distance between the empty label sets and label sets with all lahels.

---

BGC
AAPD
RCVI-V2
NYT
Micro-FI
Micro-11
Micno-FI
Macro-F1
Micro-F1
Macro-F1
Micro-Fi
Macro-F1
Hierarchy-Aware Models
TeXRCNNN
81.57
59.25
70.83
56.18
IRAGM
77.22
57.91
$3.96
63.35
74.97
60.83
HTCInfoMax
76.84
58.01
79.00 80.74
54.48
83.51
62.71
59,47
HiMarch
76.57
58.34
56.16
84.73
64.11
74.65
58.26
Instruction- Tuned Language Model
CbaGPT
27.98
51.35 2004
Pretrained Language Models
BERT
78.84
61.19
80.88
57.17
85.65
67.02
78.24
65.62
HiAGM (BERT)
79.48
62.84
80.68
59.47
85.58
67.93
78.64
66.76
HTCInfoMa: (BERT)
79.16
62.94
80.76
59.46
85.83
67.09
78.75
67.31
HiMarch (BERT)
78.89
63.19
80.42
19.23
86.33
68.66
Seg2Tree (13)
79.72
63.96
80.55
59.58
70.01
IiMalConnE (BERTyA
79.19
60.85
80,98
57.75
85.89
66.65
77.53
61.08
IIGCLR (BERT)
79.22
64.04
80.95
59.34
86.49
68.31
78.86
67.96
(BERT)
81.30
81.30
66.77
22.33
81.91 1095
61.59
'2.1
70.16
70.49
1044
80.52
70.02
12/0
Table 1: Experimental results on the four HMTC datasets. The best results are in bold and the second- best is underlined. We report the mean results across 5 runs with random seeds. Models with are those using contrastive learning. For HiAGM, HTCInfoMax and HiMatch, their works used TextRCNN (Zhou et al., 2020) as encoder in their paper, we replicate results by replacing it with BERT. represents improvement to second model; the + represent the std. between experiments.
5 Experiments
5.1 Main Results
Table presents the results on hierarchical multi- label text classification. More details can be found in Appendix A. From Table 1, one can observe that HJCL significantly outperforms the baselines. This shows the effectiveness of incorporating su- pervised contrastive learning into the semantic and hierarchical information. Note that although HG- CLR (Wang al., 2022a) introduces a strönger graph encoder and perform contrastive learning on generated samples, inevitably introduces noise into these samples and overlooks the label correla- tion between them. In contrast, HJCL uses a sim- pler graph network (GAT) and performs contrastive learning on in-batch samples only, yielding signifi- cant improvements of 2.73% and 2.06% on Macro- FI in BGC and NYT. Despite Seq2Tree's use of more powerful encoder, T5, HJCL still shows promising improvements 2.01% % and 0.48% on Macro-F1 AAPD and RCV1-V2, respectively. This demonstrates the use of contrastive model het- ter exploits the power of BERT encoder. HiMul- ConE shows drop in the Macro-F1 scores even when compared to the BERT baseline, especially NYT, which has the most complex hierarchi- cal structure. This demonstrates that our approach to extracting label-aware embedding is an impor- tant step for contrastive learning for HMTC. For the instruction tuned model, ChatGPT performs poorly, particularly suffering from minority class
Datasets and Evaluation Metries We conduct experiments four widely-used HMTC bench- mark datasets, all of them consisting of multi-path labels: Blurb Genre Collection (BGC)?, Arxiv Academic Papers Dataset (AAPD) (Yang et al., 2018), NY-Times (NYT) (Shimura et al., 2018), and RCVI-V2 (Lewis et al.). Details for each dataset are shown in Table 5. We adopt the data processing method introduced in Chen et al. (2021) remove stopwords and use the same evaluation metrics: Macro-F1 and Micro-F1.
Baselines We compare HJCL with a variety strong hierarchical text classification baselines, such as HiAGM (Zhou et al., 2020), HTCInfoMax (Deng et al., 2021), HiMatch (Chen et al., 2021). 1). Seq2Tree (Raffel et al., 2019), HGCLR (Wang al., 2022a). Specifically, HiMulConE (Zhang al., 2022a) also uses contrastive learning on the hierarchical graph. More details about their imple- mentation are listed A.2. Given the recent ad- vancement in Large Language Models (LLMs), we also consider ChatGPT pot-turbb-3.5 (Brown et al., 2020) with zero-shot prompting as a base- line. The prompts and examples of answers from ChatGPT can be found in Appendix C.

---

performance. This shows that remains challeng- ing for LLMs handle complex hierarchical in- formation, and that representation learning still necessary.
F1
5.2 Ablation Study
RCVI-V2
NYT
Ablation
Figure 3: Effects of è (left) and X (right) on NYT and
Micro-F1
Macro-FI
Micro-FI
Macro-F]
RCVI. The step size for A 0.1 and X2 is 0.2. X has
Ours
87.04
70.49
80.52
70.02
Label
86.67
69.26
79.90
69.15
smaller step size since it more sensitive changes.
6m
Insiance con.
86.83
68.38
79.71
69.28
Both con.
86.39
68.06
79.23
68.21
4p.
79.74
Gruph Fusion 85.15
67.61
79.03
67.25
Table 2: Ablation study when removing components
Moro- (99)
Marro F1 (M
on the RCVI-V2 and NYT datasets. r.m stands for removing the component: rp. stands for replace with.
To better understand the impact of the differ- ent components of HJCL performance, con- ducted an ablation study on both the RCV1-V2 and NYT datasets. The RCVI-V2 dataset has substantial testing set, which helps to minimise experimental noise. In contrast, the NYT dataset has the largest depth. One can observe in Table that without label contrastive the Macro-F1 drops notably in both datasets, 1.23% and 0.87%. The removal of HiLeCon reduces the potential for label clustering and majorly affects the minority labels. Conversely, Micro-Fi is primarily affected by the omission of the sample contrast, which prevents the model from considering the global hierarchy and learning label features from training instances of other classes, based on their hicrarchical inter- dependencies. When both loss functions are re- moved, the performance declines drastically. This demonstrates the effectiveness of our dual loss ap- proaches.
HiLeCo
LeCon
Figure 4: scores different datasets with different contrastive methods. The texts above the bar show the offset between the models to the SupCon model.
label-wise contrastive loss, respectively, Figure 3 illustrates the changes on Macro-FI when varying the values X and X2- The left part of Figure 3 shows that the performance peaks with small A values and drops rapidly as these values continue to increase. Intuitively, assigning too much weight to the instance-level CL pushes apart similar samples that have slightly different label sets, preventing the models from fully utilizing samples that share similar topics. For X2, the Fl score peaks 0.6 and 1.6 for NYT and RCVI-V2, respectively. We attribute this difference to the complexity of the NYT hierarchy, which is deeper. Even with the as- sistance of the hierarchy-aware weighted function (Eq. 3), increasing A2 excessively may result in overwhelmingly high semantic similarities among label embeddings (Gao et al., 2019). The remaining results are provided in Appendix B.
Additionally, replacing the ZMLR loss with BCE loss results a slight performance drop, showcasing the importance of considering label correlation during the prediction stage. Further analysis between BCE loss and ZLPR is shown in Appendix B.3. Finally, shown in the last row in Table 2, the removal of the graph label fusion has significant impact on the performance. The pro- jection is shown affect the generalization power without the projection head CL (Gupta et al., 2022). Ablation results on other datasets can be found in Appendix B.I.
5.4 Effect of the Hierarchy-Aware Label Contrastive loss
To further evaluate the effectiveness of HiLeCon in Eq. 4, conduct experiments by replacing it with the traditional SupCon (Khosla et al., 2020) or dropping the hierarchy difference by replacing the p(-,)) at Eq. with Hamming distance, LeCon. Figure 4 presents the obtained results and detailed
Effects of the Coefficients \ı and X2 5.3
As shown in Equation (7), the coefficients A and 2 control the importance of the instance-wing and

---

Micro-F1 RCVI-V2
Macro-F1 RCVI-V2
Method
BGC
AAPD
NYT
p-value
BGC
AAPD
NYT
p-value
HiLeCon
81.30
81.91
87.04
80,52
66.77
61.59
70.49
70.02
LeCon
80.63
81.78
86.23
79.91
3.3e-2
65.12
61.42
69.01
68.57
4.0c-2
SupCon
78.64
80.70
84.54
78.64
8.3e-3
62.44
58.86
67.53
66.97
2.8e-3
Table 3: Comparison results on different Contrastive Learning approaches on the Label embedding, performed on the 4 datasets. HiLeCon denotes our proposed method. The p-value is calculated by two-tailed I-tests.
Dataset
Model
Accp
Dcc
upper levels but missed more fine-grained labels at lower levels. To compare the performance on these problems, we measure path accuracy (Acep) and depth accuracy (Acep), which are the ratio of testing samples that have their path number and all their depth correctly predicted. (Their definitions are given the Appendix B.4). As shown in Table 4, HJCL (and its variants) outperformed the base- lines, with an offset of 2.4% on average compared with the second-boc best model HGCLR. Specifically, the Accp for HJCL outperforms HGCLR with an absolute gain of 5.5% in NYT, in which the ma- jority of samples are multi-path (ef. Table 9 in the Appendix). HJCL shows performance boosts for multi-path samples, demonstrating the effective- ness of contrastive learning.
HJCL
75.22
71.96
HJCL (w/o con)
70.94
67.62
NYT
HGCLR
71.26
70.47
BERT
70.48
65.65
HJCL
63.61
79.26
HICL (w/o con)
60.50
75.83
RCVI
HGCLR
62.99
78.62
BERT
61.90
75.60
Table 4: Measurement for Acep and D NYT and RCVI. The best scores are in bold and the second best underlined. The formula and results on BGC and AAPD are shown in Appendix B.4.
results are shown in Table 3. HiLeCon outperforms the other two methods in all four datasets with substantial threshold. Specifically, HiLeCon sig- nificantly outperforms the traditional SupCon in the four datasets by an absolute gain of 2.06% and 3.27% in Micro-Fl and Macro-F1, respectively. As shown in Table 3, the difference in both met- rics is statistically significant with p-value 8.3e-3 and 2.8e-3 by a two-tailed t-test. Moreover, the improvement from LeCon in F'l scores by consid- ering the hierarchy is 0.56% and 1.19% which statistically significant (p-value 0.033, 0.040). This shows the importance of considering label granularity with depth information.
5.6 Qualitative Analysis
Betee ditee
Basel dades
Pastvarr La Bredlt
1.
(a) Gold Labels
(b) HJCL
5.5 Results on Multi-Path Consistency
One of the key challenges in hicrarchical multi- label classification is that the input texts could be categorized into more than one path in the hi- erarchy. In this section, we analyze how HJCL leverages contrastive learning to improve the cov- erage of all meanings from the input sentence. For HMTC, the multi-path consistency can be viewed from two perspectives. First, some paths from the gold labels were missing from the prediction, mean- ing that the model failed to attribute the semantic information about that path from the sentences; and even if all the paths are predicted correctly, is only able to predict the coarse coarse-grained grained labels at
Fresabes Cil Butt 6.2
(c) HGCLR
(d) HICL (w/o con)
Chaimar. waralag that HiL Cliates was elected as the U-5- Presidert, Jane VoUtD stepping
guest hoorr state duner....
Figure 5: Case study on sample from the NYT dataset. Orange represents true positive labels; Green repre- sents false negative labels; Red represents false positive labels: Blue represents true negative labels. The +-- indicates that two nodes are skipped. Part of the input texts is shown at the bottom, the full text and prediction results are in Appendix B.
HICL better exploits the correlation be-

---

tween labels in different paths in the hierar- chy with contrastive learning. For an intu- ition see the visualization in Figure 9, Ap- pendix B.5. For example, the Fi score of America/United States is only 0.3350 for the HG- CLR method (Wang et al., 2022a). In contrast, our methods that fully utilised the label correlation information improved the Fi score 0.8176. Fig- ure 5 shows case study for the prediction results from different models. Although HGCLR is able to classify U.S. under News (the middle path), fails to take into account label similarity informa- tion to identify the United States label under the Features path (the left path). In contrast, our mod- els correctly identify U.S. and Washington while addressing the false positive for Sports under the News category.
Acknowledgements
This work partially supported by the Chang Jiang Scholars Program (I2019032). We are grateful Andrej Jovanović for helpful discussions, and to anonymous reviewers for their valuable feedback.
Limitations
Our method is based on the extraction of a label- aware embedding for each label in the given tax- onomy through multi-head attention and performs contrastive learning the learned embeddings. Although our method shows significant improve- ments, the use of label-aware embeddings scales according to the number of labels in the taxon- omy. Thus, our methods may not be applica- ble for other HMTC datasets which consist of a large number of labels. Recent studies (Ni et al., 2023) show the possible improvement of Multi- Headed Attention (MHA), which is to reduce the over-paramettrization posed by the MHA. Further work should focus on reducing the number of label- aware embeddings but still retaining the compara- ble performance.
6 Conclusion
We introduce HJCL a combination of two novel contrastive methods that better learn the represen- tation for embedding in Hierarchical Multi-Labed Text Classification (HMTC). Our method has the following features: (1) It demonstrates that con- trastive learning can help retain the hierarchy infor- mation between samples. (2) By weighting both label similarity and depth information, applying supervised contrastive learning directly at the label level shows promising improvement. (3) Evalu- ation four multi-path HMTC datasets demon- strates that HJCL significantly outperforms previ- ous baselines and shows that in-batch contrastive learning notably enhances performance. Overall, HJCL bridges the gap between supervised con- trastive learning in hierarchical structured label classification tasks in general and demonstrates that better representation learning is feasible for improving HMTC performance.
Ethics Statement
An ethical consideration arises from the underlying data used for experiments. The datasets used in this paper contain news articles (e.g. NYT & RCVI- V2), books abstract (BGC) and scientific paper abstract (AAPD). These data could contain bias (Baly et al., 2018) but were not being preprocessed to address this. Biases in the data can be learned by the model, which can have significant societal impact. Explicit measures to debias data through re- annotation or restructuring the dataset for adequate representation is necessary.
References
Ramy Baly, Georgi Karadzhov, Dimitar Alexandrov, James Glass, and Preslav Nakov, 2018 Predict- ing factuality of reporting and bias of news media sources. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3928-3539, Brussels, Belgium. Association for Computational Linguistics.
In the future, we plan to look into applying our approach in some special kinds of texts, such as arguments (Saadat-Yare et al., 2022, 2023; Chaus- son et al., 2023), news (Pan et al., 2018; Liu et al., 2021; Long et al., 2020a,b) and events (Guan et al., 2023). Furthermore, we will also further develop our approach in the setting of multi-modal (Kiela et al., 2018; Chen et al., 2022b; Huang al., 2023; Chen et al., 2022a) classification, involving both texts and images.
Siddhartha Banerjee, Cem Akkaya, Francisco Perez- Sorrosal, and Kostas Tsioutsiouliklis. 2019. Hierar- chical Transfer Learning for Multi-label Text Classifi cation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2295-63000 Florence, Italy. Association for Compu- tational Linguistics.

---

Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wen- liang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, Quyet V. Do. Yan Xu, and Pascale Fung. 2023. A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hal- lucination, and interactivity. ArXiv, abs/2302.04022.
guage Technologies. pages 3259-3265, Online. As- sociation for Computational Linguistios.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Tech- nologies, Volume (Long and Short Papers), pages 4171-41886 Minneapolis, Minnesota. Association for Computational Linguistics.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, T. Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeff Wu. Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. ArXiv, abs/2005.1416 141665.
Jun Gao, Di He, Xu Tan, Tao Qin, Liwei Wang, and Tie- Yan Liu. 2019. Representation degeneration prob- lem in training natural language generation models, ArXiv, abs/1907.
Yong Guan, Jiaoyan Chen, Freddy Lécué, Jeff Z. Pan, Juan Li, and Ru Li. 2023. Trigger-argument based explanation for event detection. In Annual Meeting of the Association for Computational Linguistics.
Chunkit Chan, Xin Liu, Cheng Jiayang, Zihan Li, Yangqiu Song. Ginny Y. Wong, and Simon Chong Wee See. 2023. Discoprompt: Path prediction prompt tuning for implicit discourse relation recogni- tion. ArXiv, abs/2305 03973.
Beliz Gunel, Jingfei Du Alexis Conneau, and Ves Stoyanov. 2020. Supervised contrastive learning for pre-trained language model fine-tuning. ArXiv, abs/201 1.01403.
Sandrine Chausson, Ameer Hassan Saadat-Yazdi, Xue Li. Jeff Z. Pan, Vaishak Belle, Nadin Kökciyan, and Björn Ross. 2023. A web-based tool for detecting argument validity and novelty. In Adaptive Agents and Multi-Agent Systems.
Kartik Gupta. Thalaiyasingam Ajanthan, Anton van den Hengel, and Stephen Gould. 2022. Understanding and improving the role of projection head in self- supervised learning. ArXiv, abs/2212.11491.
Haibin Chen, Qianli Ma. Zhenxi Lin, and Jiangyue Yan. 2021. Hierarchy-awan aware label semantics matching net- work for hierarchical text classification, In Proceed- ings of the 59th Annual Meeting of the Association for Computational Linguisties and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 4370-4379, Online. Association for Computational Linguisties.
Y. Huang, Z. Chen, J. Chen, J. Z. Pan, Z. Yao, and W. Zhang. 2023. Target-Oriented Sentiment Classifi- cation with Sequential Cross-Modal Semantic Graph. In ICANN, pages 587-599.
Ting Jiang, Deqing Wang, Leilei Sun, Zhongzhi Chen, Fuzhen Zhuang, and Qinghong Yang. 2022. Exploit- ing global and local hierarchies for hierarchical text classification. In Proceedings of the 2022 Confer- ence on Empirical Methods Natural Language Processing, pages 4030-4039, Abu Dhabi, United Arab Emirates. Association for Computational Lin- guistics.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey E. Hinton. 2020. A simple framework for contrastive learning of visual representations. ArXiv, abs/2002.05709.
Zhuo Chen, Jiaoyan Chen, Wen Zhang, Linghing Guo, Yin Fang, Yufen Huang, Yuxia Geng, Jeff Pan, Wenting Song, and Hua zeng Chen. 2022a. Meaformer: Multi-modal entity align- ment transformer for meta modality hybrid. ArXiv, 11112222.14444
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. 2020, Super- vised contrastive learning. ArXiv, abs/2004. 1362.
Zhuo Chen, Yufen Huang, Jiaoyan Chen, Yuxia Geng, Wen Zhang. Yin Fang, Jeff Z. Pan, Wenting Song, and Huajun Chen. 2022b. Duet: Cross-modal semantic grounding for contrastive zcro-shot learning. In AAAI Conference on Artificial Intelligence.
Douwe Kiela, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2018 Efficient large-scale multi- modal classification. In AAAI Conference on Artifi- cial Intelligence.
Zhongfen Deng, Hao Peng, Dongxiao He, Jianxin Li, and Philip Yu. 2021. HTCInfoMax: global model for hierarchical text classification via information maximization. In Proceedings of the 2021 Confer- ence of the North American Chapter of the Associ- ation for Computational Linguistics: Human Lan-
Kamran Kowsari, Donald E Brown, Mojtaba Hei- darysafa, Kiana Jafari Meimandi, Matthew S Gerber. and Laura E. Barnes. 2017. Hdltex: Hierarchical deep learning for text classification. In 2017 16th IEEE International Conference on Muchine Learning and Applications (ICMIA), pages 364-371.

---

Juho Lee. Yoonho Lee, Jungtack Kim, Adam R. Ko- siorek, Seungjin Choi, and Yee Whye Teh. 2018. Set transformer: A framework for attention-based permutation--nvariinn neural networks. In Interna- lional Conference on Machine Learning.
Jeff Z. Pan, Diego Calvanese, Thomas Eiter, lan Hor- rocks, Michael Kifer, Fangzhen Lin, and Yuting Zhao. 2017a. Reasoning web: Logical foundation of knowledge graph construction and query answering 12th international summer school 2016, aberdeen, uk, september 5-9, 2016, tutorial lectures. In Reasoning Web.
David D Lewis, Yiming Yang. Tony G Rose, and Fan Li. RCVI: A New Benchmark Collection for Text Categorization Research.
Jeff Z. Pan, Siyana Pavlova, Chenxi Li, Ningxi Li, Yang- mei Li. and Jinshuo Liu. 2018. Content based fake news detection using knowledge graphs. In Interna- tional Semanric Web Conference (l), volume 11136 of Lecture Notes in Computer Science, pages 669- 683. Springer.
Ham Liu, Feng Zhang, Xiaotong Zhang, Siyang Zhao, Ju Sun, Hong Yu, and Xianchao Zhang. 2022. Label- enhanced prototypical network with contrastive learn- ing for multi-label label few-shot aspect category detection. Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining.
Jeff Z. Pan, Guido Vetere, José Manuél Gómez-Pérez, and Honghan Wu. 7b. Exploiting linked data and knowledge graphs in large organisations. Springer International Publishing.
Jinshuo Liu, Chenyang Wang. Chenxi Li, Ningxi Li, Juan Deng, and Jeff Z. Pan. 2021. DTN: Deep triple network for topic specific fake news detection, Jour- nal of Web Semantics.
Colin Raffel, Noam M. Shazeer, Adam Roberts, Kather- ine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21:140:1-4: 140:66
Wanqiu Long, Xinyi Cai, James Reid, Bonnie Lynn Webber, and Deyi Xiong. 2020a. Shallow discourse annotation for chinese ted talks. In International Conference Language Resources and Evaluation.
Ameer Hassan Saadat- Yazdi, Xue Li, Sandrine Chaus- son, Vaishak Belle, Björn Ross, Jeff Z. Pan, and Nadin Kökciyan. 2022. Kevin: A knowledge en- hanced validity and novelty classifier for arguments. In Workshop on Argument Mining.
Wanqiu Long and Bonnie Webber. 2022. Facilitating contrastive learning of discourse relational senses by exploiting the hierarchy of sense relations. In Pro- ceedings of the 2022 Conference on Empirical Meth- ods in Natural Language Processing, pages 10704 10716, Abu Dhabi, United Arab Emirates. Associa- tion for Computational Linguistics.
Ameer Hassan Saadat-Yazdi, Jeff Z. Pan, and Nadin Kokeiyan. 2023. Uncovering implicit inferences for improved relational argument mining. Confer- ence of the European Chapter of the Association for Computational Linguistics.
Wanqiu Long, Bonnie Webber, and Deyi Xiong. 2020b. TED-CDB: A large-scale Chinese discourse relation dataset on TED talks. In Proceedings of the 2020 Conference on Empirical Methods in Natural Lan- guage Processing (EMNIP)), pages 2793-2803, On- line. Association for Computational Linguistics.
Mobashir Sadat and Cornelia Caragea. 2022. Hierarchi- cal multi-label classification of scientific documents. In Proceedings of the 2022 Conference Empiri- cal Methods in Natural Language Processing, pages 8923-8937, Abu Dhabi, United Arab Emirates. As- sociation for Computational Linguistics.
Ilya Loshchilov and Frank Hutter. 2017. Decoupled weight decay regularization. In International Confer- ence on Learning Representations.
Evan Sandhaus. 2008. The New York Times Annotated Corpus. Artwork Size: 3250585 KB Pages: 3250585 KB Type: dataset.
Thang Luong, Hieu Pham, and Christopher D. Manning. 2015. Effective approaches to attention- based neural machine translation. In Proceedings of the 2015 Con- ference on Empirical Methods in Natural Language Processing, pages 1412-1421, Lisbon, Portugal. As- sociation for Computational Linguistics.
Jiaming Shen, Wenda Qiu, Yu Meng. Jingbo Shang. Xiang Ren, and Jiawei Han. 2021. TaxoClass: Hi- erarchical multi-label text classification using only class names. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Tech- nologies, pages 4239-4249) Online. Association for Computational Linguistics.
Yuning Mao, Jingjing Tian, Jiawei Han, and Xiang Ren. 2019. Hierarchical text classification with reinforced label assignment. In Proceedings of the 2019 Confer- ence on Empirical Methods Natural Language Pro- cessing and the 9th International Joint Conference on Natural Language Processing LEMNLP-LJCNLL pages 445-455, Hong Kong, China, Association for Computational Linguistics.
Kazuya Shimura, Jiyi Li, and Fumiyo Fukumoto. 2018. HFT-CNN Learning hierarchical category structure for multi label short text categorization. In Proceed- ings of the 2018 Conference on Empirical Methods in Natural Language Processing. pages 811-816, Brussels, Belgium. Association for Computational Linguistics.
Jinjie Ni, Rui Mao, Zonglin Yang, Han Lei, and E. Cam- bria. 2023. Finding the pillars of strength for multi- head attention. ArXiv, 111122355.438000

---

Jianlin Su, Min Zhu, Ahmed Murtadha, Shengfeng Pan, Bo Wen, and Yunfeng Liu. 2022a. Zipr: A novel loss for multi-label classification, ArXiv, abs/2208.0299.
text classification. In Proceedings of the 2022 Con- ference on Empirical Methods in Natural Language Processing, pages 3740-3751, Abu Dhabi, United Arab Emirates. Association for Computational Lin- guistics.
Xi'an ao Su, Ran Wang, and Xinyu Dai. 2022b. Con- nearest neighbor mecha- nism for multi-labed text classification. In Proceed- ings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Pa- pers), pages 672-679, Dublin, Ireland. Association for Computational Linguistics.
trastive learning-enhancee
Jiawei Wu, Wenhan Xiong, and William Yang Wang. 2019. Learning to learn and predict: A meta-learning approach for multi-label classification. In Proceed- ings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th Inter- national Joint Conference on Natural Language Pro- cessing (EMNLP-IJCNLP)) pages 4354-4364, Hong Kong, China. Association for Computational Linguis- tics.
Varsha Suresh and Desmond Ong. 2021. Not all neg- atives are equal: Label-aware contrastive loss for fine-grained text classification. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 4381-4394 Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Pengcheng Yang. Xu Sun, Wei Li, Shuming Ma, Wei Wu, and Houfeng Wang. 2018. SGM: Sequence generation model for multi-label classification. In Proceedings of the 27th International Conference on Computationall Linguistics, pages 3915-3926, Santa Fe, New Mexico, USA. Association for Computa- tional Linguistics.
Laurens van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-sne. Journal of Machine Learning Research, 986))2577-2665.
Ashish Vaswani, Noam M. Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Neural Information Processing Systems.
Chao Yu, Yi Shen, Yue Mao, and Longjun Cai. 2022. Constrained sequence-to-tree generation for hierar- chical text classification. Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval.
Petar Velickovic. Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio', and Yoshua Bengio. 2018. Graph attention networks. In International Conference on Learning Representations (ICLR).
Shu Zhang, Ran Xu, Caiming Xiong, and Chetan Ra- maiah. 2022a. Use all the labels: A hierarchical multi-lahel contrastive learning framework. 2022 JEEF/CVF Conference on Computer Vision and Pat- tern Recognition (CVPR), pages (6699-16648.
Guoyin Wang, Chunyuan Li, Wenlin Wang, Yizhe Zhang, Dinghan Shen, Xinyuan Zhang. Ricardo Henao, and Lawrence Carin. 2018 Joint embed- ding of words and labels for text classification. In Proceedings of the 56th Annual Meeting of the As- sociation for Computational Linguistics ( (Volume 1: Long Papers), pages 2321-2331, Melbourne, Aus- tralia. Association for Computational Linguistics.
Zhenyu Zhang, Yuming Zhao, Meng Chen, and Xi- aodong He. 2022b. Label anchored contrastive learn- ing for language understanding, In Proceedings of the 2022 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, pages 1437-1449, Seattle, United States. Association for Computational Linguistics.
Tongzhou Wang and Phillip Isola. 2020. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In International Conference on Machine Learning.
Lecheng Zheng, Jinjun Xiong, Yada Zhu, and Jingrui He. 2021. Contrastive learning with complex het- erogeneity. Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Min- ing.
Weiqi Wang. Tianqing Fang. Wenxuan Ding, Baixuan Xu, Xin Liu, Yangqiu Song, and Antoine Bosse- lut. 2023. Car: incceptullzzatioo-aageeete soner for zero-shot commonsense question answer- ing. ArXiv, abs/2305. 14869.
rea-
Jie Zhou, Chunping Ma, Dingkun Long, Guangwei Xu, Ning Ding, Haoyu Zhang, Pengjun Xie, and Gong- shen Liu, 2020. Hierarchy-awan global model for hierarchical text classification. In Proceedings of the 58th Annual Meeting of the Association for Compu- tational Linguistics, pages 1106-1117, Online, Asso- ciation for C Computational Linguistics.
Zihan Wang, Peiyi Wang, Lianzhe Huang, Xin Sun, and Houfeng Wang. 2022a. Incorporating hierarchy into text encoder: a contrastive learning approach for hierarchical text classification. In Proceedings of the 60th Annual Meeting of the Association for Compustational Linguistics (Volume I: Long Papers), pages 7109-71 (109-7119,900 Dublin, Ireland. Association for Computational Linguistics.
Zihan Wang, Peiyi Wang, Tianyu Liu, Binghuai Lin, Yunbo Cao, Zhifang Sui, and Houfeng Wang. 2022b. HPT: Hierarchy-awee prompt tuning for hierarchical

---

Dataset
Avg(!.))
Train
Des
Test
Ablation Models
AAPD
BGC
Micro-F
Macro
Micro Fl
Mazro F1
BGC
3.01
58,715
14,785
18,394
AAPD
61
4.09
53,840
1,000
1,000
Ours
81.91
61.59
81.30
66.77
RCVI-V2
103
3.24
20,833
2,316
781.255
Label
80.72
NYT
7.60
23,345
5,834
7,292
cmc. lastance con.
81.79
60.47
80.85
65.89
Roth
80.47
80.57
68.12
BCB Loss
80:95
59.73
80.48
63.87
Table 5: Dataset statistics. L is the number of classes. is the maximum level of hierarchy. Avg(L.) is the average number of classes per sample. Note that the commonly used WOS dataset (Kowsari et al., 2017) was not used as its labels are single-patie only.
Graph Fasion
80.42
58.38
79.53
64.17
Table 6: Ablation study when removing components of on the AAPD and BGC. r.m. stands for removing the component; rpp stands for replace with.
Appendix for Experiment Settings
hierarchy- aware attention mechanism ob- tain the text-hierarchy representation.
A.1 Implementation Details
HTCInfoMax (Deng et al., 2021) utilises in- formation maximization to model the interac- tions between text and hierarchy.
We implement our model using PyTorch- Lightning' since it is suitable for our large batches used for contrastive learning. For fair comparison, employ the bert-base-uncased model which used by other HMTC models implement HJCL. The batch size set to 80 for all datasets. Unless noted otherwise, the Å and 22 Eq. fixed to 0.1 and 0.5 for all datasets without any hyperparameter searching. The temperature 7 is fixed at 0.1. The number of heads for multi-head attention is set to 4. We use 2 layers of GAT for hierarchy injection to BGC, AAPD and RCV1-V2; and 4 layers for NYT due its depth. The optimizer is AdamW (Loshchilov and Hutter, 2017) with learning rate of 3e-5, The early stopping set to suspend training after Macro-Fi in the validation dataset and does not increase for 10 epochs. Since contrastive learning imposed stochasticity, we performed experiments with 5 random seeds. between experiments. For the baseline models, use the hyperparameters from the original paper replicate their results. For HiMulConE. (Zhang et al., 2022a), as the model was used on the image domain, we replaced its ResNet-50 feature encoder with BERT and replicated its experiment by first training the encoder with the proposed loss and the classifier with BCE loss, with 5c learning rate.
HiMatch (Chen et al., 2021) turns the prob- lem into a matching problem by grouping the text representation with its hierarchical label representation.
Seq2Tree (Yu al., 2022) introduces a sequence-to-tree framework and turns the problem into a sequence generation task using the T5 Model (Raffel et al., 2019).
HiMulConE (Zhang al., 2022a) is the clos- est to our work, also performs contrastive learning on hierarchical labels, where their erarchy has fixed height and labels are single- path only.
HGCLR (Wang et al., 2022a) incorporates the hierarchy directly into BERT and performs contrastive learning on the generated positive samples.
Appendix for Evaluation Result and Analysis
B.1 Ablation study for BGC and AAPD
The ablation results for BGC and AAPD are pre- sented in Table 6. is worth noting that in the case of AAPD, the removal of label contrastive loss significantly affects the Micro-Fl and Macro- Fl scores in both datasets. Conversely, when the instance contrastive loss is removed, only minor changes are observed in comparison the other three datasets. This can be primarily attributed the shallow hierarchy of AAPD, which consists of only two levels, resulting smaller differences
A.2 Baseline Models
To show the effectiveness of our proposed method, HJCL, we compared with previous HMTC works. In this section, we mainly describe baselines in recent work with strong performance.
HiAGM (Zhou et al., 2020) proposes

---

Moon
Figure 6: Effects of Y (left) and X2 (right) the Micro- F1 for NYT and RCVI-V2.
Figure 7: Effects of A1 (left) and A (right) on both Micro- and Macro-F1 scores among the testing set for BGC and AAPD.
between instances. Furthermore, the results in Ta- ble 1 demonstrate that the substantial improvement in Macro-F1 for AAPD can be attributed to HiLe- Con, further highlighting the effectiveness of our Hierarchy-Aware label contrastive method. On the other hand, the results for BGC follow similar trend as RCVI-V2 where both have similar hier- archy structure (c.f. Table 5), where the removal of either loss leads to a comparable drop in per- formance. The findings presented in the last two rows of Table 6 are consistent with the performance observed in the ablation study for NYT and RCVI- V2, underscoring the importance of both ZMLR loss and graph label fusion.
Lass
BCR
ZI.RR
Model
Macro-F1
Mikro-F1
Macro-F1
BERT
85.65
67.02
86.05
67.42
IiMMach
TIGCLR
86.49
68.31
86.76
68.34
Table Experimental results on RCV-1 dataset with traditional BCE and ZI.PR loss. Best results are in bold.
B.3 BCE v. ZLPR
B.2 Appendix for Hyperparameter Analysis
In this paper, we replaced the commonly-used BCE by new loss function, ZLPR (Su et al., 2022a), it presents more balanced loss function for the multi-label classification task, achieving this by leveraging the softmax function and considering the correlations between labels, contrast the Sigmoid BCE approach proposed the original paper (Zhang et al., 2022a). We consider this char- acteristic to be fundamental as aligns with our approach of emphasizing label correlations across different paths on the hierarchy.
The hyperparameter analysis for Micro-Fl scores for NYT and RCV1-V2 shown in Figure 6. The results are aligned with the observations for Macro- F1. Moreover, the hyperparameter analysis for BGC and AAPD regarding λ₃ and A2 is presented in Figure 7. Consistent with the observations from the previous ablation study section, the instance loss has a minor influence on AAPD, with per- formance peaking at Y 0.2 and subsequently dropping. Conversely, for any value of X2, the performance outperforms the baseline at À2 highlighting its effectiveness in shallow hierarchy labels. Additionally, the changes in BGC is consis- tent with those observed in RCV1-V2, as depicted in Figure 3.
To provide fairer comparison, conducted additional experiments on strong basclines, in line with our ablation study settings. We replaced their BCE loss with the ZLPR loss on NYT and RCVI datasets. As shown in Tables and 8, ZLPR consistently demonstrated improvements across dif- ferent methods, further highlighting its effective- ness in enhancing multi-label classification. On the other side, even with the integration of the ZLPR loss function, our method continues to outperform other baseline models. This shows that it is not only the adoption of the ZLPR loss function, but the overall design that allows our model to outper- form the saate-of-the-aee
Loss Model
BCE Macro-F1
ZLPR Macro-F1
Micro-F1
Micro-FI
BERT
78.24
65.62
78.75
66.24
HGCLR
78.86
67.96
79.11
68.37
HICI.
80.52
Table 7: Experimental results on NYT dataset with traditional BCE and ZLPR loss. Best results are in bold.

---

Dataset \, APath
2
3
4
5
TipoChonolirdss
TepoProttion
Nows
BGC
94.36
0.15
AAPD
57.68
6.79
35.13
0.36
0.04
RCVI-V2
85.16
12.2
2.59
0.05
NYT
49.74
34.27
15.97
0.03
Table 9: Path statistics (%) among all datasets.
6.70
477
1.70
(a)
(b)
Figure 9: T-SNE visualisation (van der Maaten and Hin- ton, 2008) (a) HiLeCon without contrastive (b) HiLe- Con. Each color represent label-aware embeddings from different path.
SERT 122UI
BEAT HCCUU
(wn)
i1 pwilt Mataon
(a)
(b)
The Accp is the measure for the ratio of predic- tions that has all the path corrected predicted; the Acep is the measure for the ratio of paths that the prediction got it all correct. The results multi- path consistency for BGC and AAPD shown in Table 10.
Figure (a) Micro-FI and (b) Macro-FI scores on
testing data of NYT, grouped by paths in the hierarchy. HileCon is our proposed method and Hil.eCon (w/o) dropped the contrastive learning function.
B.4 Performance on Multi-Path Samples
Dataset
Model
Accp
Accp
Statistics for the number of path distributions on the four multi-path HMTC datasets are shown in Table 9. Figure 8 presents the results of the performance on samples with different paths in NYT dataset.
HJCL
63.79
72.30
HICI w/o con
60.42
68.38
BGC
HGCLR
61.46
70.93
BERT
52.99
68.85
Before we formalize Accp and Acep, we give the definition of some auxiliary functions. Given the testing datasets D {() and the pre- diction results yı, Vi S. N, where yi,y; V. the true positive labels for each sample is defined as Pox y; O ys. Then we decompose both label sets ys and for into disjoint sets where each set contains labels from a single path: Path(ý;) [Y"lY' yJ
HICL
81.42
71.62
HICI w/o con HGCLR
80.11
70.76
AAPD
80.76
71.59
BERT
77.10
68.89
Table 10: Measurement for Path Accuracy and Depth Accuracy on BGC and AAPD.
B.5 T-SNE visualisation
yj = 0}. We say that the gold label y,
To qualitatively analyse the HiLeCon, we plot the T-SNE visualisation with learned label embedding across the path, as shown in Figure 9.
and prediction y; are path consistent when:
(1
[Pati(y:)] (P)))(y))
Pathh y..)
Otherwise
B.6
Case study details
The complete news report the NYT dataset used for the case study shown in Figure 10. The com- plete set of labels for the four hicrarchy plots (Fig- ure 5) is shown in Table 11. Note that to save space, the ascendants of leaf labels are omitted since they are already self-contained within the names of the leaf labels themselves.
and say path Y; is consistent in the predictions as:
{
nyd = y
Depth (Y', (Y', .y)
Otherwise
With these two definitions, we can calculate the ratio of samples and paths that are consistent with the following formulas:
EN Pathoonsoon y.yy.un
C Discussion and Case Example for ChatGPT
Accp
(8)
-1 E Extermly.) Depth ("*)
For each prompt, the LLM is presented with in- put texts, label words structured in a hierarchical
Acep
ΣN |aah(ý))l

---

Dutree
MicroPP
Micso-R
MacrosP
Macro-R
ODD
Robertson and Buchanan, Republican Chuirmar, Rich Bond, warning that Bill Clinton was White
AAPD
50.97
41,61
36.89
30.75
6,113
elected the U.S. President Jane fonda would be sleeping the
BGC
5082
63.33
35.65
45.02
12,00
for Fidel you likdd the 1992 Republican National Convention, with lits bashing of the un Christian, you'll love the 1995 convention. Assuming, upcoming midterm elections Next time powerffll new committee chaimen in Republica-conttrale Senate will surely be festured Among them will jesse Helms, chairman of the Foreign Relations "Commitiee,
puest
'If
BCVI
65.67.0.14
39 20,842034
the
Table 12: gpt-turbo-3.5 performance details HMTC datascts. The Micro/Macro-P and Micro/Macro- refers to the precision and recall for each metric respectively. The OOD refer the ratio of "Out-of- Domain" labels in the returned answers.
A'maato, charrmnn Banking, Housing and Urban" Affairs. The Speaker the House, Newt Gingrich, will be on the platform, expanding on his theme that Americans."
Democatts enemy
New York Times, 1994
in Tables 13, 14, and 15. In the experimental stage, since RCV1-V2 contains huge testing dataset, we performed random sampling with 30,000 samples (3 x 10,000) without replacement, using a random seed of 42. The performance Table for RCV1- V2 records the mean and standard deviation (std) for the three runs. As shown in Table 12, ChatGPT mainly struggles in predicting minority labels, lead- ing to significantly lower results in Macro Precision. Meanwhile, Hallucination is well-known prob- lem ChatGPT (Bang et al., 2023), and this issuc also occurs in text classification, as demonstrated in the last column of Table 12, which represents the ratio of returned answers that are not within the provided categories list. Although Few-shot In-context Learning (Brown et al., 2020) may be able to mitigate this problem by providing small subset of training samples, the flatten hierarchical labels occupy most of the tokens, and the training samples may not fit within the maximum token limit (4096 tokens). Future work on HMTC for in-context learning should focus on finding better ways to decompose and shorten the hierarchy la- bels.
Figure 10:
The complete input text sample used for the case study in Section 5.6.
Gold Labels
Top/News/U.S.
Top/News/ MaaNees/Waahington
Top/Features/ heteeaattrree/rveeeeeeeeee wwwwe//Giddeeeesiiiiiiie
North America/ United States
PeppOppinionOOppiioot/ Patam//pppp///Crrbbtoorrr
Models
Predictions
Top/News/U.S.
Top/News/ Top/News/Washingtan
IUCL
Top/Feature Seaatures/Tave//Giuuii /twee///eeaeaaeeeee/e/eeiiaiaaie
AaaiiiaaUnniiee United States North Americal
Trpp/Opinion((ppnioonfy Fd/Contributors
T/pporss/Sports
Top/Features/ htwpe///wwwe//aeeeee/eeiiiieeee
HJCL
North Americal United States
(w/o Con)
Top/News/Sports
HGCCRR
Top/News/U.S.
Top/Opinion
Table II: Complete labels set for the case study diagram shown in Figure 5. Orange represent labels that are in the gold label set but some of its decedents were missing; Red represents the incorrect labels.
format, and a natural language command that asks it to classify the correct labels related to the texts (Wang al., 2023). We flatten the hierarchy labels following the method used by Chan al. (2023) in their prompt tuning approach for Discourse Rela- tion Recognition with hierarchical structured labels. This method maintains the hierarchy dependency by connecting labels with an arrow (+).. For exam- ple, taking the label from BGC, the label "World History" appears at level-3 in the hicrarchy with ascendants "History" and "Nonfiction". This label is flattened into words as "Nonfiction + Hlistory- World History". This dependency relation is also explicitly mentioned within the prompt. Three ex- amples for AAPD, BGC, and RCV1-V2 are given

---

Classifiy the given text into the following categories, which could belong to single multiple categories: ies: [' C'Comuuter Science', "Computer Science Performance', 'Computer Science -> Formal Languages and Automata Theory' 'Computer Science -> Robotics', Rules:
"Mathematics -> Logic']
Prompt Template
1. The label prediction must be consistent, which means predicting "A -> B" also needs to predict "A"
2. No explanation is needed, output only the categories Texts: [Input]
In this paper investigate the descr rescriptional complexity of knot theoretic problems and show upper bounds for planarity problem of signed and unsigned knot diagrams represented by Gauss words We study these problems in context of automata models infinite alphabet.
Input Texts
Answer
Computer Science Formal Languages Automata Theory, Mathematics -> Combinatorics.
Gold Labels
['cs.fl', "cs.ce', 'cs']
Table 13: Example for Question and Answer from gpt- torbo-urbo-... 3.5 from AAPD
Classifiy given text the following categories, which
could belong to single or multiple categories:
['Children's 's Books', 'Poetry', 'Fiction', 'Nonfiction' 'Teen Young Adult', "Classics', 'Humor'
'Nonfiction >> History >> World History >> Asian World History']
Prompt Template
Rules:
1. The label prediction must be consistent, which means predicting "A -> B" also needs to predict "A"
2. No explanation needed, output only the categories
Texts: [Input]
Title: Jasmine Is Babysitter (Disney Princess).
Text: An original Disney Princess Little Golden Book starring
Input Texts
Jasmine super-fun babysitter!.... each Disney Princess shows they relate today's s girl
Answer
['Children's Books', 'Fiction', Children's 's Books -> Step Into Reading']
Gold Labels
['Children's Books']
Table 14: Example for Question and Answer from gpt-turbo turbo-u.bboo-oo 3.5 from BGC

---

Classifly given text into the following categories, which could belong to single or multiple categories: ies:
'ECONOMICS', 'GOVERMENT//OCIAAL' [' COORPORATE/NNDUSTAR -> STRATEGY/PLANS",
'MARKETS',
"COPOOATE/INNUUTRIAA -> LEGAL/JUDICIAL Rules:
Prompt Template
1. The label prediction must be consistent, which means predicting "A -> B" also needs to predict "A"
2. No explanation is needed, output only the categories Texts: [Input]
stand in circus collapsed during a show in northern France Friday, injuring about 48 people, most of them children, rescue workers said. About 25 children were injured and another suffered from shock when their seating fell from under them the big top of the Zavatta circus. five adults injured was seriously hurt.
Input Texts
Answer
GOVERNMENT/SOCIA >> DISASTERS AND ACCIDENTS
Gold Labels
[GOVERMMENT/SOCIAL, DISASTERS AND ACCIDENTS]
Table 15: Example for Question and Answer from spt-turbo- for RCVI-V2


Reviews:
[{"review_id": "0NaOgWMfJC", "reviewer": "Reviewer_Jpbo", "summary": "", "questions": "Please consider to highlight the novelty and differences between your work and these close baselines mentioned below. ", "limitations": "", "rating": 2, "confidence": 2, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **Q1**: The proposed method is not novel enough, as contrastive learning has been employed in various multi-label classification works (missing references). The authors should acknowledge and discuss these existing works to establish the uniqueness and importance of their approach adequately. A more comprehensive comparison with related methods is necessary to justify the novelty and significance of their proposed approach.\n\n**A1**: We thank the reviewer for pointing out their concern on the novelty of the proposed approach and including related papers that utilize contrastive learning in the multi-label classification context. However, we note that our focus is specifically on **hierarchical multi-label classification (HMTC)**, which imposes an additional constraint due to the presence of labelled hierarchies. Specifically, we emphasize the issue of **multi-path labels**, and our approach aims to focus on learning the correlations between labels across different paths. Most previous works have ignored this aspect and focused solely on datasets with single-path labels [1, 4]. While preparing the submitted paper, we did come across some of the shared papers. However, we decided not to include them (given the space limitations), as we perceived their relevance to be limited. We acknowledge that contrastive learning is not a new concept in the realm of multi-label classification, but to the best of our knowledge, none of the existing literature explored the application of contrastive learning within the domain of HMTC. We did find one exception, [1], which we extensively discussed and compared with our work. In our discussion, we also elucidated the weaknesses of their approach and expounded on how our method addresses these shortcomings. This discussion can be found between lines 77 and 83, as well as lines 184 and 191. \n\nFurthermore, we introduced the concept of Label-level Contrastive Learning. In lines 335 to 344, we present a comparison between our approach and previous multi-class classification methods that integrated contrastive learning. Additional comparison between our method and these approaches is presented in the context of Table 1. The analysis of the results, articulated between lines 454 and 460, serves to bolster the novelty of our contribution. \n\n> Please consider to highlight the novelty and differences between your work and these close baselines mentioned below.\n- **C1**: Zhang, Tianyi, et al. \"Structural Contrastive Representation Learning for Zero-shot Multi-label Text Classification.\" Findings of the Association for Computational Linguistics: EMNLP 2022.\n- **C2**: Zhang, Yu, et al. \"Metadata-induced contrastive learning for zero-shot multi-label text classification.\" Proceedings of the ACM Web Conference 2022. 2022.\n\n**A2**: Both of these works utilize contrastive learning as the objective for zero-shot multi-label text classification, a task focused on predicting unseen labels. The first work, C1, employs randomized segmentation (RTS) to divide long texts into positive pairs. On the other hand, C2 leverages learning from web-scraped metadata. Subsequently, a standard contrastive learning function is employed for the learning process. From our understanding, these two works belong to distinct domains (zero-shot vs. hierarchical), and their primary novelty lies in their approaches to generating or sampling data not present in the training set, which is crucial for achieving zero-shot classification. Their work used the most standard CL objective, without being weighted by any similarity between the labels. This would caused problems in constructing negative batches for multi-label classification [2] as it is hard to find samples that have completely different sets of labels. This issue is also discussed in the ablation study (Section 5.4 and Table 6). Consequently, we consider it inappropriate to draw a direct comparison. \n\n- **C3**: “Wang, Ran, and Xinyu Dai. \"Contrastive learning-enhanced nearest neighbor mechanism for multi-label text classification.\" Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). 2022.” \n\n**A3**: This work proposes a $k$nn retrieval-based method for flat multi-label classification, with the addition of a contrastive learning objective to ensure that train samples with similar labels are closer to each other. The proposed approach received comparable improvement over previous work.  \n\nFirstly, even though both papers use the same dataset, it should noted that the topic of our paper and C3 are different. Our focus is on hierarchical multi-label classification, where the main challenge is how to utilize the hierarchical information in prediction; where C3 ignored the hierarchy information. This highlights a clear separation between our work and C3; that is why we did not mentioned it in the submission. Secondly, C3 proposes a contrastive learning function weighted by the dynamic coefficient $\\beta_{ij}$, which is the normalization of the similarity label vectors between samples. This idea is similar to the _LeCon_ method we discussed in Section 5.4, where we analyze different types of label-wise CL and _LeCon_ is the version of _HiLeCon_ without hierarchical information. The results in Figure 4 show a consistent improvement over 4 datasets and showed statistically significance with p-value testing in Table 6.\n\n- **C4**: “Huang, Wei, et al. \"Exploring Label Hierarchy in a Generative Way for Hierarchical Text Classification.\" Proceedings of the 29th International Conference on Computational Linguistics. 2022.” \n\n**A4**: This work proposes a generative method, PAAM-HiA-T5, for solving the HMTC problem. The proposed method involves converting the labels in a hierarchical format into a flattened one by utilizing Breadth-First Search (BFS). The T5 model is subsequently employed as a generative model, considering both text sequences and upper-level label predictions while generating labels for the current level. This approach shares similarities with the methodology discussed in reference [3], which is thoroughly compared with in our submission. However, an additional path-adaptive attention mechanism is introduced in PAAM-HiA-T5 to enhance the weighting of the currently generated path, thereby reducing path inconsistency issues. Consequently, PAAM-HiA-T5 outperforms the prior work [3], which also leveraged T5 as its foundational model. \n\nUnlike the three works above, this work does utilize the hierarchical  information and addresses the HMTC problem in a generative method, using the T5 model. It’s worth noting that our method, HJCL, outperforms the PAAM-HiA-T5 in RCV-1 and NYT (except the Micro-F1 for RCV-1 with a tiny difference) even with a weaker base model (ours: BERT-base, their: T5). The major difference is in the Macro-F1 for the NYT dataset, with an absolute 5-point higher. This highlights that the contrastive approach can better exploit base  models for the task, while enhancing the performance for minority labels. \n\nWe agree with the reviewer that this reference should be included in the paper and compared with in Table 1, since  it can be seen as descendant work of Seq2Tree [3]. This will be done in the revision. \n\n[1] Zhang et al. Use All The Labels: A Hierarchical Multi-Label Contrastive Learning Framework. CVPR 2022. \n\n[2] Not All Negatives are Equal: Label-Aware Contrastive Loss for Fine-grained Text Classification (Suresh & Ong, EMNLP 2021) \n\n[3] Yu, Chao et al. “Constrained Sequence-to-Tree Generation for Hierarchical Text Classification.” Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval (2022): n. pag.\n\n[4] Wang, Ran, and Xinyu Dai. \"Contrastive learning-enhanced nearest neighbor mechanism for multi-label text classification.\" Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The proposed task is interesting and has potential application in real-life scenarios.\n\n- The paper conduct extensive experiments on multiple HMTC datasets, providing evidence of the practicality of HJCL.\n\n- The visualization aspect is well-executed, aiding readers in gaining a better understanding of the presented findings.\n", "reasons_to_reject": "The proposed method is not novel enough, as contrastive learning has been employed in various multi-label classification works (missing references). The authors should acknowledge and discuss these existing works to establish the uniqueness and importance of their approach adequately. A more comprehensive comparison with related methods is necessary to justify the novelty and significance of their proposed approach.\n\n\n\n\n", "questions_to_authors": "Please consider to highlight the novelty and differences between your work and these close baselines mentioned below. ", "paper_topic_and_main_contributions": "The paper presents HJCL, a Hierarchy-aware Joint Supervised Contrastive Learning method for hierarchical multi-label text classification (HMTC) that effectively addresses the issue of noisy sample generation and achieves promising results on multiple datasets, demonstrating the effectiveness of contrastive learning in HMTC. \n\n\n\n", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "Zhang, Tianyi, et al. \"Structural Contrastive Representation Learning for Zero-shot Multi-label Text Classification.\" Findings of the Association for Computational Linguistics: EMNLP 2022. 2022.\n\n\nZhang, Yu, et al. \"Metadata-induced contrastive learning for zero-shot multi-label text classification.\" Proceedings of the ACM Web Conference 2022. 2022.\n\nWang, Ran, and Xinyu Dai. \"Contrastive learning-enhanced nearest neighbor mechanism for multi-label text classification.\" Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). 2022.\n\nHuang, Wei, et al. \"Exploring Label Hierarchy in a Generative Way for Hierarchical Text Classification.\" Proceedings of the 29th International Conference on Computational Linguistics. 2022.\n\n"}}, {"review_id": "fgMlGMLC7G", "reviewer": "Reviewer_46re", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **Q1**: The authors did not release data and code to facilitate future research. \n\n**A1**: Thanks for the valuable comments. As announced in the last sentence of the abstract, we will release our code and data processing code upon acceptance. This follows the usual practice in EMNLP. As proof of this, we have uploaded our codebase with detailed documentation in an anonymized repository: https://anonymous.4open.science/r/HJCL-EMNLP.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper proposes a new method called HJCL that bridges the gap between supervised contrastive learning and HMTC, which can improve the accuracy of HMTC models.\n- The paper employs both instance-wise and label-wise contrastive learning techniques and carefully constructs batches to fulfill the contrastive learning objective, which can reduce the noise introduced during the generation of samples.\n- The paper conducts extensive experiments on four multi-path HMTC datasets and achieves promising results, demonstrating the effectiveness of Contrastive Learning on HMTC.\n- Very good visualizations and well-organized writing make the entire manuscript easy to follow.", "reasons_to_reject": "- The authors did not release data and code to facilitate future research.\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes a new method called HJCL that bridges the gap between supervised contrastive learning and HMTC. The framework also employs both instance-wise and label-wise contrastive learning techniques and carefully constructs batches to fulfill the contrastive learning objective. Extensive experiments on four multi-path HMTC datasets have shown that HJCL has achieved promising results, demonstrating the effectiveness of Contrastive Learning on HMTC.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "5H1JVAK8Z7", "reviewer": "Reviewer_9pSu", "summary": "", "questions": "My primary concern revolves around the extent of improvement achieved by implementing the ZLPR loss [1] in HMTC. Based on the findings from Table 2 and Table 5, when the ZLPR loss is substituted with the BCE loss, there is an approximate 1% decrease in the F1 score across all datasets. As the other baselines did not incorporate the ZLPR loss in their experiments, it would be more appropriate to consider the results obtained from the BCE loss as the main reference for a fair comparison.\n\n[1] Su et al. ZLPR: A Novel Loss for Multi-label Classification. 2022.", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **Q1**: Certain notations employed in this paper lack clear definitions, posing challenges to readers' comprehension.  \n\n  \n\n**R1**: Thanks for the valuable comment. For the final version we will improve the presentation of the paper, focusing on consistency, self-containment, and clarity.   \n\n  \n\n> **Q2**: The utilization of the ZLPR loss function, distinct from other baseline methods, may introduce bias and result in an unfair comparison.  \n\n  \n\n**R2**: Response is put together inside **R4**. \n\n  \n\n> **Q3**: Considering the existence of prior work focusing on hierarchical contrastive learning methods [1], the novelty of the proposed approach within the contrastive learning domain may be ambiguous.  \n\n  \n\n**R3**: Thanks for your comment on the novelty of the proposed approach. Indeed, as pointed out, HiMulConE [1] is the work most related to our submission. However, as discussed in the submission (lines 77 – 83, 184 – 191), [1] has some important shortcomings:  \n\n  \n\n1. The method proposed by Zhang _et al._ [1] is restricted by the assumption of a *fixed-depth hierarchy* i.e., it assumes all paths in the hierarchy have the same length. Note that this does not apply to most HMTC datasets (e.g. NYT and BGC) as not all paths from the root to leaf have the same depth (different labels have different granularity). This limited the generalization of their approach. Our method does not have this limitation.  \n\n  \n\n2. In addition to the last points, the HiMulConE method is only applied to single-path hierarchy labels (i.e., for all labels $y_{i}$ in the target label set $Y^{\\text{gold}}$, there would be at most one descendant label $y_{j} \\in Y^{\\text{gold}}$ s.t. $depth_{y_{j}} = depth_{y_{i}} + 1$) and its contrastive method is based on this assumption. However, as shown in the statistics of the HMTC dataset (Table 7), most of the datasets are multi-path, meaning label set contain labels from more than one path. Therefore, their method is not designed for the HMTC task. Instead, our method accounts for the idea of multi-path and considers the correlation between labels at different paths. \n\n  \n\n3. To better understand our novelty, we replicated the experiment with HiMulConE, cf. Table 1. As discussed between line 454 – 460, the HiMulConE model performs even poorer than the BERT baseline. This shows that its assumption of fixed depth and considering each sample as a single embedding damages the performance.  \n\n  \n\nGiven these identified downsides and the observed decline in performance, we strongly believe that our approach effectively bridges the contrastive learning gap in the broader context of the HMTC task. We experimentally showed this on common HMTC datasets which have multi-path structures with varying path-depths. Furthermore, the ablation study, focused on path and depth consistency (Section 5.5), further strengthened the novelty of our method compared to the state-of-the-art. \n\n  \n\n> **Q4**: My primary concern revolves around the extent of improvement achieved by implementing the ZLPR loss [1] in HMTC. Based on the findings from Table 2 and Table 5, when the ZLPR loss is substituted with the BCE loss, there is an approximate 1% decrease in the F1 score across all datasets. As the other baselines did not incorporate the ZLPR loss in their experiments, it would be more appropriate to consider the results obtained from the BCE loss as the main reference for a fair comparison.  \n\n  \n\n**R4**: Thank you for the valuable suggestions. ZLPR presents a more balanced loss function for the multi-label classification task, achieving this by leveraging the softmax function and considering the correlations between labels, in contrast to the sigmoid + BCE approach proposed in the original paper [2]. We consider this characteristic to be fundamental as it aligns with our approach of emphasizing label correlations across different paths on the hierarchy. \n\n  \n\nTo provide a fairer comparison, we conducted additional experiments on strong baselines, in line with our ablation study settings. We replaced their BCE loss with the ZLPR loss on the NYT and RCV1 datasets. As shown in Tables 1 and 2, ZLPR consistently demonstrated improvements across different methods, further highlighting its effectiveness in enhancing multi-label classification. Furthermore, even with the integration of the ZLPR loss function, our method continues to outperform other baseline models. This shows that it is not only the adoption of the ZLPR loss function, but the overall design that allows our model to outperform the state-of-the-art. \n\n   \n\n| Model | Micro-F1(BCE) | Macro-F1(BCE) | Micro-F1(ZLPR) | Macro-F1(ZLPR) | \n|---------|--------------|---------------|--------------|---------------| \n| BERT       | 78.24 | 65.62      | 78.75      |  66.24     | \n| HiMatch       | -  | -     | - | -     | \n| HGCLR      |  78.86 | 67.96      | 79.11  | 68.37  | \n| HJCL      |  **79.74** | **69.26**      | **80.52**      |  **70.02**     | \n\n  \n\n**Table 1**: Experimental results on NYT dataset with traditional BCE and ZLPR loss. Best results are in **bold**.\n\n  \n\n| Model | Micro-F1(BCE) | Macro-F1(BCE) | Micro-F1(ZLPR) | Macro-F1(ZLPR) | \n|---------|--------------|---------------|--------------|---------------| \n| BERT       | 85.65 | 67.02      | 86.05      |  67.42     | \n| HiMatch       |86.33 | 68.66     | 86.47      |  68.98     | \n| HGCLR      |  **86.49** | 68.31      | 86.76      |  68.34     | \n| HJCL      |  86.42 | **69.24**      | **87.04**      |  **70.49**     | \n\n**Table 2**: Experimental results on RCV-1 dataset with traditional BCE and ZLPR loss. Best results are in **bold**.\n\n  \n\n[1] Zhang et al. Use All The Labels: A Hierarchical Multi-Label Contrastive Learning Framework. CVPR 2022. \n\n[2] Su et al. ZLPR: A Novel Loss for Multi-label Classification. 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper presents an effective supervised contrastive learning approach that leverages the label hierarchy to learn representations.\n2. The proposed approach demonstrates outstanding performance across all benchmark datasets, surpassing the baselines by a substantial margin.\n3. The experiments conducted in the study are both comprehensive and sufficient, effectively revealing the motivation behind the method.", "reasons_to_reject": "1. Certain notations employed in this paper lack clear definitions, posing challenges to readers' comprehension.\n2. The utilization of the ZLPR loss function, distinct from other baseline methods, may introduce bias and result in an unfair comparison.\n3. Considering the existence of prior work focusing on hierarchical contrastive learning methods [1], the novelty of the proposed approach within the contrastive learning domain may be ambiguous.\n\n[1] Zhang et al. Use All The Labels: A Hierarchical Multi-Label Contrastive Learning Framework. CVPR 2022.", "questions_to_authors": "My primary concern revolves around the extent of improvement achieved by implementing the ZLPR loss [1] in HMTC. Based on the findings from Table 2 and Table 5, when the ZLPR loss is substituted with the BCE loss, there is an approximate 1% decrease in the F1 score across all datasets. As the other baselines did not incorporate the ZLPR loss in their experiments, it would be more appropriate to consider the results obtained from the BCE loss as the main reference for a fair comparison.\n\n[1] Su et al. ZLPR: A Novel Loss for Multi-label Classification. 2022.", "paper_topic_and_main_contributions": "This paper introduces Hierarchy-aware Joint Supervised Contrastive Learning (HJCL), a novel approach aimed at addressing the challenge of representation learning in Hierarchical Multi-Label Text Classification (HMTC). HJCL incorporates a ZLPR loss and two label-aware contrastive loss terms, effectively mitigating the under-exploitation of label information in traditional contrastive learning methods. Specifically, HJCL extracts label-aware embeddings from the input features through multi-head attention and devises various strategies to construct contrastive pairs based on the label hierarchy, enabling both instance-wise and label-wise contrastive learning. Extensive experiments conducted on four HMTC datasets demonstrate the superior performance of HJCL compared to other baselines, establishing its effectiveness in the recent state-of-the-art approaches.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. Enhancing the formulation of the background techniques. L199: utilizing distinct notations to represent input length and the number of samples. L224: providing a clear definition of the attention mechanism, L225: offering explanations for the meaning of variables $d$ and $j$.\n2. L231: referring to label-aware embeddings as label embeddings might be misleading since, in common sense, the label embeddings should not depend on the input features.\n3. L321: While it can be inferred that $\\mathcal{N}_\\ell$ represents the set of negative samples at layer $\\ell$, an explicit statement clarifying this relationship is necessary."}}]

Author rebuttal:

Meta-review: Summary: This paper presents a new method called Hierarchy-aware Joint Supervised Contrastive Learning (HJCL), which aims to tackle the challenge of representation learning in Hierarchical Multi-Label Text Classification (HMTC). HJCL uses multi-head attention to extract label-aware embeddings from input features and employs different strategies to create contrastive pairs based on the label hierarchy, allowing for both instance-wise and label-wise contrastive learning. Extensive experiments conducted on four HMTC datasets demonstrate the effectiveness of the proposed approach.

Strength: This paper presents a promising method to improve the accuracy of HMTC models. The paper is well-written and easy to follow.  Extensive experiments on four multi-path HMTC datasets are conducted.  All reviewers agree that this is a solid contribution but the excitement is relatively limited.
Decision: Accept-Findings