Dual Prompt Tuning based Contrastive Learning for Hierarchical Text Classification (2024)

Abstract: Hierarchical text classification aims at categorizing texts into a multi-tiered tree-structured hierarchy of labels. Existing methods pay more attention to capture hierarchy-aware text feature by exploiting explicit parent-child relationships, while interactions between peer labels are rarely taken into account, resulting in severe label confusion within each layer. In this work, we propose a novel Dual Prompt Tuning (DPT) method, which emphasizes identifying discrimination among peer labels by performing contrastive learning on each hierarchical layer. We design an innovative hand-crafted prompt containing slots for both positive and negative label predictions to cooperate with contrastive learning. In addition, we introduce a label hierarchy self-sensing auxiliary task to ensure cross-layer label consistency. Extensive experiments demonstrate that DPT achieves significant improvements and outperforms the current state-of-the-art methods on BGC and RCV1V2 benchmark datasets. 1

heading: 1 Introduction
text: As a specialized sub-task of text classification, hierarchical text classification (HTC) has a wide range of applications in the realistic scenarios, such as intent recognition in dialogue system, commodity and book management (Cevahir and Murakami, 2016; Aly et al., 2019), where a large number of categories are organized into a tree-structured hierarchy. The ultimate goal of HTC is to categorize texts or documents from the topmost level to the finest granularity precisely. Due to the challenges of large-scale, imbalanced and complex label hierarchy (Mao et al., 2019), simply transferring flat multi-label text classification algorithms to deal with HTC tasks often fails to achieve sufficient performance.
1Code is publicly available at https://github.com/ ccx06/Dual-Prompt-Tuning-for-HTC.
Full use of the hierarchical structure of labels is the key to achieving well-performing classification in HTC tasks, which facilitates the model in predicting labels that align with hierarchical relationship. As shown in Figure 1, current mainstream methods (Zhou et al., 2020; Deng et al., 2021; Chen et al., 2021; Zhu et al., 2023) apply a dualencoders framework to model the text and hierarchical structure separately, and then fuse them to obtain hierarchy-enhanced text features. Wang et al. (2022b) first proposes a hierarchy-aware prompttuning method, which incorporates the label hierarchy encoded by the Graph Attention Network into a soft prompt to bridge hierarchy and flat gap.
However, most studies pay close attention to exploit relations that explicitly displayed in the hierarchy, while internal interactions between "peer labels", which refer to a group of labels at the same hierarchical layer, are often neglected. PeerHTC (Song et al., 2023) recently tries to explore latent relevancy among peer labels with a complicated
12146
two-stage training procedure, in which peer and adjacent level-wise features are separately extracted by Graph Convolutional Neural Networks. Nevertheless, alleviating confusion within peer labels, especially fine-grained ones that share the same parent node at lower level, still remains challenging and highly valued.
To this end, we propose a novel Dual Prompt Tuning (DPT) method, aimed at alleviating label confusion between peer labels. We put forward a Hierarchy-aware Peer-label Contrastive Learning (HierPCL) approach to extract discriminative pair-wise representations. In detail, we create an original dual prompt template containing both positive and negative label slots, and then perform label-wise contrastive learning on the embeddings of both two types of slots. Dual prompt is multifunctional, targeted for predicting positive labels and recognizing incorrect but confused negatives at each hierarchical layer. We furthermore incorporate a rank loss component into the contrastive loss function to enhance label consistency. Moreover, we design an adaptive hard negative sampling strategy and a hierarchy-injected label representation encoding method to further boost the performance of HierPCL.
Besides, we introduce a simple and effective label hierarchy self-sensing auxiliary task to keep our model in the best sense of holistic hierarchical structure. The basic idea is to internalize structural hierarchy knowledge to ensure the cross-layer consistency of the final path predictions. Instead of directly injecting label hierarchy into text semantics, we perform multi-task learning collaborating with label prediction to identify the consistency and correctness of each candidate label path.
The contributions are summarized as follows:
• We propose a Dual Prompt Tuning method for HTC tasks to address label confusion between peers at each hierarchical layer, magnifying the power of prompt.
• We put forward a Hierarchy-aware Peer-label Contrastive Learning approach based on the dual prompt, which contributes to obtaining aligned and discriminative features on prompt slots.
• We evaluate our proposed methods on four popular benchmark datasets against the strong baselines. Experimental results demonstrate the advantage of our proposal.

heading: 2 Related Work

heading: 2.1 Hierarchical Text Classification
text: The HTC algorithms can be generally divided into local and global approaches (Zangari et al., 2023). Local approaches (Wehrmann et al., 2018; Banerjee et al., 2019) construct multiple classifiers for different partitions of the label hierarchy usually in the "top-down" flow. Although there is a certain degree of connection between multiple classifiers, it is inevitable to lose the holistic structure information of label hierarchy. Global approaches use a single classifier to classify all labels with hierarchical dependencies simultaneously. Early works simplify the HTC task into a flat multi-label classification task, discarding the inherent hierarchical information in taxonomic labels. Later on, specialized hierarchy-aware methods are proposed. HiAGM (Zhou et al., 2020), HTCInfoMax (Deng et al., 2021), HiMatch (Chen et al., 2021), and HiTIN (Zhu et al., 2023) all employ the dual-encoders framework, which utilizes a text encoder and a structure encoder to learn the representations of texts and labels respectively, and then derives enhanced text embeddings based on both textual and structural information. Great progress has been made by leveraging advanced algorithms from other domains, i.e., applying sequence generative manners (Zhao et al., 2022; Ning et al., 2023; Huang et al., 2022) to mitigate label inconsistency phenomenon, data generation strategies (Wang et al., 2023) to enrich text diversity, contrastive learning methods (Wang et al., 2022a; Ji et al., 2023) to strengthen semantic expression, and prompt-tuning paradigm (Wang et al., 2022b; Ji et al., 2023) to exploit the potential of pre-trained language models (PLMs) (Devlin et al., 2019; Brown et al., 2020; Raffel et al., 2023; Liu et al., 2023b).

heading: 2.2 Prompt Tuning
text: Prompt Tuning (Schick and Schütze, 2021; Liu et al., 2023a) refers to the technique of tuning pretrained language model by reconstructing downstream task into cloze test task, which bridges the gap in goals between fine-tuning and pre-training stages. It involves two main steps: (1) prompt template construction which generates a template containing special tokens, and (2) label word verbalizer design which defines a function from token embedding to answer words. There are two types of template construction methods. Hard prompt
methods (Shin et al., 2020; Gao et al., 2021a; Han et al., 2022) directly concatenate explicit discrete tokens with the original text and maintain them unchanged throughout entire training process, which do not introduce any parameters. Soft prompt methods (Qin and Eisner, 2021; Lester et al., 2021; Gu et al., 2022; Liu et al., 2023c) convert prompt words into a group of continuous vectors as the template and update the vector based on specific contextual semantics and task objectives during training. The prompt-based method HPT (Wang et al., 2022b) adopts a soft prompt for HTC tasks, inserting a fixed number of learnable virtual label words into the input text. HierVerb (Ji et al., 2023) proposes a Multi-verbalizer (Multi-Verb) framework which integrates the hierarchical information, bringing significant performance improvement under fewshot settings.

heading: 2.3 Contrastive Learning
text: Contrastive learning (He et al., 2020; Chen et al., 2020) aims to pull anchor sample close to its positive samples while push it apart from negative samples, which has been proven to elevate the alignment and uniformity of feature space. Contrastive learning has various forms, differing in the construction of positive and negative pairs and loss formulas. Under self-supervised settings, positive samples are usually obtained through data augmentations or repeating dropout mask twice (Gao et al., 2021b) operation. Under supervised settings, positives are other samples of the same category (Khosla et al., 2020). Negative samples are selected from the remaining samples within a batch in self-supervised learning or samples belonging to distinct categories in supervised learning. Prototypical Contrastive Learning (Li et al., 2021) is proposed to enhance semantic discrimination and balance. It encourages instances to be closer to their assigned class prototypes, which can be established as the label semantics (Ma et al., 2022), the average embeddings of instances (Xiao et al., 2021) as well as learnable parameters (Cui et al., 2022).

heading: 3 Methodology
text: In this section, we present a detailed description of our proposed DPT model to address HTC tasks. As shown in Figure 2, our model is based on prompttuning framework with Multi-verbalizer. The principal innovations of DPT are twofold, including
(1) the implementation of Hierarchy-aware Peerlabel Contrastive Learning to obtain rich discriminative features, and (2) the incorporation of Label Hierarchy Self-sensing auxiliary task to enhance encoder’s ability for an in-depth understanding of label hierarchy structure.

heading: 3.1 Preliminary
text: Given a set of inputs D = {t1, t2, ..., tN} where ti = {xj}nj=1 denotes a text composed of n words, and a predefined hierarchical label set Y which is commonly organized as a tree-like taxonomy structure G, the goal of HTC is to select labels for ti at each layer starting from the root label node of G. Assuming L is the maximum depth of G, the labels {y1, y2, ...} of an input text correspond to single or multiple paths of the label tree, each of which typically consists of no more than L continuous individual labels with hierarchical relationship within G.

heading: 3.2 Dual Prompt Tuning
text: Prompt template is utilized to wrap the original text to generate a new form of model input in prompt tuning paradigm. For example, the given text ti is converted to "[CLS] It was 1 level: [MASK] 2 level: [MASK] ... L level: [MASK]. ti [SEP]" (Ji et al., 2023). Different from vanilla prompt, we exploit a dual prompt template to reserve two types of slot positions, instead of only insert positive label slots. For instance, a common dual prompt template is formulated as follow:
T = {[CLS] ti [SEP] It belongs to [MASK] -...-[MASK] rather than [MASK]-...-[MASK][SEP]}
(1)
The number of [MASK] repetitions of positive or negative slots is equal to the depth of the label hierarchy L. In this paper, we define the [MASK] token inserted in the prompt template at the position of label slot as "label mask token". In the above example, positive label mask tokens locate between "It belongs to" and "rather than", while negative label mask tokens are behind "rather than". We’d like to emphasize that the number of prompt words introduced by dual prompt template is linearly related to the maximum depth of label hierarchy rather than the number of full labels.
Consistent with other competitive methods, we employ BERT (Devlin et al., 2019) as the backbone of our model to encode input texts and obtain all
token embeddings:
V = BERT(T (ti)) (2)
Let vpl and v n l respectively represent the embeddings of positive and negative label mask token at the l-th layer. We inherit HierVerb (Ji et al., 2023) to adopt a depth-oriented Multi-verbalizer framework mapping label mask token embeddings {vpl }Ll=1 to label words. Probability distribution of ti can be expressed as:
Z = {zl}Ll=1 = {V er1(vp1), ..., V erL(vpL)}
(3)
where the l-th verbalizer V erl acts in predicting l-level labels. More details about Multi-verbalizer framework can be found in Ji et al. (2023).

heading: 3.3 Hierarchy-aware Peer-label Contrastive Learning
text: To extract hierarchy-aware dicriminative feature on the basic of dual prompt tuning, the ideal embeddings at label mask tokens should satisfy the following intents: (1) Token embeddings of the positive label mask tokens should be close to representations of their positive labels, and far away from negative labels in the feature space. The same desire applies to token embeddings of negative label mask tokens. (2) The semantic similarity between high-level label and its ground truth sub-label is expected to be greater than that with other sub-labels,
further greater than that with sub-labels of other nodes at the same hierarchical layer. Based on the above, we propose a Hierarchy-aware Peer-label Contrastive Learning (HierPCL) method to capture latent semantic relevancy between peer labels as well as parent-child labels.
Objective of HierPCL The basic idea of HierPCL is to encourage the embeddings of the label mask tokens encoded by PLM closer to the representations of their positive labels which should be filled in the label slots of the template. The objective function of HierPCL consists of three components: positive label contrastive learning, negative label contrastive learning and cross-hierarchical rank loss.
(1) Positive label contrastive learning is performed on positive label mask tokens. The target positives are M ground truth labels while the negatives are the sampled K negative labels and negative label mask token. The loss function is formulated as:
LpCL = − 1
L
L∑
l=1
log
∑M m=1 exp(s(v p l , r
p l,m)/τ)∑
u∈Ap l exp(s(vpl , u)/τ)
(4)
where rpl,m and r n l,k respectively denote the representation vectors of the m-th positive labels and the k-th negative label in the l-th layer of sentence ti. s(·) represents cosine similarity function, and τ is the temperature coefficient. All participants above are denoted as Apl := {{r p l,m}Mm=1, {rnl,k}Kk=1, vnl },
including representation vectors of positive and negative labels, and embeddings on negative label mask token at the same hierarchical layer.
(2) Negative label contrastive learning is performed on negative label mask tokens. Opposite to LpLC , the target positives are negative labels of this instance while the negatives are ground truth labels and positive label mask token. Let Anl := {{r p l,m}Mm=1, {rnl,k}Kk=1, v p l }, the loss function is formulated as:
LnCL = − 1
L
L∑
l=1
log
∑K k=1 exp(s(v n l , r
n l,k)/τ)∑
u∈An l exp(s(vnl , u)/τ)
(5)
(3) Cross-hierarchical rank loss aims to align label token embeddings with the representations of their sub-labels. In other words, high-level labels tend to have higher semantic similarities with positive child-labels compared to negative childlabels and other peer labels. The loss function is formulated as:
LR = L−1∑
l=1
( M∑
m=1
K∑
k=1
max(0, s(vpl , r n l+1,k)− s(vpl , rpl+1,m))
+ ∑
k
∑
k̂
max(0, s(vpl , r n l+1,k̂)− s(v p l , r n l+1,k))
)
(6)
where rn l+1,k denotes the representation vector of the negative label at (l + 1)-th layer whose parent label is positive at layer l, while rn
l+1,k̂ denotes that
of other negative label that does not belong to a positive label at the next higher level.
Finally, the loss function of HierPCL is formulated as follow:
L1 = αLpCL + (1− α)LnCL + βLR (7)
where α and β are hyper-parameters used for balancing relative weights of the three components.
Top-K Hard Negative Sampling The selection strategy of negative samples is critical to contrastive learning. Since the estimation of label predictions can be considered as a reliable source for generating hard negatives, we devise an adaptive self-produced hard negative sampling strategy under the guidance of the model’s own predictions during training. It adopts the top K hard negative labels according to confidence scores ranked in descending order at each hierarchical layer. In our experiments, K is set to 10% of the number of labels at each layer, which achieves a balance between performance and memory consumption.
The effects of different settings of K are described in Appendix B in detail.
Hierarchy-injected Label Representation Label representation vectors are learned through the shared PLM without prior statistics. To avoid the side impact of the overlapping interaction between label names and text words, we assign an unique fabricated symbol for each label, like "L0", and add these symbols to the vocabulary list used in model training. To incorporate hierarchy information to label representation, we flatten the parent-child hierarchy of a label to form a label sequence, as follow:
Q = {[CLS] W [SEP] W f [SEP] {W c} [SEP]} (8)
Assuming W is the symbol of current label, then W f means the parent label symbol of current label W , and {W c} is on behalf of symbols of all child labels. "Root" and "None" are used as fictitious tokens when the parent label or child label doesn’t exist. We use the embedding on W enriched with hierarchical dependencies as the label representation r.

heading: 3.4 Label Hierarchy Self-sensing Task
text: For the purpose of elevating the model’s perceptual ability of holistic hierarchical structure, we introduce a Label Hierarchy Self-sensing auxiliary task, consisting of two sub-tasks: (1) determining whether the predicted label nodes at each level can form valid label paths, and (2) determining whether the predicted label paths are completely correct. A simple base unit of feed-forward neural network is utilized on the top of [CLS] token following PLM to execute the auxiliary task. The consistency and correction loss functions are respectively defined based on the Binary Cross Entropy (BCE) (De Boer et al., 2005) function:
Lcon = BCE(ȳcon, p̄con) (9)
Lcor = BCE(ȳcor, p̄cor) (10) where p̄con represents the probability of that the predictions can fully correspond to label paths in the label tree, and p̄cor represents the probability of that predictions are all right. We retrieve all label paths from the predicted label nodes output by verbalizers. ȳcon = 1 if all predicted label nodes can exactly form label paths. ȳcor = 1 if all combined label paths are the positive labels of the sentence. Otherwise, the value of ȳcon or ȳcor is 0.
Finally, the loss function of the auxiliary task is formulated as:
L2 = Lcon + Lcor (11)

heading: 3.5 Multi-task Training
text: Overall, multi-task training objective is to minimize the weighted combination of classification loss, peer-label contrastive learning loss, label hierarchy self-sensing loss and MLM loss retaining from BERT pre-training. The classification loss function can be contingent upon specific circumstances. For the sake of universality, the standard Binary Cross-Entropy function is employed:
LCLS = L∑
l=1
BCE(yl, zl) (12)
Final joint loss can be formulated as:
L = LMLM + LCLS + λ1L1 + λ2L2
(13)
where λ1 and λ2 are hyper-parameters.

heading: 4 Experiment

heading: 4.1 Experiment Setup
text: Datasets We conduct experiments on 4 benchmark datasets: Web-of-Science (WoS) (Kowsari et al., 2017), NYTimes (NYT) (Sandhaus, 2008), RCV1-V2 (Lewis et al., 2004) and Blurb Genre Collection (BGC) 2 (Aly et al., 2019) . The label taxonomy of WoS is single-path whereas the remaining three datasets are for multi-path HTC. The detailed statistics of these datasets are listed in Table 1.
Evaluation Metrics The performance of our methods is evaluated by popular Micro-F1 and Macro-F1 metrics. In addition, path-constrained C-MicroF1 and C-MacroF1 metrics proposed by Yu et al. (2022) are used to measure the hierarchical path consistency for comprehensive evaluation, in which an output label is considered as correct only when all its ancestor nodes are accurately predicted.
Implementation Details The backbone of our model is initialized with bert-base-uncased3. The batch size is set to 32 for BGC and 16 for other datasets. The AdamW optimizer is used with the
2https://www.inf.uni-hamburg.de/en/inst/ab/lt/ resources/data/blurb-genre-collection.html
3https://huggingface.co/bert-base-uncased
learning rate of 2e-5 for WoS and 3e-5 for others. We apply early stopping strategy with 5 patient epochs. For fair comparison, we perform the same data processing and spliting methods as HPT. The reported results of our main experiments are the average score of 5 runs over different random seeds. Experimental settings of hyper-parameters are described in Appendix A.
Baselines We compare our methods with the following advanced HTC methods:
• HiAGM, HTCInfoMax and HiMatch are dual-encoders based classic methods. They derive joint embeddings on text and labels from interactive fusion or matching mechanisms.
• HGCLR incorporates label hierarchy into text encoder through hierarchy-guided contrastive learning between texts and their generated positive samples with the most closest label paths.
• Seq2Tree and PAAM-HiA-T5 treat HTC as a sequence generation task. Seq2Tree designs a constrained decoding strategy with dynamic vocabulary to ensure label consistency. PAAM-HiA-T5 proposes a multi-level sequential label generative T5 model with a path-adaptive attention mechanism to focus on label dependency prediction.
• HPT exploits the effects of prompt-tuning by a dynamic virtual template and a zerobounded multi-label cross entropy loss. It achieves the current state-of-the-art performance on most datasets.
• HiTIN introduces the structural entropy to construct a coding tree for the label hierarchy and then builds a novel structure encoder to enhance text representations.

heading: 4.2 Main Result
text: Experimental results are shown in Tabel 2. Our model consistently outperforms previous advanced approaches across 3 datasets except for WoS. On WoS dataset with label depth of 2, our proposed DPT achieves comparable results with HPT but decreased performance compared to PAAM-HiA-T5 model. Our model establishes the state-of-the-art results on RCV1-V2 and BGC datasets. It improves 0.5% and 0.76% absolute Micro-F1 and MacroF1 on RCV1-V2 dataset compared to the current best results. The performance boost of Micro-F1 and Macro-F1 on BGC reaches 0.53% and 1.52% over HPT model. The significant improvements on Macro-F1 metric suggest that our model excels in identifying sparse labels. On NYT dataset, our model surpasses HPT on Micro-F1 by 0.14% but slightly lower on Macro-F1.
Without introducing any additional network parameters to extract the semantics of labels and their hierarchical structures, our model surprisingly outperforms previous methods of encoding label names and label hierarchies using GNNs. Compared to HGCLR, which employs instance-level contrastive learning with complex positive sample generation operation, DPT achieves performance gains across all datasets.

heading: 4.3 Results on Label Consistency
text: For HTC tasks, cross-layer label consistency is an essential evaluation factor, which signifies that multiple labels predicted by the model at each layer should conform to the predefined hierarchical structure. Table 3 illustrates the label consistency performance of our proposed DPT and the state-of-the-art HPT model. DPT improves consistency of label hi-
erarchy on RCV1-V2 and BGC by a large margin, respectively exceeding HPT by 1.49% and 1.61% on C-MacroF1 metric. Although our model focuses more on the interaction between peer labels at the same layer, the knowledge of label hierarchy has also been internalized. The accuracy of both individual label nodes and united label paths has been improved, indicating that our methods are reasonable and efficient.

heading: 4.4 Results on Imbalanced Hierarchy
text: To further clarify the superiority of our methods, we explore the performance on the imbalanced hierarchy. Following long-tailed learning setting, we sort the training set in descending order based on the quantity of class instances and evenly cluster all categories into head, middle, and tail groups. The visualization results on Macro-F1 metric of different groups are shown in Figure 3. It’s evident that the performance of DPT is better than that of HPT on imbalanced hierarchy in RCV1-V2 and BGC datasets. DPT shows significant improvements on tail classes with few training samples, demonstrating the effectiveness of our methods in eliminating the impact resulting from imbalanced distribution.

heading: 4.5 Ablation Study
text: To investigate the effects of each component of our proposed model, we implement different variants and conduct experiments on BGC and RCV1-V2 dataset. The Results are shown in Table 4 and Tabel 5 respectively. Upon only employing the HierPCL module, the performance in all metrics realizes considerable enhancement and are superior to the current state-of-the-art models, confirming its significant effectiveness and reliability. By removing negative contrastive part, the scores undergo sharp declines, which demonstrates that negative label contrastive learning plays a prominent role in HierPCL. As a strong contrast, we replace our self-produced hard negative sampling with random sampling, resulting in decrease in metrics, which validates the advantage of our negative sampling strategy. The cross-hierarchical rank loss in HierPCL and label hierarchy self-sensing auxiliary task are evidenced to improve the model’s performance, specifically with regard to the Macro-F1 metric.

heading: 4.6 Insight into Case Effects
text: In order to gain insight into practical effects of our model, we conduct detailed case studies on the test set. We define 3 types of label errors from the perspective of multi-label classification, as follows:
• Excessive, refers to the label which is unnecessarily identified as one of the positive labels for the instance.
• Misjudged, refers to the label which is mistakenly identified as a positive label while the instance actually belongs to another peer label.
• Missed, refers to the label that the model has failed to recall.
We separately calculate the distribution of label error types for baseline model (typical prompttuning with Multi-Verb framework) and our improved model on the test set. As shown in Figure 4, we find that the optimization effects of our model are manifested in recalling missed labels, correcting misjudged labels and removing excessive labels, respectively accounting for 44.69%, 40.21% and 15.10% of the proportion. It demonstrates the strong power of DPT to capture discriminative representation and further relieve label confusion. Some specific cases are illustrated in Appendix D.

heading: 4.7 Analysis of Computational Complexity
text: We conduct experiments with an NVIDIA Tesla V100 on time efficiency and computational resources on RCV1-V2 dataset. The quantitative results are listed in Appendix C. Compared to promptbased HPT model, our approach necessitates longer training time, primarily attributed to contrastive learning, which inherently possesses a high time
complexity and slightly elevates computational resources requirements. Despite all this, both factors remain within acceptable range. Furthermore, label-wise contrastive learning has superior time efficiency than instance-level contrastive learning. We devise an effective negative sampling strategy to reduce computational costs. When performing model inference, our methods do not involve any additional time-consuming steps, such as loss calculations, and exhibit faster speed compared to models that employ GNN as the structure encoder.

heading: 5 Conclusion
text: In this paper, we present a novel Dual Prompt Tuning method for HTC tasks. Firstly, we propose a Hierarchy-aware Peer-label Contrastive Learning approach to alleviate confusion between peer labels. An original dual prompt template is created with slots for both positive and negative label, on which contrastive learning is performed at each layer. Secondly, to further strengthen knowledge understanding of label hierarchy structure, we design a Label Hierarchy Self-sensing auxiliary task to identify consistency and correctness of model predictions. Experimental results illustrate that our proposed DPT method achieves significant improvements on popular HTC datasets. It excels in precisely recognizing negative labels and contributes to obtaining hierarchy-aware discriminative features. Notably, DPT exhibits remarkable effectiveness in preserving label path consistency and addressing imbalanced hierarchy challenge.

heading: 6 Limitations
text: In our work, the applicability of DPT is restricted by the scale of hierarchical labels, considering the following two factors: (1) The sequence length of input text for PLMs is constrained to a maximum value, and adding prompts consumes some tokens. However, the number of prompt words introduced by DPT is only linearly related to the depth of the label tree rather than the number of full labels, which has minimal impact. (2) As outlined in Section 3.3, when computing label representation vector, the maximum allowed number of child labels {W c} is close to the maximum sequence length of the PLM. Although DPT applies to most scenarios, it is recognized that our methods may require further improvement to accommodate huge-scale hierarchical label systems.

heading: A Hyper-parameter Settings
text: We list the hyper-parameter settings of four datasets in Table 6 for reproducibility.

heading: B Performance of Different Negative Sampling Ratio
text: To elaborate on the rationality behind selecting a K value as 10% of the label count at each layer, we compare the performance when the negative sampling rate is set at 10% and 100%. From Table 7 and Table 8, it’s obvious that 10% negative labels are sufficient, retaining the vast majority of accuracy on RCV1-V2 and even surpassing the performance of using all negatives on BGC dataset. It indicates that blindly increasing the number of negative samples is not always effective. We will explore the impact of positive and negative label ratio in the future work.

heading: C Computational Complexity of DPT
text: Table 9 compares computational complexity between our DPT model and the current state-ofthe-art prompt-based HPT model on RCV1-V2 dataset.The results show that our model has faster inference speed and acceptable training computational costs.

heading: D Case Study
text: DPT performs Peer-label Contrastive Learning at each level, which enhances the model’s representation abilities. Introduction of cross-hierarchical rank loss and label hierarchy self-sensing auxiliary task improves label consistency. To look into the practical effects, we conduct adequate case studies on the BGC dataset. Compared to the strong baseline HPT model, improvements of DPT are mainly reflected in recalling missed labels, correcting misjudged labels, removing excessive labels, and further correcting label inconsistencies. Table 10 provides some examples. Note that labels predicted by DPT model in the Table 10 are consistent with the ground truth labels.

heading: Future Work
text: We will explore the impact of positive and negative label ratio in the future work