Hierarchical Verbalizer for Few-Shot Hierarchical Text Classification
Year: 2023

Abstract:
Due to the complex label hierarchy and intensive labeling cost in practice, the hierarchical text classification (HTC) suffers a poor performance especially when low-resource or fewshot settings are considered. Recently, there is a growing trend of applying prompts on pretrained language models (PLMs), which has exhibited effectiveness in the few-shot flat text classification tasks. However, limited work has studied the paradigm of prompt-based learning in the HTC problem when the training data is extremely scarce. In this work, we define a path-based few-shot setting and establish a strict path-based evaluation metric to further explore few-shot HTC tasks. To address the issue, we propose the hierarchical verbalizer ("HierVerb"), a multi-verbalizer framework treating HTC as a singleor multi-label classification problem at multiple layers and learning vectors as verbalizers constrained by hierarchical structure and hierarchical contrastive learning. In this manner, HierVerb fuses label hierarchy knowledge into verbalizers and remarkably outperforms those who inject hierarchy through graph encoders, maximizing the benefits of PLMs. Extensive experiments on three popular HTC datasets under the few-shot settings demonstrate that prompt with HierVerb significantly boosts the HTC performance, meanwhile indicating an elegant way to bridge the gap between the large pre-trained model and downstream hierarchical classification tasks. 1

heading: 1 Introduction
text: Hierarchical text classification (HTC) is a longstanding research problem due to the wide range of real applications (Mao et al., 2019). However, prior works could still suffer poor performance in practice due to the nature of its sophisticated
∗Work done during an internship at Xiaobing.AI †Corresponding Author
1Our code and few-shot dataset are publicly available at https://github.com/1KE-JI/HierVerb.
label hierarchy as well as the requirement of largescale data annotation before training the model. Therefore, solving the HTC under the low-resource (Wang et al., 2022b) or few-shot setting becomes an urgent research topic.
Existing state-of-the-art HTC models focus on inserting label hierarchy features through graph encoders and then fuse the features into the input layer (Wang et al., 2022b) or output layer (Zhou et al., 2020) of a text encoder such as Bidirectional LSTM or pre-trained language models (PLMs), as shown in Figure 1(a). And there is a trend of taking advantage of PLMs (Chen et al., 2021; Wang et al., 2022b) as the backbone of the text encoder through a fine-tuning paradigm. Despite the success of PLMs (Devlin et al., 2019; Raffel et al., 2020) in extensive NLP-related tasks, recently, a series of studies (Petroni et al., 2019; Davison et al., 2019; Chen et al., 2022) suggest that it’s helpful to elicit
2918
the knowledge contained in PLMs and point out the fine-tuning paradigm is suboptimal in few-shot settings due to distinct training strategies between the pre-training and fine-tuning stages. Inspired by "incontext learning" proposed by GPT-3 (Brown et al., 2020), lots of prompt-based (Petroni et al., 2019; Gao et al., 2021a; Schick and Schütze, 2021; Qin and Eisner, 2021) methods were proposed to bridge the gap between pre-training and downstream tasks via stimulating pre-trained model knowledge with a few hard or soft prompts. In prompt-based tuning, the input is usually wrapped through a natural language template and the tasks are converted as masked language modeling (MLM) for PLM. For example, in the sentiment classification task, the original input x will be wrapped as "x. It was [MASK]”. The objective is to utilize MLM to predict the word that fills the [MASK], and subsequently employ a verbalizer to map the predicted word to the final classification (e.g. "positive" -> label "Positive").
Although remarkable performances have been achieved via prompt tuning on the flat text classification where labels have no hierarchy, its effects on HTC problems remain unclear, as discussed in HPT (Wang et al., 2022b). As shown in Figure 1(b), HPT proposed a hierarchy-aware prompt tuning method that incorporates the label hierarchy knowledge into soft prompts through graph representation and achieves the new state-of-the-art results on several HTC popular datasets. However, even though the low-resource setting experiment was considered in HPT, the commonly used K-shot setting was not investigated. The limitation lies in the absence of a uniform definition of the K-shot setting in HTC. Besides, the way to utilize PLMs in few-shot settings through soft prompts and fuse hierarchy by graph encoder into the PLMs harms tapping the full potential of PLMs. Hence, it is crucial to exploit a new method to elicit knowledge from PLMs in a hierarchy-aware manner for few-shot learning.
Inspired by the prior works on verbalizer design (Gao et al., 2021a; Schick and Schütze, 2021) between model outputs and labels, as shown in Figure 4(a) and 4(b), which makes promising improvements over prompt-based tuning, it is natural to raise this question: is there any verbalizer design method specific to the HTC problems? The most current works can be mainly divided into three kinds of verbalizers: manual verbalizers, search-
based verbalizers, and soft verbalizers. However, the main difference between previous works on verbalizers is the way of embedding the semantic space and they are all based on a strong assumption that there is no hierarchical dependency between downstream task labels, which raises a gap between rich flat prior knowledge in PLM and downstream task hierarchies. Thus these verbalizers are not suitable for hierarchical classification tasks, lacking awareness of hierarchy in their architectural design. To address these issues, we introduce a hierarchical-aware verbalizer (HierVerb) combined with the prompt tuning method to fully exploit the hierarchical knowledge within labels. The major contributions of this paper can be summarized as follows:
• To our best knowledge, we are the first to define the path-based few-shot setting on hierarchical text classification tasks and propose a path-based evaluation metric to further explore the consistency problem in HTC tasks.
• We propose HierVerb for few-shot HTC, which integrates the hierarchical information into the verbalizers through the flat hierarchical contrastive learning and hierarchy-aware constraint chain to better leverage the pretrained language model for few-shot learning.
• Experimental results demonstrate that HierVerb significantly outperforms the current state-of-the-art HTC methods on three popular benchmarks (WOS, DBPedia, and RCV1-V2) under extreme few-shot settings (i.e., K <=8), validating the effectiveness of its design.

heading: 2 Related Work

heading: 2.1 Hierarchical Text Classification
text: Current works for HTC focus on finding ways to insert the hierarchical label knowledge into the model, which proves to be beneficial for the problem induced by the imbalanced and large-scale label hierarchy faced in HTC problems (Mao et al., 2019). Several works (Zhang et al., 2022; Wu et al., 2019; Mao et al., 2019) applied the label-based attention module or utilized the meta-learning and reinforcement learning methods to leverage the label structure. However, as pointed out in HiAGM (Zhou et al., 2020), such methods mainly concentrate on optimizing decoding results based on the
constraint of hierarchical paths, it proposed to encode the holistic label structure with hierarchy encoders (graph or tree structure) which demonstrate to improve performance to a greater extent. Following the line of this research, Chen et al. (2021) exploited the relationship between text and label semantics using matching learning, and Wang et al. (2021) explicitly enriched the label embedding with concepts shared among classes. Yet since the label hierarchy representation remains unchanged regardless of the input, later works like HGCLR (Wang et al., 2022a) and HPT (Wang et al., 2022b) chose to migrate label hierarchy into text encoding instead of separately modeling text and labels. In addition to this, HPT achieves state-of-the-art by exploiting pre-trained language models through prompt tuning methods. Although the methods above are designed for HTC problems and promptbased techniques are applied, the frequently faced few-shot issues in HTC are less investigated, not to mention a suitable solution working well on limited training samples in a hierarchy-aware manner.

heading: 2.2 Prompt Tuning
text: Recent years have observed the widespread and powerful use of pre-trained language models (PLMs) in various downstream NLP tasks (Devlin et al., 2019; Qiu et al., 2020; Han et al., 2021). Prompt engineering goes a step further by designing a prompt template to take the power of PLMs to unprecedented heights, especially in few-shot settings (Liu et al., 2021). Later works focus on automatically discovering better hard prompts described in a discrete space to use in the querying process (Jiang et al., 2020; Gao et al., 2021a). Besides, there come with many methods that learn continuous soft prompts directly in the feature space of PLMs (Li and Liang, 2021; Lester et al., 2021; Qin and Eisner, 2021). Such continuous prompts reduce the hassle of constructing template words and transform them into parameterized embeddings.

heading: 2.3 Verbalizer Design
text: Verbalizers aim to reduce the gap between model outputs and label words, which has always been a critical issue in prompt-based tuning. Most of the current works leverage human written verbalizers (Schick and Schütze, 2021) that prove to be effective to build bridges between them. However, these approaches are highly biased towards lexical semantics of manual verbalizers and require both
domain expertise of downstream tasks and understanding of the PLMs’ abilities (Schick et al., 2020). Schick et al. (2020) and other studies (Gao et al., 2021a; Shin et al., 2020) have designed searchbased verbalizers for better verbalizer choices during the training optimization process, intending to reduce the bias caused by personal vocabulary and the cost of intensive human labor. Another line of researches (Hambardzumyan et al., 2021; Cui et al., 2022) claims it is hard to find satisfactory label words by searching large vocabulary with few examples and proposes to insert learnable embedding vectors as soft labels/verbalizers optimized during the training process. Nevertheless, the verbalizer design methods for hierarchical labels are less explored in previous works.

heading: 3 Preliminaries

heading: 3.1 Traditional HTC
text: In traditional HTC task, the structure of candidate labels yi ∈ Y are predefined as a Directed Acyclic Graph (DAG) H = (Y, E), where Y is the label set and E denotes the hierarchical connections within the labels. Specifically, H is a tree-like structure where every node except the root has one and only one parent. Hence the predicted hierarchical labels for one input sample correspond to single- or multipath taxonomic labels in H. It is worth noting that the HTC task is often viewed as a multi-label problem. Therefore the standard HTC task can be defined as follows: given an input text x={xt}Tt=1 and a label set Y, HTC aims to find a subset y from Y, in other words, to find one label path or multiple paths in H, for x.

heading: 3.2 Few-shot HTC
text: The few-shot problem has been extensively studied on tasks such as text classification, image segmentation, and named entity recognition (NER), while few works focus on the few-shot HTC task, which we call Few-HTC. It is easy to perform sampling strategies in flat single-label text classification to select K examples for each class added to the support set of K-shot learning. However, this sampling method is difficult to directly apply to HTC because an input sample may contain multiple labels. Hence it is harder to strictly meet the requirement of K shots for each corresponding class (Ding et al., 2021).
Inspired by the few-shot settings in named entity recognition (Yang and Katiyar, 2020; Ding et al., 2021) where they regard entity types as basic classes and sample few-shot sets based on each class through greedy sampling algorithms, we can define our few-shot settings based on the label paths in H since multiple slots in NER are analogous to multiple label paths in HTC. Figure 2 shows how we perform path-based sampling for building a Few-HTC support set.
Formally, the task of K-shot HTC is defined as follows: given a text x={xt}Tt=1 and a K-shot support set S for the target mandatory-leaf (Bi and Kwok, 2012) path set CT , the goal is to predict all golden paths on the label hierarchy tree for x. We design a greedy sampling method specifically for HTC problems and the details of obtaining CT and the support set S from the original HTC datasets are shown in Algorithm 1 to make sure each label path has exactly K-shot examples. To the best of our knowledge, we are the first to apply path-based few-shot settings on the HTC tasks.

heading: 4 Hierarchical Verbalizer
text: In this section, we will introduce the proposed hierarchy-aware verbalizer in detail. We incorporate hierarchical information through our multiverbalizer framework with prompt templates to elicit rich prior knowledge within the PLMs. Figure 3 shows the overall architecture of our proposed HierVerb. We first obtain the hidden states of the multiple mask tokens to represent the sentence and then project it to the verbalizer’s space of different label layers.
Algorithm 1 Greedy sampling for Few-shot HTC Input: shot K, original HTC dataset X{(x,y)} with label hierarchyH Output: K-shot support set S after sampling 1: CT ← //Initialize the original set of mandatory −
leaf paths 2: while ori_length ̸= cur_length do 3: ori_length← //Obtain the length of CT 4: Count the frequency of each Ci in X 5: Remove paths {Ci} with frequency less than K 6: Remove samples containing {Ci} in X 7: cur_length← //Obtain the length of CT 8: end while 9: {Ci : Ai} ← //Count the frequency of each Ci
appeared individually in the filtered dataset X 10: Sort the path set CT based on A 11: S← ϕ//Initialize an empty support set 12: {Counti} ← //Initialize the counts of all paths in
CT to zero 13: for i = 1 to |CT | do 14: while Counti < K do 15: Sample(x, y) ∈ Xs.t.Ci ∈ y,w/o replacement 16: S← S ∪ {(x, y)} 17: Update {Countj}∀ Cj ∈ y 18: end while 19: end for 20: return S

heading: 4.1 Multi-verbalizer Framework
text: Since the label hierarchy is a tree structure in our problem, we think of HTC as a single-label or multi-label classification task performed at multiple levels, following Wang et al. (2022b). In this way, we can easily construct templates based on the depth of the hierarchy tree. Given a piece of training text x and the label hierarchy H with a depth of D, the template p is written simply as "[CLS] It was 1 level:[MASK] 2 level:[MASK]...D level:[MASK]. x [SEP]". We use multiple [MASK] tokens for corresponding multi-level label predictions. Note that the number of [MASK] tokens is equal to the number of layers of H.
For better learning of hierarchical verbalizer and text representation in few-shot settings, we use BERT (Devlin et al., 2019) as our text encoder. For an input text x wrapped with the template T:
Tprompt(x) = {[CLS] It was ti ... tD. x [SEP]} (1) where ti means "i level:[MASK]". Note that our template T is a dynamically wrapped sentence containing as many t as the number of hierarchy layers. We feed the input x wrapped with the template T to the encoder of the BERT to obtain the hidden states h1:n:
h1:n = BERT(Tprompt(x)1:n) (2)
where h1:n ∈ Rn×r and r is the hidden state dimension of BERT and n is the length of Tprompt(x). For convenience, we pick out a subset {hd}(d ∈ [1, ..., D]) which is the set of hidden state vectors corresponding to all [MASK] tokens.
On top of this, we use multi-verbalizer for depthoriented learning and construct each verbalizer based on the full set of labels for the corresponding layer. Thus we have a list of verbalizers V = {Vd}(d ∈ [1, ..., D]). Each verbalizer is created as a virtual continuous vector Wd ∈ Rr×ld where ld is the number of labels of d-th layer and we initialize the embedding Wd of each Vd by averaging the embeddings of its corresponding label tokens and label tokens of all its descendants in H.
In our framework, the d-th mask is connected to the d-th verbalizer to play the role of predicting the d-th layer label.
We denote the distribution of the wrapped sentences in the corpus as O. The probability distribution of all labels yd on the layer d is:
PO(yd|Tprompt(x),D = d) = q(hdWd + bd) (3)
where Wd ∈ Rr×ld and bd ∈ Rld are weights and bias and q is a function used to convert logits into probabilities. Hence the predicted probability of
text i on label j of d-th layer is:
pdij = PO(yd = j|Tprompt(x),D = d) (4)
Following previous work (Zhou et al., 2020; Wang et al., 2022a), we use a binary cross-entropy loss function for multi-label classification. However, the definition of multi-label in our framework is slightly different from these works. The multi-label problem whose ground truth is a single path on the hierarchical dependency tree H can be redefined as a single-label prediction problem at each layer with the help of the multi-verbalizer. For such a single-path prediction, the loss function is defined as:
LCidj = −ydijlog(pdij) (5)
Instead, for multi-path problems:
LCidj = −ydijlog(pdij)− (1− ydij)log(1− pdij) (6)
To sum up, for each input text i, we can calculate the loss of the multi-verbalizer framework as:
LC = D∑
d
ld∑
j
LCidj = D∑
d
ld∑
j
LC(pdij , y d ij) (7)

heading: 4.2 Hierarchy-aware Constraint Chain
text: In order to reduce the gap between the training objective of the pre-trained model and the hierarchical objective, we first use the hierarchical constraint chain to solve this problem.
According to the label dependency tree H, we maintain a parent-to-child mapping −→ M between layers:
−→ Md(y d j ) = {yd+11 , yd+12 , ..., yd+1n } (8)
where yd is a label j belonging to the d-th layer and {yd+1n } are its corresponding children nodes at the (d+1)-th layer.
Thus the propagated probability of text i on label j of d-th layer can be obtained through:
p̃dij = (1− β)pdij + β ∑ pd+1 ij̃ , j̃ ∈ −→Md(j) (9)
which is implemented to quantify constraints from descendant nodes where β controls the degree of descendant node constraints. Since we are propagating from the bottom up, our computational constraints gradually propagate upward from the leaf nodes of the hierarchy tree. The loss of the constraint chain can be defined as:
LHCC = D∑
d
ld−1∑
j
LC(p̃dij , y d ij) (10)

heading: 4.3 Flat Hierarchical Contrastive Loss
text: Secondly, we design the flat hierarchical contrastive loss objective to learn the hierarchy-aware matching relationship between instances, instead of the relationship between instances and labels as proposed in Chen et al. (2021). It is non-trivial to match different instances due to the sophisticated semantics of each instance in the hierarchical setting. Given input sentence representation and the label hierarchy, there are two main goals we want to achieve through optimization: (1) For sentence pairs, the representations of intra-class correspondences at each level should obtain higher similarity scores than inter-class pairs. (2) The similarity between lower-level representations of intra-class pairs deserves more weight than that of relatively high-level ones. To achieve our goal, we flatten the hierarchy into a multi-level lattice structure and define our objective function based on the SimCSE estimator (Gao et al., 2021b), which is widely used in contrastive learning.
Denote B = {(Xn, {Y d}n)} as one batch where {Y d}n is the original labels in d-th layer, n ∈ N, d ∈ D, where N denotes the batch size and D denotes the maximum depth of the label hierarchy H. Following SimCSE, we can have 2N sets of hidden vectors for all corresponding [MASK] tokens Z = {z ∈ {hd} ∪ {h̃d}} where h̃d is simply obtained by feeding original text into the encoder for the second time. Any sentence pairs in one batch can be defined as P = [(Xa, {Y d}a), (Xb, {Y d}b)], and we keep a lattice label matrix:
Md(a, b) = { 1, {Y d}a ∩ {Y d}b ̸= ϕ 0, {Y d}a ∩ {Y d}b = ϕ (11)
Thus the final flat hierarchical contrastive loss function is defined as:
LFHC = −1
N2D2 D∑ d d∑ u 2N∑ n log
exp( ∑
n′ S(h u n,h u n′ )Mu(n,n ′)) exp( ∑ n′ S(h u n,h u n′ )) × 1 2(D−d)×α
(12) where S is cosine similarity function, hdn is the hidden states of d-th [MASK] for sentence n, and α controls the relative penalty importance of different layers.
Considering that once Md(n, n′) equals to one, all Mu(n, n′) can be assured to be equal to one by reason of the tree structure. Thereafter it assigns more weight to the contrastive loss of the lower layer whose d value is larger, and α intensifies the differentiation between all layers. This results in the inequality Distance d1 < d2 < d3 < d4 in Figure 3.

heading: 4.4 Classification Objective Function
text: Overall, our final training objective is the combination of multi-verbalizer framework loss, constraint chain loss, and flat hierarchical contrastive loss.
L = LC + λ1LHCC + λ2LFHC (13) where λ1 and λ2 are the hyperparameters controlling the weights of corresponding loss and HCC and FHC stand for Hierarchy-aware Constraint Chain and Flat Hierarchical Contrastive Loss respectively.

heading: 5 Experiments

heading: 5.1 Experiments Setup
text: Experimental settings As mentioned in Preliminaries, we focus on few-shot settings that only K samples for each label path are available for training on a new HTC task called Few-HTC in this work. In order to better study the few-shot generalization ability of the model under different scales of training data, we conduct experiments based on K ∈ {1,2,4,8,16}.
Datasets and Implementation Details We evaluate our proposed method on three widely used datasets for hierarchical text classification: Webof-Science (WOS) (Kowsari et al., 2017), DBpedia (Sinha et al., 2018) and RCV1-V2 (Lewis et al., 2004). WOS and DBPedia are for single-path HTC while RCV1-V2 includes multi-path taxonomic labels. The statistic details are illustrated in Table 1. For implementation details, please refer to Appendix A.
Evaluation Metrics Similar to previous work, we measure the experimental results with MacroF1 and Micro-F1. To further evaluate the consistency problem between layers, we adopt path-constrained MicroF1 (C-MicroF1) and path-
constrained MacroF1 (C-MacroF1) proposed in Yu et al. (2022) which we refer to collectively as Cmetric. In C-metric, a correct prediction for a label node is valid only if all its ancestor nodes are correct predictions, otherwise, it is regarded as a misprediction. However, in the case of path splitting based on the mandatory-leaf nodes, the metric is still not sufficient to provide a comprehensive evaluation of hierarchical path consistency, because it ignores the correctness of a node’s children nodes. Therefore, we propose a new path-constrained evaluation method based on the perspective of path correctness, which is called P-metric (PMacro-F1 and PMicro-F1). The details of our P-metric are shown in Appendix B.
Baselines We select a few recent state-of-the-art works as baselines: HiMatch (Using BERT as encoder) (Chen et al., 2021), HGCLR (Wang et al., 2022a) and HPT (Wang et al., 2022b). We also perform the vanilla fine-tuning method on the Fewshot HTC task, which we refer to as Vanilla FT in the following.

heading: 5.2 Main Results
text: Main experimental results are shown in Table 2. As is shown, HierVerb wins over all comparison models by a dramatic margin under nearly all sit-
K Method WOS
PMicro-F1 PMacro-F1 CMicro-F1 CMacro-F1
uations. Appendix C more intuitively shows the performance gap between different models.
In the case of no more than 4 shots on WOS, 8.9%, 9.18%, and 6.91% micro-F1 absolute improvement and 19.27%, 18.3%, and 16.87% macroF1 absolute improvement from the best baseline methods are achieved, respectively. Under 1-shot situations, compared with all baseline models, there is an average of 57.58% micro, 74.79% macro-F1 absolute improvement on DBPedia, and 20.46% micro-F1, 2.06% macro-F1 absolute improvement on RCV1-V2. Although the RCV1-V2 dataset provides no label name which has a negative effect on our verbalizer initialization, our method still achieves state-of-the-art on both Micro-F1 and Macro-F1 under almost all few-shot experiments.
There are three main reasons why HierVerb performs better under the few-shot setting: (1) Not require additional learning parameters. Previous methods like HPT and HGCLR improve the performance by adding extra parameters to the GNN layers, which could lead to overfitting for few-shot settings; (2) Multi-Verb is better than the single-flat verb. The previous methods are to first stretch the hierarchical label into a flattened one-dimensional space and then do multi-label prediction, more like a normal multi-label classification task with hierarchical dependencies on labels. In contrast, HierVerb advocates preserving the original hierarchical concept in the architecture through a multi-verb framework. (3) Our hierarchical loss is optimized
from a semantic perspective for better generalization.

heading: 5.3 Consistency Between Multi-layers
text: Table 3 further studies the consistency performance. Since our method is optimized from a semantic perspective, more consideration is given to the potential semantic dependency between different labels rather than directly fitting specific downstream data, our method still maintains excellent consistency performance in the absence of sufficient labeled training corpora. It is clear that HGCLR and BERT (Vanilla FT) using the direct fitting method only achieve 0 points in PMicro-F1 and PMacroF1 under the 1 shot setting. As for HPT, extra graph parameter learning hurts the generalization of PLMs. The complete experiments and analyses on the other two datasets are shown in Appendix D.

heading: 5.4 Ablation Study
text: The main parts of our work are the multi-verbalizer framework, hierarchy-aware constraint chain, and
flat hierarchical contrastive loss. To illustrate the effect of these parts, we test our model by gradually removing each component of our model at a time by default, as shown in Table 4. We implement Vanilla Soft Verbalizer (Hambardzumyan et al., 2021) in our own version which we refer to as SoftVerb in the following for convenience. Similar to HierVerb, the SoftVerb also uses multiple [MASK] tokens, but only uses a single flat verbalizer to map the label. Compared to SoftVerb which uses a single flat verbalizer, using multi-verbalizer and integrating hierarchical information into the verbalizer of each layer through FHC and HCC leads to better performance.

heading: 5.5 Effects of Model Scales
text: In previous experiments like § 5.2, we show that HierVerb is powerful on bert-base-uncsaed. To further study the ability of HierVerb to utilize the prior knowledge of the pre-trained language model, we conduct experiments on bert-large-uncased. Table 5 demonstrates that HierVerb consistently outperforms all baseline models in all shot settings. We find that the gap is even significantly larger for HierVerb and all other baseline models compared to using bert-base-uncased. For example, under 1-shot setting, HierVerb achieves a 27.92% increase in macro-F1 and an 11.54% increase in micro-F1, compared with HPT. But in the case of bert-base-uncased, the improvements of macro-F1 and micro-F1 are 19.27% and 8.9% respectively, which further emphasizes that our model is superior to all baseline models in the ability to mine the prior knowledge of the language model, and this effect is more significant when the scale of the language model increases.

heading: 5.6 Performance Benefit in a Full-shot Setup
text: We conduct experiments on HierVerb in a full-shot setting. Instead of carefully selecting hyperparameters, we directly use the parameter set from the few-shot settings. For baseline models, we reproduce their experiments according to the settings in their original paper. Although HierVerb is designed to be more favored for few-shot settings, the performance of full-shot setup is still quite competitive compared with HPT. As shown in Table 6, our overall micro-F1 score is only 0.10 lower than HPT (which requires to learn extra parameters of GNN), while achieving a macro-F1 score 0.13% higher than HPT. In fact, HierVerb outperforms BERT (Vanilla FT) and HiMatch by a significant
margin.

heading: 6 Conclusion
text: In this paper, we define the few-shot settings on HTC tasks and a novel evaluation method based on the perspective of path correctness, which is valuable in practical applications. We propose a novel approach to adapt flat prior knowledge in PLM to downstream hierarchical tasks. The proposed HierVerb learns hierarchical-aware verbalizers through flat contrastive learning and constraint chain, which elegantly leverages the prior knowledge of PLMs for better few-shot learning. We perform few-shot settings on HTC tasks and extensive experiments show that our method achieves state-of-the-art performances on 3 popular HTC datasets while guaranteeing excellent consistency performance.
Limitations
Since the appearance of large pre-trained models such as GPT-3 (Brown et al., 2020), there has been a wave of using large models without fine-tuning to do in-context learning directly to complete various NLP tasks, or to freeze the parameters of large models and then only optimize task-oriented parameters. The proposed HierVerb is a lightweight method especially suitable for the case of insufficient labeled training data, but it is difficult to directly extend to a large-scale language model (i.e, >=175B) because large language models are hard to fine-tune in many situations. In future work, we plan to study our method on a larger scale language model in which only parts of parameters specific to downstream HTC tasks need to be learned and further, extend our model to the zero-shot learning scenario.
Ethics Statement
All datasets for our research are publicly available and all experimental results are based on three different random seeds. We obtain these experimental results using the experimental setup mentioned in this work. For the sake of energy saving, we will not only open source the few-shot datasets under all random seeds and the code, but also release the checkpoints of our models from the experiments to reduce unnecessary carbon emissions.

heading: B Path-based Evaluation Metric
text: Specifically, in P-metric, we evaluate the confusion matrix of all label path ids instead of the original label ids. Besides, only if all {yi} labels involved in one path are predicted accurately, the corresponding path id is regarded as correct in the confusion matrix. We count the total number of golden labels as Countgold and at the same time record the predicted labels that do not form a complete path with other predicted labels as invalid and count their
Labels
total as Countinvalid. We define:
γ = 1− 2× ( 1 (1 + e−a) − 0.5) (14)
where a = CountinvalidCountgold and multiply γ with PMacroF1 and PMicro-F1 obtained from the confusion matrix to get our final PMacro-F1 and PMicro-F1 so that we can penalize the evaluation score to get a fairer evaluation when the model smartly predicts a particularly large number of labels that do not form a complete path, considering that we are building confusion matrix based on the path. Figure 5 shows the inconsistency problem.

heading: C Performance Gap between Different Models
text: The performance gap on three datasets between different models is clearly shown in Figure 6-8. The gap keeps growing as the shots become fewer. It can be clearly seen that both HierVerb’s Micro-F1 and Macro-F1 change very slightly from 1 to 16 shots on DBPedia while other models are particularly dependent on the increase of labeled training samples.

heading: D Complete Consistency Experiments
text: We further conduct consistency experiments on two other datasets. The results are shown in Ta-
Fungus
1 2
43 65
7 98 1110
NaturalE
Earthquake SolarEclipse
Species
Plant
Event
ConiferFilmFestival
SocietalE
Root
0
Eukaryote
(a) Fungus
1 2
43 65
7 98 1110
NaturalE
Earthquake SolarEclipse
Species
Plant
Event
ConiferFilmFestival
SocietalE
Root
0
Eukaryote
(b) Fungus
1 2
43 65
7 98 1110
NaturalE
Earthquake SolarEclipse
Species
Plant
Event
ConiferFilmFestival
SocietalE
Root
0
Eukaryote
(c) Fungus
1 2
43 65
7 98 1110
NaturalE
Earthquake SolarEclipse
Species
Plant
Event
ConiferFilmFestival
SocietalE
Root
0
Eukaryote
(d)
Figure 5: An example of taxonomic hierarchy for HTC. Assume we have an input sentence to predict, then we have (a) Golden labels of the input sentence, (b) and (c) Predicted labels lead to "label path inconsistency", and (d) Correctly predict all the labels in the path consisting of the node set {1,3,7}. We pick out these predicted labels in (c) that cannot form a complete path with any other predicted labels and record them as Yinvalid={1,3,10}, then we calculate the size of the set Yinvalid and add it to Countinvalid.
K Method DBPedia RCV1-V2
PMicro-F1 PMacro-F1 CMicro-F1 CMacro-F1 PMicro-F1 PMacro-F1 CMicro-F1 CMacro-F1
1 Ours 83.56 77.96 89.80 81.78 - - 39.41 5.16 HPT 61.08 57.80 82.84 66.99 - - 21.92 2.87 HGCLR 0.0 0.0 28.05 0.24 - - 23.26 1.04 Vanilla FT 0.0 0.0 28.08 0.24 - - 19.37 1.02
2 Ours 88.58 86.35 93.61 88.96 - - 45.11 12.32 HPT 82.36 81.41 92.31 86.43 - - 38.24 7.00 HGCLR 54.55 3.72 67.70 26.41 - - 24.24 0.89 Vanilla FT 53.83 3.71 67.72 26.89 - - 23.60 0.81
4 Ours 91.90 91.38 95.74 92.87 - - 54.67 23.80 HPT 87.61 87.04 94.50 90.42 - - 50.68 20.54 HGCLR 55.34 3.76 67.54 28.60 - - 44.74 9.02 Vanilla FT 55.15 3.74 67.44 28.32 - - 22.42 0.63
Table 8: Consistency experiments on the DBPedia and RCV1-V2 datasets using two path-constraint metrics. PMicro-F1 and PMacro-F1 are our proposed path-based consistency evaluation P-metric. Since the label distribution of the original test set of RCV1-V2 is not mandatory-leaf in H while WOS and DBPedia are, we use only C-metric on RCV1-V2 to evaluate the consistency performance. We report the mean F1 scores (%) over 3 random seeds. Bold: best results. All experiments use their respective metrics as a signal for early stopping.
ble 8. In all experiments, HGCLR and Vanilla FT consistently perform poorly on both P-Metric and C-Metric, while HierVerb and HPT achieved relatively high results, indicating that the prompt-based method can better use the prior knowledge in the pre-trained model to elicit potential semantic associations between natural language texts of all labels belonging to the same path.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
section Limitations
7 A2. Did you discuss any potential risks of your work? Our work is only for academic research purposes.
3 A3. Do the abstract and introduction summarize the paper’s main claims? section Abstract and section 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 7 Did you use or create scientific artifacts? Left blank.
B1. Did you cite the creators of artifacts you used? No response.
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? No response.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? No response.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? No response.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? No response.
B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. No response.
C 3 Did you run computational experiments? section 5
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? section 5
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? section 5
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? section 5
7 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? We haven’t used the existing packages for evaluation. We use the code written by ourselves. The code we use will publish the code upon acceptance.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: In future work, we plan to study our method on a larger scale language model in which only parts of parameters specific to downstream HTC tasks need to be learned and further, extend our model to the zero-shot learning scenario