SlowGuess/NAACL_2021_OCR huggingface dataset:
Hui Liu $^{1}$  Danqing Zhang $^{2}$  Bing Yin $^{2}$  Xiaodan Zhu $^{1}$
<sup>1</sup>Ingenuity Labs Research Institute & ECE, Queen's University, Canada
{hui.liu, xiaodan.zhu}@queensu.ca
$^{2}$ Amazon.com Inc, Palo Alto, CA, USA
{danqinz, alexbyin}@amazon.com
Exploiting label hierarchies has become a promising approach to tackling the zero-shot multi-label text classification (ZS-MTC) problem. Conventional methods aim to learn a matching model between text and labels, using a graph encoder to incorporate label hierarchies to obtain effective label representations (Rios and Kavuluru, 2018). More recently, pretrained models like BERT (Devlin et al., 2018) have been used to convert classification tasks into a textual entailment task (Yin et al., 2019). This approach is naturally suitable for the ZS-MTC task. However, pretrained models are underexplored in the existing work because they do not generate individual vector representations for text or labels, making it unintuitive to combine them with conventional graph encoding methods. In this paper, we explore to improve pretrained models with label hierarchies on the ZS-MTC task. We propose a Reinforced Label Hierarchy Reasoning (RLHR) approach to encourage interdependence among labels in the hierarchies during training. Meanwhile, to overcome the weakness of flat predictions, we design a rollback algorithm that can remove logical errors from predictions during inference. Experimental results on three real-life datasets show that our approach achieves better performance and outperforms previous non-pretrained methods on the ZS-MTC task.
Multi-label text classification (MTC) is a basic NLP problem that underlies many real-life applications like product categorization (Partalas et al., 2015) and medical records coding (Du et al., 2019). The labels in the output space are often interdependent and in many applications organized in a hierarchy, as shown in the example in Figure 1. A significant challenge for real-life development of MTC applications is severe deficiencies of annotated data for each label in the hierarchy, which demands better solutions for zero-shot learning.
Figure 1: An example of label hierarchy and predictions with logical errors. Circled labels are model predictions without incorporating label hierarchy.
The existing zero-shot learning for multi-label text classification (ZS-MTC) mostly learns a matching model between the feature space of text and the label space (Ye et al., 2020). In order to learn effective representations for labels, a majority of existing work incorporates label hierarchies via a label encoder designed as Graph Neural Networks (GNNs) that can aggregate the neighboring information for labels (Chalkidis et al., 2020; Lu et al., 2020).
Recently, pretrained models like BERT (Devlin et al., 2018) have been widely used as strong matching models due to their superior representation ability (Qiao et al., 2019). They have been applied to convert a classification task to a textual entailment task, by treating the text to be classified as the premise, and its label as the hypothesis, which is naturally suitable for the ZS-MTC study (Yin et al., 2019). However, the problem of this approach is that pretrained models cannot generate individual vector representations for labels—a label is coupled with the corresponding text in learning joint representation—thus conventional methods, like GNNs which utilize the label hierarchy to obtain better label representations, cannot be directly applied to pretrained models, making them underexplored in the existing research.
Although pretrained models have shown potential on ZS-MTC, as discussed above, it is not intuitive to introduce structural information of label hierarchies to the learning procedure. Flattening
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1051-1062 June 6-11, 2021. ©2021 Association for Computational Linguistics
all the labels without considering their hierarchical structures, however, will result in predictions that contain logical errors, which are known as the class-membership inconsistency (Silla and Freitas, 2011). The problem will be even more salient for pretrained models because they only take the literal tokens of the labels as input. An example with logical errors is shown in Figure 1. Without label hierarchy information, the model correctly predicts *Bikes* as a true label, but fails to predict its parent label, *Sporting Goods*. Meanwhile, the model does not choose the label *Local Services* while predicting its child label *Bike Repair* due to the fact that *Bike Repair* has tokens similar to those in the input text.
To overcome the forementioned weakness, we propose a Reinforced Label Hierarchy Reasoning (RLHR) approach to introduce label structure information to pretrained models. Instead of regarding labels to be independent, we cast ZS-MTC as a deterministic Markov Decision Process (MDP) over the label hierarchy. An agent starts from the root label and learns to navigate to the potential labels by hierarchical deduction in the label hierarchy. The reward is based on the correctness of the deduction paths, not simply on the correctness of each label. Thus the reward received by one predicted label will be determined by both the label itself and other labels on the same path, which can help to strengthen the interconnections among labels. Meanwhile, we find that the hierarchical inference method (Huang et al., 2019) will broadcast the errors arising at the higher levels of label hierarchies. Thus we further design a rollback algorithm based on the predicted matching scores of labels to reduce the logical errors in the flat prediction mode during inference. We apply our approach to different pretrained models and conduct experiments on three real-life datasets. Results demonstrate that pretrained models outperform conventional non-pretrained methods by a substantial margin. After being combined with our approach, pretrained models can attain further improvement on both the classification metrics and logical error metrics<sup>1</sup>. We summarize our contributions as follows:
- We demonstrate that pretrained models outperform conventional methods on ZS-MTC.
- We design a novel Reinforced Label Hierarchy Reasoning (RLHR) approach and a
matching-score-based rollback algorithm to introduce the structural information of label hierarchies to pretrained models in both the training and inference stage.
- Experiments with different pretrained models are performed on three real-life datasets. We show the effectiveness of our proposed approach and provide detailed analyses.
Exploiting the prior distribution of the label space has proven to be an effective method to tackle the multi-label text classification problem because it can provide the model with information about the label structure. Mao et al. (2019); Huang et al. (2019) took the explicitly represented label hierarchy as the structural information, while Wu et al. (2019) assumed the prior distribution to be implicit and trained their model to learn the distribution during learning.
Leveraging the label hierarchy to tackle ZS-MTC has shown to be promising in previous work, which mostly aimed to learn a matching model between texts and labels. Chalkidis et al. (2020, 2019); Xie et al. (2019) adopted Label-Wise Attention Networks to encourage interactions between text and labels. Rios and Kavuluru (2018); Lu et al. (2020) used Graph Neural Networks to capture the structural information in the label hierarchy. However, few existing works investigate the effectiveness of pretrained models on the ZS-MTC task, despite pretrained models being effective as matching models for many natural language processing tasks (Ma et al., 2019; Qiao et al., 2019; Nogueira et al., 2019).
The logical error problem in flat predictions has been widely discussed in previous MTC work (Silla and Freitas, 2011; Wehrmann et al., 2018; Mao et al., 2019), which is mostly solved through a hierarchical procedure during inference. In our work, we will investigate such a method and see that the hierarchical inference method is not optimal for pretrained models on the ZS-MTC task because it broadcasts errors top-down in the label hierarchy.
Path reasoning is effective for exploiting explicit relationships in structured data, which can be combined with reinforcement learning, e.g., knowledge graph reasoning (Wan et al., 2020; Xian et al., 2019; Xiong et al., 2017). We propose to introduce the label hierarchy to pretrained models through path reasoning, with the aim to strengthen the intercon
1Code and data available at https://github.com/layneins/Zero-shot-RLHR
nections between labels. To the best of our knowledge, our work is the first to improve pretrained models through label hierarchies for ZS-MTC.
3.1 Label Hierarchy Reasoning
In general, a label hierarchy is defined as  $\mathcal{G} = (\mathcal{L},\mathcal{E})$ , where  $\mathcal{L}$  and  $\mathcal{E}$  are a set of labels and relations, respectively. The latter represent parent-child relations between labels. The root of  $\mathcal{G}$  is a special label  $\mathbb{R}$ . A data instance  $x$  is defined as a tuple  $(T,P)$  with  $T$  as the input text and  $P = \{p_{1},p_{2},\dots ,p_{N}\}$  as deduction paths, and a path  $p_i = \{\mathbb{R},l_i^1,\dots ,l_i^{K - 1},l_i^K\}$  where  $l_i^k\in \mathcal{L}$  is at the  $k^{th}$  layer of  $\mathcal{G}$  and  $l_i^{k - 1}$  is the parent of  $l_i^k$ . A deduction path must be contiguous, starting with  $\mathbb{R}$ , and is not required to terminate at a leaf label.
3.2 Zero-shot Multi-label Text Classification
Let  $\mathcal{L}_s$  and  $\mathcal{L}_u$  denote the seen and unseen labels, respectively, where  $\mathcal{L}_s\cup \mathcal{L}_u = \mathcal{L}$ . Given a training set  $\mathcal{D}^s = \{\pmb {x}_i^s\}_{i = 1}^{N_1}$  where the labels of  $\pmb{x}_i^s$  are all seen labels, we aim to learn a matching model  $f(\mathcal{D}^s;\theta)$  and make prediction on  $\mathcal{D}^u = \{\pmb {x}_i^u\}_{i = 1}^{N_2}$ . Some deduction paths of  $\pmb{x}_i^u$  consist of seen labels while some contain both seen and unseen labels. Notice that the children of an unseen label are also unseen labels. Evaluations on  $\mathcal{D}^u$  will be conducted in two settings: (1) evaluate the performance on  $\mathcal{L}_u$ , which is known as the zero-shot (ZS) setting, and (2) evaluate the performance on  $\mathcal{L}_s\cup \mathcal{L}_u$ , which is the generalized zero-shot (GZS) setting (Huynh and Elhamifar, 2020).
The goal of our RLHR approach is to learn a policy  $\mathcal{P}$  that can make more consistent predictions by traversing the label hierarchy  $\mathcal{G}$  to generate deduction paths. Given a training instance  $x$ , an agent will start from the root  $\mathbb{R}$  and follow  $\mathcal{P}$  at each time step to extend the deduction paths by navigating to the children labels at the next level. By measuring the correctness of the generated deduction paths with reinforcement learning (RL), the label hierarchy is introduced to the model during the training time and the interconnections of labels will hence be strengthened, which can help to reduce logical errors in prediction. As we will show in our experiments, hierarchical inference, which is used in previous work (Mao et al., 2019), will propagate the errors occurring at the high levels of
hierarchies during inference, resulting in inferior performance. Thus we still adopt the flat prediction during inference, but further design a rollback algorithm based on the structure of  $\mathcal{G}$  and the predicted matching scores. We will introduce the details of our proposed RLHR and the rollback algorithm in the following subsections.
4.1 Base Model
Our base model adopts pretrained models  $m$ , e.g., BERT (Devlin et al., 2018), which have proven to be effective in matching modelling. Given the input text  $T$  and the label  $l$ , we follow Yin et al. (2019) by transforming the text-label pair into textual entailment representation as "[CLS]  $T$  [SEP] hypothesis of  $l$ ". The hidden vector  $v_{cls}$  of [CLS] is regarded as the aggregate representation and will be used in the classification layer to calculate the matching score  $ms$ . The overall calculation process of  $ms$  is abbreviated as:
$$
m s = \mathcal {M} (T, l) \tag {1}
$$
If  $ms \geq \gamma$  where  $\gamma$  is a threshold, we then say  $T$  belongs to label  $l$ . In experiments  $\gamma$  is set to be 0.5.
4.2 Reinforced Label Hierarchy Reasoning (RLHR)
Different from vanilla pretrained models that rely on flat prediction during training, we propose to formulate the ZS-MTC task as a deterministic Markov Decision Process (MDP) over label hierarchies. For the input text, the agent trained by RLHR will predict  $M$  deduction paths from the root label  $\mathbb{R}$ . When all deduction paths are generated, the rewards will be received, which are determined by the correctness of the paths. An overall illustration of the RLHR approach is shown in Figure 2. We introduce the details of the RL modules in this subsection.
4.2.1 States
Maintaining just one deduction path for one data instance will result in an inefficient learning process. However, the number of potential deduction paths will increase exponentially as the model goes deeper into the lower levels of the hierarchies. To maintain a good trade-off between computational resources and time efficiency, we keep the beam of deduction paths to be  $M$ . Thus for a data instance  $x$ , the global state  $S^k$  at step  $k$  is composed of the sub-states of  $M$  deduction paths:
$$
S ^ {k} = \left\{s _ {i} ^ {k} \right\} _ {i = 1} ^ {M} \tag {2}
$$
Figure 2: An example of our RLHR approach with  $M = 4$ . Green circles are the ground truth labels.  $p_1, p_2, p_3$ , and  $p_4$  are four sampled deduction paths, where  $p_3$  ends before it arrives at a leaf label.
The sub-state  $s_i^k$  for deduction path  $p_i$  at step  $k$  is defined as a tuple  $(T, l_i^k)$ , where  $T$  is input text and  $l_i^k$  is the label.
4.2.2 Actions
The complete action space  $A_{i}^{k}$  of sub-state  $s_{i}^{k}$  is defined as all possible child labels of label  $l_{i}^{k}$ :
$$
A _ {i} ^ {k} = \left\{l \mid l \in C \left(l _ {i} ^ {k}\right) \right\} \tag {3}
$$
where  $C(l_i^k)$  denotes the child labels of  $l_i^k$ . For the deduction path  $p_i$  at the time step  $k$ , an action  $a_i^k$  is to select one label  $l_i^{k+1}$  from  $A_i^k$ . Notice that the agent may not select any labels from  $A_i^k$ , which means path  $p_i$  ends before it arrives at a leaf label and a "stop" action is taken. By adding this "early stop" mechanism, we can make the agent automatically learn when to stop assigning new labels to the deduction paths.
4.2.3 Policy
We parameterize the action  $a_i^k$  by a policy network  $\pi (\cdot |s,A;\theta)$  where  $\theta$  is parameters. For deduction path  $p_i$  at time step  $k$ , the policy network takes as input the state  $s_i^k$  and the corresponding action space  $A_{i}^{k}$ , emitting the matching score of each action in  $A_{i}^{k}$ , which is calculated by the base pretrained model  $\mathcal{M}$ . Finally an action  $a_{k}$  is sampled based on the matching score distribution of the actions in  $A_{i}^{k}$ . The calculation is formulated as follows:
$$
\pi \left(\boldsymbol {a} _ {i} ^ {k} \mid s _ {i} ^ {k}, A _ {i} ^ {k}; \theta\right) = \left\{\mathcal {M} (T, l) \mid l \in A _ {i} ^ {k} \right\} \tag {4}
$$
$$
a _ {i} ^ {k} \sim \pi \left(a _ {i} ^ {k} \mid s _ {i} ^ {k}, A _ {i} ^ {k}; \theta\right) \tag {5}
$$
4.2.4 Reward
In our approach, the reward is based on the correctness of a complete deduction path. Instead of treating all labels to be flat, our approach encourages the interdependence among the labels. The reward received by a label  $l_{i}^{k}$  is not only decided by the correctness of itself but also the correctness of
other labels on the same deduction path  $p_i$ . Given the golden deduction paths  $\hat{P} = \{\hat{p}_1, \hat{p}_2, \dots, \hat{p}_N\}$ ,  $p_i$  will obtain a positive reward if  $p_i$  is in  $\hat{P}$  or  $p_i$  is a sub-path of a path in  $\hat{P}$ . Formally the reward of path  $p_i$  is defined as:
$$
r _ {i} = \left\{ \begin{array}{l l} \lambda \cdot 1, & \text {i f} p _ {i} \subseteq \hat {p} _ {j} \text {w h e r e} \hat {p} _ {j} \in \hat {P} \\ - 1, & \text {o t h e r w i s e}, \end{array} \right. \tag {6}
$$
where  $\lambda$  is a hyper-parameter for scaling. Under most circumstances, the number of wrong deduction paths will be greater than the correct ones. The problem will be even more severe for the MTC tasks because the distribution of positive labels and negative labels is usually imbalanced given a data instance  $x$ . A larger  $\lambda$  can encourage the model to focus more on the correct paths.
Notice that our approach differs from existing methods which adopt hierarchical classification (Sun and Lim, 2001; Peng et al., 2018). A hierarchical classification method based on the label hierarchy can only cast the influence from parent label to child label, while in our approach the influence is mutual between parent label and child label, which can hence strengthen the reasoning ability of the models.
4.2.5 Optimization
Our goal is to learn a stochastic policy  $\pi$  that maximize the expected total reward  $J(\theta)$  of the M sampled deduction paths, which can be formulated as:
$$
J (\theta) = E _ {\pi (a \mid s)} [ \sum_ {i = 1} ^ {M} r _ {i} (\boldsymbol {s}, \boldsymbol {a}) ] \tag {7}
$$
where  $\theta$  is the parameter of policy network. We adopt policy gradient (Sutton et al., 2000) as the optimization algorithm which updates  $\theta$  as:
$$
\theta \leftarrow \theta + \eta \nabla_ {\theta} \tilde {J} (\theta) \tag {8}
$$
where  $\eta$  is the discount learning rate. Since there are multiple deduction paths for one data instance, the gradient can be approximated by
$$
\tilde {J} (\theta) = \frac {1}{M} \sum_ {i = 1} ^ {M} \sum_ {k} \log \pi \left(a _ {i} ^ {k} \mid s _ {i} ^ {k}; \theta\right) \cdot \left(r _ {i} - r _ {b}\right) \tag {9}
$$
$r_b$  is a constant for the stabilization of the training procedure, for which we use the average reward of the last training epoch in our experiments.
4.3 Inference Rollerback
Existing methods mostly adopt the hierarchical inference method (Mao et al., 2019), which will avoid logical errors, i.e., class-membership inconsistency (Silla and Freitas, 2011), but bring a serious problem: the prediction errors made at the high levels of a hierarchy are often severely propagated to the lower levels. For instance, if a correct label at the first layer is missing, then all the descendant labels will not be considered during inference. This will no doubt harm the performance. On the contrary, if the model still makes flat prediction, all labels will be visited during inference, while more logical errors will probably arise.
To overcome the forementioned weaknesses, we propose a rollback algorithm during the inference stage based on the predicted matching scores of all labels. For a data instance  $x$ , we obtain the predicted labels in flat prediction mode as  $P$ , which consists of two parts: (1) labels that can form complete deduction paths, and (2) labels with logical errors, which we denote as  $P_{e} = \{l_{1}^{k_{1}}, l_{2}^{k_{2}}, \dots, l_{N}^{k_{N}}\}$ . For a label  $l_{i}^{k_{i}} \in P_{e}$ , we extract its deduction path from  $\mathcal{G}$  as  $p_{i} = \{\mathbb{R}, l_{i}^{1}, \dots, l_{i}^{k_{i}-1}, l_{i}^{k_{i}}\}$  and their corresponding predicted matching scores  $\{1, ms_{i}^{1}, \dots, ms_{i}^{k_{i}-1}, ms_{i}^{k_{i}}\}^{2}$ . Meanwhile we set a rollback threshold  $\mu^{k}$  for the labels in the  $k^{th}$  layer of  $\mathcal{G}$ , where  $\{\mu_{k}\}$  are hyper-parameters tuned on the development set. As long as the matching scores meet the requirements
$$
\{m s _ {i} ^ {j} \geq \mu^ {j} \} _ {j = 1} ^ {k _ {i} - 1},
$$
we add the labels in  $p_i$  back to  $P$ . Otherwise label  $l_i^{k_i}$  will be removed from  $P$ .
The motivation behind this matching-score-based rollback algorithm is that for a label hierarchy  $\mathcal{G}$ , the labels at higher-level hierarchy contain more training instances but their meaning are more
Dataset	Docs	Labels
#Train	#Dev	#Test	Avg(|L|)	seen unseen
Yelp	187153	10858	10858	3.80	466	71
WOS	36397	5294	5294	2.00	122	28
QCD	177423	12277	12277	4.69	243	93
Table 1: Dataset Statistics.  $\mathrm{Avg}(|L|)$  denotes the average number of labels in one data instance.
abstract, while the labels at lower levels are more specific such as the labels "Active Life" and "Bike Rentals" in Figure 1. Pretrained models just take as input the literal tokens of a label and thus are possible to obtain a better performance on certain labels at the lower levels than those at higher levels.
We conduct experiments on three real-life datasets from different domains; the details are provided in Table 1. Yelp<sup>3</sup> is a customer review dataset, in which we need to classify customer reviews into correct business categories. WOS (Kowsari et al., 2017) is a scientific paper dataset which provides the abstracts of published papers and the corresponding topics. QCD is a query classification dataset we create for the ZS-MTC task. It is composed of search queries and target product types, which is collected from e-commerce websites. The layer numbers of the label hierarchies in Yelp, WOS and QCD are 4, 2, and 3, respectively. For examples of the three datasets, please refer to Appendix A.1.
5.1.2 Implementation Details
We test our proposed approach with two pretrained models, BERT (Devlin et al., 2018) and DistilBERT (Sanh et al., 2019). For BERT, we use the uncased base version, which is of 12-layer transformer blocks, 768-dimension hidden state, 12 attention heads and 110M parameters in total. For DistilBERT, it contains 6-layers transformer blocks, 768-dimension hidden state and 12 attention heads, totally 66M parameters. For training, we use Adam (Kingma and Ba, 2014) for optimization and learning rate is set to 1e-6. Meanwhile we adopt early stopping to avoid overfitting on the training data.  $\lambda$  is set to 30 on Yelp, 20 on QCD, and 5 on WOS,
Root label  $\mathbb{R}$  always has a matching score 1.
Method	Setting	Yelp	WOS	QCD
Ma-F	Mi-F	EBF	Err↓	Ma-F	Mi-F	EBF	Err↓	Ma-F	Mi-F	EBF	Err↓
CNN	ZSGZS	0.331.31	2.0214.97	16.35	0.3211	0.367.00	4.4329.58	28.22	0.2977	5.029.66	6.5826.22	26.94	2.9386
CNN+LWAN	ZSGZS	4.244.67	7.1519.26	19.38	0.9303	0.546.81	4.5329.00	26.88	0.3079	5.0210.03	7.0928.86	28.24	4.3923
ZAGCNN	ZSGZS	17.9416.30	18.7525.97	28.24	1.3136	12.0219.59	17.1736.37	24.72	2.5827	5.2223.85	10.0142.52	40.65	2.0212
DistilBERT	ZSGZS	41.4221.29	40.3328.18	30.44	0.4039	70.6968.03	65.1963.64	55.18	0.5178	23.6824.43	24.9534.29	33.57	1.0854
+RLHR	ZSGZS	42.1626.95	43.8740.43	40.85	0.3347	74.5671.65	72.4468.05	61.06	0.4732	24.5826.10	27.7938.37	37.46	0.8389
BERT	ZSGZS	44.4923.38	42.6131.53	34.59	0.3755	77.8774.69	77.2770.56	56.69	0.1983	28.1827.04	27.4537.20	36.88	1.2497
+RLHR	ZSGZS	45.4632.09	48.2649.75	49.52	0.2952	78.4675.51	79.1972.62	64.43	0.2488	28.3228.67	28.8041.08	39.99	1.1984
Table 2: Results of different methods on the three datasets under two settings. Ma-F, Mi-F, EBF, and Err denote Macro-F1, Micro-F1, Example-based F1, and logical error rate, respectively. ZS and GZS denote the zero-shot and generalized zero-shot setting.  $\downarrow$  means the lower the better. Bold numbers indicate the best results for each metric. All the results are acquired under the flat prediction.
which we will discuss more in Section 5.3.4. We set  $M$  to 5 with DistilBERT and 3 with BERT by trading off between training time and GPU memory usage.
The RL training procedure is unstable and slow if the agent is trained from scratch (Silver et al., 2016). So with both BERT and DistilBERT, we pretrain the policy network in flat prediction mode on the training data with the learning rate of 1e-5.
5.1.3 Evaluation Metrics