SlowGuess/ACL_2020_OCR huggingface dataset:
Jie Zhou $^{1,2*}$ , Chunping Ma $^{2}$ , Dingkun Long $^{2}$ , Guangwei Xu $^{2}$ , Ning Ding $^{3}$ , Haoyu Zhang $^{4}$ , Pengjun Xie $^{2}$ , Gongshen Liu $^{1\dagger}$
$^{1}$ School of Electronic Information and Electrical Engineering, Shanghai Jiao Tong University  
 $^{2}$ Alibaba Group,  $^{3}$ Tsinghua University,  $^{4}$ National University of Defense Technology  
{sanny02,lgshen}@sjtu.edu.cn,  
{kunka.xgw,chengchen.xpj}@taobao.com  
{chunping.mcp,dingkun.ldk}@alibaba-inc.com  
{dingn18}@mails.tsinghua.edu.cn, {zhanghaoyu10}@nudt.edu.cn
Hierarchical text classification is an essential yet challenging subtask of multi-label text classification with a taxonomic hierarchy. Existing methods have difficulties in modeling the hierarchical label structure in a global view. Furthermore, they cannot make full use of the mutual interactions between the text feature space and the label space. In this paper, we formulate the hierarchy as a directed graph and introduce hierarchy-aware structure encoders for modeling label dependencies. Based on the hierarchy encoder, we propose a novel end-to-end hierarchy-aware global model (Hi-AGM) with two variants. A multi-label attention variant (HiAGM-LA) learns hierarchy-aware label embeddings through the hierarchy encoder and conducts inductive fusion of label-aware text features. A text feature propagation model (HiAGM-TP) is proposed as the deductive variant that directly feeds text features into hierarchy encoders. Compared with previous works, both HiAGM-LA and HiAGM-TP achieve significant and consistent improvements on three benchmark datasets.
Text classification is widely used in Natural Language Processing (NLP) applications, such as sentimental analysis (Pang and Lee, 2007), information retrieval (Liu et al., 2015), and document categorization (Yang et al., 2016). Hierarchical text classification (HTC) is a particular multi-label text classification (MLC) problem, where the classification result corresponds to one or more nodes of a taxonomic hierarchy. The taxonomic hierarchy is commonly modeled as a tree or a directed acyclic graph, as depicted in Figure 1.
Existing approaches for HTC could be categorized into two groups: local approach and global
Input text: ["David Beckham's new book will be published."
Figure 1: This short sample is tagged with news, sports, football, features and books. Note that HTC could be either a single-path or a multi-path problem.
approach. The first group tends to construct multiple classification models and then traverse the hierarchy in a top-down manner. Previous local studies (Wehrmann et al., 2018; Shimura et al., 2018; Banerjee et al., 2019) propose to overcome the data imbalance on child nodes by learning from parent one. However, these models contain a large number of parameters and easily lead to exposure bias for the lack of holistic structural information. The global approach treats HTC problem as a flat MLC problem, and uses one single classifier for all classes. Recent global methods introduce various strategies to utilize structural information of top-down paths, such as recursive regularization (Gopal and Yang, 2013), reinforcement learning (Mao et al., 2019) and meta-learning (Wu et al., 2019). There is so far no global method that encodes the holistic label structure for label correlation features. Moreover, these methods still exploit the hierarchy in a shallow manner, thus ignoring the fine-grained label correlation information that has proved to be more fruitful in our work.
In this paper, we formulate the hierarchy as a directed graph and utilize prior probabilities of label dependencies to aggregate node information. A hierarchy-aware global model (HiAGM) is pro
*This work was done during intern at Alibaba Group.
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1106-1117 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
posed to enhance textual information with the label structural features. It comprises a traditional text encoder for extracting textual information and a hierarchy-aware structure encoder for modeling hierarchical label relations. The hierarchy-aware structure encoder could be either a TreeLSTM or a hierarchy-GCN where hierarchical prior knowledge is integrated. Moreover, these two structure encoders are bidirectionally calculated, allowing them to capture label correlation information in both top-down and bottom-up manners. As a result, HiAGM is more robust than previous top-down models and is able to alleviate the problems caused by exposure bias and imbalanced data.
To aggregate text features and label structural features, we present two variants of HiAGM, a multi-label attention model HiAGM-LA and a text feature propagation model HiAGM-TP. Both variants extract hierarchy-aware text features based on the structure encoders. HiAGM-LA extracts the inductive label-wise text features while HiAGM-TP generates hybrid information in a deductive manner. Specifically, HiAGM-LA updates the label embedding across the holistic hierarchy and then employs node outputs as the hierarchy-aware label representations. Finally, it conducts multi-label attention for label-aware text features. On the other hand, HiAGM-TP directly utilizes text features as the input of the structure encoder in a serial dataflow. Hence it propagates textual information throughout the overall hierarchy. The hidden state of each node in the entire hierarchy represents the class-specific textual information.
The major contributions of this paper are:
- With the prior hierarchy knowledge, we adopt typical structure encoders for modeling label dependencies in both top-down and bottom-up manners, which has not been investigated for hierarchical text classification.
- We propose a novel end-to-end hierarchy-aware global model (HiAGM). We further present two variants for label-wise text features, a hierarchy-aware multi-label attention model (HiAGM-LA) and a hierarchy-aware text feature propagation model (HiAGM-TP).
- We empirically demonstrate that both variants of HiAGM achieve consistent improvements on various datasets when using different structure encoders. Our best model outperforms the state-of-the-art model by  $3.25\%$  of MacroF1 and  $0.66\%$  of Micro-F1 on RCV1-V2.
- We release our code and experimental splits of Web-of-Science and NYTimes for reproducibility.
Existing works for HTC could be categorized into local and global approaches. Local approaches could be subdivided into local classifier per node (LCN) (Banerjee et al., 2019), local classifier per parent node (LCPN) (Dumais and Chen, 2000), and local classifier per level (LCL)(Shimura et al., 2018; Wehrmann et al., 2018; Kowsari et al., 2017). Banerjee et al. (2019) transfers parameters of the parent model for child models as LCN. Wehrmann et al. (2018) alleviates exposure bias problem by the hybrid of LCL and global optimizations. Peng et al. (2018) decomposes the hierarchy into subgraphs and conducts Text-GCN on n-gram tokens.
The global approach improves flat MLC models with the hierarchy information. Cai and Hofmann (2004) modifies SVM to Hierarchical-SVM by decomposition. Gopal and Yang (2013) proposes a simple recursive regularization of parameters among adjacent classes. Deep learning architectures are also employed in global models, such as sequence-to-sequence (Yang et al., 2018), meta-learning (Wu et al., 2019), reinforcement learning (Mao et al., 2019), and capsule network (Peng et al., 2019). Those models mainly focus on improving decoders based on the constraint of hierarchical paths. In contrast, we propose an effective hierarchy-aware global model, HiAGM, that extracts label-wise text features with hierarchy encoders based on prior hierarchy information.
Moreover, the attention mechanism is introduced in MLC by Mullenbach et al. (2018) for ICD coding. Rios and Kavuluru (2018) trains label representation through basic GraphCNN and conducts mutli-label attention with residual shortcuts. AttentionXML (You et al., 2019) converts MLC to a multi-label attention LCL model by label clusters. Huang et al. (2019) improves HMCN (Wehrmann et al., 2018) with label attention per level. Our HiAGM-LA, however, employs multi-label attention in a single model with a simplified structure encoder, reducing the computational complexity.
Recent works, in semantic analysis (Chen et al., 2017b), semantic role labeling (He et al., 2018) and machine translation (Chen et al., 2017a), shows the improvement on sentence representation of syntax
<https://github.com/Alibaba-NLP/HiAGM>
Figure 2: Example of the taxonomic hierarchy. The number indicates the prior probability of label dependencies according to the training corpus.
encoder, such as Tree-Based RNN (Tai et al., 2015; Chen et al., 2017a) and GraphCNN (Marcheggiani and Titov, 2017). We modify those structure encoders for HTC with fine-grained prior knowledge in both top-down and bottom-up manners.
Hierarchical text classification (HTC), a subtask of text classification, organizes the label space with a predefined taxonomic hierarchy. The hierarchy is predefined based on holistic corpus. The hierarchy groups label subsets according to class relations. The taxonomic hierarchy mainly contains the tree-like structure and the directed acyclic graph (DAG) structure. Note that DAG can be converted into a tree-like structure by distinguishing each label node as a single-path node. Thus, the taxonomic hierarchy can be simplified as a tree-like structure.
As illustrated in Figure 2, we formulate a taxonomic hierarchy as a directed graph  $G = (V, \vec{E}, \vec{E})$  where  $V$  refers to the set of label nodes  $V = \{v_1, v_2, \ldots, v_C\}$  and  $C$  denotes the number of label nodes.  $\vec{E} = \{(v_i, v_j) | i \in V, j \in child(i)\}$  is the top-down hierarchy path and  $\vec{E} = \{(v_j, v_i) | i \in V, j \in child(i)\}$  is the bottom-up hierarchy path. Formally, we define HTC as  $H = (X, L)$  with a sequence of text objects  $X = (x_1, x_2, \ldots, x_N)$  and an aligned sequence of supervised label sets  $L = (l_1, l_2, \ldots, l_N)$ .
As depicted in Figure 1, each sample  $x_{i}$  corresponds to a label set  $l_{i}$  that includes multiple classes. Those corresponding classes belong to either one or more sub-paths in the hierarchy. Note that the sample belongs to the parent node  $v_{i}$  in the condition pertaining to the child node  $v_{j} \in child(i)$ .
4 Hierarchy-Aware Global Model
As depicted in Figure 3, we propose a Hierarchy-Aware Global Model (HiAGM) that leverages the fine-grained hierarchy information and then aggregates label-wise text features. HiAGM consists of a traditional text encoder for textual information and a hierarchy-aware structure encoder for hierarchical label correlation features.
We present two variants of HiAGM for hybrid information aggregation, a multi-label attention model (HiAGM-LA) and a text feature propagation model (HiAGM-TP). HiAGM-LA updates label representations with the structure encoder and generates label-aware text features with multi-label attention mechanism. HiAGM-TP propagates text representations throughout the holistic hierarchy, thus obtaining label-wise text features with the fusion of label correlations.
4.1 Prior Hierarchy Information
The taxonomic hierarchy describes the hierarchical relations among labels. The major bottleneck of HTC is how to make full use of this established structure. Previous studies directly utilize this hierarchy path in a static method based on a pipeline framework, hierarchical model or label assignment model. In contrast, based on Bayesian statistical inference, HiAGM leverages the prior knowledge of label correlations regarding the predefined hierarchy and corpus. We exploit the prior probability of label dependencies as prior hierarchy knowledge.
Suppose that there is a hierarchy path  $e_{i,j}$  between the parent node  $v_i$  and child node  $v_j$ . This edge feature  $f(e_{i,j})$  is represented by the prior probability  $P(U_j|U_i)$  and  $P(U_i|U_j)$  as:
$$
\begin{array}{l} P \left(U _ {j} \mid U _ {i}\right) = \frac {P \left(U _ {j} \cap U _ {i}\right)}{P \left(U _ {i}\right)} = \frac {P \left(U _ {j}\right)}{P \left(U _ {i}\right)} = \frac {N _ {j}}{N _ {i}}, \tag {1} \\ P (U _ {i} | U _ {j}) = \frac {P (U _ {i} \cap U _ {j})}{P (U _ {j})} = \frac {P (U _ {j})}{P (U _ {j})} = 1. 0, \\ \end{array}
$$
where  $U_{k}$  means the occurrence of  $v_{k}$  and  $P(U_{j}|U_{i})$  is the conditional probability of  $v_{j}$  given that  $v_{i}$  occurs.  $P(U_{j} \cap U_{i})$  is the probability of  $\{v_{j}, v_{i}\}$  occurring simultaneously.  $N_{k}$  refers to the number of  $U_{k}$  in the training subset. Note that the hierarchy ensures  $U_{k}$  given that  $v_{child(k)}$  occurs. We rescale and normalize the prior probabilities of child nodes  $v_{child(k)}$  to sum total to 1.
Figure 3: The overall structure of our hierarchy-aware global model. HiAGM consists of a text encoder and a hierarchy-aware encoder. The dataflows of structure encoders are illustrated in the grey dashed box. Two variants, as HiAGM-LA and HiAGM-TP, are presented in black dashed boxes, respectively.
4.2 Hierarchy-Aware Structure Encoder
Tree-LSTM and graph convolutional neural networks (GCN) are widely used as structure encoders for aggregating node information in NLP (Tai et al., 2015; Chen et al., 2017a; He et al., 2018; Rios and Kavuluru, 2018). As depicted in Figure 3, HiAGM models fine-grained hierarchy information based on the hierarchy-aware structure encoder. Based on the prior hierarchy information, we improve typical structure encoders for the directed hierarchy graph. Specifically, the top-down dataflow employs the prior hierarchy information as  $f_{c}(e_{i,j}) = \frac{N_{j}}{N_{i}}$  while the bottom-up one adopts  $f_{p}(e_{i,j}) = 1.0$ .
Bidirectional Tree-LSTM Tree-LSTM could be utilized as our structure encoder. The implementation of Tree-LSTM is similar to syntax encoders(Tai et al., 2015; Zhang et al., 2016; Li et al., 2018). The predefined hierarchy is identical to all samples, which allows the mini-batch training method for this recursive computational module. The node transformation is as follows:
$$
\boldsymbol {i} _ {k} = \sigma (\boldsymbol {W} _ {(i)} \boldsymbol {v} _ {k} + \boldsymbol {U} _ {(i)} \widetilde {\boldsymbol {h}} _ {k} + \boldsymbol {b} _ {(i)}),
$$
$$
\boldsymbol {f} _ {k, j} = \sigma (\boldsymbol {W} _ {(f)} \boldsymbol {v} _ {k} + \boldsymbol {U} _ {(f)} \boldsymbol {h} _ {j} + \boldsymbol {b} _ {(f)}),
$$
$$
\boldsymbol {o} _ {k} = \sigma (\boldsymbol {W} _ {(o)} \boldsymbol {v} _ {k} + \boldsymbol {U} _ {(o)} \tilde {\boldsymbol {h}} _ {k} + \boldsymbol {b} _ {(o)}),
$$
$$
\boldsymbol {u} _ {k} = \tanh  (\boldsymbol {W} ^ {(u)} \boldsymbol {v} _ {k} + \boldsymbol {U} ^ {(u)} \widetilde {\boldsymbol {h}} _ {k} + \boldsymbol {b} ^ {(u)}), \tag {2}
$$
$$
\boldsymbol {c} _ {k} = \boldsymbol {i} _ {k} \odot \boldsymbol {u} _ {k} + \sum_ {j} \boldsymbol {f} _ {k, j} \odot \boldsymbol {c} _ {j},
$$
$$
\boldsymbol {h} _ {k} = \boldsymbol {o} _ {k} \odot \tanh  (\boldsymbol {c} _ {k}),
$$
where  $\pmb{h}_k$  and  $\pmb{c}_k$  represent the hidden state and memory cell state of node  $k$  respectively.
To induce label correlations, HiAGM employs a bidirectional Tree-LSTM by the fusion of a childsum and a top-down module:
$$
\widetilde {\boldsymbol {h}} _ {k} ^ {\uparrow} = \sum_ {j \in c h i l d (k)} f _ {p} (e _ {k, j}) \boldsymbol {h} _ {j} ^ {\uparrow},
$$
$$
\widetilde {\boldsymbol {h}} _ {k} ^ {\downarrow} = f _ {c} \left(e _ {k, p}\right) \boldsymbol {h} _ {p} ^ {\downarrow}, \tag {3}
$$
$$
\boldsymbol {h} _ {k} ^ {b i} = \boldsymbol {h} _ {k} ^ {\uparrow} \oplus \boldsymbol {h} _ {k} ^ {\downarrow},
$$
where  $h_k^\uparrow$  and  $h_k^\downarrow$  are separately calculated in the bottom-up and top-down manner as  $h_k = \mathrm{TreeLSTM}(\widetilde{h_k})$ .  $\oplus$  indicates the concatenation of hidden states. The final hidden state of node  $k$  is the hierarchical node representation  $h_k^{bi}$ .
Hierarchy-GCN GCN (Kipf and Welling, 2017) is proposed to enhance node representations based on the local graph structural information. Some NLP studies have improved Text-GCNs for rich word representations upon the syntactic structure and word correlation(Marcheggiani and Titov, 2017; Vashishth et al., 2019; Yao et al., 2019; Peng et al., 2018). We introduce a simple hierarchy-GCN for the hierarchy structure, thus gaining our aforementioned fine-grained hierarchy information.
Hierarchy-GCN aggregates dataflows within the top-down, bottom-up, and self-loop edges. In the hierarchy graph, each directed edge represents a pair-wise label correlation feature. Thus, those dataflows should conduct node transformations with edge-wise linear transformations. However, edge-wise transformations shall lead to overparameterized edge-wise weight matrixes. Our Hierarchy-GCN simplifies this transformation with a weighted adjacent matrix. This weighted adjacent
matrix represents the hierarchical prior probability. Formally, Hierarchy-GCN encodes the hidden state of node  $k$  based on its associated neighbourhood  $N(k) = \{n_k, child(k), parent(k)\}$  as:
$$
\begin{array}{l} \boldsymbol {u} _ {k, j} = a _ {k, j} \boldsymbol {v} _ {j} + \boldsymbol {b} _ {l} ^ {k}, \\ \boldsymbol {g} _ {k, j} = \sigma \left(\boldsymbol {W} _ {g} ^ {d (j, k)} \boldsymbol {v} _ {k} + \boldsymbol {b} _ {g} ^ {k}\right), \tag {4} \\ \boldsymbol {h} _ {k} = \operatorname {R e L U} \left(\sum_ {j \in N (k)} \boldsymbol {g} _ {k, j} \odot \boldsymbol {u} _ {k, j}\right), \\ \end{array}
$$
where  $\pmb{W}_{g}^{d(k,j)}\in \mathbb{R}^{dim}$ ,  $\pmb{b}_l\in \mathbb{R}^{N\times dim}$ , and  $\pmb{b}_{g}\in \mathbb{R}^{N}$ .  $d(j,k)$  indicates the hierarchical direction from node  $j$  to node  $k$ , including top-down, bottom-up, and self-loop edges. Note that  $a_{k,j}\in \mathbb{R}$  denotes the hierarchy probability  $f_{d(k,j)}(e_{kj})$ , where the self-loop edge employs  $a_{k,k} = 1$ , top-down edges use  $f_{c}(e_{j,k}) = \frac{N_{k}}{N_{j}}$ , and bottom-up edges use  $f_{p}(e_{j,k}) = 1$ . The holistic edge feature matrix  $\pmb{F} = \{a_{0,0},a_{0,1},\dots,a_{C - 1,C - 1}\}$  indicates the weighted adjacent matrix of the directed hierarchy graph. Finally, the output hidden state  $\pmb{h}_k$  of node  $k$  denotes its label representation corresponding to the hierarchy structural information.
4.3 Hybrid Information Aggregation
Previous global models classify labels upon the original textual information and improve the decoder with predefined hierarchy paths. In contrast, we construct a novel end-to-end hierarchy-aware global model (HiAGM) for the mutual interaction of text features and label correlations. It combines a traditional text classification model with a hierarchy encoder, thus obtaining label-wise text features. HiAGM is extended to two variants, a parallel model for an inductive fusion (HiAGM-LA) and a serial model for a deductive fusion (HiAGM-TP).
Given a document  $x = (\pmb{w}_1, \pmb{w}_2, \dots, \pmb{w}_s)$ , the sequence of token embedding is firstly fed into a bidirectional GRU layer to extract text contextual feature. Then, multiple CNNs are used for generating n-gram features. The concatenation of n-gram features is filtered by a top-k max-pooling layer to extract key information. Finally, by reshaping, we can obtain the continuous text representation  $S = (s_1, \ldots, s_n)$  where  $s_i \in \mathbb{R}^{d_c}$  and  $d_c$  indicates the output dimension of the CNN layer.  $n = n_k \times n_c$  refers to the multiplication of top-k number and the number of CNNs.
Hierarchy-Aware Multi-Label Attention The first variant of HiAGM is proposed based on multi-label attention, called as HiAGM-LA. Attention
mechanism is usually utilized as the memory unit in text classification (Yang et al., 2016; Du et al., 2019). Recent LCL studies (Huang et al., 2019; You et al., 2019) construct one multi-label attention-based model per level so as to avoid optimizing label embedding among different levels.
Our HiAGM-LA is similar to those baselines but simplifies multi-label attention LCL models to a global model. Based on our hierarchy encoders, HiAGM-LA could overcome the problem of convergence for label embedding across various levels. Label representations are enhanced with bidirectional hierarchical information. This local structural information makes it feasible to learn label features across different levels in a single model. Formally, suppose that the trainable label embedding of node  $k$  is randomly initialized as  $L_{k} \in \mathbb{R}^{d_{l}}$ . The initial label embedding  $L_{k}$  is directly fed into structure encoders as the input vector of aligned label node  $x_{k}$ . Then, the output hidden state  $h \in \mathbb{R}^{C \times d_{c}}$  represents as the hierarchy-aware label features. Given text representation  $S \in \mathbb{R}^{n \times d_{c}}$ , HiAGM-LA calculates the label-wise attention value  $\alpha_{ki}$  as:
$$
\alpha_ {k j} = \frac {e ^ {\boldsymbol {s} _ {j}} \boldsymbol {h} _ {k} ^ {T}}{\sum_ {j = 1} ^ {n} e ^ {\boldsymbol {s} _ {j}} \boldsymbol {h} _ {k} ^ {T}}, \boldsymbol {v} _ {k} = \sum_ {i = 1} ^ {n} \alpha_ {k i} \boldsymbol {s} _ {i}, \tag {5}
$$
Note that  $\alpha_{ki}$  indicates how informative the  $i$ -th text feature vector is for the  $k$ -th label. We can get the inductive label-aligned text features  $\mathbf{V} \in \mathbb{R}^{C \times d_c}$  based on multi-label attention. Then it would be fed into the classifier for prediction. Furthermore, we could directly use the hidden state of hierarchy encoders as the pretrained label representations so that HiAGM-LA could be even lighter in the inference process.
Hierarchical text feature propagation Graph neural networks are capable of message passing (Gilmer et al., 2017; Duvenaud et al., 2015), learning both local node correlations and overall graph structure. To avoid the noise from heterogeneous fusion, the second variant obtains label-wise text features based on a deductive method. It directly takes text features  $S$  as the node inputs and updates textual information through the hierarchy-aware structure encoder. This variant mainly conducts the propagation of text features, called as HiAGM-TP. Formally, node inputs  $V$  are reshaped from text features by a single linear transformation:
$$
\boldsymbol {V} = \boldsymbol {M} \boldsymbol {S}, \tag {6}
$$
where the trainable weight matrix  $M \in \mathbb{R}^{(n \times d_c) \times (C \times d_v)}$  transforms text features  $S \in \mathbb{R}^{n \times d_c}$  to node inputs  $V \in \mathbb{R}^{C \times d_v}$ .
Given the predefined structure, each sample would update its textual information throughout the same holistic taxonomic hierarchy. In a mini-batch learning manner, the initial node representation  $V$  is fed into the hierarchy encoder. The output hidden state  $h$  denotes deductive hierarchy-aware text features as the input of the final classifier. Compared with HiAGM-LA, the transformation of HiAGM-TP is conducted on textual information without the fusion of label embedding. Thus, the structure encoder would be activated in both training and inference procedures for passing textual messages across the hierarchy. It could converge much easier but has slightly higher computational complexity than HiAGM-LA.
4.4 Classification
We flatten the hierarchy by taking all nodes as leaf nodes for multi-label classification, no matter it is a leaf node or an internal node. The final hierarchy-aware features are fed into a fully connected layer for prediction. HiAGM is complementary with recursive regularization(Gopal and Yang, 2013) as  $L_{r} = \sum_{i\in C}\sum_{j\in child(i)}\frac{1}{2} ||\pmb{w}_{i} - \pmb{w}_{j}||^{2}$  for the parameters of the final fully connected layer. For multi-label classification, HiAGM uses a binary cross-entropy loss function:  $L_{c} = -\sum_{i = 1}^{N}\sum_{j = 1}^{C'}[y_{ij}\log (y_{ij}') + (1 - y_{ij})\log (1 - y_{ij}')]$  where  $y_{ij}$  and  $y_{ij}'$  are the ground truth and sigmoid score for the j-th label of the i-th sample. Thus, the final loss function is  $L_{m} = L_{c} + \lambda \cdot L_{r}$ .