SlowGuess/NAACL_2021_OCR huggingface dataset:
Jiaming Shen, Wenda Qiu, Yu Meng, Jingbo Shang, Xiang Ren, Jiawei Han
$\diamond$  University of Illinois at Urbana-Champaign, IL, USA,  $\nabla$  University of California, San Diego, CA, USA
$\triangle$  University of Southern California, CA, USA
$\diamondsuit$  {js2, qiuwenda, yumeng5, hanj} @illinois.edu  $\nabla$  jshang@ucsd.edu  $\triangle$  xiangren@ucs.edu
Hierarchical multi-label text classification (HMTC) aims to tag each document with a set of classes from a class hierarchy. Most existing HMTC methods train classifiers using massive human-labeled documents, which are often too costly to obtain in real-world applications. In this paper, we explore to conduct HMTC based on only class surface names as supervision signals. We observe that to perform HMTC, human experts typically first pinpoint a few most essential classes for the document as its "core classes", and then check core classes' ancestor classes to ensure the coverage. To mimic human experts, we propose a novel HMTC framework, named TaxoClass. Specifically, TaxoClass (1) calculates document-class similarities using a textual entailment model, (2) identifies a document's core classes and utilizes confident core classes to train a taxonomy-enhanced classifier, and (3) generalizes the classifier via multi-label self-training. Our experiments on two challenging datasets show TaxoClass can achieve around 0.71 Example-F1 using only class names, outperforming the best previous method by  $25\%$ .
Hierarchical multi-label text classification (HMTC) aims to assign each text document to a set of relevant classes from a class taxonomy. As a fundamental task in NLP, HMTC has many applications such as product categorization (Goumy and Mejri, 2018), semantic indexing (Li et al., 2019), and fine-grained entity typing (Xu and Barbosa, 2018).
Most existing methods address HMTC in a supervised fashion — they first ask humans to provide many labeled documents and then train a text classifier for prediction. Many classifiers have been developed with different deep learning architectures such as CNN (Kim, 2014), RNN (You et al., 2019), Attention Network (Huang et al., 2019), and achieved decent performance when trained on massive human-labeled documents. Despite such a
Figure 1: An exemplar document tagged with five classes. Here, if we are able to pinpoint this document's most essential classes, crafted cereal and baby cereal, as core classes, we can check their ancestor classes in the taxonomy and recover all the true classes.
success, people find that applying these methods to many real-world scenarios remains challenging as the human labeling process is often too time-consuming and expensive.
Recently, more studies have been developed to address text classification using smaller amount of labeled data. First, several semi-supervised methods (Gururangan et al., 2019; Berthelot et al., 2019) propose to use abundant unlabeled documents to assist model training on labeled dataset. Although mitigating the human annotation burden, these methods still require a labeled dataset that covers all classes, which could be too expensive to obtain when we have a large number of classes in HMTC. Second, some weakly-supervised models exploit class indicative keywords (Meng et al., 2018; Zeng et al., 2019; Mekala and Shang, 2020) or class surface names (Meng et al., 2020; Wang et al., 2020) to derive pseudo-labeled data for model training. Nevertheless, these models all assume each document has only one class and all class surface names (or class indicative keywords) must appear in the corpus, which are too restrictive for HMTC.
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4239-4249 June 6-11, 2021. ©2021 Association for Computational Linguistics
In this paper, we study the problem of weakly-supervised hierarchical multi-label text classification where only class surface names, a class taxonomy, and an unlabeled corpus are available for model training. This setting is closer to how humans resolve the HMTC problem — we perform classification by understanding each class from its surface name rather than learning from labeled documents. We observe that when asked to assign multiple classes to a document, humans will first pinpoint most essential "core classes" and then check whether their ancestor classes in the taxonomy should also be tagged. Taking the document in Fig. 1 as an example, humans can quickly identify this review text is clearly about "baby cereal" and "crafted cereal", which are the core classes. After assigning these two most essential classes to the document, people continue to check the core classes' ancestor classes and find "feeding" as well as "baby food" should be tagged.
Motivated by the above human labeling process, we propose TaxoClass, a weakly-supervised HMTC framework including four major steps. First, we calculate the document-class similarity using a pre-trained textual entailment model (Yin et al., 2019). Second, we identify each document's core classes by (1) selecting candidate core classes that are most similar to the document at each level in a top-down fashion, and (2) choosing  $\langle$ document, candidate core class\rangle pairs that are salient across the whole unlabeled corpus. Third, we derive training data from document core classes and use them to train a text classifier. This classifier includes a document encoder based on pre-trained BERT (Devlin et al., 2019), a class encoder capturing class taxonomy structure, and a text matching network computing the probability of a document being tagged with each class. Finally, we generalize this text classifier using multi-label self-training on all unlabeled documents.
Contributions. To summarize, our major contributions are as follows: (1) We propose a weakly-supervised framework TaxoClass that only requires class surface names to perform hierarchical multi-label text classification. To the best of our knowledge, TaxoClass is the first weakly-supervised HMTC method. (2) We develop an unsupervised method to identify document core classes based on which a text classifier can be learned. (3) We conduct extensive experiments to verify the effectiveness of TaxoClass on two real-world datasets.
In this section, we introduce the notations and present our task definition.
Notations. A corpus  $\mathcal{D} = \{D_1, \ldots, D_N\}$  is a text collection where each document  $D_i \in \mathcal{D}$  is a sequence of words. A class taxonomy  $\mathcal{T} = (\mathcal{C}, \mathcal{R})$  is a directed acyclic graph where each node represents a class  $c_j$  and each directed edge  $\langle c_m, c_n \rangle \in \mathcal{R}$  indicates that parent class  $c_m$  is more general than the child class  $c_n$ . In this work, we assume each class  $c_j$  has a surface name  $s_j$  (either a word or a phrase) that serves as the weak supervision signal.
Task Definition. Given an unlabeled corpus  $\mathcal{D}$ , a class hierarchy  $\mathcal{T} = (\mathcal{C},\mathcal{R})$ , and class surface names  $\mathcal{S} = \{s_j\}_{j = 1}^{|\mathcal{C}|}$ , our task is to learn a text classifier  $f(\cdot)$  that maps a new document  $D_{new}$  to its target  $\mathbf{y} = [y_1,\dots ,y_{|\mathcal{C}|}]\in \mathcal{V} = \{0,1\}^{|\mathcal{C}|}$  where  $y_{j}$  equals to 1 if this document is categorized with class  $c_{j}$  and 0 otherwise.
Discussion. When the number of classes  $|\mathcal{C}|$  is large (as it is in many HMTC applications), we can no longer assume all class surface names in  $\mathcal{S}$  will explicitly appear in the given corpus  $\mathcal{D}$  as done in most previous studies (Meng et al., 2019; Li et al., 2019; Wang et al., 2020). This is because many class names are actually summarizing phrases provided by humans (e.g., "grocery & gourmet food" in Fig. 1). As a result, we need to design a method that works under such a scenario.
3 Our TaxoClass Framework
Our TaxoClass framework consists of four major steps: (1) document-class similarity calculation, (2) document core class mining, (3) core class guided classifier training, and (4) multi-label self-training. Fig. 2 shows our framework overview and below sections discuss each step in more details.
3.1 Document-Class Similarity Calculation
We take a textual entailment approach (Yin et al., 2019) to calculate the semantic similarity between each  $\langle$ document, class\rangle pair. This approach imitates how humans determine whether a document is similar to a class or not — we read this document, create a hypothesis by filling the class name into a template (e.g., "this document is about _"), and ask ourselves to what extent this hypothesis is correct, given the context document.
In this work, we adopt a pre-trained textual entailment model that inputs a document  $D_{i}$  as the
Figure 2: Our TaxoClass framework overview. We first calculate document-class similarities using a textual entailment model (Sect. 3.1). Then, we identify document core classes (Sect. 3.2) and train a taxonomy-enhanced text classifier (Sect. 3.3). Finally, we generalize the classifier via multi-label self-training (Sect. 3.4). The "shared model parameters" indicates that we do self-training on the same model learned using our identified core classes.
"premise", a template filled with a class name  $s_j$  as the "hypothesis", and outputs a probability of how likely this premise can entail the hypothesis. We treat this probability  $\mathbf{P}(D_i \to c_j)$  as the document-class similarity  $sim(D_i, c_j)$ . More specifically, we use Roberta-Large-MNLI as our textual entailment model which utilizes the pretrained Roberta-Large as its backbone and is fine-tuned on the MNLI dataset.
3.2 Document Core Class Mining
When asked to tag a document with a set of classes from a class taxonomy, humans will first pinpoint a few classes that are most essential to this document. We refer to those most essential classes as the "core classes" and identify them in below two steps.
3.2.1 Core Class Candidate Selection
We observe that on average each document is tagged with a small set of classes from the entire class taxonomy. Therefore, we first reduce the search space of core classes using a top-down approach (c.f. Fig. 3). Given a document  $D$ , we start with the "Root" class at level  $l = 0$ , find its two children classes that have the highest similarity with  $D$ , and add them into a queue. Then, for each class at level  $l$  in the queue, we select  $l + 2$  classes from its children classes that are most similar to  $D$ . After all levels  $l$  classes are processed, we aggregate all selected children classes and choose  $(l + 1)^{2}$  classes (at level  $l + 1$ ) with the highest path score
$(ps)$  defined below:
$$
p s (R o o t) = 1,
$$
$$
p s \left(c _ {j}\right) = \max  _ {c _ {k} \in P a r \left(c _ {j}\right)} \left\{p s \left(c _ {k}\right) \cdot s i m \left(c _ {j}, D\right) \right\}, \tag {1}
$$
where  $\text{Par}(c_j)$  is class  $c_j$ 's parent class set. All chosen classes (at level  $l + 1$ ) will be pushed into the queue and we stop this process when no class in the queue has further children. Finally, all classes that have entered the queue, except for the "Root" class, consist of the core class candidate set. We use  $\mathbb{C}_i^{\text{cand}}$  to denote the candidate core class set of document  $D_i$ .
3.2.2 Confident Core Class Identification
For each document, we identify its core classes from the above selected candidate set based on two observations. First, a document usually has higher similarity with its core class  $c$  than with the parent and sibling classes of  $c$ . Take the document  $D_{2}$  in Fig. 2 as an example, the similarity between  $D_{2}$  and its core class "crib" is 0.95, much higher than the similarity between  $D_{2}$  and core class's parent class "nursery" (0.6) as well as core class's sibling classes. Based on this observation, we define the "confidence score" of a candidate core class  $c$  for a document  $D$  as below:
$$
\operatorname {c o n f} (D, c) = \operatorname {s i m} (D, c) - \max  _ {c ^ {\prime} \in \operatorname {P a r} (c) \cup S i b (c)} \left\{\operatorname {s i m} (D, c ^ {\prime}) \right\}, \tag {2}
$$
where  $Sib(c)$  represents the sibling class set of  $c$ .
Our second observation is that the similarity between a document  $D$  and its core class  $c$  is salient from a corpus-wise perspective. Namely, if a class  $c$  is a document  $D$ 's core class, the confidence score
'https://huggingface.co/ roberta-large-mnli
Figure 3: Top-down core class candidate selection.
$conf(D, c)$  is higher than the median confidence score between class  $c$  and all documents tagged with  $c$  (denoted as  $\mathcal{D}(c)$ ). Formally, we have:
$$
\operatorname {c o n f} (D, c) \geq \operatorname {m e d i a n} \left\{\operatorname {c o n f} \left(D ^ {\prime}, c\right) \mid D ^ {\prime} \in \mathcal {D} (c) \right\}. \tag {3}
$$
According to this observation, we check each class in document  $D_{i}$ 's candidate core set  $\mathbb{C}_i^{cand}$  and add classes that satisfy the above criteria into the final core class set  $\mathbb{C}_i$ . Note here this core class set  $\mathbb{C}_i$  could be empty when document  $D_{i}$  does not have any confident core class.
3.3 Core Class Guided Classifier Training
Based on identified document core classes, we train one classifier for hierarchical multi-label text classification. Below we first introduce our classifier architecture and then present our training method.
3.3.1 Text Classifier Architecture
We design our classifier to have a dual-encoder architecture: one document encoder maps document  $D_{i}$  to its representation  $\mathbf{D}_i$ , one class encoder learns class  $c_{j}$ 's representation  $\mathbf{c}_{j}$ , and one matching network returns the probability of document  $D_{i}$  being tagged with class  $c_{j}$ .
Document Encoder. In this work, we instantiate our document encoder  $g_{\mathrm{doc}}(\cdot)$  to be a pretrained BERT-base-uncased model (Devlin et al., 2019) and follow previous work (Chang et al., 2019; Meng et al., 2020) to use the [CLS] token representation as the document representation.
Class Encoder. For class encoder  $g_{\mathrm{class}}(\cdot)$ , we follow (Shen et al., 2020) and use a graph neural network (GNN) (Kipf and Welling, 2017) to model the class taxonomy structure. This taxonomy-enhanced class encoder can capture both the textual information from class surface names and structural information from the class taxonomy.
Given a class  $c_{j}$ , we first obtain its ego network that includes its parent and children classes in the class taxonomy, as shown in Fig. 4. Then, we input this ego network to a GNN that propagates
Figure 4: Taxonomy-enhanced class encoder.
node features over the network structure. The node features are initialized with the pre-trained word embeddings of class surface names<sup>3</sup>. The propagation mechanism updates the feature of a node  $u$  by iteratively aggregating representations of its neighbors and itself. Formally, we define a GNN with  $L$ -layers as follows:
$$
h _ {u} ^ {(l)} = \operatorname {R e L U} \left(\sum_ {v \in N (u)} \alpha_ {u v} ^ {(l - 1)} \mathbf {W} ^ {(l - 1)} h _ {v} ^ {(l - 1)}\right), \tag {4}
$$
where  $l \in \{1, \dots, L\}$ ,  $N(u)$  includes node  $u$ 's neighbors and itself,  $\alpha_{uv}^{(l-1)} = \frac{1}{\sqrt{|N(u)||N(v)|}}$  is a normalization constant (same for all layers), and  $\mathbf{W}^{(l-1)}$  are learnable parameters.
After obtaining individual node features, we combine them into a vector representing the whole ego network  $G$  as follows:
$$
h _ {G} = \frac {1}{| G |} \sum_ {u \in G} h _ {u} ^ {(L)}. \tag {5}
$$
As this ego network is centered on class  $c_{j}$  and encodes its both textual and structural information, we treat this final graph representation as the class representation  $\mathbf{c}_j$ .
Text Matching Network. Based on the document representation  $\mathbf{D}_i$  and the class representation  $\mathbf{c}_j$ , we use a log-bilinear text matching model to compute the probability of document  $D_i$  being tagged with class  $c_j$  as follows:
$$
p _ {i j} = \mathbf {P} \left(y _ {j} = 1 \mid D _ {i}\right) = \sigma \left(\exp \left(\mathbf {c} _ {j} ^ {T} \mathbf {B} \mathbf {D} _ {i}\right)\right), \tag {6}
$$
where  $\sigma (\cdot)$  is the sigmoid function and  $\mathbf{B}$  is a learnable interaction matrix.
3.3.2 Text Classifier Training
We use our discovered document confident core classes to train a text classifier. One intuitive strategy is to treat each document's core classes as positive classes and all the remaining classes as negative classes. However, this strategy has a high false
2We have also tried using "average" but empirically found that using "median" is better and more robust to outliers.
3For multi-gram class names, we use their averaged word embeddings.
Algorithm 1: TaxoClass Framework.
Input: An unlabeled corpus  $\mathcal{D}$ , a class taxonomy  $\mathcal{T}$  with class names  $\mathcal{S}$ , an entailment model  $\mathcal{M}$ , total number of batches  $B$ .
Output: A trained classifier  $f(\cdot)$ .
1 Use model  $\mathcal{M}$  to compute document-class similarity (c.f. Sect. 3.1);
2 Obtain document core classes  $\{(D_i,\mathbb{C}_i)\mid D_i\in \mathcal{D}\}$  (c.f. Sect. 3.2);
3 Train classifier  $f(\cdot)$  with Eq. (8);
4 for  $i$  from 1 to  $B$  do
5 if  $i\bmod 25 = 0$  then   
6 Update  $Q$  with Eq. (10);   
7 Train classifier  $f(\cdot)$  with Eq. (9);
8 Return  $f(\cdot)$ ;
negative rate because some non-core classes could still be relevant to the document (c.f. Fig. 1).
We observe a document's multiple labeled classes usually have some ancestor-descendent relations in the class hierarchy  $\mathcal{T} = (\mathcal{C},\mathcal{R})$ . This implies that given a document's core class, its parent class and some of its children classes are also likely to be tagged with this document. Therefore, we introduce all core classes' parent classes into the positive class set and exclude their children classes from the negative class set. Formally, given a document  $D_{i}$  with its core class set  $\mathbb{C}_i$ , we define its positive and negative class set as follows:
$$
\mathbb {C} _ {i} ^ {p o s} = \left(\bigcup_ {c _ {j} \in \mathbb {C} _ {i}} P a r \left(c _ {j}\right)\right) \cup \mathbb {C} _ {i}, \tag {7}
$$
$$
\mathbb {C} _ {i} ^ {n e g} = \mathcal {C} - \mathbb {C} _ {i} ^ {p o s} - \bigcup_ {c _ {j} \in \mathbb {C} _ {i}} C h d (c _ {j}),
$$
where  $Chd(c_{j})$  is class  $c_{j}$ 's children class set. Finally, we train our classification model using the below binary cross entropy (BCE) loss:
$$
\mathcal {L} = - \sum_ {\substack {i = 1 \\ \mathbb {C} _ {i} \neq \emptyset}} ^ {| \mathcal {D} |} \left(\sum_ {c _ {j} \in \mathbb {C} _ {i} ^ {p o s}} \log p _ {i j} + \sum_ {c _ {j} \in \mathbb {C} _ {i} ^ {n e g}} \log (1 - p _ {i j})\right), \tag{8}
$$
where “ $\emptyset$ ” indicates an empty set and we exclude the documents without any confident core class from the loss calculation.
3.4 Multi-label Self-Training
After training the text classifier based on document core classes, we propose to further refine the model via self-training on the entire unlabeled corpus  $\mathcal{D}$  for better generalization. The idea of self-training (ST) (Xie et al., 2016) is to iteratively use the model's current prediction  $P$  to compute a
Dataset	# Train	# Test	# Classes
Amazon-531	29,487	19,685	531
DBPedia-298	196,665	49,167	298
Table 1: Dataset statistics. Supervised methods are trained on the entire training set. Weakly-supervised methods are trained by treating the training set as unlabeled data. All methods are evaluated on the test set.
target distribution  $Q$  which guides the model for refinement. In general, the ST objective is expressed with the KL divergence loss as below:
$$
\mathcal {L} _ {S T} = \mathrm {K L} (Q | | P) = \sum_ {i = 1} ^ {| \mathcal {D} |} \sum_ {j = 1} ^ {| \mathcal {C} |} q _ {i j} \log \frac {q _ {i j}}{p _ {i j}}. \tag {9}
$$
The target distribution  $Q$  is constructed by enhancing high-confidence predictions while downweighting low-confidence ones:
$$
q _ {i j} = \frac {p _ {i j} ^ {2} / \left(\sum_ {i} p _ {i j}\right)}{p _ {i j} ^ {2} / \left(\sum_ {i} p _ {i j}\right) + \left(1 - p _ {i j}\right) ^ {2} / \left(\sum_ {i} \left(1 - p _ {i j}\right)\right)}. \tag {10}
$$
Different from the previous studies (Meng et al., 2018; Yu et al., 2020), our target distribution  $Q$  can be applied to multi-label classification problem as it normalizes the current predictions  $P$  for each individual class. Intuitively, this equation can enhance high-confidence predictions while downweighting low-confidence predictions. This is because if example  $i$  is more confidently labeled with class  $j$  than other examples, we will have a large  $p_{ij}$  that dominates the  $\sum_{i} p_{ij}$  term. Consequently, Eq 10 computes a large  $q_{ij}$ , which further pushes the model to predict class  $j$  for example  $i$ .
In practice, instead of updating the target distribution  $Q$  for every training example, we update it every 25 batches and train the model with Eq. (9), which makes the self-training process more efficient and robust. We summarize our TaxoClass framework in Algorithm 1.
We use two public datasets from different domains to evaluate our method: (1) Amazon-531 (McAuley and Leskovec, 2013) contains 49,145 product reviews and a three-level class taxonomy consisting of 531 classes; and (2) DBPedia-298 (Lehmann et al., 2015) includes 245,832
4This hyper-parameter controls the update frequency. Empirically, we find our model is insensitive to this hyperparameter (in the typical value range of 10-100).
Wikipedia articles and a three-level class taxonomy with 298 classes. Documents in both datasets are lower-cased and truncated to has maximum 500 tokens. We list the data statistics in Table 1.
To the best of our knowledge, we are the first to study weakly-supervised HMTC problem and there is no directly comparable baseline under the exact same setting as ours. Therefore, we choose a wide range of representative methods that are most related to TaxoClass and adapt them to our problem setting, described as follows.
- Hier-doc2vec (Le and Mikolov, 2014): This weakly-supervised method first embeds documents and classes into a shared semantic space, and then recursively selects the class of the highest embedding similarity with the document in a top-down fashion. We set the embedding dimensionality to be 100 and use the default value for all other hyper-parameters.
- WeSHClass (Meng et al., 2019)<sup>7</sup>: Another weakly-supervised method that generates pseudo documents to pre-train a text classifier and bootstraps the pre-trained classifier on unlabeled documents with self-training. The class surface names are treated as the "class-related keywords" in this method. For the pseudo document generation step, we use its internal LSTM language model. We treat all classes in its returned class path as the output classes.
- SS-PCEM (Xiao et al., 2019)<sup>8</sup>: This semisupervised method uses a generative model to generate documents based on a class path sampled from the class taxonomy. Both labeled and unlabeled documents are used to fit this generative model via the EM algorithm. Finally, it uses the posterior probability of a test document to predict its labeled classes. Among different base classifiers, we choose their author reported best variant PCEM in this study. We use  $30\%$  of labeled training documents for this method.