Title: END-TO-END HIERARCHICAL TEXT CLASSIFICATION WITH LABEL ASSIGNMENT POLICY

Abstract: We present an end-to-end reinforcement learning approach to hierarchical text classification where documents are labeled by placing them at the right positions in a given hierarchy. While existing "global" methods construct hierarchical losses for model training, they either make "local" decisions at each hierarchy node or ignore the hierarchy structure during inference. To close the gap between training/inference and capture label dependencies in an end-to-end manner, we propose to learn a label assignment policy to determine where to place the documents and when to stop. The proposed method, HiLAP, directly optimizes metrics over the hierarchy, makes inter-dependent decisions during inference, and can be combined with different text encoding models for end-to-end training. Experiments on three public datasets show that HiLAP yields an average improvement of 33.4% in Macro-F1 and 5.0% in Samples-F1, outperforming state-of-the-art methods by a large margin. 1

1. INTRODUCTION:

In recent years there has been a surge of interest in leveraging taxonomies and hierarchies to organize and classify text documents, leading to the development of hierarchical text classification (HTC) methods-methods that can predict for a document multiple appropriate labels (which together constitute a sub-tree) in a given hierarchy. These methods have found a wide range of applications such as question answering (Qu et al., 2012) , online advertising (Agrawal et al., 2013) , and scientific literature organization (Peng et al., 2016) . In contrast to traditional "flat" classification, the key challenge of HTC lies in modeling the inter-dependent, large-scale, and imbalanced label space.

Due to the complexity of HTC, how to better utilize the label hierarchy remains an open problem. HTC methods are traditionally divided into three categories, namely flat, local, and global approaches (Silla & Freitas, 2011) . Flat approaches generally ignore the label hierarchy. Some only predict labels at the leaf nodes and then add all the ancestors of the predicted leaf nodes. Others ignore the hierarchy and perform standard multi-label classification, in which inconsistencies (i.e., one label is predicted but its ancestors are not) may occur and post-processing is thus needed. Local approaches train a set of local classifiers per node/per parent node/per level, which function independently and (usually) make predictions in a top-down order to avoid inconsistencies. Traditional global approaches (Cai & Hofmann, 2004; Vens et al., 2008; Silla Jr & Freitas, 2009) are largely modified based on specific flat models and rely on static, human curated features as input. In addition, many existing global approaches make unrealistic assumptions of the problem as in flat approaches. For example, Hierarchical-SVM (Cai & Hofmann, 2004) requires that all possible labels are on the leaf nodes and the heights of leaf nodes are the same.

Recent approaches (Kim, 2014; Lai et al., 2015; Yang et al., 2016) to text classification mainly focus on flat classification and have been shown to be very effective. However, their performance in HTC is relatively less studied. Even if the classification task is essentially hierarchical, prior work (Gopal & Yang, 2013; Johnson & Zhang, 2014; Peng et al., 2016; 2018) still makes flat and independent predictions and utilizes intuitive constraints, such as the embeddings of one label and its parent should be close. One recent framework (Wehrmann et al., 2018) attempted to leverage both local and global information. However, it uses static features and its inference is essentially flat, which may lead to inconsistencies. STOP is taken t = 0 t = 1 t = 2 t = 6

Figure 1: An illustrative example showing our label assignment policy. At t = 0, the document x i is placed at the root label and the policy would decide if x i should be placed to its two adjacent (denoted by purple) labels. At t = 1, x i is placed at label 1, which adds another three adjacent labels as the candidates. At t = 6, the stop action is taken and the label assignment process is thus terminated. We then take all the labels where x i has been placed (a sub-tree consisting of label 0, 1, ..., 5) as x i 's document labels.

In this paper, we present an end-to-end reinforcement learning approach to HTC where documents are labeled by placing them at the right positions in a label hierarchy. We propose HiLAP, a principled global framework that learns a label assignment policy to determine where to place the documents and when to stop. HiLAP optimizes its policy by exploring the label hierarchy, in which training and inference follow the same routine and inter-dependent decisions are made. Compared to flat and local approaches, HiLAP achieves better effectiveness because it examines the global hierarchical structure during both training and inference phases. Compared to most existing global approaches, HiLAP has more flexibility in that it has no constraints on the structure of the hierarchy. Furthermore, the label assignment policy of HiLAP ensures that its predictions are always consistent and no post-processing is needed.

HiLAP can be combined with different text representation learning models and trained in an endto-end fashion. We select three representative text encoding models as the base models to evaluate the effectiveness of HiLAP. Experiments on three public datasets from different domains show that combining HiLAP with existing representation learning models yields an average performance improvement of 33.4% in Macro-F1 and 5.0% in Samples-F1, outperforming state-of-the-art HTC methods by a large margin. In particular, ablation study shows that HiLAP is especially beneficial to those unpopular labels at the bottom levels.

2. LEARNING LABEL ASSIGNMENT POLICY FOR END-TO-END HIERARCHICAL TEXT CLASSIFICATION:

This section presents the proposed end-to-end reinforcement learning approach to hierarchical text classification. We first introduce our label assignment policy including the design of its actions and rewards, and then describe the details of policy learning.

2.1. HIERARCHICAL LABEL ASSIGNMENT:

We define a label hierarchy H = (L, E) as a tree or DAG (directed acyclic graph)-structured hierarchy with node set L (i.e., the labels), and edge set E (which indicates the parent-child relationship between labels). Taking a set of documents {x 1 , x 2 , ..., x N } and their document labels {L 1 , L 2 , ..., L N } ∈ L as input, we aim to learn a policy P to place each documents x i to its labels L i on the label hierarchy H. Specifically, the policy P puts x i at the root label in the beginning and at each time step, decides which label x i should be further placed to, among all the adjacent labels of where x i has been placed, until a special stop action is taken. An illustration of our label assignment policy is shown in Figure 1 . We define one base model B as a mapping f that converts each raw document x i to a finite dimensional vector as its representation, i.e., the document embedding e d ∈ R D (D denotes the embedding size). B can be any neural text representation learning model and its output e d is used as the input of the policy P. The challenge, compared to standard classification setup, is that we need to model E, i.e., the relationship between labels. 

2.2. REINFORCEMENT LEARNING FOR LABEL ASSIGNMENT:

To learn the label assignment policy, we train a policy network to determine where to place the documents and when to stop as follows.

Actions We regard the process of placing a document x i to the right positions on the label hierarchy as making a sequence of actions. Specifically, we define that an action a t at time step t is to select one label l t from the action space A t and place x i to that label l t . We denote the children of label l t as C(l t ). The action space A t consists of all the adjacent labels of where the document x i has been placed. In this way, for example, HiLAP can first place x i to a label at level 3 if the confidence (probability) of that label is higher and then place it to another label at level 1.

At the beginning of each episode, x i is placed at the root label l 0 and the action space A t = C(l 0 ), i.e., all the labels at level 1. When x i is placed at another label l 1 , its children C(l 1 ) would then be added to the action space A t . In addition, one stop action e stop ∈ R C (C denotes the embedding size) is added to the action space A t so that the model can learn when to stop placing document x i to new labels. Intuitively, when the confidence of placing x i to another label is lower than the stop action, the label assignment process would be terminated. In short, the size of A t is |{C(l 0 ), C(l 1 ), ..., C(l t ), stop} -{l 1 , l 2 , ..., l t }|. Note that in other local/global approaches the predictions on different paths are independent while in HiLAP the inter-dependencies of labels across different paths and levels are considered and we optimize the metrics over the hierarchy by providing the policy network with rewards that capture the overall quality of its label assignment.

Rewards HiLAP receives rewards from the environment as feedback for its actions. Different from existing work where each label of one sample is treated independently, HiLAP captures the quality of all the labels assigned to each sample x i by rewarding the agent with the per-sample F1 (see Section 3.1 for details):

F1 xi = 2×P x i ×R x i P x i +R x i

, where P xi and R xi denote per-sample precision and recall respectively that compare the predicted labels with gold labels of x i .

Instead of waiting until the end of the label assignment process and comparing the predicted labels with the gold labels, we use reward shaping (Mao et al., 2018) , i.e., giving intermediate rewards at each time step, to accelerate the learning process. Specifically, we set the reward r of x i at time step t to be the difference of its F1 score between current and last time step:

r xi t = F1 xi t -F1 xi t-1 . If current F1

is better than that at last time step, the reward would be positive, and vice versa. The cumulative reward from current time step to the end of an episode would cancel the intermediate rewards and thus reflect whether current action improves the overall label assignment of one sample or not. As a result, the learned policy would not focus on the current placement but have a long-term view that takes following actions into account.

Policy Network:

We parameterize each action a t by a policy network π(a | s; W). The architecture of HiLAP is shown in Figure 2 . For each document, its representation e d is generated by the base model B. For each label, a label embedding l ∈ R C is randomly initialized and updated during training. To model the label relationship E, the embeddings of the document e d and current label l t are concatenated and projected to a vector s t ∈ R C via a two-layer feed-forward network. s t has the same size as the label embedding l and we use s t as the state representation of the document at current label l t . By stacking the action embeddings, we can obtain an action matrix A t with size |{C(l 0 ), C(l 1 ), ..., C(l t ), stop} -{l 1 , l 2 , ..., l t }| × C. A t is multiplied with the state embedding s t , which outputs the probability distribution of actions. Finally, an action a t is sampled based on the probability distribution of the action space:

s t = ReLU(W 1 l ReLU(W 2 l [e d ; l t ])), π(a | s; W) = softmax(A t s t ), a t ∼ π(a | s; W).

We use REINFORCE (Williams, 1992) , one instance of the policy gradient methods, as the optimization algorithm. To reduce variance, 10 rollouts for each training sample are run and the rewards are averaged. In addition, we adopt a self-critical training approach (Rennie et al., 2017). For each document x i , two label assignments are generated: Lxi is sampled from the probability distribution, and Lxi , the baseline label assignment, is greedily generated by choosing the action with the highest probability at each time step. We use r( Lxi ) -r( Lxi ) as the actual reward, which ensures that the policy network learns to place the document to positions with higher F1 score than the greedy baseline. At the time of inference, we greedily select labels with the highest probability as Lxi .

2.3. TOP-DOWN SUPERVISED PRE-TRAINING:

It is known that reinforcement learning models often suffer from high variance during training. Instead of learning from scratch, we use supervised learning to pre-train our framework. We denote the supervised variant as HiLAP-SL. While most parameters of HiLAP-SL are shared with HiLAP, its action space and way of exploring of the label hierarchy H are dissimilar.

The main difference is that HiLAP-SL explores the label hierarchy H in a top-down manner. At each time step t, the document goes down one level on the hierarchy. HiLAP-SL concentrates on the local discrimination of labels with the same parent. The local per-parent label probability distribution p SL t is generated as follows.

p SL t = σ(C t s t ), where σ denotes the sigmoid function, and C t ∈ R |C(lt)|×C denotes the action space of HiLAP-SL, i.e., an embedding matrix consisting of the children of current label l t (rather than all the labels where x i has been placed as in HiLAP).

Another difference is that in HiLAP the actions are sampled and thus the documents might be placed to wrong labels, while in HiLAP-SL only the ground-truth positions are traversed during training. Specifically, if there are K(≥ 1) ground-truth labels at the same level, the document embedding e d would be cloned K times following each label and K different paths would be generated independently. The loss function of HiLAP-SL is defined as follows.

L l = T t=0 L t ,

where T is the lowest label's height of one sample (T may vary by samples) and L t estimates the binary cross entropy over the candidate labels C(l t ) at each time step t. HiLAP-SL works as if there were a set of local classifiers, although most of its parameters (except for the label embeddings l t ) are shared by all the labels so that one does not need to actually train a set of classifiers. During inference, HiLAP-SL follows the same top-down routine as in training using the per-parent label probability p SL t and thus no post-processing is needed for inconsistency correction. All the parameters of HiLAP are shared with HiLAP-SL and can be initialized by the pre-trained HiLAP-SL model except for the embedding of the stop action e stop (which is randomly initialized).

2.4. COMBINING FLAT, LOCAL, AND GLOBAL INFORMATION FOR POLICY LEARNING:

We further add a flat component to our framework as a regularization of the base model. Specifically, the flat component is a simple feed-forward network consisting of a fully connected layer and the sigmoid function. It projects the document embedding e d to a label probability distribution p f of all the labels L on the hierarchy.

p f = σ(W f e d ).

The combination of the base model and the flat component is exactly the same as a flat model and ensures that the document representation e d learned by the base model B has the capability of flat classification among all the labels L. We use a flat loss L f to measure the binary cross entropy over all the labels as in the flat models. Combining the flat loss with local loss, the supervised loss in HiLAP-SL is defined as

L SL = λL f + (1 -λ) T t=0 L t ,

where λ ∈ [0, 1] is the mixed ratio. Similar to Celikyilmaz et al. (2018) , we also found that mixing a proportion of the supervised loss is beneficial to the learning process of HiLAP. Further combining the global information, the total loss of HiLAP is defined as L mixed = L RL + αL SL , where α is a scaling factor accounting for the difference in magnitude between L RL and L SL . While we do not directly use the flat component during inference, it helps the learning process of the base model and improves the performance of both HiLAP-SL and HiLAP, which we will show in Section 3.5.

3. EXPERIMENTS:

We evaluate the benefits of our framework against a number of state-of-the-art HTC approaches, with the goal of answering the following questions:

Q1 How does our proposed method (HiLAP) compare to state-of-the-art HTC approaches?

Q2 How does HiLAP compare to other hierarchical classification frameworks when the same base models are adopted by all the frameworks? Q3 How do different components in HiLAP contribute to its performance in terms of the popular(sparse) labels?

3.1. DATASETS AND EVALUATION METRICS:

We conduct experiments on three public and commonly used datasets from different domains. The first two datasets are related to news categorization, including RCV1 (Lewis et al., 2004) and the New York Times (NYT) annotated corpus (Sandhaus, 2008) . We follow the original training/test split for RCV1 and sub-sample NYT due to its large size. The third dataset is the Yelp Dataset Challenge 20182 , which consists of a subset of Yelp businesses and their reviews. We use the Yelp Business Categories3 as the label hierarchy and predict the categories of one business using its reviews. For each dataset, there may be more than one label at each level and the lowest labels of a sample may not be at the leaf nodes. A summary of the datasets is shown in Table 1 and further details can be found in Appendix A. We use standard metrics (Johnson & Zhang, 2014; Peng et al., 2016) for HTC including Micro-F1, Macro-F1, and Samples-F14 . Micro-F1 measures the overall precision/recall and favors labels with more samples. Macro-F1 calculates the F1 scores of all the labels and performs an unweighted average over them. Similarly, Samples-F1 calculates the F1 scores of all the samples and averages them ( i F1 x i

N:

). Recall that F1 xi is used as the reward in HiLAP.

3.2. BASE MODELS:

In our experiments, three representative text classification models with different characteristics are selected as the base models to prove the robustness and versatility of HiLAP. As one will see, HiLAP consistently improves the base model through better exploration of the label hierarchies.

TextCNN (Kim, 2014) is the classic convolutional neural network for text classification. In our implementation, TextCNN is composed of one convolutional layer with three kernels of different sizes (3, 4, 5), followed by max pooling, a dropout layer, and a fully-connected layer. We chose TextCNN because it was one of the first successful and well used neural-based models for text classification.

HAN (Yang et al., 2016) first learns the representation of sentences by feeding words in each sentence to a GRU-based sequence encoder (Bahdanau et al., 2014) and then feeds the representation of the encoded sentences into another GRU-based sequence encoder, which generates the representation of the whole document. Attention mechanism such as word attention and sentence attention is also used. We chose HAN because it uses RNNs instead of CNNs and is shown to be effective on the Yelp Review datasets (Zhang et al., 2015) .

bow-CNN (Johnson & Zhang, 2014) employs bag of words (multi-hot zero-one vectors) as input and directly applies CNN to high-dimensional text data. It learns the representation of small text regions (rather than single words) for use in classification. We chose bow-CNN since it does not use any word embeddings as in TextCNN and HAN. In addition, bow-CNN achieved the state-of-the-art performance on the RCV1 dataset (Lewis et al., 2004) .

3.3. BASELINES:

We compare our framework with state-of-the-art HTC methods. The traditional methods that we compare with are Support Vector Machines (SVM) and its hierarchical variants. Specifically, SVM performs standard multi-label classification using one-vs-the-rest (OvR) strategy. Leaf-SVM treats each leaf node as a label and adds the ancestors of predicted leaf nodes. Other variants include HSVM (Tsochantaridis et al., 2005) , Top-Down SVM (TD-SVM) (Liu et al., 2005) , and Hierarchically Regularized Support Vector Machines (HR-SVM) (Gopal & Yang, 2013) . The neural-based methods that we compare with include HLSTM (Chen et al., 2016) , HR-DGCNN (Peng et al., 2018) and HMCN (Wehrmann et al., 2018) . We also compare with the base models, namely, TextCNN (Kim, 2014) , HAN (Yang et al., 2016) , and bow-CNN (Johnson & Zhang, 2014) to see how much we could improve upon them via better exploration of the hierarchy.

3.4. IMPLEMENTATION DETAILS:

For each base model, we follow its original implementation and fix the hyper-parameters for different datasets. We randomly sample 10% from the training set as development set (Johnson & Zhang, 2014; Peng et al., 2018) . We set batch size to 32 and use the first 256 tokens of each document for representation learning. We use a constant threshold (0.5) for all the labels. All the models are trained using an Adam optimizer with initial learning rate 1e-3 and weight decay 1e-6. We use pretrained GloVe word vectors (Pennington et al., 2014) with dimensionality 50 as word embeddings for TextCNN and HAN. We limit the vocabulary to the most frequent 30000 words in the training data and generate multi-hot vectors as the input of bow-CNN. For our framework, we set the size of W 2 l to 500 and the sizes of W 1 l and label embedding l t to 50.

3.5. EXPERIMENTAL RESULTS:



Main Results:

Table 2 and 3 compare the performance of HiLAP to the state-of-the-art HTC baselines. These results provide positive answers to our question Q1: On the RCV1 dataset, HiLAP (HAN) achieves similar performance to HR-DGCNN even though the original flat HAN is worse than HR-DGCNN. Our HiLAP (TextCNN) outperforms most baselines in Macro-F1 and HiLAP (bow-CNN) achieves the best performance on all the three metrics. 5 On the NYT dataset, similar results are observed: TextCNN and HAN are both improved when combining with HiLAP and HiLAP (bow-CNN) again achieves the best performance. On the Yelp dataset, HiLAP (HAN) achieves the best Micro-F1 and Samples-F1, while HiLAP (bow-CNN) obtains the highest Macro-F1. Interestingly, a simple SVM outperforms several neural-based models, indicating that traditional feature-based methods still play an important role in HTC. Performance Comparison using the Same Text Encoding Model To answer Q2 and Q3, we compare different frameworks that support the use of exactly the same base model and Figure 3 summarizes the comparison results. 6 As one may notice, due to the extreme imbalance of the datasets, directly applying a flat multi-label classification model may suffer from low Macro-F1, i.e., the predictions of flat models are inevitably biased to the most popular labels. HMCN also has the same issue, resulting in Macro-F1 scores lower than 10 when combining with some base models. In contrast, HiLAP significantly outperforms the baselines especially in Macro-F1, which implies that our policy network is better at tackling labels with relatively few samples. On the downside, it is also observed that HiLAP-SL may have a negative effect in terms of Micro-F1 (although it is usually marginal compared with the gain in Macro-F1). However, such negative effects are eliminated by HiLAP through better exploration of the label hierarchy H. Overall, HiLAP obtains the highest performance on 24/27 results across three datasets, three base models, and three evaluation metrics. achieves close performance to Flat Component Only on Micro-F1 and Samples-F1 and even higher Macro-F1 than HiLAP-SL-NoFlat. HiLAP-NoSL is initialized by the pre-trained HiLAP-SL model without mixing the supervised loss during its training. We can see that using the reinforced loss alone still improves the performance on all the three metrics. After removing the flat loss during the training of HiLAP, HiLAP-NoFlat shows slightly lower performance than the full HiLAP model, indicating that the flat component serves as a regularization of the base model and is beneficial to the overall performance.

We also analyze the source of the performance gains by dividing the labels based on their levels and number of supporting samples. Figure 4 shows one comparison of absolute Macro-F1 differences between various methods and the base model. We observe similar trends for other setups/metrics and omit them for better view. As depicted in Figure 4 , HiLAP and HiLAP-SL are especially beneficial to those unpopular labels at the bottom levels.

4. RELATED WORK:

Hierarchical text classification and general hierarchical classification approaches have been developed for many applications. In the biomedical domain, medical subject headings (MeSH) indexing, which is to assign a set of MeSH main headings to citations, has been studied for years (Liu et al., 2015; Peng et al., 2016) . In addition, there are plenty of methods focusing on the hierarchical prediction of protein and gene functions (Clare & King, 2003; Silla Jr & Freitas, 2009; Secker et al., (Sandhaus, 2008) dataset. We show the per-level gains on the left, in which L1, L2, and L3 denote the levels of the hierarchy. We divide the labels into three equal sized categories, namely P1, P2, and P3, in a descending order based on their number of samples, and show their gains on the right. Cerri et al., 2016) . Another line of work concentrates on document categorization. Both traditional methods (Lewis et al., 2004; Gopal & Yang, 2013) and neural methods (Johnson & Zhang, 2014; Peng et al., 2018) have been proposed to classify the topics of newswire and web content (Dumais & Chen, 2000; Sun & Lim, 2001) , categories of laws and patents (Bi & Kwok, 2015; Cai & Hofmann, 2004; Rousu et al., 2005) .

Many previous works (Liu et al., 2005; Xue et al., 2008; Sun & Lim, 2001 ) train a set of local classifiers and make predictions in a top-down manner. In particular, (Bi & Kwok, 2015) develop Bayes-optimal predictions that minimize the global risks with the trained model but the model is still locally trained. Such local approaches are not popularly used among recent neural-based hierarchical classification models (Johnson & Zhang, 2014; 2016; Peng et al., 2018) since it is infeasible to train many neural classifiers locally.

Global methods, on the other hand, train only one classifier. Although global methods are desirable, they are relatively less studied due to the complexity of the problem. Existing global models are generally modified based on specific flat models. Hierarchical-SVM (Cai & Hofmann, 2004; 2007; Qiu et al., 2009) generalizes Support Vector Machine (SVM) learning based on discriminant functions that are structured in a way that mirrors the label hierarchy. One limitation is that Hierarchical-SVM only supports balanced tree (all possible labels are presumed to be at the same height in their experiments). Hierarchical naive Bayes (Silla Jr & Freitas, 2009) modifies naive Bayes by updating weights of one's ancestors as well whenever one label's weights are updated. There are other global methods that are based on association rules (Wang et al., 2001) , C4.5 (Clare & King, 2003) , kernel machines (Rousu et al., 2005) , and decision tree (Vens et al., 2008) . Constraints such as the regularization that enforces the parameters of one node and its parent to be similar (Gopal & Yang, 2013; 2015) are also proposed to leverage the label hierarchy while maintaining scalability. However, their use of hierarchies is somewhat limited.

In addition to the most relevant prior studies mentioned above, dataless classification (Ha-Thuc & Renders, 2011; Song & Roth, 2014; Meng et al., 2018) leverages the labels in the hierarchy as weak supervision and requires almost no training data. Yu et al. (2014) ; Bhatia et al. (2015) scale to hundreds of thousands of labels but do not assume and leverage the existence of label hierarchies.

5. CONCLUSION:

We proposed an end-to-end reinforcement learning approach to hierarchical text classification where documents are labeled by placing them at the right positions in the label hierarchy. The proposed framework makes consistent and inter-dependent predictions, in which any neural-based representation learning model can be used as a base model and a label assignment policy is learned to determine where to place the documents and when to stop. Experiments on three public datasets of different domains showed that our approach outperforms state-of-the-art hierarchical text classification methods significantly. In the future, we will explore the effectiveness of the proposed framework on other base models and forms of data (e.g., images). We also plan to mix more losses covering other aspects in the objective function and test whether they could further improve the performance of our framework.

:

Code and data will be released on GitHub upon acceptance.

https://www.yelp.com/dataset/challenge

https://www.yelp.com/developers/documentation/v3/all category list

We take the name Samples-F1 from sklearn.metrics.f1 score. It is referred to as Example-based F1 inPartalas et al. (2015), EBF inPeng et al. (2016) and Edge-F1 inMao et al. (2018).

Our results are not directly comparable withJohnson & Zhang (2014) andLewis et al. (2004) due to implementation details and the fact that they tune the threshold for each label using k-fold cross-validation. See Appendix B for more discussions of the baselines.

For HMCN, we replace its static features with a base model for fair comparison. The original HMCN + HAN failed to learn and we removed its batch normalization. See Appendix B for more details.

https://www.csie.ntu.edu.tw/ cjlin/libsvmtools/multilabel/

Figures and Tables:

Figure 3: Performance comparison of different frameworks using the same base model as input. We improved HMCN + HAN by removing its batch normalization.

Figure2: The architecture of the proposed framework HiLAP. One CNN model(Kim, 2014) is used as the base model for illustration. The document embedding e d generated by the base model is combined with the label embedding l t and used as the state representation s t , based on which actions are taken by the policy network.

Table content: <table/>

Summary of the three datasets. |L| denotes the number of labels in the label hierarchy. Avg(|L i |) and Max(|L i |) denote the average and maximum number of labels of one sample, respectively.

Table content: <table><tr><td colspan="6">Dataset Taxonomy |L| Avg(|L i |) Max(|L i |) Training</td><td>Test</td></tr><tr><td>RCV1</td><td>Tree</td><td>103</td><td>3.24</td><td>17</td><td>23,149</td><td>781,265</td></tr><tr><td>NYT</td><td>Tree</td><td>115</td><td>2.52</td><td>14</td><td>25,279</td><td>10,828</td></tr><tr><td>Yelp</td><td>DAG</td><td>539</td><td>3.77</td><td>32</td><td>87,375</td><td>37,265</td></tr></table>

Comparison results on the RCV1 dataset. * denotes the results reported inPeng et al. (2018) on the same dataset split. Note that the original results of RCV1 inGopal & Yang (2013) are not comparable because they used a different label hierarchy.

Table content: <table><tr><td/><td>Method</td><td colspan="3">Micro-F1 Macro-F1 Samples-F1</td></tr><tr><td/><td>Leaf-SVM  *</td><td>69.1</td><td>33.0</td><td>-</td></tr><tr><td>Flat</td><td>SVM HLSTM  *  (Chen et al., 2016) TextCNN (Kim, 2014)</td><td>80.4 67.3 76.6</td><td>46.2 31.0 43.0</td><td>80.5 -75.8</td></tr><tr><td/><td>HAN (Yang et al., 2016)</td><td>75.3</td><td>40.6</td><td>76.1</td></tr><tr><td/><td>bow-CNN (Johnson &amp; Zhang, 2014)</td><td>82.7</td><td>44.7</td><td>83.3</td></tr><tr><td>Local &amp; Global</td><td>TD-SVM (Liu et al., 2005) HSVM  *  (Tsochantaridis et al., 2005) HR-SVM  *  (Gopal &amp; Yang, 2013) HR-DGCNN  *  (Peng et al., 2018) HMCN (Wehrmann et al., 2018) HiLAP (TextCNN) HiLAP (HAN)</td><td>80.1 69.3 72.8 76.1 80.8 78.6 75.4</td><td>50.7 33.3 38.6 43.2 54.6 50.5 45.5</td><td>80.5 ---82.2 80.1 77.4</td></tr><tr><td/><td>HiLAP (bow-CNN)</td><td>83.3</td><td>60.1</td><td>85.0</td></tr></table>

Results of various methods on the NYT and Yelp datasets.

Table content: <table><tr><td/><td>Method</td><td/><td>NYT</td><td/><td/><td>Yelp</td><td/></tr><tr><td/><td/><td colspan="6">Micro-F1 Macro-F1 Samples-F1 Micro-F1 Macro-F1 Samples-F1</td></tr><tr><td/><td>SVM</td><td>72.4</td><td>37.1</td><td>74.0</td><td>66.9</td><td>36.3</td><td>68.0</td></tr><tr><td>Flat</td><td>TextCNN (Kim, 2014) HAN (Yang et al., 2016)</td><td>69.5 62.8</td><td>39.5 22.8</td><td>71.6 65.5</td><td>62.8 66.7</td><td>27.3 29.0</td><td>63.1 67.9</td></tr><tr><td/><td>bow-CNN (Johnson &amp; Zhang, 2014)</td><td>72.9</td><td>33.4</td><td>74.1</td><td>63.6</td><td>23.9</td><td>63.9</td></tr><tr><td>Local &amp; Global</td><td>TD-SVM (Liu et al., 2005) HMCN (Wehrmann et al., 2018) HiLAP (TextCNN) HiLAP (HAN) HiLAP (bow-CNN)</td><td>73.7 72.2 69.9 65.2 74.6</td><td>43.7 47.4 43.2 28.7 51.6</td><td>75.0 74.2 72.8 68.0 76.6</td><td>67.2 66.4 65.5 69.7 68.9</td><td>40.5 42.7 37.3 38.1 42.8</td><td>67.8 67.6 68.4 72.4 71.5</td></tr></table>

Ablation Study of Framework Components in HiLAP To better understand Q3, We show the ablation analysis of our framework in Table4. Using Flat Component Only degenerates our framework to the flat baseline. By comparing the results of Flat Component Only and HiLAP-SL-NoFlat (a variant of HiLAP-SL without flat loss), we further confirm that flat approaches are likely to neglect sparse labels, which results in low Macro-F1. By combining the two components, HiLAP-SL

Table content: <table><tr><td/><td>80</td><td>78.1</td><td>76.6</td><td>77.8</td><td>78.6</td><td colspan="3">TextCNN HMCN Flat</td><td>78.6</td><td>75.8</td><td>77.2</td><td>80.1</td><td colspan="2">80 77.4</td><td>75.3</td><td>75.4</td><td>75.4</td><td/><td colspan="2">HAN HMCN Flat</td><td>78.9</td><td>76.1</td><td>76.7</td><td>77.4</td><td colspan="2">80 80.6</td><td>82.7</td><td>82.5</td><td>83.3</td><td colspan="3">bow-CNN HMCN Flat</td><td>80.8</td><td>83.3</td><td>83.0</td><td>85.0</td></tr><tr><td/><td/><td/><td/><td/><td/><td/><td colspan="2">HiLAP-SL</td><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td colspan="2">HiLAP-SL</td><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td colspan="2">HiLAP-SL</td></tr><tr><td>RCV1</td><td>40 60</td><td/><td/><td/><td/><td>33.4</td><td>43.0 HiLAP 45.1</td><td>50.5</td><td/><td/><td/><td/><td>40 60</td><td/><td/><td/><td/><td>38.9</td><td>40.6 HiLAP 41.7</td><td>45.5</td><td/><td/><td/><td/><td>40 60</td><td/><td/><td/><td/><td>39.4</td><td>44.7 HiLAP 55.3</td><td>60.1</td></tr><tr><td/><td>20</td><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td>20</td><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td>20</td><td/><td/><td/><td/><td/><td/><td/></tr><tr><td/><td>0</td><td colspan="4">Micro-F1</td><td colspan="3">Macro-F1</td><td colspan="4">Samples-F1</td><td>0</td><td colspan="4">Micro-F1</td><td colspan="3">Macro-F1</td><td colspan="4">Samples-F1</td><td>0</td><td colspan="4">Micro-F1</td><td colspan="3">Macro-F1</td><td>Samples-F1</td></tr><tr><td>NYT</td><td>20 30 40 50 60 70</td><td>68.4</td><td>69.5</td><td>68.8</td><td>69.9</td><td>30.0</td><td colspan="2">39.5 HMCN 40.6 43.2 Flat HiLAP-SL HiLAP</td><td>70.3</td><td>71.6</td><td>71.2</td><td>72.8</td><td>20 30 40 50 70 60</td><td>63.7</td><td>62.8</td><td>60.5</td><td>65.2</td><td>21.8</td><td colspan="2">22.8 HMCN 23.2 28.7 Flat HiLAP-SL HiLAP</td><td>65.9</td><td>65.5</td><td>64.0</td><td>68.0</td><td colspan="2">20 40 80 73.4 60</td><td>72.9</td><td>74.0</td><td>74.6</td><td>34.6</td><td colspan="2">33.4 HMCN Flat HiLAP-SL 47.2 51.6 HiLAP</td><td>74.2</td><td>74.1</td><td>75.5</td><td>76.6</td></tr><tr><td/><td>10</td><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td>10</td><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/></tr><tr><td/><td>0</td><td colspan="4">Micro-F1</td><td colspan="3">Macro-F1</td><td colspan="4">Samples-F1</td><td>0</td><td colspan="4">Micro-F1</td><td colspan="3">Macro-F1</td><td colspan="4">Samples-F1</td><td>0</td><td colspan="4">Micro-F1</td><td colspan="3">Macro-F1</td><td>Samples-F1</td></tr><tr><td>Yelp</td><td>20 30 40 50 60 70</td><td>55.9</td><td>62.8</td><td>62.1</td><td>65.5</td><td/><td colspan="2">27.3 HMCN 32.4 37.3 Flat HiLAP-SL HiLAP</td><td>57.3</td><td>63.1</td><td>62.7</td><td>68.4</td><td>20 30 40 50 60 70</td><td>70.8</td><td>66.7</td><td>67.0</td><td>69.7</td><td>18.7</td><td colspan="2">29.0 HMCN 30.1 38.1 Flat HiLAP-SL HiLAP</td><td>72.0</td><td>67.9</td><td>68.7</td><td>72.4</td><td>20 30 40 50 70 60</td><td>58.0</td><td>63.6</td><td>64.9</td><td>68.9</td><td/><td colspan="2">23.9 HMCN 25.4 42.8 Flat HiLAP-SL HiLAP</td><td>59.3</td><td>63.9</td><td>66.2</td><td>71.5</td></tr><tr><td/><td>10</td><td/><td/><td/><td/><td>6.7</td><td/><td/><td/><td/><td/><td/><td>10</td><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td>10</td><td/><td/><td/><td/><td>7.3</td><td/><td/></tr><tr><td/><td>0</td><td colspan="4">Micro-F1</td><td colspan="3">Macro-F1</td><td colspan="4">Samples-F1</td><td>0</td><td colspan="4">Micro-F1</td><td colspan="3">Macro-F1</td><td colspan="4">Samples-F1</td><td>0</td><td colspan="4">Micro-F1</td><td colspan="3">Macro-F1</td><td>Samples-F1</td></tr></table>

Ablation study of HiLAP-SL and HiLAP when combining with bow-CNN(Johnson & Zhang, 2014) on the RCV1 dataset(Lewis et al., 2004).

Table content: <table><tr><td>Method</td><td colspan="3">Micro-F1 Macro-F1 Samples-F1</td></tr><tr><td>Flat Component Only</td><td>82.7</td><td>44.7</td><td>83.3</td></tr><tr><td>HiLAP-SL-NoFlat</td><td>81.0</td><td>52.1</td><td>81.7</td></tr><tr><td>HiLAP-SL</td><td>82.5</td><td>55.3</td><td>83.0</td></tr><tr><td>HiLAP-NoSL</td><td>83.2</td><td>59.3</td><td>85.0</td></tr><tr><td>HiLAP-NoFlat</td><td>83.0</td><td>59.8</td><td>84.7</td></tr><tr><td>HiLAP</td><td>83.3</td><td>60.1</td><td>85.0</td></tr></table>

2010;

Table content: <table><tr><td/><td>/HYHOEDVHG0DFUR)*DLQV</td><td/><td colspan="2">3RSXODULW\EDVHG0DFUR)*DLQV</td><td/></tr><tr><td>$EVROXWH0DFUR)'LIIHUHQFH</td><td>+L/$3 +L/$36/ +0&amp;1</td><td>$EVROXWH0DFUR)'LIIHUHQFH</td><td>+L/$3 +L/$36/ +0&amp;1</td><td/><td/></tr><tr><td>/</td><td>/</td><td>/</td><td>3</td><td>3</td><td>3</td></tr><tr><td colspan="6">Figure 4: Level-based and popularity-based Macro-F1 gains compared to bow-CNN (Johnson &amp;</td></tr><tr><td colspan="2">Zhang, 2014) on the NYT</td><td/><td/><td/><td/></tr></table>

Peer review (iclr2019):
 This papers uses the label hierarchy to drive the search process over a set of labels using reinforcement learning. The approach offers clever and promising techniques to force the inference process in structured classification to converge, but experiments seem to lack apple-to-apple comparisons. However, I think the authors should rather present this work as structured classification, as labels dependencies not modeled by the hierarchy are exploited, and as other graph structure could be exploited to drive the RL search. I tend to see hierarchical classification as an approach to multi-label classification justified by a greedy decomposition that reduced both training and test time. This view has been outmoded for more than an decade, first as flat approaches became feasible, and now as end-to-end structured classification is implementable with DNNs (see for instance David Belanger work with McCallum) Compared to other structured classification approaches whose scope is limited by the complexity of the inference process, this approaches is very attractive. The authors open the optimization black box of the inference process by adding a few very clever tricks that facilitate convergence: - Intermediate rewards based on the gain on F1 score - Self critical training approach - "Clamped" pre-training enabled by the use of state embeddings that are multiplied my a transition to any state in the free mode, and just the next states in the hierarchy in the clamped mode - Addition of a flat loss to improve the quality of the document representation While those tricks may have been used for other applications, they seem new in the context of hierarchical/multi-label/structured classification. While the experiments appear thorough, they could be the major weakness of this paper. The results the authors quote as representative of other approaches seem in fact entirely reproduced on datasets that were not used on the original papers, and the authors do not try an apple-to-apple comparison to determine if this *reproduction* is fair. None of the quoted work used the 2018 version of Yelp, and I could only find RCV1 Micro-F1 experiments in Johnson and Yang, who report a 84% micro-F1, far better than the 76.6% reported on their behalf here, and better than the 82.7% reported by the authors. I read note 4 about the difference in the way the threshold is computed, but I doubt it can explain such a large difference. I did not check everything, but could not find and apple-to-apple comparison? Have the network architecture been properly optimized in terms of hyper-parameters? In particular, having tried Kim CNN on large label sets, I suspect the author settings using a single layer after the convolution is sub-optimal. I concur with the following paper than an additional hidden layer is essential: Liu et al "Deep Learning for Extreme Multi-label Text Classification". I also note the 32 batch size could be way too small for sparse label sets (I tend to use a batch size of 512 on this type of data).