Year: 2018

Abstract:
We present an end-to-end reinforcement learning approach to hierarchical text classification where documents are labeled by placing them at the right positions in a given hierarchy. While existing “global” methods construct hierarchical losses for model training, they either make “local” decisions at each hierarchy node or ignore the hierarchy structure during inference. To close the gap between training/inference and capture label dependencies in an end-to-end manner, we propose to learn a label assignment policy to determine where to place the documents and when to stop. The proposed method, HiLAP, directly optimizes metrics over the hierarchy, makes inter-dependent decisions during inference, and can be combined with different text encoding models for end-to-end training. Experiments on three public datasets show that HiLAP yields an average improvement of 33.4% in Macro-F1 and 5.0% in Samples-F1, outperforming state-of-the-art methods by a large margin.1

heading: 1 INTRODUCTION
text: In recent years there has been a surge of interest in leveraging taxonomies and hierarchies to organize and classify text documents, leading to the development of hierarchical text classification (HTC) methods—methods that can predict for a document multiple appropriate labels (which together constitute a sub-tree) in a given hierarchy. These methods have found a wide range of applications such as question answering (Qu et al., 2012), online advertising (Agrawal et al., 2013), and scientific literature organization (Peng et al., 2016). In contrast to traditional “flat” classification, the key challenge of HTC lies in modeling the inter-dependent, large-scale, and imbalanced label space.
Due to the complexity of HTC, how to better utilize the label hierarchy remains an open problem. HTC methods are traditionally divided into three categories, namely flat, local, and global approaches (Silla & Freitas, 2011). Flat approaches generally ignore the label hierarchy. Some only predict labels at the leaf nodes and then add all the ancestors of the predicted leaf nodes. Others ignore the hierarchy and perform standard multi-label classification, in which inconsistencies (i.e., one label is predicted but its ancestors are not) may occur and post-processing is thus needed. Local approaches train a set of local classifiers per node/per parent node/per level, which function independently and (usually) make predictions in a top-down order to avoid inconsistencies. Traditional global approaches (Cai & Hofmann, 2004; Vens et al., 2008; Silla Jr & Freitas, 2009) are largely modified based on specific flat models and rely on static, human curated features as input. In addition, many existing global approaches make unrealistic assumptions of the problem as in flat approaches. For example, Hierarchical-SVM (Cai & Hofmann, 2004) requires that all possible labels are on the leaf nodes and the heights of leaf nodes are the same.
Recent approaches (Kim, 2014; Lai et al., 2015; Yang et al., 2016) to text classification mainly focus on flat classification and have been shown to be very effective. However, their performance in HTC is relatively less studied. Even if the classification task is essentially hierarchical, prior work (Gopal & Yang, 2013; Johnson & Zhang, 2014; Peng et al., 2016; 2018) still makes flat and independent predictions and utilizes intuitive constraints, such as the embeddings of one label and its parent should be close. One recent framework (Wehrmann et al., 2018) attempted to leverage both local and global information. However, it uses static features and its inference is essentially flat, which may lead to inconsistencies.
1Code and data will be released on GitHub upon acceptance.
In this paper, we present an end-to-end reinforcement learning approach to HTC where documents are labeled by placing them at the right positions in a label hierarchy. We propose HiLAP, a principled global framework that learns a label assignment policy to determine where to place the documents and when to stop. HiLAP optimizes its policy by exploring the label hierarchy, in which training and inference follow the same routine and inter-dependent decisions are made. Compared to flat and local approaches, HiLAP achieves better effectiveness because it examines the global hierarchical structure during both training and inference phases. Compared to most existing global approaches, HiLAP has more flexibility in that it has no constraints on the structure of the hierarchy. Furthermore, the label assignment policy of HiLAP ensures that its predictions are always consistent and no post-processing is needed.
HiLAP can be combined with different text representation learning models and trained in an endto-end fashion. We select three representative text encoding models as the base models to evaluate the effectiveness of HiLAP. Experiments on three public datasets from different domains show that combining HiLAP with existing representation learning models yields an average performance improvement of 33.4% in Macro-F1 and 5.0% in Samples-F1, outperforming state-of-the-art HTC methods by a large margin. In particular, ablation study shows that HiLAP is especially beneficial to those unpopular labels at the bottom levels.

heading: 2 LEARNING LABEL ASSIGNMENT POLICY FOR END-TO-END HIERARCHICAL TEXT CLASSIFICATION
text: This section presents the proposed end-to-end reinforcement learning approach to hierarchical text classification. We first introduce our label assignment policy including the design of its actions and rewards, and then describe the details of policy learning.

heading: 2.1 HIERARCHICAL LABEL ASSIGNMENT
text: We define a label hierarchy H = (L,E) as a tree or DAG (directed acyclic graph)-structured hierarchy with node set L (i.e., the labels), and edge set E (which indicates the parent-child relationship between labels). Taking a set of documents {x1, x2, ..., xN} and their document labels {L1, L2, ..., LN} ∈ L as input, we aim to learn a policy P to place each documents xi to its labels Li on the label hierarchy H. Specifically, the policy P puts xi at the root label in the beginning and at each time step, decides which label xi should be further placed to, among all the adjacent labels of where xi has been placed, until a special stop action is taken. An illustration of our label assignment policy is shown in Figure 1. We define one base model B as a mapping f that converts each raw document xi to a finite dimensional vector as its representation, i.e., the document embedding ed ∈ RD (D denotes the embedding size). B can be any neural text representation learning model and its output ed is used as the input of the policy P . The challenge, compared to standard classification setup, is that we need to model E, i.e., the relationship between labels.

heading: 2.2 REINFORCEMENT LEARNING FOR LABEL ASSIGNMENT
text: To learn the label assignment policy, we train a policy network to determine where to place the documents and when to stop as follows.
Actions We regard the process of placing a document xi to the right positions on the label hierarchy as making a sequence of actions. Specifically, we define that an action at at time step t is to select one label lt from the action space At and place xi to that label lt. We denote the children of label lt as C(lt). The action space At consists of all the adjacent labels of where the document xi has been placed. In this way, for example, HiLAP can first place xi to a label at level 3 if the confidence (probability) of that label is higher and then place it to another label at level 1.
At the beginning of each episode, xi is placed at the root label l0 and the action space At = C(l0), i.e., all the labels at level 1. When xi is placed at another label l1, its children C(l1) would then be added to the action space At. In addition, one stop action estop ∈ RC (C denotes the embedding size) is added to the action space At so that the model can learn when to stop placing document xi to new labels. Intuitively, when the confidence of placing xi to another label is lower than the stop action, the label assignment process would be terminated. In short, the size of At is |{C(l0), C(l1), ..., C(lt), stop} − {l1, l2, ..., lt}|. Note that in other local/global approaches the predictions on different paths are independent while in HiLAP the inter-dependencies of labels across different paths and levels are considered and we optimize the metrics over the hierarchy by providing the policy network with rewards that capture the overall quality of its label assignment.
Rewards HiLAP receives rewards from the environment as feedback for its actions. Different from existing work where each label of one sample is treated independently, HiLAP captures the quality of all the labels assigned to each sample xi by rewarding the agent with the per-sample F1 (see Section 3.1 for details): F1xi = 2×P
xi×Rxi Pxi+Rxi , where P xi and Rxi denote per-sample precision and recall respectively that compare the predicted labels with gold labels of xi.
Instead of waiting until the end of the label assignment process and comparing the predicted labels with the gold labels, we use reward shaping (Mao et al., 2018), i.e., giving intermediate rewards at each time step, to accelerate the learning process. Specifically, we set the reward r of xi at time step t to be the difference of its F1 score between current and last time step: rxit = F1 xi t − F1 xi t−1. If current F1 is better than that at last time step, the reward would be positive, and vice versa. The cumulative reward from current time step to the end of an episode would cancel the intermediate rewards and thus reflect whether current action improves the overall label assignment of one sample or not. As a result, the learned policy would not focus on the current placement but have a long-term view that takes following actions into account.
Policy Network We parameterize each action at by a policy network π(a | s;W). The architecture of HiLAP is shown in Figure 2. For each document, its representation ed is generated by the base model B. For each label, a label embedding l ∈ RC is randomly initialized and updated during training. To model the label relationship E, the embeddings of the document ed and current label lt
are concatenated and projected to a vector st ∈ RC via a two-layer feed-forward network. st has the same size as the label embedding l and we use st as the state representation of the document at current label lt. By stacking the action embeddings, we can obtain an action matrix At with size |{C(l0), C(l1), ..., C(lt), stop} − {l1, l2, ..., lt}| × C. At is multiplied with the state embedding st, which outputs the probability distribution of actions. Finally, an action at is sampled based on the probability distribution of the action space:
st = ReLU(W1l ReLU(W 2 l [ed; lt])),
π(a | s;W) = softmax(Atst), at ∼ π(a | s;W).
We use REINFORCE (Williams, 1992), one instance of the policy gradient methods, as the optimization algorithm. To reduce variance, 10 rollouts for each training sample are run and the rewards are averaged. In addition, we adopt a self-critical training approach (Rennie et al., 2017). For each document xi, two label assignments are generated: L̃xi is sampled from the probability distribution, and L̂xi , the baseline label assignment, is greedily generated by choosing the action with the highest probability at each time step. We use r(L̃xi) − r(L̂xi) as the actual reward, which ensures that the policy network learns to place the document to positions with higher F1 score than the greedy baseline. At the time of inference, we greedily select labels with the highest probability as L̂xi .

heading: 2.3 TOP-DOWN SUPERVISED PRE-TRAINING
text: It is known that reinforcement learning models often suffer from high variance during training. Instead of learning from scratch, we use supervised learning to pre-train our framework. We denote the supervised variant as HiLAP-SL. While most parameters of HiLAP-SL are shared with HiLAP, its action space and way of exploring of the label hierarchy H are dissimilar.
The main difference is that HiLAP-SL explores the label hierarchy H in a top-down manner. At each time step t, the document goes down one level on the hierarchy. HiLAP-SL concentrates on the local discrimination of labels with the same parent. The local per-parent label probability distribution pSLt is generated as follows. pSLt = σ(Ctst), where σ denotes the sigmoid function, and Ct ∈ R|C(lt)|×C denotes the action space of HiLAP-SL, i.e., an embedding matrix consisting of the children of current label lt (rather than all the labels where xi has been placed as in HiLAP).
Another difference is that in HiLAP the actions are sampled and thus the documents might be placed to wrong labels, while in HiLAP-SL only the ground-truth positions are traversed during training. Specifically, if there are K(≥ 1) ground-truth labels at the same level, the document embedding ed would be cloned K times following each label and K different paths would be generated independently. The loss function of HiLAP-SL is defined as follows.
Ll = T∑
t=0
Lt,
where T is the lowest label’s height of one sample (T may vary by samples) and Lt estimates the binary cross entropy over the candidate labels C(lt) at each time step t. HiLAP-SL works as if there were a set of local classifiers, although most of its parameters (except for the label embeddings lt) are shared by all the labels so that one does not need to actually train a set of classifiers. During inference, HiLAP-SL follows the same top-down routine as in training using the per-parent label probability pSLt and thus no post-processing is needed for inconsistency correction.
All the parameters of HiLAP are shared with HiLAP-SL and can be initialized by the pre-trained HiLAP-SL model except for the embedding of the stop action estop (which is randomly initialized).

heading: 2.4 COMBINING FLAT, LOCAL, AND GLOBAL INFORMATION FOR POLICY LEARNING
text: We further add a flat component to our framework as a regularization of the base model. Specifically, the flat component is a simple feed-forward network consisting of a fully connected layer and the
sigmoid function. It projects the document embedding ed to a label probability distribution pf of all the labels L on the hierarchy.
pf = σ(Wfed).
The combination of the base model and the flat component is exactly the same as a flat model and ensures that the document representation ed learned by the base model B has the capability of flat classification among all the labels L. We use a flat loss Lf to measure the binary cross entropy over all the labels as in the flat models. Combining the flat loss with local loss, the supervised loss in HiLAP-SL is defined as LSL = λLf + (1 − λ) ∑T t=0 Lt, where λ ∈ [0, 1] is the mixed ratio. Similar to Celikyilmaz et al. (2018), we also found that mixing a proportion of the supervised loss is beneficial to the learning process of HiLAP. Further combining the global information, the total loss of HiLAP is defined as Lmixed = LRL + αLSL, where α is a scaling factor accounting for the difference in magnitude between LRL and LSL. While we do not directly use the flat component during inference, it helps the learning process of the base model and improves the performance of both HiLAP-SL and HiLAP, which we will show in Section 3.5.

heading: 3 EXPERIMENTS
text: We evaluate the benefits of our framework against a number of state-of-the-art HTC approaches, with the goal of answering the following questions:
Q1 How does our proposed method (HiLAP) compare to state-of-the-art HTC approaches?
Q2 How does HiLAP compare to other hierarchical classification frameworks when the same base models are adopted by all the frameworks?
Q3 How do different components in HiLAP contribute to its performance in terms of the popular(sparse) labels?

heading: 3.1 DATASETS AND EVALUATION METRICS
text: We conduct experiments on three public and commonly used datasets from different domains. The first two datasets are related to news categorization, including RCV1 (Lewis et al., 2004) and the New York Times (NYT) annotated corpus (Sandhaus, 2008). We follow the original training/test split for RCV1 and sub-sample NYT due to its large size. The third dataset is the Yelp Dataset Challenge 20182, which consists of a subset of Yelp businesses and their reviews. We use the Yelp Business Categories3 as the label hierarchy and predict the categories of one business using its reviews. For each dataset, there may be more than one label at each level and the lowest labels of a sample may not be at the leaf nodes. A summary of the datasets is shown in Table 1 and further details can be found in Appendix A.
We use standard metrics (Johnson & Zhang, 2014; Peng et al., 2016) for HTC including Micro-F1, Macro-F1, and Samples-F14. Micro-F1 measures the overall precision/recall and favors labels with more samples. Macro-F1 calculates the F1 scores of all the labels and performs an unweighted average over them. Similarly, Samples-F1 calculates the F1 scores of all the samples and averages them ( ∑ i F1 xi
N ). Recall that F1 xi is used as the reward in HiLAP.
2https://www.yelp.com/dataset/challenge 3https://www.yelp.com/developers/documentation/v3/all category list 4We take the name Samples-F1 from sklearn.metrics.f1 score. It is referred to as Example-based F1 in Par-
talas et al. (2015), EBF in Peng et al. (2016) and Edge-F1 in Mao et al. (2018).

heading: 3.2 BASE MODELS
text: In our experiments, three representative text classification models with different characteristics are selected as the base models to prove the robustness and versatility of HiLAP. As one will see, HiLAP consistently improves the base model through better exploration of the label hierarchies.
TextCNN (Kim, 2014) is the classic convolutional neural network for text classification. In our implementation, TextCNN is composed of one convolutional layer with three kernels of different sizes (3, 4, 5), followed by max pooling, a dropout layer, and a fully-connected layer. We chose TextCNN because it was one of the first successful and well used neural-based models for text classification.
HAN (Yang et al., 2016) first learns the representation of sentences by feeding words in each sentence to a GRU-based sequence encoder (Bahdanau et al., 2014) and then feeds the representation of the encoded sentences into another GRU-based sequence encoder, which generates the representation of the whole document. Attention mechanism such as word attention and sentence attention is also used. We chose HAN because it uses RNNs instead of CNNs and is shown to be effective on the Yelp Review datasets (Zhang et al., 2015).
bow-CNN (Johnson & Zhang, 2014) employs bag of words (multi-hot zero-one vectors) as input and directly applies CNN to high-dimensional text data. It learns the representation of small text regions (rather than single words) for use in classification. We chose bow-CNN since it does not use any word embeddings as in TextCNN and HAN. In addition, bow-CNN achieved the state-of-the-art performance on the RCV1 dataset (Lewis et al., 2004).

heading: 3.3 BASELINES
text: We compare our framework with state-of-the-art HTC methods. The traditional methods that we compare with are Support Vector Machines (SVM) and its hierarchical variants. Specifically, SVM performs standard multi-label classification using one-vs-the-rest (OvR) strategy. Leaf-SVM treats each leaf node as a label and adds the ancestors of predicted leaf nodes. Other variants include HSVM (Tsochantaridis et al., 2005), Top-Down SVM (TD-SVM) (Liu et al., 2005), and Hierarchically Regularized Support Vector Machines (HR-SVM) (Gopal & Yang, 2013). The neural-based methods that we compare with include HLSTM (Chen et al., 2016), HR-DGCNN (Peng et al., 2018) and HMCN (Wehrmann et al., 2018). We also compare with the base models, namely, TextCNN (Kim, 2014), HAN (Yang et al., 2016), and bow-CNN (Johnson & Zhang, 2014) to see how much we could improve upon them via better exploration of the hierarchy.

heading: 3.4 IMPLEMENTATION DETAILS
text: For each base model, we follow its original implementation and fix the hyper-parameters for different datasets. We randomly sample 10% from the training set as development set (Johnson & Zhang, 2014; Peng et al., 2018). We set batch size to 32 and use the first 256 tokens of each document for representation learning. We use a constant threshold (0.5) for all the labels. All the models are trained using an Adam optimizer with initial learning rate 1e-3 and weight decay 1e-6. We use pretrained GloVe word vectors (Pennington et al., 2014) with dimensionality 50 as word embeddings for TextCNN and HAN. We limit the vocabulary to the most frequent 30000 words in the training data and generate multi-hot vectors as the input of bow-CNN. For our framework, we set the size of W2l to 500 and the sizes of W 1 l and label embedding lt to 50.

heading: 3.5 EXPERIMENTAL RESULTS
text: Main Results Table 2 and 3 compare the performance of HiLAP to the state-of-the-art HTC baselines. These results provide positive answers to our question Q1: On the RCV1 dataset, HiLAP (HAN) achieves similar performance to HR-DGCNN even though the original flat HAN is worse than HR-DGCNN. Our HiLAP (TextCNN) outperforms most baselines in Macro-F1 and HiLAP
(bow-CNN) achieves the best performance on all the three metrics.5 On the NYT dataset, similar results are observed: TextCNN and HAN are both improved when combining with HiLAP and HiLAP (bow-CNN) again achieves the best performance. On the Yelp dataset, HiLAP (HAN) achieves the best Micro-F1 and Samples-F1, while HiLAP (bow-CNN) obtains the highest MacroF1. Interestingly, a simple SVM outperforms several neural-based models, indicating that traditional feature-based methods still play an important role in HTC.
Performance Comparison using the Same Text Encoding Model To answer Q2 and Q3, we compare different frameworks that support the use of exactly the same base model and Figure 3 summarizes the comparison results.6 As one may notice, due to the extreme imbalance of the datasets, directly applying a flat multi-label classification model may suffer from low Macro-F1, i.e., the predictions of flat models are inevitably biased to the most popular labels. HMCN also has the same issue, resulting in Macro-F1 scores lower than 10 when combining with some base models. In contrast, HiLAP significantly outperforms the baselines especially in Macro-F1, which implies that our policy network is better at tackling labels with relatively few samples. On the downside, it is also observed that HiLAP-SL may have a negative effect in terms of Micro-F1 (although it is usually marginal compared with the gain in Macro-F1). However, such negative effects are eliminated by HiLAP through better exploration of the label hierarchy H. Overall, HiLAP obtains the highest performance on 24/27 results across three datasets, three base models, and three evaluation metrics.
Ablation Study of Framework Components in HiLAP To better understand Q3, We show the ablation analysis of our framework in Table 4. Using Flat Component Only degenerates our framework to the flat baseline. By comparing the results of Flat Component Only and HiLAP-SL-NoFlat (a variant of HiLAP-SL without flat loss), we further confirm that flat approaches are likely to neglect sparse labels, which results in low Macro-F1. By combining the two components, HiLAP-SL
5Our results are not directly comparable with Johnson & Zhang (2014) and Lewis et al. (2004) due to implementation details and the fact that they tune the threshold for each label using k-fold cross-validation. See Appendix B for more discussions of the baselines.
6For HMCN, we replace its static features with a base model for fair comparison. The original HMCN + HAN failed to learn and we removed its batch normalization. See Appendix B for more details.
achieves close performance to Flat Component Only on Micro-F1 and Samples-F1 and even higher Macro-F1 than HiLAP-SL-NoFlat. HiLAP-NoSL is initialized by the pre-trained HiLAP-SL model without mixing the supervised loss during its training. We can see that using the reinforced loss alone still improves the performance on all the three metrics. After removing the flat loss during the training of HiLAP, HiLAP-NoFlat shows slightly lower performance than the full HiLAP model, indicating that the flat component serves as a regularization of the base model and is beneficial to the overall performance.
We also analyze the source of the performance gains by dividing the labels based on their levels and number of supporting samples. Figure 4 shows one comparison of absolute Macro-F1 differences between various methods and the base model. We observe similar trends for other setups/metrics and omit them for better view. As depicted in Figure 4, HiLAP and HiLAP-SL are especially beneficial to those unpopular labels at the bottom levels.

heading: 4 RELATED WORK
text: Hierarchical text classification and general hierarchical classification approaches have been developed for many applications. In the biomedical domain, medical subject headings (MeSH) indexing, which is to assign a set of MeSH main headings to citations, has been studied for years (Liu et al., 2015; Peng et al., 2016). In addition, there are plenty of methods focusing on the hierarchical prediction of protein and gene functions (Clare & King, 2003; Silla Jr & Freitas, 2009; Secker et al., 2010;
Cerri et al., 2016). Another line of work concentrates on document categorization. Both traditional methods (Lewis et al., 2004; Gopal & Yang, 2013) and neural methods (Johnson & Zhang, 2014; Peng et al., 2018) have been proposed to classify the topics of newswire and web content (Dumais & Chen, 2000; Sun & Lim, 2001), categories of laws and patents (Bi & Kwok, 2015; Cai & Hofmann, 2004; Rousu et al., 2005).
Many previous works (Liu et al., 2005; Xue et al., 2008; Sun & Lim, 2001) train a set of local classifiers and make predictions in a top-down manner. In particular, (Bi & Kwok, 2015) develop Bayes-optimal predictions that minimize the global risks with the trained model but the model is still locally trained. Such local approaches are not popularly used among recent neural-based hierarchical classification models (Johnson & Zhang, 2014; 2016; Peng et al., 2018) since it is infeasible to train many neural classifiers locally.
Global methods, on the other hand, train only one classifier. Although global methods are desirable, they are relatively less studied due to the complexity of the problem. Existing global models are generally modified based on specific flat models. Hierarchical-SVM (Cai & Hofmann, 2004; 2007; Qiu et al., 2009) generalizes Support Vector Machine (SVM) learning based on discriminant functions that are structured in a way that mirrors the label hierarchy. One limitation is that Hierarchical-SVM only supports balanced tree (all possible labels are presumed to be at the same height in their experiments). Hierarchical naive Bayes (Silla Jr & Freitas, 2009) modifies naive Bayes by updating weights of one’s ancestors as well whenever one label’s weights are updated. There are other global methods that are based on association rules (Wang et al., 2001), C4.5 (Clare & King, 2003), kernel machines (Rousu et al., 2005), and decision tree (Vens et al., 2008). Constraints such as the regularization that enforces the parameters of one node and its parent to be similar (Gopal & Yang, 2013; 2015) are also proposed to leverage the label hierarchy while maintaining scalability. However, their use of hierarchies is somewhat limited.
In addition to the most relevant prior studies mentioned above, dataless classification (Ha-Thuc & Renders, 2011; Song & Roth, 2014; Meng et al., 2018) leverages the labels in the hierarchy as weak supervision and requires almost no training data. Yu et al. (2014); Bhatia et al. (2015) scale to hundreds of thousands of labels but do not assume and leverage the existence of label hierarchies.

heading: 5 CONCLUSION
text: We proposed an end-to-end reinforcement learning approach to hierarchical text classification where documents are labeled by placing them at the right positions in the label hierarchy. The proposed framework makes consistent and inter-dependent predictions, in which any neural-based representation learning model can be used as a base model and a label assignment policy is learned to determine where to place the documents and when to stop. Experiments on three public datasets of different domains showed that our approach outperforms state-of-the-art hierarchical text classification methods significantly. In the future, we will explore the effectiveness of the proposed framework on other base models and forms of data (e.g., images). We also plan to mix more losses covering other aspects in the objective function and test whether they could further improve the performance of our framework.

heading: A DATASET DETAILS
text: In this section, we describe the details of the datasets. The RCV1 dataset (Lewis et al., 2004) is a manually labeled newswire collection of Reuters News from 1996 to 1997. Its news documents are categorized with three aspects: industries, topics, and regions. We use its topic-based label hierarchy for classification as it has been well used in prior work (Gopal & Yang, 2013; Johnson & Zhang, 2014; Peng et al., 2018; Wehrmann et al., 2018). There are 103 categories and four levels in total including all labels except for the root label in the hierarchy.
The NYT annotated corpus (Sandhaus, 2008) is a collection of New York Times news from 1987 to 2007. Due to its large size, we randomly sampled 36107 documents from all the news documents, and further split them into training and test set of 25279 and 10828 samples, respectively. We use the first 3 layers in the hierarchy and keep the labels with at least 40 supporting samples.
For the preprocessing of the Yelp dataset, we first removed categories that have fewer than 100 businesses and then businesses that have fewer than 5 reviews. We concatenated (at most) the first 10 reviews of each business as its representation. We set the training/test ratio to 70%/30%, which results in a training set of 87,375 samples and a test set of 37,517 samples. This is an even more challenging task because the reviews are usually written in an informal way and it is more imbalanced than the RCV1 or NYT datasets (e.g., label Restaurants has 32,357 businesses in the training set while Retirement Homes has 23).

heading: B PERFORMANCE ANALYSIS OF BASELINES
text: There are several things to note in terms of the performance of the baselines. First, our results are not directly comparable to Lewis et al. (2004); Johnson & Zhang (2014) due to implementation details and the fact that they tune the threshold for each label using scutfbr. According to the implementation in LibSVM7, the scutfbr threshold tuning algorithm uses two nested 3-fold cross validation for each of the 103 labels and the classifier is trained 3× 3× 103 = 927 times, which is infeasible in our case.
Secondly, we found that the performance of HMCN (Wehrmann et al., 2018) is sometimes much lower than expected. After tuning their model, we observed that if we first do a weighted sum of the local and global outputs and then apply the sigmoid function, HMCN’s performance becomes much better (see Table 5) than doing them in the opposite order as in the paper. In addition, we found that HMCN + HAN (Yang et al., 2016) would result in extremely low performance. We had to remove HMCN’s batch normalization to make it compatible with HAN. Combining HMCN with other base models did not encounter similar issues.
Thirdly, our implementation of TextCNN (Kim, 2014) and HAN (Yang et al., 2016) shows better performance than those reported in Peng et al. (2018) due to implementation details. A comparison can be found in Table 6.
7https://www.csie.ntu.edu.tw/ cjlin/libsvmtools/multilabel/

Review summaries:
This paper presents an end to end rl approach for hierarchical text classification. The paper proposes a label assignment policy for determining the appropropriate positioning of a document in a hierarchy. It is based on capturing the global hierachical structure during training and prediction phases as against most methods which either exploit the local information or neural net approaches which ignore the hierarchical structure. It is demonstrated the method particularly works well compared to sota methods especially for macro-f1 measure which captures the label weighted performance. The approach seems original, and a detailed experimental analysis is carried out on various datasets.