End-to-End Hierarchical Text Classification with Label Assignment Policy

1 INTRODUCTION . In recent years there has been a surge of interest in leveraging taxonomies and hierarchies to organize and classify text documents , leading to the development of hierarchical text classification ( HTC ) methods—methods that can predict for a document multiple appropriate labels ( which together constitute a sub-tree ) in a given hierarchy . These methods have found a wide range of applications such as question answering ( Qu et al. , 2012 ) , online advertising ( Agrawal et al. , 2013 ) , and scientific literature organization ( Peng et al. , 2016 ) . In contrast to traditional “ flat ” classification , the key challenge of HTC lies in modeling the inter-dependent , large-scale , and imbalanced label space . Due to the complexity of HTC , how to better utilize the label hierarchy remains an open problem . HTC methods are traditionally divided into three categories , namely flat , local , and global approaches ( Silla & Freitas , 2011 ) . Flat approaches generally ignore the label hierarchy . Some only predict labels at the leaf nodes and then add all the ancestors of the predicted leaf nodes . Others ignore the hierarchy and perform standard multi-label classification , in which inconsistencies ( i.e. , one label is predicted but its ancestors are not ) may occur and post-processing is thus needed . Local approaches train a set of local classifiers per node/per parent node/per level , which function independently and ( usually ) make predictions in a top-down order to avoid inconsistencies . Traditional global approaches ( Cai & Hofmann , 2004 ; Vens et al. , 2008 ; Silla Jr & Freitas , 2009 ) are largely modified based on specific flat models and rely on static , human curated features as input . In addition , many existing global approaches make unrealistic assumptions of the problem as in flat approaches . For example , Hierarchical-SVM ( Cai & Hofmann , 2004 ) requires that all possible labels are on the leaf nodes and the heights of leaf nodes are the same . Recent approaches ( Kim , 2014 ; Lai et al. , 2015 ; Yang et al. , 2016 ) to text classification mainly focus on flat classification and have been shown to be very effective . However , their performance in HTC is relatively less studied . Even if the classification task is essentially hierarchical , prior work ( Gopal & Yang , 2013 ; Johnson & Zhang , 2014 ; Peng et al. , 2016 ; 2018 ) still makes flat and independent predictions and utilizes intuitive constraints , such as the embeddings of one label and its parent should be close . One recent framework ( Wehrmann et al. , 2018 ) attempted to leverage both local and global information . However , it uses static features and its inference is essentially flat , which may lead to inconsistencies . 1Code and data will be released on GitHub upon acceptance . In this paper , we present an end-to-end reinforcement learning approach to HTC where documents are labeled by placing them at the right positions in a label hierarchy . We propose HiLAP , a principled global framework that learns a label assignment policy to determine where to place the documents and when to stop . HiLAP optimizes its policy by exploring the label hierarchy , in which training and inference follow the same routine and inter-dependent decisions are made . Compared to flat and local approaches , HiLAP achieves better effectiveness because it examines the global hierarchical structure during both training and inference phases . Compared to most existing global approaches , HiLAP has more flexibility in that it has no constraints on the structure of the hierarchy . Furthermore , the label assignment policy of HiLAP ensures that its predictions are always consistent and no post-processing is needed . HiLAP can be combined with different text representation learning models and trained in an endto-end fashion . We select three representative text encoding models as the base models to evaluate the effectiveness of HiLAP . Experiments on three public datasets from different domains show that combining HiLAP with existing representation learning models yields an average performance improvement of 33.4 % in Macro-F1 and 5.0 % in Samples-F1 , outperforming state-of-the-art HTC methods by a large margin . In particular , ablation study shows that HiLAP is especially beneficial to those unpopular labels at the bottom levels . 2 LEARNING LABEL ASSIGNMENT POLICY FOR END-TO-END HIERARCHICAL TEXT CLASSIFICATION . This section presents the proposed end-to-end reinforcement learning approach to hierarchical text classification . We first introduce our label assignment policy including the design of its actions and rewards , and then describe the details of policy learning . 2.1 HIERARCHICAL LABEL ASSIGNMENT . We define a label hierarchy H = ( L , E ) as a tree or DAG ( directed acyclic graph ) -structured hierarchy with node set L ( i.e. , the labels ) , and edge set E ( which indicates the parent-child relationship between labels ) . Taking a set of documents { x1 , x2 , ... , xN } and their document labels { L1 , L2 , ... , LN } ∈ L as input , we aim to learn a policy P to place each documents xi to its labels Li on the label hierarchy H. Specifically , the policy P puts xi at the root label in the beginning and at each time step , decides which label xi should be further placed to , among all the adjacent labels of where xi has been placed , until a special stop action is taken . An illustration of our label assignment policy is shown in Figure 1 . We define one base model B as a mapping f that converts each raw document xi to a finite dimensional vector as its representation , i.e. , the document embedding ed ∈ RD ( D denotes the embedding size ) . B can be any neural text representation learning model and its output ed is used as the input of the policy P . The challenge , compared to standard classification setup , is that we need to model E , i.e. , the relationship between labels . 2.2 REINFORCEMENT LEARNING FOR LABEL ASSIGNMENT . To learn the label assignment policy , we train a policy network to determine where to place the documents and when to stop as follows . Actions We regard the process of placing a document xi to the right positions on the label hierarchy as making a sequence of actions . Specifically , we define that an action at at time step t is to select one label lt from the action space At and place xi to that label lt. We denote the children of label lt as C ( lt ) . The action space At consists of all the adjacent labels of where the document xi has been placed . In this way , for example , HiLAP can first place xi to a label at level 3 if the confidence ( probability ) of that label is higher and then place it to another label at level 1 . At the beginning of each episode , xi is placed at the root label l0 and the action space At = C ( l0 ) , i.e. , all the labels at level 1 . When xi is placed at another label l1 , its children C ( l1 ) would then be added to the action space At . In addition , one stop action estop ∈ RC ( C denotes the embedding size ) is added to the action space At so that the model can learn when to stop placing document xi to new labels . Intuitively , when the confidence of placing xi to another label is lower than the stop action , the label assignment process would be terminated . In short , the size of At is | { C ( l0 ) , C ( l1 ) , ... , C ( lt ) , stop } − { l1 , l2 , ... , lt } | . Note that in other local/global approaches the predictions on different paths are independent while in HiLAP the inter-dependencies of labels across different paths and levels are considered and we optimize the metrics over the hierarchy by providing the policy network with rewards that capture the overall quality of its label assignment . Rewards HiLAP receives rewards from the environment as feedback for its actions . Different from existing work where each label of one sample is treated independently , HiLAP captures the quality of all the labels assigned to each sample xi by rewarding the agent with the per-sample F1 ( see Section 3.1 for details ) : F1xi = 2×P xi×Rxi Pxi+Rxi , where P xi and Rxi denote per-sample precision and recall respectively that compare the predicted labels with gold labels of xi . Instead of waiting until the end of the label assignment process and comparing the predicted labels with the gold labels , we use reward shaping ( Mao et al. , 2018 ) , i.e. , giving intermediate rewards at each time step , to accelerate the learning process . Specifically , we set the reward r of xi at time step t to be the difference of its F1 score between current and last time step : rxit = F1 xi t − F1 xi t−1 . If current F1 is better than that at last time step , the reward would be positive , and vice versa . The cumulative reward from current time step to the end of an episode would cancel the intermediate rewards and thus reflect whether current action improves the overall label assignment of one sample or not . As a result , the learned policy would not focus on the current placement but have a long-term view that takes following actions into account . Policy Network We parameterize each action at by a policy network π ( a | s ; W ) . The architecture of HiLAP is shown in Figure 2 . For each document , its representation ed is generated by the base model B . For each label , a label embedding l ∈ RC is randomly initialized and updated during training . To model the label relationship E , the embeddings of the document ed and current label lt are concatenated and projected to a vector st ∈ RC via a two-layer feed-forward network . st has the same size as the label embedding l and we use st as the state representation of the document at current label lt. By stacking the action embeddings , we can obtain an action matrix At with size | { C ( l0 ) , C ( l1 ) , ... , C ( lt ) , stop } − { l1 , l2 , ... , lt } | × C. At is multiplied with the state embedding st , which outputs the probability distribution of actions . Finally , an action at is sampled based on the probability distribution of the action space : st = ReLU ( W1l ReLU ( W 2 l [ ed ; lt ] ) ) , π ( a | s ; W ) = softmax ( Atst ) , at ∼ π ( a | s ; W ) . We use REINFORCE ( Williams , 1992 ) , one instance of the policy gradient methods , as the optimization algorithm . To reduce variance , 10 rollouts for each training sample are run and the rewards are averaged . In addition , we adopt a self-critical training approach ( Rennie et al. , 2017 ) . For each document xi , two label assignments are generated : L̃xi is sampled from the probability distribution , and L̂xi , the baseline label assignment , is greedily generated by choosing the action with the highest probability at each time step . We use r ( L̃xi ) − r ( L̂xi ) as the actual reward , which ensures that the policy network learns to place the document to positions with higher F1 score than the greedy baseline . At the time of inference , we greedily select labels with the highest probability as L̂xi .

Summary: This paper presents an end to end rl approach for hierarchical text classification. The paper proposes a label assignment policy for determining the appropropriate positioning of a document in a hierarchy. It is based on capturing the global hierachical structure during training and prediction phases as against most methods which either exploit the local information or neural net approaches which ignore the hierarchical structure. It is demonstrated the method particularly works well compared to sota methods especially for macro-f1 measure which captures the label weighted performance. The approach seems original, and a detailed experimental analysis is carried out on various datasets.