Modeling label correlations implicitly through latent label encodings for multi-label text classification

1 INTRODUCTION . Multi-label text classification ( MLTC ) aims to associate each text sample with a set of labels . It can be applied to a wide spectrum of scenarios , including text categorization ( Schapire & Singer , 2000 ) , information retrieval ( Gopal & Yang , 2010 ) and web mining ( Zhang & Zhou , 2013 ; Liu et al. , 2020 ) . Unlike single-label text classification , in MLTC a text is usually associated with more than one label . Typically , these labels correlate with each other . Utilizing their correlations effectively can benefit their predictions significantly ( Tsoumakas et al. , 2009 ; Gibaja & Ventura , 2015 ) . However , due to the high-order nature of these correlations , it is a challenging task ( Zhang & Zhou , 2013 ) . There are different ways to deal with this challenge . A relatively earlier line of works models the correlations independently of text context ( Prabhu & Varma , 2014 ; Jernite et al. , 2017 ; Peng et al. , 2018 ; Singh et al. , 2018 ) . A representative class in this line is the tree-structure methods . Typically , they build a hierarchical structure for the label space statistically or according to taxonomy . During predictions , they usually utilize the modeled label correlations through some regularization methods ( Peng et al. , 2018 ; Singh et al. , 2018 ) . Since the structure of labels that heavily limits the correlations is defined in advance , they have introduced a strong inductive bias that suggests that labels correlate with each other following a tree structure unrelated to text context . But in real tasks , the labels and the text often correlate with each other closely in a complicated way . The correlations between 1The source code is provided in the supplementary file . labels can be quite different depending on text contexts . It may not be appropriate to simplify their correlations a priori too much . Another representative class of works in this line is the label embedding methods . In computer vision , researchers have used label embeddings to capture label correlations to enhance performances on zero-shot learning and other tasks ( Palatucci et al. , 2009 ; Yogatama et al. , 2015 ; Ma et al. , 2016 ) . Inspired by them , Wang et al . ( 2018 ) introduce joint label embeddings to model label relations and label-text relations to enhance text classification performances . Their work is extended by Xiao et al . ( 2019 ) . In these works , the correlations between labels are modeled by the similarities between label embeddings ; they are learned adaptively . Such correlation modeling is more flexible and expressive compared with the tree structure methods . And they do not only model label correlations but also model label-text relations . However , they still model label correlations independently of context . A relatively recent line of works not only embeds labels but also encodes them contextually through attention-based encoding ( or encoding-decoding ) mechanisms . Such mechanisms allow them to jointly model the correlations between labels , and between labels and the text context with less a priori limits . This line of works can be further divided into three major classes : sequence-tosequence ( Nam et al. , 2017 ; Yang et al. , 2018 ) , sequence-to-set ( Yang et al. , 2019 ; Tsai & Lee , 2020 ) and joint encoding ( Zhang et al. , 2021 ) . The sequence-to-sequence methods are first in the line . They are based on the well-known sequenceto-sequence framework . Generally , they encode text by deep neural networks and then decode them to targeted labels one by one . They lead to a breakthrough in this area by eliminating the inductive bias that label correlations are independent of text context . But due to the nature of the classic seq2seq framework , they also introduce another inductive bias : there is an order among the labels associated with each text . However , it is difficult to define such orders appropriately . And this issue is critical to these methods ( Vinyals et al. , 2015 ) . To overcome the difficulties in defining appropriate label orders , the sequence-to-set methods ( Yang et al. , 2019 ; Tsai & Lee , 2020 ) are proposed . Different from seq2seq methods that calculate loss each time a label is generated , they use reinforcement learning to delay loss calculation to the time when the whole label sequence is generated . Since the label sequences are constructed adaptively during training by reinforcement learning , it can obtain them more reasonably . However , the labels are essentially a set rather than a sequence . Constructing the label sequence adaptively still regards the labels as a sequence rather than a set . It does not eliminate the label-sequence inductive bias completely . Very recently , a new method ( Zhang et al. , 2021 ) ( it is the most related work to ours ) encodes the text and the labels jointly within a unified encoding process using BERT ( Devlin et al. , 2018 ) . It performs extra tasks on the label encodings to further model label correlations . By doing so , it removes the label-sequence inductive bias and obtains state-of-the-art results . But their work still suffers from the following three issues . ( 1 ) Their extra tasks are designed based on the following consideration : in the context of a given text , the co-occurred labels should have stronger mutual correlations . Since unrelated but common labels can co-occur frequently , we think this inductive bias may still be excessive . We call this issue label-correlation overload . ( 2 ) Their extra task plcp models label-correlations pairwisely rather than globally ; it may also be an oversimplification . ( 3 ) Although jointly encoding actual labels and texts has addressed the label-sequence inductive bias elegantly , it can be less effective and even impossible when the number of actual labels is very large . This issue limits its applications heavily . All the above methods share a common fundamental characteristic : they model label correlations explicitly . As one may observe from the above discussions , due to the high-order , contextual and non-sequential nature of label correlations , it is difficult to model them explicitly without introducing improper inductive bias . To overcome this difficulty , this paper uses latent label encodings to model actual labels and their correlations implicitly . Specifically , it pads a set of latent labels rather than actual labels in front of the inputs to BERT , and then maps the contextual encodings of these latent labels to actual class labels cooperatively . The label-label and label-text correlations regarding the actual labels are modeled indirectly through these latent-label encodings and their correlations . Since the correlations between these latent-label encodings , and their relations with actual labels and the text context are all trained adaptively , it imposes fewer a priori limits and provides more flexibility . Furthermore , by encoding latent labels rather than actual labels , our method can well address the forementioned three issues of the most related work to ours ( Zhang et al. , 2021 ) : ( 1 ) Our labelcorrelations are learned adaptively with few a priori limits , this can avoid label-correlation overload introduced by empirical extra tasks ; ( 2 ) By extensively utilizing latent-label encodings and their relations in each label ’ s prediction , we address the oversimplification of pairwise label-correlation modeling . ( 3 ) The number of latent-label is an adjustable hyper-parameter that can be adjusted to accommodate specific tasks , usually it is much smaller than the number of actual labels . This feature addresses the issue of massive-label well . General-purpose text classification algorithms are not specially designed for MLTC , such as TextCNN ( Kim , 2014 ) and BERT ( Devlin et al. , 2018 ) . They do not consider label correlations . In a sense , they can also be regarded as modeling label correlations implicitly . We think that the advantage of our algorithm over them in capturing label correlations lies in the following aspects : compared with their latent neurons , our latent label encodings are structural and high-orderly correlated . These features make it more expressive in modeling label correlations , just like the actual label encodings do . We summarize the major contributions of this paper as follows : ( 1 ) To the best of our knowledge , all the previous state-of-the-art works for MLTC regard label correlations as a key challenge . They model label correlations explicitly , which lacks flexibility and hence is prone to suffer from inappropriate inductive bias . Differently , we use latent label encodings to model label correlations implicitly . The difference between our work and the previous state-ofthe-art works in modeling label correlations is fundamental , it provides a new line of thought . ( 2 ) Our algorithm outperforms the state-of-the-art results on two widely used benchmark datasets by a large margin . Feature study shows that it significantly outperforms using actual label encodings for classification . Error analysis indicates that the advantage of our algorithm lies in label-correlation utilization rather than document representation . The inspiring results demonstrate the great potential of modeling label correlations implicitly . ( 3 ) Feature study further reveals that contrary to the other token embeddings , sometimes pretraining latent label embeddings can lead to significant performance loss instead of promotion . This suggests that they are more related to task information ( i.e. , the actual labels ) than the other tokens . 2 METHODOLOGY . Given a text T containing m sequential word tokens w0 , w1 , . . . , wm−1 , wi ∈ V ( 0 ≤ i ≤ m− 1 ) , a MLTC task aims to determine a label set LS = { l0 , l1 , . . . , lp−1 } , lj ∈ L ( 0 ≤ j ≤ p− 1 , |L| = n ) , m is the length of the text , LS is the set of labels associated with the text , p is the number of labels in LS , V is the vocabulary of word tokens , L is the complete set of all the candidate labels , n is the cardinality of L. Our method to solve this problem is straightforward : it pads k latent labels ll = ( ll0 , ll1 , . . . , llk−1 ) in front of the input to BERT and maps the encodings of these latent labels to actual labels through a dense neural network . Figure 1 shows its framework . In the following , we describe them in detail . Input As illustrated in Figure 1 , the input is a concatenation of latent labels and text tokens . Specifically , let the latent labels be ll = ( ll0 , ll1 , . . . , llk−1 ) , the tokenized text be ( w0 , w1 , . . . , wm−1 ) , the input then is ( ll0 , ll1 , . . . , llk−1 , w0 , w1 , . . . , wm−1 , [ SEP ] ) . These latent labels do not exist in the vocabulary of BERT ; their initial embeddings are generated randomly . The number k of latent labels is a hyperparameter . Since all these latent labels share the same position embedding 0 , they are a set rather than a sequence . Usually , the number of latent labels is much smaller than that of the actual labels . It makes the input to BERT more concise and easier to train compared with using actual label embeddings . Also , the condensed representations of latent labels makes it easier for them to correlate with each other closely . BERT within-task further pretraining ( optional ) The setting of the BERT part is the same as traditional BERT . It is well known that in-domain further pretraining BERT can enhance its adaptability to the specific contexts . So , we employ an optional within-task further pretraining process that follows the within-task pretraining procedure introduced in ( Sun et al. , 2019 ) . Since the latent labels are task-targeted , their embeddings are not pretrained during this process as other tokens do . As shown by our experiment , pretraining them can decrease performance significantly ( see 4.2 ) . Note that this is only an optional process ; our algorithm can still outperform the state-of-the-art results by a large margin without this process . Output and Loss As illustrated in Figure 1 , let the contextual encodings of these latent labels be Hl = [ hll0 , hll1 , . . . , hllk−1 ] . They are first flattened and then fed to a single layer dense network to obtain the predictions : ŷ = sigmoid ( W To flatten ( Hl ) + bo ) ( 1 ) Wo ∈ Rkd×n and bo ∈ Rn are the trainable weights and bias of the dense neural network respectively , d is the dimension of the encodings , n is the number of actual labels . For the recent state-of-the-art LACO algorithm ( Zhang et al. , 2021 ) , its final prediction on each label is determined only by the encoding of the label itself . On the contrary , our prediction on each label is determined by all the encodings of these latent labels cooperatively . The advantage of our method is twofold . Firstly , it utilizes not only the label encodings but also their correlations . Secondly , the encodings of these latent labels are shared among all the predictions ; compared with predicting each label separately , this general sharing can enhance robustness . Like previous works , we use Binary Cross Entropy as the loss function for the multi-label text classification tasks : Lmltc = − n∑ i=1 [ qi ln pi + ( 1− qi ) ln ( 1− pi ) ] ( 2 ) where pi = P ( yi|D ) is the probability of label i predicted by the model ( yi ∈ L ) , and qi ∈ { 0 , 1 } indicates whether yi ∈ LS .

Summary: This paper consider multi-label text classification problem and propose a cross attention Transformer encoder to model the correlation between latent labels and input text sequence. The hidden states of those latent labels are concatnate as input to layers of MLP for the classification head. The experiment results show marginal gain over the baselines.