DEIM Forum 2023 1a-1-5

Utilizing Keyphrase Generation and Semantic Similarity for Extreme MultiLabel Text Classification
Xiangting DAI†

Mizuho IWAIHARA‡

Graduate School of Information, Production and Systems, Waseda University
2-7 Hibikino, Wakamatsu ku, Kitakyushu-shi, Fukuoka, 808-0135 Japan
E-mail: †daixt@akane.waseda.jp, ‡iwaihara@waseda.jp
Abstract Extreme multi-label classification (XMC) is an important problem of assigning relevant labels to a text from a vast label domain.
The existing XMC methods utilize machine learning or deep learning methods to extract dense representation from the input text and selectlabels from a vast label set. There are two problems in the existing methods: 1) The performance of abstract label extraction is poor, and 2)
statistical sampling of negative labels is reducing the accuracy. Recently, BERT or other pre-trained language models are extensively used asa document encoder for the XMC problem, whereas text2text models have not been used. In this paper, we propose a method based on
generating keyphrases by a text2text language model, rather than retrieving labels from the label set. Keyphrase generation is the process of
predicting both present and absent keyphrases from a given document. The vast majority of generated keyphrases can be found in the wholelabel set, but a portion of generated keyphrases is not in the label set. We consider replacing non-existing labels by existing labels. Finally,
generated candidate labels are ranked for selecting the final labels, where a bi-encoder-based label ranking model is trained with false labels
not in the label set, and produces sentence-pair relevance scores. Our experiments over XMC benchmark collections show notable superioritiesof our approach.

Keyword Extreme Multi-Label Classification, Keyphrase generation, Pre-trained Model, Semantic Similarity

1. Introduction

the success of pre-trained language models like BERT [7],Multi-label text classification is assigning more than one

RoBERTa [22], and XLNet [25], which have a remarkablelabel to a document, which is a common task in real world

performance because of their contextualized semantic

applications,commodity

representation of texts. In various NLP tasks, XMC

recommendation [25] and tagging Wikipedia articles [17].

methods like LightXML [19] utilize pre-trained language

Also, in this scenario the set of possible labels from whichmodels for text representations and dynamic negative label

document labels are chosen can be extremely large, whichsampling to avoid the long tail, sparse label space of XMC.

is called extreme-multi label classification (XMC) [9] [17]Although the methods mentioned above have obtained[19]. In the XMC task, the target document shall besalient results in XMC tasks, we also find that the existing

assigned one or more labels from a numerous label

methods have yet to exploit semantic relatedness of labelcollection. XMC datasets have many labels, ranging fromtexts for XMC tasks. Virtually, most methods directly use

4K in the EUR-Lex dataset and 3M labels in the Amazon3M

BOW features rather than the raw original label texts.

dataset [10].

Meanwhile,

Various methods have been proposed in recent years to

information and large-scale spare vector representations oftackle XMC, including traditional machine learning and

BOW features, the problems of high computing costs andstatistical methods like DiSMEC [18] and FastXML [23].

lack of semantic relations between labels are evident.

such

as

e-commerce

becauseofthesuperficial

semantic

Indeed, with the development of deep learning, more

Nowadays, almost all of existing methods try to utilize a

researches have focused on deep learning methods such as

pre-trained language model to obtain text representations

XML-CNN [9]. However, these methods are commonly

for retrieving labels from the label set. A number ofbased on certain statistical representation model like Bag-

methods that utilize pre-trained language models forof-words (BOW) or shallow neural network models that canobtaining text vectors, like LightXML, merely use theproduce word embeddings like word2vec. On the otherencoder structure of the transformer. Indeed, the pre-hand, recent methods, such as AttentionXML [17], havetrained model composed with a transformer encoder has aused hierarchical label trees to classify candidate labelsbrilliant performance for various NLP tasks. However, theyfrom a large label set. However, these approaches do not

are inapplicable to generation tasks such as summarizationconsider utilizing the raw label texts. In recent years, with

and question answering.
In recent years, with the remarkable progress of the pretrained language models [7], a vast number of sequence-tosequence models have appeared, such as BART [14], which
is trained by corrupting texts with random noise andreconstructing the original document.
In recent years, more and more NLP tasks can beconverted into a unified pattern, converting most of NLP
tasks into the text-to-text formulation. Text-to-text modelT5 [3][12] shows the capacity of reframing all NLP tasks
into a unified text-to-text format. Accordingly, it isfeasible to recall labels from a large label set by a text-to-

Figure 1: the comparison between our method and conventional

text language model. For instance, there exist researches

method.

on keyphrase generation with a text-to-text pre-trained
language model [1][21]. So, we are naturally enlightened

conventional methods are shown in Figure 1.

by this direction. There exists potential for utilizing a text-

We believe the generation model can be finetuned to

to-text pre-trained language model for generating labels forgenerate texts that are semantically close to the predefined

XMC.

labels. Since our method utilizes text information of labels,

To address the abovementioned approaches, we propose a

we can also compute semantic relatedness between label

method based on keyphrase generation and semantic

names.

similarity for XMC tasks. Firstly， we try to use a text-to-

In this paper, we proposed a brand-new method based on

text language model to generate keyphrases from input

keyphrase generation and semantic similarity for the

texts. These keyphrases can be regarded as concise

extreme multi-label classification task. We evaluate ourrepresentations of the input text. Since the text-to-text

model on four public datasets of the XMC task. Comparedmodel may generate keyphrases which do not belong to thewith the state-of-the-art model, the result from ourlabel set, we need to select labels close to these non-proposed method shows a competitive result.

existent labels.
There are two strategies for non-existent labels. First is

2. Related Work

directly removing the non-existent labels. However, this

2.1 Machine Learning method

way may lose the semantic information obtained from thePrevious XMC methods conventionally use machine

text-to-text model. Therefore, our method uses a morelearning techniques to obtain labels from the label set byflexibleto

BOW and TF-IDF features. The enormous number of labelstransform non-existent labels. Sentence-transformer is ameans huge spare vector dimensions. There are three basic

framework based on Sentence-BERT, which can obtain

directions: One-vs-all (OVA) method, label tree-based

embeddings from sentences and calculate semanticalmethod, and embedding-based method. OVA methods likesimilarities between sentences. We use a sentence-

Parable [24], DiSMEC [18], PD-Sparse [6] apply binarytransformer to calculate the similarity between eachclassification tasks to each label. This method has several

generated keyphrase which is not in the label set and eachproblems, such as computing cost and model size.

approachbysentence-transformer[15]label in the label set. Moreover, we replace all non-existent

Another category is the label tree-based method, whichlabels with labels having the highest semantic similarities.

focuses on decreasing computing cost caused by the large

Finally, the candidate labels are ranked to choose the final

label number in the OVA method. The method likepredicted labels.

FastXML [23] and CRAFTML [20] can build a hierarchicalandtree with partitioned labels clusters. The tree-basedclassification tasks into generation tasks by a text-to-text

methods' difficulties are the tree structure's design and how

Ourcontributionisconverting

label

recalllanguage model. On the other hand, our label recalling
method based on keyphrase generation is distinct from anyother existing label recalling method of XMC tasks. The
comparison and difference between our method and otherFigure 2: The overview of our proposed model based on keyphrases generation and semantic similarity.

to build an efficient and accurate classifier for each treemodel. In this paper, text-to-text models are finetuned byleaf node. Most of label tree-based methods use k-means

specific formats and can generate keyphrases by input text.

[18] or other clustering algorithms to partition labels.

The keyphrase format is like a label in the XMC tasks.

The final category is embedding methods, which aim to

2.4 Sentence-transformer

transfer high label dimension into lower dimension space.

Sentence-transformer is an improved model of the pre-Some modified embedding methods like SLEEC [13]trained-BERT model that operates at the sentence levelutilizes label compression and decompression on label[16]. It has been shown to be effective for a wide range ofembeddings for XMC tasks.

tasks, including semantic textual similarity and document

2.2 Deep learning methodclassification. Sentence-transformer can encode sentences'

Deep learning models have proved their capability andsemantics and contextual relationships, resulting in a

potential in various NLP tasks. Compared with the classicalstrong performance on various benchmarks. There are twomachine learning based method, the deep learning method

types of sentence-transformer models: Cross-encoder,

have made significant progress in text information

which can deliver two sentences into a transformer model

extraction and representation. However, they need higher

and generate a value of output as the relevance score of thefloat computing resources like GPU and TPU. Generally,

sentencemachine learning methods will use BOW features as theembedding of sentences, and to cosine similarities of these

representation of input text. In contrast, deep learning

embeddings are used to evaluate semantic relatedness of

methods like CNN-XML and LightXML can apply deep

the sentences.

learning models to represent input texts, which can gather

3. Methodology

more contextualized semantic information.

Figure 2 shows the overview of our proposed new model

2.3 text-to-text language modelfor XMC tasks. This model can be separated into three

The pre-trained language model has become the mostparts: Keyphrase generation, label combining, and label

popular topic in the NLP field for the last few years. Someranking. Firstly, we should train a text-to-text model forexisting methods, Like LightXML, have used pre-trained

keyphrase generation from documents and their groundlanguage models like BERT and RoBRETa for text

truth labels. Next, we utilize a method to resolve the non-representation. On the other hand, the text-to-text models

existent label problem and obtain filtered predicted labels

like BART, T5, and Pegasus have different structures

after the combining part. Finally, we consider using thecompared with each other model. BART uses a denoising

method of semantic comparison to sort the labels from theautoencoder

last step.

architectureandbidirectional

attention

pair,

and

bi-encoder,

which

can

generatemechanisms. T5 can be finetuned for a wide range of

3.1 Problem definition

language tasks. And Pegasus focuses on extracting gap-

Based on the information above about the extreme multi-

sentences from a document with more parameters than thelabel classification, we should use a more specificother two models.

expression to represent it. In XMC tasks, we denote theCurrently, there is no research exploring the possibility of

XMC datasets as 𝐷! . The 𝐷! consists of a set of tuples

a text-to-text model for XMC tasks. Diya, A [1] discussed(𝑑" , 𝐿" ) including documents and their ground truth label

the keyphrases generating tasks by finetuning the BART

sets, where d i is the i-th document and 𝐿" = {𝑙# , 𝑙$ , … , 𝑙" } isthe set of ground truth for i-th document. Otherwise, 𝐷!

each document as the input text and raw label texts of thealso have a whole label set S L that includes all labels thatraw document. The pre-trained text-to-text model can

appear in all training and testing dataset.

predict keyphrases after training processing by our specific

3.2 Keyphrase Generation

data format like Figure 3. In this part, we need to

In This part, we proposed a method that can be performed

preprocess input text data first. The training dataset for thein text2text style downstream task to get some preliminary

text-to-text model should be reformatted into two parts:

keyphrases which can be divided into raw labels in XMC

documents and labels. At the beginning of each document,tasks. These raw labels are close to the real labels rather

we add a ‘summarize:’ prefix to guide the text-to-text

than indexes in the XMC tasks. Based on the real labels

model to generate keyphrases.

pipeline in our method, some datasets of XMC tasks, suchas Amazon-3M, are hard to evaluate because these datasetsneed complete real label information.
We train a standard text2text model for keyphrase
generation tasks. Considering the model size and themasked and reconstruction strategy, we choose BART
model as the primary language model, which has abidirectional encoder and autoregressive decoder [14] andissuitable

for

text

generation

basic

Furthermore, before the training step, we must prepare themathematical expression for keyphrases generation islabels corresponding to each document. So, each training

given a sequence of input variables𝑋 = [𝑥# , 𝑥$ , … , 𝑥% ] and asequence of output
𝑌1 = [𝑦3# , 𝑦3$ , … , 𝑦3& ] based on therecord has a complete raw input text which is adhered to aprobability estimation:

training data, we feed training data into the text-to-text&*#

𝑃5𝑌16𝑋; 𝜃9 = ;"+#where 𝑦3'tasks.

The

Figure 3 an example of keyphrase format for XMC dataset.

𝑃(𝑦3" |𝑦3':&)# , 𝑥#:% ; 𝜃)(1)prefix and a ground truth label set. After combining themodel for training. Otherwise, considering the length of
input text is longer than the max-length of text-to-text

is the <BOS> token and the 𝑦3&*# is the <EOS>

models, we use a default configuration of truncation of

token, 𝜃 is the parameters of the model. Otherwise, the?
loss function is the cross-entropy of predicting sequence 𝒀

text-to-text models rather than modify it for longer text toand ground truth sequence which consists of a set of labels.

trained models can obtain contextualized more semantic

The loss function can be represented by:information and can be easier to recall the abstract labels0

not in the input document but in the ground truth label set/

𝐿 = − A A 𝑀, 𝑦,,. log5𝑦3,,. 9

simplify more models as far as possible. Text-to-text pre-(2),+# .+#of the input text.
The other important point is the configuration ofwhere 𝑇 is the length of ground truth sequence, 𝑁 is theprediction. After training the text-to-text language model

length of predicted keyphrases sequence, 𝑀, is the masked

with our specific training data and settings, we can use thevalue for the position 𝑗, 𝑦,,. is the ground truth sequence,

pre-trained model to generate keyphrases. The keyphrases

𝑦3,,. is the 𝑗-th token of predicted keyphrases sequence.

generated by the text-to-text model contain a few labels

Keyphrases are a word sequence that can be shown thethat are the ground truth labels of input text. However,concise representations of the input texts with different

because of the feature of the text-to-text model, the modelorganizational format compared with the label set of acan generate redundant and very similar labels, which will

XMC dataset. The processing of mapping keyphrases into

influence

XMC labels is:

hyperparameters to avoid a mass of very similar labels in
34%5&2"6# 1$
2{ 𝑌1.1, 𝑌.1 , … , 𝑌1.1
} J⎯⎯⎯⎯⎯L {𝐿1#784 , 𝐿1$784 , … , 𝐿1𝒕784 }

(3)

the

prediction

results.

Weshouldadjustthe predicted keyphrases.
3.3 Label combination

where 𝑌1.1 ={𝑦3# , 𝑦3$ , … , 𝑦3& } is the predicted keyphrases, and𝐿1784 = {𝑙N# , 𝑙N$ , … , 𝑙N&! }, 𝑛 and 𝑛: are the number of predicted

generating part, it is necessary to transform the keyphrases

token of keyphrases and the number of mapped labels, 𝑡 is

into existing labels. The non-existent labels are not uselessthe number of instances in the dataset.

in our results. Compared with redundant and very similar

Because of the text-to-text feature, we need raw text oflabels, the non-existent but different meanings between

Since non-existent labels can be generated in thelabels can express abundant semantic information of input
texts. With this in mind, the new method we proposed isreplacing these labels with the most similar labels in thewhole label set. We call the most similar label as thesimilarity candidate label.
For each non-existed label, we can obtain its label
embedding

𝑢&;&

by

BERT

model.

Meanwhile,theembeddings of whole labels of the dataset can be expressed

Figure 5: How to compute label cosine similarity with label set andas 𝑈< = [𝑢# , 𝑢$ , … , 𝑢/ ], and 𝑁 is the number of labels. The

find most similar label from the whole label set in dataset.

most similar label for one non-existent label is:

In the bi-encoder model, we can first compute all label

𝑙847=564 = argmax5cos(𝑣&;& , 𝑢" )9"∈[#,/](4)embeddings of the label set and use these embeddings to
calculate the cosine similarity with each non-existent label.

Where 𝑣&;& is the embedding of non-existent label and𝑙847=564 is the most similar label that is used to replace a

input text, more than one predicted and combined labels

non-existent label. The cosine similarity cos (𝑣&;& , 𝑈< ) is

can have the same similar candidate label. We need to findthe is the vector product.

the second semantically similar label for these labels to

Figure 4 shows the basic flow of replacement for non-avoid the conflict of label combining.

existent labels. The core of combining is finding the most

3.4 Label Ranking

similar label in the whole label set to replace a non-existentIn the XMC tasks, we need to recall as many ground truthlabel generated by the text-to-text model. Regarding thelabels as possible. So, the accuracy of the labels that arelabel format, we can regard a label as a short sentence thatrecalled is crucial. Our investigation found that thecan easily use an encoder transformer model to obtain

sentence-transformer model’s capability of embedding andembedding vectors and compute the semantic similarity.

semantic similarity can also be used in label ranking.

Due to the enormous number of labels in the datasets ofMaybe this part is close to the original embedding method

XMC tasks, if we compute the semantic similarity of eachwith a pre-trained language model. For each document, welabel with all labels in the dataset, the time cost is

first embed the document and the combined labels. Then

unacceptable. So, we choose the bi-encoder [15] as thewe calculate the cosine similarity between the document

model for computing similarity candidate labels, as shown

embedding and each combined label and rank these labels

in Figure 5.

by the cosine similarity score. This process is similar to the

Furthermore, in a group of generated labels from one

label combing part and can be expressed by:
𝐿185&. = 𝑠𝑜𝑟𝑡5cos5𝑣A;6 , [𝑙N# , 𝑙N$ , … , 𝑙N/ ]99

(5)
N
N
N
where [𝑙# , 𝑙$ , … , 𝑙/ ] are the combined labels that have been
replaced all non-existent labels, 𝐿185&. is ranked by cosine
similarity scores between document and combined labels.
Figure 6 shows an example of the ranking part of our work.

Figure 4: An example of replacement for non-existent labels . Red
labels are ground truth labels of current document, and the blue label‘inkling’ is non-existent label. The red label ‘inklings’ is the replacedlabel from the whole label set. Black labels are negative labels.

Figure 6: A example of ranking part. Each combine labels arecomputed the semantic similarity with input document by sentencetransformer. Red means this label is the ground truth label for thisdocument, black labels are negative labels.

Training data for the ranking model has three parts: 1. Raw
input document, 2. combined labels, and 3. each score of

4.3 Experiment Settings

combined labels. Instead of using labels from keyphrase

We try more than one text-to-text models for XMC tasks

generation, we use the labels from combined labels because

in the keyphrase generation part. However, depending on

the labels of the generating part have many kinds of noise,differentsuch as replicative labels and non-existent labels, which

different performances. We utilize BART-base [14], T5[3],will disturb the accuracy of the sentence-transformer

and Pegasus [12] as the text-to-text language model withmodel.

5e-5 learning rate, 2(large size) or 4(base size) training

datasets,

varioustext-to-text

modelshave

Unlike the combining part, we can train the sentence-

batch size, and five training epochs for all datasets. As fortransformer with negative samples. As for the training part,the combining part, we chose a bi-encoder as the model,

the simplest method is assigning the ground truth label 1

which is used to compute the embedding of labels.

score and the negative sample label score 0.

Regarding the ranking part, because of the requirement ofsupervised data for training bi-encoder, we must use labels

4. Experimentsthat are predicted by the pre-trained text-to-text model with

4.1 Evaluation Metrics

training documents. We choose all-MiniLM-L6-v2 as theRegard to XMC tasks, the P@k (Precision at k) is a verybi-encoder model with 64 batch size. We also choose cross-

explicit value for evaluating model performance. In our

encoder/stsb-RoBERTa-base as the cross-encoder model

works, the P@k value can be defined as:with 24 batch sizes in our experiments. Both two models

𝑃@𝑘 =

1
𝑘

are trained with four epochs.

.

A

(6)

𝑦=

Considering the time cost, for Wiki500K and AmazonCat13K, we do not use all training data to train the text-to-text=∈85&."(CD)is the true labels value, 𝑦3 is the final

model and ranking model. We randomly select 5% train

output of a model. Also, 𝑙 is the index in the top-k output

data of Wiki500K and 10% train data of AmazonCat-13K

labels.

to train the text-to-text models and ranking models.

where 𝑦 ∈{0,1}<

4.2 Datasets
Since we need to compute the semantic similarity of

5. Results and Analysis

labels with each other, it is necessary to obtain the original

We evaluate our model on four public datasets with raw

raw label texts and document texts, rather than the label

documents

index and BOW features given in XMC datasets. The raw

AmazonCat-13K, and Wiki500K. Our raw document texts

input documents of XMC datasets can be found easily, but

come from AttentionXML [17]. The raw label texts of

the raw label texts are hard to be found. Almost all of raw

AmzaconCat-13K, Wiki500K, and Wiki10-31K are also

documents of XMC datasets can be found, but a significant

from Attention XML. The raw label texts of EUR-Lex are

portion of XMC datasets provide raw documents andfrom (Johannes et al. 2010) [5].

andlabels:

EUR-Lex

4K,

Wiki10-31K,

numerical label indexes instead of real labels. Fortunately,

The performance metric is P@k (Precision@k), a simple

some datasets, like EURLex-4K [4], Wiki10-31K [5],and valuable evaluation metric used in XMC tasks. Table 2

AmazonCat-13K [10], and Wikipedia-500K [10], provide

shows the performance of our method in XMC tasks andthe raw label texts. Therefore, our main experiments will

the comparisons with other methods.

be performed on these four datasets. Table 1 shows the

Since there is no previous method based on a text-to-text

statistics of the datasets from AttentionXML [17].

model of keyphrase generation for XMC tasks, we compare

Table 1: The statistics of four datasets. N t r a i n and N t e s t mean the

our method with DiSMEC, AnnexML, Parabel, XML-CNN,

number of instances of training set and testing set. N l a b e l refers toAttentionXML, and LightXML. The results of all thethe total number of labels in dataset. D t f i d f : the dimension of TF-

baselines

are

from

AttentionXML

andLightXML.

Considering the fair comparison, all the results on our

IDF feature.

NtrainNtest

NlabelDtfidfmethod in table 2 are by facebook/bart-base model except

EUR-Lex

15,449

3,865

3,956

186,104

Wiki500(using Pegasus), because BART with bi-encoder

Wiki10-31K

14,146

6,616

30,938

101,938

obtained the high score in Wiki10-31K, and AmazconCat-

AmazonCat-13K

1,186,239

306,782

13,330203,882

13K.

WIki500K

1,779,881

769,421501,008

2,381,304

We also compared with an up-to-date XMC research XRR

[11], which is a complex system and uses semantic andHowever, the performance of AmazonCat-13K could bestatistics information for XMC tasks. Although XRR

better because of its different categories. Simultaneously,

method provide overwhelming performance, our method

our method is close to the new deep-learning-based

shows a highest score of P@1 in wiki10-31K.

methods like AttentionXML and LightXML.

Our method outperformed the almost traditional one-vsall and word embedding methods in EUR-Lex, Wiki10-31K,

5.1 Difference combination of keyphrases generationand Wiki500K. Moreover, our method shows the great

and label ranking.

performance in P@3 of Wiki500K rank only second to XRR.

In relative work, we have shown the difference between

Table 2: The comparison with the results of all baseline and our
method.

BART, T5, and Pegasus. We also run different models,consisting of different text-to-text models and sentencetransformers in EUR-Lex 4K. The results are given
in Table 3. Pegasus with cross-encoder obtained thehighest score of P@1, but the P@3 and P@5 are not thebest scores. T5 with cross-encoder and Pegasus obtained
the highest score of P@3 and P@5 separately. However,Pegasus (with 568M parameters) has a longer training
time and larger model size compared with BART-base andT5-base.
Table 3: The comparison with the results of all baseline and our
method. Cross-encoder: using cross-encoder to replace bi-encoder
ranking part.

EUR-Lex

P@1

P@3

P@5

BART

84.09

69.31

53.07

BART+cross-encoder

84.11

69.84

53.16

T5

83.54

70.22

55.19

T5+cross-encoder

84.91

71.40

55.32

Pegasus

83.70

69.14

56.67

Pegasus+cross-encoder

85.25

70.77

55.92

6. Conclusion and Future Work
We proposed a brand-new method for extreme multi-label
classification tasks in this paper. Compared with existing
methods, there are two innovative points in our method.
The first is using keyphrase generation to replace classicallabel recalling. The second is utilizing the semantic
representation to evaluate the relevance between labels andthe original document based on raw label texts. We
evaluate our method with various combinations in four
public datasets of the XMC task. Our method's results
outperformed most of traditional XMC methods and wereclose to SOTA. Compared with SOTA, our method utilizes
a different way of label recalling, and the subsequent
processing combining and ranking parts are also based on
semantical similarities.
In future work, we can try more modified and customized
text-to-text models for keyphrase generation, to enhance
the performance of keyphrase generation. On the other
hand, the ranking performance still has an ample room forimprovement.

7. Acknowledgment
Our work got help from Hao Shen, who have graduatedfrom Iwaihara Lab, Waseda University and provided apartial idea and prototype of our work.

References[1] Diya, A. and Mizuho, I., 2022. Keyphrase generation
by utilizing BART finetuning and BERT-basedranking. In DEIM Forum.
[2] Zubiaga, A., 2012. Enhancing navigation on
wikipedia
withsocialtags. arXiv
preprint
arXiv:1202.5469.
[3] Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang,
S., Matena, M., Zhou, Y., Li, W. and Liu, P.J., 2020.
Exploring the limits of transfer learning with a unified
text-to-texttransformer. J.
Mach.
Learn.
Res., 21(140), pp.1-67.
[4] Loza Mencía, E. and Fürnkranz, J., 2008, September.
Efficient pairwise multilabel classification for largescale problems in the legal domain. In Joint European
Conference on Machine Learning and Knowledge
Discovery in Databases (pp. 50-65). Springer, Berlin,
Heidelberg.
[5] Loza Mencía, E. and Fürnkranz, J., 2010. Efficient
multilabel classification algorithms for large-scale
problems in the legal domain. In Semantic Processingof Legal Texts (pp. 192-215). Springer, Berlin,
Heidelberg.
[6] Yen, I.E.H., Huang, X., Ravikumar, P., Zhong, K. andDhillon, I., 2016, June. Pd-sparse: A primal and dual
sparse approach to extreme multiclass and multilabel
classification.
In International
conferenceonmachine learning (pp. 3069-3077). PMLR.
[7] Devlin, J., Ming-Wei C., Kenton L., and Kristina T.
"Bert: Pre-training of deep bidirectional transformersforlanguage
understanding." arXiv
preprint
arXiv:1810.04805 (2018).
[8] Hartigan, J.A. and Wong, M.A., 1979. Algorithm AS
136: A k-means clustering algorithm. Journal of theroyal statistical society. series c (applied
statistics), 28(1), pp.100-108.
[9] Liu, J., Chang, W.C., Wu, Y. and Yang, Y., 2017,
August. Deep learning for extreme multi-label textclassification.
In Proceedings
of
the
40th
international ACM SIGIR conference on research anddevelopment in information retrieval (pp. 115-124).
[10] McAuley, J. and Leskovec, J., 2013, October. Hidden
factors and hidden topics: understanding rating
dimensions with review text. In Proceedings of the
7th ACM conference on Recommender systems (pp.
165-172).
[11] Xiong, J., Yu, L., Niu, X. and Leng, Y., 2023. XRR:
Extreme multi-label text classification with candidate
retrieving
anddeepranking. Information
Sciences, 622, pp.115-132.
[12] Zhang, J., Zhao, Y., Saleh, M. and Liu, P., 2020,
November. Pegasus: Pre-training with extracted gapsentencesforabstractive
summarization.
In International Conference on Machine Learning (pp.
11328-11339). PMLR.
[13] Bhatia, K., Jain, H., Kar, P., Varma, M. and Jain, P.,
2015. Sparse local embeddings for extreme multilabel classification. Advances in neural information
processing systems, 28.
[14] Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M.,
Mohamed, A., Levy, O., Stoyanov, V. and Zettlemoyer,L., 2019. Bart: Denoising sequence-to-sequence pretraining for natural language generation, translation,and comprehension. arXiv preprint arXiv:1910.13461.
[15] Reimers, N. and Gurevych, I., 2019. Sentence-bert:

Sentenceembeddings
usingsiamesebertnetworks. arXiv preprint arXiv:1908.10084.
[16] Dekel, O. and Shamir, O., 2010, March. Multiclassmultilabel classification with more classes thanexamples.
In ProceedingsoftheThirteenthInternational Conference on Artificial Intelligenceand Statistics (pp. 137-144). JMLR Workshop andConference Proceedings.
[17] You, R., Zhang, Z., Wang, Z., Dai, S., Mamitsuka, H.
and Zhu, S., 2019. Attentionxml: Label tree-based
attention-aware deep model for high-performanceextreme multi-label text classification. Advances in
Neural Information Processing Systems, 32.
[18] Babbar, R. and Schölkopf, B., 2017, February. Dismec:
Distributed sparse machines for extreme multi-label
classification. In Proceedings of the tenth ACM
international conference on web search and data
mining (pp. 721-729).
[19] Jiang, T., Wang, D., Sun, L., Yang, H., Zhao, Z. andZhuang, F., 2021, May. Lightxml: Transformer withdynamic negative sampling for high-performanceextrememulti-label
textclassification.
In Proceedings of the AAAI Conference on Artificial
Intelligence (Vol. 35, No. 9, pp. 7987-7994).
[20] Siblini, W., Kuntz, P. and Meyer, F., 2018, July.
Craftml, an efficient clustering-based random forest
for extreme multi-label learning. In International
Conference on Machine Learning (pp. 4664-4673).
PMLR.
[21] Ju, Y. and Iwaihara, M., 2022. Unsupervised
Keyphrase Generation by Utilizing Masked Words
Prediction and Pseudo-label BART Finetuning.
In International Conference on Asian Digital
Libraries (pp. 21-34). Springer, Cham.
[22] Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen,
D., Levy, O., Lewis, M., Zettlemoyer, L. and Stoyanov,V., 2019. Roberta: A robustly optimized bert
pretraining
approach. arXiv
preprint
arXiv:1907.11692.
[23] Prabhu, Y. and Varma, M., 2014, August. Fastxml: A
fast, accurate and stable tree-classifier for extreme
multi-label learning. In Proceedings of the 20th ACM
SIGKDD international conference on Knowledge
discovery and data mining (pp. 263-272).
[24] Prabhu, Y., Kag, A., Harsola, S., Agrawal, R. andVarma, M., 2018, April. Parabel: Partitioned label
trees for extreme classification with application to
dynamic search advertising. In Proceedings of the2018 World Wide Web Conference (pp. 993-1002).
[25] Yang, Z., Dai, Z., Yang, Y., Carbonell, J.,
Salakhutdinov, R.R. and Le, Q.V., 2019. Xlnet:
Generalized autoregressive pretraining for language
understanding. Advances in neural information
processing systems, 32.