Fast Multi-Resolution Transformer Fine-tuning for Extreme Multi-label Text Classification
Authors: Jiong Zhang
Wei-Cheng Chang
Hsiang-Fu Yu
Inderjit S Dhillon
Venue: NeurIPS 2021 Poster (2021)
Keywords: transformers
extreme multi-label text classification

Abstract: Extreme multi-label text classification~(XMC) seeks to find relevant labels from an extreme large label collection for a given text input. Many real-world applications can be formulated as XMC problems, such as recommendation systems, document tagging and semantic search. Recently, transformer based XMC methods, such as X-Transformer and LightXML, have shown significant improvement over other XMC methods. Despite leveraging pre-trained transformer models for text representation, the fine-tuning procedure of transformer models on large label space still has lengthy computational time even with powerful GPUs. In this paper, we propose a novel recursive approach, XR-Transformer to accelerate the procedure through recursively fine-tuning transformer models on a series of multi-resolution objectives related to the original XMC objective function. Empirical results show that XR-Transformer takes significantly less training time compared to other transformer-based XMC models while yielding better state-of-the-art results. In particular, on the public Amazon-3M dataset with 3 million labels, XR-Transformer is not only 20x faster than X-Transformer but also improves the Precision@1 from 51% to 54%.

Fast Multi-Resolution Transformer Fine-tuning for Extreme Multi-label Text Classification
Jiong Zhang Amazon j giongz0amazon. com
Wei-cheng Chang Amazon ChanweicOamazon. com
Hsiang-fu Yu
Amazon
rofu.yu@gmail, com
Inderjit Dhillon UT Austin & Amazon inderjit@cs utezas. edu
Abstract
Extreme multi-label text classification (XMC) seeks to find relevant labels from extreme large label collection for a given text input. Many real-world applications can be formulated as XMC problems, such recommendation systems, document tagging and semantic search. Recently, transformer based XMC methods, such as X- Transformer and LightXML,, have shown significant improvement over other X.CC methods. Despite leveraging pre-trained transformer models for text representation, the fine-tuning procedure of transformer models on large label space still has lengthy computational time even with powerful GPUs. In this paper, we propose novel recurxive approach, Tr-Trrnsformerr accelerate the procedure through recursively fine-tuning transformer models on series of multi-resolution objectives related to the original XMC objective function. Empirical results show that XR- Transformer takes significantly less training time compared to other transformer- based XMC models while yielding better state-of-the results. particular, on the public Amazon-3M dataset with million labels, XR-T TR-Transformer is not only 20x faster than X- X-Transformer but also improves the Precision @ from 51% 54%. Our code is publicly available at https: h github....... com/ coonn/mmn/ppa amzn/pecos
Introduction
Many real-world applications such as open-domain question answering [12) e-commerce dynamic search advertising [3] 4 and semantic matching [5], can be formulated an extreme multi-label text classification (XMC) problem: given a text input, predict relevant labels from an enormous label collection of size In these applications, L ranges from tens of thousands millions, which makes it very challenging to design XMC models that are both accurate and efficient to train. Recent works such as Parabel B], Bonsai (6). XR-Linear [7] and AttentionXML g exploit the correlations among the labels to generate label partitions or hierarchical label trees (HLTs) which can be used to shortlist candidate labels to considered during training and inference. While these methods are scalable in terms of the size of the label collection, most of them rely only on statistical representations (such as bag-of-words) or pooling from pre-generated token embeddings (such word2vee) to vectorize text inputs.
In light of the recent success of deep pretrained transformers models such as BERT 9 XLNet [IO
propose to fine-tune pre-traine trained transformer models on XMC tasks to obtain new state-of-the--e results over the aforementioned approaches. Although transformers are able to better capture semantic meaning of textual inputs than statistical representations, text truncation often needed in practice to reduce GPU memory footprint and maintain model efficiency. For example, X-Transformer truncates input texts
and RoBerta [1] in various NLP applications, X-Transformer [2] and LightXML 1
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

to contain the first 128 tokens before feeding it into transformer models. Efficiency of transformer fine-tuning poses another challenge for XMC applications. Directly fine-tuning transformer models on the original XMC task with very large label collection is infeasible as both the training time and the memory consumption are linear in L. In order to alleviate this, both X-Transformer and LightXML adopt a similar approach to group I. labels into K clusters of roughly equal size denoted by B and fine-tune transformers on the task to identify relevant label clusters (instead of labels themselves). B R vL and K R v L, then both the training time and the memory requirement of the fine-tuning can be reduced to ((VT.) from (T.). However, as pointed out in [8], the model performance would deteriorate due to the information loss from label aggregation. Thus, both X- Transformer and LightXML still choose a small constant B ( S 100) as the size of the label clusters. As result, transformers are still fine-tuned on task with around L/ 100 clusters, which leads to a much longer training time compared with non transformer based models. For example, it takes X- X-Transformer 23 and 25 days respectively to train Amazon -M and Wiki-500K even with 8 Nvidia V100 GPUs.
To address these issues, we propose XR- TR-Transformer, an XMC architecture that leverages pre-trained transformer models and has much smaller training cost compared to other transformer-based XMC methods. Motivated by the multi-resolution learning in image generation [14 and curriculum learning 17], we formulate the XMC problem as a series of sub-problems with multi-resolution label signals and recursively fine-tune the pre-trained transformer on the coarse-to-fine objectives. In this paper, our contributions are as follows:
We propose XR-Trans Transformrr a transformer based framework for extreme multi-label text classification where the pre-trained transformer is recursively fine-tuned on a series of easy- to-hard training objectives defined by a hierarchical label tree. This allows the transformers to be quickly fine-tuned for a XMC problem with a very large number label collection progressively.
To get better text representation and mitigate the information loss in text truncation for transformers, we take into account statistical text features in addition to the transformer text embeddings in our model. Also, a cost sensitive learning scheme by label aggregation is proposed to introduce richer information on the coarsified labels.
We conduct experiments on 6 public XMC benchmarking datasets and our model takes significantly lower training time compared to other transformer- based XMC models to yield better tfate-of-the-ee results. For example, we improve the state-of af-thhe-rt Pree@ result Amazon-3M established by X-Transformer from 51.20% to 54.04% while reducing the required training time from 23 days to 29 hours using the same hardware.
2 Related Works
Sparse Linear Models with Partitioning Techniques. Conventional XMC methods consider fixed input representations such sparse TF-IDF features and study different partitioning techniques or surrogate loss functions on the large output spaces to reduce complexity. For example, sparse linear one-versus-all (OVA) methods such DiSMEC [8]. PPD-Sparse [19 20). ProXMI (21
explore parallelism to solve OVA losses and reduce the model size by weight truncations.
The inference time complexity of OVA models is linear in the output space, which can be greatly improved by partitioning methods or approximate nearest neighbor (ANN) indexing on the label spaces. Initial works on tree-based methods 22 23 reduce the OVA problem to oneveerses-oome (OVS) with logarithmic depth trees. Down that path, recent works on sparse linear models including
Parabel (3. eXtremeText (24). Bonsai (6), XReg (25), NAPKINXC 226 27 and XR-Linear 77] partition labels with B B-array hicrarchical label trees (HLT), leading to inference time complexity that is logarithmic in the output space. On the other hand, low-dimensional embedding- based models often leverage ANN methods to speed up the inference procedure. For example, AnnexML. (28) and SLICE (29 consider graph-based methods such HNSW 330 while GLaS [3 considers product quantization variants such as ScaNN (32]
Shallow saabedding-based Methods. Neural- XMC models employ various network archi- tectures to learn semantic embeddings of the input text. XML-CNN [33 applies one-dimensional CNN on the input sequence and use the BCE loss without sampling, which is not scalable to XMC problems. AttentionXMI. [8] employs BiI. and label-aware attention as scoring functions. For
better scalability to large output spaces, only small number of positive and hard negative labels
2

---

are used in model GPU training. Shallow embedding based methods [34 38 use word embedding lookup followed by shallow MLP layers to obtain input embeddings. For instance, MACH 34
learns
MLP layers on several smaller XMC sub-problent induced by hashing tricks on the large label space.
Similarly, De [35] and its variant (i.e., DECAF (6)., GalaXC (37, ECLARE 3X) pre-train MLP encoders on XMC sub-problem induced by label clusters. They freeze the pre-trained word embedding and learn another MLP layer followed by a linear ranker with sampled hard negative labels from HNSW [30]. Importantly, shallow embedding- methods only show competitive performance on short-text XMC problems where the number of input tokens is small
Deep Transformer Models. Recently, pre-trained Transformer models 1911 have been applied to XMC problems with promising results [12] [3 39I. X- X-Transformer [12] considers a two-stage approach where the first stage transformer-basee duced by balanced label clusters, and the second stage sparse TF-IDF is combined with the learned neural embeddings the imput to linear OVA models. APLC-XLNet 39 fine-tunes XLNet encoder on adaptive imbalanced label clusters based on label frequency similar to Adaptive Softmax [40]. LightXML [13 fine-tunes Transformer encoders with the OVA loss function end-to-end via dy- namic negative sampling from the matching network trained on label cluster signals. Nonetheless, Transformer-based XMC models have larger model size and require longer training time, which hinders its practical usage on different downstream XMC problems.
encoders are Iearned on XMC sub-problems in-
3 Background Material
We assume we are given a training set yx) where x; E D is the ith input document and Y: {0, 134 is the one hot label vector with yi, indicating that label é is relevant instance i. The goal of eXtreme Multi-label Text Classification (XMC) is to learn function f : D x [L] + R, such that f(x, E) denotes the relevance between the input x and the label In practice, labels with the largest k values are retrieved as the predicted relevant labels for given input The most straightforward model is one-versus-all (OVA) model:
f(x, f) = w (xx; & E [L],
(1)
where W [WI...., WL] RdxL are the weight vectors and Φ D Rd is the text vectorizer that maps x to d-dimensional feature vector. (-)) could be a deterministic text vectorizer, such as the bag-of-words (BOW) model or Term Frequency-Inveee vectorizer with learnable parameters. With the recent development in deep learning, using pre-traine trained transforner as the text vectorizer has shown promising results in many XMC applications [12 [3] 39). When L is large, however, training and inference of OVA model without sampling would be prohibitive due to the (L) time complexity.
Document Frequency (TFIDF) model, or
To handle the extremely large output space, recent approaches partition the label space to shortlist the labels considered during training and inference. In particular, 12 13 [4]
35 39 follow three
stage framework: partitioning, shortlisting, and ranking. First, label features are constructed to group labels into K clusters C E {0, 1] where Cr, cluster. Then a shortlisting model learned to match input x to relevant clusters in an OVA setting:
denotes that label é is included in the k-th
g(x, k) = W 4 (x); k E [K].
(2)
Finally, a classification model with output size L is trained on the shortlisted labels:
f(x, f) = w (x); l E S,(x),
(3)
where Sy (x) C [L] is the label set shortlisted by g(x, .). In the extreme case where only one label cluster is determined to be relevant to a input x, the training and inference cost x would be O(K + )). which in the best case scenario is O(VL) when K VL.
For transformer based methods, the dominant time the evaluation of (x)). But K being too big or too small could still be problematic. Empirical results show that the model performance deteriorates when clusters are too big [8] This is because that the signals coming from B labels within the same cluster will be aggregated and not distinguishable, where B is the cluster size. Therefore, B cannot be too big to ensure a reasonable label resolution for fine-tuning. Also, as pointed out in [2), fine-tuning transformer models on large output spaces can be prohibitive. As result, the label clusters need to
3

---

Fast Fine-tuning Low label resolution
XR- ra-nTrnsfrrmrr
Slow Fine-tuning High label resolution
10
10
Label Resolution (in log-scale)
X-Transformer
No-fine-suring
OVA Transformer Classifier
LightXML
Figure 1: Illustration of fine-tuning on XMC tasks of different label resolutions. For an XMC task with low label resolution, fine-tuning can be fast but model performance might deteriorate due to large deviation from the original XMC task. In practice, X- X-Transformer and LightXML. adopt XMC task with a relatively higher label resolution ensure reasonable model performance at the cost of longer training time. The proposed XR- Re-Transformer leverages multi-resolution learning and model bootstrapping that achieves both fast fine-tuning and good model performance.
be constructed in way to balance the model performance and fine-tuning efficiency. In practice, both the transformer-based XMC models, such as X- K-Transformer and LightXML. adopt a small fixed constant the cluster size B (s 100), which means that training the shortlisting model g(x, k)) is still very time consuming as the number of clusters z L/B.
Proposed Method: XR-Transformer
As noted above, the shortlisting problem 2 is itself XMC problem with slightly smaller output size 1 where the cluster size. In XR-Transformer, we apply the same three stage framework L recursively on the shortlisting problem until a reasonably small output size is reached BD. We can therefore follow the curriculum learning scheme and fine-tune the pre-trained transformers progressively on the sub-XMC problems with increasing output space { BD fine-tuning task, the candidate label set is shortlisted by the final model at the previous task. The recursive shortlisting ensures that for any input, the number of candidate labels to include in training and inference is ((B) and therefore the total number of considered labels O(B log (L)). Also, we leverage the multi-step fine-tuning and use the embedding generated at the previous task to bootstrap non pre-trained part for the current task. We now describe the model design detail.
J. I ' .J. At each
Hierarchical Label Tree (HLT). Recursively generating label clusters D times is equivalent to building HLT [41] of depth D. We first construct label features Z I. [xxd This could be done by applying text vectorizers on label text or from Positive Instance Feature Aggregation (PIFA):
Ze = |/vell
Ve
where v.
Σ ((x)), i
()
where 4 D + Rd it the text vectorizer. Then we follow similar procedures as S and 13 and use balanced k-means (k B) to recursively partition label sets and generate the HLT top-down fashion. The HLT is represented with series of indexing matrices (C(t) ) such that
C( (0, IjKi where Ko and Kn L. Equivalently, once (D) constructed, the HLT can be built from bottom up through joining B adjacent clusters together.
Multi-resolutie Output Space. Multi-resolution learning has been explored in different contexts such as computer vision [4 For instance, using an output scheme with coarse-to- resolutions results in better image quality for generative adversarial networks [15] [6 As another example in meta learning, [43] learns multiclass models via auxiliary meta classes by collapsing existing classes. Nevertheless, mulli-rrsolution learning has not been well-explored in the XMC literature. In XR- Transformer, we leverage the label hierarchy defined by the HLT and train the transformer model on multi-resolltion objectives.
The XMC task can be viewed as generating an 1-D image y E {0, 13 with binary values based on input text Just like coarsified image could be obtained by a max or mean pooling of nearby pixels,

---

the coarse label vector can be obtained by max poollin of labels which are nearby in the label feature space. Once the HLT is constructed using label features, the true labels at layer Y E {0, 1 K. can be determined by the true labels of the child clusters t through max-pooling like operation:
binarize (1-1))
(5)
1, and 1/( = ye is the original label matrix. This forms a series of learning signals with coarse-to-fine
resolution and can be used to generate learning tasks with easy-to-hard objectives.
Direct use of the binarized Y E {0, 1] 1x in Eq (5) results in information loss when merging several positive labels into one cluster. Ideally, a cluster containing several positive children is more relevant than a cluster with only one positive child. To add this lower level information to higher level learning objectives,
introduce the relevance matrix R(O E R N. K; for layer of the XMC sub-problem where R defines the non-negative important weight for ith instance to 2th cluster. Different from cost-sensitive learning 44 for MLC (CSMLC) setting 45 where there is only one cost matrix explicitly derived by evaluation metrics such as Fl score, in KR-Transformer, we consider the usage of cost-sensitive learning where the relevance matrices are recursively induced by the HLT structure. Specifically, given an HLT, we recursively construct relevance matrices for 1= 1,.....1 D:
R R C(*+)
and (D) = Y()), Motivated by [48] we adopts the row-wise & normalized relevance matrix:
R
y 1. 1,
if
R R lh
a
otherwise,
where a is the hyper parameter to balance positive and negative weights.
Label Shortlisting. During training. XR- ER-Transformer only focuses on discriminating the labels or clusters that have high chance of being positive. A necessary condition for label layer to be positive is that its parent label at level / - 1 is positive. Therefore, an intuitive approach would be to only train on the output space shortlisted by positive clusters of the parent layer. However, in practice we found this approach sometimes leads to sub-optimal result during inference with beam search. As an effort to balance explore and exploit, we further include the top-k relevant clusters determined by the model learned on the parent layer to mimic the beam search during inference. Thus at layer t, the labels considered during training are shortlisted by the parent layer f 1:
(t-11) = Top (t) T (X, ((t-1)), K),
(7)
( (t) binarize ((tri) C(OT) binarize Y ((O))
(8)
where the Top(:; k) operator zeros out elements in matrix except the top-k largest values in each row. For each instance x,, only non-zero indices of M, will be included into the training objective.
We can therefore define a series of learning objectives for level & E (1, D} as:
min W00 ,9
Σ
CY (00 ie w!!T Φ(x;, θ)) + 1
where is a point-wise loss such as hinge loss, squared hinge loss or BCE loss, W(!), θ are the model weights to be Iearned.
Text Representation. Most previous works on XMC construct text feature representation in one of two ways: statistical feature representations and semantic feature representations. Although the latter, in particular transformer models, have shown promising results on various NLP benchmarks, the self-attention mechanism makes transformers unscalable w.r.t. sequence length. To ensure efficiency, input texts are usually truncated [12 13 which result in loss of information. On the other hand, the statistical features, such as TFIDF, are fast to construct with the whole input taken into consideration. In XR-Transformer, we use a combination of these two feature representations and each component a complement of lost information for the other one:
(x) Ann (x, +) T Leat (x, θ) = [ (x)]l'
(10)
[ Dd (x, +)
5

---

Algorithm 1: Iterative Le,
Input :X, Y, C, 0, P
m, f- C.shape
if then
M omes(N, m)
if 0 is not fixed then
0" Optimize
9 with & Ddnnn initialize + 0
else
L 0"
W + argminw E Et=
((0) Ri L(Yre, We T Peat (x,, + W|]
else
prev f binarize YC)
M binarize (anarize(PCC] (PC ) binarize (Y pren CT)
if 0 is not fixed then
0- Optimize with 4 = Pann initialize é =
else
L 0" -
W argminw E EeM R0 W t coo (x,, 03)) A/|
Return: : W
Algorithm 2: XR- TR-Transformer training
Input :X, Y, pre-trained transformer & Pdnn (-. 0p)
Z, = v |lel: where v. = Exm.e= Pefsel (x,)), V e [4]
{C + k-mmean--lusteringgZZ Z)
Generate label hierarchy (Y)( J using
0" = 0o. P None
for t 1.2.3,.... D do
W, 0* + Jeerative_Larn(X Y C(9,0" 0", P)
P + Top(W catt X, k)
Z, ve ll where v, Σ =1 4 ont (x) [L]
(C(0)2 k-means-clustering(())
Generate label hierarchy (Y P. using
Fix 0" P None
for t 1, 2. D do
(t). &- Iterative LeannXX, y() 0r,
P - TTp(W(() Deat (x)),,)))) k))
Return & ", (C(0) (W(1)))
where Lenn , θ) is the transformer parametrized by θ. Once the text representation is constructed, predictions be made by simply applying linear projection top of the text representation through
Training with bootstrapping. The training of XR- TR-Transformer consists of three steps. At first, preliminary HLT is constructed using raw statistical features. Then a pre-trained transformer model is fine-tuned recursively from low resolution output to high resolution. At each layer i, fine-tuning objective is optimized with initialization θ (t-1))- the best transformer weights of layer t 1. (0)) denotes the pre-trained transformer weights.
Unlike the transformer warmed-up with pre-trained weights, the projection weights (()) is trained from scratch without good initialization. At the beginning of fine-tuning, gradient flow through these cold-start (usually randomly initialized) weights will usually worsen the pre-trained components.

---

We leverage the recursive learning structure to tackle this issue by model bootstrapping. Concretely, w(1) is initialized as:
w Frest
argmin
Σ E.M L.
R C
w dann (X, (((-1)())) + 1/
(11)
In practice, [1]) is fast to compute since the semantic text feature for the previous layer Dear [X, (((-1)+) is already computed and thus (TT) can be solved very quickly on CPUs with variety of parallel linear solvers, such as LIBLINEAR [49].
Once the fine-tuning complete, the refined HLT constructed with the text representation that combines statistical text feature and fine-tuned semantic text embeddings. Then the ranking models are trained on top of the combined text features for the final prediction. The detailed training procedure is described in Algorithm and
'he inference cost of KR-Transformer consists mainly of two parts: cost to compute transformer embedding and to retrieve relevant labels through beam search. Therefore, the inference time complexity is (T dimension and Idan is the time to compute & Pann (x) for given input. Note that even the inference done with beam search through the refined HLT, the transformer text embedding only need be computed once per instance.
Inference.
kd log(L)), where l is the beam size, is the concatenated feature
Connections with other tree based methods. Although methods such AttentionXML g also train on supervisions induced by label trees, the final model is a chain of sub-models which each on is learned on single-resolution. In particular, given hierarchical label tree with depth D, AttentionXXIL will train D different text encoders on each layer of the tree where as XR-Transformer trains the same transformer encoder progressively all layers of the tree. This difference leads longer inference time for AttentionXML than XR-Transformer since multiple text encoders need to be queried during inference, shown in the comparison in the inference time in Appendix A.4.2
5 Experimental Results
We evaluate IR-Transformer on public XMC benchmarking datasets: Eurlex-4K. Wiki10-31K. AmazonCat-13Ke Wiki-500K, Amazon-670K, Amazon-3M. Data statisties given in Table 1 For fair comparison, we use the same raw text input, sparse feature representations and same train-test split AttentionXML [ and other latest works [12 [3] The evaluation metric is Precision @k (P@k), which is widely-used in XMC literature Precision @k (PSP@) defer to Appendix A.4.3 which focus more tail labels' performance.
8
I 28 The results of Propensity-scon score
Table 1: Data statistics. Ntrain Neett refer to the number of instances in the training and test sets, respectively. L: the number of labels. L: the average number of positive labels per instance. average number of instances per label. fift:; the sparse feature dimension of &ifidf (.). These six publicly available benchmark datasets, including the sparse TF-IDF features are downloaded from httpss: : //github. com/ haamm/yorrh/aae AttentionXMi. which are the same as AttentionXML. B X- Transformer [12] and LightXML [13] for fair comparison.
Dataset
Ntrain
Ntest
L
L
defidf
Eurlex-4K
15,449
3,865
3,956
5.30
20.79
186,104
Wiki1 0-KKK
14,146
6,616
30,938
18.64
8.52
101,938
AmazonCat-190 3K
1.186,239
306,782
13,330
5.04
448.57
203,882
Wiki-500K
1,779,881
769,421
501,070
4.75
16.86
2,381,304
Amazon-670K
490,449
153,025
670,091
5.45
3.99
135,909
Amazon-3M
1,717,899
742,507
2,812,281
36.04
22.02
337,067
Baseline Methods. We compare XR-Transformer with state-of-the-art (SOTA) XMC methods:
AnnexMI. 28), DiSMEC [18), PfastreXML. 41, Parabel 3 eXtremeTTxx 24, Bonsai (50) XMI.-
CNN 33 XR-Lincar 77, AttentionXML
8 X- Transforrerr [2] and LightXML [3] We obtain
7

---

Table 2: Comparison of KR-Transformer with recent XMC methods six public datasets. Results with a trailing reference are taken from [ Table 3] and 7 Table 3]. We obtain the results of AttentionXML", LightXMLL X- K-Transformer' and XR- TR-Trssfformer on the same vectorized feature matrix provided [8] Due to GPU memory constraint, LightXML is not able to run on Amazon-3M. The PSP@k results are available in Appendix A.4.3
Methods
P@L 1
P@3
P@5
P@1
P@3
P@5
P@1
P@3
P@5
Eurlex-4K
Wiki10-31K
AmazonCat 13K
AnnexML
28
79.66
64.94
53.52
86.46
74.28
64.20
93.54
78.36
63.30
DiSMEC
18
83.21
70.39
58.73
84.13
74.72
65.94
93.81
79.08
64.06
XML
73.14
60.16
50.54
83.57
68.61
59.10
91.75
77.97
63.68
Parabel 3
82.12
68.91
57.89
84.19
72.46
63.37
93.02
79.14
64.51
TextmeeText 24
79.17
66.80
56.09
83.66
73.28
64.51
92.50
78.12
63.51
Bonsai [50
82.30
69.55
58.35
84.52
73.76
64.69
92.98
79.13
64.46
XML-CNN
33
75.32
60.14
49.21
81.41
66.23
56.11
93.26
77.06
61.40
XR-Linear 7
84.14
72.05
60.67
85.75
75.79
66.69
94.64
79.98
64.79
AttentionXML
86.93
74.12
62.16
87.34
78.18
69.07
95.84
82.39
67.32
X- T-Transformer
87.61
75.39
63.05
88.26
78.51
69.68
96.48
83.41
68.19
LightXML.
87.15
75.95
63.45
89.67
79.06
69.87
96.77
83.98
68.63
XR- Transformer"
88.41
75.97
63.18
88.69
80.17
70.91
96.79
83.66
68.04
Wiki-500K
Amazon-67000 670K
Amazon-3M
AnnexML
28
64.22
43.15
32.79
42.09
36.61
32.75
49.30
45.55
43.11
DiSMEC
18
70.21
50.57
39.68
44.78
39.72
36.17
47.34
44.96
42.80
XML 41
56.25
37.32
28.16
36.84
34.23
32.09
43.83
41.81
40.09
Parabel
68.70
49.57
38.64
44.91
39.77
35.98
47.42
44.66
42.55
e TextemeTexxt
24
65.17
46.32
36.15
42.54
37.93
34.63
42.20
39.28
37.24
Bonsai [50
69.26
49.80
38.83
45.58
40.39
36.60
48.45
45.65
43.49
XML-CNN 33
33.41
30.00
27.42
XR-L.inerr
65.59
46.72
36.46
43.38
38.40
34.77
47.40
44.15
41.87
AttentionXML *
76.74
58.18
45.95
47.68
42.70
38.99
50.86
48.00
45.82
X- *
77.09
57.51
45.28
48.07
42.96
39.12
51.20
47.81
45.07
LightXMI."
77.89
58.98
45.71
49.32
44.17
40.25
XR- Transformer"
79.40
59.02
46.25
50.11
44.56
40.64
54.20
50.81
48.26
Table 3: Comparing training time (in hours) of DNN-based methods that produce the SOTA results in The number following the model indicates the number of ensemble models used.
Table 2
Dataset
AttentionXMI--3
X-Transformer.9
LightXMI--3
XR-Transformerr-3
Eurlex-4K
0.9
7.5
16.9
0.8
Wiki10 M-ki10-31K
1.5
14.1
26.9
1.5
AmazonCat- 13K
24.3
147.6
310.6
13.2
Wiki-500K
37.6
557.1
271.3
38.0
Amazon-670K
24.2
514.8
159.0
10.5
Amazon-3M
54.8
542.0
29.3
most baseline results from g Table 3] and 77 Table 3] except for the latest deep learning based algorithms 8
To have fair comparison on training time, we use the same hardware (i.e.. AWS p3. 16xlarge and the same inputs (i.e., raw text, vectorized features, data split) to obtain the results of AttentionXML, X-Transformer and LightXML. The parammee of XR-Trrame XR-Transformer and more empirical results are included in Appendix A.3
13.
The comparisons of Precision @k (P@k) and training time are shown in Table 2 and Table 8 respectively. The proposed XR-Transformer follows AttentionXML. and LightXML. to use an ensemble of models, while X-Transformer uses an ensemble of models [12] More details about the ensemble setting can be found in Appendix A.3 The proposed XR-Trans Transformer
Model Performance.

---

Table 4: Single model comparison of DNN based XMC models. Training time p3.pp...1... 13.16glarge with Nvidia V100 GPUs Teain are reported for AttentionXMLL T-Transformer and XR- Transformer, whereas time single Nvidia V100 GPU Troain is reported for LightXML and AR-Transformer.
Dataset
Method
P@ 1
P@3
P@5
Tromn
T
AttentionXML-11
87.1
77.8
68.8
0.5
X- X-Transformer-1
87.5
77.2
67.1
3.5
Wiki10-31K
LightXML-1
87.8
77.3
68.0
6.7
XR TR-nssssfomer-
88.0
78.7
69.1
1.3
0.5
AttentionXML-1
75.1
56.5
44.4
12.5
Wiki-500K
X X-Transformer-1
44.8
40.1
34.6
56.0
LightXML-1
76.3
57.3
44.2
89.6
XR 1R-TTrasffommr-11
78.1
57.6
45.0
29.2
12.5
AttentionXML-1
45.7
40.7
36.9
8.1
Amazon-670k
X-Transfomeer-1
44.8
40.1
34.6
56.0
LightXML-1
47.3
42.2
38.5
53.0
XR Transformer-
49.1
43.8
40.0
8.1
3.4
framework achieves new SOTA results in 14 out of 18 evaluation columns (combination of datasets and P@k) k), and outperforms competitive methods on the large datasets. Next, we show the training time of XR-Transformer is significantly less than other DNN-base models.
Training Cost. Table X shows the training time for these DNN- models. To have fair compar- ison, all the experiments are conducted with float32 precision on AWS p3. 16xlare instance with Nvidia V100 GPUs except for LightXMII, which was run on single V100 GPU since multi-GPU training is not implemented. XR-Transformer consumes significantly less training time compared with other transformer based models and the shallow BiLSTM model AttentionXML On Amazon- 3M, XR-Transformer has 20x speedup over X-Transformer while achieving even better P@ k. Finally, in table we compare TR-Transformer with LightXML under the single model setup (no ensemble), where IR-Transfommer still consistently outperforms LightXML in P@k and training time.
Table 5: Comparing XR- UR-Transformer with Pre-Trained and word2vec embeddings concatenated with TF-IDF features.
Methods
P@
P@3
P@5
P@ 1
P@3
P@5
P@ 1
P@3
P@5
Eurlex-4K
Wiki10-31K
AmazonCat 13K
TF-IDF
84.14
72.05
60.97
85.75
75.79
66.69
94.64
79.98
64,79
word2vec +TF-IDF
84.35
71.27
59.10
86.11
76.92
66.45
94.53
79.44
63,94
Pre- pre-Tranned +TF-IDF
84.92
71.40
59.36
85.78
78.30
68.33
95.05
80.12
64.53
XR- Transformer
88.41
75.97
63.18
88.69
80.17
70.91
96.79
83,66
68.04
Wiki-500K
Amazon-670K
Amazon-3M
TF-IDF
65.59
46.72
36.46
43.38
38.40
34.77
47.40
44.15
41.87
word2vec +TF-IDF
68.21
48.16
37.54
44.04
39.07
35.35
47.51
44.49
42.19
Pre-Trained Trained +TF-IDF
70.18
49.82
38.75
44.55
38.91
34.77
49.66
46.41
43.96
XR-Transformer
79.40
59.02
46.25
50.11
44.56
40.64
54.20
50.81
48.26
Comparison of Different Semantic Embeddings. To provide more empirical justifications, that the improvement performance comes from better semantic embedding rather than the introducing of TF-IDF features, further tested models using Pre-Traine Trained Transformer and word2vec embeddings concatenated with the same TF-IDF features.
Table summarizes the performance of these models on all 6 datasets. In particular, word2vec using token embedding from word2vec- word2vec-google-newe google-nes--330 and for Tre-Trained we use the same setting as XR- TRR-nssfformrr (3-model ensemble). On large datasets such as Wiki-500K/Amazon 670K/Amazon-3M. Pre-Trained +TF-IDF has marginal improvement compared to the baseline TF-IDF features. Nevertheless, proposed XR- XR-Transformer still enjoy significant gain compared to Pre-Traie Trainei +TF-IDF. This suggests the major improvement from learning more powerful neural semantic embeddings, rather than the use of TF-IDF.

---

Effeet of Cost-Sensitive Learning. In Table 6 we analyze the effect of cost sensitive learning on four XMC datasets with the largest output spaces. On most datasets, cost sensitive learning aggregated labels yields better performance than those without. We also show that cost-sensitive learning not only beneficial to IR-Transformer, but also useful to its linear counterpart XR- Linear 7. See Appendix 1.4.1 for more results.
Table 6: Ablation of cost-sensitive learning on the single IR-Transformer model with or without Cost Sensitive (CS). Precision 1,3,5 P(@k) and Recall @ 1.3,5 (R @k) are reported.
Dataset
Method
P@
P@3
P@5
R@
R@3
R@s
XR Transformer- (wwo CS)
86.8
77.6
68.8
5.2
13.6
19,8
Wiki10-31K
XR- IR-Transformer-1
88.0
78.7
69.1
5.3
13.8
19.9
XR- TR-rrasffomer- 1 (w/o CS)
77.6
57.4
44.9
25.8
48.1
57.8
Wiki-500K
XR- IR-Transformer-
78.1
57.6
45.0
26.1
48.5
58.1
XR- Re--ransformer-11 (wwo
(w/o CS)
49.1
43.8
40.0
10.3
25.6
37.7
Amazon-670K
XR Transformer-1
49.0
43.7
39.9
10.4
25.7
37.7
Transformer- (w/o CS)
50.2
47.6
45.4
8.4
12.4
Amazon-3M
XR- Transformer-1
52.6
49.4
46.9
3.8
9.3
13.6
Effect of Label Resolution and Text Rep- resentation. Next, we compare the effect of label resolution the quality of the fine- tuned transformer embeddings. We fine- tune transformer models in non-recuriive manner on a two layer HLT with differ-
WIR10-31KK
Amazon-670K
B
Rus
48
40
86
leaf cluster size. Then the fine-tuner transformer embeddings are used along or in combination with TF-IDF features produce the predictions with refined HLT. From Figure 2 we can observe that larger cluster size will result in worse semantic features. Figure also shows that combin- ing semantic features Pann with statistical features & Pifidf could in general improve the model performance.
41
1600
400
Figure Comparison of BERT model fine-tuned with different label resolution. Larger cluster size means lower label resolution. Note that Peat is the normalized concatenation of Peftdy and o Pann.
6 Conclusion and Future work
In this paper, we have presented TR-Transformer approach, which is an XMC architecture that leverages multi-resolution objectives and cost sensitive learning to accelerate the fine-tuning of pre-trained transformer models. Experiments show that the proposed method establishes new state- of-the-art results on public XMC datasets while taking significantly less training time compared with earlier transformer based methods. Although the proposed architecture designed for XMC, the ideas can be applied to other areas such information retrieval or other DNN models such as CNNs/ResNNess. Also, more extensive study is required to understand why the coarse-to- to-fie scheme would lead to not only faster training but better overall quality. A hypothesis is that the problem is being solved at multiple scales hence leading to more robust learning deep transformer models.
10

---

References
[1] Wei-Cheng Chang, Felix X. Yu, Yin-Wen Chang, Yiming Yang, and Sanjiv Kumar. Pre-training tasks for embedding- based large-scale retrieval. In International Conference Learning Representations, 2020.
[2] Kenton Lee, Ming-Wei Meig-Weicc Chang, and Kristina Toutanova. Latent retrieval for weakly supervised open domain question answering. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL), 2019.
[3] Yashoteja Prabhu, Anil Kag, Shrutendra Harsola, Rahul Agrawal, and Manik Varma. Parabel: Partitioned label trees for extreme classification with application dynamic search advertising. In WwW, 2018.
[4] Yashoteja Prabhu and Manik Varma. Fastxml: A fast, accurate and stable tree-classifier for extreme multi-label learning. In KDD, 2014.
[5] Wei-Cheng Chang, Daniel Jiang, Hsiang-Fu Yu, Choon-Hui Teo, Jiong Zhang, Kai Zhong, Kedarnath Kolluri, Qie Hu, Nikhil Shandilya, Vyacheslav levgrafov, Japinder Singh, and Inderjit S Dhillon. Extreme multi-label learning for semantic matching in product search. In KDD. ACM, 2021.
[6] Sujay Khandagale, Han Xiao, and Rohit Babbar. Bonsai: diverse and shallow trees for extreme multi-label classification. Machine Learming, 109(1 (1211))209992119 2020.
[7] Hsiang-Fu Yu, Kai Zhong, and Inderjit S Dhillon. PECOS: Prediction for enormous and correlated output spaces. arXiv preprint arXiv:2010.058/1 2020.
[8] Ronghui You, Zihan Zhang, Ziye Wang, Suyang Dai, Hiroshi Mamitsuka, and Shanfeng Zhu. AttentionXMII: Label tree-based attention-aware deep model for high-performance extreme multi-label text classification. Advances Neural Information Processing Systems, pages 5812-5822, 2019.
[9] Jacob Devlin, Ming-Wos Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), 2019.
[10] Zhilin Yang, Zihang Dai, Yiming Yang. Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V Le. XLNet: Generalized autoregressive pretraining for language understanding. In NIPS, 2019.
[11] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer. and Veselin Stoyanov. RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint arXiv: 1907. 2019.
[12] Wei-Cheng Chang, Hsiang-Fip Yu, Kai Zhong. Yiming Yang, and Inderjit S Dhillon. Taming pretrained transformers for extreme multi-label text classification. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 3163-3171, 2020.
[13] Ting Jiang, Deqing Wang, Leilei Sun, Huayi Yang. Zhengyang Zhao, and Fuzhen Zhuang. LightXML: Transformer with dynamic negative sampling for gegh-performance extreme multi- label text classification. In AAAI, 2021.
[14] Wei-Sheng Lai, Jia-Bin Huang, Narendra Ahuja, and Ming- Ming-Hsuan Yang. Deep laplacian pyramid networks for fast and accurate super-resolution. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017.
[15] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation. In ICLR, 2018.
[16] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the JEFF/CVE Conference on Computer Vision and Pattern Recognition, pages 4401-4410, 2019.

---

[17] Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th annual international conference on machine learning, pages 41-48, 2009.
[18] Rohit Babbar and Bernhard Schölkopf. DiSMEC: distributed sparse machines for extreme multi-label classification. In WSDM, 2017.
[19] Ian EH Yen, Xiangru Huang, Kai Zhong, Pradeep Ravikumar, and Inderjit S Dhillon. PD- Sparse: A primal and dual sparse approach to extreme multiclass and multilabel classification. In International Conference on Machine Learning (ICML), 2016.
[20] Ian EH Yen, Xiangru Huang, Wei Dai, Pradeep Ravikumar, Inderjit Dhillon, and Eric Xing. PPDsparse: A parallel primal-dual sparse method for extreme classification. In KDD, ACM, 2017.
[21] Rohit Babbar and Bernhard Schölkopf. Data scarcity, robustness and extreme multi-label classification. Machine Learning. pages 1-23, 2019.
[22] Anna E Choromanska and John Langford. Logarithmic time online multiclass prediction. Advances in Neural Information Processing Systems, 28:55-63, 2015.
[23] Hal Daumé III, Nikos Karampatziakis, John Langford, and Paul Mineiro. Logarithmic time moe-a In International Conference Machine Learning, pages 923-932. PMLR, 2017.
[24] Marek Wydmuch, Kalina Jasinska, Mikhail Kuznetsov, Róbert Busa-Fekete, and Krzysztof Dembezynski. A no-regret generalization of hierarchical softmax to extreme multi-label classification. NIPS, 2018.
[25] Yashoteja Prabhu, Aditya Kusupati, Nilesh Gupta, and Manik Varma. Extreme regression for dynamic search advertising. In Proceedings of the 13th International Conference on Web Search and Data Mining, pages 456-4666 2020.
[26] Kalina Kobuus,, Marek Wydmuch, Krzysztof Dembezynski, Mikhail Kuznetsov, and Robert Busa-Fekete. Probabilistic label trees for extreme multi-label classification. arXiv preprint 2rX.vv2009.112/11 2020.
[27] Kalina -oobus, Marek Wydmuch, Devanathan Thiruvenkatachari, and Krzysztof Dem- bezynski. Online probabilistic label trees. In International Conference on Artificial Intelligence and Statistics, pages 1801-1809. PMLR, 2021.
[28] Yukihiro Tagami. AnneXML: Approximate nearest neighbor search for extreme multi-label classification. In Proceedings of the 23rd ACM SIGKDD international conference on knowledge discovery and data mining, pages 455-464, 2017.
[29] Himanshu Jain, Venkatesh Balasubramanian, Bhanu Chunduri, and Manik Varma. SLICE: Scal- able linear extreme classifiers trained on 100 million labels for related searches. In Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining, pages 528-55 -36.. ACM, 2019.
[30] Y. A. Malkov and D. A. Yashunin. Efficient and robust approximate nearest neighbor search using hierarchical navigable small world graphs. IEEE Transactions on Pattern Analysis and Machine Intelligence, 42(4):824-836, 2020.
[31] Chuan Guo, Ali Mousavi, Xiang Wu, Daniel N Holtmann- Satyen Kale, Sashank Reddi, and Sanjiv Kumar. Breaking the glass ceiling for embedding-based classifiers for large output spaces. In Advances in Neural Information Processing Systems, pages 4944-4954, 2019.
[32] Ruiqi Guo, Philip Sun, Erik Lindgren, Quan Geng, David Simcha, Felix Chern, and Sanjiv Kumar. Accelerating large-scale inference with anisotropic vector quantization. In International Conference Machine Learning, pages 3887-3896. PMLR, 2020.
[33] Jingzhou Liu, Wei-Cheng Chang, Yuexin Wu, and Yiming Yang. Deep learning for extreme multi-label text classification. Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 115-124. ACM, 2017.
12

---

[34] Tharun Kumar Reddy Medini, Qixuan Huang, Yiqiu Wang, Vijai Mohan, and Anshumali Shrivastava. Extreme classification in log memory using count-min sketch: A case study of amazon search with 50m products. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' d'Alché- Bue, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
[35] Kunal Dahiya, Deepak Saini, Anshul Mittal, Ankush Shaw, Kushal Dave, Akshay Soni, Himan- shu Jain, Sumeet Agarwal, and Manik Varma. DeepXML:: A deep extreme multi-label learning framework applied short text documents. In Proceedings of the 14th ACM International Conference on Web Search and Data Mining, pages 31-39, 2021.
[36] Anshul Mittal, Kunal Dahiya, Sheshansh Agrawal, Deepak Saini, Sumeet Agarwal, Purushottam Kar, and Manik Varma. DECAF: Deep extreme classification with label features. In Proceedings of the 1 14th ACM International Conference Web Search and Data Mining, pages 49-57, 2021.
[37] D. Saini, A.K. Jain, K. Dave, J. Jiao, A. Singh, R. Zhang, and M. Varma. GalaXC: Graph neural networks with labelwise attention for extreme classification. In Proceedings of The Web Conference, April 2021.
[38] A. Mittal, N. Sachdeva, S. Agrawal, S. Agarwal, P. Kar, and M. Varma. ECLARE: Extreme classification with label graph correlations. In Proceedings of The ACM International World Wide Web Conference, April 2021.
[39] Hui Ye, Zhiyu Chen, Da- Ha- Wang, and Brian Davison. Pretrained generalized autoregressive model with adaptive probabilistic label clusters for extreme multi-label text classification. In International Conference on Machine Learning, pages 10809-108 19. PMLR, 2020.
[40] Armand Joulin, Moustapha Cissé, David Grangier, Hervé Jégou, al. Efficient softmax approximation for gpus. In International Conference on Machine Learning, pages 1302-1310.
PMLR, 2017.
[41] Himanshu Jain, Yashoteja Prabhu, and Manik Varma. Extreme multi-label loss functions for recommendation, tagging, ranking & other missing label applications. In KDD, 2016.
[42] Marco Pedersoli, Andrea Vedaldi, Jordi Gonzalez, and Xavier Roca. A coarse-to-fin fo- approach for fast deformable object detection. Pattern Recognition, 48(5):1844 1853, 2015.
[43] Shikun Liu, Andrew Davison, and Edward Johns. Self-Supervised generalisation with meta auxiliary learning. H. Wallach, H. Larochelle, Beygelzimer, F. d'Alché-Buc Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
[44] Charles Elkan. The foundations of cost-sensitive learning. In International joint conference on artificial intelligence, volume 17, pages 973-978. Lawrence Erlbaum Associates Ltd, 2001.
[45] Chun-Liang Li and Hsuan-Tien Lin. Condensed filter tree for cost-sensitive multi-label classifi- cation. In International Conference on Machine Learning, pages
PMLR, 2014.
[46] Kuan-Hao Huang and Hsuan- Lin. Cost-sensitive label embedding for multi-label classifi- cation. Machine Learning, 1755)):725- 2017.
[47] Hsuan-Tien Lin. Advances in cost-sensitive multiclass and multilabel classification. In Pro- ceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 31 3187-3188 2019.
[48] Aditya K Menon, Ankit Singh Rawat, Sashank Reddi, and Sanjiv Kumar. Multilabel reductions: what is my loss optimising? In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Bue, E. Fox, and R. Gameett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
[49] Rong-En Fan, Wei Chang, Cho-Jui Hsieh, Xiang-Rui Wang, and Chih-Jen Lin. LIB- LINEAR: a library for large linear classification. Journal of machine learning research, 9AAu)):181111874 2008.
13

---

[50] Sujay Khandagale, Han Xiao. and Rohit Babbar. BONsAI-diverse and shallow trees for extreme multi-label classification. arXiv preprint arXiv:1904.08249 2019.
[51] Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. Proceedings of the International Conference on Learning Representations, 2014.
14


Reviews:
[{"review_id": "_AnSaDq5ge3", "reviewer": "Reviewer_YuK3", "summary": "The authors propose an approach for extreme multi-label classification  that is based on a transformer network and leverages a reduction approach that breaks down the problem to smaller ones via a hierarchical approach that constructs a label tree. The authors use both the representation of the transformer as well as classical TF-IDF ones as the final one for the model. The approach is validated in 6 large datasets and compared with other state-of-the-art approaches.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Using tree structures for extreme multilabel classification is not new as an approach. The authors can refer to approaches like the ones developed by Langford and team (strangely enough these works are not cited). So, reductions are not new.\n\nWhat it seems to be new is the multi-resolution learning. The authors have some results on the size of the clusters but is not very clear how much this way of training helps. I think what is missing here is a deeper exploration in terms of experiments. For, example is interesting that the TF-IDF features alone do not show variation based on the cluster size while DNN ones yes. The authors should put some more effort here to better understand why this happens.\n\nImprovements are rather small, even though the proposed approach improves in most of the cases. Training time is greatly improved though.\nWe need to be in a really large output space to see a good difference in the metrics. It would be nice if the authors could elaborate a bit more on this.\n\nWould it be possible also to have the results of Table 2 with/wo TF-IDF features? \n\nWhy the authors did not try to compare with simple concatenations of TF-IDF and aggregations based on a pre-trained language model or even a word2vec approach.\n\nIs after-all the use of the simple statistical features that helps to improve the performance? ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Still the approach is expensive and to be honest my take away if that TF-IDF is still a powerful scheme.\n\nNo negative societal impact.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2.5"}, "extra_text": {}}, {"review_id": "RZC918ps895", "reviewer": "Reviewer_ihmk", "summary": "To accelerate the fine-tuning procedure of transformer models on XMC, the paper proposes XR-Transformer, which is recursively fine-tuned on coarse-to-fine multi-resolution tasks and cost sensitive learning. Experiments on the benchmarks show that XR-Transformer generally outperforms previous methods and is faster than the competing transformer-based models.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper proposes a novel method to accelerate fine-tuning transformer-based models for XMC, which is inspired by the multi-resolution learning in image generation and curriculum learning. This technique allows for fast fine-tuning and good performance. Whereas concatenating neural features and TF-IDF features to produce the predictions is already discussed in the X-Transformer paper and thus is not new.\n\nThe paper is technically sound and well written overall. The method is generally well established. The experiment results seem to be solid. The DNN-based methods use the same inputs for fair comparisons, and the running time is also well reported. Since the results of the DNN-based methods are slightly different from the original X-Transformer and LightXML papers, it could be better to give the number of run times, the standard deviation of the results, and the statistical significance. In addition, there is a slight gap between the training time of ensemble model (Table 3) and single model (Table 4), this might be improved by adding the standard deviation.\n \n **Question for the authors:**\n \n1. The text inputs are truncated to 128 tokens, while the documents are generally longer on average. As the training time is significantly reduced, perhaps XR-Transformer could use longer input text to obtain even better results?\n2. The authors mention the PSP and tail labels, do the coarse and fine resolutions have connections with the label frequency? Such as higher resolutions focus more on tail labels. Are the HLT and indexing matrices $C$ related to the tail labels? I would like to see more discussion about this.\n\n**Typo:**\n\n- Line 66  model -> models\n- Line 247 The **comparison** of ... **are** ...", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": " The paper should not pose any negative societal impacts.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4.5"}, "extra_text": {}}, {"review_id": "G-J3NA01t90", "reviewer": "Reviewer_9rJq", "summary": "The authors address the problem of fine-tuning transformers models for extreme multi-label text classification (XMLTC). Even with pre-trained transformer models, fine-tuning it to the task of predicting for large label space is computationally expensive and requires significant GPU resources. The authors propose a new routine to train the X-Transformers-like model that significantly reduces that computational cost. The routing resembles AttentionXML learning routine, where one model is trained by tree level to predict meta-labels at that level. Training starts from the top level of the tree, and the model for the next level is initialized with weights of the final model from the previous level. Here underlying transformer model is first tuned on the task of predicting the same meta-labels. It starts with tuning using meta-labels from the top level and then continues tuning layer by layer. In addition, the authors apply a few additional techniques like cost-sensitive training (positive meta-labels are weighted by a number of relevant labels they consist) and bootstrapping of output weights for meta-labels/labels based on current transformers embeddings. After tuning, new label-partitioning is constructed, and the model is retrained with fixed transformer weights. The authors evaluate the attractiveness of their method on six popular benchmark datasets. New routing leads to a significant reduction in training time at the same often achieving better predictive performance.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n+ The organization of this paper is good, and it's easy to understand. \n+ The paper has clear motivation.\n+ Proposed method significantly reduces the computational cost of training achieving better or similar results to its parent method X-Transformers and SOTA LightXML and AttentionXML. \n+ Reduction of training cost makes XMLC more accessible cost, which is vital for further research and practitioners deploying such models.\n+ Proposed training routine should be applicable to other label tree-based approaches methods with transformers and possibly other types of underlining deep models.\n+ Solid experimental evaluation that proves the attractiveness of the proposed method.\n+ The appendix seems to consist of enough details to reproduce this work and results.\n\nWeaknesses:\n- I don't find any serious issue in this work. It seems to me that the scope of contribution is a bit limited since the proposed approach is composed of elements that were already explored by the XC community, but this is a very successful combination. \n\nOther comments:\n* \"Nevertheless, multi-resolution learning has not been well-explored in the XMC literature. In XR-Transformer, we leverage the label hierarchy defined by the HLT and train the transformer model on multi-resolution objectives.\" - I would argue with that statement, as I mentioned in summary, your routine resembles AttentionXML approach. And while it's not directly multi-resolution learning, I would personally acknowledge it here more.\n* After reading the appendix (Table 6 to be precise), I realized that the initial Hierarchical Label Tree (HLT) used for tuning is different from the refined HLT (I mean pre-leaf structure). The initial is more \"wide\", which makes sense since you want to have more fine-grained meta-labels. But I don't find this stated in the main text. Maybe it's worth commenting on it.\n* I think that Recursive_Learn procedure could be a bit more readable and general if presented in an iterative fashion, where the whole HLT is already given.\n* There is a mistake in Table 4, Row \"Wiki-500, X-Transformers-1\" is the same as the row \"Amazon-670K, X-Transformers-1\". \n\n---\n\nEdit after the authors' response: I think the authors responded well to the reviewers' comments. I would like to keep my rating and recommend this paper for acceptance.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No issues with negative social impact were found.", "review_text": "", "extra_scores": {"ethical_concerns": "No ethical issues were found.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "5wAYNND8lUR", "reviewer": "Reviewer_HoYN", "summary": "This paper proposes a coarse-to-fine learning framework to solve the problem of extreme multi-label text classification.  Experiments show the model not only accelerates existing models 20x times but also achieves a slightly better performance on the dataset with an extremely large label set. The techniques are sound and the empirical results are promising.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Advantages:\n1. The paper is well-written and well-organized. The authors include enough background information and clearly illustrate the efficiency-resolution trade-off in this task. Readers without prior knowledge of the XMC problem can easily follow it.\n2. The paper is well-motivated. The proposed method is natural, sound, and potentially helpful in other problems.\n3. The model achieves state-of-the-art performance on 14 out of 18 metrics on 6 datasets.\n4. The authors will publish their codes.\n5. Though the empirical improvements over the baseline models are not very significant on many datasets, the great time efficiency makes the model more applicable in many real-world scenarios.\n\n\nWeaknesses:\n1. Since the proposed method incorporates TF-IDF into the text representation and incorporating TF-IDF is often helpful in improving model performance, these improvements may mainly come from TF-IDF instead of the proposed framework. The superiority of the proposed framework would be more convincing if the authors enhance baselines models with TF-IDF and compare the proposed XR-transformer with the enhanced baselines OR compare the unenhanced-XR-transformer with baseline models.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "1. The first figure in the Appendix that overviews the model architecture is relatively hard to understand. It would be very helpful for readers to understand this paper if the model architecture is presented in a more understandable way.\n2. To the best of my knowledge, this paper has no potential negative societal impact.\n3. The authors do not address the limitations mentioned in Sec. 6 (understand why the coarse-to-fine scheme leads to better performance). The experiments discussed in above weaknesses 1 may help them better understand where the improvements come from.", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper introduces an improved version of a transformer-based algorithm for extreme multi-label classification. The empirical results show significant speed-ups and very competitive predictive performance. The paper is well-written and the proposed modifications are sound. The reviewers were satisfied by the responses given by the authors. 
Decision: Accept (Poster)