Enhancing Tail Performance in Extreme Classifiers by Label Variance Reduction
Authors: Anirudh Buvanesh
Rahul Chand
Jatin Prakash
Bhawna Paliwal
Mudit Dhawan
Neelabh Madan
Deepesh Hada
Vidit Jain
SONU MEHTA
Yashoteja Prabhu
Manish Gupta
Ramachandran Ramjee
Manik Varma
Venue: ICLR 2024 poster (2024)
Keywords: Extreme Classification
Extreme Multi-Label Learning
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Abstract: Extreme Classification (XC) architectures, which utilize a massive One-vs-All (OvA) classifier layer at the output, have demonstrated remarkable performance on problems with large label sets. Nonetheless, these architectures falter on tail labels with few representative samples. This phenomenon has been attributed to factors such as classifier over-fitting and missing label bias, and solutions involving regularization and loss re-calibration have been developed. This paper explores the impact of label variance - a previously unexamined factor - on the tail performance in extreme classifiers. It also develops a method to systematically reduce label variance in XC by transferring the knowledge from a specialized tail-robust teacher model to the OvA classifiers. For this purpose, it proposes a principled knowledge distillation framework, LEVER, which enhances the tail performance in extreme classifiers with formal guarantees on generalization. Comprehensive experiments are conducted on a diverse set of XC datasets, demonstrating that LEVER can enhance tail performance by around 5\% and 6\% points in PSP and coverage metrics, respectively, when integrated with leading extreme classifiers. Moreover, it establishes a new state-of-the-art when added to the top-performing Renee classifier. Extensive ablations and analyses substantiate the efficacy of our design choices. Another significant contribution is the release of two new XC datasets that are different from and more challenging than the available benchmark datasets, thereby encouraging more rigorous algorithmic evaluation in the future. Code for LEVER is available at: aka.ms/lever.

Published as a conference paper at ICLR 2024
ENHANCING TAIL PERFORMANCE IN EXTREME CLAS- SIFIERS BY LABEL VARIANCE REDUCTION
Anirudh Buvanesh', Rahul Chand", Jatin Prakash, Bhawna Paliwal, Mudit Dhawan Neelabh Madan, Deepesh Hada, Vidit Jain, Sonu Mehta, Yashoteja Prabhu Manish Gupta, Ramachandran Ramjee, Manik Varma
Microsoft
{t- abuvanen,,, t t-rahulchand, t-japrakash, bhawna, t t-mdhawan t-nmadan, deepeshhada, jainvidit, conu.mehta, yprabhu gmanish, ramjee, manik]emicrosoft. com
ABSTRACT
Extreme Classification (XC) architectures, which utilize massive One-vs-All (OvA) classifier layer at the output, have demonstrated remarkable performance problems with large label sets. Nonetheless, these architectures falter tail la- bels with few representative samples. This phenomenon has been attributed to fac- tors such as classifier over-fitting and missing label bias, and solutions involving regularization and loss re-calibration have been developed. This paper explores the impact of label variance previously unexamined factor on the tail perfor- mance in extreme classifiers. It also develops a method to systematically reduce label variance in XC by transferring the knowledge from a specialized tail-robust teacher model to the OvA classifiers. For this purpose, it proposes a principled knowledge distillation framework, LEVER, which enhances the tail performance in extreme classifiers with formal guarantees generalization. Comprehensive experiments are conducted on diverse set of XC datasets, demonstrating that LEVER can enhance tail performance by around 5% and 6% points in PSP and coverage metrics, respectively, when integrated with leading extreme classifiers. Moreover, it establishes new state-of-the- of--hh--ttttttt- when added to the top-performing Renée classifier. Extensive ablations and analyses substantiate the efficacy of our design choices. Another significant contribution is the release of two XC datasets that are different from and more challenging than the available bench- mark datasets, thereby encouraging more rigorous algorithmic evaluation the future. Code for LEVER is available at: laka ms/levr
INTRODUCTION
Extreme Classification (XC) addresses tasks where a data point is mapped to the most relevant sub- seí of labels from large label space. Deep architectures that comprise a neural network encoder followed by massive One-vs-All (OvA) classification layer at the output have become the de-facto standard for contemporary XC algorithms and have demonstrated remarkable results several large-scale applications (Agrawal et al. 2013 Yadav et al. 202T] Chang et al., 2020 Beygelzimer al. 2009 Babbar & Schölkopt 2017). Despite this progress, such over-parameterized OvA classi- fication layers has also been known to overfit and underperform on labels with limited representative samples, also known the tail labels (Wei et al.) 2021). As result, bulk of such tail labels, which often provide niche and highly informative results for a test sample (Jain et al.) (016)) are incorrectly classified thus diminishing their aggregate utility for practical application.
The challenge of enhancing the tail performance of extreme OvA classifiers has been the focus of some recent studies. These investigations have identified multiple factors that contribute the hardness of tail labels and proposed solutions to alleviate them. Some works have addressed the
Primary Authors. Coneepondence to: Anirudh Buvanesh Kanirudhbilus wttphhhbbb110@@mmmmmmmmmm com? Yashoteja
Prabhu wyprabhuemicroooft..oo

---

Published as a conference paper at ICLR 2024
concern of overfitting to data-scard tail labels by constraining the capacity of tail classifiers through regularization tricks (Guo et al. 2019). A separate line work has studied the effects of false negatives, also known missing labels, the tail performance and proposed to appropriately amend the classifier training loss through propensity-scoring techniques (Qaraei et al. 2021
This paper brings to light another important yet previously unexamined factor behind the under- performance of tail OvA classifiers, namely label variance (Sec. (3). Typically, the ground truth of an XC dataset is constructed by approximating complex label distribution that arises in source application with a discrete sample of labels. For example, in a recommendation task, the ground truth is defined as the set of items clicked by each user within a specified time period. However, in general, the ground truth can vary from one data sampling period to another, as a user's 's interests can fluctuate with time. Similarly, in expert annotation based data, employing fewer experts to reduce annotation costs can introduce variance in the ground truth (aka. label variance) owing to inter-annotator disagreements. Large label variance particularly harmful for the tail classifiers' performance as they have to rely sparse ground truth and the approximation errors can have drastically magnified effects with low sample counts.
In recent work, Menon et al. (2021a) studied the problem of label variance in the context of multi- class classification and retrieval, and further note that a teacher-to-stee to-tudent knowledge distillation strategy can be used to improve the generalization performance of the student model. This paper borrows the hasic ideas from Menon et al.] (021a) and extends them to the more challenging Extreme Classification setting through several key innovations. First, it theoretically formalizes the performance degradation in OvA classifiers owing to label variance, specifically quantifying the magnified effect on the tail classifiers. Second, whereas MMnnn et al. (2021a) assumes the pre- existence of a teacher, this paper learns its own Siamese-style teacher model that is optimized for tail performance, and further develops a principled knowledge distillation strategy to effectively teach the downstream OvA classifiers. The resulting approach, LEVER, is demonstrated to improve tail classifiers' performance by around 5% and 6% points in terms of PSP and coverage metries, which also advances the state-of-the-ar in XC.
Another independent contribution of this paper is the public release of two new datasets for algo- rithmic benchmarking in XC. Traditionally, performance in XC is mostly assessed the public datasets available from (Bhatia et al., 2016). These datasets appear to share a common property that the data points associated with label are fairly similar to each other in their semantic intents, making these datasets less challenging to learn. In contrast, the real-world upplications of XC can be more diverse in their properties and complexity. To encourage more rigorous algorithmic eval- uation, the new datasets are constructed with the property that a label can be associated with data points of vastly different intents. These datasets, termed as multi-intent datasets, are inspired by real applications, are more challenging, and can unlock exciting research problems in the future.
This paper makes the following key contributions: 1. Identifies the problem of label variance which adversely affects the performance of tail classifiers in XC. 2. Proposes a principled LEVER ap- proach mitigate the label variance effects on tail classifiers in XC (Sec. 3.2.. 3. Develops an effective Siamese-style model as tail teacher with LEVER (Sec. (.3) 4. Conducts extensive ex- perimentation using multiple sfate-of-the-art baselines and diverse benchmarks to demonstrate the utility and generality of the proposed approach (Sec. 5). 5. Releases two new multi-intent datasets for robust experimentation in XC (Sec. 4
2
RELATED WORK
2.1 EXTREME CLASSIFICATION
Recent advancements in XC have leveraged deep network-based representations like LSTM YYou et al. 2018), Transformer (Zhang et al. 2021 Jiang et al. 2021) or customized architectures (Dahiya et al. 2021b) to generate rich semantic representations of inputs. These are then assigned to ap- propriate labels via an OvA classifier layer. To facilitate efficient learning with large label sets, techniques such as multi-staged encoder refinement (Dahiya et al. 2021a Zhang al. 202 Tiang ct al. (02TT, hierarchical label search, and bard-negative sampling (Dahiya et al. [023a 2021b; Zhang et al.) 2021] [riag et al.) 22021] Mittal al.) (021a) have been introduced. Furthermore, simultaneous training of the deep encoder and OvA classifiers has been demonstrated to boost per-
2

---

Published as a conference paper at ICLR 2024
formance in leading XC approaches like DEXA (Dahiya et al. 2023b). ELIAS Gupta al. 2022
CascadeXML (Kharbanda et al., 2022) and Renée (Jain et al. (023). However, despite these ad- vancements, many of these approaches share common limitation: a decline in performance for tail labels, which is the primary focus of this paper.
2.2 ENHANCING TAIL PERFORMANCE IN XC
Extreme classifiers have been observed to under-perfone on tail labels with limited representative samples. This phenomenon has been attributed to various factors, and several approaches have been proposed to address them.
Over-fitting of OvA Classifiers: OvA classifiers, which employ a distinct classifier for each label, are massively parameterized in scenarios with large label sets. Consequently, they are susceptible to overfitting on tail labels with scarce representative samples. In response. various classifier reg- ularization techniques have been introduced. For instance, ProXML (Babbar Schölkopf 2019) employs L1 Ll-reuularizer, and GLaS (Guo et al. (219)) uses label-decorrcllation based regularizer.
Bias due to Missing Labels: XC datasets, which are often too large for exhaustive labeling, missing or false negative labels are frequent issue. These missing labels introduce systematic biases into the ground truth and are known to significantly impact tail labels. Strategies to address tail
labels typically involve estimating the missing propensities for labels first and then recalibrating the
loss through simple weighting (Jain et al.) 2016 Wei et al.) [021] Wydmuch et al.) [021]] Schultheis
al. 2022). The phenomenon of missing label bias is distinct from that of label variance.
Data Scarcity in Tail Labels: XC datasets contain tail labels with a limited number of positive data samples. To mitigate this scarcity, data augmentation techniques like TAUG (Wei et al.) 2021 and Gandalf (Kharbanda et al. 2024) have been proposed. However, these methods lack formal guarantees and do not perform consistently across different datasets as shown this paper (Ta- ble 2 Another line of work leverages label-side features to improve the tail label prediction per- formance (Xiong et al. 2020, Dahlya et al.) 2021a] [223a] Jain al. [2023). Approaches like NGAME (Dahiya et al. 2023a) share information between semantically similar labels by placing dense embedding space using a Siamese encoder. However, these methods primarily focus on enhancing encoder robustness and do not explicitly address the qual- ity of subsequent OvA classifiers. Our proposed model shares similarities with these approaches through its use of Siamese teacher but distinguishes itself by learning specialized teacher model suitable for distillation and developing a principled approach to improve tail OvA classifiers.
them close to each other
In addition to these known issues, this paper introduces label variance tant, consideration pertaining to tail performance in XC. A closely related work is the study around uncertainty quantification in extreme classification (Jiang et al. (2023) because variance can intrin- sically be viewed as an uncertainty measurement. But in this work, we attempt to mitigate variance rather than just estimate it. It important to differentiate the label variance discussed here from the variance described in (Babbar & Schölkopf 2019). The latter addresses variance from the perspec- tive of lack of commonality between the features of train and instances. In contrast, our focus on label variance pertains to inaccuracies in the ground truth relevance scores.
additional, but impor-
3 LEVER: LABEL VARIANCE REDUCTION IN EXTREME CLASSIFICATION
Label variance is a measure of approximation errors introduced in the ground truth of a dataset due the discrete data sampling process. These errors can negatively impact the performance of trained classifiers, particularly those on the tail. This section introduces LEVER, principled approach based on knowledge distillation designed to alleviate label variance and enhance the generalization capabilities of One-vs-All (OvA) classifiers. An effective teacher model for distillation based on Siamese-style encoder is also proposed.
3.1 PRELIMINARIES
Extreme Classification (XC) maps data point space X onto label space represented y {0, 1)", where L is the number of labels, potentially reaching into the millions. A deep extreme classification architecture typically includes deep encoder En which generates semantically rich
3

---

Published as a conference paper at ICLR 2024
representation Ex(x) for any given input data point E X. This is followed by One-vs-All classifier layer {wil which sorts the labels based on w, Ex(x) scores and predicts the highest scoring labels the most relevant ones for x.
Different strategies have been employed for training such a deep architecture including stagewise training where encoder and classifiers are optimized in two successive stages, and end-to-end train- ing where both are optimized jointly. For this paper, we assume a stagewise training schedule. Furthermore, the focus will be primarily on the second stage of OvA classifier training during which encoder is assumed to be already trained and held fixed. As result, each OvA classifier is trained independently of others which also simplifies the theoretical analysis. For brevity, we drop the en- coder symbol Eg and directly use x to refer to a data point's 's embedding from the encoder over which OvA classifiers are applied.
For data point x, let (x) y(x)) Vy {0, 1) represent the true and complete distribution of label relevance which accurately captures the stochasticities inherent in the user preferences or annotator judgments. Note that this distribution sums up to 1 over all label subsets. Unfortunately, the full relevance distribution seldom available and is instead approximated with a discrete sample of labels P(Y (x) = y|x). The approximation error due to this sampling is captured by the following expression for label variance:
Vy Wyxxyy Eylxly [
Vui VvixxB/ [r]] E Ely P(y 1|x) P = =[x))
The second expression denotes the variance in the marginal relevance of label to point x, term that is particularly useful in analyzing One-vs-All classifiers. larger variance indicates that the imprecision in sampled label is more.
To train the classifier for label 1, we first construct a training set denoted as D = [x,, ya) and solve binary classification problem with ya the target label for x;. For simplicity, present the analysis for a single classifier, with the understanding that the same holds for all classifiers. To avoid confusion, we omit subscript where it is not necessary. The binary classification objective minimizes the following empirical risk of classification:
N
1
R = min
Σ
Clys. w xx) Tx;)
N
1=1
with, C(y. = = w (11-y)f(0) w Tx)
(2)
Here, f represents a convex classification surrogate such as hinge loss or logistic loss (Qaraei et al., 2021]. Using a weight factor C' >> is standard practice in imbalanced classification appropriately balance the relative importance of positive and negative samples for label. This is particularly important for a tail label with a few positives, denoted by number where:
S N
Ex Pz) ~
where, Ps = P(y 1/x)
Following the standard practice (aakade et al. w and the input vector x are bounded by [w] W and ||x||| B respectively. Additionally, we assume that the function f exhibits Lipschitz continuity with a Lipschitz constant L.
2008), we assume that the norms of the weight vector
The generalization performance of a trained classifier evaluated by its true population risk. A lower value of this risk indicates superior predictive capability:
R = Ex,y [R(,, w x)]
(4)
3.2 LEVER FRAMEWORK
The deviation between empirical and true risks formally measures a classifier's generalization gap, with smaller values indicating better test-time generalization. Following (Maurer & Pontil 2009). we express the generalization gap in terms of data- deta-dependent bounds based on label variance. Ap- plying Bennett's 's inequality, suggested in the reference, with simplifications relevant the prob- lem at hand, provides us with the following result. Note that all the proofs are available from the supplementary Sec. A
4

---

Published as a conference paper at ICLR 2024
Theorem 1. Let My be the uniform covering number (Menon et al. 2021a corresponding to the classification loss L. Then, given the definittions established earlier, For any d (0, 1), with probability at least - over sampling the data points {x}
R R O(V V wT)] + IXI x [[[CC
log(MN/0)
log( log(MN/0) N
N
where. VxL(P2: wTx x) and [y(y are the variances in the loss function contributed by x, and conditional variance of y respectively.
Lemma 1. Assuming the loss weighting factor C defined in Eq. 2 as C N where S is the threshold defined in Eq. and N is the number of training points, the variance term V Ex ( B)² in Theorem is bounded by NYLWB)
g.
Theorem establishes a strong dependence between the classifier performance and the variance in labels V y / | with larger values of the latter degrading the effectiveness of the trained classifiers. Furthermore, Lemma shows that smaller positive sample count S can amplify the adverse effect of label variance which makes the tail classifiers more prone to label variance-related degradation. Now, if we have access to precise estimates of marginal relevance, denoted by p3 = Elyl x). we can replace with Pz, effectively reducing the label variance term to O. This forms the intuition behind LEVER which employs an additional teacher network to provide accurate estimates of px-
In practice however, obtaining perfect teacher is infeasible both due to modeling and compu- tational hardness issues. As a result, the ability to robustly leverage a partially biased teacher to improve the target student model is essential for the practical utility of LEVER. To enable this, propose the following variant of LEVER where an imperfect teacher's relevance estimates are used for regularizing the original loss with discrete labels:
1 N
min
Clyi, w Tx:)
C(p,, T Tx,)
(6)
where úú are the relevance estimates outputted by the teacher model, and À is a regularization hyper- parameter. The above formulation aims to trade off variance errors due to y: with the bias errors due to P. to attain the lowest overall generalization error. The following theorem shows that, for an appropriate choice of X, the risk of the resulting classifier is lower than when trained on either y: or D; alone:
Theorem 2. Let R. R be the population risk and empirical risk for a binary classification loss L. Let MN be the uniform covering number (Menon et al. (021a) corresponding to C. Also, let the teacher be imperfect with maximum possible error in relevance estimates bounded by E = Ppx
Clys, w xx)x + Pxll lloe. Then, by solving the regularized optimization problem R. min N Ei=1 E
L EEN, E(pr, w xz) and setting À to minimize population risk- for any ô E (0, 1), the following inequality holds with probubility at least 1 8 over sampling the data points {x) under the assumption of a reasonably small teacher error (E);
c² +ct log(MN/8)
X
RSR.
+
Va
Va-a
DV 2-
N
log(M /8)
CLWE B = ECLWB (8)
where, == V2
b= Vllgg(MN//
N
N
Note that when c = 0, x = which is equivalent to training on pure teacher estimates. Also, when In other words, the bound over population risk
0 <<csb. / a + S min & is tighter than when À 0 or À 1. Therefore, trading off the teacher's 's bias with label variance by setting an appropriate 0 < < 1 can lead to better generalization than pure training with either original ground truth biased teacher estimates label targets.
3.3 A STALESE-STYLE TEACHER FOR LEVER
Recent studies have shown that Siamese Networks, when used as input encoders, exhibit strong performance tail labels (Dahiya et al. 2021al 2023a Jain et al. 2023). This success can be
5

---

Published as a conference paper at ICLR 2024
attributed to the ability of Siamese encoders leverage label correlations by utilizing label-side features. These features, often presented as descriptive text or structured graphs over labels, are commonly found in XC applicationn. In fact, most recent XC datasets have started to incorporate them (Bhatia et al. 2016). Consequently, this allows for the sharing of information between se- mantically similar labels, effectively addressing the problem of data scarcity in tail labels. It is important to note, however, that a standalone Siamese model is insufficient as it tends to under-fit data-rich head labels, thereby compromising overall prediction quality. This paper, therefore, pro- poses the use of Siamese Networks as teachers within the LEVER framework to enhance the tail performance of one-vs-all classifiers. By employing LEVER, we can improve the tail performance of one-vs-all classifiers without compromising their already excellent head accuracies.
Siamese encoder, Eg, is trained to map the features of data points, denoted as {x and label features, represented {zif= into common embedding space. The objective of this mapping is to ensure that labels relevant to a given data point are positioned closer in the embedding space, while those that are irrelevant are distanced. Typically, this is achieved by minimizing triplet loss 12 X - z x; + A]+, where k and 1 are a negative and positive samples, respectively, for label I and A a margin enforced for better generalization (Dahiya et al., 2021a 2023a). However, the triplet-loss is not probabilistically calibrated and does not provide reliable marginal relevance targets for training student. To address this, we leverage a logistie-loss based objective that is found to be well-calibrated:
Σ
log(1 + c² N X6-2/ xx++A
min
LET. MEX
VEX+
The following theorem demonstrates the calibration property of Eq. assuming that the loss can be fully minimized, i.e., loss between each positive-negativy pair is minimized.
Theorem 3. Consider a label z, and a pair of data points Xa; Xb. Let Pa Ph be the probabilities
that the label is relevant to points a, b respectively. Then, assuming that Eq. iis fully minimized, the expected loss in Eq. 9 is minimized for Pa. = 1/(1 e- -xTxxxxxx xxtel), p = 1/(1 + E -xx" (xxx))))).
The above result shows a direct connection between the Siamese model's scores and relevance prob- abilities, which can be exploited as teacher targets. The parameter e is hyper-parameter, and is fitted by cross cross-validation validation. While the above strategy provides well-calibrated scores, we empirically observe that simple score mapping strategies, such as Pa cos Sim(z, ++1 where cos Sim represents the cosine similarity, also work equally well.
To make training tractable, we follow the negative mining strategy used in NGAME (Dahiya al. 2023a). Motivated by recent works that under-sample (or oversample) model inputs (Menon et al. 20216) address dataset imbalance, wise approach, in which mini-batcher are made from labels rather than points. This adjustment leads to the up-sampling of tail labels, thereby increasing their importance during training. Empiri- cally, we find that a teacher trained via this strategy exhibits better tail performance. Subsequently, the one-vs-all (OvA) classifier distilled from this teacher outperforms the OvA classifier distilled from the Siamese teacher trained with pisnt-wise sampling, both in precision (+0.37% on average in P@] 1) and in PSP (+1.7% on average in PSP@ 1), as detailed in Table 12 in the appendix.
modify NGAME's point-wise sampling strategy to label-
4 CONTRIBUTED DATASETS
Motivation Performance evaluation of XC algorithms has largely relied on public benchmark datasets available from (Bhatia et al.) 2016). In these datasets, the data points associated with label tend to be fairly similar to each other in their semantic intents. We refer to these as single- intent datasets. For example, in AF-AAmmzonTills-1133 associated with "formal shirts for men" or "casual shirts for men". In contrast, several real-world XC applications belong multi-intent setting where the label can be associated with data points of vastly different intents. For instance, in query auto-completion (Yadav et al. 2021 where the prefix of search query needs to be mapped to completing suffixes, a suffix ".. "..book" might start with ei- ther "face.." or "note.." as prefix thus leading to completely different final queries. Such multi- intent datasets can be challenging for XC but are under--epresented among existing benchmarks. Addition- ally, the datasets we release exhibit significant imbalances compared to existing benchmarks, with
the label "clothing for men" might be

---

Published as a conference paper at ICLR 2024
7- 11% of the labels accounting for 80% of the positive instances (refer Table 3). This imbalance poses multiple challenges. First, methods like GLaS (Guo et al. 2019) and Gandalf (Kharbanda) et al. 2024), which depend on label correlations for regularization or data augmentation, struggle due to the sparse correlations among tail labels when imbalance is high (refer Table 2 Second, classifier-hased methods may achieve high precision by focusing the head labels, but this results in poor performance on tail metrics such as coverage (refer 2) We believe that the contributed datasets will promote further study into developing methods that are robust across various dataset settings.
Contributed datasets: Two new datasets, LF-AOL-270K and LF- -ikiHierrrrch-- 1M are curated. LF-AOL-270K involves the query auto-completion task of matching query prefix with complet- ing suffixes. It is curated from publically available AOL search logs (Pass et al. 2006). I.F- WikitHierarchy-M MM involves the taxonomy completion task (Benaouicha et al.) 2016) of matching a Wikipedia category to its parent categories (Zesch & Gurevych] 2007). This dataset is motivated by the real-world application of query-to-ad keyword matching where a keyword can subsume the intent of its query thus giving rise to hierarchical association structures. Complete dataset creation details and dataset statistics are provided in appendix Sec. B.3
EXPERIMENTS AND RESULTS
diverse set of datasets, encompassing both full-text and short-text feature scenarios, as well as novel multi-intent datasets. Specifically, we utilized three full-text datasets (LF-Amazon-13) 131K, L.F-Wikipedia-500KK I.F-WikiSeeAlso--22000 datasets (LF-AmazonTittle 131 K, L--AAmzoon l'itles-1.3MM and two new multi-intent datasets (LF- WikiHierarchy- IM and LF-AOL-270K) For detailed dataset statisties, please refer to Table the appendix. Additionally, we evaluate LEVER on a large proprietary query-to-keyword matching dataset with 20M labels (refer Sec. B.2 in the appendix for more details).
Datasets: LEVER was evaluated
two short-text
To assess the test-time performance, standard evaluation metrics were used, namely precision @k (P@k. k= 1, 3, and 5) and its gropensity-weighee variant @SP@k (with k=1, 3, and 5). Detailed definitions for these metrics can be found in (Bhatia et al., 2016). Additionally, following the recommendations in (Schultheis et al. 2022), we also included coverage @k (C@k) as an important metric to evaluate the tail performance.
Evaluation Metrics:
Baselines We applied LEVER to improve multiple strong OvA-based baselines, including Cas- cadeXML (Kharbanda et al. 2022), ELIAS (Gupta et al.) (022), and Renée (Jain et al.) 2023). for demonstrating its effectiveness and generality. We also compared LEVER to other competing tail-
enhancement techniques including eguuarization-based methods such as GLaS (Guo et al. 2019)
and Le-reuularization, data augmentation methods like TAUG (Wei et al., 202I) and Gandalf KKarr banda et al. 2024). and propensity weighting approaches such as Re-rank (Wei et al. 2021 For comprehensive details on model hyper-parameters, please refer to Sec. D in the appendix.
LEVER Implementation Details As discussed in Sec. LEVER uses Siamese teacher to obtain relevance estimates, p. Using the relevance estimates an augmented dataset Dang is created by adding each label as a document, resulting in dataset comprising N + L documents and L labels. Document and label embeddings from the Siamese teacher are then used to add TI nearest labels, and Ta nearest documents for particular label. total, T 77 + Ta elements are added for each label. Empirically, we find that not adding documents (Ta = 0) leads to performance similar to that of adding documents in most cases. For more details on LEVER's dyper-paramedt refer See. D.7.1 in the appendix.
Performance on SOTA OvA methods Table demonttteesssssttess LEVER's effectiveness when applied to leading classifier-based XC methods, including CascadeXML, ELIAS, and Renée. LEVER con- sistently improves P@ and PSP@ on average by 2% and 5%, respectively, across all base models and datasets. When applied to Renée, LEVER achieves new state-of-the-art, increasing PSP@) by up to 5% while maintaining comparable precision. Notably. LEVER proves highly effective highlighting its importance when data is limited. T3 in appendix further illustrates LEVER's gains on a proprietary dataset con- taining 20M labels. Larger improvements in ELIAS and Cas are attributed to these models not explicitly utilizing label features during training or initialization. In contrast, Renée, which uses
on smaller datasets A.F-AmmaonTitles... 13 KK A.-AAmazon-1311

---

Published as a conference paper at ICLR 2024
Table I: LEVER can be applied improve any OvA-based approach. When used with leading OvA approaches LEVER consistently boosts tail performance across all benchmarks, increasing PSP on average by 5.3% while maintaining comparable precision (1.4% gain on average). Coverage metrics (reported in Table 5 in the appendix) show similar trends with average gain of 6.5%.
Model
LF-Amazon Titizonnniitee-11 13IK
L.F-Amazon-B 13IK
P@
P@3
P@5
PSP@1
PSP@3
PSP@5
P@1
P@3
P@5
PSP@
PSP@3
PSP@5
ELIAS
37.28
25.18
18.14
28.95
34.45
39.08
43.03
29.27
21.20
33.49
40.80
46.76
ELIAS LEVER
42,86
28.37
20.16
36,30
41.05
45.43
47.38
32.24
23.22
38.97
46.74
52.79
CascadeXML
36,28
24.88
18.18
26.50
33.21
38.81
43.76
29.75
21.58
34.05
41.69
47.96
CascadeXML + LEVER
43.58
28.79
20.63
36.24
41,83
46,95
48.24
32,82
23.73
39.09
47.55
54.18
Renée
46.05
30.81
22.04
38.47
44.87
50.33
48.05
32.33
23.26
39.32
47.10
53.51
Renéo + LEVER
46.44
30.83
21.92
39.70
45.44
50.31
49.19
33.30
24.04
40.64
48.48
54.87
LF- Wi-WWipppdi---00
LF Amazzon Titles-1.3M
P@
P@9
P
PSP@8
PS
PSP@5
P@
P6
5
PSP@1
PSP
PSP@8
ELIAS
81.94
62.71
48.75
33.58
43.92
48.67
47.48
42.21
38.60
18.79
23,20
26.06
ELJAS LEVER
82.44
63.88
50.03
36.94
49.28
55.03
48.91
43.17
39.28
23.68
27.43
29.72
CascadeXML
43.29
47.14
37.73
15.92
20.23
23.16
CascadeXML LEVER
80.10
60.41
46.44
36.79
46.65
50.99
47.98
42.02
38.12
20.06
24,51
27.28
Renée
84.95
66.25
51.68
37.10
50.27
55.68
56.10
49.91
45.32
28.56
33,38
36.
Renéo + LEVER
85.02
66.37
51.98
42.93
$5.00
60.29
56.01
49.43
44.85
33.55
36.82
38.81
LF LF-AO1-2770
LF- L.-WikiHierarchy-- 1M
P@1
P@3
P@5
PSP@1
PSP@3
PSP@5
P@
P@
P@5
PSP@
PSP@3
PSP@5
ELIAS
40.83
22.33
14.91
13.29
21.46
25.22
95.27
94.25
92.45
17.15
24.41
30.01
LEVER
94.02
91.97
28.27
36.80
42.13
Cas
14.82
23.19
94.88
93.69
91.79
16.03
22.87
CascadeXML LEVEER
39.41
21.78
14.99
11.96
21.30
27.59
94.77
93.54
91.56
20.14
27.49
33,01
Renée
40.97
23.34
15.85
14.76
26.45
32.19
95.01
93.99
92.24
19.69
27.36
33.20
Renée + LEVER
41.70
24.76
17.07
20.38
37.07
45.13
95.19
93.91
92.07
24.76
32.63
38.15
Table 2: Comparison of LEVER with other tail specific XC approaches. LEVER outperforms regularization and augmentation-bassd methods by average of 4% in coverage and 3% in PSP.
I.F-Amzzon 13KK
LF-AOOL-270O
C@3
C@5
P
PSP@3
PSP@5
C@1
C@3
PSP@
PSP@3
PSP@5
Renée
31.31
53.50
61.03
38.47
44.87
50.13
12.40
29.77
36.53
14.76
26.45
32.19
+TAuG
29.47
51.52
15.72
26.74
32.35
Renée BoW
30.03
51.78
59.17
36.96
42.86
48.09
12.67
34.12
43.45
15.58
30.28
37.90
Renée L2Reg
31.66
53.65
60.80
38.74
44.53
49.49
21.07
26.27
12:21
20.09
24.36
Renée GLaS
31.90
54.02
61.15
38.74
44.53
49.49
12.36
29.41
36.06
14.67
26.11
36.75
Renée Gandall
33.17
55.36
62,22
40.49
45,83
50.96
12.63
29.82
36.31
15.10
26.64
32.17
Renée LEVER
32.50
54.59
61.42
39.70
45.44
50.31
17.43
42.54
52.01
20.38
37.07
45.14
LL--Wikpeeiia-500K
LE-WakiHierarchyyll
Renée
22.90
50.08
61.59
37.10
50.27
55.68
6.72
11.49
14.65
19.69
27.36
33.20
Renée TAUG
19.88
44.74
56.13
33.76
46.54
52.16
3.59
7.19
9.94
16.95
24.06
29.69
Renée BaW
22.92
49.64
61.40
36.66
49.79
55.55
7.84
14.77
18.39
24.25
31.10
36.30
Renée 1.2Rag
26.52
53.95
65.14
39.55
52.42
57.43
5.61
9.96
12.98
18.56
25.90
31.49
Ronéo GLaS
23.45
52.02
63.90
37.27
31.54
37.15
6.89
11.82
15.08
20.07
27.82
33.70
Gandalf
23.09
49.87
13.17
17.52
36.09
Ronée LEVER
29.46
58.53
70.29
42.93
$5.00
60.29
9.32
16.41
20.29
24.76
32.63
38.15
the NGAME encoder for initialization, shows comparatively modest gains with LEVER. More-
over, Table 4 in the appendix illustrates the performance of LEVER when combined with XReg (Prabhu et al. 2020), an extension of Parabel, showcasing that LEVER can effectively combine with non- Don-DNN--a based methods too.
Comparison with Tail Extreme Classification Methods: In Table 2 present comparative analysis of Renée + LEVER against leading tail sabel-specialized methods. Note that these ap- proaches be easily integrated with OvA classifiers without any architectural modifications.
8

---

Published as a conference paper at ICLR 2024
These methods can be broadly categorized into two classes: (1) regularization- such GLaS and Le-reuulaiizationn GLaS promotes the proximity of classifiers for labels with similar ground truths, while I2-regularization introduces additional L2 loss between tail expert label embed- dings and label classifiers. (2) Augmentation-baset such as TAUG and Gandalf, which introduce additional training data for lahels. Detailed comparisons with other prominent Extreme Classifica- tion methods, including ER-Transformer (Zhang et al. 2021). ELIAS, CascadeXML, NGAME, and ECLARE (Mittal al. 2021b), are provided in Table 7 within the appendix. Our primary focus here is on tail label performance, hence we report PSP and coverage metrics.
LEVER consistently outperforms the second- method by an average margin of 4% in coverage and 3% in PSP. Notably. on datasets characterized by significant skew and multi-intent scenarios, LEVER exhibits substantial gains in comparison to approaches like GL and Gandalf, which rely on ground truth data to model label correlations. For example, in the query completion task on the AOL dataset, the label "who wrote To Kill a Mockingbird" co-oceurs with labels like "wholesale I-shirts" or "who am I" as they share the prefix "who". Training classifiers with such diverse targets can lead to associations between dissimilar labels, hampering classifier training. Using Bag of Words (BoW) features from label text to model label connections alleviates the multi-intent and skew issue to some extent, as observed when Renée+ BoW performs better than Renée+ GLaS/Gandalf in LF-AOI-270K and LLF-Wiiilieraachy--ll associations between labels and documents through a tail-expert Siamese network, surpassing raw text-based methods.
However, LEVER goes further by learning semantic
Comparison with Siamese Teacher: Table 15 in the appendix compares LEVER against its cor- responding Siamese encoder-based teacher. LEVER utilizes the teacher to improve OvA perfor- mance on the tail without degrading the classifier performance on the head labels. As a result, the student model in LEVER can surpass its own teacher in overall performance since it outperforms the Siamese teacher on the head labels while more-or-less equalizing on the tail.
Comparison with an ensemble of OvA classifier and tail-expert: To combine the strengths of OvA classifiers and encoder, another option might be to consider an ensemble model that uses predictions from the OvA model for head labels and the encoder predictions for the tail labels. Ta the appendix compares LEVER with an ensemble of OvA (Renée) and Siamese Encoder. LEVER outperforms the ensemble on both precision and tail metrics. A more detailed discussion of this is provided in Sec. C.3 of the appendix.
Choice of expert encoder: LEVER utilizes a 6-layer DistilBert as an expert encoder. In Table II in the appendix we show results for two other light- encoders: 3-layer MiniLM (Wang al.) 2020) and Astec Encoder (Dahiya et al. 2021b). We observe that a superior expert encoder leads to improved performance in both P and PSP.
Effect of varying :: Table 21 and Figure 7 in appendix shows effect of varying 7 on LEVER's performance, Increasing improves performance on tail, while it hurts head and torso labels.
LEVER Computational Cost: Since LEVER is a training time-only modification, it leaves the inference costs unchanged while increasing the training time average by 3.1x. Table 23 in the appendix shows the training time for different models and datasets when combined with LEVER. Note that in ELIAS and CascadeXML, where the train times increase by a greater margin, the gains provided by LEVER are also higher (avg. +6.1% increase in PSP and +2% increase in P). Tables and 26 in the appendix show the break down of the train times for Renée, ELIAS, and Cascade MiscadeXMLrrn respectively.
24 25
6 CONCLUSIONS
This paper presented novel approach to address the challenges of tail performance in Extreme Clas- sification (XC) by focusing on label variance, a previously unexplored factor. It proposed LEVER framework for leveraging tail-robust teacher model to systematically reduce label variance, thereby enhancing the performance of one-vs-all classifiers. further developed an effective instantiation of this framework using specialized Siamese teacher model. Experimental results on various XC datasets demonstrated significant improvements in tail performance metrics when LEVER was in- tegrated with leading extreme classifiers, and advanced the state-of sare--of-he--at art in XC. Finally, this paper also released two new and multi-inten intti--ieettteeee datasets for robust benchmarking in XC.

---

Published as a conference paper at ICLR 2024
REFERENCES
Rahul Agrawal, Archit Gupta, Yashoteja Prabhu, and Manik Varma. Multi-label learning with mil- lions of labels: Recommending advertiser bid phrases for web pages. In Proceedings of the 22nd international conference on World Wide Web, pp. 13-24, 2013.
Rohit Babbar and Bernhard Schölkopf. Dismec: Distributed sparse machines for extreme multi-label classification. In Proceedings of the Tenth ACM International Conference on Web Search and Data Mining. WSDM 17, pp. 721-729 New York, NY, USA. 2017. Association for Computing Machinery, ISBN 978 450346757. doi: 10.1145/3018611.3088844 URL https://dol:
org/ 125//10.10 10..1145/200 30055/30166611 0101113011111 3018741]
Rohit Babbar and Bernhard Schölkopf. Data scarcity, robustness and extreme multi-label classifica- tion. Machine Learning, 108(8):1329-135, 2019.
Mohamed Benaouicha, Mohamed Ali Hadj Taieb, and Malek Ezzeddine. Derivation of "is a" tax- onomy from wikipedia category graph. Eng. Appl. Artif. Intell., 50:265-286, 2016.
Alina Beygelzimer, John Langford, Yuri Lifshits, Gregory Sorkin, and Alex Strehl. Conditional probability tree estimation analysis and algorithms. In Proceedings of the Twenty-Fifth Confer- ence on Uncertainty in Artificial Intelligence, pp. 51-58, 2009.
Kush Bhatia, Kunal Dahiya, Himanshu Jain, Purushottam Kar, Anshul Mittal, Yashoteja Prabhu, and Manik Varma. The extreme classification repository: Multi-label datasets and code, 2016.
URL http:tt::/// httt:://mmaikvarrnn ottp://ooolodddd//////aaaooooooo
dwwpl///waaad/////oooooooo XC/ www.Reessitto...ooooooo
Wei-Cheng Chang, Hs Yu, Kai Zhong, Yiming Yang, and Inderjit S Dhillon. Taming pre- trained transformers for extreme multi-label text classification. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining, pp. 3163-3171, 2020.
Kunal Dahiya, Ananye Agarwal, Deepak Saini, K Gururaj, Jian Jiao, Amit Singh, Sumeet Agarwal, Purushottam Kar, and Manik Varma. Siamesexml: Siamese networks meet extreme classifiers with 100m labels. In International Conference Machine Learning, pp. 2330-2340. PMLR, 2021a.
Kunal Dahiya, Deepak Saini, Anshul Mittal, Ankush Shaw, Kushal Dave, Akshay Soni, Himanshu Jain, Sumeet Agarwal, and Manik Varma. Deepxml: A deep extreme multi-label learning frame- work applied to short text documents. In Proceedings of the 14th ACM International Conference on Web Search and Data Mining, pp. 31-39, 2021b.
Kunal Dahiya, Nilesh Gupta, Deepak Saini, Akshay Soni, Yajun Wang. Kushal Dave, Jian Jiao, Gururaj K, Prasenjit Dey, Amit Singh, Deepesh Hada, Vidit Jain, Bhawna Paliwal, Anshul Mittal, Sonu Mehta, Ramachandran Ramjee, Sumeet Agarwal, Purushottam Kar, and Manik Varma. Ngame: Negative mining- aware
for extreme classification. In Proceed- ings of the Sixteenth ACM International Conference on Web Search and Data Mining, WSDM '23, pp. 258-266, New York, NY, USA, 2023a. Association for Computing Machinery. ISBN 10.4//1111
9781450394079. doi: 10.1145/3339997.3500992 URL
09995997.370333
Kunal Dahiya, Sachin Yadav, Sushant Sondhi, Deepak Saini, Sonu Mehta, Jian Jiao, Sumeet Agarwal, Purushottam Kar, and Manik Varma. Deep encoders with auxiliary parameters for extreme classification. In In Proceedings of the ACM SIGKDD Con- ference on Knowledge Discovery and Data Mining, Long Beach, California, August wttp:///wessaeeeeeerrr/apa/aaaaa
2023b. URL https: www.::/wwicooooooooommom deep- Saae---ncoeeesstttbaaalllllyyy.p htw.:/ateeeettteeoeeeeeeeeeeeeaa
httpp::/www.iicroooff..
Chuan Guo, Ali Mousavi, Xiang Wu, Daniel N Holtmann- Satyen Kale, Sashank Reddi, and Sanjiv Kumar. Breaking the glass ceiling for embedding- classifiers for large output spaces. Advances in Neural Information Processing Systems, 32, 2019.
Nilesh Gupta, Patrick Chen, Hsiang-Fu Yu. Cho-Jui Hsich, and Inderjit Dhillon. Elias: End-to-end learning to index and search in large output spaces. Advances in Neural Information Processing Systems, 35:19798-19988 2022.
10

---

Published as a conference paper at ICLR 2024
Himanshu Jain, Yashoteja Prabhu, and Manik Varma. Extreme multi-label loss functions for recom-
mendation, tagging. ranking & other missing label applications. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pp. 935-944, 2016.
Vidit Jain, Jatin Prakash, Deepak Saini, Jian Jiao, Ramachandran Ramjee, and Manik Varma. Renee: End-to-end training of extreme classification models. In Proceedings of Machine Learning and Systems, pp. To appear, 2023.
Jyun- Yu Jiang, Wei-Cheng Chang, Jiong Zhang, Cho-Jui Hsieh, and Hsiang-Fu Yu. Uncertainty
quantification for extreme classification. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '23, pp. 1649-1659, New York, NY. USA. 2023. Association for Computing Machinery. ISBN 978 doi: 10.1145/3539618.3599 /ttp////LA////////////////////// 355144/33355611888888788 3591780
Ting Jiang, Deqing Wang, Leilei Sun, Huayi Yang. Zhengyang Zhao, and Fuzhen Zhuang. Lightxml: Transformer with dynamic negative sampling for high-performance extreme multi-label text clas- sification. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp- 7987-7994, 2021.
Sham Kakade, Karthik Sridharan, and Ambuj Tewari. On the complexity of lin- ear prediction: Risk bounds, margin bounds, and regularization. In Advances in neural information processing systems. Curran Associates, Inc., 2008. URL
nww.ipsssssp... cttpsseccppppppppeee files/ httee///appe/220/EELLLL 33699688805540338594777778888883 Pare
Siddhant Kharbanda, Atmadeep Banerjee, Erik Schultheis, and Rohit Babbar. Cascadexml: Re- thinking transformers for end-to-end multi-resolution training in extreme multi-label classifica- tion. In S. Koyejo, S. Mohamed, A. Agarwal. D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 2074-2087. Curran Associates, Inc., 2022. URL httt:s:: ngs neurips cc cwpppppppppppccccppppp oasprissspeppppeeeeeeeeee//prrre file FB12//000755aa880000744444333333
Siddhant Kharbanda, Devaansh Gupta, Erik Schultheis, Atmadeep Banerjee, Vikas Verma, and Ro- hit Babbar. Gandalf: Learning label correlations in extreme multi-label classification via label features, 2024.
net. htw.l/peneeeeeeeeeeeereeeeeeeeee
Gyuwan Kim. Subword language model for query auto-completion. 005,09999900599, 2019.
arXiv preprint
Andreas Maurer and Massimiliano Pontil. Empirical bernstein bounds and sample variance penal- ization. arXiv preprini arXiv:0907.37400, 2009.
Aditya K Menon, Ankit Singh Rawat, Sashank Reddi, Seungyeon Kim, and Sanjiv Kumar. A statistical perspective on distillation. In Proceedings of the 38th International Conference on Machine Learming, 2021a.
Aditya Krishna Menon, Sadeep Jayasumana, Ankit Singh Rawat, Himanshu Jain, Andreas Veit, and Sanjiv Kumar. Long-tail learning via logit adjustment. In International Conference on Learning Representations, 2021b. URL htttttt:/
Bhaskar Mitra and Nick Craswell. Query auto-coppletion for rare prefixes. CIKM '15 Proceedings of the 24th ACM International on Conference on Information and Knowledge Management, 2015.
Anshul Mittal, Kunal Dahiya, Sheshansh Agrawal, Deepak Saini, Sumeet Agarwal, Purushottam Kar, and Manik Varma. Decaf: Deep extreme classification with label features. In Proceedings of the 14th ACM International Conference on Web Search and Data Mining, pp. 49-57, 2021a.
Anshul Mittal, Noveen Sachdeva, Sheshansh Agrawal, Sumeet Agarwal, Purushottam Kar, and Manik Varma. Eclare: Extreme classification with label graph correlations. In Proceedings of the Web Conference 202/, pp. 3721-3732, 2021b.
Greg Pass, Abdur Chowdhury, and Cayley Torgeson. A picture of search. Proceedings of the 1st international conference Scalable information systems, 2006.

---

Published as a conference paper at ICLR 2024
John Platt. Probabilistic outputs for support vector machines and comparisons to regularized likeli- hood methods. Adv. Large Margin Classif., 10, 06 2000.
Yashoteja Prabhu, Aditya Kusupati, Nilesh Gupta, and Manik Varma. Extreme regression for dynamic search advertising. In Proceedings of the 13th International Conference on Web Search and Data Mining, WSDM '20, pp. 456-464, New York, NY, USA, 2020. Association for Computing Machinery. ISBN 9781450368223. doi: 10.1145/33311113377766. URL
http://///oi...11.11111111.1 337773333631113377663 3371768
Mohammadreza Qaraei, Erik Schultheis, Priyanshu Gupta, and Rohit Babbar. Convex surrogates for unbiased loss functions in extreme classification with missing labels. In Proceedings of the Web Conference, pp. 3711-3720, 2021.
Erik Schultheis, Marek Wydmuch, Rohit Babbar, and Krzysztof Dembezynski. On missing labels, long-tails and propensities in extreme multi-label classification. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 1547-1557, 2022.
Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. Minilm: Deep self- attention distillation for task-agnostic compression of pre-trained transformers, 2020.
Tong Wei, Wei- Wei Tu, Yu-Feng Li, and Guo-Ping Yang. Towards robust prediction tail labels. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pp. 1812-1820, 2021.
Marek Wydmuch, Kalina Jasinska-Kobus, Rohit Babbar, and Krzysztof Dembezynski. Propensity- scored probabilistic label trees. In Proceedings of the International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 2252-2256, 2021.
Lee Xiong, Chenyan Xiong, Ye Li, Kwok Kwok-Fung Tang, Jialin Liu, Paul Bennett, Junaid Ahmed, and Arnold Overwijk. Approximate nearest neighbor negative contrastive learning for dense text retrieval. In International Conference on Learning Representations, 2020.
Nishant Yadav, Rajat Sen, Daniel N Hill, Arya Mazumdar, and Inderjit Dhillon. Session- aware query auto-completion using extreme multi-label ranking. In Proceedingy of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pp. 3835-3844, 2021.
Ronghui You, Suyang Dai, Zihan Zhang, Hiroshi Mamitsuka, and Shanfeng Zhu. Attentionxml: Extreme lulti-label text classification with multi-label attention based recurrent neural networks. arXiv preprint arXiv: 1188//1117777 137:138-187 2018.
Torsten Zesch and Iryna Gurevych. Analysis of the Wikipedia category graph for NLP applications. In Proceedings of the Second Workshop on TextGraphs: Graph- Based Algorithmss for Natural Language Processing, pp. 1-8, Rochester, NY, USA, 2007. Association for Computational Lin- guistics. URI. htttttt:: a antho www.org////o.. 0200
Jiong Zhang, Wei-Cheng Chang, Hsiang-Fu Yu, and Inderjit Dhillon. Fast multi-resolution trans- former fine-tuning for extreme multi-label text classification. Advances in Neural Information Processing Systems, 34:7267-7280, 2021.
Ruohong Zhang, Yau-Shian Wang, Yiming Yang, Donghan Yu, Tom Vu, and Likun Lei. Long- tailed extreme multi-label text classification by the retrieval generated pseudo label descrip- tions. In Andreas Vlachos and Isabelle Augenstein (eds.). Findings of the Association for Com- putational Linguistics: EACL 2023, pp. 1092-1106, Dubrovnik, Croatia, May 2023, Associ- ation for Computational Linguistics. doi: 10.18653/v1/2233.fmmm http:///222...nnnnsoeeeeee findings-cacll... URL https: lacllantholoog.org///oo/o winiiingdiiiienenneneiiennn eacl.88 81
12

---

Published as a conference paper at ICLR 2024
CONTENTS
Introduction
Related Work
2.1
Extreme Classification
2.2
Enhancing Tail Performance XC
3
LEVER: Label Variance Reduction in Extreme Classification
Preliminaries
3.2
LEVER Framework
3.3
Siamese-Style Teacher for LEVER
4
Contributed Datasets
5
Experiments and Results
Conclusions
Theoretical Proofs
14
B
Dataset details
17
Dataset Statisties
17
B.2
QK-20M Dataset
17
B.3
Multi-intent dataset preparation
17
B.3.1
LF-AOL-270K
17
B.3.2
LF-WikiHHirrachh--MM IM
18
Additional results
20
LEVER's performance on non-DNN methods
20
Comparison with SOTA and Tail XC methods
20
Comparison with ensemble between tail Expert and OvA classifier
20
Effect of Re- Re-ranking on LEVER and other Tail XC approaches
22
Ablations
22
D Model Details and Hyperparameters
25
Tail expert Siamese Encoder
25
D.2
ELIAS
26
CascadeXML
28
D.4
Renée
29
D.5
ReRank TAUG
30
D.6
Gandali
30
13

---

Published as a conference paper at ICLR 2024
D.7 LEVER
31
D.7.1 Hyperparameters
D.7.2 Training time
32
A THEORETICAL PROOFS
Theorem 1. Let R. R be the population risk and empirical risk for a binary classification loss L. Let My be the uniform covering number (Menon al., (2021a) corresponding to L. Then, given the definitions established earlier, for any 5 E (0, 1), the following inequality holds with probability least 1 - over sampling the data points {x]
R S R + 1000
(V Vx C((= w wxx] + Ex V [x
log(Mx/0) log(MN/0)
N
N
(10)
where, VxL(Pa, w x) and Vy[y|x] are the variances in the loss function contributed by sampling of data points {x] and conditional sampling of labels {.) respectively.
Proof. Applying Proposition 2. from (Menon et al. 2021a to our setting gives the following initial result:
R R R 0 OV x) Mog /8//N + log(MMx/8/NN)
(11)
The following simplifications be made by leveraging basic probabilistic calculus:
Vx ( w x]] V w x|]]| + ELIV, w x]] (by law of total variance)
=Vx[EvICyy, w x)]x]] + Ex [V,yf x) + (1 v)f(0, w x)|x]]
-VxCC(p=: w x)] B w x) -(0,, w x
VxIC((p: w Tx)] [
(12)
where, d, (CS(1, w., - xx))
& ( w T x) + (2(0, 2C. w wTx)./(0, Tx)
Since f 2 0 we get,
d < C² (21 x) + (2 w Tx) x)
(13)
Using 13 in 12 gives,
Vxy(C(y, w x)] < Vx([(p>. w x)] + & ((,, x) (x)++ ((,,
(14)
Assuming f(0,0) f(1,0) fo (a small constant), and applying the Lipschitz continuity of gives,
fly. w x) - 0() s L/w LWB Ay E (0.
(y.wTx 5 LWB fo
(15)
Using 15 in gives.
V Vx L(y, x)] < V2[C(pz. T + +I(WBB fo)
0 (x(((((p w T VV
(16)
Using 16 in 11 completes the proof,
R R 0 V w Txx)] Ex (bb1
log(MN/4 /6)
log(Ms) (N/8)
N
(17)
14

---

Published as a conference paper at ICLR 2024
Lemma 1. Assuming the loss weighting factor C' is defined in Equation as C N where S the threshold defined in Equation and N the number of training point, the variance term
V Ex V (CWWB in Theorem is bounded by N(LW BY
Proof. Recall from Section gives,
3.1 that P= P(y = 1/x) and Ex (p S S. Using V Volulx] = P= P=(1-pz)) P&)
(N
NLWB
== Bx
Ex(r2]
(18)
Theorem 2. Let R. R be the population risk and empirical risk for binary classification loss L. Let My be the uniform covering number (Menon et al. (021a) corresponding to L. Also, let the teacher be imperfect with maximum possible error in relevance estimates bounded by E llx px Dxll Then, solving the following regularized optimization problem:
N Σ
A N
1- X
min R₂
Σ Clyn, w x;) +
Cipe w TTx))
(19)
w
N
and setting À to minimize population risk will given the following bound for any E (0. 1), the following inequality holds with probability at least 1 - 5 over sampling the data points {x]==*
C A
R
+ V
+
(20)
bV 6-
b
where, a Ve log log(MN/0)
Tog(MN/H N
b VSCLWB
c ECLWB (21)
Proof. Teacher tends to be imperfect with relevance estimates P.- this case, let us train the classifier using targets 8e Ay + (1 (1- x)p.. Let the corresponding population and empirical risks when trained 8a be R. R. respectively. Then, the following holds:
R, R R. + R, R.
R |-RR + RR R.ll
The first term can bounded as follows:
R- R. = ExEyix C(yz, w x) - L((gz + (1- \) w x)
Exx L(PE, w x) L(>p= (1 (1-))pr. w
T x)
Assuming, L(y, w x) Cl, w xx)
-y) Txx
R R = Ex[(1 A) - (Cf( w Tx) - f(00w =x)]l < (1 >) Palloo max (CS(1, w T x) -S(0, w xx)
< (1 A)E(C () + fo)
~ ((11 A)
(22)
where E is the upper bound over the error in the teacher's 's relevance estimates.
The second term R, R can be bounded by applying A
15

---

Published as a conference paper at ICLR 2024
RS S R. + +0 +0 V My (1 X
[lg( log(Mx/0)
N
Now, Vu (1 Alp x Vu(Avix] [ x
R. R. +0 V2 X2 S(CLWB) B)²
log(MN/0)
log Log(NN/1 /0)
N
N
N
S
where, N
Enpr Ex y
(23)
As result:
R +0 22 V V
log(MN /8)
log(MN/0)
O((1
A)ECLW B)
N
N
(24)
As À is regularization hyper-parameter, it value needs be set so minimize the generalization error Theoretically, this can be achieved by solving:
min a X262 + (1
A)c
where, a V log(MN/8)
N
CLWB
b VSlog(MN// (8)
N
ECLWB
(25)
Let's assume a reasonably small bias in teacher estimate. Specifically, let < b which means that the error due to teacher bias is relatively smaller than the error due to label variance.
Now, taking the derivative w.r. À and setting it to 0, we get:
C
X
(26)
b 1
c
2
Va+ X222 + +1 (1- - A)c = / a- -
+c
(27)
b
Theorem 3. Given label z, and pair of data points X, Xt. Let Pa. Pb be the probabilities that the label is relevant to points b respectively. Then, assuming that (9) is fully minimized, the expected loss in (9) is minimized for Pa 11(1 e- xx+ ), pb 1/(1 e xxte))
Proof. The expected loss between the triplet is given by:
Pa(1 - (1) log(1 xx x; xx) + po Pa) log(1 + xxxxxx-aTo
Assuming A = z xs. z x. and taking the gradient w.r.t z gives,
(xb xa)
+ pb pa) e- (x, - xb)
=P -pb)
1 1t
1
Xi. - Xa (e (pp -po) - -6 - -pa)
(28)
1 c
16

---

Published as a conference paper at ICLR 2024
Setting Pa 1/ e
xx (x))x+)))) ). 1/(1 e xo)) in 28 gives,
(c xa'xo-z'' (x'xx-='x0)(( (c xx+o))
(c- (x+0))
X6 Xa
+ e
(1+e-(alx (xxxx))))) (1+e xa. (-
0
(29)
From 29 we see that the derivative of the loss is 0 when Pa 1/(1 + e xxx'xx+o0 cx(xxxx)))) x)) Pb
1/(1 + (x x)- thus minimizing the expected loss, Note that this calibration strategy is in line with posthoc calibration strategies discussed in Platt (0000), where model learned, and then a parametrized sigmoid function is fit to learn the relevance probabilities.
B DATASET DETAILS
B.1 DATASET STATISTICS
Table shows the statistics of benchmark datasets including the newly contributed multi-intent datasets.
Table 3: Dataset Statistics. Pos-80% is an imbalance metric (Schultheis et al. 2022) defined as minimum fraction of class labels that retain 80% positive labels in the dataset. Lower value corresponds to higher skew.
Dataset
Train Docs
Does
Labels
Avg. LabelvDoc Avg. Does/Tabel
AF-AmazoTTidee 131K
294,805
134,835
131,073
2.29
5.15
47.5
Existing
L.F-Amazo-113
294,805
134,835
131,073
2.29
5.15
47.5
LF WikiSeeAlso 320K
693,082
177,515
312,330
2,11
4,68
37.4
WikiieddiaaaS0k
1.813,391
783,743
501,070
4.77
LLF-AmazoTTites-- 1.0ss---.... 1.3
2,248,619
970,237
1,305,265
22.20
38.24
28.9
New
-F-OOK-270KK
3,922.479
519,352
272,825
2.01
28.83
11.6
LF- LJ-WiikiHierarchy-MMM
1,589,378
397,952
976,214
25.98
42.31
7.3
B.2 QK-20M DATASET
Query Keyword (QK) matching is an essential element in applications such sponsored search. In these applications, users express their intent by querying a search engine, while advertisers bid on relevant phrases from the same domain, referred as keywords. The retrieval (or matching sys- tem) is responsible for matching user queries to relevant advertisements. To train query-keyword matching system, we build a dataset using click logs from Bing. We begin by considering 20M pop- ular advertiser bid phrases (or keywords), and then corresponding each keyword we add relevant queries to the ground truth based on whether there was a user click on the keyword when the user searched for that particular query.
B.3 MULTI-INTENT DATASET PREPARATION
B.3.1 LF-AOL-270K
Task Description:
Query auto-completion involves matching query prefix to completing suffixes, egg. given a prefix, 'cheap nike s' recommending suffix completions like 'shoes', 'shirts' etc. LF- AOL-270K is curated from AOL search logs (Pass et al. (006) for the task of query auto-completion where (prefix, suffix) pairs are modeled as (doc, label) pairs. Retrieved suffixes from this task can be combined with user prefixes to get full query completions proposed in (Mitra & Craswell 2015).
Dataset generation: The dataset generation process involved three steps (i) Pre-processing, (ii) Prefix-suffix generation, and (iii) Post-processing.
Queries in AOL search logs were de-duplicated and -on-alphanumrric characters removed. Queries with less than three characters were filtered since compleeinnt is rarely required for those. Additionally, steps prescribed in (Kim) 2019) were followed for pre-pree pre-processing and train-test splits ereation.
Pre-processing:
17

---

Published as a conference paper at ICLR 2024
Prefix-suffix generation: After pre-proce preppreeessng shortlist of the top 10M popular suffixes was derived from the train split, based on their frequency in queries. These suffixes are popular n-grams (word-level) up to 100 characters appearing the end of queries. Sampling was done to ensure that each train query has at least one suffix from 10M suffix shortlist. Ground truth suffixes were added for sampled prefixes yielding 9.3M suffixes and 5.67M distinct prefixes (training points). Using the 10M suffix shortlist, the process of sampling prefixes was repeated in the test split, resulting in 460K suffixes.
Tost-processing Train-test leakage was avoided by removing all prefixes in the test set that ap- peared in the train set. The suffix (label) set is derived from the intersection of train and test suffixes to have fixed label set. Finally, the dataset contains 272K labels (suffixes), 3.9M training points (prefixes), and 519K test points (test prefixes).
Code to create the dataset from raw AOL search logs is available here -.
B.3.2 JFF-WIKIHHERARRHHY-I
Task description: Taxonomy completion task involves matching category with its generalized parent categories. A.--WikkHierrarcy--MM uses Wikipedia categories to build a taxonomy completion task where documents are categories and labels are its parent categories. Articles in Wikipedia are assigned categories, which serve as semantic tags. (e.g. 'FIFA World Cup 2022' article has a category tag of 'Football'). These categories are arranged in a taxonomy-like like structure where each category is linked to zero or more parent categories. The parent of category its direct generalization, e.g. the category 'Football' has direct parent categories 'Athletic Sports', "Team sports' and 'Ball Games', This taxonomy-lic structure is called Wikipedia Category graph (WCG) and has been well studied in (Zesch & Curevych 2007 Benaouicha et al. 2016).
Dataset generation The dataset generation process involved four steps (i) Raw data collection, (ii) Pre-processing, (iii) Label set generation from WCG and (iv) Post-processing.
Raw data collection: The WCG is created by using the English Wikimedia dump as of 03/23 dump contains the list of all Wikipedia categories and their links.
? The
Pre-processing:
To create the WCG we first filter out all meta categories used for Wikipedia mainte- nance, e.g. 'Wikipedia missing topics', 'Wikipedia new articles', 'Categories for renaming' etc. The complete list of filtered meta- meta-categories are released as part of the code. Post filtering, the resulting WCG is directed acyelic graph with 1,993,526 categories (nodes) and 5,781,011 edges. Each edge is a document-labe pair.
Label set generation from WCG: The WCG in its current form only contains direct parents and misses out on potentially important ground truth information. For example, the category 'Foot- ball' will not have categories like 'Sports', 'Athletic Sports', and 'Team activities' as its labels as they are not its direct parents. On the other hand, adding all reachable nodes as labels leads to vague document-label pairs. For example, starting from FFootball' one can reach the category 'Cos- mopolitan mammals' as follows: 'Football' + 'Athletic sports' + "Sports by type' + 'Sports' + 'Entertainment" - 'Human activities' - 'Humans' -> "Cosmopolitan mammals'.
To maximize the relevant ground truth document-label pairs while also avoiding wrong matches like 'Football' + "Cosmopolitan mammals' we limit the traversal to a maximum depth of 3 which gave the optimal trade-off (i.e. maximize true positives while avoiding false positives). Thus, in the above example, only categories up to 'Sports' are added as labels. Please refer to Fig. [ for more clarity. Subsequently, we get 1,987,330 documents and 976,214 14 labels with 51,643,812 edges between them. Note that both the number of documents and labels are less than the total number of categories (1,993,526). Some categories will not have a parent and therefore won't 't be added document. Similarly, categories that are not parents of any category will not be added as labels. The final dataset is created by taking an 80%-20% random train-test split.
The dataset contains categories that occur as both documents and labels. For example, the category 'Football' occurs both as a label (for 'Football clubs". 'History of Football', etc), and as a document. Since a category will never have itself as a label we filter off pairs like
Post-processing:
18

---

Published as a conference paper at ICLR 2024
Teams
Sports
Games
Depth
1
Physical Skill
Team activities
Sports by type
Games by type
Balls
Depth
T
sports
Athletic sports
Ball game
Depth
Football
Figure 1: Snapshot WCG graph starting from category 'Football', All categories (nodes) reach- able from 'Football" till the depth of 3 are added to its ground truth label set.
LF-AmazonTTitls-1131
LF-AOL-270K
Encoder
Encoder
OvA (Renée)
OvA (Renée)
Contribution P8S
Contribution R@5
2
2
2
213
2
2
Training Points Label
Training Points Label
(a)
(b)
LF- Fi-Wiiipddii-SOOKK
LF HT-Wiilltteerrrcyy-11
20
Encoder
Contribution PiS
OvA (Renée)
Contribution to Pgs
OvA (Renée)
15
10
21
29
27
2 2
2
210
2'
2
Avg Training Points per Label
Avg Training Points per Label
(c)
(d)
Figure 2: P@5 comparison of Siamese Encoder (blue) and OvA Classifier Renée (orange) ho- mogeneous (LF-AmazonTTttls- 131K: K: Fig. 2a LF-Wikipedia-500K: Fig. 2c and heterogeneous datasets (LF-AOL-270K: Fig. 2bb LLF-iiiHieraarhyy- IM: Fig. (d]] Labels are partitioned into
cqui-volume bins based on their frequencies along the X-axis. The difference in performance (on both head and tail) is wider for heterogeneous datasets.
'Football' Football' during evaluation to not unfairly penalize Siamese-based models that rank such pairs at the top.
The LF-WWikiHieerrchyy- IM dataset is available here
19

---

Published as a conference paper at ICLR 2024
C ADDITIONAL RESULTS
C.I LEVER'S PERFORMANCE ON NON-DNN METHODS
Table illustrates the performance of LEVER when combined with XReg (Prabhu et al. 2020
an extension of Parabel, showcasing that LEVER can effectively combine with non-DNN based methods.
Table 4: Performance Comparison of XReg and XReg + LEVER on LF-AOL-270K and LF- AmazonTiddes-I31K
Dataset
Model
P@1 I
P@3
P@5
PSP@
PSP@3
PSP@5
C@1
C@!
C@5
I.F.AmazonTilll Tittle--iittl-11111 131K
XReg
33.1
22.3
16.0
24.5
29.4
31.54
20.22
36.63
42.84
XReg LEVER
24.7
17.6
31.4
25.84
43.47
1.F-A01-270KK
XReg
27.0
14.3
9.9
7.0
11.0
14.1
4.18
10.58
14.44
XReg LEVER
26.1
14.3
10.1
9.2
17.9
24.0
6.79
20.59
28.65
C.2 COMPARISON WITH SOTA AND TAIL XC METHODS
Table 5 demonstrates the enhanced performance achieved by applying LEVER top- perfffrrfming Extreme Classification (XC) methods, including ELIAS, CascadeXML, and Renée. On average PSP metrics are boosted by 5%, Coverage improves by 6.5% and Precision improves by 1.4%.
Table presents a comparison of LEVER with various OvA-based methods
ELIAS, and CascadeXML) and Siamese encoder methods (NGAME and ECLARE). It's worth not- ing that for WikiHierarchy-IM. OvA and Siamese approaches exhibit significant trade-offs between precision and tail metrics.
C.3 COMPARISON WITH ENSEMBLE BETWEEN TAIL EXPERT AND OVA CLASSIFIER
In the ensemble model, for each data point, the encoder and OvA model provide shortlist of top-k labels along with their prediction scores. These two shortlists (containing a total of up to 2k labels) need be combined into single shortlist of & labels by tie- fie-breaking elaborated below. First, the labels with a frequency more than the cut-off are considered from the OvA's shortlist. Similarly, the labels with frequency less than the cut-off are considered from the encoder's shortlist. Cut-off's derived on the basis of the cross-over points between Encoder and Renée in the decile wise plots shown in Fig. [ Then, the two resulting shortlists scores from both models and retaining only the & overall highest-scoring labels. Table 8 shows that LEVER clearly outperforms the ensemble model in 3 out of 4 datasets across all metrics. In the case of LF- WikiHiierrccrh -IM, the ensemble model shows gains in coverage metrics (~4-5%), this comes the expense of significant loss Precision (~30%). Figure compares the performance of LEVER with the ensemble model and here we see clear dip in the torso deciles. To better understand why ensemble curve doesn't it exactly mimic the OvA curve before the cutoff and encoder curve after the cutoff, consider the following toy example:
combined by considering the assigned label
Assume a dataset D with labels which are partitioned into 3 deciles (head, torso, and tail deciles).
Out of 8 labels. 3 belong to the head decile (Hi, H2, Ha), 2 belong to the torso decile (O1, O2) and the remaining 3 belong to the tail decile (Ti, T2, Tz). The cut-off threshold partitions the label set into 2 sets: (i) labels with frequency greater than cut-off: (H1, H2, H3, O2) and labels with frequency less than cut-off: TO,,T,,T, Tz). Assume a data point d has ground truth labels: (H1, ; H2, O1, O2, Ti).
Below we list the predictions of different models in the format of "label ID:model score"
Top-5 encoder predictions (Ti 0.8, T2 0.6, T3 0.4, O, 0.2, O2 0.1).
Top-5 OvA predictions (H₃ 0.7, H2 : O.5, Hy 0.3, O2 : 0.2, 0, 0.1)
To compute the ensemble model predictions, first restrict the predictions of the individual models based on the cutoff frequency. ie Encoder's predictions are restricted to (O,, Ti T2, T₂) and OvA predictions restricted to (H,, H2, H3, O2). This gives the following filtered shortlists:
20

---

Published as a conference paper at ICLR 2024
Table 5: Using LEVER with leading OvA approaches improves their tail label performance con- sistently across benchmarks, with an average gain of 5% in PSP and 6.5% in coverage (C), while maintaining comparable precision (P) with an average gain of 1.4%.
AF-AmazonTinbc-1311 TitKes-331K
Model
P@1
P@5
PSP@1
PSP@3
PSP@5
C@1
C03
Ce5
ELIAS
37.28
25.18
18.14 14
28.95
34.45
39.08
23.73
42.36
49.06
ELIAS LEVER
42.86
28.37
20.16
36.30
41,05
45,43
29.81
49.88
56.25
CascadeXML
36.28
24.88
18.18
26.50
33.21
38.81
21.38
40.58
48.46
CascadeXML LEVER
43.58
28.79
20.63
36.24
41.83
46,95
29.43
50.61
57.90
Renée
46.05
30.81
22.04
38.47
44.87
50.33
31.31
53.50
61.03
Renée + LEVER
46.44
30.83
21.92
39.70
45.44
50.31
32.50
$4.59
61.42
LF LF-AAmzzon-le 131K
ELIAS
24.27
21.20
33:49
27.04
49.10
57.34
ELIAS LEVER
23.22
55.27
CascaleXML
43.76
29.75
21.58
34.05
41.69
47.96
27.30
50.18
58.81
CascadeXML + LEVER
48.24
32.82
23.73
39.09
47.55
54.18
31,26
55.97
64.81
Renée
48.05
32.33
23.26
39.32
47.10
53.51
31.49
55.81
64.61
Renée + LEVER
49.19
33,30
24.04
40.64
48.48
54,87
32,39
56.81
65.20
ELJAS
27.36
20.66
23.83
13.30
27.72
ELIAS LEVER
45.99
30.28
22.78
30,00
34.16
37.52
16.56
33,06
41.34
CascadeXML.
30.21
18.72
14.05
12,46
14.15
16.25
6.70
13.38
17.72
CascaleXML LEVER
38.84
25.43
19.36
21.62
25.85
29.45
12.01
25.12
32.59
Renée
47.79
31.73
23.82
31.13
36.49
40.37
17.02
35.32
44.56
Renée LEVER
47.89
31.52
23.53
32.44
37.45
40.99
17.78
36.33
45.31
LF-Wikipediaa-00K
ELIAS
81.94
62.71
48.75
33.58
43.92
48.67
19.62
41.30
51.36
ELIAS LEVER
82.44
63.88
50.03
36.94
49.28
55.03
23.55
50.81
63.04
CascadeXMI.
77.00
58.30
45. 10
31
39.35
43.29
1.5778
33.07
41.46
CascadeXMI. + -LEVER
80.10
60.41
46.44
36.79
46.65
50.99
23.99
49.16
60.13
Renée
84.95
66.25
51.68
37.10
50.27
55.68
22.90
50.08
61.59
Renée LEVER
42,93
58.53
70.29
LF-AOL-270KK
ELIAS
40.83
22.33
14.91
13:29
21.46
25.22
10.46
22.85
27.06
ELIAS + LEVER
40.85
22.83
15.57
13.68
24.30
30.43
10.52
26.33
33.56
CascadeXML
41.20
22.12
14.82
12.58
19.53
23.19
8.73
19.47
23.47
Cascale LEEVER
39.41
21.78
14.99
11. 11.96
21.30
27.59
7.86
22.11
29.89
Renée
23.34
32.19
29.77
36.53
Renée LEVER
41.70
24.76
17.07
20.38
17.43
52.01
I.F.WikiFFierarchy.-hh
ELIAS
95.27
94.25
92.45
17.15
24.41
30.01
4.00
7.78
10.49
ELIAS LEVER
94.02
91.97
89.50
28.27
36.80
42.13
10.78
18.88
23.03
Cas
94.88
93.69
91.79
16.03
22.87
28.17
3.12
6.17
8:52
LEVEE
94.77
93.54
91.56
20.14
27.49
11.22
14.13
Renée
95.01
93.99
92.24
19.69
27.36
33.20
6.72
11.49
14.65
Renée + LEVER
95.19
93.90
92.07
24.76
32.63
38.15
9.32
16.14
20.29
LF-Amazon T'tlee-1.3M
ELIAS
47.48
42.21
38.60
18.79
23.20
26.06
11.53
21.45
27.33
ELIAS LEVER
48.91
43.17
39.28
23.68
27.43
29.72
15.10
26.65
32.84
CascadeXML
41.43
37.73
16.75
21.95
CascadeXML. LEVER
47.98
42.02
38.12
20.06
24.51
27.28
12.36
22.57
28.52
Remée
56.10
49.91
45.32
28.56
33.38
36.14
17.31
30.60
37.59
Ronéc + LEVER
56.01
49.43
44.85
33.55
36.82
38.81
21.03
35.70
42.78
Encoder: (7) 0.8,, 0.6, T3 0.4,0, 0.2)
OvA: (H1 : 0.7, Hy 0.5, H3 0.3, O2 0..)
Next, we combine and sort the labels based on the scores from both the encoder and OvA as follows:
(T₂ 0.8, H1 0.7, T2 : 0.6, H2 0.5, T3 0.4, H3 0.3, O2 0.2, 0₁ 0.2)
Finally, retain only the t highest scoring labels our final ensemble predictions:
21

---

Published as a conference paper at ICLR 2024
Table 6: Using LEVER with leading OvA approaches improves their tail performance consistently across benchmarks in Macro-F1 (+4.3% on avg.). Macro-precision (+4.1% on avg.), and Macro- Recall (+5..9%% on avg.). Following ZZang et al. (2023), h values for datasets were chosen based on average labels per point for the dataset. We use k=3 3 for LF-Amazon Titles 131K, LF-A IK LF LF-AOL-270K, k=5 for LF- LF-Wikipedia- 1.3M and LF- LF-iiiiHHierrcchy-ll
k= for TT-AmmzzonTiles
Model
LF-Amazon Mitlzsiiitles13111
LF-Amazon-131k
LF-Wikipediaa-Sue
F1@k
P@r
Rok
F1
Pork
R@L
61.
P@rk
Rork
ELIAS LEVER
24.23
22.82
31.70
28.48
26.46
37.93
27.98
27.97
35.17
29.53
27.70
38.21
33.36
31.16
43.35
44.68
CascadeXML
22.23
20.80
30.36
28.89
26.74
38.83
20.04
20.13
26.75
CascadeMML + LEVER
29.12
26.91
39.04
33.29
30.76
44.13
31.40
31.61
41.70
Renée
32,19
30.36
41.55
32.55
29.95
43.88
35.94
36.78
43.76
Renée + LEVER
33.35
31.78
42.45
35.69
34,05
45.04
40.54
40.71
51.36
Model
LF-Amazon 1.MM
LF-AOL-270K
LE-Wikittierarchy 1M
Fl@k
P@kk
R@kk
F1 @
P@kk
R@k
Fl 0Pk
P@kk
R@k
ELIAS
16.43
15.73
24.68
10.69
9.45
16,18
20.71
23.26
22.43
ELIAS LEVER
18.85
17.92
28.58
13.97
12.87
19.41
26.99
28.75
30.53
CascadeXML
13.58
13:23
22.06
9.00
8:25
13.46
17.73
20.91
19.22
CML LEVVER
16.02
27.02
11.03
15.74
21.17
Renée
24.79
24.16
35.42
17.70
16.88
22.53
24.48
27.59
26.04
Reude LEVER
26.72
25.73
37.04
22.38
20.40
30,18
27.89
30.67
30.23
Model
L.F-WikiSee
Fl@K
P@k
R@k
ELIAS
17.67
16.83
22.46
ELIAS LEVER
22.06
21.21
27.26
Cas
5.86
5.20
9.78
CascndeMML LEVER
14.81
13.90
20.00
Renće
23.28
22.17
29.30
LEVER
30.12
Ensemble predictions: (T1 0.8, H1 : 0.7, T2 : 0.6, H2 : 0.5, Ts 0.4) Table 9 shows the contri- bution to P@5 for different models across the three deciles. Note that the example is in line with our observations in Fig. 3 where (i) Encoder performs better on tail deciles (blue curve), (ii) OvA models perform better on head deciles (orange), (iii) Ensemble (green) between Encoder and OvA models perform comparably to Encoders on tail deciles and OvA based models on head deciles but incurs significant losses in torso deciles, (iv) Performance of the ensemble model can be worse than the individual models (c.g. ensemble P@5 < OvA P@5 in toy example). If the Ensemble model were to dominate both Encoder and OvA models it should have achieved decile- contributions of (2/5, 2/5, 1/5) which is not the case. On average, the torso labels are ranked relatively lower by both models since neither model specializes in them. Further, when combined using the proposed ensemble these labels get more aggressively down-voted.
C.4 EFFECT OF RE-RANKING ON LEVER AND OTHER TAIL XC APPROACHES
Table TO illustrates the impact of post-hoc reranking using inverse propensity scores, on LEVER and other Tail XC approaches. The application of reranking shows varying degrees of trade-offs between precision and tail metrics across different models. Notably, while LEVER attains superior performance in tail metries for three out of four datasets, there is a trade-off in precision compared to other methods in the LF- WikiHierarchy- 1M dataset.
C.5 ABLATIONS
Effect of Teacher Model: Table demonstrates the impact of employing various encoders as a tail expert. We conduct a comparison with two alternative encoders: (i) MiniLM, a 3-layer transformer
22

---

Published as a conference paper at ICLR 2024
Table 7: Comparison between LEVER and leading OvA-based methods such as TrrTransformer, ELIAS, and CascadeXML, as well as Siamese encoder-based methods like NGAME and ECLARE. Note that for LF- WikiHierarchy-IMM OvA and Siamese- based methods display significant trade-offs between precision and tail metrics. Siamese-bar based methods score much higher in PSP numbers (+16 on average) but lag behind precision (-14 on average) when compared to OvA-based methods.
P@i
P@B
P@5
PSP@
P
PSP@5
C@1
Ce3
C@5
LF-Amazon Titles- 131K
XR- ER-Trrnsformer
38.10
25.57
18.32
28.86 86
34,85
39.59
20.24
40.70
48.87
SOTA XC Methods
ELIAS
37.28
25.18
18.14
28.95
34.45
39.08
23.73
42.36
49.06
CascadeXMI.
36.28
24.88
18.18
26.50
33,21
38.81
21.38
40.58
48.46
ECLARE
41,40
27.58
19.82
34.22
39.69
44.63
27.91
48.38
55.48
NGAME
46,58
30.41
21.49
39.54
44.77
49.59
32.35
53.78
60.73
Renée
46.05
30.81
22.04
38.47
44.87
50.33
31.31
53.50
61.03
Renée +TAUG
21.15
47.85
29:47
51.52
58.68
Renéc + BoW
42.95
29.18
21.03
36.96
42,86
48.09
30.03
51.78
59.17
XC
Methods
Renée + L2Reg
45.19
29.92
21.29
38.47
44.23
49.24
31.66
53.65
60.80
Renée + GL
45.35
30.03
21.33
38.74
44.53
49:49
31,90
54.02
61.15
Renée + Gandalf
45.86
30.53
21.79
40.49
45.83
50.96
33.17
55.36
62.22
Renée + LEVER
46.44
30.83
21.92
39.70
45.44
50.31
32.50
$4.59
61.42
JP-AOL-27UK
XR- formmeerrrr
26.31
8.83
23.07
29.44
SOTA XC Metheds
21.46
10.46
22.85
27.06
CasendeXML
41.20
22.12
14.82
12.58
19.53
23.19
8.73
19:47
23.74
ECLARE
28.53
16.18
11.55
10.11
18.69
24.58
7.41
20.59
28.11
NGAME
39.44
22.29
15.30
16.33
29.63
37.06
14.28
34.70
43.84
Renée
40.97
23.34
15.85
14.76
26.45
32.19
12:40
29.77
36.53
Renée +TAUG
40.40
22.80
15.56
15.72
26.74
32.35
12.46
29.26
35.88
Renée + BoW
41.11
23.91
16.41
15.58
30,28
37.50
12.67
34.32
43.45
Tail XC Methods
Renée 1.Regg
39.83
21.75
14.71
12:21
20.09
24.36
8.67
21.07
26.27
Ronée GLaS
23.25
15.78
14:67
26.
31,75
12.36
29.41
36.06
Ronée Gandalf
40.63
23:01
15.58
26.64
32.17
12.63
29.82
36.31
LEVER
17.07
37.07
45.14
42.54
52.01
1.F.
J.F-Wiippedii..em 500K
XR- frk-foanformmer
81.62
61.38
47.85
33.58
42.97
47.81
19:05
40.05
50.66
SOTA XC Methods
ELIAS
81.94
62.71
48.75
33.58
43.92
48.67
19.62
41.30
51.36
CascadeXML
77.00
58.3
45.10
31.25
39.35
43.29
15.78
33.07
41.46
NGAME
84,32
65.59
51.41
39.88
50.74
57.09
26.22
51.42
64.79
Renée
84.95
66.25
51.68
37.10
50,27
55.68
22.90
50.08
61.59
Renée +TAUG
50.32
52.16
19.88
44.74
56.13
Renée + BoW
84.43
66.09
51.74
36.66
49.79
55.55
22.92
49.64
61.40
Tai XC Methods
Renée + L2Reg
84.57
66.05
51.50
39.55
52.42
57.43
26.52
53.95
65.14
Renée + GS
84.85
66.63
$2.09
37.27
$1..54
57.15
23.43
52.02
63.90
Renée + Gandalf
84.59
66.07
51.63
37.05
49.99 99.944
55.31
23:09
49.87
61.24
Renée + LEVER
85.02
66.37
51.98
42.93
55.00
60.29
29.46
58.53
70.29
L.F Wikii lierarchy
IM
XR-Trursformer
94.26
92.39
15.96
23.04
28.6
2.98
6.23
8.86
SOTA XC Methods
92.45
30.01
4.00
7.78
10.49
CascadeXMI.
91.79
16.03
22.87
28.17
6.17
ECLARE
90.95
89.14
86.90
15.70
22.41
27.65
2.57
5.94
9.30
NGAME
83.16
78.24
73.90
38.43
44.22
47.93
7.83
22.59
29.25
Renée
95.01
93.99
92.24
19.69
27.36
33.20
6.72
11.49
14.65
Renée +TAUG
95.34
94.45
92.27
16:95
24.06
29.69
3.59
7.19
9.94
Ronée BoW
93.92
92.04
90.27
24.25
31.10
36.30
7.84
14.77
18.39
Tail XC Methods
Renée 1.2egg
94.68
93.45
91.56
18.56
2.5.90
31.49
5.61
9.96
12.98
Ronée GLaS
20.07
27.82
33.70
6.89
11.82
15.08
Renée Gandalf
30.05
36.09
13.17
17.52
Renée LEVER
95.19
93.90
92.07
24.76
32.63
38.15
9.32
16.41
20.29
model, and (ii) Astec, which learns a projection matrix from sparse Bag of Words (BoW) features to dense embedding space. The results highlight that the choice of a superior teacher substantially enhances the performance of LEVER.
Effeet of sampling strategy: LEVER makes use of NGAME Module (Dahiya et al. 2023a) trained using mini-batches of labels instead of documents. The modification helps specialize the Siamese
23

---

Published as a conference paper at ICLR 2024
Table 8: Comparison of LEVER with an ensemble of OvA and tail expert encoder. LEVER outperforms the ensemble consistently on all metrics for 3 out of 4 datasets Note that for LF- WikiHierarchy- 1M even though the ensemble improves coverage, the drop in precision is very large (31 I % on average).
P@3
P@5
PSP
PSP@
PSP@5
C@3
C@5
LF-Amazzn
Ensemble
42,98
26.84
18.23
37.24
41.58
44.69
30.38
51.57
57.54
Ronéo LEVER
46.44
30.83
21.92
39.70
45.44
50.31
32.82
55.11
61.94
I.F-AOL--270K
Ensemble 35,20
20.33
13.69
19.43
36.98
44.74
17.19
44,80
54.87
Renée LEVER 41.71
24.77
17.07
20.38
37.07
45.14
17.43
42.54
52.01
LF- LF--WkipeddlaS00
Fasemble
82.55
61.96
46.65
79.82
51.29
55.76
25.72
58.46
72.17
Renée LEVER 85.02
66.42
$2.05
42.50
54.86
60.20
29.46
58.53
70.29
I.F.Wiliifierarehyy..
Ensemble
67.48
62,65
58.39
28.08
31.75
34.01
10.86
20.82
25.89
LEVER
95.19
92.07
24.79
32.74
38.29
9.08
16.12
20.02
Table 9: P@5 performance for different models across deciles.
Model Head Decile P@5
Torso Decile P@s
'Tail Decile P@5
Overall P@5
Encoder
0/5
2/5
I/5
3/5
(V5
Ensemble
2/5
0/5
1/5
3/5
Table 10: Performance comparison of LEVER with other tail XC approaches LEVER outperforms other tail XC methods in tail metrics 3 out of 4 datasets. while LEVER attains superior perfor- mance in tail metrics for three out of four datasets, there a trade-off in precision compared to other methods in the LF- Wi-Wiiiiiierrreyy--ll 1M dataset.
Dataset
Model
P@1
P@3
P@B
Ps@1
Ps0
P:@5
C@1
C
C@5
Renée
46.05
30.81
22.04
38.47
44,87
50.33
31.31
53.50
61.03
Rerank
19,99
50.78
32,90
JF-AmazonTiills-111
L2Rgg ReRank
44.89
29.71
21.14
39.99
44.58
49.29
33.18
54.48
61.19
GLaS ReRank
45.06
29.82
21.17
40.18
44.83
49.49
33.36
54.78
61.48
Gandalf ReRank
44.17
30.29
21.90
40.98
46.09
51.19
33.61
56.27
62.97
LEVER ReRank
45.36
30.67
21.95
41.13
46.00
50.85
33.91
55.79
62.31
Renée
40.97
23.34
15.85
15.06
26. 36
31.97
12.40
29.77
36.53
Rerank
41.53
24.11
16.44
20.21
31.11
37.24
20.27
36.13
43.01
LF-AOL-270K
L2Reg ReRank
40.25
22.43
15.25
15.16
23.59
28.81
13.69
26.38
32.56
GLaS ReRank
41.41
24.01
16.37
19.93
30.71
36.82
20.05
35.71
42.56
Gandalf ReRank
40.87
23.49
15.94
20.70
30.68
36.22
20.61
35.47
41
LEVER ReRank
39.60
24.23
16.92
28.20
41.58
49.33
27.40
48.95
57.62
Renée
84.95
66.25
51.68
37.10
50.27
55.68
22.90
50.08
61.59
Rerank
79.28
63.56
50.80
53.44
16
59.06
41.58
59.52
67.17
JF-WiiipediaaS00k
1.2Ray ReRank
79.30
63.64
50.69
57.67
58.06
60.32
45.03
62.90
70.14
GLaS ReRank
80.20
64.74
51.50
53.22
56.85
60.07
41 .49
60.17
68.55
Gandalf ReRank
58.36
39.07
58.00
66.23
LEVER ReRank
75.34
62.07
50.26
59.15
60.29
62.95
47.24
68.40
75.94
Renée
95.01
93.99
92.24
19.69
27.36
33,20
6.62
11.39
14.56
Rerank
89.95
89.94
88.86
44.15
52.89
58.47
18.15
30.53
35.16
LF JF.iiiiilieranccyy- IM
12Rey ReRank
91.18
90.92
89.51
41.22
49.41
55,00
16.34
27.78
32.73
GLaS + ReRank
93.09
92.42
91.00
46.38
$4.75
60.17
18.78
31.41
36.07
Gandalf ReRank
51.77
57.79
61.37
20.21
34.82
I.VERR ReRank
86.27
84.51
83.69
52.17
58.92
63.59
19.60
34,84
40.58
encoder towards tail labels. Renée + LEVER denotes the model that uses NGAME encoder encderr with mini-batches of documents to augment the training data. Table [2] shows the effect of sampling strategy by comparing Renée LEVER and Renée + LEVER. Renée LEVER outperforms Renée + LEVER by upto 2% in PSP while being comparable in precision.
24

---

Published as a conference paper at ICLR 2024
LF-Wikipedia-5000
Contribution to P@5
4
Ense
Fance Lever
2
2
27
2
2
Avg Training Points per Label
Figure 3: Performance comparison of a Tail Expert Encoder (blue). an OvA Classifier (orange), an Ensemble of Expert Encoder and OvA Classifier (Green) and LEVER- OvA Classifier (red) in the presence of label skew. Labels are partitioned into equi-volume bins based on their frequencies along the X-axis. OvA overfits to tail labels with few training points. Encoder leverages label meta- data to improve on tail but underfits to head. Ensemble mode suffers on torso labels. LEVER combines the strengths of both OvA and Encoder to perform well on all labels. The macro prefix has been omitted for the sake of brevity.
Table 11: Comparison of different encoders: 3-layer MiniLM and Astec, and their effects on LEVER performance. The Astec encoder learns a projection matrix that maps sparse Bag-of Bag-of-Words features to a dense embedding space. A superior teacher leads to improved performance in both Precision and tail metrics, namely PSP and coverage. Note that for all we add the same number of neighbours for each label across all teachers.
P@11
P@3
P@S
PSP
PSP@3
PSP@S
CØ1
C83
C'5
LF-Amazon TilK
16.96
MinI L33 Encoder
23.86
21.65
16.82
20.22
32.44
39.26
17.20
41.80
50.82
DistiBFERT-1A Encoder
20.77
Ronée
46.05
30.81
22.04
38.47
44.87
50.33
31.31
53.50
61.03
Renée + LEVER (Astec)
42.76
28.97
20.97
36.09
42.25
47.82
29.54
$1.29
59.01
Renée LEVER (MinLL--L3)
45.26
30.40
21.82
38.28
44.67
50.09
31.22
53.73
61.11
Renée + LEVER (DisiIBERT..6)
46.44
30.83
21.92
39.70
45.44
50.31
32.82
55.11
61.94
1F-AAmzzotTiilles-1....1
Astec Encoder
36.14
30.25
26.32
28.12
29.00
29.29
18.38
31.56
37.83
MiniL. L.3 Encoder
32.10
26.86
23.43
25.48
26.20
26.48
16.81
29.32
35.46
DistiBFRTT-L. Encnler
42.27
36.16
31.63
35.62
38.11
38.87
22.37
38.93
46.98
Renée
56.10
49.91
45.32
28.56
33.38
36.14
17.61
30.60
37.59
(Astec)
49.30
39.26
Renée + LEVER (MiniLM-L.3)
50.24
44.01
40.08
32.73
35.90
37.73
20.09
35.55
43.23
Renée + LEVER (DislIBERT-LL)
56.01
49.43
44.85
33.55
36.82
38.81
21.03
35.70
42.78
Effeet of varying 7: The hyperparameter tuned using validation set that contains 5% of the training data. The best value of obtained is then used to train LEVER on complete training data. Fig. 7 shows the effect of varying т on LE VEVER'S performance. It can be seen that increasing т leads to better performance on tail labels, while it hurts the head or torso labels.
Effeet of varying The hyperparameter controls the importance between the two loss terms. Table 14 shows the effect of varying X, and Figures show their corresponding decile-wise plots.
D MODEL DETAILS AND HYPERPARAMETERS
D. TAIL EXPERT SIAMESE ENCODER
NGAME's
(Daiiya al. 2023a) hyperparameters include:
25

---

Published as a conference paper at ICLR 2024
Table 12: Siamese teacher models, when trained with a document-wise sampling strategy (Stamese Encoderaoc) perform better in P@ 1 by 3% on average but are inferior in PSP@1 by 10% com- pared to label-wise trained teachers (Siamese Encoderar). However, OvA classifiers, when dis- tilled from Siamese Encoderıи (Renée LEVER()), are comparable in Precision while more ac- curate in PSP@ by 1.7% compared to OvA classifiers distilled from Siamese Encoder (Renée LEVER doc), thus highlighting the importance of tail-specialized Siamese teacher model.
Dataset
Model
P@1
P@3
P@5
PSP@1
PSP@
PSP@5
Siamese Encodoraue
43.13
28.99
20.73
38.72
43.93
48.81
Siamese Encoderial
41.31
28.70
20.77
39.21
44.61
AFAAmoooTiiees 131K
Renée + LEVER
46.05
30.81
22.04
38.47
44.87
50.33
LEVER
30.83
21.92
39.70
Siamese Encoder
81.96
60.72
46.23
48.76
55.87
58.25
L.F. Wikiipdiia
Siamese Engnderaad
67.82
45.66
34.31
60.77
57.26
57.20
Renée + LEVER doc
85.09
65,94
$1.69
40.17
$2.65
58.18
LEVER
66.37
51.98
42.93
Siamese Eacoder
59.08
53,96
49.58
50.82
50,20
50.16
LF Wikiiiierarchyy
Siamese Enguderrot
66.82
50.64
55.42
75.63
73.02
70.52
Renée + L.VER doc
95.02
94.06
92.28
23.64
31,28
36.89
Ronée LEVER
95.19
93.90
92.07
24.79
32.74
38.29
Siamese Encoderaue
45.83
39.94
AF-AmazooTitles-L.... 13M
Siamese Encoderaad
42.27
36.16
31.63
38.62
38.11
38.87
Renée + LEVER
55.02
48.94
44.82
31.86
36.42
38.75
Renée LEVER
56.01
49.43
44.85
33.55
36.82
38.81
Table 13: P and PSP Comparison of NGAME, Renée, and Renée + LEVER on QK-20M Dataset
P@
P@3
P@5
PSP@1
PSP@3
PSP@
NGAME
69.94
52.72
44.81
48.24
55.63
58.71
Rende
72.14
54.87
47.02
50.75
58.90
62.56
71.70
$4.48
46.56
54.74
63.36
cluster-s2:
Mini-batches in NGAME are created from clusters of similar documents (or labels). To build a batch B documents (or labels) we pick B/ B/cluster-sz clusters.
cluster-freq: Denotes the frequency of refreshing the clusters using updated embeddings.
y: Denotes the margin enforeed while training with contrastive loss.
1r: Learning rate for the encoder.
bsz: Denotes the size of mini-batches.
epochs: Denotes the number of epochs for which the NGAME module is trained.
To train the tail-expert NGAME module closely follow the settings from (Dahiya et al.) (023a) NGAME utilizes a 6-layer DistilBERT architecture. Table T6 shows the hyperparameters used on benchmark as well newly contributed datasets.
D.2 ELIAS
ELIAS's (Gupta et al. 2022) hyperparameters include:
C: Denotes the number of clusters in the index graph.
a: Multiplicative hyperparameter that controls the effective number of clusters that can get activated for a given input get activated for a given input.
3: Multiplicative hyperparameter that controls the effective number of labels that can get as- signed to particular cluster.
Controls the row-wise sparsity of the adjacency matrix.
: Controls importance of classification loss L. and shortlist loss Cg in the final loss.
K: Denotes the shortlist size, label classifiers are only evaluated on top-K shortlisted labels.
b: Denotes the beam size.
26

---

Published as a conference paper at ICLR 2024
Effect of varying À when Renée is combined with LEVER. The equal weightage (0.5) gives the best performance. Increasing À weighs the hard labels more, resulting performance that gets closer to the base classifier, i.e., PSP worsens, and Precision remains more or less unaffected. Decreasing also helps only up to a certain point, i.e., A= we believe this is because our teacher is not perfect, and we strike balance between hard and soft labels. Figures 4 S and 6 show the dese plots corresponding to these values. Note that for LF-AmazonTitles IK the effect of varying À is minimal.
Table 14:
LF JF-Amazoo
P@
P@
P@5
PSP@
PSP@3
PSP@5
C@1
C83
C@5
0.33
46. 15
30.76
21.87
39.7
45.33
50.13
32.56
54.48
61.24
0.50
46.44
30.83
21.92
39.70
45.44
50.31
32.82
55.11
61.94
0.66
46.57
30.87
21.93
39.59
45,46
50.36
32.31
54.49
61.48
0.80
46.58
30.88
21.92
39.37
45.39
50.33
32.06
54,38
61.42
LF- JF-Wikkpediaa-50k
PG
PGB
PG6
PSP@
PSP@9
PSP6
CW01
CW03
CW05
0.33
84.66
65.94
51.54
40.51
53.4
58.75
26.88
56.01
67.94
52.05
42.50
60.22
0.66
84.96
66.51
52.18
39.34
53.53
59.46
25.66
55.78
68.33
84.84
66.42
52.14
38.0
53.09
59.16
24.96
55.08
67.79
LFF-001-270K
P@3
P@5
PSP@
PSP@3
PSP@5
C@1
C@3
C65
0.33
41.14
24
16.5
17.24
32.31
40.02
14.14
36.67
45.84
0.50
41.70
24.78
17.07
20.38
37.07
45.13
17.43
42.54
52.01
0.66
41.44
24.41
16.76
16.76
32.69
40.47
14.16
37,38
46.54
16.49
37.64
13.16
Table 15: Renée (OvA) and Siamese trained encoder exhibit different trade-offs on in precision and tail-metrics (PSP, Coverage). LEVER improves the tail performance of Renée (+5% on average in PSP and +3% average in coverage) while retaining comparable precision.
P@3
P@5
PSP
PSP@
PSP@5
C@5
FT-AmazomTitless1135
Siamese Encoder
41.33
28.71
20.77
39.24
44.62
49.52
32,83
55.11
61.95
Renée
46.05
30.81
22.04
38.47
44.87
50.33
53.50
61.03
Renée LEVER
46.44
30.8
21.92
39.70
45.44
50.31
32.50
54.59
61.42
LF-AOL-270K
Siamese Encoder
23.24
15.67
11.68
25.41
36.24
43.43
27.49
47.45
55.58
Renée
40.97
23.34
15.85
14.76
26.45
32.19
12.40
29.77
36.53
LEVER
17.07
45.14
LF-Wikipedla--000
Siamese Enceder
67.81
45.65
34.31
60.76
57.25
57.20
48.76
71.41
78.59
Renée
84.95
66.25
51.68
37.10
50.27
55.68
22.90
50.08
61.59
Rendo LEVER
85.02
66.37
51.98
42.93
$5.00
60.29
29.46
58.53
70.29
L.F-WiklHierarehy-
Siamese Enceder
66.82
60.64
55.42
75.63
73.02
70.54
21.82
42.93
50.62
Renée
95.01
93.99
92.24
19.69
27.36
33.20
6.72
11.49
14.65
LEVER
95.19
93.90
92.07
24.76
27

---

Published as a conference paper at ICLR 2024
LF-Wikipedia-500K
8
0.50
0.66
Contribution to P@5
7
6
5
4
211
29
27
2
23
Avg Training Points per Label
Figure 4: Effect of varying hyperparameter on LEVER's head and tail performance LF- Wikipedia-500K
X LF-AmazonTittes1131
A- 0.50
4.0
Contribution to P@5
À 0.80
3.5
3.0
2.5
2.0
1.5
26
2
2
23
2
2
Avg Training Points per Label
Figure 5: Effect of varying hyperparameter À on LEVER's head and tail performance LF- AmazonTitles- 127K. Here the choice of À has minimal affect on the final performance of LEVER therefore all four plots are closely superimposed.
epochs: Denotes the total number of epochs (i.e. including stage and stage 2 training). LR,, LRw:
Denotes the learning rate used for the transformer encoder and the rest of the model.
baz:
denotes the batch-size of the mini-batches used during training
We closely follow the setting used in (Gupta et al. 2022). ELIAS uses a 6-layer Distil-BERT encoder. Note that the NGAME encoder only used to augment the ground truth with labels similar to particular label, is not used in any other way while training ELIAS. Table T7 shows the hyperparameters used on the benchmark as well as newly contributed datasets.
D.3 CASCADEXML
CascadeXML's Kharbanda et al. 2022) hyperparameters include:
28

---

Published as a conference paper at ICLR 2024
LF-AOL-270K
À 0.33
À 0.50
3.5
X 0.66
Contribution to P@5
À 0.80
3.0
2.5
2.0
1.5
1.0
19
26
213
2 200
27
24
Avg Training Points per Label
Figure 6: Effect of varying hyperparameter À on LEVER'S head and tail performance LF-AOL- 270K
Table 16: Hyperparameters of tail-expert NGAME module. X indicates use of random mini- batches.
Dataset
cluater-a2 cluster-freq
T.R
baz
Spochs
AmazonTiills- 131K
J.-Amazon1131 131K
512
2 10
700
400
1F-AOL-270K
X
X
0.05
2 10
3200
300
LF- B-WikiSccAlso-220
512
0,3
2 10
1024
300
Wikipedia 500K 500K
0.3
2 10
512
WikiHiernrryy
P-AAmazonTitls-I...
2 10
1600
400
Ep: Number of epochs CascadeXML is trained for.
bs Denotes the batch size used for training.
label resolution: Denotes the BERT layers and clustering size used at each resolution.
dropout: Dropout used at each resolution.
shortlist size: Cluster size used at each resolution.
LR2, LRw: Denotes the learning rate used for the transformer encoder and weight vectors.
We closely follow the setting used in Kharbanda et al. 2022). Cascade uses 12-layer BERT encoder. Note that the NGAME encoder is only used to augment the ground truth with labels similar to particular label, it is not used in any other way while training CascadeXML. Table 18 shows the hyperparameters used on benchmark as well as newly contributed datasets.
D.4 RENÉE
Renée's s (Jann al. 2023) hyperparameters include:
epochs: Denotes the total number epochs for which Renée trained.
dropout: Denotes the probability randomly dropping the encoder outputs in order to regularise the network.
warmup: Warmup steps is the number of training iterations over which both the encoder and the classifier learning rates linearly increased from 0 the maximum value.
29

---

Published as a conference paper at ICLR 2024
Table 17: Hyperparameters ELIAS
Dataset
C
Autias
K
b
Epoche
LRg
LRw
bss
LF-Amazon Titks-131K
2048
10
0.05
60
10
10
512
LI-Amazon-13 -3IK
2048
10
150
1000
0.05
2000
20
70
10
5
1024
2F-AOL-270K
4096
10
150
1000
0.05
2000
20
70
3 10
1 10
8192
LF WisiSeeAlso 320K
4096
10
150
1000
0.05
2000
20
40
5 10
5 10
1024
8192
10
0.05
20
40
5
LE Wikiiiierrnchy-
16384
30
1024
LF JF-Amazon Titles- 1.3M
16384
10
150
1000
0.05
2000
20
40
2
10
1 1024
Table 18: Hyperparameters of CascadeXML
Datasct
bes
Label Resclution
Dropout
Shortlist
LRg LRw
IF-AmazonTitles
(5.6):210
(8):218
(10):216 131073
0.2,022, 0.35,
2 210, 210
le
LF-Amazn 131K
{5.6):29
18):212
[10):216
2 2..27.0
LF-AOLL-20K
96
(5.6):2*9
(8):243
(10):2'6
12:272825
0.2, 0.3.0..55330000
2 22.2"000 240
le
le
I. AilieeeeeAllooS2D
12
64
(5.6):220
(8):248
-(10)2116 -12: 312330
0.2, 0.25, 0,35, 0.5
2 2112211 212
le
le
LF-Wikipedin-500K
12
256
(5.6):210
(8):248
-(10)21e - 12: 501070
0.2, 0.25, 0.35, 0,5
2 2
le
le
LB-WikHHicracch--1M
96
(5.6):249
(8):245
(10):216
12: 976214
0.25 0.35, 0.5
210.210.20
1e
1e
Jm-amzonTTitls-
(78).215
(10):2'6
220,221
1e
16-3
LR LRw: Denotes the learning rate used for the transformer encoder and the classifier layer.
bsz: Denotes the batch size of the mini-batches used during training.
clf-wd: Weight decay for fully connected layer parameters.
Table 19: Hyperparameters of Renée
Dataset
#pocha
Bropout
Na Narmpp
I.Rg
L,Rw
haz
cl f-ud
AF--moon
100
0.85
5000
1 x 10
x 10
512
1 10
LE Amazonn 131K
100
0.85
5000
x 10
x 10
512
x 10
LF-AO1-270K
100
0.60
20000
1 x 10
1
10
1024
10
100
0.75
5000
2 10
2
2048
x
Wikipedia 500K
10
10
10
L. WikiHieenrchy- IM
100
0.70
20000
x 10
2
10
1024
10
Fr-AAmzonTiilssI..11
100
0.70
15000
x 10
1 x
10
1024
x 10
We closely follow the setting used in (Jain et al., 2023). Renée uses a 6-layer Distil-BERR encoder.
Table shows the hyperparameters used on benchmark as well as newly contributed datasets.
D.5 RERANK + TAUG
ReRank TAUG (Wei al. 2021 hyperparameters include:
Expli:: Denotes the proportion of labels that will be considered as head labels. The original dataset D containing L labels is split into datasets Dj, and D,- Dj, contains headmost eptis
labels and their associated training points, while D, contains the remaining L Esplit L labels along with their associated training points.
n-aug: Denotes the number of additional data points that will be generated for each data point in De.
Pdrop: Denotes the probability of dropping a token from the data point.
Pswop: Denotes the probability of swapping two randomly chosen tokens.
rerank strtegy:: Denotes the multiplicative factor used to re-rank scores. We use the label inverse propensity factor perform re-ranking.
D.6 GANDALF
Gandalf's Kharbanda et al. 2024) hyperparameters include:
30

---

Published as a conference paper at ICLR 2024
Table 20: Hyperparameters of Re-rank TAUG
Datasct
aug
Pine
JF-AmazonTitie- 111ks-131K
0.90
8
0.30
0.30
270K
0.65
0.20
0.20
LF Wikipedia 500K
0.90
0.10
0.10
LF- Hi-Wiiiiiirrrcchy-MM IM
0.90
0.20
0.20
shreshold: Denotes the threshold used to filter out labels obtained from the normalized
label correlation graph during augmentation.
We closely follow the settings used in (Kharbanda et al. 2024) and use threshold of 0.1 for all datasets.
D.7 LEVER
D.7.1 HYPERPARAMETERS
LEVER uses the parameter т to control the number of entities (data points or labels) that are added for each label. The hyper-parametts from Theorem 3 was set 0, as this value worked consistently well across datasets. Table 22 displays the values of т, Td, and 7i for both benchmark and newly contributed datasets. We observed that setting т = 0 yields satisfactory results for most cases, with the exception of LFF-Wikipedia-500K for which we set = 12. A value of X = 0.5 was employed for experiments. Table 2T illustrates the effect of varying v, and Table demonstrates the effect of varying A.
Table 21: Performance variation in P. PSP and Coverage when is varied in LEVER for LF- Wikipedia-500K and LF- WikiHierrrchy-- 1M. Figure 7 shows the decile-wise plots corresponding to these values. As more neighbours added, the tail metries (PSP and Coverage) improve while the Precision remains constant or slightly drops.
I. LF-Wikipedia-S00KK
PSP@8
PSP@3
PSP@5
C99
18.1
84.96
66.26
51.68
37.10
50.27
55.68
22.90
50.08
61.59
20
39.5
85.14
66.90
52.35
38.39
52.33
58.04
24.72
53.45
65.43
45
64.5
85.02
66.43
52.05
42.51
$4,86
60.20
29.08
58.28
70.09
LF LF-Wikifierarchh--h
PPL
P@1
P83
P85
PSP
PSP@3
PSP@5
CW1
C'03
C95
43.3
95.01
93.99
92.24
19.69
27.36
33.20
6.62
11.39
14.56
92.07
24.79
32.74
20.02
47.7
94.97
93.18
90.98
26.37
34,99
40.51
10.08
18:34
22.74
LF (F-Wikipedia-50k
LF-WIKiHlerarchy-1MM
Contribution P@S
Contribution to P@SS
10
T=20
211
2a
27
2
213
2
2
Avg Training per Label
Avg Training Points per Label
Figure 7: Effect of varying hyperparameter T on LEVER's head and tail performance on LF- Wikipedia- and LF--WikiHiierrchhy-MM.
31

---

Published as a conference paper at ICLR 2024
LF-WikiHierarchy-1M
ELIAS
ELIAS + LEVER
Renée
Contribution to P@5
Renée LEVER
15
10
5
2
20
27
24
Avg Training Points per Label
Figure 8: In ELIAS there is bigger drop in head performance as compared to Renée when LEVER applied. This therefore translates to a much larger increase in ELIAS+LEV ER''S 's tail performance for this dataset, note that for ELIAS the performance improvement starts from the torso labels itself.
Table 22: LEVER's Hyperparameter T on benchmark datasets
Dataset
I.F-Amazon-133 131K
20
20
LF-AOL-270K
100
LF- ALoo
Wikipedin 50OK 500K
I.F. Wikilieranchy IM
Ii-Ammzon/itles--l
15
15
D.7.2 TRAINING TIME
Table 23
shows the training time for different models and datasets when combined with LEVER. Note that in ELIAS and CascadeXML, where the train times increase by a greater margin, the gains provided by LEVER are also higher (avg. +6. 1% increase in PSP and +2
Table 23: Training time (in hours) for different models single NVIDIA V100 GPU. The average training time increases by 3. and in the worst case, by 8.9x. For LEVER counterparts, this includes the time for training the teacher model, generating soft labels, and using/training the OvA classifier.
Dataset
Renée
Renéet LEVER
ELIAS
ELIAS+ LEVER
CascudeMML. CascaleXMLL+ LEVER
LF-amooniiltess
20.11
LF-Amazon-1311
42.77
46.51
19.44
65.62
4,60
41.31
LI-A0L-270K
136.22
138.20
60.67
171.20
42.12
152.00
LF. WikiSeeAlso 320K
86.42
95.19
25.33
111.46
12.40
88.52
I.F. JF-Wiiipedin-SOp
154.93
184.05
138.67
226.72
29.58
89.72
JF-WikiHierarchy--ll IM
31.44
40.15
61.17
35.91
LF- (itazzo/ittess..333
154.39
186.45
40.00
158.23
70.00
202.93
Average Time Inc.
1.14x
3.29x
4,86x
Table 24 25
show the runtime break down when LEVER. Note that the training time of the Siamese Teacher is less than what is reported in (Dahiya et al., (023a) as that includes the time taken to train both the NGAME Encoder and NGAME Classifier.
26
32

---

Published as a conference paper at ICLR 2024
Table 24: Renée LEVER training time (in hrs) on a single NVIDIA V100 GPU. Training LEVER involves three steps (a): Time to train the Siamese teacher, (b): Time to construct soft labels and (c): Time to train the Renée.
Dataset
Siamese Teacher
Solt Labels
Renée
Total
AT-AmmzonTilee Titles 13IK
11.82
8.22
20.11
L.F-Amazon- 13IK
34.44
0.07
12.00
46.51
LF-AOL-270K
108.00
138.20
LE Wi--eeSeeeAlo--2e
69.86
0,25
25.07
95.19
Wikipedia 500K
I.F. WikiHieranchyy IM
19.33
1.04
19.78
40.15
LF-AmazonTitles
93.50
92.22
186.45
Table 25: ELIAS LEVER training time (in hrs) single NVIDIA V100 GPU. Training LEVER involves three steps (a): Time to train Siamese teacher, (b): Time to construct soft labels and (c): Time to train the ELIAS.
Dataset
Siamess Teacher
Soft Labels
ELIAS
Total
LF-AmazonTitls-1311
11.82
0.07
6.93
18.82
131K
65.62
LF-A0L-270K
108.00
0.20
63.00
171.20
320K
LE JFF-Wiipedia-5000
50.26
0.45
176.00
226.72
LE Wikllienacchy
IM
19.33
1.04
40.80
61.17
LF- Antzaoniitees-l11
1.3M
93.50
0.73
64.00
158.23
Table 26: CascadeXML + LEVER training time (in hrs) on a single NVIDIA V100 GPU. Training LEVER involves three steps (a): Time train the Siamese teacher, (b): Time to construct soft labels and (c): Time to train the MM.
Dataset
Siamese Teacher
Soft Labels
CascadeXML.
Total
Ll=Amazoniites-111
17.14
LF Am-.oonzon 131K
34.44
0.07
6.80
41.31
LF-AOL-270K
108.00
0.20
43.80
152.00
LF WikiSeeAlo 320K
69.86
0.25
18.40
88.52
I.F. J.F-Wiiipeiin-O00 S00K
50.26
0.45
39.00
89.72
LF-WikiHienncchy..D
19.33
1.04
15.54
35.91
J-AmmzoaTHtlee-L.3
93.50
0.73
108.70
202.93
33


Reviews:
[{"review_id": "bPwOn1D8KL", "reviewer": "Reviewer_Uo9h", "summary": "The paper presents a methodology for improving performance on tail-labels. The main observation behind the proposed approach is that there is a significant variance in the label distribution due to the finite and even scarce number of data points for tail-labels in the extreme classification. The paper builds a connection with an existing work (Menon et al. 2021), on which it heavily relies, for most of the motivation, to propose a variance reduction strategy (Theorem 1 in the paper & discussion thereafter) and hence improvement in the generalization error. \n\nThe focus of the approach (generalization analysis) is on the last/classification layer of the network. The above generalization analysis leads to an augmented objective which, in addition to standard hard labels, also consists of a loss term consisting of soft labels from a teacher model, for which recent frameworks based on Siamese training are exploited. The proposed approach is tested on a range of datasets from the extreme classification repository, and it is shown that the proposed approach leads to significant improvements in the prediction performance in terms of P@k and PSP@k metrics. The augmentation strategy is also compared to existing methods for data augmentation in extreme classification to further demonstrate its general applicability.", "questions": "As mentioned above in the weaknesses section", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The paper attempts to address a key shortcoming of existing methods in extreme classification i.e. performance of sota methods on tail-labels, which despite its importance, doesn’t get much focus of research.\n\n2. The experimental results of the paper are quite impressive, and significant gains on a range of datasets, and methods are shown, thereby demonstrating its general applicability. Overall, the experimental setup is quite detailed.\n\n3. The paper also contributes two additional datasets (LF-AOL-270K, and LF-WikiHierarchy-1M), which may be helpful for the community.", "weaknesses": "1. The main shortcoming of the paper is the lack of novelty in the main idea and the approach :\n\n1a) In terms of the content i.e. the main theoretical results such as Theorem 1, its proof, the idea to reduce variance for improving generalization, using soft-labels in teacher-student setup has already been explored in the existing work of Menon et al 2021. It seems that the paper attempts to build a loose bridge between this earlier theoretical work to build a weighting scheme and soft-labels (equation 9). \n\n1b) The paper claims that the variance issue for tail-labels has not been explored so far. However, this seems not quite true, as a similar concern has been raised in the earlier work Babbar and Scholkopf 2019, where there was a similar argument that due to the lack of data in tail-labels, for a given class label, there is high variance between the input features between two samples. \n\n2) There also seems to be a lack of consistency in terms of using the symbols and notation in many places. For instance, X is used in Section 3.1 to denote a (random?) instance in the input space $\\mathcal{X}$. However, the same is used to limit the norm of x in the same section. For equation (1), should the LHS also not be conditioned on x i.e. V[y|x]. It is not clear why there are two different symbols for the sub-script (small-case) x. Why the equation (6) is defined in terms of a trained classifier w*, while the standard generalization results are stated in terms of all possible classifiers in the hypothesis class. The theorem statement needs to be on a better formal footing as done in the original paper Menon et al. 2021, the current version is incomplete and unclear.\n\n3. In terms of properly citing related work, the paper falls somewhat short. For instance, the paper introduces the need to use calibrated losses without a proper justification to use calibrated losses or a reference thereof. It does not seem to be in the sense of the word used in the last sentence of page 1. Another concern is the lack of setting the right context for the cited papers.  As mentioned above, the high variance problem for tail-labels has also been explored. As another instance, the work of Schultheis et al. 2022 is cited but the correct context is in the related work section when discussing Missing and Tail-labels in Section 2.2, and not only later down in discussing coverage as a metric. Even though the paper has been cited, it is not in the right context. In terms of the main classifier, the focus of the paper is on one-vs-rest approaches, while DiSMEC which initiated this approach in extreme settings isn’t referred.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "O6ZTIrtlbZ", "reviewer": "Reviewer_Cq1J", "summary": "The paper introduces a new extreme multi-label classification algorithm called LEVER, that aims to improve the existing methods by proposing training with a loss function that combines loss calculated on observed hard labels and soft labels coming from the teacher model, which is assumed to have superior performance on the tail labels. The teacher model used by the authors in the empirical part of the paper is a Siamese-style neural network that leverages label features and is trained with a logistic-loss-based objective, which was found to yield calibrated estimates of the marginal probabilities of labels. The authors demonstrate the attractiveness of the proposed approach in the exhaustive empirical comparison when they used 4 popular XMLC benchmarks, introduced 2 new benchmarks, and combined the proposed approach with 3 SOTA methods. The proposed approach, in many cases, significantly improves the performance on standard precision@k, propensity-scored precision@k, and coverage@k.", "questions": "1. As I mentioned in the weakness section, I would like to see how different values of $\\lambda$ impact the final performance and what the trade-off curve between head-labels and tail-labels performance looks like. Could the authors comment on that?\n2. On the LF-AOL-270K dataset, LEVER combined with ELIAS yields extremely high improvement on standard precision@k, especially at @5. These seem almost unrealistic when compared with scores of other methods. Are these numbers for sure correct? If yes, do the authors have any explanation for this result?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-written and easy to follow.\n2. The proposed method is simple to apply to a wide range of XML classifiers.\n3. The exhausting empirical comparison proved the attractiveness of the proposed method.\n4. In addition to the main results, the authors provide a wide array of additional experiments in the appendix.\n5. Two new datasets additionally strengthen the contribution of this work.", "weaknesses": "1. The proposed method is just a combination of loss with hard and soft labels, which is a simple idea.\n2. I find the theoretical results rather simplistic, expected, and being there to serve as just justification for the applied method rather than its original motivation.\n3. This part of the paper especially confirms that for me:\n\n   > Notably, if we have precise estimates of marginal relevance, denoted by $p_x = E[y|x]$, we can replace $y$ with $p_x$, effectively reducing the variance term to 0 and thereby improving classifier generalization. This principle forms the foundation for the LEVER framework, which employs an additional teacher network to provide accurate estimates of $p_x$.\n\n   If we have a model that provides good estimates $p_x$ then we don't need to train another one, the XMLC task is already solved! The strength of the method seems to lay in the properly selected trade-off between loss calculated on hard observed labels and soft labels coming from the teacher network that seems to be much better on tail-labels thanks to leveraging labels-side information. \n\n4. From the appendix, I understand that $\\lambda = 0.05$ (the variable that weights the hard and soft part of the loss) was used for all the experiments. Since the choice of $\\lambda$ seems to be crucial for the method. I find the lack of further comments on it and experiments demonstrating its impact the biggest weakness of this paper.\n\nMy score is 6, but I believe the paper is even a bit above that.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jns9oikZjC", "reviewer": "Reviewer_Kx6v", "summary": "This paper aims to improve the tail labels performance of the extreme multi-label classification (XMC) problems. The paper propose LEVER, a knowledge distillation framework, that learns student OVA classifiers with binary relevance true labels and soft-labels generated by teacher bi-encoders. The author claims that bi-encoders have better performance on tail labels, and hence using their soft labels as additional signals to learn the student OVA classifiers can help reduce the variance for tail labels. Empirically, LEVER demonstrated consistent improvement upon competitive OVA classifiers on a wide-range of XMC datasets.", "questions": "## 1. Calibrated Scores\nIts well known that pairwise ranking losses (e.g., Equation (10) of this submission) is shift-invariant hence may not produce calibrated scores, as pointed out by [1]. Why not considering point-wise loss functions or hybrid objectives [1] (listwise Softmax + pointwise BCE) to produce more calibrated scores?\n\n## 2. Experiment Settings and Results\n(1) What's the model size for each method in Table 1 and Table 2? The model size should include every component needed to do inference. For example, does ELIAS and ELIAS+LEVER have the same model size, as LEVER is just in-place modifying the OVA classifiers of LEVER? If not, I am concern about the performance gain of LEVER is due to additional model capacity. \n\n(2) In Table 1, ELIAS+LEVER achieved >20% absolute gain on LF-AOL-270K in P@3 and P@5. If not a typo, any insight why such significant gain? Similar questions to ELIAS+LEVER on LF-Wiki-1M, PSP metrics.\n\n(3) On page 9, the author claims the training time of LEVER only increased by at most 2x. Does that include (a) training time of teacher bi-encoders (b) prediction time of teacher bi-encoder to generate soft-labels (c) training time of student OVA classifiers which are trained by not only sparse ground truth labels but also dense soft-labels? If so, what's the detailed breakdown in terms of those three components?\n\n(4) Suppose the ground truth label matrix is a sparse $N \\times L$ matrix. How dense are the soft-labels generated by LEVER? Does the performance of LEVER vary when using different top-$k$ soft-labels per input?\n\n## 3. Macro-averaged Evaluation Metrics\nTo properly measure the performance of long-tailed labels, text classification community often consider Macro-average Precision/Recall/F1 metrics [2,3,4]. The author should also report Macro-averaged metrics to further validate the major claim of LEVER, which is the the performance gain on tailed-labels.\n\n\n## Reference\n\n[1] Yan et al. Scale Calibration of Deep Ranking Models. KDD 2022.\n\n[2] Zhang et al. Long-tailed Extreme Multi-label Text Classification by the Retrieval of Generated Pseudo Label Descriptions. EACL 2023.\n\n[3] Yang et al. A re-examination of text categorization methods. SIGR 1999.\n\n[4] Lewis et al. RCV1: A New Benchmark Collection for Text Categorization Research. JMLR 2004.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The paper writing is easy to follow\n2. Empirical results are strong", "weaknesses": "1. Its questionable that using \"pairwise\" logistic loss (Eq.10) lead to calibrated scores\n2. Some experiment settings are not clear from the main text. See detailed questions below.\n3. Missing proper evaluation metrics (i.e., Macro-average Precision/Recall/F1) for tail-labels performance", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kpgvNw0JN9", "reviewer": "Reviewer_UBHy", "summary": "This paper investigates a novel factor, label variance, in the context of tail performance in extreme classifiers. To address this issue, the authors propose LEVER, a knowledge distillation framework that leverages a robust teacher model to reduce label variance. Experimental results demonstrate that LEVER significantly improves tail performance, achieving approximately 5% and 6% increases in PSP and Coverage metrics, respectively, when integrated with state-of-the-art extreme classifiers.", "questions": "1. Can the teacher model be adapted to the current task? How does its performance compare? What are the differences between directly using a better model for distillation and utilizing the results of other models as auxiliary information in this paper?\n2. Refer to the weaknesses mentioned.", "limitations": "", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 1, "contribution": 2, "strengths": "1. The paper give a proof of the correlation between the generalization performance of a classifier and the variance in labels.\n2. Use a Siamese- style model as a teacher to help reduce the label variance effects.\n3. The extensive experiment results are promising.\n4. The paper contribute two new datasets to the field.", "weaknesses": "1. The readability of the paper is not strong, and the formatting is uncomfortable. For example, the abstract should generally be a single paragraph, there are too many blank spaces before and after Section 2 Related Work, and the spacing of equations is large. With ample space in the appendix, the full text does not fill up the nine pages. Improvements are needed in basic writing formatting and readability.\n2. The paper has limited novelty. From a personal perspective, the innovation lies in using a teacher model to predict probabilities. The rest of the paper mainly demonstrates the significant impact of tail classes. From the perspective of innovative design, it is not convincing. It is also unclear why the teacher model can provide accurate estimates of $p_x$.\n3. There is limited description regarding fair comparisons. After introducing the teacher model, the training process and time will be affected. The main part should devote more space to introducing the teacher model, analyzing its performance, and comparing the training time with other models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)