MOGIC: Metadata-infused Oracle Guidance for Improved Extreme Classification
Authors: Suchith Chidananda Prabhu
Bhavyajeet Singh
Anshul Mittal
Siddarth Asokan
Shikhar Mohan
Deepak Saini
Yashoteja Prabhu
Lakshya Kumar
Jian Jiao
Amit S
Niket Tandon
Manish Gupta
Sumeet Agarwal
Manik Varma
Venue: ICML 2025 poster (2025)
Keywords: Extreme Classification
XC
XML
XMC
Transformers
Metadata
Auxiliary information
Information Retrieval
Regularization
Supervised Learning
Memory-based Models
Neural Networks
Primary area: general_machine_learning->representation_learning
TL;DR: Oracle guided enhancement of memory representations improves task performance

Abstract: Retrieval-augmented classification and generation models benefit from *early-stage fusion* of high-quality text-based metadata, often called memory, but face high latency and noise sensitivity. In extreme classification (XC), where low latency is crucial, existing methods use *late-stage fusion* for efficiency and robustness. To enhance accuracy while maintaining low latency, we propose MOGIC, a novel approach to metadata-infused oracle guidance for XC. We train an early-fusion oracle classifier with access to both query-side and label-side ground-truth metadata in textual form and subsequently use it to guide existing memory-based XC disciple models via regularization. The MOGIC algorithm improves precision@1 and propensity-scored precision@1 of XC disciple models by 1-2% on six standard datasets, at no additional inference-time cost. We show that MOGIC can be used in a plug-and-play manner to enhance memory-free XC models such as NGAME or DEXA. Lastly, we demonstrate the robustness of the MOGIC algorithm to missing and noisy metadata. The code is publicly available at [https://github.com/suchith720/mogic](https://github.com/suchith720/mogic).

MOGIC: Metadata-infused Oracle Guidance for Improved Extreme Classification
Suchith Chidananda Prabhu" "1 Bhavyajeet Singh" 2
Anshul Mittal Siddarth Asokan
Shikhar Mohan Deepak Saini Yashoteja Prabhu
Lakshya Kumar
Jian Jiao
Amit S Niket Tandon? Manish Gupta 3 Sumeet Agarwal
Manik Varma
Abstract
augmented generation (RAG) Leewis et al
2020 Gao et al.
2024 Fan et al. 2024). context enrichment also benefits classification tasks (Guu et al. 2020b Guo et al. 2023 Mo- han al., 2024 Long et al. 2022). In this paper, we focus using memory to improve classification, and in particular scenarios wherein the label space is extremely large (typi- cally of the order of millions), i.e., eXtreme Classification (XC) (Mohan et al. 2024).
classification and generation models benefit from early-stage fusion of high- quality text-based metadata, often called memory, but face high latency and noise sensitivity. In extreme classification (XC), where low latency is crucial, existing methods use late-stage fusion for efficiency and robustness. To enhance accu- racy while maintaining low latency, we propose MOGIC, a novel approach to metadata-infinm oracle guidance for XC. We train an early-fusion oracle classifier with access to both query-side and label-side ground-truth metadata in textual form and subsequently use it guide existing memory-based XC disciple models via regular- ization. The MOGIC algorithm improves preci- sion@ and propensity-scorcs precision @ I of XC disciple models by 1-2% on six standard datasets, at no additional inference-time cost. We show that MOGIC can be used in plug-and-play manner enhance memory-free XC models such NGAME or DEXA. Lastly, we demonstrate the robustness of the MOGIC algorithm to missing and noisy metadata. The code is publicly available at /thttpp:///gtthue www.:ttwww....ouooooeee.eoeeooot th720/mogic
Real-world XC tasks often involve short-text queries, such as matching queries to advertiser-biddee keywords (for sponsored search ads (Dahiya et al.)
2021 Jain et al.) 2016
Prabhu al. 2018b) queries product titles (for prod-
uct recommendations (Dahiya et al.
202 Medini et al.
2019 Mittal et al. 2021 or queries to webpage titles (for tagging (Babbar & Schölkopf 2017 Chang et al. 2020 You et al. 2019)). These XC tasks, characterized by an extremely large label space, often suffer from data sparsity, having very few training examples per label on average. In such XC tasks, auxiliary metadata can offer relevant diverse information, which can be leveraged to obtain better un- derstanding of the input queries, thereby making accurate predictions. For example, on the query-to-ad-keywont pre- diction task, the query-side metadata can be obtained by mining organic search webpage titles clicked in response to the query on the search engine, while on the Wikipedia categories prediction task, other Wikipedia article titles con- nected to the original page via hyperlinks could serve the metadata. For a detailed overview of the preliminaries, and definitions of key terms, please refer to Appendix A
1. Introduction
Context enrichment is the process of incorporating meta- data from external sources (referred to memory for model, comprising memory items) to improve the overall performance on a task. While such methods have gained popularity in generative modeling in the form of retrieval-
Memory-based augmentation in XC remains a challeng- ing problem due to the need for accurate and fast mem- ory access. The low latency requirements are imposed by real-world applications. Ideal/golden memory item link- ages are typically available during training, referred to as ground-trute truth metadata. However, these golden linkages are seldom available during real-world deployment, particularly in real-world scenarios, where 80-85 of the traffic cold-start setting, as reported (Mittal et al.) 2025). Our work demonstrates that methods that rely entirely on high- quality ground-truth linkages of metadata are susceptible
'Equal contribution Yarrii School of Artificial Intelligence, IIT Delhi, India "Microsoft Research, India Mirccsooft, India "Mircosoft, USA. Correspondence to: Suchith Chidananda Prabhu <suchith720@ gmail.commm
the
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

MOGIC: Metadata-infused Oracle Guidance for Improved Extreme Classification
Table 1: A comparion of predictions from MOGIC (OAK), OAK, MOGIC (NGAME), and NGAME models, and the ground -ruth and oracle predictions, on the Wikipedia See Also prediction task. Legend: Black indicates ground truth, Red indicates incorrect predictions, Green indicates correct predictions, and Blue indicates missing labels (as per a human judge).
QUERY
Grass court
Tangbe
GROUND TRUTH LABELS
Clay court. Carpet court, Hardcourt
Mustang District, Kali Gandaki Gorge, Kali Gandaki River, Upper Mustang, Gandaki River
GROUND TRUTH QUERY METADATA
Tennis temninology, Sports rules and regulations, Tennis count surfaces
Populated places Mustang District
PREDICTED QUERY METADATA
Courts by typo, Landforms, Grasslands
Populated places Cameroon, Communes Township level divisions of Hehei
Cameroon,
ORACLE PREDICTIONS
Clay court, Carpet court, Hardoourt, Plexicushion, DecaTurf
Mustang District, Kali Gandaki Gorge, Kali Gandaki River, Upper Mustang, Mustang Caves
OAK PREDICTIONS
Femii Ghostriders, Garland, Texas, List of Nevada state pris- ons, Ronald Reagan Boyhood Home, West End (Richmond, Virginia)
Desalpar, Second Franco- List of currently crupting volcanoes
Franco-Dahomean Dahomean War, Vladivostok, Kitenge,
MOGIC (OAK) PREDICTIONS
Clay court, Carpet court, Hardcourt, Video arcade, U.S. Men's
Mustang District, Kali Gandaki Gorge, Kali Gandaki River, Upper Mustang, Gandaki River
Clay Court Championships
NGAME PREDICTIONS
National Register of Historie Places listings in Camberland County, North Carolina, Shangri La (Doris Dukej, Vauxhall, National Register of Historic Places listings in Perry County.
Five kings of Wa, Piteran, Wemale, Oyo Empire, List of light- houses Togo
Alabama
MOGIC (NGAME) PREDICTIONS
Clay court, Carpet court, Hardcourt, Stadium, Riding ball
Mustang District, Upper Mustang, Mustang Caves, List munic- ipalities in Andhra Pradesh, Muktinath
noise and fail to make accurate predictions, even when these linkages are slightly perturbed (see [5 To address the cold-start challenge, prior work focused on predicting meta- data during inference as an auxiliary task. This approach typically involves training a separate model on ground-truth metadata to solve this auxiliary task. However, several chal- lenges persist, and can be characterized into: (a) Sensitivity to retrieved metadata: Low quality retrievals from memory lead to noisy augmentations to the query, degrading task performance. For example. Cuconasu et al. (2024); Yoran et al.] (2024) and Yu et al.) (2024) showed that, for text-based carly-fusion models, robustness to retrieved memory item quality is critical for performance. (b) Relationship between latency and the form of metadata infusion: Textual meta- data offers superior interpretability over embedding- based metadata, but can lead to high inference-time fusion latency as it causes increase in the sequence length of the input to the transformer.
pared to early-fusion models (achieving precision @ I of 38.92 compared to 47.63 for early-fusion). This highlights key trade-off: while early-fusion models may struggle with noisy metadata, they demonstrate superior performance when given clean, high-quality metadata. In this work, we aim to leverage this insight, using early-fusion models to improve the performance of memory- XC models. Fur- thermore, this XC setting uniquely allows for incorporating label-side metadata within the memory framework, feature that has not been fully utilized in existing approaches.
In this paper, we design a novel memory-based approach for XC that utilizes both query and label-side metadata and maintains low inference latency, by employing a combina- tion of early-fusion of text-based metadata and late-fusion of memory items. Our approach involves two phases of training: (a) Oracle training, and (b) Oracle-guided disciple training. We call this two-stage metadata-infused oracle guidance framework for improved extreme classification as MOGIC. In the first phase, we train an early-fusion Oracle classifier which is provided access to both query-side and label-side ground-truth metadata in textual form. In the second phase, the oracle used to guide the training of any existing semory-based XC disciple model (such as OAK or NGAME (Dahiya et al. 2023a)), by means of novel regularization loss. Table | shows two examples of queries with ground truth labels; predicted and ground truth memory items; the oracle predictions, and predictions from OAK, NGAME, and their MOGIC counterparts on the Wikipedia related articles prediction task. We observe that for the query "Grass Court", "Courts by type" is a relevant memory
State-of-the-art retrieval augmentation approaches struggle to balance accuracy and efficiency in XC. Recent works such as OAK (Mohan et al. (2024) and PINA (Chien et al.] 2023) have shown that using metadata can improve online classification performance. However, each approach has its limitations. For example, OAK's late-stage embedding- based fusion method improves generalization and reduces inference latency when handling noisy predicted metadata (achieving precision @1 of 33.71 compared to 28.49 for dataset). How-
early-fusion on A ever, even when provided with the ground-truth metadata, the OAK model produces suboptimal representations com-
2

---

MOGIC: Metadata-infused Oracle Guidance for Improved Extreme Classification
item. However, some predicted metadata (e.g., "Landforms" and "Grasslands"" mislead OAK, leading to incorrect pre- dictions about geographical places, NGAME also exhibits poor performance in this scenario. In contrast, MOGIC, trained with our novel regularization, effectively retains the original intent of the query. This is because MOGIC lever- ages the best of both worlds- - maintains low inference latency and effectively handles noisy predicted metadata through late-fusion, while simultaneously learning to mimic the superior embeddings of the early-fusion oracle through our novel regularization loss. This combined approach mit- igates drawbacks of both late- and early-fusion methods, resulting in improved performance. Appendix [H] contains a detailed case study on how metadata and oracle-guided training help MOGIC rectify errors from OAK.
fier, such as NGAME (Dahiya et al. 2023a encoder, is used to retrieve the metadata, and single transformer atten- tion layer is used to fuse the representations of the query and the retrieved metadata. The proposed MOGIC algorithm leverages a text-based early-fusion model to improve the representations of the memory items in OAK. Along another direction, models such as DEXA (Dahiya et al., 2023b) ag- gregate information from the neighborhood of the encoder representations to form the context. Consequently, MOGIC can also be applied atop XC models such as NGAME and DEXA to improve performance (cf. Table 6).
Retrieval-augmennte Generation (RAG): The RAG paradigm has become the defacto approach for incorporat- ing metadata for context enrichment in generative models, with the application typically being that of question answer- ing. Prior to RAG, models such REALM (Guu et all, 2020a) leveraged external knowledge sources to improve the accuracy of the transformer encoders using a retriever that selected relevant documents or passages from the mem- ory, while an encoder fused the input text and memory items to compute an enriched embedding. RAG-based ap- proaches (Lewis al., 2020 Akyurek et al. 2023 Zhang et al. 2023 Radhakrishhan et al. 2024 Muennighoff et al. 2024 combine pre-trained parametric and non- non-parametric memory for language generation. In RAG settings, the mem- ory. typically text-based, is infused with the query at the imput Yang et al. 2018:
In summary, contributions are as follows:
We propose MOGIC, Metadata-infuuse Oracle Guidance framework for Improved Extreme Classification, that maintains real-world infer- ence latency, while achieving 1-2% improvement over state-of-the-ar XC models.
We develop novel regularization loss that leverages the oracle model to guide the training of memory- based disciple. This approach enables the disciple to demonstrate performance gains while maintaining low inference latency, resulting in a 50% reduction in com- putational costs compared to the oracle.
Karpukhin ct al. 2020; Qu et al. 2021 Lan et al. 2023 Lála et al. 2023 Yan et al. 2024
By means of extensive experimentation on six popular XC datasets, we show that (a) MOGIC improves accu- racy significantly in terms of standard metrics such as precision, NDCG and propensity-scored precision atop both memory-based models such as OAK as well as memory-free -free models such as DEXA and NGAME; (b) MOGIC is robust to missing and noisy metadata com- pared to the oracle, and (c) MOGIC (OAK) achieves state-of-the-arr metrics across all datasets.
Other approaches incorporate task- specific memory, such as tabular data (Zha et al. 2023 Luo et al. 2023) or knowl- edge graphs (Gaur et al. 2022 He et al. 2024). We observe that retrieval-augmented models benefit from early-fusion, and high-quality metadata, but suffer from high inference la- tency and poor robustness to noise. MOGIC makes a novel contribution by introducing the textual early-fusion of meta- data into XC models while respecting latency constraints.
Guided Representation Learning: Transferring capabil- ities via context-following from large language models (L.LMs) to smaller ones has been widely studied (Kim & Rush 2016 Gupta & Agrawal 2022 Xu et al.) 2024). In the generative setting, models such as Alpaca (Taori et al.) 2023), Vicuna (Chiang et al., 2023), Self-instruct (Wang ct al. 2023), etc.. have been shown to use supervised instruction-followingg fine-tuning to improve generation, where the tuning data is generated using LLMs. On the -.-.- classification task, AugGPT (Dai et al., 2025 employs a teacher LLM to rephrase input sentence to im- prove general and clinical-domain classification perfor- mance. Various other works (Gilard et al. 2023 He ct al. 2023 Gao et al. 2023 Chenglin et al. 2024 et al. 2023) have considered guidance for LLM--baed classification in the context of annotation generation, data clustering and curation, etc., but do not target the XC setting. The Ora-
2. Related Work
Extreme Classification:
XC a crucial component in rank-
ing and recommendation systems (You et al.
2019
Guo
let al. Dahiya et al., [02T Mittal et al.
2019
2021 Saini
let al. 2021 Gupta et al.) 2023 Mohan et al. 2024). XC
approaches learn a classifier associated with each label, treat- ing each of the labels as classes in the multi-label setting, with features obtained either via classical approaches such as bag-of-words (Babbar & Schölkopf 2017 Prabhu et al.] 2018b), decision trees (Prabhu et al., 2018b) or via deep- learning techniques that leverage either pre-trained (Jain ct al. 2019) or learned (You et al.) 2019 Jiang et al. 2021 Dahiya et al. 2023a) models. The closest approach ours is that of OAK (Mohan et al. 2024), wherein an XC classi-
3

---

MOGIC: Metadata-infused Oracle Guidance for Improved Extreme Classification
cle guidance framework in MOGIC can be viewed an instantiation of guided representation learning, wherein the text-based early-fusion Oracle provides supervision for downstream model such as OAK. MOGIC is therefore or- thogonal to these existing guidance- based approaches.
the disciple in the second phase of the MOGIC frame- work. Typically the oracle is computationally ex- pensive, but highly accurate gaabedding-based model. The oracle accepts as input. queries. labels and their associated metadata, and generates high-quality repre- sentations, while being trained/finetuned for the XC task. These embeddings are subsequently used to guide the disciple via a guidance loss. In MOGIC, we con- sider DistilBERT (Sanh et al., 2219)) LLaMA-2 (Tou: vron et al. 2023) and Phi-2 Javaheripi et al. 2023)
3. The MOGIC Approach
Before we describe the MOGIC algorithm, we first intro- duce the notation used in this paper.
finetuned on the XC task as oracles.
Notation: Consider the task of query-to-label subset pre- diction, as common in the XC setting. Let L denote the total number of labels present, and Q, the total number of queries. Let xg. Z be the textual descriptions of the query and the label, indexed by q and respectively. For each query Xg. its ground-truth label vector is yy E {[1,++1], where lyght dgt 1 if label I is relevant to query q and Vet = 1 otherwise. Let Ag be the textual descriptions of the auxiliary memory item (indexed by k), and M de- note the total number of memory items. For each query X, or label Z,, the vector representation of it's ground truth memory-items is denoted by an/1 with a E {-1, +1 where a +1 if memory item k is relevant and vice-versa. We assume that the label and the memory memory-item sets remain unchanged between training and testing.
3. Task-specific Loss Function:
The loss, denoted by Loisciple, is associated with the task for which the dis- ciple is being trained. For instance, in OAK, Loisciple is a triplet margin loss (Mohan et al. 2024).
4. Guidance Loss Function: The guidance is provided by the oracle to the disciple using two loss functions: LAlignnent and Lratching (cf. Section 3.3
MOGIC is a highly modular framework which can accom- modate different choices of the oracle model, the disciple model and its task-specific loss function. The following section discusses the training of the oracle and the guidance loss employed to regularize the disciple.
In summary, X def (x U ( U (A denote all the textual information, comprising Q labeled queries, L labels, and M memory items. The dataset then denoted by
3.2. MOGIC Phase 1: Oracle Training
To train highly accurate XC oracle, three components are critical: (a) The task-specific loss function; (b) Super- vised training data, and (c) Auxiliary metadata, which can enhance the quality of label and query. Prior work has predominantly focused on designing effective task-specific loss functions (Dahiya al., 2023a Gupta et al. 20233
D {(Xq: ya. an) 9= (Zr, ai ) { Within this setting, the XC problem is one of predicting relevant set of labels y, associated with each query X,, while leveraging memory items. We now present the MOGIC framework.
Kharbanda et al. (023). We leverage the standard triplet loss with bn-batch negative sampling for training our oracle model. The oracle is trained by using early-fusion, wherein the query, label, and their associated metadata are provided at the input layer via simple text concatenation (with appro- priate delimiter tokens). The input is then projected onto an embedding space Rd using an encoder E For exam- plc, given query q with textual description Xg. the input to the oracle is the query text concatenated with its asso- ciated m memory items, i.e. X, X,I1Ap.11.... .I.....A.A..... denotes the delimiter token, and the corresponding
3.1. The MOGIC Framework
MOGIC comprises four main components (a) The base XC model (disciple), either memory-based -baeed, or memory-free free; (b) The oracle O for guidance. (c) The XC-task-specific loss function; and (d) The guidance loss function. In this paper, we primarily focus on memory- based XC models, and in particular, OAK. Please refer to Figure [ for an overall understanding of the integration of OAK into the MOGIC framework. Appendices B Bnd provide additional details. The four blocks are described below.
where || embedding is given as x; = Egg (X On the label side, the eetaadtta-enrichen label representation is computed as z Egg (Zi), where Zi is defined in manner similar to X The optimization objective for the oracle is:
1. Disciple (D): The disciple (in this case, OAK) is a trainable XC architecture with parameters 0D such that this encoder model takes in query or label and outputs a d-dimensional embedding Eop: X + 1d11 that lies on the unit sphere Sd-1,
0o aarg C [xx {zi hez, ( (BotHEEQ,EL) 0
2. Oracle (O): The oracle is an encoder model (either a large language model (LLM) or a small language model (SLM) with parameters to and used to guide
where L can be any discriminative loss function that pushes relevant labels closer to and irrelevant labels farther away from the query in their joint embedding space. Based on
4

---

MOGIC: Mctadata-infused Oracle Guidance for Improved Extreme Classification
L iggeentt
Lystchang
Query (X²): Gummy Candy
MEMORY BANK
C
Edo
Meladatai A Gummi Candies
Query(Xgg) Gummy Candy
Metadatal A, Candy
EvD
Lerectple
Label(2])
E
Eoo
Label (2z): Jelly Bean
Jelly Bean
Metadata(Ar. Gelatin desserts
DISCIPLE (OAK)
ORACLE GUIDANCE
ORACLE
Figure 1: The MOGIC training framework can be used with any XC approach. In this figure, MOGIC is used over the OAK disciple and its task-specific loss. An oracle's (LLaMA-2, Phi-2 or DistilBERT) embeddings are used to regularize the representation of OAK using the guidance loss. The green box represents the OAK disciple architecture, comprising encoder (Eep). memory bank and the combiner (C). The oracle comprises the encoder (Eeo). The MOGIC approach involves training the disciple model on three loss: (a) The task loss L (b) The oracle-disciple ranking alignment loss Lalignennt and (c) The oracle-discipe embedding matching loss Lxatching
empirical success, we use a triplet-loss-based optimiza- tion, which has demonstrated superior performance of dual- encoder-based models:
and (b) A task-specific loss function over which the disciple is trained. In MOGIC. we propose two additional loss terms, namely the Alignment loss and the Matching loss, to pro- vide oracle-guidance to the disciple for learning superior embeddings. We describe these two loss components below.
STriplet (xx)): {zi), (Uge))
Σ Bor =-n=-1
x
Zn - x
2p + 71+,
1. Al iggnent:: The Al Aligmment loss focuses on aligning the rankings of the oracle and the disciple. To enforce this, MOGIC introduces a triplet margin loss between the oracle's 's query and the disciple's 's label embeddings, and vice versa, as follows.
where Zp and nn are the embeddings of the positive and negative label, respectively, Xy is the query embedding, and y is the margin. We omit superscripts and indexing subscripts in the equation above for notational simplicity.
For the LLaMA-2 and Phi-2 SLMs, we perform LoRA fine- tuning for the specific XC task using the corresponding supervised training data and simple prompting (ef. Ap- pendix D). For DistilBERT, we perform standard finetuning.
Lalignnent = Lrriplet ((x,): (2) (Uee))
(I)
+ Ciriplet
(x {2r) [voe))
Recall that the asterisk in the superseript indicates an embedding from the oracle model.
3.3. MOGIC Phase 2: Oracle-guided Disciple Training
The oracle typically demonstrates high accuracy on the downstream XC task due to its larger size and access to privileged information (ground truth textual metadata) not accessible to the disciple. However, they are impractical to deploy in any real-world application due to one or more of the following limitations: (a) High computational cost during deployment; (b) Slow inference time, or (c) A lack of robustness to noisy metadata. Therefore, we use guidance, in the form of embeddings from the oracle, to regularize chosen disciple model. The inspiration for training a disci- ple model using an oracle is drawn from from knowledge distillation, however, the proposed framework has several key differences compared to traditional distillation, as dis- cussed in Appendix C.I While we present experimental results using multiple disciples in Section 4 here, we base our discussion around the OAK disciple as it has been shown to be the state-of-the-ar for XC tasks.
Matching: The Matching loss focuses on ensuring that the disciple mimics the oracle's 's embeddings. To enforce this in MOGIC, we introduce 1.2 loss be- tween the oracle query embeddings and the disciple query embeddings (and similarly for labels) as follows:
LMatching
Σ ||x 9EQ
x1112
+ Σ || IEL
zi (2)
where X, (X,) and 2; (2) are the query and label embeddings of the disciple, respectively. Finally, MOGIC combines the aforementioned two losses with the disciple's 's task-specific loss function as follows:
LYOGIe LDisciple a Laligement 3 LMatching
where 3 are tunable hyper-parameters and set to 1.0 and 0. respectively in our experiments. Appendix provides a detailed sensitivity analysis over various values of a and 3. During the guidance training phase, only the disciple is trained, while the oracle remains frozen.
Disciple training comprises two key components: (a) An embedding generator which provides embeddings Xg and zı, associated with given query X, and label Z,, respectively
5

---

dataset and its MOGIC: Metadata-infused Oracle Guidance for Improved Extreme Classification
3.4. Theoretical Justification of Oracle-Guided Losses
Wikipedia dumps dated 2022-05-20. LF-WikiTides--00k and -iFiiiippedia-500K (full-text version of the former) are datasets where, given Wikipedia article/page, the task is to predict the Wikipedia categories that the article should be tagged with. Other Wikipedia article titles connected to the original page via hyperlinks in the article are used as metadata. Similarly, the task in the JF--WikiSeAAll IstT'il 320K and L.F-WikiSeeAlso-222 former) datasets is, given Wikipedia article/page, to predict the other Wikipedia articles to be recommended in the "See Also" section. The Wikipedia categories that these articles are tagged with are used as metadata in this scenario. In a similar vein, the Amazon datasets are created from publicly available Amazon Product review dump Given a product title, the task in the full-text version, A.--Amazon-1311 is predict products that are likely to be bought together. The product category that these products are tagged with serve as the metadata.
Oracle-guided training, where a frozen oracle guides the training of disciple model through the Alignment and Matching losses, enhances the disciple's 's accuracy and ro- bustness, by enabling implicit knowledge transfer and im- proving convergence with fewer training samples. We the- oretically analyze the impact of minimizing these losses on the disciple's accuracy and measure the sample com- plexity under the following simplifying assumptions: (a) A Lipschit--contimuuus binary classification loss (with con- stant K) replacing the triplet loss, and (b) Embeddings are norm-bounded by B. The disciple is dual encoder with parameters 0D= = {8%. f,), where b% E F and f%/, E G represent the query and label tower parameters, respectively. The hypothesis classes F and G have bounded complexities, characterized by Rademacher constants Rg and R₁. The following is our key result:
(full text version of the
Titless-
Theorem 1. Given MOGIC's problem setting, if the disciple model is trained by minimizing the oracle- guided loss L = LAt grment + K B. Lmatching on the training set DD ~ D with N samples, then for some X > O and any ō E [0, 1]. the following inequality holds true with probability at least
Implementation Details:
We initialize all oracle and disci- ple encoders with a DistilBERT checkpoint pre-trained on the MS-MARCO dataset (Chen et al.] 2024) and fine-tune it. Table TO Appendix M2 summarizes the various hyper- parameters used for each dataset. We remark that MOGIC uses golden linkages for the metadata only at training time, whereas at inference time, these metadata linkages are in- duced (i.e., for a new query, links to metadata are predicted by the disciple). All models were trained using the Py library on a machine with 4 AMD MI200 GPUs.
1 8:
E ((x2))))) [Coiscioll] S (((( [Coracie]
4K N (Rg + RR) + 2
log(} x N
Proof. The proof is provided in the Appendix B
4.2. Results
The above theorem shows that, post oracle-guided training, the disciple's 's expected population loss tends to be close to oracle's 's population loss itself, thus inheriting strong ora- cle accuracy. This implies good training efficiency for the proposed MOGIC framework.
We now present the experimental results and conduct an ablation study to analyze the contributions of different com- ponents of our proposed framework (ef. Appendix G.2 for further details).
Main results on benchmark datasets: MOGIC is com- pared against state-of-the-art XC and dense retrieval ap- proaches in Table MOGIC leads to state-of-the-ar ac- curacy on multiple datascts. These accuracy gains are at- tributed to gradient regularization from the oracle model.
4. Experiments and Results
We now present the experimental setup, and results on train- ing various disciple models using the MOGIC framework.
In particular, MOGIC (OAK) outperforms OAK by 1-2% in P@ 1 and 2-3% in propensity metrics. In addi- tion to OAK, MOGIC also outperforms graph-based ap- proaches such as GraphFormers (Yang et al.) (021) and GraphSage (Hamilton et al.] (017) by 8%, even those these baseline approaches are given an unfair advantage with ac- cess to ground-truth memory items during inference. Note that MOGIC makes no changes to the input, but only the training procedure is improved with the inclusion of a novel
4.1. Datasets and Experimental Setup
The XML Repository (Bhatia et al.) 2016) provides various
public XC datasets which have been thoroughly studied in the literature (You et al., 2019
Guo et al. 2019 Dahiya Saini et al. 2021 Gupta
ett al.
2021
Mittal et al.
2021
let al.
2023 Mohan et al.
2024). However, very few of
them Mohan et al. 2024| Chien et al. 2023) offer ground
truth metadata. To address this, we attach ground truth
metadata from the original dumps to existing XC datasets.
Table Appendix F. summarizes the dataset statistics.
https://csswebb http:///wwwebbuuuucdd/ddd///d/ https//.www......c.all/aaaatttaa
amazon_reviews
The Wikipedia datasets were created from publicly available
6

---

MOGIC: Mctadata-infused Oracle Guidance for Improved Extreme Classification
Table 2: Main Results. Results on public benchmark datasets. MOGIC up to 2% more accurate as compared to baselines. Results on full-text datasets are in Appendix G.5 For details on evaluation metrics, see Appendix F4
17.5
MOGIC (OAK)
15.0
Method
P@1
P@5
N@5
PSP@5
7.5
LF WikiSee Aiso Titles320KK
5.0
MOGIC (OAK)
34.62
17.93
27.44
33.18
OAK
33.71
17.12
24.53
30.83
DEXA
32.91
16.77
24.63
29.55
NGAME
32.64
16.6
23.44
29.87
complete
4
(#312K( (#74K) (#31K) (#9K) (#1K)
ANCE
30.79
15.36
25.14
28.73
(1.49) (3.94) (9.14) (30.98) (200.78)
DEXML
29.9
14.80
22.80
25.70
Quantiles
(Increasing Freq.)
GraphFormers
21.94
11.79
24.02
22.70
GraphSAGE
23.13
8.26
25.12
18.73
Figure Quantile wise-comparison of MOGIC and base-
lines for LF- EE-WikisseellsoTTnees-222
MOGIC (OAK)
47.28
18.55
34.97
26.12
Table 3: Choice of oracle models. Performance comparison of MOGIC on the LF-WikkSSeAllsoTilles different oracle models. The DistilBERT oracle outperforms the LoRA-finetned SLM oracles. Nevertheless, via the MOGIC framework, the disciple is capable of leveraging the SLM oracles' signals to improve its task performance.
OAK
44.82
17.67
33.72
24.90
considering
DEXA
47.41
17.62
33.64
24.03
NGAME
39.04
16.08
30.75
23.03
ANCE
29.68
12.51
25.10
21.18
GraphFormers
24.53
11.33
20.35
19.53
GraphSAGE
21.14
11.3
22.61
11.82
LF-AmazonTitless TithnzoonTille-1133 131K
Oracle params)
Finetuning
P@1 p@s
Nēs PSP@1 PSP@5
MOGIC (OAK)
47.01
22.40
49.51
50.33
MOGIC (OAK)
OAK
46.42
21.88
49.06
49.78
DEXA
46.42
21.59
49.00
49.65
DistiBERT (65M)
full
34.62
17.98
35,70
27.44
33.18
NGAME
46.01
21.47
48.67
49.43
I.aMA-2 (7B)
LoRA
34,64
17.93
35.71
27.28
33.02
Phi-2 (2.718)
LoRA
34.34
17.73
27.09
regularization loss. Furthermore, since MOGIC is regular- ization framework, it leads to no additional inference cost (cf. Appendix G.3
Oracle-only Performance
DistiBERT (65M)
full
47.63
22.75
48.37
36.71
41.45
LLaMA-2 (7B)
LoRA
34.20
16.21
33.14
30.46
31.92
Phi-2 (2.7B)
LoRA
33.32
15.48
31.87
29.75
30.61
Figure 2 presents a quantile-wise comparison of MOGIC The lefimost
and baselines on LF- Wikiisee bin contains the highest fraction of rare (tail) labels whereas the rightmost bin contains the most popular (head) labels. MOGIC (OAK) gives consistent gains in tail bins and com- parable results in head bins (see Appendix F.3 for details).
Ablations loss functions: To understand the importance of each loss term in the second phase of MOGIC training, we ablate by removing each of the three losses, one by one. Lastly, we remove both the oracle-guidance-based loss func- tions (Matching and Ali iigmment). The results, presented in Table suggest that all loss components are important and the best accuracy achieved when all components are used.
Choice of oracle model: MOGIC's improvement can be attributed the oracle model uses to guide the gradi- ents of the disciple. In Table 3 we show results with three different models as the oracle: A finetuned DistilBERT (which is our recommended choice of the Oracle), and two LoRA-finetuned SLMs, Phi-2 and LLaMa-2. A learnable linear projection layer downsizes the SLM embeddings for computing the Alignment and Matching losses. A disciple regularized with DistilBERT performs on par with the SLM oracles. For all experiments, we use tetadata-infused Dis- tiBBERT as oracle due to its faster training and inference times. This choice increases efficiency without sacrificing disciple accuracy. We compare against SLM oracles applied to the task in Appendix G.6
Impact of using ground-truth metadata: A practical sys- tem uses metadata predicted by the disciple, given query. Hence, all experimental results thus far for MOGIC have been reported considering predicted metadata linkages. In MOGIC we observe a large gap in performance of Oracle and MOGIC (disciple) model, which we attribute to the oracle's 's access to ground-truth links to the metadata during both training and testing. We therefore hypothesize that in ideal scenario, using ground-truth metadata test time could boost overall XC task accuracy. To validate this, we replace predicted metadata in MOGIC with ground-truth metadata during inference and observe that the gap in per-
7

---

MOGIC: Mctadata-infused Oracle Guidance for Improved Extreme Classification
Table 4: Ablations on loss functions. We present ab- lations on Aligment and Matching losses. The MOGIC (OAK) configuration considers all three loss components: Disciple + Alignment + Matching.
Table 6: MOGIC is applicable to any XC disciple. On LF- WikiSeeAAssTittes- of the base algorithm by 1-2% in P@1 1. For results on L.F. Amazon Titles-1311 dataset, refer Appendix G.4
320K, MOGIC improves accuracy
Loss terms 2
P@5
N@8
PSP@1
PSP@5
Models
N@5
PSP@1
PSP@5
I.F- L.-
MOGIC (OAK)
34.62
17.93
27.44
35.70
33,18
OAK
33.71
17.12
24.53
30,81
MOGIC (OAK)
34.62
17.93
27.44
35.70
33,18
Disciple + Alignment
34.12
17.66
26.72
35.16
32.57
MOGIC (NGAME)
32.37
16.38
26.87
33.16
31,08
Disciple + Matching
34.11
17.63
26.81
35.24
32.40
NGAME
30.72
15.42
25.18
31.56
28.88
Disciple
33.71
17.12
24.53
33.83
30.83
MOGIC (DEXA)
16.92
34,00
31.82
Alignment + Matching
32.70
16.92
26.03
33.60
31.30
DEXA
31.57
16.14
25.64
32.71
29.99
LF-WikTiiles:500
Table 7: Robustness to missing metadata: Results com-
MOGIC (OAK)
47.28
18.55
34.97
27.29
26.12
paring the performance of MOGIC (OAK) when reducing the size of memory bank on L.F AlsoTse- to simulate missing metadata. Size (%) denotes the percent- age of the metadata retained. As the memory bank size is decreased by randomly removing items, MOGIC's perfor- mance decreases only slightly (both with predicted and with ground truth metadata) but Oracle suffers significantly.
Disciple + Aligrment
45.22
17,58
33.49
27.24
25.10
Disciple Matching
46,03
16.86
32,86
26.87
24.19
Disciple
44.82
17.67
33.72
25.79
24.90
Alignment + Matching
44.93
17.40
33.18
26.87
24.73
Table 5: Impact of using ground truth metadata. We at- tribute the large gap between the oracle and disciple models' performance to the availability of ground-truth metadata during training and inference. We observe that providing ground-truth metadata at test time to MOGIC (OAK) re- duces the gap in performance between the oracle and disci- ple from 8% to 6%. Additionally MOGIC is robust to noisy (predicted) metadata but the oracle is not. The oracle accu-
Size (%)
P@1
P@5
N@5
PSP@1
PSP@5
MOGIC Predicted metadata
100
34.62
17.93
27.44
35.70
33.18
34.54
17.87
27.36
35.60
33.08
racy drops by 17% in P@ on LF-WikiSSeeAlssTileee----
60
34.38
17.81
27.29
35.47
32.98
40
34.17
17.72
27.22
35.28
32.85
MOGIC Ground -taund metadata
Models
Metadata Source
P@1
P@5
N@5
PSP@1
PSP@5
100
36.94
19.12
29.00
38.42
35.07
Ground truth
36.94
19.12
29.00
38.42
35.07
MOGIC (OAK)
80
36.69
19.02
28.82
38.17
34.92
Prodicted
34.62
17.93
27.44
35.70
33.18
60
36.47
18.86
28.70
37.85
34,66
Ground truth
47.63
22.75
48.37
36.71
41.45
40
35.91
18.59
28.45
37.24
34.26
Oracle
Prodicted
25.09
12.88
19.31
26.05
23.33
Oracle
formance between the oracle and disciple decreases from 8% to 6% on the LL-W dataset.
100
47.63
22.75
48.37
36.71
41.45
80
39.22
19.07
30.44
40.15
35.08
35.08
17.30
27.58
36.13
32.08
The ability to extract useful information from noisy meta- data is a desirable quality for disciple. We can measure this via model's ability to handle predicted metadata dur- ing inference. Table shows that using predicted metadata leads to decrease of just ~122% for MOGIC (OAK) across metrics, showing that it is fairly robust. However, the ora- cle accuracy drops by 22% in P@ when it uses predicted metadata, likely due to the oracle's 's reliance on ground -rrth metadata during training. Although MOGIC receives guid- ance from relatively less-robust oracle, the mechanisms in our paale-guidance-bee significantly more robust disciple.
40
30.64
15.43
24.45
31.80
28.92
and DEXA. T that MOGIC provides 1-2% im- provement in precision and NDCG, and 2-3% improvement in PSP over the base XC algorithms.
4.3. Robustness Analysis
We now analyze the robustness of the MOGIC framework to both missing metadata and noisy metadata.
Robustness to missing metadata: MOGIC uses memory items to improve query representation and regularize XC models. To simulate missing metadata, we decrease the size of the memory bank by randomly removing items (i.e., by randomly subsampling the set of retrieved memory items). In Table 7 we present the performance of MOGIC (OAK) when the size of the memory bank is reduced. We observe
training enables learning
MOGIC is applicable to any XC disciple: While we present the main results using OAK as the disciple model, to test the general applicability of the MOGIC framework, we experiment with two other popular XC disciples: NGAME
8

---

MOGIC: Metadata-infused Oracle Guidance for Improved Extreme Classification
Table 8: Robustness to missing metadata: MOGIC is more robust to noisy metadata that the oracle it is guided by. Introducing noise in the fused metadata at inference time can lead to up to 20% reduction in accuracy of the oracle, since early-fusion models rely on high-quality metadata at the imput unlike the late-fusion-bast based MOGIC model. The performance of MOGIC (OAK) decreases only slightly, indicating that models trained with MOGIC are robust to noise in the metadata.
ring high inference latency. This is achieved via two-phase training. The first phase trains oracle using textual meta- data infusion both on the query well as the label side. The second phase involves guiding the training of a disciple model using embeddings from the oracle classifier.
A key reason for the effectiveness of our method is that MOGIC, trained with the proposed regularization strategy, effectively retains the original intention of the query. leverages the best of both worlds: maintaining low inference latency and gracefully handling noisy or missing metadata via late-fusion, while simultaneously learning to mimic the superior embeddings of the oracle through our regulariza- tion loss. This combined approach mitigates the respective drawbacks of both late- and early-fusion strategies, leading to consistently superior performance.
Noise %
P@1 1
P@5
N@5
@
PSP@5
MOGIC (OAK)
0
36.94
19.12
29.00
38.42
35.07
20
36.26
18.80
28:66
37.69
34.61
40
35.62
18.44
28.36
36,90
34,08
60
34.92
18.12
27.94
36.19
33.59
Through extensive experimentation on six popular bench- mark XC datasets, we demonstrated that the MOGIC signif- icantly outperforms state-of-the-ar XC models, achieving improvements in terms of precision, NDCG, and propensity- scored precision. Moreover, MOGIC exhibits robustness to missing and noisy metadata, making it a valuable tool for real-world applications.
Oracle
0
47.63
22.75
48.37
36.71
41.45
20
34.80
16.83
26.67
35.64
30.73
40
26.75
13.1
20.45
27.56
23.87
60
18.65
9.31
14.29
19.44
17.02
that, as the size of memory bank is decreased, MOGIC's performance decreases only marginally (both with predicted as well as with ground-truth metadata) but the oracle per- formance worsens significantly. We observe that the drop in accuracy is larger for the oracle model in comparison to MOGIC, which we attribute to MOGIC's robustness to noise in the memory items.
In conclusion, MOGIC represents significant advancement in the field of extreme classification, offering a practical and effective solution for incorporating metadata to enhance performance. Our work highlights the potential of oracle- guided training in improving the robustness and accuracy of memory- based models in challenging classification tasks.
Robustness to noisy metadata: To further understand the relationship between the oracle's 's and MOGIC's ' perfor- mance under the action of noise, we consider the following setting. First, while predicting, we take the ground-truth metadata from both the oracle and the MOGIC models. Subsequently, for every query, we inject varying levels of noise (from 0% to 60%) to the ground-truth metadata, and measure the degree of robustness of both the models to such noise. Noise is added by randomly replacing a certain percentage of ground-truth metadata items with irrelevant ones. Table shows that, we increase noise, the XC task performance decreases for both the models, signifying the importance of clean metadata. However, MOGIC's down- stream performance decreases only slightly, while the ora- cle's performance decreases significantly, demonstrating the robustness of MOGIC. We also observe that MOGIC (OAK) robust noise in the metadata even during training, and the drop in its performance is negligible. For additional details, refer to Appendix G.7
A Acknowledgments
We gratefully acknowledge the financial support provided by the Yardi School of Artificial Intelligence at the Indian Insti- tute of Technology Delhi, and the Prime Minister's Research Fellowship. We also thank HPC, IIT Delhi and Microsoft Research India for providing computational resources.
Impact Statement
Our usage of data and terms of providing service to peo- ple around the world has been approved by our legal and ethical boards. In terms of social relevance, our research helping millions of people find the goods and services that they are looking for online with increased efficiency and a significantly improved user experience, both via sponsored search ads as well as via e-commerce search. This facilitates purchase and delivery without any physical contact which is important given today's 's social constraints. Furthermore, our research is increasing the revenue of many small and medium businesses including mom and pop stores while also helping them grow their market and reduce the cost of reaching new customers.
5. Conclusion
We introduce MOGIC, a novel framework for enriching query representations using relevant metadata without incur-
9

---

MOGIC: Mctadata-infused Oracle Guidance for Improved Extreme Classification
References
Cuconasu, F., Trappolini, G., Siciliano, F., Filice, S., Cam pagnano, C.,, Maarek, Y., Tonellotto, N., and Silvestri, F. The power of noise: Redefining retrieval for RAG systems. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Infor- mation Retrieval, volume 17 of SIGIR 2024, pp. 719-729, ACM, July 2024, doi: 10. 10.11445/322677233557884 URL http://dx.doi org/ 001.1.1111556211 011.01111111/36627722336333
Akyurek, A. F., Akyurek, E., Kalyan, A., Clark, P., Wi- jaya, D. T., and Tandon, N. RL4F: Generating natural language feedback with reinforcement learning for repair- ing model outputs. In Proceedings of the 6l Annual Meeting of the Association for Computational Linguis- tics ( (Volume 1: Long Papers). pp. 7716-7733, Toronto, Canada, July 2023. Association for Computational Lin- URL
guistics. doi: 10.186553/11/20233aallloooz
Dahiya, K., Saini, D., Mittal, A., Shaw, A., Dave, K., Soni, A., Jain, H., Agarwal, S., and Varma, M. DeepXML: A Deep Extreme Multi-Label Learning Framework Applied to Short Text Documents. In WSDM, 2021.
Babbar, R. and Schölkopf, B. DiSMEC: Distributed Sparse Machines for Extreme Multi-label Classification. In WSDM, 2017.
Dahiya, K., Gupta, N., Saini, D., Soni, A., Wang. Y., Dave, K., Jiao, J., Gururaj. K., Dey, P., Singh, A., Hada, D., Jain, V., Paliwal, B., Mittal, A., Mehta, S., Ramjee, R., Agarwal, S., Kar, P., and Varma, M. Ngame: Negative mining-award aware mini-batching for extreme classification. In WSDM, March 2023a.
Babbar, R. and Schölkopf, B. Data scarcity, robustness and extreme multi-label classification. Machine Learning, 108(8):1329-135: 2019.
Bhatia, K., Dahiya, K., Jain, H., Mittal, A., Prabhu, Y., and Varma, M. The Extreme Classification Repository: Multi- label Datasets & Code, 2016. http:/tttt:/mikvaaattt htg//domlLoadd/////MMLPPppssssss
Dahiya, K., Yadav, S., Sondhi, S., Saini, D., Mehta, S., Jiao, J., Agarwal, S., Kar, P., and Varma, M. Deep encoders with auxiliary parameters for extreme classification. In Proceedings of the 29th ACM SIGKDI Conference on Knowledge Discovery and Data Mining, pp. 358-367, 2023b.
htmll
Chang. W.-C., Y., Zhong, K., Yang. Y., and Dhillon, I.-S. Taming Pretrained Transformers for Extreme Multi- label Text Classification. In KDD, 2020.
Dai, H., Liu, Z., Liao, W., Huang, X., Cao, Y., Wu, Z.., Zhao, L., Xu, S., Zeng, F., Liu, W., al. Auggpt: Leveraging chatgpt for text data augmentation. IEEE Transactions Big Data, 2025.
Chen, Q., Geng, X., Rosset, C., Buractaon, C., Lu, J., Shen, T., Zhou, K., Xiong, C., Gong, Y., Bennett, P., et al. Ms marco web search: A large-scale scale information-rich web dataset with millions of real click labels. In Companion Proceedings of the ACM Web Conference 2024, pp. 292- 301, 2024.
Fan, W., Ding, Y., Ning, L., Wang, S., Li, H., Yin, D., Chua, T.-S., and Li, Q. A survey on rag meeting Ilms: Towards retrieval- -ugmente- large language models. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 6491-6501, 2024.
Chenglin, I.., Chen, Q., Li, I.., Wang, C., Tao, F., Li, Y., Chen, Z., and Zhang, Y. Mixed distillation helps smaller language models reason better. In Al-Onaizan, Y., Bansal, M.. and Chen, Y.-N. (eds.), Findings of the Association for Computational Linguistics: EMNLP 2024, pp. 1673- 1690, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10. 10.1888111/2244 findings-emmlp.91. URL 2024. f findings-dennnnnn 91/
Gao, J., Pi, R., Yong, L.,, Xu, H., Ye, J., Wu, Z.,, ZHANG, W., Liang, X., Li, Z.., and Kong, L. Self-guided noise-free data generation for efficient zero-shot learning. In The Eleventh International Conference on Learning Represen- tations, 2023. URL https:://opereeee id=h50pjGd_1o6
org//
wet/forun
Chiang, W.-L., Li, Z., Lin, Z., Sheng, Y., Wu, Z., Zhang, H., Zheng, I.,, Zhuang, S., Zhuang, Y., Gonzalez, J. F.,, Stoica, I., and Xing. E. P. Vicuna: An open-source chatbot impressing GPT-4 with 90%* ChatGPT qual- ity, March 2023. URL 2023 30 viiuna/
Gao, Y., Xiong, Y., Gao, X., Jia, K., Pan, J., Bi, Y., Dai, Y., Sun, J., Wang. M., and Wang, H. Retrieval- auggeettd generation for large language models: A survey. arXiv 2024. URL htttts/////
preprint org/abs/2312.0 10997
Gaur, M., Gunaratna, K., Srinivasan, V., and Jin, H. ISEEQ: Information seeking question generation using dynamic meta-information retrieval and knowledge graphs. In Proceedings of the AAAI Conference on Artificial Intelli- gence, volume 36, pp. 10672-10680) 2022.
Chien, E., Zhang, J., Hsieh, C., Jiang. J., Chang, W., Milenkovic, O., and Yu, H. PINA: Leveraging side infor- mation in eXtreme multi-labes classification via predicted instance neighborhood aggregation. In ICML. 2023.
10

---

MOGIC: Metadata-infused Oracle Guidance for Improved Extreme Classification
Gilardi, F., Alizadeh, M., and Kubli, M. ChatGPT out- performs crowd workers for text-annotation tasks. Pro- ceedings of the National Academy of Sciences, 120(30), 2023. doi: 10. 10.10773/mmas
Jain, H., Prabhu, Y., and Varma, M. Extreme Multi-label Loss Functions for Recommendation, Tagging. Ranking and Other Missing Label Applications. In KDD, August 2016.
URL https:// www. www.pnas... wowg///o//aaai a00 http://ooo/abs//0.10////pppa 23315/0005..2251161111 230501
Jain, H., Balasubramanian, V., Chunduri, B., and Varma, M. Slice: Scalable Linear Extreme Classifiers trained on 100 Million Labels for Related Searches. In WSDM, 2019.
Guo, C.,, Mousavi, A., Wu, X., Holtmann-Rice D.-N., Kale, S., Reddi, S., and Kumar, S. Breaking the Glass Ceil- ing for Embedding-Banl saaddd Classifiers for Large Output Spaces. In NeurlPS, 2019.
Javaheripi, M., Bubeck, S., Abdin, M., Aneja, J., Bubeck, S., Mendes, C. C. T., Chen, W., Del Giorno, A., Eldan, R., Gopi, S., et al. Phi-2: The surprising power of small language models. Microsoft Research Blog, 2023.
Guo, Z., Cheng, S., Wang, Y. Li, P., and Liu, Y. Prompt-guided retrieval augmentation for non- knowledge-intene intensive tasks. In Findings of the Association for Computational Linguistics: ACL 2023, pp. 10896- 10912, Toronto, Canada, July 2023. doi: 10. 10.18853/v1/ 2023. wwwniinngnnscc.699 URL https://acclanthologg org/2023. Findings-acli occ..993. 693
Jiang, T., Wang. D., Sun, L., Yang, H., Zhao, Z., and Zhuang, F. LightXMI.: Transformer with Dynamic Negative Sam- pling for High-Performance Extreme Multi-label Text Classification. In AAAI, 2021.
Gupta, M. and Agrawal, P. Compression of deep learn- ing models for text: A survey. ACM Transactions on Knowledge Discovery from Data (TKDD), 16(4):1-55, 2022.
Karpukhin, V., Oguz, B., Min, S., Lewis, P., Wu, L., Edunov, S., Chen, D., and Yih, W.-t. Dense passage retrieval for open-domain question answering. In EMNLP, 2020.
Kharbanda, S., Banerjee, A., Gupta, D., Palrecha, A., and Babbar, R. Inceptionxml: A lightweight framework with synchronized negative sampling for short text extreme classification. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 760-769, 2023.
Gupta, N., Khatri, D., Rawat, A. S., Bhojanapalli, S., Jain, P., and Dhillon, I. S. Efficacy of dual-encoders for extreme multi-label classification. In ICLR, 2023.
Guu, K., Lee, K., Tung, Z., Pasupat, P., and Chang, M. Retrieval augmented language model pre- pre-training. In International conference on machine learning. pp. 3929- 3938. PMLR, 2020a.
Kim, Y. and Rush, A. M. Sequence-level knowledge distil- lation. In Proceedings of the 2016 Conference on Empiri- cal Methods in Natural Language Processing, pp. 1317- 1327, Austin, Texas, November 2016. Association for Computational Linguistics. doi: 10. 10.18653//1/D16-1133 1139.
Guu, K., Lee, K., Tung. Z., Pasupat, P., and Chang, M.- W. Realm: Retrieval-augmented language model pre- training, In ICML, 2020b.
URL
Hamilton, L., Ying, R., and Leskovec, J. Inductive rep- resentation learning on large graphs. In Proceedings of Neural Informa-
Lála, J., O'Donoghue, O., Shtedritski, A., Cox, S., Ro- driques, S. G., and White, A. D. PaperQA: retrieval- augmented generative agent for scientific research. arXiv preprint, arXiv:2312.07559 2023. URL https://
the st International Conference tion Processing Systems, NIPS' 17, pp. 1025-1035, Red Hook, NY, USA, 2017. Curran Associates Inc. ISBN 9781510860964.
arxiv.org/ant www/absg/////ooooo 2312 07559
Lan, T., Cai, D., Wang, Y., Huang, H., and Mao, X.-L. Copy is all you need. In The Eleventh International Conference on Learning Representations, 2023, URL https://openrevieet nett/forum11d-CROO0ANN..
He, X., Tian, Y., Sun, Y., Chawla, N., Laurent, T,, LeCun, Y., Bresson, X., and Hooi, B. G-retriever: Retrieval- augmented generation for textual graph understanding and question answering. Advances Neural Information Processing Systems, 37:132876-132907 2024.
Lewis, P., Perez, E.,, Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.-t., Rocktäschel, T., et al. Retrieeall-uugmented generation for knowledge- intensive nlp tasks. Advances in Neural Information Pro- cessing Systems, 33:9459-9474, 2020.
He, Z., Ribeiro, M. T., and Khani, F. Targeted data gen- eration: Finding and fixing model weaknesses. In Pro- ceedings of the 61st Annual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Pupers), pp. 8506-8522 2806, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653//1/2033. acl-long.474 URL acl-long.47
Li, J., Gui, L., Zhou, Y., West, D., Aloisi, C., and He, Y. Distilling ChatGPT for explainable automated stu- dent answer assessment. In Findings of the Associa- tion for Computational Linguisties: EMNLP 2023, pp.

---

MOGIC: Metadata-infused Oracle Guidance for Improved Extreme Classification
6007-6026, Singapore, December 2023. Association for Computational Linguistics. doi: 10. 10.18653/w1/2033 findings-emnlp.399 URL https://acllntthlloo org/2023. "iindinng-eeniii 399
Qu, Y., Ding, Y., Liu, J., Liu, K., Ren, R., Zhao, W. X., Dong, D., Wu, H., and Wang, H. Rocketqa: An optimized training approach to dense passage retrieval for open- domain question answering. 2021.
Long, A., Yin, W., Ajanthan, T., Nguyen, V., Purkait, P., Garg, R., Blair, A., Shen, C.. and van den Hengel, A. Retrieval augmented classification for long-tail visual recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 6959- 6969, 2022.
Radhakrishnan, P., Chen, J., Xu, B., Ramaswami, P., Pho, H., Olmos, A., Manyika, J., and Guha, R. Knowing when to bridgingg large language models and data. arXiv preprint arXiv:2409. 13741, 2024.
Saini, D., Jain, A., Dave, K., Jiao, J., Singh, A., Zhang. R., and Varma, M. GalaXC: Graph Neural Networks with Labelwise Attention for Extreme Classification. In WWW, 2021.
Luo, Z. Xu, C., Zhao, P., Geng, X., Tao, C., Ma, J., Lin, Q., and Jiang, D. Augmented large language mod- els with parametric knowledge guiding. arXiv preprint, arXiv:2305.cm 205407730 2023. URL https://arxiv.orgg
Sanh, V., Debut, L., Chaumond, J., and Wolf, T. DistiBBRRT, a distilled version of BERT: smaller, faster, cheaper and lighter. ArXiv, 2019.
Medini, T. K. R., Huang, Q.. Wang, Y., Mohan, V., and Shri- vastava, A. Extreme Classification in Log Memory using Count-Min Sketch: A Case Study of Amazon Search with 50M Products. In NeurlPS, 2019.
Taori, R., Gulrajani, I., Zhang, T., Dubois, Y., Li, X., Guestrin, C., Liang, P., and Hashimoto, T. B. Stanford al- paca, 2023. URL https:://crfm. stan wwt.foo..dddddd....ee edu/2023/
03/13/alpaca. html
Mittal, A., Dahiya, K., Agrawal, S., Saini, D., Agarwal, S., Kar, P., and Varma, M. DECAF: Deep Extreme Classifi- cation with Label Features. In WSDM, 2021.
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine- tuned chat models. arXiv preprint arXiv:2307.09288 2023.
Mittal, A., Mohan, S., Saini, D., Asokan, S., Prabhu, S. C., Kumar, L., Malhotra, P., Jiao, J., Singh, A., Agarwal, S., Chakrabarti, S., Kar, P., and Varma, M. Graph regularized encoder training for extreme classification. In Companion Proceedings of the ACM on Web Conference 2025, WWW '25, pp. 394-402, New York, NY, USA, 2025, Association for Computing Machinery. ISBN 9798400713316. doi: 10. 10.1145/7701776637153300 URL 1145/3701716.371222
Wang. Y., Kordi, Y., Mishra, S., Liu, A., Smith, N. A., Khashabi, D., and Hajishirzi. H. Self-instruct: Aligning language models with self eeff-generated instructions. In The s1 Annual Meeting Of The Association For Computa- tional Linguistics, 2023.
Xu, X., Li, M., Tao, C., Shen, T., Cheng, R., Li, J., Xu, C., Tao, D., and Zhou, T. A survey on knowledge distillation of large language models. arXiv preprint, arXiv:2402.1311 16, 2024. URL https://arxiv.. org/ abs/ hts/2402..33166
Mohan, S., Saini, D., Mittal, A.. Chowdhury, S. R., Paliwal, B., Jiao, J., Gupta, M., and Varma, M. OAK: Enrich- ing document representations using auxiliary knowledge for extreme classification. In Proceedings of the Inter- national Conference on Machine Learning (ICML). July 2024.
Yan, S.QQ., Gu, J.-C., Zhu, Y., and Ling, Z.-H. Cor- rective retrieval augmented generation. arXiv preprint arXiv:2401. 15884, 2024. URL htt.htt/p///iiiii orr/ abs/2401.
Muennighoff, N., Hongjin, S., Wang, L.. Yang, N., Wei, F., Yu, T., Singh, A., and Kiela, D. Generative representa- tional instruction tuning. In ICLR 2024 Workshop: How Far Are We From AGI, 2024.
Yang, J., Liu, Z., Xiao, S., Li, C., Lian, D., Agrawal, S., Singh, A., Sun, G., and Xic. X. Graphformers: Gnn- nested transformers for representation learning on textual graph. NeurlPS, 34:28798-28811 2021.
Prabhu, Y., Kag, A., Gopinath, S., Dahiya, K., Harsola, S., Agrawal, R., and Varma, M. Extreme multi-label learning with label features for warm-start tagging, ranking and recommendation. In WSDM, 8a.
Yang. Z., Qi, P., Zhang, S., Bengio, Y., Cohen, W., Salakhut- dinov, R., and Manning, C. D. HotpotQA: A dataset for di- verse, explainable multi-hop question answering. In Pro- ceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2369-2380 Associ- ation for Computational Linguistics, October-Novenie 2018. doi: 10. 10.18653/v 11511/61/DDI..... 1259.
Prabhu, Y., Kag, A., Harsola, S., Agrawal, R., and Varma, M. Parabel: Partitioned label trees for extreme classification with application to dynamic search advertising. In WWW, 1b..
12

---

MOGIC: Mctadata-infused Oracle Guidance for Improved Extreme Classification
Yoran, O., Wolfson, T., Ram, O., and Berant, J. Mak- ing retrieval-augmented language models robust to ir- relevant context. In The Twelfth International Confer- ence on Learning Representations, 2024. URL https:
You, R., Dai, S., Zhang, 7.,, Mamitsuka, H., and Zhu, S. AttentionXML: Extreme Multi-Label Text Classification with Multi-Label Attention Based Recurrent Neural Net- works. In NeurlPS, 2019.
Yu, W., Zhang, H., Pan, X., Cao, P., Ma, K., Li, Wang, H., and Yu, D. Chain-of-note Enhancing robustness in retrieval-augmenter -ugmented language models. In Al-Onaizan Y. Bansal, M., and Chen, Y.-N. (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. pp. 14672-14685, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10. 10.18665/v11 12055///10022442.mmmmmmmmmma 813. URL
mmam.m/m emnlp-main
813/
Zha, L., Zhou, Li, L.,, Wang, R., al. TableGPT: Towards unifying tables, nature language and commands into one GPT. arXiv preprint, 20t.::::00000077444 2023. URL wttp://wwwg//bb///00000000
Zhang, P., Xiao, S., Liu, Z., Dou, Z., and Nie, J.-Y. Retrieve anything to augment large language models. arXiv preprint, arXiv:2310007554, 2023. URL https: //rxxiv.org/ab//21100//55
13

---

MOGIC: Mctadata-infused Oracle Guidance for Improved Extreme Classification
Appendix
Table of Contents
Preliminaries
15
Visualization of MOGIC (OAK)
16
C End-to-end Walkthrough of the MOGIC Framework
17
Relation between MOGIC and Knowledge Distillation
18
D) Prompt for SLMs
18
E Theorefical Proofs
E, Notations
19
Derivations
F Experimental Setup
22
F1
Dataset Statistics
22
F2
Hyper-paramen pneerpparantter and Training Details
22
Label Quantile Creation
F44
Evaluation Metries
23
G
Additional Experimental Results
23
G.1
Sensitivity Analysis of the Loss Hyperparameters
23
G.2
Contribution of Components Performance Gain
................................
24
G.3
Elficiency Analysis
24
G.4
Generalization of MOGIC Other Datasets Across Disciples
................................
25
G.5
Experimental Results on Full-text Datasets
26
G.6
Comparisons with Pure LLM based Approaches
26
Dependency on Metadata During MOGIC Training
26
G.8
Experimental Results AatAAmaoooiiil-----ii.ii
27
Case Study: Effect of Metadata in MOGIC (OAK)
28
14

---

MOGIC: Metadata-infused Oracle Guidance for Improved Extreme Classification
A. Preliminaries
Ground-truth metadata:
Besides the query text, often, a variety of auxiliary information available many domains, e.g., frequenly clicked webpages for search queries in sponsored search, previously searched queries for web search query auto-completion, etc. Auxiliary information available from disparate but related tasks often have relevant diverse information that the input query does not, which can be leveraged to provide better predictions. We call such auxiliary information as ground-truth metadata. For example, sponsored search ads task that involves seeer-to-ad-keye prediction, the query-side metadata is obtained by mining the organic search webpage titles clicked in response to the query on the search engine, while on the Wikipedia categories prediction task, other Wikipedia article titles connected to the original page via hyperlinks could serve as the metadata.
Early-fusion: When the query and metadata tokens are concatenated in the original text form itself (initial stage of processing) rather than in embedding form, we call it early-fusion. This approach contrasts with late-fusion, where these combined at later stages.
Late-stage fusion:
When the query and metadata embeddings are combined after their tokens have been processed through multiple Transformer encoder layers, we call this combination as late-stage fusion.
Oracle: An oracle is a model with access to metadata information during both training and inference. These models are characterized by their superior metric values, which is their sole purpose, without any consideration for model size or computational cost (measured in FLOPs). Due to their impractical computational demands, oracles cannot be deployed real-world applications.
Disciple: A disciple is frugal model that attempts to mimic the performance of an oracle model. While not as superior as oracle, disciple excels in terms of model size and computational cost, making it suitable for deployment in real-world applications.
This is an oracle model which is given access to ground truth metadata. Metadata- Oracle is the crux of our MOGIC framework, In the two stage method, Metadata-infused oracle training forms the first stage. In this stage, an oracle model is trained using both query-side and label-side ground-truth metadata. This metadata is in textual form and is used enhance the training process.
Metadata-infused oracle:
Memory-based models versus Memory-free models: Memory-based XC models are models that have access to memory (metadata). Very few XC methods are memory-based On the other hand, most of the XC methods do not leverage metadata at all and are therefore called memory-free methods.
Query-side metadata: Additional auxiliary information related to the input query is called query-side metadata. For example, in T for the query "Grass court", query-side metadata can be "Tennis terminology", "Sports rules and regulations", "Tennis court surfaces".
Label-side metadata: Additional auxiliary information related label is called label-side metadata. For example, for the label "Clay court", label-side metadata is "Tennis terminology", "Sports rules and regulations", "Clay", "Tennis court surfaces".
Missing labels: XC tasks involve a vast number of labels, making it impractical for annotators to mark all potential labels. This inherent limitation often results in missing labels, which are labels that should have been included in the ground truth but were inadvertently omitted.
Memory bank:
The memory bank is a collection of vector embeddings of all metadata associated with both the queries and labels, parameterized by 03s. Formally, memory bank is represented by M RMX,, each memory item is mapped to a row in the matrix which we call its memory item representation m, E R.D. Here ((-)0x)) returns relevant memory items for query X₂ and label 2,, i.e., mg/1 E MM(x// // /Z/00s)).
Rademacher complexity: Rademacher complexity constants Rg R₁ are estimated as the average empirical loss of minimizing the hypothesis class on a data sample with randomly annotated labels i.e. labels are generated by a purely random Bernoulli distribution with probabilities 0.5 to be either positive or negative. In intuitive terms, the smaller the values of R2, Rú, the less prone are the query tower and label tower to overfit the finite training data, and consequently the accuracy on the test set is expected to be better.
15

---

MOGIC: Metadata-infused Oracle Guidance for Improved Extreme Classification
B. Visualization of MOGIC (OAK)
Anligneent Lmarching
Query (X₂): Gummy Candy
MEMORY BANK
400000
C
EgO
Metadatai A Gummi Candies
Query(Xa (X²) Gummy Candy
Metadataí Candy
Edp
Disciple
LabeliZa (4) Jelly Bean
Eop
EAO
Label (Za): Jelly Bean
Metadata A; Gelatin desserts
DISCIPLE (OAK)
ORACLE GUIDANCE
ORACIE
Figure 3: MOGIC (OAK) training framework, depicted in Figure presented to facilitate a detailed explanation.
In this figure, we explain the detailed architecture of the proposed MOGIC framework. The input query is "Gummy Candy". and the expected label is "Jelly Bean". The overall framework comprises of 4 different parts: (1) disciple, (2) task specific loss functions, (3) guidance from the oracle to the disciple, and (4) the oracle itself. The framework comprises of 2 stages: oracle training and disciple training. The oracle training leverages both the query-side as well as the label-side ground truth metadata. As shown in the Figure 3 the query-side ground truth metadata includes 2 memory items: "Gummi Candies" and "Candy". The label side ground truth metadata includes just one memory item in this case: "Gelatin Desserts". This rich metadata is concatenated with the query and the label in the text form to train the oracle. In stage 2, we train the disciple based on guidance from the oracle. This stage 2 illustrated in detail in the Figure The disciple denoted by the green box, consists of an encoder, memory bank and combiner C. At train time, the disciple first encodes both the query well as the label using the same encoder. Since the disciple has to be low latency, it must involve late fusion. For late fusion, we need embedding representations of both the query well as the metadata items. To obtain embedding of the metadata items, the query is also sent to the memory bank. The combiner actually performs the late fusion using cross attention and outputs an enhanced query representation. Of course while training the disciple, we need to ensure that the task specific loss minimized. This loss tries to maximize the similarity between the embedding the label and the enhanced query representation, and illustrated in the task part of the Figure 3 The guidance part of the Figure shows how the guidance passed on by the oracle to the disciple using 4 different loss functions. These loss functions try maximize the similarity between (a) oracle's query representation and the disciple's query representation (b) oracle's ' label representation and the disciple's 's label representation (c) oracle's ' query representation and the disciple's 's label representation, and (d) oracle's 's label representation and the disciple's 's query representation.
16

---

MOGIC: Metadata-infused Oracle Guidance for Improved Extreme Classification
C. End-to-end Walkthrough of the MOGIC Framework
Disciple
Task
Guidance
Ornele
Coort type
Laadbern
G
Teans scraiedagy
Query:
Nigrever
Letering
8
Memory: Tennis teminology.
Tennis court surfaces
Query:
Graolaah
4 Label: Clay Court
C00
Label: Clay Court
Aiggeeet
Masching
Memory: Clay, surfices,, Clay boumaments
Figure 4: This is a detailed version of Figure 0r. represents the parameters of the classifier that predicts metadata for given query. Our denotes the memory bank containing memory items corresponding to query and label metadata. OD represents the parameters of the base encoder in disciple. C signifies the combiner module. Collectively, OL. AM. 0D. and constitute the parameters of the disciple, denoted as 0p.
This section provides a more detailed walkthrough of MOGIC framework as depicted in Figure] | Figure 4 shows more detailed version of the architecture and it will help us walk through an example on the training and the inference of the MOGIC framework. Let us consider the query "Grass Court" mentioned in [able The framework has three major blocks query processing block, label processing block and oracle representation block. The query processing block and label processing block are part of the disciple.
Query processing block: involves the following steps
The query "Grass Court" sent to the memory bank to retrieve relevant metadata, including "Courts by type", "Landforms", and "Grasslands". The memory bank contains vector representations for each of these metadata. These vector representations are then sent for further processing.
Metadata retrieval:
Query encoding: The query "Grass court" is passed through the disciple encoder to obtain its vector representation.
Query enrichment: The query representation is fused with the metadata representation using a cross-attention layer to create an enriched query representation.
Label processing block: similar to the above query processing
Label encoding:
The label "Clay Court" is passed through the base encoder to obtain its vector representation. This encoder is shared between query and the label.
Label enrichment: The label vector representation is further enriched by combining with separate "Clay Court" free parameter.
The query "Grass Court" is concatenated with its associated ground truth metadata "Tennis terminology", "Sports rules and regulations". "Tennis court surfaces" to form super query "Clay Court Tennis terminology Sports rules and regulations Tennis court surfaces". This super query is passed through the oracle encoder to obtain its vector representation. Similarly "Clay court" is concatenated with "Clay", "Tennis court surfaces" and "Clay tournaments" and passed to the shared oracle encoder to obtain its vector representation.
Oracle representation block:
The above blocks are then used for training and inference,
Inference: During inference for the query "Grass Court" we compute its query representation using the query processing block and then calculate its similarity with all label representations, present in the dataset, computed from the label processing
17

---

MOGIC: Mctadata-infused Oracle Guidance for Improved Extreme Classification
block, including "Alabama", "Clay Court", "Carpet Court", "Hardcourt" and "Henry Moore", to determine their relevance the query using cosine similarity distance.
Training: Unlike inference, training uses all the blocks and involves the following steps:
We compute query and label representations for both the query "Grass Court" and the label "Clay Court" using the query processing block and the label processing block.
Vector representations:
Triplet loss: We then apply triplet loss the query and label representations, as common in retrieval methods.
Oracle regularization: To further regularize the disciple model, we introduce additional triplet loss terms (a) between the query "Grass Court" oracle representation and the "Clay Court" label representation. (b) between the label "Clay Court" oracle representation and the "Grass Court" query representation. (c) mean squared error (MSE) loss to minimize the distance between the "Grass Court" oracle representation and its query representation and the "Clay Court" oracle representation and its label representation.
What makes MOGIC (OAK) perform better than OAK?
We observe that for the first example "Grass Court", "Courts by type" is good memory item, but some predicted memory items ("Landforms" and "Grasslands") are misleading. This is the same metadata which is used both by OAK well as MOGIC (OAK). Unfortunately, the misleading metadata causes OAK to produce bad predictions about geographical places like "Texas, List of Nevada state prisons, Ronald Reagan Boyhood Home, West End (Richmond, Virginia)". The gracle-guddane fortunately helped MOGIC (OAK) to avoid paying attention to the misleading metadata and therefore MOGIC (OAK) ends up predicting accurate labels like "Clay court, Carpet court, Hardcourt". Even the label "U.S. Men't 's Clay Court Championships" is somewhat relevant. With MOGIC regularized training. MOGIC (OAK) was able retain the original intent of the query and predicted various type tennis courts.
C.1. Relation between MOGIC and Knowledge Distillation
The disciple training in MOGIC is novel variant of knowledge distillation (KD). Compared to standard KD, it differs in multiple ways. First, the metadata is provided early concatenation textual form for the oracle, but is used to train free parameters in the disciple's 's memory via novel regularization framework. This can be viewed knowledge distillation from an early-concatentation model to a two-tower model. While all disciple models benefit from this framework, the additional parameters present in the memory-base -based disciples such MOGIC (OAK) demonstrate the most gains. Furthermore, the oracle has access to ground-truth metadata, which is privileged information that is not available to the disciple model (unlike standard KD). In this MOGIC framework, the oracle can either be larger, or of the same size as the disciple. Table presenns comparisons between larger (Lsee, 2.7B/7B sized) oracle and a disciple-sizee sized (65M sized), DistilBERT oracle. For more details, please refer to Section 4.2
D. Prompt for SLMs
Given the title of a wikipedia article and the correspending categories of that article on wikipedia, your task is to predict the titles all articles which are likely to be listed in the see also section of the mentioned article. Output the separated list of titles of the articles in the also section of the given article.
\ela'e Input \newline
\#y#'e Title agricultural science \newline \PYAre Categories agriculture. agronony
\PARUP*# Task Output \newline
MAMAI Predicted titles \newline
agricul tural sciences basic tapics, agriculture ministry, agrascology. american society of agronony, genonics of domestication, history of agricultural science, institute of food and agricultural assessment of agricultural science and technology for developnent, national ffa organization, agricultural science.
international
Listing 1:
This prompt aims to fine-tune small language models (LLaMA-2 and Phi-2) using LoRA to predict the titles of articles likely to appear in the 'See Also' section of a Wikipedia article given its title and associated categories.
18

---

MOGIC:
Metadata-infused Oracle Guidance for Improved Extreme Classification
E. Theoretical Proofs
E.1. Notations
Let 2 = y.) {2 be the training dataset, where X,, Zi are the raw text features of 2th query and (th label respectively, and y, E {0, 1).h S is the binary label vector for the 2th query.
Let the oracle be dual encoder model denoted by parameters 0o {%) ()). Similarly, let the disciple also be dual encoder model denoted by parameters 0p {0h: ). Given the raw text query and label samples X and Z, let the frozen oracle embeddings be denoted as x" Eso (X), z = Ego (Z), and the trainable disciple embeddings be denoted by
x Eep (X), Eáp (Z).
Let = [Zi...., 2 be the matrix of all label embeddings stacked together. Consider the loss Yx)
y:) {z) 12))) =
1 Σ 1= A x; Y:) to be generic loss function that is separable over query samples. Note that the triplet loss used by N
MOGIC falls in this family of loss functions, and therefore the analysis presented below holds true for it.
Let the query and label towers of the disciple model 8D = 16% 0b) belong to hypothesis classes F,G whose complexities be bounded by Rademacher constants Rg, Rt respectively.
The Rademacher complexity constants R2 and are scalar values that quantify the complexity or capacity of the hypothesis classes corresponding the query tower and the label tower, respectively. We use the standard definitions of Rademacher constants from the Statistical Learning Theory for Binary Classification problems
Under the above setting, the Alignment and Matching losses can be expressed as:
Lxligment
=
{zi + [(((x)
(4)
2
1 1
1
Chatching
|2
- x
1/
+
|z]
27
(5)
N
i=1
1=1
E.2. Derivations
The below lemma shows that the triplet loss used in MOGIC is upper-bounded by a linear combination of the Alignment and Matching loss, assuming that the individual pairwise loss terms comprising the triplet loss are Lipschitz continuous by themselves.
Lemma 2. Let P be the number of positive labels for a given query and, E(s.,yt) P-L-P) Ep.e(t..... L) 1
+1)1( = -1)ggsiin - Sip) be the triplet loss for a query which is decomposable over all relevant-irrelevan label pairs. g is Lipschitz-continuuxi with constant K, then the following inequalities hold true:
2. L
[(((x,);)) (2) 2i(x) (2))),,)) +
|2] - z ll
(6)
L
1=1
N
2 K B
||x: x1112
(7)
N
eecs https///.ss.saaaaaeoocs/csss//ss hatp:///dd__rdecceeee.....
https://wwen http://wwwwccccueeee///sse https:/wwwwraccccccccc//am//anna
19

---

MOGIC: Metadata-infused Oracle Guidance for Improved Extreme Classification
Proof. Let 8,, 8 be score vectors. Then,
v(. - N().
(8)
1
Σ
(8kp = +1)1(ym = -1)(g(sin - -sip) g( (pp)
(9)
P. PT. P) P)
p.qEll.....
1
Σ
1(Vip 11) = 11) 8ip) =((6 k(p!a
(10)
P. P)
K
Σ
1(yip = +1) = -1)([sin - s'ml + +1ssp sipp))
(11)
P. PL-P) P)
PI
If = z x, and s -Z"x,, Xi, then:
y.) y()..
(12)
K. B
Σ
(Vip = +V) = -1)(11an
2,l12 + 112p - 25il2))
(13)
N P. (L-P)
(I..........
K.B
2NP(L
P)
L.
|2| 2i12 12
(14)
N. P. PL
L
1=1
2 K B
2 zi 12
(15)
As result, the following inequality holds true:
2.K B
[(((x))))) (2))))) S (I
+
zi l/
(16)
L
1=1
Similarly, sf 2 x, and s; = 2T TT,,, then:
((((s y.) A(5
(17)
2.K K.KK B
Σ
1/m +1)1(Mm = -1)[[x x;112
(18)
N P. -(
p...(l.....)
2 K B
| x; 11
(19)
N
As result, the following inequality holds true as well:
N
2 K B
2 (2))E)) S L(1( (2)ta)
||x;
x1112
(20)
N
i=1
Lemma 3. Assume a realizable setting where, for some 0p = 0, x z" holds for all x, z. Now, let Op 0p be another value of OD which minimizes the pracle-gideed population loss Laiignment + XLratching. Its corresponding embeddings are denoted by x, z. Further, assume that all the embeddings are bounded by ||x||2, [z||2, ||x 112, || 1|2 - B.
Then, for X K- B, the following inequalities hold:
2()( (2))-2) S LAligment + A Luatchhng S [(((()))))) (2)
(21)
20

---

MOGIC: Metadata-infused Oracle Guidance for Improved Extreme Classification
Proof. By averaging the two inequalities in Lemma 2 we get the following result:
L( M() (2)) (2)))))
(22)
- (2([(x, y:)) (2i)==)
2(((x),))) (2)
1 N N
+ K. B T, 1=1
||| zi 22
| - 12
(23)
LAl grmett + K B Lratching
(24)
Next, note that x, z are the values of x, z which minimize the oracle-guided loss. Due this and the realizable setting assumption:
Laisgment + K. B Listching
(25)
= 1 ([((((()))))) (2i
[(((x) (a))
N
+ K. B( Σ 22 - 12
| - x 112)
(26)
$ (Cll(x) y:)
(1) )|ا)
(z)())))
N
+ B Σ
|2 27 1l2
+
Σ || - x; 22)
(27)
N
1=1
c(((x)
(28)
The above proves the two inequalities.
However, optimizing the population-level vrall-guiied loss is not feasible as we are often restricted to a finite training sample size. Now, empirical loss optimization on the finite training set introduces some error. The following lemma bounds this error:
Lemma 4. Let the disciple model be trained by minimizing the orucle-guided loss on the training set D holds:
[Z Let the empirical training risk attained by this minimization be E, then the following inequality
2K N
log( ( N
min E(C 8n
-)
231 <
(R, + Ri) + V
(29)
where L. is the population-tevel sracle-guided training loss.
Proof. Proof uses the standard ideas of ghost sampling and Rademacher complexity bounding, along with some well-known properties of Rademacher complexity. Note here that ming, E.L = LAlsgnnent + A LMatchiig with x, z embeddings, thus
connecting to Lemma 3
Theorem 5. Given the problem setting described above, the disciple model is trained by minimizing the pracle-guided loss
== Laignnenent + X- Lyatching on the training set D = yx), ( then for some > and any å E [o, 1]. the following inequality holds true with probability at least 8:
E Exyy (Z x.y) S (( x" x) + 4K
log( N
(R, + Ri) + 2
(30)
N
Proof. Proof involves a simple algebraic combination of the results in Lemmas and
21

---

MOGIC: Metadata-infused Oracle Guidance for Improved Extreme Classification
F. Experimental Setup
In this appendix, we discuss details regarding the dataset statistics, training hyper- papar-paramtters and the evaluate metrics used.
F.. Dataset Statistics
Table 9: A summary of the dataset statistics in terms of the queries (Q), labels (L) and memory items (M). The Avg. queries per label is computed as the average value of the number of positive labels associated with each query in the dataset. Similarly, to find the Avg. labels per query, given a label, we identify the number of queries for which that label and LF-WikiSeeAlso-322 tasks, the Wikipedia categories that these articles are tagged with are used as metadata. For LF- wiw.iiiiittsSSOO00 and L.- JFFWiiipedia-5000 tasks, the Wikipedia article titles connected to each original page via hyperlinks in the article are used as metadata. For AmazonTillss 131K and AF-Amazon-131 131K tasks, categories associated with the product are metadata.
a positive, and compute the average of those numbers. For AF-
Dataset
* Train Queties
Y Labels
A Test Queties
Avg. Queriesfabel
Avg. Lubelsiquery
Metadata Types
Mecadasa
MetadataQQuzry
I.F. 371
693K
312K
17K
2,11
4.67
eategrey
656K
4.89
LF- WikdSesAlso 320K
693K
312K
177K
2.11
calegoey
656K
4.89
WikiEddes-SOO
18M
501K
753K
hyper-link
2.1M
13.95
LP- Al--Wiipedia
13M
501K
783K
4.74
17.15
hyper-link
2.1M
13.95
294K
131K
134K
5.15
2.29
category
210K
4,64
LF-Ammuon-DD
294K
13IK
134K
5.15
2.29
caterory
210K
4,64
F.2. Hyper-parameters and Training Details
Table [0 showw hyper-parametees used in MOGIC to regularize XC models. SLMs were obtained from the HuggingFace model repository. Phi-2 (2.7B parameters) was retrieved from wtt/m/etaa-llaammammaa----11111- 2 (7B parameters) was retrieved from httpp:://tgggtiiii htt.as////uugiiigeeecc//oeettees
and LLaMA-
Table 10: Hyper-parameter values for MOGIC on all datasets to enable reproducibility. MOGIC code will be released publicly. Most hyperparameters were set to their default values across all datasets. LR is learning rate. Margin y = 0.3 was used for contrastive loss. A cell containing the symbol indicates that that cell contains the same hyperparameter value present in the cell directly above it.
Batch Size
Encoder epochs
Encoder LR
Bert seq. length
Dataset
S
by
Lmux
LF- WikiSccAlls/TTnees522K
1024 t
300 t
0.0002
32
LF- WiOT
t
LF- F..iiiiSeeAllo32200
t
256
LF Wikipedia-500K
t
f
1
F.3. Label Quantile Creation
For Figure 2 labels were divided into 5 equi- squi-voluminous quantiles. To each label I E [L]. a popularity score V/ = if Va = +1} was assigned by counting number of training datapoints tagged with that label. The total volume of all labels was computed as Vist def Σie(L) Vi. Labels were arranged in decreasing order of their popularity score Vi. Five label quantiles were then created so that the volume of labels each bin, sum of popularity of labels, is roughly & Vir Thus, labels were collected in the first bin in decreasing order of popularity till the total volume of labels in that bin exceeded Vix/5 at which point the first bin was complete and the second bin was created by selecting remaining labels in decreasing order popularity till the total volume of labels in the second bin exceeded Vix and so on. For example, for the LFF-WikiTitles-000 500k dataset, the five bins were found to contain approximately 1K, 9K, 30K, 84K, 375 labels respectively. Note that the first bin contains very few s 1K labels since these are head labels and small number of them quickly racked up a total volume of as Vior/5 / whereas the last quantile contains more than 100x more labels at around 375K labels since these are tail labels and so a lot more of them are needed to add up to a total volume of Vor/5.
22

---

MOGIC: Metadata-infused Oracle Guidance for Improved Extreme Classification
F.4. Evaluation Metries
Performance has been evaluated using propensity scored precision@* ls and nDCG@ k. which are unbiased and more suitable metric in the extreme multi-label, setting (Jain et al. 2016 Babbar & Schölkopt 2019 Prabhu et al. 2018a|b) The propensity model and values available The Extreme Classification Repository (Bhatia al. 2016) were used. Performance has also been evaluated using vanilla precision @ k and nDCG @ & (with k 1, and 5) for extreme classification.
Let y E IR4 denote the predicted score vector and E {0, 13 denote the ground truth vector (with {0, 1) entries this time instead +1 entries, for sake convenience). The notation ranky (y) C. [L] denotes the set labels with highest scores the prediction score vector y and ||yl|, denotes the number of relevant labels the ground truth vector. Then we have:
P@k =
Σ
(31)
k (granku
PSP@k
yt
(32)
k lgranky (5)
Pl
DCG@k ko ((r.))
Σ
31 log(i + 1)
(33)
PSDCG@K 1
(34)
ko
((..)) (P), KyErrnkk (x)ce log(i +
DCG@k
nDCG@k min((..1lyll))
(35)
Ei=
log(++1)
PSnDCG@k PSDCG@K
(36)
Σ logl+1
FN@k = (ieranke0))
(37)
||yl|
Here, Pt is propensity score of the label calculated as described J Jain et al. (2016).
G. Additional Experimental Results
We now present additional experimental results and various ablation experiments pertaining MOGIC.
G.1. Sensitivity Analysis of the Loss Hyperparameters
Results in Table 2 used (a, 3) = 01)) which leads to the best performance. This section presents the ablations on the choice of and 3 on the LF-WikiSSeellsoTitees 320K dataset, with the values of all other hyperparameters defined in Appendix F2 The following table summarizes the results. We also observe that the performance of MOGIC is generally robust to the choice of (a, 3) with variance of 00108 in P@ 1, when comparing across all the choices considered. Figure presents the results in the form of a heat map for better visualization.
Table 11: Sensitivity analysis over the a and / hyperparameters, considering a e 1, and B E {0.1, 1,
8
P@1
P@5
N@5
PSP@1
PSP@5
0.1
0.1
34.23
17.75
35.40
27.05
32.67
0.1
1.0
34.34
17.74
35.45
27.07
32.63
10.0
33.70
17.47
34.83
26.23
32.07
1.0
0.1
34.62
17.93
35.70
27.44
33.18
I.0
1.0
34.56
17.91
35.66
27.32
33.12
1.0
10.0
34.05
17.61
35.05
26.55
32.42
10.0
0.1
34.16
17.57
35.04
26.84
32.39
10.0
1.0
34.11
17.56
35.00
26.72
32.32
10.0
10.0
33.61
17.30
34.47
25.89
31.66
23

---

MOGIC: Mctadata-infused Oracle Guidance for Improved Extreme Classification
P@1
34.6
0.1
34.4
34.2
R 10
34.0
10.0
33.8
1.0
10.0
B
Figure 5: A plot of Table 11 which presents sensitivity analysis over the a and 3 hyperparameters, considering all combinations of a E (0 10) and 1, 10). Red denotes high values, whereas Blue denotes low values.
Table 12: The synergistic combination of the various components within MOGIC leads the observed performance gains. This table shows ablations of MOGIC with different components WikiSee 320K dataset.
Setting
Models
P@L
P@S
N@5
PSP@1
PSP@5
MOGIC (OAK)
34.62
17.93
35.70
27.44
33.18
2
MOGIC (OAK) + Oracle w/o Metadata
34.09
17.43
34.90
26.95
32.13
Early Fusion (similar REALM)
28.49
14.52
29.46
22.26
26.52
4
MOGIC (OAK) Early Fusion
29.30
14.88
29.92
22.21
26.87
G.2. Contribution of Components to Performance Gain
To systematically evaluate the individual contributions of the various components within the MOGIC framework, we conduct a series of ablation studies in this section (cf. Table (2).
Contribution metadata in oracle training (Setting in Table 12): We train the MOGIC (OAK) model using an oracle which has not been trained using any metadata. As seen Table [2] this experiment yields results that are better than standard OAK but not as good as the proposed MOGIC (OAK) where the oracle has access to metadata during training. This shows that while individually, the regularization itself can result better performance of the disciple, training oracle with access to metadata further improves the performance.
Early fusion in disciple model (Settings 3 and 4 in Table 12 We choose to perform a late fusion of the metadata information in the disciple model by passing the query representation and memory items through a single layer combiner. An alternate choice could be to perform early fusion of metadata. To validate this, we perform two experiments, (a) only carly fusion in disciple (Setting 3), where the predicted metadata concatenated with the input and (b) both early and late fusion (Setting 4) where alongside concatenating the predicted metadata input (Table 12). we also perform the fusion using the combiner layer. Both of these approaches perform significantly worse than our proposed framework. This is because late fusion adds robustness to incorrect predictions in the metadata.
G.3. Efficiency Analysis
We now report the training and inference time and comparisons between oracle and MOGIC (OAK). Tables I3 and 14 summarize these numbers. As already note in Section 4 MOGIC is merely regularization framework, and we observe that inference times for given baseline disciple model and its MOGIC variant are identical. There is marginal increase in the training time due to computation of the additional regularization terms in the loss function.
24

---

MOGIC: Mctadata-infused Oracle Guidance for Improved Extreme Classification
Table 13: Training and inference time of MOGIC (OAK) and DistilBERT oracle on different datasets.
Dataset
Inference (in ms)
Training (in hrs)
Inference (in ms)
Training (in hrs)
MOGIC (OAK)
DistilBERT Oracle
LF-WikSSeeAloofitss220K
14.06
25
24.70
45
LL-Wik
13.72
41
26.79
69
131
13.66
25.21
Training: All models were trained on AMD 4xMI200 GPUs. MOGIC (OAK) utilizes a context length of 32 for short-text datasets and 256 for full-text datasets. The Oracle model was trained with context lengths of 128 and 256 for short and full-text datasets, respectively. The training time for the DistilBERT-haa oracle was approximately 2x that of MOGIC (OAK), primarily attributed the increased context length. This longer context length results larger attention matrices, thereby increasing the computational cost of backpropagation.
Inference: We evaluate the inference latency of MOGIC (OAK) CPU using 8 threads, processing single query with 2 metadata vectors within a PyTorch implementation. MOGIC (OAK) demonstrates significantly faster inference times
compared to the oracle model.
Table 14: Real world deployment latency of MOGIC (OAK) and its OAK encoder in milliseconds.
Algorithm
Mean
p95
p99
p999
OAK encoder
10.30 14.95
20.05
22.59
23.49
MOGIC (OAK)
22.34
25.07
27.08
To validate real-world feasibility, we further assess end-to-end production latency with inference on CPU, processing a single query with two metadata vectors. Through model optimizations, we significantly reduce the latency gap between the OAK encoder and MOGIC (OAK) inference, demonstrating the feasibility of deploying MOGIC (OAK) in real-world scenarios.
G.4. Generalization of MOGIC on Other Datasets Across Disciples
MOGIC can generalize across both disciples and datasets. To validate this, we now include results on training MOGIC with the DEXA and NGAME disciples on LF-Amazon-1331 dataset. Table summarrizzs these results, wherein we observe that MOGIC demonstrates performance gains across both disciples.
Table 15: MOGIC is general framework, and can be extended to any base XC algorithm to improve its accuracy. Along with WikiSee TitKss330K (Table , MOGIC also shows gains over other disciples on AF-AmazonTitles K31K. observe MOGIC can improve accuracy of the base algorithm by 1-2% in P@L
Models
P@I
P@5
N@s
PSP@1
PSP@5
MOGIC (OAK)
47.01
22.40
49,51
40.62
50.33
OAK
46.42
21.88
49.06
39.76
49.78
MOGIC (NGAME)
44,27
21.26
47.48
39.48
49.18
NGAME
43.44
21.16
47.10
39.00
49,00
MOGIC (DEXA)
45.43
21.70
48.49
39.91
49.95
DEXA
44.47
21.34
47.65
39.25
49.08
25

---

MOGIC: Metadata-infused Oracle Guidance for Improved Extreme Classification
G.S. Experimental Results on Full-text Datasets
Table 16: Results full-text benchmark datasets demonstrate that MOGIC achieves up 2% higher accuracy compared baseline methods, similar its performance on short-text datasets. Results on short-text datasets are presented in Table 2 For details on evaluation metrics, please refer to Appendix F4
Method
P@1
P@S
N@5
PSP@5
P@1
P@S
N@s
PSP@5
P@1
P@5
N@5
PSP@5
IF-WikiSecAlso 320K
LF Wikipedia- 5O0K
LF-Amazon- 131K
MOGIC (OAK)
49.62
24.26
50.49
43.17
85.34
51.50
77.85
61.74
50.05
23.72
52.87
53,80
OAK
48,57
23.28
49.16
40.44
85.23
50.79
77.26
60.80
48,36
22.20
51.27
52.21
DEXA
47.11
22.71
47.62
38.78
84.92
50.51
76,8
58.33
46,64
22.06
50.38
NGAME
46.4
18.05
46.64
33.33
84.01
49.97
75.97
57.04
46.53
22.02
49.58
50.45
ANCE
45.64
17.32
45.43
32.83
77.92
40.95
68.7
57.33
DEXMI.
85.78
50.53
77.11
58.97
GraphFormers
18.14
8.81
20.81
20.98
31.10
14.00
24.87
21.83
GraphSAGE
19.30
10.82
22.67
23.5
32.53
15.5
25.33
19.14
G.6. Comparisons with Pure LLM-based Approaches
Table compares MOGIC against LLaMA- and Phi-based oracles without disciple models, when LoRA-finctune for label generation (XC task). Generative LMs cannot be directly employed for the XC task, and therefore, for these experiments we earry out classification by comparing the embeddings associated with final token of the query and the labels from the last layer of the SLM. Results reported in Table 17 summarizes this experiment. From Table 17 we observe that the (65M sized) MOGIC disciple, trained with a BAsiiBERT/LLLMAA-AE
oracle outperforms the 7B-scale fine-tuned SLMs used directly without any disciple. We attribute this worse performance to the fact that these SLMs were pre-trained for text generation and then adapted for label generation. However, despite their performance, their embeddings, when used for regularization in the MOGIC framework, result in improved performance the disciple models.
To further investigate performance comparisons against language models, we also include comparisons against LLaMA, Phi, and GPT when used directly for label generation, with and without LoRA fine-tuning.
Table 17: Comparison of DistilBERT, an embedding-based method, with SLMs trained for text generation.
Model
Task Evaluation Method
P@1
P@5
PSP@1
PSP@5
DistiBBERT (ef. Table
3
Embedding based
47.63
22.75
36.71
41.45
LLaMA Metadata (LoRA) (ef. Table
Embedding based
34.20
16.21
30.46
31.93
Phi Metadata (LoRA) ) (ef. Table
Embedding based
33.32
15.48
29.75
30.61
LLaMA + Metadata (LoRA)
Generative
9.427
9.065
12.86
9.982
Phi Metadata (LoRA)
Generative
8.267
8.031
10,25
7.689
GPT + Metadata
Generative
14.57
12.26
16.86
12.92
G.7. Dependency on Metadata During MOGIC Training
As discussed Table g (cf. Section 4.3
disciples trained with an oracle are more robust to noisy/missing metadata in comparison to their oracle counterparts during test time. This robustness is not limited to noise at inference-time, but also noise present in the metadata during training time. To simulate training scenario where reliable metadata linkages are available, consider the following setting: We train MOGIC wherein 50% of the metadata is corrupted (i.e., replaced with randomly selected metadata from the corpus). The following table summarizes the linker, the oracle and disciple models' performance. We observe that MOGIC (OAK) is more robust to noisy metadata, even during training, and its drop performance is negligible. The MOGIC model trained with noisy metadata continues to be state of the art, outperforming the other baselines.
26

---

MOGIC: Metadata-infused Oracle Guidance for Improved Extreme Classification
Table 18: Linker metrics with 50% noise-added metadata, where 50% metadata for each query is replaced by random metadata.
Method
P@1
P@5
N@s
PSP@1
PSP@5
P@1
P@S
N@5
PSP@1
PSP@5
LF- LF-WikkkScAlssoTTitss-2200
LF- LL-WWikTTillo-5000K
Ground-truth metadata
46.16
20.97
36.48
28.75
25.37
20.74
13.72
15.90
7.38
8.29
Noisy metadata
27.98
9.90
20.82
21.62
14.10
10.06
4.99
6.44
7.12
5.48
Table 19: Oracle metrics with 50%-noisy metadata
Method
P@1
P@5
N@5
PSP@1
PSP@5
P@1
P@5
N@s
PSP@1
PSP@5
I.F- J.--WiiiSeeelllo/tee3322
L.F-WikTitlcs-5006
Ground-truth Oracle
47.48
22.64
48.18
36.60
41.27
64.32
29.92
50.58
37.41
39.75
Noisy Oracle
41.23
19.92
42.17
30.94
35.77
59.85
26.98
46.47
34.58
36.00
Table 20: Performance of MOGIC (OAK) with 50%-noisy metadata during training
Method
P@1 1
P@S
N@5 PSP@1 PSP@5
P@1
P@S
N@s
PSP@1 PSP@5
LF Liss 320K
LF WikiTiddes-500 500K
MOGIC (OAK) (ef. Table 2
34.62
17.93
35.70
27.44
33.18
47.28
18.55
34.97
27.29
26.12
MOGIC (OAK) with 50%-noisy noisy metadata
34.29
17.68
35.37
27.89
33.07
46.68
18.53
34.83
27.44
26.27
G.8. Experimental Results on LE-AAaazonTitles-1..
We present the results of MOGIC (OAK) on the LF-AmazonTitles 1.3M dataset, which is an extremely large-scale extreme classification (XC) benchmark containing approximately 1.3 million labels.
Table 21: Results on JF-AmazonTitles 1.3M benchmark datasets
Method
Metadata
P@1
P@5
N@5
PSP@1
PSP@3
PSP@5
MOGIC (OAK)
Dump Category
48.93
38.49
46.45
35.78
38.92
40.59
OAK
Dump Category
48.91
38.13
46.07
34.65
37.53
39.04
MOGIC (OAK)
GPT Category
50.95
39.95
48.19
36.28
39.50
41.22
OAK
GPT Category
49.46
38.61
46.62
34.92
37.86
39.41
We constructed metadata for the L.F-AmazzoTTitles1..3M dataset using two different approaches:
1. Product-dump parsing: We parsed the Amazon product data dump to extract auxiliary information related to each product. Among the various fields available, we selected the product category as the metadata to be used in our experiments.
2. GPT-generated categories: We used GPT-4o-mini to generate category labels for each product, which were then used as metadata.
Empirically, we observed that the GPT-generated categories yielded the most significant performance improvements. However, the generated categories were not always consistent. To ensure compatibility with algorithm, performed conflation step to normalize and standardize these categories before use. For the conflation, we identified all tail entities (having fewer than 5 samples per entity) and replaced these labels with their closest semantic match from the entire entity set. A threshold was applied before the conflation to ensure that no two unrelated categories are merged together. This was done to ensure that the erroncous entities generated due to stochastic nature of language models are merged together with a more general entity label.
27

---

MOGIC: Metadata-infused Oracle Guidance for Improved Extreme Classification
H. Case Study: Effect of Metadata in MOGIC (OAK)
In our framework, metadata provided as early concatenation textual form to the oracle, which is then used to regularize the free parameters of the disciple's memory during training. While OAK suffers from overgeneralization and semantic drift-often selecting popular or tangentially related labels- (OAK) leverages metadata to guide its predictions toward domain-specife and contextually appropriate labels. Metadata serves three main functions:
Disambiguation: helping resolve terms with multiple meanings.
Semantic grounding: encouraging consistency in domain or region.
Error pruning: reducing the model's tendency to retrieve globally frequent but contextually irrelevant entities.
Let us consider the two queries presented in Table from LF- Ais where the task is, given the title of Wikipedia article, predict related Wikipedia articles listed in its See Also section. In the following, examine how metadata helps MOGIC rectify errors from OAK.
1. Grass court
For the query "Grass court", the ground truth labels include specific types of tennis courts (Clay court, Carpet court, Hardcourt), which are all surface types in the sport of tennis. The predicted metadata included Courts by type, Landforms and Grasslands. Out of which Courts by type is the only relevant metadata which links to the ground-truth labels.
OAK predictions are completely off-topic, retrieving labels such as Fernie Ghostriders (a Canadian ice hockey team), Garland, Texas (a geographic location), and Ronald Reagan Boyhood Home (a historical site). These results reflect semantic drift: the model picks up on vague or shallow lexical or associative signals related to "court" but fails to situate it in the correct sports domain.
MOGIC (OAK), contrast, correctly predicts all three ground truth labels and only makes minor errors (e.g., Video arcade, U.S. Men's Clay Court Championships). This improvement comes from the regularization-based training of MOGIC which helps the disciple focus the right metadata using the guidance received from the oracle model during training.
This example highlights how metadata helps disambiguate multi-sent sense terms (e.g., "court") and offers semantic grounding by restricting the model's attention to the correct conceptual subspace.
2. Tangbe
The query "Tangbe" refers to a village in the Mustang District of Nepal. The ground truth labels reflect this geographical and cultural context: Mustang District, Kali Gandaki Gorge, Upper Mustang, etc. The predicted metadata includes Populated places in Cameroon, Communes of Cameroon and Township-level divisions of Hebei which are all irrelevant.
OAK predictions include entities like Desalpur (an archaeological site India), Vladivostok (a city in Russia), and Kitenge (a textile), which are geographically and topically irrelevant. The model appears to guessing from broad set of vaguely place-related or historical entities without any clear regional grounding, which is mostly likely due to error propagation from the linker.
In contrast, MOGIC (OAK) correctly retrieves all five ground truth labels, suggesting precise understanding of the regional context. This highlights the error pruning capability of MOGIC.
28


Reviews:
[{"review_id": "4t426KYjPG", "reviewer": "Reviewer_5PFN", "summary": "The authors propose a framework for building a disciple model which can perform extreme multi-label classification with the assistance of RAG-like metadata. This pipeline, MOGIC, is two-phase: in phase (1) an oracle with access to high-quality, ground-truth metadata is trained. In phase (2), a smaller, \"disciple\" model participates in knowledge \"distillation-like\" training to mimic the predictions of the oracle model but also make predictions on relevant metadata which might help with the downstream prediction. Through MOGIC, the disciple model is both robust to noisy metadata and performant on XML tasks such as WikiTitles-500K and AmazonTitles-131K.\n\n## Update after rebuttal\n\nI am choosing to maintain my positive score in light of the rebuttal.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed review. Please find our response to your comments below.\n\n1. **Clarity on Rademacher constants**\n* **Response**: The Rademacher complexity constants $R_q$ and $R_l$ in Theorem 1 are scalar values which quantify the complexity or capacity of the hypothesis classes corresponding to the query tower and label tower, respectively. We use the standard definitions of Rademacher constants from the Statistical Learning Theory for Binary Classification problems (ref: https://www.cs.cmu.edu/~ninamf/ML11/lect1117.pdf).\n* *Explanation on Rademacher complexity:* Mathematically, Rademacher complexity constants $R_q,R_l$ are estimated as the average empirical loss of minimizing the hypothesis class on a data sample with randomly annotated labels i.e. labels are generated by a purely random Bernoulli distribution with probabilities 0.5 to be either positive or negative. In intuitive terms, the smaller the values of $R_q, R_l$, the less prone are the query tower and label tower to overfit the finite training data, and consequently the accuracy on the test set is expected to be better.\n\n2. **Novelty of  disciple training and clarity of oracle architecture**\n* **Response**: The disciple training in MOGIC is a novel variant of knowledge distillation (KD). Compared to standard KD, it differs in multiple ways. First, the metadata is provided as early concatenation in textual form for the oracle, but is used to train free parameters in the disciple's memory via a novel regularization framework. This can be viewed as KD from an early-concatenation model to a two-tower model. While all disciple models benefit from this framework, the additional parameters present in the memory-based disciples such as MOGIC (OAK) demonstrate the most gains. Furthermore, the oracle has access to ground-truth metadata, which is privileged information that is not available to the disciple model (unlike standard KD). In this MOGIC framework, the oracle can either be larger, or of the same size as the disciple. Table 3 of the main manuscript presents comparisons between larger (LLM-based, 2.7B/7B sized) oracle and a disciple-sized (65M sized), DistilBERT oracle (cf. Lines 367-380, column 1).\n\n3. **Clarifying XML Labels and Data Scarcity**\n\n* **Response**: We will update the introduction of the manuscript accordingly. We will also include a detailed example, as suggested by you, in the appendix in the revised draft.\n\n4. **Testing on Large-Scale XML Datasets like WikiLSHTC-325K or Amazon-670K**\n\n* **Response**: Datasets such as WikiLSHTC-325K (which contains 325K labels) do not contain raw text or label features associated with labels, and therefore have not been directly used for training encoder-based models such as NGAME, DEXA, DEXML, ANCE etc. This is why we chose to report on LF-WikiTitles-500k and LF-Wikipedia-500K dataset (which has 501K labels), which are from the same distribution and have the same task as WikiLSHTC (category prediction), but contain a larger label set, with label features/text available. Similarly, to be aligned with the baseline methods, we use LF-Amazon-131k dataset instead of Amazon-670K. To demonstrate MOGIC's effectiveness on larger-scale datasets, we are working to include experiments on the LF-AmazonTitles-1.3M dataset in the final version of the manuscript. While the results are not yet available due to the resource-intensive process of generating metadata and training models, we aim to include them before the conclusion of the rebuttal phase.\n\n5. **Comparison against SLEEC.**\n\n* **Response**: SLEEC does not use label-text for classification, and therefore performs poorer than methods such as Parabel [1] on multiple standard datasets. A direct comparison against SLEEC is challenging, as we were unable to find an up-to-date implementation of the algorithm. Therefore, we choose to compare MOGIC against Parabel, which has been shown to outperform SLEEC across datasets[1]. We present these comparisons on LF-AmazonTitles-131K, LF-WikiSeeAlsoTitles-320K and LF-WikiTitles-500K. These results are reported in the table below, with performance numbers for Parabel obtained from ECLARE[2]:\n\n* *Performance of MOGIC(OAK) and Parabel on different benchmark datasets*\n\n|\t|   P@1 |   P@5  | PSP@1 | PSP@5 |\n|--:|--:|--:|--:|--:|\n|  **LF-AmazonTitles-131K** |   |\t|\t|   |\n| MOGIC(OAK)  | 47.01  | 22.40  | 40.62 | 50.33 |\n| Parabel | 32.6 | 15.61 | 23.27 | 32.14 |\n|  **LF-WikiSeeAlsoTitles-320K** |   |\t|\t|   |\n| MOGIC (OAK) | 34.62 | 17.93 | 27.44 | 33.18 |\n| Parabel | 17.68 |  8.59 | 9.24 | 11.8 |\n|  **LF-WikiTitles-320K** |   |\t|\t|   |\n| MOGIC (OAK) | 47.28 | 18.55 | 27.29 | 26.12 |\n| Parabel | 40.41 |  15.42 | 15.55 |  15.35 |\n\n\n[1] Prabhu, Yashoteja, et al. \"Parabel: Partitioned label trees for extreme classification with application to dynamic search advertising.\" World Wide Web Conference 2018.\n\n[2] Mittal, Anshul, et al. \"ECLARE: Extreme classification with label graph correlations.\" theWebConf  2021.", "claims_and_evidence": "The results, especially the P@1 numbers, indicate the effectiveness of the approach. It is to my understanding that through this strategy, lightweight architectures such as DistilBERT can perform on par with Phi-2 and Llama-2. It is clear through the experiments section that usage of high-quality textual metadata is capable of training a high-quality disciple model.", "methods_and_evaluation_criteria": "I do believe that the proposed method makes sense for the problem, especially in the age of LLMs. Classical XML architectures and papers from 2016-2024 did not extensively consider the addition of metadata for help with classification, but given the typically lackluster results on popular XML datasets (due to their extreme difficulty), calling RAG-like methodology is intuitive. The datasets are appropriate, though I would have liked to have seen performance on even more challenging datasets such as Amazon-670k (see questions below).", "theoretical_claims": "I did not check the correctness of the proof, though the bound makes sense as it becomes tighter with more metadata and samples. The authors should probably explain what the Rademacher constants are for those readers less familiar with PAC theory (are these just Rademacher splitting dimensions?). \n.", "experimental_designs_or_analyses": "The experimental designs appear to be fair and routine. The datasets are standard and frequently-encountered in XML literature. The augmentation of the XML samples and labels with metadata is new to me, but the construction of the metadata makes sense.", "relation_to_broader_scientific_literature": "Extreme multi-label classification is applicable to the broader recommender systems community. In my opinion, XML is still generally unsolved and challenging, so the contribution of new algorithms which pushes the P@1 on these datasets is meaningful and useful to the broader machine learning community.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Other Strengths:\n-I find the contribution useful as it re-frames the XML problem within the landscape of LLMs and RAG. It's a fresh perspective.\n\nOther Weaknesses:\n-The disciple training doesn't seem entirely novel. It resembles knowledge-distillation, except it's not immediately clear if the disciple architecture is smaller or simply the same as the oracle.", "other_comments_or_suggestions": "The authors could try to present a simple example of the features and labels for an XML problem. It likely won't be immediately clear to a reader outside of this field that the label is an enormously-sized, highly sparse binary vector corresponding to classes -- this is only briefly covered. The authors should also try to further emphasize hat these problems are difficult because there are very few samples per label.", "questions_for_authors": "Is it possible to run tests on extremely challenging datasets such as WikiLSHTC-325K and Amazon-670K? I would be interested in seeing how metadata is constructed for these datasets and how MOGIC holds up. If not, could the authors explain why these datasets are not suitable for the described framework? I am also interested in seeing if a gold-standard method such as SLEEC can be defeated via MOGIC.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WXWwMtempP", "reviewer": "Reviewer_FoCK", "summary": "The paper introduces MOGIC, a framework for improving extreme classification (XC) by leveraging metadata through a two-phase training approach. XC involves tasks with extremely large label spaces (e.g., product recommendations, Wikipedia tagging) where metadata can enhance accuracy but faces challenges like noise and latency. Existing methods use late-stage fusion for efficiency but underperform when metadata is clean. MOGIC trains an early-fusion Oracle model with access to ground-truth metadata to guide a disciple model (e.g., OAK, NGAME) via regularization. This approach improves precision and robustness without increasing inference latency, achieving 1–2% gains on six datasets.\n\n## update after rebuttal\nThanks for the responses and extra sensitivity analysis which partially addresses my questions. I will keep my positive rating of 3 considering the overall quality of this paper.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed review. Below are our responses to your comments.\n\n1. **Comparisons with LLM-based approaches**\n* **Response**: We have already included comparisons of MOGIC against LLaMA and Phi-based Oracles without disciple models, when LoRA-finetuned for label generation (XC task) in Tab 3 (and row 1-3 below). Generative LMs cannot be directly employed for the XC task, and therefore, for these experiments we carry out classification by comparing the embeddings associated with final token of the query and the labels from the last layer of the SLM. Results reported in Tab. 3 of the paper summarizes this experiment (cf. Lines L367-380, col 1).  We observe that the (65M sized) MOGIC disciple, trained with a DistilBERT/LLaMA-based oracle outperforms the 7B-scale fine-tuned SLMs used directly without any disciple. We attribute this worse performance to the fact that these SLMs were pre-trained for text generation and then adapted for label generation. However despite their performance, their embeddings, when used for regularization in the MOGIC framework, result in improved performance of the disciple models. We will improve the clarity of this discussion.\n\n*  To further investigate performance comparisons against language models, as part of the rebuttal, we also include comparisons against LLaMA, Phi, and GPT when used directly for label generation (row 4-6). We observe that oracle models trained specifically for the XC task perform better than much larger LMs. Also, for the XC task, using the embedding from LMs is better than mapping their generations to the label set since the generations often contain labels which do not belong to the label set.\n\n*Oracle metrics on LF-WikiSeeAlsoTitles-320K*\n||P@1|P@5|\n|-|-|-|\n|DistilBERT (65M) (cf.Table3)|47.63|22.75|\n|LLaMA-2 (7B) (LoRA) -Embed (cf.Table3)|34.20|16.21|\n|Phi-2 (2.7B) (LoRA) -Embed (cf.Table3)|33.32|15.48|\n|LLaMA-2 (7B) (LoRA) Gen|9.427|9.065|\n|Phi-2 (2.7B) (LoRA) Gen|8.267|8.031|\n|GPT 4-o Gen|14.57|12.26|\n\n2. **Metadata dependency during training**\n* **Response**: As discussed in Tab. 8 (cf. Sec 4.3), disciples trained with an Oracle are more robust to noisy/missing metadata in comparison to their Oracle counterparts during inference. This robustness is not limited to noise at inference, but also noise present in the metadata during training. To simulate a training scenario wherein reliable metadata linkages are not available, we consider the following setting: We train MOGIC on LF-WikiSeeAlsoTitles-320K dataset wherein 50% of the metadata is replaced with a randomly selected metadata from the corpus. The following table summarizes the linker, the Oracle and disciple models' performance. We observe that MOGIC (OAK) is more robust to noisy metadata, even during training with negligible drop in performance. The MOGIC model trained with noisy metadata, continues to be SOTA, outperforming other baselines.\n\n*Linker metrics with 50% noise-added metadata, where 50 % metadata for each query is replaced by random metadata.*\n||P@1|P@5|N@5|PSP@1|PSP@5|\n|-|-|-|-|-|-|\n|Ground-truth metadata|46.16|20.97|36.48|28.75|25.37|\n|Noisy metadata|27.98|9.90|20.82|21.62|14.10|\n\n*Oracle metrics with 50%noisy metadata*\n||P@1|P@5|N@5|PSP@1|PSP@5|\n|-|-|-|-|-|-|\n|Ground truth Oracle|47.48|22.64|48.18|36.60|41.27|\n|Noisy Oracle|41.23|19.92|42.17|30.94|35.77|\n\n*Performance of MOGIC (OAK) and OAK with 50%noisy metadata during training*\n||P@1|P@5|N@5|PSP@1|PSP@5|\n|-|-|-|-|-|-|\n|MOGIC(OAK) (cf.Table2)|34.62|17.93|35.70|27.44|33.18|\n|MOGIC(OAK) with 50%noisy metadata|34.29|17.68|35.37|27.89|33.07|\n\n3. **Sensitivity of Loss Balancing Hyperparameters**\n* **Response**: We now include a sensitivity analysis over the $𝛼$ and $β$ hyperparameters, considering all combinations of $𝛼\\in\\{0.1,1,10\\}$ and $β\\in\\{0.1,1,10\\}$. We performed a sensitivity analysis prior to our experimentation and found that $(𝛼,β) = (1,0.1)$ leads to the best performance, which are the values we use for the experiments in the paper. We now present the ablations on the choice of $𝛼$ and $β$ on the LF-WikiSeeAlsoTitles-320K dataset, with all other hyperparameters as defined in Appendix H. This table summarizes the results. We observe that $(𝛼,β) = (1,0.1)$ results in the best performance. However, the performance of MOGIC is generally robust to the choice of $(𝛼,β)$ with a variance of 0.108 in P@1, when comparing across all the choices considered. We will improve the clarity of this hyperparameter choice in the final version of the manuscript, and include this ablation in the Appendix.\n\n|$𝛼$|$β$|P@1|P@5|N@5|PSP@1|PSP@5|\n|-|-|-|-|-|-|-|\n|0.1|0.1|34.23|17.75|35.40|27.05|32.67|\n|0.1|1.0|34.34|17.74|35.45|27.07|32.63|\n|0.1|10.0|33.70|17.47|34.83|26.23|32.07|\n|1.0|0.1|34.62|17.93|35.70|27.44|33.18|\n|1.0|1.0|34.56|17.91|35.66|27.32|33.12|\n|1.0|10.0|34.05|17.61|35.05|26.55|32.42|\n|10.0|0.1|34.16|17.57|35.04|26.84|32.39|\n|10.0|1.0|34.11|17.56|35.00|26.72|32.32|\n|10.0|10.0|33.61|17.30|34.47|25.89|31.66|", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "1. The method leverages both query-side and label-side metadata, enriching representations bidirectionally. Examples show improved label prediction by incorporating contextual metadata from both ends.  \n2. MOGIC demonstrates consistent performance gains across multiple datasets and metrics. It improves precision@1, NDCG, and propensity-scored metrics over state-of-the-art models like OAK. The method is validated on six benchmarks, showing broad applicability.", "theoretical_claims": "There are some theoretical proofs which seem to be correct. One typo is that, the symbol k in Inequality (11) should be K.", "experimental_designs_or_analyses": "Six datasets are used for the experiments. Three different LLMs are tested and many baselines are compared. I think the experimental evaluation is quite comprehensive.", "relation_to_broader_scientific_literature": "This work is related to extreme classification. And it is also related to RAG in LLMs.", "essential_references_not_discussed": "Nil.", "other_strengths_and_weaknesses": "1. The approach is model-agnostic, enhancing both memory-based (OAK) and memory-free (NGAME, DEXA) XC models. Plug-and-play compatibility allows integration into existing architectures. Flexibility in Oracle choice (DistilBERT, Phi-2) balances performance and efficiency.  \n2. Theoretical analysis justifies the regularization losses, linking disciple performance to Oracle-guided training. Bounds on population loss show the disciple converges toward Oracle accuracy with finite samples.  \n3. MOGIC maintains low inference latency by avoiding early-fusion overhead. Training costs are manageable, and inference matches base models’ speed. Experiments confirm no latency increase compared to OAK.  \n4. Robustness to missing or noisy metadata is demonstrated through quantile-wise analysis and noise injection tests. The disciple outperforms the Oracle when metadata is perturbed, showing resilience to real-world conditions.", "other_comments_or_suggestions": "Comparisons with pure LLM-based approaches (without disciple models) are missing. Larger Oracles might outperform MOGIC if computational constraints are relaxed, but this trade-off is not quantified.", "questions_for_authors": "1. The Oracle’s dependency on high-quality metadata during training limits performance if metadata is sparse or biased. While MOGIC handles inference-time noise, training assumes reliable metadata linkages, which may not hold in all scenarios.  \n2. Hyperparameters α and β for loss balancing are set empirically. The paper does not analyze sensitivity to these choices, risking suboptimal tuning in new applications.  \n3. The framework may overfit Oracle biases, particularly if the Oracle’s metadata integration is noisy. Ablation studies show performance drops without metadata, hinting at potential over-reliance.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "Nil."}}, {"review_id": "naPOmGAzxj", "reviewer": "Reviewer_3zWG", "summary": "This paper mainly explores methods to enhance classification performance using metadata in the task of Extreme Classification. Experiments on six popular benchmark datasets show that the method significantly improves the model performance.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their detailed review. Please find our response to your comments below.\n\n1. **The two-phase training process involving Oracle training and Oracle-guided disciple training might introduce additional complexity.**\n* **Response**: We agree with the reviewer that the two-stage training framework introduces some additional complexity compared to single-stage baselines. However, the improvements in performance and the increased robustness achieved through regularization in this two-stage approach present a reasonable tradeoff for the added complexity. Moreover, this complexity is limited to the training phase (a one-time cost), while the deployed disciple models remain highly efficient at inference.\n\n2. **Although it shows good performance on the tested datasets, its scalability to extremely large-scale or rapidly evolving datasets remains to be seen.**\n\n* **Response**: The MOGIC approach can be readily scaled to larger datasets. However, there are no high-quality rapidly-evolving dataset for XC, to the best of our understanding. We believe this is a strong area for future research. The Wikipedia-500k is one of the largest datasets used by the XC community, comprising 1.8M training samples and 501K labels, while many other XC datasets such as EURLex-4.3K, Bibtex or AmazonCat-13K have significantly fewer labels (such as 4.3K in the EURLex-4.3K dataset or 13K in the AmazonCat-13K dataset). While LF-AmazonTitles-1.3M is another viable choice, we reported results on Amazon-131k, since metadata in this setting is not readily available, and must be generated using GPT-based approaches. In light of your comments, we are also working to include experimentation on the LF-AmazonTitles-1.3M dataset. While we will include these results in the final version of the manuscript, given the additional resources involved in generating the metadata and subsequently training the models, we do not have the results ready to report at this time, but are striving to incorporate them before the end of the rebuttal discussion phase.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "Yes, in Theoretical Justification of Oracle-Guided Losses", "experimental_designs_or_analyses": "Yes", "relation_to_broader_scientific_literature": "This paper makes solid contributions to the area of Extreme Classification.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "1. The proposed MOGIC framework is innovative as it combines early-fusion of text-based metadata and late-fusion of memory items. \n2. The two-phase training process involving Oracle training and Oracle-guided disciple training might introduce additional complexity.\n3. Although it shows good performance on the tested datasets, its scalability to extremely large-scale or rapidly evolving datasets remains to be seen.", "other_comments_or_suggestions": "See weakness above", "questions_for_authors": "See weakness above", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper introduces MOGIC, a method for improving Extreme Classification (XC). It is trains an "teacher" model with privileged, high-quality metadata and then distills into a student XC model with regularization. Experiments were conducted on standard XC benchmarks such as LF-AmazonTitles and LF-WikiTitles and showed performance gains in their setting without increasing inference latency. All the reviewers were positive about the paper and still raised questions about comparisons with purely LLM-based methods, robustness to unreliable metadata during training, hyperparameter sensitivity, and the method's performance on larger-scale datasets. The authors provided additional results showing robustness and addressing reviewers' initial concerns about scalability and practicality. Reviewers acknowledged the responses positively, maintaining their support for the paper based on clarified and supplemented points. Please incorporate these new results and reviewer feedback in the final version.
Decision: Accept (poster)