Rejuvenating Cross-Entropy Loss in Knowledge Distillation for Recommender Systems
Authors: Zhangchi Zhu
Wei Zhang
Venue: ICLR 2026 Poster (2026)
Keywords: Knowledge Distillation
Recommender System
Model Compression
Cross-Entropy
Primary area: other topics in machine learning (i.e., none of the above)

Abstract: This paper analyzes Cross-Entropy (CE) loss in knowledge distillation (KD) for recommender systems. KD for recommender systems targets at distilling rankings, especially among items most likely to be preferred, and can only be computed on a small subset of items. Considering these features, we reveal the connection between CE loss and NDCG in the field of KD. We prove that when performing KD on an item subset, minimizing CE loss maximizes the lower bound of NDCG, only if an assumption of closure is satisfied. It requires that the item subset consists of the student's top items. However, this contradicts our goal of distilling rankings of the teacher's top items. We empirically demonstrate the vast gap between these two kinds of top items. To bridge the gap between our goal and theoretical support, we propose **R**ejuvenated **C**ross-**E**ntropy for **K**nowledge **D**istillation (RCE-KD). It splits the top items given by the teacher into two subsets based on whether they are highly ranked by the student. For the subset that defies the condition, a sampling strategy is devised to use teacher-student collaboration to approximate our assumption of closure. We also combine the losses on the two subsets adaptively. Extensive experiments demonstrate the effectiveness of our method. Our code is available at https://github.com/BDML-lab/RCE-KD.

Published as a conference paper at ICLR 2026
REJUVENATING CROSS-ENTROP! Loss IN KNOWL- EDGE DISTILLATION FOR RECOMMENDER SYSTEMS
Zhangchi Zhu', Wei Zhang 1.-
East China Normal University SShnggaai Innovation Institute zczhul stu ecnu. edu. en, zhangwei. twwtt1llueeee 00100mma1mmmmm com
ABSTRACT
This paper analyzes Cross-Entropy (CE) loss in knowledge distillation (KD) for recommender systems. KD for recommender systems targets at distilling rankings, especially among items most likely to be preferred, and can only be computed small subset of items. Considering these features, we reveal the connection between CE loss and NDCG in the field of KD. We prove that when performing KD on an item subset, minimizing CE loss maximizes the lower bound of NDCG, only if an assumption of closure is satisfied. It requires that the item subset consists of the student's top items. However, this contradicts our goal of distilling rankings of the teacher's top items. We empirically demonstrate the vast gap between these two kinds of top items. To bridge the gap between our goal and theoretical support, we propose Rejuvenated Cross-Entres It splits the top items given by the teacher into two subsets based on whether they are highly ranked by the student. For the subset that defies the condition, a sampling strategy is devised to use teacher-student collaboration to approximate our assumption of closure. We also combine the losses the two subsets adaptively. Extensive experiments demonstrate the effectiveness of our method. Our code www./D/.m///////
for Knowledge Distillation (RCE-KD) KD).
available /tttps://github com BDML- 1atilll-aaRRRE-E- RCE-KD
INTRODUCTION
Recently, as the scaling law in recommender systems (Zhai et al., 2024) has been increasingly recognized, many researchers have proposed extremely large models (Ohsaka & Togashi 2023 Zhai et al. 2024) to achieve higher recommendation accuracy. However, the increase model size inevitably incurs high storage costs and inference latency, causing higher maintenance costs and lower user satisfaction.
To improve the inference efficiency and decrease the storage cost of recommendation models with- out sacrificing their recommendation accuracy, knowledge distillation (KD) for recommender sys- tems (Kang et al. 2020) Sunn et al.) 2024) has attracted attention. KD (Hinton al., 2015) is an approach for model compression. It aims to transfer knowledge from a pre-trained large teacher to small student. Once training is complete, only the small student is used for inference. Among existing works on KD, response- based KD (Hinton et al. (015) encourages students to mimic the teacher's s predictions and has gained extreme attention due to its excellent performance. As a popular loss for response-based KD methods, Cross-Entropy (CE) loss is very important. Most response- response-based KD methods (Huang et al. 2022 Cui et al., 2023) in Computer Vision (CV) and Natural Language Processing (NLP) are based CE loss. However, little work has been done to use analyze CE loss in KD for recommender systems. Note that KD for recommender systems has two unique features: 1) It focuses more on rankings than specific scores, especially among the teacher's top items (Kang jt al. 2020). 2) KD can only be conducted on small subset items since the quantity of all the items is very large. These features make the compatibility of CE loss in KD for recommender systems questionable. To obtain an initial insight into the performance of CE loss, we present the results of vanilla CE loss and several response-baad based KD methods in Figure To cover as many types of loss functions as possible, we consider the wisn loss (i.e., CD (Lee et al. 2019)), wisr loss (i.e., UnKD (Chen et al. 2023)), and RRD-based losses (Kang et al. 2020) (a list-wise loss, i.e..
"Corresponding author.

---

Published as a conference paper at ICLR 2026
V 1
CD
UnKD
RRD
HetComp
ANN vanilla CE
X
RCE-KD
0.12
0.0400
NDCG@20
0.0375
0.11
0.0350
0.10
0.0325
MF
MF
LGCN LGCN
HS
HSTU
0.0300
MP
MF
LGCN LGCN Yelp
HSTU HSTU
Gowalla
Figure I: Performance comparison of different KD methods. We report the results in three homoge- neous Teacher- - Student settings.
RRD (Kang et al. 2020). HetComp (Kang et al. 2023 ). and our method RCE-KD. In vanilla CE loss, we compute CE loss using the teacher's 's top items. We find that vanilla CE loss is often inferior to all baselines. This result contrasts with the extensive use of CE loss for KD in other fields.
Considering the features of recommender systems and the surprisingly poor performance of CE Loss, we analyze CE loss in KD for recommender systems. Firstly, we extend the connection between CE loss and NDCG to full-item KD, where CE loss computed using all items. We theoretically prove that minimizing CE loss maximizes the lower bound of NDCG with the relevance scores proportional to the teacher's predicted scores. This suggests strong motivation for using CE loss in KD.
However, full-item KD is not practical due to the extremely large number of items. In real-woo world scenarios, CE loss could only be computed on a subset of items (i.e., the partial-item KD), such as the teacher's is top items in vanilla CE. For this case, we define partial NDCG, which only considers rankings within subset of items. Then, we prove that CE loss bounds partial NDCG. However, holds only if the item subset satisfies our assumption of closure (Assumption 4.3) It requires that all items that the student ranks higher than any item in the subset are also in the subset. This assumption emphasizes the effect of the student's 's top items. Recall that goal is to distill rankings among the teacher's ' top items (Reddi et al., (021). Unfortunately, we observe that the top items given by the teacher are usually ranked low by the student. This makes it difficult for the teacher's top items to satisfy our assumption of closure. Thus, vanilla CE cannot bound partial NDCG and performs poorly,
To fully unleash the potential of CE loss by re-establishing its connection with NDCG, propose Rejuvenated Cross-Entropy for Knowledge Distillation (RCE-KD), which consists of four key points: It divides the teacher's top items into two subsets: the subset that consists of items also ranked high by the student and the one that consists of the rest of the items. 2) For the first subset, we distill rankings among these items by using CE loss directly on the student's top items. 3) For the second subset, we design a sampling strategy to sample from the student's 's top items and compute CE loss on a new item set that approximately satisfies Assumption 4.3 4) The fusion weights of the losses on these two subsets are adaptively updated based on their size. With the above improvements, we can nearly completely unleash the potential of CE loss while ensuring high training efficiency.
To sum up, the key contributions of our work are follows:
We theoretically extend the connection between CE and NDCG to the field of KD for recommender systems in real scenarios, where KD is performed on an item subset. Specifically, we first define partial NDCG, which measures ranking ability on a subset of items. Then, we prove that minimizing CE loss given item subset maximizes the partial NDCG on it. We also give the critical assumption made on the item subset for the conclusion to hold.
Based on the analysis, propose RCE-KD unleash the potential of CE loss fully while ensuring high training efficiency. It splits the top items of teachers and calculates the loss separately. A dynamic weighting method is devised to adaptively fuse the losses all subsets.
Extensive experiments are conducted on three public datasets and both homogeneous and heteroge- neous KID settings to demonstrate the superiority of the proposed approach.
2

---

Published as a conference paper at ICLR 2026
2 RELATED WORK
2.1 KNOWLEDGE DISTILLATION FOR RECOMMENDER SYSTEMS
Existing KD methods fall into three categories: response-based, feature-based, and relation-based.
Response-based methods focus on teachers' predictions. CD (Lee et al. 2019) samples unobserved items from distribution associated with their rankings predicted by students, and distills with a point- wise loss. RankDistil (Reddi et al. [202]) enables students to mimic teachers by sampling high-ranking items predicted by teachers and calculating multiple forms of loss functions on them. RRD (Kang 2020) adopts list-wise loss to maximize the likelihood of the teacher's ' recommendation list. Note that RRD could be regarded as the extension ListMLE loss (Xia al. 2008 to the top-K setting (Xia et al.) 2009). Based on RRD, DCD (Lee & Kim 202T) uses the discrepancy between the teacher and student model predictions to decide which ledgl to distill. HetComp (Kang et al.)
ct al.
2023) transfers the ensemble knowledge of heterogeneous teachers by constructing easy-to-hard
knowledge sequences from the teachers' trajectories.
Feature-based methods focus on the intermediate representations of the teacher. FreqD (Zhu & Zhang 2024) defines knowledge as different frequency components of the features and proposes
emphasizing important knowledge by graph filtering. PCKD (Zhu & Zhang 2025) observes that projectors in caature-based KD interrupt user preference contained in the features and designs two regularization terms restrict the projectors.
Relation-based methods focus on the relationships between different items. HTD (Kang et al. 2021 distills the sample relation hierarchically alleviate the capacity gap between the student and teacher.
Our work compensates for the lack of theoretical analysis of CE loss in response- methods. Based theoretical analysis, we design aplit-and-fusion paradigm with novel sampling strategy and adaptive loss fusion mechanism to enhance vanilla CE loss, thereby unlocking its full potential.
2.2 CONNECTION BETWERN CE Loss AND NDCG
Recently, many studies CCoo et al.
2007
Ravikumar et al.,
2011 Bruch et al. 2019 Wu et al.
2024 Yang al. 2024 Xu et al. 2024b) learning-to-rank (LTR) have focused the impact of different surrogate loss functions on NDCG. Among them, CE loss is of particular interest due to
its wide range of applications. As pioneer, ListNet (Cao et al. 2007) introduces CE loss into LTR by defining the top-one probability. Then, (Bruch et al. 2019) for the first time proves that CE loss bound on NDCG when considering binary ground- truth labels. Subsequently, work has been done to improve CE loss based on this conclusion. For example, PSL (Yang et al. 2024) changes the surrogate activations, and SCE (Xu et al., 2024b) increases the weight of negative samples in CE loss to achieve tighter bound of NDCG. Another work relevant to us is (Wu et al. 2024). It reveals the pros and cons of sampled CE loss for item recommendation and also relates it to NDCG. However, these methods mentioned above hardly address the case of non-binary ground-truth labels. Moreover, they either do not focus on the scenarios that need item sampling or simply use uniform sampling without making any assumptions about the items being sampled. This makes them entirely inapplicable for KD, where we take the teacher's 's predictions as labels and emphasize the top-ranked items.
3 PRELIMINARY
3.1 TOP-N RECOMMENDATION
This work focuses on the top-N recommendation with implicit feedback. Let LA and I denote the user and item sets, respectively. Then, [ A recommendation model scores the items not interacted with by the user and recommends N items with the largest scores. We use Tui denote the score of interaction (u, i) predicted by the recommendation model and use Tu E [ to denote the predicted scores of all items for user In this paper, we use superscripts and T to denote the student and the teacher, respectively. In the following sections, we default our analysis any u. l4 if not specified.
and [I] are the number of users and items, respectively.
3

---

Published as a conference paper at ICLR 2026
3.2
LOSS FOR KNOWLEDGE DISTILLATION
Given an item set Ju for each user it E Ut CE loss in KD for recommender systems is computed as:
LCE = [ue Σ Σ
ar Jh)) log (rus. J"),
(I)
JEJ"
where r and r denote the scores predicted by the teacher and the student, respectively, (r J exp(rTi)/ Ejeg" exp(rig) denotes the softmax over item set J Similarly
(r J") = exp(ra)) Σjer exp(rig). Note that for each user u, we only have access to
sampled item subset since it is computationally intractable over the entire item set I (Sun et al. 2024
CONNECTION BETWEEN CE LOsS AND RANKING IMITATION IN KD
This section reveals the connection between CE loss and ranking imitation in the field of KD. As starting point, we extend the connection between CE and NDCG to the full-item KD, where the CE loss computed using all items. Note that although the conclusion promising. the full-item KD is not practical due to the extremely large number of items. Therefore, we further analyze the connection between CE loss and partial NDCG in partial-item KD, where CE loss computed using only subset of items. Finally, we demonstrate the challenges when using CE loss as distillation loss by showing the large differences in the student's and teacher's top items.
4.1 ANALYSIS IN FULL-ITEM KD
This section studies the full-item KD, where CE loss is computed on the entire item set, i.e., Ju Given ground- tround-truth relevance scores vector y e R!Z with yı denoting the score of item i, and the predicted permutation л, NDCG is defined as:
NDCG v) = DCG( v)
DCG(
where से is the ideal ranked list (where items sorted according to y). DCG is defined as follows:
2
DCG(
(3)
loga(1 + (1)
where क -1) the rank item i.
In the following theorem, we show that minimizing CE loss maximizes the lower bound of NDCG, where the relevance scores of items are proportional the scores predicted by the teacher.
Theorem 4.1. Suppose that compute CE loss the entire item I and take the teacher's predicted scores (i.e., v.). ) as the target. In that case, we maximize a lower bound of NDCG, with the teacher's transformed predictive scores y logg(a (rr"')) + 1) being the relevance scores. Here o(-) T denotes the softmax function and o or ) (x)(rTT)) Σ JEZ exp(r")).
The proof is provided in Appendix B.1 Theorem 4. demonstrates that when minimize CE loss over the entire item set, the student can imitate the teacher terms of NDCG, This theorem gives an intuitive explanation of the rationality of using CE loss distillation loss.
4.2 ANALYSIS PARTIAL- ITEM KD
Although the above conclusion is promising, can only afford CE loss with a sampled item subset in real-world scenarios. This section shows that CE loss must involve both the teacher's 's and the student's predicted top items make the student benefit from the teacher's 's ranking ability.
Firstly, we define the partial NDCG to describe NDCG the partial-item KD scenario. only focuses the rankings within the item subset.
4

---

Published as a conference paper at ICLR 2026
MF
VF
LGON - LGGN
HSTU HSTU
LGGNN MF
HSTU MF
Ranking from S
Reginning
500-
End
500
750-
500
500
250
0.
100
0
100
100
100
0
100
Ranking from T
Figure 2: Relationship between rankings given by the teacher (shown æ-axis) and the student
(shown in y-axis). Items sorted in decreasing order according to the teacher's 's rankings.
Definition 4.2 (Partial NDCG). Given an item set T the partial NDCG on To (denoted as NDCG,.) is defined as follows:
(T,G).(A,1) _ (CG(T,3331)
where
(yJ-); = yi ff to otherwise,
(5)
denotes the truncated y that only retains the scores corresponding the items Ju, and Ju is the corresponding ideal runked list.
Then, to draw a promising conclusion analogous the full-item KD, we must make a not mild but critical assumption about the item subset J"
Assumption 4.3 (Closure of J"). For each item in J", we assume that all items that are considered by the student be ranked higher than are also in J". Formally,
U
c
(6)
NETT
where π -i) is the rank of item predicted by the student.
Finally, we have the following theorem that connects CE loss and partial NDCG.
Theorem 4.4. Given an item set J S that satisfies Assumption 4.3 minimizing CE loss on T maximizes a lower bound of NDCG where the relevance scores y (o 1)) Ju
The proof is provided in Appendix B.2 Note that Theorem 4.1 can be regarded as a special case of Theorem (e.eorem4.44]iie when Tu T. Previous works (Kang et al. 2020 Reddi et al.] (02T) find that the student can learn the rankings of top items given by the teacher, it benefits from the teacher's ranking ability. In other words, they expect to connect their distillation losses with NDCG where Ju involves the teacher's top items. Our theorem gives method with theoretical support for accomplishing that purpose. That is, T" must also involve enough top items provided by the student.
4.3 CHALLENGE IN PARTIAL-ITEM KD
According our analysis, J" must satisfy Assumption 4.3 for the connection between CE loss and partial NDCG to hold. However, it is difficult to satisfy this assumption if we do not explicitly consider the student's top items. Specifically, student's ' and the teacher's is rankings at the beginning and end of the training. The student is trained with vanilla CE loss, which is computed using the teacher's ' top items. The dataset is CiteULike. Detailed analysis and results on all datasets are given in Appendix A. From the results, we find that:
Figure 2 we report the relationship between the
Observation 4.5. The teacher's 's top items are very likely to be ranked low by the student, especially the beginning of the training.
As a result, if we compute CE loss only on the teacher's 's or the student's top items, we cannot bound partial NDCG on the teacher's 's top items. Moreover, if we simply add the student's 'ss top items to an item subset that initially contains the teacher's top items to make satisfy the assumption of closure, it will result in very large item subset.
5

---

Published as a conference paper at ICLR 2026
5
REJUVENATED CROSS-ENTRONE FOR KNOWLEDGE DISTILLATION
5.1 OVERVIEW OF RCE-KD
To unleash the potential of CE loss distilling rankings among the teacher's top items, we propose RCE-KD, novel approach involving both the teacher's and the student's top items in KD. The key is to split the teacher's s top items into two subsets based on whether or not an item is ranked high by the student. Then, we try to make both item subsets satisfy Assumption 4.3 exactly approximately.
Let O _ arg (rp (r")) and Q - arg ( denote the sets of top-K items predicted by the teacher and the student, respectively. We aim to transfer the teacher's ranking ability over 2 (Kang et al. 2020 Lee et al. 2019 Tang & Wang 2018). In RCE-KD,
propose to separate Q into Two subsets. The first subset is the intersection between 2 and 2 The second subset contains the remaining items in 2 Formally, (QT)1 & O n Q and (ot)2 (((()
5.2
LOss FOR (QT)
For the first subset, we transfer the knowledge within it by computing CE loss on O Formally,
L1
(r 2 log (r )
uEll iE 2
Note that 2 satisfies Assumption 4.3 Therefore, Li make the student benefit from the teacher's ranking ability by exactly bounding NDCGgg. Since (Q") is subset of O it encourages the student to learn the rankings among (QT))-
5.3 Loss FOR (2T)²
For the second subset, we propose approximately maximize NDCG (Q)) by computing CE loss the union of (2..): 2 and a set of randomly sampled items. The probability of each item being sampled is defined as follows: For each item in (Oh) )2, we raise the scores of all items ranked higher than i in the student's predicted ranking by After iterating the entire [ 2, let žj denote the score of item j. Then, the probability of item j to be sampled is given by P5 x e Vj e O where is hyperparameter and is fixed 10 in our experiments.
Note that the sampling strategy is adaptive due to: 1) When the student assigns low rankings to all items in (2,) )2, we sample nearly uniformly from the entire item set I, allowing us cover more items in multiple training epochs. 2) In contrast, we sample from highly ranked items when the student can already assign higher rankings items in (2²)) 2. According Theorem 4.4 these highly ranked items play a greater role in maximizing the partial NDCG on ( (QT)² )) and enable us to efficiently approximate the fulfillment of Assumption 4.3
Using the above sampling strategy, we sample L items and combine them with (o )2 to form the set A (note that resample in each epoch). Then, CE loss is computed on A as follows:
L2
1 U
Σ
Σ
a(r A²) lo) ( A").
(8)
NEW iE.A"
5.4 ADAPTIVE LOSS FUSION
Note that L1 and Le play different roles. Li focuses on (oT), 1. consisting top items considered by both the student and teacher. The goal of these items is to distill their fine-grained rankings, which is done by exactly maximize the partial NDCG, On the contrary, Le focuses on items not well-mastered by the student. The goal for these items is to improve their rankings, which is done by making the student imitate the teacher's 's ranking on (QT.) )2 and randomly sampled items.
To combine the two losses, we propose adaptive weighting scheme. Specifically, the final loss is
LRCE-KD (1 7) La + L2.

---

Published as a conference paper at ICLR 2026
Table 1: Recommendation performance. The best results are in boldface, and the best baselines underlined. Improv.b denotes the relative improvement of RCE-KD over the best baseline. LGCN stands for LightGCN. A paired (-test is performed over independent runs for evaluating p-value (s 0.05 indicates statistical significance..
T-+8
Method
10
Nir 10
Cielilikke
R4F20
N88/00
R 10
NW 10
Gowalla
R:
N::
R& 10
Yelp
R0 20
N0
Teacher
0.0283
0,0155
0,0442
0.0198
0.1088
0.0907
0.1544
0.1053
0.0394
0.0253
0,0660
0.0339
Student
0.0177
0.0098
0.0284
0.0128
0.0946
0.0820
0.1329
0.0839
0.0488
04222
0.0586
0,0299
CD
0.0239
0.0131
030158
0.0855
0.1559
0.026
DONTD
RRD
0.0135
0.0362
0.0169
0.0977
0.0861
0.987
00362
0/0330
0.026
0.0319
MP MP
HerComp
DCD
0,0255
0.0254
0.0136
0.0135
0.0391
0.0375
0.1733
0.0177
0.1007
1022
0.0871
0.0874
0.1427
0.1413
00999
010010
00377
0.031
08240
9,2445
0,039
0/644
0.0331
0.0330
TARec
0.0254
0.0133
0.0380
0.0111
0.1012
0.0870
0.1416
0.0993
0,0380
04034
0,035
0.0927
RCE-KD
0,0278
0,0152
0.0194
0,1082
0.0905
0.1525
0.1047
0.0400
0,0259
0,0667
0.0345
Imprend
9.02%
11.78%
10.23%
9.60%
5.25%
3.55%
6.7%
3.60%
4.44%
521%
3.57%
4.23%
p-valur
1.71e-4
1,986-5
5.476-4
7.126-5
4.22--5
2.79
8.228-4
3.376-3
6.326-4
1.236-4
1.73e-3
3.176-5
Teacher
0.0296
0.0160
0.0461
0,0205
0,1236
0.1035
0.1730
0.1190
0.0482
0.0276
0.0716
0.0367
Student
0.0113
0.0344
0,98
0.1580
0.1069
00363
0,0255
0,0211
0,0308
CD
0.0234
0.0125
0.0351
0000
01112
0
0.192
01100
0.0385
0.0337
0,0699
0,03
RRD
0.0247
0.0125
0.0359
0.058
011142
0,969
011627
0.1109
0,0391
0.0245
0.0671
0.0338
LOCN-LGCN
HetComp
DCD
0.0248
0.0243
0.0124
0.0127
0.0360
0.0362
0.0155
0.0160
0.1149
0.1150
0,000
0,071
0.1636
0.1108
011100
0.0405
0.0403
0.0256
0.0247
0.0776
0.891
0.0350
0.0340
TARec
0.0121
0.0350
0.055
0.1630
0.0099
0.0258
0,0911
0.0352
RCE-KD
0.0262
0.0133
0.0377
0.9171
0.119
0,1011
011681
04163
0.9431
UA277
0,0716
0J0369
Improub
4.73%
4.125
6.21%
4008
50095
27556
18355
3.18%
1.17%
4.8%%
p-oauue
4.72e-1
7,33c
9.87e-1
3.73e-3
1.3ee3
4,52e1
2.00e
7.38c-1
4.52e-3
3.37c:3
8.66e-
7.92e-1
Teacher
0,0463
0.0291
0.0613
00333
0.112
60901
0.1625
090060
0,0832
0.0000
0,0417
Student
0.0262
0.0159
0.0371
00009
0.0944
00781
01416
01W23
0.0991
00254
0.064
0.0342
CD
0.0428
0.0264
01038
01817
01527
019999
01001
RRD
0.0453
0.0263
0.0562
0.297
0,1048
0.0835
0.1538
0.1002
0,023
0,0244
0,7122
0.0373
HSTU -STTU
DCD
0,0292
6.050g
0,000
0.1541
11010
00400
00728
0,0382
HeiComp
0.4700
0.0299
0.0009
60311
0.1099
0.080000
01552
R1D18
00000
08785
00719
0.0377
TARec
0.0447
0.0270
0.0521
0.1053
0
0.1557
01000
004577
0,0280
0
0.0369
RCE-KD
0,0524
0.0325
0,8670
0.0366
0.1106
0.0902
0.1594
0.1058
0,0459
0,0305
0.0734
0,0400
Imprend
11.49%
8.70%
10.02%
10,57%
4.34%
5.25%
3.44%
3.62%
3.61%
5.17%
3.57%
1.5%%
4.71%
p-value
1.73e-1
4.22c-1
8.92e
9.92--1
3.576
8.20e-3
7.32e-1
3.71e-5
2:23e-3
2.38e-1
Teacher
0.0296
0.0160
0.0461
0.0005
0. 0,1236
0.1035
0.1730
0.1190
0,0432
0.0276
0.0716
0.0367
Student
0.0177
0.0098
0.0284
00028
0.0946
0.0030
0.1329
0.939
0.0348
0.0222
0.086
0.0299
CD
0.0133
0.0365
0.159
00064
00000
0.0312
RRD
0,0247
0.0137
0.0367
69009
8800
0
01070
00377
90200
00622
0,0300
LGCN-MF
HetComp
DCD
0.0260
0.0263
0.0142
0.0139
0.0402
0.0387
00000
01110
01123
0.9943
0.906
01000
011002
01098
0.0992
0.0399
9.0258
0.026
0,0441
0.6677
0.0344
0.0347
RCE-KD
TARec
0,0254
0.0137
0.0156
0.0437
0.370
00033
0.120
01110
0.0900
110013
0.1605
011677
0.1100
0.1163
01H19
0.0106
0,0363
0
00663
0.0350
Impracb
837%
9.88
871%
6300
4555%
5,44%
2.05%
4.7%%
2.86%
p-ualue
1.77-5
1.926-
4.29c-
6.990-
4 4.33645
1.236-3
3.23c-4
1.7164
6.820-1
9.05-4
1.036-3
Teacher
0,0463
0.0291
0,0613
0,0333
0.1124
0,0901
0.1625
01003
0,314
0,0000
0,0417
Student
0.0177
0.0098
6,0284
00128
0.0946
0,020
0.139
00039
00348
B0222
00586
000299
C13
0.0361
0.0209
0.0002
M0231
000844
0.1521
11002
B12266
01043
130170
RRD
0.0379
0.0224
0.0520
0.0270
0.1084
0.0831
0.1511
0.1018
08400
00221
0.0334
0.0359
HSTU >MP
HetComp
DCD
0.0411
0.0401
0.0253
0.0239
0.0533
0.0524
0.0295
0.278
0.1066
0,1078
0.0854
0.0840
0.1531
0,1552
0.1029
0.1031
0.041
0.0449
0.790
0.0297
0.0799
0.0765
0.0392
0,0382
TARec
0.0416
0.0258
00038
0,3000
0.1059
0.0838
0.1527
0.0223
00437
0A295
0.7762
0.0378
RCE-KD
0.0457
0.0285
0.0595
0.0324
0.1128
0.0905
0.1624
0.1065
0.0485
0A336
0
0.0419
Imprev.b b
9.86%
10.47%
10.59%
8.00%
4.64%
5.97%
4.64%
3,30%
7.06%
6.40%
5.23%
6.8%%
Puulue
6.73c-5
5.66e-1
1.93e-1
7.1le:5
1.37e-3
2.52e-1
3.31e4
2.83e:3
107c-1
3.88--
1.17e:3
1.92r-1
where y is updated at the beginning of each epoch by the following equation:
7 = exp (-3. ((2")
(10)
where |. /enotes the cardinality of the set and / is a hyperparameter. When (( )1| is small, we make the student overlap more with the teacher's 's top items by increasing the weight of Lz. Otherwise, we assign a large weight to & because it is more useful when we want to distill fine-grained rankings.
In Appendix C.5 we provide extensive, multi-level experiments that thoroughly validate the superior- ity of our adaptive 7-scheduulngg strategy.
Finally, the total loss for training the student is given by
L = Lase + A LICE-KD
(11)
where Lase is the loss of the base recommendation model, such as BPR loss. À is a hyperparameter.
6 EXPERIMENTS
Section 6.1 first introduces the experimental settings. The implementation details are shown in
Appendix C.
Then, the overall performance comparison is shown in Section 6.2 Consequently,
7

---

Published as a conference paper at ICLR 2026
investigate the training efficiency of all compared KD methods in Section 6.3 The ablation study is conducted in Section 6.4 To verify our sampling strategy's efficiency for approximating Assumption 4.3 conduct experiments in Appendix C.2 We present hyperparameter analysis in
C.5 to validate the superiority of our adaptive schedulingg strategy. Moreover, in Appendix D.1 and Appendix [D.2] we demon- strate the effectivvness of applying our method to sequential recommendation and multi-modal recommendation, respectively, showcase its generalization capability for recommendation tasks. Finally, we also visualized the evolution of NDCG during training in Appendix A.2 to demonstrate that RCE-KD KD successfully bounds NDCG as theoretically expected.
Appendix C..4 Then, we provide multi-level experiments in Appendix
6.1 EXPERIMENTAL SETTINGS
Datasets. We conduct experiments on three public datasets, including Cite (Wang et al. 2013 Kang et al. 2022 2021). Gowalla (Cho ct al. 2011]; Tang & Wang] 2018 Lee et al. [2019), and Yelp2018 (Lee et al. 2019) Kweon et al. 2021 Detailed statisties and methods of constructing training and test sets are given in Appendix C.1
Evaluation Protocols. Per the custom, we adopt the full-ranking evaluation achieve an unbiased evaluation. We employ Recall (Recall @ N) and normalized discounted cumulative gain (NDCG@) N) and report the results for N E {10, 20). We conduct five independent runs for each configuration and report the average results.
Baselines. We compare our method with five response-basee KD methods: CD (Lee et al. 2019).
RRD Kang et al. 2020), DCD (Lee & Kim 2021), HetComp (Kang et al. 2023 and TARec Zhuang et al. 2025). The introduction of these methods is in Appendix C.I
Backbones. We refer previous works (Chen al. 2023 Kang al.
2020
2021
and use
MF (Rendle et al. (012)) and LightGCN (He et al. 2020). We also add HSTU
(Zhai et al. 2024 as a
new backbone, which is popular generative recommendation model.
Teacher/Student. For each backbone, we create two instances, one large and one small. We use the large instance the teacher and the small one as the student. Details are provided in Appendix C.1
6.2 PERFORMANCE COMPARISON
The performance of all methods is provided in Table From the results, observe that: Different KD methods perform differently. We find that CD performs poorly compared to other methods. We attribute this CD using pair-wise loss to align teachers' and students' predictions. In training recommendation models, pair-wise loss is usually less effective than list-wise losses, such as RRD loss and CE loss.
Our method significantly outperforms all other methods in all cases, suggesting it effectively aligns the teacher's and student's 's predictions and utilizes the teacher's predictions enhance the student. This also demonstrates that utilizing CE loss for KD and using teacher and student predictions to collaboratively decide on sampling strategies are effective.
In all scenarios, students can perform similarly to teachers. This suggests that with the proper knowledge distillation approach, we can significantly reduce the model size and improve the model's inference efficiency with little to degradation of the model's recommendation accuracy.
Table 2: The comparison of the training time (seconds) per epoch.
CiteULake LighIGCN
Gowalla LightGCN
Yelp LightGCN
Method
MF
HSTU
MF
HSTU
MF
HSTU
Student
4.25
5.47
8.12
27.33
50.11
58.72
26.93
36.53
49.29
CD
14.37
20.80
29.03
82.77
137.63
201.70
77.72
126.38
210.69
RRD
19.67
23.81
39.09
132.37
167.90
231.12
119.49
152.27
271.86
DCD
21.37
26.82
38.63
145.87
158.60
241.35
108.37
162.92
292.00
HetComp
16.32
24.87
40.03
137.62
144.53
239.07
121.36
156.04
281.91 111
RCE-KD
15.23
21.79
33.62
141.72
221.65
81.17
129.74
233.38
8

---

Published as a conference paper at ICLR 2026
Table 3: The comparison of GPU Memory (GB) required by method and comparison methods.
CiteULike LightGCN
Gowalla TightGCN
Yelp LighIGCN
Method
MF
HSTU
MF
HSTU
MF
HSTU
Student
0.39
0.60
3.11
0.45
1.08
1.10
0.45
0.81
1.36
CD
1.07
2.52
6.27
6.27
8.81
19.37
4.99
7.02
17.62
RRD
0,92
2.42
6.81
5.23
8.64
19.93
4.85
6.30
19.01
DCD
1,41
2,93
7.22
7.89
9.37
21,52
6.03
7.22
20.89
HetComp
1.09
2.69
6.97
6.78
9.02
19.99
5.87
7.01
19.97
RCE-KD
1.05
2,47
6.52
6.37
8.90
19.98
5.41
6.90
18.74
6.3 TRAINING EFFICIENCY
In this section, report the training efficiency of our method and comparison methods. Since TARec involves a two-stage training process, we did not include it in the comparison. All results are obtained by testing with PyTorch
GeForce RTX 3090 GPU.
In RCE-KD, we only need add the cost of time and space required for random sampling on top of CE loss. Therefore, it has very high training efficiency. To empirically validate the training efficiency of our method, we report the training time and storage cost of our method and comparison methods. The results are presented in Table and Table 3 The method Student denotes that train the student model without KD. Note that since we save the teacher's 's predictions before KD and simply load the predictions without rerunning the teacher during KD, the architecture of the teachers does affect the training inference. Therefore, we only report the results with different students.
From the results, find that:
All KD methods inevitably increase training costs. In most cases, all KD methods have similar training costs. We believe this is attributed to the fact that they all follow similar pattern of sampling subset of items before computing the loss functions.
Among all baseline methods, find that CD and RRD have smaller training costs than others. We believe this is because CD and RRD are simpler and require fewer intermediate computational processes. However, they do perform well as the more complex methods. This forces baselines to face a trade-off between training cost and recommendation accuracy.
Our method has similar training efficiency as CD and RRD. This can be attributed to the simplicity of our method. Moreover, we empirically find that the number of items that need to sampled by our method is often smaller than that of other methods, significantly reducing the cost required in the sampling phase. Together, these two make our method highly efficient in training.
/ RCE-KD
w/o S
V-A w/o
w/o sep
KNNY w/ const
X88 full-CE
0.12
0.040
NDCG@20
NDCG@20
0.035
0.10
HST
0.030
HST
Gowalla
Yelp
Figure 3: Ablation study Gowalla and Yelp, including the results in three homogeneous Teacher + Student settings.
6.4 ABLATION STUDY
RCE-KD consists of three key components: 1) It divides the teacher's top items into two subsets; 2) It computes CE loss items selected from both the teacher's ss and the student's top items; 3) An adaptive mechanism is proposed to combine the losses on these subsets. To validate the effectiveness of these key components, design four variants: 1) RCE-KD w/o sep does not compute losses for

---

Published as a conference paper at ICLR 2026
(QT) and (1T),, separately. It only computes CE loss A" U Q 2) RCE-KD w/o S only aligns the predictions on the teacher's predicted top items, i.e. 2 3) Similarly, RCE-KD w/o only aligns the predictions the student's ' predicted top items, l.c., Q 4) RCE-KD w/ const replaces the adaptive weight derived from (q)(T0) with a constant hyperparameter y. We also compare with CE loss computed the full item set, denoted as full-CE, to validate the effectiveness of our method.
Figure 3 shows the results of these four variants on Gowalla and Yelp, and three Teacher/Student settings. The results of the remaining settings are provided in Appendix C.3 We find that all variants are inferior to the original RCE-KD, which demonstrates the effectiveness of all key components. Moreover, RCE-KD w/o S usually performs worse than RCE-KD w/o T. We believe that the reason is that the top items given by the student can exactly satisfy Assumption 4.3 while the top items given by the teacher do not. The superiority of RCE-KD w/ const over RCE-KD w/o T demonstrates the necessity of involving top items from both the student and the teacher. The superiority of RCE-KD over RCE-KD w/ const and RCF.KDD w/o sep validates the effectiveness of our adaptive weighting scheme and the necessity of splitting out the two subsets and treating them separately. Finally, our method performs even slightly above full-CE in most cases, due to a tighter bound on NDCG than full-item CE (Xu al. 2024a
CONCLUSION
This paper analyzes CE loss the real KD scenario for recommender systems, where loss is computed using subset of items. We prove that CE loss bounds NDCG. It makes CE loss suitable for recommender systems, where rankings are essential. We also theoretically provide a critical assumption about the item subset, on which CE loss is computed, for the conclusion hold. Based on the above analysis, we propose RCE-KD to fully unleash the potential of CE loss by approximately satisfying the assumption through teacher-student collaboration. Extensive experiments on both homogeneous and heterogeneous settings demonstrate the effectiveness of our method.
ACKNOWLEDGMENTS
This work was supported in part by National Natural Science Foundation of China (No. 62572198 and 92270119).
REFERENCES
Sebastian Bruch, Xuanhui Wang. Michael Bendersky, and Mare Najork. An analysis of the softmax cross entropy loss for learning-to-ren with binary relevance. In Proceedings of the 2019 ACM SIGIR international conference on theory of information retrieval, pp. 75-78, 2019.
Zhe Cao. Tao Qin, Tie-Ya Yan Liu, Ming-Feng Tsai, and Hang Li. Learning to rank: from pairwise approach to listwise approach. In Proceedings of the 24th international conference on Machine learning, pp. 129-136, 2007.
Gang Chen, Jiawei Chen, Fuli Feng, Sheng Zhou, and Xiangnan He. Unbiased knowledge distillation for recommendation. In Proceedings of the Sixteenth ACM International Conference on Web Search and Data Mining, pp. 976-984, 2023.
Eunjoon Cho, Seth A Myers, and Jure Leskovec. Friendship and mobility: user movement in location- based social networks. In Proceedings of the 17th ACM SIGKDD international conference Knowledge discovery and data mining, pp. 1082-1090, 2011.
Jiequan Cui, Zhuotao Tian, Zhisheng Zhong, Xiaojuan Qi, Bei Yu, and Hanwang Zhang. Decoupled kullback-leibler divergence loss. arXiv preprint arXiv:2305. 13948, 2023.
Jacob Devlin, Ming-Wor Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume (long and short papers), pp. 4171-4186, 2019.
10

---

Published as a conference paper at ICLR 2026
Ruining He and Julian McAuley. Ups and downs: Modeling the visual evolution of fashion trends with one-class collaborative filtering. In proceedings of the 25th international conference on world wide web, pp. 507-517, 2016.
Xiangnan He, Kuan Deng, Xiang Wang. Yan Li, Yongdong Zhang, and Meng Wang. Lighteen: Simplifying and powering graph convolution network for recommendation. In Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval, pp. 639-648, 2020.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint arXiv: 02.531.,,,, 2015.
Tao Huang, Shan You, Fei Wang, Chen Qian, and Chang Xu. Knowledge distillation from stronger 6 2022.
teacher. Advances in Neural Information Processing Systems,
SeongKu Kang, Junyoung Hwang, Wonbin Kweon, and Hwanjo Yu. De-rrd: A knowledge distillation framework for recommender system. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management, pp. 605-614, 2020.
SeongKu Kang, Junyoung Hwang, Wonbin Kweon, and Hwanjo Yu. Topology distillation for recommender system. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pp. 829-839) 2021.
SeongKu Kang, Dongha Lee, Wonbin Kweon, and Hwanjo Yu. Personalized knowledge distillation for recommender system. Knowledge-Band Systems, 239:107958, 2022.
SeongKu Kang, Wonbin Kweon, Dongha Lee, Jianxun Lian, Xing Xie, and Hwanjo Yu. Distillation from heterogeneous models for top-k recommendation. In Proceedings of the ACM Web Conference 2023, pp. 801-811, 2023.
Wang-Cheng Kang and Julian McAuley. Self-attentive sequential recommendation. In 2018 IEEE international conference on data mining (ICDM), pp. 197-206 IEEE, 2018.
Wonbin Kweon, SeongKu Kang, and Hwanjo Yu. Bidirectional distillation for top-k recommender system. In Proceedings of the Web Conference 2021, pp. 3861-3871, 2021.
Jae-woong Lee, Minjin Choi, Jongwuk Lee, and Hyunjung Shim. Collaborative distillation for top-n recommendation. 2019 IEEE International Conference on Data Mining (ICDM), pp. 369-378. IEEE, 2019.
Youngjune Lee and Kee-Eung Kim. Dual correction strategy for ranking distillation in top-n recommender system. Proceedings of the 30th ACM International Conference on Information & Knowledge Management, pp. 3186-3190, 2021.
Julian McAuley, Christopher Targett, Qinfeng Shi, and Anton Van Den Hengel. Image- based recommendations on styles and substitutes. In Proceedings of the 38th international ACM SIGIR conference on research and development in information retrieval, pp. 43-52, 2015.
Naoto Ohsaka and Riku Togashi. Curse of" low" dimensionality in recommender systems. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 537-547, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. International conference on machine learning, pp- PmLR, 2021.
Pradeep Ravikumar, Ambuj Tewari, and Funho Yang. On ndeg consistency of listwise ranking methods. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pp. 618-626 JMLR Workshop and Conference Proceedings, 2011.
Sashank Reddi, Rama Kumar Pasumarthi, Aditya Menon, Ankit Singh Rawat, Felix Yu, Seungyeon Kim, Andreas Veit, and Sanjiv Kumar. Rankdistil: Knowledge distillation for ranking. In International Conference Artificial Intelligence and Statisties, pp. 2368-2317 PMLR, 2021.
11

---

Published as a conference paper at ICLR 2026
Nils Reimers and Iryna Gurevych. Sentence-bert. Sentence embeddings using siamese bert-networks. arXiv preprint arXiv: 1908. 10084, 2019.
Steffen Rendle, Christoph Freudenthaler, Zeno Gantner, and Lars Schmidt-Thieme Bpr: Bayesian personalized ranking from implicit feedback. arXiv preprint arXiv:/205. 220.22688 2012.
Wenqi Sun, Ruobing Xie, Junjie Zhang, Wayne Xin Zhao, Leyu Lin, and Ji-Rong Wen. Distillation is all you need for practically using different pre-trained recommendation models. arXiv preprint arXiv:2401. 0007007777 2024.
Jiaxi Tang and Ke Wang. Ranking distillation: Learning compact ranking models with high per- formance for recommender system. In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining, pp. 2289-2298, 2018.
Hao Wang, Binyi Chen, and Wu-Jun Li. Collaborative topic regression with social regularization for tag recommendation. In IJCAI, volume 13, pp. 2719-27122 2725, 2013.
Wei Wei, Lianghao Xia, and Chao Huang. Multi-relational contrastive learning for recommendation. In Proceedings of the 17th ACM conference recommender systems, pp. 338-349, 2023.
Wei Wei, Jiabin Tang, Lianghao Xia, Yangqin Jiang, and Chao Huang. Promptmm: Multi-modal knowledge distillation for recommendation with prompt-tuning. In Proceedings of the ACM Web Conference 2024, pp. 3217-3228, 2024.
Jiancan Wu, Xiang Wang, Xingyu Gao, Jiawei Chen, Hongcheng Fu, and Tianyu Qiu. On the effectiveness of sampled softmax loss for item recommendation. ACM Transactions on Information Systems, 42(4):1 2024.
Fen Xia, Tie-Yan Liu, Jue Wang, Wensheng Zhang, and Hang Li. Listwise approach to learning to rank: theory and algorithm. In Proceedings of the 25th international conference Machine learning, pp. 1192-1 199, 2008
Fen Xia, Tie- Yan Liu, and Hang Li. Top-k consistency of learning rank methods. Advances in Neural Information Processing Systems, 22:2098-2106, 2009.
Cong Xu, Jun Wang, and Wei Zhang. Stablegen: Decoupling and reconeiling information propagation for collaborative filtering. IEEE Transactions on Knowledge and Data Engineering, 2023.
Cong Xu, Zhangchi Zhu, Jun Wang, Jianyong Wang. and Wei Zhang. Fairly evaluating large language model-based recommendation needs revisit the cross-entropy loss. arXiv preprint arXiv:2402.0621 02440200211, 16, 2024a.
Cong Xu, Zhangchi Zhu, Jun Wang, Jianyong Wang, and Wei Zhang. Understanding the role of cross-entropy loss in fairly evaluating large language model-based recommendation. arXiv preprint arXiv:2402.062166 2024b.
Weiqin Yang, Jiawei Chen, Xin Xin, Sheng Zhou, Binbin Hu, Yan Feng, Chun Chen, and Can Wang. Psl: Rethinking and improving softmax loss from pairwise perspective for recommendation. arXiv preprint arXiv:2411.00 00163, 2024.
Jiaqi Zhai, Lucy Liao, Xing Liu, Yueming Wang, Rui Li, Xuan Cao, Leon Gao, Zhaojie Gong, Fangda Gu, Michael He, et al. Actions speak louder than words: Trillion- parammerr sequential transducers for generative recommendations. 2024.
Xin Zhou, Hongyu Zhou, Yong Liu, Zhiwei Zeng. Chunyan Miao, Pengwei Wang, Yuan You, and Feijun Jiang. Bootstrap latent representations for multi-modal recommendation. In Proceedings of the ACM web conference 2023, pp. 845-854, 2023.
Zhangchi Zhu and Wei Zhang. Exploring feature-based knowledge distillation for recommender system: A frequency perspective. arXiv preprint arXiv:2411. 10676, 2024.
Zhangchi Zhu and Wei Zhang. Preference-conniistent knowledge distillation for recommender system. IEEE Transactions on Knowledge and Data Engineering, 2025.
Ziyi Zhuang, Hanwen Du, Hui Han, Youhua Li, Junchen Fu, Joemon Jose, and Yongxin Ni. Bridging the gap: beacher-assisted wasserstein knowledge distillation for efficient multi-modal recommendation. Proceedings of the ACM on Web Conference 2025, pp. 2464-2475, 2025.
12

---

Published as a conference paper at ICLR 2026
MF MF Se
LGGN
L.GNN CleLLike FISTU
HSSUU on CleUT lkk
LOON MF CheUTiike
HSTC
MF Chilll iko
600
Reginning
750
800-
M V
600
750
/ hm
400-
500
600-
400-
500
200-
250
400
200-
250
0
Ranking given by the student
90
100
0
50
100
0
50
100
30
100
50
100
MP
MP Gowallaa
LGGCN -LGCN on Goowalla
HSTU HSTU on Gowalll
LGCNN MF on Gowailla
HIS TU
MF on Goowalla
600
200-
End
Beginning
400
400-
300-
200-
mMMM
200-
100
200
200
100
0-
0-
0-
50
100
0
50
100
50
100
0
50
100
50
100
MI - MI on Yelp
LGCN - LGGN Yelp
IISTU - IISTU om Yelp
LGCN - MF Yelp
IESTU - on Yelp
200
200
300
200
100
100
200
Ar App
100
Reginning
100
Eind
04
04
04
0
50
100
0
50
100
50
100
50
100
0
50
100
Ranking given by the teacher
Figure 4: Relationship between rankings given by the teacher (shown in zr-axis) and the student (shown in y-axis) on all datasets. Items are sorted in decreasing order according to the teacher's rankings.
A VISUALIZATIONS
A.I RELATIONSHIP BETWEEN THE RANKINGS GIVEN BY THE STUDENT AND THE TEACHER
To investigate whether we can satisfy Assumption 4.3 without explicitly considering the student's top items, we report the rankings given by the student and the teacher. The student trained with vanilla CE loss, which is computed using the teacher's top items. The items are sorted in decreasing order according the rankings given by the teacher. The results on all three datasets are provided in Figure In each subfigure, we give two lines. The grey one represents the results at the beginning of training (after about 0.2x total epoch number of rounds of training). The red one represents the results after training is complete.
The results show similar trends all cases. Concretely, we observe that: 1) There a significant positive correlation between the rankings given by the teacher and the student. This suggests that through knowledge distillation, students do learn some the teacher's ranking results. 2) For top items given by the teacher (ranked higher than 100), students often give lower rankings (lower than 100 and even 200 on CiteULike). 3) The phenomenon is particularly acute at the beginning of training.
A.2 TIGIITER NDCG BOUND VERIFICATION
To empirically verify that our improvement to the CE loss indeed enables us to bound NDCG more tightly, as predicted by our theory, we conduct a comprehensive analysis of the training dynamics across all three recommendation datasets: CiteULike, Gowalla, and Yelp. Specifically, we monitor the NDCG@ 10 metric on the training set throughout the training process, comparing our improved CE loss with the original CE loss under various teacher-student architecture combinations.
Figure presents the training curves of NDCG@ 10 for both methods across five distinct knowledge distillation scenarios: MF-MF
HSTU~HSTU LGCN and HSTU-MF. The experimental results consistently demonstrate three key advantages of our RCE-KD:
Higher NDCG values: Across all datasets and teacher-student combinations, RCE-KD consistently achieves higher NDCG@ 10 values throughout the training process compared to Vanilla CE. This indicates that approach consistently and stably improves NDCG throughout the entire training
13

---

Published as a conference paper at ICLR 2026
MF-MF CikeLike
AGCN-LGGN CikeLLãe I85UU-HSIU Like
LGCN-MF CiteLLike
HSTU-MF CikeLiko
0.8
0.8
0.8
0.8
0.8
0.4
0.4
0.4
0.4
0.4
Vamilla CE
0.2
RCE-KD
0.2
0.2
0.2
0.2
0.0
0.0
0.0
0.0
200
200
200
0
200
0
200
MF MF-MFFum on Gowalls
LOCN-LOCN on Gowalla
UISUU-HH FISTU on Gowalla
LGCN N- un Gowalls
USSU on Gowelly
NDCG@10
RCL-KD
0.6
0.6
0.6-
0.6-
0.6
0.2
0.2
0.2
0.2
0.2
100
50
50
100
50
100
Yelp
-LGN Yelp
Yelp
Yelp
0.8
Vanilla CF
0.8
0.8
0.8
RCE-D
0.6
0.4
0.4
0.4
0.4
0.4
0.0
0.0
0.0
0.0
0.0
Epoch
Figure 5: Training curves of NDCG@ 10 the training set for RCE-KD and Vanilla CE across three datasets (CikeULike, Gowalla, Yelp) and five teacher-student architecture combinations.
process. Moreover, the higher NDCG achieved by method after training convergence demonstrates its superior ability to optimize NDCG.
Faster growth rate: The training curves reveal that RCE-KD exhibits steeper learning curve, indicating a more rapid improvement in NDCG during the early training epochs. This accelerated learning be attributed to the more informative gradient signals provided by RCE-KD. This observation directly supports our theoretical analysis that RCE-KD provides tighter bound on the ranking metric, as reflected by the superior optimization of NDCG during training.
Faster convergence: In addition to achieving higher peak performance, RCE-KD demonstrates faster convergence to optimal NDCG value in most cases. The training curves show that RCE-KD reaches its plateau earlier than Vanilla CE, suggesting that the method not only achieves better final performance but also requires fewer training epochs reach convergence.
Based on the above observations, we can confidently assert that our method indeed achieves tighter bounds on NDCG, as predicted by our theoretical analysis. Consequently, enables students to learn the ranking capabilities of teachers more effectively, making it more suitable for knowledge distillation in recommender systems.
B PROOFS
B. PROOF THEOREM 4.1
Proof. By inserting y = log,(a(r)) + 1) into the definition of DCG in Eq. 3
we have
a ); log, + (1))
DCG( v) = Σ
(12)
iET
14

---

Published as a conference paper at ICLR 2026
Then, similar to the proof for Theorem 3 in (Bruch et al. 2019).
have
o )
DCG( y)
(13)
loga (1 + (1)
a(r?) )
(14)
12 π (1)
exp(rat)
Σ ((r)):
(15)
ENEZ (xp(ri)))
where r is the student's predictive score vector that derives the permutation π.
For the ideal DCG, we have
( T ):
DCG(T, y)
(16)
TET loge
σ (r.T)
)[
(17)
IET
(18)
Finally,
DCG( y)
log NDg y) = log
(19)
DCG y)
exp(rist
2 log
Σ (rr")
(20)
Σ KEE exp(ris (rsk))
IEI
exp(rast)
Σ
a(r
log
(21)
ENEI exp(rruk) (akk
where the final incquality holds because of Jensen's is inequality. We complete the proof by noting that the right-hand side of the final inequality is the negative of CE loss.
B.2 PROOF OF THEOREM 4.4
Proof.
DCG(T, MgT)
NDCG (u y)
(22)
D Vyn)
z DDGG(TT,
(Due to DCG(
I 1.)
(rr.T))
(23)
log, 1+ π (i))
KEJ"
(24)
π 11
1E3
Σ
(25)
Σ R. 10)5a 100) exp(r - ).))
16
exp(r
Σ
(26)
(--))))--1(1))) exp(r)))
exp(r-S)
2 Σ
(
(27)
Ejeg exp(ra))
1E
15

---

Published as a conference paper at ICLR 2026
Table 4: Statistics of the preprocessed datasets.
Dataset
#Users
#Items
#Interactions
#Sparsity
CiteULike
5,219
25,181
125,580
99.89%
Gowalla
29,858
40,981
1,027,370
99.92%
Ye1p2018 18
41,801
26,512
1,022,604
99.91% %
Table 5: Dimensions of teachers and students for MF and LighIGCN.
CiteULike LightGCN
Gowalla
Yelp
Model
MF
MF
LightGCN
MF
LightGCN
Teacher
400
2000
300
2000
300
1000
Student
20
20
20
20
20
20
Therefore,
exp(r
log (T,)) 2 log Σ a(r");
(28)
ΣjEJ exp(rig)
exp(rTi)
exp(r
= log
Σ
log Σ
a(r)))
(29)
Σ JJ exp(riT)
Σ Ljero exp(r
SET
jEJ
exp(rir)
exp(rei)
log
log Σ
a(rr))
(30)
JEJU exp(rit)
EjEf" exp(rr.)
uzj
jEJ
exp(rit)
exp(rai)
log
log Cga,
(31)
Cj exp(ri))
Σ Ej exp(r S
where Cz. & Ejern is constant, given J"
Note that by minimizing CE loss on J", which is defined as follows:
exp(r Ejeru exp(rig) TT log
exp(r=i)
Σ T
(32)
EJEJ exp(r (8
we also maximize
Σ exp(r.T) vi :EJ* Ejera exp(r
log exp(r=))
+ log Czu,
(33)
Σjer exp(r
because Cgu a constant when Ju is fixed.
C MORE EXPERIMENTAL RESULTS
C.I EXPERIMENTAL SETTINGS
Datasets. We conduct experiments three public datasets, including CiteULike Wang al.
2013 Kang et al. 2022 2021). Gowalla (Cho et al. 20011 Tang & Wang 2018 Lee et al. 2019). and 18 (Lee
2019 Kweon al. 2021).
Following the previous method (Xu et al. 2023 we filter out users and items with less than 10
interactions and then split the rest chronologically into training, validation, and test sets in 8:1:1 ratio. The statistics of the preprocessed datasets are summarized in Table
16

---

Published as a conference paper at ICLR 2026
Table 6: The Number of transformer blocks (#Block) and number of heads (#Head) for HSTU.
CiteULike
Gowalla
Yelp
Model
#Block
#Head
#Block
#Head
#Block
#Head
Teacher
4
8
Student
1
1
1
Table 7: Overlap rate at the beginning (2% of total training epochs), midpoint (20% of total training epochs), and end (100% of total training epochs) of training. Denoted OV @ 100 respectively.
OV @2, OV @ 20, and
CikeULike
Gowalla OR@ 20
Yelp @R@20
T-S
OR@2
OR 20
OR@ 100
OR@2
OR@ 100
OR@2
OR@ 100
MF-MF
0.57
0.89
0.98
0.69
0.93
0.98
0.64
0.94
0.95
LGCN-LGCN
0.67
0.94
0.97
0.60
0.90
0.96
0.71
0.95
0.97
HSTU-HHST
0.69
0.96
0.98
0.67
0.92
0.97
0.73
0.93
0.98
LGCN-MP
0.52
0.92
0.95
0.56
0.90
0.96
0.62
0.93
0.95
HSTU-MM
0.97
0.61
0.92
0.95
0.67
0.95
0.96
Teacher/Student. For each backbone, create two instances, large and small. We the large instance as the teacher and the small one as the student. For the large instance, we increase the longer improves and adopt the model with the
model size until the recommendation performance best performance. For the small instance, we select the hyperparameters to enlarge the performance gap between the student and the teacher.
Concretely, for MF and LightGCN, we choose different embedding dimensions for the teacher and the student while keeping other hyperparameters the same. The detailed embedding dimensions are provided in T5 As for HSTU, we decrease the number of transformer blocks and the number of heads obtain the student model. The final number of blocks and heads for HSTU is given in Table
In addition to homogeneous settings, we consider two heterogeneous settings where teachers and students have different architectures: 1) LightGCN as the teacher and MF as the student, and 2) HSTU as the teacher and MF as the student.
Implementation Details. We implement all the methods with PyTorch and use Adam the optimizer. Before distillation, we save teacher's predictions and load them during KD instead of rerunning the teacher. In the case of using HSTU the student, fix the batch size to 128. In other cases, we fix it to 2048. For our method, the weight decay is selected from {le-3, le-5, Je-7]. The search is selected from
space of the learning rate is f1e-3, le-4). 3 is selected from (0.5,1,3,5,7.9 100,500, 1000). We 10.5,1,5, 10,50, 100,0,00000000 100,000,500000 5500000,, 10000))000000000 K and I. are both selected from 110,50,000,500, conduct early stopping according the NDCG@ 20 on the validation and stop training when the NDCG @ 20 does not increase for 30 consecutive epochs. All hyperparameters of the compared baselines are tuned to ensure optimal performance.
Baselines. We compare our method with the following knowledge distillation methods:
CD (Lee al., 2019) samples unobserved items with ranking-relatee distribution and uses point-wise KD loss.
RRD (Kng et al.
2020) adopts a list-wise loss to maximize the likelihood of the teacher's
recommendation list.
DCD (Lee & Kim 2021
corrects what the student has failed to predict with a dual correction loss
accurately.
2023 guides the student model by transferring easy-to-hard knwoledge sequences generated from the teacher's trajectories.
HetComp Kang et al.
TARec (Zhuang et al. 2025) proposes a teacher-assisted Wasserstein Knowledge Distillation framework that contains two-stage distillation bridge the gap between the teacher and student.
17

---

Published as a conference paper at ICLR 2026
/
RCE-KD
+ill w/o
|, w T
w/o scp
ANNY
w/ const
XXX
full-CE
0.04
NDCG@200
0.03
0.02
LOCN
CiteULike
0.12
NDCG@20
0.11
0.10
LGCN LGCN
HSTU
LGCN MF
HSTU
Gowalla
NDCG@20
0.040
0.035
0.030
MF
HSTU
HSTU
MF
Yelp
Figure 6: Ablation study on all datasets and all Teacher + Student settings. The average NDCG@3 and standard deviation over independent runs are provided.
C.2 APPROXIMATE EFFICIENCY OF ASSUMPTION 4.3
In Theorem 4.4 we demonstrate that the relationship between CE loss and NDCG can only be
established when Assumption 4.3 holds. To address the practical limitation of precisely satisfying Assumption 4.3 in real-world scenarios, we devise a novel sampling strategy for [ ) in Section 5.3
This strategy enables the extended A to closely approximate Assumption design experiments to validate the efficiency of this approximation. Specifically. we compute the degree of overlap between the set we constructed (i.e., AA") and the ideal set as training progressed. Formally, we take the [Ap- items given by the student as the ideal set (denoted as Idea") because strictly satisfies the closure assumption. Then, in Table 7 we show the overlap rate between A and the ideal set I dea" at the beginning, midpoint, and end of the training. The overlap rate is computed d.
In this section,
as OV A n Idea" /|A
From the results in Table 7 observed that during the early stages of training (approximately 2% of total training epochs), a high overlap rate (exceeding 60%) is typically achieved. As training progresses, the overlap rate increases rapidly, reaching approximately 95% by the mid-trainin phase (around 20% of total training epochs). By the end training, the overlap rate reached approximately 98%.
18

---

Published as a conference paper at ICLR 2026
MF MM
LOCN LOCN
HSTL HSTL
MP MP-MF
1GCN-11O
RSTT IIST
Civll//ee
Gowall
Yelp
Creliiie
Govals
ND00
0,040
0040
0.11
01333
0.10
Effect of A.
(b) Effect of
LCCN TOCN
HNTT NSTTU
LOCN L.GNN
HSTU HSTU
CillTike
Groalh
Ydl
Chefil Rir
Gowala
Yilp
ND008220
0.040
0.000
0.0
0.885
0.10
1000 K
1000
1000
1003
100 1000
1000
X
L
(c) Effect of K.
(d) Effect of L.
Figure 7: Hyperparameter study on three datasets. We the results on three homogenous Teacher - Student settings. The average NDCG @ 20 and standard deviation over independent runs are provided.
LCCN-MP
FISTU - MP
LGCN -GGCN
RSTU -HSTU
CldULic
Gonalls
Cheliiik
Govall
Yel
N0C00000
0.02
0,24
0.045
01110
0.02
0.109
0,035
DL1D5
0.011
7
(a) Effect of X.
(b) Effect of B
MF
Clelliie
Govalla
Yel
ChdlLike
Gonall
Vel
NDCG620
600
0.115
0000
0.
0,000
0.110
0.105
0.25
0.00
100
TJO XO0D
3001000
100
500
100 900
500 m00
K
K
K
1.
(c) Effect of K.
(d) Effect of L.
Figure 8: Hyperparameter study on three datasets. We report the results on two heterogencous Teacher Student settings. The average NDCG@2 20 and standard deviation over 5 independent runs are provided.
C.3 ABLATION STUDY
This section presents additional results of the ablation study. In Figure 6 we give the results on all KD settings and all datasets. The results suggest similar trends Figure 3 Specifically, we find that all variants are inferior to the original RCE-KD, demonstrating the effectiveness of all key components. Moreover, RCE-KD w/o S usually performs worse than RCE-KD w/o T. We believe that the reason is that the top items given by the student can exactly satisfy Assumption the top items given by the teacher do not. On the other hand, the superiority of RCE-KD w/ const over RCE-KD w/o T demonstrates the necessity of involving top items from both the student and the teacher. The superiority of RCE-KI over RCE-KD w/ const and RCE-KD w/o sep validates the effectiveness of our adaptive weighting scheme and the necessity of splitting out the two subsets and treating them separately. Finally, our method performs even slightly above full-CE in most cases, due to tighter bound on NDCG than full-item CE (Xu et al. 2024a).
while
19

---

Published as a conference paper at ICLR 2026
CiteUlike
Gowalla
Yelp
Mcan of's,
0.4
0.2
100
100 Epoch
0
50
100
Epoch
Epoch
Figure 9; Training curves of the mean of Su across three datasets (CiteULike, Gowalla, and Yelp).
C.4 HYPERPARAMETER ANALYSIS
Effects of X. We À to balance the impact of our KD loss and the base loss in Eq. [T]. ET. In Figure 7(a) and Figure 8(a) we report the effect of X. The results suggest that the suitable values of vary across different datasets. For general, the best choice of À lies in (5, 10, 50).
Effects of (. In Eq. 10). we use 33 for computing the adaptive weight. In Figure 7(b) and Figure [(b) we analyze the effect of B. The best choice of 3 lies {3, 5, 7). We find that both large or too small |3 will lead to worse performance because neither of them takes into account both subsets (i.e., (2T.): ) and (OT)) )2) in the same time.
Effects of K. We define O" and Q as the set of items with top- K scores predicted by the teacher and the student, respectively. Here, the hyperparameter K affects the size of these two subsets. In Figure TigureF(cc)an and Figure [[c) we analyze the effect of K. We observe that K is optimal at 50 or 100. If K too small, will result in key items being ignored; if K too large, it will introduce too much noise. Thus, choosing suitable K will benefit the performance.
Effects of When constructing A for the second loss L2, we sample L items through our proposed sampling strategy. Figure 7(d) and Figure S(d) analyze the effect of L. We find that the optimal value of is 100. We also find that the performance less sensitive to the change of than K. However, since large L inevitably introduces larger training cost, suggest choosing suitable by considering both the recommendation accuracy and the training inference.
C.5 JUSTIFICATION OF ADAPTIVELY SCHEDULING
In this section, we justify our r-rule through three principals: (1) decreasing, (2) user-specific, and (3) exponential.
C.5.1 L WEIGHT MUST INCREASE OVER TRAINING
Our distillation objective is
CRCE-RD = (1-7)C1 + 722.
(34)
Let
G VLLIl, G2 |V
(35)
Then the gradient contribution of L1 is
(1 (1-7)G1
dw dy
wi (7) (1 (1- + +G2²
< 0.
(36)
Thus decreasing 7 strictly increases the influence of Li-
We compare three schedules: 1) fixed y, 2) decreasing 7, and 3) increasing 7. The results are given in Table
20

---

Published as a conference paper at ICLR 2026
CiteUlike- Beginning
CiteUlike Middle
CiteUlike End
600
600
600
Count
400
400
400
200
200
0.0
0.2
0.4
0.6
0.00
0.25
0.50
0.75
0.2
0.4
0.6
0.8
Gowalla Beginning
Gowalla Middle
Gowalla End
Count
5000
5000
5000
2500
2500
2500
0.0
0.2
0.4
0.2
0.4
0.2
0.4
Yelp- Beginning
Yelp Middle
Yelp End
3000
3000
3000
Count
2000
2000
2000
1000
1000
1000
0.0
0.2
0.4
0.0
0.2
0.4
0.2
0.4
Su
Su
Su
Figure 10: The histogram of at the beginning, middle, and end of training.
Table Recommendation performance of different scheduling methods: fixed 7 decreasing 7
and 3) increasing
CitelT like
Gowalla
Yelp
T-S
Schedule
Recall 20
NDCG@20
Recall
NDCG@20
@ecall@
NDCG@2 20
fixed
0.0393
0.0169
0.1440
0.0996
0.0627
0.0324
MF-MF
decreasing
0.0410
0.0174
0.1459
0.1003
0.0634
0.0327
increasing 0.0372
0.0164
0.1421
0.0990
0.0624
0.0318
fixed "
0.0337
0.0157
0.1630
0.1110
0.0671
0.0347
LGCN-1I.GCN
decreasing y
0.0342
0.0160
0.1639
0.1117
0.0674
0.0350
increasing
0.0335
0.0152
0.1621
0.1106
0.0669
0.0342
fixed
0.0615
0,0330
0.1532
0.1021
0.0724
0.0374
HSTU-HSTU
dccreasing y
0.0626
0.0341
0.1540
0.1024
0.0726
0.0380
increasing 0.0603
0.0319
0.1527
0.1008
0.0717
0.0366
We find that only the schedules that increase -i-wehhht with epoch consistently improve NDCG, This matches our principal of amplifying the influence of L1 and confirms that C₁ should progressively dominate training proceeds.
C.5.2 WHY y MUST BE ASER-SPECIFIC-
WHY USING s, IS JUSTIFIED
In the previous section, y changes with epoch because analyze global learning dynamics. To enable user-specific adaptivity, however, epoch alone is insufficient: different users progress at different speeds. Thus we define
1( 1l
Su
Q
(37)
which measures how well user aligns with the teacher.
FigureP|shows that on average, s, increases monotonically with training proceeding. This implies that making L1 Lieihttht increase with 8, is consistent with (and indeed equivalent to) the global
21

---

Published as a conference paper at ICLR 2026
Table 9: Performance comparison between global and user-specific scheduling strategies under
different seacher-student settings.
Cike
Gowalla
Yelp
T--
Schedule
Recall@20
NDCG@20
2ecall@20
NDCG@20
Recall @0
NDCG@20
Global
0.0410
0.0174
0.1459
0.1003
0.0634
0.0327
MF-MF
User-specific
0.0419
0.0181
0.1477
0.1012
0.06488
0.0333
Global
0.0342
0.0160
0.1639
0.1117
0.0674
0.0350
LGCN-1L.GCN Uscr-specific
0.0363
0.0164
0.1664
0.1133
0.0699
0.0357
Global 0.0626
0.0341
0.1540
0.1024
0.0726
0.0380
HSTU-HSTU
User-specific
0.0643
0,0350
0.1567
0.1032
0.0734
0.0388
Table 10: Performance comparison between linear and exponential scheduling strategies under different seacher-student settings.
CiteULike
Gowalla
Yclp
T S
Schedule
20
NDCG@20
Recall@20
NDCG@20
Recall Φ 20
NDCG@20
Linear 0.0419 0.0431
0.0181
0.1477
0.1012
0.0648
0.0333
MF-MF
Exponential
0.0194
0.1525
0.1047
0.0667
0.0345
Linear 0.0363
0.0164
0.1664
0.1133
0.0699
0.0357
LGCN LLGCN
Exponential
0.0377
0.0171
0.1681
0.1163
0.0716
0.0369
Linear
0.0643
0.0350
0.1567
0. 0.1032
0.0734
0.0388
IISTU~IISTU
Exponential
0.0670
0.0366
0.1594
0.1058
0.0754
0,0400
decision that L1 should increase with training proceeding. The crucial difference that Su captures heterogeneous per-user progress, whereas epoch does not.
Because the distribution of {su} remains highly dispersed during training (as shown in Figure 10p. hard users retain low 8,. while easy users obtain high 8a. A global forces both types of users to adopt the same Li/L2 weight ratio, which is suboptimal. Thus we adopt the user-specific schedule
Yu = as + b, L1-weight= = 1
(38)
Table shows that this user-specific rule significantly outperforms global y.
C.5.3 WHY THE INCREASE SHOULD EXPONENTIAL
Having established monotonic increase in L -eeweghht, we compare two increasing schedules:
Linear: Min (su) as +
Exponential: Texp (sw) = -1-e--0 33
The exponential form provides a phase transition, yielding low weight of Ci initially (allowing Co to explore) and rapid change later (letting L, dominate).
Table 10 shows that the exponential rule consistently outperforms the linear rule, validating its principled nature.
C.5.4 FINAL JUSTIFICATION
From the analyses above, proper schedule must:
1. Increase Le-weight over the course training.
2. Adapt per user, which requires using Su-
3. Follow an exponential pattern.
Our final rule the minimal functional form that satisfies all three principles. Alternative schedules violate at least one of these principles and perform worse
22

---

Published as a conference paper at ICLR 2026
Table 11: The Number of transformer blocks (#Block) and number of heads (#Head) for SASRec.
CiteULike
Gowalla
Yelp
Model
#Block
#Head
#Block
#Head
#Block
#Head
Teacher Student
4
8
1
1
1
D APPLICATION ON MORE RECOMMENDATION TASKS
D.1 APPLICABILITY IN SEQUENTIAL RECOMMENDATION
Our method can be easily applied to other recommendation scenarios, such sequential recommen- dation. To verify this, we construct sequential recommendation datasets using the datasets in our paper. Specifically, we take each user's last interaction as the test item, the second-to-las interaction as the validation item, and the previous interactions as training items. To further enrich our evaluation, we introduce the classical sequential recommendation model SASRec (Kang & McAuley 2018) as an additional backbone. include new distillation setting in which both the teacher and the student based on SASRec. The detailed configurations for this setup provided in Table
In Table [2] we report the performance of all methods under five knowledge distillation settings. The results demonstrate that approach still significantly outperforms all baseline methods sequence recommendation tasks. Compared to the best baseline method, our method achieves improvements ranging from minimum of 2.74% to a maximum of 10%. This performance is comparable to results on top-N recommendation tasks presented in the main text, indicating the strong generalization capability of our method across recommendation tasks.
D.2 APPLICABILITY IN MULTI-MODAL. MAA.MMDDDRRROOMMEEAAAAAAA RECOMMENDATION
To further examine whether our approach generalizes to multi-modal recommendation scenarios, we conduct experiments on three widely used multi-modal recommendation datasets:
The Netflix dataset (Wei et al. 2024) provides user-item interaction logs collected from the streaming platform. Each movie accompanied by multimodal metadata, most notably poster images and titles. Visual representations are derived using the CLIP-ViT encoder (Radford al. 2021). while textual information is transformed into embeddings via pre-trained BERT model (Devlin et al. 2019).
Netflix:
Tiktok: The Tiktok micro-video dataset (Wei et al. 2023) offers interaction histories together with three complementary types of content: visual, acoustic, and textual. Visual and audio signals from short videos are processed into 128-dimensional feature vectors, and the accompanying captions are embedded using Sentence- BERT (Reimers & Gurevych 2019).
Electronics: The Amazon Electronics dataset (He & McAuley 2016 McAuley et al. 2015)
consists of user reviews and item metadata from the consumer electronics category. Images represented using 4.0%6-dimensoonal features extracted by pre-trained CNN-based models (He & McAuley 2016). Textual information formed by integrating titles, descriptions, category labels. and brand attributes is encoded as 384-dimensional embeddings using Maatence-BERR) (Reimers & Gurevych/ 2019).
In this task of knowledge distillation for multi-modal recommender systems, we compare our method with three baselines. Specifically, we select two state-of-the-an knowledge distillation methods for multi-modal recommender systems: PromptMM (Wei et al. (2024) and TARec (Zhuang et al.) 2025). PromptMM introduces soft prompt-tuning. together with disentangled multi-modal list- wise distillation and modality-aware re-weighting mechanism. TARec proposes a seacher-assisted Wasserstein Knowledge Distillation framework that contains two-stage distillation to bridge the gap between the teacher and student. We also utilize HetComp (Kang al. 2023), the state-of-the-arr knowledge distillation method for general recommender systems as a baseline. Note that our method and HetComp rely solely the teacher's is logits for distillation, whereas PromptMM and TARec use both logits and embeddings. To ensure a fair comparison, we disable their embedding-leve. distillation components and retain all other parts of their methods.
23

---

Published as a conference paper at ICLR 2026
Table 12: Recommendation performance sequential recommendation task. The best results in boldface, and the best baselines are underlined. Improv.b denotes the relative improvement of RCE-KD the best bascline. LGCN stands for LightGCN. A paired t-lest is performed over independent runs for evaluating p-value (S 0.05 indicates statistical significance).
CiteULike
Gowalla
Yelp
Method
Recall
NDCG@
Recall@ 10
NDCG@10
Recall 10
NDCG@
Teacher
0.0077
0.0063
0.0310
0.0208
0.0153
0.0080
Student
0.0046
0.0032
0.0161
0.0081
0.0097
0.0058
CD
0.0067
0.0054
0.0274
0.0175
0.0140
0.0071
RRD
0.0069
0.0057
0,0281
0,0182
0.0139
0.0069
DCD
0.0073
0.0060
0.0292
0.0189
0.0142
0.0071
MF >MF
HetCCom
0.0072
0.0059
0.0289
0.0187
0.0146
0.0074
TARec
0.0070
0.0056
0.0285
0.0184
0.0140
0.0071
RCE-KD
0.0078
0.0063
0.0305
0.0201
0.0150
0.0077
Improv.
6.85%
5.00%
4,45%
6.35%
2.74%
4.05%
p-value
3.9-4
8.6e-4
5.5e-3
3.7e-4
6.7e-3
7.7e-4
Teacher
0.0083
0.0066
0.0401
0.0279
0.0167
0.0089
Student
0.0051
0.0040
0.0217
0.0154
0.0103
0.0064
CD
0.0066
0.0050
0.0349
0.0250
0.0144
0.0073
RRD
0.0068
0.0051
0.0354
0.0251
0.0146
0.0077
DCD
0.0070
0.00SS
0.0368
0.0261
0.0154
0.0082
LGCN-LGCN
lee
0.0071
0.0055
0.0363
0.0259
0.0148
0.0079
TARee
0.0068
0.0052
0.0366
0.0261
0.0157
0.0083
RCE-KD
0.0075
0,0060
0.0383
0.0269
0.0165
0.0887
Improv.b
5.63%
9.09%
4.08%
3.07%
5.10%
4.82%
p-value
9.6c-5
9.0c-4
3.1c-4
8.7c-4
7.2c-4
3.8c-4
Teacher
0.0091
0.0067
0.0325
0.0214
0.0159
0.0087
Student
0.0054
0.0042
0.0160
0.0086
0.0100
0.0067
CD
0.0073
0.0055
0.0250
0.0169
0,0133
0.0070
RRD
0.0079
0.0057
0.0241
0.0165
0.0129
0.0069
DCD
0.0083
0.0064
0.0269
0.0170
0.0143
0.0074
HctComp
0.0086
0.0066
0.0281
0.0174
0.0136
0.0070
TARec
0.0080
0.0059
0.0271
0.0171
0.0130
0.0067
RCE-KD
0.0092
0.0070
0.0303
0.0181
0.0154
0.0078
Improv.b
6.98%
6.05%
7.83%
4.02%
7.69%
5/41%
p-value
1.3e-3
3.7e-4
6.7e-4
5.5e-5
3,9e-3
1.
Teacher
0.0102
0.0072
0.0331
0.0217
0.0172
0.0098
Student
0.0063
0.0049
0.0164
0.0087
0.0110
0.0071
CD
0.0102
0.0067
0.0285
0.0151
0.0129
0.007S
RRD
0.0092
0.0065
0.0270
0.0157
0.0134
0.0078
DCD
0.0095
0.0069
0.0277
0.0162
0.0141
0.0082
HSTU~HHTT
HetComp
0.0099
0.0073
0.0282
0.0170
0.0149
0.0084
TARec
0.0093
0.0067
0.0271
0.0157
0.0140
0.0081
RCE-KD
0.0111
0.0078
0.0309
0.0184
0.0158
0.0990
Improu.
8.82%
6.85%
8.42%
8.24%
6.04%
7.14%
p-value
5.7e-4
8.9e-4
9.0e-5
2.0e-3
9.7e-4
4.1e-3
Teacher
0.0102
0.0072
0.0331
0.0217
0.0172
0.0098
Student
0.0046
0.0032
0.0161
0.0081
0.0097
0.0058
CD
0.0070
0.0053
0.0287
0.0188
0.0153
0.0079
RRD
0.0077
0.0059
0.0301
0.0200
0.0160
0.0091
DCD
0.0082
0.0061
0.0303
0.0200
0.0164
0.0091
HF
HetComp
0.0089
0.0065
0.0310
0.0207
0.0163
0.0089
TARce
0.0083
0.0062
0.0298
0.0197
0.0164
0.0093
RCE-KD
0.0097
0.0068
0.0329
0.0214
0.0173
0.0998
Improv.b
8,99%
4.62%
6.13%
3.38%
5.49%
5.38%
p-value
4.2c-4
9,3c-3
5.3c-4
9.2c-5
6.2c-3
7.4c-4
We treat the multi-mode teacher model BM3 (Zhou et al., 2023) as teacher backbones since
performs well in recent work (Zhuang et al. 2025). ). Per the custom of recent work (Wei et al. [0244] Zhuang et al. 2025). we use the ID-based models MF and LightGCN as students. This yields our teacher- Saacher-sstudent configurations: BM3 MF, BM3 -
Table 15 summarizes the experimental results. In all settings, RCE-KD significantly improves over the students and consistently outperforms all baseline methods. In most cases, method approaches the teacher's performance, remaining just slightly lower, or even slightly higher, while other methods farther behind. These results confirm that distillation approach remains effective when
24

---

Published as a conference paper at ICLR 2026
Table 13: Statistics of datasets with multi-modal item Visual (V), Acoustic (A), Textual (T) contents.
Dataset
Netflix
Tiktok
Electronics
Modality
V
T
V
A
T
V 4096
T 384
Feat. Dim.
512
768
128
128
768
User
43,739
14,343
41,691
Item
17,239
8,690
21,479
Interaction
609,341
276,637
359,165
Sparsity 99.919%
99.778%
99.960%
Table 14: Dimensions of models multi-modal recommendation task.
Dataset
Netflix
Tiktok
Electronics
Model
BM3
MF
LighGGCN
BM3
MF
LightGCN
BM3
MF
LightGCN
Dimension
400
20
20
200
20
20
200
20
20
Table 15: Recommendation performance on multi-moda recommendation task. The best results are in boldface, and the best baselines are underlined. Improv.b denotes the relative improvement of CCE-KD over the best baseline. LGCN stands for LightGCN. A paired t-test is performed over independent runs for evaluating y-value (S 0.05 indicates statistical significance).
Netflix
Tiktok
Electronics
T-S
Method
Recall
NDCG@20
Recall@20
NDCG@20
Recall
NDCG@20
Teacher
0.1913
0.0809
0.0707
0.0254
0.0361
0.0152
Student
0.1420
0.0498
0.0411
0.0153
0.0182
0.0073
HetComp
0.1783
0.0755
010659
0.0238
0.0320
0.0133
BM3-MF
PromptMM
0.1732
0,0728
0.0221
0.0297
0.0120
TARec
0.1792
0.0760
0.0230
0.0333
0.0139
RCE-KD
0.1884
0.0797
0.0702
0.0253
0.0347
0.0146
Improv.b
5.13%
4.87%
6.53%
6.30%
4.20%
5,04%
p-value
2.0e-3
5.9e-4
1.7e-3
9.7e-5
6.6e-3
2.8e-4
Teacher
0.1913
0.0809
0.0707
0.0254
0.0361
0.0152
Student
0.1507
0.0537
0.0562
0.0189
0,0211
0.083
HetComp
0.1811
0.0747
0.0629
0.0221
0.0317
0.0129
BM3-LLGCN
PromptMM
0.1779
0.0728
0.0603
0.0215
0.0323
0.0137
TARec
0.1834
0.0753
0.0651
0.0235
0.0341
0.0144
RCF-KD
0.1907
0.0804
0.0255
0.0359
0.0153
Improv.b
3.98%
6.77%
8.29%
8.51%
5.28%
6,25%
p-value
7.9e44
4.0e-3
9.4e-4
5.9e-3
7.3e-4
8.2e-5
transferring knowledge from multi-modal teachers to lighter ID-based students. Our experimental results demonstrate that the proposed method is not limited to distillation within ID-based models.
25


Reviews:
[{"review_id": "uMPbmrnTea", "reviewer": "Reviewer_xgs9", "summary": "This paper studies the cross-entropy (CE) loss in the field of recommendation knowledge distillation, and proposes a new adaptively-weighted distillation loss.\n\nWhile the topic is important and the paper presents a method that is conceptually reasonable, I have some concerns on the unclear motivation and technical contribution.  As a result, I initially assign a score of 4, but I could raise my score if my concerns have been well addressed.", "questions": "Please refer to weaknesses.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "This work has the following strengths:\n\n1.\tThis work studies on an important problem.\n\n2.\tThe proposed method appears to be reasonable.\n\n3.\tExtensive experiments on real-world datasets have been conducted to verify the efficacy of the proposed method.", "weaknesses": "However, this work also has some limitations:\n\n1.\tMy major concerns lie on the motivations. 1) Why is optimizing NDCG considered critical in the context of knowledge distillation? While NDCG is a standard metric for evaluating recommendation performance, it may not be an ideal loss function for distillation purposes. 2) Observation 4.5 is interesting, but it is derived solely from the CiteULike dataset, making the evidence less robust and limiting generalizability. 3) Assumption 4.3 imposes a relatively strong condition that may be difficult to satisfy. Is this assumption strictly necessary to establish the theoretical connection between CE and NDCG? Could weaker or more relaxed conditions be used instead? 4) There appears to be a gap between the theoretical analysis and the proposed method. Does the proposed method strictly satisfy Assumption 4.3? Can the proposed loss function tightly bound the NDCG metric as claimed?\n\n\n2.\tMy another concerns lie on the technical contribution: 1) The theoretical relationship between NDCG and CE loss has been explored in prior work (e.g., [a1]). Connecting NDCG and CE in the specific setting of knowledge distillation may not constitute a highly novel contribution unless unique theoretical challenges are clearly articulated. What are the distinctive theoretical hurdles addressed in this work, and what new insights are provided? 2) The proposed method appears to be a relatively simple distillation trick. While simplicity is not inherently undesirable and sometimes could be a good properties, a deeper theoretical justification is needed to explain why this strategy is particularly effective or elegant for this task. What are the inherent merits or underlying principles that make it well-suited here?\n\n3.\tI also have some minor concerns on the experiments: 1) The baselines are relatively dated (before 2023). Incorporating more recent state-of-the-art methods would strengthen the evaluation. 2) The experiments are conducted only on three datasets. Including more datasets would better. \n\n[a1] On the effectiveness of sampled softmax loss for item recommendation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aP3IEEbmc8", "reviewer": "Reviewer_PnSg", "summary": "This paper studies why Cross-Entropy (CE) loss performs poorly in knowledge distillation for recommender systems. The authors prove CE loss only works well when the item set includes the student model's top-ranked items, but this conflicts with the goal of distilling the teacher's top items. They propose RCE-KD, which splits the teacher's top items into two groups and applies different sampling strategies. Experiments show their method consistently outperforms existing distillation approaches across multiple datasets and model configurations.", "questions": "1. Why were these three datasets chosen? The Amazon dataset is a widely-used benchmark in recommendation research but is absent here. Could the authors justify this choice or include Amazon datasets in the evaluation?\n2. Why are standard sequential recommendation models like SASRec and GRU4Rec not included as baselines? These are established benchmarks, and their absence makes it difficult to assess the competitiveness of the proposed approach.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The paper provides rigorous theoretical analysis by establishing the connection between CE loss and partial NDCG in recommender system KD, identifying the critical closure assumption that determines when CE loss effectively bounds ranking performance.\n2. The paper is well-structured and easy to follow.\n3. This paper presents experimental results on multimodal recommendation scenarios, demonstrating their effectiveness.", "weaknesses": "1. The motivation for model compression in recommendation systems is questionable. Unlike CV/NLP models that require edge deployment, recommendation systems typically operate on cloud infrastructure where model size is rarely a bottleneck. The authors fail to provide compelling evidence that model size poses a significant problem in recommendation scenarios. Without clear justification for why compression is necessary in this cloud-based deployment context, the practical value of this work remains unconvincing.\n2. The teacher models used in this study are relatively modest in size, with the largest containing fewer than 300M parameters. This contradicts the fundamental premise of knowledge distillation, which typically involves compressing truly large, unwieldy models like LLMs or large vision transformers. With teacher models of this scale, standard optimization and deployment techniques should suffice, making the proposed distillation approach potentially unnecessary for the problem at hand.\n3. Current trends in recommendation system distillation focus on using large multimodal models as teachers to guide smaller ID-based models—a more practical scenario given the genuine computational overhead of multimodal architectures. However, this paper exclusively employs ID-only models for both teacher and student, which represents a less compelling use case. The authors should justify this choice or adopt the more realistic multimodal-to-ID distillation setting.\n4. Recent work [1] has demonstrated the effectiveness of Wasserstein distance for knowledge distillation in recommendation systems. However, the authors do not compare their approach against alternative distance metrics, including Wasserstein-based methods. \n\n\n[1] Zhuang, Ziyi, et al. \"Bridging the Gap: Teacher-Assisted Wasserstein Knowledge Distillation for Efficient Multi-Modal Recommendation.\" Proceedings of the ACM on Web Conference 2025. 2025.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EiLuxvfRWv", "reviewer": "Reviewer_TXd5", "summary": "The paper is about Knowledge Distillation (KD) of Recommender Systems (RS).\\\nCross-Entropy (CE) loss is the most widely adopted loss function for KD in other domains, however, it underperforms in RS (Figure 1).\\\nThe author shows that it is because we need to use all items for CE, to align the KD with NDCG.\\\nHowever, in practice, we cannot use all items for CE, we need to sample some items for the computational burden.\\\nTo this end, they propose to use top-ranked items both from the teacher and the student.", "questions": "Please refer to Weaknesses.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The motivation is sound\n- KD is an important task for RS, since the model size increases with the number of increasing users and items.\n- Also, CE is the most widely adopted loss function for KD in other domains.\n\n2. The proposed method is well-formulated\n- I like Section 4, where the authors show that full CE is the proper way to align KD with NDCG.\n- Also, we know that full-item CE is infeasible in practice. So they also present partial-item CE and some conditions for that.\n- Lastly, \n\n3. The paper demonstrates the superiority of the proposed method with experiments on real-world datasets.", "weaknesses": "1. Additional experiment result\n- A comparison including\\\n(1) vanilla CE\\\n(2) full-item CE\\\n(3) partial-item CE (top-ranked from Teacher)\\\n(4) partial-item CE (top-ranked from Student)\\\n(5) partial-item CE (top-ranked from Teacher or Student, i.e., the union of them)\\\n(6) RCE-KD\\\nwould be beneficial to clearly present the motivation/superiority of the proposed method.\n\n2. Minor comments\n- Figure 1 needs to include RCE-KD, to show that vanilla CE actually can be improved with a simple modification.\n- Citation format should be revised (\\cite{} vs \\citep{}).\\\nFor example, in the first sentence of Introduction, \"Recently, with the scaling law in recommender systems **Zhai et al. (2024)** being gradually discovered\"\\\nIt should be \"Recently, with the scaling law in recommender systems **(Zhai et al., 2024)** being gradually discovered\"\n- Typo in line 275: \"The first subset is the **interaction** between Q^T_u and Q^S_u\" -> **intersection**", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4DYf3wMk57", "reviewer": "Reviewer_irWM", "summary": "This paper proposed an improved version of cross entropy loss for knowledge distillation", "questions": "Can the proposed knowledge distillation method be applied to multi-modal recommender systems or just recommender system without modality features?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The theoretical claims seem interesting.\n2. The presentation of the paper is clear.\n3. Experimental results demonstrate that the proposed method can show performance improvement.", "weaknesses": "1. It will be better if more baselines specialized for knowledge distillation in recommender systems can be compared such as [1][2]. However, I think the paper be of interest even without these comparisons, and the comparison will strengthen the claims.\n2. The design of gamma update rule in Eq 10 seems somewhat ad-hoc. Maybe more principled method can be developed to balance this weight parameter.\n3. The theory part is interesting but might not be entirely novel. The fact that optimizing recommendation loss function is connected to the ranking metric NCDG [3] is relatively known in the community, and in my opinion the majority of knowledge distillation loss derivation in this paper is centered around this.\n\nReferences\n\n[1] PromptMM: Multi-Modal Knowledge Distillation for Recommendation with Prompt-Tuning\n\n[2] Unbiased knowledge distillation for recommendation\n\n[3] On Optimizing Top-𝐾 Metrics for Neural Ranking Models", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)