Learning Cascade Ranking as One Network
Authors: Yunli Wang
ZhenZhang
Zhiqiang Wang
Zixuan Yang
Yu Li
Jian Yang
Shiyang Wen
Peng Jiang
Kun Gai
Venue: ICML 2025 poster (2025)
Keywords: Cascade Ranking Optimization
Surrogate Loss Optimization
Multi-Stage Coordination
End-to-End Training
Recommendation Systems
Primary area: applications->everything_else
TL;DR: We propose LCRON, which improves cascade ranking systems by introducing an end-to-end training method that aligns with the overall system objective and enhances collaboration among stages.

Abstract: Cascade Ranking is a prevalent architecture in large-scale top-k selection systems like recommendation and advertising platforms. Traditional training methods focus on single-stage optimization, neglecting interactions between stages. Recent advances have introduced interaction-aware training paradigms, but still struggle to 1) align training objectives with the goal of the entire cascade ranking (i.e., end-to-end recall of ground-truth items) and 2) learn effective collaboration patterns for different stages. To address these challenges, we propose LCRON, which introduces a novel surrogate loss function derived from the lower bound probability that ground truth items are selected by cascade ranking, ensuring alignment with the overall objective of the system. According to the properties of the derived bound, we further design an auxiliary loss for each stage to drive the reduction of this bound, leading to a more robust and effective top-k selection. LCRON enables end-to-end training of the entire cascade ranking system as a unified network. Experimental results demonstrate that LCRON achieves significant improvement over existing methods on public benchmarks and industrial applications, addressing key limitations in cascade ranking training and significantly enhancing system performance.

Learning Cascade Ranking as One Network
Yunli Wang' Zhen Zhang'
Zhiqiang Wang'
Zixuan Yang' Kun Gai
Yu Li
Jian Yang? Shiyang Wen
Peng Jiang'
Abstract
Westing Daige Searted
Selveted
Cascade Ranking is prevalent architecture in large-scale top-k selection systems like recom- mendation and advertising platforms. Traditional training methods focus on single-stage optimiza- tion, neglecting interactions between stages. Re- cent advances have introduced interaction-aware training paradigms, but still struggle to 1) align training objectives with the goal of the entire cas- cade ranking (i.e., end-to-end recall of ground- truth items) and 2) learn effective collaboration patterns for different stages. To address these challenges, we propose LCRON, which intro- duces novel surrogate loss function derived from the lower bound probability that ground truth items are selected by cascade ranking, ensuring alignment with the overall objective of the sys- tem. According to the properties of the derived bound, we further design an auxiliary loss for each stage to drive the reduction of this bound, leading to a more robust and effective top-k se- lection. LCRON enables end-to-end training of the entire cascade ranking system unified network. Experimental results demonstrate that LCRON achieves significant improvement over existing methods on public benchmarks and in- dustrial applications, addressing key limitations in cascade ranking training and significantly en- hancing system performance.
Golechee
Moveniin Shace
Figure I. A typical cascade ranking architecture, including four stages: Matching, Pre-ranking, Ranking, and Mix-ranking, The red points represent the ground truth for the selection.
utilization and performance through a multi-stage, funnel- like filtering process. A typical cascade ranking system comprises multiple stages, including Matching, Pre-ranking, Ranking, and Mix-ranking, as illustrated in Figure 1. The objective to select ground truth items (referred to the red points in Figure 1) as the final outputs.
Early traditional training approaches for cascade ranking systems often optimize each stage in isolation, construct- ing samples, designing learning objectives, and defining proxy losses separately (Crammer & Singer, 2001; Burges et al., 2005; Li al., 2007: Covington al., 2016; Wang et al., 2018 8; Ma al., 2018; Sheng et al., 2023; Wu et al., 2024). This fragmented approach overlooks the interactions between stages, leading to suboptimal alignment with the overall system objective. Specifically, two key challenges arise: 1) Misalignment of Training Objectives: Cascade ranking aims to collaboratively select all relevant items from the ground truth across multiple stages. However, each stage's ' learning targets optimized by pointwise pairwise losses are often more strict than the collaborative goal of cascade ranking. This misalignment may lead to ineffi- ciency, particularly in recommendation scenarios where the model capacity is typically limited. 2) Lack of Learning to Collaborate: During online serving, different stages in a cascade ranking system will interact and collaborate with each other. Efficient interactions and collaborations are crucial for improving the overall performance of the cascade ranking system. For example, Retrieval model can preemptively avoid items that the ranking model tends to overestimate, while the Ranking model can accurately identify ground-truth items from the recalled set. However, when models for different stages are trained independently,
1. Introduction
Cascade ranking has emerged prevalent architecture in large-scale top-k selection systems, widely adopted in indus- trial applications such as recommendation and advertising platforms. This architecture efficiently balances resource
Kuaishou Technology, Beijing, China "Beihang Univer- sity, Beijing, China IIndependent, Beijing, China. Correspon- dence to: Yunli Wang, Jian Yang <wangyunli Staishou.com. ilaya@ buaa.cdu.cn..
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Learning Cascade Ranking as One Network
they lack the ability to learn these interactions and collabo- rations, which may lead to degraded testing performance.
by ARF (Wang et al., 2024), we use differentiable sort- ing techniques as the foundation of our surrogate losses. singte inherits the main idea of the Reetoz loss in ARF while addressing its limited use of the soft permutation ma- trix generated by differentiable sorting. Finally, we combine different losses of LCRON in a UWL (Kendall et al., 2018) form to reduce the number of hyperparameters, enhancing its practicality and robustness.
Recent works have attempted to address these challenges. ICC (Gallagher et al., 2019) is an early study to partially ad- dress challenges I) and 2) by fusing the predicted scores of different stages and optimizing them jointly through Lamb- daRank (Burges, 2010). However, its stage-wise interaction remains unidirectional, restricting the learning of bidirec- tional collaboration. It also suffers from limited sample space, as it focuses only on exposed items. RankFlow (Qin et al., 2022) introduces an iterative training paradigm that dynamically determines the training samples for each stage by its upstream stage to address challenge 2). Although RankFlow reports significant improvements over both in- dependent training and ICC, its iterative training process may lead to increased complexity and instability during training. FS-LTR (Zheng et al., 2024) tackles challenge 2) by learning online patterns of interactions and collab- orations with full-stage training samples and outperforms RankFlow. However, neither RankFlow nor FS-LTR ex- plicitly addresses the misalignment of training objectives. ARF (Wang al., 2024) emphasizes that learning targets should align with the objective of cascade ranking and pro- poses a new surrogate loss for Recall based on differentiable sorting. However, ARF focuses on single stage and can- not fully address challenges 1) and 2). Additionally, ARF does not fully utilize the information of the soft permutation matrix, further limiting its performance.
To verify the effectiveness of our method, we conduct exten- sive experiments on both public and industrial benchmarks. We conduct public experiments on RecFlow (Liu et al., 2025), the only public benchmark based on a real-world rec- ommendation system that contains multi-stage samples of cascade ranking. The results on the public benchmark show that LCRON outperforms all baselines under the stream- ing evaluation (where for any day t as test, its training data comes from the beginning up to day t - 1), indicat- ing the effectiveness and robustness of our method. The ablation study highlights the role of different components of LCRON. The industrial experimental results show that L.CRON consistently outperforms the best two baselines in public experiments on end-to-end Recall. We further con- duct an online A/B experiment in a real-world advertising system. Compared to FS-LTR, LCRON brings about 4.10% % increase in advertising revenue and a 1.60% in- crease in the number of user conversions, demonstrating that our approach has significant commercial value for real-world cascade ranking systems.
To the best of our knowledge, no existing approach si- multaneously addresses both challenges, highlighting the need for more comprehensive solution.
2. Problem Formulation
We first introduce the formulation of a cascade ranking system. In such systems, a large initial set of candidate items is processed through a series of consecutive filtering stages to identify the most relevant or optimal results efficiently, Each stage applies a specific model M; to evaluate and select subset of items from the input set for the next stage. Let T denote the total number of stages in the cascade, with an initial candidate inventory size of go. For any given stage i, we define the sample space of input candidates as Qi-1, which contains Φ:-1 items. After processing by the i-th stage model, the output consists of q, selected items. Note that qi typically decreases as i increases.
To address these challenges, we propose LCRON (Learning Cascade Ranking as One Network), introducing two types of novel surrogate losses. First, we propose a novel surro- gate loss L.2e. which is the lower bound of differentiable approximation of the survival probability of ground-truth items through the entire cascade ranking system. Lez- di- rectly aligns the learning objective with the global goal of the cascade ranking system, while naturally learn- ing effective collaboration patterns for different stages. However, optimizing Leze alone may lead to insufficient su- pervision for individual stages, especially when the survival probability at a particular stage is close to O. In addition, we derive the lower bound and find that the tightness of the bound is highly related to the consistency of different stages. Therefore, we design a surrogate loss Lsingic towards the Recall of each single stage, which enforces the model to distinguish ground-truth items from the entire candidate set rather than the filtered subset from upstream stages. T.singte can tighten the theoretical bound of Leze and provides additional effective supervision when the survival proba- bility of ground -tuth items in Leze is close to O. Inspired
The models in the cascade ranking system are typically trained using specific paradigms, with training samples de- rived from the system itself. To rank the items, we define F&a (S) as the ordered terms vector of set S sorted by the score of model M in descending order, and (S) K] as the top K terms of (S). With the set of trained models (M, s S T). the final output set C.Sout of the cas- cade ranking system can be obtained through a sequential filtering operation. This process can be formulated as:
2

---

Learning Cascade Ranking as One Network
et al., 2024). While these methods are widely adopted, they often fail to align training objectives with the global goal of cascade ranking and overlook the interactions between different stages.
Coout Fm (oo)l: ml M: qr)
(1)
We define the ground truth set ot as the set of items con- sidered relevant or optimal based on user feedback or expert annotations. The goal of training paradigms for cascade ranking is optimize the Recall of CS2t using training data collected from the system, which can be formulated as Eq 2. where @r is the size of CSSuut. IC is the size of C'Sgt, and K. < qr. Here, 1.) is the indicator function that returns 1 if the condition is true and O otherwise.
Recently, several works have attempted to address these challenges by proposing interaction-awary train- ing paradigms to jointly train the entire cascade ranking system. ICC (Gallagher al., 2019) fuses the predictions of different stages and optimizes the fusion score using LambdaRank (Burges, 2010). However, it suffers from limited sample space and unidirectional stage-wise wise interac- tion. RankFlow (Qin et al., 2022) introduces an iterative training paradigm. Each stage is trained with samples gen- erated by its upstream stage and distills knowledge from its downstream model. While RankFlow reports significant improvements over ICC, its iterative training process may increase complexity and instability. FS-LTR (Zheng et al., 2024) argues that each stage model should be trained with full-stage samples. It trains the cascade ranking system using full-stage samples and LambdaRank loss, achieving better results than RankFlow. However, both FS-LTR and RankFlow fail to fully align with the global goal of cascade ranking. Another approach, ARF (Wang et al., 2024), em- phasizes the importance of aligning learning targets with the Recall of cascade ranking and proposes surrogate losses based on differentiable sorting to optimize Recall. How- ever, ARF focuses only on a single stage and assumes that downstream models are optimal, limiting its applicability. In this paper, we propose LCRON for end-to-end alignment with the global objective of cascade ranking, addressing the aforementioned challenges.
(iitem,ECCap M ECS,))
Recall@K@@q
((Se))
Most previous works design training sets and methodologies for different stages separately, while few attempt to build universal training paradigms, as detailed in Section 3. In this paper, we propose an all-in-one training paradigm for cascade ranking systems, which addresses the limitations of previous approaches and is detailed in Section 4.
3. Related Work
3.1. Learning Methodologies for Cascade Ranking
Cascade ranking (Wang et al., 201 I; Li et al., 2023) is widely used in online recommendation and advertising systems to balance performance and resource efficiency. It employs multiple models with varying capacities to collaboratively select top-k items from the entire inventory. Early traditional training approaches for cascade ranking systems often opti- mize each stage separately, with distinet training sample or- ganization, learning objectives, and surrogate losses. There are three common learning tasks in cascade ranking systems: 1) probability distribution estimation (e.g., PCTR), which aims to optimize the accuracy of probability estimation and order and uses surrogate losses such as BCE, BPR, or their hybrid (Zhou et al., 2018; Ma et al., 2018; Sheng et al., 2023; Huang et al., 2022). The training samples include both pos- itive and negative samples after exposure. 2) continuous value estimation (e.g., video playback time, advertising payment amount), which usually employs surrogate losses such as ordinal regression to optimize the model (Niu et al., 2016; Lin et al., 2023). It focuses on learning continuous values after specific user behaviors occur (c.g., viewing time after video exposure). 3) learning-to-rank, which more commonly used in the retrieval stage of cascade ranking systems, with the entire or partial order of the ranking stage as the ground truth. It leverages methods such as pointwise, pairwise, and listwise approaches (Crammer& & Singer, 2001; Li et al., 2007: Covington et al., 2016; Wang et al., 2018: Thonet et al., 2022; Tang et al., 2022; Wu al., 2024; Wang
In addition, some recent works on cascade ranking systems offer complementary perspectives that could potentially be integrated with joint training approaches. FAA (Li et al., 2023) focuses on feature consistency of cascade ranking by aligning stage-wise representations via attention, while LCRON complements this approach by introducing end- to-end loss functions for global optimization. SRCR (Za- mani et al., 2022) focuses on improving two-stage cascade systems but employs a non-lcarnable component (BM25) for retrieval and employs BERT (Devlin et al., 2019) for ranking and just jointly optimizes the number of retrieved documents N and the ranking model. This differs from LCRON, which enables end-to-end joint learning across fully learnable cascade stages. These two approaches are conceptually complementary, suggesting that future work could explore joint optimization of both model parameters and system-level decision variables such as retrieval quota.
3.2. Differentiable Techniques for Hard Sorting
Differenniable sorting techniques provide continuous relax- ations of the sorting operation, enabling end-to-end training
3

---

Learning Cascade Ranking as One Network
within deep learning frameworks. Early work by Grover al.9) introduced NeuralSort, which approximates hard sorting by generating unimodal row-stochastic matrix. Cuturi et (2019) further formulated the sorting process as an optimal transport problem with entropic regularization, enabling smooth approximations of ranks and sorted values. Subsequently, Prillo & Eisenschlos(2020) proposed Soft- Sort, a more lightweight and efficient approach for differen- tiable sorting. Recent advances, such as those in (Petersen et al., 2021; 2022; Sander et al., 2023; Cuturi et al., 2019), have further improved the performance and efficiency of differentiable sorting operators. These techniques have been widely adopted in various domains, including computer vi- sion (Grover et al., 2019; Blondel et al., 2020; Sander et al., 2023), online recommendation and advertising (Swezey et al., 2021; Pobrotyn & Bialobrzeski, 2021; Wang et al., 2024). In this paper, we leverage differentiable sorting oper- ators to construct the surrogate losses of I. LCRON. It employs the soft permutation matrix produced by differentiable sort- ing methods. Thus, those differentiable sorting methods that do not produce a soft permutation matrix, such as Fast- Sort (Blondel et al., 2020) and OT (Cuturi et al., 2019), can not be the foundational component of LCRON.
== (u, ((itemy. [ N))
3
(3)
Mi ((itemmy) v.) 0 S j and item, E 2.}
1-0
where n, is the number of samples drawn from Q, and N = no + ny + m2 + R33. In LCRON, the construction of full-stage training samples is a fundamental component of our approach. While specific sampling strategies and hyper- parameters (e.g. n4) may influence the performance, their detailed analysis is beyond the scope of this paper. In both public and online experiments, all methods are evaluated on the same sampled datasets to ensure a fair comparison.
Let R3 denote the descending rank index (where a higher value indicates better rank) for the pair (u, ilemy) within its sampled stage. The label yj is determined by both the stage order and the rank within the stage. Specifically, for any two pairs (u, item,) and (u, item,). the label y, and yo satisfy the following condition:
[w > us) 1(S > S,) V (1(S: S)) > R2)) (4)
where S, S, indicates that item, belongs to a later stage than itemy, and R₃ > R2 indicates that item, has a higher rank than item, within the same stage.
4. Methodology
In this section, we introduce our proposed method LCRON, which is the abbreviation of "Learning Cascade Ranking as One Network". In Section 4.1, we give the organization of full-stage training samples, which mainly follows Zheng et (2024).. In Section 4.2, we introduce a novel loss Taze that directly optimizes the lower bound of a differentiable approximation of Equation 2, ensuring alignment with the overall objective of cascade ranking. In section 4.3, we discuss the limitations of Les and introduce Lsingle as an auxiliary loss for each single stage to tighten the theoretical bound of Le and provide additional effective supervision.
4.2. End-to-end Surrogate Loss for Top-K Cascade Ranking
Our goal is design efficient surrogate loss that aligns with the Recall Kg@q metric of the entire cascade ranking system. We can transform the problem of optimizing the Recall@K.@q of cascade ranking into a survival probabil- ity problem, allowing the model to estimate the probability that the ground truth is selected by the cascade ranking. Note that K. the size of CSS.t. Let M,(D) E RIxN de- note the prediction vector of M, on the training data D. Let PM, represent the probability vector of each ad in D being selected by the cascade ranking for top-q: selection. qi is the quota of M,, as mentioned in Section 2. The sum of M, should be q. Let the survival probability vector output by the system be P and let the label y be a binary vector indicating whether it the ground truth. We can employ a cross-entropy loss of P&p and y to optimize the probability of the ground truth being selected by the cascade ranking as shown in Equation 5:
4.1. Sull-Stage Training Samples of Cascade Ranking
For simplicity, we illustrate our method using a two- stage cascade ranking system (T 2), which can be readily extended to systems with additional stages. We downsample items from all stages of the cascade ranking system to construct full-stage training samples, primarily following FS-LTK (Zheng et al., 2024).
Let M1 and M2 denote the retrieval and ranking models, respectively. The input space of Mi, the input and out- put spaces of M2 are Qo. 21. and Q2. We denote the ground-truth set CgSt, also referred to as Q3. For given impression, let represent the user information and item, represent the item information. Let y, denote the label for the pair (u, item,) and y = (y1, ......, )N). The training samples for one impression can be formulated as Eq 3:
CF(P - ((vin((PPg)))
(5)
+ (1-1))m(11- -
Let E (0, 13. denote the sampling result from pot
M, and
4

---

Learning Cascade Ranking One Network
let Px represent the probability of sampling л. Then, Px can be formulated as Eq 6:
selection obtained through hard sorting (represented
binary posterior observation, i.e., 1/0). To enable gradient- based optimization, we can relax M (g:) into the stochastic PM, and optimize it via maximum likelihood. Specifically, this can be achieved by relaxing the hard permutation matrix P (which sorts items in descending order) into a soft permu- tation matrix P via differentiable sorting techniques (Grover et al., 2019; Prillo & Eisenschlos, 2020; Petersen al., 2021). Differentiable sorting methods typically generate P as row-stochastic matrix (each row sums to 1) by apply- ing row-wise softmax with temperature parameter As T - 0, P converges to the hard permutation matrix P.
H M, );
P EsciNnIs =r I (M,))
(6)
Then PCs can be expressed as Eq 7:
(PMs C T) PCs = BRNPt (T,, (/.PMM2)(10 (1, PM, M2
where P and Cs are vectors. ( denotes dot product, and @ denotes element-wise product. 1 represents vector where all elements are Due to the intractability of directly optimizing Pes caused by sampling and integration opera- tions, we aim to find an approximate surrogate for CCs- We define Pog II: PM It can be shown that Pas serves as an lower bound for paa as demonstrated in Eq. 8, since < (T, MMa (11, ) S always holds:
Specifically, let Pi,, E [0, be the soft permutation matrix for model Mi, where (P),),, ),. represents the soft probability that item &o is ranked at position j. Then, can formulate the top-q, selection probability P and the end-to-end loss Leze of the cascade ranking system as Eq 10 and Eqq 11:
Σ3=1 (P) PMm)s:
M.
(10)
8 8p(E ,- (P) )4:))
MM O P0s = ExNPt (T, (T,P (1, PM2
(P )is
2 BaN PM, O
Lize - Σ Ullmn
(8)
① OpplE. (Pta (Pin)en)) )a,))
(11)
pac
Σ (P) M.
M.
Σ - u1
III 2 spl Pi Ma,ee. )..))
where ب. represents the element-wise division operator. In Eq 10, we perform an element- wise division to ensure that the probability values are normalized, as some differen- tiable sorting operators cannot guarantee column- nor- malization, such as NeuralSort (Grover et al., 2019) and SoftSort (Prillo & Eisenschlos, 2020). "sp" denotes that a stop gradient is needed during training. The end-to-end loss Le in Eq. 11 directly optimizes the joint survival probability through all cascade stages.
The next question is how to obtain a differentiable M, enabling to optimize P To achieve this, introduce the permutation matrix as the foundation of our approach. For a given vector I and its sorted counterpart y. there exists a unique permutation matrix P such that = Pr. The elements of P are binary, taking values of either 0 or Let Pl) denote the permutation matrix that sorts the vector (.) in descending order. Specifically, (P))., 1 indicates that zj is the i-th largest element in x.
Traditional losses often impose strict constraints, which be- come problematic when model capacity is insufficient to satisfy all constraints. In contrast, L.ze directly aligns the learning objective of the goal of cascade ranking, allow- ing models to prioritize critical rankings while tolerating minor errors in less important comparisons. In addi- tion, when stage assigns a low score to a ground-truth item, Le not only optimizes that particular stage but also encourages other stages to improve their scores for the same ground-trut truth item. This mechanism enables an efficient collaborative pattern to be learned across stages, enhancing the overall survival probability of ground- truth items in the cascade ranking system. Importantly, this collaboration is bidirectional, overcoming the limitation of
Using the permutation matrix, the top-k elements of D selected by Mi can be formulated as:
& Mt(K) = Σ (Ptals
(9)
where (k) E RR 1xn is a binary vector indicating whether each item is selected by the model, and (Pi)); represents the j-th row of the permutation matrix for model M.
is evident that P&; can be interpreted as the distribution of M (g:). where M (q:) denotes the deterministic top-q:
5

---

Learning Cascade Ranking as One Network
ICC (Gallagher et al., 2019), which only allows unidirec- tional dependency from one stage its pre-stages (e.g.. from the ranking stage to the retrieval stage).
Table 1. Dataset Statistics.
Stage
Users
Impressions
Itens per impression
the range of labels
rank pos
38,195
6,062,344
10
[1.20]
rank_neg
38,193
6,06223348
10
[21,23]
4.3. Auxiliary Loss of Single Stage for Tightening the Bound
coarse nog
38,193
6,062,348
10
[22,22]
premannneg
38,193
6,062,348
10
[23,23)
Although Leze directly optimizes the joint survival probabil- ity of ground-truth items through the entire cascade ranking system, it may suffer from two weaknesses: 1) Leze is de- rived as a lower bound of the joint survival probability (see Section 4.2), the tightness of this bound could influence the overall performance, but Leze can not optimize this bound itself, 2) it may suffer from insufficient supervision when the survival probability at particular stage is close to O. This issue is particularly pronounced during the initial training phase, where if all stages assign low scores to ground-truth items, the gradients across all models would be small. This situation could make it difficult to properly warm up the models, resulting in slow convergence or suboptimal learn- ing performance.
Leze 2a
L
Lingte 2,32
M,
L Ma
single
+
+ (002((a87))
(13)
272
5. Experiments
5.1. Experiment Setup
We conduct comprehensive experiments on both public and industrial datasets. We conduct public experiments to ver- ify the effectiveness of our proposed method and perform ablation studies along with in-depth analysis. We conduct online experiments to study the impact of our method on real-world cascade ranking applications. Here we mainly describe the setup for public experiments, and details of online experiments are described in section 5.5.
According to Eq 8, the bound depends on the magnitude of 22 and the equation holds as 22 approaches 1.
(T.PM,
Public Benchmark. We conduct public experiments based on RecFlow (Liu et al., 2025), which, to the best of our knowledge, is the only public benchmark that collects data from all stages of real-world cascade ranking sys- tems. RecFlow includes data from two periods (denoted as Period I and Period 2), spanning 22 days and 14 days, respectively. While Period primarily designed for studying the distribution shift problem in recommendation systems, which is beyond the scope of this paper, we focus on Period I data as our testbed. To train the two-stage cas- cade ranking, we adopt four stages of samples: rank rank_neg, coarse_neg, and prerank.neg. Table 1 sum- marizes the dataset statistics.
Since Ex-P, (T) = PM, it is evident that optimizing the
difference between (D) and M2( can help tighten the bound. We provide detailed analysis in appendix A.
To address these issues, we propose L for each sin- gle model, which is shown in Eq 12. Lingle provides the same supervision for both retrieval and ranking mod- els, thereby optimizing the consistency of M1(M) and M2(D) and leading to a tight bound of Leze- L forces each model to distinguish mround-truth items from the entire inventory, offering effective yet not overly strict additional supervision for each model. This helps address the insufficient supervision situation that may occur in the Leze loss during the initial training phase. Furthermore, unlike LRetaz in ARF, which uses only the top-m rows of the soft permutation matrix, Laingle imposes supervision signals over the entire soft permutation matrix. This pro- vides more comprehensive supervision information and is expected to facilitate better model learning.
singte
Cascade Ranking Setup. We employ a typical two- stage cascade ranking system as the testbed, utilizing DIN (Zhou et al., 2018) for the ranking model and DSSM (Huang et al., 2013) for the retrieval model. The retrieval and ranking models are completely parameter- isolated, ensuring no parameters are shared between stages. This design eliminates potential confounding ef- fects from parameter sharing, enabling a fair comparison between different methods. In this setup, the retrieval model selects the top 30 items, and subsequently, the ranking model chooses the top 20 items out of these 30 for "exposure". Each impression contains 10 ground-truit truth items, referred to rank pos, which serve as the ground truth for exposure evaluation. This setup aligns with the data structure of the benchmark, ensuring evaluation con- sistency with real-world cascade ranking scenarios.
Σ ugln( E (Ptt)) 2 1p(E 11 M.MMM. (Pt.)es) 1a:))
L L,M, single
=
3
(12)
ES= (Pid)s
Σ(1 - 111-1))m(1
2 Sp (PM
Inspired by ARF (Wang et al., 2024), we simply employ UWL (Kendall et al., 2018) to balance the Leze and Liingle to reduce the number of hyper--pare parameters. For the two- stage cascade ranking, the final loss is formulated as Eq 13, where o, 3 and 7 are trainable scalars.
Evaluation. We employ Recall@ defined in Equa- tion 2 as the golden metric to evaluate the overall per-
6

---

Learning Cascade Ranking as One Network
formance of the cascade ranking system. Corresponding the cascade ranking and public benchmark setup, the and k for the evaluation are 20 and 10, respectively. In order to explore the impact of different baselines on model learning at different stages, we also evaluate the Recall@k@m and N DCG'G of each model on the en- tire inventory of candidate items as auxiliary observation metries for analysis.
5.2. Competing Methods
We compare our method with the following state-of-the-art methods in previous studies.
Binary Cross-Entropy (BCE). We treat the "rank pos" samples as positive and others as negative to train a BCE loss. The retrieval model is trained with all stages sam- ples in Table 1. The ranking model is trained with only rank-pos and rank neg samples, following the classic set- ting of cascade ranking systems. The rank ppss is regarded as positive samples, and others are regarded as negative samples. It is denoted as "BCE" in the following.
Following the mainstream setup for evaluating recommen- dation datasets, we use the last day of Period 1 as the test set to report the main results of our experiments (Sec- tion 5.3), while the second-to sooondd-tt-lstttt day serves as the valida- tion set for tuning the hyperparameters (e.g., the tempera- ture parameters of ICC, ARF, and LCRON; the a of Rank- Flow; the top-k and smooth factor of FS-Lambdal.oss .oss) of different methods. Considering that industrial scenarios commonly employ streaming training. we further evaluate the performance of different methods by treating each day as a separate test set (in Section 5.4). In this setting, when day t is designated the test set, the corresponding training data encompass all days from the beginning of Period 1 up to day 1 1-1.
rank_neg
ICC. It's an early study for joint learning the models of cascade ranking (Gallagher et al., 2019), which fuse multi-stage predictions and optimize them by Lamb- daRank (Burges, 2010).
RankFlow. Qin et (l022) propose RankFlow, a method that iteratively updates the retrieval and ranking models, achieving better results than ICC (Gallagher et al., 2019). The training sample space for the ranking model is de- termined by the retrieval model, and the ranking model's predictions are distilled back to the retrieval model.
LCRON Setup. We employ NeuralSort (Grover et al., 2019) as the differentiable sorting operator, aligning with the baseline method ARF to ensure a fair comparison. We tune the hyper-paramene controls the temperature of NeuralSort. We set 91 and 92 in Les. to 10 during training. Although this value does not exactly match the quotas of the cascade ranking setup, it represents trade-off between gradient optimization stability and maintaining the interpretability of the loss due to the properties of the permutation matrix. A larger qi is more likely to lead to gradient conflicts during training. We further provide a detailed discussion of this limitation in Section E.
FS-LTR. Zheng al.(2024) propose method to train all models in cascade ranking system using full-stage training samples and teaming-tto-rnk surrogate losses, achieving better results than RankFlow (Qin et al., 2022). Since Zheng et al. primarily focuses on the orga- nization of training samples rather than the design of surrogate losses, we design two representative variants to cover a wide spectrum of LTR losses. The first vari- ant, FS-RankNet, utilizes the RankNet (Burges al., 2005) loss, a classic pairwise learning-to- to-ringgttt-rakk method. The second variant, FS-LambdaLoss utilizes the Lamb- dal Loss (Wang et al., 2018) loss, an advanced listwise LTR method. These variants represent two fundamental paradigms in LTR, ensuring a comprehensive comparison.
T on the validation set, which
Implementation Details. We utilize the training pipeline and implementations of DSSM (Huang et al., 2013) and DIN (Zhou al., 2018) models provided by the open- source code from (Liu et al., 2025), and we primarily focus on implementing the training loss functions of base- lines and our method. The Multi-layer Perceptron (Rosen- blatt, 1958) of the user and item towers in DSSM are set to be [128, 64, 32]. The architecture of DIN's MLP is [128, 128. 32, 1]. All offline experiments are imple- mented using PyTorch 1.13 in Python 3.7. We employ the Adam optimizer with learning rate of 0.01 for training all methods. Following the common practice in online recommendation systems (Liu et al., 2025; Zhang et al., 2022), each method is trained for only one epoch. The batch size is set to 1024. The source code of our public experiments is publicly available?.
ARF. is designed to adapt to varying model capacities and data complexities by introducing an adaptive target that combines "Relaxed" loss and "Global" loss (Wang et al., 2024). We use ARF loss train both the retrieval and ranking models. To further improve ARF, we intro- duce ARF-v2 as an enhanced baseline, which replaces the relaxed loss of ARF with our proposed Leingle (Sec- tion 4.3). We set the hyper-parameters corresponding to the q of the cascade ranking system. For the retrieval model, we set the parameters m and k: to 30 and 10, re- spectively. For the ranking model, we set these parameters to 20 for m and 10 for k.
5.3. Main Results
Table presents the main results of the public experiments conducted on RecFlow. LCRON significantly outperforms
7

---

Learning Cascade Ranking as One Network
Table 2. Main results of public experiments on RecFlow. Fach method was run 5 times, and the results are reported as meantstd. indicates the best results. Bold numbers indicate that LCRON shows statistically significant improvements over the baselines, as determined by a t-test at the 5% significance level. Note that the Recall @ 10020 of Joint is the golden metric for the whole cascade ranking system. The test set is the last day, with the remaining data used for training.
Table 3. Ablation study of I.CRON. * indicates the best results. Each method was run 5 times, and the results are reported as mean std. Bold numbers indicate that LCRON shows sta- tistically significant improvements over each ablation model, as determined by a t-test at the 5% significance level. The test set the last day, with the remaining data used for training.
Metreshierns
Recate 103-0)
Recats 100
NorrasaT
Recame TERF W
00 1090
63757 : 10999
1330 10300
-
0
000033
07235 10.000
003933 = 000007
6:7135 B0000
00051
(1736
10999 10006
10709
Socel 11m
Kacall
SDC05
Rocel
BC5
0,83560 0005
0.3410e50007
0.N135d 0001
059006600004
0,7150e0 0015
ICC
0.8 18823
0.9108804903
c.999c00 18313
Ronkshow
0.861740 1007
0786144 1997
0750843-0008
0.863943:9366
0.717460 0010
CA85440 0931
0.955-d 1006
0.977 -0 (035)
D477044-1905
0:708745 0933
Methods
PS-Lambehi.co
ARM
0.85588) 0016
0:9084011800
0.8660820005 8
036804211907
0,733640 8027
0/601140 18277
0.007 2010000
0.W61 10:0005
0.71902000077
0:5177401100
0.55
ICC
ARF-2
0.667600
0.867963/0039
0.7259e0
0.71526000288
rankflow
LCRON (00n)
08735508866
038755510004
0033
671311690009
0.92
F5 Raniaeet
ANF
Brokto-end Recaill
ARF V2
FS L5-Lambdalass
all baselines on the end-to-end (joint) Recall, demonstrating its effectiveness in optimizing cascade ranking systems. No- tably, while LCRON does not dominate all individual stage metrics (e.g., the Recall of the Ranking model), it achieves substantial improvements in joint metries, highlighting the importance of stage collaboration. This aligns with our de- sign philosophy: LCRON fully considers the interaction and collaboration between different stages, enabling significant gains in joint performance even with modest improvements in single-stage metrics.
0.50
LCRON
0.70
0.14
0.72
20 32 14 Day
16
2926
Figure 2. The evaluation results of different methods on RecFlow, in streaming manner.
The results also reveal interesting insights about the base- lines: 1) While PS-LambdaLoss shows strong performance, FS-RankNet performs poorly under the same sample orga- nization. This indicates that the choice of sarming-to-rane methods plays critical role in optimizing cascade ranking systems. 2) ARF-v2 outperforms ARF in all metrics, sug- gesting that Lsingle effectively mitigates the disadvantages of the Rettz loss in ARF, as discussed in Section 4.3.
reflect real-world industrial scenarios, where models are typically trained in an online, streaming manner. To provide a more comprehensive evaluation, we conduct additional experiments under a streaming training setup. where each day is treated as a separate test set. Specifically, when day t is designated as the test set, the training data encompass all days from the beginning up day 1- The results, shown in Figure 2, reveal two key observations: 1) In the initial phase (first 10 days), LCRON exhibits rapid convergence, matching the performance of ARF-v2 and significantly out- performing other baselines. 2) In the later phase (last 12 days), LCRON not only surpasses all baselines but also demonstrates a widening performance gap over time, indi- cating its superior adaptability and robustness in long-term industrial applications. These findings underscore LCRON's ability to achieve both fast convergence and sustained per- formance improvements, making it a practical and effective solution for real-world cascade ranking systems.
5.4. In-depth Analysis
We first conduct an ablation study to evaluate the contribu- tion of each component in LCRON. Specifically, we analyze the impact of Leze and Lsaggl by removing them individu- ally. The results are summarized in Table 3. The ablation essen-
study demonstrates that both Leze and Lyingle tial components of LCRON. While their removal may not lead to statistically significant changes in all the individual ranking or retrieval metrics, the joint evaluation metric con- sistently shows a statistically significant improvement. This suggests that the interaction effects between the two stages of the cascade ranking system are optimized more effec- tively with the collaboration of Leze and Lsingle, enhancing the overall performance.
Furthermore, we investigate the impact of manually tuning the weights of Leze and Lanngl LCRON. While the UWL formulation designed reduce hyperparameter tuning costs, it does not guarantee optimal performance theoret- ically. Table 4 shows the results of fixing L,ze to 1 and varying the weight of Leingle- We observe that different weight configurations yield varying performance, with most configurations outperforming the baselines. The best man- ual configuration achieves a joint Recall @ of 0.8733,
In Section 5.3, we report the results tested on the last day of the dataset. Although this is a mainstream test setting for recommendation benchmarks (Zheng et al., 2024; Wang et al., 2024; Liu et al., 2025), we argue that may not fully
8

---

Learning Cascade Ranking as One Network
Table 4. Experimental results of LCRON under hxxd-weight con- figurations of Leze and Luingte. We fix Leze to and evaluate different weights for Laingle The test set the last day, with the remaining data used for training.
Table 5. Industrial experimental results for 15 days real-world advertising system. Each method was allocated 10% of the online traffic. For online metries, we calculate the relative improvement of other methods compared to FS-LambdaLLoss as the baseline.
weight of Terzlle 150
Sone
Ricsi T1PP
Ranking
NIX
TOva 1I
Retieaal
NEN 1D
Method/Mecric
Offline Metrics Joint Recall
Online Metrics
0.8712
0.5713
0.7244
0,N51]
0.7122
Revenue
Ad Conversions
11 1850
0.8108
(17227
0,9635
0.2119
PS-LanbdaLoss ARF-v2
0.8210 0.8237
0,8000
0.8108
087279
0.9654
02113
018723
0,8725
0.7284
0.8951
0.7136
+1.66%
+0.65%
(18731
0.8730
0.8131
0.8730
0.7280
0.7295
0.8654
0.W90
0.7130
0,7120
I.CRON (ours) 0.8289
+4.1%
+1.6%
0.5720
0,718
0.7233
0,6699
0.2118
08773
0.930
01729
0.W677
0.240
resources, we opted select only the two best-performing baselines from the public experiment, along with LCRON for the online A/B test. Each experimental group was al- located 10% of the online traffic. Each model was trained using an online learning approach and was deployed online after seven days of training, followed by a I 15-day online A/B test. Due to the space limitation, implementation details are described in appendix B. The results of the online exper- iments, shown in Table 5, demonstrate that our LCRON model achieves significant improvements in both revenue and ad conversions compared to the two baseline methods, FS- LSS-LabddaLoss and ARF-v2. Notably, LCRON delivers superior performance in both public and industrial experi- ments, where variations in data size and model architecture are present. This indicates not only the effectiveness but also the robust generalization capabilities of our approach. Based on the results of rigorous A/B testing, I.CRON has been fully rolled out on the Kuaishou advertising platform since January 2025. The two models trained under LCRON have successfully replaced the primary pathways (i.c., those with the highest weight) in the Matching and Pre-ranking stages, marking a major milestone in its industrial deployment.
which is almost the same as UWL--bsse LCRON. This sug- gests that while manual tuning can yield competitive results, UWL provides robust and efficient way to combine Leze and single without extensive hyperparameter search.
Overall, these in-depth analyses, including the ablation study, streaming evaluation, and weight tuning experiments, collectively demonstrate the robustness and practicality of LCRON in real-world cascade ranking systems. The abla- tion study confirms the necessity of both Leze and single for optimizing the interaction effects between ranking and retrieval stages. The streaming evaluation results further validate LCRON's ability to adapt to dynamic, real-world scenarios, while the weight tuning experiments highlight the efficiency of the UWI. (Kendall et al., 2018) formu- lation in reducing hyperparameter tuning efforts without sacrificing performance. These findings solidify LCRON as a strong candidate for industrial applications, offering a balanced combination of performance, adaptability, and ease of deployment.
In addition, we conducted several supplementary experi- ments to further validate the effectiveness and robustness of LCRON under various settings. These include studies on cascade ranking systems with more than two stages (I' > 2). the impact of different differentiable sorting operators, sensi- tivity analysis on hyperparameter T, as well as performance under varying learning rates and batch sizes. These ex- periments not only demonstrate LCRON's scalability to multi-stage cascade ranking systems (see Appendix C.2), but also its compatibility with alternative differentiable sort- ing techniques (see Appendix C.3). Moreover, LCRON exhibits stable performance across a wide range of values (see Appendix C.4) and consistently outperforms baselines under different training config configurations (see Appendix C.5), highlighting its practicality and robustness for real-world applications. Due to space constraints, detailed settings, results, and analysis are provided in Appendix C.
6. Conclusion
In this paper, we present LCRON, a novel training frame- work for cascade ranking systems, which incorporates two complementary loss functions: Leze and Leingle- Leze opti- mizes a lower bound of the survival probability of ground- truth items throughout the cascade ranking process, ensuring theoretical consistency with the system's 's global objective. To address the limitations of Lezes we introduce Lsingles which tightens the theoretical bound and provides additional supervisory signals to enhance stage-wise learning. We conduct extensive experiments on both public (RecFlow)
and industrial benchmarks, as well as online A/B testing, demonstrating that LCRON significantly improves end-to- end Recall, advertising revenue, and user conversion rates. This work has the potential to make broad positive impact across a range of applications, such as recommendation, ad- vertising, and search systems. We also provide an in-depth discussion of the limitations and potential future directions of this work in Appendix E.
5.5. Online Deployment
To study the impact of LCRON on real-world industrial ap- plications, we further deploy LCRON in the advertising sys- tem of Kuaishou Technology. Due to the scarcity of online

---

Learning Cascade Ranking One Network
Acknowledgements
Gallagher, L.,, Chen, R., Blanco, R., and Culpepper, J. S.
Joint optimization of cascade ranking models. In WSDM,, pp. 15-23, 2019.
We thank Lili Mou for his insightful discussion and sugges- tions on the early study of this work. We thank Jiangwei Guo, Daqing Chang. Qi Chen, Yangrui Wang, Hang Chen, Xiang He, Miaochen Li, Jian Hou, Qinglong Li, and Qian He for their contributions to the construction of data streams, enabling us to build full-stage training samples. We also thank Kai Zheng and Qi Liu for their assistance with the RecFlow benchmark, and Jupan Li, Jianghua You, Caiyi Xu, and Xu He for their help with the training and serving pipeline. Finally, we sincerely appreciate the valuable feed- back and comments from the anonymous reviewers, which have helped improve the clarity and quality of the paper.
Grover, A., Wang, E., Zweig, A., and Ermon, S. Stochastic optimization of sorting networks via continuous relax- ations. In ICLR, 2019.
He, K., Zhang, X., Ren, S., and Sun, J. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In ICCV, pp. 1026-1034, 2015.
Huang. P.S.., He, X., Gao, J., Deng, I.,, Acero, A., and Heck, Learning deep structured semantic models for web search using clickthrough data. In CIKM, pp. 2333-2338, 2013.
Impact Statement
Huang. S., Wang, Y., Mou, L.., Zhang, H., Zhu, H., Yu, C., and Zheng, B. MBCT: tree-based feature-awart binning for individual uncertainty calibration. In WWW, pp. 2236- 2246, 2022.
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which feel must be specifically highlighted here.
Jamal, M. A. and Qi, G.-J. Task agnostic meta-leaming for few-shot learning. In CVPR, pp. 11719-11727, 2019.
References
Kendall, A., Gal, Y., and Cipolla, R. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. In CVPR, pp. 7482-7491, 2018.
Blondel, M., Teboul, O., Berthet, Q., and Djolonga, Fast differentiable sorting and ranking, In ICML, pp. 950-959, 2020.
Lee, J.-J. and Yoon, S. W. Xb-maml: Learning expandable basis parameters for effective meta-lcarning with wide task coverage. In A/STATS, pp. 3196-3204, 2024.
Burges, C. J. From ranknet to lambdarank to lambdamart: overview. Learning, pp. 81, 2010.
Burges, C. C., Shaked, T., Renshaw, E.,, Lazier, A., Deeds, M., Hamilton, N., and Hullender, G. N. Learning to rank using gradient descent. In ICML, pp. 89-96, 2005.
Li, P., Burges, C. J. C., and Wu, Q. Mcrank: Learning to rank using multiple classification and gradient boosting. NeurlPS, pp. 897-904, 2007.
Chen, Z., Ngiam, J., Huang, Y., Luong, T., Kretzschmar, H., Chai, Y., and Anguelov, D. Just pick a sign: Opti- mizing deep multitask models with gradient sign dropout. NeurlPS, $3:2039-2053 2020.
Li, Z., Tao, C., Feng, J., Shen, T., Zhao, D.. Geng, X., and Jiang, D. FAA: fine-grained attention alignment for cascade document ranking. In ACL, pp. 1688-1700, 2023.
Lin, X., Chen, X., Song, L.., Liu, J., Li, B., and Jiang, P. Tree based progressive regression model for watch-time prediction in vhort-video recommendation. In KDD, pp. 2023.
Covington, P., Adams, J., and Sargin, E. Deep neural net- works for youtube recommendations, In RecSys, pp. 191- 198, 2016.
Crammer, K. and Singer, Y. Pranking with ranking. In NeurIPS, pp. 641-647, 2001.
Liu, Q., Zheng, K., Huang, R., Li, W., Cai, K., Chai, Y., Niu, Y., Hui, Y., Han, B., Mou, N., Wang, H., Bao, W., Yu, Y., Zhou, G., Li, H., Song, Y., Lian, D., and Gai, K. Recflow: An industrial full flow recommendation dataset. In ICLR, 2025.
Cuturi, M., Teboul, O., and Vert, J.-P. Differentiable ranking and sorting using optimal transport. NeurIPS, 32, 2019.
Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. Bert: Pre-training of deep bidirectional transformers for lan- guage understanding. In NAACL, pp. 4171-4186, 2019,
Liu, S., Johns, E., and Davison, J. End-to-end multi-task learning with attention. CVPR, pp. 1871-1880, 2019.
Ma, X., Zhao, L., Huang, G., Wang. Z., Hu, Z., Zhu, X., and Gai, K. Entire space multi-tast model: An effective approach for estimating post-click conversion rate. In SIGIR, pp. 1137-1 140, 2018.
Finn, C., Abbeel, P., and Levine, S. Model-agnostic meta- learning for fast adaptation of deep networks. In ICML, pp. 1126-117 1666-1115 1135, 2017.
10

---

Learning Cascade Ranking One Network
Niu, Z., Zhou, M., Wang, L., Gao, X., and Hua, G. Ordinal regression with multiple output for age estimation. In CVPR, pp. 4920-4928, 2016.
Wang, Y., Wang, Z., Yang. J., Wen, S., Kong, D., Li, H., and Gai, K. Adaptive neural ranking framework: Toward maximized business goal for cascade ranking systems. In WWW, pp. 3798-38880 2024.
Petersen, Borgelt, C., Kuehne, H.,, and Deussen, O. Dif- ferentiable sorting networks for scalable sorting and rank- ing supervision. In ICML, pp. 8546-85555 2021.
Wang, Z., Tsvetkov, Y., Firat, O., and Cao, Y. Gradient vac- cine: Investigating and improving multi-task optimization in massively multilingual models. In ICLR, 2021.
Petersen, F,, Borgelt, C., Kuehne, H., and Deussen, O. Monotonic differentiable sorting networks. In ICLR, 2022.
Wu, J., Wang, X., Gao, X., Chen, J., Fu, H., and Qiu, On the effectiveness of sampled softmax loss for item recommendation. TOIS, 42(4):1-26, 2024.
Pobrotyn, P. and Biatobrzeski, R. NeuraIndeg: Direct opti- misation of ranking metric via differentiable relaxation of sorting. CoRR, abs/2102.07831, 2021.
Zamani, H., Bendersky, M., Metzler, D., Zhuang, H., and Wang. Stochastic retrieval- reranking. In SIGIR, pp. 81-91, 2022.
Prillo, and Eisenschlos, J. Softsort: A continuous relax- ation for the argsort operator. ICML, pp. 7793-7802, 2020.
Zhang, Z.-Y., Sheng, X.-R., Zhang, Y., Jiang, B., Han, S.. Deng, H., and Zheng, B. Towards understanding the over- fitting phenomenon of deep click-through rate models. In 2022.
Qin, J., Zhu, J., Chen, B., Liu, Z., Liu, W., Tang, R., Zhang, R., Yu, Y., and Zhang, Rankflow: Joint optimization of state cascade ranking systems as flows. In SIGIR, pp. 814-824, 2022.
CIKM, pp.
Zheng, K., Zhao, H., Huang, R., Zhang, B., Mou, N., Niu, Y., Song, Y., Wang, H., and Gai, K. Full stage learning to rank: A unified framework for multi-stage systems. In wwW. pp. 3621-3631 2024.
Rosenblatt, The perceptron: a probabilistic model for information storage and organization the brain. Psy- chological review, 65(6):386, 1958.
Zhou, G., Zhu, X., Song, C.,, Fan, Y., Zhu, H., Ma, X., Yan, Y., Jin, J., Li, H., and Gai, K. Deep interest network for click thrkuhh rate prediction. KDD, pp. 1059-1068, 2018.
Sander, M. E., Puigcerver, J., Djolonga, J., Peyré, G., and Blondel, M. Fast, differentiable and sparse top-k: a convex analysis perspective. In ICML, pp. 29936, 2023.
Sheng, X.-R., Gao, J., Cheng, Y., Yang, S., Han, S., Deng, H., Jiang, Y., Xu, J., and Zheng, B. Joint optimization of ranking and calibration with contextualized hybrid model. In KDD, pp. 4813-4822. 2023.
Swezey, R. E., Grover, A., Charron, B., and Ermon, S. Pirank: Scalable learning to rank via differentiable sorting. In NeurIPS, pp. 21644-21654, 2021.
Tang, Y., Bai, W., Li, G., Liu. X., and Zhang, Y. Croloss: towards customizable loss for retrieval models in rec- ommender systems. In CIKM, pp. 1916-1924, 2022.
Thonet, T., Cinar, G., Gaussier, E., Li, M., and Renders, J.-M. Listwise learning to rank based on approximate rank indicators. In AAAI, volume 36, pp. 8494-8502, 2022.
Wang, L., Lin, J., and Metzler, D. A cascade ranking model for efficient ranked retrieval. In SIGIR, pp. 105-114, 2011.
Wang, X., Li, C., Golbandi, N., Bendersky, M., and Na- jork. M. The lambdaloss framework for ranking metric optimization. In CIKM, pp. 1313-1322, 2018.

---

Learning Cascade Ranking One Network
A. Theoretical Analysis for the Gap Between PCs and P
Here, we provide a more detailed theoretical analysis of the gap between Pes and its lower bound Pcs and explain how Tssingle contributes to tightening this bound.
The gap between Pes and Pog can be formulated Eq. 14:
A
CS
CS
M2 O π
ET
M2 ①
(T,, Ptt
= En
pda
Q2
M2
(T,, P
M2
ETTP
92
(14)
M2
(TT, MMg)
92
M2
E PT (T,, Ma
PM
Q2
(PM, Pdp
A
Here, we have derived a theoretical upper bound for A, denoted as A' Note that PM. E [0, PM. E [0, 1] and
PM = 42. > PM, = 91-
Next, we consider how the relationship between the model outputs might affect A' For given P treating PM2 the only variable to minimize A. it is evident that the following condition must be satisfied:
{
1 i e argTop (K=q1)
(PM,);
(15)
otherwise
where (PM)
M, ); represents the i-th element of the vector :M and arg ArgTopK(PM)) denotes the indices of the top K elements
of p the consistency of the entire output distributions across different models also achieves the same goal, as this is stronger constraint. Furthermore, if (PM, ) is a binary vector (i.e., its elements are either 0 or 1), then A achieves its minimum value
'M,' This indicates that if the top q2 sets of the two models are consistent, it helps reduce A' Of course, optimizing
(A' = 0) when Eq. 15 is satisfied.
The single-stage loss I M; single (Eq. 12) directly optimizes the ranking consistency of each model with the same supervision (ground-truth labels), thereby implicitly aligning Mi (D) and M2(D). This helps to reduce A' indirectly optimizing the bound A. It is worth noting that there are many methods to optimize the consistency of model outputs, such as model distillation or minimizing the KL divergence between outputs. The Lsingle we designed not only optimizes the consistency of output distributions across different models but also mitigates potential gradient vanishing issues in Teze under certain circumstances, providing additional effective supervision signals.
B. Implementation Details of Online Experiments
In this section, we provide additional details on the implementation of the online experiments. Our online system consists of four stages: Matching, Pre-ranking, Ranking, and Mix-ranking, However, for the purpose of this study, we focus two-stage cascade ranking setup, comprising the Matching (Retrieval) and Pre-ranking stages (illustrated in Figure 1). This design choice is motivated by several practical and methodological considerations: 1) Generality of Two-Stage Setup: The two-stage cascade ranking setup (Matching and Pre-ranking) does not lose generality, as it captures the core challenges of cascade ranking optimization, which are fundamental to any multi-stage ranking system. The insights gained from this setup
12

---

Learning Cascade Ranking as One Network
can be extended to systems with more stages. 2) Practical Constraints:
Conducting experiments on the entire four-stage system would introduce significant engineering challenges, particularly in data construction and model deployment. For instance, aligning the format of training logs and deployment details across all stages is non-trivial. Deploying and evaluating a full four-stage cascade ranking system in an online environment would require substantial infrastructure support, which is beyond the scope of this study.
Regarding the features, we utilize approximately 150 sparse features and 4 dense features. Sparse features are represented by embeddings derived from lookup tables, with each ID embedding having a dimensionality of 64. Dense features, on the other hand, directly use raw values or pre-trained model outputs, with total dimensionality of 512. The Retrieval model includes all features used in the Pre-ranking model, except for combined features. Combined features refer to the process of integrating multiple individual features into a new, unified feature set to improve model performance. This technique captures interactions between different variables, providing richer information than each feature could individually. The combined features also fall under the category of sparse features. In our experiments, there are 20 combined features. In our experiments, models of different stages do not share any parameters.
Regarding the model architectures, we adopt DSSM (Huang et al., 2013) for the retrieval stage and MLP (Rosenblatt, 1958) for the Pre-ranking stage. For DSSM (Huang al., 2013), the FFN layers' size of both the user and item towers is [1024,256,2566 The layer size of the MLP (Rosenblatt, 1958) is [1024.112.12..], We employ PRelu (He al., 2015) as the activation function for hidden layers. We use Helnit (He et al., 2015) to initialize all the training parameters. We adopt batch normalization for each hidden layer, and the normalization momentum is 0.999.
Regarding the training samples, we organize our training samples according to Section 4.1. We collect samples from the Matching, Pre-ranking, and Ranking stages, with items that succeed in the Ranking stage treated as ground-truth items. For each impression, we collect 20 items for training (i.e., N 20 in Section 4.1). The sample distribution across stages is defined by no 5, n1 722 8, and ni 2.
The models are trained in an online streaming manner. During each day of training, 20 billion (user, ad) pairs were processed. The optimizer is AdaGrad with a learning rate of le-2. All parameters are trained from scratch, without any pre-trained embeddings. The batch size is set to 4096 for both the retrieval and Pre-ranking models. Both the training and serving frameworks are developed based on TensorFlow. We use LCRON to train the retrieval and pre-rank model together and save unified checkpoints. Subsequently, by reconstructing two metadata files, these two models are deployed separately. During deployment, each model loads only the parameters corresponding to its own structure from the saved checkpoint during the joint training phase.
C. More Experimental Details and Results on the Public Benchmark
C.1. Implementation and Hyper-parameters Tuning Details of Baselines
Since none of the baseline methods have been evaluated on the RecFlow (Liu et al., 2025) dataset under cascade ranking settings, all baseline results were obtained by re-implementing or adapting the source code under the same experimental conditions as our proposed LCRON, rather than directly citing results from previous papers. For FS- Sa-Rann and FS- Lambdal we adapt standard implementations from the TF-Ranking library to PyTorch versions. For other baselines, when open-source code was available and runnable, used it directly; otherwise, we implemented the baselines based on the descriptions in their respective papers. To ensure fair comparison, all methods were evaluated using the same common hyperparameters, such as learning rate, batch size, optimizer, initialization method, etc.
We performed a grid search on the main hyperparameters for all methods to ensure fair comparisons. We report the best results for the baselines. Specifically, the parameters we tuned include: temperature for ICC 10.05.0.1.0..,1..0) 1.O), tau for ARF and LCRON (1,20.50.100.2) 1000); alpha (0.0.25.0.5,0.75. for RankFlow; and top-k (10.20.30,40) and smooth factor (0.00.5,,.5,0075,.) for FS-LambdalLss. BCE and FS RS-RRnnNet do not have independent hyperparameters.
C.2. Experiments on Three-stage Cascade Ranking
In the main text, we only show the experiments on two-stage cascade ranking scenarios. Although LCRON is readily extended to cascade ranking systems with more than two stages, which is emphasized in Section 4.1, there is no experimental evidence for this claim. To further verify the scalability of LCRON for T' > 2 stages, we also conduct experiments under the three-stage cascade ranking setting, which is also typical setting for T in real-world cascade ranking applications. Limited
13

---

Learning Cascade Ranking as One Network
Table Experimental results for three-stage cascade ranking. Each method was run times, and the results are reported as meantstd. * indicates the best results. Bold numbers indicate that LCRON shows statistically significant improvements over the baselines, as determined by a t-test the 5% significance level. The test set the last day, with the remaining data used for training.
Joint Recall 20 f
Ranking
Pre- Pre-ranking
Retrieval
Method/MMetic
Recall & 10@:20 1
NDCG@ 10
Recall 30
NDCG@ 1
&ecall 10@-0 f
NDCG@ 10
BCE
0..195
0.6574=0.0011
0.5.14
0.8814+0.0007
0.6382+0.0000
0.9709=0.0000
0.630ee0.0019
ICC
0.638610.0071
0.619620.0122
0.5794=0.0053
0.7682+0.0485
0.49251000045
0.852610.0467
0.4754=0.0679
RunkFlow
07308=000005
0.7230=0.000
0,6400=0.00ml
0.8729#0.001 0014
0.6396=0.000 0008
0.9611#0.00M
0.6265=0.0013
FS. RankNee
0.6200
0.6224 0008
0.57766=00.000
0.803840.0000
0.5733+0.0000
0.93733+000008
0.5678=0.000X
FS- FS-LambdaLoss
0.7319=0.0038
0.7292:0.00 2.42
0.6431#0.0014*
0.8803+0.000 0.009
0.6443+0.0024
0.9662#0.0012
0.6297=0.0022
ARF 0.7256=0.0004
0.7251 0000
0,5675=0.0003
0.8712+0.0000 0008
0,5099 0074
0,9612+0.0004
0,4268=0.000 0031
ARF-v2 LCRON
0.738240.0020
0.7285 10005
0.6430e0.0015
0.877760.0064
0.643840000011
0.964910.0029
0.6284a0.0033
0.7390=0.0008
0.7338=0.0000
0.6017=000009
0.8859#0.0007
0,6010+0.0031
0.9678:0.0012
0,5758=0.006
Table 7. Experimental results for directly using differentiable sorting operators to learning the rank, and test LCRON with different diffcrentiable sorting operators. The experiments are under the settings of the two-stage cascade ranking. Each method was run 5 times, and the results are reported meansstd. indicates the best results. The test set the last day, with the remaining data used for training.
Joint
Ranking
Retrieval
Method/Metric
Recall TOW
Recall 106620 1
NDCG 10
Recalfer 106330 T
NDCG 10
SoftSort
0.8103=0.0013
0.8138=00.0111
0.714840.0006
0.9386440 0003
0.7066:40. 0005
NouralSort
0,8210=0.0016
0.8233+0.0007
0.7138+0.0004
0.946940.0010
0.6979+0.0013
LCRON(Softorr)
0.8723=0.0008
0.8720=0.0009
0.724620.0008
0.9703=0.0015*
0.7035+0.0266
LCCRNNNcuralSSon) 0.8732=00.0005
0.8731+00.004'
0.7292+0.0008*
0,970010.0004
0.7152=0.0009
by our industrial scenario, we can only implement two-stage experiments. Thus, we only conduct the three-stage experiments on the public benchmark. Concretely, we constructed a three-stage cascade ranking system (T = 3) on RecFlow (Liu et al., 2025), using its prerank_neg_ coarse_neg, rank_neg, rerank_neg, and rerank samples. The samples of rerank_pon are treated as ground truth. The three stages utilized DSSM (Huang et al., 2013), MLP (Rosenblatt, 1958), and DIN (Zhou et al., 2018) architectures, respectively, which are widely adopted in industrial recommendation systems for Matching, Pre-ranking, and Ranking stages. The results are shown in Table 6, formatted as meantstd. LCRON still significantly outperforms the baselines on end-to-end recall, suggesting the scalability of LCRON to cascade ranking systems with more than two stages.
C.3. Experiments with Different Differentiable Sorting Operators
In the ablation study in Section 5.4, we separately validate the effects of Lene and Lsingle. It is also curious about the performance of directly applying differentiable sorting techniques. To further validate the effectiveness of LCRON, we conduct experiments that solely use differentiable sorting techniques (i.e., aligning model predictions with label permutation matrices through CE loss), which share the same underlying rationale as FS-RankNe in making models fit complete orders. In addition, LCRON's compatibility with different differentiable sorting techniques is also curious. Thus, we further conduct experiments that test SoftSort (Prillo & Eisenschlos, 2020) as an alternative to NeuralSort (Grover et al., 2019).
All experimental results are shown in Table 7, formatted as meantstd. Each method was run five times. In Table 7, "NeuralSort" and "SoftSort" represent the results of evaluating standalone the NeuralSort (Grover et al., 2019) and SoftSort (Prillo& & Eisenschlos, 2020) operators respectively. "LCRON(NeuralSori"" and "LCRON(Sof(Sor))) represent the results of LCRON that employs NeuralSort and SoftSort its foundation, respectively.
The results show that LCRON significantly outperforms directly using the differentiable sorting operators, further validating the effectiveness of our proposed method. Another fact is that LCRON(Sof(Sort) also achieves significantly better performance than baselines. These experiments verify LCRON's effectiveness and generalization capability across differentiable sorting operators, also suggesting that LCRON's effectiveness could benefit from more advanced differentiable sorting techniques.
C.4. Sensitivity Analysis on Hyper-parameters of LCRON
We analyze the sensitivity of LCRON to its hyperparameter т, which controls the smoothness of NeuralSort (Grover et al., 2019). Table 8 summarizes the results for different values of r. The best performance is achieved at r = 50. Even
14

---

Learning Cascade Ranking One Network
Table Sensitivity analysis results of the hyper-parameter of L.CRON the public benchmark, under the settings of the two-stage cascade ranking. The test set is the last day, with the remaining data used for training.
Joint
Ranking
Retrieval
T
Recall @ 10@201
Recall @ 10@ 20
NDCG@10
Recall @ 10 30
NDCG@ 10
0.8701
0,8701
0.7276
0,9677
0.7071
20
0.8709
0.8708
0,7285
0,9696
0.7145
0.8732
0.8729
0.7291
0,9700
0.7151
100
0.8722
0,8719
0,7292
0,9707
0,7155
200
0.8721
0.8719
0.7278
0.9703
0.7156
1000
0.8716
0,8711
0.7285
0.9711
0.7197
Table Sensitivity analysis with learning rate of 0.001 and batch size of 512, under the settings of the two-stage cascade ranking. The test set is the last day, with the remaining data used for training. indicates the best results. The number in bold means that our method outperforms all the baselines on the corresponding metric.
Joint
Ranking
Retrieval
Method/Metric
Recall @i T0@20
Recall@ 10 2
NDCG@ 1
Recall 10@30
NDCG 10
BCR ICC
0.8181
0.8000
0.6700
0.9576
0.6918
0.7644
0.7662
0.6648
0.8825
0.5094
RankFlow
0.8326
0.8272
0.6909
0,9535
0.6922'
FS- RankNet
0.7537
0.7543
0.6546
0.9184
0.6547
PS-LambdaLoss
0.8289
0,8250
0.6878
0,9558
0.6899
ARF 0.8288
0.8256
0,6316
0.9515
0,5115
ARF-v2
0.8302
0.8295
0.6907
0.9550
0.6838
LCRON 0.8396
0.8380
0.6944
0,9573
0.6839
suboptimal values 7 within a wide range (e.g., = 100 or 200) yield significant improvements over the baselines, demonstrating the robustness LCRON hyper-parameter choices. From the results, we observe that the performance seems to exhibit an unimodal trend with respect to т, peaking at r 50 and gradually decreasing as moves away from this value. This unimodal behavior provides practical guidance for hyperparameter tuning in real-world industrial applications, suggesting that moderate value of is likely to yield near-optimal performance.
C.5. Sensitivity Analysis on Batch Size and Learning Rate
common all methods, namely batch size and learning rate. Coneretely, we perform the sensitivity analysis four hyperparameter configurations: (1) learning rate=0.001 with batch size=512; (2) learning rate=0.001 with batch size=2048; (3) learning rate=0.02 with batch size=512; (4) learning rate=0.02 with batch size=2048. The experimental results are shown in Table Tables 9 to 12, respectively. is noteworthy that these methods show different degrees of effectiveness at different learning rates and batch sizes, among which ICC particularly sensitive to these hyperparameters, revealing that its gradient more likely to be too sharp or vanish. Most of these methods achieved the best results under learning rate=0.02 and batch size=512. can be seen that our method consistently achieves optimal results, demonstrating the robustness of our approach.
We also conduct sensitivity analysis experiments the main
D. Runtime and Space Complexity Analysis
This section presents runtime and space complexity analysis of the discussed methods, aiming to evaluate their practicality in real-world applications.
The time and space complexity of BCE is O(DN), where N the number of sampled items within single impression and D is the number of impressions in the training. RankFlow involves sequential training stages, using BCE and distillation losses (each with O(DN) complexity). The serial nature of these steps introduces only constant multiplicative factor, preserving the overall ODDN complexity. Differentiable sorting techniques such as NeuralSort and SoftSort typically have a per-impression time and space complexity of O(N2), where N is the number of items sampled within single impression. This complexity is on par with classic learning- toriinngtt-ramm to-k methods like RankNet (Burges et al., 2005) and Lambdal Loss (Wang et al., 2018). As a result, methods based on differentiable sorting including ICC, FS- DS-RankNet, DS-LambdaLoss. ARF, ARF-v2, and LCRON also exhibit an overall time and space complexity of O(DN2).
15

---

Learning Cascade Ranking One Network
Table 10. Sensitivity analysis with learning of 0.001 and batch size of 2048, under the settings of the two-stage cascade ranking. The test set is the last day, with the remaining data used for training. 'indicates the best results. The number in bold means that our method outperforms all the baselines on corresponding metric.
Joint
Ranking
Retrieval
Method/Metric
Recall @ 10@201 f
Recall 10 20
NDCG@ 10
Recall 10@000
NDCG
BCE
0.8106
0.7947
0,6629
0.9484
0,6808
ICC
0.7459
0,7490
0.6497
0.8754
0.5278
RankFlow
0.8166
0.8135
0.6857"
0.9438
0.6811"
TS-RRakNet
0.7533
0.7554
0.6526
0.9111
0.6463
FS-Lambdal
0.8194
0.8172
0.6843
0.9467
0.6737
ARF
0.8174
0.8148
0.6269
0.9443
0.5039
ARF-v2
0.8202
0.8193
0.6824
0.9463
0.6632
LCRON
0.8247
0.8219
0.6771
0.9508
0.6750
Table 11. Sensitivity analysis with a learning rate of 0.02 and batch size of 512, under the settings of the two-stage cascade ranking. The set is the last day, with the remaining data used for training. * indicates the best results. The number in bold means that our method outperforms all the basclines on the corresponding metric.
Joint
Ranking
Retrieval
Method/Metric
Recall @ 10@ 20
Recall@ 10@ 20
NDCG@ 10
Recall 10@ 30 1
NDCG@ 10
BCE
0.8637
0.8522
0.7138
0.9673*
0.7027
ICC
0.4972
0,5037
0,4151
0,7426
0.4020
RankFlow
0.7935
0.7983
0.6914
0.9238
0.6607
FS-RankNet
0.8768
0.8772
0,7409
0.9641
0.7049*
FS-LambdaLoss
0.8778
0.8792
0.7430
0.9658
0.6968
ARF
0.8704
0.8739
0.6843
0,9583
0.5147
ARF-v2
0.8776
0,8803
0,7345
0,9635
0.6935
LCRON
0.8841
0.8859
0.7396
0.9671
0.6969
In real-world applications, is usually small (e.g., 20 in our system). Moreover, the main computational cost comes from the model's prediction generation, which independent the specific loss function used. As result, the computational cost from the O(DN²) operations in the loss function remains manageable. Under such settings, LCRON should incur no additional training overhead compared to baseline methods. To validate this, we conducted experiments RecFlow using A800 GPUs and recorded the GPU memory usage and runtime. Taking two-stage cascade ranking experiments (N is 40) on RecFllow an example, the maximum of GPU memory usage for BCE, ICC, RankFlow, FS- TS-RRnkNet, FS-LambdaLoss, ARF, ARF-v2, and LCRON was approximately 37.7 / 38.2 38.2 2/ 37.7 38.2 37.3 37.3 / 38.2 GB, respectively. The one-epoch runtimes were 5358 5376/ / 5057 5104 / 5076 / 5362 5573 5418 seconds, respectively. All methods show comparable runtime performance, with minor differences likely due to runtime environmental factors rather than algorithmic complexity. terms of GPU memory usage, model parameters dominate the consumption. Differences space complexity among the various loss functions have a negligible impact on overall GPU memory usage.
In summary, the analysis demonstrates that LCRON does not introduce significant computational memory overhead compared to existing methods, making it well-suited for industrial deployment.
E. Limitations and Future Work
E.1. Limitations of Using Soft Permutation Matrix to Express the Probability Top-k Set Selection
For permutation matrix A, the elements Alj. i] and Alk, i) represent the probabilities that the i-th element of the vector ranks j-th and k-th, respectively. For any & k, the values of Alj, i] and Alk, i] are mutually exclusive; in other words, an increase in value will suppress the other. This characteristic leads expresses the probability of an item appearing in the top-q: set, when optimizing the probability of the ground-truth item (assumed to be the i-th item) using our proposed loss, LCRON tends to simultaneously increase both A i] and A A i). This results in gradient conflicts and cancellations, slowing down the optimization process and degrading performance. Moreover, this issue becomes more pronounced as increases.
challenge in LCRON: although Equation 10
As mentioned in the main text, to improve performance, we set @n and d2 in Equation 10. While this somewhat
16

---

Learning Cascade Ranking One Network
Table 12. Sensitivity analysis with learning rate of 0.02 and batch size of 2048, under the settings of the two-stage cascade ranking. The test set is the last day. with the remaining data used for training. 'indicates the best results. The number in bold means that our method outperforms all the baselines on the corresponding metric.
Joint
Ranking
Retrieval
Method/Metric
Recall @ 10@20 f
Recall 10 10 f
NDCG@ 1
Recall 10 f
NDCG 10
BCE
0.8582
0.8469
0.7086
0.9691
0.7061
ICC
0.5061
0.4927
0.4151
0.7615
0,4647
RankFlow
0.8722
0.8729
0.7360
0.9625
0.6910
TS-RaakNet
0.7884
0,7924
0,6875
0,9277
0.6630
FS-Lambdal Loss
0.8726
0.8728
0.7368
0.9689
0.7141
ARF
0.8667
0.8680
0,6786
0.9632
0.5364
ARF-v2
0.8725
0.8730
0.7323
0.9691
0.7157
LCRON
0.8785
0.8785
0.7327
0.9713
0.7132
compromises the physical meaning of the end-to-end loss, it achieves better trade-off between alignment with physical meaning and ease gradient optimization. we align @i and d2 with the cascade ranking setup (i.e., setting @1 = 30 and 92 = 20 for the end-to-end loss), the end-to-end recall of LCRON corresponding to Table 2 would decrease to 0.8714 + 0.0008.
We believe that future research should explore differentiable operators directly designed for top-k set selection, rather than constructing the probability distribution based sorting results. Such operators could potentially resolve the conflict issues observed in LCRON. Alternatively, another feasible approach might involve introducing carefully designed weights for different Alj i] and Alk, i] terms in Equation 11, thereby mitigating the conflict problem.
E.2. Limitations of the Fusion of Multiple Surrogate Losses
LCRON adopts UWL (Kendall et al., 2018) strategy to fuse different surrogate losses (i.e., Leze and Laingte). As shown in Table 2 and Table 4, UWL demonstrates robustness by reducing the need for manual hyperparameter tuning without compromising performance. However, it is worth noting that UWL makes an implicit assumptin--orggnnalyy proposed in its paper-that each loss follows a Gaussian distribution. This assumption may not hold in practice, making UWL more of heuristic approach that primarily aims balancing the scales of different loss terms rather than providing theoretically grounded fusion.
It remains unclear whether this strategy can maintain its effectiveness and robustness across broader scenarios, such as those involving varying model capacities or data complexities. Therefore, exploring more principled approaches for combining multiple losses in LCRON is an important direction for future work.
We argue that the problem of loss combination in LCRON can be viewed as a special case of multi-task learning, where the goal is to combine multiple objectives in way that better optimizes the final target. Advanced techniques from the fields of multi-task learning (Liu al., 2019; Chen al., 2020; Wang al., 2021) and meta-learning (Finn al., 2017; Jamal & Qi, 2019; Lee & Yoon, 2024) are likely to offer valuable insights into improving both the effectiveness and generalization of the LCRON framework. As discussed in ARF (Wang et al., 2024), certain sub-losses may carry overlapping or hierarchical semantic meanings. Different settings such such as models with varying capacities or datasets with different levels of complexity-man benefit from distinct weighting strategies. Designing a dedicated meta-learner that adapts the loss weights based on the characteristics of the model and data might lead to more universally robust solution.
E.3. Computational Complexity Limitations of LCRON for Cascade Top-K Selection
LCRON utilizes a soft permutation matrix generated through differenntablee sorting techniques, resulting in an O(n²) computational complexity. Although an O(n²) complexity is generally acceptable (as many mainstream learning-to to-rank methods, such as Lambdal Loss (Wang et al., 2018), also exhibit this complexity). it still falls short compared to the theoretical optimal complexity of (n) for hard top-k selection problems.
A key direction for future research could be to explore how to directly relax hard top-k selection methods to obtain differentiable operators for top-k selection and cascade top-k selection with complexity of O(n). Moreover, investigating how more computationally efficient algorithms can be combined with data scaling-up up strategies- and their potential impact on industrial applications-is also likely to be of great significance.
17


Reviews:
[{"review_id": "n4ogwb0PP0", "reviewer": "Reviewer_yid9", "summary": "The paper \"Learning Cascade Ranking as One Network\" introduces LCRON, a novel approach for training cascade ranking systems in an end-to-end manner. Traditional cascade ranking architectures suffer from misalignment between training objectives across different stages and discrepancies between training and testing environments. The paper proposes a new surrogate loss function that optimizes the lower bound of the survival probability of ground-truth items through all stages, ensuring a better alignment of training objectives. The authors also introduce an auxiliary loss for each stage to improve robustness. Experimental results on public (RecFlow) and industrial benchmarks demonstrate that LCRON outperforms existing approaches in terms of recall and conversion metrics, achieving a 4.1% increase in advertising revenue and a 1.6% increase in user conversions in a real-world deployment.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks very much for the detailed and insightful review.\n\nFor the questions:\n\n1) In cascade ranking systems, **we can often explicitly define the ground-truth, thus optimizing the end-to-end recall directly maximizes selection efficiency. So we treat it as the golden metric**. Other intermediate metrics, such as precision or diversity, are less critical in this context. Recall and NDCG for single stage are also intermediate metrics for observation and analysis.\n2) All baseline results were obtained by re-implementing or adapting the source code under the same experimental conditions as our proposed LCRON, rather than directly citing results from previous papers. **Since none of the baseline methods have been evaluated on the RecFlow dataset under cascade ranking settings. For FS-RankNet & FS-LambdaLoss, we adapt standard implementations from the TF-Ranking library to PyTorch versions. For other baselines, when open-source code was available and runnable, we used it directly; otherwise, we implemented the baselines based on the descriptions in their respective papers**. All methods were evaluated using the same common hyperparameters (lr and batchsize, optimizer, initialization method, etc.) to ensure fair comparison.\n3) For the public experiments, the training data organization for the two-stage cascade ranking is described in lines 284-297. There is no additional data pre-processing step. I assume you might be asking how the data filtering is performed in a cascade ranking system, i.e., how many items the retrieval model selects to pass to the ranking model and how many items the ranking model then selects as the final output. These specific settings are detailed in lines 298–312 of the draft.\n4) **Yes, we performed a grid search on the main hyperparameters for all methods to ensure fair comparisons. For the baselines, we reported the best results, and for LCRON, we included a sensitivity analysis in the appendix. Specifically, the parameters we tuned include: temperature for ICC (0.05,0.1,0.5,1.0), tau for ARF and LCRON (1,20,50,100,200,1000); alpha (0,0.25,0.5,0.75,1) for RankFlow; and top-k (10,20,30,40) and smooth factor (0,0.25,0.5,0.75,1) for FS-LambdaLoss**. BCE and FS-RankNet do not have independent hyperparameters. Regarding the learning rate (lr) and batch size, since all methods used the same setting for fair comparison and results on industrial applications are typically not very sensitive to these hyperparameters, we did not experiment with different lr and batch sizes. **Considering your concern, we ran an additional four sets of experiments to validate performance under different LR and batch sizes, rerunning all eight methods for each set, resulting in 32 experimental runs**. Due to time limitations, each method was run only once. Statistical significance can be assessed by referring to the mean±std and p-value from other significance tests. Due to space limitations, we only show the End-to-End Recall under different LR and batch sizes:\n\n|Method|bs=512,lr=0.001|bs=2048,lr=0.001|bs=512,lr=0.02|bs=2048,lr=0.02|\n|-----|-----|-----|------|------|\n|BCE|0.8181|0.8106|0.8637|0.8582|\n|ICC|0.7644|0.7459|0.4972|0.5061|\n|RankFlow|0.8326|0.8166|0.8768|0.8722|\n|FS-RankNet|0.7537|0.7533|0.7935|0.7884|\n|FS-LambdaLoss|0.8289|0.8194|0.8777|0.8726|\n|ARF|0.8288|0.8174|0.8704|0.8667|\n|ARF-v2|0.8302|0.8202|0.8776|0.8725|\n|LCRON|**0.8396**|**0.8247**|**0.8841**|**0.8785**|\n\n**It can be seen that our method achieves consistently optimal results, demonstrating the robustness of our approach**. These sensitivity analysis details and the hyperparameter tuning specifics for the baselines will be added to the appendix.\n\n5) Please refer to our response to Reviewer VkMd (for weakness 1).\n\n6) Please refer to our response to Reviewer VSAL. The improvement of LCRON over other baselines is statistically significant.\n\nFor the missing reference: \n\nThank you for highlighting these works. [1] is already introduced in related work. [1] and [3] are differentiable sorting methods, but they don't produce the permutation matrix, making them incompatible as foundation components for LCRON compared to NeuralSort and SoftSort. We will discuss this in related work. Our current comparisons include state-of-the-art single-stage recall optimization methods (e.g., ARF) and joint learning approaches for cascade ranking (ICC, RankFlow, FS-RankNet, FS-LambdaLoss). [2] and [4] focus on optimizing ranking metrics (e.g., NDCG and Precision) for single-stage models, which differ from our end-to-end cascade ranking objective. We will explicitly highlight this distinction in the related work section.\n\nDue to space limitations, we only show the key results (e.g., end-to-end recall) in the rebuttal text. Full additional results can be found in this anonymized github link: https://anonymous.4open.science/r/2025038594/\n\nIf you have any further questions or concerns, we will make every effort to provide further clarification.", "claims_and_evidence": "The main claims of the paper are:\n- LCRON aligns training objectives across all cascade ranking stages: Supported by the proposed surrogate loss function, which explicitly optimizes the recall of the entire system rather than individual stages.\n- LCRON improves end-to-end recall compared to existing methods: Empirical results from RecFlow and industrial benchmarks confirm higher recall scores.\n- LCRON enhances commercial performance in real-world applications: A/B testing in a real-world advertising system shows notable revenue and conversion improvements.\n\nWhile these claims are mostly well-supported, the empirical results focus primarily on recall metrics, and it would be useful to evaluate additional ranking quality metrics (e.g., precision, diversity).", "methods_and_evaluation_criteria": "The paper presents a clear research question: How can cascade ranking be trained end-to-end while aligning training objectives across all stages? The hypothesis, which proposes that a surrogate loss can improve ranking alignment and recall, is consistent with the methodology and results. The experimental design is appropriate for addressing this question.\n\n*Baselines*. The paper compares LCRON against state-of-the-art methods such as BCE, ICC, RankFlow, FS-LTR, and ARF. The chosen baselines are well-justified, covering both simple (BCE) and advanced (RankFlow, FS-LTR) approaches. However, it is unclear whether baseline results were obtained from previous papers or rerun under the same conditions. Explicit clarification on this would improve reproducibility.\n\n*Evaluation Metrics*. The primary evaluation metric is Recall@K@M, which is relevant for cascade ranking. The authors also report NDCG@K as a secondary metric. While these metrics align with the research goal, additional discussion on trade-offs between precision, recall, and ranking diversity would be beneficial.\n\n*Data Collection and Preprocessing*. The dataset choice (RecFlow) is appropriate, as it contains multi-stage ranking samples.\nData preprocessing steps (e.g., filtering of interactions) are not detailed. If any pruning was performed, the justification should be included.\n\n*Data-Splitting and Generalization*. The train-test split is conducted over time, which is standard for ranking models.\nAll models appear to be trained on the same splits, but cross-validation techniques are not explicitly mentioned.\n\n*Hyperparameter Optimization*. The optimization strategy is briefly discussed but lacks detail on parameter ranges and tuning procedures. It is unclear how many configurations were tested or how hyperparameters were selected, which could impact result reproducibility.\n\n*Experiment Execution and Sensitivity Analysis*. The experimental setup appears fair, but hardware details (e.g., GPU models, memory) are missing. There is no explicit discussion of statistical significance tests (e.g., p-values or confidence intervals) in Table 2, which would strengthen the analysis.\nSensitivity analysis is limited to the temperature parameter in differentiable sorting, but other key hyperparameters (e.g., learning rate, batch size) are not explored.", "theoretical_claims": "The theoretical justification for LCRON’s surrogate loss function appears sound. The derivation of the lower bound on the survival probability of ground-truth items is well-structured and aligns with the recall optimization objective.", "experimental_designs_or_analyses": "The experimental design is strong and well-structured, with:\n- Comparisons against state-of-the-art baselines (BCE, ICC, RankFlow, FS-LTR, ARF)\n- Public and industrial benchmarks\n- Ablation studies to test the contribution of each component\n- Streaming evaluation to simulate real-world training conditions\n\nA few areas for improvement are (i) Computational cost analysis: how does LCRON’s training time compare to existing methods?\n(ii) Robustness to dataset shifts: given that RecFlow spans multiple time periods, an explicit analysis of temporal generalization would be beneficial.", "relation_to_broader_scientific_literature": "The paper builds upon and improves several prior works in cascade ranking, particularly: LambdaRank and Learning-to-Rank approaches, Differentiable sorting techniques (NeuralSort, SoftSort), Interaction-aware training methods (RankFlow, FS-LTR).\n\nThe work is well-grounded in prior literature.", "essential_references_not_discussed": "While the paper covers major references, it could benefit from discussing additional works on differentiable sorting and ranking, such as:\n- Blondel, M., Teboul, O., Berthet, Q., & Djolonga, J. (2020, November). Fast differentiable sorting and ranking. In International Conference on Machine Learning (pp. 950-959). PMLR.\n- Pobrotyn, P., & Białobrzeski, R. (2021). Neuralndcg: Direct optimisation of a ranking metric via differentiable relaxation of sorting. arXiv preprint arXiv:2102.07831.\n- Cuturi, M., Teboul, O., & Vert, J. P. (2019). Differentiable ranking and sorting using optimal transport. Advances in neural information processing systems, 32.\n- Thonet, T., Cinar, Y. G., Gaussier, E., Li, M., & Renders, J. M. (2022, June). Listwise learning to rank based on approximate rank indicators. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 36, No. 8, pp. 8494-8502).", "other_strengths_and_weaknesses": "Other Strengths: \n- Practical Relevance: The method has clear real-world applicability in advertising and recommender systems.\n\n- Clear Paper Structure: The organization of the paper makes it easy to follow.\n\n- Empirical Strength: The industrial deployment and A/B testing strengthen the validity of claims.\n\nOther Weaknesses: \n- Computational Complexity Not Analyzed: The additional cost of using differentiable sorting techniques is not explicitly measured, which could impact scalability in large-scale applications", "other_comments_or_suggestions": "- Consider including a formal convergence proof for LCRON's optimization process to demonstrate that it will always converge. Additionally, it would be helpful to analyze the conditions under which the optimization may fail or become unstable.\n\n- It would strengthen the paper to provide formal generalization bounds for LCRON, especially in highly dynamic ranking environments, to guarantee that the model will generalize well to unseen data, beyond the empirical results.", "questions_for_authors": "1. why didn't you evaluate additional ranking quality metrics (e.g., precision, diversity)?\n2. were baseline results obtained from previous papers or rerun under the same conditions?\n3. can you detail  which data preprocessing steps (e.g., filtering of interactions) were performed? If any pruning was performed, the justification should be included.\n4. What strategy was used for hyperparameter tuning and what were the specific ranges considered for key hyperparameters? How many configurations were tested during tuning, and what criteria were used to select the final hyperparameters? Were all baselines tuned under the same conditions to ensure fair comparisons?\n5. Can you specify the computational resources used for training, such as GPU models, memory, and runtime per experiment?\n6. Why are statistical significance tests (e.g., p-values, confidence intervals) missing from Table 2? Can you provide evidence to confirm the robustness of the reported improvements?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EnH4UUHraT", "reviewer": "Reviewer_VSAL", "summary": "The paper addresses two key challenges in cascade ranking:\n(i) the misalignment of training objectives across different stages and\n(ii) the discrepancy between training and test environments caused by multi-stage ranking and filtering.\n\nTo overcome these issues, the authors propose a novel loss function comprising: an end-to-end term that optimizes the global objective of the cascade ranking system, and stage-wise terms that help models adapt to changes in the sample space distribution introduced by upstream ranking stages. The paper adopts the model architecture for all stages from RecFlow [1], focusing its contributions on refining the training loss.\n\n[1] RecFlow, ICLR'25", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks very much for the detailed and insightful review. \n\nFor the weaknesses:\n\n1) We adopted a two-stage setup (retrieval + ranking) because two-stage cascading represents the most classic form of cascade ranking, as seen in previous works like FS-LTR. From a practical perspective, to the best of our knowledge, real-world online cascade ranking systems typically employ 3–4 stages to achieve an optimal trade-off between effectiveness and efficiency. Due to real-world constraints (e.g., business requirements, team organizations), retrieval + preranking often serves as the most feasible scenario for validating and implementing multi-stage joint optimization. Thus, we believe our experiments retain generality and sufficiently demonstrate the value of our method in real-world applications. Nevertheless, we acknowledge that testing LCRON on scenarios with >2 stages could further validate its scalability and enhance the depth of this work. Due to some limitations of our industrial scenario, we can not deploy 3 or more stages experiments. We constructed a three-stage cascade ranking system and verified the effectiveness of LCRON on RecFlow. **The detailed settings and results are in the response to reviewer VkMd. Experimental results show that LCRON still significantly outperforms baselines in this three-stage setting**. \n\n2) **a)** We **aligned all baseline model architectures and isolated parameters across stages** strictly (as illustrated in lines 301-304). This **ensures performance improvements are not attributed to parameter sharing or architectural interactions**. Our public experiments used DSSM and DIN (with attention), while online experiments used DSSM and MLP—**both setups cover mainstream architectures (DSSM for Retrieval, MLP for Pre-ranking, attention-based models [e.g., DIN] for Ranking) in recommendation/advertising cascade systems**. We believe this sufficiently demonstrates LCRON’s generalization across architectures.\n**b)** In Table 2, the metrics under \"ranking\" and \"retrieval\" reflect individual model performance on full samples, not their combined cascade performance. **The comparison between FS-LambdaLoss and LCRON highlights that neglecting inter-stage interactions may lead to suboptimal cascade results despite strong standalone model performance**. This confirms that LCRON priorities end-to-end cascade effectiveness over individual model optimization. Note that end-to-end recall is the golden metric for cascade ranking, as illustrated in lines 314-322. **c) About the complexity, please see the response to reviewer VkMd (for weakness 1)**.\n\n3) We prioritized significance testing (via unpaired t-tests with 5 runs per method) for ablation studies (Table 3), where smaller performance gaps required rigorous validation. The larger margins in baseline comparisons (Table 2) inherently imply statistical significance. Furthermore, online experiments confirmed LCRON’s statistically significant superiority over strong baselines like FS-LTR and ARF-v2 in public benchmarks. We believe these results collectively justify the robustness of Table 2’s conclusions. However, to address potential concerns about the statistical significance of improvements in Table 2, we conducted additional significance tests on the baselines. **The results are shown in the following table, formatted as mean±std(p-value). LCRON demonstrates statistically significant (p-value<0.05) superiority over all baselines on joint recall (namely end-to-end recall)**.\n\n|Method/Metric|JointRecall@10@20↑|\n|-------------|-------------------|\n|BCE|0.8539±0.0006(0.0000)|\n|ICC|0.8132±0.0003(0.0000)|\n|RankFlow|0.8647±0.0007(0.0000)|\n|FS-RankNet|0.7881±0.0007(0.0000)|\n|FS-LambdaLoss|0.8666±0.0016(0.0004)|\n|ARF|0.8608±0.0006(0.0000)|\n|ARF-v2|0.8678±0.0009(0.0000)|\n|LCRON(ours)|0.8732±0.0005|\n\nFor the missing reference: \n\nThank you for highlighting this work. We notice that it focuses on improving \"retrieval + ranking\" cascade systems but employs a non-learnable retrieval component (BM25) paired with BERT for ranking. Specifically, it jointly optimizes the number of retrieved documents N and the ranking model. This differs from our focus on joint learning across fully learnable cascade stages. We will discuss this work in the Related Work section to clarify its distinctions from our approach.\n\n&nbsp;\n\nDue to space limitations, we only show the key results (e.g., end-to-end recall) in the rebuttal text. Full additional results can be found in this anonymized github link: https://anonymous.4open.science/r/2025038594/\n\nIf you have any further questions or concerns, we will make every effort to provide further clarification.", "claims_and_evidence": "The paper introduces LCRON, a novel loss function for cascade ranking that simultaneously addresses two key challenges: misalignment of training objectives across stages and discrepancies between training and testing environments. By incorporating an end-to-end loss term alongside stage-wise losses, the method aims to improve coordination across ranking stages and enhance overall system performance.\n\nExperimental results indicate that LCRON achieves the highest end-to-end recall among the tested methods. However, a key limitation of the evaluation is that all baseline methods share the same model architecture and differ only in their loss functions. While this ensures a controlled comparison, it leaves open the question of whether the improvements stem from the proposed loss function itself or from interactions with the underlying architecture. Additionally, the most competitive baseline, FS-LambdaLoss, outperforms LCRON in both ranking and retrieval stages in terms of Recall, suggesting that while LCRON improves joint performance, its per-stage effectiveness varies. Furthermore, the overall gains in end-to-end recall, though positive, are relatively small, and validating them with statistical significance tests would strengthen the claims.", "methods_and_evaluation_criteria": "The benchmark datasets used in this paper are well-suited to the cascade ranking problem, providing a realistic testbed for evaluating multi-stage ranking systems. However, the authors limit their evaluation to a two-stage setup, even though the RecFlow benchmark includes four stages. While the approach can theoretically be extended to more stages, it remains unclear how the proposed surrogate loss function performs across all stages of a fully deployed cascade ranking system. Evaluating LCRON in a true multi-stage setting would provide deeper insights into its effectiveness and scalability.", "theoretical_claims": "The paper provides a clear explanation of the proposed loss function. However, I am unable to verify the theoretical claims due to my limited expertise in extensive mathematical proofs.", "experimental_designs_or_analyses": "Please see \"Claims And Evidence\" and \"Methods And Evaluation Criteria\"", "relation_to_broader_scientific_literature": "The paper brings up two important challenges in cascade ranking and suggests using surrogate loss functions to tackle them. Since cascade ranking is widely used in industrial recommendation systems, this is a valuable contribution to the field. However, the approach would be more convincing with a stronger set of experiments, especially testing across more stages, broader set of architectures, and validating the improvements with significance tests.", "essential_references_not_discussed": "The paper discusses key references related to cascade ranking and provides a solid foundation for its contributions. But it should be noted that, some closely related work on the interaction between retrieval and ranking stages, such as Stochastic Retrieval-Conditioned Reranking (Zamani et al., ICTIR'22), is not covered. Including such references could provide additional context and help position the proposed approach within the broader landscape of retrieval and ranking research.", "other_strengths_and_weaknesses": "Strengths: \n- The paper identifies key challenges in cascade ranking - misalignment of training objectives across stages and discrepancies between training and test environments; and introduces a novel loss function which optimizes both end-to-end performance and stage-wise alignment through a novel surrogate loss function.  \n\n- The method is evaluated on both public benchmark and an industrial dataset, with online A/B testing showing a 4.1% increase in revenue and a 1.6% increase in user conversions, highlighting its practical impact.  \n\nWeaknesses:\n\n- While the RecFlow dataset includes four ranking stages, the paper evaluates LCRON only on a two-stage setup, leaving it unclear how well the method scales to fully deployed multi-stage cascade ranking systems.  \n\n- All baselines share the same model architecture and differ only in their loss functions, limiting the scope of comparison, and the strongest baseline (FS-LambdaLoss) outperforms LCRON in both retrieval and ranking stages in terms of Recall, raising questions about whether the overall gains justify the added complexity.  \n\n- The reported improvements in end-to-end recall are relatively small, and the paper does not provide statistical significance tests to confirm their robustness, making it difficult to determine whether the gains are meaningful or within the margin of variance.", "other_comments_or_suggestions": "--", "questions_for_authors": "Please see above sections", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bJIlouOipr", "reviewer": "Reviewer_abHQ", "summary": "This paper proposes LCRON (Learning Cascade Ranking as One Network), a new method for optimizing cascading sorting systems. LCRON implements end-to-end training through two agent loss functions (Le2e and Lsingle) to ensure that the goals of each stage are consistent with the overall goals of the system and enhance the interaction between stages. Experiments have shown that LCRON outperforms existing methods in both public benchmarks and industrial applications, significantly improving ad revenue and user conversion rates.\n\n## update after rebuttal\nThanks for your careful response, and I consider the previous score reasonable and will keep the previous rating.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your detailed and insightful review.\n\nFor weaknesses 1 & 2: \n\nTo the best of our knowledge, we have already compared LCRON with existing multi-stage optimization methods. **To further validate the effectiveness of solely using differentiable sorting techniques (i.e., aligning model predictions with label permutation matrices through CE loss)**, which shares the same underlying rationale as FS-RankNet in making models fit complete orders, **we add two new baselines**. We conduct experiments on NeuralSort [1] and SoftSort [2].\nIn ablation studies, we separately validated the effects of $L_{e2e}$​ and $L_{single}$. Since differentiable sorting techniques form the foundation of both $L_{e2e}$ and $L_{single}$, the results of **\"NeuralSort\"** and **\"SoftSort\"**, which evaluate standalone differentiable sorting techniques, **also serve as an ablation of LCRON**. Moreover, **we further conduct experiments using the SoftSort [2] operator as the foundation of LCRON, to study its generalization capability across different differentiable sorting operators**. All additional results are shown in the following table, formatted as mean±std(p-value). Each method was run 5 times, and we conducted t-tests between LCRON (NeuralSort) and each of the other methods.\n\n|          Methods          |    Joint Recall (Golden Metric)     |   Recall of Ranking Model    | NDCG of Ranking Model  |  Recall of Retrieval Model   |   NDCG of Retrieval Model    |\n| ------------------------- | ----------------------------------- | ---------------------------- | ---------------------- | ---------------------------- | -----------------------------|\n| **NeuralSort** |        0.8210±0.0016(0.0000)        |    0.8233±0.0007(0.0000)     | 0.7138±0.0004(0.0000)  |    0.9469±0.0010(0.0000)     |    0.6979±0.0013(0.0000)     |\n| **SoftSort**              |        0.8103±0.0013(0.0000)        |    0.8138±0.0011(0.0000)     | 0.7148±0.0006(0.0000)  |    0.9386±0.0003(0.0000)     |    0.7066±0.0005(0.0000)     |\n| **LCRON(SoftSort)**       |        0.8723±0.0008(0.0615)        |    0.8720±0.0009(0.0485)     | 0.7246±0.0096(0.3395)  |    0.9703±0.0015(0.6750)     |    0.7035±0.0265(0.3782)     |\n| **LCRON(NeuralSort)**     |            0.8732±0.0005            |        0.8731±0.0004         |     0.7292±0.0008      |        0.9700±0.0004         |        0.7152±0.0009         |\n\nThe results show that LCRON(SoftSort) can also achieve significantly better performance than baselines. These experiments verify LCRON's generalization capability across different differentiable sorting operators, also suggesting that LCRON's effectiveness could benefit from more advanced differentiable sorting techniques.\n\nWe will add these results to the new version. If you have any further questions or concerns, we will make every effort to provide further clarification.\n\nReferences:\n\n[1] Grover, A., Wang, E., Zweig, A., and Ermon, S. Stochastic optimization of sorting networks via continuous relaxations. In ICLR, 2019.\n\n[2] Prillo, S. and Eisenschlos, J. Softsort: A continuous relaxation for the argsort operator. In International Conference on Machine Learning, pp. 7793–7802. PMLR, 2020.", "claims_and_evidence": "The paper's main claims are supported by convincing evidence. Extensive experiments on four datasets demonstrate LCRON’s superiority over state-of-the-art methods across multiple metrics.", "methods_and_evaluation_criteria": "The proposed methods make sense for the problem. LCRON innovatively introduces two novel surrogate loss functions (Le2e and Lsingle) to align the training objectives across multiple stages of cascade ranking, ensuring end-to-end optimization and enhanced interaction awareness between stages. The experimental evaluation utilizes the RecFlow benchmark dataset, which is specifically designed for cascade ranking systems and includes multi-stage samples from real-world recommendation systems. Additionally, the evaluation compares LCRON against state-of-the-art baseline methods, demonstrating its effectiveness in both public benchmarks and industrial applications.", "theoretical_claims": "I checked the correctness of the proofs for theoretical claims, including the derivation of the lower bound for the survival probability of ground-truth items in the cascade ranking system.  The theoretical analysis is sound and aligns with the experimental results, showing that Lsingle effectively reduces the gap and enhances the overall performance of the cascade ranking system. No significant issues were found in the theoretical claims.", "experimental_designs_or_analyses": "I checked the validity of the experimental designs and analyses. The experiments are conducted on the RecFlow benchmark dataset, which is specifically designed for cascade ranking systems and includes multi-stage samples from real-world recommendation systems. The results are evaluated using the Recall@K@m metric, which is a standard evaluation criterion for cascade ranking systems. Additionally, the paper includes an ablation study, streaming evaluation, and online A/B testing to comprehensively validate the effectiveness of LCRON. The issues are listed behind in the Weaknesses.", "relation_to_broader_scientific_literature": "LCRON advances cascade ranking by addressing limitations of traditional methods. It builds on interaction-aware training (e.g., RankFlow, FS-LTR) and differentiable sorting (e.g., ARF), introducing novel losses (Le2e, Lsingle) for end-to-end optimization and stage-specific supervision. These contributions align with broader trends in multi-task learning and differentiable techniques, offering a robust solution for cascade ranking systems.", "essential_references_not_discussed": "There are no related works that are not currently discussed in the paper.", "other_strengths_and_weaknesses": "The paper proposes LCRON (Learning Cascade Ranking as One Network), a framework for optimizing cascade ranking systems by introducing two new surrogate loss functions (Le2e and Lsingle) to align training objectives across stages and enable end-to-end optimization. This approach is original and significant, as it addresses key limitations in traditional cascade ranking training methods, such as misaligned objectives and insufficient stage interaction. The application to real-world recommendation and advertising systems further highlights its practical significance.\nWeaknesses:\n1. Comparison with Existing Techniques: While LCRON is compared with state-of-the-art methods like RankFlow and FS-LTR, a deeper comparison with other differentiable ranking techniques or multi-stage optimization approaches could further highlight its advantages.\n2. Ablation Studies: Although the paper includes an ablation study, additional experiments to isolate the impact of individual components (e.g., the role of differentiable sorting or the interaction between Le2e and Lsingle) would strengthen the claims.", "other_comments_or_suggestions": "I would like to learn about the authors' response to the weaknesses listed above, which may give me a clearer perspective on the paper's contribution.", "questions_for_authors": "I would like to learn about the authors' response to the weaknesses listed above, which may give me a clearer perspective on the paper's contribution.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Jkzhpq56ay", "reviewer": "Reviewer_VkMd", "summary": "The paper introduces LCRON (Learning Cascade Ranking as One Network), a novel end-to-end training framework for multi-stage ranking systems. LCRON formulates cascade ranking as a unified network with a new surrogate loss that aligns all stages with the overall top-$k$ selection objective​. In particular, it derives a differentiable lower bound on the probability that ground-truth items survive through all stages, and uses this as the primary training signal for the entire cascade.\n\nTo complement the end-to-end loss, the authors design an auxiliary per-stage loss that optimizes each stage’s Recall in isolation​. This single-stage loss (inspired by prior work on differentiable ranking) ensures each model selects ground-truth items from the full candidate set, mitigating issues like gradient vanishing and tightening the bound between the surrogate objective and true cascade performance.\n\nExperiments demonstrate significant improvements with LCRON over existing cascade ranking methods. On the public RecFlow benchmark (a multi-stage recommendation dataset), LCRON achieves the best end-to-end Recall under realistic streaming evaluations​. In an industrial online advertising platform, LCRON delivered substantial business gains (e.g. +4.10% revenue and +1.60% user conversions compared to the strongest baseline)​. These results validate that aligning training with the cascade’s global objective and enforcing cross-stage consistency leads to a more effective and robust multi-stage ranking system​.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed and insightful review.\n\nFor the questions:\n\n1) Yes, the abstract intends to state that L_single helps reduce the gap between $P_{CS}$ and $\\hat{P_{CS}}$. In lines 576-588 of the appendix, we explained the conditions for the gap to be reduced to 0 (i.e., the equation in eq14 holds). It can be found that the consistency of the top q2 sets selected by the two models is a sufficient condition for the gap reducing to 0, so optimizing L_single helps to reduce the gap. Indeed, if the role of auxiliary loss can be explained more intuitively in the abstract and introduction, it will make the article easier to understand. Thank you very much for the suggestion regarding the description of the abstract. However, it is a little difficult to intuitively explain why l_single works without a mathematical description. We welcome suggestions for clearer phrasing and will revise accordingly.\n\n2) Thank you very much for your careful evaluation of our work. There should be $P_{\\pi}$ rather than ${(P_{\\pi})\\_i}$, which is indeed a typo and leads to confusion. $P_{\\pi}$ represents the probability distribution of sampling to set $\\pi$. \nWhen $\\pi$ is given, $P_{\\pi}$ is a scalar. In addition, to make the description more rigorous, all $\\pi\\sim P_{M_1}^{q_1}$ in the text should be replaced by $\\pi\\sim P_{\\pi}$, indicating that $\\pi$ is sampled from $P_{\\pi}$. \n$P_{M_1}^{q_1}$ only represents the mean vector of the sampling distribution. We will fix these typos.\n\n3) Limited by our industrial scenario, we can only implement two-stage experiments. Thanks for your recognition of the comprehensive and solid experimental part. We also believe that the experiment in the scenario of M>2 can further verify the scalability of LCRON and enhance the depth of this paper. RecFlow contains data from more stages. We constructed a three-stage (M=3) cascade ranking system on the public RecFlow benchmark, using its prerank_neg, coarse_neg, rank_neg, rerank_neg, and rerank_pos samples (rerank_pos is the ground truth). The three stages utilized DSSM, MLP, and DIN architectures, respectively. The results are shown in the following, formatted as mean±std(p-value). LCRON still outperforms the baselines (statistically significant), showing the scalability of LCRON.\n\n|Method|Joint Recall|\n|--|--|\n|BCE|0.7191±0.0005(0.0000)|\n|ICC|0.6386±0.0071(0.0000)|\n|RankFlow|0.7308±0.0005(0.0000)|\n|FS-RankNet|0.6200±0.0010(0.0000)|\n|FS-LambdaLoss|0.7319±0.0038(0.0214)|\n|ARF|0.7256±0.0004(0.0000)|\n|ARF-v2|0.7332±0.0020 (0.0076)|\n|LCRON(ours)|**0.7390±0.0008**|\n\nFor the weaknesses:\n\n1) About Complexity & Deployment:\nDifferentiable sorting techniques such as NeuralSort and SoftSort typically have an O(n²) complexity, which is the same as common LTR methods like LambdaLoss, where n refers to the number of sampled items within a single impression. In real-world applications, n is usually not too large (e.g. n=20 in our system), LCRON incurs no additional training cost compared to the baselines.\nFor public experiments, we use A800 GPU to conduct public experiments. The GPU memory used for BCE, ICC, RankFlow, FS-RankNet, FS-LambdaLoss, ARF, LCRON are 28.4/28.9/28.9/28.4/28.9/28.4/28.9GB, respectively. The runtime of one epoch of them are 5358/5376/5057/5104/5076/6145/5339s, respectively. \nDeployment requires training stages in a single TensorFlow job and loading weights into separate meta files—it is not difficult for industrial serving teams (Appendix Lines 643–646). Moreover, $\\tau$ is the sole hyper-parameter of LCRON, thus its use typically requires only a small amount of hyperparameter tuning. To sum up, LCRON incurs no significant training/deployment overhead in real-world applications.\n\n2) Scalability: Please see the response to Q3 above.\n\n3) Regarding the robustness across capacities: Our original claim intended to highlight LCRON's consistent performance across different model architectures (e.g., DSSM+MLP, DSSM+DIN), suggesting robustness on capacities. We agree that the experiments do not directly validate robustness across model capacities. To address this, we will refine this claim of the abstract in the revised version and add a discussion in the Limitations and Future Work section to guide further exploration.\n\nFor the missing reference:\nThank you for pointing out this work. We will discuss FAA in the Related Work section to enhance the comprehensiveness of related work.\n\nFor the typos:\nThank you for pointing out these typos. Additionally, we noticed that the Recall and NDCG for each single stage in the manuscript were mistakenly swapped. All noted errors will be corrected, and we will thoroughly proofread the manuscript to ensure accuracy. \n\nDue to space limitations, we only show key results in the rebuttal text. Full additional results can be found in this anonymized github link: https://anonymous.4open.science/r/2025038594/\n\nIf you have any further questions or concerns, we will make every effort to provide further clarification.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "The paper’s theoretical core is the derivation of a lower bound for the joint survival probability of ground-truth items in the cascade. This is presented by defining an approximate joint probability $\\hat{P}\\_{CS}$ as the product of per-stage selection probabilities​ and showing that $\\hat{P}\\_{CS}$ is a provable lower bound of the true cascade selection probability $P_{CS}$​. The steps in this derivation (Eq. 7–8 in the paper) appear to be sound and mathematically correct, relying on the fact that certain fractions are always $\\leq 1$. The bound is logically valid given the independence assumption between stages’ selection events.\n\nThe authors claim that the auxiliary single-stage loss “tightens the bound” – i.e., reduces the gap between the lower bound $\\hat{P}\\_{CS}$ and the true $P_{CS}$. In Appendix A, they analyze the gap $\\Delta = P_{CS} - \\hat{P}\\_{CS}$ and relate it to the consistency between stages​. The theory suggests that if each stage more reliably retrieves the relevant item (as enforced by $L_\\text{single}$ on the full candidate set), then $\\hat{P}\\_{CS}$ approaches $P_{CS}$. This reasoning is sound in principle – a more consistent cascade (each stage capturing the true item) will make the product approximation more accurate. The provided equations support this claim, though they rely on expectations over permutations and some assumptions of stage-independence.\n\nOne minor concern is that the paper could better explain in words some of the theoretical conclusions. For instance, the phrase “drive the reduction of this bound”​ is used to describe the effect of the auxiliary loss – while the math shows $L_{single}$ helps close the gap, a clearer intuitive explanation of how reducing the bound gap translates to better Recall would help. Additionally, the notation $\\left(P_{\\pi}\\right)\\_i$ introduced in the theoretical section is a bit confusing​. It represents the $i$th component of the sampling distribution $P_{\\pi}$, but it initially looks like a stray parenthesis. This could be clarified to avoid any doubt in the correctness of the notation (see questions for authors). Overall, the theoretical content is sound; it supports the method’s design well, with just a few places where more explicit justification or clearer notation would strengthen confidence.", "experimental_designs_or_analyses": "The experimental evaluation is comprehensive and solid. The authors use both offline experiments (on a public dataset) and online experiments (live A/B test in an industrial system), which is a strong indicator of the method’s practical value. The offline experiments include streaming evaluations (multiple train-test splits over time)​, which adds credibility by simulating a real production training pipeline. These design choices make the results more trustworthy and less prone to overfitting on a single static split.\n\nThe paper includes analyses to isolate the impact of each component. Notably, an ablation study removes the end-to-end loss or the single-stage losses to show their contributions. The results (as described in the text) highlight that without $L_\\text{single}$ or without $L_{e2e}$, the performance drops, confirming that both are needed for the full benefit. This analysis supports the claim that the combination of losses is effective. There’s also mention of a sensitivity analysis (in Appendix C) for hyperparameters, indicating the authors checked robustness of the results with respect to loss weighting or other parameters – a good experimental practice.\n\nOne aspect not fully explored is the effect of cascade depth. The experiments seem to focus on a two-stage cascade (likely because RecFlow has two ranked stages). It would strengthen the paper to see an experiment with $M=3$ stages or more, to verify that LCRON’s benefits carry over to longer cascades. For example, adding an extra intermediate ranking stage and evaluating whether LCRON still outperforms baselines (and how the losses scale) would address any concern that the method is tuned specifically to two stages. This is a missing analysis that would be valuable, although its absence does not invalidate the current results – it’s more about demonstrating scalability.", "relation_to_broader_scientific_literature": "This paper positions itself relative to three main recent approaches in multi-stage ranking: RankFlow (Qin et al., 2022), FS-LTR (Zheng et al., 2024), and ARF (Wang et al., 2024). RankFlow introduced joint optimization via iterative feedback between stages, and FS-LTR proposed training all stages on the full cascade data (to mitigate sample bias)​. However, neither explicitly optimizes the final-stage metric. ARF introduced a differentiable surrogate loss for Recall but only for a single stage​. LCRON clearly extends these ideas: it combines the full-cascade training philosophy of FS-LTR with the metric-driven loss of ARF, achieving an end-to-end training that RankFlow attempted, but in a single unified model rather than an iterative process. This represents a notable advancement, as no existing approach simultaneously addressed both the sample bias and objective misalignment before (as the authors point out)​.\n\nThe cascade ranking concept isn’t new (e.g., Wang et al., 2011 introduced an efficient cascade ranker; Gallagher et al., 2019 (ICC) optimized fused stage scores via LambdaRank​). LCRON differs by focusing on directly optimizing the selection probability of relevant items. Earlier methods often optimized proxy objectives (like combined scores or separate stage objectives) and could suffer from stage inconsistency or bias. By using differentiable sorting and a probabilistic formulation, LCRON provides a more principled end-to-end solution. This is a meaningful improvement on the foundations laid by those works, aligning with a broader trend in ranking research to move from heuristic multi-stage training to theoretically grounded joint optimization.\n\nCompared to FS-LTR (which is a strong recent baseline), LCRON adds the missing piece of objective alignment. FS-LTR trained all stages together but still used traditional loss functions per stage, whereas LCRON introduces losses that correspond to Recall directly​. Similarly, compared to ARF’s single-stage recall optimization, LCRON shows how to incorporate that idea across an entire cascade and handle the interactions between stages (for example, by multiplying probabilities from stage 1 and stage 2). The results in the paper demonstrate that these improvements are not just theoretical – LCRON outperforms ARF and FS-LTR in practice, indicating the combination of techniques is effective.", "essential_references_not_discussed": "Fine-Grained Stage Alignment: One relevant recent work that was referenced but not discussed in detail is the FAA: Fine-grained Attention Alignment for cascade ranking (Li et al., 2023)​. FAA addresses the cascade ranking problem by aligning representations (attention) between stages. While LCRON approaches cascade optimization from a loss function perspective, FAA’s approach is complementary – focusing on feature consistency. A brief discussion of how LCRON’s objective alignment differs from or could be combined with representation alignment (as in FAA) would strengthen the related work section.", "other_strengths_and_weaknesses": "Strengths: \n\nThe idea of training all cascade stages as one unified network with a bound-approximating loss is a notable innovation. While it builds on elements from prior work (full-stage sampling, differentiable sorting), the particular combination – especially the novel end-to-end loss formulation ($L_\\text{e2e}$) and the bound-tightening strategy with $L_\\text{e2e}$ – is original. This approach has not been explicitly done before in the literature, making it a fresh contribution.\n\nThe contributions have high significance for both research and industry. Improving multi-stage ranking has direct implications for large-scale recommender systems and search engines. The fact that LCRON showed measurable gains in a production environment (revenue and conversions) indicates that this method can impact real systems, not just benchmark scores. For the research community, LCRON’s approach could inspire more work on global objective optimization in cascades and on using differentiable ranking techniques in multi-stage pipelines. It effectively addresses a gap in the literature, so its acceptance would add valuable knowledge and potentially a new baseline for others to compare against.\n\nWeaknesses: \n\nOne weakness is the complexity introduced by the method. Training with differentiable sorting and multiple loss components (even if combined via learned weights) can be harder to implement and tune than traditional methods. The paper mitigates this by using the UWL scheme to automatically balance loss weights, but the approach still requires careful engineering (e.g., setting the softmax temperature $\\tau$ for NeuralSort). Another minor weakness is that some claims (like robustness across model capacities) were not directly verified, as noted earlier. Lastly, the method’s benefit was clearly shown for two-stage cascades; it’s an open question how it performs with more stages or in scenarios with dramatically different stage characteristics (this could be explored in future work). These weaknesses, however, are not fundamental flaws but areas to keep in mind when applying the method.", "other_comments_or_suggestions": "1. Typos/Grammar: There are a few minor typos that should be corrected for the camera-ready version. For example, the abbreviation explanation for LCRON in the introduction reads “Learnig Cascade Ranking as One Network”​, missing an “n” in “Learning.” Such small errors should be fixed for clarity. Also, “auxillary” should be “auxiliary” (noticed in a section heading or text around L1095-L1103​).\n\n2. The notation $(P_{\\pi})\\_i$​ is a bit confusing – it looks like a stray parenthesis. It would be clearer to denote this as $P_{\\pi}(i)$ or something like $P_{\\pi_i}$ if the intent is to index the vector $P_{\\pi}$. Additionally, in Equation 10 (the definition of $P_{M_i}^{q_i}$ via the soft permutation matrix), it might be clearer to denote the resulting probability as $\\hat{P}\\_{M_i}^{q_i}$since it’s obtained through an approximation (soft sorting). Consistently using the hat notation for approximated probabilities (as done for $\\hat{P}_{CS}$) would help the reader keep track of what is exact vs. relaxed.", "questions_for_authors": "1. In the abstract, you write “design an auxiliary loss for each stage to drive the reduction of this bound”​. Could you clarify this phrasing? Does it mean that optimizing the auxiliary loss empirically tightens the lower bound (i.e., increases $\\hat{P}\\_{CS}$ toward $P_{CS}$)? Please elaborate on how $L_\\text{single}$ concretely contributes to reducing the gap between the lower bound and the true joint probability – for instance, is there a way to measure or prove this reduction happens as $L_\\text{single}$ is minimized? A bit more intuition here would help: we see the math in Appendix A, but a clearer explanation of how the auxiliary loss “drives the bound’s reduction” would be appreciated.\n\n2. Notation $(P_{\\pi})\\_i$ – is this a typo? In Section 4.2, you define $P_{\\pi}$ as the probability of a sampling $\\pi$, and then use the notation $(P_{\\pi})\\_i$​. This notation was a bit hard to parse – it looks like $P_{\\pi}$ might be a vector or distribution, and you’re referring to its $i$-th component. Could you confirm what $(P_{\\pi})\\_i$ means exactly? If $P_{\\pi}$ is a distribution over sampled sets, perhaps $(P_{\\pi})\\_i$ is the probability that a particular item $i$ is included? The parentheses made it read like a possible typo. Should it be $P{\\pi_i}$ (the probability of a specific permutation) or $P(\\pi_i)$? Clarifying this would help readers follow the derivation in Eq. 6 without confusion.\n\n3. Have you considered or tested LCRON on a cascade with three stages (or more)? While RecFlow provides two-stage data, it would be insightful to know how the approach scales to an additional stage. For example, if we had a retrieval model $M_1$, an intermediate ranker $M_2$, and a final ranker $M_3$, would the LCRON formulation easily extend (with $L_\\text{e2e}$ using $\\prod_{i=1}^{3} P^{q_i}\\_{M_i}$ and each stage having $L_\\text{single}$)? If you have any preliminary results or reasoning for $M=3$, please share them. This would help convince readers (and practitioners) that LCRON generalizes beyond the two-stage scenario. Are there any expected difficulties for $M>2$ (e.g., increased gradient variance or more hyperparameters), or does it plug in seamlessly?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper presents LCRON, a principled end-to-end training framework for cascade ranking that introduces a novel surrogate loss aligning stage-wise objectives with the global top-k selection goal. The well-motivated submission addresses key limitations in current multi-stage ranking systems, namely inter-stage misalignment and training-test discrepancies. The theoretical formulation is sound, and the experimental results are strong, demonstrating consistent improvements over state-of-the-art baselines across both public and industrial benchmarks. The reviewers highlight the paper’s practical relevance, methodological novelty, and strong empirical validation. Final scores reflect a consensus leaning toward acceptance, with three reviewers recommending weak acceptance and two recommending clear acceptance. I recommend acceptance of this paper if there is room.
Decision: Accept (poster)