Unified Off-Policy Learning to Rank: a Reinforcement Learning Perspective
Authors: Zeyu Zhang
Yi Su
Hui Yuan
Yiran Wu
Rishab Balasubramanian
Qingyun Wu
Huazheng Wang
Mengdi Wang
Venue: NeurIPS 2023 poster (2023)
Keywords: learning to rank
off-policy learning
reinforcement learning
click model
TL;DR: We proposed a unified off-policy learning to rank method that is agnostic to underlying click model.

Abstract: Off-policy Learning to Rank (LTR) aims to optimize a ranker from data collected by a deployed logging policy. However, existing off-policy learning to rank methods often make strong assumptions about how users generate the click data, i.e., the click model, and hence need to tailor their methods specifically under different click models. In this paper, we unified the ranking process under general stochastic click models as a Markov Decision Process (MDP), and the optimal ranking could be learned with offline reinforcement learning (RL) directly. Building upon this, we leverage offline RL techniques for off-policy LTR and propose the Click Model-Agnostic Unified Off-policy Learning to Rank (CUOLR) method, which could be easily applied to a wide range of click models. Through a dedicated formulation of the MDP, we show that offline RL algorithms can adapt to various click models without complex debiasing techniques and prior knowledge of the model. Results on various large-scale datasets demonstrate that CUOLR consistently outperforms the state-of-the-art off-policy learning to rank algorithms while maintaining consistency and robustness under different click models.

Unified Off-Policy Learning to Rank: a Reinforcement Learning Perspective
Zeyu Zhang'
Yi Su Hui Yuan
Yiran Wu
Rishab Balasubramanian"
Qingyun Wu Huazheng Wang
Mengdi Wang'
University of Science and Technology of China Google Deepmind SPrinceton University
*Penn State University 'Oregon State University
zgkd2019zzy@mail wsw. edu. cn yisuntv@google. com
(huiyuan, mengdiw)@prinneeton edu (ykw5399, qingyun. wu] uus-psu. edu
{balasuri, huazheng- wangh@oregonstate edu
Abstract
Off Off-policy Learning to Rank (LTR) aims to optimize ranker from data collected by a deployed logging policy. However, existing off-policy learning to rank methods often make strong assumptions about how click model, and hence need to tailor their methods specifically under different click models. In this paper, unified the ranking process under general stochas- tic click models as a Markov Decision Process (MDP), and the optimal ranking could be learned with offline reinforcement learning (RL) directly. Building upon this, we leverage offline RL techniques for off-policy LTR and propose the Click Model-Agnostic Unified Off-policy Learning Rank (CUOLR) method, which could be easily applied to wide range of click models. Through dedicated formulation of the MDP, show that offline RL algorithms can adapt to various click models without complex debiasing techniques and prior knowledge of the model. Results on various large-scale datasets demonstrate that CUOLR consis- tently outperforms the state-of-the the--ar art off- policy learning to rank algorithms while maintaining consistency and robustness under different click models.
generate the click data, i.e., the
Introduction
Learning Rank (LTR) is a core problem in Information Retrieval (IR) with wide applications such web search and recommender systems 33 Traditionnl LTR methods require high-quality annotated relevance judgments for model training, which expensive, time-consuming, and may not align with actual user preferences [41] As result, learning to rank with implicit user feedback, such as logged click data, has received a huge amount of attention in both academia and industry 23 26 53].
Despite its low cost, learning to rank directly from implicit user feedback could suffer from the intrinsic noise and bias in user interactions, e.g., position bias, where an item displayed at higher position receives a higher click-through rate (CTR) than its relevance [12] To mitigate the bias in the click data, off- off-policy learning to rank methods have been proposed under different bias assumptions such as position bias [33 2. selection bias [52 and trust bias 36 9 major branch of off off-policy
learning to rank called counterfactual learning to rank achieves unbiasedness by re-weighting the samples using the inverse propensity scoring (IPS) method [46] 2 27]. To estimate the propensity from logged click data, existing works require explicit assumptions about how users examine the rank list and generate the click data, i.c., click models [12 For example, position-based model
"oorrespondence to: Yi Su and Huazheng Wang
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

assumes the probability of examining result only depends on the position; while cascade model (CASCADE) [2] assumes each click depends on the previous click, and dependent click model (DCM) I8 considers both. Different debiasing methods have been proposed to cator to specific click models, including PBM 53 26]. CASCADE; and DCM [48]. However, prior knowledge of the click model is usually unknown and the correct click model needs to be identified from user behavior data before applying an off-policy algorithm, which is challenging in complex real-world environments. Besides, many popular and powerful click models have not been studied in counterfactual learning to rank such as the click chain model (CCM) [77] and the user browsing model (UBM) [13]. It requires significant amount of work to study debiasing methods for every popular click model.
(PBM) 24
To overcome these issues, we propose to study a unified approach of off-policy learning to rank adaptable to general click models. Our key insight that the user's examination and click behavior summarized by click models has Markov structure; thus off-policy LTR under general click models can be formulated as a Markov Decision Process (MDP). Specifically, the learning to rank problem now can be viewed as an episodic RL problem 45 T. where each time step corresponds to ranking position, each action selects a document for the position, and the state captures the user's examination tendency. This formulation allows to view off-policy LTR from the perspective of offline reinforcement learning [3I], where we can leverage off-the- thelf offline RL algorithms [29 [9 [6] to optimize the ranking list. Importantly, our formulation bridges the area of off- oiflpolicy learning to rank and offline RL, allowing for the integration of ideas and solutions from offline RL to enhance the solution of the off- off-policy LTR problem.
Inspired by the formulation, we propose the Click Model-Agnostic Unified Off-policy Learning to Rank (CUOLR) method. We first construct each logged query and ranking data as an episode of rein- forcement learning following the MDP formulation. Our dedicated structure for state representation learning can efficiently capture the dependency information for examination and click generation, e.g. ranking position in PBM and previous documents in CM and DCM. The algorithm jointly learns state representation and optimizes the policy, where any off-the-shelf offline RL algorithm can be applied as a plug-in in solver. Specifically, we adapt the popular CQL algorithm [29] as an instantia- tion, which applies the conservative (pessimism) principle to Q function estimate. We evaluate algorithm on real-world learning to rank datasets [38] 7 under various click models. Compared with off-policy LTR methods that are dedicated to specific click models, our click model-agnostic method consistently outperforms the best-perforning baselines in all click models.
The contributions of this paper are summarized follows:
We formulate the off-policy LTR with biased feedback under general click model as a Markov Decision Process, and bridge the area of off off-policy learning to rank and offline reinforcement learning.
We propose CUORL, Click model-agnostic Unified Off-policy LTR method that could utilize any offline RL algorithm as a plug-in solver, and we instantiate it using CQL.
We conduct extensive empirical experiments to validate the effectiveness of our algorithm using real-world LTR datasets under different click models.
Related Work
Off-policy Learning to Rank. Off-policy Learning to Rank aims to optimize the ranking function from logged click data (23). The majority of the works aim to mitigate the bias in logged click data, known as counterfactual learning to rank or unbiased learning to rank. The debiasing methods mainly
follow inverse propensity scoring strategy (26) 49 4. [1 48 3, while there are also recent works applying doubly robust estimator to reduce variance 40 28
Cief et al. [T] proposed pessimistic off-policy optimization for learning to rank that also mitigates bias but not in unbiased way. All these methods rely on prior knowledge of the click model 12 10), while our algorithm is agnostic to general click models.
learn policy from large logged datasets where the distributional shift between the logging policy and the learned policy imposes major challenge. In this setting, different algorithms proposed, from value-based ones
Offline Reinforcement Learning. Offline RL algorithms 22 14
16 13

---

(e.g. [35 344 to policy-basse based ones (e.g. 43 42
Among the vast literature offline reinforcement
learning, the principle of pessimism/coonseraaisn 29
is an important line and has inspired
many algorithms from empirical and theoretical perspective [60 39
32
56
61
55
58 While all the
aforementioned methods can plugged into our algorithm, we choose the classic CQL. algorithm 29 with a conservative Q function on top of soft actor-critic algorithm [9]
Reinforcement Learning to Rank. Wei al. [54] first model ranking problem an MDP, where the state is the candidate document set at current rank and the action is the selected document. [57 62 have been studied under similar MDP formulation. However, [54] 57 requires relevance labels as feedback and cannot mitigate bias in click data; [62] is an online learning algorithm that learns from user interactions instead of logged data. Compared these studies, we characterize the MDP formulation from a different perspective. i.e., capture bias in the click model, and propose the offline RL algorithm with logged click feedback.
3 Reinforcement Learning to Rank: A Unified Formulation
As the majority of existing works in unbiased learning to rank focused on inferring documents' relevance from the click models, these methods are tied to specific click models adopted. In this section, we formulate learning rank with general click feedback as a Markov decision process, offering a unified and comprehensive modeling of possibly complicated user behavior. This formulation unifies LTT problems associated with different click models, under which the underlying click model is translated into the environment setup of MDP such as state transitions and rewards. opens up the possibility to employ rich class of reinforcement learning algorithms for solving LTR, which we will give greater details in the next section.
3.1 Preliminary
Click model. A key challenge of off-policy LTR lies in learning the document's attractive- ness/relevance from the implicit feedback that is biased by the user's examination behavior. To address this challenge, a range of click models have been proposed to accommodate user's 's various click behavior [1O]. In this study, we focus
general family of click models [63 30), which is marked by two characteristics: (1) Most of the mainstream click models have a "two-step" flavor that breaks user's click behavior towards some document down into the user's examination and the document's 's relevance. For each document d, the user first decides whether to examine it in the ranking list, based the specific behavior. Mathematically the user behavior is modeled as the examination probability, which generally depends on the ranking list R and the position of the document k; denote XRR, k). Once the document is examined, the user will choose whether to click it, based on the attractiveness (d)) [ (2) Any documents under the k-th position do not have an effect on x X(R, k)).
Definition (Click Model), For any rank list R and position k the attractiveness and examination probability are independent.
P(C+ = 1 = k)(((()
(I)
where Os the click indicator rank k and x(R, k) the examination probability of position le in the rank list R. For each document d, the attractiveness add) only depends on the document itself. And the attructiveness is mutually independent.
We show that classic click models such as PBM. CASCADE, DCM, and CCM are instances of Definition D with details listed in Appendix C
3.2 Learning to Rank as Markov Decision Process
In Reinforcement Learning (RL), the interactions between the agent and the environment are often described as an fiite-hhrizzn discounted Markov Decision Process M = (S, A, T, T.T.". H). The goal of the RL problem is find a policy that maximizes the value function, i.e. the discounted
We simplify the notation and assume d captures (query, document) pair information on given query.
3

---

cumulative reward
E
rrr(s(; ai) IT, 50
(2)
t=0
In what follows, we formulate each of the A, T.r, 7, H) components in the ranking scenario. Our formulation essentially differs from the existing MDP formulation of ranking [54], where the state at position k is defined as remaining documents that are yet to rank following the k
ranked ones on
the top, instead Wei al. [54] has a limited capturing of user's click behavior as a result of being
ignorant of the ordering within the top k 1 documents. From here on, we use &o E [K] to denote the
k-th position top down on a list total length K. serves as the counterpart the time step in (2) for the ranking setting.
State For each position & E [K], state sk should include and represent the current status of the ranking that the agent is faced with. Thus, we define the state at rank &t as:
Sk = [[d1, d2.................. .., dx-1) 1), k),
which is a concatenation of the established sub-ranking list up to k. denoted by (d1, d2,... d2-1). 1). and the position Here d, refers to the document presented rank with sg is initialized as [0.0]. Together with the action aş as to select document d2 presenting at rank k defining sy as (3) fully captures the user's click behavior Cg at this point. Recall (I) that P(Cx = R, XR, ())())))) where x(R, k) is determined by (d1, d2..... dg-1). To better capture the rich information in the observed state, we discuss how to attain the effective state embedding from the raw representation in Section
Action A Action ag naturally defined as the document to present at rank k given state sg. In our experiments, each action is represented by feature vector associated with the query. It is worth mentioning that the available action set A& at each k is related to state 8k as well as the specific query, unlike the common case where the action space is fixed. There are two main differences here compared with the fixed action space: (1). the available actions vary under different queries; and (2). once an action is chosen, it is removed from the candidate set Ar.
Transition ((s|ss,,) Transition maps state-action pair to the probability distribution over pos- sible next states. Given our formulation of state and action, the next state 8k-1 is deterministic given 8 [(d1, d2, dx-1) 1), k:] and ag- Formally, (S*++1/8k, ax) 1 if and only if 8 [[d1 d2, ... dd-1, ax), + 1]. Note that with this transition function, we can show that our model a rigorous MDP, where the distribution of next state only based on the current state and the action.
Reward r(s,a) Aligned with the goal LTR to maximize total clicks, we adopt the binary click reward at each position, i.e. r(sk, ag) Cg. It is easily checked this is well-defined reward from ( that the distribution r is fully determined by s and a, i..., E[r(sk, ax.)] = x(s))a((a))
Putting the components together we have formally built up our MDP formulation of LTR,, which we name as "MDP for Ranking" and denote by MR A, Tr,,,, H) with components defined as above. The rich RL literature has the convention to solve the optimal policy x² that maximizes the cumulative reward, which in our proposed MDP translates
K
Σ
r r(k, T( (1((ss
argmax
where the expectation is taken over the stochasticity in both environments MR and policy Before leveraging any RL algorithms for ranking, it is necessary to validate good policies of MR(S, A, T.r,Y. Y. K) yields good ranking lists. In the following subsection, we give rigorous theorem for this validation.
3.3 Optimizing Rank List by Optimizing Policy
Constructing rank list given policy π. With the definition of MDP, we can construct a rank list with any given policy sequentially. At each position k the state is constructed based on previous document features (manually or by sequential models). Then document (action) is chosen by the policy æ and placed at position k. where the next state determined accordingly. Repeat this process at each position until all the documents are set or the list reaches its end (K 10 for example).

---

Definition 2 (Policy Induced Ranking.). Given policy x(- s) of the induced rank list Rº as
construct
(k) f ag T(- sk).
To investigate whether the optimal rank list can be captured by optimal policy N², we start with defining the optimality of rank list.
Definition 3 (Optimality). rank list R. is optimal if and only if all documents are sorted in descending order in terms of their attractiveness, i.e.
a(R(1)) 2... a(R(K))
(((()((((K))
and (R(1), (R(K)....R(KK)) are the K most attractive documents among all, where K is the length of the list and is also called the top-K ranking.
Assumption 3.1 (Optimality of optimal ranking). Let VR(81) = E r R(k)) be the value of rank list R. and let R be the optimal rank list by Definition Then max VR (31) Vi- (81).
Assumption 3.1 is adopted from Assumption 2 in 30), which suggests optimal rank list sorted by decreasing attractiveness of documents leads to optimal rewards, which will be covered by optimal policy learned from MDP formulation. This is mild assumption classic click models such as PBM and cascade model all satisfy the assumption 63
4 Unified Off-policy Learning to Rank
The formulation of off-po learning-tor to-rnkkkt--nkkkkk (LTR), when viewed from the perspective of offline reinforcement learning, presents opportunity leverage off-the- RL algorithms to address off-policy LTR problems. In this section, we introduce a novel and unified off-policy LTR algorithm that is agnostic the underlying click model used to generate the offline training data. Our algorithm composed of three key components: (1) episodes constructed from the logged ranking data; (2). state representation learning; and (3). policy optimization via offline RL algorithms. In the following, we provide a detailed exposition of each component.
Episodes Construction. Given any logged ranking dataset, the first step is construct se- ries of episodes from the ranking data, making it compatible with the off-the-shelf RL algorithms. Specifically, our original ranking data {(g,, R2 (q, R.)) is composed of tuples with con- textual query feature gis a ranked list R; with K documents and a corresponding click vector cg R2) {0, 17 From the perspective of RL, this tuple can conceptualized as one episode, following the established MDP formulation of the ranking process. In particular, for each tuple (gi, R2, cg R,)), we transform it to a length K episode Ti
{(s) a ri with $ = (RRa(: k), = R,(k), ri = cg [k)
for Ak E [K]
Here we use R. A:] to denote the concatenation of the document feature vectors before position and [k( represents the document feature position k, similarly for cg R.) In particular, the episode т; is constructed by going through the ranked list from top to the bottom. Each state s contains all the document information before position i and the position information k represented as function of R.[: k] and & with ó being a learned embedding function that we will discuss shortly, The action step k is the document placed at position k i.e., Re(k:, with the reward at current timestep is the binary click for the corrpsonding action at position k, i.e., c,(ys, [kk). Given this, we have constructed an offline RL dataset with episodes with length K each.
State Representation Learning. In RL, state representation learning offers an efficient means of managing large, raw observation spaces, enhancing the generalization performance when encountering previously unseen states. In our particular setting, the observation space consists of raw document features up to position k: As we will show Section 5.2 utilizing raw observations as the state representation in the RI. algorithm can lead to highly sub-optimal performance, due to both unscalability with respect to & and limitations in the representation power. Rather than incorporating additional auxiliary tasks to explicitly learn state representations, we propose to jointly learn the state
5

---

representations together with the policy optimization algorithm, which aims to automatically learn state representation & that will benefit the downstream policy optimization task. For example, DQN uses multiple layers of nonlinear functions to encode the map perceptual inputs to a state embedding that could be linearly transformed into the value function. To this end, we introduce the implicit state representation learning component our off-policy learning to rank algorithm, which is composed of the following key components:
Positional Encoding: To effectively inject the position information in sj. we utilize the positional encoding technique 59], ensuring the model make use of the position information when generating clicks. Specifically, positional encoding represents each position in the ranked list by the sinusoidal function with different frequencies such that
k
2 = sin 1000025/dmmco
PE(K)22+1 cos
1000025/dmmar /dmoora
Here PE( Rd with dl being the dimension of document feature and 2i and 2: being the even and the odd entries of (E(K) with 24, 2i + E [d].
Multi-head Self-attention: The other challenge in our case is to find a specific architecture for the state representation learning that tailored for the learning rank task. As the ranked list of document features is inherently sequential, we leverage the multi-head self-attention mechanism to learn the state embedding ø. Specifically, the state ii is defined as:
$ = (RRa k)) = Concat(head,,.... head, W
where head, Attention(s: W.O 5% W.K Sk W.Y. with W.O. W,K, W. th are learnable parameters for the 7t head and W is the learnable parameter of the output layer after concatenating the results of all the / heads; At each position kt we concatenate the document features R.[: A] along with the position embedding for position k passing them as the input for the multi-head self-attention.
Joint Representation Learning and Policy Optimization. Given the constructed offline dataset and a dedicated structure for learning efficient state representations, we now demonstrate how we leverage any off-the-shelf RI. algorithm as the plug-in solver for finding the optimal policy. We use the popular offline RI. algorithm: CQL,, as an instantiation, which learns a conservative Q function and utilizes the classical soft actor-critic algorithm on top of it. Specifically, it optimizes the following lower bound of the critic (Equation 5):
[Q00- 100--arul Q0
21
i < argmin o BEDD
log exp(Qo(s,a)) - (aa (Q0((,
0
(5)
where - r = is the estimated Bellman operator, TT is the logging policy. Here we use 6' to emphasize that the parameters in the target Q network is different from policy Q network. The conservative minimizes the expected Q value for -ut-o--distribution (state, action) pairs and prevents the Q-function's 's over-estimation issue. Built upon SAC, the algorithm improves the policy πe (i.e., the actor) based on the gradient of the estimated Q function, with entropy regularization. Compared with the original CQL. algorithm, we also add the state representation learning component, and we jointly optimize the state embedding, the critic and actor parameters, with a detailed algorithm included in Algorithm[[] Once we have obtained the learned optimal policy Tē. extract the optimal ranking policy following Definition 2
5 Experiments
We empirically evaluate the performance of our proposed method CUOLR several public datasets, compared with the ofte-oo-tthearr off-policy learning-to-rant methods. Specifically, we aim to assess the robustness of our method across different click models and showcase the effectiveness of our unified framework. Beyond this, we perform the ablation study to examine the efficacy of our proposed state representation learning component.
CCodes: http://wwt...tttoooo///////////d
Neur ip22033

---

Algorithm Click Model-Agnostic Unified Off Learning to Rank (with CQL)
1: Inputs: logged ranking data {(ys, Ra, Ci Re)) length of the ranking K, batch size B,
train iteration T.
2: Initialize: policy "g and Q function Qa. embedding model (:(::)
3: for E [2'] do:
4: Randomly sample batch of queries Q with size B.
5: Construct offline RL episodes 7 = {(( oj ri] tg
$ = のe [R. kj, k), d = R.(A,, r - Ra( for vk E [K]
6: Train the Q-net (and embedding model) with loss defined in equation (5)
0 0 лα VoLoss T) + - 3b - no V Loss(O,) T)
7: Improve policy me (and embedding model) with SAC-style entropy regularization.
& + E + BnnBsTTaanztttt [Qa (s. a) log गT (a s)]
t v + nolE ann (Q a) log गH (a s)]
8: end for
9: Output: learned ranking policy म.. Q function 2z. embedding model 6 (..))
Recover the optimal ranking from the learned policy _e using Definition 2
5.1 Setup
Datasets. We conduct semi-synthetic experiments on two traditional learning-to-rante benchmark datasets: MSLR-WEBI0G and Yahoo! LETOR (set 1). Specifically, we sample click data from these real-world datasets, which not only increases the external validity of the experiments but also provides the flexibility to explore the robustness of our method over different click models. For both datasets, it consists of features representing query and document pairs with manually judged relevance labels ranging from (irrelevant) to 4 (perfectly relevant). We provide the statistics of all datasets in Appendix A Both datasets come with the train-val-test split. The train data is used for generating logging policy and simulating clicks, with the validation data used for hyperparameter selection. And the final performance of the learned ranking policy is evaluated in the test data.
Click Data Generation. We follow Joachims al. 2.5 to generate from the full-information relevance labels. Specifically, we first train a Ranking SVM [26] using 1% of the training data query we get a ranking R; and simulate the clicks based on various click models we use. As discussed in Section 3.1 there are two components for the click generation, examination probability and attractiveness of the document for the query. All click models differ in their assumptions on the examination probability. For PBM, adopt the examination probability p 1k estimated by Joachims et al. (25] through eye-tracking experiments:
click data
logging policy T. to present the initial ranked list of items. For each
xRR,, k) x(k) = P%
where 17 [0, +oo] is hyper-parameter that controls the severity of presentation biases and in our experiment, we set 1/ = 1.0 as default. For CASCADE, the examination probabilities are only dependent on the attractions of each previous document. For DCM, the Äs also simulated by the same parameters as PBM examination probability. We use the same attraction models for all click models, defined following:
(7( 20max
a(d) = = (1
where r(d) E j0, 4] is the relevance label for document d and Tmaz is the maximum relevance label. We also use e to model click noise so that irrelevant documents have non-zero probability to be treated as attractive and being clicked.
7

---

Table 1: Performance comparison with different click models on Yahoo! LETOR setl and MSLR- WEB10K. "e" and "**" indicate statistically significant improvement (p-value < 0.05 and p-value < 0.01 respectively) over the best baseline for each metric respectively.
Yahoo! LETOR
MSLR-WEB
CLICK MODEL
ALG
ERR@K
NDCG@K
ERR@K
NDCG@K
K=5
K=10
K=5
K=10
K=5
K=10
K=5
K=10
DLA
0.439
0.455
0.691
0.742
0.256
0.278
0.356
0.384
PBM
CM-IPW
0.440
0.456
0.692
0.743
0.255
0.277
0.354
0.383
IPW
0,446
0,462
0.700
0.748
0.281
0.301
0,367
0,390
CUL
0.458
0.473
0.700
0.753"
0.281
0.303
0.380"*
0.406"*
CUOLR(SAC)
0.459
0.478 "*
0.700
0.753
0.279
0.301
0.379
0.404
DLA
0.441
0.457
0.690
0.741
0.265
0.286
0.365
0.392
CASCADE
CM-IPW
0.444
0.460
0.696
0.745
0.283
0.304
0.378
0.403
IPW
0.442
0.457
0.690
0.740
0.257
0.279
0.358
0,387
CUL
0.459
0.479
0.696
0.748
0.280
0.301
0.378
0.404
CUOLR(SAC)
0.461
0.478
0.696
0.748
0.279
0.301
0.379
0.405
DLA
0.444
0.459
0.696
0.745
0.279
0.299
0.378
0.404
DCM
CM-IPW
0.447
0.463
0.704
0.752
0.284
0.304
0.379
0.402
IPW
0.444
0.459
0.697
0.746
0.278
0.299
0,363
0,387
CUOLR(CQL)
0.461 **
0.474*
0.699
0.753
0.278
0.300
0.380
0.405
CUOLR(SAC)
0.461
0.475
0.703
0.755
0.278
0.300
0.378
0.403
LOGGING 0.385
0.403
0.630
0.693
0.206
0.230
0.304
0.338
ORACLE 0.462
0.476
0.739
0.781
0.328
0.347
0.432
0.453
Baselines and Hyperparameters. We compare our CUOLR method with the following baselines: (1). Dual Learning Algorithm (DLA) 44] which jointly learns an unbiased ranker and unbiased propensity model;
(2). Inverse Propensity Weighting (IPW) Algorithm 52 26 which first learns the propensities by result randomization, and then utilizes the learned probabilitiess to correct for position bias; and (3). Cascade Model-based IPW (CM-IPW) [48] which designs a propensity estimation procedure where previous clicks are incorporated in the estimation of the propensity. is worth mentioning that (1) and (2) are designed for PBM and (3) is tailored for cascade-basee models. Besides, train LambdaMart ranking model, ORACLE for short. The performance of the logging policy (LOGGING) is also reported as the lower bound of the ranking model.
6 model with true relevance labels the upper bound for the
For baselines, we use a 2-layer MLP with width 256 and ReLU activation according to their original paper and codebase 4 51 47 For the embedding model in method, with heads. And for actors and critics in CQL and SAC algorithms, we utilize a 2-layer MLP with width 256 and ReLU activation. The conservative parameter (marked red in Equation (5) in CQL is set to 0.1. We use Adam for all methods with a tuned learning rate using the validation set. More details are provided in Appendix A
multi-head attention
Metrics. We evaluate all the methods using the full-information test set. We use the normalized Discounted Cumulative Gain (nDCG) [20] and the Expected Reciprocal Rank (ERR) [8] as evaluation metrics and report the results at position 5 and 10 demonstrate the performance of models at different positions.
5.2 Results
How does CUOLR perform across different click models, compared the baselines? To validate the effectiveness of our CUOLR method, we conducted performance evaluations across a range of click-based models, including PBM, CASCADE, DCM, and CCM. We compared our approach with state-of-the-an baselines specifically designed for these click models, namely DLA, IPW, and CM-IPW. Due to space limitation, we only show results of PBM, CASCADE, and DCM in Table with the full table shown in Appendix B
For the position-based based model, IPW demonstrates the
8

---

Table 2: Performance of CUOLR algorithm with different state embeddings. The experiments
conducted on Yahoo! LETOR set and AISLR-WEBIOK 10K, with PBM and CASCADE as click models.
Yahoo! LETOR
MSSR-WEBIIN
CLICK MODEL
STATE EMBED
ERR@K
NDCG@K
@RR@K
NDCG@K
K=5
K=10
K=5
K=10
K=5
K=10
K=5
K=10
POS
0,439
0.455
0.691
0.742
0.282
0.304
0.382
0.407
PREDOC
0.435
0.451
0,682
0.734
0.274
0.295
0.374
0.401
PBM
POS + PREDOC
0.440
0.456
0.685
0.734
0.277
0.298
0.375
0.400
ATTENTION
0.459
0.478
0.700
0.753
0.281
0.303
0.380
0.406
POS
0.426
0.442
0.668
0.724
0.260
0.283
0.360
0.391
PREDOC
0.456
0.472
0.703
0.754
0.272
0.293
0.373
0.400
CASCADE
POS+PREDOC
0,453
0.470
0,686
0.740
0.275
0.296
0,373
0.396
ATTENTION
0.461
0.478
0.696
0.748
0.280
0.301
0.378
0.404
ORACLE
0.462
0.486
0.739
0.781
0.328
0.347
0.432
0.453
best performance among the baselines, which is expected is tailored for position- methods. Similarly, CM-IPW yielded the best performance for the cascade-based methods, which aligns with its incorporation of previous document information in the propensity estimation. Remarkably, across all click models, our method, whether combined with SAC or CQL, consistently achieves the best performance in most cases. This validates the effectiveness of our unified framework and the robustness of our CUOLR algorithm. Furthermore, it is noteworthy that our method demonstrated consistent performance across different RL algorithms, verifying its resilience and adaptability to various underlying RI. solvers.
How effective is the state representation learning component in CUOLR? In this experiment, we examine different approaches for state representation learning and study how it affects the overall performance of our proposed method. We compare with the following state embeddings: (1). position- only embedding (POS), which only utilizes the position information using positional encoding; (2). wreviou-doccument-bee
embedding (PREDOC), which takes a simple average of all the document features in R[ k): (3). the concatenation of the position and the average document features up to position (POS- PREDOC), as well as the proposed learnable state representations based on multi- head self-attention (ATTENTION). ORACLE here is used to show the gap from the upper bound. The results of our experiments are presented in Table 2 (with a full table including other click models is shown in Appendix B.. For the PBM click model. it is evident that state embeddings utilizing position-based information, such as POS and POSDPREDOC, outperform other state embeddings. In contrast, for the CASCADE click model, state embeddings utilizing previous document features exhibit significantly stronger performance compared to those utilizing position information. Notably, method, CUOLR, which dynamically learns the state embeddings during policy optimization, consistently achieves comparable performance compared to using hard-coded fixed state embeddings. This highlights the necessity of leveraging state representation in off-policy LTR and underscores the effectiveness of our proposed approach.
Conclusion
In this paper, we present an off-policy learning-to-rees -aaning-to-rank formulation from the perspective of reinforce- ment learning. Our findings demonstrate that under this novel MDP formulation, RL algorithms can effectively address position bias and learn the optimal ranker for various click models, without the need for complex debiasing methods employed in unbiased learning to rank literature. This work establishes direct connection between reinforcement learning and unbiased learning rank through a concise MDP model. Specifically, we propose novel off off-policy learning-te tornkkk algorithm, CUOLR, which simultaneously learns efficient state representations and the optimal policy. Through empirical evaluation, we show that CUOLR achieves robust performance across wide range of click models, consistently surpassing existing off-policy learning-to-rant methods tailored to those specific models. These compelling observations indicate that the extensive research conducted on offline

---

reinforcement learning can be leveraged for learning to rank with biased user feedback, opening up promising new area for exploration.
Acknowledgments and Disclosure F Funding
This work was supported in part by Google Cloud Research Credits Program. Mengdi Wang acknowledges the support by NSF grants DMS-1953686, IIS- IIS-2107304, CMMI-1653435, ONR grant 1006977, and C3.AI.
References
[1] Alekh Agarwal, Nan Jiang, Sham M Kakade, and Wen Sun. Reinforcement learning: Theory and algorithms. CS Dept., UW Seattle, Seattle, WA, USA, Tech. Rep, pages 10-4, 2019.
[2] Aman Agarwal, Kenta Takatsu, Ivan Zaitsev, and Thorsten Joachims. general framework for counterfactual learning-to-rrn tornng-tt-rank... In Proceedings of the 42nd International ACM SIGIR Conference Research and Development in Information Retrieval, pages 5-14, 2019.
[3] Aman Agarwal, Xuanhui Wang, Cheng Li, Michael Bendersky, and Mare Najork. Addressing trust bias for unbiased learning-to-rank, In The World Wide Web Conference, pages 4-14, 2019.
[4] Qingyao Ai, Keping Bi, Cheng Luo, Jiafeng Guo, and W Bruce Croft. Unbiased learning to rank with unbiased propensity estimation. In The 4/st International ACM SIGIR Conference on Research & Development in Information Retrieval, pages 385-394, 2018.
[5] Jacob Buckman, Carles Gelada, and Mare G Bellemare. The importance pessimism in fixed-datase policy optimization. arXiv preprint arXiv:2009.06799, 2020.
[6] Chris J.C. Burges. From ranknet lambdarank to lambdamart: An overview. Technical Re- port MASR-TR-2010-20 June 2010. httpttts:://////m ://::////wwwww/ www.omoft.......mmmmmmmmm.w....t publication/from: frrmeooooooooooooo rww.eet-nnnn--aaeeeeeeeeeeeee 1an--lmmmddranann to 1aroo-llmmmmmaaarra hom-rddrrrr-n-------ammmmar----- anwhaaarttt-----eeeeeee ovnrveve
[7] Olivier Chapelle and Yi Chang. Yahoo! learning to rank challenge overview. In Proceedings of the learning to rank challenge, pages 1-24. PMLR, 2011.
[8] Olivier Chapelle, Donald Metlzer, Ya Zhang, and Pierre Grinspan. Expected reciprocal rank for graded relevance. In Proceedings of the 18th ACM conference Information and knowledge management, pages 621-630, 2009.
[9] Jiawei Chen, Hande Dong, Xiang Wang, Fuli Feng, Meng Wang, and Xiangnan He. Bias and de- bias in recommender system: A survey and future directions. arXiv preprint arXiv:2010.03240. 2020.
[10] Aleksandr Chuklin, Ilya Markov, and Maarten de Rijke. Click Models for Web Search. Morgan & Claypool, 2015. ISBN 9781627056489, doi: 10.2 10.2200/50054EEDI V01 1011
[11] Matej Cief, Branislav Kveton, and Michal Kompan. Pessimistic off-policy optimization for learning to rank. arXiv preprint arXiv:2206.0253 2022.
[12] Nick Craswell, Onno Zoeter, Michael Taylor, and Bill Ramsey. An experimental comparison of click position-bial bias models. In Proceedings of the 2008 international conference web search and data mining, pages 87-94, 2008
[13] Georges E Dupret and Benjamin Piwowarski. A user browsing model to predict search engine click data from past observations. In Proceedings of the 31 annual international ACM SIGIR conference on Research and development in information retrieval, pages 331-338, 2008
[14] Damien Ernst, Pierre Geurts, and Louis Wehenkel. Tree-based batch mode reinforcement learning. Journal of Machine Learning Research, 6, 2005.
[15] Dylan J Foster, Akshay Krishnamurthy, David Sinchi-Levi, and Yunzong Xu. Offline rein- forcement learning: fundamental barriers for value function approximation. arXiv preprint axX1v:211..10919, 2021.
10

---

[16] Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International conference on machine learning, pages 2052-2062. PMLR, 2019.
[17] Fan Guo, Chao Liu, Anitha Kannan, Tom Minka, Michael Taylor, Yi-Min Wang, and Christos Faloutsos. Click chain model in web search. In Proceedings of the 18th international conference on World wide web, pages 11 1-20, 2009.
[18] Fan Guo, Chao Liu, and Yi Min Wang. Efficient multiple-click models in web search. In Proceedings of the second acm international conference on web search and data mining, pages 124 131, 2009.
[19] Tuomas Haarnoja, Aurick Zhou, Pieter Abeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, pages 1861-1870. PMLR, 2018.
[20] Kalervo Järvelin and Jaana Kekäläinen. Cumulated gain- evaluation of ir techniques. ACM Transactions on Information Systems (TOIS), 20(4):422-446, 2002.
[21] Nan Jiang and Lihong Li. Doubly robust off-policy value evaluation for reinforcement learning, In International Conference on Machine Learning, pages 652-661 PMLR, 2016.
[22] Ying Jin, Zhuoran Yang, and Zhaoran Wang. Is pessimism provably efficient for offline RL? In International Conference on Machine Learning, pages 5084-5096. PMLR, 2021.
[23] Thorsten Joachims. Optimizing search engines using clickthrough data. In Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining, pages 133-142, 2002.
[24] Thorsten Joachims, Laura Granka, Bing Pan, Helene Hembrooke, and Geri Gay. Accu- rately interpreting clickthrough data as implicit feedback. In Proceedings of the 28th An- nual International ACM SIGIR Conference Research and Development in Information Retrieval, SIGIR '05, page 154-161, New York, NY, USA, 2005. Association for Com- URL https
puting Machinery. ISBN 1595930345. doi: 10.1145/107604410760000
7 //doi.org/10 10. 00445//170044.... 10776600331071663
[25] Thorsten Joachims, Laura Granka, Bing Pan, Helene Hembrooke, and Geri Gay. Accurately interpreting clickthrough data as implicit feedback. In Acm Sigir Forum, volume 51, pages 4-11, Acm New York, NY, USA, 2017.
[26] Thorsten Joachims, Adith Swaminathan, and Tobias Schnabel. Unbiased learning-to--ee -tarin with biased feedback. In Proceedings of the tenth ACM international conference on web search and data mining, pages 2017..
[27] Jae-kwang Kim and David Haziza. Doubly robust inference with missing data in survey sampling. Statistica Sinica, 24, 01 2014. doi: 10.5705/ss.2012.005
[28] Haruka Kiyohara, Yuta Saito, Tatsuya Matsuhiro, Yusuke Narita, Nobuyuki Shimizu, and Yasuo Yamamoto. Doubly robust off poolcy evaluation for ranking policies under the cascade behavior model. In Proceedings of the Fifteenth ACM International Conference on Web Search and Data Mining, pages 487-497, 2022.
[29] Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 33: 1179-1191, 2020.
[30] Tor Lattimore, Branislav Kveton, Shuai Li, and Csaba Szepesvari. Toprank: A practical algorithm for online stochastic ranking. In NeurIPS, 2018.
[31] Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.0166 2020.
[32] Gene Li, Cong Ma, and Nathan Srebro. Pessimism offline linear contextual bandits using bp confidence sets. arXiv preprint arXivv2205.10677 2022.

---

[33] Tie- Yan Liu et al. Learning to rank for information retrieval. Foundations and Trends@ in Information Retrieval, 3(3):225-331, 2009.
[34] Volodymyr Mnih, Koray Kavukeuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprini 1rXiv:13/2.5602 2013.
[35] Volodymyr Mnih, Koray Kavukeuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland. Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 518(7540):5295333 2015.
[36] Maeve O' O'Brien and Mark Keane. Modeling result-list searching the world wide web: The role of relevance topologies and trust bias. 01 2006.
[37] Harrie Oosterhuis. Doubly robust estimation for correcting position bias in click feedback for unbiased learning to rank. ACM Transactions on Information Systems, 41(3):1-33, 2023.
[38] Tao Qin and Tie- Yan Liu. Introducing LETOR 4.0 datasets. CoRR, /abs//306.259,, 2013. URL http://arxiv. org/ aa//abb//31000 2597
[39] Paria Rashidinejad, Banghua Zhu, Cong Ma. Jiantao Jiao, and Stuart Russell. Bridging offline reinforcement learning and imitation learning: tale of pessimism. Advances in Neural Information Processing Systems, 34:1 1702-11716, 2021.
[40] Yuta Saito. Doubly robust estimator for ranking metrics with post-click conversions. In Proceedings of the 14th ACM Conference on Recommender Systems, pages 92-100, 2020.
[41] Mark Sanderson et al. Test collection based evaluation of information retrieval systems. Foundations and Trends@ in Information Retrieval, 4(4):247-375, 2010.
[42] John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learming, pages 1889-1897. PMLR, 2015.
[43] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal 2017.
policy optimization algorithms. arXiv preprint
[44] Laixi Shi, Gen Li, Yuting Wei, Yuxin Chen, and Yuejie Chi. Pessimistic q-learning for offline reinforcement learning: Towards optimal sample complexity. In International Conference on Machine Learning, pages 19967-20025. PMLR, 2022.
[45] Richard Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
[46] Adith Swaminathan and Thorsten Joachims. Counterfactual risk minimization: Learning from logged bandit feedback. In International Conference on Machine Learning, pages 814 123. PMLR, 2015.
[47] Anh Tran, Tao Yang, and Qingyao Ai. Ultra: An unbiased learning to rank algorithm toolbox. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management, pages 4613-4622, 2021.
[48] Ali Vardasbi, Maarten de Rijke, and Ilya Markov. Cascade model-based propensity estimation for counterfactual learning to rank. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 2089-2092, 2020.
[49] Ali Vardasbi, Harrie Oosterhuis, and Maarten de Rijke. When inverse propensity scoring does not work: Affine corrections for unbiased learning to rank. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management, pages 1475-1484, 2020.
[50] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Fukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances neural information processing systems, 30, 2017.
12

---

[51] Nan Wang, Zhen Qin, Xuanhui Wang, and Hongning Wang. Non-clicks mean irrelevant? propen- sity ratio scoring as a correction. In Proceedings of the 14th ACM International Conference on Web Search and Data Mining, pages 481-489, 2021.
[52] Xuanhui Wang, Michael Bendersky, Donald Metzler, and Marc Najork. Learning to rank with selection bias in personal search. In Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval, pages 115-124, 2016.
[53] Xuanhui Wang, Nadav Golbandi, Michael Bendersky, Donald Metzler, and Marc Najork. Position bias estimation for unbiased learning to rank in personal search. In Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining, pages 610-618, 2018.
[54] Zeng Wei, Jun Xu, Yanyan Lan, Jiafeng Guo, and Xueqi Cheng. Reinforcement learning to rank with markov decision process. In Proceedings of the 40th international ACM SIGIR conference on research and development in information retrieval, pages 945-9455 2017.
[55] Tengyang Xie, Ching-An Cheng, Nan Jiang, Paul Mineiro, and Alekh Agarwal. Bellman- consistent pessimism for offline reinforcement learning. Advances in Neural Information Processing Systems, 34:6683-6694, 2021.
[56] Tengyang Xie, Nan Jiang, Huan Wang, Caiming Xiong, and Yu Bai. Policy finetuning: Bridging sample-efficient offline and online reinforcement learning. Advances in Neural Information Processing Systems, 34:27395-27444 2021.
[57] Jun Xu. Zeng Wei, Long Xia, Yanyan Lan, Dawei Yin, Xueqi Cheng, and Ji-Rong Wen. Reinforcement learning to rank with pairwise policy gradient. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, pages (18,-51,,20 2020.
[58] Tengyu Xu and Yingbin Liang. Provably efficient offline reinforcement learning with trajectory- wise reward. arXiv preprint arXiv:2206. 064266, 2022.
[59] Hang Yan, Bocao Deng, Xiaonan Li, and Xipeng Qiu. Tener: adapting transformer encoder for named entity recognition. arXiv preprint 191..0444, 2019.
[60] Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. Advances in Neural Information Processing Systems, 33: 13:14422-4414 2020.
[61] Andrea Zanette, Martin Wainwright, and Emma Brunskill. Provable benefits of actor-critic methods for offline reinforcement learning. Advances in Neural Information Processing Systems, 34:13626 13640, 2021.
[62] Shengyao Zhuang, Zhihao Qiao, and Guido Zuccon. Reinforcement online learning to rank with unbiased reward shaping. arXiv preprint arXiv:2201.015344 2022.
[63] Masrour Zoghi, Tomas Tunys, Mohammad Ghavamzadeh, Branislav Kveton, Csaba Szepesvari, and Zheng Wen. Online learning to rank in stochastic click models. In International Conference on Machine Learning, pages 4199-4208, 2017.
13

---

A Experiment Details
A.1 Dataset Statisties
We conducted experiments on MSLR-WER MELLR-WEB1RR 10K and Yahoo! LETOR (set 1) with semi-synthetic generated click data. Yahoo! LETOR comes from the Learn to Rank Challenge. It consists of 29,921 queries and 710K documents. Each query-document pair is represented by 700-dimensional feature and annotated with 5-level relevance label ranging from 0 to 4. MSLR-WEBI 10K dataset contains 10,000 queries and 125 retrieved documents average. Each query-document pair is represented by a .36-dimmnsional feature vector and a 5-level relevance label. The dataset is partitioned into five parts with about the same number of queries, denoted as S1, S2, S3, S4, and S5, for five-fold cross crossvvliiattion All statistics of the used datasets are summarized in Table
Table 3: Statistics of Learning to Rank Datasets
of queries
# of features
# of query-document pairs
rel level
# of folders
MSLR-WER 10K
10000
136
1250k
5
Yahoo! LETOR
29921
700
710k
5
A.2 Implementation Details
Before introducing the hyperparameters required for each algorithm, we first describe some global hyperparameters that are used commonly across all algorithms. We use batch size B 256 queries per epoch, and use nDCG@ 10 as the training objective for all baselines. We use Adam optimizer to train all the networks.
Dual Learning Algorithm (DLA) 4]. For DLA, two sub-models are being implemented: the ranking (scoring) model which is used score each document; and the propensity model which is used to estimate the propensity for each document in the rank list. We use the MLP with two hidden layers of 256 units for both of them, and other hyperparameters are shown in Table 5
Inverse Propensity Weighting (IPW) 52 26 & Cascade Model-based IPW (CM-IPW) 48
For IPW and CM-IPW, we need get the propensities from Result Randomization. In total, there are 10M random rank lists with different searching queries shown to the user (click model), and the parameters of each click model are estimated by Maximize Likelihood Estimation (MLE). Estimation details are shown in Table 4 where Cş and Ce indicate the click at rank l and before rank k respectively. The superscript (s) denotes the click of some click session 8. Besides, we implement the ranking model the same way that for DLA. Other hyperparameters are shown in Table 5
For ORACLE, we train a LambdaMART ranker Burges (6) with true labels on the training dataset and evaluate its performance the test set. We leverage the RankLib learning to rank library, and set the hyperparameters shown in Table This utilizes the full-information data and serves upper bound of the performance for all algorithms utilizing the partial-information data, such as the generated clicks.
ORACLE.
For our algorithm CUOLR, there are three sets of hyperparameters used by the following sub-models: parameters for state embedding model, RL policy, as well as RL critic, where we use 256-256 MLP to implement all the networks. Detailed hyperparameters are shown in Table [
CUOLR.
B Additional Results
In this section, present the complete results for all five click models (PBM, CASCADE, UBM, DCM, and CCM) on the two datasets: Yahoo! LETOR set and MSLR-WEB 10K. For the first two
https: http://iiiettttt..aa///////////e
https:// . sandbox. yahoo, com/
https://oourcetorge met/p/lemuee BankLib/kkLLLLa
14

---

Table 4: Details for IPW and CM-IPW.
PROPENSITY
PARAM. ESTIMATION
E.es c(2)
IPW
0.
E.es C(
P(E = 1 11 P(E - 1| ll (1 Ci(1-11)))
Eoes, I[C)
S& = { s= Ck:) =1}
CM-IPW
A
Table 5: Hyperparameters for each algorithm used in the experiment.
ALG
HYPERPARAMETERS
policy learning rate: 1c-4 propensity learning rate: le-4
DLA
loss type: softmax
policy learning rate: le-4 loss type: softmax
IPW & CM-IPW
number trees: 1000
number of leaves for each tree: 100
ORACLE
shrinkage (learning rate): 0.01
min leaf support: 50
early stop: 100
actor Icarning rate: lc-4
critic learning rate; 1c-4
actor a: le-10 (fixed)
CUOLR
soft update т: 5e-3
discount 7. 0.8
embed Iearning rate: 1c-6
embed type: multi-head attention number of heads: 8
CQL a: 1e-1 (fixed)
studies in Section B.1 and B.2, we run 5 runs with different random seeds for the Yahoo! dataset. For the MSLR- BE-WEB1OG 10K dataset which naturally comes with 5 folds, we take run for each fold and aggregate the results. In the ablation experiment of conservatism for the offline RL algorithm in Section B.3 we only run runs Yahoo! due to the time limit. In all of our experiments, we use nDCG and ERR at positions 3,5,10 as evaluation metrics.
B.1 Performance across Different Click Models
In this section, we present a comprehensive comparison between our proposed method, CUOLR, and various baseline approaches. Specifically,, we examine the efficacy of DLA and IPW, which have been specifically designed for position-based models, as well as CM-IPS, which has been tailored for cascade-based models. The comparative results are presented Table In the case of position- based models such as PBM and UBM, it is evident that IPW demonstrates the most superior performance among all the considered baselines. Conversely, when evaluating cascade models such as cascade and DCM, the utilization of CM-IPW yields improved performance, as it takes into account the propensity estimation considering prior examinations and clicks. Among the diverse click models examined, unified algorithm. CUOLR, consistently achieves the highest level of performance in terms of the ERR metrics across different positions. Furthermore, it consistently outperforms the other models in the majority of cases in terms of nDCG @ 10. This provides empirical verification of the effectiveness of our unified framework and the robustness of the CUOLR algorithm.
B.2 State Representation Ablation Experiments
In this section, we present an ablation study focusing on the state embedding utilized in our algorithm, CUOLR. We compare the effectiveness of our proposed multi-head self-attention, augmented with positional embedding, against several heuristic hard-coded basclines for state embedding. These
15

---

Table 6: Performance comparison with different click models Yahoo! LETOR setl. "e" and "**" indicate statistically significant improvement (p-value < 0.05 and p-value < 0.01 respectively) over the best baseline for each metric respectively.
CLICK MODEL
ALG
ERR@ @
ERR@S
ERR@ @ 10
nDCG@3
nDCG@s
nDCG@ 10
DLA
0.418
0.439
0.455
0,669
0.691
0.742
CM-IPW
0.418
0,440
0,456
0.669
0.692
0.743
PBM
IPW
0,424
0,446
0,462
0.678
0.700
0.748
CUOLR(CQL)
0.437
0.458
0.473
0.678
0.700
0.753*
CUOLR(SAC)
0.437**
0.459
0.478**
0.674
0.700
0.753
DLA
0.420
0.441
0.457
0.668
0.690
0.741
CM-IPW
0.422
0.444
0.460
0.674
0.696
0.745
CASCADE
IPW
0.420
0.442
0.457
0.668
0.690
0.740
CUOLR(CQL)
0.436
0.459
0.479
0.668
0.696
0.748
CUOLR(SAC)
0.440*
0.461
0.478**
0.670
0.696
0.748
DLA
0.426
0,448
0.463
0.687
0.708
0.756
CM-IPW
0.415
0.437
0.453
0.664
0.689
0.740
UBM
IPW
0.427
0.449
0.464
0.686
0.708
0.756
CULL
0.435
0.457
0.473
0.679
0.704
0.756
CUOLR(SAC)
0.435
0.458
0.473*
0.675
0.700
0.754
DLA
0.422
0.444
0.459
0.673
0.696
0.745
CM-IPW
0.426
0.447
0.463
0.683
0.704
0.752
DCM
IPW
0.422
0.444
0.459
0.674
0.697
0.746
CUOLR(CQL)
0.437
0.461
0.474*
0,676
0.699
0.753
CUOLR(SAC)
0.436**
0.461
0.475**
0.675
0.703
0.755
DLA
0.427
0.449
0.464
0.685
0.707
0.754
CM-IPW
0.416
0.438
0.453
0.659
0.684
0.736
CCM
IPW
0.429
0,450
0.465
0.689
0.710
0.757
CUOLR(COL) 0.438
0.462
0.478
0.681
0.704
0.758
CUOLR(SAC)
0.437**
0.460
0.476**
0,680
0.706
0.759
LOGGING 0.359
0.385
0.403
0.595
0.630
0.693
ORACLE
0.440
0.462
0.486
0.720
0.739
0.781
baselines include utilizing only positional information (POS), concatenating previous document in- formation (PREDOC), and a combination of positional and document information (POS+PREDOC), The evaluation performed the Yahoo dataset, and the results are summarized in Table Con- sistent with expectations, for position-based models (PBM), the most effective approach is utilizing only the positional information. Conversely, for cascade models (CASCADE), only considering the previous document information gives the best performance. In the case of more complicated models, such as DCM. CCM. and UBM, where the click model relies on both position and previous examinations, it becomes evident that incorporating combination of positional information and previous document information yields the highest performance. Among all of them, worth to point out that our proposed state representation learning consistently attains comparable performance the best baseline. Notably, our method possesses the advantage of automatically learning the optimal state representation, irrespective of the underlying assumptions of the click models.
B.3 Conservatism for Offline RL Ablation Experiments
In this section, we conduct an ablation study to investigate the influence the hyperparameter a. which governs the conservatism, in the CQL algorithm. Specifically, we examine its effects on the Yahoo! dataset by varying a across range of values: {0, 1e 3, 5e 3, 1e 2, 5e 1e 1, 5e 1, 1e0, 5e0, le1, 5c1 }. It noteworthy that when a is set to O, the CQL algorithm simplifies to the SAC algorithm [19] Remarkably, we find that the performance of CUOLR remains consistently robust across the diverse a values, as long they are not being restricted to be too close to the logging policy (i.e., large a values). This consistency underscores the effectiveness of our method, which demonstrates its ability to adapt different underlying reinforcement learning (RL.) algorithms.
Interestingly, in contrast to classical offline datasets studied in Kumar al. [29], where the conservatism parameter plays substantial role, observe that its impact is comparatively minor
16

---

Table 7: Performance comparison with different click models indicate statistically significant improvement (p-value < 0.05 and p-value < 0.01 respectively) over the best baseline for each metric respectively.
"e" and "Ma"
CLICK MODEL
ALG
ERR @3
ERR@5
ERR @ 10
nDCG@3
nDCG@S
nDCG@ 10
DLA
0.230
0.256
0.278
0.343
0.356
0.384
CM-IPW
0.229
0.255
0.277
0.341
0.354
0.383
PBM
IPW
0.257
0.281
0,301
0.356
0.367
0.390
CUOLR(CQL)
0.257
0.281
0.303
0.369 **
0.380
0.406
CUOLR(SAC)
0.255
0.279
0.301
0.368
0.379**
0.404
DLA
0.239
0.265
0,286
0.354
0.365
0.392
CM-IPW
0.259
0.283
0.304
0.367
0.378
0,403
CASCADE
IPW
0.232
0.257
0.279
0.347
0.358
0.387
CUOLR(CQL)
0.255
0.280
0.301
0.366
0.378
0.404
CUOLR(SAC)
0.255
0.279
0,301
0.368
0.379
0.405
DLA
0.257
0,280
0.300
0.369
0.377
0.400
CM-IPW
0.253
0.276
0,297
0.354
0.362
0.386
UBM
IPW
0.258
0.281
0.301
0.359
0.367
0.390
CULL
0.255
0,280
0.301
0.373
0.384
0.408
CUOLR(SAC)
0.260
0.284
0.306
0.374"
0.384"
0.408
DLA
0.254
0.279
0.299
0.368
0.378
0.404
CM-IPW
0.259
0.284
0.304
0.367
0.379
0.402
DCM
IPW
0.256
0.278
0,299
0.354
0,363
0.387
CUOLR(CQL)
0.254
0.278
0.300
0.369
0.380
0.405
CUOLR(SAC)
0.254
0.278
0.300
0.367
0.378
0.403
DLA
0.252
0.275
0.295
0.347
0.356
0.379
CM-IPW
0.227
0.249
0.270
0.305
0.316
0.342
CCM
IPW
0.255
0.278
0,298
0.351
0.360
0.383
CUOLRICOL)
0.255
0.280
0.301
0.373
0.384
0.408
CUOLR(SAC)
0.260
0.284**
0.305
0.374*
0.383
0.408
LOGGING 0.180
0.206
0.230
0.288
0,304
0.338
ORACLE 0.305
0.328
0.347
0.426
0.432
0.453
in the offline learning to rank dataset. This observation is worth further investigation for better understanding the impact of conservatism in offline LTR settings.
Besides, To figure the effect of conservatism, do experiments several click models Yahoo! dataset to compare the performance between simple SAC and CQL with the optimal conservative parameter a. The optimal
selected through grid search ranging from {le 5e 3, le 2, 5e 2, le -1 5e - 1, 1e0, 5e0, 1el, 5e1}. The results are shown in Table It's obvious that the performance of CQL with optimal a is consistently better than that of CQL, especially in the NDCG metric. This improvement further illustrates the necessity of conservatism in Off-policy learning to rank task.
9
B.4 Data Quality for Offline Reinforcement Learning Rank
In this section, we conduct an ablation study to clarify the effect of data quality to the offline RL algorithms. In usual cases, data quality important to most offline algorithms, including the offline RL algorithm. To verify the effect of data quality, we conduct experiments on the WeblOk dataset with different offline data. Specifically, on each click model, the logging policy is trained with different portions of train data: 1) SVMRank trained with 1% train data; 2) SVMRank trained with 0.01% train data; 3) random policy without pre- gr--trannin. Then the quality of the train data gathered by the three logging policy decreases, and the performance of CQL on them is displayed in Table IT] We find that the algorithm trained with data better quality (first line for each click model) performs significantly better than the other two. Besides, it's interesting that the algorithm trained with random logging data can't even beat the logging policy trained with % train data, which further highlight the significance of data quality.
17

---

CASCADE
PBM
BAR
MDP
DCM
eji examination
S/i state
de document
ay: action
Cj click
F reward
ky rank/position
CCM
Figure 1: Graphical models of different click models and our MDP formulation about the learning rank problem.
C Click Model and MDP Formulation
In this section, present comprehensive overview of different click models employed in the paper, namely PBM, CASCADE, DCM, CCM, and UBM. Additionally, we demonstrate the graphical models associated with each click model and how they could be unified into the Markov Decision Process (MDP) framework, as depicted in Figure T
PBM [24]. The position-based model is model where the probability of clicking on an item depends on both its identity and rank. The examination probability is rank-dependent only, i.e.,
X(R, k) x(k).
The cascade model assumes that the user scans rank list R. from top to bottom. If a document rank & examined and clicked, the user stops browsing the remaining documents. Otherwise, the user goes on to the next rank & + 1 with probability one. The first document d₃ always examined. The document at rank & will be examined and only if the previous k documents are not clicked. Therefore we have:
CASCADE (2).
XRR, k) - (1- (1-a((())
DCM 48 18). The dependent click model assumes that the user examines the results from top to bottom until an attractive result is found, P(Ex+1 = 1 E = 1, Ct = 0) 1, where Eş is the examination indicator at rank k. After each click, there is mank-dependent chance that the user is unsatisfied, P(Ex+1 = C 1) = At- Therefore, we have:
(RR, k) 115- (1 aR(()) (1-A,))
The click chain model (CCM) is generalization of the dependent click model where continuing to examine the results before click is not deterministic, i.c. P(Ef+1 1 E,
CCM 17
18

---

Table 8: Performance of CQL algorithm with different state embedding. The experiments
conducted on Yahoo! set 1 and click models are PBM, CASCADE, UBM, DCM, and CCM
CLICK MODEL
STATE EMBED
ERR@3
ERR@5
ERR@10
nDCG@3
nDCG@S nDCG@
POS
0.417
0.439
0,455
0.668
0.691
0.742
PREDOC
0,413
0,435
0.451
0.657
0.682
0.734
PBM
POS+PREDOC
0,418
0,460
0,456
0,663
0.685
0.734
ATTENTION
0.437
0.459
0.478
0.674
0.700
0.753
POS
0,404
0,426
0.442
0,646
0.668
0.724
PREDOC
0.434
0.456
0.472
0.679
0.703
0.754
CASCADE
POS+PREDOC
0.432
0.453
0,470
0.664
0.686
0,740
ATTENTION
0.440
0.461
0.478
0.669
0.696
0.748
POS
0.420
0.442
0.458
0.678
0.700
0.747
PREDOC
0,410
0,433
0.449
0.666
0.691
0.743
UBM
POS+PREDOC
0.416
0,439
0.454
0,676
0.700
0.750
ATTENTION
0.435
0.457
0.473
0.679
0.704
0.756
POS
0.415
0,437
0.453
0.658
0.680
0.731
PREDOC
0.435
0.457
0.473
0.682
0.706
0.758
DCM
POS+PREDOC
0.439
0.461
0.477
0.687
0.709
0.758
ATTENTION
0.437
0.461
0.475
0.675
0.703
0.755
POS
0.421
0,443
0.458
0.678
0.701
0.748
PREDOC
0,420
0,443
0.459
0,677
0.702
0.754
CCM
POS+PREDOC
0.426
0.448
0.464
0.685
0.710
0.759
ATTENTION
0.437
0.460
0.476
0.680
0.706
0.759
ORACLE 0.440
0.462
0.486
0.720
0.739
0.781
Table 9: Comparison of SAC and CQL with optimal alpha on Yahoo. To show the impact of the offline algorithm in off-policy learning to rank, we compare SAC and CQL with different degrees of conservatism. We control the degree of conservatism via the parameter (See details in Eq.5 in the main paper). The as range from (le 5e le 2, 5e 2, le 5e 1, le0, 5e0, le1, 5e1} and we do grid search to find the optimal a.
ERR@K K=S
NDCG@K
CLICK MODEL
ALG
K=3
K=10
K=3
K=S
K=10
SAC
0,437
0.459
0.478
0.674
0.700
0.753
PBM
CQL(optimal a:)
0.437
0.460
0.478
0.692
0.715
0.766
SAC
0.440
0.461
0.478
0.670
0.696
0.748
CASCADE
CQL(optimal
0.440
0.461
0.479
0.691
0.715
0.765
SAC
0.436
0.461
0.475
0.675
0.703
0.755
DCM
CQI((optimal
0.439
0.461
0.477
0.694
0.718
0.767
SAC
0.437
0.460
0.476
0.680
0.706
0.759
CCM
CQL(optimal (x)
0.438
0.462
0.479
0.691
0.712
0.766
1. a1. The probability of continuing after click is not position dependent, but relevance dependent, P(E C2 = 1) a2(1 R&) + agli, where RG is the relevance of the ith document in the rank list. Therefore, the examination probability at each position can be written as:
- 12 (1))((())) az -ag)))
The user browsing model (UBM) an extension of the PBM model with some elements of the cascade model. The whole model is position- position-based, but for the examination probability, it considers previous clicks. Specifically, the examination probability depends not only on rank of the document k. but also on the rank of the previously clicked document K,, which is modeled by set of parameters TAk, i.e. P(Fx the previous clicked document or 0 if none of them was clicked, i.c. k' max(r {O...., 1})
UBM 13]
Ci = CI...., Ct-1 = Ck-1 Tkk', where is the rank of
19

---

Table 10: Comparison of different logging policies Web10k. On each click model, compare three different logging policies: SVMRank trained with 1% train data, SVMRank trained with 0.01% train data, and random policy.
ERR@K
NDCG@K
CLICK MODEL
LOGGING
K=3
K=S
K=10
K=3
K=S
K=10
por=1 1%
0.257
0.281
0.303
0.369
0.380
0.406
PBM
por=0.01
0.214
0.239
0,262
0.324
0.339
0.367
rand
0.111
0.130
0.151
0.159
0.175
0.201
por=
0.255
0.280
0.301
0.366
0.378
0.404
CASCADE
por=0.01
0.214
0.239
0.262
0.326
0,340
0.368
rand
0.117
0.136
0.157
0.163
0.180
0.208
por= 1 %
0.254
0.278
0.300
0.369
0,380
0.405
DCM
por=0.01 %
0.212
0.238
0.260
0.324
0.338
0.367
rand
0.121
0.139
0.159
0.167
0.178
0.204
por=1%
0.255
0.280
0.301
0.373
0.384
0.408
CCM
por=0.01%
0.214
0.239
0.261
0.324
0.339
0.367
rand
0.123
0.142
0.162
0.163
0.179
0.208
=1
X k:) = Ykk
Zoghi et al. 663 showed that PBM and CM satisfied Assumption 3.1 where optimal ranking list leads to optimal total clicks. In our future work, it would be interesting to show that other click models also satisfy this assumption.
20

---

pbm
cascade
ccm
34.00
3434
0430
0428
3.24
0.22
3450
0.50
0.40
0476
3/12
3470
3.00
0.466
0462
0.600
0:89
0:670
0.7155
0.7100
0.7075
0.7025
0.7000
0.768
0.762
0.700
0.76
0.50
30'4
10'2
30"1
109
207
10a
10"1
167
107
20
10th
102
201
10"
2010
10"2
10'
10'
err_3
err 5
err 10
ndcg_3
ndcg_5
ndcg_10
Figure 2: NDCG@ 10 and ERR@ 10 CUOLR (with CQL) on Yahoo! dataset set 1, with different conservative parameters as. Each row is the performance of metric (ERR, NDCG@ @,5,11) on the four click models (PBM, CASCADE, DCM, CCM).
21


Reviews:
[{"review_id": "9hyPULy9sV", "reviewer": "Reviewer_XmTH", "summary": "In this work, the authors model the process of learning optimal ordering of ranked lists, called Learning to Rank (LTR) as a Markov decision process (MDP). This allows them to utilize techniques from reinforcement learning to solve for the policy that generates the optimal ordering. In practical applications of LTR, a pre-collected dataset (offline) is typically used which is collected using a logging policy that is not optimal (off-policy). As a result, techniques for offline off-policy reinforcement learning are used in this work.\nIn the MDP formulation, the reward is defined as an item of the ranked list being clicked. The distribution of clicks is referred to as a click model, which results in various reward functions. An RL algorithm can optimize a wide range of reward functions and allows this approach to be largely click-model agnostic. \nThe approach is empirically validated on semi-synthetically generated click datasets, and it demonstrates competitive performance across different click models compared to click-model-specific baselines for learning optimal rankings.\n", "questions": "A few questions:\n- Assumption 3.1 holds only for monotonically decreasing examination probabilities. The reference [64] proves the result in Assumption 3.1 only for PBM and cascade models, and is not the case for other click models. Why is this top-down scanning of the ranked list necessary for the working of the algorithm?\n- The robustness to change in $\\alpha$ in CQL seems to indicate that just fitting a Q function on the offline data should suffice. This would be true if the logging policy provides enough support for all actions, which is commonly the case in practice where uniform random logging policies are used. Why is a conservative algorithm (CQL) necessary for the experiments? \n- (L148-L149) How is the action set restricted to not repeat actions implementationally? \n- The gains in performance are more significant for ERR as compared to NDCG. Why is that the case?\n", "limitations": "Like any other method for LTR, the performance of the algorithm should depend on the coverage provided by the logging policy. A comment on the effect of the coverage of the logging policy and it's effects on the learnt rankings would be useful.\n\nState representations play an important role in the performance of the algorithm and in the context of this work the representations are learned via architectural choices in the Q-function estimator. The claim that (L248) that an additional component is added on top of CQL is overstated. \n", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The paper succinctly models LTR as a sequential decision-making problem, building on top of prior work,  in a manner that allows the use of transformer architecture to be employed in conjunction with an RL algorithm. This allows for a couple of benefits: 1) being agnostic of specific click models since any click distribution in the offline dataset can in theory be encoded as the reward that the RL algorithm can optimize for, and 2) allowing for flexibility in defining the state representation through the use of attention and position embeddings. \nBoth of the claims are supported by strong empirical analysis.\n", "weaknesses": "The paper extended prior formulations of LTR as an MDP to allow for using the transformers architecture, with the aim performing policy learning without click model specific methods. In addition to the existing experiments, one experiment that would highlight the point would be one that learns the optimal ranking for data generated from a non-standard click model – say a randomly picked click distribution – where all other methods fail.\n\nAdditionally, the utility of the section on the optimality of rankings (Definition 3; Assumption 3.1) is unclear. These are possibly included to tie this work to existing literature, however, this method should not require those conditions for finding an optimal policy. \nOn a related note, L122-L123 incorrectly states that Definition 1 is explained in Appendix C, which only discusses the definitions of examination probabilities of various click models. \n\nThe writing and presentation could be improved in places, a few examples being highlighted below. \nMinor/Typos:\n- “Debiasing” as caused by bias in the data, versus bias due to estimation procedure has been used interchangeably through the paper leading to a fair amount of confusion (Ex: L26-L29)\n- Equation (1) should have a $\\propto$ and not $=$\n- L121: mutually independent of what?\n- L227: $d\\\\_model$ undefined.\n- Algorithm 1: Mistake on Line 7\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for appreciating the advantages of our click model-agnostic solution.\n\n**Q:** The paper extended prior formulations of LTR as an MDP to allow for using the transformers architecture, with the aim of performing policy learning without click model specific methods. In addition to the existing experiments, one experiment that would highlight the point would be one that learns the optimal ranking for data generated from a non-standard click model – say a randomly picked click distribution – where all other methods fail.\n\n**A:** Thank you for the suggestion of testing on non-standard click models such as randomly picked click distribution. According to Definition 1, the click model at any position should only be determined by the rank of the current position and previous documents, where a randomly picked click distribution may not satisfy this assumption. We do want to point out this assumption is not restrictive as the classic PBM and CASCADE model all satisfy this, and we have already evaluated on several more complicated click models including UBM, DCM, and CCM and reported the results in the main paper and Appendix.\n\n**Q:** Additionally, the utility of the section on the optimality of rankings (Definition 3; Assumption 3.1) is unclear. These are possibly included to tie this work to existing literature, however, this method should not require those conditions for finding an optimal policy. On a related note, L122-L123 incorrectly states that Definition 1 is explained in Appendix C, which only discusses the definitions of examination probabilities of various click models.\n\n**A:** Definition 3 is used to define the optimality of rankings. Note that the goal of LTR is to find the optimal ranking list while the objective of RL method is the optimal value function. We want to emphasize that these two optimality are not necessarily aligned, and they are only equivalent when Assumption 3.1 is satisfied. For most classical click models, such as PBM and cascade model Assumption 3.1 is satisfied and we can show the equivalence between the two. Sorry for the confusion in L122-L123. As we can see in Appendix C, we explained the specific form of various examination probabilities, while the attractivenesses $\\alpha(\\cdot)$ only depend on the documents and are the same across all the click models. By showing these forms, we are explicitly showing how these models are instances of Definition 1 and could be incorporated into our unified framework. \n\n**Q:** Assumption 3.1 holds only for monotonically decreasing examination probabilities. The reference [64] proves the result in Assumption 3.1 only for PBM and cascade models, and is not the case for other click models. Why is this top-down scanning of the ranked list necessary for the working of the algorithm?\n\n**A:** As we have explained above, the optimality assumption in Assumption 3.1 is needed, as the optimal value in RL may not give an optimal ranking list. It is unclear if click models with non-monotonically decreasing examination probabilities can be solved by our framework since optimizing rewards may not align with optimizing ranking list. We empirically showed that besides PBM and cascade model, major click models like UBM, DCM, CCM (all have monotonically decreasing examination probabilities) can be effectively solved by our offline RL framework. It would be interesting to look beyond this assumption in future.\n\n**Q:** (L148-L149) How is the action set restricted to not repeat actions implementationally?\n\n**A:** Each action is a query-document pair, which is unique in the dataset.\n\n**Q:** The gains in performance are more significant for ERR as compared to NDCG. Why is that the case?\n\n**A:** Compared to NDCG, ERR is more sensitive to the relevance of top results. This observation suggests our improvements are more beneficial to the top positions of ranking list compared to baselines.\n"}, {"review_id": "NfN2uS53Uz", "reviewer": "Reviewer_fhCJ", "summary": "- Authors propose a unified off-policy reinforcement learning-based approach for learning to rank (LTR) problem that is adaptable to all general click models.\n- Authors argue that a user's behavior in terms of clicks can be modeled by a Markov Decision Process (MDP), thereby allowing them to look at LTR from the perspective of offline reinforcement learning. \n- Authors' key contribution is a novel state representation that takes into account features of all the items/documents presented to a user. Specifically, they represent state at position $k$ by $s_k = [(d_1, d_2, \\cdots, d_{k-1}), k]$ to capture the context of user behavior prior to reaching position $k$-ranked position. The action set at position $k$ is all the available items/documents that haven't been presented to the user in prior $k-1$ positions, and the reward is simply the users' click behavior when presented with action $a_k$. \n- Authors use positional encoding and self-attention layer to extract the state representation given the sequence of items/documents at prior ranked positions, i.e., $(d_1, d_2, \\cdots, d_{k-1})$. They train an end-to-end model that jointly optimizes for state representation as well as the policy.", "questions": "- Could the authors address the questions raised in the weaknesses section of the review?", "limitations": "- Authors have not explicitly addressed limitations of their approach or any potential negative societal impact in the manuscript. However, authors' proposed approach is not a significant departure from the prevalent large scale recommender systems.", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "**Motivation**\n- Authors present a strong motivation for their approach based upon the need for a unified a click model-agnostic LTR method which generalizes well to use of any RL algorithm. \n- Authors do a good job at placing their work in context of related work in the field by comparing/contrasting their approach with other works. \n_Note_: I am not updated on the state-of-the-art RL approaches for LTR, and therefore may not be aware of some recent works. \n\n**Technical Presentation**\n- Authors do a good job at formalizing the problem and providing all the relevant technical information in Sections 3 and 4 that would be required for reproducibility.\n- Authors contribution is quite easy and intuitive to understand.\n\n**Experiments**\n- Authors evaluate their approach (using CQL and SAC) in comparison to baselines such as Dual Learning Algorithm (DLA), Inverse Propensity Weighting (IPW) algorithm and Cascade-model based IPW, as well as LambdaMart model which serves an upper bound. The metrics of authors' proposed approach are significantly favorable in comparison to baselines. _Note_: Just as above, because I am unaware of the SOTA, I am unable to comment on whether the right baselines have been used for comparison.\n- Authors provide ablation study to validate the need for each component of their proposed approach.", "weaknesses": "**Is this a Markov Decision Process?**\n\nFormally, the Markovian property of MDP refers to the fact that given states $s_i$ and actions $a_i$, the equality $\\mathrm{Pr}(s_t | s_0, a_0, s_1, a_1, \\cdots, s_{t-1}, a_{t-1}) = \\mathrm{Pr}(s_t | s_{t-1}, a_{t-1})$ holds. Intuitively, it means that the probability of landing in state $s_t$ of the Markov chain only depends on the last state-action pair $(s_{t-1}, a_{t-1})$, and not the state-action pairs preceding that. \n\nGiven the authors' approach uses state representation as $s_k = [(d_1, d_2, \\cdots, d_{k-1}), k]$, it is unclear to me how/if the Markovian property still hold?\n\n**Effect of Click Data Generation**\n\nAuthors mention that they use 1% of training data to train a Ranking SVM to generate initial ranked list of items, upon which clicks are simulated using different click models. How does the usage of more/less portion of training data affect the comparative baselines and in turn the relative performance improvement provided by the proposed approach over those baselines?\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the appreciating our motivation, presentation and experiments.\n\n**Q:** Is this a Markov Decision Process? Given the authors' approach uses state representation, it is unclear to me how/if the Markovian property Pr(st|s0,a0 …st-1, at-1) = Pr(st| st-1, at-1) still holds?\n\n**A:** We would like to clarify that according to the state definition in Eq (3), the Markovian property holds. Specifically, the state only depends on the current position and previous documents. \n\n**Q:** Effect of Click Data Generation. How does the usage of more/less portion of training data affect the comparative baselines and in turn the relative performance improvement provided by the proposed approach over those baselines?\n\n**A:** Thank you for the question. Please refer to Table 1 and our answer in the general response to all reviewers for new experimental results. Due to time limit, we have compared CUORL-CQL against different logging policies and will add more baselines to the experiment in next version."}, {"review_id": "RvB8ATTW9Y", "reviewer": "Reviewer_atvb", "summary": "The paper talks about how to model user behaviors with positional biases in an online search system. The paper proposes a unified RL framework to generalize three common types of positional bias models: Position-Based Modeling (PBM), CASCADE (each click depends on the previous click), and Dependent Click Models (DCM). By capturing all previous clicks into a state, the paper suggests that standard offline RL algorithms, such as Conservative Q-Learning (CQL) and Soft Actor Critic (SAC), can be used to estimate and optimize for the positional-aware ranking policies. Simulation experiments are included to support the claims.", "questions": "Table 1. The table compares the proposed method with baseline methods, one of which is used to simulate the ground-truth user behaviors. However, in the results, the baseline method which was used to simulate the user behaviors did not perform the best in estimating the user behaviors. Is this expected? Also, the differences between all methods seem pretty small.\n\nAlgorithm 1. Line 6, where is psi defined?\n\nEq (5). What is the difference between CQL and SAC? Why is there only one equation being presented for both algorithms? More details would be appreciated for clarity purposes.", "limitations": "No, the paper did not discuss the choice of hyperparameters or the closeness of the experimental results. They seem to be common challenges with RL algorithms.", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "Originality. The paper observes commonalities in three different types of methods and proposes a unified method to combine them. The generality of the proposal is further examined by simulating different cases from each of the method and showing similar performance using the proposed method. The observation and the empirical validation feel original to me.\n\nThe authors addressed my concerns and I have adjusted my scores accordingly.", "weaknesses": "Significance. The paper made a rather limiting assumption that all users in the same search system follow a single pattern of positional bias. This may not be true, as different users may have different positional biases. A more practical approach is to insert perturbations to the ranking positions to estimate the true positional effects using IPS. To mitigate the risks of perturbations, methods have been introduced to swap only adjacent search results. Please see this paper as an example: https://www.cs.cornell.edu/people/tj/publications/agarwal_etal_19a.pdf\n\nClarity. The description of the proposed method lacks sufficient details (see additional questions). Also the experimental results (Table 1) did not seem to contain significant differences between the algorithms being considered. This leads me to wonder if the proposed RL solution is actually easy to find, or is it fundamentally difficult to implement and the results may be sensitive to hyperparameter choices. The paper did not discuss common limitations of RL algorithms.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for recognizing the originality of our unified formulation and solution.\n\n**Q:** The paper made a rather limiting assumption that all users in the same search system follow a single pattern of positional bias. This may not be true, as different users may have different positional biases. A more practical approach is to insert perturbations to the ranking positions to estimate the true positional effects using IPS. To mitigate the risks of perturbations, methods have been introduced to swap only adjacent search results.\n\n**A:** We thank the reviewer for pointing out the reference on the estimation of propensities and it is worth pointing out that in the reference paper, the probability of being examined actually only depends on the rank, and it is the same across users. Instead, we want to point out that one of our main advantages is that we do not need to learn a separate propensity estimation model. \n\n**Q:** The description of the proposed method lacks sufficient details. Also the experimental results (Table 1) did not seem to contain significant differences between the algorithms being considered. The paper did not discuss common limitations of RL algorithms.\n\n**A:** We provide details of our solution using SAC and CQL and their hyper-parameters in the Appendix (see Table 5 in Section B for details). Please refer to our answer in the general response to all reviewers regarding performance difference between SAC and CQL. \n\n**Q:** In the results, the baseline method which was used to simulate the user behaviors did not perform the best in estimating the user behaviors. Is this expected? Also, the differences between all methods seem pretty small.\n\n**A:** If we understand correctly that \"the baseline method which was used to simulate the user behaviors\" refers to IPW/DLA in PBM model and CM-IPW in Cascade model in Table 1, we would like to clarify that it does not have to perform best. Note that while designed for the specific click models, these methods still need to estimate parameters such as attractiveness and/or bias parameters in click model, and whether they can learn efficiently determines their performance. But we can see that methods designed for specific click model generally perform better than mismatched methods, e.g., CM-IPW>IPW/DLA in Cascade model.\n\n**Q:** Algorithm 1. Line 6, where is psi defined?\n\n**A:** We apologize for the confusion. $\\psi$ is but mentioned in Line 2, the suffix of $\\phi_\\psi(\\cdot,\\cdot)$. It refers to the trainable parameters in the embedding model $\\phi$.\n\n**Q:** Eq (5). What is the difference between CQL and SAC?\n\n**A:** According to Eq.5 in Line 244, the CQL algorithm overcomes the distribution shift problem in offline RL by Conservative Q-learning. In its loss function, a conservative term is added (the first term in the RHS of the equation after $\\alpha$) to constrain the difference between the logging policy and our trained policy in case of the overestimation problem. More details can be found in [1] and we will add more discussion in next version.\n\n[1] Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 33:1179–1191, 2020* "}, {"review_id": "QVTcOKyYBz", "reviewer": "Reviewer_79J3", "summary": "This paper presents a unified approach for off-policy learning to rank (LTR) that is adaptable to general click models. The authors formulate off-policy LTR as a Markov Decision Process (MDP) and leverage offline reinforcement learning (RL) techniques to optimize the ranking process. They propose the Click Model-Agnostic Unified Off-policy Learning to Rank (CUOLR) method, which can be easily applied to a wide range of click models. The authors provide empirical evidence demonstrating the effectiveness of CUOLR in comparison to state-of-the-art off-policy LTR algorithms.\n", "questions": "● Lines130-133 are hard to understand. Why not include remaining documents that are yet to rank? How will the policy perform on $s_0$, where nothing will be included as input?\n● The introduction of dynamic action space can introduce dynamics shift in RL and unstable training. Are there any specific techniques to handle this issue?\n● Why does SAC obtain similar performance with CQL when learning in static datasets? Will the issue of policy distribution mismatch lead to poor performance?", "limitations": "The formulation and evaluation are limited to off-policy/offline RL that learn from a static dataset. In applications where there are adequate online interaction data, it will be helpful to consider some online RL counterparts.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "● The paper presents an innovative and practical methodology for off-policy learning to rank. The formulation of off-policy LTR as an MDP and the use of offline RL techniques provide a comprehensive and adaptable approach for ranking optimization.\n● The authors provide insightful empirical findings, showing that the CUOLR method consistently outperforms state-of-the-art off-policy learning to rank algorithms. The results on various large-scale datasets demonstrate the effectiveness, consistency, and robustness of CUOLR under different click models.\n● The paper presents a detailed introduction on state representation learning. There are also empirical analyses on its practical effect.", "weaknesses": "● The synthetic dataset and the offline evaluation can give biased evaluation results of the algorithms.\n● There are some related works that are not mentioned in this paper. Cai, et al. [1] also apply RL techniques to recommendation systems and provide a new MDP formulation. It also includes ranking scores in the formulation that can be regarded as a downstream application to this paper. Xue, et al. [2] provides another MDP formulation of RL for optimizing long-term user engagement.\n● There remain some issues unsolved in the paper. See the questions for details.\n\n[1] Cai, Qingpeng, et al. \"Reinforcing User Retention in a Billion Scale Short Video Recommender\nSystem.\" arXiv preprint arXiv:2302.01724 (2023).\n[2] Xue, Wanqi, et al. \"PrefRec: Preference-based Recommender Systems for Reinforcing Long-term User Engagement.\" arXiv preprint arXiv:2212.02779 (2022).\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive comments on our work. \n\n**Q:** The synthetic dataset and the offline evaluation can give biased evaluation results of the algorithms.\n\n**A:**  First, same as previous off-policy learning to rank research (consistent with all baseline methods), we conduct the semi-synthetic experiments where the training data are all the direct feedback from a \"simulated\" user (the click model reflects his/her behavior) on real-world queries. This is different from offline evaluation since we have control of the environment and know the groundtruth of any ranking list (trajectory).  In this setup, the only bias comes from the click model, which is different from the bias in typical offline evaluation RL. For more details, please see the experiment setup section (section 5.1).\n\n**Q:** There are some related works that are not mentioned in this paper. Cai, et al. [1] also apply RL techniques to recommendation systems and provide a new MDP formulation. It also includes ranking scores in the formulation that can be regarded as a downstream application to this paper. Xue, et al. [2] provides another MDP formulation of RL for optimizing long-term user engagement.\n([1] Cai, Qingpeng, et al. \"Reinforcing User Retention in a Billion Scale Short Video Recommender System.\" arXiv preprint arXiv:2302.01724 (2023). [2] Xue, Wanqi, et al. \"PrefRec: Preference-based Recommender Systems for Reinforcing Long-term User Engagement.\" arXiv preprint arXiv:2212.02779 (2022).)\n\n**A:** Thank you for providing these two papers on RL for recommendation. Their settings are different from our LTR setup, we will definitely add the discussion in our next version.\n\n**Q:** Lines130-133 are hard to understand. Why not include remaining documents that are yet to rank? How will the policy perform on s0, where nothing will be included as input?\n\n**A:** According to click models, the click at each position only depends on current position and previous documents. Remaining documents that are yet to rank are considered in action set instead of our state representation. At position 0 (state $s_0$), the policy will choose the document with the highest relevance/attractiveness from all candidates.\n\n**Q:** The introduction of dynamic action space can introduce dynamics shift in RL and unstable training. Are there any specific techniques to handle this issue?\n\n**A:** Thank you for pointing out this question. We agree that the raw action space is dynamic. However, as we utilize the attention model and project the raw action space into a latent action space, there is intended to be less dynamic shift happening there. \n\n**Q:** Why does SAC obtain similar performance with CQL when learning in static datasets? Will the issue of policy distribution mismatch lead to poor performance?\n\n**A:** Please refer to our answer in the general response to all reviewers regarding the performance difference between SAC and CQL."}, {"review_id": "T4ThhMtfeK", "reviewer": "Reviewer_ynTd", "summary": "This paper aims to unify the assumptions on user behavior in a ranking called click model by formulating the ranking problem as a click-model agnostic Markov Decision Process (MDP). By doing so, the paper proposes to reduce the Off-Policy Learning to Rank (LTR) problem to a variant of offline RL, which does not require precise estimation of the click models. Under this formulation, the proposed offline RL method, which incorporates state representation learning to the well-known Conservative Q-Learning (CQL), enables more accurate estimations of ranking metrics than baseline estimators in real-world datasets.", "questions": "- In general, SAC does not perform very well in the offline RL setting and thus the offline RL algorithms, including CQL, have been proposed. However, the experiment results show that CUOLR (CQL) and CUOLR (SAC) perform competitively. Could you provide some justification for this result?", "limitations": "This limitation is not special for this paper, but classical click models and MDP both assume that the reward observed at each position are not affected by lower positions including the neighboring ones. If this assumption does not hold, LTR may introduce some bias and offline RL may also ignore some causal relation between actions and rewards.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The manuscript is easy to follow. The motivation for unifying click models and introducing the offline RL framework is clearly explained.\n\n2. The RL formulation of LTR is reasonable, and showing the connection between offline RL and Off-Policy LTR would be insightful for the LTR community.\n\n3. The ablation study on state representation learning is insightful.\n", "weaknesses": "1. While the connection between LTR and offline RL is interesting, the proposed method (CUOLR) itself is not a fundamentally new framework for offline RL. In particular, the proposed state representation learning method, which applies positional encoding and attention to the inputs, seems to be an engineering effort rather than a very novel framework.\n\n2. In experiments, an offline RL algorithm (CQL) does not show advantages over a simple RL baseline (SAC). While I acknowledge the author(s)’ contribution to formulating LTR as an RL problem, I think the algorithm has room for improvement.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer for recognizing the advantages of our unified RL formulation, connecting LTR and RL solution, and our ablation study. \n\n**Q:** The proposed method (CUOLR) itself is not a fundamentally new framework for offline RL, and the proposed state representation learning method seems to be an engineering effort rather than a very novel framework.\n\n**A:** While applying positional encoding and attention to the inputs contains certain engineering efforts, we would like to emphasize that our proposed CUOLR method is novel in terms of its unified MDP formulation that is agnostic to various click models, novel state representation design and flexible to apply any offline RL solvers. This also gives a much better performance than the tailored method for each specific click model. \n\n**Q:** Classical click models and MDP both assume that the reward observed at each position are not affected by lower positions including the neighboring ones. If this assumption does not hold, LTR may introduce some bias and offline RL may also ignore some causal relation between actions and rewards.\n\n**A:** We agree with the reviewer that classical click models assume that the reward is not affected by lower positions. It would be a challenging yet interesting future work to broaden the scope and relax the assumption."}]

Author rebuttal:
We thank all the reviewers for your detailed comments and questions, which helped us to improve our paper. We appreciate that the reviewers generally agree our unified formulation is well-motivated, our click model-agnostic approach is novel and original, and experimental results are extensive and support the claim. We have responded to your individual comments and will add the following new results and discussion in the paper.

**CQ1:**   Performance Comparison of CQL and SAC (by Reviewer XmTH, 79J3, ynTd).

**A1:** We first emphasize that the main contribution of the paper is formulating off-policy learning to rank (LTR) problem as an MDP and proposed an offline RL framework that leverages **any** off-the-shelf RL algorithm as the plug-in solver. We choose the popular CQL algorithm to conduct experiments as it can alleviate over-estimation caused by distributional shift with conservative Q-learning. However, as reviewers have noticed from the results in main paper, SAC (which is equivalent to CQL with conservative parameter $\alpha=0$) performs closely to CQL with the initially chosen fixed $\alpha=0.1$.  To further investigate this, we report the performance of SAC and CQL with optimal $\alpha$ in Table 2 of the PDF, where we search for optimal value separately for each click model. We observe that the performance of CQL with optimal $\alpha$ is much higher than simple SAC especially in NDCG. This observation suggests that distribution shift and over-estimation are still an issue in the off-policy LTR setting, and making CQL work needs additional effort of careful tuning, which is the case for other domains as well. In this paper, we have shown that the off-policy LTR problem can be unified and solved by our click-model agnostic offline RL framework with simple RL solvers like CQL and SAC. Comparing the pros and cons of various advanced RL algorithms in off-policy LTR is beyond the scope of this paper, but an interesting follow-up. 

**CQ2:** The impact of logging policy (by Reviewer XmTH, fhCJ)

**A2:** We conduct new experiments on the impact of logging policies on Web10k dataset. On each click model, we compare three different logging policies: SVMRank trained with 1% training data, SVMRank trained with 0.01% data, and random policy. The result of CUOLR-CQL with fixed $\alpha=0.1$ is reported in Table 1 of the PDF.  Not surprisingly, we can see that with a worse logging policy, the performance of learned policy also decreases. This also highlights the need for a good logging policy in offline LTR.

Meta-review: Reviewers appreciate the overall perspective of learning to rank (LTR) as an offline RL problem learned from logged click data.  While the idea of viewing LTR as an MDP is not new, the authors claim that learning from implicit feedback (click data) and the ability to leverage various click models under a unified framework is new.  Reviewers generally agree with these claims of novelty and believe this approach and use of offline RL may inspire follow-on research.  The rebuttal discussion and additional experimental results had a positive impact on reviewer opinion and there was a general consensus among all five reviewers that the paper could be accepted, though some reviewers maintain residual concerns post-rebuttal that should be addressed in the final revision.

In general, the authors are requested to incorporate points of clarification from the post-rebuttal discussion.  In addition, reviewers expressed specific concerns during final decision discussion that the following key points need to be included on revision:
- Overstatement of claims (about state representations and work in the Appendix) need to be corrected.
- Additional experiments and comments presented in the rebuttal (CQL, effect of the logging policy) need to be included (in the main paper or Appendix) and discussed in the main paper.
- Clarification of the Markovian property -- I include the reviewer's comment verbatim: "Authors representation of the state takes into account the sequence of documents, as against just the set of documents viewed by the user because the authors concatenate the representations of prior states (an order sensitive aggregation) as against a summation or mean representation of prior states (an order insensitive aggregation). In my opinion, that invalidates authors' claim about this setup being an MDP because the next state of the agregated MDP depends on the sequence of prior states, not just an aggregation of the prior states. This means that Markovian property does not hold, at least when it comes to vanilla MDP. The authors could possibly prove that it is the case for a higher order Markov Chain, but they have not."
Decision: Accept (poster)