Title: Unified Off-Policy Learning to Rank: a Reinforcement Learning Perspective
Venue: NeurIPS 2023
Accepted: yes
Keywords: learning to rank

off-policy learning

reinforcement learning

click model
Abstract: Off-policy Learning to Rank (LTR) aims to optimize a ranker from data collected by a deployed logging policy. However, existing off-policy learning to rank methods often make strong assumptions about how users generate the click data, i.e., the click model, and hence need to tailor their methods specifically under different click models. In this paper, we unified the ranking process under general stochastic click models as a Markov Decision Process (MDP), and the optimal ranking could be learned with offline reinforcement learning (RL) directly. Building upon this, we leverage offline RL techniques for off-policy LTR and propose the Click Model-Agnostic Unified Off-policy Learning to Rank (CUOLR) method, which could be easily applied to a wide range of click models. Through a dedicated formulation of the MDP, we show that offline RL algorithms can adapt to various click models without complex debiasing techniques and prior knowledge of the model. Results on various large-scale datasets demonstrate that CUOLR consistently outperforms the state-of-the-art off-policy learning to rank algorithms while maintaining consistency and robustness under different click models.

Unified Off-Policy Learning to Rank: a Reinforcement Learning Perspective

Abstract

Off-policy Learning to Rank (LTR) aims to optimize a ranker from data collected by a deployed logging policy. However, existing off-policy learning to rank methods often make strong assumptions about how users generate the click data, i.e., the click model, and hence need to tailor their methods specifically under different click models. In this paper, we unified the ranking process under general stochastic click models as a Markov Decision Process (MDP), and the optimal ranking could be learned with offline reinforcement learning (RL) directly. Building upon this, we leverage offline RL techniques for off-policy LTR and propose the Click Model-Agnostic Unified Off-policy Learning to Rank (CUOLR) method, which could be easily applied to a wide range of click models. Through a dedicated formulation of the MDP, we show that offline RL algorithms can adapt to various click models without complex debiasing techniques and prior knowledge of the model. Results on various large-scale datasets demonstrate that CUOLR consistently outperforms the state-of-the-art off-policy learning to rank algorithms while maintaining consistency and robustness under different click models.

1 Introduction

Learning to Rank (LTR) is a core problem in Information Retrieval (IR) with wide applications such as web search and recommender systems [33]. Traditional LTR methods require high-quality annotated relevance judgments for model training, which is expensive, time-consuming, and may not align with actual user preferences [41]. As a result, learning to rank with implicit user feedback, such as logged click data, has received a huge amount of attention in both academia and industry [23, 26, 53].

Despite its low cost, learning to rank directly from implicit user feedback could suffer from the intrinsic noise and bias in user interactions, e.g., position bias, where an item displayed at a higher position receives a higher click-through rate (CTR) than its relevance [12]. To mitigate the bias in the click data, off-policy learning to rank methods have been proposed under different bias assumptions such as position bias [53, 2], selection bias [52] and trust bias [36, 9]. A major branch of off-policy learning to rank called counterfactual learning to rank achieves unbiasedness by re-weighting the samples using the inverse propensity scoring (IPS) method [46, 2, 27]. To estimate the propensity from logged click data, existing works require explicit assumptions about how users examine the rank list and generate the click data, i.e., click models [12, 10]. For example, position-based model

(PBM) [24] assumes the probability of examining a result only depends on the position; while cascade model (CASCADE) [12] assumes each click depends on the previous click, and dependent click model (DCM) [18] considers both. Different debiasing methods have been proposed to cator to specific click models, including PBM [4, 53, 26], CASCADE; and DCM [48]. However, prior knowledge of the click model is usually unknown and the correct click model needs to be identified from user behavior data before applying an off-policy algorithm, which is challenging in complex real-world environments. Besides, many popular and powerful click models have not been studied in counterfactual learning to rank such as the click chain model (CCM) [17] and the user browsing model (UBM) [13]. It requires a significant amount of work to study debiasing methods for every popular click model.

To overcome these issues, we propose to study a unified approach of off-policy learning to rank adaptable to general click models. Our key insight is that the user's examination and click behavior summarized by click models has a Markov structure; thus off-policy LTR under general click models can be formulated as a Markov Decision Process (MDP) . Specifically, the learning to rank problem now can be viewed as an episodic RL problem [45, 1], where each time step corresponds to a ranking position, each action selects a document for the position, and the state captures the user's examination tendency. This formulation allows us to view off-policy LTR from the perspective of offline reinforcement learning [31], where we can leverage off-the-shelf offline RL algorithms [29, 19, 16] to optimize the ranking list. Importantly, our formulation bridges the area of off-policy learning to rank and offline RL, allowing for the integration of ideas and solutions from offline RL to enhance the solution of the off-policy LTR problem.

Inspired by the formulation, we propose the Click Model-Agnostic Unified Off-policy Learning to Rank (CUOLR) method. We first construct each logged query and ranking data as an episode of reinforcement learning following the MDP formulation. Our dedicated structure for state representation learning can efficiently capture the dependency information for examination and click generation, e.g. ranking position in PBM and previous documents in CM and DCM. The algorithm jointly learns state representation and optimizes the policy, where any off-the-shelf offline RL algorithm can be applied as a plug-in solver. Specifically, we adapt the popular CQL algorithm [29] as an instantiation, which applies the conservative (pessimism) principle to Q function estimate. We evaluate our algorithm on real-world learning to rank datasets [38, 7] under various click models. Compared with off-policy LTR methods that are dedicated to specific click models, our click model-agnostic method consistently outperforms the best-performing baselines in all click models.

The contributions of this paper are summarized as follows:

We formulate the off-policy LTR with biased feedback under general click model as a Markov Decision Process, and bridge the area of off-policy learning to rank and offline reinforcement learning.

We propose CUORL, a Click model-agnostic Unified Off-policy LTR method that could utilize any offline RL algorithm as a plug-in solver, and we instantiate it using CQL.

We conduct extensive empirical experiments to validate the effectiveness of our algorithm using real-world LTR datasets under different click models.

2 Related Work

Off-policy Learning to Rank. Off-policy Learning to Rank aims to optimize the ranking function from logged click data [23]. The majority of the works aim to mitigate the bias in logged click data, known as counterfactual learning to rank or unbiased learning to rank. The debiasing methods mainly follow inverse propensity scoring strategy [26, 49, 4, 51, 48, 3], while there are also recent works applying doubly robust estimator to reduce variance [40, 28, 37]. Cief et al. [11] proposed pessimistic off-policy optimization for learning to rank that also mitigates bias but not in an unbiased way. All these methods rely on prior knowledge of the click model [12, 10], while our algorithm is agnostic to general click models.

Offline Reinforcement Learning. Offline RL algorithms [29, 14, 44, 21, 16, 15] learn policy from large logged datasets where the distributional shift between the logging policy and the learned policy imposes a major challenge. In this setting, different algorithms are proposed, from value-based ones

(e.g. [35, 34]) to policy-based ones (e.g. [43, 42]). Among the vast literature on offline reinforcement learning, the principle of pessimism/conservatism [29, 5, 22] is an important line and has inspired many algorithms from empirical and theoretical perspective [60, 39, 32, 56, 61, 55, 58]. While all the aforementioned methods can be plugged into our algorithm, we choose the classic CQL algorithm [29] with a conservative Q function on top of soft actor-critic algorithm [19].

Reinforcement Learning to Rank. Wei et al. [54] first model ranking problem as an MDP, where the state is the candidate document set at current rank and the action is the selected document. [57, 62] have been studied under similar MDP formulation. However, [54, 57] requires relevance labels as feedback and cannot mitigate bias in click data; [62] is an online learning algorithm that learns from user interactions instead of logged data. Compared to these studies, we characterize the MDP formulation from a different perspective, i.e., capture bias in the click model, and propose the offline RL algorithm with logged click feedback.

3 Reinforcement Learning to Rank: A Unified Formulation

As the majority of existing works in unbiased learning to rank focused on inferring documents' relevance from the click models, these methods are tied to specific click models adopted. In this section, we formulate learning to rank with general click feedback as a Markov decision process, offering a unified and comprehensive modeling of possibly complicated user behavior. This formulation unifies LTR problems associated with different click models, under which the underlying click model is translated into the environment setup of MDP such as state transitions and rewards. It opens up the possibility to employ a rich class of reinforcement learning algorithms for solving LTR, which we will give greater details in the next section.

3.1 Preliminary

Click model. A key challenge of off-policy LTR lies in learning the document's attractiveness/relevance from the implicit feedback that is biased by the user's examination behavior. To address this challenge, a range of click models have been proposed to accommodate user's various click behavior [10]. In this study, we focus on a general family of click models [63, 30], which is marked by two characteristics: (1) Most of the mainstream click models have a "two-step" flavor that breaks user's click behavior towards some document down into the user's examination and the document's relevance. For each document d , the user first decides whether to examine it in the ranking list, based on the specific behavior. Mathematically the user behavior is modeled as the examination probability, which generally depends on the ranking list R and the position of the document k , denote as χ ( R , k ) . Once the document is examined, the user will choose whether to click it, based on the attractiveness α ( d ) 2 . (2) Any documents under the k -th position do not have an effect on χ ( R , k ) .

Definition 1 (Click Model) . For any rank list R and position k , the attractiveness and examination probability are independent.

where C k is the click indicator at rank k , and χ ( R , k ) is the examination probability of position k in the rank list R . For each document d , the attractiveness α ( d ) only depends on the document itself. And the attractiveness is mutually independent.

We show that classic click models such as PBM, CASCADE, DCM, and CCM are instances of Definition 1, with details listed in Appendix C.

3.2 Learning to Rank as Markov Decision Process

In Reinforcement Learning (RL), the interactions between the agent and the environment are often described as an finite-horizon discounted Markov Decision Process M = ( S, A, T, r, γ, H ) . The goal of the RL problem is to find a policy π that maximizes the value function, i.e. the discounted

cumulative reward

In what follows, we formulate each of the ( S, A, T, r, γ, H ) components in the ranking scenario. Our formulation essentially differs from the existing MDP formulation of ranking [54], where the state at position k is defined as remaining documents that are yet to rank following the k -1 ranked ones on the top, instead Wei et al. [54] has a limited capturing of user's click behavior as a result of being ignorant of the ordering within the top k -1 documents. From here on, we use k ∈ [ K ] to denote the k -th position top down on a list of total length K . It serves as the counterpart of the time step t in (2) for the ranking setting.

State S For each position k ∈ [ K ] , state s k should include and represent the current status of the ranking that the agent is faced with. Thus, we define the state at rank k as:

which is a concatenation of the established sub-ranking list up to k , denoted by ( d 1 , d 2 , . . . , d k -1 ) , and the position k . Here d i refers to the document presented at rank i with s 0 is initialized as = [() , 0] . Together with the action a k as to select document d k presenting at rank k , defining s k as (3) fully captures the user's click behavior C k at this point. Recall (1) that P ( C k = 1 | R , k ) = χ ( R , k ) α ( R ( k )) , where χ ( R , k ) is determined by ( d 1 , d 2 , . . . , d k -1 ) . To better capture the rich information in the observed state, we discuss how to attain the effective state embedding from the raw representation in Section 4.

Action A Action a k is naturally defined as the document to present at rank k given state s k . In our experiments, each action is represented by a feature vector associated with the query. It is worth mentioning that the available action set A k at each k is related to state s k as well as the specific query, unlike the common case where the action space is fixed. There are two main differences here compared with the fixed action space: (1). the available actions vary under different queries; and (2). once an action is chosen, it is removed from the candidate set A k .

Transition T ( s ′ | s, a ) Transition maps a state-action pair to the probability distribution over possible next states. Given our formulation of state and action, the next state s k +1 is deterministic given s k = [( d 1 , d 2 , · · · , d k -1 ) , k ] and a k . Formally, T ( s k +1 | s k , a k ) = 1 if and only if s k +1 = [( d 1 , d 2 , · · · , d k -1 , a k ) , k +1] . Note that with this transition function, we can show that our model is a rigorous MDP, where the distribution of next state only based on the current state and the action.

Reward r ( s, a ) Aligned with the goal of LTR to maximize total clicks, we adopt the binary click as a reward at each position, i.e. r ( s k , a k ) = C k . It is easily checked this is a well-defined reward from (1) that the distribution of r is fully determined by s and a , i.e., E [ r ( s k , a k )] = χ ( s k ) α ( a k ) .

Putting the components together we have formally built up our MDP formulation of LTR, which we name as " M DP for R anking" and denote by MR ( S , A , T , r, γ, H ) with components defined as above. The rich RL literature has the convention to solve the optimal policy π ∗ that maximizes the cumulative reward, which in our proposed MDP translates to

where the expectation is taken over the stochasticity in both environments MR and policy π . Before leveraging any RL algorithms for ranking, it is necessary to validate good policies of MR ( S , A , T , r, γ, K ) yields good ranking lists. In the following subsection, we give a rigorous theorem for this validation.

3.3 Optimizing Rank List by Optimizing Policy

Constructing a rank list given policy π . With the definition of MDP, we can construct a rank list with any given policy sequentially. At each position k , the state is constructed based on previous document features (manually or by sequential models). Then a document (action) is chosen by the policy π and placed at position k , where the next state is determined accordingly. Repeat this process at each position until all the documents are set or the list reaches its end ( K = 10 for example).

Definition 2 (Policy Induced Ranking.) . Given a policy π ( · | s ) of MR ( S , A , T , r, γ ) , construct the induced rank list R π as

To investigate whether the optimal rank list can be captured by optimal policy π ∗ , we start with defining the optimality of rank list.

Definition 3 (Optimality) . A rank list R is optimal if and only if all documents are sorted in descending order in terms of their attractiveness, i.e.

and {R (1) , · · · , R ( K ) } are the K most attractive documents among all, where K is the length of the list and it is also called the top-K ranking.

Assumption 3.1 (Optimality of optimal ranking) . Let V R ( s 1 ) = E [ ∑ K k =1 γ k -1 r ( s k , R ( k ))] be the value of rank list R , and let R ⋆ be the optimal rank list by Definition 3. Then max R V R ( s 1 ) = V R ⋆ ( s 1 ) .

Assumption 3.1 is adopted from Assumption 2 in [30], which suggests optimal rank list sorted by decreasing attractiveness of documents leads to optimal rewards, which will be covered by optimal policy learned from our MDP formulation. This is a mild assumption as classic click models such as PBM and cascade model all satisfy the assumption [63].

4 Unified Off-policy Learning to Rank

The formulation of off-policy learning-to-rank (LTR), when viewed from the perspective of offline reinforcement learning, presents an opportunity to leverage off-the-shelf RL algorithms to address off-policy LTR problems. In this section, we introduce a novel and unified off-policy LTR algorithm that is agnostic to the underlying click model used to generate the offline training data. Our algorithm is composed of three key components: (1) episodes constructed from the logged ranking data; (2). state representation learning; and (3). policy optimization via offline RL algorithms. In the following, we provide a detailed exposition of each component.

Episodes Construction. Given any logged ranking dataset, the first step is to construct a series of episodes from the ranking data, making it compatible with the off-the-shelf RL algorithms. Specifically, our original ranking data { ( q i , R i , c i ( q i , R i )) } n i =1 is composed of n tuples with contextual query feature q i , a ranked list R i with K documents and a corresponding click vector c i ( q i , R i ) ∈ { 0 , 1 } K . From the perspective of RL, this tuple can be conceptualized as one episode, following the established MDP formulation of the ranking process. In particular, for each tuple ( q i , R i , c i ( q i , R i )) , we transform it to a length K episode τ i := { ( s i k , a i k , r i k ) } K k =1 with

Here we use R i [: k ] to denote the concatenation of the document feature vectors before position k and R i [ k ] represents the document feature at position k , similarly for c i ( q i , R i )[ k ] . In particular, the episode τ i is constructed by going through the ranked list from top to the bottom. Each state s i k contains all the document information before position k and the position information k , represented as a function of R i [: k ] and k with ϕ being a learned embedding function that we will discuss shortly. The action at step k is the document placed at position k , i.e., R i [ k ] , with the reward at current timestep is the binary click for the corrpsonding action at position k , i.e., c i ( q i , R i )[ k ] . Given this, we have constructed an offline RL dataset with n episodes with length K each.

State Representation Learning. In RL, state representation learning offers an efficient means of managing large, raw observation spaces, enhancing the generalization performance when encountering previously unseen states. In our particular setting, the observation space consists of raw document features up to position k . As we will show in Section 5.2, utilizing raw observations as the state representation in the RL algorithm can lead to highly sub-optimal performance, due to both unscalability with respect to k and limitations in the representation power. Rather than incorporating additional auxiliary tasks to explicitly learn state representations, we propose to jointly learn the state

representations together with the policy optimization algorithm, which aims to automatically learn state representation ϕ that will benefit the downstream policy optimization task. For example, DQN uses multiple layers of nonlinear functions to encode the map perceptual inputs to a state embedding that could be linearly transformed into the value function. To this end, we introduce the implicit state representation learning component our off-policy learning to rank algorithm, which is composed of the following key components:

Positional Encoding : To effectively inject the position information in s i k , we utilize the positional encoding technique [50, 59], ensuring the model make use of the position information when generating clicks. Specifically, positional encoding represents each position k in the ranked list by the sinusoidal function with different frequencies such that

Here PE ( k ) ∈ R d with d being the dimension of document feature and 2 i and 2 i +1 being the even and the odd entries of PE ( k ) with 2 i, 2 i +1 ∈ [ d ] .

Multi-head Self-attention : The other challenge in our case is to find a specific architecture for the state representation learning that tailored for the learning to rank task. As the ranked list of document features is inherently sequential, we leverage the multi-head self-attention mechanism to learn the state embedding ϕ . Specifically, the state s i k is defined as:

where head i = Attention( s k · W Q i , s k · W i K , s k · W V i ) , with W Q i , W i K , W V i ∈ ψ i are learnable parameters for the i th head and W O is the learnable parameter of the output layer after concatenating the results of all the I heads; At each position k , we concatenate the document features R i [: k ] along with the position embedding for position k , passing them as the input for the multi-head self-attention.

Joint Representation Learning and Policy Optimization. Given the constructed offline dataset and a dedicated structure for learning efficient state representations, we now demonstrate how we leverage any off-the-shelf RL algorithm as the plug-in solver for finding the optimal policy. We use the popular offline RL algorithm: CQL, as an instantiation, which learns a conservative Q function and utilizes the classical soft actor-critic algorithm on top of it. Specifically, it optimizes the following lower bound of the critic (Equation 5):

where ˆ B π Q = r + γP π Q is the estimated Bellman operator , π β is the logging policy. Here we use θ ′ to emphasize that the parameters in the target Q network is different from policy Q network. The conservative minimizes the expected Q value for out-of-distribution (state, action) pairs and prevents the Q-function's over-estimation issue. Built upon SAC, the algorithm improves the policy π ξ (i.e., the actor) based on the gradient of the estimated Q function, with entropy regularization. Compared with the original CQL algorithm, we also add the state representation learning component, and we jointly optimize the state embedding, the critic and actor parameters, with a detailed algorithm included in Algorithm 1. Once we have obtained the learned optimal policy π ∗ ξ , we extract the optimal ranking policy following Definition 2.

5 Experiments

We empirically evaluate the performance of our proposed method CUOLR on several public datasets, compared with the state-of-the-art off-policy learning-to-rank methods. Specifically, we aim to assess the robustness of our method across different click models and showcase the effectiveness of our unified framework. Beyond this, we perform the ablation study to examine the efficacy of our proposed state representation learning component. 3 .

Algorithm 1 Click Model-Agnostic Unified Off-policy Learning to Rank (with CQL)

1: Inputs: logged ranking data { ( q i , R i , c i ( q i , R i )) } n i =1 , length of the ranking K , batch size B , train iteration T .

2: Initialize: policy π ξ and Q function Q θ , embedding model ϕ ψ ( · , · )

3: for t ∈ [ T ] do :

4: Randomly sample a batch of queries Q with size B .

5: Construct offline RL episodes T = { { ( s i k , a i k , r i k ) } K k =1 } i : q i ∈Q .

6: Train the Q-net (and embedding model) with loss defined in equation (5)

7: Improve policy π ξ (and embedding model) with SAC-style entropy regularization.

8: end for

9: Output: learned ranking policy π ∗ ξ , Q function Q ∗ θ , embedding model ϕ ∗ ψ ( · , · )

10: Recover the optimal ranking from the learned policy π ∗ ξ using Definition 2.

5.1 Setup

Datasets. We conduct semi-synthetic experiments on two traditional learning-to-rank benchmark datasets: MSLR-WEB10K and Yahoo! LETOR (set 1). Specifically, we sample click data from these real-world datasets, which not only increases the external validity of the experiments but also provides the flexibility to explore the robustness of our method over different click models. For both datasets, it consists of features representing query and document pairs with manually judged relevance labels ranging from 0 (irrelevant) to 4 (perfectly relevant). We provide the statistics of all datasets in Appendix A. Both datasets come with the train-val-test split. The train data is used for generating logging policy and simulating clicks, with the validation data used for hyperparameter selection. And the final performance of the learned ranking policy is evaluated in the test data.

Click Data Generation. We follow Joachims et al. [25] to generate partial-information click data from the full-information relevance labels. Specifically, we first train a Ranking SVM [26] using 1% of the training data as our logging policy π β to present the initial ranked list of items. For each query q i , we get a ranking R i and simulate the clicks based on various click models we use. As discussed in Section 3.1, there are two components for the click generation, examination probability and attractiveness of the document for the query. All click models differ in their assumptions on the examination probability. For PBM, we adopt the examination probability ρ = { ρ k } K k =1 estimated by Joachims et al. [25] through eye-tracking experiments:

where η ∈ [0 , + ∞ ] is a hyper-parameter that controls the severity of presentation biases and in our experiment, we set η = 1 . 0 as default. For CASCADE, the examination probabilities are only dependent on the attractions of each previous document. For DCM, the λ s are also simulated by the same parameters as PBM examination probability. We use the same attraction models for all click models, as defined following:

where r ( d ) ∈ [0 , 4] is the relevance label for document d and r max = 4 is the maximum relevance label. We also use ϵ to model click noise so that irrelevant documents have a non-zero probability to be treated as attractive and being clicked.

Baselines and Hyperparameters. We compare our CUOLR method with the following baselines: (1). Dual Learning Algorithm (DLA) [4] which jointly learns an unbiased ranker and an unbiased propensity model; (2). Inverse Propensity Weighting (IPW) Algorithm [52, 26] which first learns the propensities by result randomization, and then utilizes the learned probabilities to correct for position bias; and (3). Cascade Model-based IPW (CM-IPW) [48] which designs a propensity estimation procedure where previous clicks are incorporated in the estimation of the propensity. It is worth mentioning that (1) and (2) are designed for PBM and (3) is tailored for cascade-based models. Besides, we train a LambdaMart [6] model with true relevance labels as the upper bound for the ranking model, ORACLE for short. The performance of the logging policy ( LOGGING ) is also reported as the lower bound of the ranking model.

For baselines, we use a 2-layer MLP with width 256 and ReLU activation according to their original paper and codebase [4, 51, 47]. For the embedding model in our method, we use multi-head attention with 8 heads. And for actors and critics in CQL and SAC algorithms, we utilize a 2-layer MLP with width 256 and ReLU activation. The conservative parameter α (marked red in Equation (5)) in CQL is set to 0.1. We use Adam for all methods with a tuned learning rate using the validation set. More details are provided in Appendix A.

Metrics. We evaluate all the methods using the full-information test set. We use the normalized Discounted Cumulative Gain (nDCG) [20] and the Expected Reciprocal Rank (ERR) [8] as evaluation metrics and report the results at position 5 and 10 to demonstrate the performance of models at different positions.

5.2 Results

How does CUOLR perform across different click models, compared to the baselines? To validate the effectiveness of our CUOLR method, we conducted performance evaluations across a range of click-based models, including PBM, CASCADE, DCM, and CCM. We compared our approach with state-of-the-art baselines specifically designed for these click models, namely DLA, IPW, and CM-IPW. Due to space limitation, we only show results of PBM, CASCADE, and DCM in Table 1, with the full table shown in Appendix B. For the position-based model, IPW demonstrates the

best performance among the baselines, which is expected as it is tailored for position-based methods. Similarly, CM-IPW yielded the best performance for the cascade-based methods, which aligns with its incorporation of previous document information in the propensity estimation. Remarkably, across all click models, our method, whether combined with SAC or CQL, consistently achieves the best performance in most cases. This validates the effectiveness of our unified framework and the robustness of our CUOLR algorithm. Furthermore, it is noteworthy that our method demonstrated consistent performance across different RL algorithms, verifying its resilience and adaptability to various underlying RL solvers.

How effective is the state representation learning component in CUOLR? In this experiment, we examine different approaches for state representation learning and study how it affects the overall performance of our proposed method. We compare with the following state embeddings: (1). positiononly embedding (POS), which only utilizes the position information using positional encoding; (2). previous-document-based embedding (PREDOC), which takes a simple average of all the document features in R [: , k ] ; (3). the concatenation of the position and the average document features up to position k (POS + PREDOC), as well as the proposed learnable state representations based on multihead self-attention (ATTENTION). ORACLE here is used to show the gap from the upper bound. The results of our experiments are presented in Table 2 (with a full table including other click models is shown in Appendix B). For the PBM click model, it is evident that state embeddings utilizing position-based information, such as POS and POS+PREDOC, outperform other state embeddings. In contrast, for the CASCADE click model, state embeddings utilizing previous document features exhibit significantly stronger performance compared to those utilizing position information. Notably, our method, CUOLR, which dynamically learns the state embeddings during policy optimization, consistently achieves comparable performance compared to using hard-coded fixed state embeddings. This highlights the necessity of leveraging state representation in off-policy LTR and underscores the effectiveness of our proposed approach.

6 Conclusion

In this paper, we present an off-policy learning-to-rank formulation from the perspective of reinforcement learning. Our findings demonstrate that under this novel MDP formulation, RL algorithms can effectively address position bias and learn the optimal ranker for various click models, without the need for complex debiasing methods employed in unbiased learning to rank literature. This work establishes a direct connection between reinforcement learning and unbiased learning to rank through a concise MDP model. Specifically, we propose a novel off-policy learning-to-rank algorithm, CUOLR, which simultaneously learns efficient state representations and the optimal policy. Through empirical evaluation, we show that CUOLR achieves robust performance across a wide range of click models, consistently surpassing existing off-policy learning-to-rank methods tailored to those specific models. These compelling observations indicate that the extensive research conducted on offline

reinforcement learning can be leveraged for learning to rank with biased user feedback, opening up a promising new area for exploration.

Acknowledgments and Disclosure of Funding

This work was supported in part by Google Cloud Research Credits Program. Mengdi Wang acknowledges the support by NSF grants DMS-1953686, IIS-2107304, CMMI-1653435, ONR grant 1006977, and C3.AI.

References

Alekh Agarwal, Nan Jiang, Sham M Kakade, and Wen Sun. Reinforcement learning: Theory and algorithms. CS Dept., UW Seattle, Seattle, WA, USA, Tech. Rep , pages 10-4, 2019.

Aman Agarwal, Kenta Takatsu, Ivan Zaitsev, and Thorsten Joachims. A general framework for counterfactual learning-to-rank. In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval , pages 5-14, 2019.

Aman Agarwal, Xuanhui Wang, Cheng Li, Michael Bendersky, and Marc Najork. Addressing trust bias for unbiased learning-to-rank. In The World Wide Web Conference , pages 4-14, 2019.

Qingyao Ai, Keping Bi, Cheng Luo, Jiafeng Guo, and W Bruce Croft. Unbiased learning to rank with unbiased propensity estimation. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval , pages 385-394, 2018.

Jacob Buckman, Carles Gelada, and Marc G Bellemare. The importance of pessimism in fixed-dataset policy optimization. arXiv preprint arXiv:2009.06799 , 2020.

Chris J.C. Burges. From ranknet to lambdarank to lambdamart: An overview. Technical Report MSR-TR-2010-82, June 2010. URL https://www.microsoft.com/en-us/research/ publication/from-ranknet-to-lambdarank-to-lambdamart-an-overview/ .

Olivier Chapelle and Yi Chang. Yahoo! learning to rank challenge overview. In Proceedings of the learning to rank challenge , pages 1-24. PMLR, 2011.

Olivier Chapelle, Donald Metlzer, Ya Zhang, and Pierre Grinspan. Expected reciprocal rank for graded relevance. In Proceedings of the 18th ACM conference on Information and knowledge management , pages 621-630, 2009.

Jiawei Chen, Hande Dong, Xiang Wang, Fuli Feng, Meng Wang, and Xiangnan He. Bias and debias in recommender system: A survey and future directions. arXiv preprint arXiv:2010.03240 , 2020.

Aleksandr Chuklin, Ilya Markov, and Maarten de Rijke. Click Models for Web Search . Morgan &Claypool, 2015. ISBN 9781627056489. doi: 10.2200/S00654ED1V01Y201507ICR043.

Matej Cief, Branislav Kveton, and Michal Kompan. Pessimistic off-policy optimization for learning to rank. arXiv preprint arXiv:2206.02593 , 2022.

Nick Craswell, Onno Zoeter, Michael Taylor, and Bill Ramsey. An experimental comparison of click position-bias models. In Proceedings of the 2008 international conference on web search and data mining , pages 87-94, 2008.

Georges E Dupret and Benjamin Piwowarski. A user browsing model to predict search engine click data from past observations. In Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval , pages 331-338, 2008.

Damien Ernst, Pierre Geurts, and Louis Wehenkel. Tree-based batch mode reinforcement learning. Journal of Machine Learning Research , 6, 2005.

Dylan J Foster, Akshay Krishnamurthy, David Simchi-Levi, and Yunzong Xu. Offline reinforcement learning: fundamental barriers for value function approximation. arXiv preprint arXiv:2111.10919 , 2021.

Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International conference on machine learning , pages 2052-2062. PMLR, 2019.

Fan Guo, Chao Liu, Anitha Kannan, Tom Minka, Michael Taylor, Yi-Min Wang, and Christos Faloutsos. Click chain model in web search. In Proceedings of the 18th international conference on World wide web , pages 11-20, 2009.

Fan Guo, Chao Liu, and Yi Min Wang. Efficient multiple-click models in web search. In Proceedings of the second acm international conference on web search and data mining , pages 124-131, 2009.

Tuomas Haarnoja, Aurick Zhou, Pieter Abeel, and Sergey Levine. Soft actor-critic: Offpolicy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning , pages 1861-1870. PMLR, 2018.

Kalervo Järvelin and Jaana Kekäläinen. Cumulated gain-based evaluation of ir techniques. ACM Transactions on Information Systems (TOIS) , 20(4):422-446, 2002.

Nan Jiang and Lihong Li. Doubly robust off-policy value evaluation for reinforcement learning. In International Conference on Machine Learning , pages 652-661. PMLR, 2016.

Ying Jin, Zhuoran Yang, and Zhaoran Wang. Is pessimism provably efficient for offline RL? In International Conference on Machine Learning , pages 5084-5096. PMLR, 2021.

Thorsten Joachims. Optimizing search engines using clickthrough data. In Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining , pages 133-142, 2002.

Thorsten Joachims, Laura Granka, Bing Pan, Helene Hembrooke, and Geri Gay. Accurately interpreting clickthrough data as implicit feedback. In Proceedings of the 28th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval , SIGIR '05, page 154-161, New York, NY, USA, 2005. Association for Computing Machinery. ISBN 1595930345. doi: 10.1145/1076034.1076063. URL https: //doi.org/10.1145/1076034.1076063 .

Thorsten Joachims, Laura Granka, Bing Pan, Helene Hembrooke, and Geri Gay. Accurately interpreting clickthrough data as implicit feedback. In Acm Sigir Forum , volume 51, pages 4-11. Acm New York, NY, USA, 2017.

Thorsten Joachims, Adith Swaminathan, and Tobias Schnabel. Unbiased learning-to-rank with biased feedback. In Proceedings of the tenth ACM international conference on web search and data mining , pages 781-789, 2017.

Jae-kwang Kim and David Haziza. Doubly robust inference with missing data in survey sampling. Statistica Sinica , 24, 01 2014. doi: 10.5705/ss.2012.005.

Haruka Kiyohara, Yuta Saito, Tatsuya Matsuhiro, Yusuke Narita, Nobuyuki Shimizu, and Yasuo Yamamoto. Doubly robust off-policy evaluation for ranking policies under the cascade behavior model. In Proceedings of the Fifteenth ACM International Conference on Web Search and Data Mining , pages 487-497, 2022.

Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems , 33: 1179-1191, 2020.

Tor Lattimore, Branislav Kveton, Shuai Li, and Csaba Szepesvari. Toprank: A practical algorithm for online stochastic ranking. In NeurIPS , 2018.

Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643 , 2020.

Gene Li, Cong Ma, and Nathan Srebro. Pessimism for offline linear contextual bandits using ℓ p confidence sets. arXiv preprint arXiv:2205.10671 , 2022.

Tie-Yan Liu et al. Learning to rank for information retrieval. Foundations and Trends® in Information Retrieval , 3(3):225-331, 2009.

Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602 , 2013.

Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature , 518(7540):529-533, 2015.

Maeve O'Brien and Mark Keane. Modeling result-list searching in the world wide web: The role of relevance topologies and trust bias. 01 2006.

Harrie Oosterhuis. Doubly robust estimation for correcting position bias in click feedback for unbiased learning to rank. ACM Transactions on Information Systems , 41(3):1-33, 2023.

Tao Qin and Tie-Yan Liu. Introducing LETOR 4.0 datasets. CoRR , abs/1306.2597, 2013. URL http://arxiv.org/abs/1306.2597 .

Paria Rashidinejad, Banghua Zhu, Cong Ma, Jiantao Jiao, and Stuart Russell. Bridging offline reinforcement learning and imitation learning: A tale of pessimism. Advances in Neural Information Processing Systems , 34:11702-11716, 2021.

Yuta Saito. Doubly robust estimator for ranking metrics with post-click conversions. In Proceedings of the 14th ACM Conference on Recommender Systems , pages 92-100, 2020.

Mark Sanderson et al. Test collection based evaluation of information retrieval systems. Foundations and Trends® in Information Retrieval , 4(4):247-375, 2010.

John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning , pages 1889-1897. PMLR, 2015.

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 , 2017.

Laixi Shi, Gen Li, Yuting Wei, Yuxin Chen, and Yuejie Chi. Pessimistic q-learning for offline reinforcement learning: Towards optimal sample complexity. In International Conference on Machine Learning , pages 19967-20025. PMLR, 2022.

Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction . MIT press, 2018.

Adith Swaminathan and Thorsten Joachims. Counterfactual risk minimization: Learning from logged bandit feedback. In International Conference on Machine Learning , pages 814-823. PMLR, 2015.

Anh Tran, Tao Yang, and Qingyao Ai. Ultra: An unbiased learning to rank algorithm toolbox. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management , pages 4613-4622, 2021.

Ali Vardasbi, Maarten de Rijke, and Ilya Markov. Cascade model-based propensity estimation for counterfactual learning to rank. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval , pages 2089-2092, 2020.

Ali Vardasbi, Harrie Oosterhuis, and Maarten de Rijke. When inverse propensity scoring does not work: Affine corrections for unbiased learning to rank. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management , pages 1475-1484, 2020.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017.

Nan Wang, Zhen Qin, Xuanhui Wang, and Hongning Wang. Non-clicks mean irrelevant? propensity ratio scoring as a correction. In Proceedings of the 14th ACM International Conference on Web Search and Data Mining , pages 481-489, 2021.

Xuanhui Wang, Michael Bendersky, Donald Metzler, and Marc Najork. Learning to rank with selection bias in personal search. In Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval , pages 115-124, 2016.

Xuanhui Wang, Nadav Golbandi, Michael Bendersky, Donald Metzler, and Marc Najork. Position bias estimation for unbiased learning to rank in personal search. In Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining , pages 610-618, 2018.

Zeng Wei, Jun Xu, Yanyan Lan, Jiafeng Guo, and Xueqi Cheng. Reinforcement learning to rank with markov decision process. In Proceedings of the 40th international ACM SIGIR conference on research and development in information retrieval , pages 945-948, 2017.

Tengyang Xie, Ching-An Cheng, Nan Jiang, Paul Mineiro, and Alekh Agarwal. Bellmanconsistent pessimism for offline reinforcement learning. Advances in Neural Information Processing Systems , 34:6683-6694, 2021.

Tengyang Xie, Nan Jiang, Huan Wang, Caiming Xiong, and Yu Bai. Policy finetuning: Bridging sample-efficient offline and online reinforcement learning. Advances in Neural Information Processing Systems , 34:27395-27407, 2021.

Jun Xu, Zeng Wei, Long Xia, Yanyan Lan, Dawei Yin, Xueqi Cheng, and Ji-Rong Wen. Reinforcement learning to rank with pairwise policy gradient. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval , pages 509-518, 2020.

Tengyu Xu and Yingbin Liang. Provably efficient offline reinforcement learning with trajectorywise reward. arXiv preprint arXiv:2206.06426 , 2022.

Hang Yan, Bocao Deng, Xiaonan Li, and Xipeng Qiu. Tener: adapting transformer encoder for named entity recognition. arXiv preprint arXiv:1911.04474 , 2019.

Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Y Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. Advances in Neural Information Processing Systems , 33:14129-14142, 2020.

Andrea Zanette, Martin J Wainwright, and Emma Brunskill. Provable benefits of actor-critic methods for offline reinforcement learning. Advances in Neural Information Processing Systems , 34:13626-13640, 2021.

Shengyao Zhuang, Zhihao Qiao, and Guido Zuccon. Reinforcement online learning to rank with unbiased reward shaping. arXiv preprint arXiv:2201.01534 , 2022.

Masrour Zoghi, Tomas Tunys, Mohammad Ghavamzadeh, Branislav Kveton, Csaba Szepesvari, and Zheng Wen. Online learning to rank in stochastic click models. In International Conference on Machine Learning , pages 4199-4208, 2017.

A Experiment Details

A.1 Dataset Statistics

We conducted experiments on MSLR-WEB10K 4 and Yahoo! LETOR (set 1) 5 with semi-synthetic generated click data. Yahoo! LETOR comes from the Learn to Rank Challenge. It consists of 29,921 queries and 710K documents. Each query-document pair is represented by a 700-dimensional feature and annotated with a 5-level relevance label ranging from 0 to 4. MSLR-WEB10K dataset contains 10,000 queries and 125 retrieved documents on average. Each query-document pair is represented by a 136-dimensional feature vector and a 5-level relevance label. The dataset is partitioned into five parts with about the same number of queries, denoted as S1, S2, S3, S4, and S5, for five-fold cross-validation. All statistics of the used datasets are summarized in Table 3.

A.2 Implementation Details

Before introducing the hyperparameters required for each algorithm, we first describe some global hyperparameters that are used commonly across all algorithms. We use batch size B = 256 queries per epoch, and use nDCG@10 as the training objective for all baselines. We use Adam optimizer to train all the networks.

Dual Learning Algorithm (DLA) [4]. For DLA, two sub-models are being implemented: the ranking (scoring) model which is used to score each document; and the propensity model which is used to estimate the propensity for each document in the rank list. We use the MLP with two hidden layers of 256 units for both of them, and other hyperparameters are shown in Table 5.

Inverse Propensity Weighting (IPW) [52, 26] & Cascade Model-based IPW (CM-IPW) [48]. For IPW and CM-IPW, we need to get the propensities from Result Randomization . In total, there are 10M random rank lists with different searching queries shown to the user (click model), and the parameters of each click model are estimated by Maximize Likelihood Estimation (MLE) . Estimation details are shown in Table 4, where C k and C <k indicate the click at rank k and before rank k respectively. The superscript · ( s ) denotes the click of some click session s . Besides, we implement the ranking model the same way as that for DLA. Other hyperparameters are shown in Table 5.

ORACLE. For ORACLE, we train a LambdaMART ranker Burges [6] with true labels on the training dataset and evaluate its performance on the test set. We leverage the RankLib 6 learning to rank library, and set the hyperparameters shown in Table 5. This utilizes the full-information data and serves as an upper bound of the performance for all algorithms utilizing the partial-information data, such as the generated clicks.

CUOLR. For our algorithm CUOLR, there are three sets of hyperparameters used by the following sub-models: parameters for state embedding model , RL policy , as well as RL critic , where we use a 256-256 MLP to implement all the networks. Detailed hyperparameters are shown in Table 5.

B Additional Results

In this section, we present the complete results for all five click models (PBM, CASCADE, UBM, DCM, and CCM) on the two datasets: Yahoo! LETOR set 1 and MSLR-WEB10K. For the first two

studies in Section B.1 and B.2, we run 5 runs with different random seeds for the Yahoo! dataset. For the MSLR-WEB10K dataset which naturally comes with 5 folds, we take 1 run for each fold and aggregate the results. In the ablation experiment of conservatism for the offline RL algorithm in Section B.3, we only run 3 runs on Yahoo! due to the time limit. In all of our experiments, we use nDCG and ERR at positions 3,5,10 as evaluation metrics.

B.1 Performance across Different Click Models

In this section, we present a comprehensive comparison between our proposed method, CUOLR, and various baseline approaches. Specifically, we examine the efficacy of DLA and IPW, which have been specifically designed for position-based models, as well as CM-IPS, which has been tailored for cascade-based models. The comparative results are presented in Table 7. In the case of position-based models such as PBM and UBM, it is evident that IPW demonstrates the most superior performance among all the considered baselines. Conversely, when evaluating cascade models such as cascade and DCM, the utilization of CM-IPW yields improved performance, as it takes into account the propensity estimation considering prior examinations and clicks. Among the diverse click models examined, our unified algorithm, CUOLR, consistently achieves the highest level of performance in terms of the ERR metrics across different positions. Furthermore, it consistently outperforms the other models in the majority of cases in terms of nDCG@10. This provides empirical verification of the effectiveness of our unified framework and the robustness of the CUOLR algorithm.

B.2 State Representation Ablation Experiments

In this section, we present an ablation study focusing on the state embedding utilized in our algorithm, CUOLR. We compare the effectiveness of our proposed multi-head self-attention, augmented with positional embedding, against several heuristic hard-coded baselines for state embedding. These

baselines include utilizing only positional information (POS), concatenating previous document information (PREDOC), and a combination of positional and document information (POS+PREDOC). The evaluation is performed on the Yahoo dataset, and the results are summarized in Table 8. Consistent with expectations, for position-based models (PBM), the most effective approach is utilizing only the positional information. Conversely, for cascade models (CASCADE), only considering the previous document information gives the best performance. In the case of more complicated models, such as DCM, CCM, and UBM, where the click model relies on both position and previous examinations, it becomes evident that incorporating a combination of positional information and previous document information yields the highest performance. Among all of them, it is worth to point out that our proposed state representation learning consistently attains comparable performance to the best baseline. Notably, our method possesses the advantage of automatically learning the optimal state representation, irrespective of the underlying assumptions of the click models.

B.3 Conservatism for Offline RL Ablation Experiments

In this section, we conduct an ablation study to investigate the influence of the hyperparameter α , which governs the conservatism, in the CQL algorithm. Specifically, we examine its effects on the Yahoo! dataset by varying α across a range of values: { 0 , 1 e -3 , 5 e -3 , 1 e -2 , 5 e -2 , 1 e -1 , 5 e -1 , 1 e 0 , 5 e 0 , 1 e 1 , 5 e 1 } . It is noteworthy that when α is set to 0, the CQL algorithm simplifies to the SAC algorithm [19]. Remarkably, we find that the performance of CUOLR remains consistently robust across the diverse α values, as long as they are not being restricted to be too close to the logging policy (i.e., large α values). This consistency underscores the effectiveness of our method, which demonstrates its ability to adapt to different underlying reinforcement learning (RL) algorithms. Interestingly, in contrast to classical offline RL datasets studied in Kumar et al. [29], where the conservatism parameter plays a substantial role, we observe that its impact is comparatively minor

in the offline learning to rank dataset. This observation is worth further investigation for better understanding the impact of conservatism in offline LTR settings.

Besides, To figure out the effect of conservatism, we do experiments on several click models on Yahoo! dataset to compare the performance between simple SAC and CQL with the optimal conservative parameter α . The optimal α is selected through grid search ranging from { 1 e -3 , 5 e -3 , 1 e -2 , 5 e -2 , 1 e -1 , 5 e -1 , 1 e 0 , 5 e 0 , 1 e 1 , 5 e 1 } . The results are shown in Table 9. It's obvious that the performance of CQL with optimal α is consistently better than that of CQL, especially in the NDCG metric. This improvement further illustrates the necessity of conservatism in Off-policy learning to rank task.

B.4 Data Quality for Offline Reinforcement Learning to Rank

In this section, we conduct an ablation study to clarify the effect of data quality to the offline RL algorithms. In usual cases, data quality is important to most offline algorithms, including the offline RL algorithm. To verify the effect of data quality, we conduct experiments on the Web10k dataset with different offline data. Specifically, on each click model, the logging policy is trained with different portions of train data: 1) SVMRank trained with 1% train data; 2) SVMRank trained with 0.01% train data; 3) random policy without pre-training. Then the quality of the train data gathered by the three logging policy decreases, and the performance of CQL on them is displayed in Table 10. We find that the algorithm trained with data in better quality (first line for each click model) performs significantly better than the other two. Besides, it's interesting that the algorithm trained with random logging data can't even beat the logging policy trained with 1% train data, which further highlight the significance of data quality.

...

𝑐

!

𝑒

!

𝑑

!

𝑘

!

𝑐

"

𝑒

"

𝑑

"

𝑘

"

𝑐

#

𝑒

#

𝑑

#

𝑘

#

CASCADE

𝑐

!

𝑒

!

𝑑

!

𝑐

"

𝑒

"

𝑑

"

𝑐

#

𝑒

#

𝑑

#

...

PBM

𝑘

!

𝑘

"

𝑘

#

...

𝑐

!

𝑒

!

𝑑

!

𝑐

"

𝑒

"

𝑑

"

𝑐

#

𝑒

#

𝑑

#

DCM

𝑘

!

𝑘

"

𝑘

#

...

𝑐

!

𝑒

!

𝑑

!

𝑐

"

𝑒

"

𝑑

"

𝑐

#

𝑒

#

𝑑

#

CCM

𝑘

!

𝑘

"

𝑘

#

𝑟

!

𝑠

!

𝑎

!

𝑟

"

𝑠

"

𝑎

"

𝑟

#

𝑠

#

𝑎

#

...

MDP

𝑘

!

𝑘

"

𝑘

#

𝑠

$

: state

𝑎

$

: action

𝑟

$

: reward

𝑒

$

: examination

𝑑

$

: document

𝑐

$

: click

𝑘

$

: rank/position

C Click Model and MDP Formulation

In this section, we present a comprehensive overview of different click models employed in the paper, namely PBM, CASCADE, DCM, CCM, and UBM. Additionally, we demonstrate the graphical models associated with each click model and how they could be unified into the Markov Decision Process (MDP) framework, as depicted in Figure 1.

PBM [24]. The position-based model is a model where the probability of clicking on an item depends on both its identity and rank. The examination probability is rank-dependent only, i.e.,

CASCADE [12]. The cascade model assumes that the user scans a rank list R from top to bottom. If a document at rank k is examined and clicked, the user stops browsing the remaining documents. Otherwise, the user goes on to the next rank k + 1 with probability one. The first document d 1 is always examined. The document at rank k will be examined if and only if the previous k -1 documents are not clicked. Therefore we have:

DCM [48, 18]. The dependent click model assumes that the user examines the results from top to bottom until an attractive result is found, P ( E k +1 = 1 | E k = 1 , C k = 0) = 1 , where E k is the examination indicator at rank k . After each click, there is a rank-dependent chance that the user is unsatisfied, P ( E k +1 = 1 | C k = 1) = λ k . Therefore, we have:

CCM[17]. The click chain model (CCM) is a generalization of the dependent click model where continuing to examine the results before a click is not deterministic, i.e. P ( E j +1 = 1 | E j =

Table 9: Comparison of SAC and CQL with optimal alpha on Yahoo. To show the impact of the offline algorithm in off-policy learning to rank, we compare SAC and CQL with different degrees of conservatism. We control the degree of conservatism via the parameter α (See details in Eq.5 in the main paper). The α s range from { 1 e -3 , 5 e -3 , 1 e -2 , 5 e -2 , 1 e -1 , 5 e -1 , 1 e 0 , 5 e 0 , 1 e 1 , 5 e 1 } and we do grid search to find the optimal α .

1 , C j = 0) = α 1 . The probability of continuing after a click is not position dependent, but relevance dependent, P ( E j +1 | C j = 1) = α 2 (1 -R i ) + α 3 R i , where R i is the relevance of the i th document in the rank list. Therefore, the examination probability at each position can be written as:

UBM[13]. The user browsing model (UBM) is an extension of the PBM model with some elements of the cascade model. The whole model is position-based, but for the examination probability, it considers previous clicks. Specifically, the examination probability depends not only on the rank of the document k , but also on the rank of the previously clicked document k ′ , which is modeled by a set of parameters γ kk ′ , i.e. P ( E k = 1 | C 1 = c 1 , . . . , C k -1 = c k -1 ) = γ kk ′ , where k ′ is the rank of the previous clicked document or 0 if none of them was clicked, i.e. k ′ = max { r ∈ { 0 , . . . , k -1 }} :

c r = 1

Zoghi et al. [63] showed that PBM and CM satisfied Assumption 3.1 where optimal ranking list leads to optimal total clicks. In our future work, it would be interesting to show that other click models also satisfy this assumption.

pbm

cascade

dcm

ccm

0.438

0.436

0.434

0.432

0.430

0.428

0.426

0.424

0.422

0.460

0.458

0.456

0.454

0.452

0.450

0.448

0.446

0.476

0.474

0.472

0.470

0.468

0.466

0.464

0.462

0.690

0.685

0.680

0.675

0.670

0.7150

0.7125

0.7100

0.7075

0.7050

0.7025

0.7000

0.6975

0.766

0.764

0.762

0.760

0.758

0.756

0.754

0.752

0.750

10-3

10-2

10-1

100

101

10-3

10-2

10-1

100

101

10-3

10-210-1

100

101

10-3

10-2

10-1

100

101

err_3

err_5

err_10

ndcg_3

ndcg_5

ndcg_10

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper aims to unify the assumptions on user behavior in a ranking called click model by formulating the ranking problem as a click-model agnostic Markov Decision Process (MDP). By doing so, the paper proposes to reduce the Off-Policy Learning to Rank (LTR) problem to a variant of offline RL, which does not require precise estimation of the click models. Under this formulation, the proposed offline RL method, which incorporates state representation learning to the well-known Conservative Q-Learning (CQL), enables more accurate estimations of ranking metrics than baseline estimators in real-world datasets.\n\nSTRENGTHS:\n1. The manuscript is easy to follow. The motivation for unifying click models and introducing the offline RL framework is clearly explained.\n\n2. The RL formulation of LTR is reasonable, and showing the connection between offline RL and Off-Policy LTR would be insightful for the LTR community.\n\n3. The ablation study on state representation learning is insightful.\n\nWEAKNESSES:\n1. While the connection between LTR and offline RL is interesting, the proposed method (CUOLR) itself is not a fundamentally new framework for offline RL. In particular, the proposed state representation learning method, which applies positional encoding and attention to the inputs, seems to be an engineering effort rather than a very novel framework.\n\n2. In experiments, an offline RL algorithm (CQL) does not show advantages over a simple RL baseline (SAC). While I acknowledge the author(s)’ contribution to formulating LTR as an RL problem, I think the algorithm has room for improvement.\n\nQUESTIONS:\n- In general, SAC does not perform very well in the offline RL setting and thus the offline RL algorithms, including CQL, have been proposed. However, the experiment results show that CUOLR (CQL) and CUOLR (SAC) perform competitively. Could you provide some justification for this result?\n\nLIMITATIONS:\nThis limitation is not special for this paper, but classical click models and MDP both assume that the reward observed at each position are not affected by lower positions including the neighboring ones. If this assumption does not hold, LTR may introduce some bias and offline RL may also ignore some causal relation between actions and rewards.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a unified approach for off-policy learning to rank (LTR) that is adaptable to general click models. The authors formulate off-policy LTR as a Markov Decision Process (MDP) and leverage offline reinforcement learning (RL) techniques to optimize the ranking process. They propose the Click Model-Agnostic Unified Off-policy Learning to Rank (CUOLR) method, which can be easily applied to a wide range of click models. The authors provide empirical evidence demonstrating the effectiveness of CUOLR in comparison to state-of-the-art off-policy LTR algorithms.\n\nSTRENGTHS:\n● The paper presents an innovative and practical methodology for off-policy learning to rank. The formulation of off-policy LTR as an MDP and the use of offline RL techniques provide a comprehensive and adaptable approach for ranking optimization.\n● The authors provide insightful empirical findings, showing that the CUOLR method consistently outperforms state-of-the-art off-policy learning to rank algorithms. The results on various large-scale datasets demonstrate the effectiveness, consistency, and robustness of CUOLR under different click models.\n● The paper presents a detailed introduction on state representation learning. There are also empirical analyses on its practical effect.\n\nWEAKNESSES:\n● The synthetic dataset and the offline evaluation can give biased evaluation results of the algorithms.\n● There are some related works that are not mentioned in this paper. Cai, et al. [1] also apply RL techniques to recommendation systems and provide a new MDP formulation. It also includes ranking scores in the formulation that can be regarded as a downstream application to this paper. Xue, et al. [2] provides another MDP formulation of RL for optimizing long-term user engagement.\n● There remain some issues unsolved in the paper. See the questions for details.\n\n[1] Cai, Qingpeng, et al. \"Reinforcing User Retention in a Billion Scale Short Video Recommender\nSystem.\" arXiv preprint arXiv:2302.01724 (2023).\n[2] Xue, Wanqi, et al. \"PrefRec: Preference-based Recommender Systems for Reinforcing Long-term User Engagement.\" arXiv preprint arXiv:2212.02779 (2022).\n\nQUESTIONS:\n● Lines130-133 are hard to understand. Why not include remaining documents that are yet to rank? How will the policy perform on $s_0$, where nothing will be included as input?\n● The introduction of dynamic action space can introduce dynamics shift in RL and unstable training. Are there any specific techniques to handle this issue?\n● Why does SAC obtain similar performance with CQL when learning in static datasets? Will the issue of policy distribution mismatch lead to poor performance?\n\nLIMITATIONS:\nThe formulation and evaluation are limited to off-policy/offline RL that learn from a static dataset. In applications where there are adequate online interaction data, it will be helpful to consider some online RL counterparts.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper talks about how to model user behaviors with positional biases in an online search system. The paper proposes a unified RL framework to generalize three common types of positional bias models: Position-Based Modeling (PBM), CASCADE (each click depends on the previous click), and Dependent Click Models (DCM). By capturing all previous clicks into a state, the paper suggests that standard offline RL algorithms, such as Conservative Q-Learning (CQL) and Soft Actor Critic (SAC), can be used to estimate and optimize for the positional-aware ranking policies. Simulation experiments are included to support the claims.\n\nSTRENGTHS:\nOriginality. The paper observes commonalities in three different types of methods and proposes a unified method to combine them. The generality of the proposal is further examined by simulating different cases from each of the method and showing similar performance using the proposed method. The observation and the empirical validation feel original to me.\n\nThe authors addressed my concerns and I have adjusted my scores accordingly.\n\nWEAKNESSES:\nSignificance. The paper made a rather limiting assumption that all users in the same search system follow a single pattern of positional bias. This may not be true, as different users may have different positional biases. A more practical approach is to insert perturbations to the ranking positions to estimate the true positional effects using IPS. To mitigate the risks of perturbations, methods have been introduced to swap only adjacent search results. Please see this paper as an example: https://www.cs.cornell.edu/people/tj/publications/agarwal_etal_19a.pdf\n\nClarity. The description of the proposed method lacks sufficient details (see additional questions). Also the experimental results (Table 1) did not seem to contain significant differences between the algorithms being considered. This leads me to wonder if the proposed RL solution is actually easy to find, or is it fundamentally difficult to implement and the results may be sensitive to hyperparameter choices. The paper did not discuss common limitations of RL algorithms.\n\nQUESTIONS:\nTable 1. The table compares the proposed method with baseline methods, one of which is used to simulate the ground-truth user behaviors. However, in the results, the baseline method which was used to simulate the user behaviors did not perform the best in estimating the user behaviors. Is this expected? Also, the differences between all methods seem pretty small.\n\nAlgorithm 1. Line 6, where is psi defined?\n\nEq (5). What is the difference between CQL and SAC? Why is there only one equation being presented for both algorithms? More details would be appreciated for clarity purposes.\n\nLIMITATIONS:\nNo, the paper did not discuss the choice of hyperparameters or the closeness of the experimental results. They seem to be common challenges with RL algorithms.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\n- Authors propose a unified off-policy reinforcement learning-based approach for learning to rank (LTR) problem that is adaptable to all general click models.\n- Authors argue that a user's behavior in terms of clicks can be modeled by a Markov Decision Process (MDP), thereby allowing them to look at LTR from the perspective of offline reinforcement learning. \n- Authors' key contribution is a novel state representation that takes into account features of all the items/documents presented to a user. Specifically, they represent state at position $k$ by $s_k = [(d_1, d_2, \\cdots, d_{k-1}), k]$ to capture the context of user behavior prior to reaching position $k$-ranked position. The action set at position $k$ is all the available items/documents that haven't been presented to the user in prior $k-1$ positions, and the reward is simply the users' click behavior when presented with action $a_k$. \n- Authors use positional encoding and self-attention layer to extract the state representation given the sequence of items/documents at prior ranked positions, i.e., $(d_1, d_2, \\cdots, d_{k-1})$. They train an end-to-end model that jointly optimizes for state representation as well as the policy.\n\nSTRENGTHS:\n**Motivation**\n- Authors present a strong motivation for their approach based upon the need for a unified a click model-agnostic LTR method which generalizes well to use of any RL algorithm. \n- Authors do a good job at placing their work in context of related work in the field by comparing/contrasting their approach with other works. \n_Note_: I am not updated on the state-of-the-art RL approaches for LTR, and therefore may not be aware of some recent works. \n\n**Technical Presentation**\n- Authors do a good job at formalizing the problem and providing all the relevant technical information in Sections 3 and 4 that would be required for reproducibility.\n- Authors contribution is quite easy and intuitive to understand.\n\n**Experiments**\n- Authors evaluate their approach (using CQL and SAC) in comparison to baselines such as Dual Learning Algorithm (DLA), Inverse Propensity Weighting (IPW) algorithm and Cascade-model based IPW, as well as LambdaMart model which serves an upper bound. The metrics of authors' proposed approach are significantly favorable in comparison to baselines. _Note_: Just as above, because I am unaware of the SOTA, I am unable to comment on whether the right baselines have been used for comparison.\n- Authors provide ablation study to validate the need for each component of their proposed approach.\n\nWEAKNESSES:\n**Is this a Markov Decision Process?**\n\nFormally, the Markovian property of MDP refers to the fact that given states $s_i$ and actions $a_i$, the equality $\\mathrm{Pr}(s_t | s_0, a_0, s_1, a_1, \\cdots, s_{t-1}, a_{t-1}) = \\mathrm{Pr}(s_t | s_{t-1}, a_{t-1})$ holds. Intuitively, it means that the probability of landing in state $s_t$ of the Markov chain only depends on the last state-action pair $(s_{t-1}, a_{t-1})$, and not the state-action pairs preceding that. \n\nGiven the authors' approach uses state representation as $s_k = [(d_1, d_2, \\cdots, d_{k-1}), k]$, it is unclear to me how/if the Markovian property still hold?\n\n**Effect of Click Data Generation**\n\nAuthors mention that they use 1% of training data to train a Ranking SVM to generate initial ranked list of items, upon which clicks are simulated using different click models. How does the usage of more/less portion of training data affect the comparative baselines and in turn the relative performance improvement provided by the proposed approach over those baselines?\n\nQUESTIONS:\n- Could the authors address the questions raised in the weaknesses section of the review?\n\nLIMITATIONS:\n- Authors have not explicitly addressed limitations of their approach or any potential negative societal impact in the manuscript. However, authors' proposed approach is not a significant departure from the prevalent large scale recommender systems.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this work, the authors model the process of learning optimal ordering of ranked lists, called Learning to Rank (LTR) as a Markov decision process (MDP). This allows them to utilize techniques from reinforcement learning to solve for the policy that generates the optimal ordering. In practical applications of LTR, a pre-collected dataset (offline) is typically used which is collected using a logging policy that is not optimal (off-policy). As a result, techniques for offline off-policy reinforcement learning are used in this work.\nIn the MDP formulation, the reward is defined as an item of the ranked list being clicked. The distribution of clicks is referred to as a click model, which results in various reward functions. An RL algorithm can optimize a wide range of reward functions and allows this approach to be largely click-model agnostic. \nThe approach is empirically validated on semi-synthetically generated click datasets, and it demonstrates competitive performance across different click models compared to click-model-specific baselines for learning optimal rankings.\n\nSTRENGTHS:\nThe paper succinctly models LTR as a sequential decision-making problem, building on top of prior work,  in a manner that allows the use of transformer architecture to be employed in conjunction with an RL algorithm. This allows for a couple of benefits: 1) being agnostic of specific click models since any click distribution in the offline dataset can in theory be encoded as the reward that the RL algorithm can optimize for, and 2) allowing for flexibility in defining the state representation through the use of attention and position embeddings. \nBoth of the claims are supported by strong empirical analysis.\n\nWEAKNESSES:\nThe paper extended prior formulations of LTR as an MDP to allow for using the transformers architecture, with the aim performing policy learning without click model specific methods. In addition to the existing experiments, one experiment that would highlight the point would be one that learns the optimal ranking for data generated from a non-standard click model – say a randomly picked click distribution – where all other methods fail.\n\nAdditionally, the utility of the section on the optimality of rankings (Definition 3; Assumption 3.1) is unclear. These are possibly included to tie this work to existing literature, however, this method should not require those conditions for finding an optimal policy. \nOn a related note, L122-L123 incorrectly states that Definition 1 is explained in Appendix C, which only discusses the definitions of examination probabilities of various click models. \n\nThe writing and presentation could be improved in places, a few examples being highlighted below. \nMinor/Typos:\n- “Debiasing” as caused by bias in the data, versus bias due to estimation procedure has been used interchangeably through the paper leading to a fair amount of confusion (Ex: L26-L29)\n- Equation (1) should have a $\\propto$ and not $=$\n- L121: mutually independent of what?\n- L227: $d\\\\_model$ undefined.\n- Algorithm 1: Mistake on Line 7\n\nQUESTIONS:\nA few questions:\n- Assumption 3.1 holds only for monotonically decreasing examination probabilities. The reference [64] proves the result in Assumption 3.1 only for PBM and cascade models, and is not the case for other click models. Why is this top-down scanning of the ranked list necessary for the working of the algorithm?\n- The robustness to change in $\\alpha$ in CQL seems to indicate that just fitting a Q function on the offline data should suffice. This would be true if the logging policy provides enough support for all actions, which is commonly the case in practice where uniform random logging policies are used. Why is a conservative algorithm (CQL) necessary for the experiments? \n- (L148-L149) How is the action set restricted to not repeat actions implementationally? \n- The gains in performance are more significant for ERR as compared to NDCG. Why is that the case?\n\nLIMITATIONS:\nLike any other method for LTR, the performance of the algorithm should depend on the coverage provided by the logging policy. A comment on the effect of the coverage of the logging policy and it's effects on the learnt rankings would be useful.\n\nState representations play an important role in the performance of the algorithm and in the context of this work the representations are learned via architectural choices in the Q-function estimator. The claim that (L248) that an additional component is added on top of CQL is overstated.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-23"}