SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Zeyu Zhang $^{1}$  Yi Su $^{2*}$  Hui Yuan $^{3}$  Yiran Wu $^{4}$  Rishab Balasubramanian $^{5}$  Qingyun Wu $^{4}$  Huazheng Wang $^{5*}$  Mengdi Wang $^{3}$
<sup>1</sup>University of Science and Technology of China <sup>2</sup>Google Deepmind <sup>3</sup>Princeton University
4Penn State University 5Oregon State University
zgkd2019zzzy@mail.ustc.edu.cn yisumtv@google.com
{huiyuan, mengdiw}@princeton.edu {ykw5399, qingyun.wu}@psu.edu
{balasuri, huazheng.wang}@oregonstate.edu
Off-policy Learning to Rank (LTR) aims to optimize a ranker from data collected by a deployed logging policy. However, existing off-policy learning to rank methods often make strong assumptions about how users generate the click data, i.e., the click model, and hence need to tailor their methods specifically under different click models. In this paper, we unified the ranking process under general stochastic click models as a Markov Decision Process (MDP), and the optimal ranking could be learned with offline reinforcement learning (RL) directly. Building upon this, we leverage offline RL techniques for off-policy LTR and propose the Click Model-Agnostic Unified Off-policy Learning to Rank (CUOLR) method, which could be easily applied to a wide range of click models. Through a dedicated formulation of the MDP, we show that offline RL algorithms can adapt to various click models without complex debiasing techniques and prior knowledge of the model. Results on various large-scale datasets demonstrate that CUOLR consistently outperforms the state-of-the-art off-policy learning to rank algorithms while maintaining consistency and robustness under different click models.
Learning to Rank (LTR) is a core problem in Information Retrieval (IR) with wide applications such as web search and recommender systems [33]. Traditional LTR methods require high-quality annotated relevance judgments for model training, which is expensive, time-consuming, and may not align with actual user preferences [41]. As a result, learning to rank with implicit user feedback, such as logged click data, has received a huge amount of attention in both academia and industry [23, 26, 53].
Despite its low cost, learning to rank directly from implicit user feedback could suffer from the intrinsic noise and bias in user interactions, e.g., position bias, where an item displayed at a higher position receives a higher click-through rate (CTR) than its relevance [12]. To mitigate the bias in the click data, off-policy learning to rank methods have been proposed under different bias assumptions such as position bias [53, 2], selection bias [52] and trust bias [36, 9]. A major branch of off-policy learning to rank called counterfactual learning to rank achieves unbiasedness by re-weighting the samples using the inverse propensity scoring (IPS) method [46, 2, 27]. To estimate the propensity from logged click data, existing works require explicit assumptions about how users examine the rank list and generate the click data, i.e., click models [12, 10]. For example, position-based model
*Correspondence to: Yi Su and Huazheng Wang
(PBM) [24] assumes the probability of examining a result only depends on the position; while cascade model (CASCADE) [12] assumes each click depends on the previous click, and dependent click model (DCM) [18] considers both. Different debiasing methods have been proposed to cater to specific click models, including PBM [4, 53, 26], CASCADE; and DCM [48]. However, prior knowledge of the click model is usually unknown and the correct click model needs to be identified from user behavior data before applying an off-policy algorithm, which is challenging in complex real-world environments. Besides, many popular and powerful click models have not been studied in counterfactual learning to rank such as the click chain model (CCM) [17] and the user browsing model (UBM) [13]. It requires a significant amount of work to study debiasing methods for every popular click model.
To overcome these issues, we propose to study a unified approach of off-policy learning to rank adaptable to general click models. Our key insight is that the user's examination and click behavior summarized by click models has a Markov structure; thus off-policy LTR under general click models can be formulated as a Markov Decision Process (MDP). Specifically, the learning to rank problem now can be viewed as an episodic RL problem [45, 1], where each time step corresponds to a ranking position, each action selects a document for the position, and the state captures the user's examination tendency. This formulation allows us to view off-policy LTR from the perspective of offline reinforcement learning [31], where we can leverage off-the-shelf offline RL algorithms [29, 19, 16] to optimize the ranking list. Importantly, our formulation bridges the area of off-policy learning to rank and offline RL, allowing for the integration of ideas and solutions from offline RL to enhance the solution of the off-policy LTR problem.
Inspired by the formulation, we propose the Click Model-Agnostic Unified Off-policy Learning to Rank (CUOLR) method. We first construct each logged query and ranking data as an episode of reinforcement learning following the MDP formulation. Our dedicated structure for state representation learning can efficiently capture the dependency information for examination and click generation, e.g. ranking position in PBM and previous documents in CM and DCM. The algorithm jointly learns state representation and optimizes the policy, where any off-the-shelf offline RL algorithm can be applied as a plug-in solver. Specifically, we adapt the popular CQL algorithm [29] as an instantiation, which applies the conservative (pessimism) principle to Q function estimate. We evaluate our algorithm on real-world learning to rank datasets [38, 7] under various click models. Compared with off-policy LTR methods that are dedicated to specific click models, our click model-agnostic method consistently outperforms the best-performing baselines in all click models.
- We formulate the off-policy LTR with biased feedback under general click model as a Markov Decision Process, and bridge the area of off-policy learning to rank and offline reinforcement learning.
- We propose CUORL, a Click model-agnostic Unified Off-policy LTR method that could utilize any offline RL algorithm as a plug-in solver, and we instantiate it using CQL.
- We conduct extensive empirical experiments to validate the effectiveness of our algorithm using real-world LTR datasets under different click models.
Off-policy Learning to Rank. Off-policy Learning to Rank aims to optimize the ranking function from logged click data [23]. The majority of the works aim to mitigate the bias in logged click data, known as counterfactual learning to rank or unbiased learning to rank. The debiasing methods mainly follow inverse propensity scoring strategy [26, 49, 4, 51, 48, 3], while there are also recent works applying doubly robust estimator to reduce variance [40, 28, 37]. Cief et al. [11] proposed pessimistic off-policy optimization for learning to rank that also mitigates bias but not in an unbiased way. All these methods rely on prior knowledge of the click model [12, 10], while our algorithm is agnostic to general click models.
Offline Reinforcement Learning. Offline RL algorithms [29, 14, 44, 21, 16, 15] learn policy from large logged datasets where the distributional shift between the logging policy and the learned policy imposes a major challenge. In this setting, different algorithms are proposed, from value-based ones
(e.g. [35, 34]) to policy-based ones (e.g. [43, 42]). Among the vast literature on offline reinforcement learning, the principle of pessimism/conservatism [29, 5, 22] is an important line and has inspired many algorithms from empirical and theoretical perspective [60, 39, 32, 56, 61, 55, 58]. While all the aforementioned methods can be plugged into our algorithm, we choose the classic CQL algorithm [29] with a conservative Q function on top of soft actor-critic algorithm [19].
Reinforcement Learning to Rank. Wei et al. [54] first model ranking problem as an MDP, where the state is the candidate document set at current rank and the action is the selected document. [57, 62] have been studied under similar MDP formulation. However, [54, 57] requires relevance labels as feedback and cannot mitigate bias in click data; [62] is an online learning algorithm that learns from user interactions instead of logged data. Compared to these studies, we characterize the MDP formulation from a different perspective, i.e., capture bias in the click model, and propose the offline RL algorithm with logged click feedback.
3 Reinforcement Learning to Rank: A Unified Formulation
As the majority of existing works in unbiased learning to rank focused on inferring documents' relevance from the click models, these methods are tied to specific click models adopted. In this section, we formulate learning to rank with general click feedback as a Markov decision process, offering a unified and comprehensive modeling of possibly complicated user behavior. This formulation unifies LTR problems associated with different click models, under which the underlying click model is translated into the environment setup of MDP such as state transitions and rewards. It opens up the possibility to employ a rich class of reinforcement learning algorithms for solving LTR, which we will give greater details in the next section.
Click model. A key challenge of off-policy LTR lies in learning the document's attractiveness/relevance from the implicit feedback that is biased by the user's examination behavior. To address this challenge, a range of click models have been proposed to accommodate user's various click behavior [10]. In this study, we focus on a general family of click models [63, 30], which is marked by two characteristics: (1) Most of the mainstream click models have a "two-step" flavor that breaks user's click behavior towards some document down into the user's examination and the document's relevance. For each document  $d$ , the user first decides whether to examine it in the ranking list, based on the specific behavior. Mathematically the user behavior is modeled as the examination probability, which generally depends on the ranking list  $\mathcal{R}$  and the position of the document  $k$ , denote as  $\chi(\mathcal{R}, k)$ . Once the document is examined, the user will choose whether to click it, based on the attractiveness  $\alpha(d)^2$ . (2) Any documents under the  $k$ -th position do not have an effect on  $\chi(\mathcal{R}, k)$ .
Definition 1 (Click Model). For any rank list  $\mathcal{R}$  and position  $k$ , the attractiveness and examination probability are independent.
$$
P \left(C _ {k} = 1 \mid \mathcal {R}, k\right) = \chi (\mathcal {R}, k) \alpha (\mathcal {R} (k)) \tag {1}
$$
where  $C_k$  is the click indicator at rank  $k$ , and  $\chi(\mathcal{R}, k)$  is the examination probability of position  $k$  in the rank list  $\mathcal{R}$ . For each document  $d$ , the attractiveness  $\alpha(d)$  only depends on the document itself. And the attractiveness is mutually independent.
We show that classic click models such as PBM, CASCADE, DCM, and CCM are instances of Definition 1, with details listed in Appendix C.
3.2 Learning to Rank as Markov Decision Process
In Reinforcement Learning (RL), the interactions between the agent and the environment are often described as an finite-horizon discounted Markov Decision Process  $M = (S, A, T, r, \gamma, H)$ . The goal of the RL problem is to find a policy  $\pi$  that maximizes the value function, i.e. the discounted
2We simplify the notation and assume  $d$  captures (query, document) pair information on given query.
cumulative reward
$$
\mathbb {E} \left[ \sum_ {t = 0} ^ {H} \gamma^ {t} r \left(s _ {t}, a _ {t}\right) \mid \pi , s _ {0} = s \right]. \tag {2}
$$
In what follows, we formulate each of the  $(S,A,T,r,\gamma ,H)$  components in the ranking scenario. Our formulation essentially differs from the existing MDP formulation of ranking [54], where the state at position  $k$  is defined as remaining documents that are yet to rank following the  $k - 1$  ranked ones on the top, instead Wei et al. [54] has a limited capturing of user's click behavior as a result of being ignorant of the ordering within the top  $k - 1$  documents. From here on, we use  $k\in [K]$  to denote the  $k$ -th position top down on a list of total length  $K$ . It serves as the counterpart of the time step  $t$  in (2) for the ranking setting.
State  $S$  For each position  $k\in [K]$ , state  $s_k$  should include and represent the current status of the ranking that the agent is faced with. Thus, we define the state at rank  $k$  as:
$$
s _ {k} = \left[ \left(d _ {1}, d _ {2}, \dots , d _ {k - 1}\right), k \right], \tag {3}
$$
which is a concatenation of the established sub-ranking list up to  $k$ , denoted by  $(d_{1}, d_{2}, \ldots, d_{k-1})$  and the position  $k$ . Here  $d_{i}$  refers to the document presented at rank  $i$  with  $s_{0}$  is initialized as  $= [(), 0]$ . Together with the action  $a_{k}$  as to select document  $d_{k}$  presenting at rank  $k$ , defining  $s_{k}$  as (3) fully captures the user's click behavior  $C_{k}$  at this point. Recall (1) that  $P(C_{k} = 1 \mid \mathcal{R}, k) = \chi(\mathcal{R}, k) \alpha(\mathcal{R}(k))$ , where  $\chi(\mathcal{R}, k)$  is determined by  $(d_{1}, d_{2}, \ldots, d_{k-1})$ . To better capture the rich information in the observed state, we discuss how to attain the effective state embedding from the raw representation in Section 4.
Action  $\mathcal{A}$  Action  $a_{k}$  is naturally defined as the document to present at rank  $k$  given state  $s_k$ . In our experiments, each action is represented by a feature vector associated with the query. It is worth mentioning that the available action set  $\mathcal{A}_k$  at each  $k$  is related to state  $s_k$  as well as the specific query, unlike the common case where the action space is fixed. There are two main differences here compared with the fixed action space: (1). the available actions vary under different queries; and (2). once an action is chosen, it is removed from the candidate set  $\mathcal{A}_k$ .
Transition  $\mathcal{T}(s'|s,a)$  Transition maps a state-action pair to the probability distribution over possible next states. Given our formulation of state and action, the next state  $s_{k + 1}$  is deterministic given  $s_k = [(d_1,d_2,\dots ,d_{k - 1}),k]$  and  $a_{k}$ . Formally,  $\mathcal{T}(s_{k + 1}|s_k,a_k) = 1$  if and only if  $s_{k + 1} = [(d_1,d_2,\dots ,d_{k - 1},a_k),k + 1]$ . Note that with this transition function, we can show that our model is a rigorous MDP, where the distribution of next state only based on the current state and the action.
Reward  $r(s, a)$  Aligned with the goal of LTR to maximize total clicks, we adopt the binary click as a reward at each position, i.e.  $r(s_k, a_k) = C_k$ . It is easily checked this is a well-defined reward from (1) that the distribution of  $r$  is fully determined by  $s$  and  $a$ , i.e.,  $\mathbb{E}[r(s_k, a_k)] = \chi(s_k) \alpha(a_k)$ .
Putting the components together we have formally built up our MDP formulation of LTR, which we name as "MDP for Ranking" and denote by  $\mathcal{MR}(\mathcal{S},\mathcal{A},\mathcal{T},r,\gamma ,H)$  with components defined as above. The rich RL literature has the convention to solve the optimal policy  $\pi^{*}$  that maximizes the cumulative reward, which in our proposed MDP translates to
$$
\pi^ {*} = \operatorname {a r g m a x} _ {\pi} \mathbb {E} \left(\sum_ {k = 1} ^ {K} \gamma^ {k - 1} r \left(s _ {k}, \pi (\cdot \mid s _ {k})\right)\right), \tag {4}
$$
where the expectation is taken over the stochasticity in both environments  $\mathcal{MR}$  and policy  $\pi$ . Before leveraging any RL algorithms for ranking, it is necessary to validate good policies of  $\mathcal{MR}(S, A, T, r, \gamma, K)$  yields good ranking lists. In the following subsection, we give a rigorous theorem for this validation.
3.3 Optimizing Rank List by Optimizing Policy
Constructing a rank list given policy  $\pi$ . With the definition of MDP, we can construct a rank list with any given policy sequentially. At each position  $k$ , the state is constructed based on previous document features (manually or by sequential models). Then a document (action) is chosen by the policy  $\pi$  and placed at position  $k$ , where the next state is determined accordingly. Repeat this process at each position until all the documents are set or the list reaches its end ( $K = 10$  for example).
Definition 2 (Policy Induced Ranking.). Given a policy  $\pi (\cdot \mid s)$  of  $\mathcal{MR}(\mathcal{S},\mathcal{A},\mathcal{T},r,\gamma)$ , construct the induced rank list  $\mathcal{R}^{\pi}$  as
$$
\mathcal {R} ^ {\pi} (k) \gets a _ {k} \sim \pi (\cdot | s _ {k}).
$$
To investigate whether the optimal rank list can be captured by optimal policy  $\pi^{*}$ , we start with defining the optimality of rank list.
Definition 3 (Optimality). A rank list  $\mathcal{R}$  is optimal if and only if all documents are sorted in descending order in terms of their attractiveness, i.e.
$$
\alpha (\mathcal {R} (1)) \geq \dots \geq \alpha (\mathcal {R} (K))
$$
and  $\{\mathcal{R}(1),\dots ,\mathcal{R}(K)\}$  are the  $K$  most attractive documents among all, where  $K$  is the length of the list and it is also called the top- $K$  ranking.
Assumption 3.1 (Optimality of optimal ranking). Let  $V_{\mathcal{R}}(s_1) = \mathbb{E}\big[\sum_{k=1}^{K} \gamma^{k-1} r(s_k, \mathcal{R}(k))\big]$  be the value of rank list  $\mathcal{R}$ , and let  $\mathcal{R}^{\star}$  be the optimal rank list by Definition 3. Then  $\max_{\mathcal{R}} V_{\mathcal{R}}(s_1) = V_{\mathcal{R}^{\star}}(s_1)$ .
Assumption 3.1 is adopted from Assumption 2 in [30], which suggests optimal rank list sorted by decreasing attractiveness of documents leads to optimal rewards, which will be covered by optimal policy learned from our MDP formulation. This is a mild assumption as classic click models such as PBM and cascade model all satisfy the assumption [63].
4 Unified Off-policy Learning to Rank
The formulation of off-policy learning-to-rank (LTR), when viewed from the perspective of offline reinforcement learning, presents an opportunity to leverage off-the-shelf RL algorithms to address off-policy LTR problems. In this section, we introduce a novel and unified off-policy LTR algorithm that is agnostic to the underlying click model used to generate the offline training data. Our algorithm is composed of three key components: (1) episodes constructed from the logged ranking data; (2) state representation learning; and (3). policy optimization via offline RL algorithms. In the following, we provide a detailed exposition of each component.
Episodes Construction. Given any logged ranking dataset, the first step is to construct a series of episodes from the ranking data, making it compatible with the off-the-shelf RL algorithms. Specifically, our original ranking data  $\{(q_i,\mathcal{R}_i,\mathbf{c}_i(q_i,\mathcal{R}_i))\}_{i = 1}^n$  is composed of  $n$  tuples with contextual query feature  $q_{i}$ , a ranked list  $\mathcal{R}_i$  with  $K$  documents and a corresponding click vector  $\mathbf{c}_i(q_i,\mathcal{R}_i)\in \{0,1\} ^K$ . From the perspective of RL, this tuple can be conceptualized as one episode, following the established MDP formulation of the ranking process. In particular, for each tuple  $(q_{i},\mathcal{R}_{i},\mathbf{c}_{i}(q_{i},\mathcal{R}_{i}))$ , we transform it to a length  $K$  episode  $\tau_{i}\coloneqq \{(s_{k}^{i},a_{k}^{i},r_{k}^{i})\}_{k = 1}^{K}$  with
$$
s _ {k} ^ {i} := \phi (\mathcal {R} _ {i} [: k ], k), \quad a _ {k} ^ {i} := \mathcal {R} _ {i} [ k ], \quad r _ {k} ^ {i} := \mathbf {c} _ {i} (q _ {i}, \mathcal {R} _ {i}) [ k ] \quad \mathrm {f o r} \forall k \in [ K ]
$$
Here we use  $\mathcal{R}_i[:k]$  to denote the concatenation of the document feature vectors before position  $k$  and  $\mathcal{R}_i[k]$  represents the document feature at position  $k$ , similarly for  $\mathbf{c}_i(q_i,\mathcal{R}_i)[k]$ . In particular, the episode  $\tau_{i}$  is constructed by going through the ranked list from top to the bottom. Each state  $s_k^i$  contains all the document information before position  $k$  and the position information  $k$ , represented as a function of  $\mathcal{R}_i[:k]$  and  $k$  with  $\phi$  being a learned embedding function that we will discuss shortly. The action at step  $k$  is the document placed at position  $k$ , i.e.,  $\mathcal{R}_i[k]$ , with the reward at current timestep is the binary click for the corresponding action at position  $k$ , i.e.,  $\mathbf{c}_i(q_i,\mathcal{R}_i)[k]$ . Given this, we have constructed an offline RL dataset with  $n$  episodes with length  $K$  each.
State Representation Learning. In RL, state representation learning offers an efficient means of managing large, raw observation spaces, enhancing the generalization performance when encountering previously unseen states. In our particular setting, the observation space consists of raw document features up to position  $k$ . As we will show in Section 5.2, utilizing raw observations as the state representation in the RL algorithm can lead to highly sub-optimal performance, due to both unscalability with respect to  $k$  and limitations in the representation power. Rather than incorporating additional auxiliary tasks to explicitly learn state representations, we propose to jointly learn the state
representations together with the policy optimization algorithm, which aims to automatically learn state representation  $\phi$  that will benefit the downstream policy optimization task. For example, DQN uses multiple layers of nonlinear functions to encode the map perceptual inputs to a state embedding that could be linearly transformed into the value function. To this end, we introduce the implicit state representation learning component our off-policy learning to rank algorithm, which is composed of the following key components:
Positional Encoding: To effectively inject the position information in  $s_k^i$ , we utilize the positional encoding technique [50, 59], ensuring the model makes use of the position information when generating clicks. Specifically, positional encoding represents each position  $k$  in the ranked list by the sinusoidal function with different frequencies such that
$$
P E (k) _ {2 i} = \sin \left(\frac {k}{1 0 0 0 0 ^ {2 i / d _ {\mathrm {m o d e l}}}}\right), P E (k) _ {2 i + 1} = \cos \left(\frac {k}{1 0 0 0 0 ^ {2 i / d _ {\mathrm {m o d e l}}}}\right),
$$
Here  $PE(k) \in \mathbb{R}^d$  with  $d$  being the dimension of document feature and  $2i$  and  $2i + 1$  being the even and the odd entries of  $PE(k)$  with  $2i, 2i + 1 \in [d]$ .
Multi-head Self-attention: The other challenge in our case is to find a specific architecture for the state representation learning that tailored for the learning to rank task. As the ranked list of document features is inherently sequential, we leverage the multi-head self-attention mechanism to learn the state embedding  $\phi$ . Specifically, the state  $s_k^i$  is defined as:
$$
s _ {k} ^ {i} := \phi (\mathcal {R} _ {i} [: k ], k) = \operatorname {C o n c a t} (h e a d _ {1}, \dots , h e a d _ {I}) W ^ {O}
$$
where  $head_{i} = \text{Attention}(s_{k} \cdot W_{i}^{Q}, s_{k} \cdot W_{i}^{K}, s_{k} \cdot W_{i}^{V})$ , with  $W_{i}^{Q}, W_{i}^{K}, W_{i}^{V} \in \psi_{i}$  are learnable parameters for the  $i^{th}$  head and  $W^{O}$  is the learnable parameter of the output layer after concatenating the results of all the  $I$  heads; At each position  $k$ , we concatenate the document features  $\mathcal{R}_{i}[: k]$  along with the position embedding for position  $k$ , passing them as the input for the multi-head self-attention.