SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Joint Representation Learning and Policy Optimization. Given the constructed offline dataset and a dedicated structure for learning efficient state representations, we now demonstrate how we leverage any off-the-shelf RL algorithm as the plug-in solver for finding the optimal policy. We use the popular offline RL algorithm: CQL, as an instantiation, which learns a conservative Q function and utilizes the classical soft actor-critic algorithm on top of it. Specifically, it optimizes the following lower bound of the critic (Equation 5):
$$
\hat {\theta} \leftarrow \underset {\theta} {\operatorname {a r g m i n}} \alpha \mathbb {E} _ {s \sim \mathcal {D}} \left[ \log \sum_ {a} \exp \left(Q _ {\theta} (s, a)\right) - \mathbb {E} _ {a \sim \pi_ {\beta} (a | s)} \left[ Q _ {\theta} (s, a) \right] \right] + \frac {1}{2} \mathbb {E} _ {s, a, s ^ {\prime} \sim \mathcal {D}} \left[ \left(Q _ {\theta} - \hat {\mathcal {B}} ^ {\pi} \hat {Q} _ {\theta^ {\prime}}\right) ^ {2} \right] \tag {5}
$$
where  $\hat{B}^{\pi}Q = r + \gamma P^{\pi}Q$  is the estimated Bellman operator,  $\pi_{\beta}$  is the logging policy. Here we use  $\theta'$  to emphasize that the parameters in the target Q network is different from policy Q network. The conservative minimizes the expected Q value for out-of-distribution (state, action) pairs and prevents the Q-function's over-estimation issue. Built upon SAC, the algorithm improves the policy  $\pi_{\xi}$  (i.e., the actor) based on the gradient of the estimated Q function, with entropy regularization. Compared with the original CQL algorithm, we also add the state representation learning component, and we jointly optimize the state embedding, the critic and actor parameters, with a detailed algorithm included in Algorithm 1. Once we have obtained the learned optimal policy  $\pi_{\xi}^{*}$ , we extract the optimal ranking policy following Definition 2.
We empirically evaluate the performance of our proposed method CUOLR on several public datasets, compared with the state-of-the-art off-policy learning-to-rank methods. Specifically, we aim to assess the robustness of our method across different click models and showcase the effectiveness of our unified framework. Beyond this, we perform the ablation study to examine the efficacy of our proposed state representation learning component.<sup>3</sup>.
<sup>3</sup>Codes: https://github.com/ZeyuZhang1901/Unified-Off-Policy-LTR-Neurips2023
Algorithm 1 Click Model-Agnostic Unified Off-policy Learning to Rank (with CQL)
1: Inputs: logged ranking data  $\{(q_i, \mathcal{R}_i, \mathbf{c}_i(q_i, \mathcal{R}_i))\}_{i=1}^n$ , length of the ranking  $K$ , batch size  $B$ , train iteration  $T$ .
2: Initialize: policy  $\pi_{\xi}$  and Q function  $Q_{\theta}$ , embedding model  $\phi_{\psi}(\cdot, \cdot)$
3: for  $t \in [T]$  do:
4: Randomly sample a batch of queries  $\mathcal{Q}$  with size  $B$ .
5: Construct offline RL episodes  $\mathcal{T} = \left\{\{(s_k^i,a_k^i,r_k^i)\}_{k = 1}^K\right\}_{i:q_i\in \mathcal{Q}}$
$$
s _ {k} ^ {i} = \phi_ {\psi} (\mathcal {R} _ {i} [: k ], k), \quad a _ {k} ^ {i} = \mathcal {R} _ {i} [ k ], \quad r _ {k} ^ {i} = \mathbf {c} _ {i} (q _ {i}, \mathcal {R} _ {i}) [ k ] \quad \text {f o r} \forall k \in [ K ]
$$
6: Train the Q-net (and embedding model) with loss defined in equation (5)
$$
\begin{array}{l} \theta \leftarrow \theta - \eta_ {Q} \nabla_ {\theta} \operatorname {L o s s} (\theta , \mathcal {T}) \\ \psi \leftarrow \psi - \eta_ {\phi} \nabla_ {\psi} \operatorname {L o s s} (\theta , \mathcal {T}) \\ \end{array}
$$
7: Improve policy  $\pi_{\xi}$  (and embedding model) with SAC-style entropy regularization.
$$
\begin{array}{l} \xi \leftarrow \xi + \eta_ {\pi} \mathbb {E} _ {s \sim \mathcal {T}, a \sim \pi_ {\xi} (\cdot | s)} \left[ Q _ {\theta} (s, a) - \log \pi_ {\xi} (a \mid s) \right] \\ \psi \leftarrow \psi + \eta_ {\phi} \mathbb {E} _ {s \sim \mathcal {T}, a \sim \pi_ {\xi} (\cdot | s)} \left[ Q _ {\theta} (s, a) - \log \pi_ {\xi} (a \mid s) \right] \\ \end{array}
$$
9: Output: learned ranking policy  $\pi_{\xi}^{*}$ , Q function  $Q_{\theta}^{*}$ , embedding model  $\phi_{\psi}^{*}(\cdot, \cdot)$
10: Recover the optimal ranking from the learned policy  $\pi_{\xi}^{*}$  using Definition 2.
Datasets. We conduct semi-synthetic experiments on two traditional learning-to-rank benchmark datasets: MSLR-WEB10K and Yahoo! LETOR (set 1). Specifically, we sample click data from these real-world datasets, which not only increases the external validity of the experiments but also provides the flexibility to explore the robustness of our method over different click models. For both datasets, it consists of features representing query and document pairs with manually judged relevance labels ranging from 0 (irrelevant) to 4 (perfectly relevant). We provide the statistics of all datasets in Appendix A. Both datasets come with the train-val-test split. The train data is used for generating logging policy and simulating clicks, with the validation data used for hyperparameter selection. And the final performance of the learned ranking policy is evaluated in the test data.
Click Data Generation. We follow Joachims et al. [25] to generate partial-information click data from the full-information relevance labels. Specifically, we first train a Ranking SVM [26] using  $1\%$  of the training data as our logging policy  $\pi_{\beta}$  to present the initial ranked list of items. For each query  $q_{i}$ , we get a ranking  $\mathcal{R}_i$  and simulate the clicks based on various click models we use. As discussed in Section 3.1, there are two components for the click generation, examination probability and attractiveness of the document for the query. All click models differ in their assumptions on the examination probability. For PBM, we adopt the examination probability  $\rho = \{\rho_k\}_{k=1}^K$  estimated by Joachims et al. [25] through eye-tracking experiments:
$$
\chi (\mathcal {R} ^ {q}, k) = \chi (k) = \rho_ {k} ^ {\eta}
$$
where  $\eta \in [0, +\infty]$  is a hyper-parameter that controls the severity of presentation biases and in our experiment, we set  $\eta = 1.0$  as default. For CASCADE, the examination probabilities are only dependent on the attractions of each previous document. For DCM, the  $\lambda$ s are also simulated by the same parameters as PBM examination probability. We use the same attraction models for all click models, as defined following:
$$
\alpha (d) = \epsilon + (1 - \epsilon) \frac {2 ^ {r (d)} - 1}{2 ^ {r _ {\text {m a x}}} - 1}
$$
where  $r(d) \in [0,4]$  is the relevance label for document  $d$  and  $r_{max} = 4$  is the maximum relevance label. We also use  $\epsilon$  to model click noise so that irrelevant documents have a non-zero probability to be treated as attractive and being clicked.
Table 1: Performance comparison with different click models on Yahoo! LETOR set1 and MSLR-WEB10K. "**" and "***" indicate statistically significant improvement (p-value  $< 0.05$  and p-value  $< 0.01$  respectively) over the best baseline for each metric respectively.
CLICK MODEL	ALG	Yahoo! LETOR	MSLR-WEB10K
ERR@K	NDCG@K	ERR@K	NDCG@K
K=5	K=10	K=5	K=10	K=5	K=10	K=5	K=10
PBM	DLA	0.439	0.455	0.691	0.742	0.256	0.278	0.356	0.384
CM-IPW	0.440	0.456	0.692	0.743	0.255	0.277	0.354	0.383
IPW	0.446	0.462	0.700	0.748	0.281	0.301	0.367	0.390
CUOLR(CQL)	0.458**	0.473**	0.700	0.753*	0.281	0.303	0.380**	0.406**
CUOLR(SAC)	0.459**	0.478**	0.700	0.753**	0.279	0.301	0.379**	0.404**
CASCADE	DLA	0.441	0.457	0.690	0.741	0.265	0.286	0.365	0.392
CM-IPW	0.444	0.460	0.696	0.745	0.283	0.304	0.378	0.403
IPW	0.442	0.457	0.690	0.740	0.257	0.279	0.358	0.387
CUOLR(CQL)	0.459**	0.479**	0.696	0.748	0.280	0.301	0.378	0.404
CUOLR(SAC)	0.461*	0.478**	0.696	0.748	0.279	0.301	0.379	0.405
DCM	DLA	0.444	0.459	0.696	0.745	0.279	0.299	0.378	0.404
CM-IPW	0.447	0.463	0.704	0.752	0.284	0.304	0.379	0.402
IPW	0.444	0.459	0.697	0.746	0.278	0.299	0.363	0.387
CUOLR(CQL)	0.461**	0.474**	0.699	0.753	0.278	0.300	0.380	0.405
CUOLR(SAC)	0.461**	0.475**	0.703	0.755	0.278	0.300	0.378	0.403
LOGGING	0.385	0.403	0.630	0.693	0.206	0.230	0.304	0.338
ORACLE	0.462	0.476	0.739	0.781	0.328	0.347	0.432	0.453
Baselines and Hyperparameters. We compare our CUOLR method with the following baselines: (1). Dual Learning Algorithm (DLA) [4] which jointly learns an unbiased ranker and an unbiased propensity model; (2). Inverse Propensity Weighting (IPW) Algorithm [52, 26] which first learns the propensities by result randomization, and then utilizes the learned probabilities to correct for position bias; and (3). Cascade Model-based IPW (CM-IPW) [48] which designs a propensity estimation procedure where previous clicks are incorporated in the estimation of the propensity. It is worth mentioning that (1) and (2) are designed for PBM and (3) is tailored for cascade-based models. Besides, we train a LambdaMart [6] model with true relevance labels as the upper bound for the ranking model, ORACLE for short. The performance of the logging policy (LOGGING) is also reported as the lower bound of the ranking model.
For baselines, we use a 2-layer MLP with width 256 and ReLU activation according to their original paper and codebase [4, 51, 47]. For the embedding model in our method, we use multi-head attention with 8 heads. And for actors and critics in CQL and SAC algorithms, we utilize a 2-layer MLP with width 256 and ReLU activation. The conservative parameter  $\alpha$  (marked red in Equation (5)) in CQL is set to 0.1. We use Adam for all methods with a tuned learning rate using the validation set. More details are provided in Appendix A.
Metrics. We evaluate all the methods using the full-information test set. We use the normalized Discounted Cumulative Gain (nDCG) [20] and the Expected Reciprocal Rank (ERR) [8] as evaluation metrics and report the results at position 5 and 10 to demonstrate the performance of models at different positions.
How does CUOLR perform across different click models, compared to the baselines? To validate the effectiveness of our CUOLR method, we conducted performance evaluations across a range of click-based models, including PBM, CASCADE, DCM, and CCM. We compared our approach with state-of-the-art baselines specifically designed for these click models, namely DLA, IPW, and CM-IPW. Due to space limitation, we only show results of PBM, CASCADE, and DCM in Table 1, with the full table shown in Appendix B. For the position-based model, IPW demonstrates the
Table 2: Performance of CUOLR algorithm with different state embeddings. The experiments are conducted on Yahoo! LETOR set1 and MSLR-WEB10K, with PBM and CASCADE as click models.
CLICK MODEL	STATE EMBED	Yahoo! LETOR	MSLR-WEB10K
ERR@K	NDCG@K	ERR@K	NDCG@K
K=5	K=10	K=5	K=10	K=5	K=10	K=5	K=10
PBM	POS	0.439	0.455	0.691	0.742	0.282	0.304	0.382	0.407
PREDOC	0.435	0.451	0.682	0.734	0.274	0.295	0.374	0.401
POS + PREDOC	0.440	0.456	0.685	0.734	0.277	0.298	0.375	0.400
ATTENTION	0.459	0.478	0.700	0.753	0.281	0.303	0.380	0.406
CASCADE	POS	0.426	0.442	0.668	0.724	0.260	0.283	0.360	0.391
PREDOC	0.456	0.472	0.703	0.754	0.272	0.293	0.373	0.400
POS+PREDOC	0.453	0.470	0.686	0.740	0.275	0.296	0.373	0.396
ATTENTION	0.461	0.478	0.696	0.748	0.280	0.301	0.378	0.404
ORACLE	0.462	0.486	0.739	0.781	0.328	0.347	0.432	0.453
best performance among the baselines, which is expected as it is tailored for position-based methods. Similarly, CM-IPW yielded the best performance for the cascade-based methods, which aligns with its incorporation of previous document information in the propensity estimation. Remarkably, across all click models, our method, whether combined with SAC or CQL, consistently achieves the best performance in most cases. This validates the effectiveness of our unified framework and the robustness of our CUOLR algorithm. Furthermore, it is noteworthy that our method demonstrated consistent performance across different RL algorithms, verifying its resilience and adaptability to various underlying RL solvers.
How effective is the state representation learning component in CUOLR? In this experiment, we examine different approaches for state representation learning and study how it affects the overall performance of our proposed method. We compare with the following state embeddings: (1). position-only embedding (POS), which only utilizes the position information using positional encoding; (2). previous-document-based embedding (PREDOC), which takes a simple average of all the document features in  $\mathcal{R}[:,k]$ ; (3). the concatenation of the position and the average document features up to position  $k$  (POS + PREDOC), as well as the proposed learnable state representations based on multi-head self-attention (ATTENTION). ORACLE here is used to show the gap from the upper bound. The results of our experiments are presented in Table 2 (with a full table including other click models is shown in Appendix B). For the PBM click model, it is evident that state embeddings utilizing position-based information, such as POS and POS+PREDOC, outperform other state embeddings. In contrast, for the CASCADE click model, state embeddings utilizing previous document features exhibit significantly stronger performance compared to those utilizing position information. Notably, our method, CUOLR, which dynamically learns the state embeddings during policy optimization, consistently achieves comparable performance compared to using hard-coded fixed state embeddings. This highlights the necessity of leveraging state representation in off-policy LTR and underscores the effectiveness of our proposed approach.
In this paper, we present an off-policy learning-to-rank formulation from the perspective of reinforcement learning. Our findings demonstrate that under this novel MDP formulation, RL algorithms can effectively address position bias and learn the optimal ranker for various click models, without the need for complex debiasing methods employed in unbiased learning to rank literature. This work establishes a direct connection between reinforcement learning and unbiased learning to rank through a concise MDP model. Specifically, we propose a novel off-policy learning-to-rank algorithm, CUOLR, which simultaneously learns efficient state representations and the optimal policy. Through empirical evaluation, we show that CUOLR achieves robust performance across a wide range of click models, consistently surpassing existing off-policy learning-to-rank methods tailored to those specific models. These compelling observations indicate that the extensive research conducted on offline
reinforcement learning can be leveraged for learning to rank with biased user feedback, opening up a promising new area for exploration.
This work was supported in part by Google Cloud Research Credits Program. Mengdi Wang acknowledges the support by NSF grants DMS-1953686, IIS-2107304, CMMI-1653435, ONR grant 1006977, and C3.AI.
[1] Alekh Agarwal, Nan Jiang, Sham M Kakade, and Wen Sun. Reinforcement learning: Theory and algorithms. CS Dept., UW Seattle, Seattle, WA, USA, Tech. Rep, pages 10-4, 2019.
[2] Aman Agarwal, Kenta Takatsu, Ivan Zaitsev, and Thorsten Joachims. A general framework for counterfactual learning-to-rank. In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 5-14, 2019.
[3] Aman Agarwal, Xuanhui Wang, Cheng Li, Michael Bendersky, and Marc Najork. Addressing trust bias for unbiased learning-to-rank. In The World Wide Web Conference, pages 4-14, 2019.
[4] Qingyao Ai, Keping Bi, Cheng Luo, Jiafeng Guo, and W Bruce Croft. Unbiased learning to rank with unbiased propensity estimation. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, pages 385-394, 2018.
[5] Jacob Buckman, Carles Gelada, and Marc G Bellemare. The importance of pessimism in fixed-dataset policy optimization. arXiv preprint arXiv:2009.06799, 2020.
[6] Chris J.C. Burges. From ranknet to lambdarank to lambdamart: An overview. Technical Report MSR-TR-2010-82, June 2010. URL https://www.microsoft.com/en-us/research/publication/from-ranknet-to-lambdarank-to-lambdamart-an-overview/.
[7] Olivier Chapelle and Yi Chang. Yahoo! learning to rank challenge overview. In Proceedings of the learning to rank challenge, pages 1-24. PMLR, 2011.
[8] Olivier Chapelle, Donald Metlzer, Ya Zhang, and Pierre Grinspan. Expected reciprocal rank for graded relevance. In Proceedings of the 18th ACM conference on Information and knowledge management, pages 621-630, 2009.
[9] Jiawei Chen, Hande Dong, Xiang Wang, Fuli Feng, Meng Wang, and Xiangnan He. Bias and debias in recommender system: A survey and future directions. arXiv preprint arXiv:2010.03240, 2020.
[10] Aleksandr Chuklin, Ilya Markov, and Maarten de Rijke. Click Models for Web Search. Morgan & Claypool, 2015. ISBN 9781627056489. doi: 10.2200/S00654ED1V01Y201507ICR043.
[11] Matej Cief, Branislav Kveton, and Michal Kompan. Pessimistic off-policy optimization for learning to rank. arXiv preprint arXiv:2206.02593, 2022.
[12] Nick Craswell, Onno Zoeter, Michael Taylor, and Bill Ramsey. An experimental comparison of click position-bias models. In Proceedings of the 2008 international conference on web search and data mining, pages 87-94, 2008.
[13] Georges E Dupret and Benjamin Piwowarski. A user browsing model to predict search engine click data from past observations. In Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval, pages 331-338, 2008.
[14] Damien Ernst, Pierre Geurts, and Louis Wehenkel. Tree-based batch mode reinforcement learning. Journal of Machine Learning Research, 6, 2005.
[15] Dylan J Foster, Akshay Krishnamurthy, David Simchi-Levi, and Yunzong Xu. Offline reinforcement learning: fundamental barriers for value function approximation. arXiv preprint arXiv:2111.10919, 2021.
[51] Nan Wang, Zhen Qin, Xuanhui Wang, and Hongning Wang. Non-clicks mean irrelevant? propensity ratio scoring as a correction. In Proceedings of the 14th ACM International Conference on Web Search and Data Mining, pages 481-489, 2021.
[52] Xuanhui Wang, Michael Bendersky, Donald Metzler, and Marc Najork. Learning to rank with selection bias in personal search. In Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval, pages 115–124, 2016.
[53] Xuanhui Wang, Nadav Golbandi, Michael Bendersky, Donald Metzler, and Marc Najork. Position bias estimation for unbiased learning to rank in personal search. In Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining, pages 610-618, 2018.
[54] Zeng Wei, Jun Xu, Yanyan Lan, Jiafeng Guo, and Xueqi Cheng. Reinforcement learning to rank with markov decision process. In Proceedings of the 40th international ACM SIGIR conference on research and development in information retrieval, pages 945-948, 2017.
[55] Tengyang Xie, Ching-An Cheng, Nan Jiang, Paul Mineiro, and Alekh Agarwal. Bellman-consistent pessimism for offline reinforcement learning. Advances in Neural Information Processing Systems, 34:6683-6694, 2021.
[56] Tengyang Xie, Nan Jiang, Huan Wang, Caiming Xiong, and Yu Bai. Policy finetuning: Bridging sample-efficient offline and online reinforcement learning. Advances in Neural Information Processing Systems, 34:27395-27407, 2021.
[57] Jun Xu, Zeng Wei, Long Xia, Yanyan Lan, Dawei Yin, Xueqi Cheng, and Ji-Rong Wen. Reinforcement learning to rank with pairwise policy gradient. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 509-518, 2020.
[58] Tengyu Xu and Yingbin Liang. Provably efficient offline reinforcement learning with trajectory-wise reward. arXiv preprint arXiv:2206.06426, 2022.
[59] Hang Yan, Bocao Deng, Xiaonan Li, and Xipeng Qiu. Tener: adapting transformer encoder for named entity recognition. arXiv preprint arXiv:1911.04474, 2019.
[60] Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Y Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. Advances in Neural Information Processing Systems, 33:14129-14142, 2020.
[61] Andrea Zanette, Martin J Wainwright, and Emma Brunskill. Provable benefits of actor-critic methods for offline reinforcement learning. Advances in Neural Information Processing Systems, 34:13626-13640, 2021.
[62] Shengyao Zhuang, Zhihao Qiao, and Guido Zuccon. Reinforcement online learning to rank with unbiased reward shaping. arXiv preprint arXiv:2201.01534, 2022.
[63] Masour Zoghi, Tomas Tunys, Mohammad Ghavamzadeh, Branislav Kveton, Csaba Szepesvari, and Zheng Wen. Online learning to rank in stochastic click models. In International Conference on Machine Learning, pages 4199-4208, 2017.
We conducted experiments on MSLR-WEB10K  $^{4}$  and Yahoo! LETOR (set 1)  $^{5}$  with semi-synthetic generated click data. Yahoo! LETOR comes from the Learn to Rank Challenge. It consists of 29,921 queries and 710K documents. Each query-document pair is represented by a 700-dimensional feature and annotated with a 5-level relevance label ranging from 0 to 4. MSLR-WEB10K dataset contains 10,000 queries and 125 retrieved documents on average. Each query-document pair is represented by a 136-dimensional feature vector and a 5-level relevance label. The dataset is partitioned into five parts with about the same number of queries, denoted as S1, S2, S3, S4, and S5, for five-fold cross-validation. All statistics of the used datasets are summarized in Table 3.
Table 3: Statistics of Learning to Rank Datasets
# of queries	# of features	# of query-document pairs	rel level	# of folders
MSLR-WEB10K	10000	136	1250k	5	5
Yahoo! LETOR	29921	700	710k	5	2
Before introducing the hyperparameters required for each algorithm, we first describe some global hyperparameters that are used commonly across all algorithms. We use batch size  $B = 256$  queries per epoch, and use  $nDCG@10$  as the training objective for all baselines. We use Adam optimizer to train all the networks.
Dual Learning Algorithm (DLA) [4]. For DLA, two sub-models are being implemented: the ranking (scoring) model which is used to score each document; and the propensity model which is used to estimate the propensity for each document in the rank list. We use the MLP with two hidden layers of 256 units for both of them, and other hyperparameters are shown in Table 5.
Inverse Propensity Weighting (IPW) [52, 26] & Cascade Model-based IPW (CM-IPW) [48]. For IPW and CM-IPW, we need to get the propensities from Result Randomization. In total, there are 10M random rank lists with different searching queries shown to the user (click model), and the parameters of each click model are estimated by Maximize Likelihood Estimation (MLE). Estimation details are shown in Table 4, where  $C_k$  and  $C_{<k}$  indicate the click at rank  $k$  and before rank  $k$  respectively. The superscript  $\cdot^{(s)}$  denotes the click of some click session  $s$ . Besides, we implement the ranking model the same way as that for DLA. Other hyperparameters are shown in Table 5.
ORACLE. For ORACLE, we train a LambdaMART ranker Burges [6] with true labels on the training dataset and evaluate its performance on the test set. We leverage the RankLib $^{6}$  learning to rank library, and set the hyperparameters shown in Table 5. This utilizes the full-information data and serves as an upper bound of the performance for all algorithms utilizing the partial-information data, such as the generated clicks.
**CUOLR.** For our algorithm CUOLR, there are three sets of hyperparameters used by the following sub-models: parameters for state embedding model, RL policy, as well as RL critic, where we use a 256-256 MLP to implement all the networks. Detailed hyperparameters are shown in Table 5.
In this section, we present the complete results for all five click models (PBM, CASCADE, UBM, DCM, and CCM) on the two datasets: Yahoo! LETOR set 1 and MSLR-WEB10K. For the first two
<sup>4</sup>https://www.microsoft.com/en-us/research/project/mslr/
<sup>5</sup>https://webscope Sandbox.yahoo.com/
$^{6}$ https://sourceforge.net/p/lemur/wiki/RankLib/
Table 4: Details for IPW and CM-IPW.
PROPENSITY	PARAM. ESTIMATION
IPW	P(E=1|k)=θk	θk=∑s∈S C0(s)/∑s∈SC(k(s))
CM-IPW	P(E=1|k)≈P(E=1|k,C<k)=Πi<k(1-Ci(1-λi))	λk=1-∑s∈Sc I(C(s)>0)/∑s∈Sc 1, Sk={s:C(s)=1}