SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 5: Hyperparameters for each algorithm used in the experiment.
ALG	HYPERPARAMETERS
DLA	policy learning rate: 1e-4
propensity learning rate: 1e-4
loss type: softmax
IPW & CM-IPW	policy learning rate: 1e-4
loss type: softmax
ORACLE	number of trees: 1000
number of leaves for each tree: 100
shrinkage (learning rate): 0.01
min leaf support: 50
early stop: 100
CUOLR	actor learning rate: 1e-4
critic learning rate: 1e-4
actor α: 1e-10 (fixed)
soft update τ: 5e-3
discount γ: 0.8
embed learning rate: 1e-6
embed type: multi-head attention
number of heads: 8
CQL α: 1e-1 (fixed)
studies in Section B.1 and B.2, we run 5 runs with different random seeds for the Yahoo! dataset. For the MSLR-WEB10K dataset which naturally comes with 5 folds, we take 1 run for each fold and aggregate the results. In the ablation experiment of conservatism for the offline RL algorithm in Section B.3, we only run 3 runs on Yahoo! due to the time limit. In all of our experiments, we use nDCG and ERR at positions 3,5,10 as evaluation metrics.
B.1 Performance across Different Click Models
In this section, we present a comprehensive comparison between our proposed method, CUOLR, and various baseline approaches. Specifically, we examine the efficacy of DLA and IPW, which have been specifically designed for position-based models, as well as CM-IPS, which has been tailored for cascade-based models. The comparative results are presented in Table 7. In the case of position-based models such as PBM and UBM, it is evident that IPW demonstrates the most superior performance among all the considered baselines. Conversely, when evaluating cascade models such as cascade and DCM, the utilization of CM-IPW yields improved performance, as it takes into account the propensity estimation considering prior examinations and clicks. Among the diverse click models examined, our unified algorithm, CUOLR, consistently achieves the highest level of performance in terms of the ERR metrics across different positions. Furthermore, it consistently outperforms the other models in the majority of cases in terms of nDCG@10. This provides empirical verification of the effectiveness of our unified framework and the robustness of the CUOLR algorithm.
B.2 State Representation Ablation Experiments
In this section, we present an ablation study focusing on the state embedding utilized in our algorithm, CUOLR. We compare the effectiveness of our proposed multi-head self-attention, augmented with positional embedding, against several heuristic hard-coded baselines for state embedding. These
Table 6: Performance comparison with different click models on Yahoo! LETOR set1. " *" and "***" indicate statistically significant improvement (p-value  $< 0.05$  and p-value  $< 0.01$  respectively) over the best baseline for each metric respectively.
CLICK MODEL	ALG	ERR@3	ERR@5	ERR@10	nDCG@3	nDCG@5	nDCG@10
PBM	DLA	0.418	0.439	0.455	0.669	0.691	0.742
CM-IPW	0.418	0.440	0.456	0.669	0.692	0.743
IPW	0.424	0.446	0.462	0.678	0.700	0.748
CUOLR(CQL)	0.437**	0.458**	0.473**	0.678	0.700	0.753*
CUOLR(SAC)	0.437**	0.459**	0.478**	0.674	0.700	0.753**
CASCADE	DLA	0.420	0.441	0.457	0.668	0.690	0.741
CM-IPW	0.422	0.444	0.460	0.674	0.696	0.745
IPW	0.420	0.442	0.457	0.668	0.690	0.740
CUOLR(CQL)	0.436**	0.459**	0.479**	0.668	0.696	0.748
CUOLR(SAC)	0.440**	0.461*	0.478**	0.670	0.696	0.748
UBM	DLA	0.426	0.448	0.463	0.687	0.708	0.756
CM-IPW	0.415	0.437	0.453	0.664	0.689	0.740
IPW	0.427	0.449	0.464	0.686	0.708	0.756
CUOLR(CQL)	0.435**	0.457**	0.473**	0.679	0.704	0.756
CUOLR(SAC)	0.435**	0.458**	0.473**	0.675	0.700	0.754
DCM	DLA	0.422	0.444	0.459	0.673	0.696	0.745
CM-IPW	0.426	0.447	0.463	0.683	0.704	0.752
IPW	0.422	0.444	0.459	0.674	0.697	0.746
CUOLR(CQL)	0.437**	0.461**	0.474**	0.676	0.699	0.753
CUOLR(SAC)	0.436**	0.461**	0.475**	0.675	0.703	0.755
CCM	DLA	0.427	0.449	0.464	0.685	0.707	0.754
CM-IPW	0.416	0.438	0.453	0.659	0.684	0.736
IPW	0.429	0.450	0.465	0.689	0.710	0.757
CUOLR(CQL)	0.438**	0.462**	0.478**	0.681	0.704	0.758
CUOLR(SAC)	0.437**	0.460**	0.476**	0.680	0.706	0.759
LOGGING	0.359	0.385	0.403	0.595	0.630	0.693
ORACLE	0.440	0.462	0.486	0.720	0.739	0.781
bases include utilizing only positional information (POS), concatenating previous document information (PREDOC), and a combination of positional and document information (POS+PREDOC). The evaluation is performed on the Yahoo dataset, and the results are summarized in Table 8. Consistent with expectations, for position-based models (PBM), the most effective approach is utilizing only the positional information. Conversely, for cascade models (CASCADE), only considering the previous document information gives the best performance. In the case of more complicated models, such as DCM, CCM, and UBM, where the click model relies on both position and previous examinations, it becomes evident that incorporating a combination of positional information and previous document information yields the highest performance. Among all of them, it is worth to point out that our proposed state representation learning consistently attains comparable performance to the best baseline. Notably, our method possesses the advantage of automatically learning the optimal state representation, irrespective of the underlying assumptions of the click models.
B.3 Conservatism for Offline RL Ablation Experiments
In this section, we conduct an ablation study to investigate the influence of the hyperparameter  $\alpha$ , which governs the conservatism, in the CQL algorithm. Specifically, we examine its effects on the Yahoo! dataset by varying  $\alpha$  across a range of values:  $\{0, 1e - 3, 5e - 3, 1e - 2, 5e - 2, 1e - 1, 5e - 1, 1e0, 5e0, 1e1, 5e1\}$ . It is noteworthy that when  $\alpha$  is set to 0, the CQL algorithm simplifies to the SAC algorithm [19]. Remarkably, we find that the performance of CUOLR remains consistently robust across the diverse  $\alpha$  values, as long as they are not being restricted to be too close to the logging policy (i.e., large  $\alpha$  values). This consistency underscores the effectiveness of our method, which demonstrates its ability to adapt to different underlying reinforcement learning (RL) algorithms. Interestingly, in contrast to classical offline RL datasets studied in Kumar et al. [29], where the conservatism parameter plays a substantial role, we observe that its impact is comparatively minor
Table 7: Performance comparison with different click models on MSLR-WEB10K. "**" and "***" indicate statistically significant improvement (p-value  $< 0.05$  and p-value  $< 0.01$  respectively) over the best baseline for each metric respectively.
CLICK MODEL	ALG	ERR@3	ERR@5	ERR@10	nDCG@3	nDCG@5	nDCG@10
PBM	DLA	0.230	0.256	0.278	0.343	0.356	0.384
CM-IPW	0.229	0.255	0.277	0.341	0.354	0.383
IPW	0.257	0.281	0.301	0.356	0.367	0.390
CUOLR(CQL)	0.257	0.281	0.303	0.369**	0.380**	0.406**
CUOLR(SAC)	0.255	0.279	0.301	0.368**	0.379**	0.404**
CASCADE	DLA	0.239	0.265	0.286	0.354	0.365	0.392
CM-IPW	0.259	0.283	0.304	0.367	0.378	0.403
IPW	0.232	0.257	0.279	0.347	0.358	0.387
CUOLR(CQL)	0.255	0.280	0.301	0.366	0.378	0.404
CUOLR(SAC)	0.255	0.279	0.301	0.368*	0.379	0.405*
UBM	DLA	0.257	0.280	0.300	0.369	0.377	0.400
CM-IPW	0.253	0.276	0.297	0.354	0.362	0.386
IPW	0.258	0.281	0.301	0.359	0.367	0.390
CUOLR(CQL)	0.255	0.280	0.301	0.373**	0.384**	0.408**
CUOLR(SAC)	0.260	0.284	0.306*	0.374*	0.384*	0.408*
DCM	DLA	0.254	0.279	0.299	0.368	0.378	0.404
CM-IPW	0.259	0.284	0.304	0.367	0.379	0.402
IPW	0.256	0.278	0.299	0.354	0.363	0.387
CUOLR(CQL)	0.254	0.278	0.300	0.369	0.380	0.405
CUOLR(SAC)	0.254	0.278	0.300	0.367	0.378	0.403
CCM	DLA	0.252	0.275	0.295	0.347	0.356	0.379
CM-IPW	0.227	0.249	0.270	0.305	0.316	0.342
IPW	0.255	0.278	0.298	0.351	0.360	0.383
CUOLR(CQL)	0.255	0.280	0.301*	0.373**	0.384**	0.408**
CUOLR(SAC)	0.260*	0.284**	0.305**	0.374**	0.383**	0.408**
LOGGING	0.180	0.206	0.230	0.288	0.304	0.338
ORACLE	0.305	0.328	0.347	0.426	0.432	0.453
in the offline learning to rank dataset. This observation is worth further investigation for better understanding the impact of conservatism in offline LTR settings.
Besides, To figure out the effect of conservatism, we do experiments on several click models on Yahoo! dataset to compare the performance between simple SAC and CQL with the optimal conservative parameter  $\alpha$ . The optimal  $\alpha$  is selected through grid search ranging from  $\{1e - 3,5e - 3,1e - 2,5e - 2,1e - 1,5e - 1,1e0,5e0,1e1,5e1\}$ . The results are shown in Table 9. It's obvious that the performance of CQL with optimal  $\alpha$  is consistently better than that of CQL, especially in the NDCG metric. This improvement further illustrates the necessity of conservatism in Off-policy learning to rank task.
B.4 Data Quality for Offline Reinforcement Learning to Rank
In this section, we conduct an ablation study to clarify the effect of data quality to the offline RL algorithms. In usual cases, data quality is important to most offline algorithms, including the offline RL algorithm. To verify the effect of data quality, we conduct experiments on the Web10k dataset with different offline data. Specifically, on each click model, the logging policy is trained with different portions of train data: 1) SVMRank trained with  $1\%$  train data; 2) SVMRank trained with  $0.01\%$  train data; 3) random policy without pre-training. Then the quality of the train data gathered by the three logging policy decreases, and the performance of CQL on them is displayed in Table 10. We find that the algorithm trained with data in better quality (first line for each click model) performs significantly better than the other two. Besides, it's interesting that the algorithm trained with random logging data can't even beat the logging policy trained with  $1\%$  train data, which further highlight the significance of data quality.
Figure 1: Graphical models of different click models and our MDP formulation about the learning to rank problem.
C Click Model and MDP Formulation
In this section, we present a comprehensive overview of different click models employed in the paper, namely PBM, CASCADE, DCM, CCM, and UBM. Additionally, we demonstrate the graphical models associated with each click model and how they could be unified into the Markov Decision Process (MDP) framework, as depicted in Figure 1.
PBM [24]. The position-based model is a model where the probability of clicking on an item depends on both its identity and rank. The examination probability is rank-dependent only, i.e.,
$$
\chi (\mathcal {R}, k) = \chi (k).
$$
CASCADE [12]. The cascade model assumes that the user scans a rank list  $\mathcal{R}$  from top to bottom. If a document at rank  $k$  is examined and clicked, the user stops browsing the remaining documents. Otherwise, the user goes on to the next rank  $k + 1$  with probability one. The first document  $d_{1}$  is always examined. The document at rank  $k$  will be examined if and only if the previous  $k - 1$  documents are not clicked. Therefore we have:
$$
\chi (\mathcal {R}, k) = \Pi_ {i = 1} ^ {k - 1} (1 - \alpha (\mathcal {R} (i))).
$$
DCM [48, 18]. The dependent click model assumes that the user examines the results from top to bottom until an attractive result is found,  $P(E_{k + 1} = 1 \mid E_k = 1, C_k = 0) = 1$ , where  $E_k$  is the examination indicator at rank  $k$ . After each click, there is a rank-dependent chance that the user is unsatisfied,  $P(E_{k + 1} = 1 \mid C_k = 1) = \lambda_k$ . Therefore, we have:
$$
\chi (\mathcal {R}, k) = \Pi_ {i = 1} ^ {k - 1} \left(1 - \alpha (\mathcal {R} (i)) \cdot (1 - \lambda_ {i})\right).
$$
CCM [17]. The click chain model (CCM) is a generalization of the dependent click model where continuing to examine the results before a click is not deterministic, i.e.  $P(E_{j + 1} = 1 \mid E_j =$
Table 8: Performance of CQL algorithm with different state embedding. The experiments are conducted on Yahoo! set 1 and click models are PBM, CASCADE, UBM, DCM, and CCM
CLICK MODEL	STATE EMBED	ERR@3	ERR@5	ERR@10	nDCG@3	nDCG@5	nDCG@10
PBM	POS	0.417	0.439	0.455	0.668	0.691	0.742
PREDOC	0.413	0.435	0.451	0.657	0.682	0.734
POS+PREDOC	0.418	0.460	0.456	0.663	0.685	0.734
ATTENTION	0.437	0.459	0.478	0.674	0.700	0.753
CASCADE	POS	0.404	0.426	0.442	0.646	0.668	0.724
PREDOC	0.434	0.456	0.472	0.679	0.703	0.754
POS+PREDOC	0.432	0.453	0.470	0.664	0.686	0.740
ATTENTION	0.440	0.461	0.478	0.669	0.696	0.748
UBM	POS	0.420	0.442	0.458	0.678	0.700	0.747
PREDOC	0.410	0.433	0.449	0.666	0.691	0.743
POS+PREDOC	0.416	0.439	0.454	0.676	0.700	0.750
ATTENTION	0.435	0.457	0.473	0.679	0.704	0.756
DCM	POS	0.415	0.437	0.453	0.658	0.680	0.731
PREDOC	0.435	0.457	0.473	0.682	0.706	0.758
POS+PREDOC	0.439	0.461	0.477	0.687	0.709	0.758
ATTENTION	0.437	0.461	0.475	0.675	0.703	0.755
CCM	POS	0.421	0.443	0.458	0.678	0.701	0.748
PREDOC	0.420	0.443	0.459	0.677	0.702	0.754
POS+PREDOC	0.426	0.448	0.464	0.685	0.710	0.759
ATTENTION	0.437	0.460	0.476	0.680	0.706	0.759
ORACLE	0.440	0.462	0.486	0.720	0.739	0.781
Table 9: Comparison of SAC and CQL with optimal alpha on Yahoo. To show the impact of the offline algorithm in off-policy learning to rank, we compare SAC and CQL with different degrees of conservatism. We control the degree of conservatism via the parameter  $\alpha$  (See details in Eq.5 in the main paper). The  $\alpha$ s range from  $\{1e - 3, 5e - 3, 1e - 2, 5e - 2, 1e - 1, 5e - 1, 1e0, 5e0, 1e1, 5e1\}$  and we do grid search to find the optimal  $\alpha$ .
CLICK MODEL	ALG	ERR@K	NDCG@K
K=3	K=5	K=10	K=3	K=5	K=10
PBM	SAC	0.437	0.459	0.478	0.674	0.700	0.753
CQL(optimal α)	0.437	0.460	0.478	0.692	0.715	0.766
CASCADE	SAC	0.440	0.461	0.478	0.670	0.696	0.748
CQL(optimal α)	0.440	0.461	0.479	0.691	0.715	0.765
DCM	SAC	0.436	0.461	0.475	0.675	0.703	0.755
CQL(optimal α)	0.439	0.461	0.477	0.694	0.718	0.767
CCM	SAC	0.437	0.460	0.476	0.680	0.706	0.759
CQL(optimal α)	0.438	0.462	0.479	0.691	0.712	0.766
$1, C_{j} = 0) = \alpha_{1}$ . The probability of continuing after a click is not position dependent, but relevance dependent,  $P(E_{j + 1} \mid C_j = 1) = \alpha_2(1 - R_i) + \alpha_3 R_i$ , where  $R_i$  is the relevance of the  $i^{th}$  document in the rank list. Therefore, the examination probability at each position can be written as:
$$
\chi (\mathcal {R}, k) = \Pi_ {i = 1} ^ {k - 1} (1 - \alpha (\mathcal {R} (i)) \cdot (1 - \alpha_ {2} (1 - \mathcal {R} (i)) - \alpha_ {3} \mathcal {R} (i))
$$
UBM [13]. The user browsing model (UBM) is an extension of the PBM model with some elements of the cascade model. The whole model is position-based, but for the examination probability, it considers previous clicks. Specifically, the examination probability depends not only on the rank of the document  $k$ , but also on the rank of the previously clicked document  $k'$ , which is modeled by a set of parameters  $\gamma_{kk'}$ , i.e.  $P(E_k = 1 \mid C_1 = c_1, \dots, C_{k-1} = c_{k-1}) = \gamma_{kk'}$ , where  $k'$  is the rank of the previous clicked document or 0 if none of them was clicked, i.e.  $k' = \max \{r \in \{0, \dots, k-1\}\}$ :
Table 10: Comparison of different logging policies on Web10k. On each click model, we compare three different logging policies: SVMRank trained with  $1\%$  train data, SVMRank trained with  $0.01\%$  train data, and random policy.
CLICK MODEL	LOGGING	ERR@K	NDCG@K
K=3	K=5	K=10	K=3	K=5	K=10
PBM	por=1%	0.257	0.281	0.303	0.369	0.380	0.406
por=0.01%	0.214	0.239	0.262	0.324	0.339	0.367
rand	0.111	0.130	0.151	0.159	0.175	0.201
CASCADE	por=1%	0.255	0.280	0.301	0.366	0.378	0.404
por=0.01%	0.214	0.239	0.262	0.326	0.340	0.368
rand	0.117	0.136	0.157	0.163	0.180	0.208
DCM	por=1%	0.254	0.278	0.300	0.369	0.380	0.405
por=0.01%	0.212	0.238	0.260	0.324	0.338	0.367
rand	0.121	0.139	0.159	0.167	0.178	0.204
CCM	por=1%	0.255	0.280	0.301	0.373	0.384	0.408
por=0.01%	0.214	0.239	0.261	0.324	0.339	0.367
rand	0.123	0.142	0.162	0.163	0.179	0.208
$$
c _ {r} = 1
$$
$$
\chi (\mathcal {R}, k) = \gamma_ {k k ^ {\prime}}
$$
Zoghi et al. [63] showed that PBM and CM satisfied Assumption 3.1 where optimal ranking list leads to optimal total clicks. In our future work, it would be interesting to show that other click models also satisfy this assumption.
Figure 2: NDCG@10 and ERR@10 of CUOLR (with CQL) on Yahoo! dataset set 1, with different conservative parameters  $\alpha$ s. Each row is the performance of a metric (ERR, NDCG@3,5,10) on the four click models (PBM, CASCADE, DCM, CCM).