SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Off-policy learning can also be directly built on off-policy evaluation. Several work [36, 46] also propose to control the variance of the estimator caused by small logging probabilities through instance reweighing. Again, they directly use the estimated logging policy for correction, and thus performed worse than UIS as observed in our experiments. A recent study [29] assumed additional structure in the action space and proposed the marginalized IPS. Instead, our work considers the uncertainty in the estimated logging policy and thus does not add any new assumptions about the problem space.
Uncertainty-aware learning. Estimation uncertainty has been extensively studied [45, 50, 1]. In the context of on-policy reinforcement leaning and bandits [1, 48, 49], the use of uncertainty aims to strike a balance between exploration and exploitation by adopting an optimistic approach (i.e., UCB in bandits). One the other hand, most research on offline reinforcement learning/bandits [43, 4, 6] tends to be more conservative, employing techniques such as Lower Confidence Bounds (LCB) or penalizing out-of-distribution states and actions based on uncertainty to address extrapolation errors. However, these principles differ fundamentally from UIPS, which directly minimizes the mean square error of off-policy evaluation. The closed-form solution of the resulting per-instance weight in UIPS reflects how uncertainty contributes to the policy evaluation error. Moreover, Our UIPS-O and UIPS-P baselines leverage uncertainties using the two aforementioned general principles respectively. However, empirical findings indicate that blindly penalizing or boosting samples based on uncertainty is problematic. Proper correction depends on both uncertainty in logging policy estimation and the actual value of estimated logging probabilities.
In this paper, we propose a Uncertainty-aware Inverse Propensity Score estimator (UIPS) to explicitly model the uncertainty of the estimated logging policy for improved off-policy learning. UIPS weighs each logged instance to reduce its policy evaluation error, where the optimal weights have a closed-form solution derived by minimizing the upper bound of the resulting estimator's mean squared error (MSE) to its ground-truth value. An improved policy is then obtained by optimizing the resulting estimation. Extensive experiments on synthetic and three real-world datasets as well as the theoretical convergence guarantee demonstrate the efficiency of UIPS.
As demonstrated in this work, explicitly modeling the uncertainty of the estimated logging policy is crucial for effective off-policy learning; but the best use of this uncertainty is not to simply down-weigh or drop instances with uncertain estimations, but to balance it with the actually estimated logging probabilities in a per-instance basis. As our future work, it is promising to investigate how UIPS can be extended to value-based learning methods, e.g., actor-critics. And on the other hand, it is also important to analyze how tight our upper bound analysis of policy evaluation error is; and if possible, find new ways to tighten it for improvements.
Algorithm 1 UIPS
1: Input: The logged dataset  $D \coloneqq \{(x_{n},a_{n},r_{x_{n},a_{n}})|n\in [N]\}$ , the estimated logging policy model  $\hat{\beta} (a|\pmb {x}) = \frac{\exp(f_{\pmb{\theta}}(\pmb{x},a))}{\sum_{a^{\prime}}\exp(f_{\pmb{\theta}}(\pmb{x},a^{\prime}))}$ , dimension  $d$ .
2: Initialize  $M_D = \overline{I}_{d\times d}$
3: for  $n = 1$  to  $N$  do
4:  $M_D = M_D + \nabla_\theta f_\theta(\pmb{x}_n, a_n) \nabla_\theta f_\theta(\pmb{x}_n, a_n)^\top \quad \triangleright M_D$  for uncertainty calculation.
5: end for
6: for  $n = 1$  to  $N$  do
7:  $U_{\pmb{x}_{n},a_{n}} = \sqrt{\nabla_{\pmb{\theta}}f_{\pmb{\theta}}(\pmb{x}_{n},a_{n})^{\top}} M_{D}^{-1}\nabla_{\pmb{\theta}}f_{\pmb{\theta}}(\pmb{x}_{n},a_{n})$
8: end for
9: while not converge do
0: for  $n = 1$  to  $N$  do
1: Calculate  $\phi_{\pmb{x}_n,a_n}^*$  as in Theorem 3.2.
2: Calculate gradients as in Eq.(5) and updating  $\pi_{\vartheta}(a|\pmb {x})$
3: end for
4: end while
5: Output:The learnt policy  $\pi_{\varphi}(a|\mathbf{x})$
7.1 Notations and Algorithm Framework.
For ease of reading, we list important notations in Table 5 and summarize the main framework of the proposed UIPS in Algorithm 1.
Notation	Description
X	context space
A	action set
x∈Rd	context vector
a	action
rx,a	reward
π(a|x)	targeted policy to evaluate
β*(a|x)	the unknown ground-truth logging policy
β(a|x)	the estimated logging policy
V(π)	value function
D := {(xn, an, rxn, an}|n∈[N]}	logged dataset containing N samples
φ*	the optimal uncertainty-aware weight
fθ*(x,a)	the unknown ground-truth function that generates β*(a|x) = exp(fθ*(x,a)) / ∑a' exp(fθ*(x,a'))
fθ(x,a)	the estimate of fθ*(x,a) that generates β(a|x)
Bx,a	confidence interval of β(a|x)
Ux,a	uncertainty defined as |fθ*(x,a) - fθ(x,a)| ≤ γUx,a
g(xn,an)	gradient of fθ(x,a) regarding to the last layer.
Table 5: Notations
Computation Cost. The additional computation cost of UIS over IPS comes from two parts:
- Pre-calculate uncertainties (line 1-5 in Algorithm 1): This part calculates uncertainty of the logging probability for each  $(s, a)$  pair, and it only needs to be executed once. The computational cost of this step is  $O(Nd^{2} + d^{3})$ , where  $O(Nd^{2})$  is for calculating uncertainties in each  $(s, a)$  pair and  $O(d^{3})$  is for matrix inverse.
- Calculate  $\phi_{\pmb{x},a}^{*}$  during training (line 8 in Algorithm 1): It only takes O(1) time, the same computational cost as calculating IPS score.
Note that calculating the logging probability for each sample, which is essential for both UIS and IPS, takes  $O(Nd|\mathcal{A}|)$  time. Since the dimension  $d$  is usually much less than action size  $|\mathcal{A}|$  and sample size  $N$ , UIS does not introduce significant computational overhead compared to the original IPS solution.
7.2 Derivation of confidence interval of logging probability.
Given that  $|f_{\pmb{\theta}^*}(\pmb{x}, a) - f_{\pmb{\theta}}(\pmb{x}, a)| \leq \gamma U_{\pmb{x}, a}$  holds with probability at least  $1 - \sigma$  and
$$
\beta^ {*} (a | \boldsymbol {x}) = \frac {\exp (f _ {\boldsymbol {\theta} ^ {*}} (\boldsymbol {x} , a))}{Z ^ {*}}, \hat {\beta} (a | \boldsymbol {x}) = \frac {\exp (f _ {\boldsymbol {\theta}} (\boldsymbol {x} , a))}{\hat {Z}},
$$
where  $Z^{*} = \sum_{a^{\prime}}\exp (f_{\theta^{*}}(a^{\prime}|\pmb {x}))$  and  $\hat{Z} = \sum_{a^{\prime}}\exp (f_{\theta}(a^{\prime}|\pmb {x}))$ , we can get that with probability at least  $1 - \sigma$ :
$$
\begin{array}{l} \left| f _ {\boldsymbol {\theta} ^ {*}} (\boldsymbol {x}, a) - f _ {\boldsymbol {\theta}} (\boldsymbol {x}, a) \right| \leq \gamma U _ {\boldsymbol {x}, a} \\ \Leftrightarrow f _ {\boldsymbol {\theta}} (\boldsymbol {x}, a) - \gamma U _ {\boldsymbol {x}, a} \leq f _ {\boldsymbol {\theta} ^ {*}} (\boldsymbol {x}, a) \leq f _ {\boldsymbol {\theta}} (\boldsymbol {x}, a) + \gamma U _ {\boldsymbol {x}, a} \\ \Leftrightarrow \exp (f _ {\boldsymbol {\theta}} (\boldsymbol {x}, a)) \exp (- \gamma U _ {\boldsymbol {x}, a}) \leq \exp (f _ {\boldsymbol {\theta} ^ {*}} (\boldsymbol {x}, a)) \leq \exp (f _ {\boldsymbol {\theta}} (\boldsymbol {x}, a)) \exp (\gamma U _ {\boldsymbol {x}, a}) \\ \stackrel {(1)} {\Leftrightarrow} \hat {Z} \hat {\beta} (a | \boldsymbol {x}) \exp (- \gamma U _ {\boldsymbol {x}, a}) \leq \exp \left(f _ {\boldsymbol {\theta} ^ {*}} (\boldsymbol {x}, a)\right) \leq \hat {Z} \hat {\beta} (a | \boldsymbol {x}) \exp (\gamma U _ {\boldsymbol {x}, a}) \\ \stackrel {(2)} {\Leftrightarrow} \frac {\hat {Z} \exp (- \gamma U _ {\boldsymbol {x} , a})}{Z ^ {*}} \hat {\beta} (a | \boldsymbol {x}) \leq \beta^ {*} (a | \boldsymbol {x}) \leq \frac {\hat {Z} \exp (\gamma U _ {\boldsymbol {x} , a})}{Z ^ {*}} \hat {\beta} (a | \boldsymbol {x}) \\ \end{array}
$$
The step labeled as (1) is due to the modelling of  $\hat{\beta}(a|\pmb{x})$ . And the step labeled as (2) is because  $Z^{*}$  is a positive constant independent of  $\hat{\beta}(a|\pmb{x})$ . Thus with probability at least 1- $\delta$ , we have  $\beta^{*}(a|\pmb{x}) \in B_{\pmb{x},a}$  and
$$
\boldsymbol {B} _ {\boldsymbol {x}, a} = \left[ \frac {\hat {Z} \exp (- \gamma U _ {\boldsymbol {x} , a})}{Z ^ {*}} \hat {\beta} (a | \boldsymbol {x}), \frac {\hat {Z} \exp (\gamma U _ {\boldsymbol {x} , a})}{Z ^ {*}} \hat {\beta} (a | \boldsymbol {x}) \right].
$$
7.3.1 Synthetic Data
Data generation. Given the ground-truth logging policy  $\beta^{*}(a|x)$ , we generate the logged dataset as follows. For each sample in train set, we first get the embedded context vector  $\pmb{x}$  from its original feature vector  $\tilde{\pmb{x}}$ . We then sample an action  $a$  according to  $\beta^{*}(a|x)$ , and obtain the reward  $r_{\pmb{x},a} = \pmb{y}_{\tilde{\pmb{x}},a}$ , resulting bandit feedback  $(x,a,r_{x,a})$ , where  $\pmb{y}_{\tilde{\pmb{x}},a}$  is the label of class  $a$  under the original feature vector  $\tilde{\pmb{x}}$ . We repeat above process  $N$  times to collect the logged dataset. In our experiments, we take  $d = 64$ ,  $N = 100$ .
We model the logging policy as in Eq.(9), where  $\{\theta_{a}\}$  are the parameters to be estimated. To train the logging policy, we take all samples in the logged dataset  $D$  as positive instances, and randomly sample non-selected actions as negative instances as in [8].
7.3.2 Real-world Data
Statistics of datasets. The statistics of three real-world recommendation datasets with unbiased data can be found in Table 6.
Table 6: The statistics of three real-world datasets.
Dataset	#User	#Item	#Biased Data	#Unbiased Data
Yahoo R3	15,400	1,000	311,704	54,000
Coat	290	300	6,960	4,640
KuaiRec	7,176	10,729	12,530,806	4,676,570
All these datasets contain a set of biased data collected from users' interactions on the platform, and a set of unbiased data collected from a randomized controlled trial where items are randomly selected. As in [10], on each dataset, the biased data is used for training, and the unbiased data is for testing, with a small part of unbiased data split for validation purpose (5% on Yahoo R3 and Coat,
and  $15\%$  on KuaiRec). We take the reward as 1 if: (1) the rating is larger than 3 in Yahoo! R3 and Coat datasets; (2) the user watched more than  $70\%$  of the video in KuaiRec. Otherwise, the reward is labeled as 0.
We adopted a two-tower neural network architecture to implement both the logging and learning policy, as shown in Figure 2. For the learning policy, the user representation and item representation are first modelled through two separate neural networks (i.e., the user tower and the item tower), and then their element-by-element product vector is projected to predict the user's preference for the item. We then re-use the user state generated from the user tower of the learning policy, and model the logging policy with another separate item tower, following [8]. We also block gradients to prevent the logging policy learning interfering the user state of the learning policy. In each learning epoch, we will first estimate the logging policy, and then take the estimated logging probabilities as well as their uncertainties to optimize the learning policy.
Figure 2: Model architecture of the logging and the learning policy in real-world datasets
7.3.3 Implementation details.
To facilitate hyper-parameter tuning, we disentangled two  $\eta$ s in two the terms of  $\phi_{\boldsymbol{x},a}^{*}$  in Theorem 3.2, and introduce  $\eta_{1}$  and  $\eta_{2}$  to represent  $\eta$  in the first and second term respectively in our implementation. This is due to the scale of  $\eta$  in the first term is closely related to the scale of  $\lambda$ , with  $\lambda / \sqrt{\eta}$  as the truly effective hyper-parameter. But the scale of  $\eta$  in the second term is independent from  $\lambda$ .
Moreover, while  $\lambda = \mathbb{E}_{\pi_{\theta}}\left[r_{\pmb{x},a}^{2}\frac{\pi_{\theta}(a|\pmb{x})}{\beta^{*}(a|\pmb{x})}\right]$  depends on  $\pi_{\theta}$  as discussed in Eq.(6), we cannot adaptively set the value of  $\lambda$  since the ground-truth logging policy  $\beta^{*}(a|\pmb {x})$  is unknown. However, we have:
$$
\lambda := E _ {\pi} \left[ r _ {\boldsymbol {x}, a} ^ {2} \frac {\pi_ {\theta} (a | \boldsymbol {x})}{\beta^ {*} (a | \boldsymbol {x})} \right] \leq \left(\sum_ {a} \pi_ {\theta} (a | \boldsymbol {x}) ^ {4}\right) \left(\sum_ {a} \frac {r _ {\boldsymbol {x} , a} ^ {4}}{\beta^ {*} (a | \boldsymbol {x}) ^ {2}}\right) \leq \left(\sum_ {a} \frac {r _ {\boldsymbol {x} , a} ^ {4}}{\beta^ {*} (a | \boldsymbol {x}) ^ {2}}\right),
$$
where the first inequality is due to the Cauchy-Schwarz inequality and the second inequality is because  $\sum_{a}\pi_{\theta}(a|\pmb {x})^{4}\leq \sum_{a}\pi_{\theta}(a|\pmb {x}) = 1$  with  $\pi_{\theta}(a|\pmb {x})\in [0,1]$ . We denote  $\tilde{\lambda} = \left(\sum_{a}\frac{r_{\pmb{x},a}^4}{\beta^*(a|\pmb{x})^2}\right)$ , which is dataset-specific constant and independent from  $\pi_{\theta}$ . By replacing  $\lambda$  in Eq.(6) with  $\tilde{\lambda}$ , we can still minimize an upper bound of  $\mathrm{MSE}(\hat{V}_{\mathrm{UPS}}(\pi_{\vartheta}))$ , which ensures that the result of our analysis still holds. Thus, considering ease of computation and efficiency, we take a fixed  $\lambda$  during our policy learning.
We then use grid search to select hyperparameters based on the model's performance on the validation dataset: the learning rate was searched in  $\{1e^{-5}, 1e^{-4}, 1e^{-3}, 1e^{-2}\}$ ;  $\lambda, \gamma, \eta_1$  were searched in  $\{0.5, 0.1, 1, 2, 5, 10, 15, 20, 25, 30, 40, 50\}$ . And  $\eta_2$  was searched in  $\{1, 10, 100, 1000\}$ . For baseline algorithms, we also performed a similar grid search as mentioned above, and the search range follows the original papers.
Ablation study: hyper-parameter tuning. Although UIPS has four hyperparameters  $(\lambda, \gamma, \eta_1, \text{and} \eta_2)$ , one only needs to carefully finetune two of them, i.e.,  $\gamma$  and  $\eta_1^2 / \lambda$ , to obtain good performance of UIPS. This is because:
(a) Precision@5
(b) Recall@5
(c)NDCG@5
Figure 3: Effect of  $\lambda$  and  $\gamma$  on synthetic dataset with  $\tau = 0.5$
-  $\eta_{2}$  acts as a capping threshold to ensure  $\phi_{x,a}^{*} \leq 2\eta_{2}$  holds even when the corresponding propensity scores are very low. Hence, it should be set to a reasonably large value (e.g., 100).
- The key component (i.e., the first term) of  $\phi_{\pmb{x},a}^{*}$  can be rewritten in the following way. While all  $(\pmb{x},a)$  pairs will be multiplied by  $\phi_{\pmb{x},a}^{*}$ ,  $\eta_{1}$  in the numerator will not affect final performance too much, and the key is to find a good value of  $\eta_1^2 /\lambda$  to balance the two terms in the denominator:
$$
\eta_ {1} / \left[ \exp (- \gamma U _ {\boldsymbol {x}, a}) + \frac {\eta_ {1} ^ {2} / \lambda \cdot \pi_ {\boldsymbol {\vartheta}} (a | \boldsymbol {x}) ^ {2}}{\hat {\beta} (a | \boldsymbol {x}) ^ {2} \exp (- \gamma U _ {\boldsymbol {x} , a})} \right].
$$
Thus with  $\eta_{1}$  and  $\eta_{2}$  fixed, the effect of hyper-parameter  $\gamma$  and  $\lambda$  on precision, recall as well as NDCG can be found in Figure 3. We can observe that to make UIPS excel,  $B_{\boldsymbol{x},a}$  needs to be of high confidence, e.g.,  $\gamma = 25$  performed the best on the dataset with  $\tau = 0.5$ . Moreover, the threshold  $\sqrt{\lambda} /\eta_1$  cannot be too small or too large.
7.4 Experiments on the doubly robust estimators.
The doubly robust (DR) estimator [15], which is a hybrid of direct method (DM) estimator and inverse propensity score (IPS) estimator, is also widely used for off-policy evaluation. More specifically, let  $\hat{\eta}:\mathcal{X}\times \mathcal{A}\to R$  be the imputation model in DM that estimates the reward of action  $a$  under context vector  $\pmb{x}$ , and  $\hat{\beta} (a|\pmb {x})$  be the estimated logging policy in the IPS estimator. The DR estimator evaluates policy  $\pi$  based on the logged dataset  $D\coloneqq \{(x_{n},a_{n},r_{x_{n},a_{n}})|n\in [N]\}$ , by:
$$
\hat {V} _ {\mathrm {D R}} (\pi) = \hat {V} _ {\mathrm {D M}} (\pi) + \frac {1}{N} \sum_ {n = 1} ^ {N} \frac {\pi \left(a _ {n} \mid \boldsymbol {x} _ {n}\right)}{\hat {\beta} \left(a _ {n} \mid \boldsymbol {x} _ {n}\right)} \left(r _ {\boldsymbol {x} _ {n}, a _ {n}} - \hat {\eta} \left(\boldsymbol {x} _ {n}, a _ {n}\right)\right) \tag {10}
$$
where  $\hat{V}_{\mathrm{DM}}(\pi)$  is the DM estimator:
$$
\hat {V} _ {\mathrm {D M}} (\pi) = \frac {1}{N} \sum_ {n = 1} ^ {N} \sum_ {a \in \mathcal {A}} \pi (a | \boldsymbol {x} _ {n}) \hat {\eta} (\boldsymbol {x} _ {n}, a). \tag {11}
$$
Again assume the policy  $\pi(a|x)$  is parameterized by  $\vartheta$ , the REINFORCE gradient of  $\hat{V}_{\mathrm{DR}}(\pi_{\vartheta})$  with respect to  $\vartheta$  can be readily derived as follows:
$$
\begin{array}{l} \nabla_ {\boldsymbol {\vartheta}} \hat {V} _ {\mathrm {D R}} (\pi_ {\boldsymbol {\vartheta}}) = \frac {1}{N} \sum_ {n = 1} ^ {N} \left(\sum_ {a \in \mathcal {A}} \pi_ {\boldsymbol {\vartheta}} (a | \boldsymbol {x} _ {n}) \hat {\eta} (\boldsymbol {x} _ {n}, a) \nabla_ {\boldsymbol {\vartheta}} \log (\pi_ {\boldsymbol {\vartheta}} (a | \boldsymbol {x} _ {n}))\right) \\ + \frac {1}{N} \sum_ {n = 1} ^ {N} \left(\frac {\pi (a _ {n} | \boldsymbol {x} _ {n})}{\hat {\beta} (a _ {n} | \boldsymbol {x} _ {n})} \left(r _ {\boldsymbol {x} _ {n}, a _ {n}} - \hat {\eta} (\boldsymbol {x} _ {n}, a _ {n})\right) \nabla_ {\boldsymbol {\vartheta}} \log \left(\pi_ {\boldsymbol {\vartheta}} (a _ {n} | \boldsymbol {x} _ {n})\right)\right). \tag {12} \\ \end{array}
$$
The imputation model  $\hat{\eta}(\boldsymbol{x}, a)$  is pre-trained following previous work [22] with the same neural network architecture as the logging policy model. Besides the standard DR estimator, we also adapt
Table 7: Experiment results on synthetic datasets. The best and second best results are highlighted with bold and underline respectively. Two  $p$ -values are calculated: (1)  $p$ -value (UIPSDR): The  $p$ -value under the t-test between UIPSDR and the best DR baseline on each dataset; (2)  $p$ -value (UIPS): The  $p$ -value under the t-test between UIPS and the best DR baseline on each dataset.
τ = 0.5	τ = 1	τ = 2
Algorithm	P@5	R@5	NDCG@5	P@5	R@5	NDCG@5	P@5	R@5	NDCG@5
BIPS-Cap	0.5515±2e-3	0.1553±8e-4	0.6031±2e-3	0.5526±2e-3	0.1561±6e-4	0.6016±1e-3	0.5409±3e-3	0.1529±9e-4	0.5901±2e-3
UIPS	0.5589±3e-3	0.1583±9e-4	0.6095±3e-3	0.5572±2e-3	0.1571±8e-4	0.6074±2e-3	0.5432±3e-3	0.1534±8e-4	0.5946±2e-3
DR	0.3846±3e-2	0.1082±8e-3	0.4684±3e-2	0.3631±3e-2	0.1017±9e-3	0.4494±3e-2	0.3560±3e-2	0.0995±7e-3	0.4470±2e-3
MinVarDR	0.3212±3e-2	0.0908±8e-3	0.4062±3e-2	0.3240±5e-2	0.0903±1e-2	0.3905±5e-2	0.3234±5e-2	0.0910±1e-2	0.4059±4e-2
ShrinkageDR	0.4139±2e-2	0.1161±7e-3	0.4969±3e-2	0.3944±3e-2	0.1101±8e-3	0.4797±2e-2	0.4080±3e-2	0.1135±7e-3	0.4901±2e-2
UIPSDR	0.4278±2e-2	0.1200±6e-3	0.5069±2e-2	0.4008±2e-2	0.1126±7e-3	0.4847±2e-2	0.4144±2e-2	0.1162±8e-3	0.4972±2e-2
p-value (UIPSDR)	2e-1	2e-1	3e-1	6e-1	4e-1	6e-1	6e-1	4e-1	5e-1
p-value (UIPS)	6e-13	4e-13	4e-12	2e-12	1e-12	5e-12	8e-12	8e-12	2e-11
Table 8: Experiment results on real-world unbiased datasets. The best and second best results are highlighted with **bold** and **underline** respectively. Two  $p$ -values are calculated: (1)  $p$ -value (UIPSDR): The  $p$ -value under the t-test between UIPSDR and the best DR baseline on each dataset; (2)  $p$ -value (UIPS): The  $p$ -value under the t-test between UIPS and the best DR baseline on each dataset.
Yahoo	Coat	KuaiRec
Algorithm	P@5	R@5	NDCG@5	P@5	R@5	NDCG@5	P@50	R@50	NDCG@50
BIPS-Cap	0.2808±2e-3	0.7576±5e-3	0.6099±8e-3	0.2758±6e-3	0.4582±7e-3	0.4399±9e-3	0.8750±3e-3	0.0238±7e-5	0.8788±5e-3
UIPS	0.2868±2e-3	0.7742±5e-3	0.6274±5e-3	0.2877±3e-3	0.4757±5e-3	0.4576±8e-3	0.9120±1e-3	0.0250±5e-5	0.9174±7e-4
DR	0.2670±2e-3	0.7174±6e-3	0.5636±6e-3	0.2884±3e-3	0.4760±5e-3	0.4541±5e-3	0.8794±1e-2	0.0240±5e-4	0.8824±2e-2
MinVarDR	0.2272±5e-3	0.5989±1e-2	0.4525±1e-2	0.2704±4e-3	0.4434±9e-3	0.4137±6e-3	0.8640±7e-3	0.0235±2e-4	0.8657±7e-3
ShrinkageDR	0.2697±2e-3	0.7226±6e-3	0.5713±5e-3	0.2895±4e-3	0.4749±6e-3	0.4526±6e-3	0.8778±2e-2	0.0239±5e-4	0.8800±2e-2
UIPSDR	0.2721±1e-3	0.7294±6e-3	0.5750±5e-3	0.2946±4e-3	0.4854±8e-3	0.4647±8e-3	0.8849±1e-2	0.0242±4e-4	0.8896±1e-2
p-value (UIPSDR)	1e-2	2e-2	1e-1	7e-3	5e-3	2e-3	4e-1	4e-1	3e-1
p-value (UIPS)	1e-12	6e-14	6e-15	3e-1	8e-1	1e-1	2e-6	2e-6	1e-3
UIPS and the best two baselines on off-policy evaluation estimator (i.e., MinVar and Shrinkage based on the results in Table 1 and Table 4) to doubly robust setting using the same imputation model.
Table 7 and Table 8 report the empirical performance of the learned policy on the synthetic datasets and three real-world datasets respectively. For ease of comparison, we also include the experiment results of BIPS-Cap and UIPS on each dataset in these two tables. Two  $p$ -values are also provided: (1)  $p$ -value (UIPSDR): The  $p$ -value under the t-test between UIPSDR and the best DR baseline on each dataset; (2)  $p$ -value (UIPS): The  $p$ -value under the t-test between UIPS and the best DR baseline on each dataset. From Table 7 and Table 8, we can first observe that DR cannot consistently outperform BIPS-Cap: It outperformed BIPS-Cap on the Coat and KuaiRec dataset, while achieving much worse performance on the synthetic datasets and Yahoo dataset. This is because the imputation model also plays a very important role in gradient calculation as shown in Eq.(12). Its accuracy greatly affects policy learning. When the imputation model is sufficiently accurate, for example, on the Coat dataset with only 300 actions, incorporating the DM estimator not only led to better performance of DR over IPS, but also improved performance of UIPSDR over UIPS. And in particular, in this situation UIPSDR performed better than DR with the gain being statistically significant. When the imputation model is not accurate enough, for example, on the KuaiRec dataset with a large action space but sparse reward feedback, DR is still worse than UIPS, and UIPSDR also performs worse than UIPS due to the distortion of the imputation model.
7.5 Difference against DICE-type algorithms.
The DICE line of work [26, 47, 9] is proposed for off-policy correction in the multi-step RL setting with the environment following the Markov Decision Process (MDP) assumption. Although the DICE line of work does not require the knowledge of the logging policy, it is fundamentally different from our work. Given a logged dataset  $D \coloneqq \{(s_t, a_t, r_{s_t, a_t}, s_{t+1})\}$  collected from an unknown logging policy  $\beta^*(a_t|s_t)$ , DICE-type algorithms propose to directly estimate the discounted stationary distribution correction  $w_{\pi/D}(s_t, a_t) = \frac{d^\pi(s_t, a_t)}{d^\mathcal{D}(s_t, a_t)}$  to replace the product-based off-policy correction weight  $\prod_t \frac{\pi(a_t|s_t)}{\beta^*(a_t|s_t)}$ , which suffers high variance due to the series of products. While the estimation of  $w_{\pi/D}(s_t, a_t)$  is agnostic to the logging policy, it highly depends on two assumptions: (1) Environment follows an MDP, i.e.,  $s_{t+1} \sim T(s_t, a_t)$  with  $T(\cdot, \cdot)$  denoting the state transition function; (2) Each logged sample should be of a state-action next-state tuple  $(s_t, a_t, r_{s_t, a_t}, s_{t+1})$  that contains state transition information.
Table 9: Empirical performance of DICE-S on synthetic datasets.
τ=0.5	τ=1	τ=2
Algorithm	P@5	R@5	NDCG@5	P@5	R@5	NDCG@5	P@5	R@5	NDCG@5
BIPS-Cap	0.5515±2e-3	0.1553±8e-4	0.6031±2e-3	0.5526±2e-3	0.1561±6e-4	0.6016±1e-3	0.5409±3e-3	0.1529±9e-4	0.5901±2e-3
DICE-S	0.5416±4e-3	0.1520±1e-3	0.5968±4e-3	0.5508±2e-3	0.1553±7e-4	0.6010±2e-3	0.5403±2e-3	0.1526±7e-4	0.5903±1e-3
UIPS	0.5589±3e-3	0.1583±9e-4	0.6095±3e-3	0.5572±2e-3	0.1571±8e-4	0.6074±2e-3	0.5432±3e-3	0.1534±8e-4	0.5946±2e-3