SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where  $\pmb{g}(\pmb{x}_n, a_n)$  is the gradient of  $f_{\pmb{\theta}}(\pmb{x}_n, a_n)$  with respect to the neural network's last layer's parameter  $\pmb{\theta}_w \subset \pmb{\theta}$ , i.e.,  $\pmb{g}(\pmb{x}_n, a_n) = \nabla_{\pmb{\theta}_w} f_{\pmb{\theta}}(\pmb{x}_n, a_n)$ . And  $M_D = \sum_{n=1}^N \pmb{g}(\pmb{x}_n, a_n) \pmb{g}(\pmb{x}_n, a_n)^\top$ , implying  $U_{\pmb{x}_n, a_n} = \sqrt{\pmb{g}(\pmb{x}_n, a_n)^\top \pmb{M}_D^{-1} \pmb{g}(\pmb{x}_n, a_n)}$ .
3.2 Convergence of policy learning under UIPS
The following theorem provides the convergence result for UIPS, which converges to a stationary point of the expected reward function. The proof is provided in Appendix 7.9.
Theorem 3.4. Denote  $G_{\mathrm{max}}$  and  $\Phi$  as the maximum value of  $\| \frac{\partial \pi_{\vartheta}(a|\pmb{x})}{\partial \vartheta} \|$  and  $\mathbb{E}_{\beta^*} \left[ \frac{\pi_{\vartheta}^2(a|\pmb{x})}{\hat{\beta}^2(a|\pmb{x})} (\phi_{\pmb{x},a}^*)^2 \right]$  respectively, i.e.,  $\| \frac{\partial \pi_{\vartheta}(a|\pmb{x})}{\partial \vartheta} \| \leq G_{\mathrm{max}}$  and  $\mathbb{E}_{\beta^*} \left[ \frac{\pi_{\vartheta}^2(a|\pmb{x})}{\hat{\beta}^2(a|\pmb{x})} (\phi_{\pmb{x},a}^*)^2 \right] \leq \Phi$ . And denote  $V_{\mathrm{max}}$  as the finite maximum expected reward that can be achieved, and  $\varphi_{\mathrm{max}} = \max_{\pmb{x},a} \left\{ \left| \frac{\beta^*(a|\pmb{x})}{\hat{\beta}(a|\pmb{x})} \phi_{\pmb{x},a}^* - 1 \right| \right\}$ . Assume that the expected reward of  $\pi_{\vartheta}$ , i.e.,  $V(\pi_{\vartheta})$ , is a differentiable and L-smooth function w.r.t  $\vartheta$ . Denote the policy parameters obtained by Eq.(5) at iteration  $k \in [K]$  as  $\vartheta_k$ , then  $\varphi_{\mathrm{max}} \in (0,1)$  and
$$
\frac {1}{K} \sum_ {k = 1} ^ {K} \mathbb {E} [ \| \nabla V (\pi_ {\vartheta_ {k}}) \| ] ^ {2} \leq \frac {2 L V _ {\max}}{K (1 - \varphi_ {\max})} + \left(L + \frac {2 V _ {\max}}{(1 - \varphi_ {\max})}\right) \frac {G _ {\max} \sqrt {\Phi}}{\sqrt {K}},
$$
where  $\nabla V(\pi_{\vartheta})$  is the true policy gradient under ground-truth logging probability, i.e.,  $\nabla V(\pi_{\vartheta}) = E_{\beta^{*}}[\frac{\pi_{\vartheta}(a|\pmb{x})}{\beta^{*}(a|\pmb{x})} r_{\pmb{x},a}\nabla_{\vartheta}\log (\pi_{\vartheta}(a|\pmb {x}))]$ .
Theorem 3.4 shows that, as  $K \to \infty$  and with  $1 / (1 - \varphi_{\max})$  and  $\Phi$  being controlled, UISP leads policy update to converge to a stationary point where the true policy gradient  $\nabla V(\pi_{\vartheta_k})$  is zero. And fortunately, UISP is effective in controlling both  $1 / (1 - \varphi_{\max})$  and  $\Phi$ . Specifically, we denote  $\varphi_{\pmb{x},a} = \left|\frac{\beta^{*}(a|\pmb{x})}{\hat{\beta}(a|\pmb{x})}\phi_{\pmb{x},a}^{*} - 1\right|$  and  $\Phi_{x,a} = \frac{\pi_{\vartheta}^2(a|\pmb{x})}{\hat{\beta}^2(a|\pmb{x})} (\phi_{\pmb{x},a}^*)^2$ . It is clear to note that  $\lambda \varphi_{x,a}^2 + \Phi_{x,a}$  corresponds to the objective in Eq.(6) for deriving  $\phi_{\pmb{x},a}^{*}$  for each sample  $(\pmb{x},a)$ . In other words, UISP selects  $\{\phi_{\pmb{x},a}^{*}\}$  to minimize  $\varphi_{\max} = \max \{\varphi_{x,a}\}$  and  $\Phi = \mathbb{E}_{\beta^{*}}[\Phi_{\pmb{x},a}]$ , which directly accelerates the policy converge to a stationary point with the true policy gradient being zero.
In the case of BIPS in Eq.(3), we have  $\phi_{\pmb{x},a} \equiv 1$ . Although  $\Phi$  may be large due to small logging probabilities, the more concerning issue is that the requirement  $\varphi_{\max} \in (0,1)$  is no longer satisfied when  $\beta^{*}(a|\pmb{x}) \geq 2\hat{\beta}(a|\pmb{x})$ , which may happen with a non-negligible probability. Hence, the convergence of policy learning under  $\hat{V}_{\mathrm{BIPS}}$  is no better than that under UIPS.
4 Empirical evaluations
We evaluate UIPS on both synthetic data and three real-world datasets with unbiased collection. We compare UIPS with the following baselines, which can be grouped into five categories:
- Cross-Entropy (CE): A supervised learning method with the cross-entropy loss over its softmax output. No off-policy correction is performed in this method.
- BIPS-Cap [8]: The off-policy learning solution under the BIPS estimator in Eq.(3). The estimated propensity scores are further suppressed to control variance, i.e., taking  $\min \left(c, \frac{\pi_{\theta}(a|x)}{\hat{\beta}(a|x)}\right)$  as the propensity score. Setting  $c$  to a small value can reduce variance, but introduces bias.
- MinVar & stableVar [46], Shrinkage [36]: This line of work improves off-policy evaluation by reweighing each sample. For example, MinVar and stableVar reweigh each sample by  $\frac{h_{\boldsymbol{x},a}}{\sum_{a'} h_{\boldsymbol{x},a'}}$  with  $h_{\boldsymbol{x},a} = \frac{\hat{\beta}(a|\boldsymbol{x})}{\pi_{\vartheta}(a|\boldsymbol{x})^2}$  and  $h_{\boldsymbol{x},a} = \frac{\sqrt{\hat{\beta}(a|\boldsymbol{x})}}{\pi_{\vartheta}(a|\boldsymbol{x})}$  respectively, since they find that  $\pi_{\vartheta}(a|\boldsymbol{x})^2 / \hat{\beta}(a|\boldsymbol{x})$  is directly related to policy evaluation variance. Su et al. [36] propose to shrink the propensity score by  $\lambda / (\lambda + \frac{\pi_{\vartheta}(a|\boldsymbol{x})^2}{\hat{\beta}(a|\boldsymbol{x})^2})$ , which is a special case of our UIPS with  $U_{\boldsymbol{x},a} = 0$  and  $\eta = 1$ . All these methods simply treat  $\hat{\beta}(a|\boldsymbol{x})$  as  $\beta^*(a|\boldsymbol{x})$ , and none of them consider the uncertainty of  $\hat{\beta}(a|\boldsymbol{x})$ .
- SNIPS [39], BanditNet [16], POEM [38], POXM [23], Adaptive [22]: This line of work aims for more stable and accurate policy learning. For example, SNIPS normalizes the estimator by the sum of propensity scores in each batch. BanditNet extends SNIPS and leverages an additional Lagrangian term to normalize the estimator by an approximated sum of propensity scores of all samples. POEM jointly optimizes the estimator and its variance. POXM controls estimation variance by pruning samples with small logging probabilities. Adaptive proposes a new formulation to utilize negative samples.
- ApproxKNN [5] and IPS-C-TS: The line of work improves off-policy learning by applying calibration to estimated logging probabilities. ApproxKNN utilizes the K-Nearest Neighbor algorithm for calibration, which exhibits the lowest calibration error in [5]. IPS-C-TS, on the other hand, employs temperature scaling, a widely recognized and effective calibration method for probability distribution [13].
Table 1: Experiment results on synthetic datasets. The best and second best results are highlighted with **bold** and **underline** respectively. The  $p$ -value under the t-test between UIPS and the best baseline on each dataset is also provided.
τ = 0.5	τ = 1	τ = 2
Algorithm	P@5	R@5	NDCG@5	P@5	R@5	NDCG@5	P@5	R@5	NDCG@5
IPS-GT	0.5589±1e-3	0.1582±6e-4	0.6093±1e-3	0.5526±2e-3	0.1565±6e-4	0.6007±1e-3	0.5531±2e-3	0.1557±7e-4	0.6037±1e-3
CE	0.5553±6e-4	0.1573±2e-4	0.6037±5e-4	0.5510±6e-4	0.1561±2e-4	0.5995±4e-4	0.5386±2e-3	0.1524±7e-4	0.5874±2e-3
BIPS-Cap	0.5515±2e-3	0.1553±8e-4	0.6031±2e-3	0.5526±2e-3	0.1561±6e-4	0.6016±1e-3	0.5409±3e-3	0.1529±9e-4	0.5901±2e-3
MinVar	0.5340±2e-3	0.1509±6e-4	0.5857±2e-3	0.5282±2e-3	0.1491±7e-4	0.5791±2e-3	0.5036±4e-3	0.1415±1e-3	0.5543±3e-3
stableVar	0.4577±5e-3	0.1310±1e-3	0.5111±2e-3	0.5373±3e-3	0.1523±9e-4	0.5866±3e-3	0.5279±3e-3	0.1492±8e-4	0.5781±3e-3
Shrinkage	0.5526±2e-3	0.1562±7e-4	0.6024±1e-3	0.5499±4e-3	0.1545±1e-3	0.6040±3e-3	0.5347±2e-3	0.1513±6e-4	0.5824±2e-3
SNIPS	0.2616±6e-2	0.0749±2e-2	0.3150±7e-2	0.3538±5e-2	0.0987±1e-2	0.4144±6e-2	0.4379±3e-2	0.1226±9e-3	0.5177±3e-2
BanditNet	0.4011±3e-2	0.1131±8e-3	0.4830±2e-2	0.3894±4e-2	0.1095±1e-2	0.4741±3e-2	0.4122±3e-2	0.1153±8e-3	0.4934±3e-2
POEM	0.5480±2e-3	0.1539±8e-4	0.6008±2e-3	0.5502±2e-3	0.1551±6e-4	0.6000±2e-3	0.5399±2e-3	0.1526±8e-4	0.5893±2e-3
POXM	0.4006±3e-2	0.1130±8e-3	0.4828±2e-2	0.3616±4e-2	0.1019±1e-2	0.4522±4e-2	0.3816±4e-2	0.1069±1e-2	0.4680±4e-2
Adaptive	0.3831±2e-2	0.1050±4e-3	0.4382±2e-2	0.4734±4e-3	0.1325±1e-3	0.5326±3e-3	0.3936±1e-2	0.1097±4e-3	0.4368±2e-2
ApproxKNN	0.5576±1e-3	0.1580±4e-4	0.6059±2e-3	0.5527±9e-4	0.1567±1e-4	0.6010±1e-3	0.5409±2e-3	0.1532±6e-4	0.5890±1e-3
IPS-C-TS	0.5565±1e-3	0.1577±3e-4	0.6048±7e-4	0.5517±6e-4	0.1563±2e-4	0.6002±6e-4	0.5393±1e-3	0.1526±5e-4	0.5879±1e-3
UIPS-P	0.4019±3e-2	0.1131±1e-2	0.4831±3e-2	0.3904±4e-2	0.1096±1e-2	0.4749±3e-2	0.4109±3e-2	0.1149±1e-2	0.4922±3e-2
UIPS-O	0.4135±4e-2	0.1167±1e-2	0.4954±4e-2	0.3896±4e-2	0.1096±1e-2	0.4739±3e-2	0.4519±3e-2	0.1268±8e-3	0.5296±2e-2
UIPS	0.5608±2e-3	0.1589±8e-4	0.6113±3e-3	0.5572±2e-3	0.1571±8e-4	0.6074±2e-3	0.5432±3e-3	0.1534±8e-4	0.5946±2e-3
p-value	3e-3	1e-2	4e-5	2e-5	2e-1	4e-10	1e-1	5e-1	4e-2
Table 2: Performance under different uncertainties.
Actions on Samples with High Uncertainty	Actions on Samples with Low Uncertainty
Algorithm	P@5(RI)	R@5(RI)	NDCG@5(RI)	P@5(RI)	R@5(RI)	NDCG@5(RI)
CE	0.5190	0.1231	0.5526	0.5913	0.1915	0.6549
BIPS-Cap	0.5117 (-1.41%)	0.1202 (-2.33%)	0.5488 (-0.68%)	0.5913 (+0.00%)	0.1903 (-0.64%)	0.6574 (+0.39%)
Shrinkage	0.5158 (-0.62%)	0.1217 (-1.11%)	0.5505 (-0.37%)	0.5892 (-0.35%)	0.1905 (-0.55%)	0.6546 (-0.05%)
UIPS	0.5222 (+0.61%)	0.1237 (+0.50%)	0.5568 (+0.77%)	0.5994 (+1.38%)	0.1940 (+1.28%)	0.6658 (+1.66%)
- UIPS-P and UIPS-O: These are two variants of our UIPS with different ways of leveraging uncertainties. UIPS-P directly penalizes samples whose estimated logging probabilities are of high uncertainty, i.e., taking  $\phi_{\pmb{x},a} = 1.0 / \exp (\gamma U_{\pmb{x},a})$ , which follows previous work on offline reinforcement learning [43, 4]. UIPS-O adversarially uses the worst propensity score for policy learning, i.e.,  $\phi_{\pmb{x},a} = 1.0 / \exp (-\gamma U_{\pmb{x},a})$
Data generation. Following previous work [24, 23], we generate a synthetic dataset by a superviser-to-bandit conversion on Wiki10-31K dataset [7], which is an extreme multi-label classification dataset. The Wiki10-31K dataset contains approximately 20K samples. Each sample is associated with a feature vector  $\tilde{\pmb{x}}$  of 101,938 dimensions and a label vector  $\pmb{y}_{\tilde{\pmb{x}}}$  of 31K classes with more than one positive class. Let  $\pmb{y}_{\tilde{\pmb{x}},a}$  denote the label of class  $a$  under  $\tilde{\pmb{x}}$ , and we take each class as an action. The huge action space creates great challenges in off-policy learning, e.g., sparse observations, and therefore better evaluates different methods.
We split the dataset into train, validation and test sets with size 11K:3K:6K. The test set is from the official split. Since the original feature vector  $\tilde{\pmb{x}}$  is too sparse, for ease of learning, we first embedded it to dimension  $d$  by  $\pmb {x} = \pmb {W}\tilde{\pmb{x}}$ , and synthesized the ground-truth logging policy  $\beta^{*}(a|x)$  by:
$$
\beta^ {*} (a | \boldsymbol {x}) \propto \exp \left(\boldsymbol {x} ^ {\top} \boldsymbol {\theta} _ {a} ^ {*} / \tau\right), \tag {9}
$$
where  $\mathbf{W}$  and  $\{\theta_{a}^{*}\}$  are pre-trained parameters by applying a logistic regression model on the train set,  $\tau$  is a positive hyper-parameter that controls the skewness of logging distribution. A small value of  $\tau$  leads to a near-deterministic logging policy, while a larger  $\tau$  makes it flatter. More implementation details can be found in Appendix 7.3.
Evaluation metrics. To evaluate the learned policy  $\pi_{\vartheta}(a|x)$ , we calculate Precision@K (P@K), Recall@K (R@K) and NDCG@K following previous work [23, 24]. Higher P@K, R@K and NDCG@K imply a better policy.
Effectiveness of policy learning. Table 1 shows the average performance and standard deviations of all algorithms under 10 random seeds on three synthetic datasets generated under different  $\tau$ . As the ground-truth logging policy is accessible on the synthetic datasets, we included a new baseline IPS-GT, which uses the IPS estimator with the ground-truth logging probabilities. We calculated  $p$ -value under t-test between UIS and the best baseline to investigate the significance of improvement.
First, we can observe that UIS achieved similar and even better performance than IPS-GT when  $\tau = 0.5$  and  $\tau = 1$ , but performed worse than IPS-GT when  $\tau = 2$ . Despite using ground-truth
logging probabilities, IPS-GT still suffered from high variance caused by samples with small logging probabilities, which is the main cause of its worse performance when  $\tau = 0.5$  and  $\tau = 1$ . In contrast, UISP effectively controlled the negative impact of these high-variance samples, resulting in a better bias-variance trade-off.
With an increasing  $\tau$ , suggesting a decrease in the probability of selecting positive actions, most algorithms experienced a drop in performance. However, UISP consistently outperformed all other algorithms across all three datasets and metrics. Interestingly, as  $\tau$  decreases, the performance improvement of UISP became even more pronounced, despite SNIPS, BanditNet, and POXM being designed to handle small logging probabilities of positive actions.
ApproxKNN and IPS-C-TS generally achieved better performance than BIPS-Cap, implying the effectiveness of calibration of estimated logging probabilities. However, UIS still consistently outperformed both ApproxKNN and IPS-C-TS. The main reason is that calibration primarily focuses on adjusting the estimated probabilities to ensure on average the model's predictions are reliable and accurate. In contrast, UIS specifically handles the impact from each individual sample in policy learning.
UIPS also consistently outperformed Shrinkage (a special case of UIPS with uncertainties always being zero) on all three datasets, demonstrating the benefits of considering the estimation uncertainty. Finally, blindly reweighing through uncertainties, regardless of their impact on the accuracy of the resulting estimator and the learned policy, ultimately resulted in poor performance, as demonstrated by UIPS-P and UIPS-O.
Performance under different uncertainty levels. As shown in Figure 1, low-frequency samples in the logged dataset suffer higher uncertainties in their propensity estimation. Thus, we divided the test set into two subsets according to the average frequency of associated actions, where the uncertainty in the subset associated with low-frequency actions is on average  $8\%$  higher than that in high-frequency actions. Table 2 shows the results on these two subsets when  $\tau = 0.5$ . In addition, we include the results of the top three baselines that directly utilize the estimated logging policy. Table 2 clearly demonstrates that only UIPS performed better than CE on the test set with low-frequency actions, implying the distortion of inaccurately estimated logging probabilities and the effectiveness of UIPS in efficiently handling them.
Off-policy Evaluation. We further inspected whether  $\hat{V}_{\mathrm{UIPS}}$  in Eq.(4) leads to more accurate off-policy evaluation. Following previous work [29, 46, 36], we evaluated the following  $\epsilon$ -greedy policy:  $\pi(a|\boldsymbol{x}) = \frac{1 - \epsilon}{|M_x|} \cdot \mathbb{I}\{a \in M_x\} + \epsilon / |\mathcal{A}|$ , where  $M_x$  contains all positive actions associated with instance  $\boldsymbol{x}$ . For each  $\boldsymbol{x}$  in the test set, we randomly sample 100 actions following the logging policy in Eq.(9) to generate the logged dataset. Table 3 shows the MSE of the estimators to the ground-truth policy value under 20 different random seeds. From Table 3, one can observe that: 1) IPS-GT with a skewer logging policy (i.e., smaller  $\tau$ ) leads to higher MSE, consistent with previous findings [29, 46, 36]; 2) inaccurate logging probabilities result in high bias and variance, leading to much larger MSE of BIPS compared to IPS-GT. Furthermore, this distortion is particularly pronounced when the ground-truth logging policy is skewed ( $\tau = 0.5$ ); and 3) although all using the estimated logging policy,  $\hat{V}_{\mathrm{UIPS}}$  yields the smallest MSE, comparing to other baselines that are designed to improve over BIPS.
Hyper-parameter Tuning. Discussions about hyper-parameter tuning and performance of UIS under different hyper-parameters can also be found in Appendix 7.3.1.
To demonstrate the effectiveness of UIS in real-world scenarios, we evaluate it on three recommendation datasets: (1) Yahoo! R3 $^1$ ; (2) Coat $^2$ ; (3) KuaiRec [12], for music, fashion and short-video recommendations respectively. All these datasets contain an unbiased test set collected from a randomized controlled trial where items are randomly selected. The statistics of the three datasets and implementation details, e.g., model architectures and dataset splits, can be found in Appendix 7.3.2.
Following [10], we take  $K = 5$  on Yahoo! R3 and Coat datasets, and  $K = 50$  on KuaiRec dataset. The  $p$ -value under the t-test between UISP and the best baseline on each dataset is also reported to investigate the significance of improvement.
1https://webscope.sandbox.yahoo.com/
$^{2}$ https://www.cs.cornell.edu/\~schnabts/mnar/
Table 3: MSE of different off-policy estimators. A lower MSE indicates a more accurate estimator.
Algorithm	IPS-GT	BIPS	minVar	stableVar	Shrinage	UIPS
τ = 0.5	0.0875±4e-4	15.786±1.51	0.9021±7e-13	0.8612±5e-8	0.0718±5e-6	0.0210±2e-6
τ = 1.0	0.0209±8e-5	0.5510±0.388	0.9019±8e-12	0.8578±2e-7	0.1978±2e-5	0.0093±1e-6
τ = 2.0	0.0020±6e-6	0.5669±0.013	0.9015±5e-15	0.8342±5e-7	0.2952±3e-5	0.0043±4e-7
Table 4: Experimental results on real-world datasets. The best and second best results are highlighted with **bold** and **underline** respectively. The p-value under the t-test between UIPS and the best baseline on each dataset is also provided.
Yahoo	Coat	KuaiRec
Algorithm	P@5	R@5	NDCG@5	P@5	R@5	NDCG@5	P@50	R@50	NDCG@50
CE	0.2819±2e-3	0.7594±6e-3	0.6073±7e-3	0.2799±5e-3	0.4618±1e-2	0.4529±7e-3	0.8802±2e-3	0.0240±8e-5	0.8810±6e-3
BIPS-Cap	0.2808±2e-3	0.7576±5e-3	0.6099±8e-3	0.2758±6e-3	0.4582±7e-3	0.4399±9e-3	0.8750±3e-3	0.0238±7e-5	0.8788±5e-3
MinVar	0.2843±4e-3	0.7685±1e-2	0.6168±1e-2	0.2813±3e-3	0.4666±9e-3	0.4414±8e-3	0.8827±1e-3	0.0240±5e-5	0.8886±2e-3
stableVar	0.2787±2e-3	0.7499±7e-3	0.5919±7e-3	0.2840±3e-3	0.4662±5e-3	0.4393±7e-3	0.8524±7e-3	0.0231±2e-4	0.8570±4e-3
Shrinkage	0.2843±3e-3	0.7654±8e-3	0.6204±7e-3	0.2790±5e-3	0.4636±4e-3	0.4464±1e-2	0.8744±3e-3	0.0238±9e-5	0.8771±6e-3
SNIPS	0.2222±4e-3	0.5828±1e-2	0.4357±1e-2	0.2643±7e-3	0.4287±1e-2	0.4009±9e-3	0.8411±6e-3	0.0228±2e-4	0.8431±6e-3
BanditNet	0.2413±8e-3	0.6442±2e-2	0.4988±2e-2	0.2781±8e-3	0.4527±1e-2	0.4251±1e-2	0.8758±5e-3	0.0239±2e-4	0.8810±4e-3
POEM	0.2732±3e-3	0.7357±1e-2	0.5880±1e-2	0.2791±4e-3	0.4566±6e-3	0.4375±6e-3	0.7785±1e-2	0.0210±2e-4	0.7779±6e-3
POXM	0.2250±5e-3	0.5940±1e-2	0.4542±2e-2	0.2663±6e-3	0.4308±9e-3	0.4006±1e-2	0.8962±1e-2	0.0245±4e-4	0.9041±1e-2
Adaptive	0.2762±3e-3	0.7451±9e-3	0.5919±8e-3	0.2830±3e-3	0.4634±5e-3	0.4217±5e-3	0.8375±1e-2	0.0227±4e-4	0.8460±1e-2
ApproxKNN	0.2697±2e-3	0.7225±5e-3	0.5760±6e-3	0.2755±2e-3	0.4594±5e-3	0.4490±4e-3	0.8839±2e-6	0.0240±5e-5	0.8895±2e-3
IPS-C-TS	0.2816±2e-3	0.7582±5e-3	0.6114±5e-3	0.2799±3e-3	0.4625±7e-3	0.4462±6e-3	0.8781±3e-3	0.0239±1e-4	0.8749±3e-3
UIPS-P	0.1829±8e-3	0.4560±3e-2	0.3300±1e-2	0.2685±7e-3	0.4364±9e-3	0.4087±7e-3	0.8638±8e-3	0.0235±3e-4	0.8685±7e-3
UIPS-O	0.1947±3e-3	0.4959±1e-2	0.3600±8e-3	0.2657±5e-3	0.4306±9e-3	0.4146±9e-3	0.8651±8e-3	0.0235±2e-4	0.8697±7e-3
UIPS	0.2868±2e-3	0.7742±5e-3	0.6274±5e-3	0.2877±3e-3	0.4757±5e-3	0.4576±8e-3	0.9120±1e-3	0.0250±5e-5	0.9174±7e-4
p-value	4e-2	1e-2	3e-2	2e-2	6e-4	5e-5	6e-4	6e-4	1e-3
We can first observe from Table 4 that on all three datasets, the proposed UIPS achieved the highest precision, recall and NDCG. Apparently, accurate estimation of logging probabilities in real-world scenarios with large action spaces and sparse interactions is challenging to achieve, causing BIPS-Cap to underperform CE. Additionally, calibration poses difficulties in such scenarios, leading to poor performance of ApproxKNN and IPS-C-TS across all three real-world datasets. BanditNet, POEM and POXM performed better in problems with a larger action space, while MinVar, stableVar and Shrinkage as well as Adaptive suited better for scenarios with a smaller action size. Again, UIPS outperformed Shrinkage, highlighting the importance of handling uncertainty in the estimated logging policy. But simply reweighing based on uncertainties, without considering their impact on the accuracy of the resulting estimator and the learned policy, led to poor performance, as shown by UIPS-P and UIPS-O.
4.3 Comparisons against other lines of off-policy learning
We discuss about the difference between UISP and recent work with direct propensity estimation [34, 21], the DICE line of work [26, 47, 9] as well as the work on the distributionally robust off-policy learning [32, 17, 44] in Appendix 7.6, Appendix 7.5 and Appendix 7.7 respectively. Our empirical evaluations on recent work [26, 44, 34] from all three lines suggest that these lines of work cannot properly handle inaccuracies in the estimated logging probabilities that hinder policy improvement. Moreover, we also integrated the proposed UISP with the doubly-robust (DR) estimator and our results suggest the accuracy of the imputation model greatly affected policy learning under DR, but UISP still provides benefits. More details can be found in Appendix 7.4.
Our work is the first of its kind to account for the uncertainty of logging policy estimation in off-policy learning. The following two lines of work are most related to this work.
Off-policy learning. In many real-world applications, such as search engines and recommender systems, interactive online model update is expensive and risky [15]. Off-policy learning has therefore attracted great interest, since it can leverage the already logged feedback data [2, 8, 22]. The main challenge in off-policy learning is to address the mismatch between the logging policy and the learning policy. One common and widely-applied approach is to leverage the Inverse Propensity Scoring (IPS) method to correct the discrepancy between the two policies. And various methods are proposed to enhance IPS for more stabilizing learning [39, 37, 38] and improved variance control [23, 22], as well as extensions to more complex problems [40, 24].
However, all these solutions directly use the estimated logging policy for off-policy correction, leading to sub-optimal performance as shown in our experiments. A recent study on causal recommendation [10] also argues that propensity scores may not be correct due to unobserved confounders. They assume the effect of unobserved confounder for any sample can be bounded by a pre-defined hyperparameter, and adversarially search for the worst-case propensity for learning. Mapping to off-policy learning, their solution is a special case of our UIS-O variant with uncertainty as a pre-defined constant.
There were existing studies [34, 21, 26, 47, 9] also explore direct estimation of the propensity ratio to bypass estimating the logging policy. However, as discussed in Appendix 7.6 and Appendix 7.5, they demonstrate inferior performance compared to UIS. This is primarily due to either the lack of consideration for the accuracy of the estimated propensity ratio, similar to the limitations of existing IPS-type algorithms in handling inaccurately estimated logging probabilities, or the degeneration to a specific IPS estimator that suffers high variance.
Recent work on distributionally robust off-policy evaluation and learning [32, 17, 44] also addresses uncertainty in off-policy learning. However, their approach to handling uncertainty and the underlying motivation differ significantly from ours, resulting in distinct techniques employed. Further details can be found in Appendix 7.7. Additionally, experiments conducted in Appendix 7.7 demonstrate that directly adapting methods from distributionally robust off-policy learning to handle inaccurately estimated logging probabilities leads to poor performance.