SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Xiaoying Zhang $^{1}$  Junpu Chen $^{2}$  Hongning Wang $^{3}$  Hong Xie $^{4}$  Yang Liu $^{1}$  John C.S. Lui $^{5}$  Hang Li $^{1}$
$^{1}$ ByteDance Research  $^{2}$ ChongQing University  $^{3}$ Tsinghua University  $^{4}$  Chongqing Institute of Green and Intelligent Technology, Chinese Academy of Science  $^{5}$  The Chinese University of Hong Kong  $\{\text{zhangxiaoying.xy, yang.liu01, lihang.lh}\} @$ bytedance.com  $\{\text{jumpchan98, hongx87, wang.hongn}\} @$ gmail.com  $\text{cslui@cse.cuhk.edu.hk}$
Off-policy learning, referring to the procedure of policy optimization with access only to logged feedback data, has shown importance in various real-world applications, such as search engines and recommender systems. While the ground-truth logging policy is usually unknown, previous work simply employs its estimated value for the off-policy learning, ignoring the negative impact from both high bias and high variance resulted from such an estimator. And such impact is often magnified on samples with small and inaccurately estimated logging probabilities. The contribution of this work is to explicitly model the uncertainty in the estimated logging policy, and propose an Uncertainty-aware Inverse Propensity Score estimator (UIPS) for improved off-policy learning, with a theoretical convergence guarantee. Experiment results on the synthetic and real-world recommendation datasets demonstrate that UIPS significantly improves the quality of the discovered policy, when compared against an extensive list of state-of-the-art baselines.
In many real-world applications, including search engines [2], online advertisements [35], recommender systems [8, 22], only logged data is available for subsequent policy learning. For example, in recommender systems, various complex recommendation policies are optimized over logged user interactions (e.g., clicks or stay time) with items recommended by previous recommendation policies (referred to as the logging policy) [51, 14]. However, such logged data is often known to be biased, since the feedback on items where the logging policy did not take is unknown. This inevitably distorts the evaluation and optimization of a new policy when it differs from the logging policy.
Off-policy learning [41, 27] thus emerges as a preferred way to learn an improved policy only from the logged data, by addressing the mismatch between the learning and logging policies. One of the most commonly used off-policy learning methods is the Inverse Propensity Scoring (IPS) [8, 25], which assigns per-sample importance weight (i.e., propensity score) to the training objective on the logged data, so as to get an unbiased optimization objective in expectation. The importance weight in IPS is the probability ratio of taking an action between the learning and logging policies.
Unfortunately, the ground-truth logging policy is oftentimes unavailable to the learner in practice, due to reasons like legacy issues, i.e., it was not recorded in the data. Additionally, in specific situations like the healthcare domain [28] or two-stage recommender systems [8], access to the ground-truth logging policy is not feasible. One common treatment by many previous studies [35, 22, 8, 24] is to first estimate the logging policy using a supervised learning method (e.g., logistic regression,
(a) Estimated Logging Probability
(b) Uncertainty of Estimation
Figure 1: Estimated logging policy and its uncertainty under different item frequency on KuaiRec.
neural networks, etc.), and then employ the estimated logging policy for off-policy learning. In this work, we first show that such an approximation results in a biased estimator which is sensitive to data with small estimated logging probabilities. Worse still, small estimated logging probabilities usually suggest there are limited related samples in the logged data, whose estimations can have high uncertainties, i.e., being wrong with a high probability. Figure 1 shows a piece of empirical evidence from a large-scale recommendation benchmark KuaiRec dataset [12], where items with lower frequencies in the logged dataset have lower estimated logging probabilities (via a neural network estimator) and higher uncertainties at the same time. The high bias and variance caused by these samples can greatly hinder the performance of subsequent off-policy learning. We defer detailed discussions of this result in Section 2.
In this work, we explicitly take the uncertainty of the estimated logging policy into consideration and design an Uncertainty-aware Inverse Propensity Score estimator (UIPS) for off-policy learning. UIPS reweighs the propensity score of each logged sample to control its impact on policy optimization, and learns an improved policy by alternating between: (1) Find the optimal weight that makes the estimator as accurate as possible, based on the uncertainty of the estimated logging policy; (2) Improve the policy by optimizing the resulting objective function. The optimal weight for each sample is obtained by minimizing the upper bound of the mean squared error (MSE) to the ground-truth policy evaluation, with a closed-form solution. Furthermore, UIPS ensures that off-policy learning converges to a stationary point where the true policy gradient is zero; while convergence may not be guaranteed when directly using the estimated logging policy. Extensive experiments on a synthetic and three real-world recommendation datasets against a rich set of state-of-the-art baselines demonstrate the power of UIPS. All data and code can be found in https://github.com/Xiaoyinggit/UIPS.git.
2 Preliminary: off-policy learning
We focus on the standard contextual bandit setup to explain the key concepts in UIPS. Following the convention [16, 29, 36], let  $\pmb{x} \in \mathcal{X} \subseteq R^d$  be a  $d$ -dimensional context vector drawn from an unknown distribution  $p(\pmb{x})$ . Each context is associated with a finite set of actions denoted by  $\mathcal{A}$ , where  $|\mathcal{A}| < \infty$ . Let  $\pi: \mathcal{A} \times \mathcal{X} \to [0,1]$  denote a stochastic policy, such that  $\pi(a|x)$  is the probability of selecting action  $a$  under context  $x$  and  $\sum_{a \in \mathcal{A}} \pi(a|x) = 1$ . Under a given context  $x$ , the reward  $r_{\pmb{x},a}$  is only observed when action  $a$  is chosen, i.e., bandit feedback. Without loss of generality, we assume  $r_{\pmb{x},a} \in [0,1]$ . Let  $V(\pi)$  denote the expected reward of the policy  $\pi$ :
$$
V (\pi) = \mathbb {E} _ {\boldsymbol {x} \sim p (\boldsymbol {x}), a \sim \pi (a | \boldsymbol {x})} [ r _ {\boldsymbol {x}, a} ]. \tag {1}
$$
We look for a policy  $\pi(a|x)$  to maximize  $V(\pi)$ . In the rest, we denote  $\mathbb{E}_{\boldsymbol{x} \sim p(\boldsymbol{x}), a \sim \pi(a|x)}[\cdot]$  as  $\mathbb{E}_{\pi}[\cdot]$ .
In off-policy learning, one can only access a set of logged feedback data  $D \coloneqq \{(x_{n},a_{n},r_{\pmb{x}_{n},a_{n}})|n\in [N]\}$ . Given  $\pmb{x}_n$ , the action  $a_{n}$  was generated by a stochastic logging policy  $\beta^{*}$ , i.e.,  $a_{n} \sim \beta^{*}(a|\pmb{x}_{n})$  which is usually different from the learning policy  $\pi (a|x)$  [24, 40, 8]. The actions  $\{a_1,\dots ,a_N\}$  and their corresponding rewards  $\{r_{\pmb{x}_1,a_1},\ldots ,r_{\pmb{x}_N,a_N}\}$  are generated independently given  $\beta^{*}$ . The main challenge is then to address the distributional discrepancy between  $\beta^{*}(a|x)$  and  $\pi (a|x)$ , when optimizing  $\pi (a|x)$  to maximize  $V(\pi)$  with access only to the logged dataset  $D$ .
One of the most widely used methods to address the distribution shift between  $\pi(a|\pmb{x})$  and  $\beta^{*}(a|\pmb{x})$  is the Inverse Propensity Scoring (IPS) [8, 25]. One can easily get that:
$$
V (\pi) = \mathbb {E} _ {\beta *} \left[ \frac {\pi (a | \boldsymbol {x})}{\beta^ {*} (a | \boldsymbol {x})} r _ {\boldsymbol {x}, a} \right],
$$
yielding the following empirical estimator of  $V(\pi)$ :
$$
\hat {V} _ {\mathrm {I P S}} (\pi) = \frac {1}{N} \sum_ {n = 1} ^ {N} \frac {\pi \left(a _ {n} \mid \boldsymbol {x} _ {n}\right)}{\beta^ {*} \left(a _ {n} \mid \boldsymbol {x} _ {n}\right)} r _ {\boldsymbol {x} _ {n}, a _ {n}}, \tag {2}
$$
where  $\pi (a_n|\pmb {x}_n) / \beta^* (a_n|\pmb {x}_n)$  is referred to as the propensity score. Various algorithms can be readily used for policy optimization under  $\hat{V}_{\mathrm{IPS}}(\pi)$ , including value-based methods [33] and policy-based methods [19, 31, 42]. In this work, we adopt a well-known policy gradient algorithm, REINFORCE [42]. Assume the policy  $\pi (a|\pmb {x})$  is parameterized by  $\vartheta$ , via the "log-trick", the gradient of  $\hat{V}_{\mathrm{IPS}}(\pi_{\vartheta})$  with respect to  $\vartheta$  can be readily derived as,
$$
\nabla_ {\pmb {\vartheta}} \hat {V} _ {\mathrm {I P S}} (\pi_ {\pmb {\vartheta}}) = \frac {1}{N} \sum_ {n = 1} ^ {N} \frac {\pi_ {\pmb {\vartheta}} (a _ {n} | \pmb {x} _ {n})}{\beta^ {*} (a _ {n} | \pmb {x} _ {n})} r _ {\pmb {x} _ {n}, a _ {n}} \nabla_ {\pmb {\vartheta}} \log (\pi_ {\pmb {\vartheta}} (a _ {n} | \pmb {x} _ {n})).
$$
Approximation with an unknown logging policy. In many real-world applications, the ground-truth logging probabilities, i.e.,  $\beta^{*}(a|\pmb{x})$  of each observation  $(\pmb{x}, a)$  in  $D$ , are unknown. As a typical walk-around, previous work employs supervised learning methods such as logistic regression [30] and neural networks [8] to estimate the logging policy, and replaces  $\beta^{*}(a|\pmb{x})$  with its estimated value  $\hat{\beta}(a|\pmb{x})$  to get the following BIPS estimator for policy learning:
$$
\hat {V} _ {\mathrm {B I P S}} \left(\pi_ {\boldsymbol {\vartheta}}\right) = \frac {1}{N} \sum_ {n = 1} ^ {N} \frac {\pi_ {\boldsymbol {\vartheta}} \left(a _ {n} \mid \boldsymbol {x} _ {n}\right)}{\hat {\beta} \left(a _ {n} \mid \boldsymbol {x} _ {n}\right)} r _ {\boldsymbol {x} _ {n}, a _ {n}}. \tag {3}
$$
However, as shown in the following proposition, inaccurate  $\hat{\beta}(a|x)$  leads to high bias and variance in BIPS. Worse still, smaller and inaccurate  $\hat{\beta}(a|x)$  further enlarges this bias and variance.
Proposition 2.1. The bias and variance of  $\hat{V}_{\mathrm{BIPS}}(\pi_{\vartheta})$  can be derived as follows:
$$
\operatorname {B i a s} \left(\hat {\mathrm {V}} _ {\mathrm {B I P S}} (\pi_ {\boldsymbol {\vartheta}})\right) = \mathbb {E} _ {D} \left[ \hat {V} _ {\mathrm {B I P S}} (\pi_ {\boldsymbol {\vartheta}}) - V (\pi_ {\boldsymbol {\vartheta}}) \right] = \mathbb {E} _ {\pi_ {\boldsymbol {\vartheta}}} \left[ r _ {\boldsymbol {x}, a} \left(\frac {\beta^ {*} (a | \boldsymbol {x})}{\hat {\beta} (a | \boldsymbol {x})} - 1\right) \right]
$$
$$
N \cdot \operatorname {V a r} _ {D} \left(\hat {V} _ {\text {B I P S}} (\pi_ {\boldsymbol {\vartheta}})\right) = \operatorname {V a r} _ {\pi_ {\boldsymbol {\vartheta}}} \left(\frac {\beta^ {*} (a | \boldsymbol {x})}{\hat {\beta} (a | \boldsymbol {x})} r _ {\boldsymbol {x}, a}\right) + \mathbb {E} _ {\pi_ {\boldsymbol {\vartheta}}} \left[ \left(\frac {\pi_ {\boldsymbol {\vartheta}} (a | \boldsymbol {x})}{\beta^ {*} (a | \boldsymbol {x})} - 1\right) \frac {\beta^ {*} (a | \boldsymbol {x}) ^ {2}}{\hat {\beta} (a | \boldsymbol {x}) ^ {2}} r _ {\boldsymbol {x}, a} ^ {2} \right]
$$
However, smaller  $\hat{\beta}(a|x)$  usually implies less number of related training samples in the logged data, and thus  $\hat{\beta}(a|x)$  can be inaccurate with a higher probability. To make it more explicit, let us revisit the empirical results shown in Figure 1. We followed the method introduced in [8] to estimate the logging policy on KuaiRec dataset [12] and plotted the estimated  $\hat{\beta}(a|x)$  and its corresponding uncertainties on items of different observation frequencies in the logged dataset. We adopted the method in [45] to measure the confidence interval of  $\hat{\beta}(a|x)$  on each instance. A wider confidence interval, i.e., higher uncertainty in estimation, implies that with a high probability the true value may be further away from the empirical mean estimate. We can observe in Figure 1 that as item frequency decreases, the estimated logging probability also decreases, but the estimation uncertainty increases. This implies that a smaller  $\hat{\beta}(a|x)$  is usually 1) more inaccurate and 2) associated with a higher uncertainty.
As a result, with high bias and variance caused by inaccurate  $\hat{\beta}(a|\boldsymbol{x})$ , it is erroneous to learn  $\pi_{\vartheta}(a|\boldsymbol{x})$  by simply optimizing  $\hat{V}_{\mathrm{BIPS}}(\pi_{\vartheta})$ . Furthermore, this approach may also hinder the convergence of off-policy learning, as discussed later in Section 3.2.
3 Uncertainty-aware off-policy learning
Our idea is to consider the uncertainty of the estimated logging policy by incorporating per-sample weight  $\phi_{x,a}$ , and perform policy learning by optimizing the following empirical estimator:
$$
\hat {V} _ {\mathrm {U I P S}} \left(\pi_ {\boldsymbol {\vartheta}}\right) = \frac {1}{N} \sum_ {n = 1} ^ {N} \frac {\pi_ {\boldsymbol {\vartheta}} \left(a _ {n} \mid \boldsymbol {x} _ {n}\right)}{\hat {\beta} \left(a _ {n} \mid \boldsymbol {x} _ {n}\right)} \cdot \phi_ {\boldsymbol {x} _ {n}, a _ {n}} \cdot r _ {\boldsymbol {x} _ {n}, a _ {n}}. \tag {4}
$$
Intuitively, one should assign lower weights to samples whose  $\hat{\beta}(a|\boldsymbol{x})$  is small and far away from the ground-truth  $\beta^{*}(a|\boldsymbol{x})$ . We then divide off-policy optimization into two iterative steps:
- Deriving the optimal instance weight: Find the optimal  $\phi_{\pmb{x},a}$  to make  $\hat{V}_{\mathrm{UIPS}}(\pi_{\vartheta})$  approach its ground-truth  $V(\pi)$  as closely as possible, so as to facilitate policy learning. The derived optimal weight is denoted as  $\phi_{\pmb{x},a}^{*}$  (see Theorem 3.2).
- Policy improvement: Update the policy  $\pi_{\vartheta}(a|x)$  using the following gradient:
$$
\nabla_ {\vartheta} \hat {V} _ {\mathrm {U I P S}} \left(\pi_ {\vartheta}\right) = \frac {1}{N} \sum_ {n = 1} ^ {N} \frac {\pi_ {\vartheta} \left(a _ {n} \mid \boldsymbol {x} _ {n}\right)}{\hat {\beta} \left(a _ {n} \mid \boldsymbol {x} _ {n}\right)} \cdot \phi_ {\boldsymbol {x} _ {n}, a _ {n}} ^ {*} \cdot r _ {\boldsymbol {x} _ {n}, a _ {n}} \nabla_ {\vartheta} \log \left(\pi_ {\vartheta} \left(a _ {n} \mid \boldsymbol {x} _ {n}\right)\right) \tag {5}
$$
The whole algorithm framework and its computational cost, as well as important notations are summarized in Appendix 7.1.
3.1 Derive the optimal uncertainty-aware instance weight
We expect to find the optimal weight  $\phi_{x,a}$  to make the empirical estimator  $\hat{V}_{\mathrm{UIPS}}(\pi_{\vartheta})$  as accurate as possible, taking into account the uncertainty in estimated logging probabilities. Intuitively, a high accuracy of the estimator is crucial for determining the correct direction of policy learning. We follow previous work [36, 29] and measure the mean squared error (MSE) of  $\hat{V}_{\mathrm{UIPS}}(\pi_{\vartheta})$  to the ground-truth policy value  $V(\pi_{\vartheta})$ , which captures both the bias and variance of an estimator. A lower MSE indicates a more accurate estimator.
In UIPS, instead of directly minimizing the MSE, which is intractable, we find  $\phi_{\pmb{x},a}$  to minimize the upper bound of MSE. As we show later, the optimal  $\phi_{\pmb{x},a}$  has a closed-form solution which relates to both the value of  $\pi_{\vartheta}(a|\pmb{x}) / \hat{\beta}(a|\pmb{x})$  and the estimation uncertainty of  $\hat{\beta}(a|\pmb{x})$ .
Theorem 3.1. The mean squared error (MSE) between  $\hat{V}_{\mathrm{UIPS}}(\pi_{\vartheta})$  and ground-truth estimator  $V(\pi_{\vartheta})$  is upper bounded as follows:
$$
\begin{array}{l} \operatorname {M S E} \left(\hat {V} _ {\mathrm {U I P S}} (\pi_ {\boldsymbol {\vartheta}})\right) = \mathbb {E} _ {D} \left[ \left(\hat {V} _ {\mathrm {U I P S}} (\pi_ {\boldsymbol {\vartheta}}) - V (\pi_ {\boldsymbol {\vartheta}})\right) ^ {2} \right] = \operatorname {B i a s} \left(\hat {V} _ {\mathrm {U I P S}} (\pi_ {\boldsymbol {\vartheta}})\right) ^ {2} + \operatorname {V a r} \left(\hat {V} _ {\mathrm {U I P S}} (\pi_ {\boldsymbol {\vartheta}})\right) \\ \leq \mathbb {E} _ {\pi_ {\boldsymbol {\vartheta}}} \left[ r _ {\boldsymbol {x}, a} ^ {2} \frac {\pi_ {\boldsymbol {\vartheta}} (a | \boldsymbol {x})}{\beta^ {*} (a | \boldsymbol {x})} \right] \cdot \mathbb {E} _ {\beta^ {*}} \left[ \left(\frac {\beta^ {*} (a | \boldsymbol {x})}{\hat {\beta} (a | \boldsymbol {x})} \phi_ {\boldsymbol {x}, a} - 1\right) ^ {2} \right] + \mathbb {E} _ {\beta^ {*}} \left[ \frac {\pi_ {\boldsymbol {\vartheta}} (a | \boldsymbol {x}) ^ {2}}{\hat {\beta} (a | \boldsymbol {x}) ^ {2}} \phi_ {\boldsymbol {x}, a} ^ {2} \right]. \\ \end{array}
$$
As the first expectation term  $\mathbb{E}_{\pi_{\vartheta}}\left[r_{\boldsymbol{x},a}^2\frac{\pi_{\vartheta}(a|\boldsymbol{x})}{\beta^*(a|\boldsymbol{x})}\right]$  is a non-negative constant, we denote it as  $\lambda \in [0,\infty)$  when searching for  $\phi_{\boldsymbol{x},a}$ . To minimize this upper bound of MSE, the optimal  $\phi_{\boldsymbol{x},a}$  for each sample  $(\boldsymbol{x},a)$  should minimize the following,
$$
\lambda \left(\frac {\beta^ {*} (a | \boldsymbol {x})}{\hat {\beta} (a | \boldsymbol {x})} \phi_ {\boldsymbol {x}, a} - 1\right) ^ {2} + \frac {\pi_ {\boldsymbol {\vartheta}} (a | \boldsymbol {x}) ^ {2}}{\hat {\beta} (a | \boldsymbol {x}) ^ {2}} \phi_ {\boldsymbol {x}, a} ^ {2}. \tag {6}
$$
An interesting observation is that setting  $\phi_{\pmb{x},a} = \frac{\hat{\beta}(a|\pmb{x})}{\hat{\beta}^*(a|\pmb{x})}$ , i.e., turning  $\frac{\pi(a|\pmb{x})}{\hat{\beta}(a|\pmb{x})}\phi_{\pmb{x},a}$  into  $\frac{\pi(a|\pmb{x})}{\beta^*(a|\pmb{x})}$  does not result in the optimal solution of Eq.(6). This is because such a setting only reduces bias (i.e., the first term of Eq.(6)), but fails to control the second term, which is related to the variance. Moreover, we cannot directly minimize Eq.(6) due to the unknown  $\beta^*(a|\pmb{x})$ . But it is possible to obtain a confidence interval which contains  $\beta^*(a|\pmb{x})$  with a high probability, when  $\hat{\beta}(a|\pmb{x})$  is obtained via a specific estimator, e.g., (generalized) linear model or kernel methods.
Following previous work [23, 16, 22], we adopt the realizable assumption that  $\beta^{*}(a|\pmb{x})$  can be represented by a softmax function applied over a parametric function  $f_{\theta^{*}}(\pmb{x},a)$ . Moreover, the universal approximation theorem [18] states that a parametric function with sufficient capacity, when combined with a softmax function, can approximate any distribution. Then we have:
$$
\beta^ {*} (a | \boldsymbol {x}) \propto \exp \left(f _ {\boldsymbol {\theta} ^ {*}} (\boldsymbol {x}, a)\right), \hat {\beta} (a | \boldsymbol {x}) \propto \exp \left(f _ {\boldsymbol {\theta}} (\boldsymbol {x}, a)\right), \tag {7}
$$
where  $f_{\pmb{\theta}}(\pmb{x},a)$  is an estimate of  $f_{\pmb{\theta}^*}(\pmb{x},a)$ . Following the conventional definition of confidence interval [20], we define  $\gamma$  and  $U_{\pmb{x},a}$  such that  $|f_{\pmb{\theta}^*}(\pmb{x},a) - f_{\pmb{\theta}}(\pmb{x},a)| \leq \gamma U_{\pmb{x},a}$  holds with probability
at least  $1 - \delta$ , where  $\gamma$  is a function of  $\delta$  (typically the smaller  $\delta$  is, the larger  $\gamma$  is). Then  $\gamma U_{\boldsymbol{x},a}$  measures the width of confidence interval of  $f_{\boldsymbol{\theta}}(\boldsymbol{x},a)$  against its ground-truth  $f_{\boldsymbol{\theta}^*}(\boldsymbol{x},a)$ . As derived in Appendix 7.2, with probability at least  $1 - \delta$ , we have  $\beta^{*}(a|\boldsymbol{x}) \in B_{\boldsymbol{x},a}$  and
$$
\boldsymbol {B} _ {\boldsymbol {x}, a} = \left[ \frac {\hat {Z} \exp (- \gamma U _ {\boldsymbol {x} , a})}{Z ^ {*}} \hat {\beta} (a | \boldsymbol {x}), \frac {\hat {Z} \exp (\gamma U _ {\boldsymbol {x} , a})}{Z ^ {*}} \hat {\beta} (a | \boldsymbol {x}) \right],
$$
where  $Z^{*} = \sum_{a^{\prime}}\exp (f_{\theta^{*}}(a^{\prime}|\pmb {x}))$  and  $\hat{Z} = \sum_{a^{\prime}}\exp (f_{\theta}(a^{\prime}|\pmb {x}))$ .
As  $\beta^{*}(a|\pmb{x})$  can be any value in  $B_{\pmb{x},a}$  with high probability, we aim to find the optimal  $\phi_{\pmb{x},a}$  that minimizes the worst case of Eq.(6), thereby ensuring that  $\hat{V}_{\mathrm{UIPS}}(\pi_{\vartheta})$  approaches its ground-truth  $V(\pi_{\vartheta})$  under the sense of MSE, even in the worst possible scenarios. This ensures the subsequent policy improvement direction will not be much worse with high probability. Thus, we formulate the following optimization problem:
$$
\min  _ {\boldsymbol {\phi} _ {\boldsymbol {x}, a}} \max  _ {\beta_ {\boldsymbol {x}, a} \in \boldsymbol {B} _ {\boldsymbol {x}, a}} \lambda \left(\frac {\beta_ {\boldsymbol {x} , a}}{\hat {\beta} (a | \boldsymbol {x})} \phi_ {\boldsymbol {x}, a} - 1\right) ^ {2} + \frac {\pi_ {\boldsymbol {\vartheta}} (a | \boldsymbol {x}) ^ {2}}{\hat {\beta} (a | \boldsymbol {x}) ^ {2}} \phi_ {\boldsymbol {x}, a} ^ {2}. \tag {8}
$$
The following theorem derives a closed-form formula for the optimal solution of Eq.(8).
Theorem 3.2. Let  $\eta \in [\exp(-\gamma U_x^{\max}), \exp(\gamma U_x^{\max})]$ , where  $U_x^{\max} = \max_a U_{x,a}$ . The optimization problem in Eq.(8) has a closed-form solution:
$$
\phi_ {\boldsymbol {x}, a} ^ {*} = \min  \left(\lambda / \left[ \frac {\lambda}{\eta} \exp (- \gamma U _ {\boldsymbol {x}, a}) + \frac {\eta \pi_ {\boldsymbol {\vartheta}} (a | \boldsymbol {x}) ^ {2}}{\hat {\beta} (a | \boldsymbol {x}) ^ {2} \exp (- \gamma U _ {\boldsymbol {x}, a})} \right], 2 \eta / \left[ \exp (\gamma U _ {\boldsymbol {x}, a}) + \exp (- \gamma U _ {\boldsymbol {x}, a}) \right]\right).
$$
The following corollary demonstrates the advantage of UIPS. The detailed proof of Theorem 3.2 and Corollary 3.3 can be found in Appendix 7.8.
Corollary 3.3. With  $\phi_{\pmb{x},a}^{*}$  derived in Theorem 3.2,  $\hat{V}_{\mathrm{UIPS}}(\pi_{\vartheta})$  in Eq.(4) achieves a smaller upper bound of MSE than  $\hat{V}_{\mathrm{BIPS}}(\pi_{\vartheta})$  in Eq. (3).
Insights about  $\phi_{\pmb{x},a}^{*}$ . The detailed analysis of the effect of  $\phi_{\pmb{x},a}^{*}$  can be found in Lemma 7.1 in Appendix 7.8. In summary, we have the following key findings,
- For samples whose largest possible propensity score is under control: i.e.,  $\frac{\pi_{\vartheta}(a|x)}{\min B_{x,a}} < \sqrt{\lambda}$ , higher uncertainty implies smaller values of  $\pi/\hat{\beta}$ . This suggests samples of this type with positive rewards are underestimated, and the extend of underestimation increases with the estimation uncertainty. UIPS thus chooses to increase  $\phi_{\pmb{x},a}^{*}$  with uncertainty, to emphasize these long-tail positive samples.
- Conversely, for samples with large propensity scores, UIS decreases  $\phi_{\pmb{x},a}^{*}$  as the uncertainty increases, so as to prevent their distortion in policy learning.
Uncertainty estimation. Now we describe how to calculate  $U_{\boldsymbol{x},a}$ , i.e., the uncertainty of the estimated  $\hat{\beta}(a|\boldsymbol{x})$ . In this work, we choose to estimate  $\beta^{*}(a|\boldsymbol{x})$  using a neural network, because 1) its representation learning capacity has been proved in numerous studies, and 2) various ways [11, 45] can be leveraged to perform the uncertainty estimation in a neural network. We adopt [45] due to its computational efficiency and theoretical soundness. Following the proof of Theorem 4.4 in [45], given the logged dataset  $D$ , we can get with a high probability that there exists  $\gamma$  such that:
$$
\left| f _ {\boldsymbol {\theta}} \left(\boldsymbol {x} _ {n}, a _ {n}\right) - f _ {\boldsymbol {\theta} ^ {*}} \left(\boldsymbol {x} _ {n}, a _ {n}\right)\right) | \leq \gamma \sqrt {\boldsymbol {g} \left(\boldsymbol {x} _ {n} , a _ {n}\right) ^ {\top} \boldsymbol {M} _ {D} ^ {- 1} \boldsymbol {g} \left(\boldsymbol {x} _ {n} , a _ {n}\right)}
$$