SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 10: Empirical performance of DICE-S on real-world datasets.
Yahoo	Coat	KuaiRec
Algorithm	P@5	R@5	NDCG@5	P@5	R@5	NDCG@5	P@50	R@50	NDCG@50
BIPS-Cap	0.2808±2e-3	0.7576±5e-3	0.6099±8e-3	0.2758±6e-3	0.4582±7e-3	0.4399±9e-3	0.8750±3e-3	0.0238±7e-5	0.8788±5e-3
DICE-S	0.2618±4e-3	0.7010±1e-2	0.5627±9e-3	0.2686±5e-3	0.4422±6e-3	0.4265±5e-3	0.8842±2e-3	0.0241±6e-5	0.8908±2e-3
UIPS	0.2868±2e-3	0.7742±5e-3	0.6274±5e-3	0.2877±3e-3	0.4757±5e-3	0.4576±8e-3	0.9120±1e-3	0.0250±5e-5	0.9174±7e-4
We further inspected how DICE-type algorithms would work in the contextual bandit setting, which can be taken as a one-state MDP. We take DualDICE [1] as an example for illustration, and similar conclusions can be drawn for other DICE-type algorithms. We can easily derive that in the contextual bandit setting, DualDICE degenerates to the IPS estimator that approximates the unknown ground-truth logging policy  $\beta^{*}(a|s)$  with its empirical estimate from the given logged dataset  $D\coloneqq \{(s_i,a_i,r_{s_i,a_i})|i\in [N]\}$ , which inherits all limitations of IPS estimator, such as high variance on instances with low support in  $D$ .
More specifically, recall that DualDICE estimates the discounted stationary distribution correction by optimizing the following objective function (Eq.(8) in [26]):
$$
\min _ {x: S \times A \to C} \frac {1}{2} E _ {(s, a) \sim d ^ {D}} [ x (s, a) ^ {2} ] - E _ {(s, a) \sim d ^ {\pi}} [ x (s, a) ],
$$
with the optimizer  $x^{*}(s,a) = w_{\pi /D}(s,a)$ . In a multi-step RL setting, one cannot directly calculate the second expectation as  $d^{\pi}(s,a)$  is inaccessible, thus DualDICE takes the change-of-variable trick to address the above optimization problem. However, in the contextual bandit setting, let  $p(s)$  denote the state distribution,  $d^{\pi}(s,a) = p(s)\pi (a|s)$  and  $d^{D}(s,a) = p(s)\beta^{*}(a|s)$ . The optimization problem can be rewritten as:
$$
\min  _ {x: S \times A \to C} E _ {s \sim p (s)} \left[ \frac {1}{2} E _ {a \sim \beta^ {*} (a | s)} [ x (s, a) ^ {2} ] - E _ {a \sim \pi (a | s)} [ x (s, a) ] \right].
$$
With the logged dataset  $D$ , the empirical estimate of above optimization problem is:
$$
\min  _ {x: S \times A \to C} \sum_ {s} \frac {N _ {s}}{N} \left(\sum_ {a} \frac {N _ {s , a}}{2 N _ {s}} x (s, a) ^ {2} - \sum_ {a} \pi (a | s) x (s, a)\right),
$$
yielding  $x^{*}(s,a) = \pi (a|s)N_{s} / N_{s,a}$ , where  $N_{s}$  and  $N_{s,a}$  denote the number of logged samples with  $s_i = s$  and  $(s_i = s, a_i = a)$  respectively. This is actually equivalent to the IPS estimator with  $N_{s,a} / N_{s}$  as the estimate of  $\beta^{*}(a|s)$ . We refer to the above estimator as DICE-S, and Table 9 and Table 10 show the empirical performance of the policy learned through DICE-S on the synthetic and real-world datasets respectively. Similar as BIPS-Cap described in Section 4, we also clip propensity scores to control variance. One can observe from Table 9 and Table 10 that DICE-S performs worse than BIPS-Cap in all datasets except the KuaiRec dataset. Additionally, DICE-S underperforms compared to UIPS in all datasets. This is due to the fact that although DICE-S is unbiased, it only becomes accurate with numerous logged samples, and suffers high variance when logged samples are limited, resulting in its poor performance in our experiments.
7.6 Comparison against work on direct propensity estimation.
In addition to DICE, several other works [34, 21] propose methods to directly estimate the propensity ratio without requiring a behavior policy. These methods then use the propensity estimates to prioritize instances in the replay buffer for better TD learning. To compare its effectiveness, we also included a new baseline called IPS-LFIW, which implements the approach proposed in [1] to directly estimate the propensity ratio for off-policy learning.
The average performance and standard deviations of IPS-LFIW and UIS on three synthetic datasets with different  $\tau$  are reported in Table 11. Recall that smaller  $\tau$  indicates a more skewed ground-truth logging policy. The p-value under the t-test between UIS and IPS-LFIW is also provided to
Table 11: Empirical performance of IPS-LFIW and UIS on synthetic datasets
τ=0.5	τ=1	τ=2
Algorithm	P@5	R@5	NDCG@5	P@5	R@5	NDCG@5	P@5	R@5	NDCG@5
IPS-LFIW	0.5542±1e-3	0.1568±4e-4	0.6033±1e-3	0.5472±1e-3	0.1549±5e-4	0.5975±1e-3	0.5255±6e-3	0.1485±1e-3	0.5769±5e-3
UIPS	0.5608±2e-3	0.1589±8e-4	0.6113±3e-3	0.5572±2e-3	0.1571±8e-4	0.6074±2e-3	0.5432±3e-3	0.1534±8e-4	0.5946±2e-3
p-value	2e-6	5e-6	6e-9	2e-8	2e-6	4e-10	4e-7	8e-7	1e-7
investigate the significance of improvements. Notably, UIS consistently outperformed IPS-LFIW with statistically significant improvements. One major reason for the worse performance of IPS-LFIW is that it does not consider the accuracy of the estimated propensity ratio, in a direct analogy to failing to handle uncertainty in the estimated logging probabilities in existing IPS-type algorithms.
7.7 Comparison against distributionally robust off-policy evaluation and learning.
Our work is fundamentally different from the line of work on distributionally robust off-policy evaluation and learning [32, 17, 44]. This results in different objectives that guide the use of min-max optimization.
Specifically, the work in [32, 17, 44] assumes unknown changes exist between their training and deployment environments, such as user preference drift or unforeseen events during policy execution. Thus they choose to maximize the policy value (e.g.,  $\hat{V}_{\mathrm{IPS}}(\pi)$ ) in the worst environment within an uncertainty set around the training environment,
$$
\max _ {\pi} \min _ {\mathcal {U}} \hat {V} _ {\mathrm {I P S}} (\pi).
$$
As a result, their uncertainty set is created by introducing a small perturbation to the training environment. For example, the work in [32, 17] searches the worst environment  $\mathcal{P}_{\infty}$  in the  $\sigma$ -close perturbed environments around the training environment  $\mathcal{P}_0$  (Eq.(1) in [17]):
$$
\mathcal {U} (\sigma) = \{\mathcal {P} _ {1}: \mathcal {P} _ {1} <   <   \mathcal {P} _ {0} \quad \text {a n d} \quad D _ {K L} (\mathcal {P} _ {1} | | \mathcal {P} _ {0}) \leq \sigma \}.
$$
And the work in [44] adversarially perturbs the known ground-truth logging policy  $\pi_0(\cdot |\cdot)$  in searching for the worst case (Eq.(4) in [44]):
$$
\mathcal {U} (\alpha) = \{\pi_ {u}: \max  _ {a, x} \quad \max  \{\frac {\pi_ {u} (a | x)}{\pi_ {0} (a | x)}, \frac {\pi_ {0} (a | x)}{\pi_ {u} (a | x)} \} \leq e ^ {\alpha} \}.
$$
In contrast, UISP assumes the training and deployment environments stay the same, but the ground-truth logging policy is unknown. To control the high bias and high variance caused by inaccurately and small estimated logging probabilities, UISP explicitly models the uncertainty in the estimated logging policy by incorporating a per-sample weight  $\phi_{\mathbf{x},a}$  as discussed in Eq.(4). In order to make  $\hat{V}_{\mathrm{UIPS}}(\pi_{\vartheta})$  as accurate as possible despite the unknown ground-truth logging policy  $\beta^{*}(\cdot |\cdot)$ , UISP solves a min-max optimization problem in Eq.(8). This optimization problem seeks to find the optimal  $\phi_{\mathbf{x},a}$  that minimizes the upper bound of the mean squared error (MSE) of  $\hat{V}_{\mathrm{UIPS}}$  to its ground-truth value, within an uncertainty set of the unknown ground-truth logging policy  $\beta^{*}(\cdot |\cdot)$ . The closed-form solution for the min-max optimization is also derived as in Theorem 3.2.
Furthermore, observing that work in [44] also performs optimization over an uncertainty set of the logging policy, we further adapted their method to handle the inaccuracy of the estimated logging policy, by taking  $\pi_0(\cdot|\cdot)$  as the estimated logging policy, i.e.,  $\pi_0(\cdot|\cdot) = \hat{\beta}(\cdot|\cdot)$ . We name the adapted methods as IPS-UN. Table 12 demonstrates the performance of the learned policy under IPS-UN on three synthetic datasets. The results suggest that directly applying IPS-UN to handle inaccurately estimated logging probabilities is not be a feasible solution to our problem. One important reason for the worse performance of IPS-UN is that it strives to optimize for the worst potential environment, which might not be the case in our experiment datasets. On the other hand, UIPS assumes the training and deployment environments stay same and strives to identify the optimal policy with an unknown ground-truth logging policy.
7.8 Theoretical Proofs.
Proof of Proposition 2.1:
Table 12: Empirical performance of IPS-UN on synthetic datasets.
τ = 0.5	τ = 1	τ = 2
Algorithm	P@5	R@5	NDCG@5	P@5	R@5	NDCG@5	P@5	R@5	NDCG@5
BIPS-Cap	0.5515±2e-3	0.1553±8e-4	0.6031±2e-3	0.5526±2e-3	0.1561±6e-4	0.6016±1e-3	0.5409±3e-3	0.1529±9e-4	0.5901±2e-3
IPS-UN	0.4089±3e-2	0.1152±8e-3	0.4916±2e-2	0.3911±3e-3	0.1100±9e-3	0.4754±3e-2	0.3599±4e-2	0.1009±1e-2	0.4353±4e-2
UIPS	0.5589±3e-3	0.1583±9e-4	0.6095±3e-3	0.5572±2e-3	0.1571±8e-4	0.6074±2e-3	0.5432±3e-3	0.1534±8e-4	0.5946±2e-3
Proof. Because of the linearity of expectation, we have  $\mathbb{E}_D\left[\hat{V}_{\mathrm{BIPS}}(\pi_\vartheta)\right] = \mathbb{E}_{\beta^*}\left[\frac{\pi_{\vartheta}(a|\pmb{x})}{\hat{\beta}(a|\pmb{x})} r_{\pmb{x},a}\right]$ , and thus:
$$
\begin{array}{l} \operatorname {B i a s} \left(\hat {\mathrm {V}} _ {\mathrm {B I P S}} (\pi_ {\boldsymbol {\vartheta}})\right) = \mathbb {E} _ {D} \left[ \hat {V} _ {\mathrm {B I P S}} (\pi_ {\boldsymbol {\vartheta}}) - V (\pi_ {\boldsymbol {\vartheta}}) \right] \\ = \mathbb {E} _ {\beta^ {*}} \left[ \frac {\pi_ {\boldsymbol {\vartheta}} (a | \boldsymbol {x})}{\hat {\beta} (a | \boldsymbol {x})} r _ {\boldsymbol {x}, a} \right] - \mathbb {E} _ {\beta^ {*}} \left[ \frac {\pi_ {\boldsymbol {\vartheta}} (a | \boldsymbol {x})}{\beta^ {*} (a | \boldsymbol {x})} r _ {\boldsymbol {x}, a} \right] \\ = \mathbb {E} _ {\beta^ {*}} \left[ \frac {\pi_ {\boldsymbol {\vartheta}} (a | \boldsymbol {x})}{\beta^ {*} (a | \boldsymbol {x})} r _ {\boldsymbol {x}, a} \left(\frac {\beta^ {*} (a | \boldsymbol {x})}{\hat {\beta} (a | \boldsymbol {x})} - 1\right) \right] \\ = \mathbb {E} _ {\pi_ {\boldsymbol {\vartheta}}} \left[ r _ {\boldsymbol {x}, a} \left(\frac {\beta^ {*} (a | \boldsymbol {x})}{\hat {\beta} (a | \boldsymbol {x})} - 1\right) \right]. \tag {13} \\ \end{array}
$$
Since samples are independently sampled from logging policy, the variance can be computed as:
$$
\mathrm {V a r} _ {D} \left(\hat {V} _ {\mathrm {B I P S}} (\pi_ {\pmb {\vartheta}})\right) = \frac {1}{N} \mathrm {V a r} _ {\beta^ {*}} \left(\frac {\pi_ {\pmb {\vartheta}} (a | \pmb {x})}{\hat {\beta} (a | \pmb {x})} r _ {\pmb {x}, a}\right).
$$
By re-scaling, we get:
$$
\begin{array}{l} N \cdot \operatorname {V a r} _ {D} \left(\hat {V} _ {\mathrm {B I P S}} \left(\pi_ {\boldsymbol {\vartheta}}\right)\right) = \operatorname {V a r} _ {\beta^ {*}} \left(\frac {\pi_ {\boldsymbol {\vartheta}} (a | \boldsymbol {x})}{\hat {\beta} (a | \boldsymbol {x})} r _ {\boldsymbol {x}, a}\right) \tag {14} \\ = \mathbb {E} _ {\beta^ {*}} \left[ \frac {\pi_ {\boldsymbol {\vartheta}} (a | \boldsymbol {x}) ^ {2}}{\hat {\beta} (a | \boldsymbol {x}) ^ {2}} r _ {\boldsymbol {x}, a} ^ {2} \right] - \left(\mathbb {E} _ {\beta^ {*}} \left[ \frac {\pi_ {\boldsymbol {\vartheta}} (a | \boldsymbol {x})}{\hat {\beta} (a | \boldsymbol {x})} r _ {\boldsymbol {x}, a} \right]\right) ^ {2} \\ = \mathbb {E} _ {\pi_ {\vartheta}} \left[ \frac {\pi_ {\vartheta} (a | \boldsymbol {x})}{\beta^ {*} (a | \boldsymbol {x})} \cdot \frac {\beta^ {*} (a | \boldsymbol {x}) ^ {2}}{\hat {\beta} (a | \boldsymbol {x}) ^ {2}} r _ {\boldsymbol {x}, a} ^ {2} \right] - \left(\mathbb {E} _ {\pi_ {\vartheta}} \left[ \frac {\beta^ {*} (a | \boldsymbol {x})}{\hat {\beta} (a | \boldsymbol {x})} r _ {\boldsymbol {x}, a} \right]\right) ^ {2} \\ \end{array}
$$
Proof of Theorem 3.1:
Proof. We can get:
$$
\begin{array}{l} \operatorname {M S E} \left(\hat {V} _ {\mathrm {U I P S}} \left(\pi_ {\boldsymbol {\vartheta}}\right)\right) = \mathbb {E} _ {D} \left[ \left(\hat {V} _ {\mathrm {U I P S}} \left(\pi_ {\boldsymbol {\vartheta}}\right) - V \left(\pi_ {\boldsymbol {\vartheta}}\right)\right) ^ {2} \right] \\ = \left(\mathbb {E} _ {D} \left[ \hat {V} _ {\mathrm {U I P S}} (\pi_ {\boldsymbol {\vartheta}}) - V (\pi_ {\boldsymbol {\vartheta}}) \right]\right) ^ {2} + \operatorname {V a r} _ {D} \left(\hat {V} _ {\mathrm {U I P S}} (\pi_ {\boldsymbol {\vartheta}}) - V (\pi_ {\boldsymbol {\vartheta}})\right) \\ = \left(\mathbb {E} _ {D} \left[ \hat {V} _ {\text {U I P S}} \left(\pi_ {\boldsymbol {\vartheta}}\right) - V \left(\pi_ {\boldsymbol {\vartheta}}\right) \right]\right) ^ {2} + \operatorname {V a r} _ {D} \left(\hat {V} _ {\text {U I P S}} \left(\pi_ {\boldsymbol {\vartheta}}\right)\right) \\ = \mathrm {B i a s} (\hat {V} _ {\mathrm {U I P S}} (\pi_ {\vartheta})) ^ {2} + \mathrm {V a r} (\hat {V} _ {\mathrm {U I P S}} (\pi_ {\vartheta})). \\ \end{array}
$$
We first bound the bias term:
$$
\begin{array}{l} \mathrm {B i a s} (\hat {V} _ {\mathrm {U I P S}} (\pi_ {\pmb {\vartheta}})) = \mathbb {E} _ {D} \left[ \hat {V} _ {\mathrm {U I P S}} (\pi_ {\pmb {\vartheta}}) - V (\pi_ {\pmb {\vartheta}}) \right] \\ \stackrel {(1)} {=} \mathbb {E} _ {\beta^ {*}} \left[ \frac {\pi_ {\boldsymbol {\vartheta}} (a | \boldsymbol {x})}{\hat {\beta} (a | \boldsymbol {x})} \phi_ {\boldsymbol {x}, a} r _ {\boldsymbol {x}, a} \right] - V (\pi_ {\boldsymbol {\vartheta}}) \\ = \mathbb {E} _ {\boldsymbol {\beta} ^ {*}} \left[ \frac {\pi_ {\boldsymbol {\vartheta}} (a | \boldsymbol {x})}{\hat {\beta} (a | \boldsymbol {x})} \phi_ {\boldsymbol {x}, a} r _ {\boldsymbol {x}, a} - \frac {\pi_ {\boldsymbol {\vartheta}} (a | \boldsymbol {x})}{\beta^ {*} (a | \boldsymbol {x})} r _ {\boldsymbol {x}, a} \right] \\ = \mathbb {E} _ {\beta^ {*}} \left[ r _ {\boldsymbol {x}, a} \frac {\pi_ {\boldsymbol {\vartheta}} (a | \boldsymbol {x})}{\beta^ {*} (a | \boldsymbol {x})} \cdot \left(\frac {\beta^ {*} (a | \boldsymbol {x})}{\hat {\beta} (a | \boldsymbol {x})} \phi_ {\boldsymbol {x}, a} - 1\right) \right] \\ \stackrel {(2)} {\leq} \sqrt {\mathbb {E} _ {\pi_ {\boldsymbol {\vartheta}}} \left[ r _ {\boldsymbol {x} , a} ^ {2} \frac {\pi_ {\boldsymbol {\vartheta}} (a | \boldsymbol {x})}{\beta^ {*} (a | \boldsymbol {x})} \right]} \cdot \sqrt {\mathbb {E} _ {\beta^ {*}} \left[ \left(\frac {\beta^ {*} (a | \boldsymbol {x})}{\hat {\beta} (a | \boldsymbol {x})} \phi_ {\boldsymbol {x} , a} - 1\right) ^ {2} \right]} \\ \end{array}
$$
Step (1) follows the linearity of expectation and step (2) is due to the Cauchy-Schwarz inequality. We then bound the variance term:
$$
\begin{array}{l} \operatorname {V a r} \left(\hat {V} _ {\text {U I P S}} \left(\pi_ {\boldsymbol {\vartheta}}\right)\right) = \frac {1}{N} \operatorname {V a r} _ {\beta *} \left(\frac {\pi_ {\boldsymbol {\vartheta}} (a | \boldsymbol {x})}{\hat {\beta} (a | \boldsymbol {x})} \phi_ {\boldsymbol {x}, a} r _ {\boldsymbol {x}, a}\right) \\ = \frac {1}{N} \left(\mathbb {E} _ {\beta *} \left[ \frac {\pi_ {\boldsymbol {\vartheta}} (a | \boldsymbol {x}) ^ {2}}{\hat {\beta} (a | \boldsymbol {x}) ^ {2}} \phi_ {\boldsymbol {x}, a} ^ {2} r _ {\boldsymbol {x}, a} ^ {2} \right] - \left(\mathbb {E} _ {\beta *} \left[ \frac {\pi_ {\boldsymbol {\vartheta}} (a | \boldsymbol {x})}{\hat {\beta} (a | \boldsymbol {x})} \phi_ {\boldsymbol {x}, a} r _ {\boldsymbol {x}, a} \right]\right) ^ {2}\right) \\ \leq \frac {1}{N} \mathbb {E} _ {\beta^ {*}} \left[ \frac {\pi_ {\pmb {\vartheta}} (a | \pmb {x}) ^ {2}}{\hat {\beta} (a | \pmb {x}) ^ {2}} \phi_ {\pmb {x}, a} ^ {2} r _ {\pmb {x}, a} ^ {2} \right] \leq \mathbb {E} _ {\beta^ {*}} \left[ \frac {\pi_ {\pmb {\vartheta}} (a | \pmb {x}) ^ {2}}{\hat {\beta} (a | \pmb {x}) ^ {2}} \phi_ {\pmb {x}, a} ^ {2} \right] \\ \end{array}
$$
Combining the bound of bias and variance completes the proof.
Proof of Theorem 3.2:
Proof. We first define several notations:
$T(\phi_{\pmb{x},a},\beta_{\pmb{x},a}) = \lambda \mathbb{E}_{\beta^{*}}\left[\left(\frac{\beta_{\pmb{x},a}}{\hat{\beta}(a|\pmb{x})}\phi_{\pmb{x},a} - 1\right)^{2}\right] + \mathbb{E}_{\beta^{*}}\left[\frac{\pi_{\pmb{\vartheta}}(a|\pmb{x})^{2}}{\hat{\beta}(a|\pmb{x})^{2}}\phi_{\pmb{x},a}^{2}\right].$
-  $\tilde{T} (\phi_{\pmb{x},a}) = \max_{\beta_{\pmb{x},a}\in B_{\pmb{x},a}}T(\phi_{\pmb{x},a},\beta_{\pmb{x},a})$  denotes the maximum value of inner optimization problem.
-  $T^{*} = \min_{\phi_{\mathbf{x},a}} \tilde{T}(\phi_{\mathbf{x},a}) = \min_{\phi_{\mathbf{x},a}} \max_{\beta_{\mathbf{x},a} \in B_{\mathbf{x},a}} T(\phi_{\mathbf{x},a}, \beta_{\mathbf{x},a})$  denote the optimal min-max value. And  $\phi_{\mathbf{x},a}^{*} = \arg \min_{\phi_{\mathbf{x},a}} \tilde{T}(\phi_{\mathbf{x},a})$ .
-  $B_{\pmb{x},a}^{-} \coloneqq \frac{\hat{Z}\exp(-\gamma U_{\pmb{x},a})}{Z^{*}}\hat{\beta}(a|\pmb{x})$ , and  $B_{\pmb{x},a}^{+} \coloneqq \frac{\hat{Z}\exp(\gamma U_{\pmb{x},a})}{Z^{*}}\hat{\beta}(a|\pmb{x})$ .
We first find the maximum value of the inner optimization problem, i.e.,  $\tilde{T} (\phi_{\pmb{x},a})$  for any fixed  $\phi_{\pmb{x},a}$ . And there are three cases shown in Figure 4:
Case I: When  $\frac{\hat{\beta}(a|x)}{\phi_{x,a}} \geq B_{x,a}^{+}$ ,  $\tilde{T}(\phi_{x,a})$  achieves the maximum value at  $\beta_{x,a} = B_{x,a}^{-}$ . In other words,  $\tilde{T}(\phi_{x,a}) = T(\phi_{x,a}, B_{x,a}^{-})$  when  $\phi_{x,a} \leq \frac{\hat{\beta}(a|x)}{B_{x,a}^{+}}$ .
Case II: When  $B_{\pmb{x},a}^{-} \leq \frac{\hat{\beta}(a|\pmb{x})}{\phi_{\pmb{x},a}} \leq B_{\pmb{x},a}^{+}$ , i.e.,  $\frac{Z^{*}\exp(-\gamma U_{\pmb{x},a})}{\hat{Z}} \leq \phi_{\pmb{x},a} \leq \frac{Z^{*}\exp(\gamma U_{\pmb{x},a})}{\hat{Z}}$ , then  $\tilde{T}(\phi_{\pmb{x},a})$  will be the maximum between  $T(\phi_{\pmb{x},a}, B_{\pmb{x},a}^{-})$  and  $T(\phi_{\pmb{x},a}, B_{\pmb{x},a}^{+})$ .
More specifically, when  $\frac{\hat{\beta}(a|\pmb{x})}{\phi_{\pmb{x},a}} \leq \frac{B_{\pmb{x},a}^{+} + B_{\pmb{x},a}^{-}}{2}$ , i.e.,  $\phi_{\pmb{x},a} \geq \frac{2\hat{\beta}(a|\pmb{x})}{B_{\pmb{x},a}^{+} + B_{\pmb{x},a}^{-}}$ ,  $\tilde{T}(\phi_{\pmb{x},a}) = T(\phi_{\pmb{x},a}, B_{\pmb{x},a}^{+})$ .
Otherwise when  $\phi_{\pmb{x},a} < \frac{2\hat{\beta}(a|\pmb{x})}{B_{\pmb{x},a}^{+} + B_{\pmb{x},a}^{-}}$ ,  $\tilde{T} (\phi_{\pmb{x},a}) = T(\phi_{\pmb{x},a},B_{\pmb{x},a}^{-})$
Figure 4: Three cases for maximizing the inner optimization problem.
Case III: When  $\phi_{\pmb{x},a} \geq \frac{\hat{\beta}(a|\pmb{x})}{B_{\pmb{x},a}^{-}}$ . implying  $\frac{\hat{\beta}(a|\pmb{x})}{\phi_{\pmb{x},a}} \leq B_{\pmb{x},a}^{-}$ ,  $\tilde{T}(\phi_{\pmb{x},a}) = T(\phi_{\pmb{x},a}, B_{\pmb{x},a}^{+})$ .
Overall, we get that:
$$
\tilde {T} \left(\phi_ {\boldsymbol {x}, a}\right) = \left\{ \begin{array}{l l} T \left(\phi_ {\boldsymbol {x}, a}, \boldsymbol {B} _ {\boldsymbol {x}, a} ^ {-}\right), & \phi_ {\boldsymbol {x}, a} \in \left(- \infty , \frac {2 \hat {\beta} (a | \boldsymbol {x})}{\boldsymbol {B} _ {\boldsymbol {x} , a} ^ {+} + \boldsymbol {B} _ {\boldsymbol {x} , a} ^ {-}} \right] \\ T \left(\phi_ {\boldsymbol {x}, a}, \boldsymbol {B} _ {\boldsymbol {x}, a} ^ {+}\right) & \phi_ {\boldsymbol {x}, a} \in \left[ \frac {2 \hat {\beta} (a | \boldsymbol {x})}{\boldsymbol {B} _ {\boldsymbol {x} , a} ^ {+} + \boldsymbol {B} _ {\boldsymbol {x} , a} ^ {-}}, \infty\right) \end{array} \right. \tag {15}
$$
Next we try to find the minimum value of  $\tilde{T} (\phi_{\pmb{x},a})$ . We first observe that without considering constraint on  $\phi_{\pmb{x},a}$ , when
$$
\phi_ {\boldsymbol {x}, a} ^ {+} = \frac {\lambda}{\lambda \frac {\boldsymbol {B} _ {\boldsymbol {x} , a} ^ {+}}{\hat {\beta} (a | \boldsymbol {x})} + \frac {\pi_ {\varnothing} (a | \boldsymbol {x}) ^ {2}}{\hat {\beta} (a | \boldsymbol {x}) \boldsymbol {B} _ {\boldsymbol {x} , a} ^ {+}}}
$$
$T(\phi_{\pmb{x},a},B_{\pmb{x},a}^{+})$  achieves the global minimum value. However,  $\phi_{\pmb{x},a}^{+}\leq \frac{\hat{\beta}(a|\pmb{x})}{B_{\pmb{x},a}^{+}}\leq \frac{2\hat{\beta}(a|\pmb{x})}{B_{\pmb{x},a}^{+} + B_{\pmb{x},a}^{-}}$  which implies when  $\phi_{\pmb{x},a}\in \left[\frac{2\hat{\beta}(a|\pmb{x})}{B_{\pmb{x},a}^{+} + B_{\pmb{x},a}^{-}},\infty\right)$ , the minimum value of  $T(\phi_{\pmb{x},a},B_{\pmb{x},a}^{+})$  is achieved at  $\frac{2\hat{\beta}(a|\pmb{x})}{B_{\pmb{x},a}^{+} + B_{\pmb{x},a}^{-}}$ .
On the other hand, without considering any constraint on  $\phi_{\pmb{x},a}$ , the global minimum value of  $T(\phi_{\pmb{x},a}, B_{\pmb{x},a}^{-})$  is achieved at:
$$
\phi_ {\boldsymbol {x}, a} ^ {-} = \frac {\lambda}{\lambda \frac {\boldsymbol {B} _ {\boldsymbol {x} , a} ^ {-}}{\hat {\beta} (a | \boldsymbol {x})} + \frac {\pi_ {\vartheta} (a | \boldsymbol {x}) ^ {2}}{\hat {\beta} (a | \boldsymbol {x}) \boldsymbol {B} _ {\boldsymbol {x} , a} ^ {-}}}. \tag {16}
$$
Thus if  $\phi_{\pmb{x},a}^{-}\leq \frac{2\hat{\beta}(a|\pmb{x})}{B_{\pmb{x},a}^{+} + B_{\pmb{x},a}^{-}}$ $\phi_{\pmb{x},a}^{*} = \phi_{\pmb{x},a}^{-}$  since  $T(\phi_{\pmb{x},a}^{-},B_{\pmb{x},a}^{-})\leq T(\frac{2\hat{\beta}(a|\pmb{x})}{B_{\pmb{x},a}^{+} + B_{\pmb{x},a}^{-}},B_{\pmb{x},a}^{-}) =$ $T(\frac{2\hat{\beta}(a|\pmb{x})}{B_{\pmb{x},a}^{+} + B_{\pmb{x},a}^{-}},B_{\pmb{x},a}^{+}).$  Otherwise, when  $\phi_{\pmb{x},a}^{-} > \frac{2\hat{\beta}(a|\pmb{x})}{B_{\pmb{x},a}^{+} + B_{\pmb{x},a}^{-}}$  , the minimum value of  $T(\phi_{\pmb{x},a},B_{\pmb{x},a}^{-})$  is also achieved at  $\frac{2\hat{\beta}(a|\pmb{x})}{B_{\pmb{x},a}^{+} + B_{\pmb{x},a}^{-}}$  implying  $\phi_{\pmb{x},a}^{*} = \frac{2\hat{\beta}(a|\pmb{x})}{B_{\pmb{x},a}^{+} + B_{\pmb{x},a}^{-}}$
$$
\phi_ {\boldsymbol {x}, a} ^ {*} = \min  \left(\frac {\lambda}{\lambda \frac {\boldsymbol {B} _ {\boldsymbol {x} , a} ^ {-}}{\hat {\beta} (a | \boldsymbol {x})} + \frac {\pi_ {\varnothing} (a | \boldsymbol {x}) ^ {2}}{\hat {\beta} (a | \boldsymbol {x}) \boldsymbol {B} _ {\boldsymbol {x} , a} ^ {-}}}, \frac {2 \hat {\beta} (a | \boldsymbol {x})}{\boldsymbol {B} _ {\boldsymbol {x} , a} ^ {+} + \boldsymbol {B} _ {\boldsymbol {x} , a} ^ {-}}\right) \tag {17}
$$
Let  $\eta = \frac{Z^*}{\hat{Z}}$ , we can get
$$
\phi_ {\boldsymbol {x}, a} ^ {*} = \min  \left(\frac {\lambda}{\frac {\lambda}{\eta} \exp (- \gamma U _ {\boldsymbol {x} , a}) + \frac {\eta \pi_ {\boldsymbol {\vartheta}} (a | \boldsymbol {x}) ^ {2}}{\hat {\beta} (a | \boldsymbol {x}) ^ {2} \exp (- \gamma U _ {\boldsymbol {x} , a})}}, \frac {2 \eta}{\exp (\gamma U _ {\boldsymbol {x} , a}) + \exp (- \gamma U _ {\boldsymbol {x} , a})}\right). \tag {18}
$$
Note that  $\eta \in [\exp (-\gamma U_s^{\max}),\exp (\gamma U_s^{\max})]$ , since  $\hat{Z}\exp (-\gamma U_s^{\max}) \leq Z^* = \sum_{a'}\exp (f_{\theta^*}(a'|x)) \leq \hat{Z}\exp (U_s^{\max})$ .
Proof of Corollary 3.3:
Proof. Following the similar procedure as in Theorem 3.1, we can derive the mean squared error (MSE) between  $\hat{V}_{\mathrm{BIPS}}(\pi_{\vartheta})$  and ground-truth estimator  $V(\pi_{\vartheta})$  is upper bounded as follows:
$$
\begin{array}{l} \operatorname {M S E} \left(\hat {V} _ {\mathrm {B I P S}} \left(\pi_ {\boldsymbol {\vartheta}}\right)\right) = \mathbb {E} _ {D} \left[ \left(\hat {V} _ {\mathrm {B I P S}} \left(\pi_ {\boldsymbol {\vartheta}}\right) - V \left(\pi_ {\boldsymbol {\vartheta}}\right)\right) ^ {2} \right] \\ \leq \mathbb {E} _ {\boldsymbol {\pi} \boldsymbol {\vartheta}} \left[ r _ {\boldsymbol {x}, a} ^ {2} \frac {\pi_ {\boldsymbol {\vartheta}} (a | \boldsymbol {x})}{\beta^ {*} (a | \boldsymbol {x})} \right] \cdot \mathbb {E} _ {\beta^ {*}} \left[ \left(\frac {\beta^ {*} (a | \boldsymbol {x})}{\hat {\beta} (a | \boldsymbol {x})} - 1\right) ^ {2} \right] + \mathbb {E} _ {\beta^ {*}} \left[ \frac {\pi_ {\boldsymbol {\vartheta}} (a | \boldsymbol {x}) ^ {2}}{\hat {\beta} (a | \boldsymbol {x}) ^ {2}} \right]. \\ \end{array}
$$
When  $\beta^{*}(a|\pmb{x}) \in [B_{\pmb{x},a}^{-}, B_{\pmb{x},a}^{+}]$ , with  $B_{\pmb{x},a}^{-} := \frac{\hat{Z}\exp(-\gamma U_{\pmb{x},a})}{Z^{*}}\hat{\beta}(a|\pmb{x})$  and  $B_{\pmb{x},a}^{+} := \frac{\hat{Z}\exp(\gamma U_{\pmb{x},a})}{Z^{*}}\hat{\beta}(a|\pmb{x})$ , MSE  $\left(\hat{V}_{\mathrm{BIPS}}(\pi_{\vartheta})\right)$  can be further upper bounded as follows.
For ease of illustration, we set  $\lambda = \mathbb{E}_{\pi_{\vartheta}}\left[r_{\boldsymbol{x},a}^{2}\frac{\pi_{\vartheta}(a|\boldsymbol{x})}{\beta^{*}(a|\boldsymbol{x})}\right]$  and