SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
T \left(\phi_ {\boldsymbol {x}, a}, \beta_ {\boldsymbol {x}, a}\right) = \lambda \mathbb {E} _ {\beta^ {*}} \left[ \left(\frac {\beta_ {\boldsymbol {x} , a}}{\hat {\beta} (a | \boldsymbol {x})} \phi_ {\boldsymbol {x}, a} - 1\right) ^ {2} \right] + \mathbb {E} _ {\beta^ {*}} \left[ \frac {\pi_ {\boldsymbol {\vartheta}} (a | \boldsymbol {x}) ^ {2}}{\hat {\beta} (a | \boldsymbol {x}) ^ {2}} \phi_ {\boldsymbol {x}, a} ^ {2} \right].
$$
Let  $T_{\mathrm{BIPS}}^*$  denote the upper bound of MSE  $\left(\hat{V}_{\mathrm{BIPS}}(\pi_{\vartheta})\right)$ , then we can derive that
$$
T _ {\mathrm {B I P S}} ^ {*} = \left\{ \begin{array}{l l} T (1, \boldsymbol {B} _ {\boldsymbol {x}, a} ^ {+}), & \hat {\beta} (a | \boldsymbol {x}) \leq \frac {\boldsymbol {B} _ {\boldsymbol {x} , a} ^ {+} + \boldsymbol {B} _ {\boldsymbol {x} , a} ^ {-}}{2} \\ T (1, \boldsymbol {B} _ {\boldsymbol {x}, a} ^ {-}), & \hat {\beta} (a | \boldsymbol {x}) > \frac {\boldsymbol {B} _ {\boldsymbol {x} , a} ^ {+} + \boldsymbol {B} _ {\boldsymbol {x} , a} ^ {-}}{2}. \end{array} \right. \tag {19}
$$
Recall that in Theorem 3.2, we show that the upper bound of MSE  $\left(\hat{V}_{\mathrm{UIPS}}(\pi_{\vartheta})\right)$  is as follows:
$$
T _ {\mathrm {U I P S}} ^ {*} = \left\{ \begin{array}{l l} T \left(\phi_ {\boldsymbol {x}, a} ^ {-}, \boldsymbol {B} _ {\boldsymbol {x}, a} ^ {-}\right), & \phi_ {\boldsymbol {x}, a} ^ {-} <   \frac {2 \hat {\beta} (a | \boldsymbol {x})}{\boldsymbol {B} _ {\boldsymbol {x} , a} ^ {+} + \boldsymbol {B} _ {\boldsymbol {x} , a} ^ {-}} \\ T \left(\frac {2 \hat {\beta} (a | \boldsymbol {x})}{\boldsymbol {B} _ {\boldsymbol {x} , a} ^ {+} + \boldsymbol {B} _ {\boldsymbol {x} , a} ^ {-}}, \boldsymbol {B} _ {\boldsymbol {x}, a} ^ {+}\right), & \phi_ {\boldsymbol {x}, a} ^ {-} \geq \frac {2 \hat {\beta} (a | \boldsymbol {x})}{\boldsymbol {B} _ {\boldsymbol {x} , a} ^ {+} + \boldsymbol {B} _ {\boldsymbol {x} , a} ^ {-}}. \end{array} \right. \tag {20}
$$
where  $\phi_{\pmb{x},a}^{-}$  is defined in Eq.(16). Next we show that  $T_{\mathrm{UIPS}}^{*}\geq T_{\mathrm{BIPS}}^{*}$
- When  $\hat{\beta}(a|\pmb{x}) \leq \frac{B_{\pmb{x},a}^{+} + B_{\pmb{x},a}^{-}}{2}$ , we can get  $T_{\mathrm{BIPS}}^* = T(1, B_{\pmb{x},a}^{+})$  and  $\frac{2\hat{\beta}(a|\pmb{x})}{B_{\pmb{x},a}^{+} + B_{\pmb{x},a}^{-}} \leq 1$ .
If  $\phi_{\pmb{x},a}^{-} < \frac{2\hat{\beta}(a|\pmb{x})}{B_{\pmb{x},a}^{+} + B_{\pmb{x},a}^{-}}$ , then  $T_{\mathrm{UPS}}^* = T(\phi_{\pmb{x},a}^{-}, B_{\pmb{x},a}^{-}) < T(\frac{2\hat{\beta}(a|\pmb{x})}{B_{\pmb{x},a}^{+} + B_{\pmb{x},a}^{-}}, B_{\pmb{x},a}^{-}) =$
$T\left(\frac{2\hat{\beta}(a|\pmb{x})}{\pmb{B}_{\pmb{x},a}^{+} + \pmb{B}_{\pmb{x},a}^{-}},\pmb{B}_{\pmb{x},a}^{+}\right)\leq T(1,\pmb{B}_{\pmb{x},a}^{+}).$  And if  $\phi_{\pmb{x},a}^{-}\geq \frac{2\hat{\beta}(a|\pmb{x})}{\pmb{B}_{\pmb{x},a}^{+} + \pmb{B}_{\pmb{x},a}^{-}}$ $T_{\mathrm{UIPS}}^* =$
$$
T \left(\frac {2 \hat {\beta} (a | x)}{B _ {x , a} ^ {+} + B _ {x , a} ^ {-}}, B _ {x, a} ^ {+}\right) \leq T \left(1, B _ {x, a} ^ {+}\right);
$$
- When  $\hat{\beta}(a|\pmb{x}) > \frac{B_{\pmb{x},a}^{+} + B_{\pmb{x},a}^{-}}{2}$ , we can get  $T_{\mathrm{BIPS}}^* = T(1, B_{\pmb{x},a}^-)$  and  $\frac{2\hat{\beta}(a|\pmb{x})}{B_{\pmb{x},a}^{+} + B_{\pmb{x},a}^{-}} > 1$ . If  $\phi_{\pmb{x},a}^- < \frac{2\hat{\beta}(a|\pmb{x})}{B_{\pmb{x},a}^{+} + B_{\pmb{x},a}^{-}}$ , then  $T_{\mathrm{UIPS}}^* = T(\phi_{\pmb{x},a}^-, B_{\pmb{x},a}^-) \leq T(1, B_{\pmb{x},a}^-)$ , since  $\phi_{\pmb{x},a}^-$  is a global minimum. Otherwise when  $\phi_{\pmb{x},a}^- \geq \frac{2\hat{\beta}(a|\pmb{x})}{B_{\pmb{x},a}^{+} + B_{\pmb{x},a}^{-}} > 1$ ,  $T_{\mathrm{UIPS}}^* = T(\frac{2\hat{\beta}(a|\pmb{x})}{B_{\pmb{x},a}^{+} + B_{\pmb{x},a}^{-}}, B_{\pmb{x},a}^+) = T(\frac{2\hat{\beta}(a|\pmb{x})}{B_{\pmb{x},a}^{+} + B_{\pmb{x},a}^{-}}, B_{\pmb{x},a}^-)$ .
In both cases, we have  $T_{\mathrm{UIPS}}^* \leq T_{\mathrm{BIPS}}^*$ , thus completing the proof.
Lemma 7.1. Under fixed  $\pi_{\vartheta}(a|\pmb{x})$  and  $\hat{\beta}(a|\pmb{x})$ , and  $\alpha_{\pmb{x},a} = \sqrt{\frac{\lambda}{2\eta^2} - \frac{\lambda(1 - \eta)}{\eta^2}\exp(-2\gamma U_{\pmb{x},a})}$ , we have the following observations:
- If  $\frac{\pi_{\vartheta}(a|\pmb{x})}{\hat{\beta}(a|\pmb{x})} \leq \alpha_{\pmb{x},a}$ ,  $\phi_{\pmb{x},a}^{*} = 2\eta / [\exp(\gamma U_{\pmb{x},a}) + \exp(-\gamma U_{\pmb{x},a})]$ . Otherwise  $\phi_{\pmb{x},a}^{*} = \lambda / \left[\frac{\lambda}{\eta} \exp(-\gamma U_{\pmb{x},a}) + \frac{\eta \pi_{\vartheta}(a|\pmb{x})^{2}}{\hat{\beta}^{2}(a|\pmb{x}) \exp(-\gamma U_{\pmb{x},a})}\right]$ . In other words,  $\phi_{\pmb{x},a}^{*} \leq 2\eta$  always holds.
- If  $\alpha_{\pmb{x},a} \leq \frac{\pi_{\vartheta}(a|\pmb{x})}{\hat{\beta}(a|\pmb{x})}$  and  $\frac{\pi_{\vartheta}(a|\pmb{x})}{B_{\pmb{x},a}^{-}} < \sqrt{\lambda}$ , larger  $U_{\pmb{x},a}$  brings larger  $\phi_{\pmb{x},a}^{*}$ .
- Otherwise  $\phi_{\pmb{x},a}^{*}$  decreases as  $U_{\pmb{x},a}$  increases.
Proof. The following inequality validates the first observation:
$$
\frac {\lambda}{\frac {\lambda}{\eta} \exp (- \gamma U _ {\boldsymbol {x} , a}) + \frac {\eta \pi_ {\vartheta} (a | \boldsymbol {x}) ^ {2}}{\tilde {\beta} ^ {2} (a | \boldsymbol {x}) \exp (- \gamma U _ {\boldsymbol {x} , a})}} \leq \frac {2 \eta}{\exp (\gamma U _ {\boldsymbol {x} , a}) + \exp (- \gamma U _ {\boldsymbol {x} , a})}
$$
For the second and third observations,  $\alpha_{\pmb{x},a} \leq \frac{\sqrt{\lambda}}{\sqrt{2}\eta} \leq \frac{\sqrt{\lambda}}{\eta}$ . Let  $\mathcal{L}(u) = \frac{\lambda}{\eta} \exp(-\gamma u) + \frac{\eta\pi_{\vartheta}(a|\pmb{x})^2}{\hat{\beta}(a|\pmb{x})^2 \exp(-\gamma u)}$ , we can have:
$$
\nabla_ {u} \mathcal {L} (u) = - \gamma \frac {\lambda}{\eta} \exp (- \gamma u) + \gamma \frac {\eta \pi_ {\boldsymbol {\vartheta}} (a | \boldsymbol {x}) ^ {2}}{\hat {\beta} (a | \boldsymbol {x}) ^ {2}} \exp (\gamma u)
$$
To make  $\nabla_u\mathcal{L}(u)\geq 0$  , we need  $u\geq \frac{1}{\gamma}\log \left(\frac{\sqrt{\lambda}\hat{\beta}(a|\pmb{x})}{\eta\pi_{\pmb{\vartheta}}(a|\pmb{x})}\right)$  . This implies when  $U_{\pmb {x},a}\geq$ $\frac{1}{\gamma}\log \left(\frac{\sqrt{\lambda}\hat{\beta}(a|\pmb{x})}{\eta\pi_{\pmb{\vartheta}}(a|\pmb{x})}\right),\phi_{\pmb{x},a}^{*}$  will decrease as  $U_{\pmb{x},a}$  increases; otherwise as  $U_{\pmb{x},a}$  increases,  $\phi_{\pmb{x},a}^{*}$  also increases.
More specifically, when  $\alpha_{\pmb{x},a}\leq \frac{\pi_{\pmb{\vartheta}}(a|\pmb{x})}{\hat{\beta}(a|\pmb{x})}$ , we have:
$$
U _ {\boldsymbol {x}, a} \leq \frac {1}{\gamma} \log \left(\frac {\sqrt {\lambda} \hat {\beta} (a | \boldsymbol {x})}{\eta \pi_ {\boldsymbol {\vartheta}} (a | \boldsymbol {x})}\right) \Leftrightarrow \frac {\pi_ {\boldsymbol {\vartheta}} (a | \boldsymbol {x})}{\boldsymbol {B} _ {\boldsymbol {x} , a} ^ {-}} <   \sqrt {\lambda} \quad \Leftrightarrow \quad \frac {\pi_ {\boldsymbol {\vartheta}} (a | \boldsymbol {x})}{\hat {\beta} (a | \boldsymbol {x})} <   \frac {\sqrt {\lambda}}{\eta} \exp (- \gamma U _ {\boldsymbol {x}, a}).
$$
In other words, for these samples, higher uncertainty implies smaller value of  $\pi / \hat{\beta}$ , and UIS tends to boost such safe sample with higher  $\phi_{\pmb{x},a}^{*}$ .
In other cases,  $\phi_{\pmb{x},a}^{*}$  decreases as  $U_{\pmb{x},a}$  increases. This completes the proof.
7.9 Convergence Analysis
Next we provide the convergence analysis of policy improvement under UIPS.
Definition 7.2. A function  $f: \mathbb{R}^d \to \mathbb{R}$  is  $L$ -smooth when  $\| \nabla f(x) - \nabla f(y) \|_2 \leq L \| x - y \|_2$ , for all  $x, y \in \mathbb{R}^d$ .
We first state and prove a general result, which serves as the basis to complete our convergence analysis of policy improvement under UIPS. The proof is a special case of convergence proof of stochastic gradient descent with biased gradients in [3]. Suppose we have a differentiable function  $f: \mathbb{R}^d \to \mathbb{R}$ , which is  $L$ -smooth, and attains a finite minimum value  $f^* := \min_{x \in \mathbb{R}^d} f(x)$ .
Suppose we cannot directly assess the gradient  $\nabla f(x)$ . Instead we can only assess a noisy but unbiased gradient  $\zeta (x)\in \mathbb{R}^d$  at the given  $x$  of the function  $\tilde{f} (x)$
Let  $b(x) = \nabla \tilde{f}(x) - \nabla f(x)$  denote the difference between  $\nabla \tilde{f}(x)$  and  $\nabla f(x)$ , and  $\delta(x) = \zeta(x) - \nabla \tilde{f}(x)$  denote the noise in gradients. We assume that:
$$
\| b (x) \| \leq \varphi \| \nabla f (x) \| \quad \text {a n d} \quad \mathbb {E} [ \delta (x) ] = 0 \quad \text {a n d} \quad \mathbb {E} [ \| \delta (x) \| ^ {2} | x ] \leq M \mathbb {E} [ \| \nabla \tilde {f} (x) \| ^ {2} ] + \sigma^ {2} \tag {21}
$$
where the constants  $\varphi$  and  $M$  satisfy  $0 < \varphi < 1$  and  $M \geq 0$  respectively. When running stochastic gradient descent algorithms, i.e.  $x_{k + 1} = x_k - \eta_k\zeta (x_k)$ , we have the following guarantee on the convergence of  $x_{k}$  to an approximate stationary point of  $f$ .
Theorem 7.3. Suppose  $f(\cdot)$  is differentiable and  $L$ -smooth, and the assessed approximate gradient meets the conditions in Eq. (21) with parameters  $(\sigma_k, \varphi_k)$  at iteration  $k$ . Denote  $\sigma_{\max} = \max_k \sigma_k$  and  $\varphi_{\max} = \max_k \varphi_k$ . Set the stepsizes  $\{\eta_k\}$  to  $\eta_k = \min \left\{ \frac{1}{(M + 1)L}, 1 / (\sigma_{\max} \sqrt{K}) \right\}$ , after  $K$  iterations, the stochastic gradient descent satisfies:
$$
\frac {1}{K} \sum_ {k = 1} ^ {K} \mathbb {E} \left[ \| \nabla f (x _ {k}) \| ^ {2} \right] \leq \frac {2 L (f (x _ {1}) - f ^ {*})}{K (1 - \varphi_ {\max})} + \left(L + \frac {2 (f (x _ {1}) - f ^ {*})}{(1 - \varphi_ {\max})}\right) \frac {\sigma_ {\max}}{\sqrt {K}}
$$
Proof. With  $\eta_k \leq \frac{1}{(M + 1)L}$ , we have:
$$
\begin{array}{l} f \left(x _ {k + 1}\right) \leq f \left(x _ {k}\right) + \left\langle \nabla f \left(x _ {k}\right), x _ {k + 1} - x _ {k} \right\rangle + \frac {L}{2} \left\| x _ {k + 1} - x _ {k} \right\| ^ {2} \tag {22} \\ = f (x _ {k}) - \eta_ {k} \langle \nabla f (x _ {k}), \zeta (x _ {k}) \rangle + \frac {L \eta_ {k} ^ {2}}{2} \| \zeta (x _ {k}) \| ^ {2} \\ = f (x _ {k}) - \eta_ {k} \langle \nabla f (x _ {k}), \delta (x _ {k}) + b (x _ {k}) + \nabla f (x _ {k}) \rangle + \frac {L \eta_ {k} ^ {2}}{2} \| \delta (x _ {k}) + b (x _ {k}) + \nabla f (x _ {k}) \| ^ {2} \\ \end{array}
$$
By taking expectations on both side, we have:
$$
\begin{array}{l} \mathbb {E} [ f (x _ {k + 1}) ] \leq \mathbb {E} [ f (x _ {k}) ] - \eta_ {k} \mathbb {E} [ \langle \nabla f (x _ {k}), \delta (x _ {k}) \rangle ] - \eta_ {k} \mathbb {E} [ \langle \nabla f (x _ {k}), b (x) + \nabla f (x _ {k}) \rangle ] + \frac {L \eta_ {k} ^ {2}}{2} \mathbb {E} [ \| \delta (x _ {k}) + b (x _ {k}) + \nabla f (x _ {k}) \| ^ {2} ] \\ \stackrel {(1)} {\leq} \mathbb {E} [ f (x _ {k}) ] - \eta_ {k} \mathbb {E} [ \langle \nabla f (x _ {k}), b (x) + \nabla f (x _ {k}) \rangle ] + \frac {L \eta_ {k} ^ {2}}{2} \left(\mathbb {E} [ \| \delta (x _ {k}) \| ^ {2} ] + \mathbb {E} [ \| b (x) + \nabla f (x _ {k}) \| ^ {2} ]\right) \\ \leq \mathbb {E} [ f (x _ {k}) ] - \eta_ {k} \mathbb {E} [ \langle \nabla f (x _ {k}), b (x) + \nabla f (x _ {k}) \rangle ] + \frac {L \eta_ {k} ^ {2}}{2} \left((M + 1) \mathbb {E} [ \| b (x) + \nabla f (x _ {k}) \| ^ {2} ] + \sigma_ {k} ^ {2}\right) \\ \stackrel {(2)} {\leq} \mathbb {E} [ f (x _ {k}) ] + \frac {\eta_ {k}}{2} \mathbb {E} \left[ \left(- 2 \langle \nabla f (x _ {k}), b (x) + \nabla f (x _ {k}) \rangle + \| b (x) + \nabla f (x _ {k}) \| ^ {2}\right) \right] + \frac {L \eta_ {k} ^ {2}}{2} \sigma_ {k} ^ {2} \\ \leq \mathbb {E} [ f (x _ {k}) ] + \frac {\eta_ {k}}{2} \mathbb {E} [ (- \| \nabla f (x _ {k}) \| ^ {2} + \| b (x) \| ^ {2}) ] + \frac {L \eta_ {k} ^ {2}}{2} \sigma_ {k} ^ {2} \\ \stackrel {(3)} {\leq} \mathbb {E} [ f (x _ {k}) ] + \frac {\eta_ {k}}{2} (\varphi_ {k} - 1) \mathbb {E} [ \| \nabla f (x _ {k}) \| ^ {2} ] + \frac {L \eta_ {k} ^ {2}}{2} \sigma_ {k} ^ {2} \\ \end{array}
$$
where the inequality labeled as (1) is due to  $\mathbb{E}[\delta(x)] = 0$ , inequality labeled as (2) is due to  $\eta_k \leq \frac{1}{(M + 1)L}$ , and inequality labeled as (3) is due to  $\|b(x_k)\| \leq \varphi_k\|\nabla f(x_k)\|$ .
By summing over iterations  $k = 1,2,\ldots ,K$  and re-arranging the terms, we obtain:
$$
\begin{array}{l} \frac {1}{2} \sum_ {k = 1} ^ {K} (1 - \varphi_ {k}) \eta_ {k} \mathbb {E} [ \| \nabla f (x _ {k}) \| ^ {2} ] \leq f (x _ {1}) - \mathbb {E} [ f (x _ {K + 1}) ] + \frac {L}{2} \sum_ {k = 1} ^ {K} \eta_ {k} ^ {2} \sigma_ {k} ^ {2} \\ \leq f (x _ {1}) - f ^ {*} + \frac {L}{2} \sum_ {k = 1} ^ {K} \eta_ {k} ^ {2} \sigma_ {k} ^ {2} \\ \end{array}
$$
where the last inequality follows from  $f(x_{K + 1}) \geq f^{*}$ . Since  $\eta_k = \min \left\{\frac{1}{(M + 1)L}, \frac{1}{\sigma_{\max}\sqrt{K}}\right\}$ ,  $\forall k = 1, \ldots, K$ , we can obtain:
$$
(1 - \varphi_ {\max }) \eta_ {1} \sum_ {k = 1} ^ {K} \mathbb {E} [ \| \nabla f (x _ {k}) \| ^ {2} ] \leq 2 (f (x _ {1}) - f ^ {*}) + L K \eta_ {1} ^ {2} \sigma_ {\max } ^ {2}
$$
Dividing both sides of the above inequality by  $K\eta_{1}(1 - \varphi_{\max})$ , we obtain the following,
$$
\begin{array}{l} \frac {1}{K} \sum_ {k = 1} ^ {K} \mathbb {E} [ \| \nabla f (x _ {k}) \| ^ {2} ] \leq \frac {2 (f (x _ {1}) - f ^ {*}) + L K \eta_ {1} ^ {2} \sigma_ {\max } ^ {2}}{K \eta_ {1} (1 - \varphi_ {\max })} \\ \leq \frac {2 (f (x _ {1}) - f ^ {*})}{K (1 - \varphi_ {\max})} \max \{L, \sigma_ {\max} \sqrt {K} \} + L \sigma_ {\max} ^ {2} \frac {1}{\sigma_ {\max} \sqrt {K}} \\ \leq \frac {2 L (f (x _ {1}) - f ^ {*})}{K (1 - \varphi_ {\max})} + \left(L + \frac {2 (f (x _ {1}) - f ^ {*})}{(1 - \varphi_ {\max})}\right) \frac {\sigma_ {\max}}{\sqrt {K}} \\ \end{array}
$$
Given this general result, we can now prove Theorem 3.4 by showing that the gradient in UISP meets the requirements in Theorem 7.3.
Proof of Theorem 3.4:
Proof. UISP aims to maximize the expected return  $V(\pi_{\vartheta})$ . Therefore, we can utilize Theorem 7.3 by setting  $f = -V(\pi_{\vartheta})$ . Since  $f^{*} = -V_{\max}$  and the expected reward is always non-negative, it follows that  $f(x_{1}) - f^{*} \leq V_{\max}$ .
We first introduce some additional notations. Let  $\rho_{\vartheta}^{*}(\pmb{x},a) = \frac{\pi_{\vartheta}(a|\pmb{x})}{\beta^{*}(a|\pmb{x})}$  denote the propensity score under ground-truth logging policy, and  $\hat{\rho}_{\vartheta}(\pmb{x},a) = \frac{\pi_{\vartheta}(a|\pmb{x})}{\hat{\beta}(a|\pmb{x})}\phi_{\pmb{x},a}^{*}$  represent the propensity score of UIPS. Recall that  $\phi_{\pmb{x},a}^{*}$  is derived through solving the optimization problem in Eq (8). Let  $g_{\vartheta}(\pmb{x},a) = \frac{\partial\pi_{\vartheta}(a|\pmb{x})}{\partial\vartheta}$ . The true off-policy policy gradient is computed as follows:
$$
\nabla V (\pi_ {\vartheta}) = \mathbb {E} _ {\beta^ {*}} [ \rho^ {*} r g _ {\vartheta} ],
$$
For UIS, the approximate policy gradient in each batch with batch size as  $B$  is:
$$
\nabla \hat {V} _ {\mathrm {U I P S}} (\pi_ {\vartheta}) = \frac {1}{B} \sum_ {i = 1} ^ {B} \hat {\rho} _ {i} r _ {i} g _ {\vartheta} ^ {i},
$$
which is an unbiased estimate of :
$$
\nabla V _ {\mathrm {U I P S}} \left(\pi_ {\vartheta}\right) = \mathbb {E} _ {\beta^ {*}} \left[ \hat {\rho} r g _ {\vartheta} \right].
$$
To utilize Theorem 7.3, we set  $\nabla f = -\nabla V(\pi_{\vartheta})$ ,  $\nabla \tilde{f} = -\nabla V_{\mathrm{UIPS}}(\pi_{\vartheta})$ , and  $\zeta = \nabla \hat{V}_{\mathrm{UIPS}}(\pi_{\vartheta})$ . We will now demonstrate that the assumptions in Eq. (21) can be satisfied.
Let  $\varphi_{\vartheta} = \max \left\{\left|\frac{\hat{\rho}_{\vartheta}(\boldsymbol{x},a)}{\rho_{\vartheta}^{*}(\boldsymbol{x},a)} -1\right|\right\}$ , We first have:
$$
\begin{array}{l} \left\| b (\vartheta) \right\| = \left\| \nabla V _ {\mathrm {U I P S}} \left(\pi_ {\vartheta}\right) - \nabla V \left(\pi_ {\vartheta}\right) \right\| \\ = \| \mathbb {E} _ {\beta^ {*}} [ (\hat {\rho} - \rho^ {*}) r g _ {\vartheta} ] \| = \| \mathbb {E} _ {\beta^ {*}} [ \rho^ {*} (\frac {\hat {\rho}}{\rho^ {*}} - 1) r g _ {\vartheta} ] \| \\ \leq \varphi_ {\vartheta} \| \mathbb {E} _ {\beta^ {*}} \left[ \rho^ {*} r g _ {\vartheta} \right] \| \tag {23} \\ \end{array}
$$
And next we show that  $0 < \varphi_{\vartheta} < 1$
$$
\varphi_ {\vartheta} = \max  \left\{\left| \frac {\hat {\rho} _ {\vartheta} (\boldsymbol {x} , a)}{\rho_ {\vartheta} ^ {*} (\boldsymbol {x} , a)} - 1 \right| \right\} = \max  \left\{\left| \frac {\beta^ {*} (a | \boldsymbol {x})}{\hat {\beta} (a | \boldsymbol {x})} \cdot \phi_ {\boldsymbol {x}, a} ^ {*} - 1 \right| \right\} \tag {24}
$$
Recall from Eq.(17) in proof of Theorem 3.2, we have :
$$
\phi_ {\boldsymbol {x}, a} ^ {*} = \min  \left(\frac {\lambda}{\lambda \frac {\boldsymbol {B} _ {\boldsymbol {x} , a} ^ {-}}{\hat {\beta} (a | \boldsymbol {x})} + \frac {\pi_ {\boldsymbol {\varnothing}} (a | \boldsymbol {x}) ^ {2}}{\hat {\beta} (a | \boldsymbol {x}) \boldsymbol {B} _ {\boldsymbol {x} , a} ^ {-}}}, \frac {2 \hat {\beta} (a | \boldsymbol {x})}{\boldsymbol {B} _ {\boldsymbol {x} , a} ^ {+} + \boldsymbol {B} _ {\boldsymbol {x} , a} ^ {-}}\right)
$$
where  $B_{\pmb{x},a}^{-} \coloneqq \frac{\hat{Z}\exp(-\gamma U_{\pmb{x},a})}{Z^{*}}\hat{\beta}(a|\pmb{x})$ , and  $B_{\pmb{x},a}^{+} \coloneqq \frac{\hat{Z}\exp(\gamma U_{\pmb{x},a})}{Z^{*}}\hat{\beta}(a|\pmb{x})$ . Thus we have:
$$
\frac {\beta^ {*} (a | \boldsymbol {x})}{\hat {\beta} (a | \boldsymbol {x})} \cdot \phi_ {\boldsymbol {x}, a} ^ {*} = \min  \left(\frac {\lambda}{\lambda \frac {\boldsymbol {B} _ {\boldsymbol {x} , a} ^ {-}}{\beta^ {*} (a | \boldsymbol {x})} + \frac {\pi_ {\vartheta} (a | \boldsymbol {x}) ^ {2}}{\beta^ {*} (a | \boldsymbol {x}) \boldsymbol {B} _ {\boldsymbol {x} , a} ^ {-}}}, \frac {2}{\frac {\boldsymbol {B} _ {\boldsymbol {x} , a} ^ {+}}{\beta^ {*} (a | \boldsymbol {x})} + \frac {\boldsymbol {B} _ {\boldsymbol {x} , a} ^ {-}}{\beta^ {*} (a | \boldsymbol {x})}}\right) \tag {25}
$$
Since  $B_{\pmb{x},a}^{+} \geq \beta^{*}(a|\pmb{x})$ , thus  $0 \leq \frac{\beta^{*}(a|\pmb{x})}{\hat{\beta}(a|\pmb{x})} \cdot \phi_{\pmb{x},a}^{*} \leq 2$ , implying  $0 < \varphi_{\vartheta} < 1$ .
Also, since  $\nabla \hat{V}_{\mathrm{UIPS}}(\pi_{\vartheta})$  is an unbiased estimate of  $\nabla V_{\mathrm{UIPS}}(\pi_{\vartheta})$ , we have:
$$
\mathbb {E} [ \delta (\vartheta) ] = \mathbb {E} \left[ \nabla \hat {V} _ {\mathrm {U I P S}} \left(\pi_ {\vartheta}\right) - \nabla V _ {\mathrm {U I P S}} \left(\pi_ {\vartheta}\right) \right] = \mathbf {0} \tag {26}
$$
Finally, we have the following,
$$
\begin{array}{l} \mathbb {E} [ \| \delta (\vartheta) \| ^ {2} ] = \mathbb {E} \left[ \| \nabla \hat {V} _ {\mathrm {U I P S}} (\pi_ {\vartheta}) - \nabla V _ {\mathrm {U I P S}} (\pi_ {\vartheta}) \| ^ {2} \right] = \mathbb {E} \left[ \left\| \frac {1}{B} \sum_ {i = 1} ^ {B} \left(\hat {\rho} _ {i} r _ {i} g _ {\vartheta} ^ {i} - \mathbb {E} _ {\beta^ {*}} [ \hat {\rho} r g _ {\vartheta} ]\right) \right\| ^ {2} \right] \\ \leq \frac {1}{B ^ {2}} \mathbb {E} \left[ \left(\sum_ {i = 1} ^ {B} \| \hat {\rho} _ {i} r _ {i} g _ {\vartheta} ^ {i} - E _ {\beta^ {*}} [ \hat {\rho} r g _ {\vartheta} ] \|\right) ^ {2} \right] \leq \frac {1}{B} \mathbb {E} \left[ \sum_ {i = 1} ^ {B} \| \hat {\rho} _ {i} r _ {i} g _ {\vartheta} ^ {i} - \mathbb {E} _ {\beta^ {*}} [ \hat {\rho} r g _ {\vartheta} ] \| ^ {2} \right] \\ \leq \mathbb {E} _ {\beta^ {*}} \left[ \| \hat {\rho} r g _ {\vartheta} - \mathbb {E} _ {\beta^ {*}} \left[ \hat {\rho} r g _ {\vartheta} \right] \| ^ {2} \right] \stackrel {(1)} {=} \mathbb {E} _ {\beta^ {*}} \left[ \| \hat {\rho} r g _ {\vartheta} \| ^ {2} \right] - \| \mathbb {E} _ {\beta^ {*}} \left[ \hat {\rho} r g _ {\vartheta} \right] \| ^ {2} \\ \leq \mathbb {E} _ {\beta^ {*}} \left[ \| \hat {\rho} r g _ {\vartheta} \| ^ {2} \right] \leq G _ {\max } ^ {2} \Phi \tag {27} \\ \end{array}
$$
where the equality labeled as (1) is due to  $E[||Y - E[Y]||^2] = E[||Y||^2] - ||E[Y]||^2$ .
Hence, by applying Theorem 7.3, we have:
$$
\frac {1}{K} \sum_ {k = 1} ^ {K} \mathbb {E} [ \| \nabla V (\pi_ {\vartheta_ {k}}) \| ^ {2} \leq \frac {2 L V _ {\max}}{K (1 - \varphi_ {\max})} + \left(L + \frac {2 V _ {\max}}{(1 - \varphi_ {\max})}\right) \frac {G _ {\max} \sqrt {\Phi}}{\sqrt {K}} \tag {28}
$$