SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Therefore, KKT conditions (conditions 1-4) are satisfied.
B.2 Proof and Discussion the Proposition 3.5
Proposition B.2. The  $\lambda, \eta$  are dual variables and are solved via the convex optimization problem of the following formula.
$$
\mathcal {L} (\lambda , \eta) = \min  _ {\lambda \geq 0, \eta \geq 0} \mathbb {E} _ {\rho (s _ {0})} \left[ \eta \log \sum_ {a _ {t}} \pi_ {\theta} \left(a _ {t} | s _ {t}\right) \exp \left(\frac {Q ^ {r c} \left(s _ {t} , a _ {t}\right)}{\eta}\right) + \lambda^ {T} \bar {c} _ {i} + \eta \epsilon \right]. \tag {27}
$$
Proof. By rearranging terms and taking the expectation over  $a_{t}$ , Eq. (21) can be rewritten as:
$$
\exp \left(\frac {\eta + \delta}{\eta}\right) \sum_ {a _ {t}} q ^ {*} \left(a _ {t} \mid s _ {t}\right) = \sum_ {a _ {t}} \pi_ {\theta} \left(a _ {t} \mid s _ {t}\right) \exp \left[ \frac {Q ^ {r c} \left(s _ {t} , a _ {t}\right)}{\eta} \right], \tag {28}
$$
where the  $\sum_{a_t}q^* (a_t|s_t) = 1$ , thus the Eq. (28) can be rewritten as:
$$
\frac {\eta + \delta}{\eta} = \log \sum_ {a _ {t}} \pi_ {\theta} \left(a _ {t} \mid s _ {t}\right) \exp \left[ \frac {Q ^ {r c} \left(s _ {t} , a _ {t}\right)}{\eta} \right]. \tag {29}
$$
By substituting Eq. (29) and (26) into Eq. (22) to eliminate  $q(a_{t}|s_{t})$ , we can rewrite Eq. (22) as:
$$
\begin{array}{l} \mathcal {L} (\lambda , \eta , \delta) = \min _ {\lambda \geq 0, \eta \geq 0, \delta} \mathbb {E} _ {\rho (s _ {0})} \bigg [ \sum_ {a _ {t}} q ^ {*} (a _ {t} | s _ {t}) Q ^ {r c} (s _ {t}, a _ {t}) - \eta \sum_ {a _ {t}} q ^ {*} (a _ {t} | s _ {t}) \log \frac {q ^ {*} (a | s)}{\pi_ {\theta} (a | s)} \\ \left. - \delta \sum_ {a _ {t}} q ^ {*} \left(a _ {t} \mid s _ {t}\right) + \lambda^ {T} \bar {c} _ {i} + \eta \epsilon + \delta \right] \\ = \min  _ {\lambda \geq 0, \eta \geq 0, \delta} \mathbb {E} _ {\rho (s _ {0})} \left[ \sum_ {a _ {t}} q ^ {*} \left(a _ {t} \mid s _ {t}\right) Q ^ {r c} \left(s _ {t}, a _ {t}\right) - \eta \sum_ {a _ {t}} q ^ {*} \left(a _ {t} \mid s _ {t}\right) \left[ \frac {Q ^ {r c} \left(s _ {t} , a _ {t}\right) - \eta - \delta}{\eta} \right] \right. \tag {30} \\ \left. - \delta \sum_ {a _ {t}} q ^ {*} \left(a _ {t} \mid s _ {t}\right) + \lambda^ {T} \bar {c} _ {i} + \eta \epsilon + \delta \right] \\ = \min  _ {\lambda \geq 0, \eta \geq 0, \delta} \mathbb {E} _ {\rho (s _ {0})} \left[ \sum_ {a _ {t}} q ^ {*} (a _ {t} | s _ {t}) [ \eta + \delta ] - \delta \sum_ {a _ {t}} q ^ {*} (a _ {t} | s _ {t}) + \lambda^ {T} \bar {c} _ {i} + \eta \epsilon + \delta \right] \\ = \min _ {\lambda \geq 0, \eta \geq 0, \delta} \mathbb {E} _ {\rho (s _ {0})} \left[ \eta \log \sum_ {a _ {t}} \pi_ {\theta} (a _ {t} | s _ {t}) \exp \left[ \frac {Q ^ {r c} (s _ {t} , a _ {t})}{\eta} \right] + \lambda^ {T} \bar {c} _ {i} + \eta \epsilon \right]. \\ \end{array}
$$
Since Eq. (30) eliminates  $\delta$ , the dual optimization problem of  $\lambda$  and  $\eta$  can be written as:
$$
\mathcal {L} (\lambda , \eta) = \min  _ {\lambda \geq 0, \eta \geq 0} \mathbb {E} _ {\rho (s _ {0})} \left[ \eta \log \sum_ {a _ {t}} \pi_ {\theta} \left(a _ {t} \mid s _ {t}\right) \exp \left[ \frac {Q ^ {r c} \left(s _ {t} , a _ {t}\right)}{\eta} \right] + \lambda^ {T} \bar {c} _ {i} + \eta \epsilon \right]. \tag {31}
$$
B.3 Proof and Discussion the Proposition 3.6
Proposition B.3. For any  $\pi_{\mathcal{M}}(a_t|s_t)$  with supp  $\pi_{\mathcal{M}}(a_t|s_t) \subset \hat{\pi}_{\beta}, \forall s_t, a_t \in D$ , the  $Q$ -value function  $Q^r$  via iterating Eq. (12) satisfies:
$$
\hat {Q} ^ {r} (s _ {t}, a _ {t}) \leq Q ^ {r} (s _ {t}, a _ {t}) - \kappa \left[ (I - \gamma P ^ {\pi}) ^ {- 1} \frac {\pi_ {\mathcal {M}}}{\hat {\pi} _ {\beta}} \right] (s _ {t}, a _ {t}) + \left[ (I - \gamma P ^ {\pi}) ^ {- 1} \frac {C _ {r , p , \delta} R _ {m a x}}{(1 - \gamma) \sqrt {| \mathcal {D} |}} \right] (s _ {t}, a _ {t}). \tag {32}
$$
Thus, if  $\kappa \geq \frac{C_{r,p,\delta}R_{max}}{(1 - \gamma)\sqrt{|\mathcal{D}(s_t,a_t)|}}\left[\frac{\pi_{\mathcal{M}}(a_t|s_t)}{\pi_\beta(a_t|s_t)}\right]^{-1}$ , the iterative update Eq. (12) can guaranteees  $\hat{Q}^r \leq Q^r$ .
Proof. The iterative equation for updating the reward Q-value, which involves an additional minimization of the Q-value and the standard Bellman error objective, is expressed as follows:
$$
\hat {Q} _ {k + 1} ^ {r} \leftarrow \arg \min  \frac {1}{2} \mathbb {E} _ {s _ {t}, a _ {t} \sim D} \left[ \left[ \left(Q ^ {r} \left(s _ {t}, a _ {t}\right)\right) - \hat {\mathcal {B}} ^ {\pi} \hat {Q} _ {k} ^ {r} \left(s _ {t}, a _ {t}\right) \right] ^ {2} \right] + \tag {33}
$$
$$
\kappa \mathbb {E} _ {s _ {t} \sim D, a _ {t} \sim \pi_ {\mathcal {M}} (\cdot | s _ {t})} \left[ Q ^ {r} (s _ {t}, a _ {t}) \right],
$$
where the  $\hat{\mathcal{B}}^{\pi}\hat{Q}^{r}(s_{t},a_{t}) = r(s_{t},a_{t}) + \gamma \hat{Q}^{r}(s_{t + 1},a_{t + 1})$ . With importance sampling, the Eq. (33) can be written as:
$$
\hat {Q} _ {k + 1} ^ {r} \leftarrow \arg \min  _ {Q ^ {r}} \frac {1}{2} \mathbb {E} _ {s _ {t}, a _ {t} \sim D} \left[ \left[ \left(Q ^ {r} \left(s _ {t}, a _ {t}\right)\right) - \hat {\mathcal {B}} ^ {\pi} \hat {Q} _ {k} ^ {r} (s, a) \right] ^ {2} \right] + \tag {34}
$$
$$
\kappa \mathbb {E} _ {s _ {t} \sim D, a _ {t} \sim \hat {\pi} _ {\beta} (\cdot | s _ {t})} \left[ \frac {\pi_ {\mathcal {M}} (a _ {t} | s _ {t})}{\hat {\pi} _ {\beta} (a _ {t} | s _ {t})} Q ^ {r} (s _ {t}, a _ {t}) \right].
$$
Setting the derivative of Eq. (34) to 0, we obtain the following expression:
$$
\hat {Q} _ {k + 1} ^ {r} \left(s _ {t}, a _ {t}\right) = \hat {\mathcal {B}} ^ {\pi} \hat {Q} _ {k} \left(s _ {t}, a _ {t}\right) - \kappa \frac {\pi_ {\mathcal {M}} \left(a _ {t} \mid s _ {t}\right)}{\hat {\pi} _ {\beta} \left(a _ {t} \mid s _ {t}\right)}. \tag {35}
$$
Note that the Eq. (34) utilizes the empirical Bellman operator  $\mathcal{B}^{\pi}$  instead of the actual Bellman Operator  $\mathcal{B}^{\pi}$ . Following the related work [23, 31], we employ the concentration properties to determine the error. For any  $\forall s_t, a_t \in D$ , with the probability  $\leq 1 - \delta$ , the error can be written as:
$$
\left| \mathcal {B} ^ {\pi} \hat {Q} \left(s _ {t}, a _ {t}\right) - \hat {\mathcal {B}} ^ {\pi} \hat {Q} \left(s _ {t}, a _ {t}\right) \right| \leq \frac {C _ {r , p , \delta} R _ {\max }}{\left(1 - \gamma\right) \sqrt {\left| \mathcal {D} \left(s _ {t} , a _ {t}\right) \right|}}, \tag {36}
$$
where  $C_{r,p,\delta}$  is a constant depending on the concentration properties of  $r(s_t, a_t)$ ,  $p(s_{t+1}|s_t, a_t)$  and the  $\delta, \delta \in (0,1)$ . The  $(\mathcal{D}(s_t, a_t))^{-\frac{1}{2}}$  represents a vector of size  $|\mathcal{S}||\mathcal{A}|$  containing the square root inverse count of each state action transition.  $R_{max} = \max_{s_t, a_t \sim D} \{r(s_t, a_t)\}$  represents the maximum reward obtained in a single step. Based on the inequality of Eq. (36) and Eq. (35) we obtain:
$$
\begin{array}{l} \hat {Q} _ {k + 1} ^ {r} (s _ {t}, a _ {t}) = \hat {\mathcal {B}} ^ {\pi} \hat {Q} _ {k} ^ {r} (s _ {t}, a _ {t}) - \kappa \frac {\pi_ {\mathcal {M}} (a _ {t} | s _ {t})}{\hat {\pi} _ {\beta} (a _ {t} | s _ {t})} \\ \leq \mathcal {B} ^ {\pi} \hat {Q} _ {k} ^ {r} \left(s _ {t}, a _ {t}\right) + \frac {C _ {r , p , \delta} R _ {\max }}{\left(1 - \gamma\right) \sqrt {\left| \mathcal {D} \left(s _ {t} , a _ {t}\right) \right|}} - \kappa \frac {\pi_ {\mathcal {M}} \left(a _ {t} \mid s _ {t}\right)}{\hat {\pi} _ {\beta} \left(a _ {t} \mid s _ {t}\right)}. \tag {37} \\ \end{array}
$$
The fixed point in the updating process of Eq. (37) can be expressed as:
$$
\begin{array}{l} \hat {Q} ^ {r} (s _ {t}, a _ {t}) \leq \mathcal {B} ^ {\pi} \hat {Q} ^ {r} (s _ {t}, a _ {t}) + \frac {C _ {r , p , \delta} R _ {\max}}{(1 - \gamma) \sqrt {| \mathcal {D} (s _ {t} , a _ {t}) |}} - \kappa \frac {\pi_ {\mathcal {M}} (a _ {t} | s _ {t})}{\hat {\pi} _ {\beta} (a _ {t} | s _ {t})} \\ \leq r + \gamma P ^ {\pi} \hat {Q} ^ {r} \left(s _ {t}, a _ {t}\right) + \frac {C _ {r , p , \delta} R _ {\max}}{\left(1 - \gamma\right) \sqrt {\left| \mathcal {D} \left(s _ {t} , a _ {t}\right) \right|}} - \kappa \frac {\pi_ {\mathcal {M}} \left(a _ {t} \mid s _ {t}\right)}{\hat {\pi} _ {\beta} \left(a _ {t} \mid s _ {t}\right)} \\ \leq Q ^ {r} \left(s _ {t}, a _ {t}\right) - \gamma P ^ {\pi} Q ^ {r} \left(s _ {t}, a _ {t}\right) + \gamma P ^ {\pi} \hat {Q} ^ {r} \left(s _ {t}, a _ {t}\right) + \frac {C _ {r , p , \delta} R _ {\max }}{\left(1 - \gamma\right) \sqrt {\left| \mathcal {D} \left(s _ {t} , a _ {t}\right) \right|}} - \kappa \frac {\pi_ {\mathcal {M}} \left(a _ {t} \mid s _ {t}\right)}{\hat {\pi} _ {\beta} \left(a _ {t} \mid s _ {t}\right)}, \tag {38} \\ \end{array}
$$
$$
(I - \gamma P ^ {\pi}) \hat {Q} ^ {r} \left(s _ {t}, a _ {t}\right) \leq (I - \gamma P ^ {\pi}) Q ^ {r} \left(s _ {t}, a _ {t}\right) + \frac {C _ {r , p , \delta} R _ {\max }}{\left(1 - \gamma\right) \sqrt {\left| \mathcal {D} \left(s _ {t} , a _ {t}\right) \right|}} - \kappa \frac {\pi_ {\mathcal {M}} \left(a _ {t} \mid s _ {t}\right)}{\hat {\pi} _ {\beta} \left(a _ {t} \mid s _ {t}\right)}, \tag {39}
$$
$$
\begin{array}{l} \hat {Q} ^ {r} \left(s _ {t}, a _ {t}\right) \leq Q ^ {r} \left(s _ {t}, a _ {t}\right) + \left(I - \gamma P ^ {\pi}\right) ^ {- 1} \left[ \frac {C _ {r , p , \delta} R _ {\max }}{\left(1 - \gamma\right) \sqrt {\left| \mathcal {D} \left(s _ {t} , a _ {t}\right) \right|}} - \kappa \frac {\pi_ {\mathcal {M}} \left(a _ {t} \mid s _ {t}\right)}{\hat {\pi} _ {\beta} \left(a _ {t} \mid s _ {t}\right)} \right] \tag {40} \\ \leq Q ^ {r} \left(s _ {t}, a _ {t}\right) - \kappa \left(I - \gamma P ^ {\pi}\right) ^ {- 1} \frac {\pi_ {\mathcal {M}} \left(a _ {t} \mid s _ {t}\right)}{\hat {\pi} _ {\beta} \left(a _ {t} \mid s _ {t}\right)} + \left(I - \gamma P ^ {\pi}\right) ^ {- 1} \frac {C _ {r , p , \delta} R _ {m a x}}{\left(1 - \gamma\right) \sqrt {\left| \mathcal {D} \left(s _ {t} , a _ {t}\right) \right|}}. \\ \end{array}
$$
To guarantee the lower bound of  $\hat{Q}^r$ , we eliminate the potential overestimation caused by  $\frac{\tilde{C}_{r,p,\delta}R_{max}}{(1 - \gamma)\sqrt{|\mathcal{D}(s_t,a_t)|}}$  via choosing an appropriate  $\kappa$ . Since  $(I - \gamma P^{\pi})^{-1}$  is a matrix of non-negative entries, the  $\kappa$  that guarantees the lower bound of  $Q^r$  needs to satisfy:
$$
\kappa \geq \frac {C _ {r , p , \delta} R _ {\max}}{(1 - \gamma) \sqrt {| \mathcal {D} (s _ {t} , a _ {t}) |}} \left[ \frac {\pi_ {\mathcal {M}} (a _ {t} | s _ {t})}{\hat {\pi} _ {\beta} (a _ {t} | s _ {t})} \right] ^ {- 1}. \tag {41}
$$
B.4 Proof and Discussion the Proposition 3.7
Proposition B.4. When  $\pi_{\mathcal{M}}(a_t|s_t) = \pi_\theta (a_t|s_t)$ , according to Eq. (13), we obtain a lower bound for the true value of  $V^{r} = \mathbb{E}_{a_{t}\sim \pi_{\theta}(a_{t}|s_{t})}[Q^{r}(s_{t},a_{t})]$  that satisfies the following inequality:
$$
\hat {V} ^ {r} \left(s _ {t}\right) \leq V ^ {r} \left(s _ {t}\right) - \kappa \left[ \left(I - \gamma P ^ {\pi}\right) ^ {- 1} \mathbb {E} _ {a _ {t} \sim \pi_ {\theta}} \left[ \frac {\pi_ {\theta} \left(a _ {t} \mid s _ {t}\right)}{\hat {\pi} _ {\beta} \left(a _ {t} \mid s _ {t}\right)} - 1 \right] \right] \left(s _ {t}\right) + \left[ \left(I - \gamma P ^ {\pi}\right) ^ {- 1} \frac {C _ {r , p , \delta} R _ {\max }}{(1 - \gamma) \sqrt {| \mathcal {D} |}} \right] \left(s _ {t}\right). \tag {42}
$$
Thus, if  $\kappa \geq \frac{C_{r,p,\delta}R_{max}}{(1 - \gamma)\sqrt{|\mathcal{D}(s_t)|}}\left[\mathbb{E}_{a_t}\sim \hat{\pi}_\beta \left[\frac{\pi_\theta(a_t|s_t)}{\pi_\theta(a_t|s_t)} -1\right]\right]^{-1}$ , the Eq. (13) can guarantee  $\hat{V}^r\leq V^r$
Proof. The following Eq. (43) represents the iterative formula for the reward Q-value with an additional maximization term introduced under the sample distribution  $\hat{\pi}_{\beta}$ .
$$
\begin{array}{l} \hat {Q} _ {k + 1} ^ {r} \leftarrow \arg \min  \frac {1}{2} \mathbb {E} _ {s _ {t}, a _ {t} \sim D} \left[ \left[ \left(Q ^ {r} \left(s _ {t}, a _ {t}\right)\right) - \hat {\mathcal {B}} ^ {\pi} \hat {Q} _ {k} ^ {r} \left(s _ {t}, a _ {t}\right) \right] ^ {2} \right] + \tag {43} \\ \kappa \left[ \mathbb {E} _ {\substack {s _ {t} \sim D \\ a _ {t} \sim \pi_ {\mathcal{M}} (\cdot | s _ {t})}} Q ^ {r} (s _ {t}, a _ {t}) - \mathbb {E} _ {\substack {s _ {t} \sim D \\ a _ {t} \sim \hat {\pi} _ {\beta} (\cdot | s _ {t})}} Q ^ {r} (s _ {t}, a _ {t}) \right]. \\ \end{array}
$$
where  $\kappa$  is a tradeoff factor. With importance sampling, the Eq. (43) can be rewritten as:
$$
\begin{array}{l} \hat {Q} _ {k + 1} ^ {r} \leftarrow \arg \min  \frac {1}{2} \mathbb {E} _ {s _ {t}, a _ {t} \sim D} \left[ \left[ \left(Q ^ {r} \left(s _ {t}, a _ {t}\right)\right) - \hat {\mathcal {B}} ^ {\pi} \hat {Q} _ {k} ^ {r} \left(s _ {t}, a _ {t}\right) \right] ^ {2} \right] + \\ \kappa \left[ \mathbb {E} _ {a _ {t} \sim \hat {\pi} _ {\beta} (\cdot | s _ {t})} \frac {\pi_ {\mathcal {M}} \left(a _ {t} \mid s _ {t}\right)}{\hat {\pi} _ {\beta} \left(a _ {t} \mid s _ {t}\right)} Q ^ {r} \left(s _ {t}, a _ {t}\right) - \mathbb {E} _ {a _ {t} \sim \hat {\pi} _ {\beta} (\cdot | s _ {t})} Q ^ {r} \left(s _ {t}, a _ {t}\right) \right]. \end{array} \tag {44}
$$
Setting the derivative of Eq. (44) to 0, we obtain the following expression:
$$
\hat {Q} _ {k + 1} ^ {r} \left(s _ {t}, a _ {t}\right) = \hat {\mathcal {B}} ^ {\pi} \hat {Q} _ {k} \left(s _ {t}, a _ {t}\right) - \kappa \left[ \frac {\pi_ {\mathcal {M}} \left(a _ {t} \mid s _ {t}\right)}{\hat {\pi} _ {\beta} \left(a _ {t} \mid s _ {t}\right)} - 1 \right] \tag {45}
$$
By taking the expectation of both side of Eq. (45) with respect to the action, we obtain:
$$
\begin{array}{l} \hat {V} _ {k + 1} ^ {r} \left(s _ {t}\right) = \hat {\mathcal {B}} ^ {\pi} \hat {V} _ {k} \left(s _ {t}\right) - \kappa \sum_ {a _ {t}} \pi_ {\theta} \left(a _ {t} \mid s _ {t}\right) \left[ \frac {\pi_ {\mathcal {M}} \left(a _ {t} \mid s _ {t}\right)}{\hat {\pi} _ {\beta} \left(a _ {t} \mid s _ {t}\right)} - 1 \right] \tag {46} \\ = \hat {B} ^ {\pi} \hat {V} _ {k} (s _ {t}) - \kappa \sum_ {a _ {t}} \left[ \pi_ {\theta} \left(a _ {t} | s _ {t}\right) - \hat {\pi} _ {\beta} \left(a _ {t} | s _ {t}\right) + \hat {\pi} _ {\beta} \left(a _ {t} | s _ {t}\right) \right] \left[ \frac {\pi_ {\mathcal {M}} \left(a _ {t} | s _ {t}\right)}{\hat {\pi} _ {\beta} \left(a _ {t} | s _ {t}\right)} - 1 \right]. \\ \end{array}
$$
When  $\pi_{\mathcal{M}}(a_t|s_t) = \pi_\theta (a_t|s_t)$ , we obtain from Eq. (46):
$$
\hat {V} _ {k + 1} ^ {r} \left(s _ {t}\right) = \hat {\mathcal {B}} ^ {\pi} \hat {V} _ {k} \left(s _ {t}\right) - \kappa \underbrace {\sum_ {a _ {t}} \left[ \frac {\left(\pi_ {\theta} \left(a _ {t} \mid s _ {t}\right) - \hat {\pi} _ {\beta} \left(a _ {t} \mid s _ {t}\right)\right) ^ {2}}{\hat {\pi} _ {\beta} \left(a _ {t} \mid s _ {t}\right)} \right]} _ {\geq 0} - \kappa \left(\underbrace {\sum_ {a _ {t}} \pi_ {\theta} \left(a _ {t} \mid s _ {t}\right)} _ {= 1} - \underbrace {\sum_ {a _ {t}} \hat {\pi} _ {\beta} \left(a _ {t} \mid s _ {t}\right)} _ {= 1}\right). \tag {47}
$$
Since  $\sum_{a_t}\pi_\theta (a_t|s_t) = \sum_{a_t}\hat{\pi}_\beta (a_t|s_t) = 1$ , we can conclude that the value estimate  $\hat{V}_k(s_t)$  is underestimated as  $\hat{V}_{k + 1}^r (s_t)\leq \hat{B}^\pi \hat{V}_k(s_t)$  based on Eq. (47). Next, considering the sampling error, we obtain from Eq. (45) and Eq. 36:
$$
\begin{array}{l} \hat {Q} _ {k + 1} ^ {r} \left(s _ {t}, a _ {t}\right) = \hat {\mathcal {B}} ^ {\pi} \hat {Q} _ {k} ^ {r} \left(s _ {t}, a _ {t}\right) - \kappa \left[ \frac {\pi_ {\mathcal {M}} \left(a _ {t} \mid s _ {t}\right)}{\hat {\pi} _ {\beta} \left(a _ {t} \mid s _ {t}\right)} - 1 \right] \tag {48} \\ \leq \mathcal {B} ^ {\pi} \hat {Q} _ {k} (s _ {t}, a _ {t}) - \kappa \left[ \begin{array}{c} \pi_ {\mathcal {M}} (a _ {t} | s _ {t}) \\ \hat {\pi} _ {\beta} (a _ {t} | s _ {t}) \end{array} - 1 \right] + \frac {C _ {r , p , \delta} R _ {m a x}}{(1 - \gamma) \sqrt {| \mathcal {D} (s _ {t} , a _ {t}) |}}. \\ \end{array}
$$
When  $\pi_{\mathcal{M}}(a_t|s_t) = \pi_\theta (a_t|s_t)$ , the fixed point of the update process in Eq. 48 can be expressed as:
$$
\begin{array}{l} \hat {Q} ^ {r} \left(s _ {t}, a _ {t}\right) \leq \mathcal {B} ^ {\pi} \hat {Q} ^ {r} \left(s _ {t}, a _ {t}\right) - \kappa \left[ \frac {\pi_ {\theta} \left(a _ {t} \mid s _ {t}\right)}{\hat {\pi} _ {\beta} \left(a _ {t} \mid s _ {t}\right)} - 1 \right] + \frac {C _ {r , p , \delta} R _ {m a x}}{(1 - \gamma) \sqrt {| \mathcal {D} \left(s _ {t} , a _ {t}\right) |}} \\ \leq r + \gamma P ^ {\pi} \hat {Q} ^ {r} (s _ {t}, a _ {t}) - \kappa \left[ \frac {\pi_ {\theta} (a _ {t} | s _ {t})}{\hat {\pi} _ {\beta} (a _ {t} | s _ {t})} - 1 \right] + \frac {C _ {r , p , \delta} R _ {m a x}}{(1 - \gamma) \sqrt {| \mathcal {D} (s _ {t} , a _ {t}) |}} \\ \leq Q ^ {r} \left(s _ {t}, a _ {t}\right) - \gamma P ^ {\pi} Q ^ {r} \left(s _ {t}, a _ {t}\right) + \gamma P ^ {\pi} \hat {Q} ^ {r} \left(s _ {t}, a _ {t}\right) - \kappa \left[ \frac {\pi_ {\theta} \left(a _ {t} \mid s _ {t}\right)}{\hat {\pi} _ {\beta} \left(a _ {t} \mid s _ {t}\right)} - 1 \right] + \frac {C _ {r , p , \delta} R _ {\max}}{(1 - \gamma) \sqrt {| \mathcal {D} \left(s _ {t} , a _ {t}\right) |}} \tag {49} \\ \end{array}
$$
$$
(I - \gamma P ^ {\pi}) \hat {Q} ^ {r} \left(s _ {t}, a _ {t}\right) \leq \left(I - \gamma P ^ {\pi}\right) Q ^ {r} \left(s _ {t}, a _ {t}\right) - \kappa \left[ \frac {\pi_ {\theta} \left(a _ {t} \mid s _ {t}\right)}{\hat {\pi} _ {\beta} \left(a _ {t} \mid s _ {t}\right)} - 1 \right] + \frac {C _ {r , p , \delta} R _ {\max}}{(1 - \gamma) \sqrt {| \mathcal {D} \left(s _ {t} , a _ {t}\right) |}}, \tag {50}
$$
$$
\hat {Q} ^ {r} \left(s _ {t}, a _ {t}\right) \leq Q ^ {r} \left(s _ {t}, a _ {t}\right) - \kappa \left(I - \gamma P ^ {\pi}\right) ^ {- 1} \left[ \frac {\pi_ {\theta} \left(a _ {t} \mid s _ {t}\right)}{\hat {\pi} _ {\beta} \left(a _ {t} \mid s _ {t}\right)} - 1 \right] + \left(I - \gamma P ^ {\pi}\right) ^ {- 1} \frac {C _ {r , p , \delta} R _ {\max}}{(1 - \gamma) \sqrt {| \mathcal {D} \left(s _ {t} , a _ {t}\right) |}}, \tag {51}
$$
$$
\begin{array}{l} \mathbb {E} _ {a _ {t} \sim \pi_ {\theta}} \hat {Q} ^ {r} \left(s _ {t}, a _ {t}\right) \leq \mathbb {E} _ {a _ {t} \sim \pi_ {\theta}} Q ^ {r} \left(s _ {t}, a _ {t}\right) - \kappa (I - \gamma P ^ {\pi}) ^ {- 1} \mathbb {E} _ {a _ {t} \sim \pi_ {\theta}} \left[ \frac {\pi_ {\theta} \left(a _ {t} \mid s _ {t}\right)}{\hat {\pi} _ {\beta} \left(a _ {t} \mid s _ {t}\right)} - 1 \right] \tag {52} \\ + \left(I - \gamma P ^ {\pi}\right) ^ {- 1} \mathbb {E} _ {a _ {t} \sim \pi_ {\theta}} \frac {C _ {r , p , \delta} R _ {m a x}}{(1 - \gamma) \sqrt {| \mathcal {D} (s _ {t} , a _ {t}) |}}, \\ \end{array}
$$
$$
\hat {V} ^ {r} \left(s _ {t}\right) \leq V ^ {r} \left(s _ {t}\right) - \kappa \left(I - \gamma P ^ {\pi}\right) ^ {- 1} \mathbb {E} _ {a _ {t} \sim \pi_ {\theta}} \left[ \frac {\pi_ {\mathcal {M}} \left(a _ {t} \mid s _ {t}\right)}{\pi_ {\beta} \left(a _ {t} \mid s _ {t}\right)} - 1 \right] + \left(I - \gamma P ^ {\pi}\right) ^ {- 1} \frac {C _ {r , p , \delta} R _ {\max}}{(1 - \gamma) \sqrt {| \mathcal {D} \left(s _ {t}\right) |}}. \tag {53}
$$
To guarantee the lower bound of  $\hat{V}^r$ , we eliminate the potential overestimation caused by  $\frac{C_{r,p,\delta}R_{max}}{(1 - \gamma)\sqrt{|\mathcal{D}(s_t)|}}$  via choosing an appropriate  $\kappa$ . Since  $(I - \gamma P^{\pi})^{-1}$  is a matrix of non-negative entries, the  $\kappa$  that guarantees the lower bound of  $V^r$  needs to satisfy:
$$
\kappa \geq \frac {C _ {r , p , \delta} R _ {\max }}{(1 - \gamma) \sqrt {| \mathcal {D} (s _ {t}) |}} \left[ \mathbb {E} _ {a _ {t} \sim \pi_ {\theta}} \left[ \frac {\pi_ {\theta} (a _ {t} \mid s _ {t})}{\hat {\pi} _ {\beta} (a _ {t} \mid s _ {t})} - 1 \right] \right] ^ {- 1}. \tag {54}
$$
B.5 Proof and Discussion the Proposition 3.8
Proposition B.5. For any  $\pi_{\mathcal{R}}(a_t|s_t)$  with supp  $\pi_{\mathcal{R}}(a_t|s_t) \subset \hat{\pi}_{\beta}$ ,  $\forall s_t, a_t \in D$ , the  $Q$ -value function  $Q^{c_i}$  via iterating Eq. (17) satisfies:
$$
\hat {Q} ^ {c _ {i}} \left(s _ {t}, a _ {t}\right) \geq Q ^ {c _ {i}} \left(s _ {t}, a _ {t}\right) + \chi \left[ \left(I - \gamma P ^ {\pi}\right) ^ {- 1} \frac {\pi_ {\mathcal {R}}}{\hat {\pi} _ {\beta}} \right] \left(s _ {t}, a _ {t}\right) - \left[ \left(I - \gamma P ^ {\pi}\right) ^ {- 1} \frac {C _ {r , p , \delta} C _ {m a x}}{(1 - \gamma) \sqrt {| \mathcal {D} |}} \right] \left(s _ {t}, a _ {t}\right). \tag {55}
$$
Proof. Using the importance sampling, Eq. (17) can be written as:
$$
\begin{array}{l} \hat {Q} _ {k + 1} ^ {c _ {i}} \left(s _ {t}, a _ {t}\right) \leftarrow \arg \min  _ {Q ^ {c _ {i}}} \frac {1}{2} \mathbb {E} _ {s _ {t}, a _ {t} \sim D} \left[ \left[ \left(Q ^ {c _ {t}} \left(s _ {t}, a _ {t}\right)\right) - \hat {\mathcal {B}} ^ {\pi} \hat {Q} _ {k} ^ {c _ {i}} \left(s _ {t}, a _ {t}\right) \right] ^ {2} \right] - \tag {56} \\ \chi \mathbb {E} _ {s _ {t} \sim D, a _ {t} \sim \hat {\pi} _ {\beta} (\cdot | s _ {t})} \left[ \frac {\pi_ {\mathcal {R}} (a | s)}{\hat {\pi} _ {\beta} (a _ {t} | s _ {t})} Q ^ {c _ {i}} (s _ {t}, a _ {t}) \right]. \\ \end{array}
$$
Setting the derivative of Eq. (56) to 0, we obtain the following expression:
$$
\hat {Q} _ {k + 1} ^ {c _ {i}} \left(s _ {t}, a _ {t}\right) = \hat {\mathcal {B}} ^ {\pi} \hat {Q} _ {k} ^ {c _ {i}} (s, a) + \chi \frac {\pi_ {\mathcal {R}} \left(a _ {t} \mid s _ {t}\right)}{\hat {\pi} _ {\beta} \left(a _ {t} \mid s _ {t}\right)} \tag {57}
$$
Note that the Eq. (57) utilizes the empirical Bellman operator  $\hat{B}^{\pi}$  instead of the actual Bellman Operator  $\mathcal{B}^{\pi}$ . Following the related work [23, 31], we employ the concentration properties to determine the error. For any  $\forall s_t, a_t \in D$ , with the probability  $\leq 1 - \delta$ , the error can be written as:
$$
\left| \mathcal {B} ^ {\pi} \hat {Q} \left(s _ {t}, a _ {t}\right) - \hat {\mathcal {B}} ^ {\pi} \hat {Q} \left(s _ {t}, a _ {t}\right) \right| \leq \frac {C _ {r , p , \delta} C _ {\max }}{\left(1 - \gamma\right) \sqrt {\left| \mathcal {D} \left(s _ {t} , a _ {t}\right) \right|}}. \tag {58}
$$
where  $C_{r,p,\delta}$  is a constant depending on the concentration properties of  $c(s_t, a_t)$ ,  $p(s_{t+1}|s_t, a_t)$  and the  $\delta, \delta \in (0,1)$ . The  $(\mathcal{D}(s_t, a_t))^{-\frac{1}{2}}$  represents a vector of size  $|\mathcal{S}||\mathcal{A}|$  containing the square root inverse count of each state action transition.  $C_{max} = \max_{s_t, a_t \sim D} \{c_i(s_t, a_t)\}$  represents the maximum cost obtained in a single step. Based on the inequality of Eq. (58) and Eq. (57) we can obtain:
$$
\begin{array}{l} \hat {Q} _ {k + 1} ^ {c _ {i}} (s _ {t}, a _ {t}) = \hat {\mathcal {B}} ^ {\pi} \hat {Q} _ {k} ^ {c _ {i}} (s _ {t}, a _ {t}) + \chi \frac {\pi_ {\mathcal {R}} (a _ {t} | s _ {t})}{\hat {\pi} _ {\beta} (a _ {t} | s _ {t})} \\ \geq \mathcal {B} ^ {\pi} \hat {Q} _ {k} ^ {c _ {i}} \left(s _ {t}, a _ {t}\right) - \frac {C _ {r , p , \delta} C _ {\max }}{\left(1 - \gamma\right) \sqrt {\left| \mathcal {D} \left(s _ {t} , a _ {t}\right) \right|}} + \kappa \frac {\pi_ {\mathcal {R}} \left(a _ {t} \mid s _ {t}\right)}{\hat {\pi} _ {\beta} \left(a _ {t} \mid s _ {t}\right)}. \tag {59} \\ \end{array}
$$
The fixed point in the updating process of Eq. (59) can be expressed as: