Algorithm 3. As a result, the Bellman backup and the policy optimization only consider in-support actions, which mitigates the issue of learning with non-exploratory  $\mu$ . The loss functions (i.e.,  $\mathcal{L}_{\tilde{\mathcal{D}}}$ ) in Algorithm 2 and Algorithm 3 denote the squared Bellman error (with a target network) as used in fitted Q iteration [78]. We omit the details here.

In summary, PPI and PQI follow the typical policy iteration and value iteration schemes, except that the backup and the argmax are only taken within the observed actions (or actions with sufficiently evidence to be in the support when using function approximators). This idea is similar to the spirit of the later IQL algorithm [8], except IQL doesn't construct the filter function explicitly but relies instead on expectile maximization.

Proposition 11. Suppose  $\mathcal{F} = (\mathcal{S}\times \mathcal{A}\to [-V_{\mathrm{max}},V_{\mathrm{max}}])^{19}$ . For  $V_{\mathrm{max}}\geq \frac{R}{1 - \gamma}$ , PPI/PQI is R-admissible with respect any  $\tilde{\mathcal{R}}\subseteq (\mathcal{S}\times \mathcal{A}\to [-1,1])$ .

Proof. We prove for PPI; the proof for PQI follows similarly. Consider some  $\pi$  such that  $C \coloneqq \sup_{s,a} \frac{d^{\pi(s,a)}}{\mu(s,a)} < \infty$ . Suppose  $\tilde{D}$  has  $N$  transitions. Take some  $V_{\max} \geq \frac{R}{1 - \gamma}$  and some  $\bar{r}$  from  $\mathcal{R}_R(\tilde{r})$ . We use the Corollary 1 in [13]: With  $I$  large enough, it holds with probability  $1 - \delta$ ,

$$
V _ {\tilde {r}} ^ {\pi} (d _ {0}) - V _ {\tilde {r}} ^ {\hat {\pi}} (d _ {0}) \leq \tilde {O} \left(\frac {V _ {\max}}{(1 - \gamma) ^ {3}} \left(\sqrt {\frac {| \mathcal {S} | | \mathcal {A} | \ln (1 / \delta)}{N}} + \frac {\sup _ {s , a , t} d _ {t} ^ {\pi} (s , a)}{b} \epsilon_ {\mu} + (1 - \gamma) \mathbb {E} _ {\pi^ {*}, P} [ \mathbb {1} [ \mu (s, a) \leq 2 b ] ]\right)\right)
$$

where  $\epsilon_{\mu} = D_{TV}(\hat{\mu},\mu) = O(\frac{1}{\sqrt{N}})$ . Further,

$$
\begin{array}{l} (1 - \gamma) \mathbb {E} _ {\pi^ {*}, P} [ \mathbb {1} [ \mu (s, a) \leq 2 b ] ] = \sum_ {s, a} d ^ {\pi^ {*}} (s, a) \mathbb {1} [ \mu (s, a) \leq 2 b ] \\ = \sum_ {s, a} \mu (s, a) \frac {d ^ {\pi^ {*}} (s , a)}{\mu (s , a)} \mathbb {1} [ \mu (s, a) \leq 2 b ] \\ \leq C \mathbb {E} _ {\mu} [ \mathbb {1} [ \mu (s, a) \leq 2 b ] ]. \\ \end{array}
$$

We can then upper bound the performance difference above as

$$
V _ {\bar {r}} ^ {\pi} \left(d _ {0}\right) - V _ {\bar {r}} ^ {\hat {\pi}} \left(d _ {0}\right) \leq \tilde {O} \left(\frac {V _ {\max }}{\left(1 - \gamma\right) ^ {3}} \left(\sqrt {\frac {| \mathcal {S} | | \mathcal {A} | \ln (1 / \delta)}{N}} + \frac {\sup  _ {s , a , t} d _ {t} ^ {\pi} (s , a)}{b} \frac {1}{\sqrt {N}} + C \mathbb {E} _ {\mu} [ \mathbb {1} [ \mu (s, a) \leq 2 b ] ]\right)\right)
$$

We bound the latter two terms by tuning  $b$ . Notice  $\mathbb{E}_{\mu}[\mathbb{1}[\mu (s,a)\leq 2b]]\leq |\mathcal{S}||\mathcal{A}|2b$  at most. This implies

$$
\begin{array}{l} \inf  _ {b} \frac {\sup  _ {s , a , t} d _ {t} ^ {\pi} (s , a)}{b} \frac {1}{\sqrt {N}} + C \mathbb {E} _ {\mu} [ \mathbb {1} [ \mu (s, a) \leq 2 b ] ] \\ \leq \inf  _ {b} \frac {\sup  _ {s , a , t} d _ {t} ^ {\pi} (s , a)}{b} \frac {1}{\sqrt {N}} + C | \mathcal {S} | | \mathcal {A} | 2 b \\ = O \left(\frac {\sqrt {C | \mathcal {S} | | \mathcal {A} |}}{N ^ {1 / 4}}\right) \\ \end{array}
$$

Thus,

$$
V _ {\bar {r}} ^ {\pi} (d _ {0}) - V _ {\bar {r}} ^ {\hat {\pi}} (d _ {0}) \leq \tilde {O} \left(\frac {V _ {\max}}{(1 - \gamma) ^ {3}} \left(\sqrt {\frac {| \mathcal {S} | | \mathcal {A} | \ln (1 / \delta)}{N}} + \frac {\sqrt {C | \mathcal {S} | | \mathcal {A} |}}{N ^ {1 / 4}}\right)\right)
$$

We can apply this similar argument used in the previous proofs to show that this bound simultaneously applies to all  $\tilde{r} \in \mathcal{R}_R(\tilde{r})$  since the high-probability concentration analysis is only taken on the data distribution where all rewards in  $\mathcal{R}_R(\tilde{r})$  are equal. Thus, when  $V_{\mathrm{max}} \geq \frac{R}{1 - \gamma}$ , PPQ (and similarly PQI) is  $R$ -admissible.

19The original algorithms of PPI and PQI assume  $\mathcal{F} = (S\times \mathcal{A}\to [0,V_{\mathrm{max}}])$  . We note that our extension to  $[-V_{\mathrm{max}},V_{\mathrm{max}}]$  do not change the performance bound as it absorbed by the  $\tilde{O}$  notation.

53

F.7 MOReL and MOPO

MOReL [33] and MOPO [32] are two popular model-based offline RL algorithms. The two algorithms operate in a similar manner, except that MOReL [33] uses the truncation approach (similar to PPI and PQI [13]) which classifies state-action into known and unknown sets, whereas MOPO [32] uses negative bonuses (similar to VI-LCB [31]). In this section, we show both algorithms are admissible.

We denote a model as  $M = (\mathcal{S}, \mathcal{A}, P_M, r_M, \gamma)$ , where  $P_M: \mathcal{S} \times \mathcal{A} \to \Delta(\mathcal{S})$  is the model dynamics, and  $r_M: \mathcal{S} \times \mathcal{A} \to [-1, 1]^{20}$  with  $1 \leq R_{\max} < \infty$  being the reward function. The model class is denoted as  $\mathcal{M}_{model}$ .

F7.1 MOReL

We analyze a variant of MOReL which also learns the reward function, whereas MOReL in the originally paper [33] assumes the reward function is given. We suppose model reward  $r_{M}$  and model dynamics  $P_{M}$  are learned by

$$
r_{M}^{\star},P_{M}^{\star}\in \operatorname *{arg  min}_{M\in \mathcal{M}_{model}}\mathcal{E}_{\tilde{D}}(M)
$$

where

$$
\mathcal {E} _ {\tilde {\mathcal {D}}} (M) := \sum_ {(s, a, \tilde {r}, s ^ {\prime}) \in \tilde {\mathcal {D}}} - \log P _ {M} (s ^ {\prime} | s, a) + (r _ {M} (s, a) - \tilde {r}) ^ {2}.
$$

We define the set of known state-actions  $\kappa_{\xi}$  as

$$
\mathcal {K} _ {\xi} = \left\{(s, a): \sup  _ {M _ {1}, M _ {2} \in \mathcal {M} _ {\frac {\alpha}{D}} ^ {\prime}} \left\{\left| r _ {M _ {1}} (s, a) - r _ {M _ {2}} (s, a) \right| + \gamma V _ {\max } D _ {T V} \left(P _ {M _ {1}} (\cdot | s, a), P _ {M _ {2}} (\cdot | s, a)\right) \right\} \leq \xi \right\}, \tag {18}
$$

with

$$
\mathcal {M} _ {\tilde {\mathcal {D}}} ^ {\alpha} := \{M \in \mathcal {M} _ {m o d e l}: \mathcal {E} _ {\tilde {\mathcal {D}}} (M) - \min _ {M ^ {\prime} \in \mathcal {M} _ {m o d e l}} \mathcal {E} _ {\tilde {\mathcal {D}}} (M ^ {\prime}) \leq \alpha \},
$$

We note that  $\alpha$  can be chosen as  $\Theta (|\mathcal{S}|^2 |\mathcal{A}|\log^{1 / \delta})$  so that  $\mathcal{M}_{\alpha}$  contains model  $(S,\mathcal{A},P,\tilde{r},\gamma)$  with probability  $1 - \delta$  for any  $\delta \in (0,1]$  (see [40] for details).

Remark Note that we change the definition for the set of known state-action. In [33], the set is given by

$$
\mathcal {K} _ {\xi} ^ {0} = \{(s, a): | r _ {M} (s, a) - \tilde {r} (s, a) | + \gamma V _ {\max } D _ {T V} \left(P _ {M} (\cdot | s, a), P (\cdot | s, a)\right) \leq \xi \}. \tag {19}
$$

However, it is not impossible to use such a set in practice since the reward function  $\tilde{r}$  and true model  $P(\cdot |s,a)$  are not given to the learner. Our known state-action set  $\kappa_{\xi}$ , in comparison, is easier to construct. It simply measures the maximum disagreement between two data-consistent models. With a considerate choice of  $\alpha$  (see [40]),  $\kappa_{\xi}$  is always a subset of  $\kappa_{\xi}^{0}$ . Our definition also matches better with the practical implementation in [33] which uses the disagreement between any two models in an ensemble.

Then MOReL learns policy  $\hat{\pi}$  by solving the MDP  $(S, \mathcal{A}, \hat{P}, \hat{r}, \gamma)$ , where

$$
\hat {r} (s, a) = \left\{ \begin{array}{l l} r _ {M} ^ {\star} (s, a), & s, a \in \mathcal {K} _ {\xi} \\ - X, & \text {o t h e r w i s e} \end{array} \right.
$$

$$
\hat {P} (s ^ {\prime} | s, a) = \left\{ \begin{array}{l l} P _ {M} ^ {\star} (s ^ {\prime} | s, a), & \quad s, a \in \mathcal {K} _ {\xi} \\ \mathbb {1} (s ^ {\prime} = s ^ {\dagger}), & \quad \text {o t h e r w i s e} \end{array} \right.
$$

Note MOReL introduces an absorbing state  $s^{\dagger}$ .

Below we prove a performance guarantee of MOReL. The proof here is different from that in the original paper, because here we need to consider reward learning and the original proof does not provide an exact rate that converges to zero as the data size increases. (In the original paper there is a non-zero bias term that depends on the comparator policy.)

20Both MOPO and MOReL assume that the reward function is known. Here we study the variants of the two algorithms which also learn the reward function.

54

Proposition 12 (MORel Performance Guarantee). Given data  $\tilde{\mathcal{D}} = \{(s,a,\tilde{r},s')\}$  of size  $N$  of an unknown MDP  $(S,A,\tilde{r},P,\gamma)$ . Suppose  $\tilde{r},r_{M}^{\star}\in [-1,1]$  and  $(\tilde{r},P)\in \mathcal{M}_{\tilde{D}}^{\alpha}$ . Choose  $\xi = \min(1,O(V_{max}(|S|^2|\mathcal{A}|\log^{(1/\delta)}/N)^{1/4}))$ . Let  $X\geq R$ . For  $\pi$  such that  $\sup_{s,a}\frac{d^{\pi}(s,a)}{\mu(s,a)} = C < \infty$ , with probability  $1 - \delta$ , it holds that

$$
(1 - \gamma) \left(V _ {\bar {r}} ^ {\pi} (d _ {0}) - V _ {\bar {r}} ^ {\hat {\pi}} (d _ {0})\right) \leq O \left(V _ {\max } \left(\frac {| \mathcal {S} | ^ {2} | \mathcal {A} | \log (\frac {1}{\delta})}{N}\right) ^ {1 / 4}\right), \quad w i t h V _ {\max } := \frac {R}{1 - \gamma}
$$

for all  $\bar{r} \in \mathcal{R}_R(\tilde{r}) \coloneqq \{\bar{r} : \bar{r}(s, a) = \tilde{r}(s, a), \forall (s, a) \in \text{supp}(\mu)$  and  $\bar{r}(s, a) \in [-R, 1], \forall (s, a) \in \mathcal{S} \times \mathcal{A}\}$ .

Proof. We extend  $\tilde{r}$  as  $\tilde{r}(s^{\dagger}, a) = -X$  for all  $a \in \mathcal{A}$ . Let  $\hat{M} = (\hat{r}, \hat{P})$ .

First, by the optimality of  $\hat{\pi}$ , we write

$$
\begin{array}{l} V _ {\bar {r}} ^ {\pi} \left(d _ {0}\right) - V _ {\bar {r}} ^ {\hat {\pi}} \left(d _ {0}\right) \\ = V _ {\bar {r}} ^ {\pi} \left(d _ {0}\right) - V _ {\hat {M}} ^ {\pi} \left(d _ {0}\right) + V _ {\hat {M}} ^ {\pi} \left(d _ {0}\right) - V _ {\hat {M}} ^ {\hat {\pi}} \left(d _ {0}\right) + V _ {\hat {M}} ^ {\hat {\pi}} \left(d _ {0}\right) - V _ {\bar {r}} ^ {\hat {\pi}} \left(d _ {0}\right) \\ \leq V _ {\vec {r}} ^ {\pi} (d _ {0}) - V _ {\hat {M}} ^ {\pi} (d _ {0}) + V _ {\hat {M}} ^ {\hat {\pi}} (d _ {0}) - V _ {\vec {r}} ^ {\hat {\pi}} (d _ {0}) \\ \end{array}
$$

where  $V_{\hat{M}}^{\pi}$  denotes the value of policy  $\pi$  with respect to the reward  $\hat{r}$  and dynamics  $\hat{P}$ . The last inequality follows from the optimality of  $\hat{\pi}$  on  $\hat{M}$ .

Then we notice the fact that  $\mathcal{K}_{\xi} \subseteq \operatorname{supp}(\mu)$  with a small  $\xi$ . The proof of the lemma below is given in Appendix F.7.3.

Lemma 7. If  $\xi \leq 1$ ,  $\mathcal{K}_{\xi} \subseteq \text{supp}(\mu)$ .

Let  $\hat{d}^{\pi}$  denote the average state-action visitation of  $\pi$  with respect to  $\hat{P}$ .

$$
\begin{array}{l} (1 - \gamma) \left(V _ {\hat {M}} ^ {\hat {\pi}} (d _ {0}) - V _ {\bar {r}} ^ {\hat {\pi}} (d _ {0})\right) \\ = \mathbb {E} _ {s, a \sim \hat {d} ^ {\hat {\pi}}} [ \hat {r} (s, a) + \gamma \mathbb {E} _ {s ^ {\prime} \sim \hat {P} | s, a} [ V _ {\bar {r}} ^ {\hat {\pi}} (s ^ {\prime}) ] - \bar {r} (s, a) - \gamma \mathbb {E} _ {s ^ {\prime} \sim P | s, a} [ V _ {\bar {r}} ^ {\hat {\pi}} (s ^ {\prime}) ] ] \\ \leq \mathbb {E} _ {s, a \sim \hat {d} ^ {\pi}} [ \hat {r} (s, a) + \gamma \mathbb {E} _ {s ^ {\prime} \sim \hat {P} | s, a} [ V _ {\bar {r}} ^ {\hat {\pi}} (s ^ {\prime}) ] - \bar {r} (s, a) - \gamma \mathbb {E} _ {s ^ {\prime} \sim P | s, a} [ V _ {\bar {r}} ^ {\hat {\pi}} (s ^ {\prime}) ] | (s, a) \notin \mathcal {K} _ {\xi} ] \\ + \mathbb {E} _ {s, a \sim \hat {d} ^ {\pi}} [ | \hat {r} (s, a) - \bar {r} (s, a) | + \gamma V _ {\max } D _ {T V} (\hat {P} (\cdot | s, a), P (\cdot | s, a)) | (s, a) \in \mathcal {K} _ {\xi} ] \\ \leq \xi + \mathbb {E} _ {s, a \sim \hat {d} ^ {\hat {\pi}}} [ \hat {r} (s, a) + \gamma \mathbb {E} _ {s ^ {\prime} \sim \hat {P} | s, a} [ V _ {\bar {r}} ^ {\hat {\pi}} (s ^ {\prime}) ] - \bar {r} (s, a) - \gamma \mathbb {E} _ {s ^ {\prime} \sim P | s, a} [ V _ {\bar {r}} ^ {\hat {\pi}} (s ^ {\prime}) ] | (s, a) \notin \mathcal {K} _ {\xi} ] \\ \leq \xi + \mathbb {E} _ {s, a \sim \hat {d ^ {\pi}}} [ - V _ {\max } - \bar {r} (s, a) + \gamma V _ {\max } | (s, a) \notin \mathcal {K} _ {\xi} ] \\ \leq \xi \\ \end{array}
$$

where in the second inequality we use  $\mathcal{K}_{\xi} \subseteq \mathrm{supp}(\mu)$  (which implies  $\tilde{r} = \bar{r}$  in  $\mathcal{K}_{\xi}$ ).

Similarly we can show

$$
\begin{array}{l} (1 - \gamma) \left(V _ {\bar {M}} ^ {\pi} (d _ {0}) - V _ {\bar {r}} ^ {\pi} (d _ {0})\right) \\ \geq - \xi + \mathbb {E} _ {s, a \sim d ^ {\pi}} [ \hat {r} (s, a) + \gamma \mathbb {E} _ {s ^ {\prime} \sim \hat {P} | s, a} [ V _ {\hat {M}} ^ {\pi} (s ^ {\prime}) ] - \bar {r} (s, a) - \gamma \mathbb {E} _ {s ^ {\prime} \sim P | s, a} [ V _ {\hat {M}} ^ {\pi} (s ^ {\prime}) ] | (s, a) \notin \mathcal {K} _ {\xi} ] \\ \geq - \xi - 2 V _ {\max } \mathbb {E} _ {s, a \sim d ^ {\pi}} [ \mathbb {1} [ (s, a) \notin \mathcal {K} _ {\xi} ] ] \\ \end{array}
$$

Combining the three inequalities above gives

$$
(1 - \gamma) \left(V _ {\bar {r}} ^ {\pi} (d _ {0}) - V _ {\bar {r}} ^ {\hat {\pi}} (d _ {0})\right) \leq 2 V _ {\max } \mathbb {E} _ {s, a \sim d ^ {\pi}} [ \mathbb {1} [ (s, a) \notin \mathcal {K} _ {\xi} ] ] + 2 \xi
$$

By the concentratability assumption, we can further upper bound

$$
\mathbb {E} _ {s, a \sim d ^ {\pi}} [ \mathbb {1} [ (s, a) \notin \mathcal {K} _ {\xi} ] ] \leq C \mathbb {E} _ {s, a \sim \mu} [ \mathbb {1} [ (s, a) \notin \mathcal {K} _ {\xi} ] ]
$$

Therefore we have

$$
\begin{array}{l} (1 - \gamma) \left(V _ {\bar {r}} ^ {\pi} (d _ {0}) - V _ {\bar {r}} ^ {\hat {\pi}} (d _ {0})\right) \leq 2 V _ {\max } \mathbb {E} _ {s, a \sim \mu} [ \mathbb {1} [ (s, a) \notin \mathcal {K} _ {\xi} ] ] + 2 \xi \\ \leq 2 V _ {\max } \frac {\mathbb {E} _ {s , a \sim \mu} [ E _ {\sup } (s , a) ]}{\xi} + 2 \xi \\ \leq 4 V _ {\max } \frac {\sup  _ {r _ {M} , P _ {M}} \mathbb {E} _ {s , a \sim \mu} [ E (s , a ; r _ {M} , P _ {M}) ]}{\xi} + 2 \xi \\ \end{array}
$$

55

where the last step is Markov inequality,

$$
E_{\sup}(s,a) = \sup_{M_{1},M_{2}\in \mathcal{M}_{\bar{\mathcal{D}}}^{\alpha}}\bigl\{|r_{M_{1}}(s,a) - r_{M_{2}}(s,a)| + \gamma V_{\max}D_{TV}(P_{M_{1}}(\cdot |s,a),P_{M_{2}}(\cdot |s,a))\bigr \} ,
$$

and

$$
E (s, a; r _ {M}, P _ {M}) = | r _ {M} (s, a) - \tilde {r} (s, a) | + \gamma V _ {\max } D _ {T V} (P _ {M} (\cdot | s, a), P (\cdot | s, a)).
$$

Now we upper bound the expectation of  $E$  over  $\mu$ . With probability greater than  $1 - \delta$ ,

$$
\begin{array}{l} \mathbb {E} _ {\mu} [ E (s, a; r _ {M}, P _ {M}) ] \leq \sqrt {\mathbb {E} _ {\mu} [ (| r _ {M} (s , a) - \tilde {r} (s , a) | + \gamma V _ {\max} D _ {T V} (P _ {M} (\cdot | s , a) , P (\cdot | s , a))) ^ {2} ]} \\ \leq \sqrt {2} \sqrt {\mathbb {E} _ {\mu} [ (r _ {M} (s , a) - \tilde {r} (s , a)) ^ {2} + (\gamma V _ {\max}) ^ {2} D _ {T V} (P _ {M} (\cdot | s , a) , P (\cdot | s , a)) ^ {2} ]} \\ \leq (1 + \gamma V _ {\max }) \sqrt {2} \sqrt {\mathbb {E} _ {\mu} [ (r _ {M} (s , a) - \tilde {r} (s , a)) ^ {2} + D _ {T V} (P _ {M} (\cdot | s , a) , P (\cdot | s , a)) ^ {2} ]} \\ \leq V _ {\max } \sqrt {2} \sqrt {\mathbb {E} _ {\mu} [ (r _ {M} (s , a) - \tilde {r} (s , a)) ^ {2} + D _ {T V} (P _ {M} (\cdot | s , a) , P (\cdot | s , a)) ^ {2} ]} \\ \leq V _ {\max } \sqrt {2} \sqrt {\frac {| \mathcal {S} | ^ {2} | \mathcal {A} | \log (\frac {1}{\delta})}{N}} \\ \end{array}
$$

where the last step is based on Lemma 8.

Lemma 8. [34] With probability  $1 - \delta$ , for any MDP model  $M$ ,

$$
\mathbb {E} _ {\mu} [ | r _ {M} (s, a) - r (s, a) | ^ {2} + D _ {T V} (P _ {M} (\cdot | s, a), P (\cdot | s, a)) ^ {2} ] \leq O \left(\frac {\mathcal {E} _ {\mathcal {D}} (M) - \min _ {M ^ {\prime}} \mathcal {E} _ {\mathcal {D}} (M ^ {\prime}) + | \mathcal {S} | ^ {2} | \mathcal {A} | \log (\frac {1}{\delta})}{N}\right).
$$

Thus, we have

$$
\begin{array}{l} (1 - \gamma) \left(V _ {\bar {r}} ^ {\pi} (d _ {0}) - V _ {\bar {r}} ^ {\hat {\pi}} (d _ {0})\right) \leq O \left(\frac {V _ {\max } ^ {2} \sqrt {| \mathcal {S} | ^ {2} | \mathcal {A} | \log (\frac {1}{\delta})}}{\xi \sqrt {N}}\right) + 2 \xi \\ \leq O \left(V _ {\max } \left(\frac {| \mathcal {S} | ^ {2} | \mathcal {A} | \log \left(\frac {1}{\delta}\right)}{N}\right) ^ {1 / 4}\right). \\ \end{array}
$$

By Proposition 12, we can apply the previous analysis technique to MOReL to show it is admissible without addition union bounds, which leads to the corollary below.

Corollary 7. For  $X \geq R$ , MOReL is  $R$ -admissible with respect any  $\tilde{\mathcal{R}} \subseteq (\mathcal{S} \times \mathcal{A} \to [-1,1])$ .

F.7.2 MOPO

MOPO [32] is very similar to MOReL except that it uses negative bonuses (like VI-LCB) instead of truncation. In the original paper of MOPO, the authors assume the reward is given. Here we consider a variant that also learns the reward. Specifically, given  $P_M^\star$  and  $r_M^\star$  above it solves the MDP  $(\mathcal{S},\mathcal{A},P_M^\star ,\hat{r},\gamma)$ , where

$$
\hat {r} (s, a) = r _ {M} ^ {\star} (s, a) - b (s, a)
$$

While the original MOPO paper does not give a specific design of  $b(s, a)$  that is provably correct, in principle making MOPO provably correct is possible. Essentially, we need to choose a large enough bonus to cover the error of the model-based Bellman operator (defined by the estimated reward and dynamics). This would lead to a bonus of order  $V_{\mathrm{max}}$  (see the analysis of VI-LCB in Appendix F.5). Then we can proceed with an analysis that combines that of MOReL and VI-LCB to prove MOPO's performance guarantee. This can then be used to show (like the previous proofs) that MOPO is admissible. We omit the proof here.

Corollary 8. Suppose  $b(s, a) = \Theta(V_{\max})$  for  $(s, a) \notin \operatorname{supp}(\mu)$ . For  $V_{\max} \geq \frac{R}{1 - \gamma}$ , MOPO is R-admissible with respect any  $\tilde{\mathcal{R}} \subseteq (\mathcal{S} \times \mathcal{A} \to [-1,1])$ .

56

F.7.3 Proof of Technical Lemma

Proof of Lemma 7. Consider any  $(\bar{s},\bar{a}) \notin \mathrm{supp}(\mu)$ . Given any  $M = (\mathcal{S},\mathcal{A},P_M,r_M,\gamma) \in \mathcal{M}_{\hat{\mathcal{D}}}^{\alpha}$ , we construct  $M' = (\mathcal{S},\mathcal{A},P_M,r_{M'},\gamma)$  with  $r_{M'} = r_M$  for all  $(s,a) \in (\mathcal{S} \times \mathcal{A}) \setminus \{(\bar{s},\bar{a})\}$ , and  $r_{M'}(\bar{s},\bar{a}) = -\mathrm{sign}(r_M(\bar{s},\bar{a}))$ . By construction, since  $M$  and  $M'$  agrees on all state-actions except  $(\bar{s},\bar{a})$ , which is not in data support, we have  $M' \in \mathcal{M}_{\hat{\mathcal{D}}}^{\alpha}$ . However, we have that

$$
\begin{array}{l} \sup_{M_{1},M_{2}\in \mathcal{M}_{\bar{\mathcal{D}}}^{\alpha}}\bigl\{\left|r_{M_{1}}(\bar{s},\bar{a}) - r_{M_{2}}(\bar{s},\bar{a})\right| + \gamma V_{\max}D_{TV}(P_{M_{1}}(\cdot |\bar{s},\bar{a}),P_{M_{2}}(\cdot |\bar{s},\bar{a}))\bigr \} \\ \geq \left| r _ {M} (\bar {s}, \bar {a}) - r _ {M ^ {\prime}} (\bar {s}, \bar {a}) \right| + \gamma V _ {\max } D _ {T V} \left(P _ {M} (\cdot | \bar {s}, \bar {a}), P _ {M} (\cdot | \bar {s}, \bar {a})\right) \\ = \left| r _ {M} (\bar {s}, \bar {a}) - r _ {M ^ {\prime}} (\bar {s}, \bar {a}) \right| \geq 1 \geq \xi . \\ \end{array}
$$

This means that  $(\bar{s},\bar{a})\notin \mathcal{K}_{\xi}$ . Therefore  $\mathcal{K}_{\xi} \subseteq \operatorname{supp}(\mu)$ .

G Grid World

In this section, we describe the full details of the grid world study that was discussed in Section 3.1.

Environment. We consider goal-directed navigation in the environment shown in Fig. 10. We use a fixed grid world layout shown in the figure. The agent's state is given by a tuple  $(x, y, d)$  where  $(x, y) \in [5]^2$  represents the 2-d coordinate and  $d \in \{N, W, E, S\}$  encodes the North, West, East, and South direction respectively that the agent is facing. The agent's action space is  $\mathcal{A} = \{f, l, r\}$  where  $f$  denotes the action of moving to grid square in front of the agent,  $l$  denotes a left turn of 90 degrees, and  $r$  denotes a right turn of 90 degrees. The agent can go through the lava (orange wavy square) but cannot go through the wall (grey square). The goal (key) is an absorbing state. The agent gets a reward of +1 when first visiting the goal followed by a reward of 0 for then staying in the goal. The agent gets a reward of -1 for reaching lava and a reward of -0.01 for all other actions to incentivize the agent to reach the goal along the shortest safe path. We use a horizon of  $H = 20$ . The agent deterministically starts in the top-right corner shown in Fig. 10.

Dataset. We collect a dataset  $\mathcal{D}$  of 500 episodes by taking 100 identical optimal episodes along with 400 identical episodes that all touch the topmost lava field. We introduce a length bias by terminating an episode if the agent touches the lava or fails to reach the goal in  $H - 1$  steps (i.e., the agent doesn't survive). In addition to the original setting with the observed reward, we consider three additional settings where the rewards in the dataset are replaced by: (i) a constant zero reward, (ii) their negative, and (iii) a random number sampled uniformly in  $[0, 1]$ .

Figure 10: Left: A goal-directed gridworld navigation task where the agent (red triangle) has to reach the goal (key) and avoid lava (orange waves). Center: Shows the length bias that helps offline RL succeed even when rewards are entirely replaced with random or adversarial values. Right: Results on the grid world task. Behavior cloning fails to solve the task (red path) while using an offline RL method leads to success (blue path) even with wrong rewards. The opacity of the grid square overlay indicates how frequent that state is in the dataset (more opaque means more frequent).

57

Algorithm 4 PEVI  $(S, A, H, D, \beta, V_{\min}, V_{\max})$  [12]. We are given access to a set  $\mathcal{D}$  of episodes  $(\tau)$ . Additionally, we are given a hyperparameter  $\beta \in \mathcal{R}_{\geq 0}$  that controls pessimism. The hyperparameters  $V_{\min}, V_{\max} \in \mathcal{R}^H$  indicate bounds on policy value with  $V_{\min,h}$  and  $V_{\max,h}$  denoting the minimum and maximum policy values respectively for time step  $h \in [H]$ . Unlike the original PEVI algorithm of [12], we treat  $V_{\min}, V_{\max}$  as hyperparameters.

Set  $V_{H + 1} = 0$    
for  $h = H,\dots ,1$  do   
for  $s\in S$  do   
for  $a\in \mathcal{A}$  do   
 $n_h(s,a) = \sum_{\tau \in \mathcal{D}}\mathbb{1}\{s_h^\tau = s\wedge a_h^\tau = a\}$    
If  $n_h(s,a) > 0$ $\tilde{Q}_h(s,a) = \frac{1}{n_h(s,a)}\sum_{\tau \in \mathcal{D}}\mathbb{1}\{s_h^\tau = s\wedge a_h^\tau = a\} \{r_h^\tau +V_{h + 1}(s_{h + 1}^\tau)\}$ $\bar{Q}_h(s,a) = \tilde{Q}_h(s,a) - \frac{\beta}{\sqrt{n_h(s,a)}}$ $\widehat{Q}_h(s,a) = \max \{V_{\min ,h},\min \{\bar{Q}_h(s,a),V_{\max ,h}\} \}$    
Else   
 $\widehat{Q}_h(s,a) = V_{\min ,h}$    
end for   
 $\pi_h(s) = \arg \max_{a\in \mathcal{A}}\widehat{Q}_h(s,a)$ $V_{h}(s) = \widehat{Q}_{h}(s,\pi_{h}(s))$    
end for   
end for   
Return  $\pi = (\pi_{1:H})$

Methods. We evaluate two methods: behavior cloning which simply takes the action with the highest empirical count in a given state and a random action if the state was unseen, and PEVI [12] an offline RL method with guarantees for tabular MDP. We provide a pseudocode of PEVI in Algorithm 4. Unlike the original PEVI in [12], we provide minimum and maximum bounds for the value function for each time step as an additional hyperparameter. These bounds should be dictated by our CMDP framework and should penalize out-of-distribution actions. In principle, we need to set these bounds such that the algorithm is admissible. Please see Appendix D and Appendix E for details.

Intuitively, PEVI performs dynamic programming similar to a standard value iteration with two key changes. Firstly, it uses pessimistic value initialization where the learned value  $V_{h}(s)$  of a state  $s$  at time step  $h$  is set to the lowest possible return  $V_{\min,h}$  if the state is unseen at time step  $h$ . Secondly, when performing value iteration it adds a pessimism penalty of  $-\beta / \sqrt{n(s,a)}$  to the dataset reward where  $n(s,a)$  is the empirical state-action count in  $\mathcal{D}$ . This pessimism penalty ensures that the learned value function lower bounds the true value function (hence a pessimistic estimate). In our experiments, we vary  $\beta$  and set  $V_{\min,h} = \tilde{V}_{\min,h} - \beta (H - h + 1) - 1$  where  $\tilde{V}_{\min,h}$  is the minimum possible return for the given reward type that we are working with. The value of  $\tilde{V}_{\min,h}$  is  $-(H - h + 1)$  for the original reward,  $-1$  for the negative reward, and 0 for the zero reward and random reward cases). We define  $V_{\max,h} = \tilde{V}_{\max,h}$  where  $\tilde{V}_{\max,h}$  is the maximum possible return for the given reward type. The value of  $\tilde{V}_{\max,h}$  is 1 of the original reward,  $(H - h + 1)$  for the random reward and the negative reward, and 0 for the zero reward case.

Results. We show numerical results in Table 16. We visualize results in Fig. 10. The behavior cloning simply imitates the most common trajectory in the dataset which results in going to the lava (shown in red in the figure). In contrast, the PEVI is able to reach the goal in every reward situation.

Explanation. The reason why PEVI works can be understood simply by pessimistic initialization and length bias. The length bias in this setting is visualized in the center of Fig. 10. PEVI assigns the final state  $s_t'$  of a failed trajectory will get a pessimistic value of  $V_t(s_t') = V_{\min, t}$ . This value isn't updated as we never take any action in  $s_t'$  due to timeout. In our case,  $s_t'$  will be the state where the agent first visits the lava. In contrast, the final state of a successful trajectory  $s_{H+1}$  gets assigned a value of 0. If  $V_{\min}$  is sufficiently negative, then PEVI will only consider policies that stay all  $H$  steps within the data support. Due to length bias, this is not true for the non-surviving trajectories that

58

Reward Type	Behavior Cloning	PEVI
Original Reward	-1.19 ± 0.00	0.92 ± 0.00
Zero Reward	-1.19 ± 0.00	0.92 ± 0.00
Random Reward	-1.19 ± 0.00	0.92 ± 0.00
Negative Reward	-1.19 ± 0.00	0.92 ± 0.00

Table 16: Gridworld results: Mean return on 1000 test episodes. PEVI achieves the optimal return of  $V^{\star} = 0.92$ .

reach the lava. Further, the only trajectories in our case that complete end up reaching and staying in the goal. Therefore, under the assumption that  $V_{\mathrm{min},h}$  are all sufficiently small, PEVI will learn a policy reaches the goal irrespective of the correctness of the reward in the data. The sufficiency of the negativity of  $V_{\mathrm{min},h}$  is implied by our CMDP framework. Lastly, note that as the world is deterministic, we didn't need to use the pessimism arising from the  $\frac{-\beta}{\sqrt{n(s,a)}}$  penalty term.

59