SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Offline RL. In contrast to online RL, offline RL assumes access to a pre-collected offline dataset and precludes active interactions with the environment. Given the absence of further data collection, the sample complexity of pure offline RL depends heavily upon the quality of the offline dataset at hand, which has often been characterized via some sorts of concentrability coefficients in prior works (Rashidinejad et al., 2021; Zhan et al., 2022). Earlier works (Munos and Szepesvári, 2008; Chen and Jiang, 2019) typically operated under the assumption of all-policy concentrability — namely, the assumption that the dataset covers the visited state-action pairs of all possible policies — thus imposing a stringent requirement for the offline dataset to be highly explorative. To circumvent this stringent assumption, Liu et al. (2020); Kumar et al. (2020); Jin et al. (2021b); Rashidinejad et al. (2021); Uehara and Sun (2021); Li et al. (2022); Yin et al. (2021); Shi et al. (2022); Yan et al. (2023) incorporated the pessimism principle amid uncertainty into the algorithm designs and, as a result, required only single-policy concentrability (so that the dataset only needs to cover the part of the state-action space reachable by the optimal policy). With regards to the basic tabular case, Li et al. (2022) proved that the pessimistic model-based offline algorithm is capable of achieving minimax-optimal sample complexity for the full  $\varepsilon$ -range, accommodating both the episodic finite-horizon case and the discounted infinite-horizon analog. Moving beyond single-agent tabular settings, a recent line of works investigated offline RL in the presence of general function approximation (Jin et al., 2020c; Xie et al., 2021a; Zhan et al., 2022), environment shift (Zhou et al., 2021b; Shi and Chi, 2022), and in the context of zero-sum Markov games (Cui and Du, 2022; Yan et al., 2022).
Hybrid RL. While there were a number of empirical works (Rajeswaran et al., 2017; Vecerik et al., 2017; Kalashnikov et al., 2018; Hester et al., 2018; Nair et al., 2018, 2020) suggesting the perfromance gain of combining online RL with offline datasets (compared to pure online or offline learning), rigorous theoretical evidence remained highly limited. Ross and Bagnell (2012); Xie et al. (2021b) attempted to develop theoretical understanding by looking at one special hybrid scenario, where the agent can perform either of the following in each episode: (i) collecting a new online episode; and (ii) executing a prescribed and fixed reference policy to generate a sample episode. In this setting, Xie et al. (2021b) showed that in the minimax sense, combining online learning with samples generated by such a reference policy is not advantageous in comparison with pure online or offline RL. Note that our results do not contradict with the lower bound in Xie et al. (2021b), given that we exploit "partial" single-policy concentrability that implies additional structure except for the worst case. Akin to the current paper, Song et al. (2022); Wagenmaker and Pacchiano (2022) studied online RL with additional access to an offline dataset. Nevertheless, Song et al. (2022) mainly focused on the issue of computational efficiency, and the algorithm proposed therein does not come with improved sample complexity. In contrast, Wagenmaker and Pacchiano (2022) focused attention on statistical efficiency, although the sample complexity derived therein is highly suboptimal when specialized to the tabular setting.
Reward-free and task-agnostic exploration. Reward-free and task-agnostic exploration, which refer to the scenario where the agent first collects online sample trajectories without guidance of any information about the reward function(s), has garnered much recent attention (Brafman and
Tennenholtz, 2002; Jin et al., 2020a; Zhang et al., 2020a, 2021b; Huang et al., 2022). Focusing on the tabular case, the earlier work Jin et al. (2020a) put forward a reward-free exploration scheme that achieves minimax optimality in terms of the dependency on  $S$ ,  $A$  and  $1 / \varepsilon$ , with the horizon dependency further improved by subsequent works (Kaufmann et al., 2021; Ménard et al., 2021a; Li et al., 2023). In particular, the exploration scheme proposed in Li et al. (2023) was shown to achieve minimax-optimal sample complexity when there exist a polynomial number of pre-determined but unseen reward functions of interest, which inspires the algorithm design of the present paper. Moreover, reward-free RL has been extended to account for function approximation, including both linear (Wang et al., 2020; Agarwal et al., 2020; Qiao and Wang, 2022; Zhang et al., 2021a; Wagenmaker et al., 2022) and nonlinear function classes (Chen et al., 2022a).
B Pseudocode of Algorithm 1
In this section, we provide the whole procedure for the proposed hybrid RL algorithm, with several subroutines deferred to Appendix C and Appendix D.
Algorithm 1: The proposed hybrid RL algorithm.
1 Input: offline dataset  $\mathcal{D}^{\mathrm{off}}$  (containing  $K^{\mathrm{off}}$  trajectories), parameters  $N$ ,  $K^{\mathrm{on}}$ ,  $T_{\mathrm{max}}$ , learning rate  $\eta$ .
2 Initialize:  $\pi_h^1 (a\mid s) = 1 / A$  for any  $(s,a,h)$ ;  $K = K^{\mathrm{off}} + K^{\mathrm{on}}$ ; split  $\mathcal{D}^{\mathrm{off}}$  into two halves  $\mathcal{D}^{\mathrm{off},1}$  and  $\mathcal{D}^{\mathrm{off},2}$ .
/* Estimation of occupancy distributions for any policy  $\pi$  1
3 Call Algorithm 3, which allows one to specify  $\widehat{d_h^\pi}(s, a)$  for any deterministic policy  $\pi$  and any  $(s, a, h)$ .
/* Estimation of occupancy distributions of the historical data. */
4 Use the dataset  $\mathcal{D}^{\mathrm{off},1}$  to compute
$$
\widehat {d} _ {h} ^ {\text {o f f}} (s, a) = \frac {2 N _ {h} ^ {\text {o f f}} (s , a)}{K ^ {\text {o f f}}} \mathbb {1} \left(\frac {N _ {h} ^ {\text {o f f}} (s , a)}{K ^ {\text {o f f}}} \geq c _ {\text {o f f}} \left\{\frac {\log \frac {H S A}{\delta}}{K ^ {\text {o f f}}} + \frac {H ^ {4} S ^ {4} A ^ {4} \log \frac {H S A}{\delta}}{N} + \frac {S A}{K ^ {\text {o n}}} \right\}\right)
$$
for any  $(s, a, h)$ , where  $N_h^{\mathrm{off}}(s, a) = \sum_{k=1}^{K^{\mathrm{off}}/2} \mathbb{1}(s_h^k = s, a_h^k = a)$  and  $c_{\mathrm{off}} > 0$  is some absolute constant.
/* Compute a general sample-efficient online exploration scheme. */
5 Call Algorithm 5 with estimators  $\widetilde{d}^{\pi}$  to compute policy  $\pi^{\mathrm{explore}}$  and the associated weight  $\mu^{\mathrm{explore}}$ .
/* Compute an online exploration scheme tailored to the offline dataset. */
$\mathbf{6}$  for  $t = 1,\dots ,T_{\mathrm{max}}$  do
7 Compute  $\mu^t$  using Algorithm 2.
8 Update  $\pi_h^{t + 1}(a\mid s)$  for all  $(s,a,h)\in S\times \mathcal{A}\times [H]$  such that:
$$
\pi_ {h} ^ {t + 1} (a \mid s) = \frac {\exp \left(\eta \sum_ {k = 1} ^ {t} \frac {\widehat {d} _ {h} ^ {\mathrm {o f f}} (s , a)}{\frac {1}{K ^ {\mathrm {o n}} H} + \mathbb {E} _ {\pi^ {\prime} \sim \mu^ {k}} \left[ \widehat {d} _ {h} ^ {\pi^ {\prime}} (s , a) \right]}\right)}{\sum_ {a ^ {\prime} \in \mathcal {A}} \exp \left(\eta \sum_ {k = 1} ^ {t} \frac {\widehat {d} _ {h} ^ {\mathrm {o f f}} (s , a ^ {\prime})}{\frac {1}{K ^ {\mathrm {o n}} H} + \mathbb {E} _ {\pi^ {\prime} \sim \mu^ {k}} \left[ \widehat {d} _ {h} ^ {\pi^ {\prime}} (s , a ^ {\prime}) \right]}\right)},
$$
9 Set  $\mu^{\mathrm{imitate}} = \frac{1}{T_{\mathrm{max}}}\sum_{t = 1}^{T_{\mathrm{max}}}\mu^t$  and  $\pi^{\mathrm{imitate}} = \mathbb{E}_{\pi \sim \mu^{\mathrm{imitate}}}[\pi ]$
/* Sampling using the above two exploration policies. */
10 Collect  $K_{\text{imitate}}^{\text{on}}$  (resp.  $K_{\text{explore}}^{\text{on}}$ ) sample trajectories using  $\pi^{\text{imitate}}$  (resp.  $\pi^{\text{explore}}$ ) to form a dataset  $\mathcal{D}_{\text{imitate}}^{\text{on}}$  (resp.  $\mathcal{D}_{\text{explore}}^{\text{on}}$ ).
/\*Run the model-based offline RL algorithm. \*/
11 Apply Algorithm 6 to the dataset  $\mathcal{D} = \mathcal{D}^{\mathrm{off},2} \cup \mathcal{D}_{\mathrm{imitate}}^{\mathrm{on}} \cup \mathcal{D}_{\mathrm{explore}}^{\mathrm{on}}$  to compute a policy  $\widehat{\pi}$ .
12 Output: policy  $\widehat{\pi}$
C Subroutine for solving the subproblem (20b)
While (20b) is a convex optimization subproblem, it involves optimization over a parameter space with exponentially large dimensions. In order to solve it in a computationally feasible manner, we propose a tailored subroutine based on the Frank-Wolfe algorithm (Bubeck, 2015).
Before proceeding, recall that when specifying  $\widehat{d}^{\pi}$  in Algorithm 3, we draw  $N$  independent trajectories  $\{s_1^{n,h},a_1^{n,h},\ldots ,s_{h + 1}^{n,h}\}_{1\leq n\leq N}$  and compute an empirical estimate  $\widehat{P}_h$  of the probability transition kernel at step  $h$  such that
$$
\widehat {P} _ {h} \left(s ^ {\prime} \mid s, a\right) = \frac {\mathbb {1} \left(N _ {h} (s , a) > \xi\right)}{\max  \left\{N _ {h} (s , a) , 1 \right\}} \sum_ {n = 1} ^ {N} \mathbb {1} \left(s _ {h} ^ {n, h} = s, a _ {h} ^ {n, h} = a, s _ {h + 1} ^ {n, h} = s ^ {\prime}\right), \quad \forall (s, a, s ^ {\prime}) \in \mathcal {S} \times \mathcal {A} \times \mathcal {S} \tag {31}
$$
where  $N_{h}(s,a) = \sum_{n = 1}^{N}\mathbb{1}\{s_{h}^{n,h} = s,a_{h}^{n,h} = a\}$
The proposed Frank-Wolfe-type algorithm. With this set of notation in place and with an initial guess taken to be the indicator function  $\mu^{(1)} = \mathbb{1}_{\pi_{\mathrm{init}}}$  for an arbitrary policy  $\pi_{\mathrm{init}} \in \Pi$ , the  $k$ -th iteration of our iterative procedure for solving (20b) can be described as follows.
- Computing a search direction. The search direction of the Frank-Wolfe algorithm is typically taken to be a feasible direction that maximizes its correlation with the gradient of the objective function (Bubeck, 2015). When specialized to the current sub-problem (20b), the search direction can be taken to be the Dirac measure  $\delta_{\pi^{(k)}}$ , where
$$
\pi^ {(k)} = \arg \max  _ {\pi \in \Pi} f (\pi , \mu^ {(k)}) := \sum_ {h = 1} ^ {H} \sum_ {s \in \mathcal {S}} \mathbb {E} _ {a \sim \pi_ {h} ^ {t + 1} (\cdot | s)} \left[ \frac {\widehat {d} _ {h} ^ {\pi} (s , a) \widehat {d} _ {h} ^ {\text {o f f}} (s , a)}{\left(\frac {1}{K ^ {\mathrm {o n}} H} + \mathbb {E} _ {\pi^ {\prime} \sim \mu^ {(k)}} [ \widehat {d} _ {h} ^ {\pi^ {\prime}} (s , a) ]\right) ^ {2}} \right]. \tag {32}
$$
As it turns out, this optimization problem (32) can be efficiently solved by applying dynamic programming (Bertsekas, 2017) to an augmented MDP  $\mathcal{M}^{\mathrm{off}}$  constructed as follows.
- Introduce an augmented finite-horizon MDP  $\mathcal{M}^{\mathrm{off}} = (\mathcal{S}\cup \{s_{\mathrm{aug}}\} ,\mathcal{A},H,\widehat{P}^{\mathrm{aug}},r^{\mathrm{off}})$ , where  $s_{\mathrm{aug}}$  is an augmented state. We choose the reward function to be
$$
r _ {h} ^ {\text {o f f}} (s, a) = \left\{ \begin{array}{l l} \frac {\pi_ {h} ^ {t + 1} (a \mid s) \widehat {d} _ {h} ^ {\text {o f f}} (s , a)}{\left(\frac {1}{K ^ {\text {o n}} H} + \mathbb {E} _ {\pi^ {\prime} \sim \mu (k)} \left[ \widehat {d} _ {h} ^ {\pi^ {\prime}} (s , a) \right]\right) ^ {2}}, & \text {i f} (s, a, h) \in \mathcal {S} \times \mathcal {A} \times [ H ], \\ 0, & \text {i f} (s, a, h) \in \left\{s _ {\text {a u g}} \right\} \times \mathcal {A} \times [ H ], \end{array} \right. \tag {33}
$$
and the probability transition kernel as
$$
\widehat {P} _ {h} ^ {\mathrm {a u g}} \left(s ^ {\prime} \mid s, a\right) = \left\{ \begin{array}{l l} \widehat {P} _ {h} \left(s ^ {\prime} \mid s, a\right), & \text {i f} s ^ {\prime} \in \mathcal {S} \\ 1 - \sum_ {s ^ {\prime} \in \mathcal {S}} \widehat {P} _ {h} \left(s ^ {\prime} \mid s, a\right), & \text {i f} s ^ {\prime} = s _ {\mathrm {a u g}} \end{array} \quad \text {f o r a l l} (s, a, h) \in \mathcal {S} \times \mathcal {A} \times [ H ], \right. \tag {34a}
$$
$$
\widehat {P} _ {h} ^ {\mathrm {a u g}} \left(s ^ {\prime} \mid s _ {\mathrm {a u g}}, a\right) = \mathbb {1} \left(s ^ {\prime} = s _ {\mathrm {a u g}}\right) \quad \text {f o r a l l} (a, h) \in \mathcal {A} \times [ H ]. \tag {34b}
$$
- Frank-Wolfe updates. We then update the iterate  $\mu^{(k + 1)}$  as a convex combination of the current iterate and the direction found in the previous step:
$$
\mu^ {(k + 1)} = (1 - \alpha) \mu^ {(k)} + \alpha \mathbb {1} _ {\pi^ {(k)}}, \tag {35}
$$
where the stepsize is chosen to be
$$
\alpha = \frac {S}{\left(K ^ {\text {o n}} H\right) ^ {3}}. \tag {36}
$$
Stopping rule. It is also necessary to specify the stopping rule of the above iterative procedure. Throughout this paper, the above subroutine will terminate as long as
$$
\sum_ {h = 1} ^ {H} \sum_ {s \in \mathcal {S}} \mathbb {E} _ {a \sim \pi_ {h} ^ {t + 1} (\cdot | s)} \left[ \frac {\widehat {d} _ {h} ^ {\text {o f f}} (s , a)}{\frac {1}{K ^ {\text {o n}} H} + \mathbb {E} _ {\pi^ {\prime} \sim \mu^ {(k)}} \left[ \widehat {d} _ {h} ^ {\pi^ {\prime}} (s , a) \right]} \right] \leq 1 0 8 S H, \tag {37}
$$
with the final output taken to be  $\mu^{t + 1} = \mu^{(k)}$ . We shall justify the feasibility of this stopping rule (namely, the fact that this stopping criterion can be met by some mixed policy) in Appendix F.
Iteration complexity. Encouragingly, the above subroutine in conjunction with the stopping rule (37) leads to an iteration complexity no larger than
$$
\text {(i t e r a t i o n c o m p l e x i t y)} \quad O \left(\frac {\left(K ^ {\mathrm {o n}} H\right) ^ {4}}{S ^ {2}}\right) \tag {38}
$$
The proof of this claim is postponed to Section F.
Algorithm 2: Subroutine for solving the sub-problem (20b).
1 Initialize:  $\mu^{(1)} = 1_{\pi_{\mathrm{init}}}$  for an arbitrary policy  $\pi_{\mathrm{init}} \in \Pi$
2 for  $k = 1,2,\dots$  do
3 Exit for-loop if the following condition is met: // stopping criterion
$$
\sum_ {h = 1} ^ {H} \sum_ {s \in S} \mathbb {E} _ {a \sim \pi_ {h} ^ {t + 1} (\cdot | s)} \left[ \frac {\widehat {d} _ {h} ^ {\text {o f f}} (s , a)}{\frac {1}{K ^ {\text {o n}} H} + \mathbb {E} _ {\pi^ {\prime} \sim \mu^ {(k)}} \left[ \widehat {d} _ {h} ^ {\pi^ {\prime}} (s , a) \right]} \right] \leq 1 0 8 S H. \tag {39}
$$
5 | Compute the optimal deterministic policy
$\mathcal{M}_{\mathrm{off}} = (\mathcal{S}\cup \{s_{\mathrm{aug}}\} ,\mathcal{A},H,\widehat{P}^{\mathrm{aug}},r_{\mathrm{off}})$  , where  $s_{\mathrm{aug}}$  is an augmented state,
$$
r _ {h} ^ {\text {o f f}} (s, a) = \left\{ \begin{array}{l l} \frac {\pi_ {h} ^ {t + 1} (a \mid s) \widehat {d} _ {h} ^ {\text {o f f}} (s , a)}{\left(\frac {1}{K ^ {\text {o n}} H} + \mathbb {E} _ {\pi^ {\prime} \sim \mu (k)} \left[ \widehat {d} _ {h} ^ {\pi^ {\prime}} (s , a) \right]\right) ^ {2}}, & \text {i f} (s, a, h) \in \mathcal {S} \times \mathcal {A} \times [ H ], \\ 0, & \text {i f} (s, a, h) \in \left\{s _ {\text {a u g}} \right\} \times \mathcal {A} \times [ H ], \end{array} \right. \tag {40}
$$
and the augmented probability transition kernel is given by
$$
\widehat {P} _ {h} ^ {\mathrm {a u g}} \left(s ^ {\prime} \mid s, a\right) = \left\{ \begin{array}{l l} \widehat {P} _ {h} \left(s ^ {\prime} \mid s, a\right), & \text {i f} s ^ {\prime} \in \mathcal {S} \\ 1 - \sum_ {s ^ {\prime} \in \mathcal {S}} \widehat {P} _ {h} \left(s ^ {\prime} \mid s, a\right), & \text {i f} s ^ {\prime} = s _ {\mathrm {a u g}} \end{array} \quad \text {f o r a l l} (s, a, h) \in \mathcal {S} \times \mathcal {A} \times [ H ]; \right. \tag {41a}
$$
$$
\widehat {P} _ {h} ^ {\mathrm {a u g}} \left(s ^ {\prime} \mid s _ {\mathrm {a u g}}, a\right) = \mathbb {1} \left(s ^ {\prime} = s _ {\mathrm {a u g}}\right) \quad \text {f o r a l l} (a, h) \in \mathcal {A} \times [ H ].
$$
(41b)
Let  $\pi^{(k)}$  be the corresponding optimal deterministic policy of  $\pi^{(k),\mathrm{aug}}$  in the original state space.
7 Update // Frank-Wolfe update
$$
\mu^ {(k + 1)} = (1 - \alpha) \mu^ {(k)} + \alpha \mathbb {1} _ {\pi^ {(k)}}, \quad \text {w h e r e} \quad \alpha = \frac {S}{(K ^ {\mathrm {o n}} H) ^ {3}}.
$$
9 Output: the policy mixture  $\mu^{t + 1} = \mu^{(k)}$
D Useful algorithmic subroutines from prior works
In this section, we provide precise descriptions of several useful algorithmic subroutines that have been developed in recent works. The algorithm procedures are directly quoted from these prior works, with slight modification.
D.1 Subroutine: occupancy estimation for any policy  $\pi$
The first subroutine we'd like to describe is concerned with estimating the occupancy distribution  $d^{\pi}$  induced by any policy  $\pi$ , based on a carefully designed exploration strategy. This algorithm, proposed by Li et al. (2023), seeks to estimate  $\{d_h^\pi\}$  step by step (i.e., from  $h = 1,\dots ,H$ ). For each  $h$ , it computes an appropriate exploration policy  $\pi^{\mathrm{explore},h}$  to adequately explore what happens between step  $h$  and step  $h + 1$ , and then collect  $N$  sample trajectories using  $\pi^{\mathrm{explore},h}$ . These turns allow us to estimate the occupancy distribution  $d_{h + 1}^{\pi}$  for step  $h + 1$ . See Algorithm 3 for a precise description.
Algorithm 3: Subroutine for estimating occupancy distributions for any policy  $\pi$  (Li et al., 2023).
1 Input: target success probability  $1 - \delta$ , threshold  $\xi = c_{\xi}H^{3}S^{3}A^{3}\log (HSA / \delta)$ .
/* Estimate occupancy distributions for step 1. */
2 Draw  $N$  independent episodes (using arbitrary policies), whose initial states are i.i.d. drawn
from  $s_1^{n,0} \stackrel{\text{i.i.d.}}{\sim} \rho (1 \leq n \leq N)$ . Define the following functions
$$
\widehat {d} _ {1} ^ {\pi} (s) = \frac {1}{N} \sum_ {n = 1} ^ {N} \mathbb {1} \left\{s _ {1} ^ {n, 0} = s \right\}, \quad \widehat {d} _ {1} ^ {\pi} (s, a) = \widehat {d} _ {1} ^ {\pi} (s) \pi_ {1} (a \mid s) \tag {42}
$$
for any deterministic policy  $\pi : \mathcal{S} \times [H] \to \Delta(\mathcal{A})$  and any  $(s, a) \in \mathcal{S} \times \mathcal{A}$ . (Note that these functions are defined for future use and not computed for the moment, as we have not specified policy  $\pi$ .)
/\*Estimate occupancy distributions for steps 2,...,H. \*/
3 for  $h = 1$  to  $H - 1$  do
/* Collect  $N$  sample trajectories using a suitable exploration policy. */
4 Call Algorithm 4 to compute an exploration policy  $\pi^{\mathrm{explore},h}$  and compute an estimate  $\tilde{P}_h$  of the true transition kernel  $P_h$ .
/* Specify how to compute  $\widehat{d}_{h+1}^{\pi}$  for any policy  $\pi$ . */
For any deterministic policy  $\pi : \mathcal{S} \times [H] \to \Delta(\mathcal{A})$  and any  $(s, a) \in \mathcal{S} \times \mathcal{A}$ , define
$$
\widehat {d} _ {h + 1} ^ {\pi} (s) = \left\langle \widehat {P} _ {h} (s | \cdot , \cdot), \widehat {d} _ {h} ^ {\pi} (\cdot , \cdot) \right\rangle , \quad \widehat {d} _ {h + 1} ^ {\pi} (s, a) = \widehat {d} _ {h + 1} ^ {\pi} (s) \pi_ {h + 1} (a | s). \tag {43}
$$
We note, however, that Algorithm 3 requires another subroutine to compute a suitable exploration policy  $\pi^{\mathrm{explore},h}$ . As it turns out, this can be accomplished by approximately solving the following problem
$$
\hat {\mu} ^ {h} \approx \arg \max  _ {\mu \in \Delta (\Pi)} \sum_ {(s, a) \in \mathcal {S} \times \mathcal {A}} \log \left[ \frac {1}{K H} + \underset {\pi \sim \mu} {\mathbb {E}} \left[ \widehat {d} _ {h} ^ {\pi} (s, a) \right] \right] \tag {44}
$$
via the Frank-Wolfe algorithm and returning  $\pi^{\mathrm{explore},h} = \mathbb{E}_{\pi \sim \hat{\mu}^h}[\pi]$ . See Algorithm 4 for details.
D.2 Subroutine: reward-agnostic online exploration
Based on the estimated occupancy distributions specified in Algorithm 3, Li et al. (2023) proposed a reward-independent online exploration scheme that proves useful in exploring an unknown environment. In a nutshell, this scheme computes a desired exploration policy by approximately solving the following optimization sub-problem:
$$
\mu^ {\text {e x p l o r e}} \approx \arg \max  _ {\mu \in \Delta (\Pi)} \left\{\sum_ {h = 1} ^ {H} \sum_ {(s, a) \in \mathcal {S} \times \mathcal {A}} \log \left[ \frac {1}{K ^ {\mathrm {o n}} H} + \mathbb {E} _ {\pi \sim \mu} \left[ \widehat {d _ {h} ^ {\pi}} (s, a) \right] \right] \right\}. \tag {47}
$$
again using the Frank-Wolfe algorithm; the resulting policy takes the form of a mixture of deterministic policies, as given by  $\pi^{\mathrm{explore}} = \mathbb{E}_{\pi \sim \mu^{\mathrm{explore}}}[\pi]$ . This exploration policy is then employed to execute the MDP for a number of times in order to collect enough information about the unknowns. See Algorithm 5 for the whole procedure.
Algorithm 4: Subroutine for computing the exploration policy for step  $h$  in occupancy estimation (Li et al., 2023).
1 Initialize:  $\mu^{(0)} = 1_{\pi_{\mathrm{init}}}$  for an arbitrary policy  $\pi_{\mathrm{init}} \in \Pi, T_{\max} = \lfloor 50SA\log (KH)\rfloor$
2 for  $t = 0$  to  $T_{\mathrm{max}}$  do  $/*$  find the optimal policy \*/
Compute the optimal deterministic policy  $\pi^{(t),\mathsf{b}}$  of the augmented MDP
$\mathcal{M}_{\mathrm{b}}^{h} = (\mathcal{S}\cup \{s_{\mathrm{aug}}\} ,\mathcal{A},H,\widehat{P}^{\mathrm{aug},h},r_{\mathrm{b}}^{h})$  , where  $s_{\mathrm{aug}}$  is an augmented state,
$$
r _ {\mathrm {b}, j} ^ {h} (s, a) = \left\{ \begin{array}{l l} \frac {1}{K ^ {\text {o n}} H + \mathbb {E} _ {\pi \sim \mu^ {(t)}} \left[ \widehat {d} _ {h} ^ {\pi} (s , a) \right]}, & \text {i f} (s, a, j) \in \mathcal {S} \times \mathcal {A} \times \{h \}, \\ 0, & \text {i f} s = s _ {\text {a u g}} \text {o r} j \neq h, \end{array} \right. \tag {45}
$$
and the augmented probability transition kernel is defined as
$$
\widehat {P} _ {j} ^ {\mathrm {a u g}, h} \left(s ^ {\prime} \mid s, a\right) = \left\{ \begin{array}{l l} \widehat {P} _ {j} \left(s ^ {\prime} \mid s, a\right), & \text {i f} s ^ {\prime} \in \mathcal {S} \\ 1 - \sum_ {s ^ {\prime} \in \mathcal {S}} \widehat {P} _ {j} \left(s ^ {\prime} \mid s, a\right), & \text {i f} s ^ {\prime} = s _ {\mathrm {a u g}} \end{array} \quad \text {f o r a l l} (s, a, j) \in \mathcal {S} \times \mathcal {A} \times [ h ]; \right. \tag {46a}
$$
4 if  $s = s_{\mathrm{aug}}$  or  $j > h$  (46b)
Let  $\pi^{(t)}$  be the corresponding optimal deterministic policy of  $\pi^{(t),\mathrm{b}}$  in the original state space.
6 Compute // choose the stepsize
$$
\alpha_ {t} = \frac {\frac {1}{S A} g (\pi^ {(t)} , \widehat {d} , \mu^ {(t)}) - 1}{g (\pi^ {(t)} , \widehat {d} , \mu^ {(t)}) - 1}, \quad \text {w h e r e} \quad g (\pi , \widehat {d}, \mu) = \sum_ {(s, a) \in \mathcal {S} \times \mathcal {A}} \frac {\frac {1}{K ^ {\text {o n}} H} + \widehat {d} _ {h} ^ {\pi} (s , a)}{\frac {1}{K ^ {\text {o n}} H} + \mathbb {E} _ {\pi \sim \mu} [ \widehat {d} _ {h} ^ {\pi} (s , a) ]}.
$$
Here,  $\widetilde{d}_h^\pi (s,a)$  is computed via (42) for  $h = 1$ , and (43) for  $h\geq 2$
7 If  $g(\pi^{(t)},d,\mu^{(t)})\leq 2SA$  then exit for-loop. // stopping rule
8 Update // Frank-Wolfe update
$$
\mu^ {(t + 1)} = \left(1 - \alpha_ {t}\right) \mu^ {(t)} + \alpha_ {t} \mathbb {1} _ {\pi^ {(t)}}.
$$
10 Set  $\pi^{\mathrm{explore},h} = \mathbb{E}_{\pi \sim \mu^{(t)}}[\pi]$  with  $\widehat{\mu}^h = \mu^{(t)}$ . // The final exploration policy for step  $h$ .
/* Draw samples using  $\pi^{\mathrm{explore},h}$  to estimate the transition kernel. */
11 Draw  $N$  independent trajectories  $\{s_1^{n,h},a_1^{n,h},\ldots ,s_{h + 1}^{n,h}\}_{1\leq n\leq N}$  using policy  $\pi^{\mathrm{explore},h}$  and compute
$$
\widehat {P} _ {h} (s ^ {\prime} \mid s, a) = \frac {\mathbb {1} (N _ {h} (s , a) > \xi)}{\max \left\{N _ {h} (s , a) , 1 \right\}} \sum_ {n = 1} ^ {N} \mathbb {1} (s _ {h} ^ {n, h} = s, a _ {h} ^ {n, h} = a, s _ {h + 1} ^ {n, h} = s ^ {\prime}), \qquad \forall (s, a, s ^ {\prime}) \in \mathcal {S} \times \mathcal {A} \times \mathcal {S},
$$
where  $N_{h}(s,a) = \sum_{n = 1}^{N}\mathbb{1}\{s_{h}^{n,h} = s,a_{h}^{n,h} = a\}$ .
12 Output: the exploration policy  $\pi^{\mathrm{explore},h}$ , the weight  $\widehat{\mu}^{h}$ , and the estimated kernel  $\widehat{P}_h$ .
D.3 Subroutine: pessimistic model-based offline RL