SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\pi_ {\ell} (j _ {\ell}) \neq \bar {\pi} (j _ {\ell}), \quad a n d \quad \forall k \neq \ell , \pi_ {\ell} (j _ {k}) = \bar {\pi} (j _ {k}).
$$
We define the star number of  $\Pi$  as  $\dim_{\mathsf{S}}(\Pi) \coloneqq \sup_{\pi \in \Pi} \dim_{\mathsf{S}}(\Pi; \bar{\pi})$ .
Definition 12 (Threshold Dimension (Alon et al., 2019; Li et al., 2022)). Fix any stationary base policy  $\bar{\pi}$ . The threshold dimension of  $\Pi$  w.r.t.  $\bar{\pi}$ , denoted  $\dim_{\mathsf{T}}(\Pi ;\bar{\pi})$ , is the length of the longest sequence  $(j_1,\pi_1),\ldots ,(j_N,\pi_N)$  such that for every  $\ell \in [N]$ :
$$
\forall m \geq \ell , \pi_ {\ell} (j _ {m}) \neq \bar {\pi} (j _ {m}), \quad a n d \quad \forall k <   \ell , \pi_ {\ell} (j _ {k}) = \bar {\pi} (j _ {k}).
$$
We define the threshold dimension of  $\Pi$  as  $\dim_{\mathsf{T}}(\Pi) \coloneqq \sup_{\pi \in \Pi} \dim_{\mathsf{T}}(\Pi; \bar{\pi})$ .
Relationships Between Complexity Measures. From (Li et al., 2022, Theorem 8) we have the relationship for every  $\Pi$ :
$$
\left. \max  \left\{\dim_ {\mathfrak {S}} (\Pi), \dim_ {\mathsf {T}} (\Pi) \right\} \leq \dim_ {\mathsf {E}} (\Pi) \leq 4 ^ {\max  \left\{\dim_ {\mathfrak {S}} (\Pi), \dim_ {\mathsf {T}} (\Pi) \right\}} \right.. \tag {34}
$$
The lower bound is obvious from the definitions and in general cannot be improved; the upper bound also cannot be improved beyond constant factors in the exponent (Li et al., 2022).
We also remark that it is clear from the definitions that VC dimension is a lower bound on all three (eluder, star, threshold); however, VC(II) can be arbitrarily smaller.
H.2 Bounds on Spanning Capacity
Now we investigate bounds on the spanning capacity in terms of the aforementioned quantities.
Theorem 7. For any policy class  $\Pi$  satisfying Assumption 1 we have
$$
\max  \left\{\min  \left\{\dim_ {\mathsf {S}} (\Pi), H + 1 \right\}, \min  \left\{2 ^ {\lfloor \log_ {2} \dim_ {\mathsf {T}} (\Pi) \rfloor}, 2 ^ {H} \right\} \right\} \leq \mathfrak {C} (\Pi) \leq 2 ^ {\dim_ {\mathsf {E}} (\Pi)}.
$$
We give several remarks on Theorem 7. The proof is deferred to the following subsection.
It is interesting to understand to what degree we can improve the bounds in Theorem 7. On the lower bound side, we note that the each of the terms individually cannot be sharpened:
- For the singleton class  $\Pi_{\mathrm{sing}}$  we have  $\mathfrak{C}(\Pi_{\mathrm{sing}}) = \min \{K, H + 1\}$  and  $\dim_S(\Pi_{\mathrm{sing}}) = K$ .
- For the threshold class  $\Pi_{\mathrm{thres}} \coloneqq \{\pi_i(j) \mapsto \mathbb{1} \{j \geq i\} : i \in [K]\}$ , when  $K$  is a power of two, it can be shown that  $\mathfrak{C}(\Pi_{\mathrm{thres}}) = \min \{K, 2^H\}$  and  $\dim_{\mathsf{T}}(\Pi_{\mathrm{thres}}) = K$ .
While we also provide an upper bound in terms of  $\dim_{\mathsf{E}}(\Pi)$ , we note that there can be a huge gap between the lower bound and the upper bound. In fact, our upper bound is likely very loose since we are not aware of any policy class for which the upper bound is non-vacuous, i.e.  $2^{\dim_{\mathsf{E}}(\Pi)} \ll \min \left\{2^{H}, |\Pi|, 2KH\right\}$  (implying that our bound improves on Proposition 3). It would be interesting to understand how to improve the upper bound (possibly, to scale polynomially with  $\dim_{\mathsf{E}}(\Pi)$ , or more directly in terms of some function of  $\dim_{\mathsf{S}}(\Pi)$  and  $\dim_{\mathsf{T}}(\Pi)$ ); we leave this as a direction for future research.
Lastly, we remark that the lower bound of  $\mathfrak{C}(\Pi) \geq \min \{\Omega(\dim_{\mathsf{T}}(\Pi)), 2^H\}$  is a generalization of previous bounds which show that linear policies cannot be learned with  $\mathrm{poly}(H)$  sample complexity (e.g., Du et al., 2019b), since linear policies (even in 2 dimensions) have infinite threshold dimension.
<sup>8</sup>Our definition of the combinatorial eluder dimension comes from Li et al. (2022) and is also called the "policy eluder dimension" in the paper Foster et al. (2021c). In particular, it is defined with respect to a base function  $\bar{\pi}$ . This differs in spirit from the original definition (Russo and Van Roy, 2013) as well as the combinatorial variant (Mou et al., 2020), which for every  $\ell$  asks for witnessing pairs of policies  $\pi_{\ell}, \pi_{\ell}'$ . Our definition is never larger than the original version since we require that  $\pi_{\ell}' = \bar{\pi}$  to be fixed for every  $\ell \in [N]$ .
H.2.1 Proof of Theorem 7
We will prove each bound separately.
Star Number Lower Bound. Let  $\bar{\pi} \in \Pi$  and the sequence  $(j_1, \pi_1), \ldots, (j_N, \pi_N)$  witness  $\dim_S(\Pi) = N$ . We construct a deterministic MDP  $M$  for which the cumulative reachability at layer  $h_{\max} \coloneqq \min \{N, H\}$  (Definition 1) is at least  $\min \{N, H + 1\}$ . The transition dynamics of  $M$  are as follows; we will only specify the transitions until  $h_{\max} - 1$  (afterwards, the transitions can be arbitrary).
- The starting state of  $M$  at layer  $h = 1$  is  $s_{(j_1,1)}$ .
- (On-Chain Transitions): For every  $h < h_{\max}$ ,
$$
P (s ^ {\prime} \mid s _ {(j _ {h}, h)}, a) = \left\{ \begin{array}{l l} \mathbb {1} \big \{s ^ {\prime} = s _ {(j _ {h + 1}, h + 1)} \big \} & \text {i f} a = \bar {\pi} (s _ {(j _ {h}, h)}), \\ \mathbb {1} \big \{s ^ {\prime} = s _ {(j _ {h}, h + 1)} \big \} & \text {i f} a \neq \bar {\pi} (s _ {(j _ {h}, h)}). \end{array} \right.
$$
- (Off-Chain Transitions): For every  $h < h_{\max}$ , state index  $\tilde{j} \neq j_h$ , and action  $a \in \mathcal{A}$
$$
P (s ^ {\prime} \mid s _ {(\tilde {j}, h)}, a) = \mathbb {1} \Big \{s ^ {\prime} = s _ {(\tilde {j}, h + 1)} \Big \}.
$$
We now compute the cumulative reachability at layer  $h_{\mathrm{max}}$ . If  $N \leq H$ , the number of  $(s, a)$  pairs that  $\Pi$  can reach in  $M$  is  $N$  (namely the pairs  $(s_{(j_1, N)}, 1), \dots, (s_{(j_N, N)}, 1)$ ). On the other hand, if  $N > H$ , then the number of  $(s, a)$  pairs that  $\Pi$  can reach in  $M$  is  $H + 1$  (namely the pairs  $(s_{(j_1, H)}, 1), \dots, (s_{(j_H, H)}, 1), (s_{(j_H, H)}, 0)$ ). Thus we have shown that  $\mathfrak{C}(\Pi) \geq \min \{N, H + 1\}$ .
Threshold Dimension Lower Bound. Let  $\bar{\pi} \in \Pi$  and the sequence  $(j_1, \pi_1), \ldots, (j_N, \pi_N)$  witness  $\dim_{\mathsf{T}}(\Pi) = N$ . We define a deterministic MDP  $M$  as follows. Set  $h_{\max} = \min \{\lfloor \log_2 N \rfloor, H\}$ . Up until layer  $h_{\max}$ , the MDP will be a full binary tree of depth  $h_{\max}$ ; afterward, the transitions will be arbitrary. It remains to assign state labels to the nodes of the binary tree (of which there are  $2^{h_{\max}} - 1 \leq N$ ). We claim that it is possible to do so in a way so that every policy  $\pi_\ell$  for  $\ell \in [2^{h_{\max}}]$  reaches a different state-action pair at layer  $h_{\max}$ . Therefore the cumulative reachability of  $\Pi$  on  $M$  is at least  $2^{h_{\max}} = \min \{2^{\lfloor \log_2 N \rfloor}, 2^H\}$  as claimed.
It remains to prove the claim. The states of  $M$  are labeled  $j_{2},\dots ,j_{2^{h_{\mathrm{max}}}}$  according to the order they are traversed using inorder traversal of a full binary tree of depth  $h_\mathrm{max}$  (Cormen et al., 2022). One can view the MDP  $M$  as a binary search tree where the action 0 corresponds to going left and the action 1 corresponds to going right. Furthermore, if we imagine that the leaves of the binary search tree at depth  $h_\mathrm{max}$  are labeled from left to right with the values  $1.5,2.5,\dots ,2^{h_{\mathrm{max}}} + 0.5$  , then it is clear that for any  $\ell \in [2^{h_{\mathrm{max}}}]$  , the trajectory generated by running  $\pi_{\ell}$  on  $M$  is exactly the path obtained by searching for the value  $\ell +0.5$  in the binary search tree. Thus we have shown that the cumulative reachability of  $\Pi$  on  $M$  is the number of leaves at depth  $h_\mathrm{max}$  , thus proving the claim.
Eluder Dimension Upper Bound. Let  $\dim_{\mathsf{E}}(\Pi) = N$ . We only need to prove this statement when  $N \leq H$ , as otherwise the statement already follows from Proposition 3. Let  $(M^{\star}, h^{\star})$  be the MDP and layer which witness  $\mathfrak{C}(\Pi)$ . Also denote  $s_1$  to be the starting state of  $M^{\star}$ . For any state  $s$ , we denote  $\mathrm{child}_0(s)$  and  $\mathrm{child}_1(s)$  to be the states in the next layer which are reachable by taking  $a = 0$  and  $a = 1$  respectively.
For any reachable state  $s$  at layer  $h$  in the MDP  $M^{\star}$  we define the function  $f(s)$  as follows. For any state  $s$  at layer  $h^{\star}$ , we set  $f(s) := 1$  if the state-action pairs  $(s,0)$  and  $(s,1)$  are both reachable by  $\Pi$ ; otherwise we set  $f(s) := 0$ . For states in layers  $h < h^{\star}$  we set
$$
f (s) := \left\{ \begin{array}{l l} \max  \{f (\operatorname {c h i l d} _ {0} (s)), f (\operatorname {c h i l d} _ {1} (s)) \} + 1 & \text {i f b o t h (s , 0) a n d (s , 1) a r e r e a c h a b l e b y \Pi}, \\ f (\operatorname {c h i l d} _ {0} (s)) & \text {i f o n l y (s , 0) i s r e a c h a b l e b y \Pi}, \\ f (\operatorname {c h i l d} _ {1} (s)) & \text {i f o n l y (s , 1) i s r e a c h a b l e b y \Pi}. \end{array} \right.
$$
We claim that for any state  $s$ , the contribution to  $\mathfrak{C}(\Pi)$  by policies that pass through  $s$  is at most  $2^{f(s)}$ . We prove this by induction. Clearly, the base case of  $f(s) = 0$  or  $f(s) = 1$  holds. If only one child of  $s$  is reachable by  $\Pi$  then the contribution to  $\mathfrak{C}(\Pi)$  by policies that pass through  $s$  equal to the contribution to  $\mathfrak{C}(\Pi)$  by policies that pass through the child of  $s$ . If both children of  $s$  are reachable by  $\Pi$  then the contribution towards  $\mathfrak{C}(\Pi)$  by policies that pass through  $s$  is upper bounded by the
sum of the contribution towards  $\mathfrak{C}(\Pi)$  by policies that pass through the two children, i.e. it is at most  $2^{f(\mathrm{child}_0(s))} + 2^{f(\mathrm{child}_1(s))} \leq 2^{f(s)}$ . This concludes the inductive argument.
Now we bound  $f(s_{1})$ . Observe that the quantity  $f(s_{1})$  counts the maximum number of layers  $h_{1}, h_{2}, \dots, h_{L}$  that satisfy the following property: there exists a trajectory  $\tau = (s_{1}, a_{1}, \dots, s_{H}, a_{H})$  for which we can find  $L$  policies  $\pi_{1}, \pi_{2}, \dots, \pi_{L}$  so that each policy  $\pi_{\ell}$  when run on  $M^{\star}$  (a) reaches  $s_{h_{\ell}}$ , and (b) takes action  $\pi_{\ell}(s_{h_{\ell}}) \neq a_{h_{\ell}}$ . Thus, by definition of the eluder dimension we have  $f(s_{1}) \leq N$ . Therefore, we have shown that the cumulative reachability of  $\Pi$  in  $M^{\star}$  is at most  $2^{N}$ .
I Extension: Can Expert Feedback Help in Agnostic PAC RL?
For several policy classes, the spanning capacity may be quite large, and our lower bounds (Theorem 2 and 3) demonstrate an unavoidable dependence on  $\mathfrak{C}(\Pi)$ . In this section, we investigate whether it is possible to achieve bounds which are independent of  $\mathfrak{C}(\Pi)$  and instead only depend on  $\mathrm{poly}(A,H,\log |\Pi |)$  under a stronger feedback model.
Our motivation comes from practice. It is usually uncommon to learn from scratch: often we would like to utilize domain expertise or prior knowledge to learn with fewer samples. For example, during training one might have access to a simulator which can roll out trajectories to estimate the optimal value function  $Q^{\star}$ , or one might have access to expert advice / demonstrations. However, this access does not come for free; estimating value functions with a simulator requires some computation, or the "expert" might be a human who is providing labels or feedback on the performance of the algorithm. Motivated by this, we consider additional feedback in the form of an expert oracle.
Definition 13 (Expert oracle). An expert oracle  $\mathrm{O}_{\mathrm{exp}}: S \times \mathcal{A} \to \mathbb{R}$  is a function which given an  $(s, a)$  pair as input returns the  $Q$  value of some expert policy  $\pi_{\circ}$ , denoted  $Q^{\pi_{\circ}}(s, a)$ .
Definition 13 is a natural formulation for understanding how expert feedback can be used for agnostic RL in large state spaces. We do not require  $\pi_{\circ}$  to be the optimal policy (either over the given policy class  $\Pi$  or over all  $\mathcal{A}^S$ ). The objective is to compete with  $\pi_{\circ}$ , i.e., with probability at least  $1 - \delta$ , return a policy  $\widehat{\pi}$  such that  $V^{\widehat{\pi}} \geq V^{\pi_{\circ}} - \varepsilon$  using few online interactions with the MDP and calls to  $\mathrm{O}_{\mathrm{exp}}$ .
A sample efficient algorithm (one which uses at most  $\mathrm{poly}(A,H,\log |\Pi |,\varepsilon^{-1},\delta^{-1})$  online trajectories and calls to the oracle) must use both forms of access. Our lower bounds (Theorem 2 and 3) show that an algorithm which only uses online access to the MDP must use  $\Omega (\mathfrak{C}(\Pi))$  samples. Likewise, an algorithm which only queries the expert oracle must use  $\Omega (SA)$  queries because it does not know the dynamics of the MDP, so the best it can do is just learn the optimal action on every state.
Relationship to Prior Works. The oracle  $\mathrm{O_{exp}}$  is closely related to several previously considered settings (Golowich and Moitra, 2022; Gupta et al., 2022; Amortila et al., 2022). Prior work (Golowich and Moitra, 2022; Gupta et al., 2022) has studied tabular RL with inexact predictions for either the optimal  $Q^{\star}$  or  $V^{\star}$ . They assume access to the entire table of values; since we study the agnostic RL setting with a large state space, we formalize access to predictions via the expert oracle. Amortila et al. (2022) study a related expert action oracle under the assumption of linear value functions. They show that in a generative model, with  $\mathrm{poly}(d)$  resets and queries to an expert action oracle, one can learn an  $\varepsilon$ -optimal policy, thus circumventing known hardness results for the linear value function setting. Up to a factor of  $A$ , one can simulate queries to the expert action oracle by querying  $\mathrm{O_{exp}}(s,a)$  for each  $a \in \mathcal{A}$ .
I.1 Upper Bound under Realizability
Under realizability (namely,  $\pi_{\circ} \in \Pi$ ), it is known that the dependence on  $\mathfrak{C}(\Pi)$  can be entirely removed with few queries to the expert oracle.
Theorem 8. For any  $\Pi$  such that  $\pi_{\circ} \in \Pi$ , with probability at least  $1 - \delta$ , the AggreVaTe algorithm (Ross and Bagnell, 2014) computes an  $\varepsilon$ -optimal policy using
$$
n _ {1} = O \left(\frac {A ^ {2} H ^ {2}}{\varepsilon^ {2}} \cdot \log \frac {| \Pi |}{\delta}\right) \text {o n l i n e t r a j e c t o r i e s} \quad \text {a n d} \quad n _ {2} = O \left(\frac {A ^ {2} H ^ {2}}{\varepsilon^ {2}} \cdot \log \frac {| \Pi |}{\delta}\right) \text {c a l l s t o} O _ {\exp}.
$$
The proof is omitted; it can be found in (Ross and Bagnell, 2014; Agarwal et al., 2019). We also note that actually we require a slightly weaker oracle than  $\mathrm{O}_{\mathrm{exp}}$ : the AggreVaTe algorithm only queries the value of  $Q^{\pi_{\circ}}$  on  $(s,a)$  pairs which are encountered in online trajectories.
I.2 Lower Bound in Agnostic Setting
Realizability of the expert policy used for  $\mathrm{O_{exp}}$  is a rather strong assumption in practice. For example, one might choose to parameterize  $\Pi$  as a class of neural networks, but one would like to use human annotators to give expert feedback on the actions taken by the learner; here, it is unreasonable to assume that realizability of the expert policy holds.
We sketch a lower bound in Theorem 9 that shows that without realizability  $(\pi_{\circ} \notin \Pi)$ , we can do no better than  $\Omega(\mathfrak{C}(\Pi))$  queries to a generative model or queries to  $\mathrm{O}_{\mathrm{exp}}$ .
Theorem 9 (informal). For any  $H \in \mathbb{N}$ ,  $C \in [2^H]$ , there exists a policy class  $\Pi$  with  $\mathfrak{C}(\Pi) = |\Pi| = C$ , expert policy  $\pi_{\circ} \notin \Pi$ , and family of MDPs  $\mathcal{M}$  with state space  $\mathcal{S}$  of size  $O(2^H)$ , binary action space, and horizon  $H$  such that any algorithm that returns a 1/4-optimal policy must either use  $\Omega(C)$  queries to a generative model or  $\Omega(C)$  queries to the  $\mathrm{O}_{\mathrm{exp}}$ .
Before sketching the proof, several remarks are in order.
- By comparing with Theorem 8, Theorem 9 demonstrates that realizability of the expert policy is crucial for circumventing the dependence on spanning capacity via the expert oracle.
- In the lower bound construction of Theorem 9,  $\pi_{\circ}$  is the optimal policy. Furthermore, while  $\pi_{\circ} \notin \Pi$ , the lower bound still has the property that  $V^{\pi_{\circ}} = V^{\star} = \max_{\pi \in \Pi} V^{\pi}$ ; that is, the best-in-class policy  $\widetilde{\pi} := \arg \max_{\pi \in \Pi} V^{\pi}$  attains the same value as the optimal policy. This is possible because there exist multiple states for which  $\widetilde{\pi}(s) \neq \pi_{\circ}(s)$ , however these states have  $d^{\widetilde{\pi}}(s) = 0$ . Thus, we also rule out guarantees of the form  $V^{\widehat{\pi}} \geq \max_{\pi \in \Pi} V^{\pi} - \varepsilon$ .
- Since the oracle  $\mathrm{O}_{\mathrm{exp}}$  is stronger than the expert action oracle (Amortila et al., 2022) (up to a factor of  $A$ ), the lower bound extends to this weaker feedback model. Investigating further assumptions that enable statistically tractable agnostic learning with expert feedback is an interesting direction for future work.
Proof Sketch of Theorem 9. We present the construction as well as intuition for the lower bound, leaving out a formal information-theoretic proof.
**Construction of MDP Family.** We describe the family of MDPs  $\mathcal{M}$ . In every layer, the state space is  $S_{h} = \{s_{(j,h)}: j \in [2^{h}]\}$ , except at  $S_{H}$  where we have an additional terminating state,  $S_{H} = \{s_{(j,h)}: j \in [2^{H}]\} \cup \{s_{\perp}\}$ . The action space is  $\mathcal{A} = \{0,1\}$ .
The MDP family  $\mathcal{M} = \{M_{b,f^{\star}}\}_{b\in \mathcal{A}^{H - 1},f^{\star}\in \mathcal{A}^{S_H}}$  is parameterized by a bit sequence  $b\in \mathcal{A}^{H - 1}$  as well as a labeling function  $f^{\star}\in \mathcal{A}^{S_H}$ . The size of  $\mathcal{M}$  is  $2^{H - 1}\cdot 2^{2^H}$ . We now describe the transitions and rewards for any  $M_{b,f^{\star}}$ . In the following, let  $s_b\in S_{H - 1}$  be the state that is reached by playing the sequence of actions  $(b[1],b[2],\dots ,b[H - 2])$  for the first  $H - 2$  layers.
- Transitions. For the first  $H - 2$  layers, the transitions are the same for  $M_{b,f^{\star}} \in \mathcal{M}$ . At layer  $H - 1$ , the transition depends on  $b$ .
- For any  $h \in \{1, 2, \dots, H - 2\}$ , the transitions are deterministic and given by a tree process: namely
$$
P (s ^ {\prime} \mid s _ {(j, h)}, a) = \left\{ \begin{array}{l l} \mathbb {1} \big \{s ^ {\prime} = s _ {(2 j - 1, h + 1)} \big \} & \text {i f} a = 0, \\ \mathbb {1} \big \{s ^ {\prime} = s _ {(2 j, h + 1)} \big \} & \text {i f} a = 1. \end{array} \right.
$$
- At layer  $H - 1$ , for the state  $s_b$ , the transition is  $P(s' \mid s_b, a) = \mathbb{1}\{s' = s_\perp\}$  for any  $a \in \mathcal{A}$ . For all other states, the transitions are uniform to  $S_H$ , i.e., for any  $s \in S_{H-1} \setminus \{s_b\}$ ,  $a \in \mathcal{A}$ , the transition is  $P(\cdot \mid s, a) = \mathrm{Uniform}(S_H \setminus \{s_\perp\})$ .
- Rewards. The rewards depend on the  $b \in \mathcal{A}^{H - 1}$  and  $f^{\star} \in \mathcal{A}^{\mathcal{S}_H}$ .
- The reward at layer  $H - 1$  is  $R(s, a) = \mathbb{1}\{s = s_b, a = b[H - 1]\}$ .
- The reward at layer  $H$  is
$$
R (s _ {\perp}, a) = 0
$$
for any  $a\in \mathcal{A}$
$$
R (s, a) = \mathbb {1} \{a = f ^ {\star} (s) \}
$$
for any  $s \neq s_{\perp}$ ,  $a \in \mathcal{A}$ .
From the description of the transitions and rewards, we can compute the value of  $Q^{\star}(\cdot ,\cdot)$
- Layers  $1, \dots, H - 2$ : For any  $s \in S_1 \cup S_2 \cup \dots \cup S_{H - 2}$  and  $a \in \mathcal{A}$ , the  $Q$ -value is  $Q^{\star}(s, a) = 1$ .
- Layer  $H - 1$ : At  $s_b$ , the  $Q$ -value is  $Q^{\star}(s_b, a) = \mathbb{1}\{a = b[H - 1]\}$ . For other states  $s \in S_{H - 1} \setminus \{s_b\}$ , the  $Q$ -value is  $Q^{\star}(s, a) = 1$  for any  $a \in \mathcal{A}$ .
- Layer  $H$ : At  $s_{\perp}$ , the  $Q$ -value is  $Q^{\star}(s_{\perp}, a) = 0$  for any  $a \in \mathcal{A}$ . For other states  $s \in S_H \backslash \{s_{\perp}\}$ , the  $Q$ -value is  $Q^{\star}(s, a) = \mathbb{1}\{a = f^{\star}(s)\}$ .
Lastly, the optimal value is  $V^{\star} = 1$ .
Expert Oracle. The oracle  $\mathsf{O}_{\mathrm{exp}}$  returns the value of  $Q^{\star}(s,a)$ .
Policy Class. The policy class  $\Pi$  is parameterized by bit sequences of length  $H - 1$ . Denote the function  $\mathrm{bin}:\{0,1,\dots ,2^{H - 1}\} \mapsto \mathcal{A}^{H - 1}$  that returns the binary representation of the input. Specifically,
$$
\Pi := \left\{\pi_ {b}: b \in \left\{\operatorname {b i n} (i): i \in \{0, 1, \dots , C - 1 \} \right\} \right\},
$$
where each  $\pi_b$  is defined such that  $\pi_b(s) := b[h]$  if  $s \in S_h$ , and  $\pi_b(s) := 0$  otherwise. By construction it is clear that  $\mathfrak{C}(\Pi) = |\Pi| = C$ .
Lower Bound Argument. Consider any  $M_{b,f^{\star}}$  where  $b \in \{\mathrm{bin}(i): i \in \{0,1,\dots,C - 1\}\}$  and  $f^{\star} \in \mathcal{A}^{S_H}$ . There are two ways for the learner to identify a 1/4-optimal policy in  $M_{b,f^{\star}}$ :
- Find the value of  $b$ , and return the policy  $\pi_b$ , which has  $V^{\pi_b} = 1$ .
- Estimate  $\widehat{f} \approx f^{\star}$ , and return the policy  $\pi_{\widehat{f}}$  which picks arbitrary actions for any  $s \in S_1 \cup S_2 \cup \dots \cup S_{H-1}$  and picks  $\pi_{\widehat{f}}(s) = \widehat{f}(s)$  on  $s \in S_H$ .
We claim that in any case, the learner must either use many samples from a generative model or many calls to  $\mathsf{O}_{\mathrm{exp}}$ . First, observe that since the transitions and rewards at layers  $1, \dots, H - 2$  are known and identical for all  $M_{b,f^{\star}} \in \mathcal{M}$ , querying the generative model on these states does not provide the learner with any information. Furthermore, in layers  $1, \dots, H - 2$ , every  $(s,a)$  pair has  $Q^{\star}(s,a) = 1$ , so querying  $\mathsf{O}_{\mathrm{exp}}$  on these  $(s,a)$  pairs also does not provide any information to the learner. Thus, we consider learners which query the generative model or the expert oracle at states in layers  $H - 1$  and  $H$ .
In order to identify  $b$ , the learner must identify which  $(s, a)$  pair at layer  $H - 1$  achieves reward of 1. They can do this either by (1) querying the generative model at a particular  $(s, a)$  pair and observing if  $r(s, a) = 1$  (or if the transition goes to  $s_{\perp}$ ); or (2) querying  $\mathsf{O}_{\mathrm{exp}}$  at a particular  $(s, a)$  pair and observing if  $Q^{\star}(s, a) = 0$  (which informs the learner that  $s_b = s$  and  $b[H - 1] = 1 - a$ ). In either case, the learner must expand  $\Omega(C)$  queries in total in order to identify  $b$ .
To learn  $f^{\star}$ , the learner must solve a supervised learning problem over  $S_H \backslash s_\perp$ . They can learn the identity of  $f^{\star}(s)$  by querying either the generative model or the expert oracle on  $S_H$ . Due to classical supervised learning lower bounds, learning  $f^{\star}$  requires  $\Omega(\mathrm{VC}(\mathcal{A}^{S_H})) = \Omega(2^H)$  queries.
J Technical Tools
Lemma 17 (Hoeffding's Inequality). Let  $Z_{1}, \dots, Z_{n}$  be independent bounded random variables with  $Z_{i} \in [a,b]$  for all  $i \in [n]$ . Then
$$
\mathbb {P} \left[ | \frac {1}{n} \sum_ {i = 1} ^ {n} Z _ {i} - \mathbb {E} [ Z _ {i} ] | \geq t \right] \leq 2 \exp \left(- \frac {2 n t ^ {2}}{(b - a) ^ {2}}\right).
$$
Lemma 18 (Multiplicative Chernoff Bound). Let  $Z_{1}, \dots, Z_{n}$  be i.i.d. random variables taking values in  $\{0,1\}$  with expectation  $\mu$ . Then for any  $\delta > 0$ ,