SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Let us first discuss the issue of computational efficiency. While there are a number of algorithms—all based on the principle of optimism in the face of uncertainty—that provide tight sample complexity guarantees for Low-Rank MDPs in reward-based [21, 24, 14] and reward-free [9, 43] settings, these algorithms involve intractable optimization problems, and cannot be implemented efficiently even when the learner has access to an optimization oracle for the representation class  $\Phi$  [11]. This intractability arises because these algorithms implement optimism via a "global" approach, in which the algorithm explores at each round by choosing the most optimistic value function in a certain version space of candidate value functions; optimizing over this version space is challenging,
as it involves satisfying non-convex constraints with a complicated dependence on the learned representation, and because the constraints are coupled globally across layers  $h \in [H]$ .
To avoid the intractability of global optimism, several works have restricted attention to a simpler model-based setting. Here, in addition to assuming that the feature maps  $(\phi_h^\star)_{h\in [H]}$  are realizable with respect to  $\Phi$ , one assumes access to a second feature class  $\Upsilon$  capable of modeling the mappings  $(\mu_h^\star)_{h\in [H]}$ ; this facilitates direct estimation of the transition probability kernel  $T_{h}(\cdot \mid x,a)$ . For the model-based setting, it is possible to efficiently implement certain "local" forms of optimism [40, 10, 48], as well as certain non-optimistic exploration techniques based on policy covers [1]. Unfortunately, model-based realizability is a restrictive assumption, and falls short of the model-free guarantees we aim for in this work; indeed, in general, one cannot hope to estimate the feature map  $\mu_{h + 1}^{\star}$  without sample complexity scaling with the number of states. $^{4}$
When one moves from model-based learning to model-free learning, representation learning becomes substantially more challenging—both for optimistic and non-optimistic approaches. Here, a key challenge is to develop representation learning procedures that are (1) efficient, yet (2) provide meaningful guarantees when the learned features are used downstream for exploration. To our knowledge, the only proposal for a representation learning procedure satisfying both desiderata comes from the work of Modi et al. [35], who introduced a promising “minimax” representation learning objective (described in detail in the sequel; cf. Algorithm 5), which Zhang et al. [49] subsequently showed to have encouraging empirical performance. However, to provide guarantees for this objective, both works place substantial additional restrictions on the low-rank factorization. In particular, Modi et al. [35] make the so-called latent variable assumption [1], which asserts that  $\phi_h^*$  and  $\mu_h^*$  are non-negative coordinate-wise, and Zhang et al. [49] further restrict to the Block MDP model [12, 32]. Non-negativity is a substantial restriction, as the best non-negative factorization can have exponentially large dimension relative to the best unrestricted factorization [1], even when reachability is assumed (cf. Appendix H.1). The source of this restriction is the problem of how to quantify how close a learned representation  $\hat{\phi}$  is to the ground truth  $\phi^*$ , which depends strongly on the downstream exploration strategy. In what follows, we show that with the right exploration strategy, this challenge can be ameliorated, but prior to our work it was unclear whether the minimax objective could lead to meaningful guarantees in the absence of non-negativity.
3.2 The SpanRL Algorithm
Our algorithm, SpanRL, is presented in Algorithm 1. The algorithm proceeds by building a policy cover layer-by-layer in an inductive fashion. For each layer  $h \geq 2$ , SpanRL uses a policy cover  $\Psi^{(h)}$  built at a previous iteration within a subroutine, RepLearn (Algorithm 5; deferred to Appendix B) to produce a feature map  $\hat{\phi}^{(h)}$  that approximates  $\phi_h^*$ . Using this feature map, the algorithm invokes a second subroutine, RobustSpanner (Algorithm 2 in Appendix B) to produce a collection of policies  $\pi_1, \ldots, \pi_d$  that act as a barycentric spanner for the feature map, ensuring maximal coverage in a certain sense; given these policies, a new policy cover for layer  $h + 2$  is formed via  $\Psi^{(h + 2)} = \{ \pi_i \circ_{h + 1} \pi_{\text{unif}} : i \in [d] \}$ . To invoke the RobustSpanner subroutine, SpanRL makes use of additional subroutines for policy optimization (PSDP; Algorithm 3 in Appendix B) and estimation of certain vector-valued functionals (EstVec; Algorithm 7 in Appendix B). We now describe each component of the algorithm in detail, highlighting how they allow us to overcome the challenges in the prequel.
Barycentric spanners. At the heart of SpanRL is the notion of a barycentric spanner [6] as an efficient basis for exploration. We begin by defining a barycentric spanner for an abstract set.
Definition 3.1 (Awerbuch and Kleinberg [6]). Given a set  $\mathcal{W} \subset \mathbb{R}^d$  such that  $\operatorname{span}(\mathcal{W}) = \mathbb{R}^d$ , we say that a set  $\{w_1, \ldots, w_d\} \subseteq \mathcal{W}$  is a  $(C, \varepsilon)$ -approximate barycentric spanner for  $\mathcal{W}$  if for every  $w \in \mathcal{W}$ , there exist  $\beta_1, \ldots, \beta_d \in [-C, C]$  such that  $\|w - \sum_{i=1}^{d} \beta_i w_i\| \leq \varepsilon$ .<sup>5</sup>
The utility of barycentric spanners for reward-free exploration is highlighted in the following lemma.
Lemma 3.1. Suppose that Assumption 2.1 holds. If  $\Psi \subseteq \Pi_{\mathsf{M}}$  is a collection of policies such that  $\{\mathbb{E}^{\pi}\left[\phi_h^\star (\pmb{x}_h,\pmb{a}_h)\right] \mid \pi \in \Psi\} \subseteq \mathbb{R}^d$  is a  $(C,\varepsilon)$ -approximate barycentric spanner for  $\mathcal{W}_h \coloneqq \{\mathbb{E}^{\pi}\left[\phi_h^\star (\pmb{x}_h,\pmb{a}_h)\right] \mid \pi \in \Pi_{\mathsf{M}}\}$  with  $\varepsilon \leq \frac{\eta}{2}$ , then  $\Psi$  is an  $\alpha$ -policy cover for layer  $h + 1$  with  $\alpha = (2dC)^{-1}$ .
For example, in the special case of the Block MDP setting [12, 32], model-based realizability entails modeling a certain emission process, which is not required by model-free approaches.
Note that our definition is a slight generalization of [6, Definition 2.1]; the latter is recovered with  $\varepsilon = 0$ .
Lemma 3.1, proven in Appendix F.6, shows that to compute a policy cover for layer  $h + 1$ , it suffices to find a barycentric spanner for the set  $\mathcal{W}_h \coloneqq \{\mathbb{E}^\pi [\phi_h^\star (\pmb{x}_h,\pmb{a}_h)]\mid \pi \in \Pi_{\mathsf{M}}\} \subseteq \mathbb{R}^d$ . Of course, even if  $\phi_h^\star$  is known, this observation is only useful if we can compute a spanner without explicitly enumerating over the set  $\Pi_{\mathsf{M}}$ , since our goal is to develop an efficient algorithm. In what follows, we will show:
1. Using, RobustSpanner, a novel adaptation of the classical algorithm of Awerbuch and Kleinberg [6], it holds that for any  $\phi \in \Phi$ , spanner computation for the set  $\{\mathbb{E}^{\pi}[\phi(\boldsymbol{x}_h, \boldsymbol{a}_h)] \mid \pi \in \Pi_{\mathbb{M}}\}$  can be performed efficiently whenever, for any  $\theta \in \mathcal{B}(1)$ , one can (approximately) solve linear optimization problems of the form
$$
\underset {\pi \in \Pi_ {M}} {\arg \max } \mathbb {E} ^ {\pi} \left[ \theta^ {\top} \phi \left(\boldsymbol {x} _ {h}, \boldsymbol {a} _ {h}\right) \right]. \tag {5}
$$
2. Given access to policy covers  $\Psi^{(1:h)}$  for layers 1 to  $h$ , one can efficiently solve the optimization problem in (5) by appealing to the PSDP algorithm for policy optimization (Algorithm 3).
To handle the fact that  $\phi_h^\star$  is unknown, Algorithm 1 computes policies  $\pi_{1:d}$  that induce a barycentric spanner for the set  $\{\mathbb{E}^{\pi}[\hat{\phi}^{(h)}(\pmb{x}_h,\pmb{a}_h)]\mid \pi \in \Pi_{\mathbb{M}}\}$ , where  $\hat{\phi}^{(h)}\in \Phi$  is a learned feature map. In what follows, we first give a detailed explanation of the two above points, before showing how to complete the argument by learning a feature map through representation learning.
Algorithm 1 SpanRL: Volumetric Exploration and Representation Learning via Barycentric Spanner
Require: Feature class  $\Phi$  and parameters  $\varepsilon, \mathfrak{c} > 0$  and  $\delta \in (0,1)$ .  
1: Set  $\Psi^{(1)} = \emptyset$ ,  $\Psi^{(2)} = \{\pi_{\text{unif}}\}$ .  
2: Set  $n_{\text{RepLearn}} = \mathfrak{c} \cdot \varepsilon^{-2} A^2 d \log(|\Phi| / \delta)$  and  $n_{\text{EstVec}} = \mathfrak{c} \cdot \varepsilon^{-2} \log(1 / \delta)$ .  
3: Set  $n_{\text{PSDP}} = \mathfrak{c} \cdot \varepsilon^{-2} A^2 d^3 H^2 \cdot (d + \log(|\Phi| / \delta))$ .  
4: Define  $\mathcal{F} := \{f : x \mapsto \max_{a \in \mathcal{A}} \theta^\top \phi(x, a) | \theta \in \mathcal{B}(1), \phi \in \Phi\}$ .  
5: Define  $\mathcal{G} = \{g : (x, a) \mapsto \phi(x, a)^\top w | \phi \in \Phi, w \in \mathcal{B}(2\sqrt{d})\}$ .  
6: for  $h = 1, \ldots, H - 2$  do  
/* Learn feature representation for layer  $h$ . */  
7: Set  $\phi^{(h)} = \text{RepLearn}(h, \mathcal{F}, \Phi, P^{(h)}, n_{\text{RepLearn}})$ , with  $P^{(h)} = \text{unif}(\Psi^{(h)})$ . // Algorithm 6.  
/* Computing an approximate spanner using learned features. */  
8: For  $\theta \in \mathbb{R}^d$  and  $(x, a) \in \mathcal{X} \times \mathcal{A}$ , define  
[ r_t(x, a; \theta) := \left\{ \begin{array}{ll} \phi^{(h)}(x, a)^\top \theta, & \text{for } t = h, \\ 0, & \text{otherwise.} \end{array} \right. ]  
9: For each  $t \in [h]$ , set  $\mathcal{G}_t = \mathcal{G}$  and  $P^{(t)} = \text{unif}(\Psi^{(t)})$ .  
10: For  $\theta \in \mathbb{R}^d$ , define LinOpt( $\theta$ ) = PSDP(h, r_1:h, :, \theta), G_{1:h}, P^{(1:h)}, n_{\text{PSDP}}). // Algorithm 3.  
11: For  $\pi \in \Pi_M$ , define LinEst( $\pi$ ) = EstVec(h,  $\phi^{(h)}$ ,  $\pi$ , nEstVec). // Algorithm 7.  
12: Set  $(\pi_1, \dots, \pi_d) = \text{RobustSpanner(LinOpt(\cdot), LinEst(\cdot), 2, \varepsilon)}$ . // Algorithm 2.  
13: Set  $\Psi^{(h+2)} = \{\pi_i o_{h+1} \pi_{\text{unif}} : i \in [d]\}$ .  
14: Return: Policy cover  $\Psi^{(1:H)}$ .
Barycentric spanner computation via approximate linear optimization. To describe spanner computation in SpanRL, we take a brief detour and consider an abstract approach to barycentric spanner computation, which generalizes our problem. Suppose that we wish to compute a spanner for an implicitly specified set  $\mathcal{W} = \{w^z\}_{z\in \mathcal{Z}}\subseteq \mathbb{R}^d$  indexed by an abstract set  $\mathcal{Z}$ . The set  $\mathcal{Z}$  (which will be set to  $\Pi_{\mathsf{M}}$  when we return to RL) may be exponentially large and cannot be efficiently enumerated. In addition, given  $z\in \mathcal{Z}$ , we cannot explicitly compute  $w^{z}$ , and have to settle for a noisy approximation.
To allow for efficient spanner computation, we assume access to two oracles for the set  $\mathcal{W}$ , a linear optimization oracle LinOpt:  $\mathcal{B}(1) \to \mathcal{Z}$  and an index-to-vector oracle LinEst:  $\mathcal{Z} \to \mathbb{R}^d$ . We assume that for some  $\varepsilon > 0$ :
<sup>6</sup>While barycentric spanners have been used in a number of recent works on sample-efficient RL [19, 20], the motivation for their use within our algorithm and analysis are quite different; see Appendix A.
1. For all  $\theta \in \mathbb{R}^d$  with  $\| \theta \| = 1$ , the output  $\hat{z}_{\theta} \coloneqq \operatorname{LinOpt}(\theta)$  satisfies  $\theta^{\top}w^{\hat{z}_{\theta}} \geq \sup_{z \in \mathcal{Z}} \theta^{\top}w^{z} - \varepsilon$ .
2. For all  $z \in \mathcal{Z}$ , the output  $\hat{w}_z \coloneqq \operatorname{LinEst}(z)$  satisfies  $\| \hat{w}_z - w^z \| \leq \varepsilon$ .
The RobustSpanner algorithm (Algorithm 2) computes a  $(C, \varepsilon)$ -approximate spanner for  $\mathcal{W}$  using  $O(d\log(d / \varepsilon))$  total calls to LinOpt and LinEst. RobustSpanner is an error-tolerant variant of the classical spanner computation algorithm of Awerbuch and Kleinberg [6], which was originally introduced and analyzed for spanner computation with an exact linear optimization oracle. Tolerance to approximation errors in the linear optimization oracle is critical for our application to RL, where additive errors will arise in sampling trajectories, as well as estimating the feature maps  $(\phi_h^*)_{h \in [H]}$ . RobustSpanner achieves error tolerance by perturbing the vectors returned by LinOpt  $(\theta)$  in the direction of  $\theta$ , which amounts to running the classical algorithm on an  $\varepsilon$ -fattening of  $\mathcal{W}$ , and is necessary in order to ensure that the approximation error of LinOpt does not swamp the signal in directions  $\theta$  in which  $\mathcal{W}$  is too "skinny." This technique may be of independent interest; see Appendix C for additional details and formal guarantees.
Representation learning. Ideally, we would like to use RobustSpanner to construct a barycentric spanner for the set  $\{\mathbb{E}^{\pi}[\phi_h^*(\boldsymbol{x}_h, \boldsymbol{a}_h)] \mid \pi \in \Pi_{\mathsf{M}}\}$  with  $\mathcal{Z} = \Pi_{\mathsf{M}}$ . Because we do not have access to  $\phi_h^*$ , we instead apply RobustSpanner with  $\mathcal{W} := \{\mathbb{E}^{\pi}[\hat{\phi}^{(h)}(\boldsymbol{x}_h, \boldsymbol{a}_h)] \mid \pi \in \Pi_{\mathsf{M}}\}$ , where  $\hat{\phi}^{(h)}$  is a learned representation. We now describe how the feature map  $\hat{\phi}^{(h)}$  is learned, then show how to use these learned features to efficiently implement the oracles LinOpt( $\cdot$ ) and LinEst( $\cdot$ ).
To learn a representation for layer  $h$ , we use the RepLearn algorithm (Algorithm 5), which was originally introduced in Modi et al. [35], Zhang et al. [49]. The algorithm gathers a collection of triples  $(\pmb{x}_h, \pmb{a}_h, \pmb{x}_{h+1})$  by rolling in to  $\pmb{x}_h$  with a policy sampled uniformly from the policy cover  $\Psi^{(h)}$  and selecting  $\pmb{a}_h$  uniformly at random. Using this dataset, the algorithm solves a sequence of adversarial training sub-problems (Line 9 of Algorithm 5) which involve the feature class  $\Phi$  and an auxiliary discriminator class  $\mathcal{F}: \mathcal{X} \to \mathbb{R}$ . As we discuss in detail in the sequel, these sub-problems, described in (7), are amenable to standard gradient-based training methods. The sub-problems are designed to approximate the following "idealized" max-min-max representation learning objective:
$$
\hat {\phi} ^ {(h)} \in \underset {\phi \in \Phi} {\arg \min } \sup  _ {f \in \mathcal {F}} \inf  _ {w} \mathbb {E} ^ {\operatorname {u n i f} (\Psi^ {(h)}) _ {\circ_ {h}} \pi_ {\operatorname {u n i f}}} \left[ \left(\phi \left(\boldsymbol {x} _ {h}, \boldsymbol {a} _ {h}\right) ^ {\top} w - \mathbb {E} \left[ f \left(\boldsymbol {x} _ {h + 1}\right) \mid \boldsymbol {x} _ {h}, \boldsymbol {a} _ {h} \right]\right) ^ {2} \right]. \tag {6}
$$
The intuition for this objective comes from the fact that in a Low-Rank MDP, for any function  $f: \mathcal{X} \to \mathbb{R}$ , the quantity  $\mathbb{E}[f(\boldsymbol{x}_{h+1}) \mid \boldsymbol{x}_h = x, \boldsymbol{a}_h = a]$  is linear in  $\phi_h^\star(x, a)$ . Thus, if  $\mathcal{F}$  is sufficiently expressive, we may hope that  $\hat{\phi}^{(h)}$  and  $\phi^\star$  are close. We adopt the simple discriminator class  $\mathcal{F} = \{x \mapsto \max_{a \in \mathcal{A}} \theta^\top \phi(x, a) \mid \theta \in \mathcal{B}(1), \phi \in \Phi\}$ . We show that solving (6) with this choice for  $\mathcal{F}$ , which is simpler than that in Modi et al. [35], Zhang et al. [49], yields an approximation guarantee for  $\hat{\phi}^{(h)}$  that is suitable for downstream use in spanner computation for general Low-Rank MDPs.
Remark 3.1 (Improved analysis of RepLearn). To facilitate an analysis of SpanRL that does not require reachability assumptions, we use slightly different parameter values for RepLearn than in Modi et al. [35], Zhang et al. [49], and provide a tighter sample complexity bound (Theorem E.1) which may be of independent interest.
In more detail, prior work shows that the RepLearn algorithm solves a variant of (6) with  $w \in \mathcal{B}(d^{1/2} \cdot \mathrm{poly}(\varepsilon^{-1}))$ , where  $\varepsilon > 0$  is the desired bound on mean-squared error. Due to the polynomial dependence on  $\varepsilon^{-1}$ , such a guarantee would lead to vacuous guarantees when invoked within our analysis of SpanRL. Our improved analysis of RepLearn, which is based on a determinantal potential argument, shows that  $w \in \mathcal{B}(\mathrm{poly}(d))$  suffices. A secondary benefit of our improved bound is a faster rate with respect to the number of trajectories.
Putting everything together. Having learned  $\hat{\phi}^{(h)}$  using RepLearn, in SpanRL we apply RobustSpanner with  $\mathcal{W} \coloneqq \{\mathbb{E}^{\pi}\big[\hat{\phi}^{(h)}(\boldsymbol{x}_h,\boldsymbol{a}_h)\big]\mid \pi \in \Pi_{\mathbb{M}}\}$ ,  $\mathcal{Z} = \Pi_{\mathbb{M}}$ , and  $C = 2$ ; that is, we plug-in the learned representation  $\hat{\phi}^{(h)}$  for the true representation  $\phi_h^*$ . With this choice, implementing LinOpt entails (approximately) solving  $\arg \max_{\pi \in \Pi_{\mathbb{M}}} \mathbb{E}^{\pi}\big[\theta^{\top}\hat{\phi}^{(h)}(\boldsymbol{x}_h,\boldsymbol{a}_h)\big]$  for a given  $\theta \in \mathcal{B}(1)$ , and implementing the LinEst oracle entails estimating  $\mathbb{E}^{\pi}\big[\hat{\phi}^{(h)}(\boldsymbol{x}_h,\boldsymbol{a}_h)\big]$  for a given  $\pi \in \Pi_{\mathbb{M}}$ . We instantiate
Though the policies produced by the algorithm may not necessarily induce a spanner for  $\mathcal{W}_h = \{\mathbb{E}^\pi [\phi_h^* (\pmb{x}_h,\pmb {a}_h)]\mid \pi \in \Pi_{\mathsf{M}}\}$  (this would require "point-wise" representation learning guarantees, which we do not have), our analysis shows that they still suffice to build a policy cover for layer  $h + 2$
$\operatorname{LinEst}(\pi)$  as the Monte Carlo algorithm EstVec (Algorithm 7), which simply samples trajectories according to  $\pi$  and returns the sample average of  $\hat{\phi}^{(h)}(\boldsymbol{x}_h, \boldsymbol{a}_h)$ . To implement LinOpt( $\theta$ ), we appeal to PSDP (Algorithm 3). PSDP, given an arbitrary reward function  $r_{1:h}: \mathcal{X} \times \mathcal{A} \to \mathbb{R}$  and a function class  $\mathcal{G} \subseteq \{g: \mathcal{X} \times \mathcal{A} \to \mathbb{R}\}$  capable of realizing all possible value functions induced by these rewards, can use the policy covers  $\Psi^{(1:h)}$  to efficiently compute a policy  $\hat{\pi} = \operatorname{PSDP}(h, r_{1:h}, \mathcal{G}, \Psi^{(1:h)}, n)$  that approximately solves  $\arg \max_{\pi \in \Pi_M} \mathbb{E}^\pi[\sum_{t=1}^h r_t(\boldsymbol{x}_t, \boldsymbol{a}_t)]$ , and does so using polynomially many episodes; see Appendix D for details and formal guarantees. Thus, implementing LinOpt( $\theta$ ) is as simple as invoking PSDP with the rewards
$$
r _ {t} (x, a; \theta) := \left\{ \begin{array}{l l} \hat {\phi} ^ {(h)} (x, a) ^ {\top} \theta , & \text {f o r} t = h, \\ 0, & \text {o t h e r w i s e}. \end{array} \right.
$$
With this, we have all the ingredients needed for spanner computation, and the algorithm is complete.
3.3 Main Guarantee for SpanRL
The following result is the main sample complexity guarantee for SpanRL (Algorithm 1).
Theorem 3.2 (Main theorem for SpanRL). Let  $\delta \in (0,1)$  be given, and suppose that realizability holds (Assumption 2.2) and that reachability (Assumption 2.1) is satisfied with parameter  $\eta > 0$ . If  $\varepsilon = \frac{\eta}{36d^{5/2}}$  and  $\mathfrak{c} = \mathrm{polylog}(A, H, d, \log(|\Phi| / \delta))$  is sufficiently large, then the policies  $\Psi^{(1:H)}$  produced by SpanRL  $(\Phi, \varepsilon, \mathfrak{c}, \delta)$  are a  $(\frac{1}{4Ad}, 0)$ -policy cover with probability at least  $1 - \delta$ . The total number of episodes used by SpanRL is at most:
$$
\widetilde {O} \left(A ^ {4} d ^ {9} H ^ {4} (d + \log (| \Phi | / \delta)) \cdot 1 / \eta^ {2}\right).
$$
Theorem 3.2 is the first provable, model-free sample complexity guarantee for general Low-Rank MDPs that is attained by an efficient algorithm. Prior to our work, all efficient model-free algorithms required non-negative features (latent variable structure) or stronger assumptions [35, 49], even in the presence of similar reachability assumptions; see Table 1.
Remark 3.2 (On the reachability assumption). While the reachability assumption is shared by the best prior efficient algorithms [35], which require non-negativity in addition to this assumption, it is natural to ask to what extent reachability restricts the generality of the Low-Rank MDP model. In Appendix H, we show that even when reachability holds, the embedding dimension in our model can be exponentially smaller than the best embedding dimension for the best non-negative (latent variable) embedding [35]. Hence, our results are meaningfully more general than prior work.
While our guarantee is polynomial in all relevant problem parameters, improving the dependence further (e.g., to match that of the best known inefficient algorithms) is an interesting direction for future research, as is removing the reachability assumption.
Application to reward-based RL. By using the policy cover produced by SpanRL within PSDP (Algorithm 3), we can optimize any downstream reward function to error  $\varepsilon$  using poly  $(d, A, H, \log |\Phi|) \cdot 1 / \varepsilon^2$  episodes. See Appendix G for details.
Efficiency and practicality. We observe that SpanRL is simple and practical. Defining  $\mathcal{L}_{\mathcal{D}}(\phi, w, f) \coloneqq \sum_{(x, a, x') \in \mathcal{D}} (\phi(x, a)^{\top} w - f(x'))^2 + \lambda \|w\|^2$ , where  $\mathcal{D}$  is a dataset consisting of  $(x_h, a_h, r_h, x_{h+1})$  tuples, the algorithm is provably efficient whenever the adversarial objective
$$
f ^ {(t)} \in \underset {f \in \mathcal {F}} {\arg \max } \underset {\phi \in \Phi} {\max } \left\{\underset {w} {\min } \mathcal {L} _ {\mathcal {D}} \left(\phi^ {(t)}, w, f\right) - \underset {\tilde {w}} {\min } \mathcal {L} _ {\mathcal {D}} \left(\tilde {\phi}, \tilde {w}, f\right) \right\}, \tag {7}
$$
in Line 9 of RepLearn (Algorithm 5), can be implemented efficiently (note that by the definition of  $\mathcal{L}_{\mathcal{D}}$ , the "inner" minima over  $w$  and  $\tilde{w}$  in (7) can be solved in closed form). This objective was also assumed to be efficiently solvable in Modi et al. [35], Zhang et al. [49] and was empirically shown to be practical in [49]; note that the objective is amenable to standard gradient-based optimization techniques, and that  $\mathcal{F}$  can be over-parameterized. While a detailed experimental evaluation is outside of the scope of this paper, we are optimistic about the empirical performance of the algorithm in light of the encouraging results based on the same objective in Zhang et al. [49]
This is the main place where the analysis uses the inductive hypothesis that  $\Psi^{(1:h)}$  are policy covers.
Outside of representation learning, the only overhead in SpanRL is the RobustSpanner subroutine, which has polynomial runtime. Indeed, RobustSpanner requires only polynomially many calls to the linear optimization oracle, instantiated as PSDP, which is efficient whenever standard least-squares regression problems based on the class  $\Phi$  can be solved efficiently, analogous to [33, 31].