SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We highlight that MEX is not a Lagrangian duality of constrained optimization objectives within data-dependent level-sets [37, 20, 15]. In fact, MEX only needs to fix the parameter  $\eta$  across each episode. Thus  $\eta$  is independent of data and predetermined, which contrasts Lagrangian methods that involve an inner loop of optimization for the dual variables. We also remark that we can rewrite (3.1) as a joint optimization  $(f,\pi) = \operatorname{argsup}_{f\in \mathcal{H},\pi \in \Pi}V_{1,f}^{\pi}(x_1) - \eta \sum_{h = 1}^{H}L_h^{k - 1}(f)$ . When  $\eta$  tends to infinity, MEX can be reduced to vanilla Actor-Critic framework [41], where critic  $f$  minimizes estimation error and actor  $\pi$  conducts greedy policy following the critic  $f$ . In the following two parts,
we instantiate Algorithm 1 to model-based and mode-free hypotheses by specifying the loss function  $L_h^k(f)$ .
Model-free algorithm. For model-free hypothesis (Example 2.1), (3.1) becomes
$$
f ^ {k} = \operatorname * {a r g s u p} _ {f \in \mathcal {H}} \left\{\max  _ {a _ {1} \in \mathcal {A}} Q _ {1, f} \left(x _ {1}, a _ {1}\right) - \eta \cdot \sum_ {h = 1} ^ {H} L _ {h} ^ {k - 1} (f) \right\}. \tag {3.2}
$$
Regarding the choice of the loss function, for seek of theoretical analysis, to deal with MDPs with low Bellman eluder dimension [37] and MDPs of bilinear class [20], we assume the existence of certain function  $l$  which generalizes the notion of Bellman residual.
Assumption 3.1. Suppose the function  $l: \mathcal{H} \times \mathcal{H}_h \times \mathcal{H}_{h+1} \times (\mathcal{S} \times \mathcal{A} \times \mathbb{R} \times \mathcal{S}) \mapsto \mathbb{R}$  satisfies: i) (Generalized Bellman completeness) [89, 15] there exists an operator  $\mathcal{P}_h: \mathcal{H}_{h+1} \mapsto \mathcal{H}_h$  such that for any  $(f', f_h, f_{h+1}) \in \mathcal{H} \times \mathcal{H}_h \times \mathcal{H}_{h+1}$  and  $\mathcal{D}_h = (x_h, a_h, r_h, x_{h+1}) \in \mathcal{S} \times \mathcal{A} \times \mathbb{R} \times \mathcal{S}$ , it holds
$$
l _ {f ^ {\prime}} \left(\left(f _ {h}, f _ {h + 1}\right), \mathcal {D} _ {h}\right) - l _ {f ^ {\prime}} \left(\left(\mathcal {P} _ {h} f _ {h + 1}, f _ {h + 1}\right), \mathcal {D} _ {h}\right) = \mathbb {E} _ {x _ {h + 1} \sim \mathbb {P} _ {h} (\cdot | x _ {h}, a _ {h})} \left[ l _ {f ^ {\prime}} \left(\left(f _ {h}, f _ {h + 1}\right), \mathcal {D} _ {h}\right) \right],
$$
where we require that  $\mathcal{P}_h f_{h+1}^* = f_h^*$  and that  $\mathcal{P}_h f_{h+1} \in \mathcal{H}_h$  for any  $f_{h+1} \in \mathcal{H}_{h+1}$  and step  $h \in [H]$ ; ii) (Boundedness) it holds that  $\sup_{f' \in \mathcal{H}} \| l_{f'}((f_h, f_{h+1}), \mathcal{D}_h) \|_{\infty} \leq B_l$  for some constant  $B_l > 0$ .
We then set the loss function  $L_h^k$  as an empirical estimation of the generalized squared Bellman error  $\left|\mathbb{E}_{x_{h+1}}\sim \mathbb{P}_h(\cdot | x_h, a_h)\right|l_{fs}\left((f_h, f_{h+1}), \mathcal{D}_h^s\right)\|^2$ , given by
$$
L _ {h} ^ {k} (f) = \sum_ {s = 1} ^ {k} l _ {f ^ {s}} \left(\left(f _ {h}, f _ {h + 1}\right), \mathcal {D} _ {h} ^ {s}\right) ^ {2} - \inf  _ {f _ {h} ^ {\prime} \in \mathcal {H} _ {h}} \sum_ {s = 1} ^ {k} l _ {f ^ {s}} \left(\left(f _ {h} ^ {\prime}, f _ {h + 1}\right), \mathcal {D} _ {h} ^ {s}\right) ^ {2}. \tag {3.3}
$$
We remark that the subtracted infimum term in (3.3) is to handle the variance terms in the estimation to achieve a fast theoretical rate. Similar essential ideas are also adopted by [37, 78, 19, 38, 52, 2, 89, 15].
Model-based algorithm. For model-based hypothesis (Example 2.2), (3.1) becomes
$$
f ^ {k} = \underset {f \in \mathcal {H}} {\operatorname {a r g s u p}} \left\{\sup  _ {\pi \in \Pi} V _ {1, \mathbb {P} _ {f}} ^ {\pi} \left(x _ {1}\right) - \eta \sum_ {h = 1} ^ {H} L _ {h} ^ {k - 1} (f) \right\}, \tag {3.4}
$$
which is a joint optimization over the model  $\mathbb{P}_f$  and the policy  $\pi$ . In the model-based algorithm, we choose the loss function  $L_h^k$  as the negative log-likelihood loss, defined as
$$
L _ {h} ^ {k} (f) = - \sum_ {s = 1} ^ {k} \log \mathbb {P} _ {h, f} \left(x _ {h + 1} ^ {s} \mid x _ {h} ^ {s}, a _ {h} ^ {s}\right). \tag {3.5}
$$
4 Regret Analysis for MEX Framework
In this section, we establish a regret analysis for the MEX framework (Algorithm 1). We give a generic theoretical guarantee which holds for both model-free and model-based settings. We first present three key assumptions needed for sample-efficient learning with MEX. In Section 5, we specify the generic theory to specific examples of MDPs and hypothesis classes satisfying these assumptions.
Firstly, we assume that the hypothesis class  $\mathcal{H}$  is well-specified, containing the true hypothesis  $f^{*}$ .
Assumption 4.1 (Realizability). We assume that the true hypothesis  $f^{*} \in \mathcal{H}$
Then we need to make a structural assumption on the MDP to ensure sample-efficient online learning. Inspired by Zhong et al. [89], we require the MDP to have low Generalized Eluder Coefficient (GEC). A low GEC indicates that the agent can effectively mitigate out-of-sample prediction errors by minimizing in-sample errors derived from historical data. To introduce, we define a discrepancy function  $\ell_{f'}(f; \xi_h) : \mathcal{H} \times \mathcal{H} \times (\mathcal{S} \times \mathcal{A} \times \mathbb{R} \times \mathcal{S}) \mapsto \mathbb{R}$  which characterizes the error of a hypothesis  $f \in \mathcal{H}$  on data  $\xi_h = (x_h, a_h, r_h, x_{h+1})$ . Specific choices of  $\ell$  are given in Section 5 for concrete model-free and model-based examples.
Assumption 4.2 (Low Generalized Eluder Coefficient [89]). We assume that given an  $\epsilon >0$ , there exists  $d(\epsilon)\in \mathbb{R}_{+}$ , such that for any sequence of  $\{f^k\}_{k\in [K]}\subset \mathcal{H}$ ,  $\{\pi_{\exp}(f^k)\}_{k\in [K]}\subset \Pi$ ,
$$
\sum_ {k = 1} ^ {K} V _ {1, f ^ {k}} - V _ {1} ^ {\pi_ {f ^ {k}}} \leq \inf  _ {\mu > 0} \left\{\frac {\mu}{2} \sum_ {h = 1} ^ {H} \sum_ {k = 1} ^ {K} \sum_ {s = 1} ^ {k - 1} \mathbb {E} _ {\xi_ {h} \sim \pi_ {\exp} (f ^ {s})} [ \ell_ {f ^ {s}} (f ^ {k}; \xi_ {h}) ] + \frac {d (\epsilon)}{2 \mu} + \sqrt {d (\epsilon) H K} + \epsilon H K \right\}.
$$
We denote the smallest number  $d(\epsilon) \in \mathbb{R}_+$  satisfying this condition as  $d_{\mathrm{GEC}}(\epsilon)$ .
As is shown by Zhong et al. [89], the low-GEC MDP class covers almost all known theoretically tractable MDP instances, such as linear MDP [81, 39], linear mixture MDP [5, 57, 12], MDPs of low witness rank [67], MDPs of low Bellman eluder dimension [37], and MDPs of bilinear class [20].
Finally, we make a concentration-style assumption which characterizes how the loss function  $L_h^k$  is related to the expectation of the discrepancy function  $\mathbb{E}[\ell]$  appearing in the definition of GEC. For ease of presentation, we assume that  $\mathcal{H}$  is finite, i.e.,  $|\mathcal{H}| < \infty$ , but our result can be directly extended to an infinite  $\mathcal{H}$  using covering number arguments [72, 37, 49, 38].
Assumption 4.3 (Generalization). We assume that  $\mathcal{H}$  is finite, i.e.,  $|\mathcal{H}| < +\infty$ , and that with probability at least  $1 - \delta$ , for any episode  $k \in [K]$  and hypothesis  $f \in \mathcal{H}$ , it holds that
$$
\sum_ {h = 1} ^ {H} L _ {h} ^ {k - 1} \left(f ^ {*}\right) - L _ {h} ^ {k - 1} (f) \lesssim - \sum_ {h = 1} ^ {H} \sum_ {s = 1} ^ {k - 1} \mathbb {E} _ {\xi_ {h} \sim \pi_ {\exp} \left(f ^ {s}\right)} \left[ \ell_ {f ^ {s}} (f; \xi_ {h}) \right] + B \left(H \log (H K / \delta) + \log (| \mathcal {H} |)\right),
$$
where  $B = B_{l}^{2}$  for model-free hypothesis (Assumption 3.1) and  $B = 1$  for model-based hypothesis.
Such a concentration style inequality is well known in the literature of online RL with general function approximation and similar analysis is also adopted by [37, 15]. With Assumptions 4.1, 4.2, and 4.3, we can present our main result (see Appendix D.1 for a proof).
Theorem 4.4 (Online regret of MEX (Algorithm 1)). Under Assumptions 4.1, 4.2, and 4.3, by setting
$$
\eta = \sqrt {\frac {d _ {\mathrm {G E C}} (1 / \sqrt {H K})}{(H \log (H K / \delta) + \log (| \mathcal {H} |)) \cdot B \cdot K}},
$$
then the regret of Algorithm 1 after  $K$  episodes is upper bounded by
$$
\mathrm {R e g r e t} (K) \lesssim \sqrt {d _ {\mathrm {G E C}} (1 / \sqrt {H K}) \cdot (H \log (H K / \delta) + \log (| \mathcal {H} |)) \cdot B \cdot K},
$$
with probability at least  $1 - \delta$ . Here  $d_{\mathrm{GEC}}(\cdot)$  is defined in Assumption 4.2.
Theorem 4.4 shows that the regret of Algorithm 1 scales with the square root of the number of episodes  $K$  and the polynomials of horizon  $H$ , GEC  $d_{\mathrm{GEC}}(1 / \sqrt{K})$ , and log covering number  $\log \mathcal{N}(\mathcal{H}, 1 / K)$ . When the number of episodes  $K$  tends to infinity, the average regret vanishes, meaning that the output policy of Algorithm 1 achieves global optimality. Since the regret of Algorithm 1 is sublinear with respect to the number of episodes  $K$ , Algorithm 1 is proved to be sample-efficient. In Appendix C, we extend the algorithm framework and the analysis to the two-player zero-sum Markov game (MG) setting, for which we also extend the definition of GEC to two-player zero-sum MGs.
Besides, as we can see from Theorem 4.4 and its specifications in Section 5, MEX matches existing theoretical results in the literature of online RL with general function approximations [89, 67, 20, 37, 19, 2]. But in the meanwhile, MEX does not require explicitly solving a constrained optimization problem within data-dependent level-sets or performing a complex sampling procedure. This advantage makes MEX a principled approach with easier practical implementation. We conduct deep RL experiments for MEX in Section 6 to demonstrate its power in complicated online problems.
5 Examples of MEX Framework
In this section, we specify Algorithm 1 to model-based and model-free hypothesis classes for various examples of MDPs of low GEC (Assumption 4.2), including MDPs with low witness rank [67], MDPs with low Bellman eluder dimension [37], and MDPs of bilinear class [20]. For ease of presentation, we assume that  $|\mathcal{H}| < \infty$ , but our result can be directly extended to infinite  $\mathcal{H}$  using covering number arguments [72, 37, 49]. All the proofs of the propositions in this section are in Appendix E.
We note that another important step in specifying Theorem 4.4 to concrete hypothesis classes is to check Assumption 4.3 (supervised learning guarantee). It is worth highlighting that, in our analysis, for both model-free and model-based hypotheses, we provide supervised learning guarantees in a neat and unified manner, independent of specific MDP structures.
5.1 Model-free online RL in Markov Decision Processes
In this subsection, we specify Algorithm 1 for model-free hypothesis class  $\mathcal{H}$  (Example 2.1). For a model-free hypothesis class, we choose the discrepancy function  $\ell$  as, given  $\mathcal{D}_h = (x_h,a_h,r_h,x_{h + 1})$ ,
$$
\ell_ {f ^ {\prime}} (f; \mathcal {D} _ {h}) = \left(\mathbb {E} _ {x _ {h + 1} \sim \mathbb {P} _ {h} \left(\cdot | x _ {h}, a _ {h}\right)} \left[ l _ {f ^ {\prime}} \left(\left(f _ {h}, f _ {h + 1}\right), \mathcal {D} _ {h}\right) \right]\right) ^ {2}. \tag {5.1}
$$
where the function  $l:\mathcal{H}\times \mathcal{H}_h\times \mathcal{H}_{h + 1}\times (\mathcal{S}\times \mathcal{A}\times \mathbb{R}\times \mathcal{S})\mapsto \mathbb{R}$  satisfies Assumption 3.1. We specify the choice of  $l$  in concrete examples of MDPs later. In the following, we check and specify Assumptions 4.2 and 4.3 in Section 4 for model-free hypothesis classes.
Proposition 5.1 (Generalization: model-free RL). We assume that  $\mathcal{H}$  is finite, i.e.,  $|\mathcal{H}| < +\infty$ . Then under Assumption 3.1, with probability at least  $1 - \delta$ , for any  $k \in [K]$  and  $f \in \mathcal{H}$ , it holds that
$$
\sum_ {h = 1} ^ {H} L _ {h} ^ {k - 1} \left(f ^ {*}\right) - L _ {h} ^ {k - 1} (f) \lesssim - \sum_ {h = 1} ^ {H} \sum_ {s = 1} ^ {k - 1} \mathbb {E} _ {\xi_ {h} \sim \pi_ {\exp} \left(f ^ {s}\right)} \left[ \ell_ {f ^ {s}} (f; \xi_ {h}) \right] + B _ {l} ^ {2} \left(H \log (H K / \delta) + \log (| \mathcal {H} |)\right),
$$
where  $L$  and  $\ell$  are defined in (3.3) and (5.1) respectively. Here  $B_{l}$  is specified in Assumption 3.1.
Proposition 5.1 specifies Assumption 4.3 for model-free hypothesis classes. For Assumption 4.2, we need structural assumptions on the MDP. Given an MDP with generalized eluder dimension  $d_{\mathrm{GEC}}$  we have the following corollary of our main theoretical result (Theorem 4.4).
Corollary 5.2 (Online regret of MEX: model-free hypothesis). Given an MDP with generalized eluder coefficient  $d_{\mathrm{GEC}}(\cdot)$  and a finite model-free hypothesis class  $\mathcal{H}$  with  $f^{*} \in \mathcal{H}$ , under Assumption 3.1, setting
$$
\eta = \sqrt {\frac {d _ {\mathrm {G E C}} (1 / \sqrt {H K})}{(H \log (H K / \delta) + \log (| \mathcal {H} |)) \cdot B _ {l} ^ {2} \cdot K}}, \tag {5.2}
$$
$$
\operatorname {R e g r e t} (T) \lesssim B _ {l} \cdot \sqrt {d _ {\mathrm {G E C}} \left(1 / \sqrt {H K}\right) \cdot \left(H \log \left(H K / \delta\right) + \log \left(\left| \mathcal {H} \right|\right)\right) \cdot K}, \tag {5.3}
$$
with probability at least  $1 - \delta$ . Here  $B_{l}$  is specified in Assumption 3.1.
Corollary 5.2 can be directly specified to MDPs with low GEC, including MDPs with low Bellman eluder dimension [37] and MDPs of bilinear class [20]. See Appendix E.1 for a detailed discussion.
5.2 Model-based online RL in Markov Decision Processes
In this subsection, we specify Algorithm 1 for model-based hypothesis class  $\mathcal{H}$  (Example 2.2). For model-based hypothesis class, we choose the discrepancy function  $\ell$  in Assumption 4.2 and 4.3 as the hellinger distance. Given data  $\mathcal{D}_h = (x_h, a_h, r_h, x_{h+1})$ , we let
$$
\ell_ {f ^ {\prime}} (f; \mathcal {D} _ {h}) = D _ {\mathrm {H}} \left(\mathbb {P} _ {h, f} \left(\cdot \mid x _ {h}, a _ {h}\right) \| \mathbb {P} _ {h, f ^ {*}} \left(\cdot \mid x _ {h}, a _ {h}\right)\right), \tag {5.4}
$$
where  $D_{\mathrm{H}}(\cdot \| \cdot)$  denotes the Hellinger distance. We note that by (5.4), the discrepancy function  $\ell$  does not depend on the input  $f^{\prime}\in \mathcal{H}$ . In the following, we check and specify Assumption 4.2 and 4.3.
Proposition 5.3 (Generalization: model-based RL). We assume that  $\mathcal{H}$  is finite, i.e.,  $|\mathcal{H}| < +\infty$ . Then with probability at least  $1 - \delta$ , for any  $k \in [K]$ ,  $f \in \mathcal{H}$ , it holds that
$$
\sum_ {h = 1} ^ {H} L _ {h} ^ {k - 1} (f ^ {*}) - L _ {h} ^ {k - 1} (f) \lesssim - \sum_ {h = 1} ^ {H} \sum_ {s = 1} ^ {k - 1} \mathbb {E} _ {\xi_ {h} \sim \pi_ {\exp} (f ^ {s})} [ \ell_ {f ^ {s}} (f; \xi_ {h}) ] + H \log (H / \delta) + \log (| \mathcal {H} |),
$$
where  $L$  and  $\ell$  are defined in (3.5) and (5.4) respectively.
Proposition 5.3 specifies Assumption 4.3 for model-based hypothesis classes. For Assumption 4.2, we also need structural assumptions on the MDP. Given an MDP with generalized eluder dimension  $d_{\mathrm{GEC}}$ , we have the following corollary of our main theoretical result (Theorem 4.4).
Corollary 5.4 (Online regret of MEX: model-based hypothesis). Given an MDP with generalized eluder coefficient  $d_{\mathrm{GEC}}(\cdot)$  and a finite model-based hypothesis class  $\mathcal{H}$  with  $f^{*} \in \mathcal{H}$ , by setting
$$
\eta = \sqrt {\frac {d _ {\mathrm {G E C}} (1 / \sqrt {H K})}{(H \log (H / \delta) + \log (| \mathcal {H} |)) \cdot K}},
$$
then the regret of Algorithm 1 after  $K$  episodes is upper bounded by, with probability at least  $1 - \delta$ ,
$$
\operatorname {R e g r e t} (K) \lesssim \sqrt {d _ {\mathrm {G E C}} \left(1 / \sqrt {H K}\right) \cdot \left(H \log \left(H / \delta\right) + \log \left(\left| \mathcal {H} \right|\right)\right) \cdot K}, \tag {5.5}
$$
Corollary 5.4 can be directly specified to MDPs with low GEC, including MDPs with low witness rank [67]. We refer to Appendix E.2 for a detailed discussion.
In this section, we aim to answer the following two questions: (a) What are the practical approaches to implementing MEX in both model-based (MEX-MB) and model-free (MEX-MF) settings? (b) Can MEX handle challenging exploration tasks, especially in sparse reward scenarios?
Experimental setups. We evaluate the effectiveness of MEX by assessing its performance in both standard gym locomotion tasks and sparse reward locomotion and navigation tasks within the MuJoCo [71] environment. For sparse reward tasks, we select cheetah-vel, walker-vel, hopper-vel, ant-vel, and ant-goal adapted from Yu et al. [83], where the agent receives a reward only when it successfully attains the desired velocity or goal. To adapt to deep RL settings, we consider infinite-horizon  $\gamma$ -discounted MDPs and MEX variants. We report the results averaged over five random seeds. The full experimental settings are in Appendix H.
Implementation details. For the model-based variant MEX-MB, we use the following objective:
$$
\max  _ {\phi} \max  _ {\pi} \mathbb {E} _ {(x, a, r, x ^ {\prime}) \sim \beta} \left[ \log \mathbb {P} _ {\phi} \left(x ^ {\prime}, r \mid x, a\right) \right] + \eta^ {\prime} \cdot \mathbb {E} _ {x \sim \sigma} \left[ V _ {\mathbb {P} _ {\phi}} ^ {\pi} (x) \right], \tag {6.1}
$$
where we denote by  $\sigma$  the initial state distribution,  $\beta$  the replay buffer, and  $\eta^{\prime}$  corresponds to  $1 / \eta$  in the previous theory sections. We leverage the score function to obtain the model value gradient  $\nabla_{\phi}V_{\mathbb{P}_{\phi}}^{\pi}$  in a similar way to likelihood ratio policy gradient [70], with the gradient of action log-likelihood replaced by the gradient of state and reward log-likelihood in the model. Specifically,
$$
\nabla_ {\phi} \mathbb {E} _ {x \sim \sigma} \left[ V _ {\mathbb {P} _ {\phi}} ^ {\pi} (x) \right] = \mathbb {E} _ {\tau_ {\phi} ^ {\pi}} \left[ \left(r + \gamma V _ {\mathbb {P} _ {\phi}} ^ {\pi} \left(x ^ {\prime}\right) - Q _ {\mathbb {P} _ {\phi}} ^ {\pi} (x, a)\right) \cdot \nabla_ {\phi} \log \mathbb {P} _ {\phi} \left(x ^ {\prime}, r \mid x, a\right) \right], \tag {6.2}
$$
where  $\tau_{\phi}^{\pi}$  is the trajectory under policy  $\pi$  and transition  $\mathbb{P}_{\phi}$ , starting from  $\sigma$ . We refer the readers to previous works [62, 76] for a derivation of (6.2). The model  $\phi$  and policy  $\pi$  in (6.1) are updated iteratively in a Dyna [68] style, where model-free policy updates are performed on model-generated data. Particularly, we adopt SAC [30] to update the policy  $\pi$  and estimate the value  $Q_{\mathbb{P}_{\phi}}^{\pi}$  by performing temporal difference on the model data generated by  $\mathbb{P}_{\phi}$ . We also follow [62] to update the model using mini-batches from  $\beta$  and normalize the advantage  $r_h + \gamma V_{\mathbb{P}_{\phi}}^{\pi} - Q_{\mathbb{P}_{\phi}}^{\pi}$  within each mini-batch.
For the model-free variant MEX-MF, we observe from (3.2) that adding a maximization bias term to the standard TD error is sufficient for exploration. However, this may lead to instabilities as the bias term only involves the state-action value function of the current policy, and thus the policy may be ever-changing. To address this issue, we adopt a similar treatment as in CQL [42] by subtracting a baseline state-action value from random policy  $\mu = \mathrm{Unif}(\mathcal{A})$  and obtain the following objective:
$$
\max  _ {\theta} \max  _ {\pi} \mathbb {E} _ {\beta} \left[ \left(r + \gamma Q _ {\theta} \left(x ^ {\prime}, a ^ {\prime}\right) - Q _ {\theta} (x, a)\right) ^ {2} \right] + \eta^ {\prime} \cdot \mathbb {E} _ {\beta} \left[ \mathbb {E} _ {a \sim \pi} Q _ {\theta} (x, a) - \mathbb {E} _ {a \sim \mu} Q _ {\theta} (x, a) \right]. \tag {6.3}
$$
We update  $\theta$  and  $\pi$  in (6.3) iteratively in an actor-critic fashion. Due to space limits, we refer the readers to Appendix H for more implementation details of MEX-MF.
Results. We report the performance of MEX-MB and MEX-MF in Figures 1 and 2, respectively. We compare MEX-MB with MBPO [34], where our method differs from MBPO only in the inclusion of the value gradient in (6.2) during model updates. We find that MEX-MB offers an easy implementation with minimal computational overhead and yet remains highly effective across sparse and standard MuJoCo tasks. Notably, in the sparse reward settings, MEX-MB excels at achieving the goal velocity and outperforms MBPO by a stable margin. In standard gym tasks, MEX-MB showcases greater sample efficiency in challenging high-dimensional tasks with higher asymptotic returns.
We then compare MEX-MF with the model-free baseline TD3 [27]. We observe that TD3 fails in many sparse reward tasks, while MEX-MF significantly boosts the performance. In standard MuJoCo gym tasks, MEX-MF also steadily outperforms TD3 with faster convergence and higher final returns.
In this paper, we introduce a novel RL algorithm framework—Maximize to Explore (MEX)—aimed at striking a balance between exploration and exploitation in online learning scenarios. MEX is provably sample-efficient under general function approximations and is easy to implement. Theoretically, we prove that under mild structural assumptions (low generalized eluder coefficient (GEC)), MEX achieves
Figure 1: Model-based MEX-MB in sparse and standard MuJoCo locomotion tasks.
Figure 2: Model-free MEX-MF in sparse and standard MuJoCo locomotion tasks.
$\widetilde{\mathcal{O}} (\sqrt{K})$  -online regret for MDPs. We further extend the definition of GEC and MEX framework to two-player zero-sum Markov games (see Appendix C) and also prove the  $\widetilde{\mathcal{O}} (\sqrt{K})$  -online regret. In practice, we adapt MEX to deep RL methods in both model-based and model-free styles and apply them to sparse-reward MuJoCo environments, outperforming baselines significantly. We hope our work can shed light on future research of designing both statistically efficient and practically effective RL algorithms with powerful function approximation.
Zhaoran Wang acknowledges National Science Foundation (Awards 2048075, 2008827, 2015568, 1934931), Simons Institute (Theory of Reinforcement Learning), Amazon, J.P. Morgan, and Two Sigma for their supports.
[1] ABBASI-YADKORI, Y., PÁL, D. and SZEPESVÁRÍ, C. (2011). Improved algorithms for linear stochastic bandits. Advances in neural information processing systems 24.
[2] AGARWAL, A. and ZHANG, T. (2022). Model-based rl with optimistic posterior sampling: Structural conditions and sample complexity. arXiv preprint arXiv:2206.07659.
[3] AUBRET, A., MATIGNON, L. and HASSAS, S. (2019). A survey on intrinsic motivation in reinforcement learning.
[4] AUER, P., CESA-BIANCHI, N. and FISCHER, P. (2002). Finite-time analysis of the multiarmed bandit problem. Machine learning 47 235-256.