SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Zhihan Liu*
Miao Lu $^{2*}$
Wei Xiong $^{3*}$
Hao Hu
Shenao Zhang
Sirui Zheng
Zhuoran Yang
Zhaoran Wang
$^{1}$ Northwestern University  $^{2}$ Stanford University  $^{3}$ University of Illinois Urbana-Champaign
4Peking University
5Tsinghua University
$^{6}$ Yale University
{zhihanliu2027,shenaozhang2028,siruizheng2025} $@$ u.northwestern.edu
miaolu@stanford.edu,wx13@illinois.edu,hanzhong@stu.pku.edu.cn
huh22@mails.tsinghua.edu.cn, zhuoran.yang@yale.edu, zhaoranwang@gmail.com
In reinforcement learning (RL), balancing exploration and exploitation is crucial for achieving an optimal policy in a sample-efficient way. To this end, existing sample-efficient algorithms typically consist of three components: estimation, planning, and exploration. However, to cope with general function approximators, most of them involve impractical algorithmic components to incentivize exploration, such as data-dependent level-set constraints or complicated sampling procedures. To address this challenge, we propose an easy-to-implement RL framework called Maximize to Explore (MEX), which only needs to optimize unconstrainedly a single objective that integrates the estimation and planning components while balancing exploration and exploitation automatically. Theoretically, we prove that the MEX achieves a sublinear regret with general function approximators and is extendable to the zero-sum Markov game setting. Meanwhile, we adapt deep RL baselines to design practical versions of MEX in both the model-based and model-free settings, which outperform baselines in various MuJoCo environments with sparse reward by a stable margin. Compared with existing sample-efficient algorithms with general function approximators, MEX achieves similar sample efficiency while also enjoying a lower computational cost and is more compatible with modern deep RL methods. Our codes are available at https://github.com/agentification/MEX.
The crux of online reinforcement learning (online RL) lies in maintaining a balance between exploiting the current knowledge of the agent about the environment and exploring unfamiliar areas [69]. To fulfill this, agents in existing sample-efficient RL algorithms predominantly undertake three tasks: i) estimate a hypothesis using historical data to encapsulate their understanding of the environment; ii) perform planning based on the estimated hypothesis to exploit their current knowledge; iii) further explore the unknown environment via carefully designed exploration strategies.
There exists a long line of research on integrating the aforementioned three components harmoniously, to find optimal policies in a sample-efficient manner. From theoretical perspectives, existing theories aim to minimize the notion of online external regret which measures the cumulative suboptimality gap of the policies learned during online learning. It is well studied that one can design both statistically and computationally efficient algorithms (e.g., upper confidence bound (UCB), [6, 39, 12, 91]) with sublinear online regret for tabular and linear Markov decision processes (MDPs). But when it
comes to MDPs with general function approximations, most of them involve impractical algorithmic components to incentivize exploration. Usually, to cope with general function approximations, agents need to solve constrained optimization problems within data-dependent level-sets [37, 20], or sample from complicated posterior distributions over the space of hypotheses [19, 2, 89], both of which pose considerable challenges for implementation. From a practical perspective, a prevalent approach in deep RL for balancing exploration and exploitation is to use an ensemble of neural networks [75, 59, 14, 53, 44, 18, 45], which serves as an empirical approximation of the UCB method. However, such an ensemble method suffers from high computational cost and lacks theoretical guarantee when the underlying MDP is neither linear nor tabular. As for other deep RL algorithms for exploration [29, 3, 11, 9, 17, 65], such as curiosity-driven method [60], it also remains unknown in theory whether they are provably sample-efficient in the context of general function approximations.
Hence, in this paper, we are aimed at tackling these issues and answering the following question:
Under general function approximation, can we design a sample-efficient and easy-to-implement RL framework to trade off between exploration and exploitation?
To this end, we propose an easy-to-implement RL framework, Maximize to Explore (MEX), as an affirmative answer to the question. To strike a balance between exploration and exploitation, MEX propose to maximize a weighted sum of two objectives: (i) the optimal expected total return associated with a given hypothesis and (ii) the negative estimation error of that hypothesis. Consequently, MEX naturally combines planning and estimation components in just a single objective. By choosing the hypothesis that maximizes the weighted sum and executing the optimal policy with respect to the chosen hypothesis, MEX automatically balances between exploration and exploitation.
We highlight that the objective of MEX is not obtained by taking the Lagrange dual of the constrained optimization objective within data-dependent level-sets [37, 20, 15]. This is because the coefficient of the weighted sum, which remains fixed, is data-independent and predetermined for all episodes. Contrary to Lagrangian methods, MEX does not necessitate an inner loop of optimization for dual variables, thereby circumventing the complications associated with minimax optimization. As a maximization-only framework, MEX is friendly to implementations with neural networks and does not rely on sampling or ensemble.
In the theory part, we prove that MEX achieves a sublinear regret  $\widetilde{\mathcal{O}}(\mathrm{Poly}(H) \cdot d_{\mathrm{GEC}}^{1/2}(1/\sqrt{HK}) \cdot K^{1/2})$  under mild assumptions and is thus sample-efficient, where  $K$  is the number of episodes and  $H$  is the horizon length. Here  $d_{\mathrm{GEC}}(\cdot)$  is the Generalized Eluder Coefficient (GEC) [89] that characterizes the complexity of learning the underlying MDP under general function approximations. Because the class of low-GE CDPs includes almost all known theoretically tractable MDP instances, our proved result can be tailored to a multitude of specific settings with either a model-free or a model-based hypothesis, such as MDPs with low Bellman eluder dimension [37], MDPs of bilinear class [20], and MDPs with low witness rank [67]. Besides, thanks to the flexibility of the MEX framework, we further extend it to online RL in two-player zero-sum Markov games (MGs), for which we further extend the definition of GEC to two-player zero-sum MGs and establish the sample efficiency with general function approximations. Moving beyond theory and into practice, we adapt famous RL baselines TD3 [27] and MBPO [34] to design practical versions of MEX in model-free and model-based fashion, respectively. On various MuJoCo environments [71] with sparse rewards, experimental results show that MEX outperforms baselines steadily and significantly. Compared with other deep RL algorithms, MEX has low computational overhead and straightforward implementation while maintaining a theoretical guarantee.
Contributions. We conclude our contributions from three perspectives.
1. We propose an easy-to-implement RL algorithm framework MEX that unconstrainedly maximizes a single objective to fuse estimation and planning, automatically trading off between exploration and exploitation. Under mild structural assumptions, we prove that MEX achieves a sublinear regret  $\widetilde{\mathcal{O}}(\mathrm{Poly}(H) \cdot d_{\mathrm{GEC}}^{1/2}(1/\sqrt{HK})K^{1/2})$  with general function approximators, and thus is sample-efficient. Here  $d_{\mathrm{GEC}}(\cdot)$  is the generalized Eluder Coefficient (GEC) of the underlying MDP.
2. We instantiate the generic MEX framework to several model-based and model-free examples and establish corresponding theoretical results. Further, we extend the MEX framework to two-player zero-sum MGs and also prove the sample efficiency with an extended definition of GEC.
3. We design practical implementations of MEX for MDPs in both model-based and model-free styles. Experiments on various MuJoCo environments with sparse reward demonstrate the effectiveness of our proposed MEX framework.
Sample-efficient RL with function approximation. The success of DRL methods has motivated a line of work focused on function approximation scenarios. This work originated in the linear case [74, 81, 12, 39, 84, 5, 82, 57, 91, 90] and is later extended to general function approximation. Wang et al. [73] first study the general function approximation using the notion of eluder dimension [63], which takes the linear MDP [39] as a special case but with inferior results. Zanette et al. [85] consider a different type of framework based on Bellman completeness, which assumes that the class used for approximating the optimal Q-functions is closed in terms of the Bellman operator and improves the results for linear MDP. After this, Jin et al. [37] consider the eluder dimension of the class of Bellman residual associated with the RL problems, which captures more solvable problems. Another line of works focuses on the low-rank structures of the problems, where Jiang et al. [35] propose the Bellman rank for model-free RL and Sun et al. [67] propose the witness rank for model-based RL. Following these two works, Du et al. [20] propose the bilinear class, which contains more MDP models with low-rank structures [6, 67, 39, 57, 12, 91] by allowing a flexible choice of discrepancy function class. However, it is known that neither BE nor bilinear class captures each other. Dann et al. [19] first consider eluder-coefficient-type complexity measure on the Q-type model-free RL. It was later extended by Zhong et al. [89] to cover all the above-known solvable problems in both model-free and model-based manners. Foster et al. [25, 23] study another notion of complexity measure, the decision-estimation coefficient (DEC), which also unifies the Bellman eluder dimension and bilinear class. The DEC framework is appealing due to the matching lower bound in some decision-making problems, where all other complexity measures do not have. However, due to the presence of a minimax subroutine in its definition, they require a much more complicated minimax optimization oracle and cannot apply to the classical optimism-based or sampling-based methods. Chen et al. [13], Foster et al. [24] extend the vanilla DEC (to the model-free case) by incorporating an optimistic modification, which was originally referred to as the feel-good modification in Zhang [87]. Chen et al. [15] study Admissible Bellman Characterization (ABC) class to generalize BE. They also extend the GOLF algorithm and Bellman completeness in model-free RL to the model-based case by considering more general (vector-form) discrepancy loss functions to construct sharper in-sample error estimators and obtain sharper bounds compared to Sun et al. [67]. Xie et al. [79] connect the online RL with the coverage condition in the offline RL, and also study the GOLF algorithm proposed in Jin et al. [37].
Algorithmic design in sample-efficient RL with function approximation. The most prominent approach in this area is based on the principle of "Optimism in the Face of Uncertainty" (OFU), which dates back to Auer et al. [4]. For instance, for linear function approximation, Jin et al. [39] propose an optimistic variant of Least-Squares Value Iteration (LSVI), which achieves optimism by adding a bonus at each step. For the general case, Jiang et al. [36] first propose an elimination-based algorithm with optimism in model-free RL and is extended to model-based RL by [67]. After these, Du et al. [20], Jin et al. [37] propose two OFU-based algorithms, which are more similar to the lin-UCB algorithm [1] studied in the linear contextual bandit literature. The model-based counterpart (Optimistic Maximum Likelihood Estimation) is studied in Liu et al. [46], Chen et al. [13]. Specifically, these algorithms explicitly maintain a confidence set that contains the ground truth with high probability and conducts a constraint optimization step at each iteration to select the most optimistic hypothesis in the confidence set. The other line of work studies another powerful algorithmic framework based on posterior sampling. For instance, Zanette et al. [84] study randomized least-squares value iteration (RLSVI), which can be interpreted as a sampling-based algorithm and achieves an order-optimal result for linear MDP. For general function approximation, the works mainly follow the idea of the "feel-good" modification proposed in Zhang [87]. These algorithms start from some prior distribution over the hypothesis space and update the posterior distribution according to the collected samples but with certain optimistic modifications in either the prior or the loglikelihood function. Then the hypothesis for each iteration is sampled from the posterior and guides data collection. In particular, Dann et al. [19] studies the model-free Q-type problem, and Agarwal and Zhang [2] studies the model-based problems, but under different notions of complexity measures. Zhong et al. [89] further utilize the idea in Zhang [87] and extend the posterior sampling algorithm in Dann et al. [19] to be a unified sampling-based framework to solve both model-free and model-based RL problems, which is also shown to apply to the more challenging partially observable setting. In addition to the OFU-based algorithm and the sampling-based framework, Foster et al. [25] propose the Estimation-to-Decisions (E2D) algorithm, which can solve problems with low DEC but requires solving a complicated minimax subroutine to fit in the framework of DEC.
Due to the limitation of the page, we defer the remaining discussions of Relationship with reward-biased maximum likelihood estimation, Exploration of DRL, and Two-player Zero-Sum Markov Game to Appendix B.
2.1 Episodic Markov Decision Processes and Online Reinforcement Learning
We consider an episodic MDP defined by a tuple  $(\mathcal{S},\mathcal{A},H,\mathbb{P},r)$ , where  $\mathcal{S}$  and  $\mathcal{A}$  are the state and action spaces,  $H$  is a finite horizon,  $\mathbb{P} = \{\mathbb{P}_h\}_{h = 1}^H$  with  $\mathbb{P}_h:\mathcal{S}\times \mathcal{A}\mapsto \Delta (\mathcal{S})$  the transition kernel at the  $h$ -th timestep, and  $r = \{r_h\}_{h = 1}^H$  with  $r_h\colon \mathcal{S}\times \mathcal{A}\to [0,1]$  the reward function at the  $h$ -th timestep. Without loss of generality, we assume that the reward function  $r_h$  is both deterministic and known.
In the episodic MDP, the agent interacts with the environment by the following online protocol. At the beginning of the  $k$ -th episode, the agent selects a policy  $\pi^k = \{\pi_h^k : \mathcal{S} \mapsto \Delta(\mathcal{A})\}_{h=1}^H$ . It takes an action  $a_h^k \sim \pi_h^k(\cdot | x_h^k)$  at timestep  $h$  and state  $x_h^k$ . After receiving the reward  $r_h^k = r_h(x_h^k, a_h^k)$  from the environment, it transits to the next state  $x_{h+1}^k \sim \mathbb{P}_h(\cdot | x_h^k, a_h^k)$ . When the agent reaches the state  $x_{H+1}^k$ , it ends the  $k$ -th episode. Without loss of generality, we assume that the initial state  $x_1 = \underline{x}$  is fixed. Our analysis can be generalized to the setting where  $x_1$  is sampled from a distribution on  $\mathcal{S}$ .
Policy and value functions. For a policy  $\pi = \{\pi_h : S \mapsto \Delta(\mathcal{A})\}_{h=1}^H$ , we denote by  $V_h^\pi : S \mapsto \mathbb{R}$  and  $Q_h^\pi : S \times \mathcal{A} \mapsto \mathbb{R}$  its value function and state-action value function at the  $h$ -th timestep, which characterizes the expected total rewards received by the agent under policy  $\pi$  afterward, starting from some  $x_h = x \in S$  (or  $x_h = x \in S$ ,  $a_h = a \in \mathcal{A}$ , resp.), till the end of the episode. Specifically,
$$
V _ {h} ^ {\pi} (x) := \mathbb {E} _ {\pi} \left[ \sum_ {h ^ {\prime} = h} ^ {H} r _ {h ^ {\prime}} \left(x _ {h ^ {\prime}}, a _ {h ^ {\prime}}\right) \mid x _ {h} = x \right], \quad \forall x \in \mathcal {S}, \tag {2.1}
$$
$$
Q _ {h} ^ {\pi} (x, a) := \mathbb {E} _ {\pi} \left[ \sum_ {h ^ {\prime} = h} ^ {H} r _ {h ^ {\prime}} \left(x _ {h ^ {\prime}}, a _ {h ^ {\prime}}\right) \Bigg | x _ {h} = x, a _ {h} = a \right], \quad \forall (x, a) \in \mathcal {S} \times \mathcal {A}. \tag {2.2}
$$
We know there exists an optimal policy  $\pi^{*}$  which has the optimal value function for all initial states [61], that is,  $V_{h}^{\pi^{*}}(s) = \sup_{\pi} V_{h}^{\pi}(x)$  for all  $h \in [H]$  and  $x \in S$ . For simplicity, we abbreviate  $V^{\pi^{*}}$  as  $V^{*}$  and the optimal state-action value function  $Q^{\pi^{*}}$  as  $Q^{*}$ . Moreover, the optimal value functions  $Q^{*}$  and  $V^{*}$  satisfy the following Bellman optimality equation [61], given by
$$
V _ {h} ^ {*} (x) = \max  _ {a \in \mathcal {A}} Q _ {h} ^ {*} (x, a), \quad V _ {H + 1} ^ {*} (x) = 0, \tag {2.3}
$$
$$
Q _ {h} ^ {*} (x, a) = \left(\mathcal {T} _ {h} Q _ {h + 1} ^ {*}\right) (x, a) := r _ {h} (x, a) + \mathbb {E} _ {x ^ {\prime} \sim \mathbb {P} _ {h (\cdot \mid x, a)} \left[ V _ {h + 1} ^ {*} \left(x ^ {\prime}\right) \right]}, \tag {2.4}
$$
for all  $(x, a, h) \in S \times \mathcal{A} \times [H]$ . We call  $\mathcal{T}_h$  the Bellman optimality operator at timestep  $h$ . Also, for any two functions  $Q_h$  and  $Q_{h+1}$  on  $S \times \mathcal{A}$ , we define
$$
\mathcal {E} _ {h} \left(Q _ {h}, Q _ {h + 1}; x, a\right) := Q _ {h} (x, a) - \mathcal {T} _ {h} Q _ {h + 1} (x, a), \quad \forall (x, a) \in \mathcal {S} \times \mathcal {A}, \tag {2.5}
$$
as the Bellman residual at timestep  $h$  of  $(Q_h, Q_{h+1})$ .
Performance metric. We measure the online performance of an agent after  $K$  episodes by regret. We assume that the learner predicts the optimal policy  $\pi^{*}$  via  $\pi^k$  in the  $k$ -th episode for each  $k \in [K]$ . Then the regret of  $K$  episodes is defined as the cumulative suboptimality gap of  $\{\pi^k\}_{k \in [K]^2}$ ,
$$
\operatorname {R e g r e t} (K) = \sum_ {k = 1} ^ {K} V _ {1} ^ {*} \left(x _ {1}\right) - V _ {1} ^ {\pi^ {k}} \left(x _ {1}\right). \tag {2.6}
$$
The target of sample-efficient online RL is to achieve sublinear regret (2.6) with respect to  $K$ .
2.2 Function Approximation: Model-Free and Model-Based Hypothesis
To handle MDPs with large or even infinite state space, we introduce a family of function approximators. In specific, we consider a hypothesis class  $\mathcal{H} = \mathcal{H}_1 \times \dots \times \mathcal{H}_H$ , which can be specified to model-based and model-free settings respectively. Also, we denote by  $\Pi = \Pi_1 \times \dots \times \Pi_H$  as the space of Markovian policies. We now specify  $\mathcal{H}$  for model-free and model-based settings.
2We allow the agent to predict the optimal policy via  $\pi^k$  while executing some other exploration policy  $\pi_{\mathrm{exp}}^k$  to interact with the environment and collect data, as is considered in the related literature [67, 20, 89]
Algorithm 1 Maximize to Explore (MEX)
1: Input: Hypothesis class  $\mathcal{H}$ , parameter  $\eta > 0$ .
2: for  $k = 1, \dots, K$  do
3: Solve  $f^k \in \mathcal{H}$  via
$$
f ^ {k} = \underset {f \in \mathcal {H}} {\operatorname {a r g s u p}} \left\{V _ {1, f} \left(x _ {1}\right) - \eta \cdot \sum_ {h = 1} ^ {H} L _ {h} ^ {k - 1} (f) \right\}. \tag {3.1}
$$
4: Execute  $\pi_{\mathrm{exp}}(f^k)$  to collect data  $\mathcal{D}^k = \{\mathcal{D}_h^k\}_{h\in [H]}$  with  $\mathcal{D}_h^k = (x_h^k,a_h^k,r_h^k,x_{h + 1}^k)$ .
5: Predict the optimal policy via  $\pi_{f^k}$ .
6: end for
Example 2.1 (Model-free hypothesis). For model-free hypothesis class,  $\mathcal{H}$  contains state-action value functions of the MDP, i.e.,  $\mathcal{H}_h \subseteq \{f_h : S \times \mathcal{A} \mapsto \mathbb{R}\}$ . Specifically, for any  $f = (f_1, \dots, f_H) \in \mathcal{H}$ , we denote  $Q_f = \{Q_{h,f}\}_{h \in [H]}$  with  $Q_{h,f} = f_h$ . Also, we denote the corresponding optimal state-value function  $V_f = \{V_{h,f}\}_{h \in [H]}$  with  $V_{h,f}(\cdot) = \max_{a \in \mathcal{A}} Q_{h,f}(\cdot, a)$  and denote the corresponding optimal policy by  $\pi_f = \{\pi_{h,f}\}_{h \in [H]}$  with  $\pi_{h,f}(\cdot) = \arg \max_{a \in \mathcal{A}} Q_{h,f}(\cdot, a)$ . Finally, we denote the optimal state-action value function under the true model, i.e.,  $Q^*$ , by  $f^*$ .
Example 2.2 (Model-based hypothesis). For model-based hypothesis class,  $\mathcal{H}$  contains models of the MDP, i.e., the transition kernel. Specifically, we denote  $f = \mathbb{P}_f = (\mathbb{P}_{1,f},\dots ,\mathbb{P}_{H,f})\in \mathcal{H}$ . For any  $(f,\pi)\in \mathcal{H}\times \Pi$ , we define  $V_{f}^{\pi} = \{V_{h,f}^{\pi}\}_{h\in [H]}$  as the state-value function induced by model  $\mathbb{P}_f$  and policy  $\pi$ . We use  $V_{f} = \{V_{h,f}\}_{h\in [H]}$  to denote the corresponding optimal state-value function, i.e.,  $V_{h,f} = \sup_{\pi \in \Pi}V_{h,f}^{\pi}$ . The corresponding optimal policy is denoted by  $\pi_f = \{\pi_{h,f}\}_{h\in [H]}$ , where  $\pi_{h,f} = \arg \sup_{\pi \in \Pi}V_{h,f}^{\pi}$ . Finally, we denote the true model  $\mathbb{P}$  of the MDP as  $f^{*}$ .
We remark that the main difference between the model-based hypothesis (Example 2.2) and the model-free hypothesis (Example 2.1) is that model-based RL directly learns the transition kernel of the underlying MDP, while model-free RL learns the optimal state-action value function. Since we do not add any specific structural form to the hypothesis class, e.g., linear function or kernel function, we are in the context of general function approximations [67, 37, 20, 89, 15].
3 Algorithm: Maximize to Explore for Online RL
In this section, we propose Maximize to Explore (MEX, Algorithm 1) for online RL in MDPs with general function approximations. With a novel single objective, MEX automatically balances the goal of exploration and exploitation. We first give a generic algorithm framework and then instantiate it to model-free (Example 2.1) and model-based (Example 2.2) hypotheses respectively.
Generic algorithm design. In each episode  $k \in [K]$ , the agent first estimates a hypothesis  $f^k \in \mathcal{H}$  using historical data  $\{\mathcal{D}^s\}_{s=1}^{k-1}$  by maximizing a composite objective (3.1). Specifically, in order to achieve exploiting history knowledge while encouraging exploration, the agent considers a single objective that sums: (a) the negative loss  $-L_h^{k-1}(f)$  induced by the hypothesis  $f$ , which represents the exploitation of the agent's current knowledge; (b) the expected total return of the optimal policy associated with this hypothesis, i.e.,  $V_{1,f}$ , which represents exploration for a higher return. With a tuning parameter  $\eta > 0$ , the agent balances the weight put on the tasks of exploitation and exploration. The agent then predicts  $\pi^*$  via the optimal policy with respect to the hypothesis that maximizes the composite exploration-exploitation objective function, i.e.,  $\pi_{f^k}$ . Also, the agent executes certain exploration policy  $\pi_{\mathrm{exp}}(f^k)$  to collect data  $\mathcal{D}^k = \{(x_h^k, a_h^k, r_h^k, x_{h+1}^k)\}_{h=1}^H$  and updates the loss function  $L_h^k(f)$ . The choice of  $\pi_{\mathrm{exp}}(f^k)$  depends on the specific MDP structure, and we refer to examples in Section 5 for detailed discussions.