Title: Maximize to Explore: One Objective Function Fusing Estimation, Planning, and Exploration
Venue: NeurIPS 2023
Accepted: yes
Keywords: reinforcement learning; online learning; game
Abstract: In reinforcement learning (RL), balancing exploration and exploitation is crucial for achieving an optimal policy in a sample-efficient way. To this end, existing sample- efficient algorithms typically consist of three components: estimation, planning, and exploration. However, to cope with general function approximators, most of them involve impractical algorithmic components to incentivize exploration, such as data-dependent level-set constraints or complicated sampling procedures. To address this challenge, we propose an easy-to-implement RL framework called Maximize to Explore (MEX), which only needs to optimize unconstrainedly a single objective that integrates the estimation and planning components while balancing exploration and exploitation automatically. Theoretically, we prove that the MEX achieves a sublinear regret with general function approximators and is extendable to the zero-sum Markov game setting. Meanwhile, we adapt deep RL baselines to design practical versions of MEX in both the model-based and model-free settings, which outperform baselines in various MuJoCo environments with sparse reward by a stable margin. Compared with existing sample-efficient algorithms with general function approximators, MEX achieves similar sample efficiency while also enjoying a lower computational cost and is more compatible with modern deep RL methods.

Maximize to Explore: One Objective Function Fusing Estimation, Planning, and Exploration

Abstract

In reinforcement learning (RL), balancing exploration and exploitation is crucial for achieving an optimal policy in a sample-efficient way. To this end, existing sampleefficient algorithms typically consist of three components: estimation, planning, and exploration. However, to cope with general function approximators, most of them involve impractical algorithmic components to incentivize exploration, such as data-dependent level-set constraints or complicated sampling procedures. To address this challenge, we propose an easy-to-implement RL framework called Maximize to Explore ( MEX ), which only needs to optimize unconstrainedly a single objective that integrates the estimation and planning components while balancing exploration and exploitation automatically. Theoretically, we prove that the MEX achieves a sublinear regret with general function approximators and is extendable to the zero-sum Markov game setting. Meanwhile, we adapt deep RL baselines to design practical versions of MEX in both the model-based and model-free settings, which outperform baselines in various MuJoCo environments with sparse reward by a stable margin. Compared with existing sample-efficient algorithms with general function approximators, MEX achieves similar sample efficiency while also enjoying a lower computational cost and is more compatible with modern deep RL methods. Our codes are available at https://github.com/agentification/MEX .

1 Introduction

The crux of online reinforcement learning (online RL) lies in maintaining a balance between exploiting the current knowledge of the agent about the environment and exploring unfamiliar areas [69]. To fulfill this, agents in existing sample-efficient RL algorithms predominantly undertake three tasks: i) estimate a hypothesis using historical data to encapsulate their understanding of the environment; ii) perform planning based on the estimated hypothesis to exploit their current knowledge; iii) further explore the unknown environment via carefully designed exploration strategies.

There exists a long line of research on integrating the aforementioned three components harmoniously, to find optimal policies in a sample-efficient manner. From theoretical perspectives, existing theories aim to minimize the notion of online external regret which measures the cumulative suboptimality gap of the policies learned during online learning. It is well studied that one can design both statistically and computationally efficient algorithms (e.g., upper confidence bound (UCB), [6, 39, 12, 91]) with sublinear online regret for tabular and linear Markov decision processes (MDPs). But when it

comes to MDPs with general function approximations, most of them involve impractical algorithmic components to incentivize exploration. Usually, to cope with general function approximations, agents need to solve constrained optimization problems within data-dependent level-sets [37, 20], or sample from complicated posterior distributions over the space of hypotheses [19, 2, 89], both of which pose considerable challenges for implementation. From a practical perspective, a prevalent approach in deep RL for balancing exploration and exploitation is to use an ensemble of neural networks [75, 59, 14, 53, 44, 18, 45], which serves as an empirical approximation of the UCB method. However, such an ensemble method suffers from high computational cost and lacks theoretical guarantee when the underly MDP is neither linear nor tabular. As for other deep RL algorithms for exploration [29, 3, 11, 9, 17, 65], such as curiosity-driven method [60], it also remains unknown in theory whether they are provably sample-efficient in the context of general function approximations.

Hence, in this paper, we are aimed at tackling these issues and answering the following question:

Under general function approximation, can we design a sample-efficient and easy-to-implement RL framework to trade off between exploration and exploitation?

To this end, we propose an easy-to-implement RL framework, Maximize to Explore ( MEX ), as an affirmative answer to the question. To strike a balance between exploration and exploitation, MEX propose to maximize a weighted sum of two objectives: (i) the optimal expected total return associated with a given hypothesis and (ii) the negative estimation error of that hypothesis. Consequently, MEX naturally combines planning and estimation components in just a single objective. By choosing the hypothesis that maximizes the weighted sum and executing the optimal policy with respect to the chosen hypothesis, MEX automatically balances between exploration and exploitation.

We highlight that the objective of MEX is not obtained by taking the Lagrange dual of the constrained optimization objective within data-dependent level-sets [37, 20, 15].This is because the coefficient of the weighted sum, which remains fixed, is data-independent and predetermined for all episodes. Contrary to Lagrangian methods, MEX does not necessitate an inner loop of optimization for dual variables, thereby circumventing the complications associated with minimax optimization. As a maximization-only framework, MEX is friendly to implementations with neural networks and does not rely on sampling or ensemble.

In the theory part, we prove that MEX achieves a sublinear regret ˜ O ( Poly ( H ) · d 1 / 2 GEC (1 / √ HK ) · K 1 / 2 ) under mild assumptions and is thus sample-efficient, where K is the number of episodes and H is the horizon length. Here d GEC ( · ) is the Generalized Eluder Coefficient (GEC) [89] that characterizes the complexity of learning the underlying MDP under general function approximations. Because the class of low-GEC MDPs includes almost all known theoretically tractable MDP instances, our proved result can be tailored to a multitude of specific settings with either a model-free or a model-based hypothesis, such as MDPs with low Bellman eluder dimension [37], MDPs of bilinear class [20], and MDPs with low witness rank [67]. Besides, thanks to the flexibility of the MEX framework, we further extend it to online RL in two-player zero-sum Markov games (MGs), for which we further extend the definition of GEC to two-player zero-sum MGs and establish the sample efficiency with general function approximations. Moving beyond theory and into practice, we adapt famous RL baselines TD3 [27] and MBPO [34] to design practical versions of MEX in model-free and model-based fashion, respectively. On various MuJoCo environments [71] with sparse rewards, experimental results show that MEX outperforms baselines steadily and significantly. Compared with other deep RL algorithms, MEX has low computational overhead and straightforward implementation while maintaining a theoretical guarantee.

Contributions. We conclude our contributions from three perspectives.

We propose an easy-to-implement RL algorithm framework MEX that unconstrainedly maximizes a single objective to fuse estimation and planning, automatically trading off between exploration and exploitation. Under mild structural assumptions, we prove that MEX achieves a sublinear regret ˜ O ( Poly ( H ) · d 1 / 2 GEC (1 / √ HK ) K 1 / 2 ) with general function approximators, and thus is sampleefficient. Here d GEC ( · ) is the generalized Eluder Coefficient (GEC) of the underlying MDP.

We instantiate the generic MEX framework to several model-based and model-free examples and establish corresponding theoretical results. Further, we extend the MEX framework to two-player zero-sum MGs and also prove the sample efficiency with an extended definition of GEC.

We design practical implementations of MEX for MDPs in both model-based and model-free styles. Experiments on various MuJoCo environments with sparse reward demonstrate the effectiveness of our proposed MEX framework.

1.1 Related work

Sample-efficient RL with function approximation. The success of DRL methods has motivated a line of work focused on function approximation scenarios. This work originated in the linear case [74, 81, 12, 39, 84, 5, 82, 57, 91, 90] and is later extended to general function approximation. Wang et al. [73] first study the general function approximation using the notion of eluder dimension [63], which takes the linear MDP [39] as a special case but with inferior results. Zanette et al. [85] consider a different type of framework based on Bellman completeness, which assumes that the class used for approximating the optimal Q-functions is closed in terms of the Bellman operator and improves the results for linear MDP. After this, Jin et al. [37] consider the eluder dimension of the class of Bellman residual associated with the RL problems, which captures more solvable problems. Another line of works focuses on the low-rank structures of the problems, where Jiang et al. [35] propose the Bellman rank for model-free RL and Sun et al. [67] propose the witness rank for model-based RL. Following these two works, Du et al. [20] propose the bilinear class, which contains more MDP models with low-rank structures [6, 67, 39, 57, 12, 91] by allowing a flexible choice of discrepancy function class. However, it is known that neither BE nor bilinear class captures each other. Dann et al. [19] first consider eluder-coefficient-type complexity measure on the Q-type model-free RL. It was later extended by Zhong et al. [89] to cover all the above-known solvable problems in both model-free and model-based manners. Foster et al. [25, 23] study another notion of complexity measure, the decision-estimation coefficient (DEC), which also unifies the Bellman eluder dimension and bilinear class. The DEC framework is appealing due to the matching lower bound in some decision-making problems, where all other complexity measures do not have. However, due to the presence of a minimax subroutine in its definition, they require a much more complicated minimax optimization oracle and cannot apply to the classical optimism-based or sampling-based methods. Chen et al. [13], Foster et al. [24] extend the vanilla DEC (to the model-free case) by incorporating an optimistic modification, which was originally referred to as the feel-good modification in Zhang [87]. Chen et al. [15] study Admissible Bellman Characterization (ABC) class to generalize BE. They also extend the GOLF algorithm and Bellman completeness in model-free RL to the model-based case by considering more general (vector-form) discrepancy loss functions to construct sharper in-sample error estimators and obtain sharper bounds compared to Sun et al. [67]. Xie et al. [79] connect the online RL with the coverage condition in the offline RL, and also study the GOLF algorithm proposed in Jin et al. [37].

Algorithmic design in sample-efficient RL with function approximation. The most prominent approach in this area is based on the principle of 'Optimism in the Face of Uncertainty' (OFU), which dates back to Auer et al. [4]. For instance, for linear function approximation, Jin et al. [39] propose an optimistic variant of Least-Squares Value Iteration (LSVI), which achieves optimism by adding a bonus at each step. For the general case, Jiang et al. [36] first propose an elimination-based algorithm with optimism in model-free RL and is extended to model-based RL by [67]. After these, Du et al. [20], Jin et al. [37] propose two OFU-based algorithms, which are more similar to the lin-UCB algorithm [1] studied in the linear contextual bandit literature. The model-based counterpart (Optimistic Maximum Likelihood Estimation) is studied in Liu et al. [46], Chen et al. [13]. Specifically, these algorithms explicitly maintain a confidence set that contains the ground truth with high probability and conducts a constraint optimization step at each iteration to select the most optimistic hypothesis in the confidence set. The other line of work studies another powerful algorithmic framework based on posterior sampling. For instance, Zanette et al. [84] study randomized least-squares value iteration (RLSVI), which can be interpreted as a sampling-based algorithm and achieves an order-optimal result for linear MDP. For general function approximation, the works mainly follow the idea of the 'feel-good' modification proposed in Zhang [87]. These algorithms start from some prior distribution over the hypothesis space and update the posterior distribution according to the collected samples but with certain optimistic modifications in either the prior or the loglikelihood function. Then the hypothesis for each iteration is sampled from the posterior and guides data collection. In particular, Dann et al. [19] studies the model-free Q-type problem, and Agarwal and Zhang [2] studies the model-based problems, but under different notions of complexity measures. Zhong et al. [89] further utilize the idea in Zhang [87] and extend the posterior sampling algorithm in Dann et al. [19] to be a unified sampling-based framework to solve both model-free and model-based RL problems, which is also shown to apply to the more challenging partially observable setting. In addition to the OFU-based algorithm and the sampling-based framework, Foster et al. [25] propose the Estimation-to-Decisions (E2D) algorithm, which can solve problems with low DEC but requires solving a complicated minimax subroutine to fit in the framework of DEC.

Due to the limitation of the page, we defer the remaining discussions of Relationship with rewardbiased maximum likelihood estimation , Exploration of DRL , and Two-player Zero-Sum Markov Game to Appendix B.

2 Preliminaries

2.1 Episodic Markov Decision Processes and Online Reinforcement Learning

We consider an episodic MDP defined by a tuple ( S , A , H, P , r ) , where S and A are the state and action spaces, H is a finite horizon, P = { P h } H h =1 with P h : S × A ↦→ ∆( S ) the transition kernel at the h -th timestep, and r = { r h } H h =1 with r h : S ×A → [0 , 1] the reward function at the h -th timestep. Without loss of generality, we assume that the reward function r h is both deterministic and known.

In the episodic MDP, the agent interacts with the environment by the following online protocol. At the beginning of the k -th episode, the agent selects a policy π k = { π k h : S ↦→ ∆( A ) } H h =1 . It takes an action a k h ∼ π k h ( · | x k h ) at timestep h and state x k h . After receiving the reward r k h = r h ( x k h , a k h ) from the environment, it transits to the next state x k h +1 ∼ P h ( · | x k h , a k h ) . When the agent reaches the state x k H +1 , it ends the k -th episode. Without loss of generality, we assume that the initial state x 1 = x is fixed. Our analysis can be generalized to the setting where x 1 is sampled from a distribution on S .

Policy and value functions. For a policy π = { π h : S ↦→ ∆( A ) } H h =1 , we denote by V π h : S ↦→ R and Q π h : S × A ↦→ R its value function and state-action value function at the h -th timestep, which characterizes the expected total rewards received by the agent under policy π afterward, starting from some x h = x ∈ S (or x h = x ∈ S , a h = a ∈ A , resp.), till the end of the episode. Specifically,

We know there exists an optimal policy π ∗ which has the optimal value function for all initial states [61], that is, V π ∗ h ( s ) = sup π V π h ( x ) for all h ∈ [ H ] and x ∈ S . For simplicity, we abbreviate V π ∗ as V ∗ and the optimal state-action value function Q π ∗ as Q ∗ . Moreover, the optimal value functions Q ∗ and V ∗ satisfy the following Bellman optimality equation [61], given by

for all ( x, a, h ) ∈ S × A × [ H ] . We call T h the Bellman optimality operator at timestep h . Also, for any two functions Q h and Q h +1 on S × A , we define

as the Bellman residual at timestep h of ( Q h , Q h +1 ) .

Performance metric. We measure the online performance of an agent after K episodes by regret . We assume that the learner predicts the optimal policy π ∗ via π k in the k -th episode for each k ∈ [ K ] . Then the regret of K episodes is defined as the cumulative suboptimality gap of { π k } k ∈ [ K ] 2 ,

The target of sample-efficient online RL is to achieve sublinear regret (2.6) with respect to K .

2.2 Function Approximation: Model-Free and Model-Based Hypothesis

To handle MDPs with large or even infinite state space, we introduce a family of function approximators. In specific, we consider a hypothesis class H = H 1 ×··· × H H , which can be specified to model-based and model-free settings respectively. Also, we denote by Π = Π 1 ×··· × Π H as the space of Markovian policies. We now specify H for model-free and model-based settings.

Algorithm 1 Maximize to Explore ( MEX )

1: Input : Hypothesis class H , parameter η > 0 .

2: for k = 1 , · · · , K do

3: Solve f k ∈ H via

4: Execute π exp ( f k ) to collect data D k = {D k h } h ∈ [ H ] with D k h = ( x k h , a k h , r k h , x k h +1 ) .

5: Predict the optimal policy via π f k .

6: end for

Example 2.1 (Model-free hypothesis) . For model-free hypothesis class, H contains state-action value functions of the MDP, i.e., H h ⊆ { f h : S × A ↦→ R } . Specifically, for any f = ( f 1 , · · · , f H ) ∈ H , we denote Q f = { Q h,f } h ∈ [ H ] with Q h,f = f h . Also, we denote the corresponding optimal statevalue function V f = { V h,f } h ∈ [ H ] with V h,f ( · ) = max a ∈A Q h,f ( · , a ) and denote the corresponding optimal policy by π f = { π h,f } h ∈ [ H ] with π h,f ( · ) = arg max a ∈A Q h,f ( · , a ) . Finally, we denote the optimal state-action value function under the true model, i.e., Q ∗ , by f ∗ .

Example 2.2 (Model-based hypothesis) . For model-based hypothesis class, H contains models of the MDP, i.e., the transition kernel. Specifically, we denote f = P f = ( P 1 ,f , · · · , P H,f ) ∈ H . For any ( f, π ) ∈ H × Π , we define V π f = { V π h,f } h ∈ [ H ] as the state-value function induced by model P f and policy π . We use V f = { V h,f } h ∈ [ H ] to denote the corresponding optimal state-value function, i.e., V h,f = sup π ∈ Π V π h,f . The corresponding optimal policy is denoted by π f = { π h,f } h ∈ [ H ] , where π h,f = arg sup π ∈ Π V π h,f . Finally, we denote the true model P of the MDP as f ∗ .

We remark that the main difference between the model-based hypothesis (Example 2.2) and the model-free hypothesis (Example 2.1) is that model-based RL directly learns the transition kernel of the underlying MDP, while model-free RL learns the optimal state-action value function. Since we do not add any specific structural form to the hypothesis class, e.g., linear function or kernel function, we are in the context of general function approximations [67, 37, 20, 89, 15].

3 Algorithm: Maximize to Explore for Online RL

In this section, we propose Maximize to Explore ( MEX , Algorithm 1) for online RL in MDPs with general function approximations. With a novel single objective, MEX automatically balances the goal of exploration and exploitation. We first give a generic algorithm framework and then instantiate it to model-free (Example 2.1) and model-based (Example 2.2) hypotheses respectively.

Generic algorithm design. In each episode k ∈ [ K ] , the agent first estimates a hypothesis f k ∈ H using historical data {D s } k -1 s =1 by maximizing a composite objective (3.1). Specifically, in order to achieve exploiting history knowledge while encouraging exploration, the agent considers a single objective that sums: (a) the negative loss -L k -1 h ( f ) induced by the hypothesis f , which represents the exploitation of the agent's current knowledge; (b) the expected total return of the optimal policy associated with this hypothesis, i.e., V 1 ,f , which represents exploration for a higher return. With a tuning parameter η > 0 , the agent balances the weight put on the tasks of exploitation and exploration. The agent then predicts π ∗ via the optimal policy with respect to the hypothesis that maximizes the composite exploration-exploitation objective function, i.e., π f k . Also, the agent executes certain exploration policy π exp ( f k ) to collect data D k = { ( x k h , a k h , r k h , x k h +1 ) } H h =1 and updates the loss function L k h ( f ) . The choice of π exp ( f k ) depends on the specific MDP structure, and we refer to examples in Section 5 for detailed discussions.

We highlight that MEX is not a Lagrangian duality of constrained optimization objectives within data-dependent level-sets [37, 20, 15]. In fact, MEX only needs to fix the parameter η across each episode. Thus η is independent of data and predetermined, which contrasts Lagrangian methods that involve an inner loop of optimization for the dual variables. We also remark that we can rewrite (3.1) as a joint optimization ( f, π ) = argsup f ∈H ,π ∈ Π V π 1 ,f ( x 1 ) -η ∑ H h =1 L k -1 h ( f ) . When η tends to infinity, MEX can be reduced to vanilla Actor-Critic framework [41], where critic f minimizes estimation error and actor π conducts greedy policy following the critic f . In the following two parts,

we instantiate Algorithm 1 to model-based and mode-free hypotheses by specifying the loss function L k h ( f ) .

Model-free algorithm. For model-free hypothesis (Example 2.1), (3.1) becomes

Regarding the choice of the loss function, for seek of theoretical analysis, to deal with MDPs with low Bellman eluder dimension [37] and MDPs of bilinear class [20], we assume the existence of certain function l which generalizes the notion of Bellman residual.

Assumption 3.1. Suppose the function l : H×H h ×H h +1 × ( S × A × R ×S ) ↦→ R satisfies: i) (Generalized Bellman completeness) [89, 15] there exists an operator P h : H h +1 ↦→H h such that for any ( f ′ , f h , f h +1 ) ∈ H × H h ×H h +1 and D h = ( x h , a h , r h , x h +1 ) ∈ S × A × R ×S , it holds

,

where we require that P h f ∗ h +1 = f ∗ h and that P h f h +1 ∈ H h for any f h +1 ∈ H h +1 and step h ∈ [ H ] ; ii) (Boundedness) it holds that sup ′ ∥ l f ′ (( f h , f h +1 ) , D h ) ∥ ∞ ≤ B l for some constant B l > 0 .

We then set the loss function L h as an empirical estimation of the generalized squared Bellman error | E x h +1 ∼ P h ( ·| x h ,a h ) [ l f s (( f h , f h +1 ) , D s h )] | 2 , given by f ∈H k

Weremark that the subtracted infimum term in (3.3) is to handle the variance terms in the estimation to achieve a fast theoretical rate. Similar essential ideas are also adopted by [37, 78, 19, 38, 52, 2, 89, 15].

Model-based algorithm. For model-based hypothesis (Example 2.2), (3.1) becomes

which is a joint optimization over the model P f and the policy π . In the model-based algorithm, we choose the loss function L k h as the negative log-likelihood loss, defined as

4 Regret Analysis for MEX Framework

In this section, we establish a regret analysis for the MEX framework (Algorithm 1). We give a generic theoretical guarantee which holds for both model-free and model-based settings. We first present three key assumptions needed for sample-efficient learning with MEX . In Section 5, we specify the generic theory to specific examples of MDPs and hypothesis classes satisfying these assumptions.

Firstly, we assume that the hypothesis class H is well-specified, containing the true hypothesis f ∗ . Assumption 4.1 (Realizablity) . We assume that the true hypothesis f ∗ ∈ H .

Then we need to make a structural assumption on the MDP to ensure sample-efficient online learning. Inspired by Zhong et al. [89], we require the MDP to have low Generalized Eluder Coefficient (GEC). A low GEC indicates that the agent can effectively mitigate out-of-sample prediction errors by minimizing in-sample errors derived from historical data. To introduce, we define a discrepancy function ℓ f ′ ( f ; ξ h ) : H×H× ( S × A × R ×S ) ↦→ R which characterizes the error of a hypothesis f ∈ H on data ξ h = ( x h , a h , r h , x h +1 ) . Specific choices of ℓ are given in Section 5 for concrete model-free and model-based examples.

Assumption 4.2 (Low Generalized Eluder Coefficient [89]) . We assume that given an ϵ > 0 , there exists d ( ϵ ) ∈ R + , such that for any sequence of { f k } k ∈ [ K ] ⊂ H , { π exp ( f k ) } k ∈ [ K ] ⊂ Π ,

We denote the smallest number d ( ϵ ) ∈ R + satisfying this condition as d GEC ( ϵ ) .

As is shown by Zhong et al. [89], the low-GEC MDP class covers almost all known theoretically tractable MDP instances, such as linear MDP [81, 39], linear mixture MDP [5, 57, 12], MDPs of low witness rank [67], MDPs of low Bellman eluder dimension [37], and MDPs of bilinear class [20].

Finally, we make a concentration-style assumption which characterizes how the loss function L k h is related to the expectation of the discrepancy function E [ ℓ ] appearing in the definition of GEC. For ease of presentation, we assume that H is finite, i.e., |H| < ∞ , but our result can be directly extended to an infinite H using covering number arguments [72, 37, 49, 38].

Assumption 4.3 (Generalization) . We assume that H is finite, i.e., |H| < + ∞ , and that with probability at least 1 -δ , for any episode k ∈ [ K ] and hypothesis f ∈ H , it holds that

where B = B 2 l for model-free hypothesis (Assumption 3.1) and B = 1 for model-based hypothesis. Such a concentration style inequality is well known in the literature of online RL with general function approximation and similar analysis is also adopted by [37, 15]. With Assumptions 4.1, 4.2, and 4.3, we can present our main result (see Appendix D.1 for a proof).

Theorem 4.4 (Online regret of MEX (Algorithm 1)) . Under Assumptions 4.1, 4.2, and 4.3, by setting

then the regret of Algorithm 1 after K episodes is upper bounded by

with probability at least 1 -δ . Here d GEC ( · ) is defined in Assumption 4.2.

Theorem 4.4 shows that the regret of Algorithm 1 scales with the square root of the number of episodes K and the polynomials of horizon H , GEC d GEC (1 / √ K ) , and log covering number log N ( H , 1 /K ) . When the number of episodes K tends to infinity, the average regret vanishes, meaning that the output policy of Algorithm 1 achieves global optimality. Since the regret of Algorithm 1 is sublinear with respect to the number of episodes K , Algorithm 1 is proved to be sample-efficient. In Appendix C, we extend the algorithm framework and the analysis to the two-player zero-sum Markov game (MG) setting, for which we also extend the definition of GEC to two-player zero-sum MGs.

Besides, as we can see from Theorem 4.4 and its specifications in Section 5, MEX matches existing theoretical results in the literature of online RL with general function approximations [89, 67, 20, 37, 19, 2]. But in the meanwhile, MEX does not require explicitly solving a constrained optimization problem within data-dependent level-sets or performing a complex sampling procedure. This advantage makes MEX a principled approach with easier practical implementation. We conduct deep RL experiments for MEX in Section 6 to demonstrate its power in complicated online problems.

5 Examples of MEX Framework

In this section, we specify Algorithm 1 to model-based and model-free hypothesis classes for various examples of MDPs of low GEC (Assumption 4.2), including MDPs with low witness rank [67], MDPs with low Bellman eluder dimension [37], and MDPs of bilinear class [20]. For ease of presentation, we assume that |H| < ∞ , but our result can be directly extended to infinite H using covering number arguments [72, 37, 49]. All the proofs of the propositions in this section are in Appendix E.

We note that another important step in specifying Theorem 4.4 to concrete hypothesis classes is to check Assumption 4.3 (supervised learning guarantee). It is worth highlighting that, in our analysis, for both model-free and model-based hypotheses, we provide supervised learning guarantees in a neat and unified manner, independent of specific MDP structures.

5.1 Model-free online RL in Markov Decision Processes

In this subsection, we specify Algorithm 1 for model-free hypothesis class H (Example 2.1). For a model-free hypothesis class, we choose the discrepancy function ℓ as, given D h = ( x h , a h , r h , x h +1 ) ,

where the function l : H×H h ×H h +1 × ( S × A × R ×S ) ↦→ R satisfies Assumption 3.1. We specify the choice of l in concrete examples of MDPs later. In the following, we check and specify Assumptions 4.2 and 4.3 in Section 4 for model-free hypothesis classes.

Proposition 5.1 (Generalization: model-free RL) . We assume that H is finite, i.e., |H| < + ∞ . Then under Assumption 3.1, with probability at least 1 -δ , for any k ∈ [ K ] and f ∈ H , it holds that

where L and ℓ are defined in (3.3) and (5.1) respectively. Here B l is specified in Assumption 3.1.

Proposition 5.1 specifies Assumption 4.3 for model-free hypothesis classes. For Assumption 4.2, we need structural assumptions on the MDP. Given an MDP with generalized eluder dimension d GEC , we have the following corollary of our main theoretical result (Theorem 4.4).

Corollary 5.2 (Online regret of MEX : model-free hypothesis) . Given an MDP with generalized eluder coefficient d GEC ( · ) and a finite model-free hypothesis class H with f ∗ ∈ H , under Assumption 3.1, setting

then the regret of Algorithm 1 after K episodes is upper bounded by

with probability at least 1 -δ . Here B l is specified in Assumption 3.1.

Corollary 5.2 can be directly specified to MDPs with low GEC, including MDPs with low Bellman eluder dimension [37] and MDPs of bilinear class [20]. See Appendix E.1 for a detailed discussion.

5.2 Model-based online RL in Markov Decision Processes

In this subsection, we specify Algorithm 1 for model-based hypothesis class H (Example 2.2). For model-based hypothesis class, we choose the discrepancy function ℓ in Assumption 4.2 and 4.3 as the hellinger distance. Given data D h = ( x h , a h , r h , x h +1 ) , we let

where D H ( ·∥· ) denotes the Hellinger distance. We note that by (5.4), the discrepancy function ℓ does not depend on the input f ′ ∈ H . In the following, we check and specify Assumption 4.2 and 4.3.

Proposition 5.3 (Generalization: model-based RL) . We assume that H is finite, i.e., |H| < + ∞ . Then with probability at least 1 -δ , for any k ∈ [ K ] , f ∈ H , it holds that

where L and ℓ are defined in (3.5) and (5.4) respectively.

Proposition 5.3 specifies Assumption 4.3 for model-based hypothesis classes. For Assumption 4.2, we also need structural assumptions on the MDP. Given an MDP with generalized eluder dimension d GEC , we have the following corollary of our main theoretical result (Theorem 4.4).

Corollary 5.4 (Online regret of MEX : model-based hypothesis) . Given an MDP with generalized eluder coefficient d GEC ( · ) and a finite model-based hypothesis class H with f ∗ ∈ H , by setting

then the regret of Algorithm 1 after K episodes is upper bounded by, with probability at least 1 -δ ,

Corollary 5.4 can be directly specified to MDPs with low GEC, including MDPs with low witness rank [67]. We refer to Appendix E.2 for a detailed discussion.

6 Experiments

In this section, we aim to answer the following two questions: (a) What are the practical approaches to implementing MEX in both model-based ( MEX-MB ) and model-free ( MEX-MF ) settings? (b) Can MEX handle challenging exploration tasks, especially in sparse reward scenarios?

Experimental setups. We evaluate the effectiveness of MEX by assessing its performance in both standard gym locomotion tasks and sparse reward locomotion and navigation tasks within the MuJoCo [71] environment. For sparse reward tasks, we select cheetah-vel , walker-vel , hopper-vel , ant-vel , and ant-goal adapted from Yu et al. [83], where the agent receives a reward only when it successfully attains the desired velocity or goal. To adapt to deep RL settings, we consider infinitehorizon γ -discounted MDPs and MEX variants. We report the results averaged over five random seeds. The full experimental settings are in Appendix H.

Implementation details. For the model-based variant MEX-MB , we use the following objective:

where we denote by σ the initial state distribution, β the replay buffer, and η ′ corresponds to 1 /η in the previous theory sections. We leverage the score function to obtain the model value gradient ∇ ϕ V π P ϕ in a similar way to likelihood ratio policy gradient [70], with the gradient of action log-likelihood replaced by the gradient of state and reward log-likelihood in the model. Specifically,

where τ π ϕ is the trajectory under policy π and transition P ϕ , starting from σ . We refer the readers to previous works [62, 76] for a derivation of (6.2). The model ϕ and policy π in (6.1) are updated iteratively in a Dyna [68] style, where model-free policy updates are performed on model-generated data. Particularly, we adopt SAC [30] to update the policy π and estimate the value Q π P ϕ by performing temporal difference on the model data generated by P ϕ . We also follow [62] to update the model using mini-batches from β and normalize the advantage r h + γV π P ϕ -Q π P ϕ within each mini-batch.

For the model-free variant MEX-MF , we observe from (3.2) that adding a maximization bias term to the standard TD error is sufficient for exploration. However, this may lead to instabilities as the bias term only involves the state-action value function of the current policy, and thus the policy may be ever-changing. To address this issue, we adopt a similar treatment as in CQL [42] by subtracting a baseline state-action value from random policy µ = Unif ( A ) and obtain the following objective:

We update θ and π in (6.3) iteratively in an actor-critic fashion. Due to space limits, we refer the readers to Appendix H for more implementation details of MEX-MF .

Results. We report the performance of MEX-MB and MEX-MF in Figures 1 and 2, respectively. We compare MEX-MB with MBPO [34], where our method differs from MBPO only in the inclusion of the value gradient in (6.2) during model updates. We find that MEX-MB offers an easy implementation with minimal computational overhead and yet remains highly effective across sparse and standard MuJoCo tasks. Notably, in the sparse reward settings, MEX-MB excels at achieving the goal velocity and outperforms MBPO by a stable margin. In standard gym tasks, MEX-MB showcases greater sample efficiency in challenging high-dimensional tasks with higher asymptotic returns.

We then compare MEX-MF with the model-free baseline TD3 [27]. We observe that TD3 fails in many sparse reward tasks, while MEX-MF significantly boosts the performance. In standard MuJoCo gym tasks, MEX-MF also steadily outperforms TD3 with faster convergence and higher final returns.

7 Conclusions

In this paper, we introduce a novel RL algorithm frameworkMaximize to Explore ( MEX )-aimed at striking a balance between exploration and exploitation in online learning scenarios. MEX is provably sample-efficient under general function approximations and is easy to implement. Theoretically, we prove that under mild structural assumptions (low generalized eluder coefficient (GEC)), MEX achieves

0.0

0.2

0.4

0.6

0.8

1.0

Steps

1e5

0

250

500

750

1000

1250

1500

1750

Return

hopper-vel-sparse

MBPO

MEX-MB

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

Steps

1e5

0

250

500

750

1000

1250

1500

1750

walker-vel-sparse

MBPO

MEX-MB

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

1.6

Steps

1e5

0

500

1000

1500

2000

cheetah-vel-sparse

MBPO

MEX-MB

0.00

0.25

0.50

0.75

1.00

1.25

1.50

1.75

2.00

Steps

1e5

1000

0

1000

2000

3000

ant-vel-sparse

MBPO

MEX-MB

0

1

2

3

4

5

6

7

8

Steps

1e4

0

500

1000

1500

2000

2500

3000

3500

Return

Hopper-v2

MBPO

MEX-MB

0.0

0.5

1.0

1.5

2.0

2.5

3.0

Steps

1e5

0

1000

2000

3000

4000

5000

MBPO

MEX-MB

0.00

0.25

0.50

0.75

1.00

1.25

1.50

1.75

2.00

Steps

1e5

0

2000

4000

6000

8000

10000

HalfCheetah-v2

MBPO

MEX-MB

0.0

0.5

1.0

1.5

2.0

2.5

3.0

Steps

1e5

0

1000

2000

3000

4000

5000

6000

MBPO

MEX-MB

0.0

0.2

0.4

0.6

0.8

1.0

Steps

1e6

0

200

400

600

800

1000

Return

hopper-vel-sparse

TD3

MEX-MF

0.0

0.2

0.4

0.6

0.8

1.0

Steps

1e6

200

100

0

100

200

300

400

walker-vel-sparse

TD3

MEX-MF

0.0

0.2

0.4

0.6

0.8

1.0

Steps

1e6

250

0

250

500

750

1000

1250

1500

1750

cheetah-vel-sparse

TD3

MEX-MF

0.0

0.2

0.4

0.6

0.8

1.0

Steps

1e6

500

250

0

250

500

750

1000

1250

ant-goal-sparse

TD3

MEX-MF

0.0

0.2

0.4

0.6

0.8

1.0

Steps

1e6

0

500

1000

1500

2000

2500

3000

3500

Return

TD3

MEX-MF

0.0

0.2

0.4

0.6

0.8

1.0

Steps

1e6

0

1000

2000

3000

4000

5000

TD3

MEX-MF

0.0

0.2

0.4

0.6

0.8

1.0

Steps

1e6

0

2000

4000

6000

8000

10000

12000

HalfCheetah-v3

TD3

MEX-MF

0.0

0.2

0.4

0.6

0.8

1.0

Steps

1e6

0

1000

2000

3000

4000

5000

TD3

MEX-MF

˜ O ( √ K ) -online regret for MDPs. We further extend the definition of GEC and MEX framework to two-player zero-sum Markov games (see Appendix C) and also prove the ˜ O ( √ K ) -online regret. In practice, we adapt MEX to deep RL methods in both model-based and model-free styles and apply them to sparse-reward MuJoCo environments, outperforming baselines significantly. We hope our work can shed light on future research of designing both statistically efficient and practically effective RL algorithms with powerful function approximation.

References

ABBASI-YADKORI, Y., P ´ AL, D. and SZEPESV ´ ARI, C. (2011). Improved algorithms for linear stochastic bandits. Advances in neural information processing systems 24 .

AGARWAL, A. and ZHANG, T. (2022). Model-based rl with optimistic posterior sampling: Structural conditions and sample complexity. arXiv preprint arXiv:2206.07659 .

AUBRET, A., MATIGNON, L. and HASSAS, S. (2019). A survey on intrinsic motivation in reinforcement learning.

AUER, P., CESA-BIANCHI, N. and FISCHER, P. (2002). Finite-time analysis of the multiarmed bandit problem. Machine learning 47 235-256.

AYOUB, A., JIA, Z., SZEPESVARI, C., WANG, M. and YANG, L. (2020). Model-based reinforcement learning with value-targeted regression. In International Conference on Machine Learning . PMLR.

AZAR, M. G., OSBAND, I. and MUNOS, R. (2017). Minimax regret bounds for reinforcement learning. In International Conference on Machine Learning . PMLR.

BAI, Y. and JIN, C. (2020). Provable self-play algorithms for competitive reinforcement learning. In International Conference on Machine Learning . PMLR.

BAI, Y., JIN, C. and YU, T. (2020). Near-optimal reinforcement learning with self-play. arXiv preprint arXiv:2006.12007 .

BELLEMARE, M., SRINIVASAN, S., OSTROVSKI, G., SCHAUL, T., SAXTON, D. and MUNOS, R. (2016). Unifying count-based exploration and intrinsic motivation. Advances in neural information processing systems 29 .

BLANCHET, J., LU, M., ZHANG, T. and ZHONG, H. (2023). Double pessimism is provably efficient for distributionally robust offline reinforcement learning: Generic algorithm and robust partial coverage. arXiv preprint arXiv:2305.09659 .

BURDA, Y., EDWARDS, H., STORKEY, A. and KLIMOV, O. (2018). Exploration by random network distillation. arXiv preprint arXiv:1810.12894 .

CAI, Q., YANG, Z., JIN, C. and WANG, Z. (2020). Provably efficient exploration in policy optimization. In International Conference on Machine Learning . PMLR.

CHEN, F., MEI, S. and BAI, Y. (2022). Unified algorithms for rl with decision-estimation coefficients: No-regret, pac, and reward-free learning. arXiv preprint arXiv:2209.11745 .

CHEN, R. Y., SIDOR, S., ABBEEL, P. and SCHULMAN, J. (2017). Ucb exploration via q-ensembles. arXiv preprint arXiv:1706.01502 .

CHEN, Z., LI, C. J., YUAN, A., GU, Q. and JORDAN, M. I. (2022). A general framework for sample-efficient function approximation in reinforcement learning. arXiv preprint arXiv:2209.15634 .

CHEN, Z., ZHOU, D. and GU, Q. (2021). Almost optimal algorithms for two-player Markov games with linear function approximation. arXiv preprint arXiv:2102.07404 .

CHOI, J., GUO, Y., MOCZULSKI, M., OH, J., WU, N., NOROUZI, M. and LEE, H. (2018). Contingency-aware exploration in reinforcement learning. arXiv preprint arXiv:1811.01483 .

CHUA, K., CALANDRA, R., MCALLISTER, R. and LEVINE, S. (2018). Deep reinforcement learning in a handful of trials using probabilistic dynamics models. Advances in neural information processing systems 31 .

DANN, C., MOHRI, M., ZHANG, T. and ZIMMERT, J. (2021). A provably efficient model-free posterior sampling method for episodic reinforcement learning. Advances in Neural Information Processing Systems 34 12040-12051.

DU, S., KAKADE, S., LEE, J., LOVETT, S., MAHAJAN, G., SUN, W. and WANG, R. (2021). Bilinear classes: A structural framework for provable generalization in rl. In International Conference on Machine Learning . PMLR.

EYSENBACH, B., KHAZATSKY, A., LEVINE, S. and SALAKHUTDINOV, R. R. (2022). Mismatched no more: Joint model-policy optimization for model-based rl. Advances in Neural Information Processing Systems 35 23230-23243.

FILAR, J. and VRIEZE, K. (2012). Competitive Markov decision processes . Springer Science &Business Media.

FOSTER, D. J., GOLOWICH, N. and HAN, Y. (2023). Tight guarantees for interactive decision making with the decision-estimation coefficient. arXiv preprint arXiv:2301.08215 .

FOSTER, D. J., GOLOWICH, N., QIAN, J., RAKHLIN, A. and SEKHARI, A. (2022). A note on model-free reinforcement learning with the decision-estimation coefficient. arXiv preprint arXiv:2211.14250 .

FOSTER, D. J., KAKADE, S. M., QIAN, J. and RAKHLIN, A. (2021). The statistical complexity of interactive decision making. arXiv preprint arXiv:2112.13487 .

FREEDMAN, D. A. (1975). On tail probabilities for martingales. the Annals of Probability 100-118.

FUJIMOTO, S., HOOF, H. and MEGER, D. (2018). Addressing function approximation error in actor-critic methods. In International conference on machine learning . PMLR.

GUO, H., CAI, Q., ZHANG, Y., YANG, Z. and WANG, Z. (2022). Provably efficient offline reinforcement learning for partially observable markov decision processes. In International Conference on Machine Learning . PMLR.

HAARNOJA, T., ZHOU, A., ABBEEL, P. and LEVINE, S. (2018). Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Proceedings of the 35th International Conference on machine learning (ICML-18) .

HAARNOJA, T., ZHOU, A., ABBEEL, P. and LEVINE, S. (2018). Soft actor-critic: Offpolicy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning . PMLR.

HOUTHOOFT, R., CHEN, X., DUAN, Y., SCHULMAN, J., DE TURCK, F. and ABBEEL, P. (2016). Vime: Variational information maximizing exploration. Advances in neural information processing systems 29 .

HUANG, B., LEE, J. D., WANG, Z. and YANG, Z. (2021). Towards general function approximation in zero-sum markov games. arXiv preprint arXiv:2107.14702 .

HUNG, Y.-H., HSIEH, P.-C., LIU, X. and KUMAR, P. (2021). Reward-biased maximum likelihood estimation for linear stochastic bandits. In Proceedings of the AAAI Conference on Artificial Intelligence , vol. 35.

JANNER, M., FU, J., ZHANG, M. and LEVINE, S. (2019). When to trust your model: Modelbased policy optimization. Advances in neural information processing systems 32 .

JIANG, N., KRISHNAMURTHY, A., AGARWAL, A., LANGFORD, J. and SCHAPIRE, R. E. (2017). Contextual decision processes with low Bellman rank are PAC-learnable. In Proceedings of the 34th International Conference on Machine Learning , vol. 70 of Proceedings of Machine Learning Research . PMLR.

JIANG, N., KRISHNAMURTHY, A., AGARWAL, A., LANGFORD, J. and SCHAPIRE, R. E. (2017). Contextual decision processes with low bellman rank are pac-learnable. In International Conference on Machine Learning . PMLR.

JIN, C., LIU, Q. and MIRYOOSEFI, S. (2021). Bellman eluder dimension: New rich classes of rl problems, and sample-efficient algorithms. Advances in Neural Information Processing Systems 34 .

JIN, C., LIU, Q. and YU, T. (2022). The power of exploiter: Provable multi-agent rl in large state spaces. In International Conference on Machine Learning . PMLR.

JIN, C., YANG, Z., WANG, Z. and JORDAN, M. I. (2020). Provably efficient reinforcement learning with linear function approximation. In Conference on Learning Theory . PMLR.

JIN, Y., YANG, Z. and WANG, Z. (2021). Is pessimism provably efficient for offline rl? In International Conference on Machine Learning . PMLR.

KONDA, V. and TSITSIKLIS, J. (1999). Actor-critic algorithms. In Advances in Neural Information Processing Systems (S. Solla, T. Leen and K. M¨ uller, eds.), vol. 12. MIT Press.

KUMAR, A., ZHOU, A., TUCKER, G. and LEVINE, S. (2020). Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems 33 1179-1191.

KUMAR, P. and BECKER, A. (1982). A new family of optimal adaptive controllers for markov chains. IEEE Transactions on Automatic Control 27 137-146.

KURUTACH, T., CLAVERA, I., DUAN, Y., TAMAR, A. and ABBEEL, P. (2018). Modelensemble trust-region policy optimization. arXiv preprint arXiv:1802.10592 .

LEE, K., LASKIN, M., SRINIVAS, A. and ABBEEL, P. (2021). Sunrise: A simple unified framework for ensemble learning in deep reinforcement learning. In International Conference on Machine Learning . PMLR.

LIU, Q., CHUNG, A., SZEPESV ´ ARI, C. and JIN, C. (2022). When is partially observable reinforcement learning not scary? arXiv preprint arXiv:2204.08967 .

LIU, Q., YU, T., BAI, Y. and JIN, C. (2020). A sharp analysis of model-based reinforcement learning with self-play. arXiv preprint arXiv:2010.01604 .

LIU, X., HSIEH, P.-C., HUNG, Y. H., BHATTACHARYA, A. and KUMAR, P. (2020). Exploration through reward biasing: Reward-biased maximum likelihood estimation for stochastic multi-armed bandits. In International Conference on Machine Learning . PMLR.

LIU, Z., LU, M., WANG, Z., JORDAN, M. and YANG, Z. (2022). Welfare maximization in competitive equilibrium: Reinforcement learning for markov exchange economy. In International Conference on Machine Learning . PMLR.

LIU, Z., ZHANG, Y., FU, Z., YANG, Z. and WANG, Z. (2022). Learning from demonstration: Provably efficient adversarial policy imitation with linear function approximation. In International Conference on Machine Learning . PMLR.

LOPES, M., LANG, T., TOUSSAINT, M. and OUDEYER, P.-Y. (2012). Exploration in modelbased reinforcement learning by empirically estimating learning progress. Advances in neural information processing systems 25 .

LU, M., MIN, Y., WANG, Z. and YANG, Z. (2022). Pessimism in the face of confounders: Provably efficient offline reinforcement learning in partially observable markov decision processes. arXiv preprint arXiv:2205.13589 .

LU, X. and VAN ROY, B. (2017). Ensemble sampling. Advances in neural information processing systems 30 .

METE, A., SINGH, R. and KUMAR, P. (2022). Augmented rbmle-ucb approach for adaptive control of linear quadratic systems. Advances in Neural Information Processing Systems 35 9302-9314.

METE, A., SINGH, R. and KUMAR, P. (2022). The rbmle method for reinforcement learning. In 2022 56th Annual Conference on Information Sciences and Systems (CISS) . IEEE.

METE, A., SINGH, R., LIU, X. and KUMAR, P. (2021). Reward biased maximum likelihood estimation for reinforcement learning. In Learning for Dynamics and Control . PMLR.

MODI, A., JIANG, N., TEWARI, A. and SINGH, S. (2020). Sample complexity of reinforcement learning using linearly combined model ensembles. In International Conference on Artificial Intelligence and Statistics . PMLR.

MOHAMED, S. and JIMENEZ REZENDE, D. (2015). Variational information maximisation for intrinsically motivated reinforcement learning. Advances in neural information processing systems 28 .

OSBAND, I., BLUNDELL, C., PRITZEL, A. and VAN ROY, B. (2016). Deep exploration via bootstrapped dqn. Advances in neural information processing systems 29 .

PATHAK, D., AGRAWAL, P., EFROS, A. A. and DARRELL, T. (2017). Curiosity-driven exploration by self-supervised prediction. In International conference on machine learning . PMLR.

PUTERMAN, M. L. (2014). Markov decision processes: discrete stochastic dynamic programming . John Wiley & Sons.

RIGTER, M., LACERDA, B. and HAWES, N. (2022). Rambo-rl: Robust adversarial model-based offline reinforcement learning. arXiv preprint arXiv:2204.12581 .

RUSSO, D. and VAN ROY, B. (2013). Eluder dimension and the sample complexity of optimistic exploration. Advances in Neural Information Processing Systems 26 .

STADIE, B. C., LEVINE, S. and ABBEEL, P. (2015). Incentivizing exploration in reinforcement learning with deep predictive models. arXiv preprint arXiv:1507.00814 .

SUN, H., HAN, L., YANG, R., MA, X., GUO, J. and ZHOU, B. (2022). Exploit reward shifting in value-based deep-rl: Optimistic curiosity-based exploration and conservative exploitation via linear reward shaping. Advances in Neural Information Processing Systems 35 37719-37734.

SUN, H., XU, Z., FANG, M. and ZHOU, B. (2022). Supervised q-learning can be a strong baseline for continuous control. In NeurIPS 2022 Foundation Models for Decision Making Workshop .

SUN, W., JIANG, N., KRISHNAMURTHY, A., AGARWAL, A. and LANGFORD, J. (2019). Model-based rl in contextual decision processes: Pac bounds and exponential improvements over model-free approaches. In Conference on learning theory . PMLR.

SUTTON, R. S. (1990). Integrated architectures for learning, planning, and reacting based on approximating dynamic programming. In Machine learning proceedings 1990 . Elsevier, 216-224.

SUTTON, R. S. and BARTO, A. G. (2018). Reinforcement learning: An introduction .

SUTTON, R. S., MCALLESTER, D., SINGH, S. and MANSOUR, Y. (1999). Policy gradient methods for reinforcement learning with function approximation. Advances in neural information processing systems 12 .

TODOROV, E., EREZ, T. and TASSA, Y. (2012). Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ international conference on intelligent robots and systems . IEEE.

WAINWRIGHT, M. J. (2019). High-dimensional statistics: A non-asymptotic viewpoint , vol. 48. Cambridge university press.

WANG, R., SALAKHUTDINOV, R. R. and YANG, L. (2020). Reinforcement learning with general value function approximation: Provably efficient approach via bounded eluder dimension. Advances in Neural Information Processing Systems 33 6123-6135.

WANG, Y., WANG, R., DU, S. S. and KRISHNAMURTHY, A. (2019). Optimism in reinforcement learning with generalized linear function approximation. arXiv preprint arXiv:1912.04136 .

WIERING, M. A. and VAN HASSELT, H. (2008). Ensemble algorithms in reinforcement learning. IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics) 38 930-936.

WU, C., LI, T., ZHANG, Z. and YU, Y. (2022). Bayesian optimistic optimization: Optimistic exploration for model-based reinforcement learning. Advances in Neural Information Processing Systems 35 14210-14223.

XIE, Q., CHEN, Y., WANG, Z. and YANG, Z. (2020). Learning zero-sum simultaneous-move markov games using function approximation and correlated equilibrium. In Conference on learning theory . PMLR.

XIE, T., CHENG, C.-A., JIANG, N., MINEIRO, P. and AGARWAL, A. (2021). Bellmanconsistent pessimism for offline reinforcement learning. Advances in neural information processing systems 34 6683-6694.

XIE, T., FOSTER, D. J., BAI, Y., JIANG, N. and KAKADE, S. M. (2022). The role of coverage in online reinforcement learning. arXiv preprint arXiv:2210.04157 .

XIONG, W., ZHONG, H., SHI, C., SHEN, C. and ZHANG, T. (2022). A self-play posterior sampling algorithm for zero-sum Markov games. In Proceedings of the 39th International Conference on Machine Learning , vol. 162 of Proceedings of Machine Learning Research . PMLR.

YANG, L. and WANG, M. (2019). Sample-optimal parametric q-learning using linearly additive features. In International Conference on Machine Learning . PMLR.

YANG, Z., JIN, C., WANG, Z., WANG, M. and JORDAN, M. (2020). Provably efficient reinforcement learning with kernel and neural function approximations. Advances in Neural Information Processing Systems 33 13903-13916.

YU, T., QUILLEN, D., HE, Z., JULIAN, R., HAUSMAN, K., FINN, C. and LEVINE, S. (2020). Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on robot learning . PMLR.

ZANETTE, A., BRANDFONBRENER, D., BRUNSKILL, E., PIROTTA, M. and LAZARIC, A. (2020). Frequentist regret bounds for randomized least-squares value iteration. In International Conference on Artificial Intelligence and Statistics . PMLR.

ZANETTE, A., LAZARIC, A., KOCHENDERFER, M. and BRUNSKILL, E. (2020). Learning near optimal policies with low inherent bellman error. In International Conference on Machine Learning . PMLR.

ZHA, D., MA, W., YUAN, L., HU, X. and LIU, J. (2021). Rank the episodes: A simple approach for exploration in procedurally-generated environments. arXiv preprint arXiv:2101.08152 .

ZHANG, T. (2022). Feel-good thompson sampling for contextual bandits and reinforcement learning. SIAM Journal on Mathematics of Data Science 4 834-857.

ZHANG, T. (2022). Mathematical analysis of machine learning algorithms .

ZHONG, H., XIONG, W., ZHENG, S., WANG, L., WANG, Z., YANG, Z. and ZHANG, T. (2022). A posterior sampling framework for interactive decision making. arXiv preprint arXiv:2211.01962 .

ZHONG, H. and ZHANG, T. (2023). A theoretical analysis of optimistic proximal policy optimization in linear markov decision processes. arXiv preprint arXiv:2305.08841 .

ZHOU, D., GU, Q. and SZEPESVARI, C. (2021). Nearly minimax optimal reinforcement learning for linear mixture markov decision processes. In Conference on Learning Theory . PMLR.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studied reinforcement learning with general function approximation setting. The authors proposed a new MEX framework, which unifies the exploration and exploitation within a unconstrained optimization objective. Under the structural assumptions about low GEC, they established regret upper bound for learning with their framework. Moreover, they conducted experiments in Mujoco under both model-based and model-free setting and achieved promising results.\n\nSTRENGTHS:\nComputational efficiency is indeed an crucial shortage of existing provably efficient algorithm frameworks. I'm glad to see some effort to investigate how to close those gaps.\n\nThe idea is clean, the proof is sound to me. The experiments look interesting and promising.\n\nWEAKNESSES:\n(1) For the model-free setting, the objective (Eq. 3.1) may not be as easily to optimize as it looks like. By Eq. 3.3, the definition of L_h^k, solving Eq. 3.1 requires to solving a mini-max optimization problem. \nOn the other hand, those constrained optimization objectives in previous literatures (like [1] and [2]) can also be directly converted to a Lagrangian form, which can also be regarded as one objective (a with minimax optimization problem).\nI don't think it is easy to conclude the objective in this paper is indeed easier to implement than the others.\n\n(2) There is not much technique novelty. The proof techniques can be found in previous literatures. The maximization objective is quite straightforward, which just convert the confidence interval constrained optimization objective in previous literatures to a Lagrangian style objective, although there are indeed some difference (here the dual parameter is fixed).\n\n\n[1] Jin et. al., Bellman Eluder Dimension: New Rich Classes of RL Problems, and Sample-Efficient Algorithms\n\n[2] Du et. al., Bilinear Classes: A Structural Framework for Provable Generalization in RL\n\nQUESTIONS:\n(1) for the model-free setting, does there exist a general choice of $l$ function in Assumption 3.1? When Assumption 4.2 is satisfied, does $l$ function always exist?\n\nLIMITATIONS:\nN.A.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis theory paper proposes an algorithmic framework where the \"hypothesis\" (model for MB methods, Q function for MF methods) for each iteration is chosen by maximizing one objective: the hypothesis likelihood (i.e., NLL for MB methods, TD error for MF methods) plus the expected returns (value function at the initial state). The paper provides regret guarantees matching prior work. The paper proposes a practical variant of the method (building on top of MBPO and TD3) that shows some promise on sparse versions of benchmark continuous control tasks (while matching performance on the standard, non-sparse versions).\n\nSTRENGTHS:\n**Originality**: To the best of my knowledge the proposed frame work is original.\n* One prior paper that's related to the model-based version of this paper is https://arxiv.org/abs/2110.02758. This that paper lacks the theoretical contributions of this paper, and comes at the problem from a different perspective, the resulting algorithms look surprisingly similar: both learn a model that is optimized to (1) have high likelihood and (2) produce high-return policies. I'd be curious to see if the current paper also performs well on the stochastic gridworlds used in the prior paper to showcase the benefits of the \"optimistic\" model.\n\n**Clarity**: I generally found the writing clear, which is especially impressive for a large theory project packed into 9 pages. The bottom of this text block has a few suggestions for improving clarity. A few high-level things\n* The related work section is generally well written, including a pretty thorough review of prior work. One suggestion is to explain the differences from prior work. What are the limitations that this paper will address?\n* I didn't get much from reading Section 5; I'd prefer that the space be spent on more intuition for the previous section, or more empirical results.\n\n\n**Significance**: Overall, it seems like the proposed framework is a novel and potentially useful way of designing better RL algorithms. I especially appreciate the simplicity of the proposed approach. I think that the paper may be a bit closer to prior work that the paper makes it out to be (see comment under Originality, and the comment about Gumbel), but it still seems to represent somewhat of a departure from the conventional ways of thinking about RL algorithms. Like most new algorithmic frameworks, it seems like there may be a few kinks that have to be worked out empirically (how to choose $\\eta$; is there a way of implementing the model-free version without the extra CQL term). Nonetheless, it seems like the paper could inspire more work in this direction.\n\n\n**Strengths**\n* The paper is generally well written, includes a thorough review of much prior work.\n* The proposed framework is derived for both model-free and model-based algorithm.\n* An empirical version of the proposed method is applied to to reasonable continuous control tasks. This is really nice to see in a theory paper.\n\nWEAKNESSES:\n* In a few spots, I felt like the paper claims were not entirely substantiated:\n  * \"One Objective to Rule Them All\" in the title -- Without comparisons to \"all\" prior methods, I don't think this is a fair claim. Perhaps a more accurate title would be \"An Objective Fusing Estimation and Planning for Exploration\"\n  * \"algorithms predominantly undertake three tasks\" -- This seems to be describing methods based on Thompson sampling and posterior sampling, but I'm not sure that R-MAX-style methods fit this mold\n  * \"outperforms MBPO by a stable margin ... showcases greater sample efficiency\" -- I'm a bit unsure what these statements are referring to. My read of Figure 1 is that there might be statistically significant gains on 2 / 8 tasks (cheetah-vel-sparse, ant-vel-sparse). I would still vote for accepting the paper even if it only outperforms baselines on 2/8 tasks (but I think that accurately portraying the results is very important); this is especially true if we see large benefits on some tabular settings (see suggestion under Originality).\n* The paper alludes to \"data-dependent level-sets\" in a number of places, but never defines what these are. I'd recommend either including a few sentences describing similarities/differences, or moving this discussion to the appendix.\n* I have a few questions about some of the technical details (see below).\n* From a practical perspective, the model-free version of the method seems to violate conventional wisdom: many practical TD methods (e.g., TD3, MBPO) choose the Q function that has lowest Q-values, whereas the proposed method chooses the one with highest Q values. Not only does this make be a bit concerned that the theoretical framework might not always work well in practice (as evinced by the need for the additional CQL regularizer), but it also makes me wonder exactly how the model-free version of the method is implemented when combined with TD3 (does it take the min or max of the Q functions)? At least for the model-free version, one thing that might help would be to sure that using the CQL regularizer allows the _same_ method to achieve good results in both online and offline settings.\n\n\n---------------------\n\n**Minor comments**:\n* In the abstract, mention that this is going to be a theory paper. I expected this to be an empirical paper until partway through the intro, when I realized that all of the citations were referring to other theory papers. As such, I found the claim that prior methods \"involve impractical algorithmic components\" a bit strange, as prior _practical_ methods cannot require impractical components.\n* L45 -- L47: I found this part a bit unclear because it was unclear what the optimization variable is (a policy, a model, data). This point is clarified later.\n* L50: I found this part a bit unclear because it was unclear what the hypothesis class was over (policies, models, etc).\n* Some small grammar errors throughout. An automated grammar checker should catch these (e.g., dropped articles)\n* L80 -- This definition could be cut, as it's already included a few paragraphs earlier. Cutting this would make it possible to remove some of the whitespace hacks.\n* L101 -- What is \"BE\"?\n* Empirically, in many TD algorithms the TD error _increases_ as the policy improves. The standard explanation is that collecting non-zero rewards results in higher TD errors. This seems potentially related to the analysis in this paper.\n* L260 -- Including a table for comparing these prior methods might help.\n\nQUESTIONS:\n1. How important is the value of $\\eta$ for both the theoretical and empirical results?\n2. L147 \"Without loss of generality\" -- Can the authors elaborate on this point? My guess is that we can ignore stochastic rewards because we could define a new MDP that has a copy of each state for each possible reward that that state could have (and update the transition probabilities accordingly). I'm not sure why it's OK to assume that the reward function is known.\n3. There's a sense in which the proposed method resembles posterior sampling methods, by drawing a connection with Gumbel-softmax-style sampling. One way to sample a random variable is to _deterministically_ choose the value that maximizes likelihood + noise, for an appropriately chosen noise distribution (e.g., the Gumbel distribution for categorical distributions). I'm curious if we can interpret posterior sampling methods as a version of Eq. 3.1 where the first term ($V(x)$) is replaced by an appropriate noise distribution (which has nothing to do with the value). \n4. L206: Does $V_f^\\pi$ make sense for both the model-based and model-free versions of the method? For the model-based version, I'm interpreting this as the expected returns of policy $\\pi$ under model $f$. But for the model-free version, I'm unsure if this is supposed to be the expected returns of $\\pi$, or the estimate from (value function) $f$.\n5. L206, \"reduced to vanilla actor-critic\": Is this true for the model-based version of the method?\n6. Is the second term in Eq. 3.3 required?\n\nLIMITATIONS:\nLimitations are not discussed in the main paper. The appendix has a cursory paragraph on limitations. I would recommend including the limitations in the main text, and including a transparent discussion of benefits/limitations of the proposed method relative to prior methods. For example, the first term in Eq. 3.1 seems to sometimes cause optimization problems. Ensemble-based methods might be more stable.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a unified objective for optimizing regret in online RL algorithms with theoretical justification and experimental evaluation. The objective crucially dos not dependent on posterior estimation or bi-level optimization, making it easier to implement in practice than previous theory motivated algorithms. The authors provide a thorough theoretical analysis of their framework, covering both model-based and model-free approaches.\n\nSTRENGTHS:\nThe paper is very well crafted, and provides an algorithm that is both theoretically sound and empirically impactful, which is always a great achievement. Even though the presentation is necessarily dense at times, given the theoretical nature of the main contribution, the paper is fairly accessible, with the core intuitions mostly presented well. The results and the practical algorithm presented in the empirical section at the end are surprisingly intuitive and seem straightforward to implement on top of several model-based RL algorithms, not just MBPO.\n\nCaveat: I am not necessarily qualified to comment on the correctness of all mathematical details or the impact of the theoretical contributions. I am overall familiar with most concepts used (i.e. Bellman and Eluder rank) but not a core RL theory researcher, so I might be missing some details.\n\nWEAKNESSES:\nGiven the theoretical nature of the paper, the presentation is necessarily dense, which makes it hard to grasp a couple of intuitive concepts. While the authors do their best to present all main components, here are some suggestions for improvement:\n- The authors use generic functions several times, most prominently in Section 3, 3.2 and 3.3. They defer concrete realizations to later, but it would be helpful to provide these examples earlier so that readers unfamiliar with the exact approach have an easier time understanding the role that these function play. (Similarly in Section 5, 5.1)\n- The authors use a unified notation for model-free and model-based hypotheses. While this makes generic statements easier to provide, it led to some confusion on my part because I lost track of whether a model-free or model-based hypothesis was meant at some points. Potentially a specialized notation would clarify these whenever a concrete algorithmic framework is discussed.\n- While I know that this is not always the main stylistic choice in theory papers, I want to encourage the authors to explain why each assumption is used, i.e. what role they play in proofs. This is especially important for 3.1, which is very abstract and hard to relate to anything this early in the presentation.\n\nThe role of the exploration policy is not elaborated on in the concrete implementation, etc. This is promised in line 199, but I cannot find a note on the exploration policy in Section 5.\n\nThe full loss presented in 6.3 does not clarify how the additional term fulfills all the requirements introduced in Section 4. While I was able to piece together most of it, an explicit explanation here would greatly improve the ease with which the role of the assumptions can be understood. In addition, some intuitive description of what this objective optimizes would be nice to have.\n\nThe experimental section is missing a discussion on how $\\eta'$ is set in practice (also, why is $\\eta'$ used instead of $\\eta$?).\n\n211: for seek of -> for the sake of?\n\nQUESTIONS:\nAssumption 4.3 and Proposition 5: Are there requirements on the exploration policy to guarantee that sufficient coverage is obtained for the supervised learning guarantee? I might be misunderstanding the exact details here, but for the supervised guarantee to hold, it feels like some assumptions on the data is needed?\n\nSection 5 (line 279) makes it seem like the choice of $l$ depends on characteristics of the MDP. If so, what aspects of the MDP need to be know a priori to make this choice?\n\nLine 298: Is the choice of the Hellinger distance crucial, or would other choices work here as well?\n\n6.2 If a differentiable model of the environment is learned, would it be possible to obtain the gradient wrt to the value function directly, without resorting to a PG style estimator? This would be directly implementable in MBPO, as the model is fully differentiable using reparametrization.\n\nThe experimental section results miss some standard information: what implementation of MBPO was used (this is also important to obtain hyperparameters), how where the environments chosen (especially why is Humanoid missing), and what is denoted by the shaded area in the plots.\n\nSeveral times \"specific MDP structure\" is mentioned (lines201, 275, 304 [I assume here it refers to the GEC assumption?]), but the authors do not fully clarify what is meant by this. What is meant by this?\n\nLIMITATIONS:\nThe authors do not address the limitations of their work. Potential discussion points are: the reliance on structural assumptions in the MDP are not provided in-depth, and the tuning of the parameter $\\eta$.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an RL framework, maximize to explore (MEX), that requires solving an unconstrained optimization which encompasses estimation, planning, and exploration. It proposes both model-free and model-based algorithms and further extends this framework to zero-sum Markov game setting. All these algorithms are proved to be efficient. It also designs practical algorithms based on existing deep RL baselines. Experimental results show that the proposed algorithms outperform baselines in sparse reward environments.\n\nSTRENGTHS:\nThis paper proposes a general framework that is adapted to various settings while enjoying favorable theoretical guarantee. Its result generalizes previous ones and is important. It is technically sound, clearly written, and well organized.\n\nWEAKNESSES:\nThe idea of maximizing to explore is not novel, and previous work are not adequately cited. This line of research is first developed by [Kumar & Becker (1982)](https://ieeexplore.ieee.org/document/1102878) who proposes an estimation criterion that biases maximum likelihood estimation with the cost (or value) of decision. In recent years, several work ([Liu et al., 2020](https://proceedings.mlr.press/v119/liu20g.html); [Mete et al., 2021](https://proceedings.mlr.press/v144/mete21a.html); [Hung et al., 2021](https://ojs.aaai.org/index.php/AAAI/article/view/16961); [Mete et al., 2022](https://ieeexplore.ieee.org/document/9751189/); [Mete et al., 2022](https://proceedings.neurips.cc/paper_files/paper/2022/hash/3c601cd5866099648c6dc783e7f39858-Abstract-Conference.html); [Wu et al., 2022](https://proceedings.neurips.cc/paper_files/paper/2022/hash/5bcb807ae43ad0851a6ba6162a866404-Abstract-Conference.html)) have applied this method to different settings and proved some theoretical results. In particular, [Wu et al. (2022)](https://proceedings.neurips.cc/paper_files/paper/2022/hash/5bcb807ae43ad0851a6ba6162a866404-Abstract-Conference.html) gives a theoretical result similar to the one for model-based MEX in this paper. Rather than completely ignoring previous efforts, it is more appropriate to propose MEX as a further generalization of previous research. \n\nThe proof of regret bound in this paper relies on the low generalized eluder coefficient assumption. However, it is not discussed whether this assumption hold in practice especially when the hypothesis class is parameterized by neural networks.\n\nQUESTIONS:\nWhile the theory indicates that the coefficient balancing value and evidence should change over time, it is set to be a constant in practice. Why is it the case?\n\nLIMITATIONS:\nThis paper has not discussed its limitations, especially about optimization aspects of the learning objective and the practicality of assumptions.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}