Title: Provably Efficient Offline Goal-Conditioned Reinforcement Learning with General Function Approximation and Single-Policy Concentrability
Venue: NeurIPS 2023
Accepted: yes
Keywords: offline goal-conditioned RL

provably efficient algorithm

single-policy concentrability

general function approximation
Abstract: Goal-conditioned reinforcement learning (GCRL) refers to learning general-purpose skills that aim to reach diverse goals. In particular, offline GCRL only requires purely pre-collected datasets to perform training tasks without additional interactions with the environment. Although offline GCRL has become increasingly prevalent and many previous works have demonstrated its empirical success, the theoretical understanding of efficient offline GCRL algorithms is not well established, especially when the state space is huge and the offline dataset only covers the policy we aim to learn. In this paper, we provide a rigorous theoretical analysis of an existing empirically successful offline GCRL algorithm. We prove that under slight modification, this algorithm enjoys an $\tilde{O}(\text{poly}(1/\epsilon))$ sample complexity (where $\epsilon$ is the desired suboptimality of the learned policy) with general function approximation thanks to the property of (semi-)strong convexity of the objective functions. We only require nearly minimal assumptions on the dataset (single-policy concentrability) and the function class (realizability). Moreover, this algorithm consists of two uninterleaved optimization steps, which we refer to as $V$-learning and policy learning, and is computationally stable since it does not involve minimax optimization. We also empirically validate our theory by showing that the modified algorithm outperforms the previous algorithm in various real-world environments.
To the best of our knowledge, this is the first algorithm that is both provably efficient with general function approximation and single-policy concentrability, and empirically successful without requiring solving minimax optimization problems.

Provably Efficient Offline Goal-Conditioned Reinforcement Learning with General Function Approximation and Single-Policy Concentrability

Abstract

Goal-conditioned reinforcement learning (GCRL) refers to learning generalpurpose skills that aim to reach diverse goals. In particular, offline GCRL only requires purely pre-collected datasets to perform training tasks without additional interactions with the environment. Although offline GCRL has become increasingly prevalent and many previous works have demonstrated its empirical success, the theoretical understanding of efficient offline GCRL algorithms is not well established, especially when the state space is huge and the offline dataset only covers the policy we aim to learn. In this paper, we provide a rigorous theoretical analysis of an existing empirically successful offline GCRL algorithm. We prove that under slight modification, this algorithm enjoys an ˜ O (poly(1 /ϵ )) sample complexity (where ϵ is the desired suboptimality of the learned policy) with general function approximation thanks to the property of (semi-)strong convexity of the objective functions. We only require nearly minimal assumptions on the dataset (single-policy concentrability) and the function class (realizability). Moreover, this algorithm consists of two uninterleaved optimization steps, which we refer to as V -learning and policy learning, and is computationally stable since it does not involve minimax optimization. We also empirically validate our theory by showing that the modified algorithm outperforms the previous algorithm in various realworld environments. To the best of our knowledge, this is the first algorithm that is both provably efficient with general function approximation and single-policy concentrability, and empirically successful without requiring solving minimax optimization problems.

1 Introduction

Goal-conditioned reinforcement learning (GCRL) aims to design agents that are able to learn generalpurpose skills to reach diverse goals [Kaelbling, 1993, Schaul et al., 2015, Plappert et al., 2018]. In particular, offline GCRL learns goal-reaching policies by purely pre-collected data without any further interactions with the environment [Chebotar et al., 2021, Yang et al., 2022]. Since such interaction can be expensive or even unsafe in practice, offline GCRL is increasingly popular as a way to learn generalist agents in real-world environments [Lange et al., 2012, Levine et al., 2020].

Although offline GCRL is promising and achieves great success in various practical scenarios [Lynch et al., 2020, Chebotar et al., 2021, Yang et al., 2022, Ma et al., 2022b,c], designing practical algorithms that are provably efficient still remains an open question. On the practical side, an ideal algorithm should be scalable to huge (or infinite) state spaces and only require minimal dataset coverage assumptions. Moreover, the algorithm should be computationally efficient and stable (e.g., only

using regression-based methods to train policies to avoid unstable minimax optimization). On the theoretical side, we aim to provide finite-sample guarantees of the learned policy.

Unfortunately, most existing algorithms are not both theoretically and practically efficient. On the one hand, many empirically efficient algorithms do not enjoy finite-sample guarantees [Lynch et al., 2020, Chebotar et al., 2021, Yang et al., 2022, Ma et al., 2022c] or even suffer constant suboptimality in favorable settings given infinite data (e.g., Ma et al. [2022c]). On the other hand, although many previous offline RL algorithms with theoretical finite-sample guarantees can be naturally extended to offline GCRL settings, they either cannot handle general value function approximation in the presence of huge (or infinite) state spaces [Jin et al., 2021, Rashidinejad et al., 2021, Yin et al., 2021, Shi et al., 2022, Li et al., 2022], or require impractically strong dataset coverage assumptions, such as all policy concentrability [Antos et al., 2008, Munos and Szepesvári, 2008, Xie and Jiang, 2021b].

Recently, several provably efficient algorithms have been proposed under general function approximation and single-policy concentrability [Zhan et al., 2022, Cheng et al., 2022, Rashidinejad et al., 2022]. In particular, the algorithm of [Zhan et al., 2022], based on the duality form of regularized linear programming formulation of RL, only requires the realizability assumption of function class. However, they all require solving minimax optimization problems which can be difficult or computationally unstable [Daskalakis et al., 2021]. On the contrary, some practically efficient algorithms (e.g., Ma et al. [2022b,c]) do not involve minimax optimization and thus are computationally more stable. This naturally raises an important question:

Can we design an efficient offline GCRL algorithm that enjoys favorable theoretical guarantees under mild assumptions and performs well empirically in real-world scenarios without a minimax formulation?

In this paper, we answer the above question affirmatively by providing rigorous theoretical guarantees for an empirically successful offline GCRL algorithm named GoFAR proposed by Ma et al. [2022c]. We made some slight yet critical modifications to GoFAR. For deterministic MDPs, we need to carefully select the value of one hyperparameter that is set to 1 in the original GoFAR (which can be tuned in practice). For stochastic MDPs, we need to first learn the true transition model via maximum likelihood (MLE) and then plug in the learned model in the algorithm. To distinguish the difference between the original algorithm and the modified ones, we name the modified versions VP-learning (Algorithm 1).

We show that the VP-learning algorithm has both good empirical performance in real-world scenarios (already shown by Ma et al. [2022c], and we compare VP-learning and GoFAR empirically and show that our modification further improves the performance of the previous algorithm GoFAR) and favorable theoretical guarantees under mild assumptions. Specifically, it achieves ˜ O (poly(1 /ϵ )) sample complexity (where ϵ is the desired suboptimality level of the learned policy) under general function approximation with realizability-only assumption and partial data coverage with singlepolicy concentrability assumption. Moreover, the VP-learning algorithm can be decomposed into two uninterleaved learning (optimization) procedures (i.e., V -learning and policy learning), which only require solving regression problems without minimax optimization.

Note that the VP-learning algorithm can be naturally applied to single-task RL settings, and all the analysis in this paper does not rely on whether the setting is goal-conditioned. Since the original algorithm is proposed and empirically validated in goal-conditioned settings, we analyze the algorithm in goal-conditioned settings as well.

1.1 Related Work

Since our algorithm can be naturally applied to single-task offline RL settings, we discuss the related work in a broader scope, which also includes single-task offline RL.

Offline RL in tabular and linear function approximation settings. In tabular and linear settings, a line of work proposed efficient (both statistically and computationally) algorithms under singlepolicy concentrability [Jin et al., 2021, Rashidinejad et al., 2021, Yin et al., 2021, Shi et al., 2022, Li et al., 2022]. These algorithms construct uncertainty quantifiers to ensure pessimism such that policies not well covered by the dataset (which, by single-policy concentrability assumption, are thus suboptimal) suffer a large penalty. Yin and Wang [2021] also consider offline RL with single-policy

concentrability and achieve instance-dependent characterization. However, the above algorithms cannot be directly applied to many practical scenarios when non-linear function approximators are required, since it is hard to obtain uncertainty quantifiers without oracle access when function approximators are non-linear [Jiang and Huang, 2020, Jin et al., 2021, Uehara and Sun, 2021, Xie et al., 2021]. In our algorithm ( V -learning step), we use a regularizer in the form of f -divergence instead of uncertainty quantifiers to ensure pessimism, which makes our algorithm efficient in the presence of non-linear function approximators without additional oracle access.

Offline RL with all-policy concentrability. Besides huge state spaces, another central challenge for offline RL is the lack of dataset coverability. Concentrability, defined as the ratio of occupancy frequency induced by a policy to the dataset distribution, is one of the most widely used definitions to characterize the dataset coverability [Munos, 2007, Scherrer, 2014]. Many previous works require all-policy concentrability to make the algorithm efficient [Szepesvári and Munos, 2005, Munos, 2007, Antos et al., 2007, 2008, Farahmand et al., 2010, Scherrer, 2014, Liu et al., 2019, Chen and Jiang, 2019, Jiang, 2019, Wang et al., 2019, Feng et al., 2019, Liao et al., 2020, Zhang et al., 2020, Uehara et al., 2020, Xie and Jiang, 2021a]. However, in practice, it is unreasonable to require that the offline dataset can cover all candidate policies, and our algorithm only requires single-policy concentrability.

Offline RL with general function approximation and single-policy concentrability. A recent line of work, which is based on marginalized importance sampling (MIS) formulation of RL, has shown success either empirically [Nachum et al., 2019a,b, Lee et al., 2021, Kim et al., 2021] or theoretically [Zhan et al., 2022, Rashidinejad et al., 2022]. In particular, Zhan et al. [2022], Rashidinejad et al. [2022] provide finite-sample guarantees for their algorithms under general function approximation and only single-policy concentrability. Another line of work [Xie et al., 2021, Cheng et al., 2022] also proposes provably efficient algorithms based on an actor-critic formulation under similar assumptions. However, all the above algorithms require solving minimax optimization, which could be difficult or computationally unstable [Daskalakis et al., 2021]. Instead, our algorithm only involves uninterleaved regression-based optimization without minimax optimization.

Offline GCRL. In the context of offline GCRL, the sparsity of the reward is another core challenge [Kaelbling, 1993, Schaul et al., 2015]. Several previous works aim to solve the issue and show empirical success without finite-sample guarantee [Ghosh et al., 2019, Chebotar et al., 2021, Yang et al., 2022, Ma et al., 2022c]. Although there exist theoretical studies of relevant problems such as the offline stochastic shortest path problem [Yin et al., 2022], theoretical understanding of the offline GCRL problem is still lacking. The most relevant work to this paper is Ma et al. [2022c], which shows great performance in several real-world settings without minimax optimization but lacks theoretical guarantee. In this paper, we proved that a slightly modified version of their algorithm (i.e., our VP-learning algorithm shown in Algorithm 1) is provably efficient. Also, we note that Ma et al. [2022a] can be viewed as the single-task version of Ma et al. [2022c], and our algorithm and analysis can be naturally extended to single-task offline RL settings.

2 Preliminaries

Basic Notations. Throughout this paper, we use |X| and ∆( X ) to denote the cardinality and probability simplex of a given set X . We use x ≲ y to denote that there exists a constant c > 0 such that x ≤ cy , use x ≳ y if y ≲ x and use x ≍ y if x ≲ y and y ≲ x . Also, we use the standard O ( · ) notation where f ( n ) = O ( g ( n )) if there exists n 0 , C > 0 such that | f ( n ) |≤ Cg ( n ) for all n ≥ n 0 , and denote f ( n ) = Ω( g ( n )) if g ( n ) = O ( f ( n )) . For any x ∈ R , define x + ≜ max { x, 0 } ; for any general function f : X → R , define f + ( x ) = max { f ( x ) , 0 } , ∀ x ∈ X . Also, for any function f : R → R , define ¯ f ( x ) = f ( x ) -min u ∈ R f ( u ) for any x ∈ R if min u ∈ R f ( u ) exists and further overwrite the notation f + ( x ) = { f ′ ( x ) ≥ 0 } · ¯ f ( x ) where {·} is the indicator function.

✶ ✶ Markov decision process. We consider an infinite-horizon discounted Markov decision process (MDP), which is described by a tuple M = ( S , A , P, R, ρ, γ ) , where S and A denote the state and action spaces respectively, P : S × A → ∆( S ) is the transition kernel, R : S × A → ∆([0 , 1]) encodes a family of reward distributions given state-action pairs with r : S × A → [0 , 1] as the expected reward function, ρ : S → [0 , 1] is the initial state distribution, and γ ∈ [0 , 1) is the discount factor. We assume A is finite while S could be arbitrarily complex (even continuous) as in many

real-world scenarios. A stationary (stochastic) policy π : S → ∆( A ) outputs a distribution over action space for each state.

Goal-conditioned reinforcement learning. In goal-conditioned RL, we additionally assume a goal set G . Similar to Ma et al. [2022c], in goal-conditioned settings, the reward function R ( s ; g ) (as well as the expected reward function r ( s ; g ) ) and policy π ( a | s, g ) also depend on the commanded goal g ∈ G , and the reward no longer depends on the action a and is deterministic.

Each (goal-conditioned) policy π induces a (discounted) occupancy density over state-action pairs for any commanded goal d π : S × A × G → [0 , 1] defined as d π ( s, a ; g ) := (1 -γ ) ∑ ∞ t =0 γ t Pr( s t = s, a t = a ; π ) , where Pr( s t = s, a t = a ; π ) denotes the visitation probability of state-action pair ( s, a ) at step t , starting at s 0 ∼ ρ ( · ) and following π given commanded goal g . We also write d π ( s ; g ) = ∑ a ∈A d π ( s, a ; g ) to denote the marginalized state occupancy. Let p ( g ) be a distribution over desired goals, then we denote d π ( s, a, g ) = d π ( s, a ; g ) p ( g ) and d π ( s, g ) = d π ( s ; g ) p ( g ) . An important property of occupancy density d π is that it satisfies the following Bellman flow constraint:

for all s ∈ S and g ∈ G when letting d = d π for any policy π . Moreover, any d satisfying (1) is the occupancy density of a policy π d where

An important quantity associated with a policy π is the value function, which is the expected discounted cumulative reward defined as V π ( s ; g ) := E [ ∑ ∞ t =0 γ t r t | s 0 = s, a t ∼ π ( ·| s t , g ) ∀ t ≥ 0] starting at state s ∈ S with a commanded goal g ∈ G where r t = R ( s t ; g ) = r ( s t ; g ) . We use the notation J ( π ) := (1 -γ ) E s ∼ ρ,g ∼ p [ V π ( s ; g )] = E ( s,a,g ) ∼ d π [ r ( s ; g )] to represent a scalar summary of the performance of a policy π . We denote by π ∗ the optimal policy that maximizes the above objective and use V ∗ := V π ∗ to denote the optimal value function.

Offline GCRL. In this paper, we focus on offline GCRL, where the agent is only provided with a previously-collected offline dataset D = { ( s i , a i , r i , s ′ i , g i ) } N i =1 . Here, r i ∼ R ( s i ; g i ) , s ′ i ∼ P ( · | s i , a i ) , and we assume that g i are i.i.d. sampled from p ( · ) and it is common that data are collected by a behavior policy µ of which the discounted occupancy density is d µ . Therefore, we assume that ( s i , a i , g i ) are sampled i.i.d. from a distribution µ where µ ( s, a, g ) = p ( g ) d µ ( s, a ; g ) = d µ ( s, a, g ) . Note that we use µ to denote both the behavior policy and the dataset distribution. We also assume an additional dataset D 0 = { ( s 0 ,i , g 0 ,i ) } N 0 i =1 where s 0 ,i are i.i.d. sampled from ρ ( · ) and g 0 ,i are i.i.d. sampled from p ( · ) . The goal of offline RL is to learn a policy ˆ π using the offline dataset so as to minimize the sub-optimality compared to the optimal policy π ∗ , i.e., J ( π ∗ ) -J (ˆ π ) , with high probability.

Function approximation. To deal with huge state spaces, (general) function approximation is necessary for practical scenarios. In this paper, we assume access to two function classes: a value function class V ⊆ { V : S × G → [0 , V max ] } that models the value function of the (regularized) optimal policies, and a policy class Π ⊆ { π : S × G → ∆( A ) } consisting of candidate policies. For stochastic MDP (Section 3.1.2), we also need a transition kernel class P ⊆ { P : S × A → ∆( S ) } which contains the ground-truth transition kernel. Additionally, for any function f : S × G → R , we denote the operator T : R S×G → R S×A×G as ( T f )( s, a ; g ) = E s ′ ∼ P ( ·| s,a ) [ f ( s ′ ; g )] . Also, for any function V : S × G → [0 , V max ] , we define A V ( s, a ; g ) = r ( s ; g ) + γ T V ( s, a ; g ) -V ( s ; g ) .

Offline data coverage assumption. Our algorithm works within the single-policy concentrability framework [Rashidinejad et al., 2021], which is defined as below.

Definition 1 (Single-policy concentrability for GCRL) . Given a policy π , define C π to be the smallest constant that satisfies d π ( s,a,g ) µ ( s,a,g ) ≤ C π for all s ∈ S , a ∈ A and g ∈ G .

The single-policy concentrability parameter C π captures the coverage of policy π in the offline data. Our algorithm only requires this parameter to be small for π ∗ α which is a regularized optimal policy

(see Section 3 for formal definition) and is close to some optimal policy. This assumption is similar to Zhan et al. [2022] and and is much weaker than the widely used all-policy concentrability that assumes bounded C π for all π (e.g., Scherrer [2014]).

3 Algorithms

Offline GCRL can be formulated as the following program:

(3) requires solving an optimization problem over the policy space. One can also optimize over occupancy density d ( s, a ; g ) s.t. d = d π for some policy π which is equivalent to that d satisfies Bellman flow constraint (1). Therefore, the program (3) can be represented equivalently as follows:

Let d ∗ denote the optimal solution of (4), then (one of) the optimal policy can be induced by π ∗ = π d ∗ as in (2). Under partial data coverage assumptions, (4) might fail in empirical settings by choosing a highly suboptimal policy that is not well covered by the dataset with constant probability. Similar to Zhan et al. [2022], Ma et al. [2022c], a regularizer is needed to ensure that the learned policy is well covered by the dataset. Therefore, one should instead solve a regularized version of (4), which is stated as follows:

where the f -divergence is defined as D f ( d ∥ µ ) ≜ E ( s,a,g ) ∼ µ [ f ( d ( s, a, g ) /µ ( s, a, g ))] for a convex function f . Throughout this paper, we choose f ( x ) = 1 2 ( x -1) 2 as in Ma et al. [2022c], where the f -divergence is known as χ 2 -divergence under this specific choice of f and it is shown to be more stable than other divergences such as KL divergence [Ma et al., 2022c]. Let d ∗ α denote the optimal solution of (5), then the regularized optimal policy can be induced by π ∗ α = π d ∗ α as in (2). The following single-policy concentrability assumption assumes that π ∗ α is well covered by the offline dataset.

Assumption 1 (Single-policy concentrability for π ∗ α ) . Let d ∗ α be the optimal solution of (5) , and let π ∗ α = π d ∗ α as defined in (2) . We assume C π ∗ α ≤ C ∗ α where C π ∗ α is defined in Definition 1 and C ∗ α > 0 is a constant.

Under Assumption 1, it can be observed that the performance difference between the regularized optimal policy π ∗ α and the optimal policy π ∗ is bounded by O ( α ) . The following proposition formally presents this observation.

Proposition 3.1. Let d ∗ α be the optimal solution of (5) , and let π ∗ α = π d ∗ α as defined in (2) . Then under Assumption 1, it holds that J ( π ∗ ) -J ( π ∗ α ) ≤ O ( α ( C ∗ α ) 2 ) .

The proof of Proposition 3.1 is deferred to Appendix A.1. Proposition 3.1 shows that by solving the regularized program (5), we can obtain a near-optimal policy as long as α is small. The algorithm of Ma et al. [2022c] also aims to solve (5) and they simply choose α = 1 . We show empirically in Section 5 that α < 1 achieves better performance than α = 1 . In theory, we must carefully choose the value of α s.t. the suboptimality of our learned policy vanishes to 0 with a reasonable rate. Finally, as in Ma et al. [2022c], we convert (5) to the dual form, which is an unconstrained problem and amenable to solve:

Proposition 3.2 (Dual form of (5)) . The duality form of (5) is

where g ∗ is the convex conjugate of g = α · f . Moreover, let V ∗ α denote the optimal solution of (6) , then it holds

for all ( s, a, g ) ∈ S × A × G

The proof of Proposition 3.2 is shown in Appendix A.2. According to the above proposition, one can first learn the V function according to (6), and then use the learned V function to learn the desired policy by (7). We call the first step V -learning and the second step policy learning , which will be discussed in detail in Sections 3.1 and 3.2 respectively. Finally, the main algorithm, which we call VP-learning , is presented in Algorithm 1.

Algorithm 1 VP-learning

1: Input: Dataset D = { ( s i , a i , r i , s ′ i , g i ) } N i =1 , D 0 = { ( s 0 ,i , g 0 ,i ) } N 0 i =1 , value function class V , policy class Π , model class P for stochastic settings.

2: Obtain ˆ U by V-Learning (Algorithm 2 or 3 ).

3: Obtain ˆ π by policy learning (Algorithm 4) using learned function ˆ U .

4: Output: ˆ π .

3.1 V -Learning

Define

However, when the transition kernel is not deterministic, this estimator is biased and will cause an over-estimation issue since g ∗ ( x ) = αf ∗ ( x/α ) = α ( x/α +1) 2 2 -α 2 contains a square operator outside of the Bellman operator (consider estimating ( E [ X ]) 2 using 1 N ∑ X 2 i ).

Therefore, we use the original version of Ma et al. [2022c] for V -learning in deterministic dynamics (Algorithm 2 in Section 3.1.1), and a slightly modified version in stochastic dynamics (Algorithm 3 in Section 3.1.2). For both settings, we assume realizability of V ∗ α on value function class V :

Assumption 2 (Realizability of V ∗ α ) . Assume V ∗ α ∈ V .

3.1.1 V -Learning in Deterministic Dynamics

Whenthe transition kernel P is deterministic, it holds that T V ( s, a ; g ) = V ( s ′ ; g ) where P ( s ′ | s, a ) = 1 . In this case, the natural estimator (9) is unbiased and can be directly applied to the V -learning procedure. The V -learning algorithm for deterministic dynamic settings is presented in Algorithm 2.

Algorithm 2 V -learning in deterministic dynamics

1: Input: Dataset D = { ( s i , a i , r i , s ′ i , g i ) } N i =1 , D 0 = { ( s 0 ,i , g 0 ,i ) } N 0 i =1 , value function class V .

2: V -learning by solving ˆ V = arg min V ∈V ˆ L ( d ) ( V ) where

3: ˆ U ( s, a ; g ) ← r ( s ; g ) + γ ˆ V ( s ′ ; g ) -ˆ V ( s ; g ) + α

4: Output: ˆ V , ˆ U .

Now for any V , we define U V ( s, a ; g ) = r ( s ; g ) + γ T V ( s, a ; g ) -V ( s ; g ) + α = A V ( s, a ; g ) + α which can be interpreted as the advantage function of V with an α -shift. We also denote U ∗ α = U V ∗ α .

Note that besides the learned ˆ V function, Algorithm 2 also outputs a ˆ U function. By (7), one can observe that in policy learning, what we indeed need is ˆ U instead of ˆ V , and thus in the V -learning procedure we also compute this ˆ U function in preparation for policy learning.

One may challenge that ˆ U cannot be computed for all ( s, a ; g ) since we do not have knowledge of all r ( s ; g ) . However, we only need the value of ˆ U ( s i , a i ; g i ) for ( s i , a i ; g i ) contained in the offline dataset, where r i is also contained. Therefore, we can evaluate the value of ˆ U at all ( s, a ; g ) tuples requested in the policy learning algorithm.

Note that Algorithm 2 is equivalent to the first step of Ma et al. [2022c] except for the choice of α and a clip for the value of g ∗ . However, the above V -learning, as well as the original GoFAR algorithm, might suffer the over-estimation issue under stochastic dynamics, and we present algorithms suitable for stochastic dynamics in Section 3.1.2.

3.1.2 V -Learning in Stochastic Dynamics

When the transition kernel is stochastic, one cannot directly use V ( s ′ ; g ) to estimate T V ( s, a ; g ) . Since T V ( s, a ; g ) = E s ′ ∼ P ( ·| s,a ) [ V ( s ′ ; g )] , a natural idea is to learn the ground-truth transition kernel P ⋆ 1 first, and then use the learned transition kernel ˆ P to estimate T V ( s, a ; g ) : ˆ T V ( s, a ; g ) = E s ′ ∼ ˆ P ( ·| s,a ) [ V ( s ′ ; g )] . This is achievable under the following realizability assumption.

Assumption 3 (Realizability of the ground-truth transition model) . Assume the ground-truth transition kernel P ⋆ ∈ P .

The algorithm for stochastic dynamic settings is presented in Algorithm 3, where we first learn the transition kernel ˆ P , and then plug in the learned transition kernel to learn V function. Similar to Algorithm 2, we also compute ˆ U in V -learning procedure.

Algorithm 3 V -learning in stochastic dynamics

1: Input: Dataset D = { ( s i , a i , r i , s ′ i , g i ) } N i =1 , D 0 = { ( s 0 ,i , g 0 ,i ) } N 0 i =1 , value function class V , model class P .

2: Estimate the transition kernel via maximum likelihood estimation (MLE)

3: V -learning using the learned transition kernel: ˆ V = arg min V ∈V ˆ L ( s ) ( V ) with

where ˆ T V ( s, a, ; g ) = E s ′ ∼ ˆ P ( ·| s,a ) [ V ( s ′ ; g )] .

4: ˆ U ( s, a ; g ) ← r ( s ; g ) + γ ˆ T ˆ V ( s, a ; g ) -ˆ V ( s ; g ) + α

ˆ ˆ

5: Output: V , U .

3.2 Policy Learning

We now derive policy learning, the second step of the VP-learning algorithm. Note that π ∗ α = arg max π E ( s,a,g ) ∼ d ∗ α [log π ( a | s, g )] . By (7), we also have

Therefore, π ∗ α = arg max π L MLE α ( π ) where L MLE α ( π ) ≜ E ( s,a,g ) ∼ µ [ U ∗ α ( s,a ; g ) + α log π ( a | s, g ) ] . Since we already learned ˆ U , which is close to U ∗ α , we can use the following estimator for L MLE α ( π ) :

Algorithm 4 Policy learning

1: Input: Dataset D = { ( s i , a i , r i , s ′ i , g i ) } N i =1 , policy class Π , ˆ U learned by Algorithm 2 or 3.

2: Policy learning by:

3: Output: ˆ π .

The policy learning algorithm is presented in Algorithm 4, which can be viewed as a weighted maximum likelihood estimation (MLE) procedure. Finally, we make the following two assumptions on the policy class Π .

Assumption 4 (Single-policy realizability) . Assume π ∗ α ∈ Π .

Assumption 5 (Lower bound of policy) . For any policy π ∈ Π , we assume that π ( a | s, g ) ≥ τ > 0 for any ( s, a, g ) ∈ S × A × G .

Remark 1. One may consider Assumption 5 strong if τ is a constant independent of α or N . However, we allow that τ depends on α . In that case, τ can be extremely small, and any policy mixed with a uniform policy with a tiny probability satisfies this assumption. Therefore, Assumption 5 is mild.

4 Theoretical Guarantees

In this section, we provide theoretical guarantees of our main algorithm (Algorithm 1). We first show the results for V -Learning and policy learning in Section 4.1 and Section 4.2 respectively and then combine them to obtain our main theorem in Section 4.3.

4.1 Analysis of V -Learning

We mainly focus on V -learning in deterministic dynamics in this section. The analysis for stochastic dynamics is similar and presented in Appendix B.2.

As discussed in Section 3.1.1, although the first step of the algorithm is called V -learning, the main goal of this step is to estimate U ∗ α = U V ∗ α accurately. The following lemma provides a theoretical guarantee that the output of V -learning algorithm ˆ U is a good estimator of U ∗ α in positive parts:

Proof sketch. By standard concentration argument, it can be shown that the empirical estimator ˆ L ( d ) in Algorithm 2 (which is unbiased in deterministic dynamics) concentrates well on L α for all V ∈ V (Lemma 3). Therefore, by realizability of V ∗ α , the value of L α at V ∗ α and the learned V -function ˆ V are close (Lemma 4). Finally, one can observe that L α is 'semi-strongly' convex w.r.t. U V + in ∥·∥ 2 ,µ -norm, and thus we can show that ˆ U + and U ∗ α + are also close.

The complete proof of Lemma 1 is deferred to Appendix B.1. In Appendix B.2, we also show the counterpart of Lemma 1 for stochastic dynamic settings.

4.2 Analysis of Policy Learning

After obtaining an accurate estimator ˆ U + of U ∗ α + in the V -Learning procedure, i.e., ∥ ˆ U + -U ∗ α + ∥ 2 ,µ ≲ √ ϵ stat, we can use ˆ U + to perform policy learning and obtain the following guarantee:

Lemma 2 (Closeness of π ∗ α and ˆ π ) . Under Assumptions 4 and 5, with probability at least 1 -δ , the output policy ˆ π of Algorithm 4 satisfies

where ϵ MLE stat is defined in Lemma 8.

The proof of Lemma 2 is provided in Appendix C.2. This result shows that the TV distance between the regularized optimal policy π ∗ α and the output policy ˆ π by Algorithm 4 is small, which translates to a bounded performance difference between these two policies as formalized in Theorem 1.

Theorem 1 (Suboptimality of ˆ π ) . Under Assumptions 4 and 5, with probability at least 1 -δ , the output policy ˆ π of Algorithm 4 satisfies J ( π ∗ α ) -J (ˆ π ) ≤ O ( V max √ ϵ MLE stat /τ 2 ) .

The proof of Theorem 1 is deferred to Appendix C.3.

4.3 Main Theorem: Statistical Rate of Suboptimality

Theorem 1 compares the performance difference between ˆ π and the regularized optimal policy π ∗ α . Since the ultimate goal is to compare with the optimal policy π ∗ , we also need to combine this result with Proposition 3.1. By carefully choosing the value of α to balance J (ˆ π ) -J ( π ∗ α ) and J ( π ) -J ( π ∗ α ) , we can bound the suboptimality of the policy ˆ π output by Algorithm 1 compared to the optimal policy π ∗ , leading to the following main result:

Theorem 2 (Statistical rate of suboptimality (in deterministic dynamics)) . Under Assumptions 1, 2, 4 and 5, with probability at least 1 -δ , the output policy ˆ π by Algorithm 1 (with the choice of Algorithm 2 for V -learning in deterministic dynamics) satisfies

if we choose α ≍ ( V 3 max log(1 /τ ) log( |V|| Π | /δ ) τ 2 ( C ∗ α ) 3 N 1 / 4 ) 1 / 3 and assume N = N 0 .

The proof of Theorem 2 is deferred to Appendix D.1. Note that Theorem 2 provides a suboptimality rate of O (1 /N 1 / 12 ) which implies an O (1 / poly( ϵ )) sample complexity and thus is statistically efficient. A similar rate can also be obtained in stochastic dynamic settings, and we present the result in Appendix D.2. Note that our rate is slightly worse than the O (1 /N 1 / 6 ) rate in Zhan et al. [2022], and worse than the optimal rate O (1 / √ N ) in Rashidinejad et al. [2022]. We briefly discuss the intrinsic difficulty to derive an optimal convergence rate. First, we only require a realizability assumption on our function class, while Rashidinejad et al. [2022] requires a much stronger completeness assumption. Second, our optimization procedure is uninterleaved and only requires solving regression problems, while Zhan et al. [2022] and Rashidinejad et al. [2022] require solving minimax problems. Finally, Rashidinejad et al. [2022] assumes that the behavior policy is known and directly computes the policy using the knowledge of behavior policy, while our algorithm uses a more practical method, i.e., MLE, to solve the policy in the policy learning step.

We also compare our theoretical results to Ma et al. [2022c]. Theorem 4.1 of Ma et al. [2022c] provides a finite-sample guarantee for the suboptimality. However, they compare the performance of ˆ π to π ∗ α (with α = 1 ) instead of π ∗ . Since the performance gap between π ∗ and π ∗ α can be as large as a constant when α = 1 , even zero suboptimality (compared to π ∗ α ) cannot imply that the learned policy has good performance. Moreover, their theoretical analysis assumes that V ∗ can be learned with zero error, which is unreasonable in practical scenarios. We also note that they only provide a proof for deterministic policy classes, which can be restrictive in practice.

5 Experiments

In this section, we provide experimental results of our VP-learning algorithm with different choices of α under five different environments: FetchReach, FetchPick, FetchPush, FetchSlide, and HandReach [Plappert et al., 2018]. Similar to Ma et al. [2022c], the datasets for the five tasks are from Yang et al. [2022]. All the implementation details of our VP-learning are the same as GoFAR (see dataset details and implementation details in Ma et al. [2022c]) 2 , except for the value of α . Note that our VP-learning algorithm with α = 1 is equivalent to the GoFAR algorithm. Table 1 presents the discounted returns and Table 2 presents the final distances of the policies trained after 100 epochs and evaluated over 10 runs. For each environment and each α , the result was averaged over 3 random seeds. The best results of each environment are in bold.

The empirical results demonstrate the correctness of our theoretical analysis: choosing α = 1 will result in a large suboptimality of π ∗ α and thus the learned policy ˆ π . Instead, we should carefully choose the value of α to ensure a vanishing suboptimality. In practice, we can tune the value of α and typically it is less than one. In our experiments, the best α ranges over [0 . 05 , 0 . 5] .

6 Conclusions

In this paper, we theoretically analyze the VP-learning algorithm (Algorithm 1, which is based on the previous empirically successful algorithm in Ma et al. [2022c]) for both single-task and goalconditioned offline settings. This algorithm can deal with general value function approximation and only requires near minimal assumptions on the dataset (single-policy concentrability) and function class (realizability). We also provide an O (1 /N 1 / 12 ) upper bound of the suboptimality of the policy learned by the algorithm and empirically validate its effectiveness.

As for future directions, one important question is whether we can achieve the optimal suboptimality rate ˜ O (1 / √ N ) while keeping the algorithm practical without unreasonably strong assumptions.

References

Alekh Agarwal, Nan Jiang, Sham M Kakade, and Wen Sun. Reinforcement learning: Theory and algorithms. CS Dept., UW Seattle, Seattle, WA, USA, Tech. Rep , pages 10-4, 2019.

Andras Antos, Rémi Munos, and Csaba Szepesvari. Fitted Q-iteration in continuous action-space mdps. In Neural Information Processing Systems , 2007.

András Antos, Csaba Szepesvári, and Rémi Munos. Learning near-optimal policies with bellmanresidual minimization based fitted policy iteration and a single sample path. Machine Learning , 71 (1):89-129, 2008.

Yevgen Chebotar, Karol Hausman, Yao Lu, Ted Xiao, Dmitry Kalashnikov, Jake Varley, Alex Irpan, Benjamin Eysenbach, Ryan Julian, Chelsea Finn, et al. Actionable models: Unsupervised offline reinforcement learning of robotic skills. arXiv preprint arXiv:2104.07749 , 2021.

Jinglin Chen and Nan Jiang. Information-theoretic considerations in batch reinforcement learning. arXiv preprint arXiv:1905.00360 , 2019.

Ching-An Cheng, Tengyang Xie, Nan Jiang, and Alekh Agarwal. Adversarially trained actor critic for offline reinforcement learning. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning , volume 162 of Proceedings of Machine Learning Research , pages 38523878. PMLR, 17-23 Jul 2022. URL https://proceedings.mlr.press/v162/cheng22b. html .

Constantinos Daskalakis, Stratis Skoulakis, and Manolis Zampetakis. The complexity of constrained min-max optimization. In Proceedings of the 53rd Annual ACM SIGACT Symposium on Theory of Computing , pages 1466-1478, 2021.

Amir Massoud Farahmand, Rémi Munos, and Csaba Szepesvári. Error propagation for approximate policy and value iteration. In Advances in Neural Information Processing Systems , 2010.

Yihao Feng, Lihong Li, and Qiang Liu. A kernel loss for solving the Bellman equation. arXiv preprint arXiv:1905.10506 , 2019.

Dibya Ghosh, Abhishek Gupta, Ashwin Reddy, Justin Fu, Coline Devin, Benjamin Eysenbach, and Sergey Levine. Learning to reach goals via iterated supervised learning. arXiv preprint arXiv:1912.06088 , 2019.

Nan Jiang. On value functions and the agent-environment boundary. arXiv preprint arXiv:1905.13341 , 2019.

Nan Jiang and Jiawei Huang. Minimax value interval for off-policy evaluation and policy optimization. Advances in Neural Information Processing Systems , 33, 2020.

Ying Jin, Zhuoran Yang, and Zhaoran Wang. Is pessimism provably efficient for offline RL? In International Conference on Machine Learning , pages 5084-5096. PMLR, 2021.

Leslie Pack Kaelbling. Learning to achieve goals. In IJCAI , volume 2, pages 1094-8. Citeseer, 1993.

Geon-Hyeong Kim, Seokin Seo, Jongmin Lee, Wonseok Jeon, HyeongJoo Hwang, Hongseok Yang, and Kee-Eung Kim. Demodice: Offline imitation learning with supplementary imperfect demonstrations. In International Conference on Learning Representations , 2021.

Sascha Lange, Thomas Gabel, and Martin Riedmiller. Batch reinforcement learning. In Reinforcement learning , pages 45-73. Springer, 2012.

Jongmin Lee, Wonseok Jeon, Byungjun Lee, Joelle Pineau, and Kee-Eung Kim. Optidice: Offline policy optimization via stationary distribution correction estimation. In International Conference on Machine Learning , pages 6120-6130. PMLR, 2021.

Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643 , 2020.

Gen Li, Laixi Shi, Yuxin Chen, Yuejie Chi, and Yuting Wei. Settling the sample complexity of model-based offline reinforcement learning. arXiv preprint arXiv:2204.05275 , 2022.

Peng Liao, Zhengling Qi, and Susan Murphy. Batch policy learning in average reward Markov decision processes. arXiv preprint arXiv:2007.11771 , 2020.

Boyi Liu, Qi Cai, Zhuoran Yang, and Zhaoran Wang. Neural trust region/proximal policy optimization attains globally optimal policy. In Neural Information Processing Systems , 2019.

Corey Lynch, Mohi Khansari, Ted Xiao, Vikash Kumar, Jonathan Tompson, Sergey Levine, and Pierre Sermanet. Learning latent plans from play. In Conference on robot learning , pages 1113-1132. PMLR, 2020.

Yecheng Jason Ma, Andrew Shen, Dinesh Jayaraman, and Osbert Bastani. Smodice: Versatile offline imitation learning via state occupancy matching. arXiv preprint arXiv:2202.02433 , 2022a.

Yecheng Jason Ma, Shagun Sodhani, Dinesh Jayaraman, Osbert Bastani, Vikash Kumar, and Amy Zhang. Vip: Towards universal visual reward and representation via value-implicit pre-training. arXiv preprint arXiv:2210.00030 , 2022b.

Yecheng Jason Ma, Jason Yan, Dinesh Jayaraman, and Osbert Bastani. How far i'll go: Offline goalconditioned reinforcement learning via f -advantage regression. arXiv preprint arXiv:2206.03023 , 2022c.

Rémi Munos. Performance bounds in ℓ p -norm for approximate value iteration. SIAM journal on control and optimization , 46(2):541-561, 2007.

Rémi Munos and Csaba Szepesvári. Finite-time bounds for fitted value iteration. Journal of Machine Learning Research , 9(5), 2008.

Ofir Nachum, Yinlam Chow, Bo Dai, and Lihong Li. Dualdice: Behavior-agnostic estimation of discounted stationary distribution corrections. Advances in Neural Information Processing Systems , 32, 2019a.

Ofir Nachum, Bo Dai, Ilya Kostrikov, Yinlam Chow, Lihong Li, and Dale Schuurmans. Algaedice: Policy gradient from arbitrary experience. arXiv preprint arXiv:1912.02074 , 2019b.

Matthias Plappert, Marcin Andrychowicz, Alex Ray, Bob McGrew, Bowen Baker, Glenn Powell, Jonas Schneider, Josh Tobin, Maciek Chociej, Peter Welinder, et al. Multi-goal reinforcement learning: Challenging robotics environments and request for research. arXiv preprint arXiv:1802.09464 , 2018.

Paria Rashidinejad, Banghua Zhu, Cong Ma, Jiantao Jiao, and Stuart Russell. Bridging offline reinforcement learning and imitation learning: A tale of pessimism. Advances in Neural Information Processing Systems , 34:11702-11716, 2021.

Paria Rashidinejad, Hanlin Zhu, Kunhe Yang, Stuart Russell, and Jiantao Jiao. Optimal conservative offline rl with general function approximation via augmented lagrangian. arXiv preprint arXiv:2211.00716 , 2022.

Tom Schaul, Daniel Horgan, Karol Gregor, and David Silver. Universal value function approximators. In International conference on machine learning , pages 1312-1320. PMLR, 2015.

Bruno Scherrer. Approximate policy iteration schemes: A comparison. In International Conference on Machine Learning , pages 1314-1322, 2014.

Laixi Shi, Gen Li, Yuting Wei, Yuxin Chen, and Yuejie Chi. Pessimistic Q-learning for offline reinforcement learning: Towards optimal sample complexity. arXiv preprint arXiv:2202.13890 , 2022.

Csaba Szepesvári and Rémi Munos. Finite time bounds for sampling based fitted value iteration. In Proceedings of the 22nd international conference on Machine learning , pages 880-887, 2005.

Masatoshi Uehara and Wen Sun. Pessimistic model-based offline reinforcement learning under partial coverage. In International Conference on Learning Representations , 2021.

Masatoshi Uehara, Jiawei Huang, and Nan Jiang. Minimax weight and Q-function learning for offpolicy evaluation. In International Conference on Machine Learning , pages 9659-9668. PMLR, 2020.

Sara Van de Geer. Empirical Processes in M-estimation , volume 6. Cambridge university press, 2000.

Lingxiao Wang, Qi Cai, Zhuoran Yang, and Zhaoran Wang. Neural policy gradient methods: Global optimality and rates of convergence. In International Conference on Learning Representations , 2019.

Tengyang Xie and Nan Jiang. Batch value-function approximation with only realizability. In International Conference on Machine Learning , pages 11404-11413. PMLR, 2021a.

Tengyang Xie and Nan Jiang. Batch value-function approximation with only realizability. In International Conference on Machine Learning , pages 11404-11413. PMLR, 2021b.

Tengyang Xie, Ching-An Cheng, Nan Jiang, Paul Mineiro, and Alekh Agarwal. Bellman-consistent pessimism for offline reinforcement learning. Advances in neural information processing systems , 34:6683-6694, 2021.

Rui Yang, Yiming Lu, Wenzhe Li, Hao Sun, Meng Fang, Yali Du, Xiu Li, Lei Han, and Chongjie Zhang. Rethinking goal-conditioned supervised learning and its connection to offline rl. arXiv preprint arXiv:2202.04478 , 2022.

Ming Yin and Yu-Xiang Wang. Towards instance-optimal offline reinforcement learning with pessimism. Advances in neural information processing systems , 34:4065-4078, 2021.

Ming Yin, Yu Bai, and Yu-Xiang Wang. Near-optimal offline reinforcement learning via double variance reduction. arXiv preprint arXiv:2102.01748 , 2021.

Ming Yin, Wenjing Chen, Mengdi Wang, and Yu-Xiang Wang. Offline stochastic shortest path: Learning, evaluation and towards optimality. In Uncertainty in Artificial Intelligence , pages 2278-2288. PMLR, 2022.

Wenhao Zhan, Baihe Huang, Audrey Huang, Nan Jiang, and Jason Lee. Offline reinforcement learning with realizability and single-policy concentrability. In Conference on Learning Theory , pages 2730-2775. PMLR, 2022.

Junyu Zhang, Alec Koppel, Amrit Singh Bedi, Csaba Szepesvari, and Mengdi Wang. Variational policy gradient method for reinforcement learning with general utilities. arXiv preprint arXiv:2007.02151 , 2020.

A Missing Proofs for Propositions in Section 3

A.1 Proof of Proposition 3.1

Proof. Since both d ∗ α and d ∗ satisfy the Bellman flow constraint (1), due to the optimality of d ∗ α in the regularized program (5), we have

Therefore,

A.2 Proof of Proposition 3.2

Proof. By Lagrangian duality, we can obtain that the dual problem is

where p ( g ) V ( s ; g ) is the Lagrangian vector. Similar to Ma et al. [2022c],

Then (15) is equivalent to

which can be further represented as

where g = α · f . Combining the constraint that d ( s, a ; g ) ≥ 0 and the proof of Proposition 4.3 (Proposition B.2) of Ma et al. [2022c], the above program is equivalent to

B Proof for V -Learning

In this section, we provide theoretical analysis for the guarantees of V -learning in different settings (deterministic dynamics in Appendix B.1 and stochastic dynamics in Appendix B.2).

Recall that we choose f ( x ) = ( x -1) 2 2 , and thus f ∗ ( x ) = ( x +1) 2 2 -1 2 . Therefore, we have g ( x ) = αf ( x ) = α ( x -1) 2 2 and g ∗ ( x ) = α ( x/α +1) 2 2 -α 2 . Also, we define g ∗ , max = max v ∈ [ -V max ,V max +1] g ∗ ( v ) , g ∗ , min = min v ∈ [ -V max ,V max +1] g ∗ ( v ) and g ∗ , ∆ = g ∗ , max -g ∗ , min . We can easily obtain that g ∗ , ∆ ≤ α (1 + V max α ) 2 = O ( V 2 max /α ) .

For the purpose of theoretical analysis, we further define

and

Note that L α ( V ) = L 1 ( V ) + L 2 ( V ) and ˆ L ( d ) ( V ) = ˆ L ( d ) 1 ( V ) + ˆ L ( d ) 2 ( V ) .

B.1 Proof for Deterministic Dynamics

We first show that with high probability, the estimator ˆ L ( d ) ( V ) in Algorithm 2 concentrates well on L α ( V ) for all V ∈ V .

Lemma 3 (Concentration of ˆ L ( d ) 1 , ˆ L ( d ) 2 ) . Under Assumptions 1 and 2, when the dynamic of the environment is deterministic, with probability at least 1 -δ , ∀ V ∈ V , it simultaneously holds that

which immediately implies

Proof. First, fix any V ∈ V . The expectation of ˆ L ( d ) 1 ( V ) is

Also,

Also note that V ( s 0 ,i ; g 0 ,i ) ∈ [0 , V max ] and g ∗ + ( r ( s i ; g i ) + γ T V ( s i , a i ; g i ) -V ( s i ; g i )) ∈ [0 , g ∗ , ∆ ] . By Hoeffding's inequality and a union bound, we have with probability at least 1 -δ ,

Applying a union bound over all V ∈ V concludes the result.

Next, we show that the value of L α at the regularized optimal V -function V ∗ α and the learned function ˆ V is close.

Lemma 4 (Closeness of the population objective of ˆ V and V ∗ α ) . Under Assumptions 1 and 2, with probability at least 1 -δ , we have

Proof. We condition on the high probability event in Lemma 3. Note that

(1) , (3) ≤ O ( ϵ stat ) by Lemma 3 and (2) ≤ 0 by the definition of ˆ V , which completes the proof.

Now we define 'semi-strong' convexity, which is used to prove Lemma 1 and might be of independent interests.

Definition 2 (Positive semi-strong convexity) . For an arbitrary set X , assume F ⊂ { f | f : X → R } is a convex set. Let ρ ∈ ∆( X ) be a probability distribution over X . The ∥·∥ 2 ,ρ -norm of f ∈ F is defined as ∥ f ∥ 2 ,ρ = √ E x ∼ ρ [ f 2 ( x )] . Define function l : R → R and define functional L : F → R as

We say L is σ -positive-semi-strongly convex w.r.t. f in ∥·∥ 2 ,ρ -norm for some σ > 0 if

is convex w.r.t. f in ∥·∥ 2 ,ρ -norm.

Note that the common σ -strongly convex definition requires L ( f ) -σ 2 ∥ f ∥ 2 2 ,ρ to be convex. Now we prove the following property for σ -positive-semi-strong convexity.

Proposition B.1. Under the same setting of Definition 2, if L is σ -positive-semi-strongly convex w.r.t. f in ∥·∥ 2 ,ρ -norm, it holds that for any f, g ∈ F ,

Proof. By Definition 2, ˜ L ( f ) ≜ L ( f ) -σ 2 ∥ f + ∥ 2 2 ,ρ is convex, which, by the definition of convex, implies that

Plugging in the definition of ˜ L ( · ) , we have

Rearranging, we can obtain that

It suffices to show that

Note that for any x ∈ X , g + ( x ) ≥ 0 and f + ( x ) ≥ f ( x ) , which implies g + ( x )( f + ( x ) -f ( x )) ≥ 0 . Also, note that when g ( x ) ≥ 0 , we have g ( x ) -g + ( x ) = 0 , and when g ( x ) < 0 , we have g + ( x ) = 0 , which means g + ( x )( g ( x ) -g + ( x )) ≡ 0 . Therefore, we have

Finally, by semi-strong convexity of L α w.r.t. U V in ∥·∥ 2 ,µ -norm, we can show that ˆ U and U ∗ α are also close.

Proof of Lemma 1. Wecondition on the high probability event in Lemma 3. Recall that U V ( s, a ; g ) = r ( s ; g ) + γ T V ( s, a ; g ) -V ( s ; g ) + α and in deterministic dynamics, ˆ U = U ˆ V . Let U = { U V : V ∈ R |S|×|G| + } ⊆ R |S|×|A|×|G| which is a convex set by definition. Also, since U V is linear in V , we can also obtain that V can be linearly represented by U . Therefore, we can define that ˜ L α ( U V ) = L α ( V ) and ˜ L α ( U V ) -1 2 E ( s,a,g ) ∼ µ [ U 2 V ( s, a, g ) + ] is linear and thus convex in U V , which implies that ˜ L α ( U V ) is 1 -positive-semi-strongly convex w.r.t. U V and ∥·∥ 2 ,µ . Then, by Proposition B.1, we have

Since U ∗ α = arg min U ∈U ˜ L α ( U ) , by the first order optimality condition, it holds that ∇ ˜ L α ( U ∗ α ) T ( ˆ U -U ∗ α ) ≥ 0 . Combining Lemma 4, we have

B.2 Proof for Stochastic Dynamics

In stochastic dynamic settings, we first learn the ground-truth transition model and then calculate ˆ U . The following lemma provides a theoretical guarantee for maximum likelihood estimation (MLE) that ˆ P and P ⋆ are close.

Lemma 5 (Convergence rate of MLE, Van de Geer [2000]) . For any fixed δ > 0 , with probability at least 1 -δ , we have

where ˆ P is defined as in (11) . This immediately implies that

Equipped with Lemma 5, we can guarantee that for any V , the population objective L α ( V ) is close to the empirical objective ˆ L ( s ) ( V ) , which is presented in Lemma 6.

Lemma 6 (Concentration of ˆ L ( s ) ) . Under Assumptions 1 to 3, with probability at least 1 -δ , ∀ V ∈ V , it holds that

Proof. For convenience, define

Note that ˆ L ( s ) 1 ( V ) = ˆ L ( d ) 1 ( V ) and thus by Lemma 3 we have that with probability at least 1 -δ , for all V ∈ V ,

Now define

Since E ( s,a,g ) ∼ µ [ ˆ L ( s ) 2 ( V )] = ˜ L ( s ) 2 ( V ) , by Lemma 3, we have that with probability 1 -δ , for any V ∈ V , it holds that

Also, note that

By Lemma 5, with probability at least 1 -δ , for all V ∈ V ,

By rescaling δ and applying a union bound, we can obtain that with probability at least 1 -δ , (16), (17), (18) hold simultaneously. The conclusion holds by applying the triangle inequality.

Finally, we show that ˆ U + and U ∗ α + are close.

Lemma 7 (Closeness of ˆ U + and U ∗ α + in stochastic dynamics) . Under Assumptions 1 to 3, with probability at least 1 -δ ,

where ˆ U is the output of Algorithm 3, α = Ω ( log( |V||P| /δ ) √ N ) and

Proof. For convenience, let ˜ U = U ˆ V . Following the same analysis as in the deterministic case, we have ∥ U ∗ α + -˜ U + ∥ 2 ,µ ≤ O ( √ ϵ stochastic stat ) by Lemma 6. Also,

where the last inequality holds by Lemma 5. Therefore,

C Proof for Policy Learning

We provide a theoretical analysis of policy learning in this section. We first show two key lemmas in Appendix C.1, and then provide missing proofs of the main text in Appendix C.2 and Appendix C.3.

C.1 Key Lemmas

Lemma 8 (Statistical error of the weighted MLE objective) . Under Assumption 5, with probability at least 1 -δ , for any policy π ∈ Π , it holds that

Proof. For convenience, we assume that the offline dataset used in policy learning is independent of the dataset used in V -Learning. This can be easily achieved by splitting the original dataset D uniformly at random into two datasets of equal size. Also, for analysis, we define

Now we condition on the high probability event in Lemma 3, which we denote by E 1 . Then by Lemma 1, we have ∥ ∥ ∥ ˆ U + α -U ∗ α + α ∥ ∥ ∥ 2 ,µ ≤ ϵ U . By Cauchy-Schwarz inequality, we can obtain that for

any policy π ,

Also, for any fixed π ∈ Π , since E ( s,a,g ) ∼ µ [ ˆ L MLE ( π )] = ˜ L MLE ( π ) , we can obtain by Bernstein's inequality that with probability at least 1 -δ ,

Since U ∗ α ( s, a ; g ) + /α = d ∗ α ( s, a ; g ) /µ ( s, a ; g ) ≤ C ∗ α , we have ∥ U ∗ α + /α ∥ 2 ,µ ≤ C ∗ α and thus ∥ ˆ U + /α ∥ 2 ,µ ≤ ∥ U ∗ α + /α ∥ 2 ,µ + ϵ U ≤ C ∗ α + ϵ U ≤ O ( C ∗ α ) . Also,

Applying a union bound over all π ∈ Π , it holds that with probability at least 1 -δ , for all π ∈ Π

We denote the above event by E 2 . When E 1 and E 2 hold simultaneously, we have by the triangle inequality that

The conclusion holds by rescaling δ .

Lemma 9 (Closeness of MLE objective of π ∗ α and ˆ π ) . Under Assumption 4, with probability at least 1 -δ ,

Proof. We condition on the high probability event in Lemma 8. Note that

(1) , (3) ≤ O ( ϵ MLE stat ) by Lemma 8 and (2) ≤ 0 by the optimality of ˆ π in empirical MLE objective, which completes the proof.

C.2 Proof of Lemma 2

Proof of Lemma 2. We condition on the high probability event in Lemma 8. Note that

We also define

and note that L MLE α, Rel is a constant shift of L MLE α . For any π ∈ Π , we further define r π = π/π ∗ α . By Assumption 5, r π ( a | s, g ) ∈ [ τ, 1 /τ ] . Let ˜ L MLE α, Rel ( r π ) = E ( s,a,g ) ∼ d ∗ α [log r π ( a | s, g )] which is 2 τ 2 -strongly concave w.r.t. ∥·∥ 2 ,d ∗ α when r π ( a | s, g ) ∈ [ τ, 1 /τ ] . Since π ∗ α is the maximizer of L MLE α and thus the maximizer of L MLE α, Rel , we have that r π ∗ α is the maximizer of ˜ L MLE α, Rel . By strong concavity and the optimality of r π ∗ α , we have

where the last inequality holds by Lemma 9. Note that

where the last inequality holds since TV distance is upper bounded by χ 2 distance. Therefore, we can finally obtain that

C.3 Proof of Theorem 1

Proof of Theorem 1. We condition on the high probability event in Lemma 8. By performance difference lemma [Agarwal et al., 2019], we have

where the last inequality holds by Lemma 2.

D Statistical Rate of the Suboptimality in Different Settings

In this section, we analyze the statistical rate of the suboptimality of the output policy ˆ π by Algorithm 1 in different settings.

D.1 Deterministic Settings

Proof of Theorem 2. By Theorem 1 and Proposition 3.1,

Since ϵ stat ≍ V 2 max √ log( |V| /δ ) N , we can further obtain that

J

(

π

∗

)

-

J

(ˆ

π

)

D.2 Stochastic Settings

Theorem 3 (Statistical rate of the suboptimality in stochastic settings) . Under Assumptions 1 to 5, with probability at least 1 -δ , the output policy ˆ π by Algorithm 1 (with the choice of Algorithm 3 for V -learning in stochastic settings) satisfies

if we choose α ≍ ( V 3 max log(1 /τ ) log( |V||P|| Π | /δ ) τ 2 ( C ∗ α ) 3 N 1 / 4 ) 1 / 3 and assume N = N 0 .

Proof. The proof is identical to the proof of Theorem 2 except that we replace ϵ stat with ϵ stochastic stat .

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes the VP-learning algorithm to solve offline goal-conditioned RL in the context of general function approximations. The algorithm is based on a previous empirically successful algorithm proposed by [1], and the author proves the finite sample complexity for VP-learning under mild assumptions. The proposed  algorithm avoids minimax learning by using a duality formulation first identified by [1], which makes the proposed VP-learning algorithm friendly to implementation while also enjoying theoretical guarantees.\n\n**References:**\n\n[1] Ma J Y, Yan J, Jayaraman D, et al. Offline goal-conditioned reinforcement learning via $ f $-advantage regression[J]. Advances in Neural Information Processing Systems, 2022, 35: 310-323.\n\nSTRENGTHS:\n- The VP-learning algorithm proposed in the paper enjoys finite sample guarantees under a mild single policy concentrability assumption. Compared with previous works on offline RL (single-task) algorithms with general function approximations, VP-learning does not involve minimax optimization, potentially making the algorithm more easy and stable to implement.\n\nWEAKNESSES:\n- This theoretical paper aims to study the problem of offline goal-conditioned RL. But it seems that the whole theory does not essentially rely on the setting of goal-conditioned RL. The existing theoretical works this paper compares with are also for standard offline RL, which makes the motivation of the paper confusing. It's true that the standard offline RL can be interpreted as a special case of goal conditioned RL, but I hope to see more intuitions and messages on the problem of goal-conditioned RL itself.\n- The key path through which the VP-learning algorithm can avoid solving minimax optimization problems is based on [1], which makes the theoretical contributions and insights of the paper weakened.\n\nQUESTIONS:\n- Regarding Assumption 4 (single policy realizability), what is the $\\alpha$ in the equation $\\pi_{\\alpha}^{\\star}\\in\\Pi$? Since in Theorem 2 and 3 this assumption is imposed before the choice of $\\alpha$, the $\\alpha$ in the assumption seems undetermined. I think this needs clarification since the paper assumes that the policy class $\\Pi$ is finite, which necessitates a specific choice of $\\pi_{\\alpha}^{\\star}$.\n- Regarding Assumption 5 (lower bound of policy), this seems a quite strong assumption which is not needed by previous single task offline RL algorithms with general function approximations, e.g., [2, 3, 4]. The author argues that the parameter $\\tau$ can be very small, but a small $\\tau$ will also increase the upper bound of the suboptimality of VP-learning due to Theorem 2&3. Is this assumption actually  necessary for offline goal-conditioned RL? or is it only required by the specific analysis for VP-learning algorithm?\n\n**References:**\n\n[2] Zhan W, Huang B, Huang A, et al. Offline reinforcement learning with realizability and single-policy concentrability[C]//Conference on Learning Theory. PMLR, 2022: 2730-2775.\n\n[3] Uehara M, Sun W. Pessimistic model-based offline reinforcement learning under partial coverage[J]. arXiv preprint arXiv:2107.06226, 2021.\n\n[4] Xie T, Cheng C A, Jiang N, et al. Bellman-consistent pessimism for offline reinforcement learning[J]. Advances in neural information processing systems, 2021, 34: 6683-6694.\n\nLIMITATIONS:\nThe above questions are potential limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper provides a rigorous theoretical analysis to a modified version of existing offline Goal-conditioned RL algorithm, and proves that it has an $\\tilde O (poly(1/\\epsilon))$ sample complexity, where $\\epsilon$ is the desired suboptimality of the learned policy. The algorithm requires minimal assumptions on the dataset and the function class and does not involve minimax optimization.\n\nSTRENGTHS:\nThe paper makes a novel contribution by providing a theoretical analysis of an existing offline GCRL algorithm. \n\nThe paper is clearly written and well-organized. The theoretical analysis is well-explained.\n\nWEAKNESSES:\nThe derivation of this paper is flawed due to a mistake in Proposition 3.2. This mistake leads to the conclusion that strong duality holds, and we can recover the optimal policy by solving the dual problem. However, it can be seen that, at the limit of $\\alpha\\to0$, the dual problem (Equation (6)) solves a value function that evaluates the behavior policy. Therefore, solving the dual problem cannot recover the optimal value from suboptimal data, and strong duality does not hold. As a result, all subsequent derivation is invalid, and the claims are unsupported.\n\nThe mistake in Proposition 3.2 stems from a citation of Proposition 4.2 of [1] which incorrectly establishes strong duality.\n\nOverall, I think the paper would have been a valuable contribution to the field of offline RL. However, the flawed derivation is a serious issue that needs to be addressed.\n\n[1] Ma, Jason Yecheng, et al. \"Offline goal-conditioned reinforcement learning via $f$-advantage regression.\" Advances in Neural Information Processing Systems 35 (2022): 310-323.\n\n\n===Post-rebuttal Update===\n\nPreviously raised concerns have been effectively addressed. Overall, I am optimistic that, with some revisions, this paper could meet the standards for acceptance. I am pleased to raise my score from 3 to 6 and look forward to seeing your further refinement of this paper.\n\nQUESTIONS:\nCan you explain more on the raised issue?\n\nLIMITATIONS:\nThe authors of the paper have discussed the limitations of their work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper aims at improving the theoretical understanding of offline goal-conditioned RL (GCRL). In particular, this paper modifies an existing offline GCRL algorithm and shows an O^˜ (poly(1/ϵ)) sample complexity under minimal assumptions of single-policy concentrability and realizability. Their algorithm, called VP-learning, consists of two uninterleaved optimization steps and has good empirical performance while retaining computational stability. Moreover, it can also be applied in non-goal-conditioned settings.\n\nThere seems to be a theory-practice gap that this paper addresses. Namely, most provably efficient algorithms seems to require minimax optimization, while in practice that is not effective. Ideally, an algorithm should be practical with good sample complexity guarantees, which is a gap this paper aims to fill.\n\nIn particular this paper provides guarantees for a modified version of an algorithm GoFAR, with modifications in the deterministic and stochastic MDP settings. They call their modified algorithm VP-learning. While other algorithms have been shown to be efficient under single-policy concentrability and realizability assumptions, they require solving minimax optimization problems.\n\nSTRENGTHS:\nWhile other algorithms have been shown to be efficient under single-policy concentrability and realizability assumptions, they require solving minimax optimization problems. This algorithm does not. \n\nThe algorithm they develop is built off of an algorithm with good empirical performance.\n\nWEAKNESSES:\nWhile part of the pitch of the paper is that they desire an algorithm that has provable efficiency whilst also having good empirical performance, they do not have empirical results to demonstrate that their modified algorithm performs well.\n\nQUESTIONS:\nHow do you expect the modified algorithm, VP-learning to compare to GoFAR in terms of empirical performance? How do you expect the modifications will impact things adversely?\n\nLIMITATIONS:\nTo my knowledge, the authors do adequately address the limitations of the paper.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 1, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

comments: SUMMARY:
This paper establishes a rigorous theoretical analysis for offline goal-conditioned reinforcement learning algorithms (GCRL). To achieve that, the authors made a slight modification on top of an existing offline GCRL algorithm (GoFAR), achieve a polynomial sample complexity by regression instead minimax optimization.

STRENGTHS:
1. This paper is well-organized and provides a throughout survey of related work.
2. It seems to achieve an effective and reasonable improvement from the lens of theory.

WEAKNESSES:
There is no experiment to support the correctness of the theoretical analysis.

QUESTIONS:
N/A

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06