MoMA: Model-based Mirror Ascent for Offline Reinforcement Learning
Authors: Mao Hong
Zhiyue Zhang
Yue Wu
Yanxun Xu
Venue: Accepted by TMLR ()

Abstract: Model-based offline reinforcement learning methods (RL) have achieved state-of-the-art performance in many decision-making problems thanks to their sample efficiency and generalizability. Despite these advancements, existing model-based offline RL approaches either focus on theoretical studies without developing practical algorithms or rely on a restricted parametric policy space,  thus not fully leveraging the advantages of an unrestricted policy space inherent to model-based methods. To address this limitation, we develop MoMA, a model-based mirror ascent algorithm with general function approximations under partial coverage of offline data. MoMA distinguishes itself from existing literature by employing an unrestricted policy class. In each iteration, MoMA conservatively estimates the value function by a minimization procedure within a confidence set of transition models in the policy evaluation step, then updates the policy with general function approximations instead of commonly-used parametric policy classes in the policy improvement step. Under some mild assumptions, we establish theoretical guarantees for MoMA by proving an upper bound on the suboptimality of the returned policy.  
We also provide a practically implementable, approximate version of the algorithm. The effectiveness of MoMA is demonstrated via numerical studies.

Published in Transactions on Machine Learning Research (10/2024)
MoMA: Model-based Mirror Ascent for Offline Reinforcement Learning
Mao Hong
mhong26@tjhu.ed edu
Department of Applied Mathematics and Statistics
Johns Hopkins University
Zhiyue Zhang
zahan 79 edu
Deportment Applied Mathematics and Statistics Johns Hopkins University
Yue Wu
yuml6608jhu..om edu
Department of Applied Mathematics and Statistics Johns Hopkins University
Yanxun Xu
yanzun.ru@juu.mm edu
Department of Applied Mathematics and Statistics Division of Quantitative Sciences Johns Hopkins Universily
Reviewed OpenReview: https: openreview. net/ forum? id= gHUKg
Abstract
Model-based offline reinforcement learning methods (RI) have achieved state-of-the-arl per- formance in many decision- decision-making problems thanks to their sample efficiency and gener- alizability. Despite these advancements, existing model-based offline RL approaches either Tocus on theoretical studies without developing practical algorithms or rely on a restricted parametric policy space, thus not fully leveraging the advantages of an unrestricted policy space inherent to model-based methods. To address this limitation, we develop MoMA, a model-based mirror ascent algorithm with general function approximations under partial coverage of offline data. MoMA distinguishes itself from existing literature by employing unrestricted policy class. In each iteration, MoMA conservatively estimates the value func- lion by minimization procedure within confidence set of transition models in the policy evaluation step, then updates the policy with general function approximations instead of commonly-used parametric policy classes in the policy improvement step. Under some mild assumptions, we establish theoretical guarantees for MoMA by proving an upper bound on the suboptimality of the returned policy. We also provide a practically implementable, approximate version of the algorithm. The effectiveness of MoMA is demonstrated via mumerical studies.
1 Introduction
Reinforcement Learning (RL) has emerged effective approach for optimizing sequential decision making by maximizing the expected cumulative reward to learn an optimal policy through iterative online interactions with the environment. RL algorithms have made significant advances in a wide range of areas such as autonomous driving "Shaley-Shward et al. 2016), video games (Torrado et al.) 2018), and robotics (Kober ct al. 2013). However, numerous real-world problems require methods to learn only from pre-collected and static (i.e., offline) datasets because interacting with the environment can be expensive or unethical, such assigning patients inferior toxic treatments in healthcare applications [Gottesman et al. 2019

---

Published in Transactions on Machine Learning Research (10/2024)
Therefore, the development of offline RL methods, which learn an optimal policy solely from offline data without further interactions with the environment, has grown rapidly in recent decades (Levine et al. 2020).
The performance of offline RL methods often relies on the coverage of offline data. Earlier theoretical studies of offine RL usually assume that offiine data have full coverage, i..., every possible policy's 's occupancy
measure can be covered by the occupancy measure of the behavior policy that generates offline data (Munos,
2003
Antos et al.) 2008 Munos & Szepesvári,
2008
Farahmand et al.) 2010 Lange al. 2012
Ross &
Bagnell 2012 Chen & Jiang, 2019 Liu et al.
2019
Uchara et al. 2020 Xie & Jiang 2020
2021 This
assumption implies highly exploratory behavior policy that can explore all state-action pairs, a condition
rarely met in practical scenarios. Recent studics have shifted from this restrictive full coverage assumption to a more realistic partial coverage framework. This only requires the behavior policy's occupancy measure to cover that of the target policy, Methods developed under the partial coverage assumption fall into two main categories: policy constraint methods and pessimistic methods. Policy constraint methods focus on learning an optimal policy within a boundary that maintains proximity to the behavior policy (Fujimoto et al., 2019:
Kumar et. al. 2019 Wu et. al.
2019: Liu et al. 2019 Laroche et al. 2019: Nachum el al. 2019 Siegel el al.
ensuring that the performance of the learned policy is least as good as the behavior policy. On the other hand, pessimistic methods, which include both model-free
2020 Kostrikov et al., 2021 Fujimoto & Gu 2021
Kumar et al.) 2020 Liu et al.
2020 Jin et al.)
2021
Rashidinejad et al. 2021 Xie et al.
2021
Zanette
et al.
2021: Yin & Wang] 2021 Kostrikov et al.;
2021
Cheng et al.] 2022 Shi et al.]
2022
Zhang et al.
2022
and model- based approaches (Yu et al. 2020 Kidambi et al.) 2020 Chang et al. 2021 Uehara & Sun
2021
Yu al.) 2021 Rigter et al.] [022] Guo et al.
2022 Rashidinejad et al. [022] Bhardwaj et al.) [0223]
employ a principle of pessimism to penalize less-visited state-action pairs without constraining the policy,
thereby avoiding uncertain regions not covered by the offline data.
Compared to pessimistic model-free methods, a key advantage of pessimistic model-based methods lies in their ability to utilize an unrestricted policy space (Uehara & Sun 2021). Nonetheless, existing offline mosel-basod approachos (Uchara & Sun 2021 Rigter ct al. 2022 Guo et al. 2022 Rashidincjad et al., 2022 Bhardwaj el al.] 2023) either focus on theoretical studies without developing practical algorithms, or rely on a restricted parametric policy space, thereby not fully exploiting the potential of an unrestricted policy space inherent to model-based methods. The drawback of using parameterized policy classes becomes evident when the optimal policy falls outside the predefined policy class. To circumvent the constraint of parametric policy spaces, this paper aims to develop pessimistic model-based algorithm that employs an unrestricted policy class.
In this study, we introduce MoMA, a pessimistic model-based mirror ascent algorithm designed for offline RI, without the need for explicit policy parameterization. At a high level, MoMA iteratively performs two steps: conservative policy evaluation and policy improvement. In the conservative policy evaluation step, we find pessimistic Q function of the current policy at each iteration t by minimizing the Q function over a confidence set of transition models. It plays critical role in relaxing the full coverage assumption to partial coverage in offline RL. In the policy improvement step, given the pessimistic Q function from the first step, we update the policy by mirror ascent (Beck & Teboulle 2003) with general function approximations. The design of the policy improvement step is adapted from Lan (0022 to facilitate an unrestricted policy class in our model-based offline RL setting.
The theoretical framework of MoMA separates conservative policy evaluation from policy improvement. This separation is advantageous as it allows for independent investigation of statistical and computational complexities. Under this theoretical framework, we establish upper bound the suboptimality of MoMA with a characterization of statistical error, optimization error, and function approximation error, assuming a computational oracle for a constrained minimization problem in the conservative policy evaluation step. A notable distinction of our work, compared to model-free approaches like Xie et al. (0021), is that our suboptimality upper bound does not include the size of the policy class. This is a significant advantage, as it permits the assumption of an unrestricted policy class. In model-free settings, the pelicy-dependent confidence set must maintain certain properties uniformly across all policies in the class, resulting in the inclusion of the policy class size in the suboptimality upper bound. In contrast, in our model-basee approach, the confidence set is independent of the policy. This independence eliminates the need to factor in the size of the policy class, thereby removing constraints on its expansiveness, On the computational side, we develop
2

---

Published in Transactions on Machine Learning Research (10/2024)
practically implementable algorithm that serves as approximation of the theoretical algorithm. In particular, primal-dual step is designed to address the computational oracle requirement- approximately solving the constrained minimization problem in the conservative policy evaluation step.
The rest of the paper is organized as follows. Section 2 introduces some preliminary settings and definitions.
In Section g we present MoMA, the proposed algorithm, followed by Section ! which discusses its practical, Dectionnestablian
the main theoretical results for MoMA under some assumptions. Numerical results in both synthetic dataset and D4RL benchmark are included in section 0 Section 7 discusses the comparison between MoMA and several existing works in offline RL. Finally, Section concludes the paper with discussion.
approximate version suitable for implementation.
2 Preliminaries
Markov decision processes and offline RL: We consider an infinite-horizon Markov decision process (MDP) M M AS P.r, v. Ho), with continuous state space S, discrete action space A (A,.A2..... A2..., Am J. a transition dynamics P(s' 8, with 8, E S and A, reward function r: S [0, 1]. a discount. factor E [0,1), 1), and imitial state distribution Po- We assume a model space P for the transition dynamic, i.c. PE P. The reward function is assumed to be known throughout this work. A stochastic policy maps from state space to a distribution over actions, representing decision strategy to pick an action with probability (-1s)) given the current state 8, i.c. .(- s) E A(A) == {p E Rm > P; = 1, 2 0, Vi} for all s E S. Given a policy т and transition dynamics P, the value function VF(s) = Ep: 7 (5, ae)|s0 = s] denotes the expected cumulative discounted reward of x under the transition dynamics P with an initial state s and a reward function r. We use VT == Es-ua Vs) to denote the expected value integrated over S with an initial distribution Pu- The action-value function (i... Q function) is defined similarly: (x) = Bp,- 7 7'r(st, (s) 8, ag = a). Let (8, a) = (1-7) (i E
8, 0, also no) be the occupancy measure of the policy under the dynamics P. Then VB can be expressed as ((s.a)=ap Irlss, a)]. Assuming that a static offline dataset D, = {(s;, ai, ri, sf) : = 1,..., n is generated by some behavior policy under the ground truth transition dynamics Pt model- offline RL methods aim to learn an optimal policy that maximizes the value Vg. through learning the dynamics from the offline dataset without any further interactions with the environment.
One fundamental challenge in offiin RL is distribution shift (Levine et al. 2020 the visitation distribution of states and actions induced by the learned policy inevitably deviates from the distribution of offline data. The concept of coverage has been introduced to mcasure the distribution shift using the density ratio (Chen & Jiang 2019). Denote p(s, a) to be the oflline distribution that generates the state-action pairs (s,, a=);== in offline data. Full coverage means sup,,a dp- (s, a)/p(s, a) < oo for all possible policies π, which may not hold in practice. In contrast, partial coverage only assumes that the offline distribution covers the visitation distribution induced by some comparator poliey m² (Xie al., 2021), such that sup,,a dp-- (s, (s,.)//(s, a) < co. Our work aims to learn the optimal policy among all polices covered by offline data, i.., I= {N : sup,,a dz (s, als a) < xo).
Partial coverage:
3 Model-based mirror ascent for offline RL
In this section, we present MoMA, which is summarized in Algorithm 11 MoMA can be separated into two steps in each iteration: 1) In the policy evaluation step, we conservatively evaluate the updated policy through minimization procedure within a confidence set of transition models; 2) In the policy improvement step, we update the policy based on mirror ascent (MA) under the current transition model. The conservative policy optimization procedure is designed to mitigate distribution shift by penalizing the value of state-action pairs that are rarely visited, thereby addressing the high uncortainty their value estimation. This approach ensures performance under a partial coverage assumption rather than full coverage assumption. We provide details for the policy evaluation and policy improvement in section 3.1 and section 3.2 respectively.
3

---

Published in Transactions on Machine Learning Research (10/2024)
Algorithm MoMA: Model-based mirror ascent for offline RL
Input: The learning rate (ne)T=1, consistent estimate P and its corresponding on- Initialization: Initialize mo(-Is) (n).
for t = 1 to T do
Conservative policy evaluation:
Let P. = argmin Vp', where Pn... = (: az).
Policy improvement: T+41(- (1))) arg
arg max (s.))..)).p) D (mi (oo(-s))pp)) s)) vs.
PEA(A)
end for
3.1
Policy evaluation: conservative estimate of
In the policy evaluation step, inspired by Uehara & Sun 2021), we first construct confidence set
P Pn... { P: (P) S {an}
for transition models.
Here
= (P)
- (t.,(P))
where In P > I is an empirical loss function for P, depending on the oflline dataset Dn, and
P = arg min (P)
PEP
is an estimator based on L... For example, if (P) denotes the negative log-likelihood function of P, then P the maximum likelihood estimator (MLF) for P. on be understood the radius of the confidence set. Then, we find P. that minimizes the value function Vp within P Pa, which is formulated as
Pt =arganni PEP Vp.
(1)
If the radius of the confidence set 0g is set to zero, then Pt exactly corresponds to the MLE for P. In this scenario, no uncertainty penalization is considered. positive On allows us to construct a perturba- tion around the MLE. Finding the most pessimistic Pt within the confidence region implicitly imposes an uncertainty penalization for rarely visited stato-action pairs.
The minimizer P,, combined with the current policy *t, be used to evaluate Q% through Monte Carlo methods. In section 4.3 will provide an example using negative log-likelihood to illustrate the implemen- tation procedure.
The conservative policy evaluation step is designed to provide a pessimistic estimate of the value function given a policy. This idea has been employed in pessimistic model-free actor-critic algorithms (Khodadadian al. 2021 Zanette al. 2021 Xie et al. 2021), where the critic lower bounds the Q function. For example, Khodadadian et al. (2021) constructed the pessimism for Q under the tabular setting. Zanette al. (2021) assumed linear function and conservatively estimated Q by minimizing Q within confidence set of the coelficients for Q. Xie el al. (2021) conservatively estimated Q in general function approximation setting, however, the size of the function class was limited by the sample size of offline data. Compared to these model-free methods, MoMA has several advantages. First, unlike Xie et al. (2021). the size of the function class for approximation in MoMA can be arbitrarily large. Second, MoMA has no restriction on the policy class, which crucial when the optimal policy is not contained in a restricted parametric policy class. See Appendix 7 for a detailed discussion about the comparisons between our work and existing literature in the policy evaluation step.
3.2 Policy improvement: mirror ascent
In the policy improvement step, we use mirror ascent, which maximizes the function with a regularizer that penalizes the Bregman distance D(..) between the next. policy and the current, policy. We first introduce
4

---

Published in Transactions on Machine Learning Research (10/2024)
Algorithm 2 MoMA: A Practical Algorithm
Input: The learning rate V/,, P Pn....
Initialization: Initialize To (n). for 1= 1 to 7 do
Conservative policy evaluation:
Compute Pt == PaCK), where g(()) is the output from eq. 44.
Policy improvement:
Sample {s, 1 from d/g.
for j to N do
Input ( Be and sj into algorithm and output (Qual A:)
end for
Find Br, that solves eq. 6 for each i i= 1,...,
Save the parametric function (fr,(s; /= as an input for iteration t 2+1 end for
the definition of the Bregman distance. Let || ||-|| be a given norm in A(A) and A(A) R be strongly convex function with respect to (w.r.t.) || ||- Then ()::)) is Bregman distance if
D == - ي (p) + (Vw (p) p' - p)1 2 1p pll vp E A(A),
= w
where ( denotes the inner product. For clarity, we consider Q7, (s, R with i-th element Q7 As). Given a pre-specified learning rate > O, the proposed update rule is:
(+ = arg max (107(()))))) - T
(T(. oh As.
(2)
PEA(A)
Intuitively, the update rule aims to maximize the value function while ensuring minimal deviation from the previous step. The distance D measures the divergence between the updated policy and the original policy from the previous step. The parameter тe can be interpreted as the step size, indicating the weight placed on distance control. This update rule is distinct from existing literature in that it does not require any explicit policy parameterization. This feature underscores the advantage MoMA over existing model-based offline
algorithms that rely on parametric policy classos, e.g., Rigter al. (002)); Guo et al. (2022); Rashidinejad et al. 2022); Bhardwaj et al. (2023).
design corresponding algo- rithms based on specific distance mcasures if desired. Notably, natural policy gradient (NPG) (Kakado 2001 is special case of policy mirror ascent when D(...) is set to be KL divergence.
As the Bregman distance defines general class of distance measures,
We remark that for a continuous state space S, it is impossible to enumerate 2 for infinitely many states. To overcome this issue, in section 4.2 we provide a computationally efficient algorithm through function approximation, which is one of the key advantages of MoMA compared to existing literature, e.g., Algorithm in Xie al. 2021).
4
A practical algorithm
In this section, we present an implementable algorithm that approximately solves the constrained minimiza- tion problem [ summarized in algorithm 2
4.1 Primal-dual (PD) for solving the constrained optimization problem
In order to approximately solve the constrained minimization problem the policy evaluation step, we introduce its Lagrangian form:
min v
X(E, az))
(3)
A PEP
5

---

Published in Transactions on Machine Learning Research (10/2024)
Suppose the transition model P is parameterized by o, denoted as Pa- To solve gradient primal-dual method with the following update rule:
3
design model
= K1 V Vv.vv (()). ( Proja (1 + n2 (E - -nn)
(4)
Here Proja is the projection to a pre-specified interval A for A, and ( V + (En - an). Here A is a user-defined interval designed to prevent ( from reaching 0 or
((t) is the Lagrangian function:
((0)
+00.
4.2
Function approximation in MA
In the policy improvement step of algorithm | updating T2+1(- 8) for infinite number of states is computationally impossible, Qp, can only be evaluated when | [s] is known for all Although Monte Carlo estimation may be utilized in evaluating Qp; in eq. (2). the computational complexity would grow exponentially with the number of iterations T, resulting in computational inefficiency. Notably, this issue is also present in Algorithm 1 of Xie et al. (2021). In contrast, our practical algorithm cxhibits polynomial dependence on T, which is computationally efficient. detailed justification for this claim provided in Appendix
By the definition of D( s),p), the objective function in the policy improvement step of algorithm 1 is equivalent to
1 T
(07 (s. .) (Ww (me(- [s) p) T
w(p).
We define the augmented action-valte function as
(w.1(s.p) = Σ (s, = (Q,))(s,))pp)
where
A))== : QP, + Vw(TT(- [s));
T
and Qu,r(s,)) a vector with its i-th element as As). The augmented action-valte function not only incorporates information about the action-value function but also includes information about the policy itself. Following Lan 2002), we approximate Qu, p) by a parametric function fe(s. 3.) E F, such that (s(5,p)))) ( (8.P) p) lor some 87, which is sufficient approximate Qw.((s, A.) for each A; according to
Qw, (s..(s,)) .).). To this end, for each i I,... m, we introduce (s, 3r, Fi,s to approximate Qw,e(s, At), and thus
B.) = - Fesss: B.,), = (1)(s; 1=1
Here, Fe,s can be chosen as c.g. reproducing kernel Hilbert spaces (RKHS) or neural networks. For each 1,.,, m, the optimal parameter 97 can be obtained as follows,
Br E argming, g., Ea-dy
- (cd(s
(5)
Specifically, we can generate { dpp and then minimize the empirical version of 5
Bas E argming.s
(6)
where (8, A are output from algorithm (see Appendix property that (1, Br,)) AulssPt.) Qua(s, Ar) for each 1 b... m.
The computable Bog satisfies the

---

Published in Transactions on Machine Learning Research (10/2024)
With the obtained (s,,) the update rule in the policy improvement step can be written below and solved by standard optimization algorithms,
T (: 8) arg max PEA(A)
m Σ (e,
w J.
A V.
T/
Such design of function approximation enjoys several benefits. 1) The objective function in 7 is concave which can be solved by standard first-order optimization methods. 2) Compared to commonly-used para- metric policy classes, our policy class is unrestricted, ensuring it includes the optimal policy. 3) The function approximation error in eq. (5) can be made arbitrarily small by cnlarging the function classes Ft,s-
4.3
An example
though our framework is general and different settings can be considered if desired. We provide a concrete example to illustrate the implementation procedure of the proposed practical algo-
rithm P
Policy cvaluation step.
We consider the following empirical loss function L, for transition models:
(.p(P)
1 n
Σ
log Pa(s/ | 88, si)
Then p= Py is exactly the MLE, and
Pa(s'( a;)
1
(.Ps
log
8ig ai) Ps
1=1
The gradient of the Lagrangian function is:
VoLvc(Ø.A) = VOV +A (Po).
Here the model gradient V V be calculated using the proposition 2 of Rigter et al. 2022 Specifically, let
M M,, = + + (6)) x V Volog 18,)),
then
a Pp = E Bs s' Po((s,,) Ma
) (s'
a, m.).
For VeEn (Ps), we have (P,) - En V Pog 8/s a.). Then the expressions of VoLv, V V P. and VoEn(Ps) can be plugged into and output P. = Pacns.
Policy improvement step. We consider
((p) log
and introduce multi-layer neural network (s, Br,s) for approximating Qu.r(s, A.). For each i 1, m, in order to find the best Br,i, we first sample {s, } i.i.d. from d'i, then run any policy evaluation procedure such Monte Carlo (algorithm in Appendix E for Qu,r(s), Ar) for each j 1, ..., N. Using training data (sz, (w. Ar)) we can obtain Br,s by standard neural network (NN) training procedure, In addition, thanks to the form of w, we have a closed form solution to 77) which is the update rule
Te+1 (A: s) x exp(tn.f..(3333..)) (..))
for I.... m. Besides NN for (s: B(,)) for each i 1,..., m, alternative general function classes, such infiite-dimmenionnal RKHS, also be employed for function approximations.

---

Published in Transactions on Machine Learning Research (10/2024)
4.4 Efficient policy update
We provide detailed discussion of the computational complexity for each step in our MoMA algorithm here. We first consider the case w(p) = -i=1 P: logpi, which leads to closed function form of T++1 (A, () x exp (eff. (s.Be..) for each i = I,..., m. In the policy evaluation step t, given i which is the output from the (t - 1)-th iteration, we can count the number of calls of (T(()) from to ++ as by realizing we need Ty in the Monte Carlo evaluation of Vp, for = I,..., K and the sampling from de in the policy evaluation step. Specifically, for cach sampling or Monte Carlo evaluation, the effective numbers of using x is ,- which is the effective trajectory length in an infinite-horizon discounted MDP. Therefore, for each I,
in the policy evaluation step, we need to use Te for total of O KL2 times, where L denotes the number of Monte Carlo trajectorics. In the policy improvement step, we need ne the sampling of (sj. A)) and Monte Carlo evaluation of (s, A;) for each j. Therefore, we need O NL Monte Carlo trajectories starting from (83,P5) for each = I,...,N. Therefore, collectively at each in algorithm evaluate the function nx (A: 1) = exp ( (::Be-.s) /C approximately O KT.²
2 we need lo NT. limes, which
1-7
[1-3)
is independent of l. More generally, when w does not induce an explicit solution to 77), then I more steps
for gradient descent of 7 may be noeded. In that casc, running algorithm costs O IT
KL NL 1-7 (1 (1-7)) ))
operations related to policy updates, which is polynomial on all the key paramcters.
4.5 Extension to continuous action space
We now extend MoMA to handle complex RL tasks with nonlinear dynamics and continuous action spaces. Instead of considering p E (AA) introduced in section 3.2 where A is assumed to be finile, we consider a continuous action space A c Rdd in this section. Here d is the dimension of the action space and A is assumed to be compact convex set. In the continuous-action case, we consider the deterministic policy क : S + A, i.. i(s) E A is a feasible action for each state s E S.
In this case, a proposed update rule
(7(as)) 1 (T.)))) Vs. M
(s) arg max aeA
(8)
Still, since the update rule g is computationally infeasible for infinitely many s, propose version with function approximation that is similar to section
4.2 Specifically, by expanding () a), the objective w()). We also define (w.e(s, a) = Q ())
function in (8) is equivalent to QP; (s, a)+ (Vw (T)( -
1 (Vw (s) a) as the augmented action-value function. We then approximate (w.e(s, a) by a parametric function fs a: B:) E F. such that fe(s, az Bt) (s,(s for some 8 Here Fe can be RKHS Neural
Networks.
In particular, the optimal parameter 3 can be obtained as follows,
8r E argming, Blaayyay (QQee(s,a) fe(s, ae(s,
(9)
Specifically, we can generate {s,, aj 1 ~ dp, and then minimize the empirical version of 5
3. E argming, -(,,,)))))))--,,,,;3:)))))) fe(s),
(10)
where (Que(ss y are output from algorithm (see Appendix property that fels, az B.) fe(s, (;) (s. Finally, the update rule involving function approximation be written as
The computable 3, satisfics the
8

---

Published in Transactions on Machine Learning Research (10/2024)
" -w(a) h Vs E.
(5) = arg max atA
fi(s,a;B.))
(11)
A standard optimization procedure such as accelerated gradient descent method can be employed to solve 11I.
For completeness, we summarize the whole algorithm for the contimnous-action case algorithm
Algorithm 3 MoMA: A Practical Algorithm in the contimuous-action case Input: The learning rate Mt, P Pn.a..
Initialization: Initialize To = (n).
for t = 1 to 7 do
Conservative policy evaluation:
Compute Pt == PaCK) where g(K) is the output from eq. (4)
Policy improvement:
Sample {sj, وززز from d..
for to N do
Input f2-1(8, a; B(-1) and (sj, az) into algorithm and output Qw, az).
end for Find Re that solves eq. 10
Save the parametric function fi(s, a; Be) as an input for iteration t + 1.
end for
5 Theoretical analysis
In this section, we present the upper bound on the suboptimality of the learned policy &े in Algorithm in terms of sample size, number iterations and all key parameters. All proofs are presented Appendix We first present the following assumptions.
Assumption 1. The following conditions hold.
(a) (Data generation). The dataset D = (si, as, Tis &i satisfies (si, az) 1nd p with 8f ~ P*( 8ig az), where p denoles the offinne distribution induced by the behavior policy under pt.
d (p,a)
(b) (Coverage of any comparator policy T!). Cxt == sups,a p(s.a)
0c.
(e) (Realizabilily) p* P.
(d) There exist < an o(1), th S 6n o(1) such that with high probabilities P. P Pnia and
Eest = sup
E(s.a)~p11P((1 s, - p' ((.1))))l11] 6n.
(12)
PEPn.n
Assumption [(a) is related to offline data generation, common in offline RL theoretical literature. Assumption [bb) essentially requires the partial coverage of the offline distribution. The concentrability coefficient C₂ measures the distribution mismatch between the offline distribution and the occupancy measure induced by at. Assumption [c)) requires that the model class is sufficiently large such that there no model misspecification error. Assumption [d) is needed to provide fast statistical rate uniformly over the confidence set. We remark that assumption (d) is mild condition. For example, commonly empirical risk functions (c.g. negative likelio))))od) satisfy it. See proposition corollary for more details in
Appendix B
Now we provide the suboptimality upper bound for algorithm 2 in the following theorem, assuming an access to computational oracle for solving the contained minimization problem 11 in the conservative policy evaluation step. Thcoretical results for algorithm [ in which we assume no function approximation, summarized in Theorem 2 in Appendix

---

Published in Transactions on Machine Learning Research (10/2024)
Theorem 1. Under Assumption 1 if m (1 y) 2logglAl) for every fixed T, then have
Vp. Vp. p-
2
7
+
Cxt East
1
7) 7)2 (1 7)3
VT
model error
policy optiniization etror
(Sapproz /mmxx,, FL
sup
7)
(o(s)
function approximation ervor
with high probability. Here f ~ Unif Ti, .... -T where { { are output by Algorithm and Eapprox is defined in Definition A.3.
A brief proof sketch is provided here. The proof of Theorem is based on decompasition of suboptimality:
Vp.
Vp.
(vp"
Vg.
+
Vi Pe
+
(VP.
Vp.))
(13)
(a)
(6)
Term (a) can be bounded by the distance between the true dynamic model P and Pt. which converges to as the sample size increases to co. Term (b) managed through the mirror ascent update rule, which tends to 0 as the number of iterations & co. Term (c) is negative thanks to the construction of the confidence set and the definition of P.. The proof is completed by averaging both sides of the above equation over T.
The upper bound in Theorem includes three terms: a model error (depending on fixed n) coming from using offline data for estimation of the transition model, an optimization error (depending on iteration T) from the policy improvement, and a function approximation error coming from using Monte Carlo samples approximating the augmented Q function. The model error is finite-sample term that cannot be reduced under the offline setting, while the optimization error can be reduced when the number of iterations T increases. Typically, we have Eest Op(1/vr)) The function approximation error involves an approximation error Eapproz that decreases as the function class is enlarged max,,; [Fe + oc, an estimation error that scales with Op(I/VN), and a distribution mismatch sup, dp'. (s)))))))))))))))))))))))))))()) between the initial distribution and the occupancy measure induced by a single m² under pa. Indeed, if max,, FF - oc, then Sapprox by Definition Consequently, the function approximation error converge to 0 as long max,, |Frisl - and N at the same speed based its expression in Theorem | The function approximation error and the model error share similar intuitive interpretations. In the model error, Cxt measures the transfer of Eest changing from the offline distribution to the target distribution dp'.. Analogously, sup, dp- (s)/ in the function approximation error measures the transfer of Eapprox + /max,,a [Fr. from the initial distribution to the target distribution dz!..
VN
6 Numerical studies
We perform numerical studies on both an illustrative synthetic dataset and MuJoCo Todorov et al. 2012 benchmark datasets, where we extend MoMA to the continuous-action setting.
6.1 Synthetic dataset: an illustration
We design a test environment based on modified random walk with terminal goal states to generate data with partial coverage and understand how pessimism helps MoMA avoid common pitfalls faced by model- based offline RL methods. w(p) is set to be Σm Pt log(ps).
Environment and offline datasct For cach episode that starts with an initial state 80 u(-2, 2). at time particle undergoes random walk and transits according a mixture of Gaussian dynamics: v.a (( 0.1), where the discrete action E (-1,0, corresponds
8a41 Sn As (a,,a, 0.1) (.1)+
10

---

Published in Transactions on Machine Learning Research (10/2024)
to Left, Stay, and Right, respectively. We generate partially covered offine dataset collected by biased (to the left) behavioral policy 3 that penalizes over-exploitation of the MLE. The full details for the environment and the behavioral policy are given in Appendix
MoMA performance The implerenntttion follows Algorithm with details Appendix F.1 We com- pare with 1) model-based NPG, which can be seen as simplified version of MoMA without the conservative policy evaluation; 2) model-free neural fitted Q-iteration (NFQ) (Riodmiller) 2003); and 3) uniformly ran- dom policy. All algorithms are oflline trained to convergence, and then put into the environment for 1000 online evaluation episodes. We choose the number of episode steps as the metric for this shortest path problem, and report the means and standard deviations for the scores of all 4 algorithms in Table MoMA has significantly superior performance compared to the baselines, while the model-based peer NPG achieves the second best performance. This is not surprising since a learned dynamics model in general helps gen- eralization in tasks with continuous state spaces, and NFQ's lack of generalization ability is exacerbated in this partial coverage setting.
Table 1: Average episode length (+ std.) over 1000 online evaluation episodes; shorter is better.
MoMA
NPG
NFQ
Uniform
2.63 + 1.61
3.20 + 2.33
4.39 + 2.96
6.13 5.07
Contribution from pessimism To understand pessimism empirically, zoom into the state 0.1, where the optimal policy is consecutive Right, the data-supported suboptimal policy is consecutive Left, and the faulty policy centers Stay. Due to inaccurate MLE, model-based algorithm without pessimism over exploits the model and converges to the faulty action Stay. In contrast, pessimism allows MoMA to trust the model on Left which has high coverage, while cautiously modifying the model such that Stay does not lead to substantial Right movement, i.e., 1o increases (see Figure 2 in Appendix F). This behavior is clearly captured during the training process: shown in the right plot of Figure [T. while the weight of the faulty action Stay monotonically increases for NPG, it decreases from the 10th epoch for MoMA. As result, the suboptimal action weight (shown middle) eventually dominates for MoMA but vanishes for NPG, and NPG's value function (001) under the true dynamics (shown left) decreases due to the over exploitation of the learned dynamics model.
-2.5
1.0
1.0
NPG
MoMA
Suboptimal Action
Value Function
WV
0.8
Faulty Action
0.8
-2.6
0.6
0.6
0.4
0.4
-2.7
0.2
0.2
0.0
0.0
0
10
20
30
40
10
20
30
40
10
20 Epoch
30
40
Epoch
Epoch
Figure 1: MoMA vs NPG training behavior, zoomed in at state 0.1. Left: the data-supported suboptimal action weight; right: the model-mislead faulty action weight.
the value function V(0.1); middle:
we provide an illustrative example demonstrating that assuming a nonpara- metric policy class leads to better performance compared to a parametric class, highlighting scenarios where the limitations of parametric policies become evident.
Furthermore, in Appendix F.2
6.2 Continuous action D4RL benchmark experiments
We now extend MoMA to handle complex RL tasks with nonlinear dynamics and continuous action spaces approximating (s, Va continuous A rather than approximating Qu,e(s, Ar) for discrete actions.
11

---

Published in Transactions on Machine Learning Research (10/2024)
4.5 for more implementation details. We adjust the implmentation accordingly, and evaluate D4RT.
See section
Fu et al. 2020) MuJoCo benchmark datasets.
We consider the medium, medium-replay, and medium-expert datasets for the Hopper, HalfCheetah, and Walker2D tasks (all v0), respectively. We compare against SOTA model-based baseline algorithms MOPO 2020 and RAMBO (Rigter et al. 2022), and model-free baseline algorithms CQL (Kumar et al., 2020) and IQL [Kostrikov et al. 2021]. With the exception of RAMBO for which we cite the results reported in Rigter et al. (2022), train all algorithms for 1E6 steps with early stopping and with 5 different random seeds; additional experimental details are given in Appendix .3 We summarize the scores (average returns of 10 evaluation episodes) in Table 2 MoMA consistently demonstrates performance that is at least comparable to state-of-the-arl (SOTA) algorithms. Notably, in 4 out of 9 cases, our algorithm outperforms the other two model-basee RL algorithms, achieving the highest performance.
(Yu et al.
Table 2: D4RL, benchmark averaged performance over 5 random seeds (+ std.).
Ours MoMA
Model-based
Model-free
MOPO
RAMBO
IQL
CQL
Hopper, medium
42.9 + 12.9
58,2 = 15.2
92.8 6.0
101.1 = 0.5
100.6 - 1.0
Hopper, medium-replay
102.2 + 0.8
101.2 + 0.9
96.6 + 7.0
66.0 + 16.2
89.2 + 9.3
Hopper, mediummexpper
102.5 + 3.4
46.3 + 17.1
83.3 + 9.1
106.7 + 7.1
82.8 1 19.0
BetCChoetah,mediun
44.2 = 0.8
26.5 10.1
77.6 1.5
42.5 = 0.1
41.3 + 0.3
HalfCheetah, medium-replay
55.4 = 1.0
53.0 = 2.0
68.9 + 2.3
42.6 - 0.1
45.9 = 0.1
HalfCheetah, expert
98.4 +: 12.6
95.5 + 10.6
93.7 10.5
96.9 + 1.8
85.3 + 7.9
Walker2D medium
36.8 = 20.4
12.8 = 7.1
86.9 2.7
58.8 = 4.4
83.1 = 0.8
Walker2D, medium-replay
28.7 1 5.8
67.9 1 5.8
85.0 1 15.0
22.5 1 11.1
28.3 1 9.3
Walker2D, medium--xpert
98.5 1 5.5
94.6 1 16.0
68.3 20.6
108.4 + 1.8
106.3 + 15.5
As one of the anonymous reviewers suggested, the confidence intervals (CIs) reported in our study can be biased due to being based on only a small number of seeds.
As a supplement to the standard mean of evaluation scores reported in Table 2 ation scheme proposed by AAaaral et al. (2021). Their method emphasizes the use of distributional metrics, which are less sensitive to outliers, providing clearer and more robust picture of algorithm performance across different runs.
we have adopted the evalu-
Specifically, we now include the interquartile mean (IQM) and the optimality gap as aggregate metrics, along with 95% bootstrap Cls. The IQM provides a robust measure of central tendency by focusing on the middle 50% of the data, reducing the influence of extreme values. The optimality gap measures how close the performance is to an optimal policy, offering meaningful interpretation of results. Additionally, we present performance profiles based on score distributions, which offer a comprehensive view of the algorithm's performance across different scenarios. These results are detailed in Appendix F.2.
7
Comparisons with existing works
In this section, we present a discussion on the comparisons with some cxisting works in the field of offline reinforcement learning.
7.1 The policy evaluation step
In Section 3.1. we mentioned two fundamental advantages that can be attributed to the MoMA's design in the policy evaluation phase: 1) better expressiveness of the policy class, and 2) more flexibility of function approximations. We add more explanations about these two points here. First, we elaborate on better expressiveness of the policy class and more flexibility the value function class. Indeed, these two advantages mainly stem from the construction of confidence set as well as the scparation of estimation and optimization under our model-basee based framework. Specifically, the offline dataset is only used to infer the transition model rather than directly infer the value function (which also depends on a policy). Thanks to this model-based
12

---

Published in Transactions on Machine Learning Research (10/2024)
feature and the framework for the proposed algorithm, neither the size of the value function class nor that of the policy class is limited by the size of the offline dataset. In fact, the policy class in our settings can be taken large enough to contain the optimal policy, and the size of the value function class can keep growing until it contains the true value as long as we run Algorithm 4 enough times to generate sufficient Monte Carlo samples. These Teatures result. an optimal rate of O(1/v1 as shown Theorem [ which outperforms the existing work (Xie ct al. 2021). To illustrate this, we compare Corollary of Section in Xie et al. (2021) with Theorem 6.11 from our work, both discussing the suboptimality gaps under general function approximation. Corollary 5 in Xie et al. (2021) presented convergence rate relative to the offinee sample size n O D(1/n1/5), while Theorem 6.11 established a. rate of O(1/vn). Therefore, MoMA enjoys the benefits of possessing more general policy classes and value function classes, while making no sacrifice on the data efficiency.
7.2 The policy improvement step
For the policy improvement step, we have developed the first computationally efficient algorithm under general function approximations (rather than linear approximations) with a theoretical guarantee. Existing literature is only computationally efficient cither under lincar approximation scttings (Zanctte et al. 2021 Xie et al., 2021] J, or without a theoretical guarantee for the policy improvement step (Cheng al. We give detailed comparisons below.
2022
Though Algorithm 1 of Xic et al. 2021)) employs a mirror ascent method, it is not efficiently implementable when |S] 0o, since it is impossible to enumerate every s in continuous state space to update the policy when the ft in Am of Xie et al. (2021) actually needs the access to Tel. 8) for every 8. Even if finitely many Tr(s) are employed for obtaining ft via Monte Carlo methods, still incurs an exponential complexity of least Q (C"). To clearly show the difference, we exhibit our proposed algorithm and the one in Xie al. (2021)
Our proposed update rule when w(p) = Ei Pi log
T++1 (A s) exp (sefcs(s:
for each i = 1,..., m.
The update rule in Algorithm 1 in Xie et al. 2021
Tt+1 (A: 1 s) exp fife (s:B..) t (Ai 1s)
for each i = 1,...,m. m.
In the update rule of Xie et al. 2021 T++1 (Ax |) does not obtain a closed form without iteratively calling the previous iteration. In the continuous state space, this procedure is computationally inefficient. Moreover, assuming that the number of calls of (-)-) used to approximate F. (: Be.s is C, then at least Ω2 (C")) number of operations related to policies are nceded. (See 4.4 In contrast, our algorithm's 's cost related to evaluating policies O (T KL + (, ) evidently more computationally efficient.
Additionally, while Cheng al. (0022) considers parametric policy class, they do not provide theoretical analysis for the actor step. Further, Zanette et al. (021 only focuses linear approximations, which may not be applicable in more general settings.
In summary, MoMA can be utilized when policy classes and value function classes of greater generality are needed, with no sacrifice on computational efficiency.
8 Conclusion
We developed MoMA, a model-based mirror ascent algorithm for offiine RL, with general function approx- imation under the assumption of partial coverage. A key strength of MoMA is its ability to fully exploit
13

---

Published in Transactions on Machine Learning Research (10/2024)
the potential of unrestricted policy space, characteristic advantage of model-based methods. This has been achieved through combination of our theoretical framework and the application of mirror ascent techniques. Additionally, we have developed a practical, implementable version of MoMA that serves as an approximation of the thcoretical algorithm. The efficacy of this practical algorithm has been demonstrated through a series of numerical experiments. While the numerical results do not consistently outperform the SOTA, one plausible reason that the policy classes used in other offline algorithms might be sufficiently rich for these tasks. The proposed algorithu would gain more advantage in tasks where the optimal policy is harder to approximate. Improving the algorithm to consistently match the SOTA is an important area for future research.
There are several other intriguing avenues for future research. One promising area involves the development of algorithms that can theoretically address the computational challenges posed by the conservative evaluation step, particularly the need for a computational oracle. Another direction involves the integration of prior domain-spocific knowledge, such as physical principles in mechanical systems or clinical insights in medical applications, into the transition model. Incorporating Bayesian estimators within our framework could provide robust means of estimating transition models in these contexts. In addition, can consider model misspecification, c.g., P & PP analyze both the estimation error and approximation error, and investigate how they affect the suboptimality gap.
Acknowledgements
We thank the Assigned Action Editor, Shixiang Gu, and the reviewers for their insightful comments and suggestions that significantly improve this paper. Yanxun Xu is supported in part by National Science Foundation grants 1918854 and 1940107, and National Institute Health grant RO1MH128085.
References
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Marc Bellemare. Deep reinforcement learning at the edge of the statistical precipice. Advances in neural information processing systems, 34:29304-20322 2021.
András Antos, Csaba Szepesvári, and Rémi Munos. Learning near-optimal policies with bellman-residual minimization based fitted policy iteration and a single sample path. Machine Learning, 71:89 129, 2008
Amir Beck and Mare Teboulle. Mirror descent and nonlinear projected subgradient methods for convex optimization. Operations Research Letters, 313) 2003.
Mohak Bhardwaj, Tengyang Xie, Byron Boots, Nan Jiang, and Ching-An Cheng. Adversarial model for offline reinforcement learning. arXiv preprint orXiw:2302.10000 2023.
Jonathan Chang, Masatoshi Uehara, Dhruv Sreenivas, Rahul Kidambi, and Wen Sun. Mitigating covariate shift in imitation learning via offline data with partial coverage. Advances in Neural Information Processing Systems, 34:965 979, 2021.
Jinglin Chen and Nan Jiang. Information-theorede considerations in batch reinforcement learning. In International Conference on Machine Learning, pp. 1042 1051. PMLR, 2019.
Ching-An Cheng, Tengyang Xic, Nan Jiang, and Alckh Agarwal. Adversarially trained actor critic for offline reinforcement learning. arXiv preprint ar Xix:2220 02446, 2022.
Amir-massond Farahmand, Csaba Szepesvári, and Rémi Munos. Error propagation for approximate policy and value iteration. Advonces in Neural Information Processing Systems, 23, 2010.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data- driven reinforcement learning. arXiv preprint arXiv:2004.072219, 2020.
Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning. Advances in neural information processing systems, 34:20132-201 20145, 2021.
14

---

Published in Transactions on Machine Learning Research (10/2024)
Scott Fujimoto, David Meger, and Doina Precup. Off-policyy deep reinforcement learning without exploration. In International conference on machine learning, pp. 9052-20622 PMT.R, 2019.
Subhashis Ghosal and Aad Van der Vaart. Fundamentals of nonparometric Bayestan inference, volume 44. Cambridge University Press, 2017.
Omer Gottesman, Fredrik Johansson, Matthieu Komorowski, Aldo Faisal, David Sontag, Finale Doshi- Velez, and Leo Authony Celi. Guidelines for reinforcement. learning in healtheare. Nature medicine, (5(1):16-18, 2019.
Kaiyang Guo, Shao Yunfeng, and Yanhui Gong. Model-based offline reinforcement learning with pessimism- modulated dynamics belief. Advances in Neural Information Processing Systems, 35:449 461, 2022.
Ying Jin, Zhuoran Yang, and Zhaoran Wang. Is pessimism provably efficient for offiine rI? In International Conference on Machine Learning, pp. 5084 5096. PMLR, 2021.
Sham M Kakade. A natural policy gradient. Advances in neural information processing systems, 14, 2001.
Sajad Khodadadian, Zaiwei Chen, and Siva Theja Maguluri. Finite-sample analysis of off-policy natural actor-critic algorithm. In International Conference on Machine Learning, pp. 5420-5485 5431. PMLR, 2021.
Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. Morel: Model-based offline reinforcement learning. Advances in neural information processing systems, 33:21810-2188 2020.
Jens Kober, J Andrew Bagnell, and Jan Peters. Reinforcement learning in robotics: A survey. The Interna- tional Journal of Robotics Research, (2(11):1238-121) 2013.
Tya Kostrikov, Rob Fergus, Jonathan Tompson, and Ofir Nachum. Offline reinforcement learning with fisher divergence critic regularization. In International Conference Machine Learning, pp. 5774-5788 PMLR, 2021.
Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping crror reduction. Advances in Neural Information Processing Systems, 32, 2019.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforce- ment learning. Advances in Neural Information Processing Systems, 33:1179-1191, 2020.
Guanghui Lan. Policy optimization over general state and action spaces. arXiv preprint arXi::2211.16715, 2022.
Sascha Lange, Thomas Gabel, and Martin Riedmiller. Batch reinforcement learning. In Reinforcement learning: State-of-the of- of-the-art, pp. 45-73. Springer, 2012.
Romain Laroche, Paul Trichelair, and Remi Tachet Des Combes. Safe policy improvement with baseline bootstrapping. In International conference on machine learning, pp. 3652-3661. PMTR, 2019.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu, Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXim:205.. 01643, 2020.
Yao Liu, Adith Swaminathan, Alekh Agarwal, and Emma Brunskill. Off-policy policy gradient with state distribution correction. arXiv preprint arXiv: XXX::1904400 1904.08479, 73, 2019.
Yao Liu, Adith Swaminathan, Alekh Agarwal, and Emma Brunskill. Provably good batch off-policy re- inforcement learning without great exploration. Advances in neural information processing systems, 33: 1264 1274, 2020.
Alfred Mullrr Integral probability metrics and their generating classes of functions. Advances in Applied Probabilily, 29(2):429 443, 1997.
Rémi Munos. Error bounds for approximate policy iteration. In ICML, volume 3, pp. 560-567. Citeseer, 2003.
15

---

Published in Transactions on Machine Learning Research (10/2024)
Rémi Munos and Csaba Szepesvári. Finite-time bounds for fitted value itcration. Journal of Machine Learning Research, 9(5), 2008.
Ofir Nachum, Bo Dai, Nya Kostrikov, Yinlam Chow, Lihong Li, and Dale Schuurmans. Algaedice: Policy gradient from arbitrary experience. arXiv preprint arXi::112.002074, 2019.
Paria Rashidinejad, Banghua Zhu, Cong Ma, Jiantao Jiao, and Stuart Russell. Bridging offline reinforcement learning and imitation learning: A tale of pessimism. Advonces in Neural Information Processing Systems, 34:11702-11716, 2021.
Paria Rashidinejad, Hanlin Zhu, Kunhe Yang, Stuart Russell, and Jiantao Jiao. Optimal conservative offline rl with general function approximation via augmented lagrangian. ar preprint arXiv:221.000116 2022.
Martin Riedmiller. Neural fitted q iteration first experiences with a data efficient neural reinforcement learning method. Machine Learning: ECML 2005: 16th European Conference on Machine Learning, Porto, Portugal, October 3-7, 2005. Proceedings 16, pp. 317-328. Springer, 2005.
Marc Rigter, Bruno Lacerda, and Nick Hawes. Rambo-rl: Robust adversarial model-based offline reinforce- ment. learning. arXiv preprint or 2rx::2204.1122 12581, 2022.
Stephane Ross and J Andrew Bagnell. Agnostic system identification for model-based reinforcement learning. arXiv preprint Xiv: 1007, 2012.
Shai Shalev-Shwarlz, Shaked Shammah, and Amnon Shashua. Safe, multi-agent, reinforcement learning for autonomous driving. arXie preprint arXiv: 161000229,,, 2016.
Laixi Shi, Gen Li, Yuting Wei, Yuxin Chen, and Yuejie Chi. Pessimistic q-learning for offiine reinforcement learning: Towards optimal sample complexity. In International Conference on Machine Learning, pp. 19967-20025. PMLR, 2022.
Noah Y Sicgel, Jost Tobias Springenberg, Felix Berkenkamp, Abbas Abdolmaleki, Michael Neunert, Thomas Lampe, Roland Hafner, Nicolas Heess, and Martin Riedmiller. Keep doing what worked: Behavioral modelling priors for offline reinforcement learning. arXiv preprint arXiv:2002.08300 2020.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ international conference intelligent robots and systems, pp. 5026 5033. IEEE, 2012.
Ruben Rodriguez Torrado, Philip Bontrager, Julian Togelius, Jialin Liu, and Diego Perez-Liebana. Deep reinforcement learning for general video game ai. In 2018 IEFE Conference on Computational Intelligence and Games (CIG), pp. 1 8. IEEE, 2018.
Masatoshi Uehara and Wen Sun. Pessimistic model-based offline reinforcement learning under partial cov- erage. In International Conference on Learning Representations, 2021.
Masatoshi Uehara, Jiawei Huang, and Nan Jiang. Minimax weight and q-function learning for off off-policy evaluation. In International Conference on Machine Learning, pp. 9659-9668. PMLR, 2020.
Sara A Van de Geer and Sara van de Geer. Empirical Processes in M-estimation, volume 6. Cambridge university press, 2000.
Martin J Wainwright. High-dimensional statistics: A non-asymptotic viewpoint, volume 48. Cambridge
university press, 2019.
Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning, arXiv preprint XrXi::1911. 11361, 2019.
Tengyang Xie and Nan Jiang. Q* approximation schemes for batch reinforcement learning: A theoretical comparison. In Conference Uncertainty in Artificial Intelligence, pp. 550-558 PMLR, 2020.
Tengyang Xie and Nan Jiang. Batch value-function approximation with only realizability. In International Conference Machine Learning, pp. 11403-11413. PMT.R, 2021.
16

---

Published in Transactions on Machine Learning Research (10/2024)
Tengyang Xie, Ching-An An Cheng, Nan Jiang, Paul Mineiro, and Alckh Agarwal. Bellman-connistent pessimism for offline reinforcement learning. Advances in neural information processing systems,
2021.
Ming Yin and Yu-Xiang Wang. Towards instanco-optimal offine reinforcement learning with pessimism. Advances in neural information processing systems, 34:4065-4000 2021.
Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Y Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. Advances in Neural Information Processing Systems, 33:14129 14142, 2020.
Tianhe Yu, Aviral Kumar, Rafael Rafailov, Aravind Rajeswaran, Sergey Levine, and Chelsea Finn. Combo: Conservative offline model-based policy optimization. Advances in neural information processing systems, 34:28954 28967, 2021.
Andrea Zanette, Martin J Wainwright, and Emma Brunskill. Provable benefits of actor-critic methods for oflline reinforcement learning. Advances in neural information processing syslems,, 34, 2021.
Xuezhou Zhang, Yiding Chen, Xiaojin Zhu, and Wen Sun. Corruption-robust offline reinforcement learning, In International Conference on Artificial Intelligence and Statistics, pp. 3757-5773. PMLR, 2022.
A Notations and definitions
Definition A.1 (Integral Probability Metric (IPM)(Mülle) 1997
dF an IPM defined by F if
d=(P,Q) = sup JEF
E
[f()]
- E If (x)]
where P and Q are two probabilily measures.
By considering different function class F, we have the relationships between IPM and several popular mea- sures of distance, such as TV and Wasserstein distance.
Definition A.2 (Slater's condition). The problem satisfies Slater's condition if strictly fensible, that Aro E D: fi (ro) < O, == 1......m hy (xo) = O, i= 1,....P
The next definition essentially measures the size of the function classes Fe,'s. If max,, Fell is sufficiently large, then Eaprrox 0. Definition A.3 (Approximation error).
Eapprox == sup inf (s, - fe, (e.)l,,.,) P. E,
Definition A.4 (Localized Population Rademacher Complexity). (Wainwright 2019, chap. 14). For given radius and function class F, a localized population Rademacher complexity is defined as
(8; = Ee,
Σ Exf (mi)
JEF 111
where {-ili=1 are i.i.d. samples from some underlying distribution P, and {es)= are i..dd Rademacher variables laking values in {-1,+1} equéprobably, independent of the sequence (xx)==
Definition A.5 (Star-shaped function class). (Wainwright, 2019 chap. 14). ster-shaped around origin if for any f E and scalor [0. 1), the function af also belongs to F.
function class F is
Definition A.6 (Strongly convexity). is strongly conver with modulus p. if the following holds:
f(y) 2 f(x) + Vf(x) x) + 11 11y x112.
(14)
17

---

Published in Transactions on Machine Learning Research (10/2024)
B Additional theoretical results
The following proposition shows that the commonly-use used empirical risk functions satisfy Assumption [d].
Proposition 1. Consider uniformly bounded function class L(P) = [(s', 8, - I(P, ((P,,', 8,a,, P P) that is star-shaped (defined in Appendür Al around the true P Suppose 6 2 a is a solution to the inequality (o:6; S 8 where (6; ((P)) is the localized Rademacher complezity (defined in appendir A of the function class. Assume I(P, 8 a) is I--isscchitz w.r.l. P, i.e.,
I(P1, 8 - 1(22, 6', 8, 8, 8, S 1o 11) - 12
Assume further that 1(P, 8,, a) is also strongly conver w.r.t. P(s',s, s, under the norm || 11. Suppose (P 8, P( 8, s,))) S P 8, - pg ppp s', high probabilities, have p Pn.. and
88,0
8,, Let a = c1 then with
sup PEP
E ((s.a)~p s.a)
c2δn.
proposition 1 covers the commonly-used negative likelihood function classes, in which the empirical risk minimizers are exactly MLEs. We show such construction satisfies assumption [dd in the following corollary.
Corollary 1. Consider 1(, ,8,88,880) = - log P(s 8,0) and P.n - log P. Assume there exist 0,v such that
sup sup P(s'ls,a) < b, and, inf ("'ssss, 2m P'Ps'
(15)
5 2 (1 + 3) solves the following inequality for the local Rademacher complerily of P:
Rn vb+v
then assumplion 7 holds with aa = 01 and eest = cz for some constants Cı,
In the following, we prove suboptimality upper bound for the policy returned by algorithm
Theorem 2. Under assumption 1 if n) (1 (1-7)V
2 Aog((A) for every fixed T, then we have
Vp. - Vp. <
γ (1 - 7)2 (1- イ3 C't Best + (1 (1-7)2
1 2 loog A))
T
statistical crror
policy optimization error
with high probability, where a Unif (To. Ti. (T.....TT-))) WT-1).
The upper bound in Theorem 2 includes two terms: a statistical error (depending on fixed n) coming from using offline data for estimation, and an optimization error (depending on iteration T) coming from the policy improvement. Different from Theorem [ function approximation is not involved in this case. The sacrifice is that the update rule is computationally infcasible.
C Technical proofs
In this scction, we present all the technical proofs the main thcoretical results. We first prove thcorem in appendix C.11 which is a simplified version of theorem Then we prove theorem in appendix C.2 by further analyzing the effect of function approximation.
18

---

Published in Transactions on Machine Learning Research (10/2024)
C.1 Proofs of theorem 2
Proof of theorem & In this proof, we let (A) and x(6) = =- 8) for clarity. We first split the average regret into the sum of three parts. We will deal with the three parts separately.
Vp- T
E - vØ
(vÆ'-
+ +
+ Σ (v
VĀ
+
+
(V vp)
(16)
For the first term, we can upper bound Vi. Vi for each By the simulation lemma 1
p.
P.
7
E
(ead)aat (g V (6) P.(la))))) (s'
7
<
B
[| a) P((-ss,)) a111
?
7
$ (1 7)2
B (t.a)=apl || (-|s,a) - Pr(lss,))ll
(17)
$ (1 7)2
7
Cxt Cxt PP* (1s,, - a((lss,))ll
7
S (1 7)2 CztEess
where we used the definitions of Ct and Sest (see assumption
The third term in 16 negative with high probability: by assumption [d). p E Pn... with high
probability. Recall the updating rule in [. P/ = argmin PEP..an Vp. So V S Vp= for all with high probability. Then the following holds with high probability:
P
T (VP.
Vp:) S..
(18)
T
Then remains to upper bound Vp.
Vp.
By performance difference lemma,
T --1
T-1 7)
Vp.
E (y))) [Ap] (s, a)]
T(1
1=0
T-1
[AT (8, a)]
(19)
T(1 7)
t=
Σ
E (o.))aa)) E ((oa))ap))) (A
T(1 7)
To further deal with the term above, will first establish upper bound for the advantage function AT, as,
Recall the policy update rule in
(8) arg max
(Q7,(0)))))
I (
PEDm
T
or equivalently,
-(2 (8, ).p) + D (T(T
(s41(s) = arg min PEA.
(20)
1/.
By the optimality condition of 200 we have for any p E Am,
Q7 + 1 LV, (T, (m))))) , - Ret1(5) 20.
1/
19

---

Published in Transactions on Machine Learning Research (10/2024)
Note that D(Te.P) = w(p) w(T)) (Vw(T).. -²). We can explicitly write out the gradient term in the inequality above, then we get
(07 1 (VW(TTeI(s))) - (vb(t(()))))((((()))) ,p mct((s) 2 0.
(21)
By definition of D): we derive that
D( (( - D(TH)(ss)p)) (Vw(Te+(S)) - me41(s))
So 21
becomes
( -D) Tc+1(s)) (
1p.
We can rewrite it in terms advantage function:
(s..))
= - -V7 (s)
S (s)) V. +
- (( -Din D(T+++
= (1) T((s)) + 7
(D(T((s),)) ( - D(T(+1(5),P))
Let p p= x()):
(s,
1 (( (()((s) ((
(9% )
(s) Te TT(s))
1
$ (0% Te (s))
(1)) - (s)))
21/:
1 - ( (s)) - ((Te1(()) (t) (s))
1p
(s)lll - ((+)) (s)lf
(22)
s 107 (s, -1
2m
(D(TT(()) (())
(8)) (()
10 (s, -113 + 1 ()) ((T,)))))
Th
Th (D(m)()) - (((H))))))
<
2(1 - 7)2
where the third line above is because D(p',p) p) 2 2 11p-''ll (see section 3.2 For simplicity, we assume the is T--norm here. Even in the general case, recall that this norm || defined Ihm and by well known result in functional analysis, all norms
finite dimension lincar space equivalent. So
still establish step similar to the third line in (22) replacing the second term by C (|)e+(()) ne (s)
for some constant C. Then in the last line in 22 the first term changes to 3h @ and the remaining are
the same. We will see this difference does not affect the general form of the theorem, while it only changes constant.
20

---

Published in Transactions on Machine Learning Research (10/2024)
Then we use 22 to upper bound the first term in
19
B
[Ap (s,a)
(sa))app
E sndzl p+
(AP) (s, +), (s))
(23)
T-1
"/1
+ (( (D)
!
2(1 - 7)2
1=0
T-1
+ E
((Fr(s). (())
2(1 - 7)2
1=0
The second term 23 can be bounded by the following telescoping technique. By assumption, {n} is non-decreasing. Also note that the Bregman divergence non-negative, so we have
L D (T.(S).
1 (0)
T1t
T-1 + Σ (T((s)) (s)
D (To(s)) (+()
nhe Th-
1T-
(24)
t=1
1
D (T
7/0
1
Do-
Then 23
becomes
T-1 Do
E (s.))dot (A, s
1
Σ
1/ +
(25)
2(1 - 7)3
Yb
The second term in 19 modified MDPs, Mj (S.A.P. and M AS, Ph f,).. We still focus the policy mf and evaluate
be handled with simulation lemma: Let F(s,a) (s, a). Consider two (S it under both modified MDPs. Since the visitation measure only depends on the transition probabilities and the discounting factor, we can rewrite the expectation of i under visitation measure as the value function of modified MDP. Then directly apply simulation lemma:
E 1 1- (s.))aas)
E (nna)wagt AP. (s,
Va M,
Vaa
M
1 (s.))ddp
7
E
Va (8')
Ee- Vx M.
(5')
M.
21

---

Published in Transactions on Machine Learning Research (10/2024)
Note that the original reward function satisfies E [0, 1], so both QF) and V are bounded in [0, 1-y1 < (1-z.- So
Then s S r
IV
Me (6')
R ((oa))apt ) (s,a)
R (s
V (5') - Eewne (ea))))))) Ms
B (ora))aa;
1
Mr
< 1 -
7 B (s.))=a7i
Vg Me
[IP((-),,)) - (1))
(26)
$ (1 - 7)3
E (sa)) [|P((-]s,a) -P-(ss,,)-PP((()_.)))
CaIB(a (P((+)s ())))
7)3
CxtEest
where the penultimate line comes from assumption [bb, and the last step is by the definition of Eest (see assumption dd).
So now we can use
25
and 26 to control the two terms
19
T-1
T
Vp.
T
T-1
1
Σ
E
[Ap (s,a)]
+ 1
E
(a)))d - B -B(na)aap!)
[Ap] (s, a)]
(27)
T(1 7)
p-
T(1 7)
1=0
1=0
Do
Σ
ne
(1 (1-1) 7)3 C
21 - 7)
T(1
7)00
Finally, we use 17
27
to upper bound the three terms in
16
T
Vp-
Vp
T
1=0
T
(ip! vF
VØ V
(V3) - Vz:)
T-1
7
1
Do
7
< (1 7)2 CrIEest +
1)t +
+
CrI + 0
21 - 7)3
T(1 - 7)1-1)110
(1 7)3
1=0
T-1
< Ext + 9T(11-)) -
1
Σ
+
Du
1=0
C.2 Proofs for theorem
Proof of theorem 1.
We first focus on V% VA Let A= (A1, A2,.., Am), and the goal is to find an optimal
randomized policy in the probability simplex Am := {p Rm Em Pi 1, 2 0, b...,.m. Then, for given (s) E Am, use
2 (s, = R(s, A.) +7 7 / Vp. (s') P P(' 8, A,) ds' 1= 1,..., m,
(28)
1 V (T) ;= 11=
Su, A,) = 0% (s, A,) +
22

---

Published in Transactions on Machine Learning Research (10/2024)
to denote the action value function and the augmented action value function evaluated at the action A. Then for any p E Am
(w. p) = Q7 + (Vw (Te(s)) pp) /nc
m
Σ op As) + (Vu (T((s))) /m
(29)
(Qw.(())).))
where Qu, is defined m-dimensional vector with its i-th element as Qw, As).
That mcans, we want to approximate (w, which is a lincar function of 2w.e )) then we only neod to approximate Q,, A₂) for each = l,..., m. For each = -,.,, m, we consider the approximation
ft, Bra) zo Qw.e(s, A.)
(30)
where (s; Be,)) denotes some function class parameterized by 3. Then the update rule in is reduced
(
m = Vs VsE.s ...ii..s....
(s) arg max PEAm.
(31)
For simplicity, we consider an equivalent rule of 31
(.) 1 -o( Vs ES
T++1 (s) = arg min
(32)
PEAm
Here notice that since w is assumed to be strongly convex with modulus 1, and (si (), is convex function of we have a strongly convex objective function in (32 with modulus 1 n. Also, Am is convex space. Therefore the optimization procedure in (32 is meaninghul.
Then, by the optimality condition of [32 we have
1 ( } . (++1(s)) 20
(f((s: T()ss)))))))))) +
m
for any E Am- By expanding this inequality, we get
(-fols: B.).p *+)) 2 (V
Th
1 (D((F+(6))))) w(p) ((FH)1()) 7
We rewrite it as
(Qu,(s,))) ).p) -
((,11 + (c)))+(@w@u (s,, f((s; m()
(33)
1 w(pp) + - -((m) m Th
-w(Te+1(s)
th
We notice that (Qw,,s,) ).p) is exactly (s. For clarity, we denote
(.(s) (m.((s.)) 3.)
which represents the error in the approximation step in algorithm m-dimension vector. Plugging (s, (s.()) = 07 (s. (ww (m((s) .p) /n for any p, and
2 By previous notations, 6e(s) is also
23

---

Published in Transactions on Machine Learning Research (10/2024)
6((s) (u. f((s: Bs) into 33
then we have
(s,ps) - (Vw (m)()) p) /m + (6)(s).
(s, (Vw (Te (s)) Te+1 (s)) / + (5.(s), Te+1
(34)
(((++((()) w(p) D (())+168),))
m ne
By definition of D), we have
w (F)41(6)) w(p) (Vw (s)) -p) - -D( (m)(s) D (TT(s).)).
(35)
Then 34
becomes
(s, - (s.p)
(0)(s) p) (s))))))).
- ( (m)(s)).p)
(36)
q
Therefore we have
AB (e, (8)) (s.
= QP, (14 (841
(s) Qp, (s. (s)
= Qp (()(5)) - (s.p)
(37)
- p) + D (TT(+1(S).)) D (()())) - - (T((s)))pp)
2
Reorganize it and we have
A7
1 D(TT4((s))))
Ap To- ( (8c(s), 1+)((s) - p) + (T((ss))
"11
1 D (11(5)
D (TT(S))))) 1 D(Te41(s).))
I (s, 1 (s)) - (5.(s), (s) -
m
11T4(()) -s(())
21jr
(Q7) (s), 1141 ne(s)) 1
(38)
1
(s)
2ntt
1
(s). (44(8) p) + D (TT()))p))
((T+41(8),p)
71
1
< 1101 =x(s)]12
[|T)+((s)
(s))))
2m
i (TT()
1
(8r(s), (1) p) + 7)t
7t < Tt 1111 - (6,(s), (s) p) + (TT(s)).p) (sT
2
Th
where the third line above is because D(P. 1p p 2 (see section already explained in the proof of theorem 2 the remark after
3.2
The reason we assume mormorm
22
in section appendix C.1
24

---

Published in Transactions on Machine Learning Research (10/2024)
Now we return to the analysis for Vp Vp.
T E
CVV
-
T+1
1=0
T
E
Api ())
(39)
T
1
(s,a)app
1=0
T
T 1
Σ
E
Ap. (s, a)
Σ
(((aa)aapt - E B(ox))apt)) A4f ) (s.a). a).
E
T+1
1
(s.a)_pp
T
1-
1=
t=
For the first term in
39
we can let the randomizod policy p in
38
be r (()::
1
Σ
1
(x.))=pp p. (s.aa) a)
T+1
7
T
A (8, π (s))
T+1
1=0
T
Σ
E
1)t |011 (8), (s)1 -
(((6))))+1())-m(((()) a()))
(T 1)(1 - -7)
2
1=0
(40)
1 D (Te(S), (s))
(s) x(())
7/
1
Σ
1/t
1
Σ
B
smazt (5()) me+((s)
2(T 1)(1 3)3
(T 1)(1 - 7)
t=
t=
T
1 D x (8)) (=)
1
1 (s),
(8); (T(( (s),,
+ (T + 1)(1 - 7)
E
7
The second term in 40 is bounded by approximation error:
E Bwadt (6()), m++1(s))
E snaz!. (Ru.c(s,)) - Bi), a - ac+1(ss)
< E snapt (l0w,)(s,)) - f1(8; Bi) n (s)) + E gnapt (lāw,(0,)))
< 2|A|| max endatt Qu. Ad) (s. Sc,1)
< A| max V Boadpt (Qu.e(s, A.) - (e:())
det dPp (s)
< 2| max
Benddy (Qw,e(s, Ai) - (e.
2|A max;
dp (s) (8)
18 BaNdT (Qw. Ai) - (1,((3:
vTーク
25

---

Published in Transactions on Machine Learning Research (10/2024)
By the updating rule in 6
Bowd (Rw, - ft,((: fes(s:B(.s))
Fia
S min BeinBsndpp bodd (Sw, A:) - ft.i(:: + N
w.h.p.
+ Fral
S suppin inf Exwdy (S,, A₂) - Ja,(s; Bw.))) (s,((;
w.h.p.
P.z De,
N
Fi,il
approx
w.h.p.
N
where the second step follows from the analysis of standard M-cstimator (an de Geer & van de Geer and the last step is because of definition A.3
2000
Combine the previous two inequalities, then we get
2|A1
dp (s) (3)
max F1.s
endrt (be(s), - ss+1(s)) v
(Eapprox + cv
(41)
VI -
N
The third term
40
can be bounded by the same method from appendix
C.1
(see
24
So we have
Σ ( (T((s)))
1 (5) S Do-
(42)
n
By
40
we obtain the following inequality, which an upper bound for the first term 39
T
(na))apt 17, ())
T 1=
T
1
T]t
1
Σ
R
smazt (de(s), Te+1(s))
2(T +1)(1 7/3
(T + 1)(1 7)
1=
1=
+ (T + 1)(1 - 7)
E
Bandet
Σ
- D
1
(s))
1 D( (s),
(43)
p
T)t
t=0
1
Σ T)e
2|A1|
dz (s) (o(s))
(Capprox
eq
max,, [F.
2(T 1)(1 - 7)3
1- 7
-
Do
+ (T' + 1)(1 7)70
The second term in 33 can be handled by the same method from appendix C.1 (sce 26 ). So we have
E (s..))) E (s..)))dp)) A7) S
(1 7)3 Cet
7
(44)
So now we can use
43 and 44 to control the two terms in 39
respectively:
T-1
(V
v
T
T-1
1 T-1
E (s.))do!! (s, +
E (B(o.a)dd3)
B ) (s,a)
(45)
T
T
T-1
2|A1
app' (s)
Dn + 7 Crt Eest-
Σ
$ 2T 7)33
n + 1
Eapprox +
-
--
Mu((s)
T(1 7)mo (1 71-
1=0
26

---

Published in Transactions on Machine Learning Research (10/2024)
Finally, consider Vp. Vp.:
T
Vp
T
p
T
E 1=0
(v---
1 T
Vp
Vp.
+ 1=
+
-
(46)
r=0
The second term in 46 is already upper bounded by 45
The third term in 46 is negative with high probability: by assumption [c), P Pa,a, with high proba-
bility. Recall the updating rule in (1. Pt argmin Vp So Vp. S p. for all with high probability. Then the following holds with high probability:
- S..
(47)
The first term in
46
can be dealt by simulation lemma, which is same to
17 in appendix C.1
Vp. Vp. S (1 7]- E
(48)
By
45
46
47
48
T --1
T T-1
y=
1
Vp <
7
Eat + 2T - 33
Σ 7
T t=0
(1 - 7)
1=
2/A1|
dpp' (s)
maxi, Fe
+ 1 v1 11-1 -7
(Sapprox + ev
(a(s)
N
Do
+
?
Cat Eest +
T(1 - -7)170
(1 7)3
dpp (s)
maxe, Fr
e) C1 + +c2/A11/
sup
(Eapprox + ov
(s(s)
N
T-1
Do
Σ
m + T(1 71
27 7)3
t=
C.3 Proofs of proposition
Proof. Here adapt result from theorem 3 Specifically, given the conditions in proposition we have
||P - 12 S c102
and
E(((P) - I(P))) S c2
Now we prove the first result in proposition (P) where P minimize (P), and (P) ((P)s,,, 8)). We also use the notion L(P) = IEI(P) which is an population counterpart of L,. Then have
1 i.c., p E Pr.an with high probability. Consider [,*)
f..(P'') -P)
-In(P") L(P")++++0)) + L(P)) L(P) + -
(49)
=(')(P') [.) + +LP)) L(P")) + L(P")
-
(I(, (P) + ((()) - L(P")))
27

---

Published in Transactions on Machine Learning Research (10/2024)
the third term is less than 0. By (b) of theorem 3 we have
([nm(P") - L (P) +
(L,(P") Ln(P) L(P) L(P*)) S czŌn.
suu P
|P-P*
| 1P-P11e 12
Then we get
(Pt) (P)) (()'))) P*112 S c48
Therefore we get (P*) - [n(P) S c,6%, and implies
p Pn.
where a is to be co,
Next, we show that Es P((|s, - (* a)ll S cōn for every P E Pn.a. Here we incorporate assumption that Es as 11 (P, Pr) L(P) - L(P") for every P. Then we have
LPP - L(P")
= L(P) - [,(P) + Ln() LP,)) (P - L(P")
< [[P) (P) (P') I(P")) (P) [Pn))
(50)
< [(P)) - (P) + (P') - L(P") [Z. (P) [P) - [P')
< 01 + e20n + C2an
c5
Since 2 is an upper bound for TV distance, we have
sup PEPnan
Esawp (P()s,) (*(ss, a)ll S c62
And the proof is done.
C.4 Proofs of corollary 1
Proof. Let
(*(s a:)
L. (P)
P( a:)
1=1
and its population counterpart:
(* (s L(P) =B p(s' s,a) -
DKL (P"
We prove Pa E Pn,a, first. Consider - (P) where P minimize L, (P). Then we have
I.(P") - LT))
L(P) + I(P) -P') )- L(P") + -(P'')(((P))
(P)
(51)
=(a) (b) + (c)
Terms (c) can be bounded by
sup [T.n - [-)(
PEP
Again, we theorem to show that supper [T L.(P)) 62.
28

---

Published in Transactions on Machine Learning Research (10/2024)
For term (b), notice that -(P))-L(P)) L(P) (k=(P*))) Combining lemma which shows convergence rate of MLE under Hellinger distance and lermma which upper bounds KT. divergence by Hellinger distance when P has a lower bound, then we have
(6) DKL(P')PP) 8
with high probability. Then we have shown that
(Pt) S c6 =
with probability at least 1 8. This implies that P E Pa with probability at least 1 - 8.
For the second part, we show
sup (d a), P' ((((,,,))))) (,),,,,)))) < c20nn.
(52)
PEP
To see this, we bound the Hellinger distance by KT. divergence (lemma specifically, for any Pn... we
have
H2(P,P") s (L(PIIP")
T(P)) (PP))
(53)
- (P)
-
L(P))
Again, the first and the third terms are bounded by
sup [Lnn - [ - 82 PEP
The second term is bounded by a c because P E Pn.., The fourth term is equal to KT((PI I c3d by consistency of MLE in KL-divergence. And the proof is done.
D Supporting lemmas
Lemma 1 (A gencralization simulation lemma). Suppose S, A, r, Y Po are all fized. Here S and A can be infinite sels, and r S - R can be any real value function. For two arbilrary transilion models P and P. and any policy æ S + A(A), we have
V$
VI 7
E Ee(oaamd (aINP(lea) [V3(6)] - E
P(Nlaa))) VT
P
1 1
Vs E, then we further have If (s) is bounded, i.e. -C $ ( S
vz V < 20 1- Baa TTCPP(s..),
7
If Vs(s) is positive and bounded, i.e. 0 < (g( $ C, Vs E S, then
VF V < <C <
2 1-7 ( Ble.)
Proof. We first prove the first part of the lemma.
Let an) denote the visitation measure over (s. conditioning on (So sp. Ao ao) under transition
model P, i.e. d'c(,, 180, an) (1 7) 7 (S A, (0)
Then we have for any (sa, ao):
1
(7000. = 1- (1 ((s,
(54)
29

---

Published in Transactions on Machine Learning Research (10/2024)
By Bellman equation, for any (s,
(7( r((,,) + af ( (07('','))
(55)
0p(s, a) = r(s, a) + +1 f P(100) (p((',
(56)
Substitute the a(s,a) in 54 by the r(s, a) in 156
1
(7 ao) - 1-7 B( -
(p((,a)- -7B ((1 Op(l',)
(57)
By 54
and 55
we first apply
55 to the (7, ao) in 54 then apply
54
iteratively:
E (1
7
Qp (sp: ao)
r(80, an) + TT (- (00 a a-1o (s( a)]
r(sp: an) + y1E (
B Pes
Rearrange it as
-r(so, a0)
Es P( (1b50.00) 8) [Bo
7
Note that the equation above holds for any real function r: r:Sx A - R, so we can replace r(.,)) by (7,)))
- Op(00, ao) = ? E Pa (100.000) (10 a')]
(58)
y ((e1)) (8, a)).
57
58
@7 ao) (70 ao)
7
E (o0 ba Q7(5s)
B ( (s() (59)
E (s. (sylsaa
Consider the first term on right hand side:
Bo.. = E (s (ppp (1)
E(s.a) (s0 P
where (s, also, on) (= "PPT
8, A.41 (1 alSo 80, Ao an)-
30

---

Published in Transactions on Machine Learning Research (10/2024)
So 59
can be rewritten
Q7 (0) - (p0 ao)
E8 (xxa))dd5(rrloo
(x - - - 22(',
[ V (5') - ((1a(aa (/(
Finally, consider (so),, (so) and the initial distribution μ.. Recall that dz is the visitation mcasure conditioning on the initial distribution p. So we have
VØ V = Esorp (S()s) - V (su)
E Eso [7) un) (%0 ao)
? E Boo (lo))) (s (v1ls0,a)) ( (5)) P(loaa)) VT((5)
1-7
E. Eer (s') - E ( vo(())]
E 1 (ma)adB
which finishes the first part of the lemma.
Then we prove the second part: first note that
VB
Vg P
1-7 7 Blaa)ads [ - E B v7(0))
7
(60)
1 7 B(o,aaydd BeNP((aa) - E Bani((aa) (7(6))
Suppose 21, 22 two arbitrary probability distributions, and C' constant satisfying -C -C s C. By property of total variation distance, TV(11,22) =11l qalli.
By Hölder inequality
Fa f(x) - Penggg = Iʃ ((x)(((())) - 92(x))do
(61)
(1 - 22)1 s 11 1 42l1 S 2CTV(q1,9))
Apply 61 to the right hand side of 60
VØ
< 2C 1-9
E (saayaa IV)P()s..).PP(ss.)))
which concludes the second part.
Third part: Consider the special that 0 fxx) S C, then improve the upper bound in
61
31

---

Published in Transactions on Machine Learning Research (10/2024)
E Bang f() fx
/ (x)(a)(x) - - (2()))dr
(2(x))11(11)) (ooo((=))<(=)1111) / f(=)(q((z) (
(<)))dr / f(x)(22(1)- (x))) (x)))dr
Note that the right hand side, the two terms inside the absolute value sign are both non-negative, so
[aa (ax) - Exmgs[(1))
S Smax { ((0))1((0- (z(z))dz, /
-10)))(t))a(()))eeee (1))) s (2(x))dxx
Cmax ((((())-(())))(0)) (x())ddr / ( (2()))do
where the last step an equivalent definition of total variation distance (for two probability distributions).
So the factor 2 on the right hand side in 61 can be improved to in this case.
Lemma 2. If f is strongly conver with modulus and differentiable, i.e.,
f(y) 2 f(x) + Vf(x) - -x) 2 x1l?
(62)
suppose is a convex differentiable function, then f + is strongly conver function with modulus μ.
Proof. Since g is a convex function, we have
g(v) 2 + I).
(63)
Then
f(1) + +(v) 2 f(x) + )(x) + (v) + VV(a)) (v -x) + flly x112,
f())+
(64)
2
Theorem 3. Theorem 14.20 of WWinwrrigh, 2019 chap. 14) (Uniform law for Lipschitz cost functions)
Given uniformly 1bounded function class F that is star-shaped around the population minimizer f let 62 2 be any solution to the inequality
Rr (o;F)) 8².
(a) Suppose that the cost function is L-Lápschitz in its first argument. Then we have
PP (Cp L5) - P (C, Ly) 10L5, llf f* 112 8n
sup FEF
with probability greater than 1- oie c egnd6, (b) Suppose that the cost function is L-Lipschitz and y-strongly convex. Then for any function e F such that Fn (Cg- Ls) < 0, we have
| 1-111
20L
+1)6,
+1 8n
and
20L
P (Cg-
< 10L
2 62.
where both
inequalities hold with the same probahillty in part (a).
32

---

Published in Transactions on Machine Learning Research (10/2024)
Lemma 3. Corollory 14.22 of WWinwwright 2019). Given class of densities satisfying the previous condi-
lãons, let. on be any solution to the crilical inequality (14.58) such that 6% 2 (1 + +) n/2 Then the nonpara- metric density estimate f satisfies the Hellinger bound
H?2 (AI)*) - cg0
with probability greater than 1 c1e C2 bimm 1
Lemma 4. Lemma B.2 of Ghosal & Van der Vaart 2017). For every b > 0, there exists a constant 66 > 0 such that for all probability densities p and densities with 0 < d (p, q) < ppp(p/))
(1+ flr log dx(p, + log+ P
K(r:)) 2 (p.0)
+ 1 Q(x),
1
h 2
V2(p:)) 2 di (p.4)
| 1 log du (p. 9) log+ P
P
Furthermore, for every pair of probability densities p and q and any 0 < E < 0.4,
[oy [11x)
k(p:)) S (p 9) (1 2log c) +2
T(u/p sc)
[1ox)] 2)
2
V2(r: (p. a) (12 + 2 2log +
1alp se
Consequently, for every pair of probability densities p and 9,
K(p:) S (p.9)
1 [+ [ log
p
2d (p.g) 9)
p
(z(p:)) 2 ( (p.0)
1 1 log
p
2d (p.9)
E
Supporting algorithms
Monte Carlo algorithms for evaluating (s, A;) at for each i = 1,... m and (u, are provided.
F Additional results and details for the numerical studies
F.1 Synthetic dataset: an illustration
Environment and behavioral policy details For each episode that starts with an initial state 2), at time n a particle undergoes a random walk and transits according to a mixture of Gaussian dynamics: 8n+1 Sn =: As ~ VoN(HH. (1 0.1), where the discrete action a E (,1, corresponds to Teft, Stay, and Right, respectively. We choose the random walk steps ,1,-1 12, = 0, 11,0 = M1,1 = 0, P2.0 N221 as known parameters, and w/-1 = t/ = 0.6, 4/1 0.4 as the ground truth unknown model parameters that we estimate with expectation maximization (EM). We generate partially covered offline dataset collected by biased (to the left) behavioral policy B, and define goal-reaching reward function, respectively given by:
-2
-3 35850.
0.05 a = 1,
-1. 3 8 0
B(a
<< 0.05 a = 0, r(s')
0.9
-1,
8' 8'23, 3,
33

---

Published in Transactions on Machine Learning Research (10/2024)
Algorithm A Monte Carlo algorithm for evaluating (w, Ai) at
Input: The parametric function f1-1,1 (s; 11-1. Initialization: Let 8p = 8, do Ai, h O, and & r(50, A.) while TRUE do
Generate U ~ unif 1]. U < <- then
Break.
else
Sample Sh. Pi(. 9h-1;00-11.
Solve
(((h)) = arg max P'EA(A)
1, (8: Be 1,)''
(65)
n
i=1
Generate an ~ (s()))
-= 9 +r(hh, an).
h=
end if
end while
Let Q7, (s, ==
Let Qw. = =Q (s, + 1 W
Vi (s));
Algorithm A Monte Carlo algorithm for evaluating Qu,, a) for the continuous-action settings
Input: The parametric function fc-1(s, B,-1). Initialization: Let 80 = 8, oa = a, h = O, and r(80,
while TRUE do Generate U unif 1]. U <1-1 then
Break.
else
Sample sh Pi(: 0h-1,, Solve
(1(h)) = arg max [ u(a')
1
(66)
e'Ed
n
Generate an ~ (((s)).
0= 9 r an). h1
end if end while
Let (s, = q.
Let Qw.((s,)) = Q ()) + m Ve (#{(s));
for all 8 E RR. Thus, the particle is encouraged to reach either the positive or negative terminal state with the shortest path possible, with a slight favor towards the positive end if the particle starts near The offline dataset contains 50 episodes, which are sufficient for an accurate estimation of Φi but may lead to misestimation of vh and 111 Indeed, for particular dataset, while the MLE w is accurate, vo is underestimated and ii is overestimated, which could make over-cxploitation of the MLE problem.
Implementation details We implement MoMA strictly following Algorithm We choose D(:-) to be KT. divergence, reducing the policy improvement steps to natural policy gradient (NPG) mentioned in
34

---

Published in Transactions on Machine Learning Research (10/2024)
Section 3.2 We parameterize by ft,(s, B1,) A.), A,) =1,,,..... 1,.., where the features e(s, A,) are chosen
to be exponential functions.
Contribution from pessimism: accompanying figure
The accompanying figure referenced in the
study of Contribution from Pessimism in Section 6.1 is given Figure 2
Model Parameter value Epoch
0.8
0.6
0.4
0
10
20 Epoch
30
40
Figure 2: The estimated model parameter yo modified by the pessimism updates. As result, MoMA policy shifts away from the faulty action Stay.
Hyperparameters and compute information experiment are summarized table The entire run (training and evaluation) of MoMA on standard CPU takes less than one hour.
The byperparameters MoMA used in the random walk
Table 3: Hyperparameters for the random walk experiment
Hyperparameter
Value
actor steps
150
model steps
150
0.1
81
0.1
A
3.0
MC number
300
7
iterations
40
F.2
The advantage of nonparametric policy class
We provide an illustrative example to empirically validate the importance of the nonparametric policy class, demonstrating scenarios where the limitations of parametric policies are evident.
Consider an MDP with states $ = are defined as follows:
and an action space = {0, 1}. The transition dynamics
P(s' 2 1, 0.8, P(s' = 1.a = 0) 0.2, P(s' = 3 1,a 0.9, P(s' = 1 s- 1,a 1) 0.1. P(s' = k k, =1 for all k = 2,3.4.5 and 0, 1.
35

---

Published in Transactions on Machine Learning Research (10/2024)
The reward function is set follows:
r(s 1,a 1) 10,
r(s 1,a 0) = 5,
r(s = 2. 1) 10,
r(s = 2, 0) = 1,
r(s, a) = sin(s) + cos(a) for 8 = 3, 4,5 and a = 0, 1.
The discount factor y is set to 0.9. Under these settings, the optimal policy x x = 1) = 1, (5 (s= = 2) = and I (s))) = for all 3, 4, 5.
We aim compare the results of the proposed nonparametric policy class and the log-linear policy class, defined as:
(a = 0 s) <(a 5) Ta = s).
1 + 0 +0,²
Based on these settings, we applied both a parametric policy gradient method and the proposed nonpara- metric method. The results indicate that the optimal parametric policy yields sub-optimal policy with a value of 23.50, whereas the proposed nonparametric method succossfully identifies the optimal policy with value of 27.08. This demonstrates the superiority of the nonparametric policy method over pre-specified policy class.
F.3 Continuous action D4RL benchmark experiments
The hyperparamcters of MoMA used in the D4RL ex- periments are summarized in Table We train and evaluate MoMA as well as baseline algorithuns one A100 GPU for all D4RL experiments, and summarize the wall-clock times in Table 5
Hyperparameters and compute information
Table 4: Hyperparameters for the D4RL experiments
Hyperparameter
Value
3E-4
3E-4
5E-5
0.99
Table 5: D4RL benchmark wall-clock times, rounded to hours.
MoMA
MOPO
IQL
CQL
Hopper, medium
6.9
4.0
6.1
Hopper, medium-replay
8.1
3.2
2.4
5.5
Hopper, medium expert
6.9
2.5
2.3
5.5
HalfChectah,mediug
8.7
4.4
5.6
8.1
HalfCheetah, medium-replay
8.7
4.4
3.2
8.1
HalfCheetah, medium-expert
9.0
4.5
4.3
8.3
Walker2D medium
10.3
2.7
2.8
6.4
Walker2D, medium-replay
8.6
5.9
Walker2D, medium-expert
8.9
3.2
3.4
6.0
Alternative evaluation metrics As supplement to the standard mean of evaluation scores reported in Table 2 we further consider the evaluation scheme proposed by Agarwal et al. (2021). Their method addresses the need for more reliable performance evaluation in deep reinforcement learning, particularly in
36

---

Published in Transactions on Machine Learning Research (10/2024)
scenarios with limited seeds. It emphasizes the use of distributional metrics, which are less sensitive to outliers, providing clearer and more robust picture of algorithm performance across different runs.
Specifically, for our MoMA, model-based baseline MOPO, and model-free baseline CQL, we plot the aggre- gate metrics including interquartile mean (IQM), mean, and optimality gap together with 95% bootstrap confidence intervals (CIs) for each one of the 9 tasks in Figure 3 TQM has better statistical efficiency than median, while optimality gap is a robust alternative mean. Higher IQM and mean scores are better, and lower optimality gap score is better. We further supplement the aggregate metrics with performance profiles based on score distributions (Agarwal et al., 2021), defined as the fraction of runs above certain score threshold, and higher curve is better. We plot the performance profiles and bootstrap 95% confidence bands in Figure 4 For the HallCheetah environment, MoMA consistently oulperforms the two baselines across the three data settings measured by all three metrics, and exhibit uniformly higher performance profiles. For the medium-expert data setting, MoMA achieves the best or competitive results across the three environments.
194
Moen
Optimalty Gaz
12M
Mezn
Dosinaiig Ges
Me4r
Optima By G83
TEA
haffowert wedi modlyye
Nalthero radio exparit
02
Mean
Octivalty
Figure 3: Aggregate metrics on D4RL tasks with 95% bootstrap Cls.
37

---

Published in Transactions on Machine Learning Research (10/2024)
mome
mopo
mome mapo
O0
mone
mupo
0.75
Tracton oww.ccceooote. come
2.5
Eounton of coge
6.70
0.00
0.00
0.00
1.00
1.00
3.00
0:20
150
Toutow our
0.20
1.0
0.20
101
100
170
100
mome
mapo
co
moma
mapo
mome
mape
cql
1.20
1.00
1.00
0.23
0.00
Wille www.eemaaaittteooo. pa11000
Figure 4: Performance profiles based on score distributions, pointwise 95% confidence bands.
38


Reviews:
[{"review_id": "ZpoBrc3jyJ", "reviewer": "Reviewer_VumW", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n* MoMA is a blend of two historically justified RL ideas: (a) picking a conservative estimate of V using a pessimistic MDP across plausible MDPs and (b) finding a policy that maximizes a regularized expected sum of rewards.\n* The paper claims that it has a better expressiveness of the policy class and more flexibility of function approximators (though see discussion below).\n* Theoretical analysis is provided.\n\nWeaknesses:\n* General\n\t* The paper includes long proofs (in Appendix). It would be useful to:\n\t\t* Provide a sketch of the main argument in the paper's body.\n\t\t* Compare the complexity results obtained in this paper with the known facts from the literature (e.g., in the form of a table).\n\t* Slightly related to the previous item, the content of Appendix A is interesting and could be included in the main body.\n\t* The paper claims two strengths of MoMA in the form of (a) unrestricted policy class (quote from Section 3.1: \"has no restriction on the policy class, which is crucial when the optimal policy is not contained in a restricted parametric policy class\"), and (b) policy updates with general function approximations (quote from Section 3.2 \"the update rule [...] does not require any explicit policy parameterization\"). However, an approximate version of the algorithm is used for the experiments with neural networks and SGD optimization. Additionally, the results in the experiments section are not as convincing as they are claimed to be in the paper (see below).\n* Experiments:\n\t* The experiments have a small number of seeds (5).\n\t* The confidence intervals are misleading. First, only five seeds are used for the computations. Second, the CIs are constructed as the mean +/- std of the result. Such an approach is usually done with the asymptotic assumption (which is controversial given the number of seeds), in which case one assumes the CI form for the normal distribution. In this case, however, +/- std corresponds to a confidence level $\\approx 68\\%$, an uncommonly low figure. As a side-note, for the general case with a finite std, one could hope to use the Chebyshev's inequality, but it is uninformative in the case at hand, giving a trivial upper-bound of $100\\%$.\n\t* The above leads to artificially narrowed down CIs and, consequently, to overoptimistic interpretation of the results. \n\t* On page 9, the paper claims \"notably, in 4 out of 9 cases, our algorithm outperforms the other three model-based RL algorithms\" (referring to Table 2). \n\t\t* There are two model-based baselines.\n\t\t* Taking into account the overlap between confidence intervals (which are misleadingly narrow) MoMA significantly beats **model-based** baselines only in one case (Hopper, medium-expert). However, it does not significantly beat all baselines (model-based and model-free combined).\n\t* There is no comparison of wall time or compute between the methods (only how long did the MoMA training and eval take in Appendix H2)\n* Other remarks:\n\t* Undefined interval $\\Lambda$ (below equation (4)).\n\t* The form of $\\omega$ function assumed in the experiments section should be made clear.\n\t* Section 4.2 is dense with multiple inline formulas, which is hard to read and not immediately easy to follow.\n\t* For random walk, $\\gamma=0.4$ (Appendix H.1.), which seems to be unusually low.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a MoMA algorithm for offline policy training. MoMA works iteratively, with each iteration comprised of two stages. In the first stage (conservative policy evaluation), an MDP that minimizes the value function of a current policy is chosen among all plausible MDPs. In the second stage (policy improvement), the next step policy is derived as a maximizer of an expected current $Q$-value under a (soft) constraint penalizing the distance from the current policy. The paper includes theoretical and experimental results.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See above.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "yXOKL4AChl", "reviewer": "Reviewer_Pmk3", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "__Strengths__\n\n* Novel algorithm design that combines conservative policy evaluation --- involving solving for a constrained model optimization, followed by a policy improvement step using mirror ascent.\n* Detailed theoretical exposition\n* Practical implementation and ablation insights: the implementation of MoMA is complemented by numerical experiments on D4RL continuous control tasks, showcasing its effectiveness. Additionally, Figure 1 provides interesting ablative insights into MoMA's approach to value conservatism, compared to NPG.\n\n__Weaknesses__\n\n* Lack of empirical validation for theoretical claims: The paper does not empirically demonstrate the claimed benefits of MoMA’s independence from the policy class parameterization. The environments used, including those from D4RL, do not necessitate non-parametric policies, thus not substantiating the theoretical advantage.\n* Limited Value of Synthetic Data Experiment: The synthetic data experiments show overlapping confidence intervals among competing methods in Table 1, suggesting that MoMA’s advantages may not be statistically significant in controlled environments.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper presents MoMA (Model-based Mirror Ascent), an offline reinforcement learning algorithm developed to function effectively under partial data coverage without the use of parametric policy spaces. The algorithm operates through two primary steps: a conservative policy evaluation, where the Q function is minimized over a confidence set of transition models to accommodate partial coverage scenarios; and a policy improvement step, which employs mirror ascent with general function approximations, facilitating the use of an unrestricted policy class. This approach is designed to enhance the algorithm's applicability in practical settings.\n\nFrom a theoretical perspective, the authors provide an upper bound on the suboptimality of MoMA, noting that it does not incorporate the size of the policy class, which may allow for broader application compared to model-free methods. The paper also details a practically implementable version of MoMA, including a primal-dual step designed to approximately solve the constrained minimization problem. The efficacy of MoMA is evaluated through numerical studies using both synthetic data and the D4RL benchmark.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The paper already represents a valuable addition to the literature on offline reinforcement learning. To strengthen the submission further, the following revisions are suggested:\n\n1. Demonstration of non-parametric policy benefits: The theoretical benefit of MoMA operating independently of the policy class parameterization is highlighted but not empirically validated. It would be beneficial if the authors could include experiments in settings where the limitations of parametric policies are evident, or demonstrate situations where the optimal policy would likely fall outside a typical parametric class. This would substantiate the theoretical advantages discussed.\n\n2. Reevaluation of synthetic data results: The overlapping confidence intervals reported in Table 1 suggest that the differences in performance between MoMA and other methods may not be statistically significant. A revision of the experimental design, perhaps by introducing a more challenging synthetic dataset, could help in demonstrating MoMA's effectiveness more convincingly. This might involve designing experiments that more clearly showcase the algorithm's strengths in controlled settings.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "xYfph0hQMZ", "reviewer": "Reviewer_FipA", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- The paper has a solid feel and the proposed ideas seem interesting and well-motivated\n- The proposed algorithm is very general and seems amenable to many domain-specific variations\n- The article goes all the way to a practical implementation and evaluation on the challenging D4RL benchmark.\n\nWeaknesses:\n\n- The clarity of the paper is overall quite low, and the paper seems to have been crammed in the page limit, with, possibly, various authors fighting for space. As a result, some very important parts of the paper are left out (pushed to the Appendix or completely left out), such as the entire page that should be in place of \"we adjust the implementation accordingly\"\n- The order in which things are presented is sub-optimal. The paper is constantly referring to the future, omitting intuition and replacing it with \"we will show an example in Section Y\".\n- In general, there is a lack of intuition or plan behind the formulae. It starts just above Equation 1, with a paragraph that directly starts with notations involving $\\mathcal{E}_n(P)$, without giving a definition of these symbols, nor an explanation of where we are going.\n- Section 4.3, \"An example\", should be much larger (and also describe the continous-action setting). It presents several contributions, leverages existing work and is smart about how to compute the gradient of the model, but everything is super-compact and quite unreadable.\n- The numerical results are too bad for a discussion of them to be omitted. The proposed algorithm does not match the SOTA, while the paper regularly claims the theoretical superiority of the proposed method. What is the catch? Is the proposed method much more compute-efficient, stable or easy to tune that the related work? Are there easy ways to improve the algorithm to match the SOTA?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The article proposes a new method for off-line Reinforcement Learning based on a model-based conservative estimate of policies (given the offline data available), followed by a policy optimization step based on these conservative estimates. The article starts from a theoretical motivation and introduction of the work, then progresses up to a practical implementation that works with continuous states and actions, evaluated on the D4RL benchmark. These numerical experiments demonstrate that the proposed method almost matches the state of the art.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Critical: improve the clarity of the paper by introducing intuition and writing the paper in a way that the reader does not have to wait for future sections to get an \"example\" or intuition on how to use a formula\n2. Critical: discuss the empirical results to explain why MoMA is still a good algorithm even though it does not match the SOTA in performance. Matching SOTA is not required in a paper (the new RL conference does a great job at explaining why), but here, the paper regularly mentions its superiority to related approaches, and this superiority should either translate to better numerical results, or some other metric being better (compute efficiency, stability, generality).\n3. Important but not critical: the paragraph above Equation 5, that discusses the augmented value function, is unreadable. The augmentation should be made much clearer, the notation should be introduced, and ideally examples or intuition should be used.", "broader_impact_concerns": "The paper does not raise broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents a novel approach to offline reinforcement learning with theoretical foundations and practical implementations. While there are some concerns about the clarity of presentation and the strength of empirical results, the innovative aspects of the work - particularly the use of an unrestricted policy class and the combination of conservative policy evaluation with mirror ascent - make it a contribution to the field.

To address the reviewers' concerns, the authors should be encouraged to:

- Improve the clarity of the paper, especially in explaining theoretical concepts and providing intuition for the algorithms.
- Strengthen the empirical validation, particularly in demonstrating the benefits of the unrestricted policy class.
- Provide a more balanced discussion of the results, acknowledging where MoMA does not outperform existing methods and explaining potential reasons.

These revisions would enhance the paper's impact and usefulness to the TMLR audience without fundamentally altering its core contributions.
Decision: Accept (poster)