Doubly Mild Generalization for Offline Reinforcement Learning
Authors: Yixiu Mao
Cheems Wang
Yun Qu
Yuhang Jiang
Xiangyang Ji
Venue: NeurIPS 2024 poster (2024)
Keywords: offline reinforcement learning
Primary area: reinforcement_learning
TL;DR: This work proposes Doubly Mild Generalization, comprising mild action generalization and mild generalization propagation, to appropriately exploit generalization in offline RL.

Abstract: Offline Reinforcement Learning (RL) suffers from the extrapolation error and value overestimation. From a generalization perspective, this issue can be attributed to the over-generalization of value functions or policies towards out-of-distribution (OOD) actions. Significant efforts have been devoted to mitigating such generalization, and recent in-sample learning approaches have further succeeded in entirely eschewing it. Nevertheless, we show that mild generalization beyond the dataset can be trusted and leveraged to improve performance under certain conditions. To appropriately exploit generalization in offline RL, we propose Doubly Mild Generalization (DMG), comprising (i) mild action generalization and (ii) mild generalization propagation. The former refers to selecting actions in a close neighborhood of the dataset to maximize the Q values. Even so, the potential erroneous generalization can still be propagated, accumulated, and exacerbated by bootstrapping. In light of this, the latter concept is introduced to mitigate the generalization propagation without impeding the propagation of RL learning signals. Theoretically, DMG guarantees better performance than the in-sample optimal policy in the oracle generalization scenario. Even under worst-case generalization, DMG can still control value overestimation at a certain level and lower bound the performance. Empirically, DMG achieves state-of-the-art performance across Gym-MuJoCo locomotion tasks and challenging AntMaze tasks. Moreover, benefiting from its flexibility in both generalization aspects, DMG enjoys a seamless transition from offline to online learning and attains strong online fine-tuning performance.

Doubly Mild Generalization for Offline Reinforcement Learning
Yixiu Mao', Qi Wang', Yun Qu', Yuhang Jiang', Xiangyang Ji
Department of Automation, Tsinghua University wtaiiiisssssssniiee tsinghua. edu. en, xyji@tsinghua edu, cn
myx210mails
Abstract
Offline Reinforcement Learning (RL) suffers from the extrapolation error and value overestimation. From generalization perspective, this issue be attributed to the over-generalization of value functions or policies towards out-of-distribune actions. Significant efforts have been devoted to mitigating such generalization, and recent in-sample learning approaches have further succeeded in entirely eschewing it. Nevertheless, we show that mild generalization beyond the dataset can be trusted and leveraged to improve performance under certain conditions. To appropriately exploit generalization
(OOD)
offline RL, we propose Doubly Mild Generalization (DMG), comprising (i) mild action generalization and (ii) mild generalization propagation. The former refers to selecting actions in a close neighborhood of the dataset to maximize the Q values. Even so, the potential erroneous generalization can still be propagated, accumulated, and exacerbated by bootstrapping. light of this, the latter concept is introduced to mitigate the generalization propagation without impeding the propagation RL learning signals. Theoretically, DMG guarantees better performance than the in-sample optimal policy in the oracle generalization scenario. Even under worst-case generalization, DMG can still control value overestimation Empirically, DMG achieves state-of-the-an performance across Gym- Gym-MuJoCo locomotion tasks and challenging AntMaze tasks. Moreover, benefiting from its flexibility both generalization aspects, DMG enjoys a seamless transition from offline online learning and attains strong online fine-tuning performance.
certain level and lower bound the performance.
Introduction
Reinforcement learning (RL) aims to solve sequential decision-making problems and has garnered significant attention recent years [53, 67, 74, 63, 12]. However, its practical applications encounter several challenges, such risky exploration attempts [20] and time-consuming data collection phases [35]. Offline RL emerges as a promising paradigm to alleviate these challenges by learning without interaction with the environment [40, 42]. It eliminates the need for unsafe exploration and facilitates the utilization of pre-existing large-scale datasets [31, 48,48,550 59].
However, offline RL suffers from the out- distiiiiibtttoi (OOD) issue and extrapolation error [19]. From a generalization perspective, this well-known challenge can be regarded as a consequence the over-generalization of value functions or policies towards OOD actions [47]. Specifically, the potential value over-estimation at OOD actions caused by intricate generalization often improperly captured by the max operation [73]. This over-estimation will propagate to values of in-distribution samples through Bellman backups and further spread to values of OOD ones via generalization. In mitigating value overestimation caused by OOD actions, substantial efforts have been dedicated [19, 39, 38, 17] and recent advancements in in-sample learning have successfully formulated the Bellman target solely with the actions present in the dataset [37, 85, 92, 88, 21] and extracted policies by weighted
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

behavior cloning [57, 80]. As a result, these algorithms completely eschew generalization and avoid the extrapolation error. Despite simplicity, this way can not take advantage of the generalization ability of neural networks, which could be beneficial for performance improvement. Until now, how to appropriately exploit generalization in offline RI. remains lasting issue.
This work demonstrates that mild generalization beyond the dataset can be trusted and leveraged to improve performance under certain conditions. For appropriate exploitation of mild generalization, we propose Doubly Mild Generalization (DMG) for offline RL, comprising (i) mild action generalization and (ii) mild generalization propagation. The former concept refers to choosing actions in the vicinity of the dataset to maximize the Q values. However, the mere utilizzation of mild action generalization still falls short in adequately circumventing potential erroneous generalization. which can be propagated, accumulated, and exacerbated through the process of bootstrapping. To address this, we propose a novel concept, mild generalization propagation, which involves reducing the generalization propagation while preserving the propagation of RI. learning signals. Regarding DMG's implementation, this work presents a simple yet effective scheme. Specifically, we blend the mildly generalized max with the in-sample max in the Bellman target, where the former is achieved by actor-critic learning with regularization towards high-value in-sample actions, and the latter is accomplished using in-sample learning techniques such as expectile regression [37].
We conduct a thorough theoretical analysis of our approach DMG in both oracle and worst-case generalization scenarios. Under oracle generalization, DMG guarantees better performance than the in-sample optimal policy in the dataset [38, 37J. Even under worst-case generalization, DMG can still upper bound the overestimation of value functions and guarantee to output a safe policy with a performance lower bound. Empirically', DMG achieves state-of-the-art performance on standard offline RL. benchmarks [16], including Gym-MuloCo locomotion tasks and challenging AntMaze tasks. Moreover, benefiting from its flexibility in both generalization aspects, DMG can seamlessly transition from offline to online learning and attain superior online fine-tuning performance.
2 Preliminaries
RL. The environment in RL is mostly characterized as a Markov decision process (MDP), which can be represented as a tuple M (S. A, P. R, y, do), comprising the state space S, action space A. transition dynamics P S A + A(S), reward function R: : x A + [0, Rmax], discount factor [0, 1). and initial state distribution do [70]. The goal of RL is to find a policy : S + ((A) that can maximize the expected discounted return, denoted as ((r):
J(x) =
x' RR(s an)
(I)
For any policy æ, we define the value function V**(s) Fx 1 Eg 7tt R(s,, ar)loo s] and the state-action value function (Q-value function) as Q (s, E, IIt 7 "R(s a;) 30 = s, 0g a].
Offline RL. Distinguished from traditional online RL training, offline RL handles a static dataset of transitions D {(s,, a, rı. )=) and seeks an optimal policy without any additional data
collection [40, 42]. We use B(a]s) to denote the empirical behavior policy observed in D, which depicts the conditional distributions in the dataset [19]. Ordinary approximate dynamic programming methods minimize temporal difference error, according to the following loss [70]:
(TD = E,,,,."/> (Qo(s.a) R(s, a) 7 max Qu (s', (s',')))]
where nh is a parameterized policy, Qo(s, a) is a parameterized Q function, and Qu (s, Q) is a target Q function whose parameters are updated via Polyak averaging [53].
3 Doubly Mild Generalization for Offline RL
This section discusses the strategy to appropriately exploit generalization in offline RL. In Section 3.1. we introduce formal perspective on how generalization impacts offline RL and discuss the issues of
Our code is available at hutps//gihhubccom/mnoyyvv/////iu

---

over-ge serreeneralizaion and non- generalization. Subsequently, we propose the DMG concept, comprising mild action generalization and mild generalization propagation in Section 3.2. Following this, we conduct a comprehensive analysis of DMG in both oracle generalization (Section 3.3) and worst-case generalization scenarios (Section 3.4). Finally, we present the practical algorithm in Section 3.5.
3.1 Generalization Issues in Offline RL
Offline RL training typically involves a complex interaction between Bellman backup and generaliza- tion [47]. Offline RL algorithms vary in backup mechanisms train the Q function. Here we denote a generic form of Bellman backup as Tu. where is a distribution in the action space.
(oQ(s, a) = R(s, a) + +.BomP((saa)
max Q(s', aa) aa'mu(-lo")
(3)
During offline training, this backup is exclusively executed on (s, a) E DD and the values of (s, a) & D are influenced solely via generalization. A crucial aspect is that (s', a') in the Bellman target can be absent from the dataset D. depending on the choice of u. As result, Bellman backup and generalization exhibit an intricate interaction: the backups on (s, a) 'D impact the values of (8, a) & via generalization; the values of (8,a) & 'D participates in the computation of Bellman target, thereby affecting the values of (s, a) E D.
This interaction poses a key challenge in offline RL, value overestimation. The potential overestima- tion of values of (s, a) & 'D, induced by intricate generalization, tends to be improperly captured by the max operation, a phenomenon known as maximization bias [73]. This overestimation propagates to values of (s, a) E 'D through backups and further extends to values of (s, a) & 'D via generaliza- tion. This cyclic process consistently amplifies value overestimation, potentially resulting in value divergence. The crux of this detrimental process can be summarized as over-generalization.
To address value overestimation, recent advancements in the field have introduced a paradigm known as in-sample learning, which formulates the Bellman target solely with the actions present in the dataset [37, 85, 92, 88, 21]. Its effect is equivalent to choosing u. in Tu to be exactly B, i.., the empirical behavior policy observed in the dataset. Following in-sample value learning. policies are extracted from the learned Q functions using weighted behavior cloning [57, 55.. By entirely eschewing generalization in offline RL training, they effectively avoid the extrapolation error [19], strategy we term non-generalizaton.. However, the ability to generalize is critical factor contributing to the extensive utilization of neural networks [41]. In this sense, in-sample learning methods seem too conservative without utilizing generalization, particularly when the offline datasets do not cover the optimal actions in large or continuous spaces.
3.2 Doubly Mild Generalization
The following focuses on the appropriate exploitation of generalization in offline RL.
We start by analyzing the generalization effect under the generic backup operator Tu. We consider straightforward scenario, where Qa is updated to Qu by one gradient step on a single (s, a) E D with learning rate a. We characterize the resulting generalization effect on any (s, & DD as follows.
Theorem 1 (Informal). Under certain continuity conditions, the following equation holds when the learning rate a is sufficiently small and ā is sufficiently close to a:
Qu (s, â) Qo(s, à) C1 (T (s, @) Qo(s,ā) Cz - all) + +0 00100 - -013)
(4)
where C1 1) and C2 is a bounded constant.
The formal theorem and all proofs are deferred to Appendix B.
Note that Eq. (4) is the update of the parametric Q function (Qe + Q+:) state-action pairs (s, à) & D. which is exclusively caused by generalization. If à is within a close neighborhood of then C2 l| all is small. Moreover, as C₁ E [0, 1]. Eq. (4) approximates true objective Ta Qo, a), as Q0 à) is updated by a true gradient step at (s, a) & 'D. Therefore,
update towards the
"The interplay between backup and generalization does not involve states out of the dataset (Bellman target does not contain OOD states), hence do not consider (5, a) D. though the analysis of Q(3, ā) is similar.
3

---

Theorem 1 shows that, under certain continuity conditions, Q functions can generalize well and approximate true updates in a close neighborhood samples in the dataset. This implies that mild generalizations beyond the dataset can be leveraged to potentially pursue better performance. Inspired by Theorem I, we define a mildly generalized policy 3 as follows.
Definition (Mildly generalized policy). Policy B is termed a mildly generalized policy if satisfies
supp(((-1s) S supp(3(-1s)), and max min ||a - a2ll S Co. a1-3(()))) a2--B(-1s)
(5)
where B is the empirical behavior policy observed in the offline dataset.
It means that / has wider support than | (the dataset), and for any an B()ss), can find a2 ~ B(-ls) (in dataset) such that ||oı - azl S Ca. other words, the generalization of 3 beyond the dataset is bounded by when measured in the action space distance. According to Theorem 1, there is high chance that Qo can generalize well in this mild generalization area ((ass) > 0.
However, even in this mild generalization area, it inevitable that the learned value function will incur some degree of generalization error. The possible erroneous generalization can still be propagated and exacerbated by value bootstrapping as discussed in Section 3. To this end, introduce an additional level of mild generalization, termed mild generalization propagation, and propose a novel Doubly Mildly Generalization (DMG) operator follows.
Definition 2. The Doubly Mild Generalization (DMG) operator is defined
Ts, == R(s,a) + max d'~B(1P0)
Q(s',a') (1
max Q15 (d)) al~B(113)
(6)
where B is the empirical behavior policy in the dataset and is a mildly generalized policy.
Note that in typical offline RL algorithms, extrapolation error and value overestimation caused by erroneous generalization are propagated through bootstrapping, and the discount factor of this process y. DMG reduces this discount factor to Xr, mitigating the amplification of value overestimation. On the other hand, in contrast to in-sample methods, DMG allows mild generalization, utilizing the generalization ability of neural networks to seek better performance, as Theorem suggests that value functions are highly likely to generalize well in the mild generalization area.
To summarize, the generalization of DMG is mild in two aspects: (i) mild action generalization:
based on the mildly generalized policy 3, which generalizes beyond },, DMG selects actions in a close neighborhood of the dataset to maximize the Q values in the first part of the Bellman target; and (ii) mild generalization propagation: DMG mitigates the generalization propagation without hindering the propagation of RL learning signals by blending the mildly generalized max with the in-sample max the Bellman target. This reduces the discount factor through which generalization propagates, mitigating the amplification of value overestimation caused by bootstrapping.
To support the above claims, we provide a comprehensive analysis of DMG in both oracle and worst-case generalization scenarios, with particular emphasis on value estimation and performance.
3.3 Oracle Generalization
This section conducts analyses under the assumption that the learned value functions can achieve oracle generalization in the mild generalization area B(a]s) > 0, formally defined as follows.
Assumption 1 (Oracle generalization). The generalization of learned Q functions in the mild generalization area ((ass) > O reflects the true value updates according to TDMG-
The mild generalization area 3(als) > 0 may contain some points outside the offline dataset, and TDMG might query Q values of such points. This assumption assumes that the generalization at such points reflects the true value updates according to TDMG. The rationale for such an assumption comes from Theorem 1, which characterizes the generalization effect of value functions in the mild generalization area. Now we analyze the dynamic programming properties of the operators TDMG and Tim. where Tia is the in- sample Q learning operator [37, 88, 21] defined as follows.

---

Definition 3. The In-saplle Q Learning operator [37] is defined as
TinQ(s,) a) == R(s, a) + +7BB'
max a'--8t-15")
((s',a')
where B is the empirical behavior policy in the dataset.
Lemma 1. Tin is "-contraction operator in the in-sample area B(a|s) > 0 under the Loo norm.
Following Lemma 1, we denote the fixed point of TIn as Vin and its induced policy as via Here Qu is known as the in-sample optimal value function [37], which is the value function of the in-sample optimal policy Ti We refer readers to [37, 38, 88] for more discussions on the in-sample optimality.
Now we present the theoretical properties of DMG for comparison.
Theorem 2 (Contraction). Under Assumption 1, TIMa is a "-conttaccion operator in the mild generalization area M(als) > under the Loe norm. Therefore, by repeatedly applying TDMG, any initial Q function can converge to the unique fixed point QDG.
We denote the induced policy of QiMG as TDMG. whose performance is guaranteed as follows. Theorem (Performance). Under Assumption 1, the value functions of DMG and Thn satisfy:
VaDua (s) V (s), Vs E D.
(8)
Theorem 3 indicates that the policy learned by DMG can achieve better performance than in-sample optimal policy under the oracle generalization condition.
3.4 Worst-case Generalization
This section turns the analyses in the worst-case generalization scenario, where the learned value functions may exhibit poor generalization in the mild generalization area (aass) > In other words, this section considers that TDMG is only defined in the in-sample area ((ass) > and the learned value functions may have any generalization error at other state-action pairs. In this case, we use the notation TDG to tell the difference.
We make continuity assumptions about the learned Q function and the transition dynamics. Assumption 2 (Lipschitz Q). The learned Q function is -ipLiihchitz Vs D, Vai, 02 A. [Q(s,))) Q(s, 02)l & Kollai - 02ll
Assumption 3 (Lipschitz P). The transition dynamics Ki-Lipschitz Vs, s' ~ S, Val 02 A. [P(s'(s. 01) - P(s'(s,az) < KPlla1 - a2ll
For Assumption 2, continuous learned Q function is particularly necessary for analyzing value func- tion generalization and can be relatively easily satisfied using neural networks or linear models [24]. Assumption is also a common assumption in theoretical studies of RL [13, 87, 61].
Now consider the iteration starting from arbitrary function Q": Qb TDM Q and Q = -in Vk E Z+ The possible value of @t is bounded by the following results.
Theorem 4 (Limited overestimation). Under Assumption the learned Q function of DMG by iterating JDMG satisfies the following inequality
(x, S @DMc (s, S (i, + teaKo A 71- Vs, a V
1-7 -1
(9)
Since in-sample training eliminates the extrapolation error [37, 92], Q can be considered relatively accurate estimate [37]. Therefore, Theorem 4 suggests that DMG exhibits limited value overesti- mation under the worst-case generalization scenario. Moreover, the bound becomes tighter ca decreases (milder action generalization) and À decreases (milder generalization propagation). This is consistent with our intuitions in Section 3.2.
Finally, we show in Theorem 5 that even under worst-case generalization, DMG guarantees to output safe policy with performance lower bound.
5

---

Theorem 5 (Performance lower bound). Let TDNG be the learned policy of DMG by iterating TDMG- π² be the optimal policy, and ep be the inherent performance gap of the in-sample optimal policy ED J(x') (. Under Assumptions 2 and 3, for sufficiently small ca. we have
J(DMG) 2 J(x²)~ P'ppRmxx
6a - 6D.
(10)
1-7
where C/ is a positive constant.
3.5 Practical Algorithm
This section puts DMG into implementation and presents a simple yet effective practical algorithm. The algorithm comprises the following networks: policy To, target policy Tे. Q network Qa. target Q network Qo, and V network Vp-
Policy learning. Practically, we expect DMG to exhibit a tendency towards mild generalization around good actions in the dataset. To this end, we first consider reshaping the empirical behavior policy B3 be skewed towards actions with high advantage values 8rt (als) x (alss) exp(A(ss a))-
Then we enforce the proximity between the trained policy and the reshaped behavior policy to constrain the generalization area. We define the generalization set Hg follows.
lc. { KL ( (-|s)[[=(-|s)) 5
(11)
Note that forward KL allows æ to select actions outside the support of 3r, enabling Hg to generalize beyond the actions in the dataset. With I6 defined, the next step is to compute the maximal Q within IIg. To accomplish this, we adopt Actor- Critic style training [70] for this part.
max E amm (8( 8.l. n II
(12)
By treating the constraint term as a penalty, we maximize the following objective.
max D.wa (-19) Q0(8, a) KT. (1))]
(13)
Through straightffrward derivations, Eq. (13) is equivalent to the following policy training objective.
(Ø) Fa.-Daarra((e)000ss - (B (exp(a a) (s)) log lo [a(s))]
(14)
where is inverse temperature and Qu a) (s) computes the advantage function A(s,a).
Value learning. Now we turn to the implementa- tion of the TDMG operator for training value func-
Algorithm DMG
tions. By introducing the aforementioned policy,
1: Initialize Tg. TTy, Qs. Qø and Vg.
we can substitute max in TDMG with Ea-r-
2: for each gradient step do
Regarding max in TDMG, any in-sample learn-
3: Update v by minimizing Eq. (15)
ing techniques can be employed to compute the in-sample maximum [37, 88, 85, 21]. In particular,
4: Update by minimizing Eq. (16)
Update by maximizing Eq. (14)
6: Update target networks: 0' + (1-
based on IQL [37], we perform expectile regression.
go, ø'
E) E
Lv(W)= = (s,a)~
[L5 (Qor a) Vo(s))] (15)
7: end for
where Lb( = Jr- - 1uu < 0) w² and E (0, For RS 1, Vo can capture the in-sample maximal Q [37]. Finally, we have the following value training loss.
(0) = (... (Cooss.a)
R(ssa) (R(ss))))
7ABa Qo 711 -1)))))()))))) (16)
Overall algorithm.
Integrating all components, present practical algorithm in Algorithm 1.

---

4 Discussions and Related Work
Summary of offline RL work from generalization perspective. As analyzed above, DMG is featured in both mild action generalization and mild generalization propagation. Within the actor- critic framework upon which most offline RL algorithms are built, these two aspects correspond to the policy and value training phases, respectively. Action generalization concerns whether the policy training intentionally selects actions beyond the dataset to maximize Q values, while generalization propagation involves whether value training propagates generalization through bootstrapping. Table I presents a clear comparison of offline RL works in this generalization view. The table shows one representative method of each category and we elaborate on others as follows.
Table 1: Comparison of offline RL work from the generalization perspective.
IQL
AWAC
TD3BC
TD3
DMG (Ours)
Action generalization
none
none
mild
full
mild
Generalization propagation
none
full
full
full
mild
Concerning policy learning, AWR [57], AWAC [55], CRR [80], 10% BC [8], IQL [37], and other works such as [78, 9, 66, 21, 88] extract policies through weighted or filtered behavior cloning, thereby lacking intentional action generalization to maximize Q values beyond the dataset. Typical poliey-regularized offline RL methods like TD3BC [17], BRAC [84], BEAR [38]. SPOT [83], and others such as [79, 61, 72] introduce regularization terms to Q maximization objectives to regularize the trained policy towards the behavior policy and allows mild action generalization. Online RL algorithms like TD3 [18] and SAC [27] have no constraints and maximize Q values in the entire action space, corresponding to full action generalization. Regarding value training, in- in-sample learning methods including OneStep RL [7]. IQL. [37], InAC [85], IAC [92], XQL [21], and SQL [88] completely avoid generalization propagation and accumulation via bootstrapping, whereas typical offline and online RL approaches allow full generalization propagation through bootstrapping. In the proposed approach DMG, generalization is mild in both aspects.
Recently, Ma et al. [47] have also drawn attention to generalization in offline RL and the issue of over- generalization. They mitigate over- veeerennralizztton from a representation perspective, differentiating between the representations of in- in-sample and OOD state-action pairs. Lyu et al. [44] argue that conventional value penalization like CQL [39] tends to harm the generalization of value functions and hinder performance improvement. They propose mild value penalization to mitigate the detrimental effects of value penalization on generalization.
Connection to heuristic blending approaches. Our approach also relates to the framework of blending heuristics into bootstrapping [10, 81, 71, 28, 82, 22]. In offline RL, HUBL [22] blends Monte-Carlo returns into bootstrapping and acts as a data relabeling step, which reduces the degree of bootstrapping and thereby increases its performance. In contrast, DMG blends the in-sample maximal values into the bootstrapping operator. DMG does not reduce the discount for RL learning but reduces the discount for generalization propagation.
For extended discussions on related work, please refer to Appendix A.
5
Experiments
In this section, we conduct several experiments to justify the validity of the proposed method DMG. Experimental details and extended results are provided in Appendices C and D, respectively.
5.1 Main Results on Offline RL Benchmarks
Tasks. We evaluate the proposed approach on Gym-MuJoCo locomotion domains and challenging AntMaze domains in D4RL [16]. The latter involves sparse-reward tasks and necessitates "stitching" fragments of suboptimal trajectories traveling undirectedly to find path to the goal of the maze.
Baselines. Our offline RI. baselines include both typical bootstrapping methods and in-sample learning approaches. For the former, we compare to BCQ [19]. BEAR [38], AWAC [55], TD3BC [17],
7

---

Table 2: Averaged normalized scores on Gym locomotion and Antmaze tasks over five random seeds. medium, m-r medium-replay, m-e medium- expert, e expert, == random; = umaze, u-d umaze-diverse, m-p medium- play, m-d medium-diverse, 1-p- plage-play, I-d = large-diverse.
Dataset-v2
BC
BCQ
BEAR
DT
AWAC OneStep
TD3BC
CQL
IQL
DMG (Ours)
halfcheetah-m
42.0
46.6
43.0
42.6
50.4
48.3
47.0
47.4
54.9:10.2
hopper-m
56.2
59.4
51.8
67.6
59.8
87.5
59.3
53.0
66.2
100.6:1.99
walker2d-m
71.0
71.8
-0.2
74.0
83.1
84.8
83.7
73.3
78.3
92.4
halfcheetah-m-4
36.4
42.2
36.3
36,6
44.8
42.7
44.6
45.5
44.2
51.410.3
hopper-m-r
21.8
60.9
52.2
82.7
69.8
98.5
60.9
88.7
94.7
1.1.9::.4
walker2d-m-r
24.9
57.0
7.0
66.6
78.1
61.7
81.8
81.8
73.8
89.7+5.0
halfcheetah- m-e
59.6
95.4
46.0
90.7
75.6
86.7
91.114.2
hopper-m-e
51.7
106.9
50.6
107.6
100.1
108.6
98.0
105.6
91.5
110.4+3.4
walker2d-m-e
101.2
107.7
22.1
108.1
110.0
111.3
110.1
107.9
109.6
114,410.7
halfcheetah-e
92.9
89.9
92.7
87.7
81.7
88.2
96.7
96.3
95.0
95,9++.3
hopper-e
110.9
109.0
54.6
94.2
109.5
106.9
107.8
96.5
109.4
111.5
walker2d-e
107.7
106.3
106.6
108.3
110.1
110.7
110.2
108.5
109.9
114.7=10.4
halfcheetahh-r
2.6
2.2
2.3
2.2
6.1
2.3
11.0
17.5
13.1
1..
hopper-r
4.1
5.4
8.5
7.9
7.9
20.41:10.
walker2d-r
1.2
4.9
12.8
2.2
0.2
6.9
1.6
5.4
4.8+2.2
locomotion total
784.2
968.0
581.7
972.6
975.3
1041.2
1013.2
1010.2
1033.1
1182.8
antmaze-u
66.8
78.9
73.0
54,2
80.0
54.0
73.0
82.6
89.6
1.
antmaze-u-d
56.8
55.0
61.0
41.2
$2.0
57.8
47.0
10.2
65.6
7.14
antmaze-m-p
0.0
0.0
0.0
59.0
76.4
80.2
antmaze-m-d
0.0
0.0
8.0
0.0
0.2
0.6
0.2
46.6
72.8
6..
antmaze-l-p
0.0
6.7
0.0
0.0
0.0
0.0
0.0
16.4
42.0
55.4+-6.2
antmaze-1-d
0.0
2.2
0.0
0.0
0.0
0.2
0.0
3.2
46.0
58.8
antmaze total
123.6
142.8
142.0
95.4
132.2
112.6
120,2
218.0
392.4
439.4
and CQL [39]. For the latter, compare to BC [58], OneStep RL [7], IQL [37], XQL [21], and SQL [88]. We also include the sequence-modeling method Decision Transformer (DT) [8].
Comparison with baselines. Aggregated results are displayed in Table 2. On the Gym locomotion tasks, DMG outperforms prior methods most tasks and achieves the highest total score. On the much more challenging AntMaze tasks, DMG outperforms all the baselines by a large margin, especially in the most difficult large mazes. For detailed learning curves, please refer Appendix D.3. According to [56], we also report the results of DMG over more random seeds in Appendix D.2.
Runtime, We test the runtime DMG and other baselines on a GeForce RTX 3090. As shown in Appendix D.I, the runtime of DMG is comparable that of the fastest offline RL algorithm TD3BC.
Performance Improvement over In-sample Learning Approaches
DMG can be combined with various in- sample learning approaches. Besides IQL [37], we also apply DMG cent state-of-the-art in-sample algorithms, X'QL [21] and SQL [88]. As shown in Ta- ble 3 (and Table 2), DMG consistently and substantially improves upon these in-sample methods, particularly on sub-optimal datasets where generalization plays crucial role in the pursuit of better policy. This provides compelling empirical evidence that the per- formance of in- in-sample methods is largely confined by eschewing generalization beyond the dataset, while DMG effectively exploits generalization, achieving significantly proved performance across tasks.
Table 3: DMG combined with various in in-sample ap- proaches, showing averaged scores over seeds.
re-
Dataset-v2
XQL (+DMG)
SQL(+DMG)
halfchectah-m hopper-m
47.7 + 55.3
48.3 + 54.5
71.1 + 90.1
75.5 97.7
walker2d-m
81.5 88.7
84,2 89.8
halfcheetah-m-rr
44.8 - 51.1
44.8 + 51.8
hopper-m-r
97.3 + 102.5
101.7 + 101.8
walker2d-m-r
75.9 - 90.0
77.2 95.2
halfcheetah-m-ee
89.8 92.5
94.0 93.5
hopper-m-e
107.1 + 111.1
111.8
110.4
walker2d-m-e
110.1 + 111.3
110.0 109.6
725.3 - 792.7
747.5
804,2

---

walker2d medium v2
hopper medium-v2
balfehherthh medium v2
baffcheetah crndem x2
100
540
70
140
Nirmatred ieemr
Sernyles Reum
Naunaee Rouss
Naauited
60
340
280
90
253
15
Q
valle
Ratun
Quile
0.00
0.5
0.50
0.75
1.00
000
0.25
0.30
0.35
1000
0.20
0.25
0.30
0.75
1.2
0.2
0.50
6.55
1.06
Figure 1: Performance and values of DMG with varying mixture coefficient À over 5 random seeds. The crosses x mean that the value functions diverge in several seeds. As increases, DMG enables stronger generalization propagation, resulting in higher and probably divergent learned Q values. Mild generalization propagation plays a crucial role in achieving strong performance.
walkerrd medium v2
hopper medium v2
balfcheetsh medium v2
baffcheetah cundem v/
100
300
40
100
NonddigdRoo Rouu
73
520
60
Skg
29)
10
200
480
10
Aetim
A.tum
1
valle
Raticn
05 0.1 230
03
0.55 (LCCA1)
00000000
Figure 2: Performance and Q values of DMG with varying penalty coefficient over 5 random seeds. As y decreases, DMG allows broader action generalization, leading to larger learned Q values. Mild action generalization is also critical for attaining superior performance.
5.3 Ablation Study for Performance and Value Estimation
Mixture coefficient X. The mixture coefficient controls the extent of generalization propagation. We fix // 0.1 and vary À [0, 1], presenting the learned Q values and performance several tasks in Figure 1. As À increases, DMG enables increased generalization propagation through bootstrapping, and the learned Q values become larger and probably diverge. A moderate (mild generalization propagation) is crucial for achieving strong performance across datasets. Under the same degree of action generalization, mild generalization propagation effectively suppresses value overestimation, facilitating more stable policy learning.
Penalty coefficient v. The penalty coefficient y regulates the degree of action generalization. We fix À 0.25 and vary v. The results are shown in Figure 2. As decreases, DMG allows broader action generalization beyond the dataset, which results in higher learned values. Regarding performance, a moderate (mild action generalization) also crucial for achieving superior performance.
5.4 Online Fine-tuning after Offline RL
Benefiting from its flexibility in both gen- eralization aspects, DMG enjoys a seam- less transition from offline to online learn- ing. This is accomplished through a grad- ual enhancement of both action generaliza- tion and generalization propagation. Since IQL [37] has demonstrated superior online fine-tuning performance compared to previ- ous methods [55, 39] its paper, we follow the experimental setup of IQL and compare IQL. We also train online RL algorithm TD3 [18] from scratch for comparison. We use the challenging AntMaze domains [16], given DMG's already high offline perfor- mance in Gym locomotion domains. Results are presented Table 4. While online training from scratch fails in the challenging sparse reward AntMaze tasks, DMG initialized with offline pretraining succeeds in learning near-optimal policies, outperforming IQL. by significant margin. Please refer to Appendix C.2 for experimental details, and to Appendix D.4 4 for learning curves.
Table 4: Online fine-tuning results on AntMaze tasks, showing normalized scores of offline training and IM steps online fine-tuning, averaged over seeds.
Dataset-v2
TD3
IQL
DMG (Ours)
antmaze-u
0.0
89,6 + 96.2
92.4 + 98.4
antmaze-u-d
0.0
65.6 62.2
75.4 + 89.2
antmaze-m-p
0.0
76.4 89.8
89.8
80.2 + 96.8
antmaze-m-d
0.0
728 + 90.2
77.2 + 96.2
antmaze-I-p
0.0
42.0 + 78.6
55,4 + 86.8
antmaze-1-d
0.0
46.0
73.4
58.8 + 89.0
antmaze total
0.0
392.4 + 490.4
439.4
556.4

---

Conclusion and Limitations
This work scrutinizes offline RL. through the lens of generalization and proposes DMG, comprising mild action generalization and mild generalization propagation, to exploit generalization in offline RL appropriately. We theoretically analyze DMG in oracle and worst-case generalization scenarios, and empirically demonstrate its SOTA performance in offline training and online fine-tuning experiments.
While our work contributes valuable insights, it also has limitations. The DMG principle is shown be effective across most scenarios. However, when the function approximator employed is highly compatible with specific task setting, the learned value functions may generalize well in the entire action space. In such case, DMG may underperform full generalization methods due to conservatism.
Acknowledgment
We thank the anonymous reviewers for feedback on an early version of this paper. This work was supported by the National Key R&D Program of China under Grant 2018AAA0102801,, National Natural Science Foundation China under Grant 61827804.
References
[1] Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. In International conference on machine learning, pages 22-31. PMLR, 2017.
[2] Gaon An, Seungyong Moon, Jang-Hyun Kim, and Hyun Oh Song. Uncertainty- based offline reinforcement learning with diversified q-ensemble. Advances in neural information processing systems, 34:7436-7447, 2021.
[3] Chenjia Bai, Lingxiao Wang, Zhuoran Yang. Zhi-Hong Deng, Animesh Garg, Peng Liu, and Zhaoran Wang. Pessimistic bootstrapping for uncertainty-die driven offline reinforcement learning. In International Conference on Learning Representations, 2022. URL https://opeenreeiee
net/forum?id=Y40m wit.iidddcccs111nnnn_. 1Z3HnqL.
[4] Jacob Beck, Risto Vuorio, Evan Zheran Liu, Zheng Xiong, Luisa Zintgraf, Chelsea Finn, and Shimon Whiteson, A survey of meta-reinforcement learning. arXiv preprint arXiv:2301.08025 2023.
[5] Avinandan Bose, Simon Shaolei Du, and Maryam Fazel. Offline multi-task transfer rl with representational penalization. arXiv preprint arXiv:2402.12570 2024.
[6] Stephen Boyd and Lieven Vandenberghe. Convex optimization. Cambridge university press, 2004.
[7] David Brandfonbrener, Will Whitney, Rajesh Ranganath, and Joan Bruna. Offline without off-policy evaluation. Advances in Neural Information Processing Systems, 34:4933 4946334446,.. 2021.
[8] Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling, Advances in neural information processing systems, 14:15044-15097, 2021.
[9] Xinyue Chen, Zijian Zhou, Zheng Wang, Che Wang, Yanqiu Wu, and Keith Ross. Bail: Best- action imitation learning for batch deep reinforcement learning, Advances in Neural Information Processing Systems, 33:18333--13633 2020.
[10] Ching-An Cheng, Andrey Kolobov, and Adith Swaminathan. Heuristic-guided reinforcement learning. Advances in Neural Information Processing Systems, 34:1 15513555013563 2021.
[11] Ching-An Cheng, Tengyang Xie, Nan Jiang, and Alekh Agarwal. Adversarially trained actor critic for offline reinforcement learning. In International Conference on Machine Learning, pages 3852-3878. PMLR, 2022.
10

---

[12] Jonas Degrave, Federico Felici, Jonas Buchli, Michael Neunert, Brendan Tracey, Francesco Carpanese, Timo Ewalds, Roland Hafner, Abbas Abdolmaleki, Diego de Las Casas, al. Magnetic control of tokamak plasmas through deep reinforcement learning. Nature, 602(7897): 414 2022.
[13] Francois Dufour and Tomas Prieto-Rumeau. Finite linear programming approximations of constrained discounted markov decision processes. SIAM Journal on Control and Optimization, 51(2):1298-1333 1324, 2013.
[14] Francois Dufour and Tomas Prieto-Rumeat Approximation of average cost markov deci- sion processes using empirical distributions and concentration inequalities. Stochastics An International Journal of Probability and Stochastic Processes, 87(2)2773-37,, 2015.
[15] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adap- tation of deep networks. International conference on machine learning, pages 1126-1135. PMLR, 2017.
[16] Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020.
[17] Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning. Advances in neural information processing systems, 34:20132 221401, 2021.
[18] Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In International conference machine learning, pages 1587-1596. PMLR, 2018.
[19] Scott Fujimoto, David Meger, and Doina Precup. Off Off-policy deep reinforcement learning without exploration. In International conference on machine learning, pages 2052-2062. PMLR, 2019.
[20] Javier Garcia and Fernando Fernández. A comprehensive survey on safe reinforcement learning. Journal of Machine Learning Research, 16(1):1437-1480, 2015.
[21] Divyansh Garg, Joey Hejna, Matthieu Geist, and Stefano Ermon. Extreme q-learning: Maxent RL without entropy. In The Eleventh International Conference on Learning Representations,
2023. URL httpp:://ott http://pppeeeeeeeeeeeeeeee wwtppeereeiee.nee/fffereeeeeeeee wew.:forummdddSSS....e...
[22] Sinong Geng, Aldo Pacchiano, Andrey Kolobov, and Ching-An Cheng. Improving offline RL by blending heuristics. In The Twelfth International Conference on Learning Representations,
2024. URL httta:::///peerevieeee wwwtffeeemmiieeeeeeeeeeeeeeeeeee HCC.LLLLLP......LLLL............
[23] Seyed Kamyar Seyed Ghasemipour, Dale Schuurmans, and Shixiang Shane Gu. Emaq: Expected-max q-learning operator for simple yet effective offline and online rl. International Conference on Machine Learning, pages 3682-3691 PMLR, 2021.
[24] Henry Gouk, Eibe Frank, Bernhard Pfahringer, and Michael Cree. Regularisation of neural networks by enforcing lipschitz continuity. Machine Learning.
2021.
[25] Sven Gronauer and Klaus Diepold. Multi-agent deep reinforcement learning: a survey. Artificial Intelligence Review, 55(2):895 943, 2022.
[26] Shangding Gu, Long Yang, Yali Du, Guang Chen, Florian Walter, Jun Wang, and Alois Knoll. A review of safe reinforcement learning: Methods, theory and applications. arXiv preprint arXiv:2205. 10330, 2022.
[27] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, pages 1861-1870. PMLR, 2018
(28) Ehsan Imani, Eric Graves, and Martha White. An off-policy policy gradient theorem using emphatic weightings. Advances in neural information processing systems, 31, 2018.
11

---

[29] Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model-based policy optimization. Advances in neural information processing systems, 32. 2019.
[30] Natasha Jaques, Asma Ghandeharioun, Judy Hanwen Shen, Craig Ferguson, Agata Lapedriza, Noah Jones, Shixiang Gu, and Rosalind Picard. Way off-policy batch deep reinforcement learning of implicit human preferences in dialog. arXiv preprint arXiv: 00ti:/997.00000 004400 2019.
[31] Alistair EW Johnson, Tom Pollard, Lu Shen, Li-wei H Lehman, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark. Mimic-iii, a freely accessible critical care database. Scientific data, 3(1):1-9, 2016.
[32] Lukasz Kaiser, Mohammad Babaeizadeh, Piotr Milos, Blazej Osinski, Roy H Campbell, Konrad Czechowski, Dumitru Erhan, Chelsea Finn. Piotr Kozakowski, Sergey Levine, al. Model- based reinforcement learning for atari. arXiv preprint 1rXiv:1903,00374 2019.
[33] Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. Morel: Model-based offline reinforcement learning. Advances in neural information processing systems, 33:21810-21822 2020.
[34] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
[35] Jens Kober, J Andrew Bagnell, and Jan Peters. Reinforcement learning in robotics: A survey. The International Journal of Robotics Research, 32(1 1):18)1-122244 2013.
[36] Ilya Kostrikov, Rob Fergus, Jonathan Tompson, and Ofir Nachum. Offline reinforcement learning with fisher divergence critic regularization. In International Conference on Machine Learning, pages
PMLR, 2021.
[37] Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. In International Conference on Learning Representations, 2022. URL https: //openrvviww
[38] Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off- policy q-learning via bootstrapping error reduction. Advances in Neural Information Processing Systems, 32, 2019.
[39] Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances Neural Information Processing Systems, 33: 1179- 1191, 2020.
[40] Saseha Lange, Thomas Gabel, and Martin Riedmiller. Batch reinforcement learning. Reinforce- ment learning: State-of-the-art, pages 45-73, 2012.
[41] Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. nature, 521(7553):433 2015.
[42] Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.016633 2020.
[43] Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, OpenAl Pieter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed pepprratvve-oopetttiiii environments. Advances in neural information processing systems, 30, 2017.
[44] Jiafei Lyu, Xiaoteng Ma, Xiu Li, and Zongging Lu. Mildly conservative q leaannngg for offline reinforcement learning. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022. URL https:// upenreview.net.e weet:/oorumidd-Veeessss VdVYY6656670..
[45] Xiaoteng Ma, Yiqin Yang, Hao Hu, Qihan Liu, Jun Yang, Chongjie Zhang, Qianchuan Zhao, and Bin Liang. Offline reinforcement learning with value-based episodic memory. arXiv preprint arXiv:21 2021.
12

---

[46] Yecheng Ma, Dinesh Jayaraman, and Osbert Bastani. Conservative offline distributional reinforcement learning. Advances Neural Information Processing Systems, 34:19235-19247, 2021.
[47] Yi Ma. Hongyao Tang, Dong Li, and Zhaopeng Meng. Reining generalization in offline reinforcement learning via representation distinction. In Thirty-seventh Conference on Neu- ral Information Processing Systems, 2023. URL https://openrevveew http:////ppeeeeeeeeeeee/etrrrrre wwtp:/uume//ootmmeeemem mVywRIDNII.
[48] Will Maddern, Geoffrey Pascoe, Chris Linegar, and Paul Newman, 1 year, 1000 km: The oxford robotcar dataset. The International Journal of Robotics Research, 36(1):3-15, 2017.
[49] Yixiu Mao, Hongchang Zhang, Chen Chen, Yi Xu, and Xiangyang Ji. Supported trust region optimization for offline reinforcement learning. In International Conference on Machine Learning, pages 23829-23851, PMLR, 2023.
[50] Yixiu Mao, Qi Wang, Chen Chen, Yun Qu, and Xiangyang Ji. Offline reinforcement learning with ood state correction and ood action suppression. arXiv preprint 0eo::2410.994440 2024,
[51] Yixiu Mao, Hongchang Zhang, Chen Chen, Yi Xu, and Xiangyang Ji. Supported value regularization for offline reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
[52] Tatsuya Matsushima, Hiroki Furuta, Yutaka Matsuo, Ofir Nachum, and Shixiang Gu. Deployment-efficient reinforcement learning via model-based offline optimization. In- lernational Conference on Learning Representations, 2021. URL https://opeerevviee http/foruummdd3hhhhppp
[53] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 518(7540):529-533 2015.
[54] Thomas M Moerland, Joost Broekens, Aske Plaat, Catholijn M Jonker, et al. Model-based reinforcement learning: A survey. Foundations and Trends@ in Machine Learning, 16(1):1-118 2023.
[55] Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. Awac: Accelerating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006.00 200,22006.0339,, 2020.
[56] Andrew Patterson, Samuel Neumann, Martha White, and Adam White. Empirical design in reinforcement learning. arXiv preprint 230v:22004003155, 2023.
[57] Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weighhed regression: Simple and scalable off-policy reinforcement learning. arXiv preprint arXiv: 2019.
[58] Dean A Pomerleau. Alvinn: An autonomous land vehicle in a neural network. Advances in neural information processing systems, 1, 1988.
[59] Yun Qu, Boyuan Wang, Jianzhun Shao, Yuhang Jiang, Chen Chen, Zhenbin Ye, Line Liu, Junfeng Yang, Lin Lai, Hongyang Qin, et al. Hokoff: Real game dataset from honor of kings and its offline reinforcement learning benchmarks. In shirtt- Conference on Neural Information Processing Systems Track on Datasets and Benchmarks, 2023.
[60] Yun Qu, Boyuan Wang, Yuhang Jiang, Jianzhun Shao, Yixiu Mao, Cheems Wang, Chang Liu, and Xiangyang Ji. Choices are more important than efforts: Llm enables efficient multi-agent exploration. arXiv preprint arXiv:2410.0251/ 2024.
[61] Yuhang Ran, Yi-Chen Li, Fuxiang Zhang, Zongzhang Zhang, and Yang Yu. Policy regularization with dataset constraint for offline reinforcement learning. In International Conference on Machine Learning, pages 28701-28717 PMLR, 2023.
[62] Tabish Rashid, Mikayel Samvelyan, Christian Schroeder De Witt, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. Monotonic value function factorisation for deep multi-agent reinforcement learning. Journal of Machine Learning Research, 21-178):1-51, 2020.
13

---

[63] Julian Schritwieser,, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Si- mon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, al. Mastering atari, go, chess and shogi by planning with a learned model. Nature, 588(7839):604-66)) -09, 2020.
[64] Jianzhun Shao, Yun Qu, Chen Chen, Hongchang Zhang, and Xiangyang Ji. Counterfactual conservative q learning for offiin multi-agent reinforcement learning. In Thirty-seventh Con- ference on Neural Information Processing Systems, 2023. URL https: http://pppprriie..o//oo net/ forum?id (aamndddd2zmmm@@855
[65] Jianzhun Shao, Hongchang Zhang, Yun Qu, Chang Liu, Shuncheng He, Yuhang Jiang, and Xiangyang Ji. Complementary attention for multi-agent reinforcement learning. In International Conference on Machine Learning, pages 30776-30793, PMLR, 2023.
[66] Noah Siegel, Jost Tobias Springenberg, Felix Berkenkamp, Abbas Abdolmaleki, Michael Neunert, Thomas Lampe, Roland Hafner, Nicolas Heess, and Martin Riedmiller. Keep doing what worked: Behavior modelling priors for offline reinforcement learning. In International Conference on Learning Representations, 2020. URL https: httt:::://ppeervieee wepperrriie..nee/oourm net/forum? id=rke7gelttwHi.
[67] David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, al. Mastering the game of go without human knowledge. nature, 550(7676):354-3555 359, 2017.
[68] Yihao Sun, Jiaji Zhang, Chengxing Jia, Haoxin Lin, Junyin Ye, and Yang Yu. Model-bellman inconsistency for based offline reinforcement learning. In International Conference on Machine Learning, pages 33177-33199 194.. PMLR, 2023.
[69] Richard S Sutton. Dyna, an integrated architecture for learning, planning, and reacting, ACM Sigart Bulletin, 264):160-163, 1991
[70] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
[71] Richard S Sutton, A Rupam Mahmood, and Martha White. An emphatic approach to the problem of off-policy temporal-difference learning. Journal of Machine Learning Research, 17 (73):1-29, 2016.
[72] Denis Tarasov, Vladislav Kurenkov, Alexander Nikulin, and Sergey Kolesnikov. Revisiting the minimalist approach to offline reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
[73] Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q-learnig. In Proceedings of the AAAI conference on artificial intelligence, volume 30, 2016.
[74] Oriol Vinyals, Igor Babuschkin, Wojciech M Czarnecki, Michaël Mathieu, Andrew Dudzik, Jun- young Chung. David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster in starcraft using multi-agent reinforcement learning. Nature, 575(7782):350- 354, 2019.
[75] Cheems Wang, Yiqin Lv, Yixiu Mao, Yun Qu, Yi Xu, and Xiangyang Robust fast adaptation from adversarially explicit task distribution generation. arXiv preprint arXiv:2407.19525 2024.
[76] Qi Wang and Herke Van Hoof. Model-based meta reinforcement learning using graph structured surrogate models and amortized policy search. In International Conference on Machine Learning, pages 23055-23077, PMLR, 2022.
[77] Qi Wang, Yiqin Lv, Zheng Xie, Jincai Huang, et al. A simple yet effective strategy to robustify the meta learning paradigm. Advances in Neural Information Processing Systems, 36, 2024.
[78] Qing Wang, Jiechao Xiong, Lei Han, Han Liu, Tong Zhang, et al. Exponentially weighted imitation learning for batched historical data. Advances in Neural Information Processing Systems, 31, 2018.
[79] Zhendong Wang, Jonathan J Hunt, and Mingyuan Zhou. Diffusion policies as an expressive policy class for offline reinforcement learning. In The Fleventh International Conference on Learning Representations, 2023. URL https: /ttpp://////eeeirrrr/ee/ net/ http:/frummddddw/vl/mma
14

---

[80] Ziyu Wang, Alexander Novikov, Konrad Zolna, Josh S Merel, Jost Tobias Springenberg, Scott E Reed, Bobak Shahriari, Noah Siegel, Caglar Gulcehre, Nicolas Heess, al. Critic regularized regression. Advances in Neural Information Processing Systems, 33:7768-7778 2020.
[81] Albert Wilcox, Ashwin Balakrishna, Jules Dedieu, Wyane Benslimane, Daniel Brown, and Ken Goldberg. Monte carlo augmented actor-critic for sparse reward deep reinforcement learning from suboptimal demonstrations. Advances in Neural Information Processing Systems, 35: 2022.
[82] Robert Wright, Steven Loscalzo, Philip Dexter, and Lei Yu. Exploiting multi-step sample trajectories for approximate value iteration. In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2013, Prague, Czech Republic, September 23-27, 2013, Proceedings, Part I 13, pages 113-128. Springer, 2013.
[83] Jialong Wu, Haixu Wu, Zihan Qiu, Jianmin Wang, and Mingsheng Long. Supported policy optimization for offline reinforcement learning. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022. URL https: /ptp///eeeeeeeeeeeeee/ee/eeeeeee
[84] Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. arXiv preprint arXiv:1911.1131,, 2019.
[85] Chenjun Xiao, Han Wang, Yangchen Pan, Adam White, and Martha White. The in-sample soft- max for offline reinforcement learning. In The Eleventh International Conference on Learning Representations, 2023. URL https: /ppnneeeeeeeeeeeeeeeee wwwpeneeiee.eeeeoooooeeeeeeeeeee id=u- RMM
[86] Tengyang Xie, Ching-An Cheng, Nan Jiang, Paul Mineiro, and Alekh Agarwal. Bellman- consistent pessimism for offline reinforcement learning. Advances in neural information processing systems, 1146683-6944 2021.
[87] Huaqing Xiong, Tengyu Xu, Lin Zhao, Yingbin Liang, and Wei Zhang. Deterministic policy gradient: Convergence analysis. In Uncertainty in Artificial Intelligence, pages 2159-2169. PMLR, 2022.
[88] Haoran Xu, Li Jiang, Jianxiong Li, Zhuoran Yang, Zhaoran Wang, Victor Wai Kin Chan, and Xianyuan Zhan. Offline RL with no OOD actions: In-sample learning via implicit value regularization. In The Eleventh International Conference on Learning Representations, 2023. net/ formm wwweiidddueeeepsssee
URL
[89] Rui Yang, Chenjia Bai, Xiaoteng Ma, Zhaoran Wang, Chongjic Zhang, and Lei Han. RORL: Robust offline reinforcement learning via conservative smoothing. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022. URL https: httpt: h/tp::::///pppreeeeeeee net/ wttp///wwwwe/ouuuiniiieeeee Bttpiddd__dJJ_________ KE.
[90] Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. Advances in Neural Information Processing Systems, 33: 14122-144444 2020.
[91] Tianhe Yu, Aviral Kumar, Rafael Rafailov, Aravind Rajeswaran, Sergey Levine, and Chelsea Finn. Combo: Conservative offline model-based policy optimization. Advances in neural information processing systems, 34:28954-2888 -29677, 2021.
[92] Hongchang Zhang, Yixiu Mao, Boyuan Wang, Shuncheng He, Yi Xu, and Xiangyang Ji. In-sample actor critic for offline reinforcement learning. In The Eleventh International Con- ference on Learning Representations, 2023. URL dfDvoWU853R.
[93] Wenxuan Zhou, Sujay Bajracharya, and David Held. Plas: Latent action space for offline reinforcement learning. In Conference Robot Learning, pages 1719-1735. PMLR, 2021.
15

---

A Extended Related Work
Model-free offline RL. In offline RL, fixed dataset is provided and no further interactions are allowed [40, 42]. As a result, conventional off-policy RL algorithms suffer from the extrapolation error due to OOD actions and exhibit poor performance [19]. To address this challenge, various offline RL algorithms have been developed, primarily categorized into model-free and model-based approaches. In model-free solutions, value regularization methods introduce conservatism in value estimation through direct penalization [39, 36, 46, 86, 11, 64, 5 51], or via value ensembles [2. 3, 89]. Policy constraint approaches enforce proximity between the trained policy and the behavior policy, either explicitly via divergence penalties [84, 38, 30, 17, 83), implicitly by weighted behavior cloning [9, 57, 55, 80, 49], directly through specific parameterization of the policy [19, 23, 93]. Some recent efforts focus on learning the optimal policy within the dataset's 's support (known as in- support in-sample optimal policy) in a theoretically sound manner [49, 51, 83]. These approaches are less influenced by the the dataset's average quality. Another popular branch of algorithms opts for in-sample learning, which formulates the Bellman target without querying the values of any unseen actions 17, 45, 37, 85, 92, 88, 21J. Among these, OneStep RI. [7] evaluates the behavior policy via SARSA [70] and performs only one step of constrained policy improvement without off off-po evaluation. IQL [37] modifies the SARSA update, using expectile regression to approximate an upper expectile of the value distribution and enables multi-step dynamic programming. Following IQL. several recent works such as InAC [85], IAC [92], X'L [21], and SQL [88] have developed different in- in-sa learning frameworks, further enhancing the performance of in-sample learning approaches. However, this work shows that the performance of in-sample approaches is confined by eschewing generalization beyond the offline dataset. In contrast, the proposed approach DMG utilizes doubly mild generalization to appropriately exploit generalization and achieves significantly stronger performance across tasks.
Model-based offline RL. Model-based offline RL methods involve training an environmental dynamics model, from which synthetic data is generated to facilitate policy optimization [69, 29, 32]. In the context of offline RL, algorithms such as MOPO [90] and MORel. [33] propose to estimate the uncertainty within the trained model and subsequently impose penalties or constraints on state- action pairs characterized by high uncertainty levels, thus achieving conservatism in the learning process. Some model- bbsed approaches incorporate conservatism in a similar way to those model- free ones. For example, COMBO [91] leverages value penalization, while BREMEN [52] employs behavior regularization. More recently, MOBILE [68] introduces uncertainty quantification via the inconsistency of Bellman estimations within a learned dynamics ensemble. SCAS [50] proposes generic model-based regularizer that unifies OOD state correction and OOD action suppression in offline RI. However, typical model-based methods often involve heavy computational overhead [29), and their effectiveness hinges on the accuracy of the trained dynamics model [54].
Recently, Bose et al. [5] explores multi-task offline RL from the perspective of representation learning and introduced a notion of neighborhood occupancy density. The neighborhood occupancy density at given stata-action pair in the dataset for a source task is defined as the fraction of points in the dataset within a certain distance from that stata-action pair the representation space. Bose et al. [5] use this concept to bound the representational transfer error in the downstream target task. In contrast, DMG is a wildly compatible idea in offline RL and provides insights into many offline RL methods. DMG balances the need for generalization with the risk of over- generalization in offline RL. Generalization to stata-action pairs in the neighborhood of the dataset corresponds to mild action generalization in the DMG framework.
B Proofs
In this section, we provide the proofs of all the theories in the paper.
B.1 Proof of Theorem
This section presents the formal theorem for the Theorem I in the main paper, along with its proof.
We first make several common continuity assumptions for Theorem
16

---

Assumption 4 (Lipschitz Q). The learned value function Qa is Ki-Lipschitz and is upper bounded by Qmax Vs D. Va1,02 ~ A, (Q.(s,a)) Qu(s, a2)l S Kollai a2l|.
Assumption 5 (Lipschitz Q gradient). The learned value function Qe is smooth. i.e, has Kg- Lipschitz continuous gradient. Vs D, Va,, a2 ~ A. [VooQo(s (1) V VoQn(s, a2)ll - Kglla
a2/l
Assumption 6 (Bounded Q and Q gradient). Vs, [Qa(s, a)1 S max and VaQa(s, a)ll S Jmax Assumption 7 (Lipschitz P). The transition dynamics is Kp-Lipschitz Vs, ~ S, 4al. 02 ~ A. [P('' [s, a1) P( [[o)ss,02] a2)l S Kp Kpllai u22ll.
Assumption 8 (Lipschitz R). The reward function R is Kg-Lipschitz vs ~ S, Van, A. [R(s[a)) - R(s, a2)l _ KRllai a2ll.
A continuous learned Q function is particularly necessary for the analysis of value function general- ization. Since we often use neural networks or linear models to parameterize the value function Qe. Assumptions 4 and 5 can be relatively easily satisfied [24]. Assumptions 6, 7, and 8 are also common the theoretical studies of RL [13, 87, 61] and optimization [6].
Before we start the proof of Theorem 1, we prove the following lemma.
Lemma 2. Vs ~ D, Vaj, 02 ~ A. [T a1) TuQo(s, a2)l $ Krlla1 azll. where Ky is a positive bounded constant.
Proof. Vs D. Ya1, 02 ~ A.
(T.Qo(8,a)) az)
R(s,a1) R(s,a2) 02) +, 7T P(Brpl()
max almu(-1s')
((5',a)
Q(s',
a'~ul-ls")
<|R(s,a1) R(s, a2))
B, B(-aa(
max a'maa(l)0)
Q(5(,a')
-Es-P((lo.0))
max Q(s',a)
a a'-u(-|o')
= -I((s,a1) R(s, a2)1 7
Σ (P()))s,0)--
max a'>u((l>')
Q(s',a)
S|R(s,a1) R(s, az)l + Σ (((('((s, P(s)(s,02))
max Q(s' ,a) a'ma(-lo")
<KR]lui a2ll +
Σ Kpllai- - Q
=(Kn + 1kpps/@max)lay azll
where the last inequality holds by Assumptions 6, 7, and 8. Therefore, for any D, 01 A. it holds that
(n.0 a1) TuQo(s, 02)l S Kr a2ll,
(17)
where Ky KR + 7KpS// max is positive bounded constant.
We restate the scenario analyzed in Theorem 1: Qe is updated to Qar by one gradient step on a single state-action pair (s, a) DD which affects the Q-value of an arbitrary state-action pair (s, a) D. The parameter update is
0'= a(T. (o.
(18)
where a is the learning rate.
Now we start the proof of Theorem in the main paper.
Theorem 6 (Theorem 1), Under Assumptions 4 to 8, the following equation holds when the learning rate a sufficiently small and à is sufficiently close to a:
Qø (s, a) Qo a) + C1 (T.Qo(s, a) @ols, + Czllã all) -100'-013
(19)
where C1 [0, 1] and C2 [-Kę KR xKplSk max. Kº KR 7
17

---

Proof. We formalize (s. â) by Taylor expansion at the parameter A:
(s a) Qa(s, @) + VoQo(s.ā) (6' - 0) + + (16'-
- 0
(20)
By plugging Eq. (18) into Eq. (20), we have
Qo a)) = ao a) VoQo(s, a) (TaQo(s,
(10' 0l13) (21)
According to Assumption 4 and Lemma 2, it holds that
Qo(s, ā) Qo(s, a)l & Ko - all (=, (o, a)] - Kr all
(22)
(23)
where Ky Ka + SSpppp is positive bounded constant.
Therefore.
(TuQo(s, - -o(s,a)) - (acs))((s,))-(((_,i (TuQo(s,ā)
Qo(s,a)) (TuQo(s,ã) ( Qo(s, a))
KT all Kol@ã-al
As result,
TaQo a) Qo(s,a) - T.Qolls Qo(s,)) + (Kq + Ky) all (s Qo(s,a) 2 T.Qo(s, Qa(,aa) (Kq + Ky) all
Thus let
T TuQu(8 a) Qa(s, a) = - (,(s,,) where C2 [-Ko Ky. Kę + K is bounded constant.
C2llā all
(24)
Now we shift our focus to a V V& Qs a) VaQo(s, a). Let = the smoothness of Qo in Assumption holds that
According
||a|| = (v.Qo - voQo(s,a)] & Kgllä - -al--
(25)
Therefore,
voQo(s,ā) Telo(s.a)Voooll. VoQo(s,a)
(V)(o(s.a)+ voQo(s,a) VeQoo+
VoQo(s..)ll+ voQossa))
VoQo(s, -10o) VoQa - -allk
Therefore, for sufficiently close à and a such that ||a al s [VoQo(s, a))l K,, it holds that a (VoQo(s, a) VoQa a) 2 0.
On the other hand, because V is bounded by Jmax according to Assumption 6, it holds that a V VoQo(s a) VoQu(s,a) S aggax
By choosing a small learning rate such that S 1/
aV a) () <
such cases (sufficiently close ā and a, and sufficiently small a), let
C1 = VoQo(s, a) VoQo(s.a)
(26)
We have C₁ E [0, 1].
By plugging Equations (24) and (26) into Equation (21), the following equation holds.
- where C, 10, C2 I-Ke K,, Kg + K7], and K KR + plpSS
Qø (s, ā) = Qo(s, â) + C1 (TuQo(s, a) - Qo(s, ā) + Czllā - all) + O -10'' 012)
(27)
This concludes the proof.
18

---

B.2 Proofs under Oracle Generalization
We first restate the several definitions the main paper.
Definition 4 (Mildly generalized policy, Definition 1). Policy B is termed a mildty generalized policy if it satisfies
supp(B(-1s) C supp(3((1)), and max min ||a - a2ll S ca. a1-Btt)bs) a1
(28)
where 3 is the empirical behavior policy in the offline dataset.
Definition (Definition 2). The Doubly Mildly Generalization (DMG) operator is defined as
TDMGQ((.a) a) = R(s,a) a) + +7BxxP((aaa) max d'-3(115))
+ max Q(5
d'~B(188)
(29)
where 3 is the empirical behavior policy in the dataset and 3 is mildly generalized policy.
Definition 6 (Definition 3). The In-sample Q Learning operator [37] is defined
Tm)Q(s, R(s,a) + +7BpxP((saa)
max a'-D(t11s)
Q(s a')
(30)
where 3 is the empirical behavior policy in the dataset.
In this subsection, we assume that the learned value function can make oracle generalization in the mild generalization area (aass) > O, which is formally defined as follows.
Assumption 9 (Oracle generalization, Assumption 1). The generalization learned Q functions in the mild generalization area ((a)s) > 0 reflects the true value updates according to TDMG- other words, TDMG is well defined in the mild generalization area 3(als) > 0.
This assumption can be considered reasonable according to the results presented in Theorem 6 above. In such cases, we analyze the dynamic programming properties of operators Tia and TDMG-
Before we start the proofs of Lemma and Theorem 2 in the main paper, we prove lemma.
Lemma 3. For any function fi, f2. any variant X, the following inequality holds:
max fi(x) - max f2(x) 2EX
S
max (f())) - -f((x)]- xEX
(31)
Proof. Define 31 argmax.ex f1( and 22 argmax f2(x). According to the definition, the following inequality holds:
(1(22) f2(22) (11) f2(22) s f((1)) f2(21)
(32)
Therefore,
max fi (x) max fe(x)
DEX
= =Ii((z)) f2(22)
< max (12) fe(x2) (a(x2)), fa(x2)),15((z)) f2(21)))
< max [x) f2(2:) ZEd
This concludes the proof of Lemma
Lemma (Lemma 1), Tin a r-conttaction operator the in-sample area 3(ass) under the
Loe norm.
19

---

Proof. Let fi and f2 be two arbitrary functions.
For all (s,a) a) s.t. B(als) > 0, we have
a) an)
R(s,a) + +7 a.0)
max
- R(s, a) -P(1)))((111))
max f2(6',aa)
a' d'mB(1)6)
=y Es P(11.0)
max e'~3(-le"))
f1(s',a')= max fe(s',a") a'-B(-16))
SyBBmP((1sa)
max fi(d',a')-
max a'~B(i10)
f2(5',a) aa)
a a'~B(-1s")
S7BBxP((16..)
max a'~xB(-1e)
Vi(s',a')-Sstte - S2(( d')l
Sy max (5,4):8(a)))200
(f((s, a) f2(s.a) a)l
where the second inequality holds by Lemma 3.
Therefore, in the iamse area ((a)s) 0, Tiu is y-contraction operator under the L norm. This concludes the proof for Tin.
Thus, by repeatedly applying Tim any initial function can converge to the unique fixed point Qi We denote its induced policy by Tin'
(s) R( PESNP()
max (in(st,aa) a'~B(1ls))
B(als) 0,
(33)
ii (s) = argmax Qin (s,
(34)
a>-B(11s)
Here, Qi is known as the in-sample optimal value function [38, 37], which is the value function of the in-sample optimal policy ni.- We refer readers to [83, 37, 49, 51] for more discussions on the in-sample or in-support optimality.
Now we start the proof of Theorem
main paper.
Theorem (Contraction, Theorem 2). Under Assumption 9, TDMG is "-contraction operator in the mild generalization area Malss) > 0 under the Lx norm. Therefore, by repeatedly applying TDMG, any initial function can converge to the unique fixed point QTMG-
Proof. By the oracle generalization assumption (Assumption 9), TDMG is well defined in the mild generalization area ((a)s) 0.
Let f1 and f2 be two arbitrary functions. For all (s, s.t. B(ass) > 0, we have
TDMG (s,a) a) TDMG (DMGf((s,a)
=R(s,a) -Bg-P(11aa)
A max Ji(5', (d') -) a'mi(-11a")
(i(d',a') a'm3(-18))
a)) -7By
A max fe(s', a') + (1 (1-1) almB3(-1s))
max fe(s',
a'mā(cle))
fi(s',a')
max fe(s', a') a'~B(167
max a'~B(-1ar)
+ +7(1 A)BBs~P(Ha
max fi(5',a')
max fa(s'.a))
a'mB((1sr))
a'~B(-)3))
20

---

Therefore, for all (s,a) s.t. B(als) > 0,
TDMGF1(S,a) TDMGf2(s,a))
1 y P(1sa)
max ('~B(-110")
fi(s',d') - max fe(s',a') d'~B(116)
+ 7(1 P()
max fi(d,a')- al-B(-10)
max 52(5',ad)
a>~B(133)
f168,
f2(5',
max d'mB(-10")
max almB(-1a))
+ +7(1 P)Ban)((lsa)
max ('~D(-16")
fi(s',a')=
max fe(s', a
STABy-P(ssa)
max
(f) f2(('.
a'-B((la")
+ +7 11)B8))BxPPll
max a'mB(-le") Ifi(s', a') fe(('.
STABg~P(1s.a)
max (3.)):5(a())200 + +7(1 1)Bss)(((sm)
(s f2(s,a)]
max
(fi(s, fe(s.a)l
(e,0):5(ae=>00
max (o.)):
(x - f2(8,a))
where the third inequality holds by Lemma 3.
Therefore, in the mild generalization area 3(als) > 0, TDMG is a 7-contraction operator under the Loo norm. This concludes the proof.
As result, by repeatedly applying TDMG. any initial Q function can converge to the unique fixed point QDMG. We denote the induced policy of QDMG by DDMG-
@bmo(s, R(s, a) (BswP((s.n)
(b a(~B(11))))
max
B(als) 0,
(35)
DMG (s) argmax (DMG(s,a)
(36)
a>3((-1s)
Before we start the proof of Theorem 3, prove two lemmas. Lemma S. Under Assumption for any function J. the following inequality holds:
(s, 2 (s, V(s,a)
s.t. B(als) > 0.
(37)
Proof. The oracle generalization assumption (Assumption 9) implies that Tin also well defined in the mild generalization area ((ass) > O. Because supp(}(-ls)) S supp({(-ls)), Tia requires less information than TDMG- Therefore, TomG being well defined in the mild generalization area implies Tia also being well defined in that area.
According to the definitions, for all (8, s.l. Mals) > O.
TDMGf(s, a) R(s, (Bs-P((sm))
A max f(s',a + (1- a'mgl-la")
2) max a(s',a') a'mā(llr)
(38)
Tiaf a) = (((sa) +BBrwP((aa))
max f(s',a') a'~B(116")
(39)
21

---

Therefore, for all (s, s.t. B(als) > 0, we have
TDMG (s, - Tin (s,
=yB
max a'-0(111")
f(s',a') - max
f(d',a')
a'-((118)
20
where the last inequality holds because has wider support than B.
Lemma 6. Under Assumption 9, for any function f1, f2 such that fi(s,0) 2 f2(s,a),
a(s s.t. B(als) > 0, the following inequality holds:
(IMGFi(s, a) 2 a)),,f(s,a), V(s,a) s.t. B(als) >
(40)
Proof. By Assumption 9, TDMG is well defined in the mild generalization area 3(als) > O. According to the definition, for all (s, a) s.t. B(als) >
TDMG a) R(s, a) (Be-p((am)
A max f(s', + (1- max f(6', a'w3(-16")
(41)
a'mā(-1)))
fi and f2 satisfy
(18,a)) 2 fa(s,a), (() s.l. B(als) 0.
(42)
Therefore, for all (s, s.t. B(als) >
(s,a)
TDMG f2(8,
=7Banp((la))
A max f1(8',a')-2 -X max f2(5',a') a'mu((-1s")
e'mB(-lo'))
+ +PBs))((1sa)
(1- max fi(s',a')- - (1- max ad) a'~((-1)"))
a'~
20
Now we start the proof of Theorem in the main paper. Theorem 8 (Performance, Theorem 3). Under Assumption 9, the value functions of TiMe and nin satisfy:
VTmG (s) 2 (in Vs D.
(43)
Proof. We first prove the following inequality:
f(s, (TDMa)* f(s,a) (f a), Ak Zt, Yf, ((s,o) s.t. B(als) 0.
(44)
When 1, according to Lemma 5, holds that
(TDMG)' f(s, 2 (Tm)) a), Vf, a(s,a) s.t. B(als) > 0.
Suppose when 1: = i, the following inequality holds:
(TDMa)) [s, a) 2 (Tim)' (Ts)),((s,) a), VS, V(s,a) a) s.l. ((ass) > 0.
Then (TDMG)'S and (Tim)' the two functions fi, f2 that satisfy the condition in Lemma 6. Therefore, by Lemma 6, it holds that
TDMG(TDMG)+ fs,,a 2 TDMG f(s, Vf, V(s s.t. B(als) o.
(45)
22

---

Now considering (Tim)' as the function f in Lemma 5. By Lemma 5, it holds that
TDMG f 2 (Ta(( (s,,), Vf. V(s.a) s.t. B(a)s) 0.
(46)
Combining Equations (45) and (46), we have
(TDMG) )++1 f(a a) 2 (Tm))++((s, vf, V(s,a) s.t. B(als) > 0.
Therefore, for all & E zt, the following inequality holds:
(Trma)* f(s.a) 2 (Tm.) f(s. (f..) vf, V(s,o) s.t. B(als) 0.
(47)
Lemma 4 states that Ii is 7-contraction operator in the in-sample area B(als) > Thus have = lim (Tim)* f(,)) V(s, a) s.t. B(a)s) > O.
(48)
k-+00
Under Assumption Theorem states that TrM conntaacion operator in the mild generaliza- tion area ((ass) > Thus we have
@ima(s, a) = lim (TDMG)* f(s, a), v(s,a) a) s.t. B(als) > k=+00
(49)
As 3 has wider support than B, supp(((-1s)) S supp(3(-Is)) the following inequality holds by combining Equations (47) to (49):
(s. 2 (s, as s.t. B(als) O.
(50)
Therefore, for any ~ D.
VTbMe(s) (s) = ViM (8) = QiMa (bM @DMG (s, i (s))
2Qi (s) = Via (s) = Viin
where the first inequality holds because "DMG (s) argmax Vima a) and (s)
3(-ls) (thus iia (s) ((-ls)), and the second inequality holds by Equation (50).
This concludes the proof.
Theorem 8 indicates that the policy induced by the DMG operator can behave better than the in-sample optimal policy under the oracle generalization condition.
B.3 Proofs under Worst-case Generalization
In this section, we focus on the analyses the worst-case generalization scenario, where the learned value functions may exhibit poor generalization in the mild generalization area (a]ss >> 0. In other words, this section considers that TDMG is only defined in the in-sample area B(a|s) > 0 and the learned value functions may have any generalization error at other state-action pairs. In this case, we use the notation JDMG for differentiation.
In this case, we make the following continuity assumptions about the learned Q function and the transition dynamics P.
Assumption 10 (Lipschitz Q). The learned Q functiin is Lipsccitt Vs ~ D. Vaj, 02 ~ A, [Q(s. 01) Q(s, a2)) Kql(aj azll
Assumption 11 (Lipschitz P). The transition dynamics P is Kp-Lipschitz Vs, 8' ~ S, Val, 02 A.
[P(s'ls ( 41) - P(s'|s, (())) < KPllai -
#2ll
For Assumption 10, continuous learned Q function is particularly necessary for the analysis of value function generalization and can be relatively easily satisfied [24], since we often use neural networks or linear models to parameterize the value function. For Assumption 11, continuous transition dynamics is also a standard assumption in the theoretical studies RL [13, 14, 87, 61]. Several previous works assume the transition be Lipschitz continuous with respect (w.r.t) both state and action [13, 14]. In our paper, we need the Lipschitz continuity to hold only w.r.t. action.
Before we start the proof of Theorem 4, prove two lemmas.
23

---

Lemma 7. Under Assumption 10, for any function f and 8 ~ D, the following inequality holds:
f(s,a) a) max f(s,a) coKq. a~3((11))
(51)
a>)())s)
Proof. For any D, we define a", a", à' as follows:
ã" argmax f(s,a)
(52)
a~B((1s) à' argmax f(s,a) a) a~B(11>
(53)
à' argmin ||a" all a>B(11s)
(54)
According the definition of mildly generalized policy / (Definition 4), it holds that l|a a' S ca. Further by Assumption 10, it holds that
[f(x,ä)) ((,,ā') s Kollä" à'll S ka N
Therefore,
-(s,a))-f(s,") - f(s,a') S ((s,a") f(s,a') ca Vs D.
Lemma 8. For any function fi, f2 such that fi(s,a) 2 fz(s.a), v(s,a) s.t. B(als) > 0. the
following inequality holds:
(s, Tinf2(s, a), ((s.a) s.d. (aass) > O.
(55)
Proof. According to the definitions, for all (s. s.l. B(als) > 0.
Tiaf(s,)) () R(s, a) BBop(((a
max f(s',
(56)
fi and f2 satisfy
(( 2 f2(s,a), s.l. B(als) O.
Therefore, for all (s, a) s.l. 3(als) >
Tnnf(ss,a) TT,f((s,
=y =7Egsw((1en)
f1(5',a') max fa(5',a') d'mB(11s)) armBii 09)
max
20
Now we start the proof of Theorem the main paper.
We consider the iteration starting from arbitrary function Q": Q TDMGQANl and Q Tin Vk e zt. The possible value Qf is upper bounded by the following results. Theorem 9 (Limited over-estimation, Theorem 4). Under Assumption 10, the learned Q function of DMG by iterating TOMG satisfies the following inequality
(s, a) s @bmo (s, a) s (i,((, a) + deaKal (1- Vs D, V+EZ. (57)
1-7
24

---

Proof. Under worst-case generalization, TDM is only defined in the area B(als) > 0, i.e., the dataset, and may have any generalization eπror at other (s,
For any function f and any 8,0~ D,
TDMG a) [nn](s,
=R(s,a) +7Bxsp((lbn))
A max a'mB(11>>)
+10++(111 (1-
max a'mā(1ls))
M(s','))
R(s,a) -:By-P((aaa)
max a'mB(((0)
f(s',a')
=)BsmP((sa))
max f(s',a') - max f(s',a)) a'mi(()1s))
SyXcaKo
where the last inequality holds by Lemma 7.
On the other hand, because 3 has a wider support than B, we also have
TDMaf(s, [ (s,
Therefore, for any function f. the following incquality holds:
Tif a) s TDMaf(ss & Tinf(s, a) + reeaKo, Vs,a~ D.
(58)
Let ʃ in Equation (58) be Qº. We have
en, a) = (s) (s, + dea MaK (1-7), (1-)) Vs
(59)
1 1-7
This is the same as Equation (57) with 1. Therefore, Equation (57) holds when 1.
Suppose when = i, Equation (57) holds:
1-2 (s. a) (sma(s,a) S (s, + AeaKaT (1 (1- Vs, as
(60)
Then let f in Equation (58) be Q We have
Tia Timer (s, a) I @ (s, a) = TDMG (nMa s TIa a) + Pe. Vs, a ~ D.
(61)
On the one hand, according to Lemma and Equation (60), for any 8, a D, we have
Ti Tin@u
STi (s,()) + AcaKon (1
(1- +3)
1
(oinlt,.a))
deaKet (1 - -3))
=[(s,a) + 7
max
a' a'~3(-16"))
1 1-7
de Koy
-)(s,a) + +PEN)P(1e.)
max (in(s',a')
+
( (1-)))
a'm3(-1or)
1 1-7
de, Koy (1 - 7-)
+ (s, a)
1
Xe Ko
= (s.a) + 1-9
(1 (1-?²)
(62)
25

---

Combining Equations (61) and (62), for any 8, ~ D, we have
DMG
Aco Aco
5Q (s,
(1 -1) + ydeako
1-9
Qi (s, XecKor 7 (
1 1-7 +1
- () + deaKoy
(1 - (it)
On the other hand, according to Lemma and Equation (60), for any s, we have
Ti TMM (8, a) 2 Tin VinQia a) = Qitt (8,a)
(63)
Combining Equations (61) and (63), for any 8, D, we have
@MG (3,a) 2 Q (s, a).
(64)
Hence, Equation (57) still holds when & =:
Qit < Q S 1 (s, + Xe.KoT (1 (1- Vs,a D.
(65)
1-7
Therefore, Equation (57) holds for all Z+, which concludes the proof.
Since ia-sample training eliminates extrapolation error completely [37, 92], Q be considered relatively accurate estimate, Therefore, Theorem 9 indicates that DMG has limited over-estimation under the worst generalization case. Moreover, the bound gets tighter ca gets smaller (more mild action generalization) and À gets smaller (more mild generalization propagation). This consistent with our intuitions in Section 3.2.
Finally, Theorem 5 in the main paper shows that even under worst-case generalization, DMG guaranteed to output safe policy with a performance lower bound.
We give lemma before we start the proof of Theorem 5,
Lemma 9. Let and T2 be two deterministic policies. Under Assumption 11, the following inequality holds:
TV (d=1 S CKr max [s) n2(2)]]
(66)
where C/ is a positive constant and d= (s) is the state occupancy induced by x.
d" (8) = (1 7) Σ
En [ [st = s]]
(67)
t=
Proof. Please refer to Lemma A.5 in [61] and Lemma [87].
Theorem 10 (Performance lower bound, Theorem 5). Let tDNG be the learned policy of DMG by iterating JDMG, x" be the optimal policy, and ep be the inherent performance gap of the iample
optimal policy eD == (TT) ( Under Assumptions 10 and for sufficiently small ea we have
((DDM)) 2 J(T*)- CKpRuax
1-7 -
(68)
where C a positive constant.
Proof. Following previous works [38, 83,37,499 we define the in-sample optimal policy as "..
क (s)
argmax Qi a)
(69)
a>-ā(-1s)
26

---

We also use ép to denote the performance gap between the in sampee optimal policy and the globally optimal policy, which is fixed once the dataset provided.
eD = -(x") J(
(70)
We use QDMG denote the learned Q function of DMG with sufficient iteration steps M 20. And #DMG is the output policy of QDMG:
(s) = argmax QDMG(S,
(71)
a~ā(-1s)
It holds that
[(x*) J(ADMG)
=I](x²) - J(xaa) + J( J
5|((=") - -J(ri)) (J(mm))) - J JfNMMG)
=DD + [J(mf)) J(fDMG)
(72)
In the following, bound the term [J( (fDDMG)
J ((( (
E Bon [r()] 1- ta [(ss]
Σ (dRDMO (m) (c)) r(s)
1 Σ (iss)-drigg - {
|n(s)]
Rmax TV
(
(s)lldtin (s)
1-y
Rmax CKp max (s) Tin
(73)
1-7
where the last inequality holds by Lemma 9.
According to Theorem 9, QDMG satisfies the following inequality:
(s, a) s QDMa(s,a) s as + 1-7 Vs, ~ D.
(74)
means that for any (s, a) D, with sufficiently small ca. QDMG(S, a) sufficiently approximates Qi, a). By Definition 4, f is mildly generalized policy. That is, for any D, 3 satisfies
supp(B(-ls)) S supp(3(-1s)), and max min ||a - 1aal-all S Ea.
(8-18) (-
As (s) E (-1ss, it implies that we can find din E (-1ss) (in dataset) such that [|*DMG(s) amlll S Ca-
Now suppose ain is not the maximum point of Qi, (s, :) at a certain s. We use Ti. (s) to denote the maximum point of (s. Let eQr. be the gap between Qi, (s, am) and Qi, (s, (s)):
eqi (s) = Qi, min (s)) Qi, (s, am) >
(75)
By Assumption 10 (Lipschitz Q), we have
QDMG (s, #DMG QDMa(s, and S (Ql -iall S KQEa.
(76)
27

---

Therefore,
QDMG T (8)) QDMG(S, #DMG
QDMG Tin (s)) - QDMG ain) - Koca
( is) - Qi (s;
KQca
1-7
Xe Ko
=c (s)
1 1-7 KQca
where the first inequality holds by Equation (76), the second incquality holds by Equation (74), and the last equality holds by Equation (75).
Hence, for sufficiently small ca such that EQT, (s) Nea - Kaea i...
1-7
(1 - 7) (8)
6a
KΦ( - 27))
(77)
holds that QDM (s) E B(-ls). This contradicts the definition of #s) in Equation (71):
(s)) (s, QDMG fDMG > O. As (i) E 3(1ss), also satisfies
#DM argmax QDMG(s, a>-((-1s)
Therefore, ain is the maximum point of Qi, (8, In other words, the maximum point of Qi, (s, (denoted by Tin (s)) is the closest neighbor of #DMG (s) in the dataset (ô(-ls) 0):
ii = argmin ||a (DMG((s)
a~1((18))
As (s) E B(-ls), the following inequality holds by Definition 4: | Jia [s) s ca.
Therefore, we have
Rmax CKpta.
(J(m()) J(ADMG) 1-
(78)
By combining Equations (72) and (78), we have
CKpR Ryax
J(TDMG) 2 J(x*)
1-7 Ea - ED.
(79)
This concludes the proof.
C Experimental Details
C.1 Experimental Details Offline Experiments
Our evaluation criteria follow those used in most previous works. For the Gym locomotion tasks, we average returns over evaluation trajectories and 5 random seeds, while for the AntMaze tasks, average over 100 evaluation trajectories and 5 random seeds. Following the suggestions in the benchmark [16], we subtract 1 from the rewards for the AntMaze datasets. And following previous works [17, 37, 83, 88], normalize the states in Gym locomotion datasets. We choose TD3 [18] as our base algorithm and optimize a deterministic policy. Thus we replace the log likelihood in Eq. (14) with mean squared error in practice, which is equivalent to optimizing Gaussian policy with fixed variance [17]. The reported results are the normalized scores, which are offered by the D4RL benchmark [16] to measure how the learned policy compared with random and expert policy:
Icarned policy return random policy return
D4RL score 100 x
expert policy return random policy return
28

---

Table 5: Hyperparameters of DMG.
Hyperparameter
Value
Optimizer
Adam [34]
Critic learning rate
10
Actor learning rate
3 x 10 with cosine schedule
Batch size
256
Discount factor
0.99
DMG
Number of iterations
106
Target update rate
0.005
Number of Critics
2
Penalty coefficient
(0.1.10) for Gym-MuJoCo
{0.5) for Antmaze
Mixture coefficient X
0.25
Expectile
0.7 Gym-MuloCo
0.9 for Antmaze
IQL Specific
Inverse temperature a
3,0 for Gym-MuJoCo
10.0 for Antmaze
Actor
1mput255--266-outpu
Architecture
Critic
input-256-256-1
As we implement our main algorithm based on IQL [37]. we use the hyperparameters suggested in their paper for fair comparisons, i.e., T 0.7 and a 3 for Gym locomotion tasks and 0.9 and a 10 for AntMaze tasks. For the results of X'QL+DMG and SQL+DMG, we also adopt the suggested hyperparameters in their papers [21, 88] for fair comparisons. In detail, we choose / in X'LL [21] as 5.0 in medium, medium-replay, and medium-expee eepiertteepprttteee datasets, and a in SQL [88] as 2.0 medium, medium-replay datasets, and 5.0 for medium-expert datasets.
DMG has two main hyperparameters: mixture coefficient À and penalty coefficient v. We use X 0.25 for all tasks. We use 0.5 for Antmaze tasks and {0.1, 10} for Gym locomotion tasks (0.1 for medium, medium-replay, random datasets; 10 for expert and medium-expert datasets). All hyperparameters of DMG are included in Table 5.
C.2 Experimental Details in Offline-to-online Experiments
For online fine-tuning experiments, we first run offline RL for X 10th gradient steps. Then we continue training while collecting data actively in the environment and adding the data the replay buffer. We perform online fine-tuning for x 106 steps with 1 update-to-data (UTD) ratio, and collect data with exploration noise 0.1 suggested by TD3 [18]. During offline pre-training, fix the mixture coefficient À = 0.25 and the penalty coefficient v = 0.5, while the online phase, corresponds standard online RL.
we exponentially adjust and v, DMG with In the challenging AntMaze domains characterized by high-dimensional state and action spaces, as well as sparse rewards, the extrapolation error remains significant even during the online phase [83]. Therefore, we decay X from 0.25 to 0.5 and y from 0.5 to 0.005 (1% of its initial value), employing decay rate of 0.99 every 1000 gradient steps. Additionally. following previous works [83, 72], we set y = 0.995 when fine-tuning on antmaze-large datasets, for both DMG and IQL to ensure fair comparison. All other training details remain consistent between the offline RI phase and the online fine-tuning phase.
and
D
Additional Experimental Results
D.1
Computational Cost
We test the runtime of offline RL algorithms halfchettal-meiiu---elpa on GeForce RTX 3090. The results of DMG and other baselines are shown Figure 3. It takes 1.7h for DMG finish the task, which is comparable to the fastest offline RL algorithm TD3BC [17].
29

---

Runline of Algocithrss
1.7h
DT
MOPO
COL
AWAC
DMG
IQL
TD3BC
Figure 3: Runtime of algorithms on halfchectah-mciium--llaaaa on GeForce RTX 3090.
D.2 Offline Training Results of DMG on More Random Seeds
The experimental results in the main paper show the mean and standard deviation (SD) over five random seeds. According to [56]. we conduct experiments to test DMG on additional random seeds, reporting 95% confidence interval (CI) over 10 random seeds. Table shows the comparison between the new results (10seeds/55%C) and the previously reported results (5seeds/SD in Table 2) on the D4RL offline training tasks. The results show that our method achieves about the same performance as under the previous evaluation criterion.
Table 6: Comparison of DMG under different evaluation criteria on D4RL offline training tasks.
Dataset-v2
DMG (Sseeds/SD)
DMG (10seeds/55%6C
halfcheetah-m
54.910.2
54.9:1..3
hopper-m
100.6:11.9
100.5=1.0
walker2d-m
92.4112.7
92.0+11.2
halfchectah-m-r
51,4++0.3
51.4+-0.4
hopper-m-r
101.9+1.44
102.1+-0.6
walker2d-m-r
89.7+:5.0
90.3+
helfcheetah-mm-e
91.1114.2
92.912.11
hopper-m-e
110.4.13.4
109.012..6
walker2d-m-e
114.4.10.7
113.9
halfcheetah-c
95.9+0.3
95.9++0.2
hopper-e
111.5+22.2
111.8
walker2d-e
114.7=+0.4
114,5+0.3
halfcheetah-r
28.8
28.7-11.2
hopper-r
20.4
21.6:16..6
walker2d-r
4.8+2.22
7.7+3.00
locomotion total
1182.8
1187.2
antmaze-u
92.4+11.8
91.
antmaze-u-d
75.44-8.1
73.0++5.0
antmaze-m-p
80.2:15.1
80.5+12.1
antmaze-m-d
77.216..1
76.7433.6
antmaze-l-p
55.4116.2
56.7-13.6
antmaze-l-d
58.8+4..5
57.2+
antmaze total
439.4
435.9
D.3 Learning Curves of DMG during Offline Training
Learning curves during offline training Gym-MuJoCo locomotion tasks and Antmaze tasks are presented in Figure and Figure 5, respectively. The curves are averaged over 5 random seeds, with the shaded area representing the standard deviation across seeds.
30

---

D.4 Learning Curves of DMG during Online Fine-tuning
Learning curves during online fine-tume ane-ttning on Antmaze tasks are presented Figure The curves are averaged over 5 random seeds, with the shaded area representing the standard deviation across seeds.
E Broader Impact
Offline reinforcement learning (RL) presents a promising avenue for enhancing and broadening the practical applicability of RL across various domains including robotics, recommendation systems, healthcare, and education, characterized by costly or hazardous data collection processes. However, imperative to recognize the potential adverse socictal ramifications associated with any offline RL algorithm. One such concern pertains to the possibility that the offline data utilized for training may harbor inherent biases, which could subsequently permeate into the acquired policy. Furthermore, it is essential to contemplate the potential implications of offline RL on employment, given its contribution to automating tasks conventionally executed by human experts, such as factory automation tonomous driving. Addressing these challenges is essential for fostering the responsible development and deployment of offline RL algorithms, with the aim of maximizing their positive impact while mitigating negative societal consequences.
From academic perspective. this research scrutinizes offline RL through the lens of generalization, balancing the need for generalization with the risk over wver-enerralization The proposed approach DMG potentially offers researchers new perspective offline RL. Besides, DMG also holds the promise to be extended to safe RL [1, 26, 20], multi-agent RL [43, 62, 65, 60, 25], and meta RL [15, 76, 77, 75, 4].
appropriately exploiting generalization in
100
espetiv2
espet-12
10d replatt-
me-lums-22
ondom
Epbosd Rotum
Epsocc Kotum
Eppooce Momum
Epsocc
50
5o
Eptoode
0.25 0.30 0.7 1.00
8000
0.25
0.30 0.75
0.0 0.30 0.2
0.25 0.30 0.5 1.00
0.5 0.30 075 8.0
Gimdtent Stepe (K]0F)
Cincterr Stegs (x10%)
Crodiet Stepe 109)
Grncion Stege [x]0f)
Gracteen Stage (x 100)
bosper export
hopger medium- expert-22
bogpas arduuss e2ply-922
bogger mediunn-22
50
hogper- cander-addddedd 22
100
100
100
Eptoode Roun
Epaoce Kotum
50
50
Ejsocce
Epsoce
50
1100 11.25 100
0.00
0.25 0.75
805 0:30
8oo 0.25 100
8.00 0.25
075
Gmddent (x10%)
Goddeet (x10%)
Grodiet
10%)
Gindiort (x10%)
Ginclent (x 1x1)
welle:2--nsiiumm 2aponiv22
wuller26mmsiummn3
100
100
Epionde novan
E00
Sploce Roum
Eguosee jouum
Eptodee
Eguonde
50
50
50
0.25 0.30 0.7
1.00
0.00
0.3
0.30
0.73 1.00
0.03
0.25
0.30
0.25 1000
0.00
0.3
0.30
0.75
1.00
0.30
0.30 Gradien Stegs (x105)
0.23
075
Grodeut Steps [K10%)
Godeer Stegs (x10%)
Grodiest Steps (x109)
Grddian Stepe [K10f)
Figure 4: Learning curves of DMG Gym locomotion tasks during offline training. The curves averaged over 5 random seeds, with the shaded area representing the standard deviation across seeds.
31

---

umaze-12
divers--22
100
medium mediun-plaa--?
Episods Belum
Eprsods Return
Epirode Retumn
50
800
Gradient Steps (x109)
Gradient Steps (x109)
Gradient Steps (x109)
untrwwe-hine-plly--2
100 untrawe-ingedivveeee?
100
100
Epirodd Rotem
Eppiodde R.uum
Epirod: Ranuns
50
86n
0.25
00,50
0.75
1.00
0.00
0.25
0.50
0,75
1.00
0.00
0.25
0.50
0,75
1,00
Steps (x109)
Gradien Steps (x10%)
Steps (x10%)
Figure Learning curves of DMG on Antmaze tasks during offline training. The curves are averaged over 5 random seeds, with the shaded area representing the standard deviation across seeds.
anlmaze-umozo-22
100
100
medinse--ediun-pllly play--2
Epirede Retum
Epiiode Retum
Epirods Retumn
50
50
50
8.00
1.00
9.00
1.00
0.00
1.00
Gradient Steps (x10%)
Gradient Stops (x10%)
Gradicnt Steps (x10%)
antowze-lareepplay.r
ontrozee antowze-lage-ivvesoee>
100
100
100
Epioodd Return
Eppisode Idtm
Episode Rarm
95.00 0.25
0.50
0.75 1.00
9.000
0.25
0.30
0.75
1.00
9.000
0.25
0.50
0.73
1.00
Gradient Steps (x109)
Gradient Steps (x10*)
Stops (x10*)
Figure Learning curves of DMG on Antmaze tasks during online fine-tuning. The curves are averaged over 5 random seeds, with the shaded area representing the standard deviation across seeds.
32

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Please refer Section 6.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
33

---

Justification:
Please refer to Appendix B.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Please refer Appendix C.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
34

---

Answer: [Yes]
Justification: Please refer to the code in the supplemental material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: /Codeeuomiisseeeeeeee
/nips. wwtpccccpppcccaecccccccccccccclt S ssisssolicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: Please refer to Appendix
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: The results in the paper accompanied by standard deviations across multiple seeds.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
35

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Please refer to Appendix D.1.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https https://neee /ttpp/////////p////eeeeeeeeessse cc cc/public/Bthiicciuiiee
Answer: [Yes]
Justification: The research conducted in the paper conforms, in every respect, with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Please refer to Appendix
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
36

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The creators or original owners of assets used in the paper are properly credited and the license and terms of use explicitly mentioned and properly respected.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
37

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The code is well documented and anonymized.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
38


Reviews:
[{"review_id": "f3bBsxWMD6", "reviewer": "Reviewer_kJjy", "summary": "This paper studies the problem of extrapolation error and value overestimation in Reinforcement Learning (RL). The authors exploit generalization in offline RL by using mild action generalization and mild generalization propagation.", "questions": "The idea of generalization to stata-action pairs in the neighborhood of those seen in the dataset has been studied in https://arxiv.org/pdf/2402.12570 and I urge the authors to cite and state the differences to the approach in this paper.", "limitations": "This paper has no significant limitations.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The problem studied is on interest : offline RL typically suffers from study of generalization beyond datapoints seen in the dataset.\n2. Strong theoretical results complimented with rigorous experiments. I'm impressed with the number of baselines the authors have tried out.", "weaknesses": "Assumption 11 looks very restrictive. The scenarios to which the approach can be applied gets significantly reduced if the transition needs to be lipschitz in the action space (for ever state, state next pair). For example for robotics tasks, where transitions are determinisitc, the transition matrix will take values 0-1, mnaking the Lipschitz constant very large in this scenario.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the time and effort you are dedicated to providing feedback on our paper and are grateful for the meaningful comments.\n\n**Q1: Assumption 11 looks very restrictive.**\n\nWe apologize for the lack of a detailed explanation of this assumption in the paper. To our knowledge, Assumption 11 (i.e., Assumption 3) is common in the theoretical studies of RL [1,2,3,4,5]. Several previous works assume the transition to be Lipschitz continuous with respect to (w.r.t) both state and action [1,2]. In our paper, we need the Lipschitz continuity to hold only w.r.t. action. However, we also acknowledge, as you mentioned, that the assumption reduces the theoretical applicability. Despite this, DMG performs quite well empirically on those robotics tasks with deterministic dynamics. In real-world scenarios, even for deterministic dynamics, there is some noise, resulting in a narrow distribution for state transitions. In this case, the assumption still applies, although the magnitude of the Lipschitz constant may vary depending on the level of noise.\n\n**Q2: The idea of generalization to stata-action pairs in the neighborhood of those seen in the dataset has been studied in [6] and I urge the authors to cite and state the differences to the approach in this paper.**\n\nThanks for the reference. The research work [6] is crucial in the analysis of multi-task offline RL, and we will cite it and include the following discussions.\n\nRecently, theoretical advancement [6] has explored multi-task offline RL from the perspective of representation learning and introduced a notion of neighborhood occupancy density. The neighborhood occupancy density at some $(s, a)$ in the dataset for a source task is defined as the fraction of points in the dataset within a certain distance from $(s, a)$ in the representation space. The authors use this concept to bound the representational transfer error in the downstream target task. In contrast, DMG is a wildly compatible idea in offline RL and provides insights into many offline RL methods. DMG balances the need for generalization with the risk of over-generalization in offline RL. Specifically, DMG achieves doubly mild generalization, comprising mild action generalization and mild generalization propagation. Generalization to stata-action pairs in the neighborhood of the dataset is a specific realization of mild action generalization. Additionally, in our work, the neighborhood is defined based on the action space distance.\n\n**Reference**\n\n[1] Dufour, Francois, and Tomas Prieto-Rumeau. \"Finite linear programming approximations of constrained discounted Markov decision processes.\" SIAM Journal on Control and Optimization 51.2 (2013): 1298-1324.\n\n[2] Dufour, Francois, and Tomas Prieto-Rumeau. \"Approximation of average cost Markov decision processes using empirical distributions and concentration inequalities.\" Stochastics An International Journal of Probability and Stochastic Processes 87.2 (2015): 273-307.\n\n[3] Shah, Devavrat, and Qiaomin Xie. \"Q-learning with nearest neighbors.\" Advances in Neural Information Processing Systems 31 (2018).\n\n[4] Xiong, Huaqing, et al. \"Deterministic policy gradient: Convergence analysis.\" Uncertainty in Artificial Intelligence. PMLR, 2022.\n\n[5] Ran, Yuhang, et al. \"Policy regularization with dataset constraint for offline reinforcement learning.\" International Conference on Machine Learning. PMLR, 2023.\n\n[6] Bose, Avinandan, Simon Shaolei Du, and Maryam Fazel. \"Offline multi-task transfer rl with representational penalization.\" arXiv preprint arXiv:2402.12570 (2024).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ojGpN6SrEo", "reviewer": "Reviewer_RKt1", "summary": "The offline RL community has recently shown a surge of interest in in-sample offline reinforcement learning algorithms. However, these algorithms can sometimes be too restrictive, unable to leverage the generalization capability of deep neural networks. As a remedy, the authors proposed an algorithm called Doubly Mild Generalization that uses the weighted mean of the in-sample maximum and mildly generalizable maximum Q-value. Experiments conducted on various offline RL benchmarks manifest the effectiveness of the proposed algorithm.", "questions": "Please refer to the **Weaknesses** section.", "limitations": "The authors adequately addressed the limitations and potential negative societal impact of their work.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The authors provide rigorous mathematical proofs for most of their arguments. They also provide a practical version of their algorithm, which performs very well on multiple offline RL benchmarks.", "weaknesses": "1. The constants $C_1$ and $C_2$ of Theorem 1 are state-dependent.\n\n2. Lines 120-121 are difficult to understand. How does (4) relate to the generalizability of Q functions? \n\n3. The meaning of Assumption 1 is unclear at first glance. An explanation of what might prevent $\\mathcal{T}_{\\textrm{DMG}}$ from being well-defined would be helpful for the readers. Also, I think the term \"well-defined\" is misleading. There's no problem with \"defining\" the DMG operator for state-action pairs outside the dataset; it's just that they would be incorrect and not be very meaningful.\n\n### Minor comments\n\n1. $\\alpha$ is missing from Eq. (21)\n\n2. $k=\\infty$ → $k\\to\\infty$ in Equations (48) and (49)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the time and effort you are dedicated to providing feedback on our paper and are grateful for the meaningful comments.\n\n**Q1: The constants $C_1$ and $C_2$ of Theorem 1 are state-dependent.**\n\nIndeed, they are state-dependent. It is worth noting that Theorem 1 provides insight and motivates our method.\n\n**Q2: How does (4) relate to the generalizability of Q functions?**\n\nWe apologize for the confusion caused by the lack of sufficient explanation. Eq. (4) is the update of the parametric Q function ($Q_{\\theta} \\rightarrow Q_{\\theta’}$) at state-action pairs $(s,\\tilde a) \\notin \\mathcal D$, which is exclusively caused by generalization. If $\\tilde a$ is within a close neighborhood of $a$, then $C_2\\|\\tilde a-a\\|$ is small. Moreover, as $C_1 \\in [0,1]$, Eq. (4) approximates an update towards the true objective $\\mathcal T_u Q_\\theta(s,\\tilde a)$, as if $Q_\\theta(s,\\tilde a)$ is updated by a true gradient step at $(s,\\tilde a) \\notin \\mathcal D$.\n\nWe will add these explanations to the latter revision.\n\n**Q3: The meaning of Assumption 1 is unclear at first glance. Rephrase the term \"well-defined\".**\n\nThanks for the kind suggestion. Indeed, the term \"well-defined\" may not be very suitable in this context. To avoid confusion, we will delete the sentence \"In other words, $\\mathcal{T}_{\\mathrm{DMG}}$ is well defined in the mild generalization area $\\tilde \\beta(a|s)>0$.\" in Assumption 1. In addition, we will include more explanations before line 158 as follows.\n\nBecause the mild generalization area $\\tilde \\beta(a|s)>0$ may contain some points outside the offline dataset, $\\mathcal{T}_{\\mathrm{DMG}}$ might query Q values of such points. In this assumption, we assume that the generalization in the mild generalization area is correct and meaningful. The rationale for such an assumption…\n\n**Q4: Minor comments**\n\nSpecial thanks for your careful review and we will update them in the latter revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9HDqV6WOIU", "reviewer": "Reviewer_Kscy", "summary": "The paper introduces a novel approach called Doubly Mild Generalization (DMG) to tackle the well-known issue of over-generalization in Offline Reinforcement Learning (RL). Offline RL often suffers from extrapolation errors and value overestimation due to the generalization of value functions or policies towards out-of-distribution (OOD) actions. To address these challenges, the authors propose DMG, which comprises two key components: (i) mild action generalization and (ii) mild generalization propagation. Mild action generalization involves selecting actions in the close neighborhood of the dataset to maximize the Q values, while mild generalization propagation mitigates the propagation of potential erroneous generalization through bootstrapping without impeding the propagation of RL learning signals. Theoretically, DMG ensures better performance than in-sample optimal policies under oracle generalization scenarios and controls value overestimation even in worst-case generalization scenarios. Empirically, DMG achieves state-of-the-art performance on standard offline RL benchmarks, including Gym-MuJoCo locomotion tasks and challenging AntMaze tasks, and demonstrates strong online fine-tuning performance.", "questions": "- Can you explain why using max and min in Eq.5? It’s a bit confusing as the policy is normally a Gaussian or deterministic policy.\n- What about directly regularizing the KL between  $\\pi$ and $\\hat \\beta$ in Eq 13?\n- Comparison with more SOTA algorithms such as STR is suggested.\n- What does the symbol × mean in Figure 1\n- It seems the performance of DMG  when $\\lambda=1$ in Figure 1 is inferior to that of TD3BC. I’m curious what’s the potential reason for this phenomenon as it should be better or similar to TD3BC when the in-sample term is removed.", "limitations": "The authors have discussed the limitations in the draft.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- **Innovative Approach**: The concept of doubly mild generalization is novel and addresses a significant challenge in offline RL by balancing the need for generalization with the risk of over-generalization.\n- **Theoretical Rigor**: The paper provides a thorough theoretical analysis of DMG, proving its advantages in both oracle and worst-case generalization scenarios. The theoretical guarantees add robustness to the proposed method.\n- **Empirical Performance**: DMG demonstrates superior empirical results, outperforming existing methods on multiple benchmarks. This includes both standard locomotion tasks and more complex navigation tasks, showcasing the versatility and effectiveness of the approach.\n- **Seamless Transition to Online Learning**: The method's ability to smoothly transition from offline to online learning, achieving strong fine-tuning performance, is a notable strength, making it practical for real-world applications where a mixture of offline data and online interactions is common.", "weaknesses": "- **Dependence on Continuity Assumptions**: The theoretical analysis relies on certain continuity assumptions about the Q function and transition dynamics. While these assumptions are standard, they may not hold in all practical scenarios, potentially limiting the applicability of the theoretical guarantees.\n- **Potential Sensitivity to Hyperparameters**: The performance of DMG might be sensitive to the choice of hyperparameters such as the mixture coefficient (λ) and penalty coefficient (ν). A more detailed analysis of hyperparameter sensitivity would be beneficial to understand the robustness of the method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the time and effort you are dedicated to providing feedback on our paper and are grateful for the meaningful comments.\n\n**Q1: Dependence on Continuity Assumptions.**\n\nThanks for the comment. Indeed, our work relies on continuity assumptions about the learned Q function and transition dynamics. For the former, a continuous learned Q function can be relatively easily satisfied and is particularly necessary for the analysis of value function generalization. For the latter, continuous transition dynamics is also a standard assumption in the theoretical studies of RL [1,2,3,4,5]. Several previous works assume the transition to be Lipschitz continuous with respect to (w.r.t) both state and action [1,2]. In our paper, we need the Lipschitz continuity to hold only w.r.t. action.\n\n**Q2: Potential Sensitivity to Hyperparameters**\n\nThanks for the comment. We will answer this question from two aspects: the robustness of the DMG principle and the robustness of the specific DMG algorithm. We have conducted an ablation study on the mixture coefficient $\\lambda$ and penalty coefficient $\\nu$ in Section 5.3. As shown in Figures 1 and 2, DMG generally achieves the best performance with $\\lambda \\in (0,1)$ and a moderate $\\nu$. This demonstrates that our DMG principle generally outperforms previous non/full generalization propagation and non/full action generalization principles, which proves the robustness of the DMG principle. On the other hand, our specific DMG algorithm also involves less hyperparameter tuning overall compared to other algorithms. As stated in Appendix C1, we use $\\lambda=0.25$ for all tasks, and use $\\nu=0.5$ for Antmaze, $\\nu \\in \\{0.1, 10\\}$ for Gym locomotion ($0.1$ for medium, medium-replay, random datasets; $10$ for expert and medium-expert datasets)。This demonstrates the robustness of the specific DMG algorithm.\n\n**Q3: Can you explain why using max and min in Eq. (5)?**\n\nWe apologize for the confusion. Eq. (5) defines mildly generalized policy, where $\\tilde \\beta$ and $\\hat \\beta$ can be any policy beyond Gaussian or deterministic policy. The max and min in Eq. (5) mean that for any $a_1 \\sim \\tilde \\beta(\\cdot|s)$ (i.e., from the defined mildly generalized policy), we can find $a_2 \\sim \\hat \\beta(\\cdot|s)$ (i.e., from the dataset) such that $\\|a_1-a_2\\| \\leq \\epsilon_a$. In other words, in a certain state, the distance between actions taken by $\\tilde \\beta$ and its closest action in the dataset is bounded by $\\epsilon_a$.\n\n**Q4: What about directly regularizing the KL between  $\\pi$ and $\\hat \\beta$ in Eq. (13)?**\n\nThanks for the question. This is equivalent to setting the inverse temperature $\\alpha$ as 0. The comparisons between DMG ($\\alpha=0$) and the original DMG on Gym locomotion tasks are shown in the following table. Compared to DMG, DMG ($\\alpha=0$) exhibits only a small drop in performance. Therefore, DMG is robust to this hyperparameter, and the choice of aligning $\\pi$ and $\\hat \\beta^*$ in implementation is not a key factor in DMG's good performance.\n\nTable 1: Averaged normalized scores on Gym locomotion tasks over 5 random seeds.\n\n| Dataset | DMG ($\\alpha=0$) | DMG |\n| --- | --- | --- |\n| halfcheetah-m | 53.6 | **54.9** |\n| hopper-m | 90.1 | **100.6** |\n| walker2d-m | 91.0 | **92.4** |\n| halfcheetah-m-r | 50.7 | **51.4** |\n| hopper-m-r | 97.4 | **101.9** |\n| walker2d-m-r | **93.3** | 89.7 |\n| halfcheetah-m-e | **93.3** | 91.1 |\n| hopper-m-e | 106.5 | **110.4** |\n| walker2d-m-e | 110.8 | **114.4** |\n| total | 786.7 | **806.8** |\n\n**Q4: Comparison with more SOTA algorithms such as STR is suggested.**\n\nThanks for the suggestion. Due to the page limit, we report the overall performance of STR (taken from its paper) and DMG in the following table. We will include the full comparison in our latter revision. Overall, DMG achieves slightly higher performance in both Gym locomotion and Antmaze domains.\n\nTable 2: Total scores on Gym locomotion and Antmaze tasks, averaged over 5 random seeds.\n\n| Dataset | STR | DMG |\n| --- | --- | --- |\n| locomotion total | 1162.2 | **1182.8** |\n| antmaze total | 430.2 | **439.4** |\n\n**Q5: What does the symbol × mean in Figure 1?**\n\nSorry for the confusion. The crosses (x) in Figure 1 mean that the value functions diverge in several seeds.\n\n**Q6: It seems the performance of DMG when $\\lambda=1$ in Figure 1 is inferior to that of TD3BC.**\n\nWhen $\\lambda=1$, DMG, just like TD3BC, allows full generalization propagation in value training. However, policy training and the specific hyperparameter configuration of DMG still differ from TD3BC. We hypothesize this may cause the discrepancy.\n\n**Reference**\n\n[1] Dufour, Francois, and Tomas Prieto-Rumeau. \"Finite linear programming approximations of constrained discounted Markov decision processes.\" SIAM Journal on Control and Optimization 51.2 (2013): 1298-1324.\n\n[2] Dufour, Francois, and Tomas Prieto-Rumeau. \"Approximation of average cost Markov decision processes using empirical distributions and concentration inequalities.\" Stochastics An International Journal of Probability and Stochastic Processes 87.2 (2015): 273-307.\n\n[3] Shah, Devavrat, and Qiaomin Xie. \"Q-learning with nearest neighbors.\" Advances in Neural Information Processing Systems 31 (2018).\n\n[4] Xiong, Huaqing, et al. \"Deterministic policy gradient: Convergence analysis.\" Uncertainty in Artificial Intelligence. PMLR, 2022.\n\n[5] Ran, Yuhang, et al. \"Policy regularization with dataset constraint for offline reinforcement learning.\" International Conference on Machine Learning. PMLR, 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nO7DEDyzqT", "reviewer": "Reviewer_iL9e", "summary": "The paper proposes an algorithm to allow for some mild generalization to OOD actions in offline RL by proposing constraints on which actions to generalize on and by constraining bootstrapping signal to avoid overestimation. It includes results and ablations on common offline RL benchmarks.", "questions": "1. How do the authors think the insights in this paper can be used for off-policy evaluation where the behavior policy is unknown and we want to evaluate a fixed target policy? The control setting allows some flexibility in say , constraining what the optimal policy can be, but with evaluation this seems trickier.\n2. The method seems to rely on using the empirical behavior policy to constrain the policy improvement. This seems reasonable, but the paper does not study this deeply. While this question may seem separate from the question investigated in this paper, in offline RL, it is typical for multiple policies to generate the data, which means the empirical policy is a multi-modal policy. I am curious how sensitive the proposed method would be to bad estimates in the empirical policy? At the very least I think the paper should discuss this point since the algorithm makes the assumption that such an empirical policy is easily accessible.", "limitations": "Yes, the paper addresses this in Section 6.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The idea is very simple (which is a good thing!), intuitive, and can be easily integrated into existing algorithms. I think it provides nice insight into how to think about generalization and overestimation which can be useful for others.\n2. The paper does a nice job of explaining the intuition behind the method and faults with previous methods. I appreciated Table 1.\n3. The paper includes results on downstream performance metrics as well as ablation studies.", "weaknesses": "1. It is a bit concerning that the main results are over only 5 seeds. While it is commonly done in prior work, I don’t think that means it's correct. I’d refer authors to [1]. I think it would be nice to increase the number of seeds.\n2. Related to above, the variations seem to be based on standard errors (for example, Table 2) but it has typically been documented that these are unreliable and should be avoided [1] in RL research. Instead opting for some high-confidence confidence interval would be better.\n\nI would be willing to increase the score if the results were still true with more seeds.\n\n[1] Empirical Design in Reinforcement Learning. Patterson et al. 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the time and effort you are dedicated to providing feedback on our paper and are grateful for the meaningful comments.\n\n**Q1: More random seeds and use of confidence interval in evaluation.**\n\nThanks for the kind suggestions and this nice reference. [1] provides a comprehensive resource for how to do good experiments in RL. We have taken your advice and conducted experiments on 10 random seeds, reporting results based on a 95% confidence interval (CI). The following table shows the comparison between the newly obtained results (10seeds/95%CI) and the previously reported results (5seeds/SD). We also present the new learning curves on all the tasks in **Figures 1 and 2** of the PDF (attached to the global response). The results show that our method achieves about the same performance as under the previous evaluation criterion. In the latter revision, we will cite [1] and update the results accordingly.\n\nTable 1: Comparison of DMG under different evaluation criteria.\n\n| Dataset | DMG (5seeds/SD) | DMG (10seeds/95%CI) |\n| --- | --- | --- |\n| halfcheetah-m | **54.9$\\pm$0.2** | **54.9$\\pm$0.3** |\n| hopper-m | **100.6$\\pm$1.9** | 100.5$\\pm$1.0 |\n| walker2d-m | **92.4$\\pm$2.7** | 92.0$\\pm$1.2 |\n| halfcheetah-m-r | **51.4$\\pm$0.3** | **51.4$\\pm$0.4** |\n| hopper-m-r | 101.9$\\pm$1.4 | **102.1$\\pm$0.6** |\n| walker2d-m-r | 89.7$\\pm$5.0 | **90.3$\\pm$2.8** |\n| halfcheetah-m-e | 91.1$\\pm$4.2 | **92.9$\\pm$2.1** |\n| hopper-m-e | **110.4$\\pm$3.4** | 109.0$\\pm$2.6 |\n| walker2d-m-e | **114.4$\\pm$0.7** | 113.9$\\pm$1.2 |\n| halfcheetah-e | **95.9$\\pm$0.3** | **95.9$\\pm$0.2** |\n| hopper-e | 111.5$\\pm$2.2 | **111.8$\\pm$1.3** |\n| walker2d-e | **114.7$\\pm$0.4** | 114.5$\\pm$0.3 |\n| halfcheetah-r | **28.8$\\pm$1.3** | 28.7$\\pm$1.2 |\n| hopper-r | 20.4$\\pm$10.4 | **21.6$\\pm$6.6** |\n| walker2d-r | 4.8$\\pm$2.2 | **7.7$\\pm$3.0** |\n| locomotion total | 1182.8 | **1187.2** |\n|  |  |  |\n| antmaze-u | **92.4$\\pm$1.8** | 91.8$\\pm$1.6 |\n| antmaze-u-d | **75.4$\\pm$8.1** | 73.0$\\pm$5.0 |\n| antmaze-m-p | 80.2$\\pm$5.1 | **80.5$\\pm$2.1** |\n| antmaze-m-d | **77.2$\\pm$6.1** | 76.7$\\pm$3.6 |\n| antmaze-l-p | 55.4$\\pm$6.2 | **56.7$\\pm$3.6** |\n| antmaze-l-d | **58.8$\\pm$4.5** | 57.2$\\pm$2.7 |\n| antmaze total | **439.4** | 435.9 |\n\n**Q2: How do the authors think the insights in this paper can be used for off-policy evaluation where the behavior policy is unknown and we want to evaluate a fixed target policy?** \n\nThanks for this insightful question. Indeed, the evaluation setting appears trickier because the target policy is fixed. While mild action generalization no longer seems applicable in this scenario, mild generalization propagation still applies. Its direct analogue in the off-policy evaluation scenario is that: the Bellman target uses a mixture of (1) the Q-value of the action outputted by the target policy and (2) the Q-value of the nearest neighbor of that action in the dataset (or the replay buffer). This approach brings bias on the one hand, but can also be expected to have a smaller variance. An in-depth analysis of this would be an interesting direction for future work.\n\n**Q3: The method seems to rely on using the empirical behavior policy to constrain the policy improvement.**\n\nWe apologize for the confusion. Actually, our method does not require training an empirical behavior policy. The implementation of policy improvement is discussed in Section 3.5 of the paper. Here, we first define a reshaped empirical behavior policy $\\hat\\beta^*$. Then we enforce the proximity between the trained policy and the reshaped behavior policy in Eq. (13). A special design worth noting is that we use the forward KL divergence $\\mathrm{KL}(\\hat\\beta^*(\\cdot|s) \\| \\pi_\\phi(\\cdot|s))$, which not only allows $\\pi$ to select actions outside the support of $\\hat\\beta^*$ but also eliminate the need for pretraining an empirical behavior policy. Specifically, by substituting the analytical expression of $\\hat\\beta^*$ into the KL divergence and using importance sampling (the same derivation as in AWR [2]), Eq. (14) eliminates the need for pre-training a behavior model (maybe multi-modal as you mentioned) and enables direct sampling from the dataset for optimization.\n\n**Reference**\n\n[1] Patterson, Andrew, et al. \"Empirical design in reinforcement learning.\" arXiv preprint arXiv:2304.01315 (2023).\n\n[2] Peng, Xue Bin, et al. \"Advantage-weighted regression: Simple and scalable off-policy reinforcement learning.\" arXiv preprint arXiv:1910.00177 (2019).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
### **Global Response**

We thank all the reviewers for taking the time to read our manuscript carefully and for providing constructive and insightful feedback. We are encouraged by the positive comments of the reviewers, such as:

- Meaningful research problem, innovative approach, and nice insights (Reviewers kJjy/Kscy/iL9e);
- Strong theoretical results with rigorous proofs (Reviewers kJjy/RKt1/Kscy);
- Superior performance on multiple benchmarks with many baselines and several ablation studies (Reviewers kJjy/RKt1/Kscy/iL9e);
- Seamless transition to online learning and strong fine-tuning performance (Reviewer Kscy).

Meanwhile, we have been working hard to address the reviewers' concerns and questions and have provided detailed responses to the individual reviews below. We hope our response could address the reviewers' concerns. We would be more than happy to resolve any remaining questions in the time we have and are looking forward to engaging in a discussion.

Manuscript updates:

- Update the results with 10 random seeds based on [1].
- Include the discussions regarding [2].
- Add more explanations of how Eq. (4) relates to the generalizability of Q functions.
- Rephrase the term "well-defined" and include more explanations of Assumption 1.
- Correct other typos.

Reference:

[1] Patterson, Andrew, et al. "Empirical design in reinforcement learning." arXiv preprint arXiv:2304.01315 (2023).

[2] Bose, Avinandan, Simon Shaolei Du, and Maryam Fazel. "Offline multi-task transfer rl with representational penalization." arXiv preprint arXiv:2402.12570 (2024).

Meta-review: This paper studies an important problem of generalization in offline reinforcement learning. Recent in-sample algorithms reduce the extrapolation error and value overestimation problem but the resulting policy could be too restrictive. The proposed method, Doubly Mild Generalization (DMG), comprises of mild action generalization and mild generalization propagation. It balances the need for generalization with the risk of over-generalization in offline RL.

All reviewers appreciate its rigorous theoretical analysis and thorough empirical validation on multiple offline RL benchmarks. Some reviewers also consider it a novel approach in offline RL, and did a nice job in explaining the intuition behind the algorithm.

There were some shared concerns on the continuous assumption in the theory, and questions on the sensitivity of hyperparameters, and lack of discusson on related works. The rebuttal has addressed all the concerns. The reviewers reached a unanimous agreement on accepting this submission with a high confidence.
Decision: Accept (poster)