When Data Geometry Meets Deep Function: Generalizing Offline Reinforcement Learning
Authors: Jianxiong Li
Xianyuan Zhan
Haoran Xu
Xiangyu Zhu
Jingjing Liu
Ya-Qin Zhang
Venue: ICLR 2023 poster (2023)
Keywords: offline reinforcement learning
deep Q functions generalization

Abstract: In offline reinforcement learning (RL), one detrimental issue to policy learning is the error accumulation of deep \textit{Q} function in out-of-distribution (OOD) areas. Unfortunately, existing offline RL methods are often over-conservative, inevitably hurting generalization performance outside data distribution. In our study, one interesting observation is that deep \textit{Q} functions approximate well inside the convex hull of training data. Inspired by this, we propose a new method, \textit{DOGE (Distance-sensitive Offline RL with better GEneralization)}. DOGE marries dataset geometry with deep function approximators in offline RL, and enables exploitation in generalizable OOD areas rather than strictly constraining policy within data distribution. Specifically, DOGE trains a state-conditioned distance function that can be readily plugged into standard actor-critic methods as a policy constraint. Simple yet elegant, our algorithm enjoys better generalization compared to state-of-the-art methods on D4RL benchmarks. Theoretical analysis demonstrates the superiority of our approach to existing methods that are solely based on data distribution or support constraints. Code is available at https://github.com/Facebear-ljx/DOGE.

Published as a conference paper at ICLR 2023
WHEN DATA GEOMETRY MEETS DEEP FUNCTION: GENERALIZING OFFLINE REINFORCEMENT LEARNING
Jianxiong Li', Xianyuan Zhan Haoran Xu', Xiangyu Zhu', Jingjing Liu' & Ya-Qin Zhang'* 1 Institute for AI Industry Research (AIR), Tsinghua University, Beijing, China
2 Shanghai Artificial Intelligence Laboratory, Shanghai, China
11 1i-jx21 @ma saaails..... tsiinguua. edu.
ABSTRACT
In offline reinforcement learning (RL), one detrimental issue policy learn- ing is the eπror accumulation of deep Q function in out-of-distribution (OOD) areas. Unfortunately, existing offline RL methods are often over-conservative, inevitably hurting generalization performance outside data distribution. In our study, one interesting observation that deep Q functions approximate well in- side the convex hull of training data. Inspired by this, we propose a new method, DOGE cestance-sensitiig Offline RL with better GEneralization). DOGE marries dataset geometry with deep function approximators in offline RL, and enables exploitation in generalizable OOD areas rather than strictly constraining policy within data distribution. Specifically, DOGE trains state-conditioned distance function that can be readily plugged into standard actor-critic methods as a pol- icy constraint. Simple yet elegant, our algorithm enjoys better generalization compared to state-of-the-art methods on D4RL benchmarks. Theoretical anal- ysis demonstrates the superiority of our approach to existing methods that are solely based on data distribution or support constraints. Code is available at https: http//gtthhh.tt/ee////////
INTRODUCTION
Offline reinforcement learning (RL) provides a new possibility to learn optimized policies from large, pre-collected datasets without any environment interaction (Levine al., 2020). This holds great promise to solve many real-world problems when online interaction is costly or dangerous yet historical data easily accessible (Zhan al., 2022). However, the optimization nature of RL. as well as the need for counterfactual reasoning on unseen data under offline setting, have caused great technical challenges for designing effective offline RI. algorithms. Evaluating value function outside data coverage areas can produce falsely optimistic values; without corrective information from online interaction, such estimation errors can accumulate quickly and misguide policy learning process (Van Hasselt et al., 2018; 8; Fujimoto et al., 2018; Kumar al., 2019).
Recent model-free offline RL methods investigate this error accumulation challenge several ways: 1) Policy Constraint: directly constraining learned policy to stay inside distribution, or with the support of dataset (Kumar et al., 2019); 2) Value Regularization: regularizing value function to assign low values out-of-distributio (OOD) actions (Kumar et al., 2020b); 3) In-sample Learning: learning value function within data samples (Kostrikov et al., 2021b) or simply treating it as the value function of behavioral policy (Brandfonbrener al., 2021). All three schools of methods share similar traits of being conservative and omitting evaluation on OOD data, which brings benefits of minimizing model exploitation error, but the expense of poor generalization of learned policy in OOD regions. Thus, gaping gap still exists when such methods are applied to real-world tasks, where most datasets only partially cover state-action space with suboptimal policies.
Meanwhile, online deep reinforcement learning (DRL) that leverages powerful deep neural network (DNN) with optimistic exploration on unseen samples can yield high serforrffoning policies with promising generalization performance (Mnih et al., 2015; Silver et al., 2017; Degrave et al., 2022;
Corresponding authors

---

Published as a conference paper at ICLR 2023
Policy constraint
Value regularization
In-sample learning
antmare-lart waweeerrgge-----eeeee
antraaze large playw22
large-playy2
entrmare- entrmare-large- weree-lergg-------
Nomaltted Return
80
Maxinng TD3-BC
Nomalize Return
Sissing COL
Nomalize Return
Missing
10
Training Steps (x10%)
Training Staps (x10%)
Training Steps (x10%)
Figure 1: Left: Visualization of AntMaze dataset. Data transitions of two small areas on the critical pathways to the destination have been removed (red box). Right: Performance of three SOTA offline RI. methods.
Packer et al., 2018). This staring contrast propels us to re-think the question: Are we being too conservative? It is well known that DNN has unparalleled approximation and generalization abilities, compared with other function approximators. These attractive abilities have not only led to huge success in computer vision and natural language processing (He et al., 2016; Vaswani et al., 2017), but also amplified the power of RL. Ideally, in order to obtain the best policy, an algorithm should enable offline policy learning unseen state-action pairs that function approximators (e.g., 2 function, policy network) can generalize well, and add penalization only on non-generalizable areas.
However, existing offline RL methods heed too much conservatism data- related regularizations, while largely overlooking the generalization ability of deep function approximators. Intuitively, let us consider the well-known AntMaze task in the D4RL benchmark (Fu et al., 2020), where an navigates from the start to the destination in large maze. We observe that existing offline RL methods fail miserably when we remove only small areas of data on the critical pathways to the destination. As shown in Figure 1, the two missing areas reside in close proximity to the trajectory data. Simply "stitching" up existing trajectories as approximation is not sufficient to form a near-optimal policy at missing regions. Exploiting the generalizability of deep function appoximators, however, can potentially compensate for the missing information.
In our study, we observe that the value function approximated by DNN can interpolate well but struggles to extrapolate (see Section 2.2). Such an "interpolate well" phenomenon is also observed in previous studies the generalization of DNN (Haley & Soloway, 1992; Barnard & Wessels, 1992; Arora et al., 2019a; Xu et al., 2020; Florence et al., 2022). This finding motivates us to reconsider the generalization of function approximators in offline RL in the context of dataset geometry. Along this line, we discover that a closer distance between a training sample to the offline dataset often leads to smaller value variation range of the learned neural network, which effectively yields more accurate inference of the value function inside the convex hull (formed by the dataset). By contrast, outside the convex hull, especially in those areas far from the training data, the value variation range usually renders too large to guarantec a small approximation еттоr.
Inspired by this, we design a new algorithm DOGE (Distance-sensitive Offline RL with better GEneralization) from the perspective of generalization performance of deep Q function. We first propose a state-conditioned distance function to characterize the geometry of offline datasets, whose output serves as proxy to the network generalization ability. The resulting algorithm learns state-conditioned distance function as policy constraint on standard actor-critic RL framework. The- oretical analysis demonstrates the superior performance bound of our method compared to previous policy constraint methods that are based on data distribution or support constraints. Evaluations on D4RL benchmarks validate that our algorithm enjoys better performance and generalization abilities than state-of-the-ar offline RL. methods.
2 DATA GEOMETRY VS. DEEP Q FUNCTIONS
2.1 NOTATIONS
We consider the standard continuous action space Markov decision process (MDP) setting, which can be represented by tuple (8, A, P, 7), where S and A are the state and action space, P(s'|s, a) is the transition dynamics, r(s, a) is a reward function, and 7 E [0, 1) is a discount factor. The objective of the RL problem is to find a policy a(als) that maximizes the expected cumulative discounted return, which can be represented by a @ function Q7 (s. a) = E|
ex (r((s, a)]ls0 =
8, (4g a, a4 (-(1s),, 81+1 P(- a.)). The function is typically approximated by function
2

---

Published as a conference paper at ICLR 2023
1.0
1.0
1.0
High
Approximation Error
Approximation
0.5
Actin
0.0
0.0
-0.5
-0.5
-0.5
low
-1.0
-0
-5
0
5
10
10
10
10
-5
State
State
Figure 2: Approximation error of deep Q functions with different dataset geometry. Offline data are marked as white dots (Please refer Appendix E.5 for detailed experimental setup).
approximators with learnable parameters 8, such as deep neural networks. Under offline RL setting, we are only given fixed dataset DD and cannot interact further with the environment. Therefore, the parameters are optimized by minimizing the following temporal difference (TD) error:
min ())D [ a) + + 1Q7 (8', a'))) - (s,a) 0
where Qy is the target Q function, which a delayed copy of the current Q network.
2.2 INTERPOLATE VS. EXTRAPOLATE
Motivating examples. Let's first consider set of simple one-dimensional random walk tasks with different offline datasets, where agents at each step can take an action to move in the range of [-1, 1]. and the state space is straight line ranging from [-10, 10). The destination is located 8 = 10. The closer to the destination, the larger reward the agent gets (i.e., r = 1 at 8 10, r = at -10.. The approximation errors of the learned Q functions are visualized in Figure Note that the approximation errors of the learned Q functions tend to be low at state-action pairs that lie inside or near the boundaries of the convex hull formed by the dataset. Under continuous state-action space, state-action pairs within the convex hull of the dataset can be represented in an interpolated manner (referred as interpolated data), iee, Tin = > OgTi; Σx= a; 1,a; 2 O, 0, = (st, a,) E 'D; similarly, we can define the extrapolated data that lie outside the convex hull of the dataset as Tout Ex=1 3,04, where 2x 3 and 3 2 0 do not hold simultaneously.
We observe that the geometry of the datasets play special role on the approximation error of deep Q functions, or in other words, deep @ functions interpolate well but struggle to extrapolate. This phenomenon is also reflected in studies under a supervised learning setting (Haley & Soloway, 1992; Barnard & Wessels, 1992; Arora et al., 2019a; Xu et al., 2020; Florence et al., 2022), but is largely overlooked in modern offline RL.
the generalization performance of deep neural networks
Theoretical explanations. Based on advanced theoretical machinery from the generalization analysis of DNN, such neural tangent kernel (NTK) (Jacot et al., 2018), demonstrate that this phenomenon is also carried over to the offline RL setting for deep Q functions. Define ( arg mi || x; (we denote |||r|]| Fuclidean norm) as the projection operator that projects unseen data x to the nearest data point in dataset D. Theorem 1 gives theoretical explanation the "interploate well" phenomenon for deep Q functions under the NTK assumptions (see Appendix B.2 for detailed proofs):
theoretically
Theorem 1. (Value difference of deep Q function for interpolated and extrapolated data). Under the NTK regime, given unseen interpolated data Zin and an extrapolated data Tout, then the value difference of deep function for interpolated and extrapolated input data can be bounded as:
[[Qo(zn)) QP < C1( ( //mnn|laaaaa 1. [PPoo (2)))) )l1> den + +2dam)
< CC(v (5i(minn(/) [in (Z.)) (IV + 2B)
(2)
[[Qo(Fout) - (Proj ))ll < C1(v (i/i//mnni) (] Il. |Proj (Fout) Il) doan + 2d
(3)
where daan = || Projp (ro < max I| -xell < B and dzan = ||
are distances of Xin and x Fout to the nearest data points in dataset D. and C₁ are finite constants.
Theorem 1 shows that given an unseen input x. Qo(x) can be controlled by in in-sample Q value Qo (Projn and the distance ||æ Projp(x)]. The smaller the distance, the more controllable the output of deep Q functions. Therefore, because the distance dataset is strictly bounded (at
3

---

Published as a conference paper at ICLR 2023
most B for interpolated data), the approximated values interpolated data well extrapolated data near the boundaries of the convex hull formed by the dataset cannot be too far off. Moreover, as doo can take substantially larger values than dz... interpolated data generally enjoys tighter bound compared with extrapolated data, if the dataset only narrowly covers a large state-action space.
Empirical observations in Figure and Theorem both demonstrate that data geometry induce different approximation error accumulation patterns for deep Q functions. While approximation error accumulation is generally detrimental to offline RL, a fine-grained analysis is missing in previous studies about where value function can approximate well. We argue that is necessary to take data geometry into consideration when designing less conservative offline RL algorithms.
GENERALIZABLE OFFLINE RL FRAMEWORK
In this section, present algorithm DOGE (Distance-sensittive Offine RL with better GEner- alization). By introducing a specially designed state-conditie conditoond distance function to characterize the geometry of offline datasets, we can construct a very simple, less conservative and also more generalizable offline RL algorithm upon standard actor-critic framework.
3.1 STATE-CONDITIONED DISTANCE FUNCTION
As revealed in Theorem 1, the comple-to-dttern distance plays an important role in measuring the controllability of Q values. However, given arbitrary state-action sample (s,a), a), naively computing its distance to the closest data point in a large dataset can be costly and impractical. Ideally, we prefer to have a learnable distance function which also has the ability to reflect the overall dataset geometry. Based on this intuition, we design a state-conditioned distance function that can be learned in an elegantly simple supervised manner with desirable properties.
Specifically, we learn the state-conditioned distance function g(s, a) by solving the following re- gression problem, with sate-action pairs (s, a) D and synthetic noise actions sampled from the uniform distribution over the full action space A:
min (( [Banunifcaillaa
g(s, a)]²]
In practical implementation, for each (s,a) ~ D, we sample N noise actions uniformly in the action space A to train g(-). More implementation details can be found in Appendix E. Morcover, with the optimization objective defined in Eq. (4), we can show that the optimal stateccondiiooned distance function has two desirable properties (proofs can be found Appendix C):
The optimal conditiond distance function of Eq. (4) convex w.r.t. actions and an upper bound of the distance to the nute-conditioned centroid as(s) of training dataset D:
Property
g" (s. à) = En-Unif(A) [C(s,, a)llâ all]
2 l|ā -a--uuus a). a]ll = llā (8) Vā A
(5)
where C(s, Be)(tif(A))((s 2 0, p(s, a) is state-action distribution of dataset D. Given a state
p(s.a)
8 E D, the state-conditioned centroid defined as (s))) = Eu-Unsf(A (A)) (C(s, a) a). Since L2-norm convex and the non-negative combination of convex functions still convex, g" (s, à) is also convex function w.r.f. à.
Property 2.
The negative gradient the optimal state-conditioned distance function at an extrapo- lated action -ag* a), points inside the convex hull of the dataset.
From Property 1, we can see that the optimal state-conditioned distance function characterizes data geometry and outputs an upper bound the distance to the state-conditioned centroid of the training dataset. Property 2 indicates that if we use the learned distance function as a policy constraint, it can drive the learned policy to move inside the convex hull of training data. We visualize the value of the trained state- doneeoooddiooeed distance function in Figure 3. It is clear that the learned distance function can accurately predict the sample-to-datee -tttsstt centroid distance, By utilizing such distance function, constrain the policy based on the global geometric information of training datasets. This
4

---

Published as a conference paper at ICLR 2023
g (s,
Training data
CAaSEn Value
Cassee Distande Vaaae
Disanoon VAlA
Distance to training data
Action
Distance function output
2.0
State
Osaa Vakao
OASEENCE value
Aciton
-1.
0.0 State
25
0.0
2.5
(a) Illustration of as,
(b) Visualizution of g (s, a) trained on diverse 2D datasets
Figure 3: Illustration state-conditioned distance function. The output the optimal distance function the non-negative combination of the distances to all training data. G is the threshold in Eq. (6) In (b), Offline data are marked as white dots.
desirable property is non-obtainable by simply constraining the policy based sample-to tolleett--ssee tt-ss distance such as the MSE loss between policy generated and dataset actions, which can only bring local geometric information. Moreover, the learned distance function can not only predict well in-distribution states but also generalize well at OOD states.
3.2 MASTANCE-SENSITIE OFFLINE REINFORCEMENT LEARNING
Capturing the geometry of offline datasets, we construct minimalist distance-sensitive offline RL framework, by simply plugging the stteeconditionee distance function as a policy constraint into standard online actor-critic methods (such as TD3 (Fujimoto al., 2018) and SAC (Haarnoja et al.,
2018)). This results in the following policy maximization objective: s.t. B,-Doo-<(11)00(s,00 S
arg max B [Q a)]
(6)
where G' is task- deeendennddnt threshold varying across tasks. In our method, we adopt non parametric treatment by setting G as the mean output (50% quantile) of the learned distance function on the training dataset, i.e., (. [s(s, a)]. which is approximated over mini-batch samples reduce computational complexity (see Appendix G for ablation on G). The constrained optimization problem in Eq. (6) can be reformulated as:
arg max min (g- [BQ(s, a) X(g(x,a) G)]
8.l.
A 0
where À is the Lagrangian multiplier, which auto-adjusted using dual gradient descent. Following TD3+BC (Fujimoto & Gu, 2021), @ values are rescaled by 8 E.- [Q(s.aa)] to balance O function maximization and policy constraint satisfaction, controlled by a hyperparameter a. To reduce computations, the denominator of 3 is approximated over mini-batch of samples. The resulting algorithm easy to implement, In our experiments, we use TD3. Please refer to Appendix E for implementation details.
3.3
RELAXATION WITH BELLMAN-CONSISTENN COFFFICIENT
3.3.1 BELLMAN-CONSISTIE COEFFICIENT AND CONSTRAINED POLICY SET
The key difference between DOGE and other policy constraint methods lies in that DOGE relaxes the strong full coverage assumption' on offline datasets and allows exploitation on generalizable OOD areas. To relax the unrealistic full-coverage assumption, we resort to weaker condition proposed by (Xie et al., 2021a), the Bellman-consiste coefficient (Definition 1), to measure how well Bellman errors can transfer to different distributions (Theorem 2).
Denote 1lf E, f| T is the Bellman operator of policy म, defined T a) r(s, a) - s' (s.a) [Q(s', a')] == r(s, a) +> [Q(s', a a/)]. [P[.] the brief notation for is the function class of Q networks. The Bellman-consistent coefficient
Ee'nni--l') (
is defined as:
'sup,a v(s.a) p(M.)) 00, and j marginal distrihutions of the learned policy and the dataset (I.c al., 2019).

---

Published as a conference paper at ICLR 2023
Definition 1. from an arbitrary distribution v to data distribution μ. w.E.t. F and n
coefficient). We define B(v, μu, IT) to measure the distributional
QQQ 7 Q
BlU, H. F,)) sup QET QQ - - T
(8)
This definition captures the generalization performance of function approximation across different distributions. Intuitively, small value of B(v, F,, means Bellman errors for policy can accurately transfer from distribution μ to v. This suggests that Bellman errors can transfer well between two distributions even if large discrepancy exists, long the Bellman--conistent coefficient is small.
Based on Definition 1, we introduce the definition of Bellman-connistent constrained policy set. Definition (Bellmun-consistend constrained policy set). We define the Dellman-connsitten con- strained policy set as Hg. The Bellman-consistent coefficient under the transition induced by Ng can be bounded by some finite constants l(k)):
(pk: F. S ((k)
(9)
where = Pu VT,,... Th E lgg. Po is the initial state-action distribution and p is the transition operator induced by nu i.e., pm (8', a'|s, a) P(s' (ss1|s, a)T:(aa (a))
We denote the constrained Bellman operator induced by I as T Tll := r(s, a) maXrelly yPPT (s', a²)]. T can be seen Bellman operator on a redefined MDP, thus theoreti- cal results MDP also carry over, such as contraction mapping and existence of a fixed point.
3.3.2 BELLMAN CONSISTENT COFFFICIENT AND PERFORMANCE BOUND OF DOGE
We show that the policy set induced by DOGE is essentially Bellman- connsstet policy set defined in Definition 2. Meanwhile, the distance constraint in DOGE can produce small value of B and hence guarantee the learned policy deviates only to those generalizable areas.
Theorem 2. (Upper bound of Bellman-consissent coefficient). Under the NTK assumption, the Bellman-consistent coefficient B(v, F,F. π) is upper bounded as:
B(e, p. F.
< 4
(1 (s0 an) Rmax + Ci (Cavdi + di) + +2 +2--7)CrP
(CoVTs +
B1
B2
Ba
2,0
(10)
EZnDlZi [z] is the centroid of offline dataset. di = || - sool and d2 = I|a' Toll are the sample-to-centroic distances. C2 / A ||] is related to the upper bound of the input scale. Cp is the lower bound of Bellman error (square) ) for
where we denote x (s, a) and x' (s', To
under distribution μ, i.e., Ep < I|Q - -T
The RHS of Eq. (10) contains four parts: 1. B1, B, and B3. It is reasonable to assume En > 0, because of the approximation error of Q networks and the distribution mismatch between μ and B, is only dependent on the Q value Q(so: a.) at the centroid of the dataset and the max reward Rmaxx B3 is related distance d, and distribution B3 is related to d2, and P=. To be mentioned, the distance regularization DOGE compels the learned policy to output the action that is near the state-conditioned centroid of dataset, thus B2 and Ba can be driven small values. Therefore, the RHS of Eq. (10) can be bounded by finite constants under DOGE, which shows that the constrained policy set induced by DOGE essentially
constrained policy set.
Then, the performance gap between the policy learned by DOGE and the optimal policy can be bounded as given in Theorem See Appendix D.I and D.2 for the proof of Theorem 2 and 3.
Theorem 3. (Performance bound of the learned policy by DOGE). Let Qu be the fixed point of Tils, i.e., QuIe The Qils, and e = Q* -Tlle is the Bellman error the k-th iteration. [l/ = F2 [|] II]. The performance of the learned policy nn is bounded by:
27
L(lTg) sup l|calla + 1-7 a (Tg)) 420
lim |Q" - +00
--== Q=a1lpo -
(11)
(1 71
2y

---

Published as a conference paper at ICLR 2023
where ((IB)) v(1 - (1 k= Ex=
((k)) which is similar to the concentrability coefficient in BEAR (Kumar et al., 2019) but in a different form. Note that [(k) is related to the RHS of Eq. (10) and
can be driven to small value by DOGE according to Theorem 2. a
I - TQQ 1lx
is the suboptimality constant, which is similar to (IT) = Q - TQ ao in BEAR.
Compared with BEAR, DOGE allows a policy shift to some generalizable OOD areas and relaxes the strong full-coverage assumption. In addition, we have L(Ilg) < L(IT) αc PaP
j(s,a) where L(II) is the concentrability coefficient BEAR. This is evident when (u(s, a) = and pa Prt..... (s,a) > O, ((lg)) can be bounded by finite constants but L(II) set to cover more generalizable OOD areas (II C Ill) and produces a larger feasible region for optimization, lower degree of suboptimality can be achieved (i.e., a (IIg) S. (III) compared to only performing optimization on II. Therefore, we can see that DOGE enjoys tighter performance bound than previous more conservative methods when allowed exploit generalizable OOD areas.
oc. Moreover, as IIg extends the policy
4 EXPERIMENTS
For evaluation, We compare DOGE and prior offline RL methods over D4RL Mujoco and AntMaze tasks (Fu et al., 2020). Mujoco is standard benchmark commonly used in previous work. AntMaze tasks are far more challenging due the markovian and mixed-quality offline dataset, the stochastic property of environments, and the high dimensional state-action space. Implementation details, experimental setup and additional experimental results can be found in Appendix E and F.
4.1 COMPARISON WITH SOTA
We compare DOGE with model-free SOTA methods, such as TD3+BC (Fujimoto & Gu, 2021), I), CQL (Kumar al., 2020b) and IQL. (Kostrikov al., 2021b). For fairness, use the "-v2" datasets for all methods. For most Mujoco tasks, we report the scores from the IQL paper. We obtain the other results using the authors' or our implementations. For AntMaze tasks, we obtain the results CQL, TD3+BC, and IQL using the authors' implementations. For BC (Pomerleau, 1988), BCQ (Fujimoto al., 2019) and BEAR (Kumar al., 2019), we report the scores from (Fu et al., 2020). All methods are evaluated over the final 10 evaluations for Mujoco tasks and 100 for AntMaze tasks.
Table shows that DOGE achieves comparable or better performance than SOTA methods on most Mujoco and AntMaze tasks. Compared to other policy constraint approaches such as BCQ, BEAR and TD3+BC, DOGE the first policy constraint method to successfully solve AntMaze-medium and AntMaze-large tasks. Note that IQL. is an algorithm designed for multi-step dynamics programming and attains strong advantage on AntMaze tasks. Nevertheless, DOGE can compete with or even surpass IQL most AntMaze tasks, by only employing generalliationnoriennee policy constraint. These results illustrate the benefits of allowing policy learning on generalizable OOD areas.
4.2 EVALUATION ON GENERALIZATION
To evaluate the generalization ability of DOGE, we remove small areas of data from the critical pathways to the destination in AntMaze medium and large tasks, to construct an OOD dataset. The two removed areas reside in close proximity to the trajectory data (see Figure 1). We evaluate representative methods (such as TD3+BC, CQL, IQL) and DOGE on these modified datasets. Figure 4 shows the comparison before and after data removal.
For such a dataset with partial state-action space coverage, existing policy constraint methods tend to over-constrain the policy to stay inside the support of a dataset, where the optimal policy is not well-covered. Value regularization methods suffer from deteriorated generalization performance, as the value function is distorted to assign low value at all OOD areas. In-sample learning methods are only guaranteed to retain the best policy within the partially covered dataset (Kostrikov et al., 2021b). As shown in Figure 4, all these methods struggle to generalize well on the missing areas and suffer severe performance drop, while DOGE maintains competitive performance. This further demonstrates the benefits of relaxing over-conservatism in existing methods.
7

---

Published as a conference paper at ICLR 2023
Table I: Average normalized scores and standard deviations over seeds benchmark tasks
Dataset
BC
BCQ
BEAR
TD3+BC
CQL
IQL
DOGE(ourS)
hopper-r
4.9
7.1
14.2
8.5
8.3.1.22
7.910.4
21.1
halfcheetah-r
0.2
8.8
15.1
11.011111
20.0
11.2-2.9
17.8
walker2d-r
1.7
6.5
10.7
1.6=1.7
8.3+0.1
5.9
0.9 +2.4
hopper-m
52.9
56.7
51.9
59.3+442
58.5#2.1
66.2=5.7
98.622.1
halfchectah-n
42.6
47.0
41.0
48.3+10.3
44.0=#544
47.4=0.2
45,3=0.6
walker2d-m
75.3
72.6
80.9
83.7+-2.1
72.5-00.8
78.3=8.7
0.8
hopper-m-r
18.1
53.3
37.3
60.9
95.0-1644
94.7=8.6
76.2+
halfcheetah-m-r
36.6
40.4
29.7
44.6+10.5
45.5
44.2-112
42.8+0.66
walker2d-m-r
26.0
52.1
18.5
81.855.5
77.2=2.55
73.8=7.1
87.312.3
hopper-m-e
52.5
81.8
17.7
98.0119.4
105.416.8
91.5==4.3
102.7:5,2
halfcheetah-m-e
55.2
89.1
38.9
90.71:4.3
91.61288
86.7=5.3
78.718.4
walker2d-m-e
107.5
109.5
95.4
110.1
108.8+0.7
109.6+1.0
110.4+1.5
locomation total
473.5
624.9
451.3
698.5++49.0
726.1+31.7
717.4+55.9
768.6+55..4
antmaze-u
65.0
78.9
73.0
91.315.7
84.811..
88.2-1.9
97
antmazo-u-d
55.0
55.0
61.0
54.6+16.2
43.3+5.4
66.7=4.0
63.5++9.3
antmaze-m-p
0.0
0.0
0.0
0.0
65.2+4.8
70.4=5.3
80.6=6.5
antmaze-m-d
0.0
0.0
8.0
0.0
14.0=11.7
74.6=3,2
77.6=6.1
antmaze-l-p
0.0
6.7
0.0
0.0
15.3
43.5
48.2.18.1
antmaze-l-d
0.0
2.2
0.0
0.0
31.6199.5
4..6
36.419..
antmaze-total
120.0
142.8
142.0
145.9421.9
297. 297.7=49.0
389.0:266.5
403.3+400.9
Policy constraint
Value regularization
In-sample learning
DOGE (Ours)
antmaze-largeella--22
entrmaze-argg-play--ee
antmare-frre--ley--2
antmerzelargge- parpere--rrrge-aa-yyy
Nomalized Return
TD3-BC
Nomalize Return
COL
Nomalized Return
Nomalizes Return
dala DOGE
Maxoro data TD9+80
Missing dana COL
Massia data
Mixsing data DOGE
0.0
0.2
0.6
0.
1.0
0.0
0.2
0.6
0.0
0.6
1.0
0.0
0.2
04
06
05
02 04
Training Steps (x10%)
Training Steps (x10%)
Training Steps (x10%)
Training Steps (x109)
antmaaz-large-divvrre
antrrare-lrg---diverree
antrarr-large-diverre
antrmaze-large-divrrs
Nomalized Return
Full cuta 108+82
Nomelized Return
duta USL
Nomalized Return
dala KS.
Nomalized Return
Full dana DOGE
Mbaing data TD3-DC
Miaiing data COL
Massing data ICL
Milsing data DOGE
1
0.5 0.8
6.0
02
0.4
0.
Training Steps (x10%)
Training Steps (x10%)
Training Steps (x10%)
Training Steps (x10%)
Figure 4:
Generalization performance after removing data from AntMaze large tasks (see Appendix F.1 for
detailed setup and additional results on AntMaze medium tasks).
4.3
ABLATION STUDY
We conduct ablation studies to evaluate the impact of the hyperparameter the non-parametric distance threshold Eq. (6), and the number of noise actions used train the distance function. For a, we add subtract 2.5 to the original value; for G. choose 30%, 50%, 70% and 90% upper quantile of the distance values in mini-batch samples; for N, we choose N = 10, 20,30. 30.
Compared to N and we find that G' has more significant impact the performance. Figure 5b shows that an overly restrictive G (30% quantile) results in a policy set too small to cover near- optimal policies. A more tolerant G, the other hand, is unlikely to cause excessive error accumulation and achieves relatively good performance. In addition, Figure 5a and Figure 5c show that performance is stable across variations of hyperparameters, indicating that method hyperparamette--roustt.
8

---

Published as a conference paper at ICLR 2023
hafcchetal-medluueeeeerrt/
halichetta-mmelunnexerrev
Nomalized Return
Nomalized Return
Nomalized Return
100
100
75
75-
75
30%
50-
50-
=.5.0
a=
50%
28
25
70%
25
00-00
10.0
90%
N-30 30
8.00
0.25
0.50
0.75
1.00
8.00
0.25
0.50
0.75
1.00
8.00
0.25
0.50
0.75
1.00
Training Steps (x10%)
Training Steps (x10%)
Training Steps (x10%)
(a) has little effect on results
(b) Small G is harmful to results
(c) NV has little effect on results
Figure 5: Ablation results. The default parameters in our implementation are marked by * The error hars indicate min and max over 5 seeds. See Appendix G for more detailed ablation studies.
5 RELATED WORK
To prevent distributional shift and exploitation error accumulation when inferring the value function at unseen samples, a direct approach is to restrict policy learning from deviating to OOD areas. To make sure the leaned policy stays inside the distribution support of training data, These policy constraint methods either carefully parameterize the learned policy (Fujimoto et al., 2019; 9; Matsushima et al., 2020), use explicit divergence penalties (Kumar et al., 2019; Wu et al., 2019; Fujimoto & Gu, 2021; Xu et al., 2021; Dadashi et al., 2021) or implicit divergence constraints (Peng et al., 2019; Nair et al., 2020; Xu al., 2022a). The theories behind these methods typically assume full state-action space coverage of the offline datasets(Lec et al., 2019; Kumar et al., 2019). However, policy constraint under full-coverage assumption is unrealistic in most real-world settings. especially on datasets with partial coverage and only sub-optimal behavior policies. Some recent works try to relax the full-coverage assumption to partial coverage by introducing different distribution divergence metrics, but only in theoretical analysis (Liu et al., 2020; Zanette et al., 2021; Xie et al., 202 Uehara & Sun, 2021; Xie et al., 20) Our method is an enhanced policy constraint method, where we relax the full-coverage assumption and allow the policy to learn on OOD areas where networks can generalize well.
Another type of offline RL method, value regularization (Kumar et al., 2020b; Kostrikov et al., 2021a; Yu et al., 2021; Xu et al., 2022b; 2023), directly penalizes the value function to produce low values at OOD actions. In-sample learning methods (Brandfonbrener et al., 2021; Kostrikov et al., 2021b), on the other hand, only learn the value function within data or treat it as the value function of the behavior policy. Compared with our approach, these methods exercise too much conservatism, which limits the generalization performance of deep neural networks OOD regions, largely weakening the ability of dynamic programming. There are also uncertainty-bast -basd and model-based methods that regularize the value function policy with epistemic uncertainty estimated from model value function (Janner et al., 2019; Yu et al., 2020; Uchara & Sun, 2021; Wu et al., 2021: Zhan et al., 2022; Bai al., 2021). However, the estimation of the epistemic uncertainty of DNN is still an under-explored area, with results highly dependent on evaluation methods and the structure of DNN.
6 CONCLUSION
In this study, we provide new insights on the relationship between approximation error of deep Q functions and geometry of offline datasets. Through empirical and theoretical analysis, we find that deep @ functions attain relatively low approximation error when interpolating rather than extrapolating the dataset. This phenomenon motivates us to design a new algorithm, DOGE, to empower policy learning on OOD samples within the convex hull of training data. DOGE is simple yet elegant, by plugging a dataset geometry-deriven distance constraint into TD3. With such a minimal surgery, DOGE outperforms existing model-free offline RL methods most D4RL tasks. We theoretically prove that DOGE enjoys tighter performance bound compared with existing policy constraint methods under the more realistic nurtial-coverage assumption. Empirical results and theoretical analysis suggest the necessity of re-thinking the conservatism principle in offline RL algorithm design, and points to sufficient exploitation of the generalization ability of deep Q functions.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
This work is supported by National Key Research and Development Program of China under Grant (2022YFB2502004), This work is also supported by Baidu Inc. through Apollo-AIR Joint Research Center. The authors would also like thank the anonymous reviewers for their feedback on the manuscripts, Jianxiong Li would like to thank Zhixu Du, Yimu Wang, Li Jiang, Haoyi Niu, Hao Zhao and colleagues AlR-Dream group for valuable discussions.
REFERENCES
Zeyuan Allen-Zhu, Yuanzhi Li, and Yingyu Liang. Learning and generalization in overparameterized neural networks, going beyond two layers. Advances in neural information processing systems, 32, 2019.
Gaon An, Seungyong Moon, Jang-Hyun Kim, and Hyun Oh Song. Uncertainty- offline reinforcement learning with diversified q-ensemble. Advances in neural information processing systems, 34:7436-7447, 2021.
Anonymous. Lightweight uncertainty for offline reinforcement learning via bayesian posterior. In Submitted to The Eleventh International Conference on Learning Representations, 2023. URL BWGJTv. under review.
https:/ /pwpeeevvieeeeeeeeeeeeeeeeeeeeee reveereviww iew. net/ teee/forruee forum?
Sanjeev Arora, Simon Du, Wei Hu, Zhiyuan Li, and Ruosong Wang. Fine-grained analysis of optimization and generalization for overparameterized two-layer neural networks. International Conference Machine Learning, pp. 322-332. PMLR, 2019a.
Sanjeev Arora, Simon S Du, Wei Hu, Zhiyuan Li, Russ Salakhutdinov, and Ruosong Wang. On exact computation with an infinitely wide neural net. Advances in Neural Information Processing Systems, 32, 2019b.
Chenjia Bai, Lingxiao Wang, Zhuoran Yang, Zhi-Hong Deng, Animesh Garg, Peng Liu, and Zhao- ran Wang. Pessimistic bootstrapping for uncertainty-driven offline reinforcement learning. In International Conference on Learning Representations, 2021.
Etienne Barnard and LFA Wessels. Extrapolation and interpolation in neural network classifiers. IEEE Control Systems Magazine, 12(5):50-53, 1992.
Peter L Bartlett and Shahar Mendelson. Rademacher and gaussian complexities Risk bounds and structural results. Journal of Machine Learning Research, 3(Nv)):463 482, 2002.
Alberto Bietti and Julien Mairal. On the inductive bias of neural tangent kernels. Advances in Neural Information Processing Systems, 32, 2019.
David Brandfonbrener, Will Whitney, Rajesh Ranganath, and Joan Bruna. Offline rl without off-policy evaluation. Advances in Neural Information Processing Systems, 34:4933-4946, 2021.
Qi Cai, Zhuoran Yang, Jason Lee, and Zhaoran Wang. Neural lemporal-differene learning converges to global optima. Advances in Neural Information Processing Systems, 32, 2019.
Robert Dadashi, Shideh Rezaeifar, Nino Vieillard, Léonard Hussenot, Olivier Pietquin, and Matthieu Geist. Offline reinforcement learning with pseudometric learning. In International Conference on Machine Learning, pp. 2307-2318. PMLR, 2021.
Jonas Degrave, Federico Felici, Jonas Buchli, Michael Neunert, Brendan Tracey, Francesco Carpanese, Timo Ewalds, Roland Hafner, Abbas Abdolmaleki, Diego de Las Casas, et al. Magnetic control of tokamak plasmas through deep reinforcement learning. Nature, 602(7897):414-419, 2022.
Jianqing Fan, Zhaoran Wang, Yuchen Xie, and Zhuoran Yang. A theoretical analysis of deep q-learning. In Learning for Dynamics and Control, pp. 489. PMLR, 2020.
Pete Florence, Corey Lynch, Andy Zeng, Oscar A Ramirez, Ayzaan Wahid, Laura Downs, Adrian Wong, Johnny Lee, Igor Mordatch, and Jonathan Tompson. Implicit behavioral cloning. In Conference Robot Learning, pp. 168-168. PMLR, 2022.
10

---

Published as a conference paper at ICLR 2023
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:200..07119, 2020.
Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning. Advances in Neural Information Processing Systems, 34, 2021.
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor- critic methods. In International conference on machine learning, pp. 1587-1596. PMLR, 2018.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International Conference on Machine Learning, pp. 2052-2062. PMLR, 2019.
Seyed Kamyar Seyed Ghasemipour. Shixiang Shane Gu, and Ofir Nachum. Why so pessimistic? estimating uncertainties for offline rl through ensembles, and why their independence matters.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning. pp. 1861-1870. PMLR, 2018.
Pamela J Haley and DONALD Soloway. Extrapolation limitations of multilayer feedforward neural networks, In /Proceedings / IJCNN International Joint Conference on Neural Networks, volume 4, pp. 25-30. IEEE, 1992.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. Advances in neural information processing systems, 31, 2018.
Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model-based policy optimization. Advances in Neural Information Processing Systems, 32, 2019.
Iya Kostrikov, Rob Fergus, Jonathan Tompson, and Ofir Nachum. Offline reinforcement learning with fisher divergence critic regularization. In International Conference Machine Learning, pp. 5774-5783. PMLR, 202
Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. In International Conference on Learning Representations, 2021b.
Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. Advances in Neural Information Processing Systems, 32, 2019.
Aviral Kumar, Rishabh Agarwal, Dibya Ghosh, and Sergey Levine. Implicit under- parametrrization inhibits data-efficient deep reinforcement learning. In International Conference Learning Representations, 2020a.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 33:1179... 1191, 2020b.
Hoang Le, Cameron Voloshin, and Yisong Yue. Batch policy learning under constraints. In Interna- tional Conference on Machine Learning, pp. 3703-3712 PMLR, 2019.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.016433 2020.
Boyi Liu, Qi Cai, Zhuoran Yang, and Zhaoran Wang. Neural trust region/proximal policy optimization attains globally optimal policy. Advances in neural information processing systems, 32, 2019.
Yao Liu, Adith Swaminathan, Alekh Agarwal, and Emma Brunskill. Provably good batch off- oof-policy reinforcement learning without great exploration. Advances in Neural Information Processing Systems, 33:1264-1277 2020.
11

---

Published as a conference paper at ICLR 2023
Tatsuya Matsushima, Hiroki Furuta, Yutaka Matsuo, Ofir Nachum, and Shixiang Gu. Deployment- efficient reinforcement learning via model-based offline optimization. In International Conference on Learning Representations, 2020.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski. et al. Human-level control through deep reinforcement learning. nature, 518(7540):529 2015.
2015.
Ashvin Nair, Murtaza Dalal, Abhishek Gupta, and Sergey Levine. Accelerating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006.09333 2020.
Charles Packer, Katelyn Gao. Jemnj Kos, Philipp Krähenbühl, Vladlen Koltun, and Dawn Song. Assessing generalization in deep reinforcement learning. arXiv preprint
2018.
Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weighted regression: Simple and scalable off-policy reinforcement learning. arXiv preprint arXiv: 190.00011 2019.
Dean A Pomerleau. Alvinn: An autonomous land vehicle in neural network. Advances in neural information processing systems, 1, 1988.
David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. nature, 550(7676):544359) 2017.
Masatoshi Uchara and Wen Sun. Pessimistic model-based offline reinforcement learning under partial coverage. In International Conference on Learning Representations, 2021.
Hado Van Hasselt, Yotam Doron, Florian Strub, Matteo Hessel, Nicolas Sonnerat, and Joseph Modayil. Deep reinforcement learning and the deadly triad. arXiv preprint arXiv: 2018.
Vladimir N Vapnik and A Ya Chervonenkis. On the uniform convergence of relative frequencies of events to their probabilities. In Measures of complexity, pp. 11-30. Springer, 2015.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017
Yifan WU, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. arXiv preprini arXiv: 1911.1111111666 2019.
Yue Wu, Shuangfei Zhai, Nitish Srivastava, Joshua M Susskind, Jian Zhang, Ruslan Salakhutdi- nov, and Hanlin Goh. Uncertainty weighted actor-critic for offline reinforcement learning. In International Conference on Machine Learning, pp. 11319- PMLR, 2021.
Chenjun Xiao, Bo Dai, Jincheng Mei, Osear A Ramirez, Ramki Gummadi, Chris Harris, and Dale Schuurmans. Understanding and leveraging overparameterizztion in recursive value estimation. In International Conference on Learning Representations, 2021.
Tengyang Xie, Ching-An Cheng, Nan Jiang, Paul Minciro, and Alekh Agarwal. Bellman-consisten consistent pessimism for offline reinforcement learning. Advances in neural information processing systems, 34, 2021a.
Tengyang Xie, Nan Jiang, Huan Wang. Caiming Xiong, and Yu Bai. Policy finetuning: Bridg- ing sample-efficient offline and online reinforcement learning. Advances in neural information processing systems, 34, 2021b.
Haoran Xu, Xianyuan Zhan, Jianxiong Li, and Honglei Yin. Offline reinforcement learning with soft behavior regularization. arXiv preprint arXiv:2 10.0230...0...... 2021.
Haoran Xu, Li Jiang, Jianxiong Li, and Xianyuan Zhan. A policy-guided imitation approach for offline reinforcement learning. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and in Neural Information Processing Systems, 2022a. URL https: // //openrevi ew net forum? ?
Kyunghyun Cho ttteee/ooooootoo/ooooooo tccssg
12

---

Published as a conference paper at ICLR 2023
Haoran Xu, Xianyuan Zhan, and Xiangyu Zhu. Constraints penalized q-learning for safe offline reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, 2022b.
Haoran Xu, Li Jiang, Jianxiong Li, Zhuoran Yang, Zhaoran Wang, Victor Wai Kin Chan, and Xianyuan Zhan. Sparse q-learning: Offline reinforcement learning with implicit value regularization. In International Conference on Learning Representations, 2023, URL https: /oppnrevieeeeeeeeeeee/ee/ee pet/forunm id=ueYYgo2sSUU.
Keyulu Xu, Mozhi Zhang, Jingling Li, Simon Shaolei Du, Ken-Ichi Kawarabayashi, and Stefanie Jegelka. How neural networks extrapolate: From feedforward to graph neural networks. In International Conference on Learning Representations, 2020.
Pan Xu and Quanquan Gu. A finite-time analysis of q-learning with neural network function approximation. In International Conference on Machine Learning, pp- 10555-10565 PMLR, 2020.
Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Y Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. Advances Neural Information Processing Systems, 33:14129-14411 2020.
Tianhe Yu, Aviral Kumar, Rafael Rafailov, Aravind Rajeswaran, Sergey Levine, and Chelsea Finn. Combo: Conservative offline model-based policy optimization. Advances in Neural Information Processing Systems, 34, 2021.
Andrea Zanette, Martin Wainwright, and Emma Brunskill. Provable benefits of actor-critic methods for offline reinforcement learning. Advances in neural information processing systems, 34, 2021.
Xianyuan Zhan, Haoran Xu, Yue Zhang, Xiangyu Zhu, Honglei Yin, and Yu Zheng. Deepthermal: Combustion optimization for thermal power generating units using offline reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, 2022.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM, 64(3):107-115, 2021.
13

---

Published as a conference paper at ICLR 2023
A SKETCH OF THEORETICAL ANALYSIS
In this section, we present in Figure 6 sketch of the overall logical flow in our theoretical analyses and the proposed algorithm, DOGE. We start by analyzing the effects of data geometry on the generalization patterns of deep Q-functions. We find that small sample-to-dataset distance leads to a tightened Q-function approximation error and thus interpolation enjoys better generalization properties than extrapolation (Theorem 1). Motivated by this, we propose DOGE, which tries to control the upper bound of the sample-to-centroid distance to be small (Property 1) and enforces convex hull based policy constraint (Property 2). Then, we dive deeper and find that the upper bound of the Bellman-consistedn coefficient is well controlled by sample-to-centroid distance and thus DOGE enjoys bounded bellman-consister coefficient (Theorem 2). Based on these findings, we can derive tighter performance bound of DOGE as compared to support constraint methods like BEAR (Theorem 3).
How data geometry impacts Deep generalzation?
Assumption (pppandi 8.2)
Lemma 1 Hapendix 9.3) pudwternd coges
Lemma 3,4
(Apporda 8.2) Dapp smoot
Definition
Detrannccm constrainet
Definition
3.1.1
Theorem 1
Theorem
prosspation
Stral distanos leads to tightened Selimon
Deep O functions interpolate will but estrapolate
Small semple-iocenntooo leads to bounded coefficient
Theorem
(Section 1.3. Appendio 0.2)
Tigttrr seffomence bound support cornirunt methodh BEG1.
DOGE (Sectio 32) R n)] stt [e-ae--l x
DOGE minimizes upoer bound sample co-centraid and geometric aware
DODE based policy constraint
Property 1 gf (Section 1.1, Aosenda
Property 2 of g (Section 3.1, Aquenda
Optimal solution
State-conditioned distance function
minleca-- -oneallll----oollllle- al] 26.8))11
Figure Sketch of theoretical analysis
B THEORETICAL ANALYSIS OF THE IMPACT OF DATA GEOMETRY ON DEEP Q FUNCTIONS
To analyze the generalization of function approximator, one can refer to some classical methods such as Rademacher complexity (Bartlett & Mendelson, 2002) and VC-dimension (Vapnik & Chervonenkis, 2015). However, the generalization bounds that obtained by these methods are usually trivial and cannot explain the generalization behavior in the overparameterized regime (Zhang al., 2021). Recent breakthroughs in neural tangent kernel (NTK) shed light on the generalization of DNN. NTK builds the connection between the training dynamics of DNN and the solution of the kernel regression w.r.t. NTK, and is widely used recent analysis of DNN generalization (Jacot et al., 2018; Arora al., 2019b; Bietti Mairal, 2019). What's 's more, NTK is also a popular analyzing tool in the convergence and optimality of deep RL (Cai et al., 2019; Fan et al., 2020; Kumar et al., 2020a; Xiao al., 2021) and thus is used in our study.
B.1 NEURAL TANGENT KERNEL
We denote a general neural network by f(0,z) Rd + R, where 0 is all the parameters in the network and x Rd is the input. Given, training dataset {(z,, ) the parameters are optimized by minimizing the squared loss function, i.e., L(0) E (fo(v)) y:)² by gradient
14

---

Published as a conference paper at ICLR 2023
descent. The dynamies of the networks output be formulated by Lemma 1 (Lemma 3.1. of (Arora et al., 2019b)); see (Arora et al., 2019b) for the proof of Lemma 1.
Lemma 1. Consider minimizing the squared loss (()) by gradient descent with infinitesimally ax)(tt R be the network ouipuls on all xj 's al time i, and Y = (Ni)venn) be the desired outputs. Then u(l) follows the following evolution, where H(t is an x n positive semidefinite matrix whose (i. entry is
small learning rate, i,e., dB(t) - -VL((((t) Let Leru(()
88
09
du(t)
= H(t) (u(t) Y).
(12)
dt
Plenty of works (Jacot et al., 2018; Arora al., 2019b; Allen-Zhu al., 2019; Xu al., 2020) study the dynamics of the neural networks' training process and find that if the width of networks is
sufficiently large, (8) stays almost constant during training. í.e., (t) H(0). What's 's more, the neural networks' parameters are randomly initialized with certain scales and the networks width goes to infinity, (() converges to a fixed matrix K, called neural tangent kernel (NTK) (Jacot et al., 2018).
< (0f(0().))) (f((()),²))
K(x, x') Eg-w
(13)
08
00
where, W is Gaussian distribution. The training dynamics in Lemma 1 is identical to the dynamics of kernel regression under gradient flow, because K stays constant during training when the width of neural networks goes to infinity. Then, the final prediction function (f - oc, assuming u(0) 0) is equal to the kernel regression solution:
Fuck (x) = (K( K(x, xx)) K
(14)
train
where K is the X NTK for the training data (the state-action pair policy evaluation in offline RL) and stays constant during training once the training data fixed. (oos)))(oosssss
(s,a) in the
Y is the training labels (r(s, a) "E (Qo (5', a')] in offline RL). K(x, is the kernel value between test data x and training data 24. We denote the feature map of K(.,) as Φ(.), and K(#,2) = (I()), o (z²)). Then, Eq. (14) is equivalent to:
Snen (x) = ((D(x), ((x))) ...,, (2(x), ((xn)) Kirasn
(15)
B.2 IMPACT OF DATA GEOMETRY DEEP Q FUNCTIONS
In this section, we analyze the impact of data geometry deep Q functions under the NTK regime. We first introduce the smoothness property of the feature map ((x) induced by NTK (Lemma 2). Then, we introduce the equivalence between the kernel regression solution in Eq. (15) and min-norm solution (Lemma 3). Builds on Lemma and Lemma 3, Lemma 4 analyzes the smoothness of the deep @ functions. At last, we study how data geometry affects deep functions (Theorem 1).
Assumption (NTK assumption). We assume the function approximators discussed in our pa- per are two-layer fully-connected ReLU neural networks with infinity width and are trained with infinitesimally small leurning rate unless otherwise specified.
Although there exist some gaps between the NTK assumption and the real setting, NTK one of the most advanced theoretical machinery from the generalization analysis of DNN. In addition, Assumption is common in previous analysis on the generalization DNN (Jacot et al., 2018; Arora et al., 2019a; Bietti & Mairal, 2019) and the convergence of DRL (Cai et al., 2019; Liu al., 2019; Xu & Gu, 2020; Fan et al., 2020). For more accurate analysis, we should adopt more advanced analysis tools than NTK and hence leave it for future work.
We first introduce Lemma 2 (Proposition 4 of (Bietti & Mairal, 2019)), which shows the feature map (2:) induced by NTK is not Lipschitz continuous but holds weaker Hölder smoothness property.
15

---

Published as a conference paper at ICLR 2023
Lemma 2. (Smoothness of the kernel map of two-layer ReLU networks). Let & be the kernel map of the neural tangent kernel induced by a two-layer ReLU neural network, x and be two inputs. then & satisfies the following smoothness property.
[ +())l min[[]=]]
|>l) yl + 2/ yll-
(16)
Lemma (Lemma of (Xu et al., 2020)) builds the connection between the kernel regression solution in Eq. (14) and the min-norm solution. For the proof of Lemma 3, we refer the reader to(Xu et al., 2020).
Lemma 3. (Equivalence to a min-norm optimization problem). Let ((x) the feature map induced by neural tangent kernel, for any E Rd. The solution the kernel regression in Eq. (14) and Eq. (15) is equivalent to (n( = (x))² Brak. where Brok the optimal solution of a min-norm optimization problem defined as
min||33||
(17)
s.t. ((x.)² = yi. for = 1,..,,
Then, deep Q functions satisfy the following smoothness property.
Lemma 4. (Smoothness for deep Q functions)). Given two inputs x and x', the distance between these two data points is = || -x-'ppp x'|1. C1 sup 11 is a finite constant. Then the difference between the output at x and the output at can be bounded by:
[[Q((z)-Qo(d)) - Qo(x')) Ci(y (it/min[[111 | I=] 2d)
(18)
Proof. In offline RL, we denote a general Q network by Qo(z) : RISH+A + R, where 0 is all the parameters the network and x = (s, a) E RIS|+AA the brief notation for state-action pair (s, a). The Q function is trained via minimizing the temporal difference >E=1(00 (Q,( y 2 by gradient descent, where y; r(x;) + + ot) 1Q (x()) E R is the
defined as
target value.
Using kemel method from NTK, Q function can be formulated as Qo(x) = (xx) B, where ((x) is independent of the changes on training labels when NTK assumption holds. This is because as the width of neural net goes to infinity, the NTK kernel K(x, x²) =< (xx), (xx') ) > produced by this
network stays constant during training, and so is the property of the feature map (xx) (Jacot et al., 2018). So, the learning process under NTK framework is actually adjusting 3 to fit the label rather than (xx). As a result, Lemma holds when deep function satisfies NTK assumptions. Given two inputs and n', the distance between these two inputs is d = || a/'l. Based on Lemma 2, it is easy to see that
[1((()) - Qo(a')ll [| (<x')" I1
< ())(()) (zx')
(Infinity norm)
S |1 (/min([la [a'l]) |r x11
+ 2||x - z'l) (Lemma2)
(19)
= (1
|
d 2d)
< Cs(v min[[[
| d
2d)
(C : sup [|3])]))
Additionally, if we consider the delayed target and delayed actor updates during policy learning, we can assume the target value used for @ evaluation stays relatively stable during each policy evaluation step and the problem can be seen solving series of regression problems. Under this mild assumption, we can learn the actual Br, at each step (3 + Brtx and so C₁ - C1 where G sup Brukk )loo) and thus complete the proof. Similar assumptions and treatments also used
16

---

Published as a conference paper at ICLR 2023
in Section 4 of (Kumar et al., 2020a) that Q function at each iteration fit its label well, Appendix A.8 of (Xiao et al., 2021), as well as Appendix F of (Ghasemipour et al.).
Lemma 4 states the value difference of a deep Q function for two inputs is related to the distance between these two inputs. The closer the distance, the smaller the value difference.
B.2.1 PROOF OF THEOREM
Builds on Lemma 4, can combine the data geometry and analyze the impact of data geometry deep Q functions.
Proof. We first review the definition of interpolated data and extrapolated data. Under continuous state-action space, state-action pairs within the convex hull of the dataset can be represented in an interpolated manner (referred as interpolated data Tin):
Zin
CAT
Σ =a== - L, 20
(20)
Similarly, we can define extrapolated data that lie outside the convex hull of the dataset as Fout
Tout
Belrs
(21)
where Dim1 3: = 1 and B, 2 0 does not hold simultaneously.
We define (r) :- arg mitt,, ED || -x: as a projector that projects unseen data x its nearest data in dataset DD Given an interpolated data Fin and an extrapolated data Tout, the distances to their nearest data in dataset dz,,, a| Projp (1 and d = | Projp (Io )| Because interpolated data lie inside the convex hull of training data, de,, < max,,eD I z4 || S B is bounded, where value difference of deep Q function for interpolated and extrapolated data can be formulated as the following shows.
max,,.;
2,, ED ||z - -zyl a finite constant. Then. by applying Lemma 4, the
|Q0 (2in) (PPO))) C1 min( Fin
Projj dr.. + 2dz.n)
(22)
$ C1(v [[]/mmnn]] Zin
( Il) 2B)
[[Qo(Tout) (ZPPo)) II) > (CC.Ciii(((// Foud
Projj )l) deout 2d
(23)
B.3 QUANTITATIVE EXPERIMENTS ON THEOREM
In addition to the one-dimensional random walk experiments presented in Section 2.2, we conduct additional experiments on the more complex and high-dimensional MuJoCo tasks (including D4RL Hopper-medium--2, Halfcheetah-meciim--22 and Walker2d-medium--22) to provide quantitative support Theorem in particular, the pertinence of interpolation and extrapolation. We first synthesize lots of interpolated data Tin and extrapolated data Tous (x = (s, a) S x .A) and then search for their nearest data points in offline dataset D accordingly, i.e., P and Proj (Zout). Then, we can evaluate the Q-value di differences [1Qx(z) Qo (Projp (x)) (LHS of Theorem 1) at these generated data and see whether the Q- value differences align well with Theorem
For the detailed experiment setup, recall that an interpolated data point Zin is convex combination of the offline dataset, i.c., Zin >im C,T,, V4 ~ D with weights a; that satisfy Σn a; = 1, 2 0. Therefore, can interpolate the offline dataset based on a; sampled from the Dirichlet distribution to generate the interpolated data. Also, an extrapolated data point Tout is expressed as a weighted sum of the offline dataset, i.c., Zout Ei= 8, I; ~ D, but its weights 8 do not satisfy the non nogaaniatyv and the summing to constraint. Therefore, we can generate extrapolated data by setting the sign of some weights negative values and varying the weights summing to
17

---

Published as a conference paper at ICLR 2023
1. After obtaining the interpolated and extrapolated data, search for their closest data points in the offline dataset D and calculate their corresponding distance || - (roip(z)] and Q-value difference| Qo(x)= - Qo (P))jo Figure 7a shows the relationship between the distance to dataset || - Projp (a) and the Q value difference [Qx(z) Qo(Projp |I (L.SS of Theorem 1). We also report the learned state-conditioned distance value g(s, a) on these generated data in Figure 7b.
Hepper-medlum-22
tellkheetal-mediunm22
Walker 2:mmm
100000 Oxffregbelll
500
25 40 []0vbet -OUProjeiel01
600-
200
-00
802
200
100
23
39
00 Distance to Dataset [jx Projuckill
10
15
20
20
10
35 28
Distance [|x-
Proj
Distance Dataset [|x Projat
Extrapolation
Interpolation
Empirical Upper bound of 10:00000 Co(Proiz())]I
(a) Relationship hetween
| proje and [[Qo(z) Qo(Projo(z))])
Hopper-mediumuv
Hellihhetta-meddimmz?
Waller266mediumm22
ats al Value
01% Valuee
StS.
18 15
25
25
30
20
25
20 Projolxi]]
25
10
15
20
30
Distance Dataset [Xx Projz
Distance to Dataset ||x
Distance to Dataset lIx ProjekX0|
Extrapolation
Interpolation
(b) Relationship between ||
Projo (x) and g(x)
Figure 7: Quantitative experiments of Theorem on the D4RL MuJoCo-medium datasets. The red star-shaped dots are the interpolated data and the circle dots are the extrapolated data. The color of the dots represents [[Q)(z) - Qo (Projn(z)) (z))] values in (a) and g(x) values in (b), respectively. The darker the color, the smaller the corresponding value. In (a), the yellow dash line is the empirical upper bound of [x)) - Qo (PProjp
Figure 7a demonstrates that the interpolated data enjoy tighter empirical upper bound of [[Qo(z) (Projj (z)) (LHS of Theorem 1) than most of the extrapolated data. Morcover, the empirical upper bound of the Q-value difference grows with the increase of the sample-to-datase distance ||= - ProjD(z)]l ll. which deep Q function is well controlled by distance to the dataset). Figure 7b shows that the state- conditioned distance function g(s, a) can output low values for interpolated data and some near- dataset extrapolated data, and thus be used relaxed policy constraint in these OOD regions.
consistent with Theorem (the upper bound of value difference of
C STATE-CONDITIONEL DISTANCE FUNCTION
C.I PROOF OF PROPERTY
Proof. Given state-action pair from the training data (s, ~ D, we synthetic random noise actions from a uniform distribution over the action space, i.e. à ~ (A). Then the distance function g(-) trained by Eq. (24).
min E(a,))D (anunirAA) [lla all -((s, a)]?
(24)
9
18

---

to Published as a conference paper at ICLR 2023
||| - all ((s, à) can be upper bounded by some finite constants because S A is compact in analysis. The optimization problem in Eq. (24) can be reformulated as the following form according to the Fubini's Theorem.
min (n-Uni((A) [(s.aa>D llla all - g(s, a)s
(25)
Note that the objective of Eq. (25) be also written functional [[g(s, a)] with respect function in following form:
[[g(s, = /A [ E [(sa))Doll all g(s, a(s,a)l dā g(s.a.
(26)
Based on calculus of variation, the extrema (maxima or minima) of functional J [[ a)] can be obtained by solving the associated Fuler-Langrane equation (0F/3g- 0). In our case, it requires the optimal state-conditioned distance function 9 satisfies the following conditions:
i 0g*
E ((aaa) [l|à all g"(
(27)
E(
0Og [|| all - g "s, a))
(DNN is continuous)
E (s,aaD [lla - all g" a)]l -
Conditioned state E D, the optimal state-conditioned distance function in Eq. (27) satisfies the following conditions:
/ ||a allu(s, / (ss
(s,
0" à) LA llā allu(s,
Sau(s,a)da 8 EED
(28)
(8, à) / C(s,a)lá a allda, & E
where, uuss a) is the empirical distribution finite offline dataset D {(r:))= i.e., the sum of the Dirae measures >E= 82,- (ss a) D, 0.v(s,a) E (s,(s, > 0. C(s,a) = Ja p(s,a)de 2 and ʃA C(ss = Because L2- Lg-norm is convex and the non-negative combination of convex functions still convex, g" (s, à) a convex function w.Z. à. In addition. vá E A. by the Jensen inequality, we have:
m(8.0)
gs à) l| Banunista) a)all = ||a (s)ll,
(29)
where (s) Fannsny D the state-conditioned centroid of training dataset.
C.2 PROOF OF PROPERTY
Proof. The negative gradient the optimal state-conditioned distance function can be formulated as:
Vag (s, = C(s,a) la al da, Và A, D
(30)
(a a) l all
p(s, a)
da, Vā E A
JAM(s, a)da
Observe that the direction the negative gradient of (s, à) is related to the integral of vector (ā a) (points towards a). When (s, a) & DD -(a a) doesn't influence the final gradient because
19

---

Published as a conference paper at ICLR 2023
u(s, a) = 0. Therefore, -(a a) only contribute to the final gradient of g" â) for (s. a) D as pls, a) > 0. For a given & E and any extrapolated action that lies outside the convex hull of training data, the integral of vector (ā a) is basically non-negative combination of vectors -(a - a) that point toward actions a E inside the convex hull. As a result, it's easy to see that Vag" a) also points inside the convex hull formed by the data.
D THEORETICAL ANALYSIS OF DOGE
In this section, we analyze the performance of the policy learned by DOGE. We first adopt the Bellman--onssitent coefficient from (Xie et al., 2021a) to quantify the distributional shift from the perspective of deep Q functions generalization. Then, we gives the upper bound of the Bellman- consistent coefficient under the NTK regime (Appendix D.1). At last, we give the performance bound of DOGE (Appendix D.2).
D.1 UPPER BOUND OF DELLMAN-CONSISTEN COEFFICIENT
Let us first review the definition of Bellman-consistere coefficient (1, x) in (Xie et al., 2021a). We define (uU ,, F, to measure the distributional shift from an arbitrary distribution v to data distribution μt, w.E.t IF and æ. JF is the function class of Q networks.
|Q - T 1
Bl0,
sup QEF IQ T=Q12,
(31)
where the j--eiggtted norm (square) is defined as || Elll which is also applica- ble for any distribution T is the Bellman operator of policy defined as T* a) (e)))))))))))))) [Q('', a')] := r(s, a) + >PP (Q(s(,('' a')l. PT | the brief notation
r(s, a) + for (t [-]. The smaller the ratio of the Bellman error under v and μa, the more v(s,a) when (s.(). p(s.a) cc. Then give the proof of
transferable the function from μ to Theorem (Upper bound of Bellman-consister coefficient).
Proof. We denote x (s, a) and x' = (8', J. = Ea-D (r) is the centroid of offline dataset. di = || Toll and d2 = I|x' xoll are the sample-to-centrodd distances. Let μ(z) be the distribution under the offline dataset and vxx) be any distribution. Then, for the numerator in Eq. (8) and Eq. (31), we have the following incqualities.
(())][((>) -00---((2---0000000
AXA
(=)))(()) (10
SXA
< SXA
v(x) [IQ(()) - ((-->))))) (Triangle)
1(1-
JSXA v(x) - |00 (x=
-(=))) ||0---0000||00000000000000000000 Q({=))) (Triangle)
8xA
f v(z)
(==))) +r())| [Q(x) - Q(..)) + PP- Q(+
(Jensen)
SxA
T2
Ta
(32)
The RHS contains three parts: I (1 ?x,]Q(=>))1 + [|( In = 11Q(x)) Q(x and
I = (P Q(x.))l]. Because ||r( || E [0, Rmax) Vor S x A. In can be upper
20

---

Published as a conference paper at ICLR 2023
bounded as:
I (L-)))((2 Rmax
(33)
By applying Lemma 4, In is upper bounded as
I2 C. Vmin([|<]] [ + 2d1]
(34)
Z3 is upper bounded as In addition, we denote C2 /supzESXA |,/|| Then, I2 and I3 can be further upper bounded by
Is C1(- [/min(1>* ||'0|| [ + 2dg
(35)
I2 51 (Cavoli 2d1)
(36)
Ia S (2-)) d2 2d2)
(37)
The above relaxation of the upper bound in Eq. (36) and Eq. (37) is not necessary, but for notation brevity, we choose relax the upper bound by treating C2 v suspassxA ||=/]. Plug Eq. (33), Eq. (36) and Eq. (37) into the RHS of Eq. (32), we can get
1Q - T=QQ
$ / u(z) (1 (1=)))(1x) + Rmaa C1(C2v d1 2d1) + (2 (2 +2d2)
SXA
(11 -)) ao) Rmax + + (C2v + 2dh (2 73 (CoV 2dz 11:
(38)
For the denominator | in Eq. (8) and Eq. (31), because the Q function is approximated. there exists approximation error between Q and T i.Q. Q-QT=== c. addition, the distribution contains some mismatch w.r.f. the equilibrium distribution induced by policy x. Therefore, it is
reasonable to assume I T TT*ll 1 2 cp
Then, we can complete the proof by plugging the upper bound in Eq. (38) and - 1Q-T=Q11, N cu into Eq. (8) or Eq. (31).
Bl.,u,,,F
Qss, an) Rmax + Cc (Cavdi + 2d)
-))- (Cavdza + 2dz
Ep.
8g
(39)
To be mentioned, the distance regularization DOGE compels the leaned policy to output the action that near the state-conditioned centroid of dataset and thus B, and B3 can be driven to some small values. B, independent on the distributional shift. Therefore, B(u F. πτ) can be bounded by some finite constants under DOGE. Therefore, the constrained policy set induced by DOGE essentially a Bellman-consisten constrained policy set HI defined Definition 2. In addition, other policy constraint methods such BEAR (Kumar al., 2019) can also have bounded B. However, these policy constraint methods do not allow the learned policy shifts to those generalizable v(s.o)
co, which is essentially different distributions where B(v, F, F, is small but sup pls.a] with DOGE.
21

---

Published as a conference paper at ICLR 2023
D.2 PERFORMANCE OF THE POLICY LEARNED BY DOGE
Here, we briefly review the definition of the Bellman-consisten constrained policy set 1s defined in Definition 2. The Bellman-consistent coefficient under the transition induced by Πs be bounded by some finite constants ((k):
(px: μ1, F, S ((k)
(40)
where, Po is the initial state-action distribution and μ is the distribution of training data. Pk Po p .H,,, VIT T2; ... TTk E Hg and p is the transition operator on states induced by ns. Le., pd (s', a als,a) = P(s'|s,a)
We denote the constrained Bellman operator induced by Ig as T and T a) := r(s, a) maxfells yPp [Q(s', (Q)sss T can be seen as a operator in redefined MDP and hence is contraction mapping and exists fixed point. We denote Q as the fixed point of T iee. Qux = Tul Qule.
The Bellman optimal operator is
TQ(s, a) r(s, a) maxx>P"(((s") d')l
(41)
T is also contraction mapping. Its fixed point is the optimal value function Q and Q = TQ"
Then, by the triangle inequality, we have:
|| - 1="1"ll> 1l 1|Q -QUIs + QuI
QTT ll ||Qus
1Q Qul 1l
QTn ll
(42)
T11
1.2
where QT is the true value of policy Tu- nn is the greedy policy w.r.t to Qn in the Bellman- consistent constrained policy set lsg, i.e., n. = supreng Bewa(-le) [Qn(s, a)]. Qn is the Q function after n-th value iteration under the constrained Bellman operator "1 For L1 part in Eq. (42), first focus on the infinity norm.
11Q" - Q 1o00 I|TQ" T QI
IITQ" T qu t1 1lo0 II T
(43)
S IITQ' T + + - 11 (T -
a(IIs) + + - 1x
where (ITs) = ITQ -1 Qux || is the suboptimality constant. Then, we get ||Q" - -QIl S a(Ils)
a(lls) and L1 I| TT 1| < 1-7
I-y
For L2, we introduce Lemma 5, which upper bounds I!Q - -= 11, The proof of Lemma can
be get by directly replacing Q" with QiIls in the Appendix F.3. in (Le et al., 2019), because Qlls is the optimal value function under the modified MDP induced by Tll..
Lemma S. (Upper bound of error propagation). |[Q
Q=
12 can be upper bounded as
2yx11 12x11 (
113,00 <
po(ds, da)
Σ
AnA + AnAn(Que Q0)² (8.
(1 71-
(44)
where
6 = Qk+1 T
(45)
(1 n) h-1
03
for l: <
gat1
(46)
(1 - 7)7"
g0+1
22

---

Published as a conference paper at ICLR 2023
A
1-7
> (P
(poale yo pan pun-t .uett for k < n
2
m20
(47)
7
? th (P (1o7leyate yatl pau pount 15
Ag
Qo is the Q function after initialization. Note that lim (a, An (Q Q6) this term for analysis simplicity. In addition, each A₃ is probability kernel that combine pa and pri (the transition operator on states induced by the constrained optimal policy -l E Lls) and Ex ag = 1.
leave
The key part Eq. (44) SSxA A and we expand this term the following shows.
PoA =
1-
(paa k pda pan-i ...pttt) & Σ y (Ppan)m
Po
SXA
2
m20
1 1-7 2
mm
(p (pan ym p pan-1 ...pitti puck
Sxd
(48)
As Eq. (40) shows, the policy set induced by DOGE is a Bellman-ccnsistent constrained policy set Hy defined in Definition 2. Therefore, let Po be the initial state-action distribution and jr denote the distribution of training data. For any policy T1, .2,... F E 1lg, the distribution after k-th Bellman-coonsstent iteration Pk PT p* there exits some finite constants ((k), that B(pk, M, F, x) & ((k) holds. Then we can get the following inequalities.
...
IQQ TT
$ TQTTT ((k)
PK
SSuA pke S 1
(49)
(le((k) (c=Q-T-Q)
SSA
As result, by applying the result of Eq. (49) to Eq. (48), we can get
[ SXA A
(1-))
1 (1-?)
Σ r" jm --)
(50)
SSX
m²0
Plugs Eq. (50) into Eq. (44) and leaves out [anAn(Q Q0) in Eq. (44), we get
27 1
lim L3 < lim
Σ
(m + - k)aallll
(1 7)-
m20
2?(1 - 7
lim 2+000
7)
m-n-k-11 1(m + +n --)llc+11
(1 722
n++
k=0
m20
2y n
lim
L(I )2
sup | 1
++0
(1 11-7) 7)2
k00
2y
12
I([lg)) k20
sup 1|
(1 ?
(51)
where, L(1BB) = ((1 (1--)22 En=1 ((k). Then, we can bound L2 by
27 L(Ilg) sup |
lim L2
(52)
++000
(1 71
k20
With the upper bound of L, and lim L2, we can complete the proof by adding these two term together.
23

---

Published as a conference paper at ICLR 2023
27
1 1 (1lB)
lim
S (1 7)²
((Ilg) sup ||axlla + k20
(53)
27
E IMPLEMENTATION DETAILS
DOGE can build on top of standard online actor-critic algorithms such as ID3(Fujmmoto et al., 2018) and SAC(Haarnoja et al., 2018). We choose TD3 as our base because its simplicity compared to other methods. We build DOGE on top of TD3 by simply plugging the state-conditions distance function as a policy regularization term during policy training process. Then, the learning objective of policy in Eq. (7) can be formulated
i = arg max min E (QQ(s.((s)) ((g(s, G)] s.t. X 0
(54)
The Q function, policy and condiiioid distance function networks are represented by 3 layers ReLU activated MLPs with 256 units for each hidden layer and are optimized by Adam optimizer. In addition, we normalize each dimension of state to standard normal distribution for Mujoco tasks. The hyperparameters of DOGE are listed Table
Table 2: Hyperparameters of DOGE
Hyperparameters
Value
Optimizer
Adam
Standard Normalize state
True for Mujaco
False for AntMaze 256
Shared parameters
Batch size
Layers Hidden dim
3
256
Actor learning rate Critic learning rate
3 10 10
3 10 for Mujoco
1 1 10-3 for AntMaze
Discount factor ?
0.99 Mujoco
TD3
0.995 for AntMaze
Number of iterations
10"
Target update rate
0.005
Policy noise
0.2
Policy noise clipping
0.5
Policy update frequency
2
Learning rate
1 10 3 for Mujoco
1 x 10 for AntMaze
State-Conditioned Distance Function
Number of noise actions N
20
Number of iterations Ng
10 for Mujoco
10 for AntMaze
[7.5, 17.5) Mujoco
[5. 10, 70) AntMaze clipped [1, 100] 3e
DOGE
Lagrangian multiplicr learning rate
E. TD3's IMPLEMENTATION DETAILS
For the choice of the Critic learning rate and discount factor Y, we find that for AntMaze tasks, high Critic learning rate can improve the stability value function during training process. This may be because the AntMaze tasks require the value function dynamic programs more times to "stitch" suboptimal trajectories than Mujoco tasks. Therefore, we choose 1 10 and 0.995 as the Critic learning rate and discount factor for AntMaze tasks, respectively. The other implementations such as policy noise scale and policy noise clipping are the same with author's 's implementation (Fujimoto al., 2018).
24

---

Published as a conference paper at ICLR 2023
E.2 STATE- CONDITOONDD DISTANCE FUNCTION'S IMPLEMENTATION DETAILS
We sample N = 20 noise actions from a uniform distribution that covers the full action space to approximate the estimation value in Eq. (4). We find N 20 can balance the computation complexity and estimation accuracy and is the same sample numbers with CQL (Kumar et al., 2020b). The ablation of N can be found in Fig. 15. The practical training objective of the state-conditioned distance function is as follows:
1 N
min
|||a àrll 1-((,,,)))) a.))
(55)
We find that wider sample range than the max action space [-amax+ 4max] is helpful to characterize the geometry of the full offline dataset. This because some actions in the offline dataset lie at the boundary of the action space, which a narrow distribution. At this time, the noise actions may not cover the geometry information near the boundary. Therefore, than the max action space, i.e., ~ Vrrifl-aama high learning rate enables stable training process in Mujoco tasks. Therefore, we choose 1 x 10-3 and 1 x 10 as the distance function learning rate for Mujoco and AntMaze, respectively. We also observe that for Mujoco tasks, 10 iterations can already produce a relatively good state-conditioned distance function, and training more times won't 't hurt the final results. To reduce computation, we only train the state-conditioned distance function for 105 steps for Mujoco tasks.
only be sampled with little probability when sampling from
sample noise actions from uniform distribution that is times wider
30max]- For the learning rate, we find that a
E.3 HYPERPARAMETERS TUNING OF DOGE
The scale of determines the strength of policy constraint. We tune balance the trade-off between policy constraint and policy improvement. To be mentioned, a is tuned within only candidates for 20 tasks (17.5 for hopper-m, hopper-m-r and all Mujoco random datasets; 7.5 for other Mujoco datasets; 5 for antmaze-u; 10 for antmaze-u-d; 70 for other AntMaze tasks). This is acceptable in offline policy tuning following (Kumar et al., 2019; Brandfonbrener et al., 2021). To ensure numerical stability, we clip the Lagrangian multiplier À to [1, 100). We also find large initial enables stable training for Mujoco tasks but slows down AntMaze training. Therefore, the initial value of Lagrangian multiplier is 5 for Mujoco and for AntMaze tasks, respectively.
E.4 PSEUDOCODE OF DOGE
The pseudocode DOGE is listed in Algorithm 1. Changes we make based on TD3 (Fujimoto et al., 2018) 8) are marked in red. The only modification is the training process of the additional state-conditioned distance function and the constrained actor update. We can perform 1M training steps on one GTX 3080Ti GPU in less than 50min for Mujoco tasks and Ih 40min for AntMaze tasks.
Algorithm Our implementation for DOGE
Require: Dataset D. State-conditioned distance network Jg. Policy network Te and target policy network T& with o' +- Φ. Value network Qa, i 1, 2 and target value network Qo, i 1,2 with 8/ 0,. conte-condiiioned distance network training steps Ng- Policy update frequency m.
1: for = 0, 1,.. M do
2:
Sample mini-batch transitions ((si, 04 Fi, x() D
3: if t. < N2 then
4:
Maate-ooditiooeed Distance Function Update: Update Eq. (55) shows.
5: end if
6: Critic Update: Update 9g using policy evaluation method TD3.
7: if t mod m = 0 then
8: Constrained Actor Update: Update é via Eq. (54). Update target networks: 0g r0, + (1 7)0% Ø' + T + (1 7)
9:
end end for
11:
25

---

Published as a conference paper at ICLR 2023
E.5 EXPERIMENT SETUP FOR THE IMPACT OF DATA GEOMETRY ON DEEP Q FUNCTIONS
We consider an one-dimensional random walk task with a fixed-horizon (50 steps for each episode), where agents each step can move in the range of [-1, +1] and the state space is a straight ranges from 1-10, 10). The destination is located at = 10. The closer the distance to the destination, the
larger the reward that the agent can get. The discount factor 7 0.9. The reward function is defined as follows:
400 - (s' 1 10)
(56)
400
We generate offline datasets with different geometry and train the agent based on these datasets. Each synthetic dataset consists of 200 transition steps. We get the approximated Q value Q by training TD3 for 1e + 4 steps each dataset. The learning rate Actor and Critic networks are both 10-3 The other implementation details are the same the implementation of original TD3 (Fujimoto et al., 2018). The true Q function can be get by Monte-Carlo estimation. We find that the near-destination states hold higher approximation enror than that far away from the destination due to the scale of true Q value near the destination large. To alleviate the impact of Q value scale on the approximation error, we define the relative approximation error follows:
((s, a) = e(s, a) min ((s,)
(57)
where, a( (s, - Q(s, The relative error in the above definition eliminates the effect of different states the approximation error and can capture the over-estimation error that care about. We plot the relative approximation error of deep Q functions with different random seeds and data geometry in Fig. 13.
F ADDITIONAL EXPERIMENT RESULTS
F.1 COMPARISON OF GENERALIZATION ABILITY
In the well known AntMaze task in D4RL benchmark (Fu et al., 2020), where an ant needs to navigate from the start the destination in large maze. The trajectories with coordinates at x x y & [4, 13) x [7, .99 [11.5 20.5) x [11, 13] in AntMaze medium tasks and x x E [10.5, 21] x [7,9] U [19, 29.5] [15, 17] in AntMaze large tasks clipped, Fig. 8 shows.
HB ?
(a) Modified Medium AntMaze
(b) Modified Large AntMaze
Figure 8: The trajectories the offline dataset are visualized blue. Data transitions of two small areas on the crtical pathways to the destination have been removed (red box).
These clipped data counts only about one-tenth of the original dataset and lies in the close proximity of the original trajectories. Under these modified datasets, simply relaying "stitching" data transitions is not enough to solve the navigation problems. We evaluate representative policy constraint method (TD3+BC (Fujimoto & Gu, 2021)), value regularization method (CQL (Kumar al., 2020b)), in- sample learning method (IQL (Kostrikov et al., 2021b)) and DOGE (our method) on these modified datasets. The evaluation results before and after clipping the trajectories listed in Table 3. The
26

---

Published as a conference paper at ICLR 2023
learning curves for the modified AntMaze medium and AntMaze large tasks are listed in Fig. 9 and Fig.
Observe in Table 3 that existing offline RL methods fail miserably and suffer from severe performance drops. By contrast, DOGE maintains competitive performance after the modification the dataset and shows good generalization ability unknown areas.
Apart from above experiments, we also evaluate DOGE when removing only one area: [10.5, 21] x [7. 9). [10.5, 21] [7. 9] for AntMaze-late large datasets and [4, 13) [7. 9). [4, 13] [7,9) 9] for AntMaze- medium datasets. The final results can be seen in Table 4.
Table 3: The performance drop after removing the data at the only way destination.
Datatset type
TD3+BC CQL
IQL
DOGE()urs)
full data miss data
65.2:14.8 10.7
70.41...3
8..6.1..5
antmaze-m-p-v2
10.2
33.2
Performance drop 4
84%
86%
59%
full data
54.0+:1..7
74.6:13.2
77.6 6.1
antmaze-m-d-v2
miss data
8.5
7.6.5..7
40.2+32.9
Performance drop |
84%
90%
48%
full data
18.8++15.3
43.5
48.2 8.1
antmaze-1-p-v2
miss data
0
1.0+0..7
22.41:5.9
Performance drop |
100%
98%
54%
full data
31.649..
45.6.17.6
36.4119.1
antmaze-l-d-v2
miss data
0
5.243.1
44.6.111.1
Performance drop
100%
89%
60%
Policy constraint
Value regularization
In-sample learning
DOGE (Ours)
antmaaee medium medium-play.v2
antmaze medium..play.vo.
antmaze- notmaze-meedum play.v2
aattmaze medium play.v2
Nomalize Return
Full data TDG+BC
Nomalized Return
Full data COL
Nomalized Return
Full data ICL
Nomalized Return
Full data DOGE
Messing TD3+8C
Masing data COL
Maxing ciite (Q)
8 8 8 8
Maeeing dats DOGE
8 B 8
8
13
0.0
0.2
0.8
0.6
0.8
Training Steps (x10%)
Training Steps (x
Training Steps (x10%)
Training Steps (x10%)
hitmaae-medunnddverre-e/
artmaaeemmedumn-dverreee
frimaze-medinn-ddeereew
nntmaze-mellum-ddveree--
Nomalized Return
Full data TDS+BC
Nomalized Return
Full COL
Nomalized Return
Full data ISL.
Nomalized Return
Ful cata DOGE Miseing DOGE
Masiing TD3+BC
Miesing COL
Mbooing
3
8
8 & 8
35
0.0
0.2
0.4
1.0
1.9
0.8 Training Steps (x10%)
Training Steps (x10%)
Training Steps (x10f)
Training Steps (x10%)
Figure 9: Evaluation on TD3+BC(Fujinto & Gu, 2021), CQL(Kumar al., 2020b), IQL(kostrikov et al., 2021b), and DOGE (ours) before and after removing the data shown in Fig. 8a for AntMaze medium tasks.
27

---

Published as a conference paper at ICLR 2023
Table Ablation for DOGE generalization with different removal areas.
Dataset
Full dataset
One removal
Two removal
antmaze-m: m-
80.6#6.5
62.3+7.5
33.2+27.3
uutmaze-m-d-v2
77.6#6.1
41.3+42.8
40.2+32.9
antmaze-l-p-v2
48.248.1
26.4+19.4
22.4=15.9
antmaze-l-d-v2
36.4+9.1
12.3+4.2
14.6+11.1
Total score
242.8+29.8
142.3+73.9
110.4187.2
F.2 ADDITIONAL COMPARISON WITH TD3+BC
In this section, we further demonstrate the superiority of DOGE over our most related practical work TD3+BC (Fujimoto & Gu, 2021). One can find that the biggest difference between DOGE and TD3+BC lies the policy constraint used for policy optimization:
-TD3+BC: constrains the policy to minimize the MSE BC loss.
DOGE:
constrains the policy minimize the learned state-conditioned distance function g(s, a).
As discussed in Section 3.1, the learned distance function g(s, a) can capture the global geometric information of the offline dataset, while the MSE BC loss can only provide local sample-to-sampe regularization, which may be noisy, especially in datasets that contain low-quality samples. Taking Figure 10 as illustration, under strict BC constraint, policy learning on noisy low-quality samples may provide contradicting learning signals to near-optimal samples, which can cause inferior policy performance and unstable training process. By contrast, the state-conditioned distance function g(s, a) in DOGE is trained on the whole dataset and hence brings global geometric information, which is far more informative and stable as compared with the MSE BC loss.
(a) TD34BC
(b) DOGE
quality Samples
Policy
g's. e)
Low-quality Semples
Near-optimal Samples
Figure 10: Illustrations of the differences between conditioned distance function constraint of DOGE, In (a), the MSE BC constraint in TD3+BC blindly enforces the imitation behavior on any data samples, which may lead to an inferior policy in the presence of noisy low-quality samples. In (b), the state-conditioned distance function g(s, can provide more informative global dataset geometry information guide the stable learning of the policy.
MSE BC constraint TD3+BC and (b) the state-
To better illustrate the superiority of DOGE over TD3+BC, we add extra comparative experiments with TD3+BC on new of mixed-quality datasets. In halfcheetah-randden dataset, we add different proportions (1% to 20%) of the near-optimal seffcheetaa-medium--xxe expert dataset to form new mixed datasets and evaluate how TD3+BC and DOGE perform. See Figure 11 for detailed results.
Figure 11 shows that DOGE enjoys more performance gains when the random dataset involves near-optimal data, while TD3+BC is heavily influenced by the local information from the larger proportion of the low-quality random data. Moreover, TD3+BC suffers from severe oscillation and training instability, while DOGE enjoys a stable training process due to the use of the more informative state-conditioned distance constraint that captures the overall dataset geometry.
28

---

Published as a conference paper at ICLR 2023
Halfcheriah 1a
Hallihhetaahhariee--
Halfchwetahh Ratio-5%
Halfchessah Ratio-3055
Hallcheotah -at 20%
Nomaltted haum
DOGE
Nomulia Retum
COGE
Nomslove Retum
DOGG
Nanallzed Retum
0006
Trewed
8
&
9
0.20
8.90
023
100
023
058 3.76 1.00
0.8
8.30
0.3
120
020
055
375
0.2
8.30
073
120
Traniing Steps (+10
Training Steps (x10%)
Training Steps (x10%)
Training Steps kx10%)
Traning Steps Steps (x10%)
Figure 11: Comparisons between DOGE and TD3+BC on mixed datasets with different proportions of malfchectah-medinee eeeeput--epperttt dataset added into halfcheetah-randde dataset. Ratio- 1% means 1% medium- experrt dataset is added into the original haffcheetah-randomn dataset. TD3+BC suffers severe oscillation and training instability, while DOGE enjoys stable training processes and substantial performance gains.
F.3 COMPARISON WITH UNCERTAINTY-BASER METHODS
We also compare DOGE with SOTA uncertainty-baser offline RL approaches, including EDAC (An al., 2021) and PBRL (Bai al., 2021) more complex D4RL AntMaze tasks. The final results are presented in Table 5. Table 5 shows that the SOTA uncertainty- based methods are unable to provide reasonable performance performance on simpler MuJoCo tasks. A similar finding is also reported in a recent offline RL study (Anonymous, 2023).
the difficult Antmaze tasks, despite that they can achieve good
In practical implementation of EDAC and PBRL, to obtain relatively accurate uncertainty measures and achieve reasonable performance, these methods typically need dozens of ensemble Q-networks, which can be quite costly and inefficient. Moreover, heavy hyperparameter tuning is also required for them to obtain the best performance. In contrast, our method quantifies the generalization ability of the Q-function from the perspective of dataset geometry and is trained using a simple regression loss Eq. (4), which enjoys better training stability and simplicity.
Table 5: Average normalized scores over 5 seeds on Antmaze tasks
Dataset
EDAC
PBRL
DOGE(Ours)
antmaze-u-v2
0
97.0+1.8
antmaze-u-p-v2
0
63.5+9.3
antmaze-m-pp-2
0
0
80.6#6.5
antmaze-m-dd-v
0
77.626.1
antmaze-l-p-v2
0
48.2+8.1
antmaze-l-d-v2
36.419.1 0
F.4 ADDITIONAL ANALYSIS ON DISTANCE FUNCTION
We report the learning curves of the conditiood distance function g(s, a) trained on different datasets (including hopper-m-v2, halfcheetah-ne and walker2d-m-v2 in Figure 12. Our proposed state-conditioned distance function learned through simple regression task (Eq. (4)), which is very casy to train. Figure 12 shows that it reaches convergence within only IK training steps D4RL MuJoCo medium datasets.
We also change the network configurations (i.e., number of hidden layers and hidden units) of the coteecoonditonee distance function g(s, a) to investigate how the expressivity of influences the performance of the policy. Table 6 shows that DOGE achieves similar performance across different network configurations, indicating that DOGE robust to model complexity and expressivity of the state-conditioned distance function.
29

---

Published as a conference paper at ICLR 2023
hopper-m
halfcheetah-m
walker2d-m
8
Distance loss
Distance loss
Distance loss
6-
4+
4
2
2-
0.00
1.00
Training Steps (x10²)
Training Steps (x10²)
Training Steps (x102)
Figure 12: Learning curves of the state-conditioned distance function g(s, a)
Table Normalized scores of DOGE trained on distance functions with different network configu- rations. [128, 128] means g network has 2 hidden layers with 128 units. [256, 256, 256] means hidden layers with 256 units.
Dataset
[128, 128]
[256,
256]
[256, 256, 256]
hopper-m
99.4
101.4
98.6
halfcheetah-m
47.4
46.9
45.3
walker2d-m
85.3
86.4
86.8
F.5 ADDITIONAL EXPERIMENTS OF THE IMPACT DATA GEOMETRY ON DEEP FUNCTIONS
We run several experiments with different random seeds (see Figure 13). Although the approximation pattern of different random seeds is not the same, they all perform in the same manner that deep Q functions produce relatively low approximation error inside the convex hull of training data. We refer this phenomenon deep functions interpolate well but struggle to extrapolate.
1.0
1.4
Approximation Erro
1.0
0.7
Approximation Error
1.0
0.9
Approximation Error
Action
Action
0.0
0.7
0.0
0.4
0.0
0.4
-0.5
-0.5
-0.5
-1.0
-1.0
0.0
10
-5
0
10
State
State
State
1.0
0.8
Approximation Enrr
1.0
1.4
Approximation Error
10
0.9
Aoproximation Eroor
0.5
0.5
0.5
Action
Acton
0.4
0.0
0.4
005
005
-0.5
-1.0 10
0.0
-1.0
10
-5
0.0
-1.0
0.0
State
State
State
1.0
0.5
1.0
0.9
Approximation Error
1.0
0.8
Aporoxiraation Error
0.5
0.5
0.5
Approximation
Action
Action
0.0
0.4
0.4
-1.0
10
5
10
0.0
-1.
10
-5
0
0.0
-1.
10
-5
10
0.0
Figure 13:
The figures above depict the effect of different data geometries on the final deep Q functions approximation error. The training data are marked as white dots.
30

---

Published as a conference paper at ICLR 2023
G ABLATIONS
i- [[(s),;)] (see Figure 14), the non- parametric threshold G in Eq. (6) (see Figure 16) and the non- non-parametric number of noise actions N to train state-conee conditiond distance function (see Figure 15) on the performance of the final algorithm. We also conduct ablation studies on the effect G on the Lagrangian multiplier (see Figure 17).
We conduct ablation studies on the effect of a in B
For a, we add or subtract 2.5 to the original value. For N, we choose N = 10, 20, 30 to conduct experiments respectively. For G. we choose 30%. 50%, 70%. 90% and 100% upper quantile of the distance value in mini-batch samples and the results can be found in Table 7.
Table 7: Ablations on G with different quantile.
Dataset
G = 30%
G = 50%
G = 70%
G = 90%
G = 100%
hopper-r-v2
19.8+0.3
21.1+12.6
13.5+13.5
17.6+12.2
16.4+12.4
halfcheetah-r-v2
19.4+0.6
17.8+1.2
17.8+0.7
17.7+1.0
17.740.8
walker2d-r-v2
2.623.9
0.9+2.4
2.2+2.6
1.8+3.3
2.2+3.2
hopper-m-v2
44.6+5.7
98.6+2.1
99.4+0.4
91.5+9.9
32.9+54.3
halfcheetahh-m-v2
41.3#1.2
45.3+0.6
46.0+0.1
46.0+0.8
46.1+0.5
walker2d-m-v2
83.7+7.5
86.8+0.8
87.3+1.6
69.9428.9
84.2+1.0
hopper-m-r-v2
51.5+11.2
76.2+17.7
79.6+36.9
78.4+27.6
65.7+37.2
halfcheetaah-m-r-22
5.945.7
42.8+0.6
43.2+0.1
42.2+0.8
42.0+0.6
walker2d-m-r--v
28.3+14.3
87.3+2.3
87.942.4
77.8+21.6
78.6+24.1
hopper-m-e-v2
61.7+10.4
102.7+5.2
82.8+5.8
88.9+17.7
70.0+48.4
halfchectahh-m-ev22
46,945.2
78.7+8.4
75.1
73.5+13.6
69. 69.948.7
walker2d-m-o-v2
110.5+0.7
110.4#1.5
111.1#0.5
110.2+22.5
80.0+54.3
Seen from Table 7 that using different G for different tasks may achieve even better performance. Particularly, for some datasets with diverse data distributions that need to find good data from suboptimal data, a more tolerant quantile (e.g., G = 70%) can reasonably extend feasible region and increase the opportunity to find the optimal policy, such as hopper-m-r, halfcheetah-m-4 walker2d- However, an overly relaxed quantile (e.g., G = 90% and 100%)
m-r, hopper-m- halfcheetah-m increases the risk of including problematic OOD actions in policy learning, causing performance drop due to value overestimation and high variance.
By contrast, an overly restrictive quantile such as G 30% can be over-conservative and cause significant constraints violations that impede policy learning, constraints satisfaction is favored over the max-Q operation in most updates. This can be reflected in the additional results for the Lagrangian multiplier X (see Appendix E.2 for learning curves and Figure 11 for additional ablations), 1-7
where À for some tasks under G 30%. This will cause the suboptimality gap (227 (lT)) in Theorem 3 to dominate the performance bound, leading to inferior policy.
As hyperparameter tuning in practical offline RL applications without online interaction very difficult, reduce the computational load, we set G = 50% as default in non-parametric manner, since it consistently achieves good performance, and is neither too conservative nor too aggressive for most tasks.
Observe in Figure that DOGE maintains the similar performance with the changes of a on most of Mujoco tasks. At the same time, we also observe that the effect of N on the experiment is not obvious. Compared with N and a, we find that has more significant effect on the experimental results. Observe in Figure 16 that small G usually causes the policy set induced by DOGE to be too small to obtain near-optimal policy. By contrast, a large G is not likely to cause excessive error accumulation and hence maintains relatively good performance.
In addition, the ablation studies show that our method is hyperparameter-rooust and maintains good performance with changes in hyperparameters.
31

---

Published as a conference paper at ICLR 2023
hopper-random--2
hopper-medumm-eplayye
hopoer-medium-22
hogget-meduu--epprttt
Nomalited Return
100
0- 15.0
0- 17.5*
Nomallzed Return
2- 17.5
15.0
Nomalized feturn
Nomalized Return
15
20.0
75
20.00.0
03-
15.0
23
23
37.5
20.0
100
8.00
0.25
0.00
076
100
0.25
0.50
0.75
1.00
0.25
0.50
0.76
100
0.25
0.50
0.78
1.00
Training Steps (x10*)
Training Steps (x10))
Training Steps (x10")
Training Steps (x10")
Balker2d-randoman
walkeerdd
weller2d-mdium-reppie wwddimmmedldbseeeeee r2play-22
walker2d medium-u v2
walker2d mediimm expert-v2
Nomalized Return
102
am 17.3*
a- 15.0
Nomalized Return
Nomalized Return
1oc-
Nomalized Return
25-
a 20.0
75
75
70
23
25
88
10.0
0.25
0.50
0.75
1.00
0.75
0.5
0:30
7.
1.00
800
Training Steps (x10%)
Training Steps (x10%
Training Steps (x10%)
Training Steps (x109)
halfchettah-rrddom-22
halfcheetah-medium- -erietttttmiiiu---pett-t
Nomalized Return
102
o- 15.0
am 17.3*
Nomalized Return
0-5.0
7.5..5 7.5
Nomalized Return
100
am 7.5*
6-5.0
Nomalized Return
25-
a 20.0
75
16.0
25
0 10.0
75
03
20.0
0.25
0:50
0.7
1.00
0.50
0.75
8.0
0.25
0.50
0.75
1.00
0.50
0.75
1.00
Training Steps (x10%)
Training Steps (x10%)
Training Steps (x10%)
Training Steps (x10%)
Figure 14: Ablation for a. Error bars indicate min and max 5 seeds.
heopen-random-22
hopper-medimmeepllaaa.
hopper-medium-22
hogper-medium expert.v2
Nomalized Return
N- 10
Nomalized Return
Nomalized Return
20
72-
75
75-
75
Nomalized
03
15
10
10
20
30
30
8.0
0.25
0.50
0.75
1.00
0.25
0.50
7.75
1.00
800
0.25
0.50
0.75
1.00
80:
0.25
0.50
1.00
Training Steps (x10%)
Training Steps (x109)
Training Steps (x10%)
Training Steps (x109)
malker2d-randamuu
walker2d reperrr-ei
dallier2d-mediume
mallier2d-meiiu medium-expert--2
Nomalized Return
a-
Nomalized Return
105
Nomalized Return
Nomalized Return
100
23
75
25
80
20
N 20
30
30
N 30
8.00
0:25
0.50
0.75
100
8.00
0.25
0.50
0.75
100
8b0
0.55
0.50
0.7
100
0.25
0.50
0.75
100
Training Steps (x10%)
Training Steps (x10%)
Training Steps (x10%)
Training Steps (x10")
hallcheetah-randdm-22
halfeheetah-med rellal/
medic-e
halfcheetah-mediume 2
Nomalited Return
A- 10
Nomalized Return
11-10 10
Nomalized Return
-=10 10
=0- 20
11 20
- 20
25
25
N1 50
75
Nomelized
6-
83
8
N 10
23
20
8.00
0.25
0.50
0.76
1.00
0.00
0.25
0.50
0.75
1.00
0.25
0.50
0.75
1.00
0.25
0.50
0.75
1.00
Training Steps (x10%)
Training Steps (x10%)
Training Steps (x10%)
Training Steps (x10%)
Figure 15: Ablation for N. Error bars indicate min and max over seeds.
H LEARNING CURVES
The learning curves for Mujoco and AntMaze tasks are listed in Fig. 18 and Fig. The learned policies are evaluated for 10 episodes and 100 episodes each seed for Mujoco and AntMaze tasks, respectively. For AntMaze tasks, we subtract from rewards for the AntMaze datasets following (Kumar et al., 2020b; Kostrikov et al., 2021b).
32

---

Published as a conference paper at ICLR 2023
hopger-random--2
hogger-meddum-eellaan2
hosper-medium-22
hopper-medte Srogee-medium-eppere
Nomalized Return
30%
Nomalized Return
30%
Nomalized Return
Nomalized Return
50% +
50%
033
00%
70%
75
70%
73-
25
30%
30%
30%
50%
70%
70%
90%
90%
8.50
0.25
0.50
0.75
1.00
0.25
0.00
0.75
1.00
8.00
0.25
0.50
0.75
1.00
8.00
0.25
0.50
0.75
1.00
Training Steps (x10%)
Training Steps (x10%)
Training Steps (x10%)
Training Steps (x10%)
walker2d random-v2
walker2d medium-v2
walker2d-medium 2xpert-22
Nomalized Return
30%
Nomalized Return
Nomalized Retum
Nomalized Return
50% +
02
90%
70%
25
25
30%
30%
30%
50%
50%
50%.
70%
70%
70%
90%
90%
96%
8.0
0.25
0.50
0.75
1.00
0.25
050
0.75
1.00
8.00
0.25
0.50
0.75
1.00
8.m
0.25
0.5
0.55
1.00
Training
(x10%)
Training Steps (x10%)
Training (x10%)
Training Steps (x10%)
halfcheetah random-22
malichetat-medium-rephl
halfeneetah medium-v2
haffcheetah-mmellum experr--22
Nomalized Return
30%
Nomalized Return
33%
Nomalized Return
30%
50% +
52%
50%
20
90%
70%
25
90%4
70%
20
90%
70%
Nomaltzed
75
32
80
50%
30%
70%
90%
8.50
0.25
0.50
0.75
1.00
8:00
0.25
0.50
0.75
1.00
8.00
0.25
0.50
0.
1.00
6.00
5.55
0.50
0.75
1.00
Training (x10%)
Steps (x100)
Training (x10%)
Training Steps (x109)
Figure 16: Ablation for G. Error bars indicate min and max over 5 seeds.
hopger-random--22
hosper-medim-repleyeeee
hopper-medium-22
hooger-medlum-expettt22
Lagrangann
0.3
0.5(cws)
A
0.5(cur))
Lagrangan,
100
0.5(cur))
Aegrangan
0.5(ours)
80
20
30
20
8.00
0255
0.50
0.70
1.00
8.0
025
0.50
0.7g
100
800
0.25
0.50
0.76
120
800
0.25
0.50
1200
(x10")
Training Steps (x10")
Training (x10")
Training Steps (x10")
walker26-rardom-22
walker2d-mediume
welke22d-mediumm22
walker2d.madi expert.v?
G=
0.3.33
G=
0.3.33
Lagranglan
0.5wors
Lagranglan
0.56our)
Pagranglan
75-
75-
25
75
%
025
0.50
0.75
1.00
8.00
0.25
0.50
0.76
100
0.25
0.5
0.75
100
800
0.25
0.50
0.75
100
Training Steps (x10%)
Training Steps (x10%)
Training Steps (x10%)
Training Steps (x100)
hallcheetah-enddm--2
helfcheetah-meedee baabhettah-mddu---ellllaee
hallchettal-medium-2
halichesth--eddi-eexpeee 2
D
G D.3
G 0.3
6
Lagranglan
100
0.5 0.5wss
Lagranglan
6 0.5iows)
Lagranglan A
G 0.5tours)
Lag Berangann
G Stourss)
25-
75-
6 0.9
0.9
75
G
8.00
8.0
800
8.00
Training Steps (x109)
Training Stops (x10%)
Training Steps (x10*)
Training Stops (x10%)
Figure 17: Ablation for A. Error bars indicate min and max over 5 seeds.
33

---

Published as a conference paper at ICLR 2023
hegper-random-ag
hopger-medium-2
hogper: medium hogper-medium
expert
Nomalized Return
Nomalizen Return
Nomalized Return
Nomalized Return
d4
73-
75
72
M
75
03
8.00
0.25
0.50
0.75
100
8.00
0.25
0.50
0.75
1000
800
0.25
0.50
7.
160
0.25
0.50
0.75
1.00
Training Steps (x10%)
Training Steps (x10%)
Training Steps (x10%)
Training Steps (x10%)
walker2d-rannn 2alkerr2d-anddmmuz2
walker2d me
walker2d-madiume
wallimmddmedium medium-expprrt-2
Nomalized Return
Nomalized Return
Nomalized Return
72
75
72-
75
Nomalized
83
15
8.00
0.25
0.50
0.75
100
0.25
0.50
0.75
1000
800
0.25
0.50
0.
160
0.25
0.50
0.75
1.00
Training (x10%)
Training Steps (x10%)
Training Steps (x10%)
Training Steps (x10%)
tullcheetah-random-22
halfeheetah www.seemmiiiiiiiiiiiiei
helfcheetah-mediim-22
hallcheetah-medium-eee -xxperr--22
Nomalite Return
Nomalized Return
23
25
75
Nomalized
Nomelized
8
8
1
8.00
0.25
0.50
0.75
1.00
0.25
0.50
0.75
1.00
0.25
0.50
0.75
1.0
0.25
0.50
0.75
1.00
Training Steps (x10%)
Training Steps (x10%)
Training Steps (x10%)
Training Steps (x10%)
Figure 18: Learning curves for Mujoco Tasks. Error bars indicate min and max seeds.
antmaze-umaze-v2
antrmaze-reddum-pllayyy-
antmaze-large-play--2
100
Nomalized Return
Nomalized lized Return
Nomalized Return
75
75
50
50
50
25
25
25
8.00
0.25
0.50
0.75
1.00
8.00
0.25
0.50
0.75
1.00
8.00
0.25
0.50
0.75
1.00
Training Steps (x106)
Training Steps (x106)
Training Steps (x10%)
antmaze-uuaaee-lvvereew>
antmare-mediimm-diveees--
antmaxe-large-diverre-z
100
100
Nomalized Return
Nomalized Return
Nomalized Return
75
50
50
25
25
28
8.00
0.25
0.50
0.75
1.00
8.00
0.25
0.50
0.75
1.00
8.00
0.25
0.50
0.75
1.00
Training Steps (x106)
Training Steps (x106)
Training Steps (x10%)
Figure 19; Learning curves for AntMaze Tasks, Error bars indicate min and max over 5 seeds.
34


Reviews:
[{"review_id": "qTXUfCke_ou", "reviewer": "Reviewer_kL5r", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes to regularize continuous control policies trained offline by forcing them away from actions that are too far from the training data's actions. This is achieved by a learned distance function that predicts distance to the training data's actions. The authors show that such a setup has some interesting properties. Empirically this appears to yield improvements on the D4RL offline benchmark suite.", "summary_of_the_review": "I am a bit conflicted about the paper, the intuitions behind the ideas are nice. The paper spends a lot of time trying to convince us of the usefulness of a variety of bounds (often closely based on prior bounds), used to justify the approach, but I am skeptical that these bounds are reflected in the results. This is worsened by the fact that some predictions made by the theory are not properly tested in practice (see above).\n\nUpdate: the authors have addressed my concerns, and I find the additions to the paper and the appendix are useful to framing this work. I lean in favor of acceptance.", "strength_and_weaknesses": "Strengths:\n- The proposed method stems from a series of interesting intuitions\n- The proposed method seems to be able to capture something about the MDP which increases its performance in an offline RL context\n\nWeaknesses:\n- The text is hard to read at times\n- The theory within the paper relies heavily on quantities which may not scale intuitively\n- There are some strange empirical evaluation choices, in particular:\n    - If I understand correctly, the core of the proposed method is to contrain a (continuous) policy to output actions that are not \"too far\" from the training data's actions. Yet, the main way this is tested is to remove part of the _state space_ from the training data? This is not testing the right thing, and is not even testing what the theory predicts that this method should be better at.\n- It's nice to have bounds, but what's even nicer is empiricially verifying that those bounds are meaningful and/or make meaningful predictions about trained models--this is missing from this work.\n\n\nAdditional comments:\n\n> we observe that the value function approximated by DNN can interpolate well but struggles to extrapolate\n\nThere are papers showing extrapolation capabilities in deep RL, see e.g. the work of Packer et al. [1]\n\n[1] Assessing Generalization in Deep Reinforcement Learning, Charles Packer, Katelyn Gao, Jernej Kos, Philipp Krähenbühl, Vladlen Koltun, Dawn Song, 2018\n\n\n> Theorem 1\n\nI'm not sure I fully get why the authors think this theorem is important.\n\n1. The NTK assumptions provide kernel-like bounds, and what we get is litterally that the further we get from the dataset the poorer the approximation, this is already known. There is some extra work to make the result a bound over the projection, but that is also fairly straightforward.\n2. This result a priori has nothing to do with the convex hull of the training data. (3) and the equation before (2) are the same equation. The main argument seems to be that because the convex hull of a finite set of points is naturally bounded, then (2) is bounded whereas (3) is not. This argument seems moot in practice, since inputs to neural networks (or any ML system really) are almost always bounded (images lie in RGB[0,1], language is a sequence of tokens, etc).\n3. The result relies on an Euclidean norm, which quickly loses significance past a number of dimensions. In fact, it is likely that past a certain number of dimensions, the very distinction between interpolation and extrapolation lose meaning (see [2]). Take 100k images from Atari at random, then take a new image by running an agent or again at random. I'd wager that either (a) the new image is within the convex hull, or (b) its distance to the convex hull is upper bounded by Eq (2)'s $B$, because $B$ is close to 1.\n\nI'm also not sure what $x$ stands for here. The error on a $Q$ function evidently depends on whether not just the state but also the action is in the dataset. Is $x$ just the concatenation of the two? What does it mean to concatenate an Atari image and a categorical label and then take norms? What about joint angles and torsion strengths?\n\n[2] Learning in High Dimension Always Amounts to Extrapolation, Randall Balestriero, Jerome Pesenti, Yann LeCun\n\n\n\n> State-conditioned distance function\n\n> synthetic noise actions sampled from the uniform distribution\n\nThis is a strange choice, why the sampling? We know that MSE regresses to the mean, and we know the mean of $\\mathbb{E}_{x'\\sim U}[|x-x'|]$. The result (regressing to the centroid) should be the same.\n\n> It is clear that the learned distance function can accurately predict the sample-to-dataset centroid distance\n\nIn the action domain, yes, perhaps. But what about the state domain?\n\n> To evaluate the generalization ability of DOGE, we remove small areas of data from the critical pathways to the destination\n\nAs above, this does not evaluate the right thing.\n\nI am left wondering after reading this paper if a simpler baseline might be just as effective, one that doesn't require $g$. In the limit, but when each state only appears once in the training data with one associated action*, $g$ is simply the distance between $\\mu(s)$ and $a$. What happens when we replace $g(s,a)$ by $\\|a-\\mu(a)\\|$ in (6) and (7)?\n\n*In a continuous state-action space, isn't every state really only \"visited once\" anyways?\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "bATv8y_ETi", "reviewer": "Reviewer_hPVe", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors suggest that current offline RL methods are overly conservative in interpolating regions of the dataset, whereas this should primarily be applied to extrapolation. They suggest that this should be controlled by the distance, which the bound/estimate using a state-conditional action distance estimator. Satisfying this distance constraint then becomes a regularizer for policy optimization. Theoretical analyses show that this provides a tighter bound on performance as compared with a previous method, BEAR. Results are presented on D4RL, comparing with relevant model-free baselines. There are also experiments on an extrapolation version of the task, removing some parts of the state/action space.", "summary_of_the_review": "The paper presents a compelling set of empirical results, alongside theoretical analyses. While I’m fairly convinced that the method performs well against strong baselines in standard offline RL benchmarks, I found it difficult to follow some aspects of the paper, particularly with regard to motivating the approach, e.g., assumptions about the underlying environment, and translating it into a practical training technique.\n\n**UPDATE:**\n\nI appreciate the authors' detailed responses to me and the other reviewers. While there may be some unresolved concerns, particularly around the clarity of the paper, most of my main concerns have been addressed. I now feel that the paper warrants acceptance, and have increased my score from 6 --> 8.", "strength_and_weaknesses": "**Strengths**\n\n- **Strong empirical evaluation**. The authors evaluate their method, DOGE, on a standard offline RL benchmark, D4RL, comparing with the relevant baseline approaches. This is in both the standard locomotion set of tasks, as well as the more difficult AntMaze environments. In all cases, the authors’ approach generally compares favorably or performs similarly. The authors also include an analysis on the AntMaze environments, removing transitions from some regions of the state-space. This allows them to test interpolation abilities of previous methods. I view this empirical evaluation as the strongest point of the paper.\n\n- **Relatively simple method**. While the authors have included a considerable degree of theoretical analysis, the method that they arrive at is ultimately fairly simple, involving a learned, state-conditional action distance estimator trained from offline trajectories. This then enters policy optimization as an additional regularization term. I see this as no more complex than many previous methods, e.g., which learn behavior priors. This simplicity may lead to more widespread adoption of the method in the RL community, making DOGE a useful baseline for future works.\n\n- **Empirical and theoretical analysis**. The authors combine empirical and theoretical analyses. While much of the theoretical analyses are outside of my area of expertise, I appreciate that these can be important. Again, this may broaden the appeal of the paper to the wider RL community.\n\n\n**Weaknesses**\n\n- **Unclear in places**. At various points throughout the paper, I found it difficult to connect the authors’ theoretical connections and motivations to the particular method proposed. For instance, the authors’ initial motivation is framed in terms of better interpolation of the state-action space of the dataset. Yet, the authors translate this into something akin to estimating the cross entropy of the behavior policy w.r.t. a uniform policy, conditioned on the state. It’s unclear how this exactly relates to the state-action distance initially motivated by the authors. I was also left uncertain about whether the authors are arguing that the state-space, action-space, or both should be interpolated. The initial motivation leads me to believe that it’s both, whereas the proposed method seems to tackle the action-space and the empirical analyses only tackle the state-space. Along similar lines, the state-conditional action-distance penalty is introduced as a regularizer, but it’s unclear how this translates into estimating and propagating state uncertainty. I would have thought that this regularizer would by integrated over space/time by entering the definition of the Q-value (similar to KL-regularizers, e.g., in SAC), but this seems to not be the case. I also found the diagrams difficult to interpret (although I did appreciate that they were included).\n\n- **Unclear how, precisely, this tackles interpolation better than previous methods**. The authors motivate their approach from the perspective of improving the ability to interpolate regions of the dataset, using the perspective of deep Q-networks in the interpolation and extrapolation regimes. In interpolated regions, this error is bounded, whereas in extrapolation regions, it is not (although it will still be correct nearby the data manifold). This then leads the authors to propose their state-conditional action distance estimator. However, previous methods, like BEAR, use ensembles to estimate something like Q-value uncertainty, which is then penalized. Following the authors’ arguments, there should be relatively less uncertainty in interpolated regions as compared with extrapolated regions, allowing these methods to also handle interpolation. Where is my misunderstanding here? Why is this method uniquely suited to handling interpolation, whereas other methods, as the authors show empirically, are not? Ultimately, with a finite dataset and continuous action space, all methods have to interpolate for every single point — does this method just have a wider window of interpolation?\n\n- **Applicability/generality unclear**. While reading the paper, I felt as though there were a number of unstated assumptions that the authors were making. The authors initially motivate their approach by considering interpolation vs. extrapolation in (continuous-action) deep Q-networks, though it seems the same technique could not be applied to discrete action space. The tasks considered by the authors consist of relatively smooth dynamics and reward functions, thus, interpolation in an environment like AntMaze is valid, especially if it allows you to connect two regions of the state-space. However, this strikes me as ultimately a heuristic, which may or may not be respected by the actual environment. Making this heuristic assumption is useful when it works out, but it’s not a general principle that should be relied upon. For example, if we consider a reward function in the shape of an inverted Mexican hat (e.g., imagine an environment with a heat source like a campfire that keeps the agent warm, but would burn the agent if it touches it directly), but you only observe data around the high-reward outer ring, then interpolation will lead you to estimate that the center is also high-value, which is fatally wrong.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "JIAv5WZFUhJ", "reviewer": "Reviewer_z7dd", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a method for accounting for the geometry of the data in devising methods for enforcing pessimism in offline RL. To do so, the paper ends up training a state-conditioned distance function between actions, and enforces a policy constraint using the learned distance function. The hope is that this is useful when the distance function can generalize (though I have questions about this below).", "summary_of_the_review": "Overall, I am not convinced that the method is robust and reliable unless the concerns of g(s, a) are discussed. The theoretical analysis doesn't explain why g(s, a) should be good in practice or why alternatives such as value conservatism are not doing the same job. More empirical evidence and improved theoretical analysis is needed.\n\n-------\n\n## After Rebuttal\n\nThanks for the responses! I revisited Theorem 2, but I still don't see why $d_1$ and $d_2$ alone control the the Bellman concentrability coefficient tightly. There are many more terms in there that depend on the Q-function class in Theorem 2 ($Q(s_0, a_0)$, $\\epsilon_\\mu$), and in general, it cannot be said that the sample to centroid distance is alone what it takes to control this distributional shift. This just makes me think further that the approach would better benefit from a \"geometry constraint\" on the Q-function class (as it would more tightly control the Bellman concentrability coefficient) as opposed to the continuous action space distance constraint.\n\nRegarding comparison with TD3+BC conceptually -- Sorry, I am still not sure. In equation 4, it seems like $g(s, \\hat{a}) = ||\\hat{a} - a_\\text{data}||_2$ at the optimum. This means that the learned function when Equation 4 is solved exactly is measuring the L-2 distance to the dataset action (assume for a second that there is only one action at a given state in the dataset). Then isn't it identical to TD3+BC? When multiple actions are observed for a given state in the dataset, one could argue that even TD3+BC would have a similar global effect, as you would train $\\pi(s)$ to be close to $a_1$, $a_2$, ... for all actions observed in the data, which would achieve the same kind of a smoothing effect as in Figure 3 (a).\n\nIt seems like the author's intuition goes beyond a state-conditioned function to some smoothing over the dataset more generally. But this is not clearly explained, and it is not clear why other methods and TD3+BC, most directly, will also not enjoy some kind of global smoothing effect more generally with a small function class. \n\nSo I don't understand how is this true  _\"In TD3+BC, the BC term simply minimizes the absolute sample-to-sample distance between actions from the policy and the dataset, which essentially avoids the policy from producing OOD actions. Whereas in DOGE, the state-conditioned distance function  characterizes a sample-to-dataset distance that captures the overall data geometry. \"\n\nOverall, I am inclined to keep my original score. Since my score is at a disagreement from other reviewers, I will reduce my confidence by 1.", "strength_and_weaknesses": "Strength: The idea seems interesting, and new, and certainly improves over policy constraints in their experiments (TD3+BC vs their method). \n\nWeaknesses:\n\n- I think I don't understand why the approach should perform well in general -- doesn't it depend on what kind of a function we train for g(a, s)? What if g(a, s) were a table, would the method not be just TD3+BC? But the paper doesn't seem to discuss the connection between the complexity of g(s, a) and the efficacy of the method.\n\n- Taking the above reasoning, shouldn't a value function conservatism method like CQL already take into account the geometry of the data? Why should the proposed method be better than that, if it is not just for tuning issues?\n\n- The theory doesn't seem to discuss the role of g and the complexity of g. It builds on the Bellman-consistent pessimism paper, yet no comparisons to the approach from that paper and their theoretical results exists in the paper.\n\n- Empirical results: besides benchmark results, more clear ablations and analysis are needed to understand why the method actually works. Can we do ablations on the network architectures for both TD3+BC and CQL, and the proposed approach to see how all of these behave with smoothing? ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "FlUoqOYdG9d", "reviewer": "Reviewer_PeHF", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Motivated by the over conservativity of existing offline RL algorithms, the paper proposes a Distance-sensitive Offline RL with better GEneralization (DOGE) method. In fact, DOGE describes a constraint set of possible policies, relying on a state-conditioned distance function. Such a distance function accounts for the data geometry. Theoretically, a connection between DOGE and bellman-consistent coefficient is established. Empirically, DOGE is demonstrated to have comparable or better performance versus the state-of-the-art methods.", "summary_of_the_review": "I am interested to see any feedback from the author regarding the theory part of the paper. Overall, I think the paper in its current form is slightly below the acceptance bar, but I am willing to raise the rating based on further discussions.", "strength_and_weaknesses": "================= Strength =================\n\nThe paper focuses on a very important problem in RL: when offline data do not have strong data coverage, how to best exploit the logged data set. The proposed method is novel to me.\n\nThe paper is clearly written with many graphical illustrations. Most of the sections are easy to follow.\n\nEmpirical results are convincing, although I am not totally familiar with the experimental environments.\n\n================= Weakness =================\n\nThe claim \"Theoretical analysis demonstrates the superiority of our approach to existing methods that are solely based on data distribution or support constraints\" may not be well supported, as in Section 3.3, the idea is to transform the policy constraint to Bellman-consistent coefficient. Moreover, Theorem 3 is a bit hard to interpret as no concrete rate is given. It would be helpful to elaborate the claim.\n\nTheoretical results in Theorem 2 have a relatively vague connection to DOGE. At a first glance, the constraint threshold $G$ in equation (6) does not appear in Theorem 2 nor Theorem 3, making it unclear how to evaluate the advantage of DOGE. At the same time, Theorem 2 is still hard to interpret because of the complicated math expressions without intuitive explanations.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster