ODICE: Revealing the Mystery of Distribution Correction Estimation via Orthogonal-gradient Update
Authors: Liyuan Mao
Haoran Xu
Weinan Zhang
Xianyuan Zhan
Venue: ICLR 2024 spotlight (2024)
Keywords: offline reinforcement learning
imitation learning
distribution correction estimation
Primary area: reinforcement learning
TL;DR: A simple modification to DICE-based method could achieve SOTA performance and great robustness.

Abstract: In this study, we investigate the DIstribution Correction Estimation (DICE) methods, an important line of work in offline reinforcement learning (RL) and imitation learning (IL). DICE-based methods impose state-action-level behavior constraint, which is an ideal choice for offline learning. However, they typically perform much worse than current state-of-the-art (SOTA) methods that solely use action-level behavior constraint. After revisiting DICE-based methods, we find there exist two gradient terms when learning the value function using true-gradient update: forward gradient (taken on the current state) and backward gradient (taken on the next state). Using forward gradient bears a large similarity to many offline RL methods, and thus can be regarded as applying action-level constraint. However, directly adding the backward gradient may degenerate or cancel out its effect if these two gradients have conflicting directions. To resolve this issue, we propose a simple yet effective modification that projects the backward gradient onto the normal plane of the forward gradient, resulting in an orthogonal-gradient update, a new learning rule for DICE-based methods. We conduct thorough theoretical analyses and find that the projected backward gradient brings state-level behavior regularization, which reveals the mystery of DICE-based methods: the value learning objective does try to impose state-action-level constraint, but needs to be used in a corrected way. Through toy examples and extensive experiments on complex offline RL and IL tasks, we demonstrate that DICE-based methods using orthogonal-gradient updates achieve SOTA performance and great robustness.

Published as a conference paper at ICLR 2024
ODICE: REVEALING THE MYSTERY OF DISTRIBU- TION CORRECTION ESTIMATION VIA ORTHOGONAL- GRADIENT UPDATE
Liyuan Mao Haoran Xu
Weinan Zhang*
Xianyuan Zhan
1 Shanghai Jiao Tong University
2 The University of Texas at Austin
3 Institute for AI Industry Research (AIR), Tsinghua University
Shanghai Artificial Intelligence Laboratory
maoliy i jtu. edu. en, haoran. xu@utexas .dd ahanxianyuan@ar air. tsinghua. edu. en, wazhangesjtu. en
ABSTRACT
In this study, we investigate the DIstribution Correction Estimation (DICE) meth- ods, an important line of work in offline reinforcement learning (RL) and imita- tion learning (IL). DICE-based methods impose state-action-level behavior con- straint, which an ideal choice for offline learning. However, they typically perform much worse than current state-of-the-art (SOTA) methods that solely use action-level behavior constraint. After revisiting DICE-based methods, we find there exist two gradient terms when learning the value function using true- gradient update: forward gradient (taken on the current state) and backward gra- dient (taken the next state). Using forward gradient bears large similarity to many offline RL methods, and thus can be regarded as applying action-level constraint. However, directly adding the backward gradient may degenerate cancel out its effect if these two gradients have conflicting directions. To re- solve this issue, we propose simple yet effective modification that projects the backward gradient onto the normal plane of the forward gradient, result- ing in an orthogonal-gradee ods. We conduct thorough theoretical analyses and find that the projected back- ward gradient brings state-level behavior regularization, which reveals the mys- tery of DICE-based methods: the value learning objective does try to impose state-action-level constraint, but needs to be used in a corrected way. Through toy examples and extensive experiments we demonstrate that DICE-based methods using centogonal-gradiien updates (O- DICH) achieve SOTA performance and great robustness. Code is available https::/gitthei wwm lyyananllliaaaaaaooooaa OD ODICE-PPOOCcc
update, new learning rule for DICE-based meth-
complex offline RL and IL tasks,
INTRODUCTION
Offline Reinforcement Learning (RL) has attracted lots of attention as it enables learning policies by utilizing only pre-collected data without any costly or unsafe online interaction. It is a promising area for bringing RL into real-world domains, such as robotics (Kalashnikov et al., 2021). ) health- care (Tang& & Wiens 2021). and industrial control (Zhan et al.) 2022). However, offline RL is known to suffer from the value overestimation issue (Fujimoto al. 2018): policy improvement entails querying the value function on values of actions produced by the learning policy, which are of- ten unseen the offline dataset. Those (potentially) out-t--distiiiutiio (OOD) actions can cause catastrophic extrapolation errors of the value function. To remedy this issue, current state-of-the-an art (SOTA) offline RL algorithms typically impose different kinds of action-level behavior constraints into the learning objective, to limit how far the learning policy deviates from the behavior policy.
'Equal contribution, work done during internshipan at Institute for AI Industry Research, Tsinghua University. Corresponning authors.

---

Published as a conference paper at ICLR 2024
Action-level constraint can be incorporated explicitly on the policy improvement step by calculating some divergence or distance metrics (Kumar et al. 2019; Wu et al. 2019 Nair et al. 2020 Fujimoto
& Gu
2021 Li et al. 2022a). or implicitly on the policy evaluation step by regulanzing the learned
value functions (Kumar et al. 2020 Kostrikov et al. 2021a Xu et al. 2023 Garg et al. 2023).
However, action-level behavior constraint is not an ideal choice in offline RL. Although OOD states do not exist during training, they do occur at evaluation time. During evaluation, due to covari- ate shift and compounding errors, the policy could output undesired actions on encountered unseen states since it receives no learning signals on those OOD states during training. Hence, it is better to apply state and action distribution constraint simultaneously. a.k.a., state-action-levvé behavior constraint, to force the policy to avoid or recover from both OOD states and actions. To the best of our knowledge, only a few works explicitly consider state-action-level behavior constraints (Li et al., 2022b; ZZhng et al. 2022). These methods build on top of action-constrained offline RI. algorithms, with additional steps to obtain and regularize OOD states by training dynamies mod- els for OOD detection (Yang et al. 2021). Instead of manually adding the state-level constraint, there already exists an important line of work, DIstribution Correction Estimation (DICE) methods (Nachum & Dai 2020), that is derived by constraining the joint state-action visitation distribution between the learning policy and behavior policy. By applying convex duality (Boyd et al.) (2044), DICE-based methods elegantly transform the intractable lete-a constraint into a tractable convex optimization objective that can be solved using only the offline dataset.
Although built upon the more ideal state-action-level behavior constraint, DICE-based methods known to perform much worse than methods that only enforce action-level constraint (Fujimoto & Gu 2021; Xu et al. 2023 Garg et al.] 2023) on commonly-used benchmark datasets (Fu et al.) 2020). which seems to be rather counterintuitive. In this work, we aim to resolve this issue and reveal the mystery of performance deficiencies in existing DICE-based methods. We first note that the derived value learning objectives of DICE-based methods contain the bellman residual term (r + E. (V(')] (())), i.e., value function should learned using true- true-gradient update (Baird, 1993). which means the gradient of value function should be taken twice on two successor states and We find that the forward gradient term (i.e., the gradient taken on 8) is actually imposing action-level constraint, which explains why recent work (Sikchi et al. 2023) that modifies DICE- based methods using semi-grade eeai-gradient update (Sutton et al. 1998) could match the performance of action-constrained offline RL methods. Furthermore, when diving deep into the backward gradient term (i.e,, the gradient taken on s'), we find that after a small modification, the ability to enforce state-level constraint. More specifically, we project the backward gradient onto the normal plane of the forward gradient, making these two gradient terms orthogonal. We term this method orthogonal -rrddint update, by doing so, forward and backward gradients will not be canceled out by each other. Theoretically, wuthogonal-gradien update has several appealing properties. It offers better convergence properties than semi-gradient update and makes the value function learn better representations across states, alleviating the feature co-adaptation issue (Kumar (et al. 2022). while being more robust to state distributional shift (i.e., OOD states).
is endowed with
We further propose practical algorithm. O-DICE (Orthogona--DICE) that incorporates V-DICE algorithm (Lee et al.] 2021] Sikchi et al.] 2023) with aathogonal-gradeenn update. We verify the effectiveness of O-DICE in both the offline RL setting and the offline Imitation Learning (II.) setting, where only one expert trajectory is given (Kostrikov et al. 2020 Zhang et al., 2023b). O-DICE not only surpasses strong action-constrained offline RL baselines, reaching SOTA perlormance in D4RL benchmarks (Fu et al., 2020), but also exhibits lower evaluation variances and stronger robustness to state distributional shift, especially on offline IL tasks, in which OOD states are more likely to occur, and being robust to them is crucial for high performance. To aid conceptual understanding of sethogonal-gradice
update, we analyze its learned values in simple toy setting, highlighting the ad advantage of its state-action-level constraint as opposed to true- true-gradient update or action- constraint imposed by semi semi-gradient update. We also conduct validation experiments and validate the thcoretical superiority of cathhoonal-gradiion updates.
2 MYSTERY OF DISTRIBUTION CORRECTION ESTIMATION (DICE)
In this section, we give a detailed derivation about how we reveal the mystery of DICE-based meth- ods by using orthogonal-gradient update. We begin with some background introduction about differ-
2

---

Published as a conference paper at ICLR 2024
ent types of DICE-baset methods. We then dive deep into the gradient flow of UICE-based methods. We discuss the effect of two different parts in DICE's s gradient flow and find a theory-to-practicn gap. We thus propose the wrthogonalggradien update to fix this gap and instantiate practical algorithm.
2.1 PRELIMINARIES
We consider the RL problem presented as a Markov decision process (Sutton et al. 1998), which specified by tuple M(S, A. P. po: 7) consisting of state space, an action space, transition probability function, reward function, an initial state distribution, and the discount factor. The goal of RI. is find a policy x (aas)) that maximizes expected return J(T) == EI r at while
Sr satisfy ag T( T--1st) and 8t+1 P(+s),,a)) ae). [(T) has equivalent dual form PPuterman 2014) that can be written as (T) = (ss,)) [r(s, a)], where d' (s, a) = (1 >i= 2 Pr(s= = 8, Gr = a) is the discounted state-action visitation distribution. In this work, we mainly focus on the offline setting, where learning from static dataset D = (8, (i Ti. 8 1 is called offline RL and learning from a static dataset without reward labels D = { ag s 1 is called offline IL. The dataset can be heterogenous and suboptimal, we denote the empirical behavior policy of DD as ap, which represents the conditional distribution p(als) observed in the dataset, and denote the empirical discounted state-action visitation distribution of Tp as d.D
DICE algorithms (Nachum & Dai 2020) are motivated by bypassing unknown on-policy sam- ples from d' in the dual form of J (T). They incorporate [(T) with behavior constraint term Dy(d"I1dd) = ((s.a)-a [f( d" (8.a)
(s,a) } where f(x) is convex function, i.., finding x² satisfying:
= arg max max (e [r(8, a)] aD,(d"1ld))
This objective is still intractable, however, by adding the Bellman-slow constraint (Manne 1960). i.e., d(s,a)) (1) = =(1 (1=2)d((s)x(a())) + +7 Eer d(s', ('s',a')pn ")p(sss'aa'><(()
(on state-action space) or
EnEA d(s, a) = (1 ((1-)dda (1=))do(s)+ + Eirrary d(s', a" (p(sss',a') a') (on state space), we can get two cor- SeceAd(s,aa
responding dual forms which are tractable (by applying duality and convex conjugate, see Appendix fooo details).
Q-DICE
(s))s max min 7) Bad (2(1] a)l (f - Q(s, /a)]
V-DICE
min (1 7) [V()] + co [f o
(2)
where f* is the (variant of) convex conjugate of f. do the distribution of initial state po. in- crease the diversity of samples, one often extends it to dd by treating every state in a trajectory as initial state (Kostrikov et al. 2020). T (=Q(s, a) r(s, a) + "p
[Q(s', a')) and TV(s, a) r(s, a) + + [V(')] represent the Bellman operator on Q and V. respec- tively. In the offline setting, since D typically does not contain all possible transitions (s. a, s'), one actually uses an empirical Bellman operator that only backs up a single s' sample, we denote the corresponding operator as 7 and 7.
Crucially, these two objectives rely only on access samples from offline dataset D. Note that in both cases for Q-DICE and V-DICE, the optimal solution is the same as their primal formulations due to strong duality. For simplicity, we consider V-DICE as an example for the rest derivation and discussion in our paper, note that similar derivation can be readily conducted for Q-DICE as well.
2.2 DECOMPOSE GRADIENT FLOW OF DICE
In the learning objective of V-DICE, we consider a parameterized value function V with network parameters denoted as 0. In objective 2). the effeett of the linear term is clear that it pushes down the V-value of state samples equally. Whereas the effect of the nonlinear term which contains the Bellman residual term (i.e., TV a) V(s)) is unclear because both V(s') and -V(s) would contribute the gradient. We thus are interested in decomposing and investigating each part of the second term, the gradient flow could be formally written as:
Vos (TV((T(( V()))) = (r* + 7V(6') - ((s))
VOV(s'
VoV(s)
(3)
backward gradient
forward gradient
3

---

Published as a conference paper at ICLR 2024
where we consider 1 for simplicity. Here denote the two parts of the gradient "forward gradient" (on s) and "backward gradient" (on s') respectively, as shown in Eq.33, A keen reader may note that Eq. looks like algorithms in online RL that use true true- gradiend update (Barrd [995). True-gradient algorithms are known to enjoy convergence guarantees to local minima under off- policy training with any function approximator. However, in online RL., most famous deep RI. algorithms use semi- mradigradient update, i.e., applying target network on (s') to freeze the backward gradient, such as DQN (Mnih et al., 2013), TD3 (Fujimoto al., 2018 and SAC (Haarnoja al., 2017). If we modify Eq. to use semi-greee -naiieedeeett update, we have
-V))(( - -)
VoV(6)
backward gradient
grudient
where Q(s, a) denotes stoopgraddeente + +s)))) Perhaps surprisingly, we find the result in Eq. bears large similarity to the gradient flow of one recent SOTA offline RL algorithm, Exponen- tial Q-Learning (EQL, Xu et al. (2023)), which imposes an implicit action-level behavior constraint (Reverse KL. between ı and TD) with the following learning objective (with in EQL equals to 1):
min ((sD))D lexp (Q(s, a) - Vss) + [s)]
(5)
where Q is learned to optimize E ((s,a,))>D(((( a) + vV ) - ((sss)))))) we take gradient of objective (5) with respect to V. we can get exactly (4.44) (with f(z) = z log x)! This tells us that using the forward gradient alone is equal to imposing only action-level constraint. Our finding is consistent with one recent work (Sikchi et al. 2023) that finds V-DICE using semi-gradient update doing implicit policy improvement to find the best action in the dataset D.
EQL has shown superior performance on D4RL benchmark tasks, however, OptiDICE (Lee et al, 2021, whose learning objective is exactly Eq. shows significantly worse performance compared to EQL. It's obvious that the only difference between semi-DICE algorithm (EQL) and true-DICE algorithm (OptiDICH) is the backward gradient. Then gap between theory and practice occurs: although the backward gradient term is considered to be necessary from the theoretical derivation, the practical result reveals that is harmful and unnecessary.
2.3 FIX THE GAP BY ORTHOGONAL-GRADIO
UPDATE
As said, the forward gradient is crucial for implicit maximization to find the best action. However, adding the backward gradient will probably interfere with it because a state s and its successor state s' are often similar under function approximation. The backward gradient may cancel out the effect of the forward gradient, leading to a catastrophic unlearning phenomenon. To remedy this, we adopt simple idea that projects the backward gradient onto the normal plane of the forward gradient, making these two gradient terms orthogonal. The projected backward gradient can be written as
V = Vo)(s)- VoV(s) TOV(s)TVoLL VoV(8' (oV(s) [
Intuitively, the projected backward gradient will not interfere with the forward gradient while still retaining information from the backward gradient. We will further show in the next section that the projected backward gradient is actually doing state-level constraint. After getting the projected backward gradient, we add it to the forward gradient term, resulting new gradient flow,
Vos (+V(s, V(+)) = (r*) xV()') V(s)) (7 VVV - (oV(s)
where we use one hyperparameter 7 > to control the strength of the projected backward gradient against the forward gradient. We call this new update rule as orthogonal gradient update and give the illustration of in Figure We can see that the orthogonal-gradien can be balanced between true-gradient and semi-gradiem by choosing different 17.
After adding the gradient of the first linear term in objective 2 we can get the new full gradient flow of learning (2). We then propose new offline algorithm, O-DICE (Orthogonal-DiCE) that incor- porates v V-DICE algorithm with orthogonal-grann -radient update. The policy extraction part in O-DICE is the same as V-DICE by using weighted behavior cloning max, ((s.)) (w" (s.a) log (a]ss)),
where (s, a) := d= (x.a) max (0)) (s.)) (s))).
(=(a,))
4

---

Published as a conference paper at ICLR 2024
True-gradeet
V2V(5)
Practical Consideration We now describe some practical considerations in O-DICE. O- DICE could be implemented with just few adjustments to previous true/semi-DICF algo- rithms. To compute the forward and backward gradient, we apply target network to Vss") when computing the forward gradient and ap- ply target network to (s) when computing the backward gradient, a.kaa, the bidirectional tar- get network trick (Zhang et all (2019b). We also apply one trick from (Sikchi et al. (2023) that rewrites objective ( from to À as
Orthogonat- graadlnn
VoV(s)
(
Semi-gradient
(s)V(s)
Figure I: Illustration of aathhooonal-grrdiiien update
Algorithm Orthogonal DICE
Require: D, A, v), learning rate
I: Initialize Vo. Vg. TT
for t= 1, 2... do
min EE [(1 +
V())
Sample transitions a, s') ~ 'D
V
4: Calculate forward gradient
1) trades off linearly between the first term and the second term. This trick makes hyperparameter tuning easier as has a nonlinear dependence through the function f. Analogous to previous DICE algorithms, we alternate the updates of V and until con- vergence although the training of them are de- coupled. Note that different from almost all previous offline RL algorithms, we find that O-DICE does not need to use the double Q- learning trick (Fujimoto et al.) 2018). which suggests that using orthogonal-gradien update regularizes the value function better and greatly enhances training stability. We summarize the pseudo-code of O-DICE in Algorithm [
where A 10
Val V
7 VVa(s') Vo(s))
5:
Calculate backward gradient
9e Val (r VV(ss' Vy(())
6:
Calculate projected backward gradient
g = (g Os
190 Da
7:
Update 0 by
0 2 ((1- (V-V + A(g0" + +ngb))
Calculate BC weight
wss max (0, (f') (r (V+')o vo())))
Update अH by maxe [wi(s, a) log n (a|s))
10:
Update @ with exponential moving average
0 -0 + (1 -)0
11:
end for
ANALYSIS
Although our method is simple and straightforward, in this section, we give thorough theoretical analysis and use didactic toy example to show several benefits of using orthogonal -gradient update.
3.1 THEORETIC RESULTS
We hegin by introducing the notation used in our analysis. We denote the nonlinear loss O- DICE, (r(r + (V()') V(s)) as L (s, We use [s)s) to denote f" + >V('') -s))), whose gradient corresponds to the forward gradient (multiply a value). We use [g(ss)) denote ["
(+)), whose gradient corresponds to the backward gradient (multiply a value). We define the projected part and the remaining parallel part of V Lo with respect to VoL
as V (s') and V (s'),, respectively. It's easy to know that we can decompose the orthogonal- gradient update in O-DICE into two steps, which are
A' a (s) 8" 8 a V Lg(s))
where is the learning rate. We first introduce two theoretical benefits of using the orthogonal- gradient update from the optimization perspective.
Theorem 1. In orthogonal-graddeen update, L (8) L (s) = -0 (under first order approximation).
This theorem formally says the eiterference-fre property in orthogonal-gradice update, that is, the gradient change in loss Lz (s') won't 't affect loss L, (s). This means that aatooogona--rradienn update preserves the benefit of semi graddetr update while still making loss Lz(s') decrease. Note that under general setting, _ (s) L 0 holds only by orthogonal-gradien update, any other gradient modification techniques (Baird
1995 Zhang et al. 2019b] Durugkar & Stone) 2018) that
keep part of V!! r. (s') will make T.q"
1
5

---

Published as a conference paper at ICLR 2024
Theorem 2. Define the angle between (oV(s) and VoV(s') as o(s, In orthogonal saatogggna-ggddiien (cos@(s,s') s') [[VaV( >(VVo(() - [VVVV
update, if Ø(s, # O, then for all n > sin² o(e,e")
-(llooV(e>>)) we
have - (s. (s s') <0.
This theorem gives convergence property that using orthogonal--raddin update could make the overall loss objective monotonically decrease. Note that similar convergence guarantee occurs only in true-gradien update, while semi-gradient update is known to suffer from divergence issue (Zhang et al. 2019b). Actually, to have a convergence guarantee, we only need to keep the gradient within the same half-plane that VoL s') lies in. This means that can adjust 1) to make the orthogonal gradient keep in the same half-plane of the true- rue-ggradeet while keeping close to the semi-gradee gradiegadientt needed, as shown in this theorem. Note that and s' are often similar, the value of [VVo [+)]l
and |l (s')l are close to each other and the ratio [oo[( will probably > 1, which indicates vol'')
that the choice of may be fairly easy to satisfy this theorem (e.g.. 0).
From Theorem and Theorem we know that orthogonal-gradien and (s) decrease simultaneously, so as to share the best of both worlds: not only does it preserve the lctiln-level constraint semi cami-gradient update, but also owns the convergence guarantee in true- cerssonalgrrdient gradient update, we show in
update could make loss L(s, s')
gradient update. We now further analyze the influence of orthogonal the following theorem that it has a deep connection with the feature dot product of consecutive states in value function (Vo(s, s') = (s) (sV(s)).
update helps alleviate feature co-adaptation). Assume the norm of (s)(s) is bounded, i.e. Vs, m I [[))V < M. Define consecutive value curvature 8(0,8, 8") = (&V(s) V where H(s) is the Hessian matrix of V(8). Assume E(0,8, )') 21. (s'')1 where >> 0. Then in orthogonal aaatoggonn-grddiinn update, we have
Theorem 3 (Orthogonal-gratee
(8, Vor 8) -om ( - (ss))(sin 6(s, 8)) .1 mm + 3 M] (6)
where 33 is a constant close to 0 if the condition number of H (s) is small (Cheney & Kincaid 2012).
Feature co-adaptation, where V (o 8') is large so that features of different states learned by the value function become very similar, is known to be a severe issue in offline deep RL (Kumar et al.) 2021 While this theorem formally says that after using wrthogonal-gradicnn
update, the difference of V Vo s') could be bounded. In RHS of Eq. (6. 33 is a constant that captures the difference between minimal and maximal eigenvalue in H (s). ,3 will be close to 0 if neural network V(s) is numerically well-behaved (i.e., the condition number of (s) is small). which can be achieved by using some practical training tricks such as orthogonal regularization (Brock et al. 2016) or gradient penalty Gulrajani et al. 2017) during training V. Also, note that (f*)(z) > 0 always holds because f* is chosen to be monotonically increasing (see proof in Appendix B for details, we highlight that this only holds in DICE- based methods. minimizing Bellman equation such that f = x² won't 't let hold). Hence, 6q. < will probably hold, this means that using orthogonal-gradien update could help alleviate feature co-adaptation issue, learning better representations for different states. We show in the following theorem that better representation brings better state-level robustness of (s),, akin to imposing the state-level behavior constraint.
Theorem 4 (How feature co-adaptation affects state-level robustness). Under linear setting when (s)) can be expressed as VsV(s) 0, assume ( < M, then there exists some small noise E | such that (ss' + +) Vss) will have different sign with V(s') V(),, if Vo(s,s') >
M- - C- -|| for some constant C > O.
This theorem says that if feature co-adaptation happens, i.e., Vo 8') is larger than a threshold, a small noise adding to V(s') will largely change its value, or the learning signal for policy. If the sign (or value) of (s*) - (s) can be easily flipped by some noise e, not only the policy extraction part during training will be affected (as two similar s' will have totally different BC weight), but also the optimal policy induced from V is likely to be misled by OOD states encountered during evaluation and take wrong actions. As shown in Theorem 3, using orthogonal -gradieet update helps alleviate feature co- adaapttaiion and enables V to learn better representations to distinguish different states, thus being more robust to OOD states.

---

Published as a conference paper at ICLR 2024
300
300
(a) Reward distribution
(b) True-gradient V
(c) Semi-gradient V
(d) Orthogonal- -gradieen V
Figure Visualizations of the value function V learned by using different updating rules grid world toycase, the base algorithm is V-DICE, normalize the value of to [0. 100) for fair comparison, The borderline the offline dataset support is marked by white dashed lines. Orthogonal-gradicn update better distinguishes both different in-distribution actions and in-distribution states vs. OOD states.
3.2 A TOY EXAMPLE
We now use a toy example to validate the above-mentioned benefits using orthogonal-gradient update. We consider 30 30 grid world navigation task where the agent starts from the bottom- left corner (0, 0) and the goal is to reach a given area at the top ((10, 30) to (20, 30)) or right border ((30, 10) to (30, 20)). The action space contains 4 discrete actions: up, down, left, and right. Every time the agent chooses an action, it will move towards that direction for one grid, receiving -1 reward, unless it reaches the goal area, which will give different, positive reward, as shown in Figure [[a). The offline dataset is collected by running a goal-reaching policy, to add some stochasticity, with probability of 0.05 the output action will be shifted random action. We use white dashed lines in Figure to plot the borderlines of the support of the collected dataset.
We run V-DICE with semi- remi-gradient, true- guue-gradent, and aatoogonal-graieem. and visualize the learned V-value (with normalization) in Figure Figure@()-(d)), (d). The corresponding policy can be extracted implicitly by choosing reachable state (grid) with the highest V-value, a given state (grid). can be shown from the result that orthogonal- -rrddieng update could learn value function (or policy) that successfully finds a path from the start to the desired location, indicating the effect of impos- ing action-level constraint. Meanwhile, it successfully distinguishes between in-distribution states and OOD states, the values of OOD states are much lower than that of in-distribution states, which reveals better feature of the value function is learned imposing state-level constraint. Using semi- gradient update could learn a good policy to find the best action but lose OOD state robustness (the color of OOD states and in-distribution states similar, once the agent gets into OOD states, can hardly come back), while true-gradee tree-gradeent update could not even learn a good policy because of the conflicting gradient issue.
From this toycase, we can clearly observe that using orthogonal-gradicne update brings state-action- level constraint, it helps improve the robustness of OOD states while still preserving the ability to find the best dataset actions.
EXPERIMENTS
In this section, we present empirical evaluations of O-DICE. We first evaluate O-DICE against other baseline algorithms on benchmark offline RL datasets. We then conduct more in-depth investigations on the policy robustness O-DICE and other algorithms. Lastly, we evaluate O-DICE on offline IL. tasks to demonstrate its superior performance. Hyperparameters and experimental details are listed Appendix D
4.1 RESULTS ON D4RL BENCHMARKS
We first evaluate O-DICE on the D4RL benchmark (Fu et al.) 2020), including MuJoCo locomotion tasks and AntMaze navigation tasks. We compare O-DICE with previous SOTA baselines that only enforce action-level constraint. including TD3+BC (Fujimoto & Gu 202T), CQL (Kumar et al, 2020), IQL. (Kostrikov et al. 2021b), and EQL (Xu al. 2023). We also compare O-DICE with

---

Published as a conference paper at ICLR 2024
Table 1: Averaged normalized of O-DICE against other baselines. The scores are taken the final 10 evaluations with seeds. O-DICE achieves the highest score 13 out of tasks, outperforming previous SOTA offline RL methods and other DICE-based methods.
construin s(als)
constrain d' (s)
D4RT. Dataset
TD3+BC
CQL
IQL
EQL
OptiDICE
F-DVL
S-DICE
O-DICE
bulfcheelah-m
48.3
44.0
47.4 10.2
47.2
10.2
45.8 10.4
47.7
47.3
47.4.00.2
hopper-m
59.3
58.5
66.3 +5.7
74.6
#334
46.4 #3.9
63.0
61.5+2.3
86.1 +4.0
walker2d-m
83.7
72.5
72.5 18.7
83.2 1:4.6
68.1 1:4.5
80.0
78.617..1
84.9 1.2.3
hulfcheelah-m--r
44.6
45.5
44.2 11.2
44.5 10077
31.7 10.8
42.9
40.6.12.6
44.0.10.3
hopper-m-r
60.9
95.0
95.2 4:8.
98.1
20.0 =6.6
90.7
64.6
99.9 +:2.7
walker2d-m-r
81.8
77.2
76.1 1.7.3
76.6 ..3.8
17.9.5.9
52.1
37.3.5.11
83.6 1:4,1
halfcheetah-m--
90.7
90.7
86.7 1:5.3
90.6 10044
59.7 1:300
89.3
92.8.0.77
93.2 10.6
hopper-m-e
98.0
105.4
101.5 #7.3
105.5 #:2.2
51.3 #3.5
105.8
108.4199
110.8 +0.6
walker2d-m-e
110.1
109.6
110.6 -1.00
110,2 1.0.8
104.0 1.5.1
110.1
110.1.10.2
110.8 1.0.2
antmaze-u
78.6
84.8
85.5 +11.
93.2 +1.4
56.0 +2.8
83.7
88.6+22.5
94.1 +1.6
antmaze-u-d
71.4
43.4
66.7 14.0
65.0 42.3
48.6 17.4
50.4
69.315.00
79.5 13.3
antmaze-m-p
10.6
65.2
72.2 +5.3
77.5 +3.7
0.0 +0.0
56.7
50.6+11.1
86.0
+1.6
antmaze-m-d
3.0
54.0
71.0 +3.2
70.0 +4,2
0.0++0.0
48.2
66.0+3.66
82.7 +4,9
antmaze-l-p
0.2
38.4
39.6 14.5
45.6 14.8
0.0
36.0
34.019.0
55.9 +3.9
antmaze-I-d
0.0
31.6
47.5 +4.4
42.5 +5.22
0.0 +0.0
44.5
32.6
54.0 +448
other DICE-based algorithms that intend to do state-action-lvve constraint, including OptiDICE (Lee al., 202T). f-DVL (Sikchi et al., 2023) and our implementation of semi-DICE (S-DICE) which updates V-DICE using gradient update.
The result shows that O-DICE outperforms all previous SOTA algorithms on almost all MuJoCo and AntMaze tasks. This suggests that O-DICE can effectively learn strong policy from the dataset, even challenging AntMaze tasks that require learning a high-quality value function so as to "stitch" good trajectories. The large performance gap between O-DICE (using orthogonal- gradient update) and OptiDICE (using true-gradient update) demonstrates the necessity using the projected backward gradient. Moreover, O-DICE shows consistently better performance over EQL, f-DVL, and S-DICE (using semi momi-gradient update) on almost all datasets, especially on sub-optimal datasets that contain less few near-optimal trajectories. This indicates the superiority of state- action-level constraint over only action-level constraint, by learning a value function with better representation to successfully distinguish good trajectories (actions) from bad ones.
4.2 RESULTS ON POLICY ROBUSTNESS
The aggregated mean score in D4RI. can't fully reflect the robustness an algorithm because, even if the average performance reasonable, the agent may still perform poorly some episodes. Here com- pare O-DICE with S-DICE and EQL on the percent difference of the worst episode during the 10 evaluation episodes at the last evaluation (Fujimoto Gul (021]. we also plot the average value of feature dot product, (oV(s) VoV(ss), as shown Figure 3 It's obvious that robust pol- icy should achieve consistently high per-
SOICE
Figure 3: Left: percent difference of the worst episode during the 10 evaluation episodes at the last evaluation of different offline RL algorithms. Right: mean value of (oV(s) VoV(s') (s') over the dataset of S-DICE and O-DICE.
formance over multiple evaluations, i.e., having small percent difference. Moreover, Theorem 4 shows that a low value of (s)(s) VsV(s') is necessary for robustness. can be observed that O-DICE achieves much smaller percent differences than action-constraiied offline RL methods like EQL and S-DICE, which empirically verifies Theorem 4. The significantly decreased feature dot- product value V VoV(s) (s') after applying orthogonal-gradie also agrees with Theorem 3.
8

---

Published as a conference paper at ICLR 2024
4.3 RESULTS ON OFFLINE IMITATION LEARNING
We then evaluate O-DICE in the offline imi- tation learning setting, where only one expert trajectory is given (Kostrikov et al.) 2020). In this setting, OOD states are fairly easy oc- cur, and being robust to them is crucial for high performance (Rajaraman et al. al., 20220) We com- pare O-DICE with BC (Pomerleau 1989), Val- ueDICE (ostrrkov et al.) 2020) and IQLearn Garg et al. (2021). ValueDICE and IQLearn two Q-DICE algorithms that use true-gradient update and semi-gradient update, respectively. As find performance varies lot by using dif- ferent trajectories, we select four trajectories from D4RL, expert datasets, each containing 1000 transitions. It can be shown in Table 2 that although all algorithms can't match the expert perfor- mance due to extremely limited data, O-DICE consistently performs better than all other baselines. This justifies the benefits of using cethogonal-gradien update, it enables better state-level robustness against OOD states by learning better feature representation.
Table 2: Normalized return of BC, ValueDICE,
IQLearn and O-DICE on offline IL tasks.
Datnset
BC
ValueDICR
IQL
O-DICE
Tra
56.0
33.8
75.1
78.8
Hopper
Tra
47.8
Traj
50.5
24.7
47.1
55,1
Tra
87.2
90.4
98.5
Tra]
64.6
23.6
50.2
75.6
Wxxker2d-expert
Tra 43.3
17.1
40.5
65.8
Traj
Traj
33.0
40.5
44.2
61.6
RELATED WORK
Offiine RL
To tackle the distributional shift problem, most model-free offline RL methods aug- ment existing off-policy RL methods with an action-level behavior regularization term. Action-level
regularization can appear explicitly as divergence penalties (Wu et al. 2019 Kumar et al. 2019
Xu et al. 2021 Fujimoto & Gu, 2021: Cheng et al. 20233 Li et al. 2023b), implicitly through weighted behavior cloning (Wang et al. 2020, Nair et al. 2020), or more directly through careful parameterization of the policy (Fujimoto et al. 2019 Zhou et al. (020). Another way to apply action-level regularization is via modification of regularization, to encourage staying near the behavioral distribution and being pessimistic about OOD state-action pairs (Kumar et al., 2020, Kostrikov et al,, 2021a Xu et al. 2022c 2023 Wang et al. 2023] Niu et al., 2022). There are also several works incorporating action-level regulariza- tion through the use uncertainty (An et al.) 2021; Bai et al. 2021) or distance function (Li et al, 2022ab. Another line of methods, the contrary, imposes action-level regularization by performing some kind of imitation learning on the dataset. When the dataset is good enough or contains high- performing trajectories,
value learning objective to incorporate some form
can simply clone filter dataset actions extract useful transitions (Xu ct al. 2022b Chen et al. 22200; Zhang et al., [203a]] Zheng et al., 2024), or directly filter individual transitions based on how advantageous they could be under the behavior policy and then clones them Brandfonbrener et al. [222 Xu et al. 2022a). There are also some attempts to include acttin
level behavior regularization (Li al. 2022b; Zhang et al.) 2202)) however, they typically require
costly extra steps of model-based OOD state detection (Li et al. 2022b Zhang et al. 2022).
DICE-based Methods DICE-based methods perform stationary distribution estimation, and many of them have been proposed for off-policy evaluation: DualDICE (Nachum et al.) 2019a), GenDICE (Zhagg et al., (019a)), GradientDICE (Zhang et al., (020). Other lines of works con-
sider reinforcement learning: AlgaeDICE (Nachum et al.) (019b), OptiDICE (Lee et al. 202
CoptiDICE (Lee et al. 2022), f-DVL (Sikchi et al. 2023); offline policy selection: (Yang et al,
2020); offline imitation learning: ValueDICE (Kostrikoy al. 2020), OPOLO (Zhu et al. 2020
IQlearn (Gagg et al., 2021), DemoDICE (Kim et al. 2021), SmoDICE (Ma et al. 2022): reward
learning: RGM (Li et al. | (223a) All of these DICE methods either using true- rueeggraieet update or semi-gradient update, while our paper provides a new update rule: uuthogonal-gradicnn update.
6 CONCLUSIONS
In this paper, we revisited DICE-based method. We provide new understanding of DICE-based method and successfully unravel its mystery: the value learning objective in DICE does try to impose state-action-level constraint, but needs to be used in conrected way, that is orthogonal-gradie update. By doing so, Dase-based methods enjoy not only strong theoretical guarantees, but also favorable empirical performance. It achieves SOTA results both offline RI. and offline II tasks.
9

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENT
This work supported by National Key Research and Development Program of China under Grant (2022YFB2502904) The SJTU team is partially supported by Shanghai Municipal Science and Technology Major Project (202ISHZDZX0102) and National Natural Science Foundation of China (62322603, 62076161). We thank Amy Zhang and Bo Dai for insightful discussions.
REFERENCES
Gaon An, Seungyong Moon, Jang-Hyun Kim, and Hyun Oh Song. Uncertainty-based offline rein- forcement learning with diversified q-ensemble. Proc. of NeurlPS, 2021.
Lei Jimmy Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. abs/16.0,06450, 2016.
Layer normalization. CoRR,
Chenjia Bai, Lingxiao Wang, Zhuoran Yang. Zhi-Hong Deng, Animesh Garg, Peng Liu, and Zhao- ran Wang. Pessimistic bootstrapping for uncertainty-driven offline reinforcement learning, In Proc. of ICLR, 2021.
Leemon Baird. Residual algorithms: Reinforcement learning with function approximation. Machine Learning Proceedings 1995, pp. 30-37, 1995.
Stephen Boyd, Stephen P Boyd, and Lieven Vandenberghe. Convex optimization. Cambridge uni-
versity press, 2004.
David Brandfonbrener, William Whitney, Rajesh Ranganath, and Joan Bruna. Offline without off-policy evaluation. Proc. of NeurIPS, 2021.
Andrew Brock, Theodore Lim, James M Ritchie, and Nick Weston. Neural photo editing with introspective adversarial networks. arXiv preprint UrXiv:1609.07095 2016.
Xinyue Chen, Zijian Zhou, Zheng Wang, Che Wang, Yanqiu Wu, and Keith W. Ross. BAIL: best- action imitation learning for batch deep reinforcement learning. In Proc. of NeurlPS, 2020.
E Ward Cheney and David R Kincaid. Numerical mathematics and computing. Cengage Learning, 2012.
Peng Cheng, Xianyuan Zhan, Zhihao Wu, Wenjia Shoucheng Song, Han Wang, Youfang Lin, and Li Jiang. Look beneath the surface: Exploiting fundamental symmetry for sample-efficient offline rl. In sevethh Conference on Neural Information Processing Systems, 2023,
Ishan Durugkar and Peter Stone. Td learning with constrained gradients. 2018.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep
data-driven reinforcement learning. ArXiv preprint, 2020.
Scott Fujimoto and Shixiang Shane Gu. minimalist approach offline reinforcement learning. ArXiv preprint, 2021.
Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation enror in actor-critic methods. Proc. of ICMI,, pp. 1582-1591. 2018.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In Proc. of ICML, pp. 2052-2062, 2019.
Divyansh Garg, Shuvam Chakraborty, Chris Cundy, Jiaming Song, and Stefano Ermon. Iq-learn: Inverse soft-q learning for imitation. Advances in Neural Information Processing Systems, 34: 4028 4039, 2021.
Divyansh Garg, Jocy Hejna, Matthieu Geist, and Stefano Ermon. Extreme lelarrnin:: Maxent rl without entropy. In Proc. of ICLR, 2023.
Ishaan Gulrajani, Faruk Ahmed, Martín Arjovsky, Vincent Dumoulin, and Aaron C. Courville, Im- proved training of wasserstein gans. In Proc. of NeurlPS, pp. 5767-5777, 2017.
10

---

Published as a conference paper at ICLR 2024
Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep energy-based policies. In Proc. of ICML, pp. 1352-1361, 2017.
Dmitry Kalashnikov, Jacob Varley, Yevgen Chebotar, Benjamin Swanson, Rico Jonschkowski, Chelsea Finn, Sergey Levine, and Karol Hausman. Mt-opt: Continuous multi-task robotic inforcement learning at scale. ArXiv preprint, 2021.
Geon-Hyeong Kim, Scokin Sco, Jongmin Lce, Wonseok Jeon, HycongJoo Hwang, Hongseok Yang, and Kee-Eung Kim. Demodice: Offline imitation learning with supplementary imperfect demon- strations. In Proc. of ICLR, 2021.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Proc. of ICLR, 2015.
Ilya Kostrikov, Ofir Nachum, and Jonathan Tompson. Imitation learning via off off-policy distribution matching. In Proc. of ICLR, 2020.
Ilya Kostrikov, Rob Fergus, Jonathan Tompson, and Ofir Nachum. Offline reinforcement learning with fisher divergence critic regularization. In Proc. of ICML, pp. 5783, 2021a.
lly Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q- learning. ArXiv preprint, 2021b.
Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy q-learming via bootstrapping error reduction. In Proc. of NeurlPS, pp. 11761-11771, 2019. 2019.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. In Proc. of NeurlPS, 2020.
Aviral Kumar, Rishabh Agarwal, Tengyu Ma, Aaron Courville, George Tucker, and Sergey Levine.
Dr3: Value-based deep reinforcement learning requires explicit regularization. arXiv preprint arXiv:211..04716, 2021.
Aviral Kumar, Rishabh Agarwal, Tengyu Ma, Aaron C. Courville, George Tucker, and Sergey Levine. DR3: value-based deep reinforcement learning requires explicit regularization. In ICLR. OpenReview.net, 2022.
Jongmin Lee, Wonseok Jeon, Byungjun Lee, Joelle Pineau, and Kee-Eung Kim. Optidice: Offline policy optimization via stationary distribution correction estimation. In Proc. of ICML, pp. 6120- 6130, 2021.
Jongmin Lee, Cosmin Paduraru, Daniel J Mankowitz, Nicolas Heess, Doina Precup, Kee-Eung Kim, and Arthur Guez. Coptidice: Offline constrained reinforcement learning via stationary distribution correction estimation. arXiv preprint arXiv:2204.089557 2022.
Jianxiong Li, Xianyuan Zhan, Haoran Xu, Xiangyu Zhu, Jingjing Liu, and Ya-Qin Zhang. When data geometry meets deep function: Generalizing offline reinforcement learning. In The Eleventh International Conference on Learning Representations, 2022a.
Jianxiong Li. Xiao Hu, Haoran Xu, Jingjing Liu, Xianyuan Zhan, Qing-Shan Jia, and Ya Ya-Qin Zhang. Mind the gap: Offline policy optimization for imperfect rewards. arXiv preprint arXiv:2302.01666 2023a.
Jianxiong Li, Xiao Hu, Haoran Xu, Jingjing Liu, Xianyuan Zhan, and Ya- Ya-Qin Zhang. Proto: Iterative policy regularized offline-to-online reinforcement leamning arXiv preprint 2tti::3355.5633 2023b.
Jinning Li, Chen Tang, Masayoshi Tomizuka, and Wei Zhan. Dealing with the unknown: Pessimistic
offline reinforcement learning. In Conference on Robot Learning, pp. 1455-1464, PMLR, 2022b.
Yecheng Jason Ma, Andrew Shen, Dinesh Jayaraman, and Osbert Bastani. Smodice: Versatile
offline imitation learning via state occupancy matching. arXiv preprint arXiv:2202.02433, 2022.
Alan S Manne. Linear programming and sequential decisions. Management Science, 6(3):259-267 1960.
11

---

Published as a conference paper at ICLR 2024
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, loannis Antonoglou, Daan Wier- stra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602, 2013.
Ofir Nachum and Bo Dai. Reinforcement learning via fenchel-rockafellar duality, arXiv preprint arXiv:2001.01866, 2020.
Ofir Nachum, Yinlam Chow, Dai, and Lihong Li. Dualdice: Behavior-agnostic estimation of discounted stationary distribution corrections. In Proc. of NeurlPS, pp. 2315-2325, 2019a.
Ofir Nachum, Bo Dai, Πya Kostrikov, Yinlam Chow, Lihong Li, and Dale Schuurmans. Algaedice: Policy gradient from arbitrary experience. ArXiv preprint, 2019b.
Ashvin Nair, Murtaza Dalal, Abhishek Gupta, and Sergey Levine. Accelerating online reinforcement learning with offline datasets. ArXiv preprint, 2020.
Haoyi Niu, Yiwen Qiu, Ming Li, Guyue Zhou, Jianming HU, Xianyuan Zhan, et al. When trust your simulator: Dynamics-aware hybrid offline-and-onliee reinforcement learning. Advances in Neural Information Processing Systems, 35:36599-366 066599,,366,,,- 2022.
Dean A Pomerleau. Alvinn: An autonomous land vehicle neural network. In Proc. of NeurlPS, 1989.
Martin L Puterman. Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons, 2014.
Nived Rajaraman, Lin Yang, Jiantao Jiao, and Kannan Ranchandran. Toward the fundamental limits of imitation learning. Advances in Neural Information Processing Systems, 33:291 14-2424, 2020.
Harshit Sikchi, Amy Zhang, and Scott Nickum. Imitation from arbitrary experience: A dual unifi- cation of reinforcement and imitation learning methods. arXiv preprint arXiv:2302.08550, 2023.
Richard S Sutton, Andrew G Barto, et al. Introduction to reinforcement learning- MIT press Cam- bridge, 1998.
Shengpu Tang and Jenna Wiens. Model selection for offline reinforcement learning: Practical con- siderations for healthcare settings. In Proc. of ML4H, 2021.
Xiangsen Wang, Haoran Xu, Yinan Zheng, and Xianyuan Zhan. Offline multi-agent reinforcement learning with implicit global-to-local value regularization. In Thirty-seventth Conference Neu- ral Information Processing Systems, 2023.
Ziyu Wang, Alexander Novikov, Konrad Zolna, Josh Merel, Jost Tobias Springenberg, Scott E. Reed, Bobak Shahriari, Noah Y. Siegel, Çaglar Gülçehre, Nicolas Heess, and Nando de Freitas. Critic regularized regression. In Proc. of NeurlPS, 2020.
Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning- ArXiv preprint, 2019.
Haoran Xu, Xianyuan Zhan, Jianxiong Li, and Honglei Yin. Offline reinforcement learning with soft behavior regularization. ArXiv preprint, 2021.
Haoran Xu, Li Jiang, Li Jianxiong, and Xianyuan Zhan. policy-guided imitation approach for of- fline reinforcement learning. In Advances in Neural Information Processing Systems, volume 35, pp. 4085 4098, 2022a.
Haoran Xu, Xianyuan Zhan, Honglei Yin, and Huiling Qin. Discriminator-weighhee offline imitation learning from suboptimal demonstrations. In International Conference on Machine Learning, pp- 24725-24742. PMLR, 2022b.
Haoran Xu, Xianyuan Zhan, and Xiangyu Zhu. Constraints penalized q-learning for safe offline rein- forcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 8753-8760, 2022c.
12

---

Published as a conference paper at ICLR 2024
Haoran Xu, Li Jiang, Jianxiong Li, Zhuoran Yang, Zhaoran Wang, Victor Wai Kin Chan, and Xi- anyuan Zhan. Offline rl with no ood actions: In-sample learning via implicit value regularization. arXiv preprint (rx.h::3333.88000 2023.
Jingkang Yang, Kaiyang Zhou, Yixuan Li, and Ziwei Liu. Generalized out-of-distribution detection: A survey. arXiv preprint arXiv:21 10. 1134, 2021.
Mengjiao Yang, Ofir Nachum, Bo Dai, Lihong Li, and Dale Schuurmans. Off-policy evaluation via the regularized lagrangian. Advances in Neural Information Processing Systems, 33:6551-6561, 2020.
Xianyuan Zhan, Haoran Xu, Yue Zhang, Xiangyu Zhu, Honglei Yin, and Yu Zheng. Deepthermal: Combustion optimization for thermal power generating units using offline reinforcement learning. In Proceedings of the AAAI Conference Artificial Intelligence, volume 36, pp. 4680-4688
2022.
Hongchang Zhang, Jianzhun Shao, Yuhang Jiang, Shuncheng He, Guanwen Zhang, and Xiangyang Ji. State deviation correction for offline reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp.
2022.
Qin Zhang, Linrui Zhang. Haoran Xu, Li Shen, Bowen Wang, Yongzhe Chang, Xueqian Wang, Bo Yuan, and Dacheng Tao. Saformer: conditional sequence modeling approach to offline safe reinforcement learning. arXiv preprint arXiv:2301. 12203, 2023a.
Ruiyi Zhang, Bo Dai, Lihong Li, and Dale Schuurmans. Gendice: Generalized offline estimation of stationary values. In International Conference on Learning Representations, 2019a.
Shangtong Zhang, Wendelin Boehmer, and Shimon Whiteson. Deep residual reinforcement learn-
ing. arXiv preprint 1905 00002, 2019b.
Shangtong Zhang, Bo Liu, and Shimon Whiteson. Gradientdice: Rethinking generalized offline
estimation of stationary values. In International Conference Machine Learning, pp. 11194 11203. PMLR, 2020.
Wenjia Zhang, Haoran Xu, Haoyi Niu, Peng Cheng, Ming Li, Heming Zhang, Guyue Zhou, and Xianyuan Zhan. Discriminator-gudee. model-based offline imitation learning. In Conference on Robot Learning, pp. 1266-1221 PMLR, 2023b.
Yinan Zheng, Jianxiong Li, Dongjie Yu, Yujie Yang. Shengbo Eben Li, Xianyuan Zhan, and Jingjing Liu. Safe offline reinforcement learning with feasibility- -uided diffusion model. In The Twelfth International Conference Learning Representations, 2024.
Wenxuan Zhou, Sujay Bajracharya, and David Held. Latent action space for offline reinforcement learning. In Conference on Robot Learning, 2020.
Zhuangdi Zhu, Kaixiang Lin, Bo Dai, and Jiay Zhou. Off-policy imitation learning from observa- tions. Advances in Neural Information Processing Systems, 33:12402-12413 2020.
13

---

Published as a conference paper at ICLR 2024
A A RECAP OF DISTRIBUTION CORRECTION ESTIMATION
For the sake of completeness, give detailed review of DICE in this section. We won't cover all variants of DICE algorithms but only focus on how to derive the final optimization ob- jective. For more thorough discussion, please refer (Nachum & Dai 2020) and (Sikchi al. 2023
A.1 CONVEX CONJUGATE AND FDIVERGENNCE
Given convex function f IR + I, it's convex conjugate is defined by:
(") suply x f(x), R}
Sometimes proper choice of f make f have close-form solution. For example, f()) (x - 1)² can induce fw( = v + 1).
Moving forward, we'll introduce f f-divergence. Given two probability distributions over Q: P and Q. P is absolutely continuous with respect to Q, the f-divergence from to Q is defined as:
Dy()))Q) = EwEr [1
P(w) ((w)
(8)
A.2 DISTRIBUTION CORRECTION ESTIMATION
DICE algorithms consider RL problems convex programming problems with linear constraints and then apply Fenchel-Rockfeller duality or Lagrangian duality to transform the problem into
constraint- free formulation (Nachum & Dai 2020 Lee et al. 2021). More specifically, consider the following regularized RL problem:
max R (r(s, - (d) (s,
Here Dy(d" (s, (s) a)) a regularizer who encourages the agent to stay close to data support from the perspective of (s, a) occupancy measure. Directly solving =* is impossible because it's intractable to calculate d' a). However, as shown in (Nachum & Dai 2020: Lee et al. 2021), one can consider d another variable d in the optimization problem which satisties Bellman-flow constraints. Then we obtain an optimization problem that's 's simultaneously about d and n:
max aD, (d)(( (s) x.d s.t. d(s,a) = (1-7))6()))) + Σ
(10)
(e',a))
As is pointed out in Nachum & Dai 2020). the original Bellman-flow over-constrains the problem. By summing constraints a, can eliminate :Taas)) in the constraints and get another problem solely depend on d:
maxE d20
- (s,
Σ d(s,a)=(11-1 (1-7)d((s) + Σ d(dd.'"plsss.dd)..ee
(11)
s.t.
REA
(0',0')
Note that this problem we have to set the feasible region to {d Vs S, A, d(s 2 0} because can't rely on the non-negativity of to force d > 0. Applying Lagrangian duality,
14

---

Published as a conference paper at ICLR 2024
get the following optimization target following (Lee et al. 2021
min max E ((..))a(r(ss a)] -aD, (d(s, (ss
((e) d20
Σ ((s) (1-7)do(s)) +y Σ d(s',a ((sss''p(sss',a') -
(o)aa)
min max(1 7) do(ss)
ΣV V(s))= + ((saa)aa((((s
(e) d20
Σ (s) Σ d(s',a')p(sls ('d,d'psssss,a) d( (aboo(())-DD(((ssclllleee
(s'a')
(x)
min max(1 Bdd
V(*) w20
+ E a w(s (1(,,a) + [x(s)o.))(()
E gaadd [f(((s,a)]
(12)
Where w(s, a) represents (a.a) Here we note that we exchange the summation of s and (s', a') in (+) to get the final formulation. As is pointed out in (Sikchi et al. 2023), if we neglect the inner constraint w 2 0, it's obvious that the above problem could be rewritten as:
d(s,a)
min (s) (V)) Br V(*)
(
[ (r(s,a) + + (1( -
(13)
Here f represents normal convex conjugate defined in eq (7). Although some previous methods directly use this formulation as their optimization target(Ma et al. 2022). The proper way to solve the inner constrained problem is reformulating it as:
max Eg B ((s8,))(ss a, v)] Eg [[(((s,))] + Σ ((8,
(14)
Gal
8,a
Note that we discard the first term in eq 12) because it doesn't contain w, and we denote R r(s,a) a) + Eg alslss.)V((s))
V(s). Since strong duality holds, we directly solve KKT conditions and get the following conditions: w- as) 2 0: (s,ssa) a) 2 0; (s.aa) O. Applying these conditions we can derive w (s.
R(s, a, V) + n² (s, a); y² (s, (s,,)
max (0, (f") [R(s, a, V))}, which eliminates w in eq 12) and gives the final optimization target:
V-DICB: min(1 (d. (V)) + E Eg ma (f') (R)) R- f(max(0, (1') (1))) V(()
(15)
Here R is the abbreviation of R(s, a, V). In this case, f" could be interpreted as (r) =
max (v)) y - f( ma (z))), which is a variant convex conjugate. Simi- larly, we can derive Q-DICE from eq (10) with Lagrangian duality(Nachum Nachum & Dai 2020). It gives
the following optimization target:
Q-DICE: max (o)
E Eg (f)(s,,)))+++++ + [xlllsaa)xxd Q(s,
(16)
s'.a'
Extracting policy is convenient for Q-DICE since its bi-level optimization procedure Nachum et al. 2019b). For V-DIEE, previous methods mainly adopt weighted BC to extract " (Lee et al. 2021 Sikchi et al. 2023), which maximizes the following objective with respect to m:
E (( [w* [w*(s, a) log [a]ss)]
(17)
Besides that, Lee et al. 2021 proposed to use information projection for training policy, which minimizes the following objective:
(8, a)
log (aass
d
KL (d))(()) (al)) = -E mnd
log d (s.
log d
TD (als)
asx(-1e)
(8.0)
coustant for
15

---

Published as a conference paper at ICLR 2024
As this method utilizes (T|ss), it could provide sufficient information when deviates significantly from Tp. However, it needs to estimate ap with BC, which will introduce additional errors.
B PROOFS
Theorem 1. In orthogonal -rrroggonnn-ggdiienttt update, LÝ" (s) (s) (s) 0 (under first order approximation).
To give an intuitive explanation this theorem, we can consider the directions of forward and backward gradients. If (s)) happens to have component that is opposite L then the effect of (s) will be offset by VgLğ(s") because their component along the direction of (a) cancels out. Following the analysis before, clearly don't want the effect of VoLi (s) to be weakened because it represents the process of finding optimal actions. It's worth noting that although I9 (s) 1 (s) < 0 sometimes, we still don't want to preserve this effect of VoL)(s') because may cause over-learn of minimizing (s).
Proof. For the sake of simplicity in our analysis, we will consider first-order approximation when examining 14" (s) 19 (s). As adding (s) - (s) will not change the target, we perform the
following equivalent transformation:
(s)
(s) - (4) (f) - 19()) (s) 19()) -())
(s) (8" -0) (s) (0' -0)
+ (0" - T) V2th" (s) (8" 0) + (0' - -0 V
(18)
(0' 0)
2
= (s) (0" - -0) + O(a?)
Note that although two gradient steps are taken from 0 to 0", their difference is still O(a). This property consequently renders all remainders second-order infinitesimal.
Then by definition, we can substitute 0" with -on (V and get:
Volita) T VaLS(o' V (
1f" (s) - 19 (s) on VoL((() (VoLs(6)) voL((s) [ voLi(s) =0 (19)
[V)
One can easily verify that when performing truc-gradient, i.e. 0" - =oV we have:
VoV(s) (F)))))))) (/ v(s)) (Vo(s'))) = =o7-
= (V))))))++-VV)) voV(s')
(20)
Due to the fact that s and represent consecutive states, it is likely that VoV(s') and VoV(s) fall within the same half-plane, resulting in VaV(s) VoV(s') (V) > 0. This interference results in an increase in (s()s) upon the application of the backward gradient, consequently leading to unlearn- prevents this unlearning by modifying backward gradient, which can preserve the information of backward gradient while eliminating interference.
ing in the process of finding the best action.
16

---

Published as a conference paper at ICLR 2024
Definition 1:
We define (o')) and 1. the perpendicular component and parallel component of (8))())) with respect to Vol. (s). formally:
VoV(s') - (oV(s) VoV(s') (oV(s)
- V(s)) 'y
[VVV(s)112
V2V())
(21)
-1))(66)((;)'001) V(s)) VoV(s) (
VoV(s) VoV(s)
Similarly, we define V (s') and V as the perpendicular component and parallel component of VoV('') with respect to Vo (s). Note that V Vs') and V V (s') can directly obtained from LgLL and (gs's. So directly define them in eq (1))
Theorem 2. Define the angle between (sV(s) and (sV as ('), In orthogonal saaoogoon--grudiint update, if (8, 8") + O, then for all vi sin o(.,5) (cos (8, V.V(+]] (1 we
[[Vile)l VeVielll
have L (8.8) (8, <.
Proof. For simplicity of analysis, we still consider the change of (s, s') under first-order approxi- mation. Due the fact that the learning rate a is actually very small, we consider this approximation to be reasonable. Then we have:
18" (8, (3, = (VLL(())+
(22)
(V
=
First, consider a special case when (o)((s) Lg lies in the same direction as (oL((s).
In this case, (V) + v((2((+') voLi(s) 2 0. We can replace VoLā(s') = v +
V T=)(s') in 21 and get:
10" (8,
= - +4 +4
-a
- - (V +n (VoL(()) (V.Li()++ V L9()
VoLI(s)
in this case20
( L2 ( + (
by defimtionn22 20
(23)
It's obvious that in this case 10" (s, (s) I always satisfies, which means optimality preserved as long as n > O. This indicates that we only have to choose appropriate n when (Vel((s) + VoLG(1)) voLi(s) (V) + V that:
< O. Due to the fact
(o
(, (())) VoV(s)
(24)
= =- --7- (0,)) >V - V(s))]² VVV
We only have to consider the case when cos Φ 0. Note that the last equality can be derived by replacing v!! = cos / VoV(s')) [+Vl
(oV(s) In this case, we can simplify eq 22 with eq 21
17

---

Published as a conference paper at ICLR 2024
and the definition of (s) and VoLg(ss). First plugging eq 21 into eq 22
have:
{ (s, (o vorg(s))
Vo + (o')((6))
(oLi(s) VOLÍ(S') VoI-(()))
[
(25)
=a([Vv[((s)]] (o
VoLg(s)
+ [[V Voll(s) VoLg(s))
)²)
[oI[(())
Recall that:
= -
(26)
V VoLg(s') -113))=+-11)--V/0)))00
We can plug eq 26 into
25 and get:
10" (8, (s, -a (f)) VV())) ((VV)))((1V))))11-----))1 TVV(S)VVVVI
+ + VoV(S) VoV(s) (9)
[|Va [[Va
(27)
To preserve optimality, we should choose appropriate y/ to ensure this change negative. We can get the equivalent form of this condition through some straightforward simplifications, which turns into:
VaV(s)
72
(oV( sin?
(28)
[
COS 6
VoV - ([VoVV6)
sin á
It is evident that this quadratic function has a global maximum when considering VoV(s)] v.V''s
variable. In fact, the global maximum is exactly cot' 4 so once o(s, 8)) # there exists 7 cot such that (s, s') is still decrease after applying ortifenntorthooooallzziiioo
Although the result above gives lower bound for 7) independent of 0, and s', obviously the bound far from tight. fact, when we apply layer normalization (Ba et al., 2016) throughout optimization, (s)) and [s")] will be kept in small interval. This means [V+V(s)] [VVV(e will
be close to 1 and [VaV(s)] (+>)) will be larger than 1. Under this condition, any positive choice of will satisfy the condition above and thus ensure a monotonic decrease ss,
Finally, it can be easily deduced through similar reasoning that semi-gradient can't ensure the de- crease of (x, 8'). So with gradient-orrthoonnaliaa 1 (s. 8') while avoiding unlearning of minimizing (s)..
we can ensure the monotonic decrease of
Definition 2: We define (8, 8') as the angle between V Vs) and V (s'),, formally:
(oV(s)
T (s'
Φo(s, = arccos(
(29)
[ V
Note that with (s, s') can represent / (s') and V(s') more conveniently. One can easily verify V(s') cos o ( v.V(s) [ and V V(s)= sin o VoV( corresponds to the geometric interpretation of V (s') and V Vs').
v(() this also
VV+V
Definition 3: Define the Hessian of V on s and as ((s) = (sV,s), H(s') = VSV Then
define AX(A) Amin (A) Amax( where Amii (A) and Xmax (A) minimal and maximal
18

---

Published as a conference paper at ICLR 2024
eigenvalues of A respectively. Finally define the constant / appears in Theorem 3 with 3v3
4
min AA H (()) a> ( (H())) Note that when Vo smooth with respect 8, 3 close to
Lemma 1: When considering inner constraint w 2 in eq (12). i.. f" max(0, (((")((())))
y (f") The inequality (f*)vv) 2 0 holds for arbitrary convex f and y-
Proof. First considering the case when (1) < case (f))()) = O, which satisfy the inequality.
0. it's obvious that (M) -(( In this
Then considering (y")-
in this case ('(y) (y) v --)((())-w)) Note that
y f() is concave, which means its saddle point = (f))-(()) (y) is also the global maximum
point. Plugging x (f')~(()))) into y I f(x) leads to the equation:
sup a f(x), x E R) = (f') + (((F)) = (2)
(30)
This means in this case f is exactly the normal convex conjugate, while its solution (z) is ensured to be positive. Then by definition, we have:
(F-(()) (f + Av) (f)
Ay
ma Ay) x -f(x)
Ay
(31)
(f)- (y Ay) f((ff)
(1)) 110100000000000000000-0000001111
Ay
= (() >0
So for arbitrary convex f and y, have (f*((y) 2 0.
Lemma 2: If x and y are n-dimensional orthonomal vectors, n x n real symmetric matrix, then x "Ay 2 Amin Amax-
Proof. As A is real symmetric matrix, it can establish set of orthonormal basis vectors using its cigenvectors, Suppose these vectors are z==l-nn We can decompose æ and under these basis vectors, x Eis1 ai4 and y = Ej=1 byžj respectively. Then have:
Σ aab
x yy orthonomal vectors
(32)
x T Ay
)zbyzz) 23)
Azbzzz)
Σ dzaāb;
(33)
Which means x T Ay just reweight a,b; with A; and sum them together. It's clear that:
Σ baagb 2 Amax Σ
aab;) + Amin(
Σ agb,))
(34)
st azbb<0
jst. 0zdy>0
So with Ej azbs Ejst, 25b ajbj we have
bja,b; 2 (Ami -max)
Σ
a,b; > (Amn Amax ||<||||y||
(Amin -mmax)
(35)
fst. 5,0
19

---

Published as a conference paper at ICLR 2024
Theorem 3 (Orthogonal-gradien) update helps alleviate feature co-adaptation). Assume the norm of (oV(s) is bounded, i.e. Vs. m & [VoV(s)]l? < M. Define consecutive value curvature s, s V(s H(() V (s'), where (() is the Hessian matrix of V(s). Assume
E(O, 8') 2 1. Vs')11 where > O. Then in orthogonal- gradienn update, have
Vo 8') (s, 8 ك -on (f*((r VV(s') ((s))[sin² o(s, 1 m + + M. (36)
where 3 is constant close if the condition number of (s)) is small.
Proof. To check the change of (8, s') caused by backward gradient, we also conduct analysis under first-order approximation. This is reasonable due to the fact that H" on the order of O(a), where a typically represents very small learning rate. So applying the same transformation Theorem I we can get:
Vor (o.(8, = (s,) 8') 0(a2)
(37)
= (VV(()) + V (0" 0') + 0(a2)
Where Hs)) is the Hessian matrix of Vo(s) with respect to 0, i.e. (s)) Note that substi- tute V (8, s') with Vo (s, s') because their difference will only contribute to a second-order infinitesimal with respect to a, which will be absorbed into O(a²).
After replacing 0" 0 with a we have
L (s, s') (o) -amy
+ [oV())
- (r) - -vs))
(38)
VV (V.V V. + (s). V('s))
One can further utilize the following equation to replace v/ and V V(')) in 2. 3
Cos (V)(s)
V
V!V(s') cos / VVV
V.V(') sin
cos क [oV(]]
V(')
(39)
This leads the following results:
V Cos V
sin Φ cos ol| VoV(3))
V V(s')l cos 10 V.V (8) H(.)
(40)
V. V(s') (oV(s)
sin V
V(''I [vV[ H(*)
Note that as cus e V V
Vsss
VVVV are orthonormal to (VV( and H(s) is real symmetric,
VaV(s)
V.V(')
and can be futher bounded with lemma 2. This gives the following bound:
sin of cos Φl V (V'( DAmin (H()) Xmax (H()))
sin Φl V, (()( (H(ss)) - Amax (H(())
(sin Φl cos øl (s)() + sin o V() min A> ((s)) (xH)())))
(41)
The AA here is defined in Definition 3. With Vs, m [[V((s) < M we can further bound + 3 with:
(1 + cos ΦI) sin Φ- min min A\ SanfA((H(s (H(s)) AV (H(s))} M
+
(42)
20

---

Published as a conference paper at ICLR 2024
Moreover, due to the fact that & E [0, sin 6( + (cos@)) is non-negative and have upper bound 3v3
Under the assumption of consecutive value curvature, and by replacing 2 + 3 in eq 38
with eq
W gm (s, - gg (s, s') can be ultimately bounded by:
Var (s, 5') Ve (5, - -ary
- ((())(ssi .-6 m + 8- M]
(43)
It's worth noting that we can derive the final bound because (Vo(')+ (o(s)) is always non-negative, which has been proved in Lemma 1. When minimizing standard mean-square Bellman error, r + >=Vo'''') Theorem reflects the nature of the combination of DICE and aatoogonalggradiinn rather than the nature of each one separately.
+ >Vo(s)
Vo(s) could be negative and thus no upper bound is guaranteed. This means
Definition 4: Define K.max as the maximum value of the square of V.VoVo()) ' eigenvalues, i.e. Kmax max A(V. (oVo\)()())) Here we inherit X(+) from Definition 3. Similarly define Kmin-
Kmin.
Then define the constant appears in Proposition 2.1 with C' almost surely positive unless 0 is exactly V. Ve V3(')'s cigenvalue.
Note that C and C is
2Kamas
Theorem 4. (How feature co-adaptation affects state-level robustness) Under linear setting when (s) can be expressed as (sV(s) 0, assume (s)ll* I M, then there exists some small noise 8 E RISI such that V(8 + + - Vs)) will have different sign with V(d') - V(s), if Volo, >
M C 1|=1 for some constant C > 0.
Proof. We begin with deriving the upper bound for (sV VoV Expanding VoV(s)- (sV('') we can get:
(V))(() (VV)))
VoV(s') VoV(s')VVVV( (oV(s)
(44)
Plugging in the bound [VV M and V.V(s') M
K 2K
||||2
have
voV(')))
min ||
(45)
Kmax
Without loss of generality, we only analyze the case when Vss) V(s) O. Due to the fact that noise E is usually small, we can use first-order Taylor expansion to approximate V(s' + E) in the neighborhood of (s').. Then we have:
V(8' + E) VoV(s' + T
VoV(s') VVVo
(46)
= To+ Ae
Here we denote A as V. VoVo(s) for simplicity. After substituting eq (46) into V(s' + +e) - (()) we have:
ee) Vo(s) VoV(s')T0+E) - (sV
= - - vov(s)) = -
(47)
Here again, for simplicity, we denote the difference of feature VoV(s') VoV(s) as Af. Similarly (AΛT)
AAS we can get V(s') (s) T Then set & - ||eell ] TAAsT which could be interpreted as rotated E. We'll then prove that this rotated E can change the sign of (s'' +e) V(s) compared to V(s') (s) while still satisfying ( s') > M-
K2
||e|
2K
Consider the following optimization problem:
min | r A
(48)
21

---

Applying this to Published as a conference paper at ICLR 2024
Solving this problem get the whose projection under A is closest to Af. It's well known that this problem has the following close-form solution:
I (AAT)-1A- Af
(49)
Then under the condition that Af E span(A), ()), it's obvious that we can choose such that this optimization target is This also means that A (AAA
Af = Af. Plugging in
-|el ( and the above relationship into eq 47) have: (AA (
A (AAT)-
v(s +2) Vss) Af || Af
0
((AA)) (AAAA)-AAAA -IAAfI
(50)
|=|||
(1 [(AAT) ) AAFI
A
Recall that from eq 45) we have || (s) VoV(s')I < eq (50)
Kinin V
||=||-
have:
|E||
|l|l
(AA (AAT)-AAAT
A[(AAT)-AA)
|ASll
VKmax
Kmin max {[(AAT)-1A]
(51)
Kiax
K K max > )
From the definition of Kmax and Kmin it's obvious that {
Kmax Plugging
this into eq (51) have [[AА
||l|| A-AA > which means:
|E
V(s' V(s) =
Af
(52)
The last inequality comes from the case we consider that V(s') - V(s) = A < 0. Moreover, due to the fact that the choice of E keeps its norm unchanged, (o(s, > M holds. Finally, one can easily verify this under the case where V(s') - Vss) > 0.
K
||=|| still
2kmax
C MORE DISCUSSION ON GRADIENT FLOW OF DICE
In this section, we'll delve into discussing the first gradient term mentioned in section 2.2 We'll first verify that adding (1-7) V V (s)] won't ton't change our theoretical results. Then we Il show that in our practical algorithm, this term will not impact the results of projection, and thus also has no influence on the practical algorithm of O-DICE.
To begin with, we first consider Theorem 1, Theorem 3 and Theorem Theorem shows that aathogonal-gradiene will not influence the process of seeking the best actions while Theorem 3, shows orthogonal-gradient leads to better state feature representation and thus brings more ro- bustness. These two theorems are completely independent of the first gradient term. As for Theorem 2, shows that (s) + V Lyss)) lies in the same half-plane with VoLl(s) + (sLg(s One can easily verify that after adding the first term, the extended orthogonal-gradien (1 E1 (V)((s) (s) + V Lg still lies in the same half- half-plane with the extended true-gradient (1 7) Vs) + VoLi(s) + (oL)(s"). This indicates that the extended version of Theorem also holds even considering (1 Ed (s)..
Moving forward to the practical algorithm, because we replace do with dº and use empirical Bell- man operator, only one batch (s, 0, s') is needed for one gradient step. This makes (1-Xi V has exactly same direction VoL Due to the property of orthogonal projection, V, (s')) keeps the same after considering the first gradient term. Thus it has no influence both theoretically and practically.
22

---

Published as a conference paper at ICLR 2024
D EXPERIMENTAL DETAILS
More details of the practical algorithm As is shown in the pseudo-code, O-DICE doesn't need parameters for policy extraction whiich is more convenient when compared with many other state- of-the-art algorithms (Xu et al. 2023 Garg et al. 2023). Moreover, as we choose Pearson x² for has the following form:
f-divergence in practice, the corresponding f. f
(x 1)2; (v) y V + 1); (R) R
+1
(53)
We also apply the trick used in SQL (Xu et ) which removes the "+1" term in w (s, a) in (17). This trick could be seen as multiplying the residual term R by a large value. Besides that, during policy extraction, we use r(s, a) (s') (s(s) represent residual term due its relationship with seeking optimal actions. The final objective for policy extraction in our method is (ecaa)aa [max (0, r(s, a) + >V'('') Vo())) log (a]ss]]
Toy case experimental details In order to study the influence of forward and backward gradient on (s),, we ran V-DICE with different gradient types true-gradient, semi-gradient, orthogonal- gradient) separately on pre-collected dataset for 10000 steps and plotted their corresponding Vss) distributions. The pre-collected dataset contains about 20 trajectories, 800 transitions. For the net- work, we use 3-layer MLP with 128 hidden units, Adam optimizer (Kingma & Ba 2015) with learning rate of 10- We choose 5 10 10-3 as the soft update weight for Moreover, as mentioned before, V-DICE with different gradient types may need different À and r) to achieve the best perfor- mance. For the sake of fairness, we tuned hyper parameters for all gradient types to achieve the best performance.
D4RL experimental details For all tasks, we conducted our algorithm for 10th steps and reported the final performance. In MuJoCo locomotion tasks, we computed the average mean returns over 10 evaluations every 5 10 training steps. across 5 different seeds. For AntMaze tasks, we calculated the average over 50 evaluations every 2 10 training steps, also across 5 seeds. Following previous research, we standardized the returns by dividing the difference in returns between the best and worst trajectories MuJoCo tasks. In AntMaze tasks, we subtracted from the rewards.
For the network, we use 3-layer MLP with 256 hidden units and Adam optimizer (Kingma & Ba 2015) with learning rate of 10-4 for both V and x in all tasks. We use target network with soft update weight 5 10-3 for V.
We re-implemented OptiDICE (Lee al. 2021) using PyTorch and ran on all datasets. We followed the same reporting methods as mentioned earlier. Although f-DVI. (Sikchi et al. 2023) reported the performance of two variants, we only take the results of f-DVL using x" because chooses similar f as other DICE methods. Baseline results for other methods were directly sourced from their respective papers.
In O-DCIE, we have two parameters: À and r). Because larger À indicates stronger ability to search for optimal actions, we select a larger experiment with values from the range [0.2, 0.4, 0.6, 0.8, 1.0] to find the setting that gives us the best performance. The values of À and for all datasets are listed in Table 3 In offline II, we use A 0.4 and r) 1.0 across all datasets.
the value of V doesn't 't diverge. As for 7), we
23

---

Published as a conference paper at ICLR 2024
Table À and used in O-DICE
Datasct
7
halfchectah- medhhuc-n-mmmiuu--222
0.2
hopper-medium--2
1.0
malker2d-medium-
0.2
halfcheetah- medium- replayy--2p222
0.2
popper-mddimm-rellaa--
1.0
walker2d medium-replay--2
0.6
0.6
halfcheetah- medium-expert--2
0.5
0.2
0.5
1.0
walker2d medium-expert-yy
0.5
0.2
antmaze umaze-v2
0.6 1.0
0.4 1.0
antmaze-medium play--p
0,7
0.2
untmaze-mcdium-ddivree----
0.7 0.2
untmaze-large-plan
0.7 0.8
satmaze-largediives
0.8 0.8
halfchectah-meddumng
hopper-medium-v2
walker24-medlum--2
D-DICE
0-DICE
100
S-DICE
100
S-DICE
100
S-DICE
80
80
Reward
Reward
40
40
20
20
0.4
166
Stop
1c6
halfcheetah-mediu--repe
hoppe-medium-reelyyy22
wallkerz--meiium--epaayyyw
120
120
120
O.DICE
ODICE
D.DICE
100
80
Reward
Roward
60
60
20
20
0.2
0.4
0.6
0.8
1.0
0.2
0.4
0.6
0.9
1.0
0.2
0.4
0.6
0.8
1.
1e6
Step
1eb
1e6
halfcheetah medium expert v2
happer hepper-medium medium expert v2
walker2d medium expert-v2
120
D-DKCE
O-DICE DICC
O-DXCE DICE
100
SIICCE
100
100
Reward
60
60
40
40
Step
Step
Step
Figure 4: Learning curves of O-DICE and S-DICE D4RL MuJoCo locomotion datasets.
24

---

Published as a conference paper at ICLR 2024
antmaze-um umaze-v2
antmaaz-umaaz-divereeee
antmaze-medium-plaa-22
120
120
120
D-DICE
D S-DICE
D-DICE SIDICE
100
Reward
Roward
60
60
60
0
0.2
0.4
0.6
0.2
0.4
0.6
0.8
1.0
0.8
Step
106
Step
1eb
Step
1e6
antmaze-medium diverse-v2
antmaae llagee play-v2
sallmazelargg-diverseee
320
O-DICE
100
S-DICE
100
S-DICE
100
S-DICE
80
80
80
Reward
Roward
40
40
20
20
0.2
0.4
0.6
0.8
1.0
0.2
0.4
0.6
0.8
1.0
0.4
0.6
0.8
1.0
1e6
Stop
1e6
Figure 5: Learning curves of O-DICE and S-DICE on D4RL AntMaze datasets.
25


Reviews:
[{"review_id": "i9CoD2QCwX", "reviewer": "Reviewer_LpDw", "summary": "The paper proposes Orthogonal-DICE algorithm, that incorporates the V-DICE algorithm with orthogonal-gradient update. The gap between EQL and OptiDICE is analyzed and theoretical proofs are given. Experimental results show that the proposed method achieve better performance than many state-of-the-art methods.", "questions": "The paper is well-written and easy to follow. The high-level idea is clear with interesting motivation. Theoretical analysis shows the reason harm of backward gradient. The experimental result is also impressive and shows the difference in gradients clearly. I have one question about a special case when gradients are computed\n1. How do you calculate the orthogonal gradient when the angle between backward and forward gradients is more than 90 degrees? Especially when it is 180 degrees?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "see questions", "weaknesses": "see questions", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FqeI5NTikJ", "reviewer": "Reviewer_8gZ1", "summary": "This paper explores DICE methods, an important area of research in offline RL. DICE-based methods impose behavior constraints at the state-action level, which is ideal for offline learning. They show that when learning the value function using true-gradient update, there are the forward gradient on the current state and the backward gradient on the next state. And they analyze that directly adding the backward gradient may cancel out its effect if the two gradients conflict. To address this, they propose a simple modification that projects the backward gradient onto the normal plane of the forward gradient, resulting in an orthogonal-gradient update. This new learning rule brings state-level behavior regularization. Through theoretical analyses, toy examples, and extensive experiments on complex offline RL and IL tasks, they demonstrate that DICE-based methods using orthogonal-gradient updates achieve good performance and robustness.", "questions": "1. The author claims that orthogonal-gradient can help consider state-level constraints, but there is no explicit explanation in the analysis as to why this constraint exists. Please provide a detailed explanation.\n2. Is the $\\epsilon$ in Theorem 4 different for different $s$ ? If so, I think Theorem 4 is trivial since we can easily find an state-wise $epsilon(s) $ to make  $V(s’+\\epsilon)-V(s) $ flip sign. \n3. The algorithmic complexity of O-DICE has not been analyzed.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The advantages of this paper are as follows: \n1. This paper discusses the relationship between true gradient and semi-gradient for (2) and establishes an analysis of the correlation between offline and online training using semi-gradient. This provides a new perspective on why (2) is difficult to train. As a result, the paper introduces the design of orthogonal-gradient, which is logically reasonable. Additionally, the paper presents the relationship between orthogonal-gradient and feature co-adaptation, making this design even more compelling.\n2. This paper is written in a fluent manner with clear logic. The theoretical analysis is rigorous, and the experiments are abundant for comparison.", "weaknesses": "1. This paper claims that the only difference between itself and OptiDICE [1]  lies in whether to use orthogonal-gradient. However, from my understanding, there is a significant difference in the optimization objectives between OptiDICE and this paper. In [1], the corresponding optimization objective (11) includes not only the value function v but also the optimal density ratio w_v. Therefore, in terms of form, it is different from the optimization objective in this paper. As a result, the author's claim that they have found the mystery of why DICE-based methods are not practical is somewhat exaggerated.\n2 . However, I believe that since (11) still involves v(s'), it should be possible to apply orthogonal-gradient when computing its gradient. Therefore, the author should compare the original OptiDICE algorithm with and without orthogonal-gradient to show that orthogonal-gradient has individual and significant gain. \n3. Most RL algorithms involve Bellman operator operations, which require computing both forward and backward gradients. Therefore, orthogonal-gradient may not just be applicable to DICE-based methods, but also to most RL algorithms. This is much more important than improving DICE-based methods, and the author could consider this perspective.\n4. From an experimental perspective, this method requires two key hyperparameters and needs different hyperparameters for different datasets, which is a disadvantage compared to other methods.\n\n[1] Optidice: Offline policy optimization via stationary distribution correction estimation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "t9DcA4mMmO", "reviewer": "Reviewer_SNRk", "summary": "- The paper identifies two gradient terms when learning the value function using true gradients: the forward gradient (taken on the current state) and the backward gradient (taken on the next state) of OptiDICE. The authors argue that directly adding the backward gradient may lead to its degeneration or cancellation if these two gradients conflict. \n- To address this issue, the paper proposes a simple yet effective modification that projects the backward gradient onto the normal plane of the forward gradient, resulting in an orthogonal-gradient update, a novel learning rule for DICE-based methods.\n- Through toy examples and extensive experiments on complex offline RL and IL tasks, the paper demonstrates that DICE-based methods using orthogonal-gradient updates achieve state-of-the-art performance and high robustness.", "questions": "- Basically the orthogonal gradient technique used in this paper is not limited to DICE algorithms but we can adopt them on any deep RL algorithms. Can we get improvements on ordinary deep RL algorithms?\n\n- As far as I know, if we use the semi-gradient update for OptiDICE, it diverges since the second term dominates the first term, and the second term only increases the V function according to the monotonic increasing shape of f^*. It seems the trick of Sikchi et al. (2023) is the trick that makes the algorithm to work. According to the objective in the paper, it seems the algorithm is sampling the arbitrary experience instead of initial state distribution, and it seems to be weighting it much more heavily. Is there any theoretical guarantee that the proposed objective function gives similar solution to what we can get with OptiDICE?\n\n- OptiDICE actually tends to overestimate \\nu, and similar to above, if S-DICE and O-DICE do not overestimate even without double Q trick, I believe that it should be related to the Sikchi et al. (2023) trick. is Sikchi et al. (2023) applied to OptiDICE in experiments?\n\n- According to those reasons, I would like to see the results without the Sikchi et al. (2023) trick.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Very interesting perspective on poorly performing OptiDICE and proposes a novel orthogonal update algorithm\n- While the paper Is limited to DICE algorithm, the proposed orthogonal update seems to be able to be applied to other bootstrapping based deep RL methods.\n- The paper is clearly presented and easy to follow.\n- Experimental results are strong.", "weaknesses": "- The theoretical motivations for orthogonal gradient update do not seem to be sufficient. What we can know with the theoretical results presented are:\n  - If we put right \\eta, the orthogonal gradient can be no worse than semi-gradient\n  - There is a possibility that orthogonal gradient can help feature co-adaptation\n  - Based on these, O-DICE should perform on par with S-DICE on simple enough domains, and on complex domains, using DR3-like regularizations will make S-DICE to perform on par with O-DICE. Will there be additional source of better performance? Can we get better fixed point when orthogonal gradient with large \\eta is adopted (large enough to make it different from S-DICE)?\n- The Sikchi et al. (2023) trick seems to have a central role of the performance. details below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "abNL1ok80M", "reviewer": "Reviewer_g7U6", "summary": "This paper uncovers an explanation for a mysterious phenomenon among off-policy RL DICE methods: state-action level behavior constraints are more principled but action-level behavior constraints perform better empirically. The authors show how an action-level behavior constraint can be achieved from two different view points: Exponential Q-Learning (semi-gradient) designed to specifically impose an action-level constraint and the forward-gradient view put forth by the authors. This then implies that the key difference between the semi-gradient and true-gradient updates (true-DICE) is the backward-gradient. Given both the superior empirical performance of the forward gradient approaches and the desire to better match the more principled state-action approach, the authors propose to retain the backward-gradient component albeit with a twist: include the backward component but reject any part of it which aligns with the forward component (thereby removing any gradient cancellation). The authors then support this technique both theoretically (alleviates interference of forward/backward updates and feature co-adaption) and with extensive empirical validation (Fig 2 shows the method avoids overestimating V(s) outside the dataset distribution, Table 1 and Table 2 show it performs well on difficult tasks, and Fig 3 shows its worst-case performance is higher than other approaches).", "questions": "- In section 3, first paragraph, did you mean \"orthogonal-gradient\" instead of \"vertical-gradient\"?\n- Theorem 3: Given that you measure feature co-adaption with an un-normalized dot-product, couldn't the result in Theorem 3 be achieved by a shrinking of the feature vectors rather than a change in their orientation or relative representations? I assume your intention is to show that $\\nabla_{\\theta} V(s)$ and $\\nabla_{\\theta} V(s')$ are more different (or separated) under the orthogonal-gradient update (i.e., $\\theta=\\theta''$) than they are normally (i.e., $\\theta=\\theta'$).\n- Figure 2: I can see how (d) aligns well with the dataset support, but so does (b). Is (b) undesireable because the actual values V(s) across the offline states are poorly approximated (i.e., the gradient across V(s) is uninformative)? If so, can you please add that to the caption?\n- You say \"Hence, Eq.(36) < 0\" above Theorem 4? Is this a typo? Did you mean to refer to equation 6?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "I thought this paper did an excellent job of identifying and isolating a key problem, making an astute connection to research elsewhere in the literature (EQL), and then exploiting this connection to develop and evaluate a simple fix. I think the paper is mostly well-written with good theoretical and empirical support. I have a few questions regarding one of the theoretical statements and one of the figures, but otherwise have no other major concerns.", "weaknesses": "I would like to see a bit more explanation / background in the second-to-last paragraph of section 1 (Introduction) before you start discussing the bellman residual term and *true-gradient* methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)