Offline RL with No OOD Actions: In-Sample Learning via Implicit Value Regularization
Authors: Haoran Xu
Li Jiang
Jianxiong Li
Zhuoran Yang
Zhaoran Wang
Victor Wai Kin Chan
Xianyuan Zhan
Venue: ICLR 2023 notable top 5% (2023)
Keywords: Deep Reinforcement Learning
Offline Reinforcement Learning
Value Regularization
Continuous Control

Abstract: Most offline reinforcement learning (RL) methods suffer from the trade-off between improving the policy to surpass the behavior policy and constraining the policy to limit the deviation from the behavior policy as computing $Q$-values using out-of-distribution (OOD) actions will suffer from errors due to distributional shift. The recent proposed \textit{In-sample Learning} paradigm (i.e., IQL), which improves the policy by quantile regression using only data samples, shows great promise because it learns an optimal policy without querying the value function of any unseen actions. However, it remains unclear how this type of method handles the distributional shift in learning the value function. In this work, we make a key finding that the in-sample learning paradigm arises under the \textit{Implicit Value Regularization} (IVR) framework. This gives a deeper understanding of why the in-sample learning paradigm works, i.e., it applies implicit value regularization to the policy. Based on the IVR framework, we further propose two practical algorithms, Sparse $Q$-learning (SQL) and Exponential $Q$-learning (EQL), which adopt the same value regularization used in existing works, but in a complete in-sample manner. Compared with IQL, we find that our algorithms introduce sparsity in learning the value function, making them more robust in noisy data regimes. We also verify the effectiveness of SQL and EQL on D4RL benchmark datasets and show the benefits of in-sample learning by comparing them with CQL in small data regimes. Code is available at \url{https://github.com/ryanxhr/SQL}.

Published as a conference paper at ICLR 2023
OFFLINE RL WITH No OOD ACTIONS: IN-SAMPLE LEARNING VIA IMPLICIT VALUE REGULARIZATION
Haoran Xul Li Jiang Jianxiong Li'
Zhuoran Yang"
Zhaoran Wang
Victor Wai Kin Chan² Xianyuan Zhan
IIntiitte for Al Industry Research (AIR), Tsinghua University
Belinghua-Berkelee Shenzhen Institute (TBSI). Tsinghua University
3 Yale University "Northwestern University *Shanghai Artificial Intelligence Laboratory (ryanxhr, jiangli3859, zhanxi anyuan ) gmmal. com
ABSTRACT
Most offline reinforcement learning (RL) methods suffer from the trade-off between improving the policy to surpass the behavior policy and constraining the policy limit the deviation from the behavior policy as computing Q- valule using out- of-distribution (OOD) actions will suffer from errors due to distributional shift. The recent proposed In-sample Learning paradigm (i.e., IQL), which improves the policy by quantile regression using only data samples, shows great promise because learns an optimal policy without querying the value function of any unseen actions. However, it remains unclear how this type of method handles the distributional shift in learning the value function. In this work, we make a key finding that the in-sample learning paradigm arises under the Implicit Value Regularization (IVR) framework. This gives a deeper understanding of why the in-sample learning paradigm works, i.e., applies implicit value regularization to the policy. Based on the IVR framework, we further propose two practical algorithms, Sparse Q-learning (SQL) and Exponential Q-learning (EQL), which adopt the same value regularization used in existing works, but in complete in- sample manner. Compared with IQI., we find that our algorithms introduce sparsity in learning the value function, making them more robust in noisy data regimes. We also verify the effectiveness of SQL and EQL on D4RL benchmark datasets and show the benefits of in-sample learning by comparing them with CQL in small data regimes. Code is available at hhttts://githut: com/ ryanxhr/IVE
INTRODUCTION
Reinforcement learning (RL) is an increasingly important technology for developing highly capable AI systems, has achieved great success in game-playing domains (Mnih et al. 2013; Silver et al.) 2017). However, the fundamental online learning paradigm in RL is also one of the biggest obstacles to RL's widespread adoption, as interacting with the environment can be costly and dangerous in real-world settings. Offline RI. also known as batch RI, aims at solving the abovementioned problem by learning effective policies solely from offline data, without any additional online interactions. It is promising area for bringing RL into real-world domains, such robotics (Kalashnikov et al., (202). healthcare (Tang & Wiens 2021 1) and industrial control (Zhan et al. 2022). In such scenarios, arbitrary exploration with untrained policies is costly or dangerous, but sufficient prior data is available.
While most off-policy RL algorithms are applicable in the offline setting by filling the replay buffer with offline data, improving the policy beyond the level of the behavior policy entails querying the Q-function about values of actions produced by the policy, which are often not seen in the dataset. Those
actions can be deemed as adversarial examples of the Q-function, which cause extrapolation errors of the Q-function (Kumar et al., 2020). To alleviate this issue, prior model-free offline RL methods typically add pessimism to the learning objective, in order to be pessimistic about the distributional shift. Pessimism can be achieved by policy constraint, which
constrains the policy to be close to the behavior policy (Kumar et al. 2019 Wu et al. 2019 Nair
Work done while JD) Technology. Correspondence Haoran XU, Xianyuan Zhan.

---

Published as a conference paper at ICLR 2023
et al. 2020; Fujimoto & Gu, 2021 value regularization, which directly modifies the Q-function to be pessimistic (Kumar et al. 2020; Kostrikov et al.) [2021a] An et al.) 2021 Bai et al. (021). Nevertheless, this imposes trade-off between accurate value estimation (more regularization) and maximum policy performance (less regularization).
In this work, we find that we could alleviate the trade-off in out-of-sample learning by performing implicit value regularization, this bypasses querying the value function of any unseen actions, allows learning an optimal policy using in-sample learning More specifically, we propose the Implicit Value Regulazization (IVR) framework, in which a general form of behavior regularizers is added to the policy learning objective. Because of the regularization, the optimal policy in the IVR framework has closed-form solution, which can be expressed by imposing weight on the behavior policy. The weight can be computed by a state-value function and an action-value function, the state-vat value function serves as normalization term to make the optimal policy integrate to 1. It is usually intractable to find closed form of the state-value function, however, we make subtle mathematical transformation and show its equivalence to solving a convex optimization problem. In this manner, both of these two value functions can be learned by only dataset samples.
Note that the recently proposed method, IQL (Kostrikov et al. 2021b), although derived from different view (i.e., approximate an upper expectile of dataset actions given a state), remains pretty close to the learning paradigm of our framework. Furthermore, our IVR framework explains why learning the state-value function is important in IQI. and gives a deeper understanding of how IQL. handles the distributional shift: it is doing implicit value regularization, with the hyperparameter to control the strength. This explains one disturbing issue of IQL, i.c., the role of T does not have a perfect match between theory and practice. In theory, 7 should be close to to obtain an optimal policy while in practice a larger r may give a worse result.
Based on the IVR framework, we further propose some practical algorithms. We find that the value regularization terms used in CQL (Kumar et al. (020) and AWR (Peng et al. 2019) are two valid choices in our framework. However, when applying them to our framework, we get two complete in-sample learning algorithms. The resulting algorithms also bear similarities to IQL.. However, we find that our algorithm introduces sparsity in learning the state- value function, which is missing in IQL. The sparsity term filters out those bad actions whose Q-values are below a threshold, which brings benefits when the quality of offline datasets inferior. We verify the effectiveness of SQL on widely-used D4RL benchmark datasets and demonstrate the state-of-the oh-the-rrt performance, especially on suboptimal datasets in which value learning is necessary (c.g., AntMaze and Kitchen). We also show the benefits of sparsity in our algorithms by comparing with IQL. in noisy data regimes and the robustness of in-sample learning by comparing with CQL in small data regimes.
To summarize, the contributions of this paper follows:
We propose general implicit value regularization framework, where different behavior regularizers can be included, all leading to a complete in in-sae learning paradigm.
Based on the proposed framework, we design two effective offline RL algorithms: Sparse Q- Learning (SQL) and Exponential Q-learning (EQL), which obtain SOTA results benchmark datasets and show robustness in both noisy and small data regimes.
RELATED WORK
To tackle the distributional shift problem, most model-free offline RL methods augment existing off-policy methods (e.g., Q-learning or actor-critic) with a behavior regularization term. Behavior regularization can appear explicitly as divergence penalties (Wu et al.. 2019 Kumar et al.) 2019 Fujimoto & Gul 2021), implicitly through weighted behavior cloning (Wang et al.] 2020 Nair et al. 2020), or more directly through careful parameterization of the policy (Fujimoto et al.) 2018 Zhou et al. 2020). Another way apply behavior regularization is via modification of the critic learning objective to incorporate some form of regularization, to encourage staying near the behavioral distribution and being pessimistic about unknown state-action pairs (Nahhum et al. 2019 Kumar
et al. 2020 Kostrikov et al. 2021a Xu et al. 2022c). There are also several works incorporating
"The core difference hetween in-sample learning and out-of-sample learning is that in-sample learning uses only dataset actions learn the value function while out-of-sample learning uses actions produced by the policy.
2

---

Published as a conference paper at ICLR 2023
behavior regularization through the use of uncertainty (Wu et al. 2021 An et al. 2021 Bai et al. 2021) or distance function (Li et al. (023b))
However, in-distribution constraints used in these works might not be sufficient to avoid value function extrapolation errors. Another line of methods, on the contrary, avoid value function extrapolation by performing some kind of imitation learning
the dataset. When the dataset is good enough or contains nigh-performing trajectories, we can simply clone or filter dataset actions to extract useful transitions (Xu al., 2022b] Chen al. 2220)) or directly filter individual transitions based on how advantageous they could be under the behavior policy and then clones them [Brandionrrener et al. (2021); Xu al. (202T 2022a). While alleviating extrapolation errors, these methods only perform single-step dynamic programming and lose the ability to "stitch" suboptimal trajectories by multi-step dynamic programming.
Our method can be viewed as combination of these two methods while sharing the best of both worlds: SQL and EQL. implicitly control the distributional shift and learns an optimal policy by in-sample generalization. SQL and EQL are less vulnerable to erroneous value estimation as in- sample actions induce less distributional shift than out-of-sample actions. Similar to our work, IQL approximates the optimum in-support policy by fitting the upper expectile of the behavior policy's 's action-value function, however, it is not motivated by remaining pessimistic to the distributional shift.
Our method adds behavior regularization term to the RL learning objective. In online RL, there are also some works incorporating an hatropy-regularizer term into the learning objective (Haarnojal et al. 2018 Nachum al. 2017 Lee et al. 2019 Neu et al. 2017 Geist et al. 2019 Ahmed et al., 2019), this brings mulli-moddlltyy to the policy and is beneficial for the exploration. Note that the entropy-regularized term only involves the policy, it could be directly computed, resulting similar learning procedure as in SAC (Haarnoja et al.) 2018). While our method considers the offline setting and provides a different learning procedure to solve the problem by jointly learning a state-value function and an action-value function.
3 PRELIMINARIES
We consider the RL problem presented as a Markov Decision Process (MDP) (Sutton et al. 1998). which is specified by tuple M (S A, T, p. P.7)) consisting of state space, action space, transition probability function, a reward function, an initial state distribution, and the discount factor. that maximizes the expected discounted
The goal of RL is find a policy (als): S x A- [0, cumulative reward (or called return) along trajectory as
max E
Σ y'r (st, ai) so s, an a, St T (-186-1, at II (-1s:) for t 2 1
(1)
In this work, we focus on the offline setting. Unlike online RL methods, offline RL aims to learn an optimal policy from a fixed dataset DD consisting of trajectories that are collected by different policies. The dataset can be heterogenous and suboptimal, we denote the underlying behavior policy of as 1r, which represents the conditional distribution p(als) observed in the dataset.
RL methods based on approximate dynamic programming (both online and offline) typically maintain an action-valuu function (Q- (f-fucction and, optionally, state-value function (V-function), refered as Q(+, a) and (s) respectively (Haarnoja et al., 2017 Nachum et al.) 2017 Kumar et al., 2020 Kostrikov et al. 2021b). These two value functions are learned by encouraging them satisfy single-step Bellman consistencies. Define a collection of policy evaluation operator (of different policy x) on Q and V as
== (=))(s.a):=r((a)
EE [Qss',a')
(T*V) = En-o [r(s, a) + E [V(')]]]
then Q and V are learned by ming J(Q) 1 ( [T (s, a) and miny J(V) 2 EDDD [T*V - (s)3). respectively. Note that x could be the learned policy æ or the behavior policy μ4, if = μr then a ~ j and a' ~ μ are equal to D and a' D. respectively. In offline RL, since D typically does not contain all possible transitions (s, a, a'),,, one actually uses an empirical policy evaluation operator that only backs up single s' sample, we denote this operator as 7*
3

---

Published as a conference paper at ICLR 2023
In-sample Learning via Expectile Regression Instead of adding explicit regularization to the policy evaluation operator to avoid cut-of-distribunn actions, IQL uses only in-sample actions to learn the optimal Q-function. IQL uses asymmetric l loss (i.e., expectile regression) to learn the V V-function, which can be seen as an estimate of the maximum Q-value over actions that are in the dataset support, thus allowing implicit Q-learning:
min E (s.))~-D1 ((Q(s, - V(s) < <))(((s))))) -) V(s))
min ((s.,,))) (r(()((())))))) yV Q(s, ((s,a))) a)
where 1 is the indicator function. After learning Q and V, IQL extracts the policy by advantage- weighted regression (Peters et 2010 Peng et 2019 Nair al. 2020
min (s,a)~ exp (3 (Q(s, a) V(s)) ) log ((ass)
While IQL achieves superior D4RL benchmark results, several issues remain unsolved:
The hyperparameter T has gap between theory and practice: in theory should be close to 1 to obtain an optimal policy while practice larger t may give a worse result.
In IQL the value function is estimating the optimal policy instead of the behavior policy, how does IQL handle the distributional shift issue?
Why should the policy be extracted by wwaaaatg--weihheee regression, does this technique guarantec the same optimal policy as the one implied in the learned optimal Q-function?
OFFLINE RL WITH IMPLICIT VALUE REGULARIZATION
In this section, introduce framework where a general form of value regularization be implicitly applied. We begin with special MDP where behavior regularizer is added to the reward, we conduct a full mathematical analysis of this regularized MDP and give the solution of it under certain assumptions, which results in a complete in-sample learning paradigm. We then instantiate practical algorithm from this framework and give thorough analysis and discussion of it.
4.1 RENNIORRRGGULARZEDD MDPs
Like regropy-regularized RL adds an entropy regularizer to the reward (Haarnoja et al. 2018). in this paper we consider imposing general behavior regularization term to objective I and solve the following behavior-regularizee MDP problem
Σ 2 (F(s..a)) a. (
max
E
where f(.) is regularization function. is known that in entropy-regularized RL the regularization gives smoothness of the Bellman operator (Ahmed et al. [0199] Chow et al. 2018), e.g., from greedy to softmax over the whole action space when the regularization is Shannon entropy. While in our new learning objective (4), we find that the smoothness will transfer the greedy max from policy softened max (depending on f) over behavior policy μ. this enables an in-sample learning scheme, which is appealing in the offline RL setting.
In the behavior-regularizee MDP, we have modified policy evaluation operator T given by
(TF)Q(s, == r(s, a) + -EE [V(s')]
where
ExuT V(s)= =BaT [((s,a) af [(als)) p(als) ).
The policy learning objective can also expressed as max; E,-D [V()). Compared with the origin policy evaluation operator 7 Tf. is actually applying value regularization to the Q-function. However, the regularization term is hard to compute because the behavior policy μl is unknown. Fenchel-duality (Boyd et al. 2004) get sampled-based estimation if f Although f-divergence (Wu et al. 2019). this unnecessarily brings a min-max optimization belongs to the f problem, which is hard to solve and results in poor performance in practice (Nachum et al. 2019).
4

---

Published as a conference paper at ICLR 2023
4.2 ASSUMPTIONS AND SOLUTIONS
We now show that we can get the optimal value function Q' and V without knowing First, in order make the learning problems analyzable, two basic assumptions required as follows:
Assumption 1. Assume {[ass) 0 p(als) 0 that T/p is well-
Assumption 2. Assume the function f() satisfies the following conditions (0, x)) (1) f(1) = 0;
(2) hy(#)= = (x) is strictly convex; (3) f(x is differentiable.
The assumptions of f1) and x (x) strictly convex make the regularization term be positive due to the Jensen's inequality y [ (=) 2 1f(1) O. This guarantees that the regularization term minimized only when π = μ.. Because (2x strictly convex, its derivative, 11((x) = f(z) + (x)) is a strictly increasing function and thus (x) exists. For simplicity, we denote (z) (h',) (xx). The assumption of differentiability facilitates theoretic analysis and benefits practical implementation due to the widely used automatic derivation deep learning.
Under these two assumptions,
get the following two theorems:
Theorem 1. In the Rehavior-regularize MDP, any optimal policy x² and its optimal value function Q' and V satisfy the following optimality condition for all states and actions:
Q* a) = r(s,a) + >E [V* (6')
(s) -s)()
t'(ass u(als) max 95
(5)
("(al5) 2 n (T((as))
(=))= + aEawn
u(als) fr
(6)
ulals)
where (8) is normalization term so that EnEA T (als) 11
The proof is provided in Appendix C.1 The proof depends on the KKT condition where the derivative of Lagrangian objective function with respect to policy {a]ss) becomes zero at the optimal solution.
Note that the resulting formulation of Q* and V only involves U* and action samples from (*)s) can be uniquely solved from the equation obtained by plugging Eq.45) into EesA ' (a|s) which also only uses actions sampled from μ.. In other words, now the learning of Q² and V can be realized in an in-sample manner.
Theorem D also shows how the behavior regularization influences the optimality condition. If we choose such that there exists some x that (x))) < O, then can be shown from (q.(5) that the optimal policy n² will be sparse by assigning zero probability to the actions whose Q-values Q" (s, a) are below the threshold U (s) + ah²((0) and assigns positive probability to near optimal actions in proportion to their Q-values (since (r(r) is increasing). Note that π could also have no sparsity, for example, if choose f = log(z), then 9ʃ exp(x will give all elements non-zero values.
Theorem 2. Define Tj the case where in Tÿ is the optimal policy n², then Tj is 7-contraction.
The proof is provided in Appendix C.2 This theorem means that by applying Q++1 = T/ repeatedly, then sequence Q* will converge the Q-value of the optimal policy '² when lc co.
After giving the closed-form solution of the optimal value function. We now aim to instantiate a practical algorithm. In offline RL, in order to completely avoid out-of-distribution actions, want a cero-forcing support constraint, i.e., p(aas) = 0 π (als) = 0. This reminds us of the class of a-divergence (Boyd et al. 2004), which is a subset of f f-divergence and takes the following form (a E R (2
Da(u,)) r) ala - 1) Ex
π
1
a-divergence is known to be mode-seeking if one chooses a < 0. Note that the Reverse KL divergence is the limit of Da (M, x) when 0. We also obtain Helinger distance and Neyman - -dv-ieereeece as a = 1/2 and a = -1, respectively. One interesting property of a-divergence is that D., (1, T) D1-a(T.1))
5

---

Published as a conference paper at ICLR 2023
4.3 SPARSE Q-LEARNING (SQL)
We first consider the case where a - 1, which we find is the regularization term CQL adds to the policy evaluation operator (according to Appendix C. in CQL): Q(s, a) T Q(s a) [ 8)
1].
In this case, we have f()) = x - and gy(x) = 1 + 2- Plug them into Eq. and Eq. in Theorem we get the following formulation:
(= r(s, a) + EE, IV (5')
Q²(ss U (+)
<'(ass) = p(als) max
0
(8)
2
2a
(1(
= = (8) + alEaa
p(als)
where (s) needs to satisfy the following equation to make 'T" integrate to 1:
C - 2
(s)
[0] =1
Eanu
E
max
(10)
2a
It is usually intractable to get the closed-form solution of (s) from I0) however, here we make a mathematical transformation and show its equivalence to solving a convex optimization problem.
Lemma 1. We can get U*(s) (s) by solving the following optimization problem:
Eang 1(2++ Q" (s. a) U(s) C
2a > =) -
Q*(s.a) -((s)))-U())) U(s) a
(11)
2a
The proof can be easily got we set the derivative of the objective to 0 with respect to (s)). which is exactly II. Now we obtain learning scheme to get Q², U and V by iteratively updating Q, U and V following (q. objective (11) and Eq. 7 respectively. We refer to this learning scheme SQL-U, U, however, SQL-U needs to train three networks, which is bit computationally expensive.
Note that the term Ea [ p(aas)
2 in Eq. is equal to Ea-s x (als))
p(als) ]. as . optimized to
become mode-seeking, for actions sampled from x², its probability π (a should be close to the probability under the behavior policy, p(aas)) s). Note that for actions sampled from μl, x (a|s) and ((aas) may have large difference because ' (alss) may be O.
Hence in SQL we make an approximation by assuming Eusa p(a s) network as U* V* - a. Replacing U with V*, we get the following learning scheme that only needs to learn V and Q iteratively to get V* and Q":
[a|x)
1, this removes one
min B(aaaD (11 Q(s, a) V(s)
0)(11 (1+ a(s,a)= ((s)))
v(5))) 2 V(s)
(12)
2a
2a
+ +V('s) Q(s min B( ((())))))( ())
(13)
After getting V and Q. following the formulation of π in Eq. 8 can get the learning objective of policy N by minimizing the KL-divergence between π and
max E(S.a>D [1+ ((s,a) V(s) > =0)(1+ ((s,a) v(o) log (aalss
(14)
2a
2a
4.4 EXPONENTIAL Q-LEARNING (EQL)
Now let's consider another choice, + 0 which is the Reverse KL divergence. Note that AWR also uses Reverse KL divergence, however, applies it to the policy improvement step and needs to sample actions from the policy when learning the value function. In this case, get f(x) log(x)
and gr (x) exp(æ - 1). Plug them into Eq. 5) and (6) in Theorem I we have
Q" (8, ((, a) E (V* (8')

---

min Published as a conference paper at ICLR 2023
n' <a(s)) - [d(als) exp ("(,,a)-U*(( U
a (T]as) p(aas)) u(aas) ot* (a|s)
(*(s) = U* (s) + aBaryy
note that En-pl [( (als) 2 [(ale]
[ala) is equal to 1, so we get (*) U (s) + a, this eliminates
the existence of U without any approximation. Replacing U with V we get the following formulation:
(" a) a(),a) + yE,''s, |s.0 [V' (5')
x' (als) = p(als) exp Q (s) - (s)
a
Note that x² should be integrated to 1, we the same mathematical transformation did in SQL and get the closed-form solution of (s)) by solving the following convex optimization problem.
Lemma 2. We can ger V V*(s) (s) by solving the following optimization problem:
(²'(, -
V(s) a
min Banp
exp
Now the final learning objective of Q, V and
x is:
(saD)D exp Q(3,,a) (Q(s,))-((o) V(()
(15)
V
a
min B(ea,// Q
(r(s, +>V(8') Q( a))
(16)
exp Q( ((s, log (a|ss)
E (x,a)~
(17)
we name this algorithm as EQL (Exponential Q-Learning) because there is an exponential term in the learning objective.
To summarize, our final algorithm, SQL and EQL, consist of three supervised stages: learning V. learning Q, and learning म. We use target networks for Q-functions and use clipped double
Q-learning (take the minimum of two Q-functions) in learning V and т. We summarize the training procedure Algorithm
4.5 DISCUSSIONS
SQL and EQL establishes the connection with several prior works such as CQL, IQL and AWR.
Algorithm 1 Sparse or Exponential Q-Learning
Require: D, a.
Like CQL pushes down policy Q-values and pushes up dataset Q-values, in SQL. and EQL, the first term in [2) and (5) pushes up V-values if Q - V O while the second term pushes down V-values, and trades off these two terms. SQL incorporates the same inherent conservatism as CQL by adding the diveerggee to the policy evaluation operator. However, SQL. learns the value function using only dataset samples while CQL. needs to sample actions from the policy. In this sense, SQL is an "implicit" version of CQL that avoids any out-of- distribution action. Like AWR, EQL applies the KL-divergence, but implicitly in the policy evaluation step. In this sense, EQL is "implicit" version of AWR that avoids any OOD action.
1: Initialize Qa. Qor. Voo. ne
2: for L 2, .....NN do
3: Sample transitions o,, D
Update Vo by Eq. 12 Eq
using Vo, Qai
5:
Update Qs by Eq [33
(3) or Eq.
using V., Qu
6:
Update Qai by ø²
Adb
A) using Vs. Qsi
7: Update wo by Eq. 14
Eq.
T7
8:
end for
Like IQL, SQL and EQL learn both V-function and fuunction. However, IQL appears to be a heuristic approach and the learning objective of V V-function in IQL has drawback. We compute the derivative of the V V-function learning objective with respect to the residual (Q V) in SQL and IQL (see Figure in Appendix A). We find that SQL keeps the derivative unchanged when the residual
7

---

Published as a conference paper at ICLR 2023
Table 1: Averaged normalized scores of SQL against other baselines. The scores taken the final 10 evaluations with seeds. SQL EQL achieves the highest scores in out of tasks.
Dataset
10%BC
DT
One-stop
CQL
SQL
BQL
halfcheetah-m
48.4
44.0
+0.2
48.3+-0.2
47.2-0.3
hopper-m
52.9
59.3
58.5
+2.1
66.3 +5.7
75.5 -3.4
70.6+2.6
walker2d
72.5 10.8
72.5
83.2.4.44
halfcheetah m-r
10.55
14.5
hopper-m-r
60,9
95.0 4-6.4
95.2
walker2d-m-r
77.24-5.5
76.1 +-7.3
77.24-3.8
81,6+4.22
halfcheetah-mn
90.7.14.3
86.7.15.3
94.010.4
1.0.5
hopper me
110.9
107.6
98.0
111.5
miee
109.0
108.1
110.2.0.8
antmazo-u
54.6
62.8
78.9
59.2
64.3
78.6
84,8+2.3
85.5 1.9
92.2=1A
93.2 =2.2
antmazz-u-d
45.6
50,2
55.0
53.0
60.7
71.4
43,4+5.4
66.7 =4.0
74.0
2.33
70.4=2.7
antmaze----
5.4
0
0.0
0.3
10.6
65.2+-4.8
72.2 +5.3
80.2
+3.7
77.5+44.
antmaze-m-d
9.8
0
0.0
0.0
3.0
54.0=11.7
71.0 +3.2
75.1
4.2
74.0=3.7
antmaze-l-p
0.0
6.7
0.0
0.0
0,2
38.4=12.3
39.6 =4.5
50.2
4488
45.6=4.2
antmaze-l-d
6.0
2.2
0.0
0.0
0.0
31 31.6+t9.5
47.5 =4.4
52.3
-5.2
49.5=4.7
kitchen-c
33.8
43.8 1112
69.4
76.4
#8.7
70.3=7.1
kitchen-p
33,9
49.8=10.1
46.1 28.5
72.5 47.4
70.5=
kitchen-m
47,5
51.0+6.5
52.8 +4.5
67.4
+5.4
61.6=5.2
below a threshold, while IQL doesn't. In IQL, the derivative keeps decreasing as the residual becomes more negative, hence, the V-function will be over-underestimated by those bad actions whose Q-value is extremely small. Note that SQL and EQL will assign a zero or exponential small probability mass to those bad actions according to Eq. (4. and Eq. (T7), the sparsity is incorporated due to the mode-seeking behavior of <dvdieeree and KI-divergence.
Also, IQL needs hyperparameters (r and (3) while SQL only needs (a:). The two hyper- parameters in IQL may not align well because they represent two different regularizations. Note that objective (17 is exactly how IQL extracts the policy! However, the corresponding optimal V-function learning objective (I5) is not objective (2). This reveals that the policy extraction part in IQL gets different policy from the one implied the optimal Q-function.
EXPERIMENTS
We present empirical evaluations of SQL and EQL in this section. We first evaluate SQL and EQL against other baseline algorithms on benchmark offline RL datasets. We then show the benefits of sparsity introduced in SQL and EQL by comparing them with IQL noisy data regimes. We finally show the robustness of SQL and EQL by comparing them with CQL in small data regimes.
5.1 BENCHMARK DATASETS
We first evaluate our approach on D4RL datasets (Fu et al. 2020). is worth mentioning that Antmaze and Kitchen datasets include few or no near-optimal trajectories, and highly require learning a value function obtain effective policies via "stitching". We compare SQL with prior state-of-the-ard
offline RL methods, including BC (Pomerleau) 1989), 10%BC (Chen et al. 2021), BCQ (Fujimoto al. 2018 DT (Chen al., 202T). TD3+BC (Fujimoto & Gu 2021). One-step RL (Brandfonbrener et al. 202T), CQL. (Kumar al. 2020). and IQI. (Kostrikov et al. 2021a). Aggregated results are displayed Table []] MuJoCo tasks, where performance already saturated, SQL. and EQL. show competitive results to the best performance of prior methods. In more challenging AntMaze and Kitchen tasks, SQL and EQL outperform all other baselines by a large margin. This shows the effectiveness of value learning in SQL and EQL. We show learning curves and performance profiles generated by the rliable library (Agarwal et al. (021) in Appendix [
We then compare approach with other baselines in RL Unplugged (Gulcehre et al. 2000). Our approach also achieves superior performance on these datasets, show aggregated results, performance profiles and experimental details in Appendix D)
high-dimensional image- baseee-bbedddaae Atari datasets
8

---

Published as a conference paper at ICLR 2023
5.2 NOISY DATA REGIME
Walker2d
Halfcheetthh
In this section, we try to validate our hypothesis that the sparsity term our algorithm introduced learning the value function will benefit when the datasets contain a large portion of noisy transitions. To do so, make a "mixed" dataset by combining ran- dom datasets and expert dataset with different expert ratios. We test the per- formance of SQL, EQL and IQL under different mixing ratios in Fig. !
SCL
poL
IQL
185
Normalized
25%
12%
2% Expert ratio
Expert ratio
Figure 1: Performance of different methods in noisy data regimes.
It is shown that SQL and EQL outperforms IQL under all settings. The performance of IQL is vulnerable the expert ratio, it has sharp decrease from 30% to 1% while SQL and EQL still retain the expert performance. For example, in walker2d, SQL and EQL reaches near 100 performance when the expert ratio is only 5%; in halfcheetah, IQL is affected even with a high expert ratio (30%).
5.3 SMALL DATA REGIME
In this section, we try to explore the benefits of in-sample learning over out-of-sample learning. We are interested to see whether in-sample learn- ing brings more robustness than out-of-sample learning when the dataset size is small or the dataset diversity of some states small, which are challenges one might encounter when using offline RI. algorithms on real- world data.
Table 2: The normalized return (NR) and Bellman error (BR) of CQL, SQL and EQL in small data regimes.
COL
EQU
Dalaset (AniMazze)
NR
BF
NR
BE
NR
BE
Vanilla
65.2
13.1
75.1
1.6
74.0
2.2
Medium
Medium
Fasy
48.2
14.5
14.7
14.8
43.3
2.1
1.7
39.7
57.5
2.3
Hard
64.4
24.2
1.9
19.6
Vanilla
38.4
13.5
50.2
1.4
19.6
1.7
Easy
28.1
12.8
40.5
1.5
40.4
1.7
To do so, we make custom datasets by discard- ing some transitions in the AntMaze datasets.
Lange
Medium
1.3
Hard
300.5
34.2
2.6
31.6
1.6
For each transition, the closer it is to the target location, the higher probability it will be discarded from the dataset. This simulates the scenarios (i.e., robotic manipulation) where the dataset is fewer and has limited state coverage maybe not be successful and are more determined when they get closer to the target location (Kumar al. 2022). We use hyperparameter to control the discarding ratio and build three new tasks: Easy, Med um and rd, with dataset becomes smaller. For details please refer to Appendix [ We compare SQL. with CQL they use the same inherent value regularization but SQL uses in-sample learning while CQL uses out-of-sample learning,
the target location because the (stochastic) data generation policies
We demonstrate the final normalized return (NR) during evaluation and the mean squared Bellman еrror (BE) during training in Table 2 It is shown that CQL has a significant performance drop when the difficulty of tasks increases, and the Bellman error also exponentially grows up, indicating that the value extrapolation eтror becomes large in small data regimes. SQL and EQL remain a stable yet good performance under all difficulties, the Bellman error of SQL much smaller than that of CQL.. This justifies the benefits of in-sample learning, i.e., it avoids erroneous value estimation by using only dataset samples while still allowing in-sample generalization to obtain a good performance.
CONCLUSIONS AND FUTURE WORK
In this paper, we propose a general Implicit Value Regularization framework, which builds the bridge between behavior regularized and in-sample learning methods in offline RL. Based on this framework, we propose two practical algorithms, which use the same value regularization in existing works, but in complete in-sample manner. We verify the effectiveness of our algorithms on both the D4RL benchmark and customed noisy and small data regimes by comparing it with different baselines. One future work is to scale proposed framework to online RL offline imitaiton learning (Li et al.) 2023a). Another future work is. instead of only constraining action distribution, constraining the state-action distribution between d and aD as considered in Nachum et al. (019))

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGEMENT
This work is supported by funding from Intel Corporation. The authors would like to thank the anonymous reviewers for their feedback on the manuseripts.
REFERENCES
Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. An optimistic perspective on offline reinforcement learning. In Proc. ICML. pp. 104-114, 2020.
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Marc Bellemare. Deep reinforcement learning the edge of the statistical precipice. Proc. of NeurIPS, 2021.
Zafarali Ahmed, Nicolas Le Roux, Mohammad Norouzi, and Dale Schuurmans. Understanding the impact of entropy on policy optimization. In Proc. of ICML, pp. 151-160, 2019.
Gaon An, Seungyong Moon, Hang-Hyun Kim, and Hyun Oh Song. Uncertainty- based offline reinforcement learning with diversified q-ensemble. Proc. of NeurlPS, 2021.
Chenjia Bai, Lingxiao Wang, Zhuoran Yang, Zhi-Hong Deng, Animesh Garg, Peng Liu, and Zhaoran Wang. Pessimistic bootstrapping for uncertainty-driver offline reinforcement learning. In Proc. of ICLR, 2021.
Stephen Boyd, Stephen P Boyd, and Lieven Vandenberghe. Convex optimization. Cambridge university press, 2004.
David Brandfonbrenerr William F Whitney, Rajesh Ranganath, and Joan Bruna. Offline rl without off-policy evaluation. Proc. of NeurIPS, 2021.
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbecl, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Proc. of NeurlPS, 2021.
Xinyue Chen, Zijian Zhou, Zheng Wang, Che Wang, Yanqiu Wu, and Keith W. Ross. BAIL: best-action imitation learning for batch deep reinforcement learning. In Proc. NeurlPS, 2020.
Yinlam Chow, Ofir Nachum, and Mohammad Ghavamzadeh. Path consistency learning in tsallis entropy regularized mdps. In Proc. of ICML, pp. 977-987,11 2018.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. ArXiv preprint, 2020.
Scott Fujimoto and Shixiang Shane Gu. A minimalist approach offline reinforcement learning. ArXiv preprint, 2021.
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In Proc. of pp. 1582-1591, 2018
Matthieu Geist, Bruno Scherrer, and Olivier Pietquin. A theory of regularized markov decision processes. In Proc. of ICML, pp. 2pp.2166 2160-2169, 2019.
Caglar Gulcehre, Ziyu Wang, Alexander Novikov, Thomas Paine, Sergio Gómez, Konrad Zolna, Rishabh Agarwal, Josh S Merel, Daniel J Mankowitz, Cosmin Paduraru, et al. RI unplugged: A suite of benchmarks for offline reinforcement learning. In Proc. of NeurIPS, 2020.
Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep energy-based policies. In Proc. of ICML, pp. 1352-1361, 2017.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Proc. of ICML, pp. 1856-1865, 2018.
10

---

Published as a conference paper at ICLR 2023
Dmitry Kalashnikov, Jacob Varley, Yevgen Chebotar, Benjamin Swanson, Rico Jonschkowski, Chelsea Finn, Sergey Levine, and Karol Hausman. Mt-opt: Continuous multi-task robotic rein- forcement learning at scale. ArXiv preprint, 2021.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Proc. of ICLR, 2015.
Ilya Kostrikov, Rob Fergus, Jonathan Tompson, and Ofir Nachum. Offline reinforcement learning with fisher divergence critic regularization. In Proc. of ICML, pp. 5774-5783, 2021a.
llya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. ArXiv preprint, 2021b.
Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. In Proc. of NeurlPS, pp. 11761-11771, 2019.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. In Proc. of NeurlPS, 2020.
Aviral Kumar, Joey Hong, Anikait Singh, and Sergey Levine. When Should We Prefer Offline Reinforcement Learning Over Behavioral Cloning? ArXiv preprint, 2022.
Kyungjae Lee, Sungyub Kim, Sungbin Lim, Sungjoon Choi, and songhwai Oh. Tsallis reinforcement learning: A unified framework for maximum entropy reinforcement learning. ArXiv preprint, 2019.
Jianxiong Li, Xiao Hu, Haoran Xu, Jingjing Liu, Xianyuan Zhan, Qing-Shan Jia, and Ya-Qin Zhang. Mind the gap: Offline policy optimization for imperfect rewards. Proc. of ICLR, 2023a.
Jianxiong Li, Xianyuan Zhan, Haoran Xu, Xiangyu Zhu, Jingjing Liu, and Ya-Qin Zhang, When data geometry meets deep function: Generalizing offline reinforcement learning. In Proc. of ICLR, 2023b.
Timothy Lillicrap. Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In Proc. of ICLR, 2016.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602, 2013.
Ofir Nachum, Mohammad Norouzi, Kelvin Xu, and Dale Schuurmans. Bridging the gap between value and policy based reinforcement learning. In Proc. of NeurlPS, pp. 2775-2785, 2017.
Ofir Nachum, Bo Dai, Ilya Kostrikov, Yinlam Chow, Lihong Li, and Dale Schuurmans. Algaedice: Policy gradient from arbitrary experience. ArXiv preprint, 2019.
Ashvin Nair, Murtaza Dalal, Abhishek Gupta, and Sergey Levine. Accelerating online reinforcement learning with offline datasets. ArXiv preprint, 2020.
Gergely Neu, Anders Jonsson, and Vicenç Gómez. A unified view of retropy-regullrizee markov decision processes. ArXiv preprint, 2017.
Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Srrantage-weightee regression: Simple and scalable off- off-policy reinforcement learning. ArXiv preprint, 2019.
Jan Peters, Katharina Mülling, and Yasemin Altun. Relative entropy policy search. In Proc. of AAAI, 2010.
Dean A Pomerleau. Alvinn: An autonomous land vehicle in neural network. In Proc. of NeurlPS, 1989.
Takuma Seno and Michita Imai. d3rlpy: An offline deep reinforcement learning library. ArXiv preprint, 2021.
11

---

Published as a conference paper at ICLR 2023
David Silver, Julian Schrittwieser, Karen Simonyan, loannis Antonoglou. Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. nature, 2017.
Richard S Sutton, Andrew Barto, al. Introduction reinforcement learning. MIT press Cambridge, 1998.
Shengpu Tang and Jenna Wiens. Model selection for offline reinforcement learning: Practical considerations for healthcare settings. In Proc. of ML4H, 2021.
Ziyu Wang, Alexander Novikov, Konrad Zolna, Josh Merel, Jost Tobias Springenberg, Scott E. Reed, Bobak Shahriari, Noah Y. Siegel, Çaglar Gülçehre, Nicolas Heess, and Nando de Freitas. Critic regularized regression. In Proc. of NeurlPS, 2020.
Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning.
ArXiv preprint, 2019.
Yue Wu, Shuangfei Zhai, Nitish Srivastava, Joshua M. Susskind, Jian Zhang, Ruslan Salakhutdinov, and Hanlin Goh. Uncertainty weighted actor-critic for offline reinforcement learning. In Proc. of ICMI,, pp. 11319-1153 1328, 2021.
Haoran Xu, Xianyuan Zhan, Jianxiong Li, and Honglei Yin. Offline reinforcement learning with soft behavior regularization. ArXiv preprint, 2021.
Haoran Xu, Li Jiang, Jianxiong Li, and Xianyuan Zhan. policy-guided imitation approach for offline reinforcement learning. In Proc. of NeurlPS, 2022a.
Haoran Xu, Xianyuan Zhan, Honglei Yin, and Huiling Qin. peeriminnator-wiighee offlin imitation learning from suboptimal demonstrations. In Proc. of ICML, pp. 24725-24742, 2022b.
Haoran Xu, Xianyuan Zhan, and Xiangyu Zhu, Constraints penalized q-learning for safe offline reinforcement learning. In Proc. of AAAI, pp. 8753-8760. 2022c.
Xianyuan Zhan, Haoran Xu, Yue Zhang, Xiangyu Zhu, Honglei Yin, and Yu Zheng. Deepthermal: Combustion optimization for thermal power generating units using offline reinforcement learning. In Proc. of AAAI, pp. 4680-4688, 2022.
Wenxuan Zhou, Sujay Bajracharya, and David Held. Latent action space for offline reinforcement learning. In Conference on Robot Learning, 2020.
12

---

Published as a conference paper at ICLR 2023
A A STATISTICAL VIEW OF WHY SQL AND EQL WORK
Inspired by the analysis in IQL, we give another view of why SQL and EQL could learn the optimal policy. Consider estimating parameter ma for random variable X using samples from dataset D, we show that ma could fit the extrema of X by using the learning objective of V V-funccion in SQL:
(1
ma
0)
I ma
Ma
arg min ERDD 11
+
2a
20
using the learning objective of V-function in EQL:
x
ma
Ma
arg min EEWD mh
exp
+
a
In Figure 2 and Figure 3 we give an example of estimating the state conditional extrema two- dimensional random variable, as shown, - 0 approximates the maximum operator over in-support values of y given z. This phenomenon can be justified in our IVR framework as the value function becomes more optimal with less value regularization. However, less value regularization also brings more distributional shift, so we need a proper a to trade-off optimality against distributional shift.
1.00
0=0.5
a=1
0.75
0=0.11
g=0.1
Derivative of Loss
-- 0.5 (SQL)
(SQL)
a=1.5
0.50
a=10
(IQL)
0.7 (IQL)
0.75
1.5
-.000
Residual
Residual
The loss with respect to the residual (Q V) in the learning objective of V in SQL with different Center: An example of estimating state conditional extrema of voo-dimensional random variable (generated by adding random noise samples from y sin())). Each corresponds derivative loss of loss fits the extrema more with a becoming smaller. Right: The comparison SQL and IQL. In SQL, the derivative keeps unchanged when the residual is below a threshold.
Figure 2: Left:
distribution over y-
1.00
Derivative of Loss
am0.5
3.75
am0.55
2-0.5 (FQL)
a=1
a=0.1
am0.7 (EQL)
q=1.5
3.59
e=10
r-0.9 (IQL)
LOSs
T=0./ [IQL)
20 -15
-1.0
-05
0.
035
10
15
20
-2
-1
20 -13 1.0
-03
0.2 05
1.5
15 2.
Residual
Residual
The loss with respect to the residual (Q V) in the learning objective of V in EQL with different Center: An example of estimating state conditional extrema of two-dimensional random variable (generated by adding random noise samples from y sin(z)). Each z corresponds distribution over y. The loss fits the extrema more with a becoming smaller. Right: The comparison the derivative loss of EQL and IQL. In EQL, the derivative softly decreases and keeps (nearly) unchanged when the residual is below threshold.
Figure 3: Left:
B How DOES SPARSITY BENEFIT VALUE LEARNING IN SQL?
In this section, we add more experiments about the sparsity characteristic of SQL. We use a toy example in the tabular setting demonstrate how sparsity benefits value learning in SQL. We study the relationship of normalized score and sparsity ratio with a the continuous action setting to clearly show that sparsity plays an important role in the performance of SQL..
13

---

Published as a conference paper at ICLR 2023
True optimal V
IQL
SQL
10
Figure 4: Evaluation IQL SQL on the Four Rooms environment. SQL learns more optimal value function and produces better policy than IQL when the datasct heavily corrupted by suboptimal actions.
Table The relationship of normalized score (left) non-sparsity ratio (right)
MuJoCo datasets.
ratio
halfcheetah-m
0.72
48.7
0.91
48.1
0.97
1.00
hopper-m
62.5
0.84
74.5
0.90
74.1
0.98
68.5
0.99
62.4
0.99
walker2d-m
22.3
0.03
65.3
0.93
84.2
0.98
83.7
0.99
84,1
0.99
haffeheetah-m--
43.2
0.65
44.2
0.84
44.8
0.89
44.9
0.97
44.8
0.99
hopper-m-r
43.5
0.53
95.5
0.74
100.7
0.94
74.2
0.99
walker2d-me
1.00
halfcheetah-me
40.2
0.38
39.3
0.33
35.8
0.27
94.2
0.99
94.8
0.99
hopper-m-e
18.6
0.07
106.3
0.89
106.6
0.94
111.9
0.96
111.5
0,99
walker2d-m-e
9.2
0.01
110.2
0.95
111.3
0.966
109.5
0.99
111.2
0.99
mujoco-mean
30.4
0.41
69.2
0.80
76.4
0.85
78.1
0.97
79.0
0.99
Table The relationship of the normalized score (left) and non non-sparsion sparsity ratio (right) with a AntMaze datasets.
0.7
Score
Score
ratio
Score
antmaze-m-d
40.3
62.6
0.39
0.35
68.6
0.70
0.82
63.4
0.77
0.92
antmaze-m-p m-
0.0
0.03
70.2
0.32
80.2
0.62
67.5
0.81
69.3
0.83
67.3
0.75
2.0
0.88
aotmaze-I-d
35,3
0.0
55.2
0.56
50.2
0.70
39.5
0.75
18.3
0.91
20.5
0.88
4.1
0.98
antmaze-I-p
0
0.48
38.3
0.33
52.3
0.51
18.2
0.70
20.3
0.89
11.5
0.80
2.1
0.95
antmaze-mas
18.9
56.6
0.40
64.5
0.79
48.5
0.74
42.4
0.86
40.7
0.80
7.5
0.94
B. SPARSITY IN THE TABULAR SETTING
In Appendix A, show the comparison of loss's derivative of SQL and IQL., found that SQL keeps the derivative unchanged when the residual is below a threshold while IQL doesn't 't, the V-function in IQL will be over-underestimated by those bad actions whose Q-value is small.
To justify this claim, we use the Four Rooms environment, where the agent starts from the bottom-left and needs navigate through the four rooms reach the goal in the up-right corner in few steps as possible. There are four actions: A {up, down, right, left). The reward is zero each time step until the agent reaches the goal-state where receives +10. The offline dataset is collected by random behavior policy which takes each action with equal probability. We collect 30 trajectories and each trajectory terminated after 20 steps if not succeed, 7 0.9.
It can be seen from Fig. that the value learning in IQL is corrupted by those suboptimal actions in this dataset. Those suboptimal actions prevent IQL from propagating correct learning signals from the goal location to the start location, resulting underestimated --values and some mistaken Q-values. Particularly, incorrect Q-values at (1, 1) and (5, 9) make the agent fail to reach the goal. While in more identical to the true optimal ones and the agent succeeds in
SQL, --values and Q-values reaching the goal location. This reveals that the sparsity term in SQL helps to alleviate the effect of bad dataset actions and learn more optimal value function.
14

---

Published as a conference paper at ICLR 2023
B.2 SPARSITY THE CONTINUOUS ACTION SETTING
The value of non-sparsity ratio (i.e., ((s,a)>
- > 0)D is controlled by the hyperparameter In the continuous action setting, we show the relationship of the normalized score and non -ppnsiiy ratio with a in Table and Table 4 It can be seen that typically a larger gives less sparsity, sparsity plays an important role in the performance of SQL and we need to choose proper sparsity ratio to achieve the best performance. The best sparsity ratio depends on the composition of the dataset, for example, the best sparsity ratios in MuJoCo datasets (around 0.1) are always larger than those in AntMaze datasets (around 0.4), this is because AntMaze datasets are kind of multi-task datasets (the start and goal location are different from the current ones), there is large portion of useless transitions contained it is reasonable to give those transitions weights using more sparsity.
C PROOFS
C.I
PROOF OF THEOREM I
In this section, we give the detailed proof for Theorem I which states the optimality condition of the behavior regularized MDP. The proof follows from the Karush-Kuhn Muhn-Tucckr (KKT) conditions where the derivative of Lagrangian objective function with respect to policy π (als) is set zero. Hence, our main theory is necessary and sufficient.
Proof. The Lagrangian function of
4 is written as follows
(((s.a) -af [(als)
(
B. = Σ d (s) Σ (alss)
((aas))
Σ u(s)
(s)
Σ T(als) -
Σ B(als)
where d is the stationary state distribution of the policy и, and are Lagrangian multipliers for the equality and inequality constraints respectively.
Let hr(x) fx)... Then the KKT condition of follows, for all states and actions we have
S (als) S and
Σ x (als) =1
(18)
0 0 ((ass)
(19)
((als)<[als)) =
(20)
(Taas)) u(s) + B(als)
Q(s.a) - an', ppaas)
(21)
where (1) the feasibility of the primal problem, 19 is the feasibility of the dual problem, 20
results from the complementary slackness and (2I) is the stationarity condition. We eliminate d d'(s) since we assume all policies induce an irreducible Markov chain.
From 21 we can resolve π (alss)
(als) = p(aas)) gs (2 (Q(s + M(lls))
Fix a state 8. For any positive action, its corresponding Lagrangian multiplier B(ass) is zero due to the complementary slackness and Q(s, a) > u(s) + ah' must hold. For any zero-probability action, its Lagrangian multiplier ((ass) will be set such that <(als) Note that B(als) > thus Q(s,a) < u(s) + (0) must hold in this case. From these observations, क ((a]s) can be reformulated as
(a]s) ) = (aals) max ( -(ss)) o)
(22)
15

---

Published as a conference paper at ICLR 2023
By plugging (22 into (18) we can obtain equation
Bana max {or g
(Q((..)) - (=(0(+)))-u(())) .0
(23)
Note that (3 has and only has one solution denoted u² (because the LHS of (23 be seen continuous and monotonic function of u), so u² can be solved uniquely. We denote the corresponding policy π as n².
Next we aim to obtain the optimal state value V*, It follows that
(*) = (s)
Σ (als) ("(ss)) -af (
m(als)
Ia'(als) 1' u'(s) + [(lss)
n (als)
((aas)
((ass)
(als))
T (als))
(s) + a
u(als)
m(als)
2 (s) + aBarp ( 8 (mlals) (T
T
The first equality follows from the definition of the optimal state value. The second equality holds
because maximizes V The third equality results from plugging 21
To summarize, we obtain the optimality condition of the behavior regularized MDP as follows
Q" a) r(s, a) + Es [V* (s')l
<a(as)) = p(als) max
Q' (s, a) (s)) :00 ( a
.00
g1
(IT(ols)) f t (als) (Maas))
V*(s)= = u (s) + aBany
m(als)
C.2 PROOF OF THEOREM 2
Proof. For any two state value functions V1 and V22 let be the policy that maximizes Tj V.. E 1,2. Then it follows that for any state s in S.
Vi) (s)
(T V2) (s)
Ti (alss)
] max Σ x (als)
r + 7B. (V/ (8')] af ((als))
Σ (als)
r + +E. [V1 (s')) - af
((aas))
p(a)s)
(mi(als)
Σ (alss) r r (V2 (s') af (a)s))
$ Σ (alss)
1 + [V (5')) - af
(M(aas)
p(als)
Σ (c IV1 (s') V/2 (s')) $ r ||V Velloo
By symmetry, it follows that for any state & in S,
(Ti (8) - (T (s) S7|VV V2llxo
Therefore, follows that
TTV-- - TVV/l 57 V2ll.0
16

---

Published as a conference paper at ICLR 2023
D EXPERIMENTAL DETAILS
D4RL experimental details For MuJoCo locomotion and Kitchen tasks, we average mean returns over 10 evaluations every 5000 training steps, over 5 random seeds. For AntMaze tasks, we average over 100 evaluations every 0. training steps, over 5 random seeds. Followed by IQL, we standardize the rewards by dividing the difference returns of the best and worst trajectories in MuJoCo and kitchen tasks, we subtract to rewards in AntMaze tasks.
Our implementation of 10%BC is as follows, we first filter the top % trajectories in terms of the trajectory return, and then run behaviour cloning those filtered data. We re-run IQL. on all datasets and report the score of IQL by choosing the best score from [0.5, 0.6, 0.7, 0.8, 0.9, 0.99], using author-prooiddes implementation"" We re-run CQL on AntMaze datasets can be improved by carefully sweeping the hyperparameter mi -iln--weiigg in [0.5, 1, 2, 5, 10), using the PyTorch-vrrsie version implementation?) Other baseline results are taken directly from their corresponding papers.
find the performance
In SQL and EQL,, use 2-layer MLP with 256 hidden units, 2015) with a learning rate of 2 10-1 for all neural networks. Following Mnih et al.
Adam optimizer (Kingma Ba
2013 Lillicrap
al. 2016), we introduce target critic network with soft update weight 5 10
We implement
our method in the framework of JAX. The only hyperparameter a used in SQL and EQL is listed in
Table 5 The sensitivity of in SQL be found in Table and Table The sensitivity of in IQL
can be found in Table The runtime of different algorithms can be found in Table 7
Table 5: used for SQL and EQL
Env
(SQL)
(EQL)
ballcheetah-medhheee
2.0
2.0
supper-medium-22
2.0
2.0
walker2d- 2d-rediummm222222
2.0
2.0
halfcheetah- mwd-im--ddpmlaeeeeee veplay-----
2.0
2.0
hopper- wwwdmm-ddeeeeeeeeeee replaa-----
2.0
2.0
walker menlkeerd-medium--ae medium-replay--aeee
2.0
2.0
halfcheetah- -eeith----iiuueeeeeeeeeee
5.0
5.0
hupper-meddimm-<xpertt22
5.0
5.0
walker2ddmmcdium-xxpee
5.0
5.0
antmaze-umaze-v2
0.5
0.5
untmaze-umuzz-diivrese
5.0
5.0
matmaze-medium-play--222
0.5
0.5
untmazz-mccium--diieess---
0.5
0.5
untmaze-large-laay-y
0.5
0.5
autmaze-argeediverseey.
0.5
0.5
kitchen-c
2.0
2.0
kitchen-p
2.0
2.0
kitchen-m
2.0
2.0
Table The sensitivity of 7 in IQL.
0.5
0.6
0.7
0.8
0.9
0.99
hopper-m-r
57.1
71.3
95.2
74.4
59.5
hopper-m-c
99.7
101.1
94.5
22.5
13.5
30.4
walker2d-m-r
76.1
72.3
41.7
20.3
walker2d-m-e
109.94
106.7
109.6
109.3
78.2
50.3
antmaze-m-d
51.6
71.0
12.1
antmaze-m-p
0
8.0
51.2
72.1
11.5
antmaze-l-d
0
1.2
12.4
47.5
7.6
antmaze-l-p
10.4
39.6
RL Unplugged experimental details
We use d3ripy (Seno & Imai] 2021).
modularized offline
RL library that contains several SOTA offline RL algorithms and provides easy-to-use wrapper
17

---

Published as a conference paper at ICLR 2023
Table
The runtime of different algorithms.
algorithms
BC
10%BC
BCQ 60m
DT
One-step
TD3+BC
CQL
IQL
SQL
EQL
runtime
20m
20m
950m
20m
25m
80m
20m
20m
20m
for the offline Atari datasets introduced in (Agarwal et al. 2020). There are three types of Atari datasets d3rlpy: mi xed: datasets collected at the first 1M training steps an online DQN agent, medi um: datasets collected at between 9M steps and 10M training steps of online DQN agent, expert: datasets collected the last 1M training steps of an online DQN agent. To make the task more challenging, games: Breakout, Qbert and Seaquest.
only 10% or 5% of origin datasets. We choose three image-based Atari
We implement the discrete version of SQL (D-SQL) and IQL (D-IQL) based on d3rlpy. The implementation discrete CQL (D-CQL) and discrete BCQ (D-BCQ) are directly taken from d3rlpy. We consistent preprocessing and network structures to ensure a fair comparision.
For baselines, we report the score of D-IQL by choosing the best score from T in [0.5, 0.7,0.9). 0.9], we report the score of D-CQL by choosing the best score from mi min--g weiiin--wwigeee weight in 21, 5), we report the score of D-BCQ by choosing the best score from T in [0.1, , 0.3, 0.5). For D-SQL, we use a = 1.0 for all datasets.
Nosiy data regime experimental details In this experiment setting, we introduce the nois sy dataset by mixing the expert and random dataset with different expert using MuJoCo locomotion datasets. The number of total transitions of the noisy dataset is 100, 000. We provide details in Table We report the score of IQL by choosing the best score from T in 00.5, 0.6, 0.7, 0.8, 0.9).
8
Table 8: Noisy dataset of MuloCo locomotion tasks with different expert ratios.
Expert
transitions
Expert transitions
1%
100,000
1,000
99,000
5%
100,000
5,000
95,000
Walker2d
10%
100.000
10,000
90,000
20%
100,000
20,000
80,000
30%
100.000
30,000
70,000
1%
100,000
1,000
99,000
5%
100,000
5,000
95,000
Halfcheetah
100,000
20%
100,000
20,000
80,000
Small data regime experimental details We generate the small dataset using the following psedocode [ its hardness level can be found at Table We report the score of CQL by choosing the best score from min-q-weight in [0.5, 1, 2, 10).
Listing 1: The sketch of generation procedure of small data regimes with different hard levels. Given an AntMaze environment and a hardness level, discard some transitions by following the rule the Coding List. Intuitively, the closer the transition to the GOAL, the higher the probability that it will be discarded.
LEVEL ('easy', 'medium', 'hard') obs dataset observations') length dataset I' observations! 1. ssapee
POSITIONS = www.:ee__posstionncccmo
GOAL - env.get_goal1)
MINIMALL_OSIITUR
get maximal cuclidead distance
MAX EU _U_DIIS (GOAL MINIMAL POSITION) THNIMAL POSITION +*2
DIS (- (PPSIITIONS
POSITIOON +*2)
MAX EU DIS
save icx rp.random. wandom(size=lenegeeee small_data www.eetions. /el.ettiioos.fffuulllllttl// for key in dataset. (eys small_data(key) datasct [key] [save_idk]
(DIS hardness LEVEL')
18

---

Published as a conference paper at ICLR 2023
Table Details of small data regimes with different task difficulties.
Dataset (AntMaze)
Hardness
Total transitions
Reward signals
Vanilla
NA
100. 000
10,000
medium-play
Easy
56,000
800
Medium
0.07
48,000
150
Hard
45,000
Vanilla
NA
100,000
12500
Easy
0
72,000
5,000
lurge-play
Medium
42,000
1,000
Medium
0.35
37,000
500
Hard
0.38
35,000
100
hallchretal--meiiun-2
hopger-me-dam-2
wilker26-medium-2
SQL
Normalized Recurn
60
60
0.4
Steps (1e6)
Time Steps (teps
Time Steps (1e6)
halfcheotall mww.um-momlla----22222
walker2d -aliur----mi
Normalized Recurn
100
100
60
244
0.5
0.2
0.4
0.5
03
0.2
0.5
0.5
Time Stops (1c6)
Time Stops (Le6)
Time Steps (1e6)
halfchnetah mwdium-dddeemmee expert-v2
hopper-medium expert-12
vallka2ddmmedumm expert-12
Normalzzed Returr
80
80
40
0.4
Steps (1c6)
Time (1e6)
Time Steps (1c6)
Figure 5: Learning curves of SQL on D4RL MuJoCo locomotion datascts.
diverse-v2
antmaae mediumeddmmmm diverse
Normalied Return
100
SQL
100
80
20
20
20
02
0.4
0.8
0.
02
2.4
0.5
1/5
0.2
02
0.4
0.5
0.5
Time Steps (1c6)
Time Steps (1e6)
Time Steps (1cb)
madium v2
antmaze large
Normalized Return
100
100
80
20
20
0.
0.4
0.8
0.
02
1.4
0.5
155
0.3
022
0.5 Time Steps (1c6)
0.5
Time Stops (1.6)
Time Staps (1e6)
Figure 6: Learning curves of SQL D4RL AntMaze datascts.
19

---

Published as a conference paper at ICLR 2023
Ktchen-compliic--g
kitchen- partial--0
kithhen-mieee miiee-vvioo
Normalized ficuunn
100
SQL
100
110
80-
Time Steps (1e6)
Time (1e6)
Time Steps (1e6)
Figure 7: Learning curves SQL on D4RL Kitchen datasets.
Jouns
Fradton of tasks with soore ((1)
0.75
0.78
0.50
0.50
of
0.25
0.25
Figure 8: Performance profiles of CQL, IQL and SQL generated by the rliable library (Agarwal et al 2021 on D4RL datasets based distributions (left), and average score distributions (right). Shaded regions show pointwise 95% confidence hands based on percentile bootstrap with stratified sampling.
Table 10: Performance in setting with 10% (top) and 5% (hottom) Atari dataset. SQL achieves the best performance in 10 out of 12 games.
Task
D-BCQ
D-IQL.
D-CQL
D-SQL
maakkout-medium-Vo (10%)
20.1
15.1
28.0
mediu-medium-o0
395.4
3717.5
4141.5
5213.4
seaquest-mediuu--00
438.1
404.9
359.4
465.3
breakout-mixxed-v0
10.9
9.3
13.3
qbert- qhert-mixod-v0
557.5
1000.3
890.2
1244.3
scaquest-mixed-v0
300.3
326.1
337.5
330.4
breakout-mmedium--00 (5%)
13.9
qbert medium-ddiid
301.6
2788.7
3147.3
2970.6
301.9
294.5
272.5
361.3
breakout-mixed-v0)
10.5
8.7
11.8
gbert-mixed-v0
576.4
931.1
925.3
965.0
seaquest-mixed-v00
275.5
292.2
321.4
336.7
20


Reviews:
[{"review_id": "x4KXQRrJXt", "reviewer": "Reviewer_u1bC", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper introduces Implicit Value Regularization (IVR) framework, provides some interesting theoretical results that might help better understand offline methods such as CQL and IQL, and then under this framework, a new offline RL method called Sparse Q-learning (SQL) is proposed. \n\nEmpirical results from D4RL benchmark shows a consistent superior performance over competing algorithms. Authors also provided some other analysis on SQL and also compared to IQL on noisy data regime and CQL on small data regime (mixed datasets made from D4RL datasets). ", "summary_of_the_review": "Overall seems a solid paper, some interesting results and theoretical analysis. Overall writing is good. \n\nMy main concern: \n- Additional technical details and whether other competing methods also had thorough hyperparameter tuning on each task should be provided, also the computation efficiency comparison, and final hyperparameters for each dataset would be good to have in the paper. \n- I'm looking for a bit more analysis on your method, for example on the sparsity thing\n\nI'm willing to increase my score if concerns are fully addressed and if anything is wrong in my comments please point it out. \n\n=============== post rebuttal ===============\nSorry for the late reply, I thank the authors for their effort in addressing the concerns and providing the additional results. The new results seem to be more convincing. I have also checked other reviewers' comments and it seems most of the major concerns are addressed by the rebuttal. Based on that I increase my score to 8.\n", "strength_and_weaknesses": "Strengths:\n- Overall good writing quality and clarity. \n- Interesting theoretical results that might help better understand offline methods that rely on some form of regularization terms. \n- Empirical results on D4RL and other 2 comparisons seem to show SQL has stronger performance than alternative methods, and some of these methods are recent methods with good performance. \n- Technical details provided for reproducibility. \n- a lot of discussion on related works, easy to understand how it relates to prior works\n\nWeaknesses:\n- Thank you for providing technical details including your hyperparameter search range. However, I am a bit concerned that it seems you performed a hyperparameter search on each dataset individually, and then select the best-performing one. This makes sense, but for the methods you compare to, are they using the same hyperparameter for all tasks, or do they also search hyperparameter on each individual dataset and report the best one? \n- Similarly, in the case when you compare SQL to CQL and IQL on the noisy and small data regime settings, did you also do a thorough hyperparameter search for IQL and CQL? \n- Maybe I missed sth but have you provided empirical evidence and figures to demonstrate that your method has a more \"sparse\" policy than the other competitive methods? \n- What is the final hyperparameter you selected for each individual dataset? (You mentioned the best is selected for each dataset, would be good if you also report the actual numbers)\n- What about computation efficiency? How fast in wall-clock speed does your method compare to others? \n- Hyperparameter sensitivity: for example can you provide some results on how performance changes with different alpha values? This will help other researchers understand how hard it is to tune your method on new tasks. \n\nNot really important: \n- some minor typos\n\nOther Questions: \n- Will your code be open sourced? ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "vLsp7Gg59H", "reviewer": "Reviewer_HMTv", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper theoretically derives an in-sample offline RL algorithm, Sparse Q-Learning (SQL), using a behavior regularized MDP for offline reinforcement learning. SQL has interesting connections to existing offline RL algorithms including CQL and the in-sample IQL algorithm. SQL performs similarly to prior methods on MuJoCo and Antmaze tasks and better on Kitchen tasks in D4RL. SQL is shown to be superior on custom smaller datasets than CQL as well as more stable than IQL on mixture datasets containing expert and random data.", "summary_of_the_review": "Overall, the paper presents theoretically well-motivated method for addressing the ad hoc choices in IQL. However, the method doesn't seem to offer much empirical gains over IQL on the standard benchmarks. However, other than robustness results on mixture datasets (noisy data regime in Section 5.1), it is unclear whether SQL offers any benefits over IQL in terms of practical benefits. \n\nQuestions / Suggestions:\n- Can you provide empirical evidence for whether SQL provide any benefits over IQL in the small-data regime in Section 5.2?\n- In-line with the best practices for evaluation, I'd recommend the use of [rliable](https://github.com/google-research/rliable) library [1] to validate the claim for SOTA results. Furthermore, standard deviation be reported for all methods in Table 1.\n- Does the method scale to high-dimensional image-based datasets such as the Atari datasets in RL Unplugged [2]? \n- Kumar et. al (2022) showed degradation in CQL performance with prolonged training (including on the Antmaze datasets) -- would SQL  provide any benefit compared over CQL / IQL due to being a principled approach for in-sample learning?\n- Are there any guidelines to set the $\\alpha$ parameter -- given that SQL is theoretically motivated, it would be nice if the hyperparameter tuning is easier / more intuitive than existing methods like CQL / IQL.\n\nMinor:\n- Can you clarify how Jensen's inequality is applied in Equation 2?\n- The generality of the IVR framework would be seen if another value of $\\alpha$ was instantiated to derive in-sample version of another existing method or deriving a new method altogether.\n\n\nReferences:\n\n[1] Agarwal, R., Schwarzer, M., Castro, P. S., Courville, A. C., & Bellemare, M. (2021). Deep reinforcement learning at the edge of the statistical precipice. Advances in neural information processing systems, 34, 29304-29320.\n\n[2] Gulcehre, C., Wang, Z., Novikov, A., Paine, T., Gómez, S., Zolna, K., ... & de Freitas, N. (2020). Rl unplugged: A suite of benchmarks for offline reinforcement learning. Advances in Neural Information Processing Systems, 33, 7248-7259.\n\n[3] DR3: Value-Based Deep Reinforcement Learning Requires Explicit Regularization (2022). ICLR. https://openreview.net/forum?id=POvMvLi91f\n\n\n-------\n----------  Post rebuttal ---------- \n\nBased on authors's response as well as the updated paper, I am now in favor of acceptance and updated the score to 8 (accept) from 5 (below accept).", "strength_and_weaknesses": "Strengths:\n\n- The paper is well written, especially the theoretical section.\n- The proposed SQL method is theoretically well-motivated. The theory also points out interesting connections to existing methods including CQL and IQL. Furthermore, SQL only requires one hyperparameter as opposed to two in IQL.\n- The implicit value framework derived in the framework seems like a general framework for instantiating in-sample Q-learning methods based on $\\alpha$-divergences (although most methods might turn out to be intractable).\n\nWeaknesses\n\n- The improvements on the D4RL benchmark do not seem to be statistically significant -- Reported results are likely within 1 standard deviation of baseline results and thus the SOTA claim may not be valid.\n   - Error bars are not reported for any of the baselines in Table 1 and 2 -- however, the standard deviation is known to be large for D4RL datasets. See https://openreview.net/pdf?id=Y4cs1Z3HnqL for a paper that reports standard deviation for some of the baseline methods.\n   -  The proposed method seems more complex to implement than existing methods such as IQL, while resulting only in marginal gains. \n-  Possibly Unfair Comparisons: It seems that per-task hyperparameter tuning is done for SQL while the baseline methods' results seem to be copied from prior papers which used the same hyperparameter for a given domain (Kitchen / Antmaze / MuJoCo).\n- It's not clear whether \"sparsity\" plays an important role in performance of SQL -- this is not explored much in the paper despite the title.\n-  \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "0K1pMYWBvtm", "reviewer": "Reviewer_PvW6", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper studies offline reinforcement learning by proposing the Implicit Value Regularization (IVR) framework. In IVR general regularizers are added to the policy learning objective. Algorithms from prior work are analyzed in this framework showing some of their weaknesses. From these theoretical insights, a practical algorithm is derived that achieves strong results on the D4RL benchmark.", "summary_of_the_review": "This is an interesting paper both in terms of theory and practical results. Hence, I vote for acceptance.", "strength_and_weaknesses": "The paper is well-written and does a good job in explaining relevant concepts. The theoretical analysis gives interesting insights and shows also how prior algorithms (IQL and CQL) fit into this framework. Further relevant work is nicely summarized in the context of the paper and put in comparison to the proposed method. \nThe proposed method Sparse Q-learning (SQL) is derived from the theoretical framework under reasonable and well-explained assumptions.\nSQL achieve SOTA performance on the D4RL benchmark and also strong results for the noisy and the small data regime.\n\nSome typos:\np.2 \"implicit value regularization\" -> \"implicit value regularization\"\np.2 \"IQL remains much close to the...\" -> \"IQL remains much closer to the...\"\np.2 \"...similarities to IQL However...\" \"...similarities to IQL. However...\"\np.3 In the last equation defining the policy evaluation operator for V: There should be a plus instead of a minus\np.4 \"... adds a entropy...\" -> \"... adds an entropy...\"\np.5 the sentence \" U∗(s) can be uniquely solved from the equation obtained by ...\" appears twice in two consecutive paragraphs.\np.9 \"This simulates the situation where the dataset is fewer and has limited state coverage near the target location because the data generation policies maybe not be satisfied and are more determined when they get closer to the target location\"  what is meant here??", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-5%