Offline Reinforcement Learning with Value-based Episodic Memory
Authors: Xiaoteng Ma
Yiqin Yang
Hao Hu
Jun Yang
Chongjie Zhang
Qianchuan Zhao
Bin Liang
Qihan Liu
Venue: ICLR 2022 Poster (2022)
Keywords: Reinforcement Learning
Offline Learning
Episodic Memory Control

Abstract: Offline reinforcement learning (RL) shows promise of applying RL to real-world problems by effectively utilizing previously collected data. Most existing offline RL algorithms use regularization or constraints to suppress extrapolation error for actions outside the dataset. In this paper, we adopt a different framework, which learns the V-function instead of the Q-function to naturally keep the learning procedure within the support of an offline dataset. To enable effective generalization while maintaining proper conservatism in offline learning, we propose Expectile V-Learning (EVL), which smoothly interpolates between the optimal value learning and behavior cloning. Further, we introduce implicit planning along offline trajectories to enhance learned V-values and accelerate convergence. Together, we present a new offline method called Value-based Episodic Memory (VEM). We provide theoretical analysis for the convergence properties of our proposed VEM method, and empirical results in the D4RL benchmark show that our method achieves superior performance in most tasks, particularly in sparse-reward tasks.

Published as a conference paper at ICLR 2022
OFFLINE REINFORCEMENT LEARNING WITH VALUE- BASED EPISODIC MEMORY
Xiaoteng Ma'; Yigin Yang' H :u Qihan Liu',
Jun Yang' Chongjie Zhang"; Qianchuan Zhao', Bin Liang'
*Department of Automation, Tsinghua University
Instiitute for Interdisciplinary Information Sciences, Tsinghua University (ma-xt1] yangyiiggggggg qi 19, hu-h19, 1 2g2 eteeeemlmasingllllleeeee..eeee.e [yangjun60) chongiie, zhaoge, bl i inghua. eec.h en
ABSTRACT
Offline reinforcement learning (RL) shows promise of applying RL to real-world problems by effectively utilizing previously collected data. Most existing offline RL algorithms use regularization or constraints to suppress extrapolation error for actions outside the dataset. In this paper, we adopt different framework, which learns the V-function instead of the Q-function to naturally keep the learning pro- cedure within the offline dataset. To enable effective generalization while main- taining proper conservatism in offline learning, we propose Expectile V-Learning (EVL), which smoothly interpolates between the optimal value learning and be- havior cloning. Further, we introduce implicit planning along offline trajectories to enhance learned V-values and accelerate convergence. Together, we present new offline method called Value-based Episodic Memory (VEM). We provide theoretical analysis for the convergence properties of our proposed VEM method, and empirical results in the D4RL benchmark show that method achieves su- perior performance in most tasks, particularly in sparse-reward tasks. Our code is public online at https://github.com///e Yang/VEM
INTRODUCTION
Despite the great success of deep reinforcement learning (RL) in various domains, most current al- gorithms rely interactions with the environment to learn through trial and error. In real-world problems, particularly risky and safety-crucial scenarios, interactions with the environment can be expensive and unsafe, and only offline collected datasets are available, such as the expert demon- stration or previously logged data. This growing demand has led to the emergence of offline rein- forcement learning (offline RL) to conduct RL in supervised manner.
The main challenge of offline RL comes from the actions out of the dataset's support (Kumar et al, 2019 2020). The evaluation of these actions that do not appear in the dataset relies the gener- alization of the value network, which may exhibit extrapolation error (Fujimoto et al.] 2019). This can be magnified through bootstrapping. leading
estimation errors. rapidly devel- oping line of recent work (Fujimoto et al. 2019 Kumar et al., 2020 Ghasemipour et al. 2021 Yang al.) (022)) utilizes various methods constrain optimistic estimation unseen actions, such as restricting available actions with a learned behavior model (Fujimoto et al. 2019) or penalizing the unseen actions with additional regularization (Kumar et al.) 2020)). However, confining learning within the distribution of the dataset can be insufficient for reducing extrapolation errors.
Another line of methods, on the contrary, uses the returns of the behavior policy the signal for policy learning, as adopted in Wang et al. (2018): |eng et al.] (2019); CCen et al. (202)). By doing so, they keep the value learning procedure completely within the dataset. However, the behavior policy of the dataset can be imperfect and insufficient to guide policy learning. To achieve trade- off between imitation learning and optimal value learning while confines learning within the dataset,
"Equal contribution. Listing order is random.
Equul advising.

---

Published as a conference paper at ICLR 2022
Regularization/
Constraint
Expectile V-Learning
Memory
improvement
Episodic Memory
Improvement
Trajs
0
ooo
0
0
0
0
0
O
Q0(s,a) a)
(s)
0
0
0
O
O
Vo(s)
Evaluation
Update Memory
Update Target
Q-based Offline RL
V-based Episodic Memory
Figure 1: The diagram of algorithms. The left side denotes the general Q-based offline RI methods. The right side is the framework of our proposed approach (VEM). Q-based methods learns boot- strapped Q-values, but requires additional constraint or penalty for actions out of the dataset. Our method, on the contrary, learns bootstrapped V-values while being completely confined within the dataset without any regularization.
propose Expectile V-learning (EVL), which is based a new expectile operator that smoothly interpolates between the Bellman expectation operator and optimality operator.
To better solve long-horizon and sparse-reward tasks, further propose using value-based plan- ning to improve the advantage estimation for policy learning. We adopt an implicit memory-based planning scheme that strietly plans within offline trajectories compute the advantages effectively, as proposed in recent advances in episodic memory-based methods (Hu et al.) 2021). Together, we present our novel framework for offline RL, Value-based Episodic Memory (VEM), which uses ex- pectile V-learning to approximate the optimal value with offline data and conduct implicit memory- based planning to further enhance advantage estimation. With the properly learned advantage func- tion. VEM trains the policy network in simple regression manner. We demonstrate our algorithm and formal description of our algorithm provided in Algorithm [
Figure |
The contributions of this paper are threefold. First, we present a new offline V-learning method, EVL. and novel offline RL framework, VEM. EVL learns the value function through the trade-offs between imitation learning and optimal value learning. VEM uses memory-basee planning scheme to enhance advantage estimation and conduct policy learning a regression manner. Second, we theoretically analyze our proposed algorithm's convergence properties and the trade-off between contraction rate, fixed-point bias, and variance. Specifically, we show that VEM is provably con- vergent and enjoys low concentration rate with small fixed-point bias. Finally, method in the offline RI. benchmark D4RL. (Fu et al. (020). Comparing with other baselines, VEM achieves superior performance, especially in the sparse reward tasks like AntMaze and Adroit. The ablation study shows that VEM yields accurate value estimates and is robust to extrapolation errors.
evaluate
2 BACKGROUND
Preliminaries. We consider a Markov Decision Process (MDP) M defined by a tuple (S, A, P. 7),
where the state space, A is the action space, P(- s, a) x A S - R. is the transition distribution function, r(s, a) S + IR is the reward function and 7 E [0. 1) is the discount factor. We say environment is deterministic if P(s' s, 8(s' fs, for some deterministic transition function f. where 8(.) is the Dirac function. The goal of an RL agent is to learn policy : A + R. which maximizes the expectation of discounted cumulative reward: J (TT) Peo (-1x)) s..0z) 7 a.)]. where Po is the distribution of the initial states.
Value-based Offline Reinforcement Learning Methods. Current offline RL methods can be roughly divided into two categories according to types of learned value function: Q-based and V-based methods. Q-based methods, such as BCQ (Fujimoto et al. 2019), learn Q-function for policy learning and avoid selecting unfamiliar actions via constraints or penalty. On the contrary, V-based methods (Peng et al. 2019 Siegel et al. 2020 Chen et al. (0200) learns the value of be- havior policy V*(s) with the trajectories in the offline dataset D and update policy
regression problem. Based on the learned V ffunction, V -based methods like AWR (Peng et al. 2019) updates the policy using advantage-weighted regression, where each state-action pair is weighted according
2

---

Published as a conference paper at ICLR 2022
to the exponentiated advantage:
max Jx (Ø) 1 ( [log (a(al) (x) (R, (*(s:))
(1)
Episodic Memory-Based Methods. Inspired by psychobiology. episodic memory-based methods store experiences in a non-parametric table fast retrieve past successful strategies when encoun- tering similar states. Model-free episodic control (Blundell et al. 2016a) updates the memory table by taking the maximum return R(s, a) among all rollouts starting from same state-action pair (s, a). Hu et al. (2021) proposes Generalizable Episodic Memory, which extends this idea to the continuous domain, and proposes updating formula with a parametric memory Q
3
METHOD
In this section, we describe our novel offline method, value-hased episodic memory, as depicted in Figure VEM uses expectile V-learning (EVL) to learn V-functions while confines value learning within the dataset to reduce extrapolation error. EVL uses expectile operator that interpolates between Bellman expectation operator and optimality operator to balance behavior cloning and op- timal value learning, Further, VEM integrates memory-based planning to improve the advantage estimation and accelerate the convergence EVL. Finally, generalized advantage- weithagee learn- ing is used for policy learning with enhanced advantage estimation. A formal description for the VEM algorithm is shown in Algorithm| in Appendix A.
3.1 EXPECTILE V-LEARNING
To achieve a balance between behavior cloning and optimal value learning, we consider the Bellman expectile operator defined as follows:
((T7))))) arg min Ba) (T((s a)l3 (1 -1 (1-7)0((s =)]
where μ the behavior policy, 0(s, a) Ey Ey a) 7V'))) v] is the expected one- step TD error, [+ max(- 0) and [1] min(.,0) 0). This operator resembles the expectile statis- tics (Newey & Powelll 1987; Rowland et al. 2019) and hence its name. We can see that when 1/2, this operator is reduced to Bellman expectation operator, while when 7 1, this operator approaches Bellman optimality operator, as depicted in Lemma]
We the following toy example further illustrate the trade-offs achieved by EVL. Consider a random generated MDP. When the operator can be applied exactly, the Bellman optimality operator is sufficient to learn the optimal value V* However, applying operators with an offline dataset raises noise on the actual operator due to the estimation error with finite and biased data. We simulate this effeect by adding random Gaussian noise to the operator. Applying the optimality operator offline datasets can lead vere overestimation due to the maximization bias and bootstrapping. The value estimation learned by EVL, on the contrary, achieves a trade-off between learning optimal policy and behavior cloning and can close to the optimal value with proper chosen 7, as depicted in Figure 2 The noise upon the operator largely de- pends on the size of the dataset. Estimation error can be significant with insufficent data. In this case, we need a small to be conservative and be close to be- havior cloning. When the dataset is large and we are able to have accurate estimation for the operator, we can use larger to recover the optimal policy. By adjusting т, the expectile operator can ac- commodate variant types of datasets. However, the expectile operator in Equation 2 does not have
noise
300
0.0
Mean Statee Value
0.3
0.4
-100
0.3
0.1
0.6
0.7
0.0
Figure 2: Trade-offs of EVL between gen- eralization and conservatism in random MDP. The green line shows the optimal value and the blue line shows the value of behavior policy. The curve is averaged over 20 MDPs.
3

---

Published as a conference paper at ICLR 2022
closed-form solution. In practice, consider the one-step gradient expectile operator
((T)) ((T))))(s)) (s) + 2aBa-m((1s) [T (0((s, a)4+ + + -1 -1--)]0(s,a)--}
(3)
where a the step-size. Please refer to Appendix B.1 for the detailed derivation. For notational convenience, we use T to denote the one-step gradient expectile operator (T₂)² hereafter.
consider the case where the dynamics are nearly-deterministic like robotic applications, and remove the expectation over the next states in the operator. This leads practical algorithm, Expectile V-Leamnin, where we train the value network to minimize the following loss:
(v) = (ss.a, (V()) Vo
(4)
(ss) = Vo.(s) + 2a [r (8(s, a, 8') (1 ()(6(s, s))]
where V is the target value after applying one-step gradient expectile operator and o(s, a') = r(s, a) + >Vo respectively. EVL guaranteed converge with concentration rate 7 2(1 -7)a max r). Please refer to Section 4 for a detailed analysis.
(s). V-function and the target V-function are parameterized by 0 and 0',
3.2 IMPLICIT SEMORY-BASED PLANNING
Although EVL reduces the extrapolation error, it is still challenging problem bootstrap over long time horizons due to estimation errors with fixed dataset. Therefore, we propose using value- based planning to conduct bootstrapping more efficiently. We adopt an implicit memory-based planning scheme that strictly plans within offline trajectories to avoid over-optimistic estimations in the planning phase. This is aligned with recent advances in episodic memory-bascd methods (Hu ct al. 202I), but we conduct this planning expectile V-values rather than Q-values. Specifically, compare the best return far along the trajectory with the value estimates V and takes the maximum between them get the augmented return R.:
( + R141 (8++1)), if << T, Te
R₂
(5)
if t= T,
where denotes steps along the trajectory, T is the episode length, and V is generalized from similar experiences. This procedure is conducted recursively from the last step the first step along the trajectory, forming an implicit planning scheme within the dataset to aggregate experiences along and across trajectories. Further, the back-propagation process in Equation 5 be unrolled and rewritten as follows:
1 (ste)
if n n> 0, if n n=0, 0,
fG
max OCn'S Ramax
Ve,
Ve,
(6)
where n denotes different length rollout steps and Ve, = for n T.
3.3 GENERALIZED ADVANTAGE- WEIGHTED LEARNING
Based on RG calculated in Section 3.2 we can conduct policy learning in a regression form, as
adopted return-based offline RL methods (Nair et al. 2020; Siegel et al. 2020 Peng et al. 2019):
max (Ø) = (( log N (az st) (A(st,
where A(s,,a)) R ( and f is an increasing, nen-negative function. Please refer to Ap- pendix C.I for the detailed implementation of Equation 7 Note that Rt, is not the vanilla returns in the dataset, but the enhanced estimation calculated by implicit planning from V., opposed with other return based methods. Please refer to Algorithm and Section for implementation details and theoretical analysis.
4

---

Published as a conference paper at ICLR 2022
4 THEORETICAL ANALYSIS
In this section, we first derive the convergence property of expectile V-Learning. Then, we demon- strate that memory-based planning accelerates the convergence of the EVL. Finally, design toy example to demonstrate these theoretical analyses empirically. Please refer to Appendix B for the detailed proofs of the following analysis.
4.1 CONVERGENCE PROPERTY THE EXPECTILE V-LEARNING
In this section, we assume the environment is deterministic. We derive the contraction property of T7 as the following statement:
Lemma 1. For any (0, T is -y,cootracttinn, where &r =1 20 - minn(7 7).
Proof. We introduce two more operators to simplify the analysis:
(T) (s) Ea (68, a)] (T(V)) = V(()) + Ee~ulo(s, a)]
Next we show that both operators are non-expansion (e.g., IT V2 I 1 b2 Finally, we rewrite T." based on 7 and T and we prove that 77" a 7r conttracctn Please refer to Appendix B.2 for the complete proof.
Based on Lemma we give discussion about the step-size a and the fraction T:
About the step-size Generally, always want larger However, a must satisfy that V(s) + 20.7(ss, & max(r(s, a) + (V'),, V(s) and ((s) + 2a(1 - 7)6(s, a) 2 min(r(s, a + (s'),, (s)), otherwise the V-value will be overestimated. Thus, must have 200 When 2max/f.1-77
and 2a(1 7) S 1, which infers that a <
we have
mmxx(1.17)) 7, = 1 - 2a min - 7][11 7) 11 min T. 1-1
77)
About the fraction r. It is easy verify that Yr approaches to 1 when - or means that with a larger the contractive property is getting weaker. The choice of 7 makes trade- off between the learning stability and the optimality of values. We further point out that when 7 1, the Expectile V-learning degrades special case of the generalized self-imitation learning (Tang 2020), which losses the contractive property.
which
Next, we prove that T is monotonous improving with respect to Lemma 2. For any T, E (0, if 2 т, we have (VV 2 Vs
Based on the Lemma we derive that V monotonous improving with respect to Proposition 1. Let V denote the fixed point of TH.. For any T, (0, if r' 2 T, have (x)(s) 2 (s), E S. S.
Further, we derive that V gradually approaches V* with respect to Lemma 3. Let V* denote the fixed point of Bellman optimality operator T In the deterministic MDP, we have lim,-+1 V = V*
Based the above analysis, we have the following conclusion:
Remark 1. By choosing suitable we can achieve the trade-off between the contraction rate and the fixed point bias. Particularly, a larger introduces a smaller fixed point bias between V and V*, and produces a larger contraction rate "/r simultaneously.
4.2 VALUE-BASED EPISODIC MEMORY
In this part, we demonstrate that the memory based planning effectively accelerates the convergence of the EVL. We first define the VEM operator as:
(Tvem V)(s)
max (TH))-I7:V(())
15 Pimax
5

---

Published as a conference paper at ICLR 2022
0.50
Coneedan Rale
11201
Consecdo Batx
0.50
Velence
0.75
0:15
0.70
0:10
8.5
1150
(a) The maximal rollout step
(b) The different behavior policies.
Figure 3: A toy example in the random MDP. In both figures, the color darkens with a larger (T E {0.6, 0.7, 0.8, 0.9)). 0.9.) The size of the spots is proportional variable: (a) Change Remax. From magenta to blue, Rmma is set as 1,2, 4 in order. (b) Change the behavior polices Jr, where pu()) = softmax (0(((((6 (s, a/a). From light yellow dark red, the a is set as 0.1, 0.3, 1,3 3 in order.
the relative scale of the third
where mmax is the maximal rollout step for memory control. Then, we derive that multi-step esti- mation operator Tvem does not change the fixed point and contraction property of TH::
Lemma 4. Given E 10, and Pmmax N+, Tvem is "--contraction. If 2' Tvem has the same fixed point as TH.
Next, we derive that the contraction rate of Tvem depends on the dataset quality. Further, we demon- strate that the convergence rate Tvem is quicker than T even the behavior policy μ is random:
Lemma 5.
When the current value estimates (s) are much lower than the value of behavior policy, Tvem provides an optimistic update. Formally, we have
Trem Vss) V7((s) S 7" (+) ++IIV VH lo +
VI ne loo Vs E
(10)
where n (s) = arg maxgennnnac ((T ((TH)=-TAVV()) V the fixed point of (x) and it is the optimal rollout value starting from
This lemma demonstrates that Tvem can provide an optimistic update for pessimistic value estimates. Specifically, the scale of the update depends on the quality of the datasets. If the behavior policy expert, which means 4 close to V,2 Then, following the lemma, the contraction rate will be
near to 7 (s)- Moreover, if the initial value estimates are pessimistic (e.g.. the initialized value function with zeros), we will have n n*(s) R Amax, indicating that the value update will be extremely fast towards lower bound of V*. On the contrary, if μ is random, we have n (s) Rs 1 and the value update will be slow towards V.
By choosing suitable Amaxx we can achieve the trade-off between the contraction rate and the estimation variance, iee. larger Dmax yields a fast update towards a lower bound of fixed point and tolerable variances empirically. Meanwhile, the choice of Rmax does not introduce additional bias, and the fixed point bias totally controlled by
Remark 2.
4.3 TOY EXAMPLE
We design a toy example in the random deterministic MDP to empirically demonstrate the above analysis. Following (Rowland et al.] (2020). we adopt three indicators, including update variance, fixed-point bias, and contraction rate, which is shown in Figure 3 Specifically, the contraction rate sup [ V Trem V'lloo/1| - V'lloo the bias
V* and the variance
vem
E |TVV - Tvem v13]] Y/2
where Frem is the stochastic approximation of Trem and V is the fixed pointed of Tvem- First, the experimental results Figure 3(a) demonstrate that the relationship of re-tepp estimation and 7. Formally, the contraction rate decreases as becomes larger, and the fixed-point bias increases as 7 becomes smaller, which are consistent with Lemma [ and Lemma Figure ((a) also shows that the variance is positively correlated with Second, the experimental results in Figure ((b] demonstrate that the relationship of dataset quality and r. The higher dataset quality corresponds to the lower contraction rate and variance, which is consistent with Lemma 5

---

Published as a conference paper at ICLR 2022
E
10
13
-22
-12
(a) Large
(b) Medium
(c) Umaze
Figure 4: Visualization of the value estimation in various AntMaze tasks. Darker colors correspond to the higher value estimation. Each map has several terminals (golden stars) and one of which is reached by the agent (the light red star). The red line is the trajectory of the ant.
5 RELATED WORK
Offinn Reinforcement Learning. Offline RL methods (Kumar et al. 2019 Siegel et al. 2020
Argenson & Arnold [2020; Wu et al. 2021; Dadashi et al. [2021] Kostrikov et al. 202 Jin
2001 Rashidinejad al. 2021) be roughly divided into policy constraint, pessimistic value estimation, and model- based methods. Policy constraint methods aim to keep the policy to be
close the behavior under a probabilistic distance (Fujimoto al. 2019
Peng al. 2019 Nair
ct al. 2020). Pessimistic value estimation methods like CQL (Kumar et al. 2020) enforces a regu- larization constraint on the critic loss to penalize overgeneralizztion. Model- methods attempt
to learn model from offline data, with minimal modification to the policy learning (Kidambi et al.,
2020 Yu et al. 2020 Janner al., 2019). However, these methods have introduce additional be- havioral policy models, dynamics models, or regularization terms (Zhang et al.
[2020b a Lee et al.)
2021 Another line of methods uses empirical return as the signal for policy learning, which con-
fines learning within the dataset but leads to limited performance (Levine et al. 2020 Geist et al.
2019
Wang et al. 2021).
Episodic Control. Episodic control aims to store good past experiences a non- non-parametric mem- ory and rapidly latch into past successful policies when encountering similar states instead of waiting for many optimization steps (Blundell et al.) (016b).. Pritzel et al.) (2017) and Lin et al. (2018) intro- duce a parametric memory, which enables better generalization through neural networks. Our work closely related to recent advances in Hu et al. al. (202T). which adopts an implicit planning scheme to enable episodic memory updates in continuous domains. Our method follows this implicit scheme, but conducts planning with expectile V- V-values to avoid overgeneralization on actions out of dataset support.
EXPERIMENTS
In our experiments, we aim to answer the following questions: 1) How does our method performe compared to state-of-the-art offline RL algorithms on the D4RL benchmark dataset? 2) How does implicit planning affect the performance effectively reduces the extrapolation error compared with other offline methods? 4) How does the critical parameter affect the performance of our method?
sparse reward tasks? 3) Can expectile V- V-Learning
6.1 EVALUATION ENVIRONMENTS
We ran VEM on AntMaze, Adroit, and MuJoCo environments to evaluate its performance on var- ious types of tasks. Precisely, the AntMaze navigation tasks control an 8-DoF quadruped robot to reach a specific or randomly sampled goal three types of maps. The reward in the AntMaze domain is highly sparse. The Adroit domain involves controlling a 24-DoF simulated hand tasked with hammering a nail, opening a door, twirling a pen, or picking up and moving a ball. On the adroit tasks, these datasets the following, "human": transitions collected by human operator,

---

Published as a conference paper at ICLR 2022
Type
Env
VEM(Ours)
VEM(r=0.5)
BAIL
BCQ
CQL
AWR
fixed
umaze
87.5
8..0
62.5 =
78.9
74.0
56.0
play
medium
78.0:33.1
71.0112.5
40.0 15.0
0.0
61.2
play
large
57.0=1..0
45.0=2.5
23.0=5.0
6.7
11.8
0.0
diverse
umaze
78.0
75.0=5.0
75.0=1.0
55.0
84.0
70.3
diverse
medium
77.0+-2.2
60.0+5.00
50.0++10.0
0.0
53.7
0.0
diverse
large
58.0 + 2.1
48.0+-277
30.0+5.0
2.2
14.9
0.0
human
door
11.2+44.2
6.911.1
0.0
-0.0
9.1
0.4
human hammer
3.6++1.0
2.5+1.0
0.0+0.1
0.5
2.1
1.2
human relocate
1.3+0.22
0.0:1..0
0.0
0.5
2.1
-0.0
human
65.0+t2.1
55.2+3.1
32.
68.9
55.8
12.3
cloned door
3.6:10.3
0.0.10.0
0.0.0.11
0.0
3.5
cloned hammer
2.7+1.5
0.5+0.1
0.1+0.1
0.4
5.7
0.4
cloned pen
48.7+332
27.8=2.2
46.5+3.5
44.0
40.3
28.0
expert door
105.5++0.2
104.8+0.2
104.7+0.3
99.0
102.9
expert hammer
128.3++.11
102.3+5.6
123.5+3.1
114.9
39.0
expert relocate
109.8:10.2
101.0+1.5
94.4+2.7
41.6
91.5
expert
pen 111.742.6
115.24113
126.7.00.3
114.9
111.0
random
walker2d
6.244.7
6,2444.7
3.9:2.5
4.9
7.0
1.5
random
hopper
11.1-110
10.
9.8+0.11
10.6
10.8
10.2
random
halfcheetah
16.4.13.6
2.6+2.1
0.0.0.11
2.2
35.4
medium walker2d
74.0#112
16.6#0.1
73.0=1.0
53.1
79.2
17.4
medium
hopper
56.6:12.3
56.6=2..3
58.2-1.0
54.5
58.0
35.9
medium
halfcheetah
47.4+10.2
45.3+002
42.
40.7
44.4
37.4
Table 1: Performance of VEM with four offline RL baselines on the AntMaze, Adroit, and MuJoCo domains with the normalized score metric proposed by D4RL benchmark, averaged over three ran- dom seeds with + standard deviation. Scores range from 0 100, where 0 corresponds to random policy performance, and 100 indicates an expert. We use the results in Fu et al. (2020) for AWR and BCQ, and use the results in Kumar al. (2020) for CQL. The results of BAIL come from implementation according to the official code (ttpss//gthuu..co/aaaavvi/k/
"cloned": transitions collected by a policy trained with behavioral cloning interacting in the environ- ment initial demonstrations, "expert": transitions collected by fine-tuned RL policy interacting in the environment. As for the MuJoCo tasks, the datasets are "random": transitions collected by random policy,"medine "medium"" transitions collected by a policy with suboptimal performance. The complete implementation details are presented in Appendix
6.2 PERFORMANCE ON D4RL TASKS
As shown Table VEM achieves state-of-the-at performance on most AntMaze tasks and has a significant improvement over other methods on most Adroit tasks. VEM also achieves good per- formances in MuloCo domains. We find that VEM has low value estimation errors in all tasks, which promotes its superior performance. However, as a similar training framework, BAIL only has reasonable performances on simple offline tasks, such as MuJoCo. Please refer to Appendix D.2 for the complete training curves and value estimation error on D4RL.
To further analyze the superior performance of VEM the sparse reward tasks, we visualize the learned value estimation in AntMaze tasks, which is shown in Figure 4 Experimental results show that VEM has the higher value estimates on the critical place of the map (c.g.. corners) since various trajectories in the datasets are connected. The accurate value estimation leads to its success complex sparse reward tasks.
6.3 ANALYSIS OF VALUE ESTIMATION
As both Expectile V- V-Learning (EVL) and Batch Constrained Q-Learning (BCQ) (Fujmooo al. 2019) aim to avoid using the unseen state-action pairs to eliminate the extrapolation error, we re- place EVL in VEM with BCQ (named BCQ-EM) evaluate the effectiveness of the EVL module.
8

---

Published as a conference paper at ICLR 2022
The experimental results in Figure 9 in Appendix Inniidiiatee indicate that the performance of BCQ-EM is mediocre, and BCQ reaches performance significantly below VEM. We observe strong correla- tion between the training instability and the explosion of the value estimation. This result should not come as a surprise since the Adroit tasks have a larger action space compared with MuloCo domains and narrow human demonstrations. Therefore, the generative model in BCQ cannot guarantee com- pletely the unseen actions are avoided. In contrast, VEM avoids fundamentally unseen actions by keeping the learning procedure within the support of an offline dataset, indicating the necessity of the EVL module. Please refer to Appendix C for the implementation details.
We evaluate - E {0.1, 0.2. 0.9} to investigate the effect of the critical parem-ear in EVL, which is shown in Figure 7 Appendix D. The experimental results demonstrate that the esti- mated value increases with a larger т, which consistent with the analysis in Section 4. Moreover, we observe that is set at low value in some complex high-dimensional robotic tasks or narrow human demonstrations, such as Adroit-clonedhhuman, to get the conservative value estimates. How- ever, if r is set too high (e.g., 0.9 in the pen-human task), the estimated value will explode and poor performance. This as expected since the over-large T leads to the overestimation error caused by neural networks. The experimental results demonstrate that we can balance behavior cloning and optimal value learning by choosing т in terms of different tasks.
6.4 ABLATIONS
Episodic Memory Module. Our first study aims to answer the impact of memory-based planning on performance. We replace the episodic memory module in VEM with standard n-step value esti- mation (named VEM- or VEM-nstep). The experimental results Figure in Appendix [D. indicate that implicit planning along offline trajectories effectively accelerates the convergence of EVL.
Expectile Loss. In addition to the Expectile loss, explored other forms of loss. Formally, compare the Expectile loss and quantile loss, a popular form in Distributional RL algorithms (Dab-) ney al. (0188) which is shown in Figure 5 in Appendix D.I. The experimental results indicate that the Expectile loss is better since it is more stable when dealing with extreme values.
7 CONCLUSION
In this paper, we propose a novel offline RI. method, VEM, based on a new V-learning algorithm, EVL. EVL naturally avoids actions outside the dataset and provides smooth tradeoff between gen- cralization and conversation for offline learning. Further, VEM enables effective implicit planning along offline trajectories to accelerate the convergence of EVL and achieve better advantage estima- tion. Unlike most existing offline RL methods, we keep the learning procedure totally within the dataset's support without any auxiliary modular, such as environment model or behavior policy. The experimental results demonstrate that VEM achieves superior performance in most D4RL tasks and learns the accurate values to guide policy learning, especially in sparse reward tasks. We hope that VEM will inspire more works offline RL and promote practical RL methods in the future.
REPRODUCIBILITY
To ensure work is reproducible, provide our code in the supplementary materials. In the fu- ture, we will publish all source code on Github. The detailed implementation of our algorithm is pre- sented as follows. The value network is trained according to Equation[4] The actor-network is trained according to Equation] 7 The hyper-parametern parameters and network structure used in VEM are shown in Appendix C.3 All experiments
on the standard offline tasks, D4RL (https://github.com//
berkelle//4//ree//ssserrkk)))
9

---

Published as a conference paper at ICLR 2022
REFERENCES
Arthur Argenson and Gabriel Dulac-Amold. Model-based offiine planning. arXiv preprint arXiv:2008. 05556,, 2020.
Charles Blundell, Benigno Uria, Alexander Pritzel, Yazhe Li, Avraham Ruderman, Joel Z Leibo, Jack Rae, Daan Wierstra, and Demis Hassabis. Model-free episodic control. arXiv preprint arXiv:1606.044 04460, 2016a.
Charles Blundell, Benigno Uria, Alexander Pritzel, Yazhe Li, Avraham Ruderman, Joel Z Leibo, Jack Rae, Daan Wierstra, and Demis Hassabis. Model-free episodic control. arXiv preprint arXiv: 04460. 2016b.
Xinyue Chen, Zijian Zhou, Zheng Wang, Che Wang, Yanqiu Wu, and Keith Ross. BAIL: Best- action imitation learning for batch deep reinforcement learning, Advances in Neural Information Processing Systems, 33, 2020.
Will Dabney, Mark Rowland, Marc G Bellemare, and Rémi Munos. Distributional reinforcement learning with quantile regression. In Thiirt--Second AAAI Conference on Artificial Intelligence, 2018.
Robert Dadashi, Shideh Rezaeifar, Nino Vieillard, Léonard Hussenot, Olivier Pietquin. and Matthieu Geist. Offline reinforcement learning with pseudometric learning. arXiv preprint arXiv:2103.01948 2021.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4RL: Datasets for deep data-driven reinforcement learning. arXiv preprint arXivv2004.07219, 2020.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. International Conference on Machine Learning, pp. 2052-2062. PMLR, 2019.
Matthieu Geist, Bruno Scherrer, and Olivier Pietquin. theory regularized markov decision processes. In International Conference on Machine Learning, pp. 2160-2169. PMLR, 2019.
Seyed Kamyar Seyed Ghasemipour, Dale Schuurmans, and Shixiang Shane Gu. EMaQ: Expected- max Q-learning operator for simple yet effective offline and online RL. International Confer- ence on Machine Learning, pp. 3682-3691. PMLR, 2021.
Hao Hu, Jianing Ye, Zhizhou Ren, Guangxiang Zhu, and Chongiie Zhang. Generalizable episodic memory for deep reinforcement learning. arXiv preprint arXiv:2103.066 2021.
Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model- based policy optimization. Advances in Neural Information Processing Systems, 32:12519- 12530, 2019.
Ying Jin, Zhuoran Yang, and Zhaoran Wang. Is pessimism provably efficient for offline RL? In International Conference Machine Learning, pp. 5084-5096. PMLR, 2021.
Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. Morel: Model- based offline reinforcement learning. arXiv preprint arXiv:2005.0595) 2020.
Ilya Kostrikov, Rob Fergus, Jonathan Tompson, and Ofir Nachum. Offline reinforcement learning with fisher divergence critic regularization. In International Conference Machine Learning, pp. 5774-5783. PMLR, 2021.
Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy Q- learning via bootstrapping error reduction. Advances in Neural Information Processing Systems, 12:11784-11794 2019.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative Q-learning for offline reinforcement learning. arXiv preprint arXiv:2006. 20Xx:2000600479, 2020.
Jongmin Lee, Wonseok Jeon, Byung-Jun Lee, Joelle Pineau, and Kee-Eung Kim. OptiDICE: Offline policy optimization via stationary distribution correction estimation. arXiv preprint arXiv:2/06. 10783, 2021.
10

---

Published as a conference paper at ICLR 2022
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tuto- rial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643 2020.
Zichuan Lin, Tianqi Zhao, Guangwen Yang. and Lintao Zhang. Episodic memory deep Q-networks. In Proceedings of the 27th International Joint Conference on Artificial Intelligence, pp. 2433- 2439, 2018.
Ashvin Nair, Murtaza Dalal, Abhishek Gupta, and Sergey Levine. Accelerating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006.09.33 2020.
Whitney Newey and James I. Powell. Asymmetric least squares estimation and testing. Econo- metrica: Journal of the Econometric Society, pp. 819-847, 1987.
Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. eyrantage-weightee regression: Simple and scalable off-policy reinforcement learning. arXiv preprint 1rXiv: 2019.
Alexander Pritzel, Benigno Uria, Sriram Srinivasan, Adria Puigdomenech Badia, Oriol Vinyals, Demis Hassabis, Daan Wierstra, and Charles Blundell. Neural episodic control. In International Conference on Machine Learning, pp. 2827-2836. PMLR, 2017.
Paria Rashidinejad, Banghua Zhu, Cong Ma, Jiantao Jiao, and Stuart Russell. Bridging offline rein- forcement learning and imitation learning: tale of pessimism. arXiv preprint arXiv:2103. 12021, 2021.
Mark Rowland, Robert Dadashi, Saurabh Kumar, Rémi Munos, Marc G Bellemare, and Will Dab- ney. Statistics and samples in distributional reinforcement learning. In International Conference PMLR, 2019.
on Machine Learning, pp.
Mark Rowland, Will Dabney, and Rémi Munos. Adaptive trade-offss in off-policy learning. In International Conference on Artificial Intelligence and Statistics, pp. 34-44. PMLR, 2020.
Noah Y Siegel, Jost Tobias Springenberg, Felix Berkenkamp, Abbas Abdolmaleki, Michael Ne- unert, Thomas Lampe, Roland Hafner, Nicolas Heess, and Martin Riedmiller. Keep doing what worked: Behavioral modelling priors for offline reinforcement learning. arXiv preprint arXiv:2002.08336, 2020.
Yunhao Tang. Self-imitation learning via generalized lower bound Q-learning. Advances in Neural Information Processing Systems, 33, 2020.
Qing Wang, Jiechao Xiong, Lei Han, Peng Sun, Han Liu, and Tong Zhang. Exponentially weighted imitation learning for batched historical data. Advances in Neural Information Processing Sys- tems, 31:6288, 2018.
Ruosong Wang, Yifan Wu, Ruslan Salakhutdinov, and Sham M Kakade, Instabilities of offline with pre-trdined neural representation. arXiv preprint arXiv:2103.04944 2021.
Yue Wu, Shuangfei Zhai, Nitish Srivastava, Joshua Susskind, Jian Zhang, Ruslan Salakhutdinov, and Hanlin Goh. Uncertainty weighted Actor-Critic for offline reinforcement learning. arXiv preprint arXiv:2105.08
2021.
Yigin Yang, Xiaoteng Ma, Li Chenghao, Zewu Zheng, Qiyuan Zhang, Gao Huang, Jun Yang, and Qianchuan Zhao. Believe what you see: Implicit constraint approach for offline multi-agent reinforcement learning. Advances in Neural Information P Processing Systems, 34, 2021.
Tianhe Yu, Garrett Thomas. Lantao Yu, Stefano Ermon, James Y Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. MOPO: Model-based offline policy optimization. Advances in Neural Information Processing Systems, 33:14129 14142, 2020.
Ruiyi Zhang, Bo Dai, Lihong Li, and Dale Schuurmans. GenDICE: Generalized offline estimation of stationary values. arXiv preprint arXiv:2002.09072. 2020a.
Shangtong Zhang, Bo Liu, and Shimon Whiteson. GradienDDICE Rethinking generalized offline estimation stationary values. In International Conference on Machine Learning, pp. 11194- 11203. PMLR, 2020b.
11

---

Published as a conference paper at ICLR 2022
A ALGORITHM
A.1 VALUE-BASED EPISODIC MEMORY CONTROL
Algorithm Value-based Episodic Memory Control
Initialize critic networks Vo1: Vog and actor network Te with random parameters 0,, 0z, Initialize target networks 0f <- 01, 02 + 02
Initialize episodic memory M for t = 1 to I do
for E 21 do
Sample N transitions (s1, 0t, R((0) from M
Update 0; < ming, N Σ (n( ves
Update @ m Σ V log (T( (1) Rm mean, Vo, (s))
end for if mod u then
t kB: + (1 x)0%
Update Memory
end if end for
Algorithm 2 Update Memory
for trajectories in buffer M do for 8t, ae. Tt, 8t--1 in reversed(T) do
for E {1,2) do
Compute (2((0 with Equation and save into buffer M
end for end for end for
A.2 AN APPROACH FOR AUTO-TUNING т
When we have a good estimation of V*, for example, when there is some expert data in the dataset, we can auto-tune r such that the value learned by EVL is close to the estimation of V*. This can be done by calculating the Monte-Carlo return estimates of each state and selecting good return values as the estimation of optimal value V Based on this target, we develop a method for auto-tuning т.
By parameterizing sigmoid with differentiable parameter & E IR, we can auto-tune by minimizing the following loss JE] {(Ev(s) v**). If (EV(s) V 0, the differentiable parameter
& will become larger and the value estimation EV(s) will become larger accordingly. Similarly, & and (s) will become smaller if (EV(s) V*) > O. The experimental results in Figure 10 in Appendix D.I show that auto-tuning can lead to similar performance compared with manual selection.
12

---

Published as a conference paper at ICLR 2022
B THEORETICAL ANALYSIS
COMPLETE DERIVATION
The expectile regression loss (Rowland et al. 2019) is defined
ER e. r) - Ex~e [ q) (1 () s 4)) (Z 9)
(11)
where is the target distribution and the minimiser of this loss is called the r-expectile of the corresponding loss in reinforcement learning is
(0) = p. [(((s. + (s) -s)))ss(s))))){ + (1-7))((ss))))) + (V) vo(s))
(12)
= 1 [T vo(s)) (1 (1--) () [s)
Then, taking the gradient of the value objective with respect to V8(s), we have
VJv(0) Σ - Vo(s))
Vo(s)) 2(1 - 7 Vo(s)) & Vo(s))
- Vo(s))+ - 2(1 - r) Vo(s)) = Σ m(a 8) 1- -
Σ (- 2(1 (6)--
(13)
Therefore,
(ss) = Ve(s) (0)
(14)
Vo(s) 20 [T00(s a)]+ + (1- ( a)){-))))
B.2 PROOF OF LEMMA 1
Lemma 1. For any E [0, 1), Tít is a+ 7 where 7, = 1 2a(1 - 7) min (T, - T}.
Proof. Note that T1/2 the standard policy evaluation Bellman operator for μa, whose fixed point
VH. We see that for any V1, V2,
T (8) T1/ V2()
(s) + a (6 a)] (V2(s) aB
(1 (s)
-(s,a) + + (()) - >V2(s'))
(15)
< (1 V1 1/2lo0 + aaylVV - 1/2lo
(1 a(1 v - V2lloo
We introduce two more operators to simplify the analysis:
(s V(s) + E a))++
(16)
(=V(s) = V(s) + ((0
Next we show that both operators are non-expansion (i.e., 1 T V2lloo S 1[V - V22lo)). For any V1, V2, have
(s) - (+) = Vi(() - Eu
182(,, a)]+] E.-,, (6) a) Vi - a))+ + V2(s))).
(17)
The relationship between [51 (s) and
exists in four cases, which
61 2 0, 02 2 then (s,,))++++ Vi - ()) Vo(+)) = >VV((6' V2(o')).
61 < 0, 82 < 0, then
(+. Vi (s) - (b) Vo(s) = Vi(ss -v(ss).
6, 2 0, 82 < then
[)(s, + (i(s) - (162(s, a))+ + V2(())
= (r(s,a) +7V((s)) Vz(s)
(18)
< (((s,))+ - (r(s, + VV22)'))
= =>(Vi(s') - -V((s))))VV('))).
where the inequality comes from r(s,a) + +VV(('') < V2(8).
13

---

Published as a conference paper at ICLR 2022
61 < 0, 2 then
+ - (162(s, + V2(s))
= -V[ (r((,)) + -V2()')
(19)
Vs) (V(s)),(s))
where the inequality comes from r(s,a) + 7V2(5) 2 V2(5).
Therefore, we have (i) -4 < [V 1/2 With the 74,7= we rewrite T7K as
(s) (s) + (1
-1
+ 20 (V + (6 a)]-)
=(1 - 2a) (2(
(20)
And
(s)) V(s) =Bb=nlo(s,a))
V(s) + a - (V) -
(21)
= (1 -1-200 -2a)V(ss) + (s) (=)(s)).
-T*(((s)))))
We first focus < For any V1, V/2, we have
(s) (s()
=1 (1-2a)(Vi(3 - V3(s)) 20T (T - 2 (2
(1 22a 27(1 (V1 V2(()) 2 (Ti - V2)))
2a(1 27) (s) V2())
(1 2a 27(1 2a) 122/o0 + 2 a(1 >)) - -V2/l0 + 22a11 2r] V2lloc
= (1 2ar(1 (1- - v/ - V2
(22)
Similarly, when 7 > 1/2, we have (V))((()))))))) S (112111-2221111 (000)))))00000000000000011000001 -Velloo-
B.3 PROOF OF LEMMA 2
Lemma 2. For any T, E (0,1), if T,, we have T TH, Vs E S.
Proof. Based Equation 20 we have
V - (s)
= (1 (l-2aa)(ss) 2ar (
-((1 (o)) 2ar77 2a 20(1+)))+V(
(23)
= 2a(7' - (T(V - (s)(s)
2a(T' - Bo) (0(s, a)) (0(s, -|2
B.4 PROOF OF LEMMA 3
Lemma 3. Let V* denote the fixed point of Bellman optimality operator T In the deterministic MDP, we have lim V V*
Proof. We first show that V is also fixed point for The Based on the definition of T we have (*()) max, (r(s, V*))' which infers that ö(s, a) < v's S, E A. Thus, have [8(s, a)]+ (s). By setting (1 7) + 0, we eliminate the effect
TIVV (*) (*))) + Ea--.-- of TH. Further by the contractive property of T!", we obtain the uniqueness of V The proof is completed.
14

---

Published as a conference paper at ICLR 2022
B.5 PROOF OF LEMMA 4
Lemma 4. Given T & 10 and T E NT, Turm is a yy-contruction. IʃT > 2. Tvem has the same fixed point as T.
Proof. We prove the contraction first. For any V1, 1/2 we have
Tvem V1 (s(s)= Tvem V2(s)
(7
max 77 15AST
s mAX Pmmx
-
(24)
1 mAx
/ V2
$ 7 1+ V2lloo
Next we show that V the fixed point of 7 is also the fixed point of Tvem when T > 2.
1 By
definition, we have V = V, Following Lemma 2 we have V V 2 1/2 V = TUV..
Repeatedly applying T and using its monotonicity, we have TVV; 2 (T n V
0max-
Thus, we have Tvem (s) (TX
(s).
B.6 PROOF OF LEMMA 5
Lemma 5. When the current value estimates V(s) are much lower than the value of behavior policy, Tvem provides an optimistic update. Formally, have
(Tem (s) V77 S S n0 (6) ++ Via 1f + I V Vs E
(25)
where n² (s) arg ma ((TK) ((
and V is the fixed point of (6)
Proof. The lemma is a direct result of the triangle inequality, We have
T Trem (ss) V22 (TT) (T'V (s)
= (17))""t>-117111
(6) (s) + V - (7)))
S - IIV Vlll lloo V Vi
(26)
B.7 PROOF OF PROPOSITION 1
Proposition Let V denote the fixed point of TH. For any T, (0,1), if 2 T. we have (s) 2 (s), vs S.
Proof. With the Lemma 2 we have V 2 T. Since V, is the fixed point of T.", we have TVVV V.. Putting the results together, we obtain V T."V I TVV; Repeatedly applying T and using its monotonicity, we have V," THV (T V V2.
C DETAILED IMPLEMENTATION
C.1 GENERALIZED ADVANTAGE- WEIGHTED LEARNING
In practice, we adopt -eeaky-ReLU or Softmax functions.
Leaky-ReLUU:
max (+) = B(e,a)n logmola 18) (A(s,a))].
A(ssa)
if A(s,a) >
(27)
where s(A(s,
Alc.a)
if A(s, a) 50
a
15

---

Published as a conference paper at ICLR 2022
Softmax:
Axp(= a))
max (6) = (a.a>DD
log TT (a(aas)-
(28)
E Bss expl A(s,, A(s,, as))
C.2 BCQ-EM
The value network of BCQ-EM is trained by minimizing the following loss:
min (8) E (ss
(Re Qo(s),))))
(29)
1 if n > 0,
R
max 0<C So
Qin.
Qr. 1((ss, âe)
(30)
if 0,
where â corresponds to the perturbed actions, sampled from the generative model (st).
The perturbation network BCQ-EM is trained by minimizing the following loss: min (e) = ED [Qa(s, ag + (s(8, 0z +))], {a: ~ G (s))
(31)
where (a(s, a, ① is perturbation model, which outputs adjustment to action in the range [-0, Ø). We adopt conditional variational auto-encoder to represent the generative model G (s) and it is trained match the state-action pairs sampled from 'D by minimizing the cross-entropy loss-function.
C.3 SYYPE-PARRAETEE AND NETWORK STRUCTURE
Table 2: Hyper-paramene Sheet
Value
Critic Learning Rate
1e-3
Actor Learning Rate
1e-3
Optimizer
Adam
Target Update Rate (n)
0.005
Memory Update Period
100
Batch Size
128
Discount Factor
0.99
Gradient Steps per Update
200
Maximum Length d
Episode Length T
Table 3: Hyper- used in VEM across different tasks
AntMaze-fixee
umaze
medium
large
0.4
0.3
0.3
umaze
medium
large
AntMaze-diverse
0.3
0.4
0.1
Adroit-human
door
hammer 0.4
pen
0.4
0.4
Adroit-cloned
door 0.2
hammer 0.3
pen
0.1
Adroit-expert
door
hammer 0.3
pen
0.3
0.3
walker2d 0.3
halfcheetah 0.4
hopper 0.5
MuloCo-medium
walker2d 0.5
halfcheetah 0.6
hopper 0.7
MuJoCo-random
We use fully connected neural network as den units and ReLU as activation function. [(state dim, 256), (256, 256), (256, action dim)]. [(state dim, 256), (256, 256), (256, 1)]-
function approximation with 256 hid- The structure of the actor network The structure the value network is
16

---

Published as a conference paper at ICLR 2022
D ADDITIONAL EXPERIMENTS ON D4RL
D.1 ABLATION STUDY
VEMI
vay
331
VIM
Edoks Ritss
ze:
100
60
02
005
00
012
001
012
100
100
(a) door-human
(b) hammer- human
(c) relocate-human human
(d) pen-human
Figure 5: Comparison results between expectile loss and quantile loss Adroit tasks. We respec-
tively name our algorithm with expectile loss and quantile loss as VEM and VEM (abs).
VTM (0.1)
E00
750-
VEM
Reum
VEM
Secm
Grood Roum
30X:
VEM (0.7)
200
VEM
200
200
-50
0.0
0.2
0.1
0.6
018
1.0
0.0
0.2
0.0
0.5
000
01.2
D.0
0.5
055
1.0
Million Stags
Million
Staps
Milloon Steas
(a) pen-human
(b) door-human
(c) hammer-human
(0:1]
Extratto Valer
5000
VEM [0:3]
Estectioon 1ahes
Fermtin Value
(200)
(9:7)
VEM (0:8)
2000
80
0.22
11.0
0.5
200
0.4
0.5
012
06
1.0
Steps
Stugs
(d) pen-human
(c) door-human
(
hammer-human
Figure 6: The results of VEM (r) with various 7 Adroit tasks. The results in the upper row are the performance. The results in the bottom row are the estimation value.
VEM
XTM AaI
2000
acpis
Pereo-ti- porpor TDs Bouu)
20
(a) pen-human
door-human
hammer-human
relocate-human
VEW
VEM
Sareaon Ertr
eado Eree
T03-80114
Tos
TES B01--8-3
Tus espps
tbs UdE3))
Tou
5y
(2445
022
6/6
0.2
12.4
600 Step
11.1
0.0
Sten
Step.
06.
Steps
(c) pen-human
(f) hoor-human
(g) hammer-hume
(h) aelocate-humant
Figure 7: Comparison results between VEM with TD3+BC. We adopt different hyper-parameter E {0.5, 2.5, 4.5) in TD3+BC to test its performance. The upper are the performance. The results in the bottom row are the estimation error (the unit is 10)2).
17

---

Published as a conference paper at ICLR 2022
2.75
Edool hear
Edock heam
VTM
VTN
VTM
V2302mo
VIMCC00
0.
(a) medium-play
(b) medium-diverse
(c) large-play
(d) large-diverse
Figure 8: The comparison between episodic memory and n-step value estimation on AntMaze tasks.
420
338
Epedds hatrr
R00
Epooe Neters
Ma Steps
Killio Stene
Millce Steye
14.8 Stepe
(a) door-human
(b) hammer-humes
(c) relocate human
(d) pen- hen-human
6:20
Triestee
800
Tcreatic
02
0.4
072
02
0.1
0.6
0-0
010
(c) door-human
(f) hammer-human
(g)
relocate-human
(h) pen-human
Figure 9: The comparison between VEM, BCQ-EM and BCQ on Adroit-human tasks. The results
in the upper row are the performance. The results the bottom row are the estimation error, where the unit is 10
D.2 COMPLETE TRAINING CURVES AND VALUE ESTIMATION ERROR
Esuade Doure
0.2
Kiilke Steps
MB Staps
(a) Episode return
(b) T value
Figure 10: Comparison between fixed 7 (VEM) and auto-tuning 7 (VEM(auto)) in the door-human human task.
18

---

Published as a conference paper at ICLR 2022
VEMO VEMU300
VEM03 VEM)0000)
ATMD [TM[
VIVVI 22512000
Valx Eatrecce
Vike toevate
100
022
10
0.1
168lin Step
IViiion
Sten
Millce Stepe
Miloon Steps
(a) door-human
(h) hammer-human
(c) relocate-human
(d) pen-human
Figure 11: Value estimation of VEM (1mmax) in adroit-human tasks, where mmax the maximal
rollout step for memory control (see Equation 11). We set = 0.5 in all tasks.
Retr
4.30
Esude
Enowe
Exrade
600
00
36
Nillbn
Mikr
Million
(a) antmaze-umaze
(b)
antmazze-umaze- (c)
antmaze-mediumm (d)
antmaze-mediumm
diverse
play
diverse
u...
Tyoue the
400
Supe
Stppe
Miller Stepe
Millon Stp
(e) untmaze-largee play
(l) antmaze- lareeazz-lrge--iveeee
door-human
(h) hammer- hammer-human
Epsede Soom
Room
Epode Roum
200
000
Millonn Soaas
Millin Stuas
Million Stuas
Millca Stape
relocate-human
ii
pen-human
(k) door-cloned
(1) hammer-cloned
2000
VIM
2000
Epede
1000
BAA.
1000
BAt
xxc
1000
3x0
bx0
1000
100
02
05
0.5
06
05
00
02
06
08
10
0.0
86
68
Steps
Stewe
Stepe
(m) pen-eloned
(n) hopper-medium
(0) walker2d- medium
(p) cechheetta--mddiin
VIM
VEM bell
VIM
Epacce bamm
Sahoce lerm
241
0.5
0.1
36
0.
6.3
35
055
Stope
Miloan Step
(q) hopper-random
(r) walker2d-random
(s) halfchectah-random
Figure 12: The training curves of VEM and BAIL D4RL tasks.
19

---

Published as a conference paper at ICLR 2022
VCU
MPM
Ere
Gimade Erer
Erv
Ene
1.0
Milion Stew
Milorr Stzpe
Nilion Stzee
Milion Steoe
(a) antmaae--uaa umaze
(b)
antmaze- umaze- (c)
antmaze-medinne (d)
antmaze-
diverse
play
diverse
VEM
8.100
VIM
VIM
Erer
patia Eroe
Excesto Enee
Eurs-
10
b000
1illion Stege
Seps
Siess
Steps
(c) antmaze- laree--rrrepppeee
(f) antmaze- Saanmaze--rrge---ee large-diverss
g door-human
(h) bammer-human human
VEM
VEM
VEM
1000
3000
Dovorte
1340
0000
000
155
105
Stage
Stops
Staps
Staps
Mlocate-human
(
pen-human
(k) door-cloned
(I) hammer-cloned
00 Dee
Edreees Ere
100
900
044
100
000
602
100
005
000
022
00.5
(m) pen-cloned
(n) hopper-medium
(0) malker2d-medium
(p) halfchectah-mette
VTM
VTM
VFM
12.0 Ene
6030
Servatoo Ener
0.2
0.5
(g) hopper-random
(r) walker2d-random
(s) halfcheetalh-randon
Figure 13: The value estimation error of VEM on D4RL tasks. The estimation error refers to the average estimated state values minus the average returns.
20


Reviews:
[{"review_id": "quP9tWmuJJn", "reviewer": "Reviewer_M22A", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Advantages:\n1. This paper provides a new perspective to evaluate actions out of the dataset's support. Traditional attempts focused on Q-based methods which require additional constraints or penalties for actions out of the dataset. This paper learns bootstrapped V-values while being completely confined within the dataset without any regularization.\n2. The Value-based Episodic Memory is simple yet efficient. The value-based planning to conduct bootstrapping is efficient. The adaption of episodic memory-based methods is appropriate. The adaption of return-based offline RL methods is effective.\n3. The theoretical analysis is convincing and enhanced the persuasion of the claims.\n4. The experiments are extensive and supportive. The critical experimental parameters are provided, and thus there should be no issues with the repeatability of the experiments. \n\nDisadvantages:\n1. There are quite a few contributions of this paper, but none of them are significant. The first contribution (the most important one), using V-function to substitute Q-function, doesn't show distinct advantages intuitively. Why the V-function is better than Q-function? Simply by adding no additional constraint or penalty for actions out of the dataset? If so, what are the basic reasons? Is it because the constraint or penalty in Q-function difficult to tackle or not reasonable? The contribution of balance between imitation learning and optimal value learning is good, but it is also trivial by just combining two losses together. The implicit memory-based planning and generalized advantage-weighted learning are direct adaptions from existing work.\n2. The Equation formations should be unified. For example, Equation (2) should include (s,a) in the Dirac function.", "summary_of_the_paper": "In this paper, the authors intend to derive offline reinforcement learning policies by learning the V-function, instead of the Q-function, so as to balance the imitation learning and optimal value learning. To achieve this goal, the author proposes Expectile V-learning (EVL) to smoothly interpolate between the Bellman expectation operator and optimality operator. Based on the learned value, the authors propose Value-based Episodic Memory (VEM) to approximate the optimal value with offline data and conduct implicit memory-based planning to further enhance advantage estimation. The authors design theoretical analysis, a toy example, and empirical experiments to validate the proposed methods.", "summary_of_the_review": "The Expectile V-learning (EVL) is new, the Value-based Episodic Memory is simple yet efficient, the theoretical analysis is convincing, and the experiments are supportive. However, my major concern is that none of the contributions are significant. I am looking forward to the authors' response, especially justifications on the novelty and effectiveness of  EVL.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "jiReeBVFOmU", "reviewer": "Reviewer_y86L", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros: \n1. The paper is clearly written and easy to understand.\n2. The method is technically sound. The authors provide theoretical guarantees of the convergence of the VEM and also show that the memory-based planning module improves the convergence rate.\n3. The empirical results of VEM show that it can outperform prior methods in many of the D4RL tasks.\n\nCons:\n1. I'm not sure why the authors didn't evaluate VEM on the full set of D4RL mujoco datasets, e.g. medium-replay, medium-expert and expert datasets. VEM is also not evaluated on the kitchen dataset. Including these would give a more clear sense of how VEM compares to prior methods.\n2. Several important baselines are missing such as [1,2,3,4,5,6]. These methods along with CQL seem to be better than VEM on most of the mujoco tasks while [3] seems to obtain strong adroit results and [6] obtains good antmaze results. The authors should perform a thorough comparison between VEM and these approaches.\n3. VEM seems to require per-task tuning with online rollouts as the authors show different $\\tau$ values on different tasks/datasets. This could be problematic since it is typically impractical and unsafe for offline RL to evaluate the policy online and per-task tuning would make it even worse.\n4. The memory-based planning module seems a bit orthogonal to the main approach, which is the expectile regression. It is directly adapted from prior work and added on top of the method. It would be interesting to see how other offline RL methods perform with the memory-based planning module.\n\n[1] Fujimoto, Scott, and Shixiang Shane Gu. \"A Minimalist Approach to Offline Reinforcement Learning.\" arXiv preprint arXiv:2106.06860 (2021).\n\n[2] Kostrikov, Ilya, et al. \"Offline reinforcement learning with fisher divergence critic regularization.\" International Conference on Machine Learning. PMLR, 2021.\n\n[3] Wu, Yue, et al. \"Uncertainty Weighted Actor-Critic for Offline Reinforcement Learning.\" arXiv preprint arXiv:2105.08140 (2021).\n\n[4] Brandfonbrener, David, et al. \"Offline RL Without Off-Policy Evaluation.\" arXiv preprint arXiv:2106.08909 (2021).\n\n[5] Chen, Lili, et al. \"Decision transformer: Reinforcement learning via sequence modeling.\" arXiv preprint arXiv:2106.01345 (2021).\n\n[6] Ajay, Anurag, et al. \"Opal: Offline primitive discovery for accelerating offline reinforcement learning.\" arXiv preprint arXiv:2010.13611 (2020).", "summary_of_the_paper": "This paper presents a new offline RL algorithm that leverages expectile regression for value learning and performs AWR-style policy learning with the value function learned with the expectile loss and memory-based planning. The proposed method, dubbed VEM, is able to interpolate between learning optimal Bellman operators and behavior cloning, preventing overestimation in Bellman backups. VEM is shown to converge and the episodic memory-based planning module is able to both theoretically improve the convergence rate and also empirically improve the performance. VEM achieves the best or comparable performance on most of the D4RL tasks.", "summary_of_the_review": "Given the comments in the previous section, I think it would be good if the authors can address the raised issues. Based on its current status, I would vote for a weak reject.\n\n_____________________________\npost rebuttal update: I've read the response and most of my concerns are addressed. I will raise my score to a 6.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "jcZuKUUQ5hS", "reviewer": "Reviewer_yPrQ", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\nThis work is well organized, the major idea is clear and valid. Besides, the proposed method is efficient and extensive experiments are provided to prove the effectiveness.\n\nWeaknesses:\nThe proposed framework seems to be similar to work [1], can you compare in detail the difference and the superiority of the proposed method? \n\nBesides, the major idea of the proposed method is also similar to the work [2]. Though work [2] is not designed for offline RL, can you also compare the major idea of the proposed method and this work?\n\n[1]https://arxiv.org/pdf/2106.08909.pdf\n[2] https://arxiv.org/pdf/2101.08152.pdf\n\nThe most recent baseline methods used are proposed in 2020, can you compare with the methods proposed in 2021, like [3]?\n[3] https://arxiv.org/pdf/2106.08909.pdf\n", "summary_of_the_paper": "This work proposed a new offline reinforcement learning framework, where the value function is employed to do update. Specifically, the proposed algorithm used optimal value learning and behavior cloning. Theoretical guarantees about the convergence of the proposed algorithm are provided. Besides, experiments on D4RL tasks are provided to show the effectiveness of the proposed method.", "summary_of_the_review": "Overall, I think this is a good paper. But I am a little concerned about the novelty of this work. So I will give weak acceptance.\n\n_____________________________________________________________\n\nI have read the author's responses. My major concern has been solved. I decide to raise my score to 8.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "PVl4Y0BXHS4", "reviewer": "Reviewer_3Tyw", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. The paper proposes a principled way to prevent overestimation in value function in offline RL by using expectile value function learning.\n2. It proposes using episodic memory to obtain better estimates of returns for AWR during policy learning phase\n3. The paper provides theoritical guarantees for the method and shows improved performance on subset of D4RL tasks.\n\nConcerns:\nThe proposed method feels like implicit/non-parameteric offline model based RL given it uses implicit planning to obtain better targets during policy learning phase. Hence, my main concern is that authors should compare their proposed method to model based offline RL methods like MoREL (kidambi et al., 2020), Combo (Yu et al., 2021).\n\nReferences:\n1. MOReL: Model-Based Offline Reinforcement Learning. Kidami et al., 2020.\n2. COMBO: Conservative Offline Model-Based Policy Optimization. Yu et al., 2021.", "summary_of_the_paper": "The paper proposes to reduce the overestimation in offline reinforcement learning by (i) learning $V(s)$ function instead of $Q(s,a)$ (and then using AWR to get the policy) to be within support of data and (ii) using expectile learning to train value function that interpolates between optimal value learning and BC. Furthermore, to obtain better advantage estimation for AWR during policy learning, the paper proposes to compare best return along the trajectory with value estimates and take the maximum between the two (i.e. $R_t = r_t + \\max(R_{t+1}, V(s_{t+1}))$). Finally, the paper provides theoretical guarantees for the proposed method and shows improved performance on a subset of D4RL tasks. ", "summary_of_the_review": "Weighing the strengths and concerns of the paper, I am recommending weak accept as of now.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "N44F5uOrkUq", "reviewer": "Reviewer_FzBf", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Pros\n* The idea of using expectile Bellman operator to interpolate between expected Bellman operator and Bellman optimality operator is interesting.\n* The authors has conducted experiments in more than 20 environments and the proposed method's performance either surpasses SOTA methods or on-par with them.\n\n-----\n## Cons/questions/suggestions [C/Q/S]\n* [C1]  The clarity of the manuscript could be significantly improved as some of the discussions/explanations are either not precise or consistent with each other. __Example 1__ On page 3, the authors wrote \"The value estimation learned by EVL, on the contrary, achieves a trade-off between learning optimal policy and behavior cloning and can be close to the optimal value with proper chosen ... The estimation error can be significant when the dataset is small, and EVL needs a smaller to be more conservative and closer to behavior\". When the dataset is small, I understand that one want to decrease the $\\tau$ to be more conservative. But it's not clear to me whether the smaller the better in this case. Also, when the authors say EVCL needs to be closer to behavior cloning, does that mean we should just choose $\\tau = 0.5$ in this case? __Example 2__ The optimal values for $\\lambda$ are mostly below 0.5 as seen in table 3. Based on the reasoning given by authors, this seems to indicate that these datasets used in the experiments are \"small\" because we require small $\\lambda$ to obtain good performance. However, the datasets in D4RL are quite large and often much bigger than most real datasets. I feel these results are inconsistent with the authors' discussion in the aforementioned example . __Example 3__ On page 3, the authors wrote \" VEM uses expectile V -learning (EVL) to learn V -functions while avoiding extrapolation error in the action space.\" This claim seems to be too strong for me. As \"avoid\" indicates completely removing the extrapolation error, while EVL actually just serves to reduce the extrapolation error. I think this claim should be more precise.  __Example 4__. On page 3, the authors wrote \"in real-world problems, the dynamics are often nearly deterministic\". For robotics applications, this might be true. However, for a lot of other real-life applications, for example, healthcare, industrial control, autonomous driving, recommender systems, etc, I don't think the dynamics is anywhere near deterministic. \n* [C2] The authors emphasize that EVL could fundamentally avoid unseen actions. I don't see why this is the case. I feel the policy network learned through advantage-weighted loss (eqn.7) can defintiley give out-of-distribution actions when taking the argmax due to inaccurate advantage estimations. Please clarify on this.\n* [Q1] In Figure 10 (b - e), the value estimation error never decreased, and the optimal estimations were obtained at 0th step. This seems to indicate that the value networks failed at learning in the corresponding environments. However, in table 1, good performances are reported for these environments. Could the authors explain why this is the case? \n* [Q2] AWR is a value-based offline RL method and is probably the most relevant baseline. I'm wondering whether the authors could provide some explanations on why AWR failed at four out of six dataset types for antmaze while VEM performs well for all six.\n* [S1] For equation 2, the authors mention that when $\\tau\\rightarrow 1$, the Bellman expectile operator approaches the Bellman optimality operator. However, the lemma 3 for this was not mentioned until page 5. I suggest adding a sentence referring readers to lemma 3 for this important observation.\n* [S2] When $\\lambda=0.5$, VEM is essentially behavior cloning + implicit planning. I think this baseline should be listed separately in table 1 to help readers see the importance of introduced flexibility for value learning with the Bellman expectile operator.\n\n----\n## Minor comments \n* [Typo] Page 8, \"Therefore, the generative model in BCQ cannot __guarantees__ completely\" -> guarantee\n* [Plot] Figure 4 is quite hard to read. The authors could probably just get rid of the floor pattern and make it a different color that has higher contrast with the value pixels.", "summary_of_the_paper": "The authors proposed a new offline RL method called VEM that combines value expectile learning and implicit planning, with the goal of reducing extrapolation error in the action space. The proposed value expectile learning is an interpolation between Bellman expectation operator and Bellman optimality operator. By adjusting the hyperparemeter, one can adjust the tradeoff between sticking to behavior policy or deviate from it to potentially obtain better performance. On standard offline RL benchmarks, the authors demonstrate that the proposed method achieve SOTA performance. ", "summary_of_the_review": "Though the idea of balancing between behavior cloning and optimal value learning through the Bellman expectile operator is interesting, the clarity of the current manuscript still require some improvement. If the authors could address my concerns and questions, I'd be happy to adjust the score.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "1fjbEinowA", "reviewer": "Reviewer_aRTY", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The main contribution of this paper is the introduction of the expectile operator as a smooth interpolation between behavior cloning and value learning in offline RL. Basically, the interpolation is controlled by a parameter $\\tau$, so that when $\\tau = 1/2$, the operator is reduced to taking expectation, and when $\\tau = 1$, the operator is equivalent to the Bellman optimality operator. The authors argue that such an operator is useful in offline RL, in which case learning algorithms need to carefully balance behavior cloning and value learning to avoid extrapolation error. The authors also prove nice properties of the introduced operator, and provide empirical results to justify the effectiveness of the proposed approach. \n\nAlthough this paper introduces an interesting idea, I still have the following concerns.\n\n(i) the Bellman expectile operator is not well-defined. It seems to me that when $\\tau = 1$, any value $v$ that is sufficiently large would achieve the same minimum value ($0$). In this case, how should we define the Bellman expectile operator? \n(ii) The authors seem to assume that the reader is familiar with prior work on value-based offline RL. For example, in Section 2.2, $R_t$ is not defined. The algorithm in Section 3.2 is pretty hard to understand without background on episodic memory-based methods. The authors should at least give an overview on episodic memory-based methods before diving into their new methods. \n(iii) It is unclear to me how one should choose $\\tau$ in practice. Note that in the offline setting, one cannot simply try different $\\tau$ and pick the one with the best performance, since in the offline RL setting, it is assumed that the agent does not have access to online samples. How is $\\tau$ chosen in the empirical evaluation? The paper can be greatly improved the authors could give some guideline on how to pick $\\tau$ in practice. \n\nDue to the above concerns, my current recommendation is a \"weak reject\". However, I am open to raise my score if the authors could resolve my concerns described above. \n\n", "summary_of_the_paper": "In this paper, the authors propose to use the expectile operator as a smooth interpolation between behavior cloning and optimal value learning in offline RL. Based on this operator, the authors develop a new offline method called Value-based Episodic Memory. The authors provide theoretical analysis and empirical results for the developed method. ", "summary_of_the_review": "See above. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Most of the reviewers think this paper is clearly a valuable addition to ICLR based on the convincing theoretical analysis and extensive experimental results. Please refer to reviewers's review for more detailed discussions of the pros and cons of the paper.
Decision: Accept (Poster)