Dealing with the Unknown: Pessimistic Offline Reinforcement Learning
Authors: Jinning Li
Chen Tang
Masayoshi Tomizuka
Wei Zhan
Venue: CoRL2021 Poster (2021)
Keywords: Offline Reinforcement Learning
Out-of-Distribution States

Abstract: Reinforcement Learning (RL) has been shown effective in domains where the agent can learn policies by actively interacting with its operating environment. However, if we change the RL scheme to offline setting where the agent can only update its policy via static datasets, one of the major issues in offline reinforcement learning emerges, i.e. distributional shift. We propose a Pessimistic Offline Reinforcement Learning (PessORL) algorithm to actively lead the agent back to the area where it is familiar by manipulating the value function. We focus on problems caused by out-of-distribution (OOD) states, and deliberately penalize high values at states that are absent in the training dataset, so that the learned pessimistic value function lower bounds the true value anywhere within the state space. We evaluate the PessORL algorithm on various benchmark tasks, where we show that our method gains better performance by explicitly handling OOD states, when compared to those methods merely considering OOD actions.

Dealing with the Unknown: Pessimistic Offline Reinforcement Learning
Jinning Li, Chen Tang, Masayoshi Tomizuka, Wei Zhan
Department of Mechanical Engineering
University of California, Berkeley
United States
{jinning caen,cheaaaaaan tang, tomi zunaz w
Abstract: Reinforcement Learning (RL.) has been shown effective in domains where the agent learn policies by actively interacting with its operating envi- ronment. However, we change the RL scheme to offline setting where the agent only update its policy via static datasets, one of the major issues in offline reinforcement learning emerges, i.e. distributional shift. We propose a Pessimistic Offline Reinforcement Learning (PessORL) algorithm actively lead the agent back to the area where it is familiar by manipulating the value function. We focus on problems caused by diistribuuuttni (OOD) states, and deliberately penalize high values at states that are absent in the training dataset, so that the learned pessimistic value function lower bounds the true value anywhere within the state space. We evaluate the PessORL algorithm on various benchmark tasks, where we show that our method gains better performance by explicitly handling OOD states, when compared to those methods merely considering OOD actions.
Keywords: Offline Reinforcement Learning, Outtof-Distiibution States
Introduction
Reinforcement learning (RL), especially with high-capacity models such as deep nets, has shown its power in many domains, c.g., gaming, healtheare, and robotics. However, typical training schemes of RL algorithms rely on active interaction with the environments. It limits their applications in domains where active data collection is expensive or dangerous (e-g., autonomous driving). Recently, offline reinforcement learning (offline RL) has emerged as promising candidate to overcome this barrier. Different from traditional RL methods, offline-RL. learns the policy from a static offline dataset collected without iterative interaction with the environment, Recent works have shown its ability in solving various policy learning tasks [1, 2, 3]. However, offline RL. methods suffer from several major problems. One of them is distributional shift. Unlike online RI. algorithms, the state and action distributions are different during training and testing. As a result, RL agents may fail dramatically after deployed online. For example, in safety-critiaal applications such as autonomous driving, overconfident and catastrophic extrapolations may occur in out-of-distriiution (OOD) scenes [4].
Many prior works [5, 6, 7, 8, 9, 10] try to mitigate this problem by handling OOD actions. They discourage the policies visit OOD actions by designing conservative value functions, or estimating the uncertainty of Q-functions. Although constraining the policy can implicitly mitigate the problem of state distributional shift, few works have adopted measures to explicitly handle OOD states during the training stage. In this work, we propose the Pessimistic Offline Reinforcement Learning (PessORT)) framework to explicitly limit the policy from visiting both unseen states and actions. We refer to the states or the actions that are not included in the training data as the unseen states or the unseen actions.
Our PessORL framework is inspired by the concept of pessimistic MDP in [11], where the reward is significantly small for unseen state-action pairs. We aim limit the magnitude of the value function at unseen states, so that the agent can avoid or recover from unseen states. It is then crucial to precisely detect OOD states and shape the value function at those states. Since prior methods on OOD actions are derived from similar concept, we can adapt their approaches to handle OOD states. There are
5th Conference on Robot Learning (CoRI. 2021), London, UK.

---

mainly two approaches in the literature. One is to estimate the epistemic uncertainty of Q-function
and subtract from the original Q-function to get a conservative Q-function [6, 7. 8, 9. 10]. The other is to regularize the Q-function during the learning process [5]. The first method is highly sensitive to
the trade-off' between the uncertainty estimation and the original Q-function [12, 13] and the quality of uncertainty estimation [14].
Therefore, we follow the second approach, and add conservative regularization term the policy evaluation step of PessORL to shape the value function. We prove that PessORL learns a pessimistic value function that lower bounds the true value function, and forces the policy to avoid or recover states and actions. We evaluate the PessORL algorithm on various benchmark tasks. The performance of our method matches the state-of-the fe--the-art art offiie RL methods. In particular, we show that, by explicitly handling OOD states, we can further improve the policy performance compared to those methods merely considering OOD actions.
from
2 Related Works
A big challenge for offline reinforcement learning methods to deal with the problems caused by unvisited states or actions in training data, which is also known distributional shift. In model-free offline reinforcement learning, some works used importance sampling to fill the gap between the learned policy and the behavior policy in the training dataset [15, 16. 17, 18, 19J. There are also many works constrained the learned policy to be similar to the behavior policy by explicit constraints in the training dataset [6, 20, 21. 22, 23], so that the agent can avoid out-of-distribution actions during test time. The work in [13] proposed a latent space to constrain the policy to avoid deviating from the training data support. One further step to make the agent avoid actions that may cause itself deviate from the training data support is to get a conservative value function and thus a conservative policy. The works in [7, 8, 6, 9, 10] estimate the uncertainty of the learned Q-function, and then directly subtract it from the Q-function to get a conservative Q-function. Another way to get a conservative Q-function is to regularize the Q-function in the optimization problem during the learning process [5]. In model-basee reinforcement learning (MBRL), there are also many algorithms that constrain the exploitation in the environment with effective uncertainty estimation methods [24, 25, 26, 27, 28, 11]. It is considered to be mature and reliable to detect OOD actions and states by methods from MBRL. Most of the aforementioned methods focus on OOD actions but not have explicit mechanism to deal with OOD states. In this paper, we focus on OOD states and propose a method to learn a pessimistic value function by adding regularization terms when updating Q-functions, and follow the works in the MBRL domain to establish the module to detect OOD states in our algorithm.
3 Background
3.1 Offline Reinforcement Learning
Given Markov decision process (MDP), an RL agent aims to maximize the expectation of cumulative rewards. The MDP represented by a tuple M = AS,A, P, r, 7). where S is the state space, A is the action space, P ; S x A x S + [0, 1] is the transition function, r S x A + IR is the reward function, and y is the discount factor. Typical RL algorithms optimize the policy using experience collected when interacting with the environment. Unlike those online learning paradigms, RL algorithms rely solely on a static offline dataset, denoted by D {($) ai, s(+))
In this work, we focus on proaamiirrorrmmnin---- -based RI. algorithms under the offline setting, where we extract policy from learned value function for the underlying MDP in the training data. Standard Q-learning method estimates an approximate Q-function parametrized by 0, i.e. Qo(s,, at). In each iteration, the Q-function is updated as follows:
1 Q+ g++l + arg min
2 F...D' (Q(s.a) - (aiiaa)--r@@" ()s (policy evaluation),
(1)
Q
where B is the empirical Bellman update operator defined as:
8 Q a) = r(s, a) + "1Ea
( at)
(2)
For discrete action space, we define * as the optimal policy induced by the learned Q-function, Le. *a|s)) 5 a = arg maxa (s. a)). In this case, f= collides into the Bellman optimality
2

---

operator. When the action space is continuous, we follow actor-critic algorithms to approximate the optimal policy by executing a policy improvement step after policy evaluation in each iteration:
#h1 arg max B.
[ ( a)) (policy improvement).
(3)
x
In the rest of the paper, we denote EQQ, Bd (0))) E Es (Q(ss,a) - 2 (3(.
1
as the
Bellman update error for simplicity.
3.2 Uncertainty-Base Methods and Pessimistic Value Functions
By observing Eqn. 1, obvious that the Q-function Qe is never evaluated or updated at states or actions that never appear in the dataset. The agent may behave unexpectedly or unpredictably at those unseen states actions during test time. For grnamic-programming way to address the issue of unseen actions is to estimate the epistemic uncertainty of Q-function and subtract it from the original Q-function [6, 7, 8, 9, 10). The uncertainty is estimated based on an ensemble of learned Q-functions, and the final conservative Q-function becomes Qc a) (e))(Q(((s
approaches. one
EQ-Pp (Q) (((s, s, ( aanee (P)())), where Une defined to be some uncertainty estimation metric, and PD(() the distribution over possible Q-functions. Because the uncertainty metric directly methods is highly sensitive to the quality of uncertainty estimation. Meanwhile, is difficult to find an ideal a to balance the original Q-function and Une.
subtracted, uncertainty-base
Another way is to regularize the Q-function at the step of policy evaluation. representative example is Conservative Q-Learning (CQL) [5]. Assuming that the dataset D is collected with behavior policy Th (als), and ** (a(s) is the learned policy at iteration k; the policy evaluation step in CQL becomes:
Q++1 arg min (Ba (0(s a)]
a))) + E(Q, B Q5)
(4)
In the rest of the paper, we denote C(Q) as the cost term adopted from the CQL, i.e., C(Q) =
"Pen (s, a)] - EEwDD aroi (ai (S, a)]) is worth noting that the aforementioned methods all focus on OOD actions, but they do not have explicit mechanism deal with OOD states, which motivates us to develop the PessORI framework in this work.
Pessimistic Offline Reinforcement Learning Framework
In this section, we introduce the PessORL framework to mitigate the issue state distributional shift. In particular, we propose novel conservative regularization term in the policy evaluation step. It can then be integrated into Q-learning or actor-critic algorithm, which will be described in Sec.
4.1 How To Deal With OOD States
Assuming the dataset D is collected with a behavior policy To (a(a)), and the states s are distributed according distribution ( in the dataset, we propose to solve the problem caused by state distributional shift by augmenting the policy evaluation step in CQL [5] with a regularization term scaled by a trade-off factor s:
min e (a) [Q(s, a)] (s) (Q(s, a))) B (T)))+((())) (5) Q
where d (s) is a particular state distribution of our choice.
The idea use the minimization term a (Q(s, a)]) to penalize high values at unseen states in the dataset, and the maximization term e (E ()).a++((aas)) (ais))((s (s,, a))) to cancel the penalization at in-distribution states. The regularized Q-function could then push the agent towards regions close to the states from the dataset, where the values are higher. To achieve it, we need to find a distribution (s) that assigns high probabilities states far away from the dataset, and low probabilities to states near the training dataset. We will instantiate practical design of d ds) in Sec. 5. For now, we just assume d (s) assigns high probabilities to OOD states.

---

4.2 Theoretical Analysis
In this section, we analyze the theoretical properties of the proposed policy evaluation step. and more details can be found in Appendix A.
The proof
We define E as the iteration of policy evaluation, i.e. Q* denotes the optimized Q-function the k-th iteration obtained by PessORL. Q" is defined to be the true Q-function under policy i rTals) in the underlying MDP without any regularization. The true Q-function can be written in a recursive form via the exact Bellman operator, 8T, as Q² -BQ² We define V as the value function under policy π () (*((s) = Eaca (a|s) [Q" (s, a)]. For the true value function V in the underlying MDP, we also have V BV
We first introduce the theorem that the learned value function is a lower bound of the true one without considering the sampling error defined in the Lemma A.I.
Theorem 4.1 Assume we can obtain the exact reward function r(s, a) and the transition function T(s'(s, a) of the underlying MDP. Let (s)s) = lim (s). Then vs E S, the learned value
function via Eqn. 5 is lower bound of the true one, i.e., (T((s) S (T),,, if the ratio & satisfies
&
ग (als) (als) - 1)
(d) d (s)
π (als)
S min Σ (alss)
d
T3 a
It is worth noting that the learned value function still lower bounds the true value function for any state and action the training datasets, i.c. 8, a E DD even when we consider the sampling error defined in the Lemma A. Further details are shown in Corollary A.1. We have reward or transition pair collected at unseen states or actions outside the training dataset, so is impossible to bound the error outside the training dataset when consider the sampling error introduced by the reward function and the transition function.
We can now step further and show that the values at OOD states are lower than those at in-distribution states based on the learned value function. The proof can be found in Appendix A.3.
Theorem 4.2 For any state s S. if > 0 is sufficiently large, then the expectation of the learned value function via Eqn. 5 under the state marginal d (s) in the training data is higher than that under d (s), i.e., Egndd (V=) < (s) (s))..
During training time, we can at least evaluate Q-values of OOD actions based on in-distribution states. However, there is actually no information about immediate rewards at OOD states, thus no information about Q-values. Intuitively, under offline settings, the best we can do to mitigate the problem of OOD states to suppress values these OOD states, and raising values in-distribution states, so that the agent can be attracted to the area where it is familiar near the training data. Thm. 4.2 indeed tells us PessORL models value function that assigns smaller values OOD states compared to those at in-distribution states. Optimizing policy under such a value function is similar to forcing the policy to avoid unknown states and actions.
In summary, PessORL can learn a pessimistic value function that lower bounds the true value function. Furthermore, this value function assigns smaller values OOD states compared those at in-distribution states, which helps the agent avoid or even recover from OOD states.
Implementing the Algorithm
In this section, we introduce practical PessORL algorithm based Eqn. 5. This algorithm simply modifies the policy evaluation step of Deep Q-Learning or Soft Actor-Critic algorithms, which is easy to implement.
5.1 Detecting OOD states
In prior to designing the algorithm, we need to choose a proper ds(s,, which requires a tool for OOD state detection. Following [11. 14, 29], use bootstrapping to detect OOD states. In

---

particular, we train a bag of Gaussian dynamics models [29] P2..P..... where each model is (.(- a) = (iss fa, (s, a), ). The function fa. outputs the mean difference between the next state and the current state, and Σs. models the standard deviation. OOD states are detected by estimating the uncertainty of bootstrap models at given state S. Concretely,
define 1 E for (s. a) is the where fo a) =
1. - (s. - (s. )
Ux (s) = (als)(als)
mean of outputs of all fØr- and the actions are drawn from a policy distribution A high un (s) value indicates the state is more likely to be an unseen state. Given a set of sampled states {$1, .2.............................. Sp uss))
we can define a discrete distribution over it using ug (s): ((s,) = Σ, u(s,) i 1, n, which assign high probabilities to OOD states. the following section, we will use it to construct the distribution (s).
5.2 Practical Implementation of PessORL
We now introduce practical PessORL algorithm. In practice, to obtain a well-defined distribution d we add an additional optimization problem over d into the original optimization problem. The resulting optimization problem for the policy evaluation step is:
min max (e (Esrd (Q(s, a)] Eged (s), (Qs)) + R(dd)] Q du
+ + B Q + C(Q),
(6)
where R(d is a regularization term inspired by [5] in order to stabilize the training. If we choose R(d)) = - (d (s) ((s)), where ((s) is the distribution we obtained from uncertainty estima- tions, then (s) x ((s) exp (v* (s) where V** (s) (als) [Q(s, a)]. The resulting d intuitively reasonable, because it assigns high probabilities OOD states with high uncertainty estimations. In particular, d assigns higher probabilities to states with high values, because we expect to penalize harder them than those with low values already. With this choice of d in
Algorithm 1: Pessimistic Offline Reinforcement Learning (PessORL)
1 Initialize: A Q network Qa parametrized by 0, A target network Qa = Qa parametrized by 0, policy network Tip parametrized by yp, and bag of dynamics models (fi, Pzy..., ... P..) to detect OOD states;
2 // Dynamies Models Training (Models are used by un, (s) to detect OOD states in the policy evaluation step)
3 for step in range(0, M) do
Train dynamics models (P), P2....,PA Pr) according the transitions in the dataset DD so that we can later obtain an uncertainty estimation model u, (s) in the policy evaluation step;
P..
5 end
// Policy Evaluation and Improvement
7
for step range(O, N) do Update Qo according to Eqn. with learning rate eg and Ux (s):
8
9
0, + 0,-1 eg (0)
Update Ti according the soft actor critic style objective and learning rate ex: PP & :P-- + Eg (s) [Qo(s, a) - log n. (a|s)]:
11
12
if mod target_upder == 0 then
13
Soft Update the target network 0, <- (1--)00-1 + 70,-1
end
end
5

---

Eqn. 6, we obtain the following PessORL policy evaluation step:
min J(Q) = min E log Σ ((s) exp (v+((s) - E Eand ( (V [s)]
Q
+ E(Q B Q+) C(Q).
(7)
The first term in Eqn. is very similar to weighted softmax values over the state space, penalizes the softmax value over the state space, but also considers the distances between sample points and the training data. The two terms following the trade-off factor & is actually trying to decrease the discrepancy between the softmax value over OOD states and the average value over in-distribution states, Intuitively, it should enforce the learned value function to output higher values at in-distribution states, and lower values at out-o--distribution states. The logsumexp term in Eqn. 7 also mitigates the requirement for an accurate uncertainty estimation ((s) over the entire state space. Only those states with high values contribute to the regularization.
The complete algorithm shown in Algorithm 1. space which requires a policy network here, and note that if the action space is discrete, then we no longer need a policy network but just an implicit policy based on the learned Q-function. We implement PessORL. on top of CQL [5], with its default hyperparameters. We also apply Lagrangian dual gradient descent to automatically adjust the trade-off factor e. During the training process of offline reinforcement learning algorithms such as CQL and PessORL, we only have access to the dataset DD instead of d= (s) and J ( s). Therefore, we follow the convention in reinforcement learning community and approximated all expectations by Monte Carlo estimation in Eqn. 7,
We include the version for continuous action
6 Experiments
We compare our algorithm to prior offline algorithms: two state-of-the-arr offline RL, algorithms BEAR [6] and CQL [5]; two baselines adapted directly from online algorithms, actor-critic algorithm TD3 [30] and DDQN [31]; and behavior cloning (BC). The TD3 baseline is applied when the action space is continuous, whereas DDQN is trained when the action space is discrete. We evaluate each algorithm on a wide range of task domains, including tasks with both continuous and discrete state and action space. All baselines are run with the default code and hyperparameters from the original repositories. In particular, we are interested in the comparison between our algorithm with COL, because we essentially add an additional state regularization term the original CQL framework.
6.1 Performance on Various Environments
Pointmass Mazes. The task for the agent in this domain is to learn from expert demonstrations to navigate from a random start to a fixed goal. The expert dataset, which contains around 1000 trajectories all from the same start point to the same goal, is collected by online trained RL policy. During the test time of offline RL algorithms, we reset the start to a random point in the state space and the goal to the same fixed point as the dataset. In this way, the performance of the agent at unseen states are evaluated.
Before showing the performance, we first check if the OOD states detection is accurate, and hence, if we can successfully penalize high values at unseen states training datasets. We evaluate the effectiveness of the OOD states detection method based on the accuracy of uncertainty estimation in the environment Pointmass. Figure ((b) and (c) are visualizations of the training datasets and estimated uncertainty (s), both of which have the coordinate systems the same as that in the map (figure ((a)). We use different colors in figure I(b) and (c) to represent different values at each point in the map. The uncertainty estimations tend to be high (yellow areas in Fig. I(c)) in area where the state density is low (blue areas in Fig 1(b), and vice versa. This trend empirically shows that our uncertainty estimations are reasonable. We can trust them to detect OOD states when training offline RL algorithms.
We include the learning curves in figure 1(d), in which we evaluate each algorithm based on 3 random seeds, and report the average return. The shaded area represents the standard deviation of each evaluation. As we can see in the figures, PessORL. outperforms other baselines in both hard and super hard environments. PessORI. benefits from the augmented policy evaluation step in Eqn. 7. The learned value function produces high values at areas that have low uncertainty estimations, and low

---

312
(a-1)
(b-1)
(c-1)
(d-1)
Toae Deauity
(a-2)
(b-2)
Figure 1: (a) The whole map of the environment; (b) The state density in training dataset; (c) The visualization of uncertainty estimation; (d) The learning curves. The top (1) and the bottom row (2) are corresponding to PointmassHard-Vy and PointmasSuperiHard-a respectively. We can see that almost all trajectories in the training datasets located around the optimal trajectory from the start to the goal the yellow areas (b), indicating they are collected by a near-optimal policy.
values highly uncertain areas (OOD states). Therefore, the agent can be "attracted" the high value areas from low value and unfamiliar areas.
Gym Tasks. In this domain, focus the locomotion environments from MuloCo, including Walker2d-v2, Hopper-v2, Halfcheetah-v2, and Ant-v2. Unlike Pointmass environment, we directly adopt the d4rl datasets [32] our training data in the gym domains. We include four different types of datasets in our experiments, namely, "medium", "medium-expert", "random", and "expert". The "medium", "random", and "expert" dataset are all collected by single policy, which is either early-stopping trained, or randomly initialized, or fully trained expert policy. The "medium-ews eexuert"" dataset generated by mixing mediocre and expert quality data. We show the normalized scores averaged over random seeds for all methods gym domain in table 1. We directly ran all baselines
Table
Performance on Gym and Adroit Domains
Domain
Task
BC
TD3
BEAR
CQL
PessORL
hopper-mediine walker2d medium halfcheerah-medium
29.71 1: 2.43
0.80 1.11
50.08
4.49
63.04
8.64
76.39 1: 4.80
13.06 +: 1.74
4.91
1.08
34.20
8.57
70.67
0.95
75.95
2.96
35.81 0.96
24.60
0.79
0.79
41.25 + 2.45
0.11 48.66
49.21
+ 0.59
medium
83.42
45.78
51.29
85.31
hopper medium expert
85.22 1: 1.38
12.12 0.64
37.67
1.84
105.84 3.62
112.80 1: 4.30
walker2d malkeerd--medum--ee -modium-exxert
16.12 + 0.58
2.006
2.46 17.29
76.27 + 13.33
89.67 + 6.73
halfcheetab medium- expert
37.04
-1.41
12.24
Gym
ant medium expert
66.49 +: 0.76
-6.5544
0.92
8.72 31.31
34.44 4: 30.96
49.95 +: 16.76
hopper-random walkerr2d 2alldood-
9.41 1.41
8.47 8.47 0.32
9.29
2.58
10.36 - 3.68
10.82 + 1.11
0.54
6.04 1.08
0.72
4:
balfcheetah-randuue
2.25 1: 0.45
1: 0.59 -37.31
27.95
2.16 1: 0.28
26.62
1.28 26.65 8.65
28.58 1 0.96
ant-random
26.00
24.05
hopper-expert
109.82
1.81 1.04
0.78 4.57
104.41 + 10. 26
110.67 +: 8.60
2alker2d-expert
60.66 2.26
-0.95 0.58
21.37 + 3.52
105.55 - 2.91
109.37 + 4.94
halfcheetah expert
1..55
7.67
80.19
20.16
ant-expert
73,57 + 3.28
55.07
1.99
44.65 = 10.39
59.04 # 20.66
60.54 =: 17.83
pen-human door human commer-human
34.46 +
-3.83 0.43
35.62 1.34
54.49 7.58
63.07 4.36
1.46 1: 0.06
1 0.01
-0.33
0.05
1.86 0.29
2.28
0.14
1.35 + 0.00
0.212
0.46 - 0.03
3.91 -03-0.34 0.34
4,24 + 0.28
Adroit
relecate-human
0.01
0.32
0..00
0.15
0.33
pen-cloned
23.58 1: 1.14
33.91 0.336
28.92 9.62
35.23 1 11.03
39.02 1: 9.25
door-clonedt
0.15 4: 0.08
0.00
-0.16 0.04
1.72 0.14
1.69 4: 0.35
hammer-cloned
0.40 + 0.07
0.00
+ 0.34
0.53
0.95 0.38
relocate-cloned
-0.21 0.11
-0.14 1 0.08
-0.23 0.13
-0.28 - 0.57
00.26 + 0.21

---

from their original repositories with their default parameters, and we only report the average scores we actually obtained. As we can see in the table, PessORL outperforms all other offline RL methods on majority of tasks on gym domains. PessORI. works especially well with mediocre quality datasets according to the results. In fact, it is one of the advantages of offline RI. methods over behavior cloning on medium quality datasets, because offline RL methods take advantage of the information both from the reward and the underlying state and action distributions in training datasets, instead of simply imitating behavior policies as behavior cloning. Medium quality datasets are also considered to be similar to real-world datasets. Therefore, it is important for an offline RL method to perform well in medium quality datasets. We also note that PessORL shares some good properties with CQL, such as satisfying performance on mixed quality datasets. PessORL and CQL both outperform other offline methods on medium--exeer datasets with PessORL better between them. The reason is that offline RL methods can "stitch" [32] different trajectories from different policies together according to the information from the reward.
Adroit Tasks The adroit domain [33] provides more challenging tasks than the Pointmass environ- ment and the gym domain. The tasks include controlling 24-DoF simulated Shadow Hand robot to twirl a pen, open a door, hammer nail, and relocate a ball. Similar to the datasets in the gym domain, we also directly use the d4rl datasets as the training datasets in our experiments. The performance of PessORL and all baselines is shown in table 1. The normalized scores of all methods are average returns on 5 random seeds. We note that PessORL has better performance than other baselines adroit domains. It is a great advantage for PessORL to learn useful skills from human demonstrations on these high dimensional and highly realistic robotic simulations.
basca-Trackatv92
topper meskum 4b
6.2 Discussions and Limitations
The main contribution of this work is to explic- itly limit the values at OOD states, so that the learned policy can act conservatively at OOD states and drives the agent back to the famil- iar areas near the training data. We are inter- ested to see if our framework can indeed in- duce different behavior OOD states. We use A = (V - Eg-DDV [V)) as metric to evaluate it at each iteration. If A is close to zero, then intuitively it indicates the values OOD states are lower than those at in- distribution states. In Fig. 2. we plot As at each iteration in hopper-medium-vV As is shown in the figure, PessORI. successsuly limits A to be non-positive, which meets our goal in this work and aligns with the statement in theorem 4.2.
1900
1300
-500
3
20
6.2
Pul
(8)
(b)
Figure 2: (a) The learning curves in hopper-
medium-vO. (b) The discrepancy As as a function of gradient steps for PessORL, CQL, and BEAR.
On the gym domain, we notice that the performance of PessORL and CQL on datasets containing expert trajectories is not satisfying, often not as good BC. We believe it is because of overly conservative value estimation. In fact, it is widely believed that conservative methods suffer from underestimation [14]. The conservative objective function in Eqn. sometimes assign values that too low to OOD states and actions. Besides, the uncertainty estimation method cannot be guaranteed to be precise on high- high-dimensional spaces. It is actually possible future work direction to solve the underestimation and uncertainty estimation problems in conservative methods.
7 Conclusion
We propose a Pessimistic Offline Reinforcement Learning framework deal with out-of-distribution states. In particular, we add a regularization term in policy evaluation step to shape value function, so that we can improve its extrapolation to OOD states. We also provide theoretical guarantees that the learned pessimistic value function lower bounds the true one and assigns smaller values to OOD states compared to those at in-distribution states. We evaluate the PessORL algorithm on various benchmark tasks, where we show that our method gains better performance by explicitly handling OOD states compared to those methods merely considering OOD actions.
8

---

References
[1] N. Jaques, A. Ghandcharioun, J. H. Shen, C. Ferguson, A. Lapedriza, N. Jones, S. Gu, and R. Picard. Way off off-policy batch deep reinforcement learning of implicit human preferences in dialog. arXiv preprint
2019.
[2] D. Kalashnikov, A. Irpan, P. Pastor, J. Ibarz, A. Herzog, E. Jang, D. Quillen, E. Holly, M. Kalakr- ishnan, V. Vanhoucke, et al. Scalable deep reinforcement learning for vision-based robotic manipulation. In Conference on Robot Learning, pages 651-673, PMLR, 2018.
[3] G. Kahn, P. Abbeel, and S. Levine. Badgr: An autonomous self- self-supervised learning- based navigation system. IEEE Robotics and Automation Letters, 612::1112-119, 2021.
[4] A. Filos, Tigkas, R. McAllister, N. Rhinehart, S. Levine, and Y. Gal. Can autonomous vehicles identify, recover from, and adapt to distribution shifts? In International Conference on Machine Learning, pages 3145-3153. PMLR, 2020.
[5] A. Kumar, A. Zhou, G. Tucker, and S. Levine. Conservative q-learning for offline reinforcement learning. arXiv preprint arXi::2006.0477 0477777776077, 2020.
[6] A. Kumar, Fu, G. Tucker, and Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. arXiv preprint 1rxXiv:900600000 00949, 2019.
[7] I. Osband, C. Blundell, A. Pritzel, and B. Van Roy. Deep exploration via bootstrapped dqn. arXiv preprint arXiv: 600.04661, 2016.
[8] B. Eysenbach, Gu, Ibarz, and S. Levine. Leave no trace: Learning to reset for safe and autonomous reinforcement learning. arXiv preprint arXiv: 1,0782, 2017.
[9] R. Agarwal, D. Schuurmans, and M. Norouzi. An optimistic perspective offline reinforcement learning. In International Conference on Machine Learning, pages 144-114. PMLR, 2020.
[10] B. O'Donoghue, I. Osband, R. Munos, and V. Mnih. The uncertainty bellman equation and exploration. In International Conference on Machine Learning, pages 3836-3845, 2018.
[11] R. Kidambi, A. Rajeswaran, P. Netrapalli, and T. Joachims. Morel: Model-based offline reinforcement learning, arXiv preprint arXiv:2005.0595/ 2020.
[12] Y. Wu, G. Tucker, and O. Nachum. Behavior regularize offline reinforcement learning. arXiv preprint arXiv: 1911, 1361, 2019.
[13] W. Zhou, S. Bajracharya, and D. Held. Plas: Latent action space for offline reinforcement learning. arXiv preprint arXhv:2011.07211 2020.
[14] S. Levine, A. Kumar, G. Tucker, and Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643, 2020.
[15] D. Precup. Eligibility traces for off-policy policy evaluation. Computer Science Department Faculty Publication Series, page 80, 2000.
[16] Gu, E. Holly, T. Lillicrap, and S. Levine. Deep reinforcement learning for robotic manipula- tion with asynchronous off-policy updates. In 20/7 IEEE international conference robotics and automation (ICRA), pages 3389-3396. IEEE, 2017.
[17] J. Huang and N. Jiang. From importance sampling to doubly robust policy gradient. In International Conference Machine Learning, pages 4434 4443. PMLR, 2020.
[18] S. Pankov. Reward-estimation variance elimination in sequential decision processes, arXiv preprint 10Xiv:1811.0622. 2018.
[19] C.-A. Cheng, X. Yan, and B. Boots. Trajectory-wise control variates for variance reduction in policy gradient methods. In Conference on Robot Learning, pages 1379-1394. PMLR, 2020.
[20] S. Fujimoto, D. Meger, and D. Precup. Off-policy deep reinforcement learning without explo- ration. In International Conference Machine Learning, pages 0062-2062. PMLR, 2019.

---

[21] N. Y. Siegel. J. T. Springenberg, F. Berkenkamp, A. Abdolmaleki, M. Neunert, T. Lampe, R. Hafner, N. Heess, and M. Riedmiller. Keep doing what worked: Behavioral modelling priors for offline reinforcement learning. arXiv preprini arXiv:2002.08336, 2020.
[22] X. B. Peng, A. Kumar, G. Zhang, and S. Levine. Advantage-weighet regression: Simple and scalable off-policy reinforcement learning. arXiv preprint arXiv:1910.00117,, 2019.
[23] A. Nair, M. Dalal, A. Gupta, and S. Levine. Accelerating online reinforcement learning with offline datasets. arXiv preprint ar (rxx:200060093599 2020.
[24] R. Sutton. Dyna, an integrated architecture for learning, planning, and reacting, ACM Sigart Bulletin, 2(4):160-163, 1991.
[25] M. Watter. J. T. Springenberg. J. Boedecker, and M. Riedmiller. Embed to control: A locally linear latent dynamics model for control from raw images. arXiv preprint 1rXiv:1506.07355 2015.
[26] M. Zhang, S. Vikram, L. Smith, P. Abbeel, M. Johnson, and S. Levine. Solar: Deep struc- tured representations for model-based reinforcement learning. In International Conference on Machine Learning, pages 7444-7453. PMLR, 2019.
[27] D. Hafner, T. Lillicrap. Fischer, R. Villegas, Ha, H. Lee, and J. Davidson. Learning latent dynamics for planning from pixels. In International Conference on Machine Learning, pages 2555-2565. PMLR, 2019.
[28] M. Janner, J. Fu, M. Zhang, and Levine. When to trust your model: Model-based policy optimization, arXiv preprint arXiv:/906.08253, 2019.
[29] K. Chua, R. Calandra, R. McAllister, and S. Levine. Deep reinforcement learning in a handful of trials using probabilistic dynamics models. In Bengio, H. Wallach, H. Larochelle, K. Grau- man, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018. URL httpttts::/pppcceediiite neurips.... cc cc/paper/2018/fiil/ hate://///acaccet777/ooaooaa0000
pdf.
[30] S. Fujimoto, H. Hoof, and D. Meger. Addressing function approximation error in actor-critic methods. In International Conference on Machine Learning, pages 1587-1596. PMLR, 2018.
[31] H. Van Hasselt, A. Guez, and D. Silver. Deep reinforcement learning with double q-learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 30, 2016.
[32] Fu, A. Kumar, O. Nachum, Tucker, and Levine. D4rl: Datasets for deep data-driven reinforcement learning, 2020.
[33] A. Rajeswaran, V. Kumar, A. Gupta, G. Vezzani, J. Schulman, E. Todorov, and S. Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. arXiv preprint arXiv: (xX:v:1709.uu/ 700.70090000 2017.
10


Reviews:
[{"review_id": "znQkDfw8g8", "reviewer": "Reviewer_2d4v", "summary": "Offline reinforcement learning (RL) is an important topic, especially in the robotic community. In fact, in robotic applications, online  RL can be unsafe, as untrained policies can harm the system. Offline RL allows an expert (either a program or a human) to provide some demonstration of the task. The output of the RL algorithm would be, ideally, an agent that solves optimally the desired task leveraging on the suboptimal demonstrations.\n\nThe literature on the topic is rich. One of the main problems, is that the RL systems tends to make assumptions on unseen state-action pairs. When the policy moves towards actions that are not seen in the dataset (out of distribution OOD), the agent might end up in an unknown region of the state-action space. To avoid this, many algorithms attempt to provide a sort of regularization or constraint to avoid OOD actions.\n\nThe contribution of this paper is to introduce a similar regularization also on OOD states. The effect of such regularization term is to penalize OOD states, such that the policy avoids them.\n\nTo this end, the authors introduces a penalization term inspired by conservative $Q$-learning (CQL). Furthermore, they introduce a distribution that has high probability on OOD states, such that the penalization term is stronger in OOD states.\n\nThe paper provides both a theoretical analysis that shows that the value estimate is pessimistic for OOD states, and an empirical analysis that supports the claims of the paper.\n\n", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "__Strenghts__\n\nThe paper provides an effective algorithm to regularize OOD states. The idea seems to be original and well-motivated.\nThe regularization term seems to be effective (at least in the empirical analysis).\n\n__Weaknesses__\n\nIn the first place, I am doubtful about the usage of $d^\\pi$. In particular, I would not focus on assigning high probability to states that have high uncertainty, rather than to quantify the state-distribution induced by the policy (let's call it $d^\\pi$).\nIn fact, this technique would penalize the states that are visited by the \"current policy\" but that are not contained in the dataset. \nQuantify (or sampling from) $d^\\pi$ can be done by sampling by the model $s_{t+1} = s_t + f(s_t, a_t)$ introcued in the paper, or using methods based on DICE for example [1].\n\nTable 1 does not provide confidence intervals, therefore it is difficult to judge the statistical significance of the improvement.\n\n[1] https://arxiv.org/pdf/1906.04733.pdf", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper is, in my opinion, incremental. There is already a large work on offline reinforcement learning that focuses on OOD actions. \nI still think that the paper is worth being accepted at the conference, as the method proposed is logical and original.\n\n\nUPDATE\n==\n\nI have read the other reviews and the authors' responses. I am convinced that the authors did a good job in answering and incorporate the suggestions made. ", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "issues": "I'd be happy to understand why $d^\\phi$ does not approximate the current state distribution (which seems to me more natural).\n\nI would like to see Table 1 with confidence intervals.", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {}}, {"review_id": "u72K6f24j8-", "reviewer": "Reviewer_YDuP", "summary": "This paper proposes a method named PORL to handle OOD states and actions in the offline RL setting. But constructing an OOD state distribution, penalizing values from OOD distributions, the offline learned policy can avoid visiting OOD states, and as a result, show impressive performance in various domains.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n(1) It's a good to directly handle OOD states in offline RL policy evaluation and improvement, instead of implicitly handing OOD actions. \n(2) It is reasonable to leverage uncertainty estimation to construct OOD state distributions. Experimental results also demonstrate its effectivenesses.\n(3) Experimental results on the D4RL benchmark is promising.\n\nWeaknesses:\n(1) Since every states in the offline dataset are seen states, how can we say that some states are not seen when their estimated uncertainty is too high or exceeds a certain threshold? During the training procedure, every state sampled from the offline dataset is a seen state! To generate unseen states, you can use model-based methods such as MOPO.\n\n(2) Theorem 4.1 requires that epsilon is small, while Theorem 4.2 requires that epsilon is sufficiently large. It seems like a conflict.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The motivation of this paper is reasonable and the results are promising. But the authors need to make some basic points more clear, e.g., what they refer to when they say \"unseen states\"? ", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "Already mentioned in \"Strengths and weaknesses\".", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {}}, {"review_id": "mhIHXnWKT9g", "reviewer": "Reviewer_DVzS", "summary": "One of the problems with offline RL is distribution shift, namely the fact that at evaluation time the agent visits parts of the state space which were not part of the training set. The authors propose a solution that builds on top of Conservative Q-Learning (CQL) and constructs a value function with high probability values inside the training dataset, and low probability for Out-Of-Distribution states. They achieve this by detecting OOD states based on their uncertainty and use that to construct a probability distribution $d_\\phi$ with the abovementioned properties, that acts as a regularizer for the Q-function. The distribution $d_\\phi$ heavily penalizes states based on two factors: their uncertainty estimate and their value estimate. This means that states with a higher value will be heavily penalized if their uncertainty is greater than zero. ", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is well written and very clear. The experiments are well designed and many environments are presented.\nThe claims are supported by theoretical guarantees and by strong experimental performances against different baselines.\nI would suggest including some references related to OOD detection based on states (see below).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "This paper provides an intuitive solution to the problem of distribution shift in batch RL. The claims are well supported by theoretical and empirical evidence. For this reason, I recommend acceptance.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "Fair: Some knowledge of the area"}, "extra_text": {"issues": "- How is the score normalized in table 1?\n- There is a work that considers pessimistic initialization of the Q-function called OPIQ [1] and presented last year at ICLR. It doesn't explicitly consider OOD scenarios, but it is related to having a pessimistic value of the Q-function for unseen state-action pairs.\nIt could be of interest to show how PORL compares to that.\n- It could also be interesting to see how PORL compares to other methods that use different types of OOD detection. Did you try some of the approaches mentioned in the related work?\n\n[1] \"Optimistic Exploration even with a Pessimistic Initialisation\" Tabish Rashid, Bei Peng, Wendelin Boehmer, Shimon Whiteson, ICLR2020"}}, {"review_id": "0otyzd4XD9y", "reviewer": "Reviewer_Qikv", "summary": "This submission proposed Pessimistic Offline Reinforcement Learning (PORL), an\noffline-RL algorithm which addresses issues associated with out-of-distribution\n(OOD) states, while previous work only focused on OOD actions.  PORL extends\nprior work by incorporating a regularization term to lower the values\nassociated with unseen states, and which is inspired by a prior regularization\nterms which was designed to lower the values associated with unseen actions.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* Extends similar methods for offline-RL in interesting and non-trivial ways.\n* I strongly appreciate the focus on theory, rather than presenting only\n  empirical results.\n\nWeaknesses:\n* Seems like the authors did not perform hyper-parameter selection during the\n  evaluation, which is a huge red flag.\n* A few key details are missing both in the theory and in the practical\n  implementation (see Issues section).\n* Some notation is confusing and/or not properly defined, or contains mistakes.\n* Not very related to robot learning, and no robot experiments (not even\n  simulated).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "While I generally like the work and the main contribution of the submission, as\nit stands, the submission has a number of flaws and issues which need to be\naddressed.  They are not all equally important, and the two main ones which\nreally concern me are the lack of good hyperparameter selection (or a\ndescription of how the PORL hyperparameters are chosen), and the questionable\nrelevance to the community of robot learning.\n\nMy current recommendation is based on the submission as it stands, and will\ngladly update it based on the author response.", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "1: Not very relevant to the robot learning community.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {"issues": "Major:\n\n* Equations (1) and (2) do not describe Q-learning, as only the Bellman\n  *evaluation* operator is used, rather than the Bellman *optimality* operator.\n  Equations (1) and (2) will approximately lead to the value function of the\n  agent which was used to perform the data collection.  However, there is no\n  underlying assumption made in offline-RL (or in this paper) that the\n  data-collecting agent is optimal or near-optimal.\n\n* Why does $d^\\phi(s)$ need to specifically be a distribution which assigns low\n  probabilities to OOD states?  Can it not simply assign a uniform probability\n  to all states?  In that case, the states present in the dataset will still\n  have $d^{\\pi_\\beta}(s) > d^\\phi(s)$, and the states not present in the\n  dataset will still have $d^{\\pi_\\beta}(s) < d^\\phi(s)$.  Isn't it possible\n  that this would work well enough, without the need to learn the distribution\n  $d^\\phi(s)$?\n\n* It is not obvious whether $\\hat V^\\pi(s) \\leq V^\\pi(s)$ implies $\\hat\n  Q^\\pi(s, a) \\leq V^\\pi(s, a)$;  some discussion / proof should be provided.\n\n* It is not clear how Theorem 4.2 implies that the PORL models a pessimistic\n  MDP as formally defined in [10], as claimed in lines 163-164.  In fact, the\n  formal definition of pessimistic-MDPs requires components, such as an\n  additional absorbing state \"HALT\", which does not appear at all in this\n  submission.  Finally, it is not even clear why this should matter.  The\n  important property seems to simply be that $\\hat V^\\pi(s) < V^\\pi$.  Why is\n  it important that PORL learns values which are hypothetically associated with\n  pessimistic MDPs?\n\n* According to Theorem 4.1, $\\varepsilon$ should be small enough to satisfy the\n  inequality.  According to Theorem 4.2, $\\varepsilon$ should be large enough\n  to satisfy the equality in eq.(21) from the appendix.  However, no practical\n  method is given to find a $\\varepsilon$ which satisfies both conditions, or\n  to even verify that a given $\\varepsilon$ satisfies such conditions.  It is\n  also not proven whether there can be a $\\varepsilon$ which is both small\n  enough and large enough to satisfy both conditions.\n\n* \"All baselines are run with the default code and hyperparameters from the\n  original repositories.\"  Were these hyperparameters optimized for the same\n  environments used in your evaluation?  It doesn't seem likely.  If so, this\n  is a huge flaw in the evaluation.  How are the hyperparameters of PORL\n  chosen?\n\nMid:\n\n* The problems associated with distributional shift, OOD states and actions,\n  could be described better, e.g., in lines 24-26, it is claimed that\n  distributional shift might cause problems, but what kind of problems is not\n  well described (yet).\n\n* Neither $\\mu$ nor its role are properly described.  It is particularly\n  confusing that $\\mu$ is described to match the state-marginal, while in Eq.4,\n  it appears as the state-conditional \"policy\" $\\mu(a\\mid s)$.\n\n* According to Eq.4, both CQL and PORL assumes access to the behavior policy\n  $\\pi_\\beta$, which is a significantly stronger requirement compared to simple\n  access to the dataset $\\mathcal{D}$.  Or is it simply the case that the\n  dataset actions (which are sampled according to the behavior policy) are used\n  directly?  The details of how objective $J$ of Eq.7 is estimated should be\n  provided (which expectations are computed exactly?  which are approximated\n  using a Monte Carlo estimation).\n\n* in lines 124 and 125, the statement \"it is empirically impossible to derive a\n  solution to maximize the KL divergence\" needs a citation or further\n  justification.\n\n* In Eq.5, why only penalize a single action-value associated with unseen\n  states, as opposed to all action-values associated with unseen states?  I.e.,\n  why only use the action $a\\sim\\hat \\pi^k(a\\mid s)$?\n\n* Theorem 4.1 contains confusing statements which switch between $Q$ and $V$.\n  \"For any action and any state ..., i.e. $\\hat V^\\pi(s) \\leq V^\\pi(s)$ ...\".\n\n* In Theorem 4.1, the definition of $\\hat V^\\pi$ was unclear.  I assume the\n  intention was to define $\\hat V^\\pi$ as the convergence value $\\hat V^\\pi(s)\n  = \\lim_{k\\to\\infty} \\hat V^k(s)$, but the way the sentence is worded it seems\n  like there is an assumption that $\\hat V^k$ converges to a specific value.\n\n* A number of issues are present in the appendix, usually related to the\n  authors skipping or not describing steps sufficiently.  Since this is the\n  appendix, the authors should feel free to use enough space to be very clear,\n  and complement the math with written descriptions of what is happening,\n  without the burden of limited pages:\n\n  * In line 384 of the appendix, \"Setting the derivative to zero\".  The\n    derivative with respect to what?  It is not clear.  Also, how is the\n    derivative of the two regularization terms computed?\n\n  * In line 386 of the appendix, \"It is obvious that the last two terms of Eqn.\n    8, ..., is indefinite\".  It is not obvious at all, and it is also unclear\n    what the authors mean by \"indefinite\".  It cannot be a real number?  Since\n    this is the appendix, the authors should feel free to use as much space as\n    possible to be as clear as possible.\n\n  * In line 400 of the appendix, \"Hence, we have ... $\\hat V^{k+1} \\leq\n    \\mathfrak{B}^\\pi \\hat V^k$, which implies that each value-function update\n    is a contraction.\"  I am not sure if this is a typo, but this does not seem\n    to align with the formal definition of contraction, which is an operator\n    which guarantees the outputs associated with any two inputs are closer to\n    each other than the inputs to each other.  What do you mean by\n    \"contraction\" here?\n\n  * Eq 13 in the appendix suddenly mixes point-wise and vector notation.\n\nMinor:\n\n* PORL is an unfortunate acronym, since it conflicts with the widely used\n  Partially Observable Reinforcement Learning (PORL).\n\n* Calling Q-learning \"dynamic-programming\" (happens a few times) is inaccurate.\n\n* Notation of policy in line 84 does not work for discrete actions;  should be\n  $\\delta\\left[ a = \\arg\\max_a \\ldots \\right]$.\n\n* $\\mathcal{E}$ and $\\mathcal{C}$ should be defined closer to their respective\n  equations, i.e., i.e., $\\mathcal{E}$ should be defined near Eq.1, and\n  $\\mathcal{C}$ should be defined near Eq.4.\n\n* The notation is at times a little bit redundant, e.g., $s\\sim\\mathcal{D}$\n  represents a state sampled from the dataset, and $s\\sim d^{\\pi_\\beta}$\n  represents the same exact thing.\n\n* Not accurate to use the same operator $\\mathfrak{B}^\\pi$ for both\n  state-values $V$ and action-values $Q$.\n\n* Pessimistic MDPs should be formally described in the Background section."}}]

Author rebuttal:

Meta-review: The reviewers generally found the paper interesting, well-motivated, and significant. I am happy that the authors used the rebuttal period efficiently, and have improved the submission in a number of ways based on reviewers' initial concerns.
Decision: Accept (Poster)