Venue: CoRL 2021 Conference

Reviews:

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This submission proposed Pessimistic Offline Reinforcement Learning (PORL), an
offline-RL algorithm which addresses issues associated with out-of-distribution
(OOD) states, while previous work only focused on OOD actions.  PORL extends
prior work by incorporating a regularization term to lower the values
associated with unseen states, and which is inspired by a prior regularization
terms which was designed to lower the values associated with unseen actions.
strengths_and_weaknesses: Strengths:
* Extends similar methods for offline-RL in interesting and non-trivial ways.
* I strongly appreciate the focus on theory, rather than presenting only
  empirical results.

Weaknesses:
* Seems like the authors did not perform hyper-parameter selection during the
  evaluation, which is a huge red flag.
* A few key details are missing both in the theory and in the practical
  implementation (see Issues section).
* Some notation is confusing and/or not properly defined, or contains mistakes.
* Not very related to robot learning, and no robot experiments (not even
  simulated).
summary_of_recommendation: While I generally like the work and the main contribution of the submission, as
it stands, the submission has a number of flaws and issues which need to be
addressed.  They are not all equally important, and the two main ones which
really concern me are the lack of good hyperparameter selection (or a
description of how the PORL hyperparameters are chosen), and the questionable
relevance to the community of robot learning.

My current recommendation is based on the submission as it stands, and will
gladly update it based on the author response.
issues: Major:

* Equations (1) and (2) do not describe Q-learning, as only the Bellman
  *evaluation* operator is used, rather than the Bellman *optimality* operator.
  Equations (1) and (2) will approximately lead to the value function of the
  agent which was used to perform the data collection.  However, there is no
  underlying assumption made in offline-RL (or in this paper) that the
  data-collecting agent is optimal or near-optimal.

* Why does $d^\phi(s)$ need to specifically be a distribution which assigns low
  probabilities to OOD states?  Can it not simply assign a uniform probability
  to all states?  In that case, the states present in the dataset will still
  have $d^{\pi_\beta}(s) > d^\phi(s)$, and the states not present in the
  dataset will still have $d^{\pi_\beta}(s) < d^\phi(s)$.  Isn't it possible
  that this would work well enough, without the need to learn the distribution
  $d^\phi(s)$?

* It is not obvious whether $\hat V^\pi(s) \leq V^\pi(s)$ implies $\hat
  Q^\pi(s, a) \leq V^\pi(s, a)$;  some discussion / proof should be provided.

* It is not clear how Theorem 4.2 implies that the PORL models a pessimistic
  MDP as formally defined in [10], as claimed in lines 163-164.  In fact, the
  formal definition of pessimistic-MDPs requires components, such as an
  additional absorbing state "HALT", which does not appear at all in this
  submission.  Finally, it is not even clear why this should matter.  The
  important property seems to simply be that $\hat V^\pi(s) < V^\pi$.  Why is
  it important that PORL learns values which are hypothetically associated with
  pessimistic MDPs?

* According to Theorem 4.1, $\varepsilon$ should be small enough to satisfy the
  inequality.  According to Theorem 4.2, $\varepsilon$ should be large enough
  to satisfy the equality in eq.(21) from the appendix.  However, no practical
  method is given to find a $\varepsilon$ which satisfies both conditions, or
  to even verify that a given $\varepsilon$ satisfies such conditions.  It is
  also not proven whether there can be a $\varepsilon$ which is both small
  enough and large enough to satisfy both conditions.

* "All baselines are run with the default code and hyperparameters from the
  original repositories."  Were these hyperparameters optimized for the same
  environments used in your evaluation?  It doesn't seem likely.  If so, this
  is a huge flaw in the evaluation.  How are the hyperparameters of PORL
  chosen?

Mid:

* The problems associated with distributional shift, OOD states and actions,
  could be described better, e.g., in lines 24-26, it is claimed that
  distributional shift might cause problems, but what kind of problems is not
  well described (yet).

* Neither $\mu$ nor its role are properly described.  It is particularly
  confusing that $\mu$ is described to match the state-marginal, while in Eq.4,
  it appears as the state-conditional "policy" $\mu(a\mid s)$.

* According to Eq.4, both CQL and PORL assumes access to the behavior policy
  $\pi_\beta$, which is a significantly stronger requirement compared to simple
  access to the dataset $\mathcal{D}$.  Or is it simply the case that the
  dataset actions (which are sampled according to the behavior policy) are used
  directly?  The details of how objective $J$ of Eq.7 is estimated should be
  provided (which expectations are computed exactly?  which are approximated
  using a Monte Carlo estimation).

* in lines 124 and 125, the statement "it is empirically impossible to derive a
  solution to maximize the KL divergence" needs a citation or further
  justification.

* In Eq.5, why only penalize a single action-value associated with unseen
  states, as opposed to all action-values associated with unseen states?  I.e.,
  why only use the action $a\sim\hat \pi^k(a\mid s)$?

* Theorem 4.1 contains confusing statements which switch between $Q$ and $V$.
  "For any action and any state ..., i.e. $\hat V^\pi(s) \leq V^\pi(s)$ ...".

* In Theorem 4.1, the definition of $\hat V^\pi$ was unclear.  I assume the
  intention was to define $\hat V^\pi$ as the convergence value $\hat V^\pi(s)
  = \lim_{k\to\infty} \hat V^k(s)$, but the way the sentence is worded it seems
  like there is an assumption that $\hat V^k$ converges to a specific value.

* A number of issues are present in the appendix, usually related to the
  authors skipping or not describing steps sufficiently.  Since this is the
  appendix, the authors should feel free to use enough space to be very clear,
  and complement the math with written descriptions of what is happening,
  without the burden of limited pages:

  * In line 384 of the appendix, "Setting the derivative to zero".  The
    derivative with respect to what?  It is not clear.  Also, how is the
    derivative of the two regularization terms computed?

  * In line 386 of the appendix, "It is obvious that the last two terms of Eqn.
    8, ..., is indefinite".  It is not obvious at all, and it is also unclear
    what the authors mean by "indefinite".  It cannot be a real number?  Since
    this is the appendix, the authors should feel free to use as much space as
    possible to be as clear as possible.

  * In line 400 of the appendix, "Hence, we have ... $\hat V^{k+1} \leq
    \mathfrak{B}^\pi \hat V^k$, which implies that each value-function update
    is a contraction."  I am not sure if this is a typo, but this does not seem
    to align with the formal definition of contraction, which is an operator
    which guarantees the outputs associated with any two inputs are closer to
    each other than the inputs to each other.  What do you mean by
    "contraction" here?

  * Eq 13 in the appendix suddenly mixes point-wise and vector notation.

Minor:

* PORL is an unfortunate acronym, since it conflicts with the widely used
  Partially Observable Reinforcement Learning (PORL).

* Calling Q-learning "dynamic-programming" (happens a few times) is inaccurate.

* Notation of policy in line 84 does not work for discrete actions;  should be
  $\delta\left[ a = \arg\max_a \ldots \right]$.

* $\mathcal{E}$ and $\mathcal{C}$ should be defined closer to their respective
  equations, i.e., i.e., $\mathcal{E}$ should be defined near Eq.1, and
  $\mathcal{C}$ should be defined near Eq.4.

* The notation is at times a little bit redundant, e.g., $s\sim\mathcal{D}$
  represents a state sampled from the dataset, and $s\sim d^{\pi_\beta}$
  represents the same exact thing.

* Not accurate to use the same operator $\mathfrak{B}^\pi$ for both
  state-values $V$ and action-values $Q$.

* Pessimistic MDPs should be formally described in the Background section.
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: One of the problems with offline RL is distribution shift, namely the fact that at evaluation time the agent visits parts of the state space which were not part of the training set. The authors propose a solution that builds on top of Conservative Q-Learning (CQL) and constructs a value function with high probability values inside the training dataset, and low probability for Out-Of-Distribution states. They achieve this by detecting OOD states based on their uncertainty and use that to construct a probability distribution $d_\phi$ with the abovementioned properties, that acts as a regularizer for the Q-function. The distribution $d_\phi$ heavily penalizes states based on two factors: their uncertainty estimate and their value estimate. This means that states with a higher value will be heavily penalized if their uncertainty is greater than zero. 
strengths_and_weaknesses: The paper is well written and very clear. The experiments are well designed and many environments are presented.
The claims are supported by theoretical guarantees and by strong experimental performances against different baselines.
I would suggest including some references related to OOD detection based on states (see below).
summary_of_recommendation: This paper provides an intuitive solution to the problem of distribution shift in batch RL. The claims are well supported by theoretical and empirical evidence. For this reason, I recommend acceptance.
issues: - How is the score normalized in table 1?
- There is a work that considers pessimistic initialization of the Q-function called OPIQ [1] and presented last year at ICLR. It doesn't explicitly consider OOD scenarios, but it is related to having a pessimistic value of the Q-function for unseen state-action pairs.
It could be of interest to show how PORL compares to that.
- It could also be interesting to see how PORL compares to other methods that use different types of OOD detection. Did you try some of the approaches mentioned in the related work?

[1] "Optimistic Exploration even with a Pessimistic Initialisation" Tabish Rashid, Bei Peng, Wendelin Boehmer, Shimon Whiteson, ICLR2020
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: Offline reinforcement learning (RL) is an important topic, especially in the robotic community. In fact, in robotic applications, online  RL can be unsafe, as untrained policies can harm the system. Offline RL allows an expert (either a program or a human) to provide some demonstration of the task. The output of the RL algorithm would be, ideally, an agent that solves optimally the desired task leveraging on the suboptimal demonstrations.

The literature on the topic is rich. One of the main problems, is that the RL systems tends to make assumptions on unseen state-action pairs. When the policy moves towards actions that are not seen in the dataset (out of distribution OOD), the agent might end up in an unknown region of the state-action space. To avoid this, many algorithms attempt to provide a sort of regularization or constraint to avoid OOD actions.

The contribution of this paper is to introduce a similar regularization also on OOD states. The effect of such regularization term is to penalize OOD states, such that the policy avoids them.

To this end, the authors introduces a penalization term inspired by conservative $Q$-learning (CQL). Furthermore, they introduce a distribution that has high probability on OOD states, such that the penalization term is stronger in OOD states.

The paper provides both a theoretical analysis that shows that the value estimate is pessimistic for OOD states, and an empirical analysis that supports the claims of the paper.


strengths_and_weaknesses: __Strenghts__

The paper provides an effective algorithm to regularize OOD states. The idea seems to be original and well-motivated.
The regularization term seems to be effective (at least in the empirical analysis).

__Weaknesses__

In the first place, I am doubtful about the usage of $d^\pi$. In particular, I would not focus on assigning high probability to states that have high uncertainty, rather than to quantify the state-distribution induced by the policy (let's call it $d^\pi$).
In fact, this technique would penalize the states that are visited by the "current policy" but that are not contained in the dataset. 
Quantify (or sampling from) $d^\pi$ can be done by sampling by the model $s_{t+1} = s_t + f(s_t, a_t)$ introcued in the paper, or using methods based on DICE for example [1].

Table 1 does not provide confidence intervals, therefore it is difficult to judge the statistical significance of the improvement.

[1] https://arxiv.org/pdf/1906.04733.pdf
summary_of_recommendation: The paper is, in my opinion, incremental. There is already a large work on offline reinforcement learning that focuses on OOD actions. 
I still think that the paper is worth being accepted at the conference, as the method proposed is logical and original.


UPDATE
==

I have read the other reviews and the authors' responses. I am convinced that the authors did a good job in answering and incorporate the suggestions made. 
issues: I'd be happy to understand why $d^\phi$ does not approximate the current state distribution (which seems to me more natural).

I would like to see Table 1 with confidence intervals.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper proposes a method named PORL to handle OOD states and actions in the offline RL setting. But constructing an OOD state distribution, penalizing values from OOD distributions, the offline learned policy can avoid visiting OOD states, and as a result, show impressive performance in various domains.
strengths_and_weaknesses: Strengths:
(1) It's a good to directly handle OOD states in offline RL policy evaluation and improvement, instead of implicitly handing OOD actions. 
(2) It is reasonable to leverage uncertainty estimation to construct OOD state distributions. Experimental results also demonstrate its effectivenesses.
(3) Experimental results on the D4RL benchmark is promising.

Weaknesses:
(1) Since every states in the offline dataset are seen states, how can we say that some states are not seen when their estimated uncertainty is too high or exceeds a certain threshold? During the training procedure, every state sampled from the offline dataset is a seen state! To generate unseen states, you can use model-based methods such as MOPO.

(2) Theorem 4.1 requires that epsilon is small, while Theorem 4.2 requires that epsilon is sufficiently large. It seems like a conflict.
summary_of_recommendation: The motivation of this paper is reasonable and the results are promising. But the authors need to make some basic points more clear, e.g., what they refer to when they say "unseen states"? 
issues: Already mentioned in "Strengths and weaknesses".
review_title: null

Meta-review: The reviewers generally found the paper interesting, well-motivated, and significant. I am happy that the authors used the rebuttal period efficiently, and have improved the submission in a number of ways based on reviewers' initial concerns.
Decision: Accept (Poster)