Title: Where Off-Policy Deep Reinforcement Learning Fails
Authors: Scott Fujimoto
David Meger
Doina Precup
Venue: ICLR 2019
Keywords: reinforcement learning
off-policy
imitation
batch reinforcement learning

Abstract: This work examines batch reinforcement learning--the task of maximally exploiting a given batch of off-policy data, without further data collection. We demonstrate that due to errors introduced by extrapolation, standard off-policy deep reinforcement learning algorithms, such as DQN and DDPG, are only capable of learning with data correlated to their current policy, making them ineffective for most off-policy applications. We introduce a novel class of off-policy algorithms, batch-constrained reinforcement learning, which restricts the action space to force the agent towards behaving on-policy with respect to a subset of the given data. We extend this notion to deep reinforcement learning, and to the best of our knowledge, present the first continuous control deep reinforcement learning algorithm which can learn effectively from uncorrelated off-policy data.

TL;DR: We describe conditions where off-policy deep reinforcements algorithms fail and present a solution.

Review discussion:
This paper studies extrapolation error in off-policy batch reinforcement learning (RL), where the extrapolation error refers to the overestimation of the value for the state-action pairs that are not in the training data.
The authors propose batch-constrained RL, where the policy is optimize under the constraint that, at each state, only those actions that have been taken in that state in the training data are allowed.  This is then extended to continuous space, where it allows only the state-action pairs that are close to a state-action pair in the training data.
When there is no such action for a given state, the action that is closet to a feasible action at that state is selected.
It makes intuitive sense that the proposed approach works well as long as we only encounter state-action pairs that are closed to one of the state-action pairs in the batch.
However, I do not expect that this is always the case.  The proposed method is to simply choose the closest action in the batch.
Then why does the proposed approach perform well?  Is it because the experiments are performed under rather deterministic settings?  How often are no state-action pairs found in the neighbor?  Is there any mechanism for recovering from "not in the batch"?
The paper would be much stronger if it study this challenge of "not in the batch" more in depth.  Technical contributions in the present paper are rather limited.
A key assumption in the discrete case is that whole episodes are in the batch.  This is rather restricting, because in many applications, it is infeasible to collect a whole episode, and parts of many episodes are collected from many agents.
Although this assumption is stated, it would be nice to emphasize by also stating that the theorems do not hold when this assumption does not hold.  The assumption becomes less important for continuous case, because of approximation.  It might be interesting to study the performance of the proposed approach when the assumption does not hold in the continuous case.
We would like to thank the reviewer for their time, feedback and thoughts. A concern presented by the reviewer was limited technical contribution. We would like to re-emphasize our contribution towards the introduction and analysis of extrapolation error in off-policy learning. Our paper provides important insight into the working of deep reinforcement learning with finite amounts of data, or the purely “exploitative” setting, as well as imitation learning with noisy demonstrations.
> It makes intuitive sense that the proposed approach works well as long as we only encounter state-action pairs that are closed to one of the state-action pairs in the batch.
However, I do not expect that this is always the case.  The proposed method is to simply choose the closest action in the batch.
Then why does the proposed approach perform well?
In regions with no data at all, there is no possible mechanism for recovery because the agent, and any possible agent, will not have trained in this region. Taking the action of the closest state-action pair is an oversimplification of our method, which likely stems from the lack of clarity in our original version of Section 4.2, which has been re-written. Our BCQ algorithm produces deep network policies that can be evaluated across the entire state space and considers both the similarity of the action to the batch as well as the expected value of the action.
That being said, it is important to take actions which are “close” in the Bellman update to minimize the extrapolation error in the value estimate.
Otherwise, as shown in Section 3.2, there can be deterioration in performance even in regions of certainty.
That is, a non-batch-constrained off-policy reinforcement learning algorithm may fail if exposed to any uncertain regions during training. Our algorithm performs well by reducing the error into the system. Informally, our value estimates are more accurate.
In experiments where BCQ may take actions leading it to unseen states, such as in the experiments with an expert behavioral policy without exploration, we find that there is sufficient generalization to regions with less data to still perform well, while stabilizing the value function.
For future work, an interesting extension of the algorithm would be to bias it towards regions of certainty, through an optimism-under-certainty heuristic, the polar-opposite to many exploration algorithms. This occurs implicitly in our algorithm as mimicking previously taken actions is more likely to lead to regions of certainty, but could be enforced more strongly.
> A key assumption in the discrete case is that whole episodes are in the batch.  This is rather restricting, because in many applications, it is infeasible to collect a whole episode, and parts of many episodes are collected from many agents.
The data doesn't need to be collected in episodic fashion, rather, that there is sufficient coverage. Collecting data in episodes is one way to ensure this, but not specifically required. This is a weaker assumption than assumptions necessary for standard Q-learning, as we no longer require visitation over all possible state-action pairs.
Authors consider a problem of off-policy reinforcement learning in a setting explicitly constrained to a fixed batch of transitions.
The argument is that popular RL methods underperform significantly in this setting because they fail to account for extrapolation error caused by inevitably sparse sampling of the possible action-state space.
To address this problem, authors introduce the notion of batch-constrained RL which studies policies and associated value functions only on the state-space covered by the available training data.
For practical applications a deep RL method is introduced which enables generalisation to the unseen states and actions by the means of function approximation.
I find the problem studied in the paper very important. It is indeed strongly connected to the idea of imitation learning which has been studied previously, but I like the explicit point from which authors see the problem.
The experimental results seem quite appealing to justify use of the proposed approach.
However, on the clarity side the paper should be improved before publication.
The interplay between action generating VAE G_w(s) and \pi is unclear to me.
First, what does it mean that G(s) is trained to minimise the distance D_A?
If G(s) is a VAE, then it is trained to minimise the corresponding variational lower bound, how is minimisation of the distance over actions is incorporated here? And what exactly is this distance?
Similarly, what does “D_S will be defined by the implicit distance induced by the function approximation” exactly mean?
Other comments / questions:
Page 6: “Theorem 1 implies with access to only
a subset of state-action pairs in the MDP, the value function… This suggests batch-constrained policies
are a necessary tool for combating extrapolation bias.”
This might be true, but it does not follow from the Theorem 1 as it only applies to the batch Bellman operator and not the standard one used in most methods.
Corollary 1 and 2: What is Q^* here?
Page 7, first sentence: should there be if A_s, e != \emptyset?
Epsion-Batch-constrained policy iteration: would the beam search actually maximize Q function? This needs to be proven or at least discussed.
I don’t see how is epsilon used in the iteration scheme.  This needs to be clarified.
Equation 11: the subscript of the max operator looks weird, should there be just a_i?
Figure 4: where is “True value” curve on the plots?
The notation \pi(s, a; \Phi) used throughout the paper is confusing and can be interpreted as a joint distribution over states and actions.
As I said, currently the paper does not appear to be easy to follow to me and even if it does contain important ideas, I believe they must be communicated in a clearer way.
I am eager to revise my evaluation if authors make substantial effort to improve the paper.
We would like to thank the reviewer for thorough review and constructive feedback. The issues with clarity largely stemmed from Section 4.2, which we agree with the reviewer was not as clear as it could be. This section has been re-written and will hopefully satisfy the reviewer. We have removed superfluous details and simplified the presentation of Section 4.2. We believe these changes better streamlines the introduction of the (unchanged) algorithm, and better justifies some of the algorithmic choices. Other small adjustments to notation and clarity have been made throughout the paper, with regards to both your comments as well as the other reviewers.
Further Responses to Comments:
> Page 6: “Theorem 1 implies with access to only a subset of state-action pairs in the MDP, the value function… This suggests batch-constrained policies are a necessary tool for combating extrapolation bias.” This might be true, but it does not follow from the Theorem 1 as it only applies to the batch Bellman operator and not the standard one used in most methods.”
The claim we intended to make was not that batch-constrained policies are necessary, but rather suggest that they are likely, or potentially, necessary. We have clarified this in the paper.
> Figure 4: where is “True value” curve on the plots?
Initially we left out the true value curve to allow for a larger figure, putting more emphasis on the results. We have re-added the true value curve.
Summary:
Proposes BCRL for learning from a fixed collection of off-policy experience (I'll call this the "training data"). BCRL attempts to avoid backing up values from states that are not present in the training data, on the assumption that the current estimates of these values are likely to be inaccurate. In the continuous state-action case, this is accomplished by training a generative model to propose, given a state `s`, an action `a` such that a transition similar to `(s, a)` is in the training data. A secondary policy is then trained to perturb the proposed action within a constrained region to maximize value. BCRL outperforms DDPG and DQN when learning from fixed data, but BCRL is slightly worse than behavior cloning at learning to reproduce an expert policy that does not take exploration actions.
Review:
The overall approach is sound. The problem of extrapolation is intuitively obvious, but not something I had thought about before. I think typically exploration would correct the problem since states with over-estimated values would become more likely to be reached, giving an opportunity to get a better estimate.
The learning setting is closer to imitation learning than to what I would call RL, since the BCRL approach essentially avoids extrapolation error by ignoring the parts of the problem that are not represented in the training data. The well-known problem with behavior cloning is compounding errors once the agent strays into areas of the state space that are far from the training data. To me "off-policy RL" implies that the goal is to learn a complete policy from off-policy data. I think the "competitors" to which BCRL should be compared are imitation learning algorithms address noisy demonstrations, and not so much off-policy RL algorithms. It would also be interesting to see the generalization performance of BCRL outside of its training data.
The BCRL idea might be applicable in a conventional RL setting as well, since the initial stages of learning could be subject to a similar extrapolation error until there has been enough exploration. A comparison to something like TRPO in this setting would be interesting.
The paper is well-written with good coverage of related literature. There are a few points where the technical content is imprecise, which I note below.
Comments / Questions:
* Could one obtain a similar effect to BCRL by simply initializing the value estimates pessimistically?
* Sec 4.1: Since B is a set of (s, a, s', r) tuples, what does it mean for a state s' to be "in B"? Similar question for state-action tuples (s, a).
* As you note in the appendix, the construction in Sec 4.1 is essentially creating a new MDP that contains only the transitions that occur in the training data. I'd suggest stating as much in the main paper for intuition.
* Sec 4.2 / 5: The perturbation constraint \Phi is set to 0.05 in the experiments. Since the actions in these control problems are vectors, what does a scalar constraint correspond to? How is the constraint enforced during learning?
* What are the distance functions D_S and D_A?
Pros:
* A good approach to applying RL methods in the "imitation-like" setting. I've seen similar things attempted before, but this method makes more sense.
Cons:
* The learning setting is more like "fuzzy" behavior cloning from noisy data than off-policy RL. Experimental comparison against more-sophisticated imitation learning approaches is missing.
We would like to thank the reviewer for their helpful comments and positive feedback. We have added an experimental result to the supplementary material to distinguish ourselves further from imitation learning algorithms and made several clarifying statements and adjustments based on your recommendations.
One con listed was missing comparisons against other state of the art imitation learning algorithms which are robust to noisy demonstrations.
However, to the best of our knowledge, we are not aware of any which satisfy the batch setting, where no further data is collected, while also setting no requirements on data being labelled expert vs. non-expert. One algorithm which does satisfy these conditions is [1], but only operates in with discrete actions, making it weak baseline in a continuous control benchmark, where independent discretization would be required. If there was a particular algorithm you had in mind when writing the review, we would be happy to include it in the final paper.
We also note the line between off-policy and robust imitation is fairly thin.
For example, in the tabular setting, our approach can learn from the set of data that includes all state-action pairs, similarly to off-policy learning. All state-action pairs, of course, also includes expert actions and could be considered a robust imitation learning algorithm as well. An expert behavioral policy is necessary for the data collection process to be sufficiently interesting, as a purely randomly policy doesn’t cover enough of the state space for it to be possible to learn meaningful behavior. To further demonstrate the effectiveness of our algorithm as an off-policy algorithm, we included results with a purely random behavioral policy on a pendulum and reacher task in the supplementary material B, where the state space can be sufficiently covered by taking random actions.
Further Responses to Questions/Comments:
> Could one obtain a similar effect to BCRL by simply initializing the value estimates pessimistically?
Essentially yes, especially in the tabular setting, however, this would slow learning as it may take many updates to “wash away” initial negative bias.
Furthermore, in a function approximation setting, maintaining an optimistic or pessimistic initialization over many timesteps is impractical and often implausible.
Finally, for a fixed, non-batch-constrained policy, this also gives biased estimates. Introducing the notion of batch-constrained gives some understanding to when the policy would be biased vs.
when it wouldn't.
> Sec 4.1: Since B is a set of (s, a, s', r) tuples, what does it mean for a state s' to be "in B"? Similar question for state-action tuples (s, a).
s' in B is shorthand for (s, a, s', r) in B for some s, a, r. We have added a clarifying sentence in the background.
> As you note in the appendix, the construction in Sec 4.1 is essentially creating a new MDP that contains only the transitions that occur in the training data. I'd suggest stating as much in the main paper for intuition.”
At your recommendation we have added this to the main paper.
> * Sec 4.2 / 5: The perturbation constraint \Phi is set to 0.05 in the experiments. Since the actions in these control problems are vectors, what does a scalar constraint correspond to? How is the constraint enforced during learning?
This correspond to \Phi * I * tanh() following the final layer. We have added a clarifying sentence in the supplementary.
References:
[1] Gao, Yang, Ji Lin, Fisher Yu, Sergey Levine, and Trevor Darrell. "Reinforcement learning from imperfect demonstrations." arXiv preprint arXiv:1802.05313 (2018).
We would like to take this opportunity to thank each reviewer again. We found that the quality of the reviews was high and the reviewers made insightful commentary, each with different flavors with respect to the paper. As this paper introduces the first analysis into the batch setting with deep function approximation, it makes sense that there would be small issues in clarity, and as displayed by our lengthy related works, there are many possible interpretations to where, and how, our work is insightful or significant. We have carefully responded to each reviewer and have made many small updates throughout the paper to improve the clarity and dispel any confusion about the task as well as our approach.
As mentioned in our response to Reviewer 2, almost all significant issues with clarity were from Section 4.2 which we have re-written to better justify the design choices made when approximating the batch-constraint in a continuous setting. We have expanded the introduction on extrapolation error to be more explicit on its origin.
Additionally, although no reviewer took issue with the original title, we have taken the opportunity to modify the title to be more informative towards the contents of the paper. We believe these changes address the reviewers’ concerns and greatly improve the quality of the paper. We will continue to edit the paper before the deadline and our happy to respond to further comments, questions or concerns.

Meta-review:
role: AC
data: metareview: The paper proposes batch-constrained approach to batch RL, where the policy is optimized under the constrain that at a state only actions appearing in the training data are allowed.  An extension to continuous cases is given.

While the paper has some interesting idea and the problem of dealing with extrapolation in RL is important, the approach appears somewhat ad hoc and the contributions limited.

For example, the constraint is based on whether (s,a) is in B, but this condition can be quite delicate in a stochastic problem (seeing a in s *once* may still allow large extrapolation error if that only observed transition is not representative).  Section 4.1 gives some nice insights for the special finite MDP case, but those results are a little weak (requiring strong assumption that may not hold in practice) --- an example being the requirement that s' be included in data if (s,a) is in data and P(s'|s,a)>0 [beginning of section 4.1].

In contrast, there are other more robust and principled ways, such as counterfactual risk minimization (CRM) for contextual bandits (http://www.jmlr.org/papers/v16/swaminathan15a.html).  For MDPs, the Bayesian version of DQN (the cited Azizzadenesheli et al., as well as Lipton et al. at AAAI'18) can be used to constrain the learned policy as well, with a simple modification of using the CRM idea for bandits.  Would these algorithms be reasonable baselines?

Decision: Reject