Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This paper tackles the offline RL setting, in which a policy is learned entirely from historical data, rather than through real-time online interaction with a simulator. However, existing algorithms such as CQL, while able to learn conservative lower bounds on the true Q-values, learn point estimates, making it difficult to also account for risk. Key contributions include:
1. bridging the gap between distributional RL and conservative value-based RL (CQL) for the first time, in which a lower bound on the quantiles of the return distribution can be learned through a penalization approach (CODAC). 
2. two key theoretical results ensure that the method works as expected. Theorem 3.6 shows that CODAC will learn quantile estimates that lower bound the quantiles of the true return distribution. Intuitively, a lower bound is preserved when integrating over this distribution, such as when computing CVaR objectives. Theorem 3.8 shows gap-expansion, in which the degree of conservativeness is less for in-distributions actions than it is for out-of-distribution actions.
3. A practical actor-critic algorithm is developed based on the min-max formulation of the original penalized objective in CQL, but in the distributional RL setting. This algorithm is evaluated on a variety of robotics tasks, to see whether CODAC can learn conservative quantiles while being able to take risk into account.
main_review: originality: I believe the paper has sufficient novelty for NeurIPS, being a novel combination of distributional RL and conservative Q-learning. This novel contribution also solves an important problem of learning risk-sensitive policies from offline data, which is a complex but important problem. existing methods are well discussed, and it is clear how this method differs in problem setting from CQL and work on distributional imitation learning. 

clarity: overall, the paper is very well written and the motivation for studying this problem is well argued. some minor details below:

1. line 76 "paratemerizing" --> "parameterizing"
2. the notation in section 2, from line 92 is very dense, and some notation isn't well defined: is D(s, a) the data set restricted to s, a pairs? 
3. it could really help readers to provide a simple motivating example early on (before section 2, or at least before going into section 3) for incorporating distributions of the return/risk in the offline RL setting, along with visualizations of what the proposed method CODAC is designed to do (e.g. manage risk while learning offline).

quality: the theoretical results are intuitive, appealing and easy to understand, particularly in the context of existing algorithms in offline RL and distributional RL. the experimental evaluation is quite thorough, testing the ability of CODAC to learn both risk-neutral and risk-sensitive policies, on data-sets of various quality, and baselines appear to be well chosen and comprehensive (see limitations for comments). 

significance: offline reinforcement learning and risk-sensitive distributional RL are both very active and emerging topics in reinforcement learning. based on this observation, I believe this paper will have a very high impact in the research community. 
limitations_and_societal_impact: tightness of lower bounds: original work on CQL shows that the learned bounds are reasonably tight lower bounds on the Q-values. does a similar result hold for the quantiles as well? 

visualization and interpretation of results: the results are quite hard to interpreted for the stochastic domains from tables alone. it is also hard to understand what kind of behavior the agent is learning on the stochastic domains. is it possible to visualize the learned behaviors similar to the first two domains? 

failure modes: it is also hard to understand why the algorithm fails (e.g. risk sensitive cheetah task?). can any commentary be provided to discuss why the algorithm fails here and what could be done in the future to address this?

choice of risk measures: why is CVaR the approach used here, given it is not novel? it would be nice to provide further reasoning. existing work on distributional RL often allows other risk measures to be defined through other transforms of tau (e.g. Wang transform, etc.) Can the current work be applied for other such risk measures? Are most/all well studied risk measures quantile-based or are some choices not amenable to the analysis in this paper? it doesn't seem that exponential utilities or certainty-equivalents are possible for example, but I may be wrong. 

overall: I believe this is a strong paper with non-trivial contributions. the overall approach makes sense and the theoretical results are quite intuitive. furthermore, the limitations of the approach (tuning alpha, etc) are anticipated but they are discussed in the conclusion, where it is noted that such limitations are not exclusive to this work, so I do not see this as an issue.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: At its core, the paper introduces an algorithm called conservative distributional evaluation (CDE). In the tabular settings, the CDE algorithms theoretically guarantees to converge to a fixed point whose quantile function lower bounds that of the true return distribution, also its fixed point is more conservative for rare actions in the dataset (the property that is called gap-expanding). These are desired properties for offline scenarios with full coverage (see the comments bellow). CDE is then used in an actor-critic framework, which is called Conservative Offline Distributional Actor Critic (CODAC). This gets a min-max objective: the inner loop chooses the current policy to maximize the CDE objective, and the outer loop minimizes the CDE objective for this policy. Some experimental results are followed to support the theory. 
main_review: -- The **far too strong** assumption that the dataset $\mathcal{D}$ has full coverage is ignored to discuss (Assumption 3.1). In real-world offline settings (such problem as in healthcare) we almost always deal with situations where only a tiny fraction of the state-action space is covered by $\mathcal{D}$. When $\mathcal{D}$ has full coverage, the problem is simply an **off-policy** learning problem (rather than offline) with nearly complete exploration. This is identical to the *online* case where there is an unknown behavioural policy for each experienced trajectory and where all state-action pairs are seen frequently enough. I found this case very misleading to be called **offline**. 

-- The penalty term in (5) is sampled-based. In particular, $F^{-1}_{Z}(\tau)$ with uniform $\tau$ may simply get a large value by some chance; hence, incurring a large penalty even when $Z$ is not distributed close to uniform. This may cause an unnecessary instability. Why not using other measures of uncertainty which are deterministic function of distribution’s shape, such as moments of $Z$ or entropy (as entropy is a super-exponential lower-bound for all even moments). Such measures can be computed directly from the quantile values; the regularization term in (5) will then only contain an expectation over $\mathcal{D}$.

-- Proof of Lemma 3.4 (in the Appendix) -> the first term in expressions after line 518 -> expectation over $\tau$ should introduce an extra constant coming from the uniform distribution when writing the integral over $\tau$, which is missing. Further, $\alpha$ also is missing in the second line. (They shouldn’t change the result when the derivative is set to zero.)

-- Theorem 3.6 -> left-hand-side of the inequality must also be $F^{-1}$. 

-- Theorem 3.6 -> A return distribution must sum to one, regardless. Intuitively, if the learned distribution lower bounds the original distribution in all the quantiles it cannot sum to one (unless at least in one (discrete) quantile it overestimates with significant amount such that it compensates for the rest). This theorem sounds counter-intuitive. Am I missing something? (This needs clarification in the text as well).  

-- After equation (6) -> It is not clear where $\mu$ comes from and how sensitive the results are to this policy. A discussion here is needed. 


Other comments:

-- Line 100: In general, $[V_{min}, V_{max}] \subseteq [\frac{R_{min}}{1-\gamma}, \frac{R_{max}}{1-\gamma}]$ instead of being equal. For instance, in a goal-seeking MDP, $[V_{min}, V_{max}] = [R_{min}, R_{max}]$.

-- Line 103 -> what do you mean by “its correspondence to policy”? If it refers to the optimal polic/value, then fix the last expression by adding asterisk. 

-- Line 107 -> definition of $\hat{\pi}_{\beta}$: right hand side should be conditioned on (s,a). 

-- Equation (5) -> what is $U$? this should be uniform distribution (?), yet in the previous section you used Uniform(.). If yes, it is better to be consistent. 

-- Careful proofread is required. I saw at least a couple of sentences with no full-stop at the end.
limitations_and_societal_impact: This is a mostly theoretical work. Not at a level to directly cause societal impact.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper introduces a general purpose offline distributional reinforcement learning algorithm CODAC. They prove that CODAC obtains conservative estimates of the return quantile, which translate into lower bounds on values. They provide sound theoretical and thorough empirical analysis for the proposed method.
main_review: The paper is well organized and well written in general. The results presented in the paper looks interesting and has potential of bringing good impact in risk-averse RL.

What is x in equation (2). Is there an intuitive way to interpret what's happening here?

The quantile based formulation is varsatile and is a good addition for risk based optimization. One can get VaR, CVaR and so on simply by updating the quantile. CVaR is known to be not time-consistent. Or in other words, a recursive decomposition does not hold and the policy is history dependent for CVaR optimization in MDPs, unless the state space is augmented [1]. But this paper is providing a recursive formulation without augmenting the state space, even for CVaR. An analytical study in this matter would be greatly appreciated.

The theoretical results presented in the paper is rigorous and looks good from a high level. Though I did not get the chance to verify the correctness of all the proofs presented in the paper.

In description of figure 1, the inital states are not represented by 'blue' color, as stated. May be green?

The empirical study presented in the paper is detailed and well thought. It does provide a comprehensive overview on the behavior of CODAC compared to other baseline methods.


[1] Risk-Sensitive and Robust Decision-Making: a CVaR Optimization Approach. Yinlam Chow, Aviv Tamar, Shie Mannor, Marco Pavone.
limitations_and_societal_impact: Limitations and impacts are addressed.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper adapts distributional RL to the offline setting by penalizing the predicted quantiles of the return for out-of-distribution actions. It proposes a CODAC algorithm for this purpose, proves the convergence of the proposed alg in tabular cases, and shows that it can learn risk-averse policies from offline data collected purely from risk-neutral policies. 
main_review: **Strengths**: 
-	The use of distributional RL for offline learning is a promising direction and this paper explores just that. I think such a combination makes an important bit to further improve the performance of offline RL which is becoming a big deal for RL in practical settings. 
-	The experimental results on risk-sensitive and risk-neutral policies are promising though expected 
**Weaknesses**:
-	The approach proposed is quite simple and straightforward without much technical innovation. For example, CODAC is a direct combination of CQL and QR-DQN to learn conservative quantiles of the return distribution. 
-	Some parts of the paper need clearer writing  (more below) 

**Comments and questions**: 
-	I think in a paragraph from lines 22-30 when discussing distributional RL, the paper lacks relevant literature on using moment matching (instead of quantile regression as most DRL methods) for DRL (Nguyen-Tang et al AAAI’21, “Distributional Reinforcement Learning via Moment Matching”). I think this should be properly discussed when talking about various approaches to DRL that have been developed so far, even though the present paper still uses quantile regression instead of moment matching. 
-	More explanation is needed for Eq (5). For example, what is the meaning of the cost $c_0(s,a)$? (e.g., to quantify out-of-distribution actions)
-	The use of $s’$ and $a’$ when defining $\hat{\pi}_{\beta}$ at line 107 might cause confusion as $\mathcal{D}$ contains $(s,a,r,s’)$. 
-	This paper is about deriving a conservative estimate of the quantiles of the return from offline data where the conservativeness is for penalizing out-of-distribution actions. In the paper, they define OOD actions as those are not drawn from \hat{\pi}_{\beta}(.|s) (line 109) but in Assumption 3.1. they assume that \hat{\pi}_{\beta}(a|s) > 0, i.e., there is no OOD actions. Thus, what is the merit of the theoretical result presented in the paper? 
limitations_and_societal_impact: The authors have adequately addressed the limitations and social impact of their work.
review_title: null

Meta-review: All reviewers agree that this is a good paper with sufficient novelty and relevance to the NeurIPS community. Reviewers had several questions, which were adequately answered by the authors. Therefore, I would recommend the *acceptance* of this paper.

I encourage the authors to consider the feedback by reviewers in their revision of the paper, including the followings:

- discuss the tightness of the lower bound
- discuss what other risk measures CODAC can be applied to
- discuss that CVaR's not being time-consistent, and how you avoid it.
- cite and compare with the literature on moment matching for DRL
- Fix the typos, including the one in Theorem 3.6
Decision: Accept (Poster)