Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: The submission investigates the risk averse objective in offline RL. Usually, the parametric uncertainty is the main source of worries in offline RL, and dealing with the stochastic uncertainty on top of it, in order to account for risk aversion, is very challenging. The authors expose clearly their method and algorithm, even though we sometimes would have liked a bit more argumentation on why this and why not that. Since no theoretical analysis is provided, the only validation is empirical. It is rather complete regarding both the settings and the domains. The results are quite impressive, in particular in the offline setting. For all these reasons, I recommend to accept the submission.

Please answer/address these comments in the rebuttal/final version:
* 1- It is unclear whether d^\beta is defined as the empirical distribution in the batch, or the true distribution. I believe that it would have been helpful in general to formalize more the distributions definitions.
* 2- U(.) is not introduced.
* 3- First sentence of 3.2: It is frequent in offline RL to use stochastic policies in order to leverage the risk taken in the face of parametric uncertainty. It is therefore rather odd to state here that only deterministic policies are considered. Even more when we notice in 3.3 that the actual policy is stochastic, since b is sampled from the stochastic behavioral policy.
* 4- Please indicate the performance of the behavioural policy (average and cvar) in the experiments.
review_title: Official Blind Review

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary
The paper studies the problem of safe reinforcement learning, where we want to learn risk-averse policies in the offline setting. It proposes “Offline Risk Averse Actor Critic” (ORAAC) which performs competitively as risk-neural agent, and outperforms D4PG based baseline as a risk-averse agent on D4RL benchmark. The algorithm involves modifying the losses to learn risk-averse actor, distributional critic and a VAE-based imitative policy.


Reasons for the score:
I vote in favour of accepting the paper. The paper studies safety aspects of learning algorithms which are competitive as a risk-averse agent in offlineRL. I would strongly encourage the authors to include numbers for baselines like CQL/BEAR. Though not risk-averse, these algorithms are conservative by design and are strong baselines in offlineRL.


Strengths:
+ The problem is well motivated and the ideas are presented clearly. The authors perform extensive experiments, ablations and provide empirical evidence where ORAAC outperform baselines like OD4PG on D4RL benchmark tasks.
+ Though risk-averse by design, the learned policy is competitive as a risk-neutral agent.

Weaknesses:
- Comparison to SOTA baselines in the risk-neutral scenario are missing. Though CQL/BEAR are not risk-averse my design, these algorithms are competitive in the risk-neutral setting. It would be critical to know how ORAAC performs in comparison to such baselines.
- Discussion on the choice of hyperparameters and experiment design are under-explored. Elaborating on this in section 4 would significantly improve the readability of the paper.

[1] CQL : https://arxiv.org/abs/2006.04779
[2] BEAR : https://arxiv.org/abs/1906.00949

review_title: Study the problem of safety in offline RL, and propose a novel algorithm for learning risk-averse policy

final_score: rating: 5: Marginally below acceptance threshold
review_content: The authors propose an RL algorithm for learning risk-averse policies from offline data. Empirically, it is shown that it can outperform some existing risk-neutral approaches on a number of challenging robotic control tasks under risk-sensitive performance measures. Although the empirical results are encouraging, the theoretical properties of the proposed algorithm are unclear and therefore it is not clear how easy it can be implemented in other tasks.

Overall, the paper is easy to read and the presentation is clear.

The authors address a very important issue that is faced by RL practitioners. Learning risk-averse policies in a fully offline setting is inherently ill-posed and the burden is in incorporating sufficiently strong prior and bias in regularizing the learning system. Unfortunately the current paper falls short in this regard, where the only idea here is to use imitation learning in the form of autoencoders. So the main contribution in terms of new idea is rather incremental.

Furthermore, the theoretical properties of the proposed algorithm are largely unclear. For example, with respect to a fixed distribution for the start state $S_1$, the optimal CVaR policy for Eq.(1) can be non-stationary. The authors restrict the policy search to the space of stationary deterministic policies. Now suppose that the neural network is over-parameterized, and assuming arbitrarily large training set, which policy does the algorithm converge to? Does the algorithm converge?

Considering the 3 gradient updates in Algorithm 1, one wonders how to choose all those parameters to make it work, and whether a single step-size parameter is enough. In particular, $\lambda$ seems hard to choose. How sensitive are the empirical results with respect to the choice of these parameters? In the truly offline case, one presumably can only perform fine-tuning using a separate validation set, how should this be done here?


review_title: Encouraging empirical results. Unclear theoretical properties, hard to implement in new tasks.

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: ##########################################################################

Summary:

This paper proposes O-RAAC, an offline RL algorithm that minimizes the Conditional Value-at-Risk (CVaR) of the learned policy's return given  a dataset by a behavior policy. It learns a distributional critic, a VAE for imitation learning, and an actor that perturbs the VAE output to minimize the risk given by the distortion operator D (Here it uses CVaR).


Several experiments were performed to show the effectiveness of O-RAAC, both with a simple 1-d driving environment and with a modified version of D4RL dataset.


 
##########################################################################
Pros: 

The paper is well written, with comparisons with competing methods throughout. This makes the connection with those methods clear and easy to understand. The discussion around the design choices and the tradeoffs are especially insightful. (e.g. Huber loss over l1 or l2; VAE vs vanilla BC; etc.)

The contribution to risk averse offline reinforcement learning is novel, with the use of CVaR and the idea of a perturbed version of a imitation learning actor. With the assumption that all the risks is captured within the reward. 


##########################################################################

Suggestions:

The risk-neutral experiment in 4.3 was a bit confusing. I was not sure whether you measured the O-RAAC’s risk-neutral performance by setting alpha = 0, or you measured its performance with alpha=0.1, but using a risk-neutral metric. I think you did the latter after a few more reading. More clarification either in Table 2 or in 4.3 would be great. 

in 3.3, you said "...is a perturbation model that is optimized maximizing the actor loss (5)," This part was confusing as well, since one would normally minimize a loss. In this paper, actor loss = risk aversion, so maximizing risk aversion makes sense, but I wouldn't call it a loss. 

review_title: Recommendation to Accept

final_score: rating: 6: Marginally above acceptance threshold
review_content: ## Brief Summary
The authors developed an risk-averse RL algorithm with operates in offline contexts. The main idea consists in putting together the benefits from 3 previous works in the distributional, risk-averse and offline RL fields:
- the DDPG distributional extension (Barth-Maron et al., 2018), called D4PG
- the Implicit Quantile Network (Dabney et al., 2018), called IQN
- actor using an imitation learning component with perturbation to control the bootstrappinge error (Fujimoto et al. 2019)
The online version of the resulting algorithm, called RAAC, is then tested on a toy problem. 
Later, the full algorithm (O-RAAC) is tested on 3 MuJoCo tasks, using offline data from the the D4RL dataset (Fu et al., 2020).
The performance is analysed on both a risk-averse and risk-neutral point of view.

## Strong Points
- The state-of-the-art is correctly reported.
- The paper is clear and easy to follow.
- The experimental analysis is correct, complete and interesting:
    + baselines seem to be competitive: they are also augmented with the VAE layer
    + they  the analysis is sound and provides interesting insights.
- The proposed approach seems to outperform the baselines on the optimized measure: moreover, as noticed in also in previous work, introducing risk-aversion seems to be useful also to increase the expected return. 

## Weak Points

- The algorithm is obtained by composing existing approach: it is not clear to me whether there is a novel methodological contribution or not.
- The authors claim that their approach is general for any risk-measure, however, the experiments are conducted only for the CVaR case and with a specific percentile.
- The results in the appendix does not seem to match the ones in the main paper.

## Recommendation
My reccomendation is to accept the paper since it provides an interesting approach which puts together, in an original way, techniques from different areas of RL, even if it is not clear whether there is a novel methodological contribution. The shown results seems to be relevant, however, it is still not clear whether they are 

## Questions for the authors
- Performances seem to be worse in the supplementary material, can you explay why?
- Could you provide experiments with different quantiles for CVaR?
- Could you provide experiments with different risk-measures?
- Is it possible to evaluate in a qualitative way the behavior obtained by the agents in the learned task when using the risk-aversion or not?

## Additional Feedback
- It would be useful to state the dependecies of your approach in a clearer way.
review_title: An interesting approach that put together state-of-the-art techniques in order to obtain a strong risk-averse offline learner

Meta-review: This paper proposes O-RAAC, an offline RL algorithm that minimizes the Conditional Value-at-Risk (CVaR) of the learned policy's return given a dataset by a behavior policy. The reviews are generally positive with most agreeing that the paper presents interesting empirical results. 

The experiments are limited to simpler domains, and could be extended to include harder tasks from other continuous control domains. Some examples could be domains such as in Robosuite (http://robosuite.ai/) or Robogym (https://github.com/openai/robogym). These environments have higher dimensional systems with more clearer safety settings. 
Agreeably, asking for comparisons with unpublished results may be unfair, however, it would be recommended to authors to include additional comparisons with latest methods in Offline/Batch-RL, including the ones which don't guarantee risk, such as CQL, BRAC, CSC.

Further, The theoretical properties of the proposed algorithm are largely unclear. It would help to analyze the effect of both convergence rates, and fixed points, further what is the effect of addition of risk, does the algorithm converge to a suboptimal solution or get there slower. Finally empirical reporting of cumulative number of failures (discrete count) during training as well as during evaluation would be very useful to practitioners. 

Other relevant and concurrent papers to potentially take note of:
Distributional Reinforcement Learning for Risk-Sensitive Policies (https://openreview.net/forum?id=19drPzGV691)
Conservative Safety Critics for Exploration (https://openreview.net/forum?id=iaO86DUuKi)

I would recommend acceptance of the paper. I would strong encourage release of sufficiently documented and easy to use implementation.  Given the fact that the main argument is empirical utility of the method, it would be limit the impact of this work if readers cannot readily build on O-RAAC. 

Decision: Accept (Poster)