Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors propose SDAC, a safe RL method based on trust regions that uses TD($\lambda$) for the distributional critic and SAC-style updates.
They provide an elegant and memory-efficient way to approximate TD($\lambda$).
They propose to perform gradient integration in order to avoid getting stuck due to unmet constraints.
The proposed method was tested on Safety Gym with very nice results.

strength_and_weaknesses: ### Strengths

The paper involves technical elements that the authors explain in a clear way.
For instance, the introduction is super clear and provides a nice overview of the existing methods. In the background, the notations are clear and precise. The figures do a great job at facilitating the comprehension. The related work is exhaustive and well-organized.

The results on Safety Gym are quite nice, showing increasing or equal scores while reducing the number of violated constraints, which illustrates the empirical pertinence of the proposed method.

### Weaknesses

The paper is quite dense and I think this aspect could be improved. Some proofs might appear in the appendix instead.

On the title: use the term low-bias instead of low-biased?

On the RL method:
* It should be made clearer that the recursive formula (eq.7) is particularly useful to implement the method
* Fig.1 is nice, but a clarification as to why a projection is needed, i.e. different supports for distinct time-steps, would be welcome in the text 
* why the off-policy version of TRPO? This choice should be motivated.
* Why TRPO over PPO? PPO enforces a trust region by modifying the policy gradient, while TRPO requires backtracking. If I understand correctly this is because backtracking includes the linearized constraints as well? This choice could be better motivated in the text.
* What is the cost of gradient integration in practice?

On experiments:
* There are many hyperparameters for the resulting method, how hard is it to hyperoptimize? I think a discussion on that aspect would benefit the work.

clarity,_quality,_novelty_and_reproducibility: Clarity: clear but quite dense, some design choices are not well explained in the current state

Quality: good

Novelty: several elements are novel or their combination is novel (efficient TD($\lambda$), distributional critics, SAC-style surrogates, gradient integration).

Reproducibility: no code is available as far as I know, a computation cost analysis is missing
summary_of_the_review: This is a nice paper, featuring good writing and a genuine effort at making technical points digest to the reader.

The experiments are varied and convey a lot of information. They do a good job at validating the claims of the paper.

The proposed algorithm seems super heavy (k-distributional critics + backtracking + quadratic programming for gradient integration) but the computational cost is not quantified in this work, which I think is regrettable. 

In the current state, it is somehow hard to see how the community will use this work if no open-source implementation and/or not more comments on computational cost and sensitivity of the hyperparameters.

For all these reasons, though the proposed method is inventive and sound, I only borderline recommend for acceptance. There are low hanging fruits that the authors could choose to address to increase the potential impact of the work.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper concerns safe (deep) reinforcement learning. In particular, the authors present an approach that take the constrained MDP setting, where on top of a reward function, a cost function is given that conveniently reflects safety concerns. The learning goal is then to maximize the reward, while the cost needs to stay below a certain thresholed. In this work, the authors employ actually a set of cost functions. The actual approach presented here entails a distributional RL method that uses trust region optimization, like in other prior works. Here, however, the multipe constraints cause problems, and the authors use (1) a target distribution to estimate constraint biases, and (2) something called the ‘parameterization trick’ to develop a dedicated actor-critic techniqe. The approaches are evaluated by means of several environments from the Safety Gym, and further robotics experiments.

strength_and_weaknesses: ++ Important problem fitting for ICLR
++ Novel approach with multiple cost functions
++ Interesting actor-critic technique

— The paper is at parts very hard to follow
— Unclear utility of multiple cost functions in the Safety Gym environments

clarity,_quality,_novelty_and_reproducibility: The biggest weakness of the paper is, at least to me, the clarity of writing. While I believe that the approach is sensible (good quality), and that it is novel, I find the paper extremely hard to follow. I had to read the introduction at least three times to have an idea of the approach, and the tense technical description does not help me. I think with more structure and more intuition, the paper can definitely be improved. 

I have some specific questions, and I am happy to re-evaluate my judgement after the discussion.

— computing the target distribution in an off-policy setting sounds a bit like ‘safe policy improvement’ (SPI), coined by Thomas et al. This offline RL technique estimates a better policy than the data-collecting behavior policy from a limited amount of data. Some clarification or comparison there would be nice.

— Is the overall methods ‘safe’ or reliable, that is, are we guaranteed to get a final policy that is safe? Can anything be said about a trade-off between safety and rewards?

— In the first part of the experiments, I do not understand the multiple constraint setting. Are there even multiple constraints? And if not, why is the method performing so much better than state-of-the-art baselines, who are supposed to perform well on the single constraint setting. 

— Could it be an interesting experiment to shape a cost function that aims to capture the multiple constraints, and then show the differences in safety?

summary_of_the_review: Good approach, clarity can be improved.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The author develop an efficient trust region-based safe RL algorithm with multiple constraints.
1. introduce a TD(λ) target distribution combining multiple-step distributions to estimate constraints with low biases
2. present a memory-efficient method to approximate the TD(λ) target distribution using quantile distributions
2. to increase the efficiency of the trust region method, we propose novel surrogate functions which use the reparameterization trick with Q-functions
strength_and_weaknesses: Strength
1. introduce a TD(λ) target distribution combining multiple-step distributions to estimate constraints with low biases
2. present a memory-efficient method to approximate the TD(λ) target distribution using quantile distributions
3. propose novel surrogate functions which use the reparameterization trick with Q-functions to increase the efficiency of the trust region method,

Weaknesses

1. I don't understand what figure 1 is trying to say, could the author be more clear?
2. The authors believe that Figure 3 can be clearly represented, "the result can be interpreted as excellent", but from this figure, I have no way to see the cost limit, as well as fine metrics, intelligent roughly, the method of this paper is roughly put in the upper left, but Point Goal and Car Goal seem to overlap in the upper left.
3. I carefully read through the experimental results of the article, and in Appendix D.1, Figure 8, in Point Goal and Point Button, WCSAC is significantly better than the SDAC proposed in this paper in terms of score, cost rate, and Total CVs, and this seems to be consistent with the Safe RL perspective. From the Safe RL perspective, this seems to be in great conflict with what is discussed in the abstract, "From extensive experiments, the proposed method shows minimal constraint violations while achieving high returns compared to existing safe RL methods." 
    I suggest that the authors should rework Figure 3 or put a more refined result, Figure 8, in the text, because the ambiguous and unrefined Figure 3 leads to a misunderstanding of the performance of the algorithm.
4. In Figure 8, why is there if a big difference between cvpo in Button's task and Goal's task? cvpo's reward in Goal's task is even negative, while it does have a considerable reward in Button's task.
5. As far as I know, \alpha is the discount factor corresponding to cost, so why the comparison in Figure 9 and Figure 10 lack CPO and CVPO.
Based on the above doubts, I have doubts about the contradictory claims in this article and the results of the experiments, and the authors don't seem to disclose their code in the appendix material. Also based on the list of hyperparameters provided by the authors (and no other baseline details of the experimental parameters are provided), I think the reproducibility of the experimental results of this work is very low.

clarity,_quality,_novelty_and_reproducibility: Clarity, Quality, Novelty, and Reproducibility can be seen above.
summary_of_the_review: I have many doubts about the experimental results and opinions in this article, and I hope the authors can explain them in detail.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a method for safe RL or more general constrained RL. Constraints are modelled by the mean-std method. As RL algorithm the authors use TRPO and use a multi-step distributional critic.
Apart from combining these methods, the paper seems to have three independent contributions: 
1. The authors extend the distributional RL approach based on quantile regression and Wasserstein distance to the multistep setting.
2. Then replace the advantage function with the q-function in estimating the trust region.
3. Finally, they propose a gradient-based method to find starting policies in the trust region.
Experiments are conducted on the safety Gym environment and compare to other safe RL approaches w.r.t. constraint violations and performance.
strength_and_weaknesses: strong points:
* The paper combines and extends several SOTA techniques.
* Results indicate a superior trade-off between constraint violations and performance

weak points:
* The paper's contributions are not easily distinguishable from the combined basis methods.
* it is unclear how the contributions relate to each other. Some of them seem disconnected and not really specific to safe RL. In particular, using a distributional critic seems valid for any AC method. 
* Some of the methods are not well-motivated. For example, I did not get why replacing the advantage function with the q-function is beneficial. 

After reading the author's comments and the revised paper, explaining why these methods address open problems of constrained RL with trust regions is more precise. 

clarity,_quality,_novelty_and_reproducibility: In general, the paper is clear. However, it is advisable to have a look at the papers the proposed SDAC method is built on top of.
The technical quality is high, and the authors seem to have the ambition to use sota-methods for any part of their method.
The paper does provide code and provides hyperparameters and settings in the appendix. Thus, reproducing the results should be no major problem,
The novelty of the approach is kind of hard to assess. The authors have engineered a well-performing safe RL method. However, the contributions do not seem to follow a single general idea but are tailor-made to make the combination to trust-region-based safe RL and distributional methods work.
summary_of_the_review: The paper proposes a well-performing safe RL-method which combines several recent approaches such as trust-region-based safe RL and distributional critics. Experimental results are also quite convincing. 
On the other hand, some of the contributions are not well-motivated. For example, the authors claim that it is a good idea to use q-values instead of the advantage function. However,  as the advantage function is the q-function minus the value function and it was introduced to lower the variance, it is kind of unclear why this is done. Later, they add entropy regularization which makes more sense, but it is not very exciting as it is a well-known technique. 
In general, many improvements seem to be a relatively straightforward combination of existing techniques and a better explanation on why the proposed extension yield a major technical contribution would be desireable.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposes a trust-region based method of safe reinforcement learning (RL), which integrates several state-of-the-art techniques and is empirically shown to outperform existing safe RL methods.  More specifically, the proposed approach 
1. learns distributional TD(\lambda) to balance the bias and variance of the estimated return, where a projection technique is used to keep the number of quantile points small,
2. uses Q-functions rather than advantage functions for SAC-like surrogates, and
3. uses gradient integration to deal with multiple constraint.
The experiments are conducted on safety gym environments with single constraint to compare against existing trust-region based safe RL methods that can deal with single constraints as well as locomotion tasks with multiple constraints to highlight the advantages of proposed method over standard (non-safe) RL that replaces the reward with the weighted sum of reward and safety.  Moreover, ablation study shows the effectiveness of each technique in the proposed approach.

The main strength of the paper is in the integration of state-of-the-art techniques in a way that it outperforms existing safe RL methods.  In particular, the advantages of the proposed approach over existing trust-region based safe RL methods are clearly demonstrated in the experiments.  Given the importance of safety in RL, the method that achieves the state-of-the-art performance is a valuable contribution.

The major weakness in the original submission was clarity (particularly introduction and experimental results), which however has been resolved satisfactory in the latest version of the paper.

One of the remaining weaknesses is in its limited novelty.  The distributional TD(\lambda) has also been studied in the prior work, as is also discussed in the paper.  While the projection technique may be new in the context of learning quantile-based distributional TD(\lambda), it is not particularly novel.  Also, the advantage of TD(\lambda) over Monte Carlo return is not so significant from the ablation study.  Q-functions are certainly used in many context of RL, while it is good to know that it is a particularly good choice in safe RL.  Gradient integration has some novelty, and its effectiveness over a naive method (with respect to the number of steps needed to satisfy all constraints) is clearly demonstrated, but the contribution by itself is limited.

Overall, weaknesses slightly outweigh strengths for this paper.  A suggestion for improvement is to empirically compare per-step computational cost of the proposed approach and baselines.  While the computational cost analysis was added in Appendix E, quantitative differences are unclear.
justification_for_why_not_higher_score: The proposed method integrates several state-of-the-art techniques, but each technique has limited novelty.
Decision: Reject