Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: 

This paper extends the stochastic latent actor-critic method to safe reinforcement learning by a safety critic. The experiments show the competitive performance of the proposed methods over existing approaches on benchmark pixel input tasks. 
strength_and_weaknesses: 

Strength:

The presentation of the paper is very clear. 

The experimental results are also sufficient to show the effectiveness of the proposed methods.



Weakness:

The main weakness is that the proposed method is not novel. The author claimed that 3 contributions of the method. They are minor to me. In my view, this paper extends SLAC to safe RL case by introducing the cost mechanism to the prediction target and discounted cost return, respectively. None of them are novel.


clarity,_quality,_novelty_and_reproducibility: 

The method of the paper is very clear. 

this paper is a combination of existing methods to me and is not novel.

The author also presents open-source code which is good for reproducibility.


summary_of_the_review: 

Generally, this paper is a combination of existing methods to me and is not novel. The paper takes a lot of content on introducing existing methods (constrained POMDP, SLAC). Using the Lagrange multiplier is also not novel for safe RL [1,2].





[1] Geibel, P. and Wysotzki, F. Risk-sensitive reinforcement learning applied to control under constraints. CoRR, abs/1109.2147, 2011. U

[2] Stooke A, Achiam J, Abbeel P. Responsive safety in reinforcement learning by pid lagrangian methods[C]//International Conference on Machine Learning. PMLR, 2020: 9133-9143.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes an algorithm for solving safety-constrained decision problems with unknown dynamics that are also partially observable, which is one of the most difficult type of decision problems. The approach is to learn a safety-constrained POMDP model that has latent state, and also includes a safety critic. All components of the model are estimated simultaneously from experienced transitions, by modifying the training objective from only maximizing cumulative discounted reward to also maintain cumulative costs, in the sense of safety violations, under a specified threshold. Empirical evaluation in several environments shows computational advantages with respect to existing baseline algorithms.
strength_and_weaknesses: The paper attempts to solve one of the most difficult decision problems possible, and achieves favorable empirical results in comparison to existing algorithms. Moreover, it does it by learning a latent state model in the form of a POMDP, which makes the paper very suitable for this conference. Although the verification is in simulation and on relatively simple navigation problems, the dimensionality of the observation space is high, because observations are images, so solving this kind of problems is far from trivial, especially in the partially observable setting.
clarity,_quality,_novelty_and_reproducibility: The contribution of the paper appears to be novel and original. An anonymized GitHub repository is provided for the purposes of reproducibility. The paper is generally well written, but the notation is quite confusing, and can hardly be understood without referring to other papers. The POMDP model includes several kinds of latent variables z, distinguished by superscripts, which apparently do not mean powers, but something else. It doesn't help that the function p_\psi is overloaded in five different ways in Equation 2, so it is hard to tell which parameter does what. Similarly, in Equations 4 and 5, it is not at all clear what M is - probably not the model, because the parameters \psi appear to be defining the model. Is it perhaps a data set of samples? But then, D is used for that in Algorithm 1. 
summary_of_the_review: This paper appears to be a good contribution to the field of learning and solving POMDP problems, with the further complication of safety constraints. The explanations of the algorithm and the mathematical notation can be improved.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes an approach for safe reinforcement learning from pixel observations. This is framed as a POMDP problem where the agent receives separate cost signals from the environment in addition to rewards. At a high-level, this work adapts the Stochastic Latent Actor-Critic framework (which learns a latent representation along with a dynamics model in the latent space) to handle safety constraints. Particularly, soft constraints since the agent can violate them at every time step for a cost. Thus the resulting learned policy becomes "safe" by minimizing some notion of expected cumulative cost in addition to maximizing the expected return. In this work, safety is considered through the addition of a separate safety critic in the policy objective that predicts the expected cost return given a latent state and an action. 
strength_and_weaknesses: The trained policy is empirically evaluated on a set of challenging, fairly-recent safety-gym environments (Ray et al. 2019). The method seems to learn safe, competitive policies in most of the environments (with DoggoGoal1and PointPush as exceptions). However, it is difficult to understand whether these policies are any good. A potential quick fix here is to plot a line similar to the budget line on the cost return plots but for the policy. This line would represent the reward return for the worst policy that still solves the problem. Later in the paper, there is a discussion on computational requirements where the running time of the method is compared to that of LAMBDA (As et al. 2022). However, it is not clear to the reader whether or not this difference was due to implementation details or computational complexity.  
clarity,_quality,_novelty_and_reproducibility: The paper is well-written. Furthermore, the presented modifications to the SLAC framework onto the safety-constrained setting are straightforward, making the method practical. Equations (2) and (3) along with the preceding paragraph could be more explicit. For example, what do the superscripts of the latent variables z_i^j mean?
summary_of_the_review: This paper presents a straightforward set of modifications to the SLAC framework onto the safety-constrained setting. This would make implementation and benchmarking fairly simple since the environments used are also standard in this setting.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper extends SLAC [1] to make it applicable to constrained POMDPs. The proposed approach combines model-free RL with a latent representation that is learned by a generative model. Constraints are satisfied by learning a constraint critic and optimizing a Lagrangian, which boils down to extending the regular SAC objective by another critic component that is weighted by a Lagrange multiplier. The method performs on par with the state-of-the-art on the safety-gym benchmark

---

[1] Stochastic Latent Actor-Critic: Deep Reinforcement Learning with a Latent Variable Model, https://arxiv.org/pdf/1907.00953.pdf
strength_and_weaknesses: **strengths**

* The paper proposes a method for safe RL in POMDPs which is competitive with the state-of-the-art.
* The method mostly consists of previously established work that is combined in a new way. I see this as a strength.

**weaknesses**

* The performance gap to previous work is not clear. As such, the paper is not an improvement over the current state of this setup. This is my main reason for the score I've given to the paper.
clarity,_quality,_novelty_and_reproducibility: * The authors point out that the pessimistic ensembling techniques used in LAMBDA [2] might be overly pessimistic. While I agree that this is a risk, I would like to point out that LAMBDA is a model-based algorithm where policy and critic are trained on model rollouts. These will have some error, and the pessimism might be necessary in that setup. SLAC, on the other hand uses model-free RL (with the exception of the latent representation) and pessimistic/optimistic approximations might simply not be necessary here. The performance gap between safe SLAC and LAMBDA appears quite small and I don't see a strong evidence to support the claim that "Safe SLAC is less pessimistic than LAMBDA".
* Regarding the performance gap between safe SLAC and LAMBDA, I'm not sure it is a good decision to use Figure 3. It is somewhat difficult to read due to the outlier caused by DoggoGoal1. Figure 5 is easier to read in comparison, though at the cost of removing DoggoGoal1 from the benchmark. Perhaps the authors could give the reader some intuition about the policy's behavior. You are saying that the policy achieves a very high reward by acting unsafely. What does this mean here? Does it completely ignore constraints in a way that would lead to hazardous behavior in a real-life setup? The point is, either the policy's behavior is acceptable or it is not. If it is not, then it shouldn't appear in the comparison, as it makes it look like safe SLAC has a huge performance edge over LAMBDA. If it is, then you should point out why. If it is difficult to judge whether the policy behaves acceptably, I would advise using Figure 5 instead of Figure 3.
* The authors suggest that using the latent state instead of a history of interactions as input to the policy works better. I can completely see why this would be the case, though I think it should be pointed out that this strategy will not work on any given POMDP, since some POMDPs require the agent to learn to actively acquire new information (e.g. the heaven and hell task from [3]), which is only possible by working with interaction histories (or sufficient statistics thereof) or belief states, which the latent state does not necessarily substitute. I don't see this as a huge issue, since POMDPs come in a large variety, and using the latent state will be a viable approach for many of those, but this shortcoming should be addressed in the text.

---

[2] Constrained Policy Optimization via Bayesian World Models, https://arxiv.org/pdf/2201.09802.pdf

[3] Monte Carlo POMDPs, https://papers.nips.cc/paper/1999/file/299570476c6f0309545110c592b6a63b-Paper.pdf
summary_of_the_review: This paper is an iteration over SLAC, which modifies that work with a constraint critic, allowing application to constrained POMDPs. The method consists mostly of existing approaches combined in a new form, which has the benefit that each component has a history of related work supporting it. That being said, the method does not have a clear edge over the closest related work (LAMBDA). Thus, I see this paper as a new approach for attaining state-of-the-art performance in a benchmark for safe-RL. I believe this is still a contribution worthy of acceptance, since it will widen researchers' knowledge of which methods work and which do not.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper describes a new technique for safe RL in the context of partially observable environments. This is done by using a latent variable with a safety critic.

Strengths:
* The paper proposes a method for safe RL in POMDPs that is competitive with the state-of-the-art.
* The revised version of the paper is clear

Weaknesses:
* The approach is a combination of exiting techniques, but the combination and its application are novel
* The improvement with respect to previous techniques is okay but not significant

Overall, this is good work that advances the state of the art in safe partially observable RL.   This work will be of interest to the RL community.
justification_for_why_not_higher_score: * The approach is a combination of exiting techniques, but the combination and its application are novel
* The improvement with respect to previous techniques is okay but not significant
justification_for_why_not_lower_score: * The paper proposes a method for safe RL in POMDPs that is competitive with the state-of-the-art.
* The revised version of the paper is clear
* This is good work that advances the state of the art in safe partially observable RL.   This work will be of interest to the RL community.
Decision: Accept: poster