Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: #### Summary:
The paper focuses on soft-constrained RL techniques and proposes a meta-gradient approach for the same. It first extends the RCPO (Tessler et al)  algorithm using the methodology of DDPG (Lillicarp et al) to propose an off-policy version of RCPO (called RC-D4PG). The main contribution of the work is the proposal of two new meta-gradients based algorithms for the soft-constrained RL problem that are able to find a good trade-off between constraint violation and maximizing returns. The first proposed algorithm - Meta-L - is based on a meta-learning based adaptive update rule for the Lagrange multiplier's learning rate. The second algorithm is based on similar principles but instead focuses on adapting the reward-shaping update in a meta manner. The author's show the empirical evidence of their method's strengths on a bunch of continuous control based simulator tasks. 

#### Strengths:

- **Setting**: The paper shifts focus on the soft-constrained RL (a relaxed CMDP) setting and provide good motivation to work on these setting rather than sticking to the much harder CMDP setting. This makes sense especially for the choice of problems the authors are interested in such as saving energy costs, find some trade-off between two metrics, etc.

- **Method**: One of the biggest problems with the Lagrange Optimization-based CMDP algorithms is that the optimization of the Lagrange multiplier is tricky (and most times very brittle). I like the approach of the authors that aims to adaptively modify the rate of change of the Lagrange multiplier, and I believe that it is a step forward in the right direction.

- **Empirical results**: The authors have strong empirical results for their methods, and it seems that the meta-gradient based approach is able to find the trade-off for the more successfully when compared to the baselines. 

- **Analysis**: The empirical analysis of the behavior of the proposed algorithms has been insightful as a reader. I appreciate the extensive experiments and particularly enjoyed the section Algorithm behavior analysis.
 

#### Weakness:
 
- **Reproducibitlity**: There is no mention of code release. Though, they provide the architectural details in the appendix it is difficult to take the author's claim at face-value when the implementations of the Deep-RL-based methods aren't released. 

- **Outer-losses**: The authors propose and test a variety of outer-loss and pick the best one here. I can see why the authors chose this approach, however, from an outsider's perspective, this means for any new application where this algorithm needs to be deployed, a lot of effort and computing needs to go first in finding the appropriate outer loss (on top of hyper-parameter optimization!). It'll be great to have some insights into the choice of outer-loss, other than the hyper-parameter based search that is currently being used.  

- **Solution quality**: No comments have ever been mentioned about the quality of the solution reached. It'll be nice to have some guarantees associated with the solution quality in this case as the setting is motivated by safety and constraint violation.  


#### Questions for rebuttal:

- Why is the Meta-parameter update for MetaL (in Algorithm 1, Line 17) different from the derived gradient update in Theorem 1 ?
- Does updating the Lagrange multiplier adaptively makes the initial learning rate selection process more cumbersome or fragile? 
- Why did the authors choose the off-policy extension of RCPO to build upon, when the original RCPO was purely online? I'm curious to know what breaks (if at all anything breaks) with the original online formulation.  

review_title: Recommendation to Accept

final_score: rating: 7: Good paper, accept
review_content: The paper presents two soft-constrained rl approaches built on top of D4PG. Specifically, they use meta gradients for the lagrange multiplier learning rate (MetaL), and use meta gradients for reward shaping (MeSH).

I found the paper to very clearly written, the main algorithm MetaL is clearly presented, and the results are fairly conclusive: the meta gradient approach proposed in the paper works better than the tested baselines. The introduction motivates the different real-world problems very nicely, e.g., hard vs. soft constraints. As someone with a lot of deep RL experience, but not a lot of constrained RL experience, I found the authors did a very good job at explaining all the relevant background. I also really appreciated the detailed experimental analysis of the approach at the end of 6.1 – it highlights exactly why the method works well. 

One critique I do have, is that it would be great to have the intuition for the MeSH update in the main paper. Otherwise, since the results indicate that it performs worse than MetaL across the board – perhaps it would be best to relegate the method to the appendix and change the presentation of the paper, e.g., change “we propose two meta gradient methods”  etc… I feel like the paper would be stronger without MeSH as it takes away from the overall message. 

Another issue I have is that 3 seeds might not be enough to make any meaningful conclusions. More seeds would be needed to make any statistical comparison between metal and Rs-0.1 in table 1. Nevertheless, as highlighted RS-0.1 fails at humanoid – which makes sense – you would need to tune the penalty parameter for each domain (although 0.1 actually works quite well on ¾ domains). Which is why meta gradient approaches make sense. 

Overall, assuming the authors add extra seeds and perform the statistical significance testing – I think this paper would have a large impact at ICLR.

Small notes:

On the presentation side of the results, I find figures 2 and 3 to be hard to interpret at a glance. It would be better to compare against the best baseline – instead of ~6 of them. 

One final note, \bar \lambda = 1000 corresponds to the upper bound of the reward. Unless the algos are consistently achieving this upper bound, \bar \lambda seems very high. Ideally, a bunch of lambdas should be tested. This feels arbitrary.

review_title: well written paper - could have a large impact

final_score: rating: 7: Good paper, accept
review_content: This paper proposes a simple approach to soft-constrained deep RL optimization using the unconstrained Lagrangian, by meta-learning the learning rate of the Lagrange multiplier. The authors include extensive evaluation, comparisons and ablations on 4 mujoco domains that establish the efficacy of the approach and provide insight into why it works better. 

Pros:
1. Proposes a solution to the important problem of soft constraint optimization in deep RL, which is a challenge to be addressed for real world deployment. This work casts the unconstrained Lagrangian problem in the meta-learning framework, where the inner update adapts the learning rate for the Lagrange multiplier, and the outer loop optimizes the overall loss which takes into account both reward and constraint penalty. This adaptable learning rate prevents the optimizer from fitting too soon to the constraints and enables it to find a solution that can give higher overall performance (sum of reward and penalty). 

2. The extensive evaluation includes comparison to prior approaches for constraint optimization in deep RL (soft-constraint with fixed Lagrangian multipliers and hard-constraint with learnable multipliers), tests with varying degree of constraint violations in the environment, ablations with variants of the outer loss across 4 mujoco domains in the Real-world RL suite. The authors also include an analysis of how the and Lagrange multiplier and its learning rate evolve in a setting with high number of safety violations, to show the benefit of using the adaptable learning rate.


Cons : 
1. The idea of adapting the learning rate in the inner loop of a meta-learning algorithm is not novel, and is often used to enhance performance. With that said, this work convincingly shows that using this simple idea is effective for constrained optimization in the meta-learning framework. 

2. The paper introduces another variant of their main approach, but doesn't discuss it adequately in the main paper, and it's unclear why it does worse than the proposed approach.

3. The paper could be further strengthened by experiments on real world domains, where soft-constrained optimization is a critical challenge.
 
review_title: Simple approach to soft-constrained deep RL optimization with exhaustive, convincing evaluation

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper addresses the soft constraints problem in RL. The problem is formulated as a Lagrangian optimization following Tesslaer et al. (2018) where the constraint is treated as a penalty in the reward. The base solution to the Lagrangian optimization is D4PG. 
To adapt the learning rate of the Lagrangian multiplier and find a good trade-off between reward and penalty, the paper customizes the usage of meta-gradient method (Xu et al., 2018) to the problem in this paper. Two variants, MetaL and MeSH are designed.

The problem being addressed is an important topic in RL, the method proposed seems to be novel (but the intuition behind it is not clear), while the empirical evaluation is not convincing to me.

Strengths:
1) Addressing an important problem in RL, potentially could draw attention to the RL community
2) The paper provides a good motivation of addressing soft constraints in RL
3) Proposed method is somewhat novel

Weaknesses:
1) In the paper the motivation of using meta-gradient to solve the formulated Lagrangian optimization is only explained once at the beginning of Page 4 "Our intuition is that a learning rate gradient that takes into account the overall task objective and constraint thresholds will lead to improved overall performance." However it is clear to me what explicitly do you want to achieve? Are you trying to find the "ground truth" $\bar{\lambda}$ (i.e., 1000 in the experiments)? Does not seem to be the case; Do you want to somehow learn a "robust" policy that works well for all $\bar{\lambda}$ values? If so it is expected that the authors would show the empirical results on different $\bar{\lambda}$ values, and the proposed method works well in all of them.
2) Following 1), the paper evaluates different forms of outer loss and show that using a critic-only outer loss yields the best performance. What is the intuition behind this? It would be good that the authors can have a more in-depth discussion on this.
3) In the empirical evaluations, the paper defines a "penalized return". This is very un-intuitive -- in reality there is not always such quantized trade-off between reward and penalty of constraint violation. If there is, then one could directly add that to the objective. It would be more interesting to see, for example, given the same reward value, the proposed method always outperforms the baselines in penalty; or the other way round. The "penalized return" metric is therefore un-convincing to me.
4) Assuming the penalized return metric makes sense. From Figure 3 it appears that the performances of the baselines sometimes are very close to the proposed method. What puzzles me is that the performances of the baselines vary dramatically across domains. Can the authors elaborate more on why this happens?

Questions: See weaknesses points 1) 2) and 4)

Less important points:
1) In table 1 it seems that overall performance of RS-D4PG monotonically increases w.r.t. $\lambda$ values. I am curious to see what happens when $\lambda$ is even smaller.
2) Page 3, Line 2, $J_{obj}^{\pi}(\theta)$ -> $\tau$ and $\eta$ are missing in the bracket
3) Line 4 at paragraph D4PG: $Q_T(s, ...)$ -> s'

review_title: Proposed methods is novel, evaluation is not considerably convincing (edit)

Meta-review: The paper looks at the soft-constrained RL techniques and proposes a meta-gradient approach.
One of the biggest problems with the Lagrange Optimization-based CMDP algorithms is that the optimization of the Lagrange multiplier is tricky
The proposed solution and empirical results have promise. The reviewers broadly agree on their evaluation and the major concerns on comprehension, additional experiments and as well as comparison with baselines have been addressed in the rebuttal. 

- Convergence rate and quality of fixed point reached.
The authors mention convergence to local optima but omit the quality of this solution from perspective of safety. It would be useful to include a discussion on the topic, with potential references to concurrent work. 
Other relevant and concurrent papers to potentially take note of:
- Risk-Averse Offline Reinforcement Learning (https://openreview.net/forum?id=TBIzh9b5eaz)
- Distributional Reinforcement Learning for Risk-Sensitive Policies (https://openreview.net/forum?id=19drPzGV691) 
- Conservative Safety Critics for Exploration (https://openreview.net/forum?id=iaO86DUuKi)

I would recommend acceptance of the paper based on empirical results, conditional on release of sufficiently documented and easy to use implementation. 
Given the fact that the main argument is empirical utility of the method, it would be limit the impact of this work if readers cannot readily build on this method. 
Decision: Accept (Poster)