Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: In this paper, the authors explored the duality between density function and value function in the setting of density constrained RL. Based on this, the author proposed a new safe constrained policy optimization algorithm by jointing optimizing policy and lagrangian multipliers of the density constraints, with an extra effort to estimate the stationary state density of current policy $\pi$ using kernel density estimation. The empirical results demonstrate that the proposed algorithm  is effective in several mujoco benchmark and  autonomous electric vehicle controlling.  I believe exploring the duality between density function and value function in the  setting in density constrained reinforcement learning is new, and may be helpful in safe reinforcement learning. 

Followings are my detailed questions and comments:

- I have a major concern about estimating the marginal (or stationary) state function of the policy $\pi$. The author use kernel density estimation to estimate $\rho(s)$, which I think deserve more discussion in this paper, and at least some empirical experiments should be conducted to verify its accuracy. Moreover, since the estimation requires to use kernels, it would be good to see the empirical results of choices of kernels and bandwidth of the kernel. 

- All the derivation and the stop criterion is based on infinite number of samples, which will not guarantee that the constrain is satisfied using finite mc samples to estimate (since you don't have the transition, you can only use samples to estimate). Will there be any guarantee that the exact estimation of density and constrain is upper bounded by some empirical estimation, such that we can use mc sampling to estimate the density or constraints?

- The duality between the density function and value function is well-known in the community, and has drawn great attention recently in the policy evaluation community [1, 2, 3, 4], which I think the authors should have some discussion on this in the related work,  and to see if the techniques can be used to estimate the density functions.

Overall I think the paper proposed a new perspective for density constrained rl, which I think is interesting and is publishable if my above concerns are well addressed. 

--------------------
I will update my scores if my questions are addressed:)

[1] Nachum, Ofir, et al. "Dualdice: Behavior-agnostic estimation of discounted stationary distribution corrections." Advances in Neural Information Processing Systems. 2019.

[2] Tang, Ziyang, et al. "Doubly robust bias reduction in infinite horizon off-policy estimation.", ICLR 2020.

[3] Uehara, Masatoshi, Jiawei Huang, and Nan Jiang. "Minimax weight and q-function learning for off-policy evaluation." arXiv preprint arXiv:1910.12809 (2019).

[4] Nachum, Ofir, and Bo Dai. "Reinforcement learning via fenchel-rockafellar duality." arXiv preprint arXiv:2001.01866 (2020).


review_title: Review for Density Constrained RL

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper studied density constrained reinforcement learning (DCRL), which compared with standard RL has constraints on the stationary state distribution. It proposes a primal-dual optimization method and proves its optimality. 


The theoretical analysis in this paper assumes $
P_a(s^\prime, s)$ is perfectly known whenever needed.  In fact, in this case formulations (3), (4), (5), (6) are simple deterministic convex (linear) optimization (regardless if density constraints are introduced or not).  The duality $J_d$ and $J_p$ duality follows directly from duality for convex programs.   The developed algorithm, Algorithm 1, is also simple application of (primal)-dual subgradient method for convex programs. If the D_s update and  \rho_s^\pi update are done exactly, i.e., assuming perfect knowledge of  P_a(s^\prime, s), then the theorems in this paper is just simple consequence of well-established convex optimization results. (I looked into the proof in the appendix. It seems the analysis indeed assumes perfect $P_a(s^\prime, s)$ in the analysis.)

Overall, I think the technical contribution and novelty of this are marginal.
review_title: Concerning the novelty

final_score: rating: 7: Good paper, accept
review_content: Summary:
This paper proposes to solve constrained RL through the lens of setting constraints on state density, rather than setting constraints on value functions. The main contributions are (1) a proof of the duality between the density function and Q-value function in constrained RL, and (2) a primal-dual algorithm, called DCRL, for solving the constrained density problem.

Recommendation:
This paper tackles an important problem, and presents a theoretically-grounded novel perspective on constrained RL. I'm leaning toward accept, but have several suggestions regarding the empirical evaluation, that are detailed below.

Pros:
* The paper is clearly written and well-motivated.
* It seems easier to express constraints in terms of the state density function rather than Q-values.
* The empirical evaluation contains a comparison to several state-of-the-art approaches for constrained RL.
* DCRL shows modest improvements in the experiments, compared to the baselines.

Cons:
* I would like to see a comparison against the simple yet effective baseline of PPO combined with Lagrangian relaxation, with respect to constraints on Q-values.
* I would like to see results on more challenging constrained RL tasks, where the state density function is harder to estimate, for instance the Safety Gym task suite [1]. These tasks are particularly interesting because CPO (surprisingly) performs worse on them than the simpler PPO-Lagrangian approach. I'm curious whether DCRL would struggle as well.

Comments/questions regarding notation:
* Using $P_a(s, s')$ and $R_a(s, s')$ in the definition of an MDP (Section 2) is odd; I recommend using the typical notation of $P(s, a, s')$ and $R(s, a, s')$.
* In the definition of MDPs, $V^\pi(s)$ is incorrectly referred to as _reward_ / _total reward_, but it should instead be referred to as the _return_ or _expected cumulative discounted reward_.
* It's not clear that the $[0, \infty]$ for the density function $\rho$ corresponds to $t$ (in the first sentence of the Density Functions section).
* Why are there two $s$'s in $\rho_s^\pi(s)$ (e.g., in Equations 1 and 2). Should the subscript be a $\gamma$ instead?
* What does it mean to "linearly interpolate" $\sigma_+$ and $\sigma_-$ (at the end of Section 3.3)?

[1] Ray et al. Benchmarking Safe Exploration in Deep Reinforcement Learning. 2019.
review_title: Novel perspective on constrained RL, with modest improvements

final_score: rating: 7: Good paper, accept
review_content: The submission introduces the dual formulation of a Constrained MDP optimization. While their algorithm does not guarantee that the constraints are satisfied during learning, the experiments show that they are reasonably respected, and yield high returns.

Despite the typos and approximations I could identify, I found the exposition of the submission very clear, the arguments are convincing, and the experiments are compelling. For all these reasons, I recommend to accept it. I rate my confidence low because I have some doubts about the novelty of Theorems 1 and 2.

My main regret is the lack of discussion of the results, both on the theoretical and empirical side. Indeed:
- Theorems 1 and 2: isn't it immediate since J_d=J_p? Is it really novel?
- Proposition 1: I'm surprised the failure modes of Proposition 1 are not more discussed. How can it not converge? Can't we guarantee monotonic improvement? Likewise, how can it converge to an unfeasible solution? Can't we enforce the feasibility during the iteration? Does it only happen when no feasible solution is ever encountered?
- Experiments: what explains the apparent superiority of the primal-dual formulation as opposed to the other algorithms? What are their failure modes? Too conservative? Not safe enough? Does it depend on the setting of some hyperparameters? Isn't it simply because the constraint is defined on the density level and that Algorithm 1 is the only one designed to solve this?

Typos and minor comments:
- P_a and R_a are SxS->_
- I feel that straightening the d, like with \dd from physics package would make the integrals more readable.
- why is the density function defined as [0,\infty)xS->R_>0, isn't [0,\infty)=R_>0? What is it used for? Shouldn't it be simply S->R_>0?
- I would advise to rename \rho_s to \rho, because s is used as a variable everywhere, for instance in Eq. 2 \rho_s(s') looks like it depends on s.
- equation 3 \tho_s^\pi should be just \rho.



review_title: Official Blind Review #3

Meta-review: The reviewers raised several theoretical and empirical questions about the paper. During the rebuttals, the authors seem to  successfully address the experimental issues, in particular those raised by Reviewers 1 and 2. However, the theoretical concerns have mainly remained unanswered. Reviewer 2 has a major concern about the convergence of Algorithm 1, both Reviewers 3 and 4 (in particular Reviewer 3) have several questions about the theoretical analysis and see the technical contribution of the work relatively low. These are not minor issues and require a major revision to be properly addressed. So, I suggest that the authors take the reviewers comments into account, revise their work and properly address the issues raised by the reviewers, and prepare their work for an upcoming conference. 
Decision: Reject