Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: 
In this paper, the author combines robust MDP and constrained MDP for continuous control tasks.
  
1.First of all, I found such a combination is straightforward. I didn’t see any insight from such combinations

2.The introduction on “constrained model misspecification (CMM)” appears to be unnatural. In control engineering, the perturbation to certain parameters can be treated as an uncertainty; disturbance can appear anytime. And most of the time, such “misspecification” is unknown and can be quantified by some metrics, such as certain norms. Any real-world environment could have such an issue. Many factors need to be considered to get rid of CMM. Therefore, the claimed contribution on “mitigating CMM” is not rigorous and maybe over-claimed.

3.The key contribution is from Theorem1. This is useful to show the algorithm convergence. However, any guarantee of safety and robustness can not be theoretically given, either during training or inference. Theorem 1 does not imply these guarantees.

4.The Lagrangian method is standard to deal with the constrained problem. Theorem 3 is not necessary.

5.The authors should avoid using “real-world” unless real-world experiments are performed. I don’t believe simulations using cart pole, walker, and quadruped are neither enough for illustration nor represent “real-world”, even though the name of the software the authors use is named after “real-world”.

review_title: Lack of  theoretical contributions and insights

final_score: rating: 5: Marginally below acceptance threshold
review_content: This manuscript studies the problem of robust and constrained reinforcement learning and proposes two new objectives for incorporating constraints and robustness to misspecified models into RL training. The advantage of these objectives is that their associated Bellman operators are contractive, which enables the use of value-function based methods. 

Overall, I vote to reject this manuscript for the reasons detailed below. 

Pros: 
- Constraints and misspecified models (or models that change over time) are real barriers to deploying RL in many applications. Therefore, the authors study an important problem that has garnered a lot of attention. 

- The manuscript offers some theoretical footing and some empirical evidence for the proposed methods. 

Cons:
- The discounted penalized cost on the constraints does not enforce the constraints. In my opinion, having a discount factor does not make sense in this case since constraints violations in the future would be discounted. Why wouldn't it be a problem to violate a constraints in the future? I understand that this approach was considered before, but that does not make it a good idea. 

- The theoretical results seem to be immediate consequences of the work by Tamar et al, 2014. For example, to show that the sup Bellman operator is a contraction we only need to note that T_sup = - T_inf when we consider the reward -c. Similarly, T_R3C = T_inf when we consider the reward for the T_inf operator to be r - lambda * c. Therefore, all the results follow immediately from Tamar et al. 2014. Given these observations, why is necessary to prove the results in the appendix? 

- I do not find the empirical results compelling. I present my concerns in the next bullet points. 

- Only three random seeds were used for evaluation which is inadequate for capturing the variance of RL algorithms (see [Henderson et al. "Deep reinforcement learning that matters", 2017] and [Islam et al., "Reproducibility of Benchmarked Deep Reinforcement Learning Tasks for Continuous Control," 2017]). Therefore, comparisons such those presented in Table 2 are not meaningful. Even looking at the values in Table 2 one can see that the mean reward for R3C-D4PG is within a standard deviation of the mean reward of other methods. Similar complaint for the plots shown in Figure 2. 

- Although I saw in the appendix the constraint set for each of the tasks considered, I am not sure what the constraint violation costs are. Could the authors clarify this? In particular, I do not know how to interpret the costs shown in the last column of Table 2. How serious is a constraint violation with a cost of 0.113? Also, what are the standard deviations for the constraint costs? Is the difference between 0.113 and 0.128 meaningful? 

- Why does it make sense to average performance metrics across tasks in Table 2? Doesn't each task have it's own scale of rewards and costs? I appreciate that there is a breakdown of performance per task in Figure 2. 

- Finally, what was the motivation for considering only the MPO and D4PG methods? Would it be possible to try a larger collection of methods on the proposed objectives? 

Minor issues:
- All plots are difficult to read. 

- Curly R in the first paragraph of section 2.1 is not defined. 

- Second line of the second paragraph in section 2.1 states that "C: S x A -> R^K is a K dimensional vector." However, "C: S x A -> R^K" is a map. Also, are multidimensional costs used in the rest of the manuscript? It seemed like all costs were scalars. 

- In Definition 1, the values functions V and V_C are never defined (although clear from context). 

- The notation \bf{V} (s) = r(s, \pi(s)) + \gamma \bf{V}(s') throughout the paper does not make sense. s' is a random variable whereas \bf{V} (s) and r(s, \pi(s)) are deterministic.

- In the Metrics paragraph, below Table 2, "different" -> "difference"

- Above equation 10 in appendix A.3, the sentence "The full derivation ..." is redundant.

----
Update after rebuttal:

I appreciate the authors' answers and revisions of the manuscript. The theoretical presentation is clearer with the new notation and I appreciate the improved Figure 2. 

I appreciate that the authors followed my suggestion to evaluate on more than 3 random seeds. Statistically, however, 5 random seeds are not much different. I was envisioning using at least 30 random seeds. Is computation a major bottleneck? Maybe a simpler and faster method could be used to showcase the benefits of the new objective. 

Overall, the new version of the paper is better, but I think the empirical evaluation and the writing should be further improved. I updated my score accordingly. 

review_title: Recommendation to Reject

final_score: rating: 4: Ok but not good enough - rejection
review_content: The paper suggests two approaches to combine the concepts of robust Markov decision processes (MDPs) with that of constrained MDPs. In the first approach, called R3C, a worst-case setting is used for both the expected total discounted rewards criterion and the constraints on the state-action pairs. The robustness is defined with respect to all possible choices (from an uncertainty set) of transition-probability functions. In the second approach, called RC, only the constraints should be robust against all possible transition probabilities. The paper studies the value functions and the corresponding Bellman operators of these problems and argues that, in both cases, these operators are contractions in the supremum norm. Finally, numerical experiments are presented on RWRL problems, such as the cart-pole and the walker, showing the effect of using the redefined operators.

The general problem that the paper studies (namely, robust constrained MDPs) is nice and worth investigation, but the offered combination is straightforward, the theoretical results are weak, and the paper is poorly written (see below). Therefore, the current form of the paper is substandard and needs major improvements. 

Several notations and concepts are not specified, starting from the state and action spaces of the MDP. For example, it is not clear whether these spaces are finite, or otherwise, what structure is assumed about them (the minimal assumption that one needs is that they are measurable spaces). The uncertainty set itself is not defined, just the notation is used. The  constraint function $C$ is defined as $S \times A \to \mathbb{R}^K$, but then few lines later in the definition of $J_C^{\pi}$ we simply have $\sum_{t=0}^{\infty} \gamma^t c_t \leq \beta$, without actually stating what $c_t$ is. If $c_t$ is defined as $c_t := C(s_t, a_t)$, then $\beta$ should be a vector to make the above inequality meaningful, with the notation that $\leq$ means coordinate-wise less than or equal. However, things like these should be guessed by the reader as the paper lacks proper definitions.

In Section 2.3.1, about the R3C part, in the second equation after (1), the obtained results are dubious, as ${\bf V}(s')$ should not depend on $s'$, as $s'$ is just a random variable with respect to an expectation is taken (see the definition of the classical Bellman operator). Also function ${\bf V}$ is not defined in the paper, the reader should guess its meaning from the appendix. In section 2.3.1, in equation (2), it is not clear with respect to what probability distribution the expectation is taken in $V(s)$. Is there a special element of the uncertainty set, a "nominal" model?

The structure of the paper is also a bit chaotic. For example, there is an "Experiments" and also an "Experimental Results" part, both containing results of various experiments. Moreover, Sections 6 and 7 should be subsections of Section 5, etc.
review_title: An important topic, but the results and the presentation are substandard

final_score: rating: 4: Ok but not good enough - rejection
review_content: The standard Reinforcement Learning framework is limited in many ways, and numerous variants have been introduced to deal with aspects such as partial observability, temporal abstraction, safety, domain transfer, etc. Yet, these issues are often studied separately and it is often unclear how to combine them together. This is the ambitious challenge taken by this paper, which attempts to bridge the two separate settings of Robust MDPs, which aim at considering ambiguity in the dynamics, and Constrained MDPs, which aim at enforcing the satisfaction of a constraint on an expected cost signal. The authors propose the formulation of two objectives, that merge the two aspects and include both a worst-case evaluation over the ambiguity set and a constraint violation penalty term. The ways of dealing with both issues are fairly standard (Lagrangian relaxation of the constraints with alternating optimization, and worst-case evaluation over a finite set of simulated transitions in practice), but their combination seems novel and relevant. These objectives come with the corresponding Bellman Expectation operators, which allow to evaluate the current policy (critic) and provide a feedback (gradient) for the actor to ensure robust constraint satisfaction. The applicability of the proposed approaches is demonstrated on a benchmark of Mujoco tasks, where they are shown to compare favorably to several baselines.

My main concerns lie with the definitions and results of Section 2.3, which I think generally lack rigour and clarity, which sheds doubts on the validity of the claimed results.
1. The authors start by defining the R3V value function $\mathbb{V}$, as a bootstrap of two other values $V$ and $V_c$, that haven't been defined. I was initially confused because they are denoted as if they do not depend on the policy $\pi$, so I first thought these referred to optimal value functions (which would need to be appropriately defined, especially $V_c$ since the costs are constrained rather than optimized), but they seem to be in fact the expected returns for the policy $\pi$ (i.e. the value functions of a policy $\pi$ as opposed to the optimal value functions).
2. Likewise, do the values $V$ and $V_c$ in definition 1 depend on the dynamics $p$? It seems so, but it should be written explicitely.
3. The derivation of A.2 seems a bit sloppy, since the last term in line 4 is identified as$ \mathbb{V}$while it does not strictly correspond to the definition 1.
4. The next state $s'$ is a random variable that depends on the dynamics $p$, and thus subject to the robust inf/sup over $P$ in the objective (1), but in the derivation A.2 and the resulting R3C Bellman operator of definition 2, it is considered as a deterministic variable in which the R3V value can be evaluated freely (without any expectation over $p$, nor inf of $p$ over $P$).
4. In Theorem 1, the R3V values $\mathbb{U}$ and $\mathbb{V}$ are described as functions of $S \to\mathbb{R}^d$, but they were defined as functions of $\to\mathbb{R}$ in definition 1. Also, $d$ is not defined.
5. According to definition 2, the R3V Bellman operator applied to a real function $\mathbb{V}$ simply consists in multiplicating \mathbb{V} by the discount gamma and adding the penalized reward $r - \lambda c$. But then, this is exactly the same as the RC Bellman operator of definition 4. The difference between the two frameworks lies in how the policy value $\mathbb{V}$ is defined (regarding the presence or absence of $\inf_{p\in P}$ before $V^{p,\pi}$), but these differences are not involved when we consider arbitrary functions $\mathbb{V}: S \to\mathbb{R}$ on which to apply the Bellman operators. I feel like  the authors intended the definitions 1 and 3 to be seen somehow as *operators* rather than *functions*, which could allow to retain the sup/inf in the definition of $\mathcal{T}^\pi_{R3C}$ and $\mathcal{T}^\pi_{RC}$, but it is a mere speculation and certainly not what is written in the paper.


In conclusion, this paper comprises a clear motivation, promising insights and encouraging results. But in the present state of vagueness of the theoretical framework, I cannot recommend acceptance. Of course, it may only be a misunderstanding from my part merely related to presentation/clarity issues and not deeper flaws in the reasoning, in which case I am ready to update my rating upon clarification by the authors.

**Minor remarks**:
* Since the definitions and results of Section 2.3 are claimed as novelties, I think they should not be listed as part of the Background section
* After theorem 1, a paragraph states that the $\mathcal{T}_{R3C}$ operator can be used in a Value Iteration algorithm, which is not the case since it is a Bellman expectation operator, not a Bellman optimality operator. If I am not mistaken, it can only be applied in a Policy Iteration scheme.
* The first paragraph of Appendix A3 is self-referencing
* Typo in (20): use norm $||$ instead of absolute value $|$
review_title: Encouraging results, but the theory section needs more work

Meta-review: # Quality:
I personally feel that the comment from Reviewer1 regarding "real-world" is a minor but valid point. Even after the rebuttal, the abstract seems to suggest that the proposed algorithm is effective to solve real-world challenges. Maybe further rephrasing or explicitly stating that the experiments are in simulation might help to clarify this point.

Reviewer3 also raised valid points regarding the experimental evaluation and the use of just 3 seeds. Given the struggle in reproducibility in RL and the shady experimental practices from even leading AI companies (e.g., cherry-picking of seeds), it is paramount that experiments follow strong methodologies and good practices. As such, the experimental results presented in this manuscript should be strengthened.

# Clarity:
All the reviewers pointed out that the paper writing should be improved. Although the authors significantly improved the manuscript during the rebuttal period. Several reviewers suggested that the manuscript should be further polished before publication.

# Originality:
The two proposed approaches are novel to the best of the reviewers and my knowledge. Two reviewers pointed out that the theoretical results should be explained more thoroughly and to clearly differentiate from prior work.

# Significance of this work: 
The paper deal with an important and timely topic. Although the work could be very impactful for real-world applications, there is no real-world application. Hence it is difficult to gauge the significance of the work. 

# Overall:
The paper does not feel quite ready for publication yet. A clearer presentation and extended experiments would certainly improve the quality of the manuscript.
Decision: Reject