Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper studies off-policy reinforcement learning, where there is a simulation environment with uncertainty parameters and the set of their possible values. The authors provided a solution of off-policy Actor-Critic which optimizes the worst-case performance and evaluated the proposed approach empirically.
strengths_and_weaknesses: This paper has a good motivation, which aims to utilize the uncertainty estimation in order to guarantee the well-behaving of the learning policy from the simulated environment to the real environment. The proposed solution in this paper is to that is to optimize the worst-case performance over policies.


The idea of optimizing the worst-case performance is actually not a novel idea in offline RL. There exist several existing works [1,2] that have explicitly discussed the same idea. Therefore, I feel it is needed to include the discussion of offline RL in the related work section. 

In addition, the empirical evaluation doesn't seem to be able to verify whether the proposed algorithm achieves the main goal of this work---migrating the learning policy from the simulated domain to the real environment. It seems the experiments only focus on the final performance of some fixed environment, rather than the migration from simulation to the real world.

References:

[1] Bellman-consistent Pessimism for Offline Reinforcement Learning

[2] Provable benefits of actor-critic methods for offline reinforcement learning
questions: It would be helpful if the author could address my two main concerns: 1) lack of discussion/comparison of existing offline RL works, especially for the ones that share the same idea; 2) the empirical evaluation doesn't seem to match the goal/motivation of this work well.
limitations: see "Strengths And Weaknesses"
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This work studies the problem of optimizing the worst-case performance in a given uncertainty parameter set.  They propose an off-policy DeepRL algorithm for this problem setting (M2TD3) that solves a tri-level optimization problem using GAN-like gradient descent approach. They show the effectiveness of their proposed approach on a variety of MuJoCo tasks against different baselines. 
strengths_and_weaknesses: # Stregths 
- Strong empirical results: The proposed approach has strong empirical results, and the comparison with the baselines seems thorough. 

# Weaknesses 
- Formulation: Certain aspects of the formulation are not explained in detail. For instance, the choice of only considering deterministic policies when the behaviour policy can be stochastic is not clear. Additionally, the assumption in lines 82-84 should be introduced as a separate assumption and accompanied by proper motivation. In the current form, there is a disconnect between the motivation in the introduction (Real-world cases) and the formulation and the empirical studies. 

- Presentation: Most of the major contributions of the work seem to be algorithmic. However, most of the algorithmic details have been pushed to the Appendix. I think the related work and some of the tables could have been pushed to the Appendix to introduce the algorithm in the main text. Additionally, another baseline (softM2TD3) seems to work really well on these tasks however not many details are provided about it in the main text.

- Related work: I think RobustMDPs should also be included in the related work.

- Hyper-parameter overhead and reproducibility: The paper introduces many algorithmic components and new hyper-parameters, but there is not much discussion of the overhead of introducing these hyper-parameters ($\lambda_{\theta, \eta}, d_{tre}$). There is also no accompanying code for the work. 
questions: - Can you please expand on the motivation on the formulation? Please check the questions in the weaknesses above. 
- Is there any formal claim of statement in Line 182? 
- The part about the uncertainty parameter sampler is not clear. Can you please expand on it more (lines 248-251)?
limitations: The authors should discuss more the implications of assumption in lines 82-84.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This work is concerned with robustness to model misspecification in reinforcement learning. The authors propose a modified objective function to the Twin-Delayed Deep Deterministic Policy Gradient (TD3) algorithm, whereby the critic takes as input an uncertainty parameter of the environment, and the actor optimizes the policy for a worst-case realization of the uncertainty parameters in a predetermined set. The latter optimization is done simultaneously in a gradient descent-ascent fashion as opposed to prior work (Robust Adversarial RL — RARL) which performs each optimization completely alternatingly. The work is validated by thorough experimental results that show improved worst-case performance of the proposed approach compared to a variety of relevant baselines.

strengths_and_weaknesses: Application of robust RL to the deep learning regime is an ongoing open problem that has been addressed to some extent in this work. I find the approach reasonable, well-explained, and placed in the context of the existing literature. The empirical analysis is also quite thorough. On the other hand, I feel a weakness of the work is in its similarity to RARL. Although the proposed approach of simultaneous gradient ascent-descent is reasonable, an advantage of this over RARL would be in better optimization. However this is not analyzed theoretically or experimentally. The authors do mention that the difficulty in optimization may explain why Domain Randomization (DR) demonstrates better worst-case performance than M2TD3, and I feel that this highlights a pretty big problem with the proposed approach: the optimization can become too difficult to achieve the stated objective. As a result, I feel that analyzing this aspect of the work would greatly strengthen the reasons for adoption of this algorithm.

Summary of points and some additional notes are provided below. 

Strengths:
- Thorough empirical analysis
- Details of algorithm well-explained and sensible
- I felt that the authors explained the limitations and did sufficient comparison with existing literature

Weaknesses:
- Need to be able to set simulator settings randomly, and we get robustness only to these settings. This is common in the literature but it still faces similar issues as domain randomization, etc., in that simulation to real setting changes need to be known in advanced.
- There is no theoretical analysis of the convergence properties of the proposed algorithm for simple settings. These results may be impossible in the deep neural network regime but it would inspire confidence in the algorithm to have some result showing that we can expect convergence with direct or linear parameterization (similar to those done by [1], [2], etc)

Writing:
- Would probably be better to put Related Works after describing the algorithm because it was hard to understand all the details that were being compared without knowing the algorithm first. 
- L228: $\max_\eta J_t$ should be $\min_\eta J_t$ ?
- Notationally I found the use of $\omega$ and $\eta$ as the uncertainty parameters confusing. I think it would make the reading easier if only one letter is used. 

Minor:
- abstract has a few grammatical errors
- L37 — extra “has been reported”
- L265: appears to have a typo “optimize theta and omega optimizes (2) …”

[1] Agarwal, Alekh, et al. "Optimality and approximation with policy gradient methods in markov decision processes." Conference on Learning Theory. PMLR, 2020.
[2] Qiu, Shuang, et al. "On finite-time convergence of actor-critic algorithm." IEEE Journal on Selected Areas in Information Theory 2.2 (2021): 652-664.
questions: Some clarifying questions that I had for the authors:
- L68: what’s the difference between $q_\omega$ and $p_\omega$ defined on L60? 
-  Could you comment on the differences between (5) and (2). I think there should be some form of correction factor in Eqn (5) as $\eta$ is different from $\omega$ sampled from $\alpha$ (similar to importance sampling in off-policy). I'm also wondering if using the state distribution according to randomly sampled $\omega$ would lead to instability because the underlying MDP is always changing, as compared to Eqn (2) where $\omega$ is fixed for the state data and the Q function.
- L229 - L235: Why aren’t all $\eta$’s updated at each step instead of only the worst one? Is it to keep diversity of $\eta$’s?


limitations: I think the authors do a reasonable job of discussing and addressing limitations of the work. 
- One other limitation that I see is in designing of the set $\Omega$ such that $\omega^* \in \Omega$, and also to have $\omega$ be a configurable parameter in the simulation environment. This is common practice in robust approaches, however.

- Also as noted by the authors, small dimensions (1-3) of uncertainty parameters were tested due to the exponentially growing size of the uncertainty parameter set with increasing dimensions. Two improvements in this area could be to design the set of parameters in a more directed manner (than a uniform cover), or to do the search in this space could more efficiently than exhaustive search.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: In this paper, the authors consider the gap from simulation environment to the real-world environment and aim to improve the robustness of policy. From a perspective of the worst-case performance on the uncertainty parameter set, they develop M2TD3 to solve a max-min optimization problem. In my opinion, this paper is well organized and the result is of abundant experiments. However, some problems should be concerned, mainly about the solution of the max-min optimization problem, the experiment settings and the selection of the baselines.
strengths_and_weaknesses: Strengths
	Clear problem formulation and very well related work classification;
	Abundant experiments and demonstratable results;
	Clear structure and understandable writing.

Weaknesses
	The authors claim that they develop a robust policy for sim2real scenario, but they suppose that the uncertainty parameter $\omega$ is fixed in each episode. I think this setting is more like transfer learning, which requires that the policy can adapt to tasks with different parameters. To make the policy more robust, the authors should consider the continuous changed uncertainty at each time step in one episode. And, in RARL, the uncertainty is present by an adversarial policy, which also interacts with the environment constantly. Thus, please explain the different setting compared with general robust RL.
	When solving the max-min optimization problem (3), the authors consider that the protagonist policy (robust policy) can observe the uncertainty (the same as the action of the adversarial policy defined in RARL) and add the uncertainty $\omega$ into the Q value function. In a sense, this observable uncertainty modelled in the Q value function makes the designed algorithm more like the domain randomization methods. And, the competitive results from DR(TD3) has the closest performance even DR(TD3) can exceed M2TD3 in the average performance. I think that the authors should add more detail comparison and analysis with DR(TD3).
	The proposed refresh strategy and the uncertainty parameter sampler may be some common methods for stable training policy and exploration noise for environment. 
	Some new works should be considered.
[1] Jian P, Yang C, Guo D, et al. Adversarial Skill Learning for Robust Manipulation[C]//2021 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2021: 2555-2561.
[2] Yu L, Wang J, Zhang X. Robust Reinforcement Learning under model misspecification[J]. arXiv preprint arXiv:2103.15370, 2021.
[3] Zhang H, Chen H, Boning D, et al. Robust Reinforcement Learning on State Observations with Learned Optimal Adversary[C]//International Conference on Learning Representation (ICLR). 2021.
[4] Kamalaruban P, Huang Y T, Hsieh Y P, et al. Robust reinforcement learning via adversarial training with langevin dynamics[J]. Advances in Neural Information Processing Systems, 2020, 33: 8127-8138.
[5] Zhang H, Chen H, Xiao C, et al. Robust deep reinforcement learning against adversarial perturbations on state observations[J]. Advances in Neural Information Processing Systems, 2020, 33: 21024-21037.

questions: Please see the questions in Weaknesses.
limitations: I think that this work would be more convincing if the authors can test the performance of M2TD3 with constantly changed uncertainty, because most of previous works are compatible with this setting. Further, what would be the impact on the algorithm if the fixed $\omega$ setting in one episode is removed.
review_title: null

Meta-review: metareview: This work proposes an actor-critic algorithm that optimizes the worst-case performance over a set of models. A key step is in incorporating the uncertainty parameter in the action-value function, a critic that estimates such a function, and an actor that searches over both a policy as well as the uncertainty parameter leading to the worst performance within the uncertainty set.

The algorithm is built on top of TD3. The paper provides several heuristics to ensure a good performance. It does not have any theoretical guarantee, which is a bit of concern for a work on robustness, but have a relatively comprehensive empirical results, showing the effectiveness of the proposed algorithm and its variants.

We have three reviewers on the accept side and one on the reject side: ViYo is a borderline reject, g5qE and tGzm are borderline accept, and Lcvb is an accept. Based on their review, and my own reading of the paper, I believe this paper can be accepted.

I explain some of the concerns and provide some suggestions to the authors.

ViYo's two main concerns are:
1. Paper not discussing related ideas in the offline RL literature.
2. The empirical evaluation not supporting the main goal of the paper: "migrating the learning policy from the simulated domain to the real environment"

The authors revised the paper and added some citations to the offline RL literature, so point (1) is addressed.

Regarding point (2), I disagree with the reviewer. It is true that the authors don't provide any experiment with real physical systems when sim2real is actually needed. But they have reported the worst-case performance of the obtained policy in the environment, which is the second best thing in this context, and is reasonable to me.
Since the reviewer did not engage with the authors to express their reaction to the response, I interpret the response as satisfying.

Some other concerns are:

- This work considers a fixed uncertainty parameter within an episode, as opposed to some other work on robustness that allows the uncertainty to change at every time step (tGzm).

This is a design choice of this paper, which I believe is reasonable. The authors revised the paper and added a footnote on p2 to clarify this.

- "Most of the algorithmic details have been pushed to the Appendix" (g5qE).

I agree with this. The paper regularly refers to an appendix for more details. Sometimes the detail can be easily postponed to an appendix, but sometimes they were just interrupting the flow of the paper. I believe having an algorithm box in the main body of the paper itself would be very helpful to understand how the method works.

A few other comments based on my own reading of the paper:
- Eq (6) is not the empirical version of Eq. (4), which is the Bellman error.
- Does M3DDPG algorithm reduces to this algorithm if the opponent-agent's policy mu_{\hat{w}} is defined to be a constant \hat{w}, as opposed to a state-dependent function?

As a general suggestion, I believe the clarity of the paper has some room for improvement. There have been a few misunderstandings in the reviews, some of them resolved and some not, which points me to the possibility that the paper might not be as clear as it can be.
confidence: Certain
Decision: Accept