Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a strong and efficient robust training framework for RL to directly estimate the optimizes the worst-case reward of a policy under p attacks. The authors empirically show that the proposed robust RL framework achieves the STOA results under strong attacks.
strengths_and_weaknesses: The paper addresses an important problem of robust RL, and the proposed method is clear and the paper is well-organized. Some analysis such as the robustness explanation of the proposed method is interesting.

However, the technical novelty of the paper is limited. For instance, the worst-case bellman operator is similar to the worse-case attack defined in [54, 41]. The worst case value estimation should be the most interesting part of the paper, while it’s just based on standard convex estimation for the worst case attack, but the DNN based RL the Q function is not convex and there are lots of strong assumptions in this process, which makes the estimation less valuable. 

questions: It is great to see that empirically the proposed robust RL framework achieves high robustness. Is there any analytical results for the robustness? say, the lower bound of the robustness as a function of the convexity approximation of the worst-case attack？
At the beginning of the paper, the authors mentioned that the explanation results of the proposed framework make sense, while it is not evaluated in the evaluation section. Are there interesting findings and observations on general games regarding the explanation of the proposed framework?
limitations: The technical novelty of the paper is limited due to the direct estimation of the worst-case adversary/value. Maybe it is helpful for the authors to clarify the technical novelties of the paper, especially compared to [54,41] in addition to the conceptual differences.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper introduces a robust training algorithm for RL. The key idea is to use a worst-case Bellman operator that allows calculation of the worst-case expected return of a policy under an adversarial attack. The new value function can then be used to find policies via both value-based methods or policy gradient ones. The authors include a weighting scheme to guide NN robustness towards more critical states.
strengths_and_weaknesses: The paper introduces a critical idea in robust training in RL. The proposed way of finding the robust policy by a value-based method directly and without a separate attacker module is significant. According to the paper's literature review, it has been a crucial idea missing in prior work. The authors do a good job empirically evaluating the new algorithm compared to prior work. The fact that the method can be applied to many existing algorithms is a nice property. Overall, the paper is well-written and impactful.

As a weakness, I wish the paper included some theoretical insight. Knowing the dynamics of the new operator even in finite MDPs is interesting. For example, the convergence of worst-case values for a fixed policy is intuitive and established, but when the policy is being updated, the dynamics are not as clear. 

Related to uncertainty in non-stationary policy case, I wonder if it is necessary to combine robust values with natural ones via $\kappa_{wst}$. In WorcA-DQN, the algorithm uses worst-cases values for most of the training phase and then it is increased. A more discussion about such design choices would be appreciated.

==== Post Rebuttal ===
I thank the authors for their clarifications on my concerns. My comment on $\kappa_{wst}$ was properly answered. However, my score remains the same.
questions: How necessary is $\kappa_{wst}$ for the convergence of the algorithm, for example in WorcA-DQN? Does $\kappa_{wst} = 0$ work?

How sensitive is the algorithm to the schedule of $\kappa_{wst}$?
limitations: There is not much discussion in this regard. More explanation would benefit the paper.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper proposes an efficient RL training framework that is robust against $\ell_p$ adversarial attacks on states. Different from the prior work that requires additional training for an RL adversary, the proposed method directly estimates the worst-case Q values without training an extra RL adversary. The paper introduces a new Bellman operator named the Worst-attack Bellman operator and computes the worst-case Q values by minimizing the Bellman error. Given the worst-attack Q-function, a policy is optimized to maximize the Q-function via standard RL algorithms such as PPO or DQN. Finally, the paper proposes a policy regularization technique that selectively minimizes the action changes under adversarial perturbations based on the state importance weight $w$. The experimental results show the effectiveness of the proposed method in MuJoCo and Atari environments.
strengths_and_weaknesses: Strengths:
- The proposed method achieves state-of-the-art performance with a large margin.
- I think introducing a new Bellman Operator is novel and the authors formalize well its definition and properties.
- The paper is well-organized and easy to read.
- The proposed framework is simple and effective, which can be easily combined with existing RL algorithms.

Weaknesses:
- I could not find any critical problem in this paper, but some minor problems should be addressed (see Questions below).
questions: **Estimating Worst-attack Value**

To compute the target of the Bellman error, one has to solve the problem of minimizing the Q-function, which can be infeasible when the action space is continuous. The authors say that this can be solved approximately via gradient descent (line 203), but running multi-step gradient descent for every Q-learning step is computationally expensive (like adversarial training). Could you please provide more details on how to solve this minimization problem? If the authors use an additional policy network (RL adversary) to approximate the minimization problem analogous to standard actor-critic algorithms, the authors should clearly explain the difference between the proposed method and the prior work that trains an RL adversary explicitly.

**Convex relaxation technique**

I am just wondering about how the convex relaxation technique can scale to environments with high-dimensional input spaces such as Atari. In [1, 2], this technique is only tested on small networks trained on datasets with small input sizes, such as MNIST and CIFAR10. Can this technique be applied to larger networks such as ResNet used in IMPALA [3]?

If these two problems are well-addressed in the rebuttal, I am willing to raise my score.

[1] Wong and Kolter, Provable Defenses against Adversarial Examples via the Convex Outer Adversarial Polytope, ICML 2018. \
[2] Zhang, et al., Towards Stable and Efficient Training of Verifiably Robust Neural Networks, ICLR 2020. \
[3] Espeholt, et al., IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures ICML 2018.

---
Post-rebuttal: Since all of my concerns have been well addressed, I decide to raise my score from 6 to 7.
limitations: I cannot find any potential negative impact of this paper.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper presents a new algorithm for training deep reinforcement learning agents robust against input perturbations. It is based on using certified bounds to provide attack-agnostic defense, while also combining with a learned worst case value function to incentivize agents to choose safe policies. Its results are evaluated on MuJoCo and Atari environments showing equal or better performance and efficiency as existing state of the art robust training algorithm, with especially large gains against stronger attacks.
strengths_and_weaknesses: Strengths:
- Novel and good ideas, especially the usage of learned worst-attack critic, as well as state-importance weights. 
- Worst attack critic and Worst-case aware policy optimization provide an efficient and powerful solution to the issue of defending against powerful attacker with long-term goals.
- Extensive empirical evaluation and good empirical results, especially against stronger attacks.
- Clear writing and overall good discussion of related work

Weaknesses:
- Some strange design choices (see questions for more details) 
- Missing an important connection to related work: Worst Attack Value described in section 4.1(when using convex relaxation to solve A_adv) is mathematically the same as Greedy Worst Case reward evaluation metric defined in Algorithm 1 of [32] (up to discount term). This is not an issue for the novelty of the paper as it is used for very different purposes, but the connection should be acknowledged and discussed in the paper.
- Minor issues with the mathematical notation used to describe worst attack Bellman operator:
Specifically, using convex relaxation to approximate A_adv does not solve the worst attack Bellman operator as defined in Eq. 1 but instead gives a lower bound for it. While this is somewhat discussed in the Appendix, this distinction should be clarified in the main text.
- Paper could be even stronger if evaluated on more realistic settings, such as real world applicable robotics.
questions: In figure 5, is attack applied at every step during training, or just used to evaluate checkpoints in "test time"? If yes, are there experiments without train-time attacks as certification based defenses like [32, 54](and WocaR if I understand correctly) don’t require train time attacks to be robust?

Weird design choices:
- Why is it necessary to train a separate clean policy Q for WocaR-DQN? This is not necessary for comparable baselines [32, 54], why is it needed now?
- Appendix Line 741-742: “For reproducibility, the reported results are selected from 30 agents for different training methods with medium performance due to the high variance in RL training.” Is this common practice? What are the results when averaging over all runs, not just median 20?

Minor errors to be fixed:

- Line 256: L_{reg} in Equation (4) → L_{wst} in Equation (4)
- Figure 5: The colors of SA-PPO and RADIAL-PPO are too similar, hard to tell which one is which
- Appendix table 2: Walker first two columns should bold RADIAL-PPO instead of SA-PPO
limitations: Limitations are not really discussed, even though authors claim to do so in Appendix B. Some limitations that could be mentioned are that their methods are restricted defending against to l_p - norm bounded attacks which may not be the most useful setting for real world use.

Discussion of potential impact: 
"Our goal is to efficiently improve RL robustness, which does not have any negative impacts." - While I agree with this in principle, the wording is too strong here, instead should say something like we don't think has any negative impacts. Plausible negative effect could for example be that more robust rl algorithms speed up adoption of robots to replace some occupations, causing mass unemployment.

### Edit:

Thank you for the detailed response. While these responses have not changes my overall view of the paper, they have addressed my outstanding concerns and clarified certain parts of the paper. In response I have adjusted my score for presentation from 3->4.
review_title: null

Meta-review: metareview: This paper introduces a novel adversarial training method that directly computes a worst-case performance under budget bounded attacks during the training process. As a result, the method is more sample efficient and achieves state-of-the-art performance across a number of test cases. 

The reviewers agree that the contributions are novel and well validated, making this paper a clear acceptance. 

confidence: Certain
Decision: Accept