Revisiting Domain Randomization Via Relaxed State-Adversarial Policy Optimization
Authors: Yun-Hsuan Lien
Ping-Chun Hsieh
Yu-Shuen Wang
Venue: Submitted to ICLR 2023 (2023)

Abstract: Domain randomization (DR) is widely used in reinforcement learning (RL) to bridge the gap between simulation and reality through maximizing its average returns under the perturbation of environmental parameters. Although effective, the methods have two limitations: (1) Even the most complex simulators cannot capture all details in reality due to finite domain parameters and simplified physical models. (2) Previous methods often assume that the distribution of domain parameters is a specific family of probability functions, such as a normal or a uniform distribution, which may not be correct. To enable robust RL via DR without the aforementioned limitations, we rethink DR from the perspective of adversarial state perturbation, without the need for re-configuring the simulator or relying on prior knowledge about the environment. We point out that perturbing agents to the worst states during training is naive and could make the agents over-conservative. Hence, we present a Relaxed State-Adversarial Algorithm to tackle the over-conservatism issue by simultaneously maximizing the average-case and worst-case performance of policies. We compared our method to the state-of-the-art methods for evaluation. Experimental results and theoretical proofs verified the effectiveness of our method.

Under review as a conference paper at ICLR 2023
REVISITING DOMAIN RANDOMIZATION VIA RELAXED
POLICY OPTIMIZATION
Anonymous authors Paper under double-blind review
ABSTRACT
Domain randomization (DR) is widely used in reinforcement learning (RL) bridge the gap between simulation and reality through maximizing its average returns under the perturbation of environmental parameters. Although effective, the methods have two limitations: (1) Even the most complex simulators cannot capture all details in reality due finite domain parameters and simplified physical models. (2) Previous methods often assume that the distribution of domain parameters is a specific family of probability functions, such as a normal or uniform distribution, which may not be correct. To enable robust RL via DR without the aforementioned limitations, we rethink DR from the perspective of adversarial state perturbation, without the need for re-configuring the simulator relying on prior knowledge about the environment. We point out that perturbing agents to the worst states during training is naïve and could make the agents over-conservative. Hence, we present a Relaxed State-Adversarill Algorithm to tackle the over-conservatism issue by simultaneously maximizing the average-case and worst-case performance of policies. We compared our method to the state-of-the--e methods for evaluation. Experimental results and theoretical proofs verified the effectiveness of our method.
INTRODUCTION
Most reinforcement learning (RL) agents are trained in simulated environments due the difficulties of collecting data in real environments. However, the domain shift, where the simulated and real environments are different, could significantly reduce the agents' performance. To bridge this "reality gap". domain randomization (DR) methods perturb environmental parameters (Tobin et al., 2017] Rajeswaran al., 2016 Diang al., 2021), such as the mass or the friction coefficient, to simulate the uncertainty in state transition probabilities and expect the agents to maximize the return over the perturbed environments. Despite its wide applicability, DR suffers from two practical limitations: (i) DR requires direct access to the underlying parameters of the simulator, and this could be infeasible only off-the-shelf simulation platforms are available. (ii) To enable sampling of environmental parameters, DR requires a prior distribution over the feasible environmental parameters. However, the design of such a prior typically relies on domain knowledge and could significantly affect the performance in real environments.
To enable robust RL via DR without the above limitations, we rethink DR from the perspective of adversarial state perturbation, without the need for re-configuring the simulator relying on prior knowledge about the environment. The idea is that perturbing the transition probabilities can be equivalently achieved by imposing perturbations upon the states after nominal state transitions. To substantiate the idea of state perturbations, a simple and generic approach from the robust optimization literature (Ben-Tal Nemirovski, 1998) is taking worst-case viewpoint and perturbing the states to nearby states that have the lowest long-term expected return under the current policy (Kuang et al. 202T). While being natural solution, such a worst-case strategy could suffer from severe over-conservatism. We identify that the over-conservative behavior results from the tight coupling between the need for temporal difference (TD) learning in robust RL and the worst-case operation of state perturbation. Specifically: (1) In robust RL, the value functions are learned with the help of bootstrapping in TD methods since finding nearby worst-case states via Monte-Carlo sampling is NP-hard (Ho et al. 2018 Chow et al. 2015 Behzadian et al.) 2021). (2) Under the worst-case state perturbations, TD methods would update the value function based on the local minimum within neighborhood of the nominal next state and is, therefore, completely unaware of the value of the

---

Under review as a conference paper at ICLR 2023
(e) Naive Worst-Case State-Adversarial Approach
(b) Relaxed State Adversaavllllllll Approach
x
+
x
X
x
x6
t-4
1-10
122
1-6
1-10
1-12
Figure 1: We illustrate the over-conservative issue of the naive worst-case state-adversarial policy optimization using a x 4 shortest-pathh grid world environment. The star, cross, and dot represent the goal, the trap, and the initial state, respectively. The terminal rewards of the trap and the goal are -10 and O. We use an arrow to represent the action that has the highest value at each state. Multiple arrows in a state indicate that the actions have equal Q-values. We also use the color indicate the value of the best action at each state. In (a), the agent trained by the naïve worst-case state-adversarial approach fails learn how reach the goal. What's worse, under TD updates, the worst-case state perturbation makes the trap state indistinguishable from other states. As a result, the agent ultimately adversarall
learns to move toward the trap state after 12 training iterations. In (b), our relaxed approach avoids the over-conservatism issue by considering both the average-cass and worst-case environments. We refer readers to Appendix A. for the step-by by-step evolution of the value functions.
nominal next state. As a result, the learner could fail to identify explore those states with potentially high returns. To further illustrate this phenomenon, we consider a toy grid world example of finding the shortest path toward the goal, shown in Figure aa). Although the goal state has a high value, the TD updates cannot propagate the value to other states since all nominal state transitions toward the goal state are perturbed away under the worst-case state-adversaria method. What's 's the agent ultimately learns to move toward the trap state due to the compounding effect of TT updates and worst-case aaveraadvesa perturbations. Notably, in addition to the grid world environment, such trap terminal states also commonly exist in various RL problems, such as the locomotion tasks in MuJoCo. As a result, there remains one critical unanswered question in robust RL: how fully unleash the power of the state- aate-addvrsarial model in robustifying RL algorithms without suffering from over-conservatismm
worse,
To answer this question, we introduce relaxed State-adversale perturbations. Specifically: (1) Instead of taking& pure worst-case perspective, we simultaneously consider both the average-case and worst-case scenarios during training. By incorporating the average-case scenarios, the TD updates can successfully propagate the values of those potentially high-return states to other states and thereby prevent the cver-oonservative behavior (Figure [[)).. (2) To substantiate the above idea, we introduce a relaxed ttate-adversare transition kernel, where the average-case environment can be easily represented by the interpolation of the nominal and the worst-case environments. Under this new formulation of DR, each interpolation coefficient corresponds to a distribution of state adversaries. (3) Besides, based this formulation, theoretically quantify the performance gap between the average-case and the worst-case environments; and prove that maximizing the average- case performance can also benefit the worst-case performance. (4) Accordingly, we present Relaxed state-ad addte--dvessariall policy optimization, bi-level framework that optimizes the rewards of the two cases alternatively and iteratively. One level updates the policy to maximize the average-case performance. and the other updates the interpolation coefficient of the relaxed state-adversaria transition kernel to increase the lower bound of the return of the worst-case environment.
RELATED WORK
Robust Markov Decision Process (MDP) and Robust RL. Robust MDP aims to maximize rewards in the worst situations if the testing environment deviates from the training environment (Nilim &
Cl Ghaoui 2005 Tyengar 2005 Wiesemann et al. 2013). Due to the large searching space, the complexity of robust MDP grows rapidly when the dimensionality increases. Therefore, Tamar et al. approximated dynamic programming to scale up the robust MDPs paradigm. Roy et al. (2017) extended the method to nonlinear estimation and guaranteed the convergence to regional minimum. Afterward, the works of (Wang & Zou 2021 Badrinath & Kalathil 202T)
(214) developed
2

---

Under review as a conference paper at ICLR 2023
study the convergence rate when applying function approximations under assumptions. Derman et al. (202T) showed that the regularized MDPs are a particular instance of robust MDPs with uncertain rewards. They solved regularized MDPs rather than robust MDPs to reduce computation complexity. Grrand-Clément & Kroer (2020) developed efficient proximal updates to solve the distributionally robust MDP via gradient descent and improved the convergence rate. However, although several approximations were presented, such model environments are still too restrictive, and they cannot be used solve real-world problems.
Adversary in Observations. Even a small perturbation to observations may significantly degrade agents' performance because deep neural networks are vulnerable to inputs constructed by adversaries (Huang et al. 2017). Therefore, methods were presented to train agents under environments with adversarial attacks to improve their robustness (Kos & Song, 2017 Pattanaik et al. 2018). To guarantee lower-hound performance, the works of [Lütjens et al., 2020, Wang et al. 2019) adopted the idea of certified defense used classification problems. When making discrete actions, agents are certifiably robust to adversaries in observation within the distance noom)) Since most real-world problems are continuous, there were also methods (Weng et al. 2019 Zhang et al. 2020; Oikarinen
al. 2021 Zhang et al. 2021 presented to improve agents' robustness for continuous actions.
Domain Randomization. Environments can induce the uncertainty of transition probabilities. To simulate this circumstance, can perturb the environmental parameters of simulator to reasonably change transition probabilities when training agents (Huang et al.] 2021] Tobin et al.. 2017: Tiang
al. 202T Igl et al. 2019: Cobbe et al., 2019). Specifically, Tobin al.
(017)) randomly sampled
environmental variables and optimized the agents' average reward. Given that a significant pertur- bation may fail the training, Cobbe et al. (2019) increased the level of difficulty step by step when training agents to improve their average rewards. Jiang et al. (202T) further considered the expected return the optimal case and introduced monotonic robust policy optimization to maximize the average-cass and worst-case returns simultaneously. Since perturbing transition probabilities through environmental parameters demands prior knowledge, Kuang et (202T) transferred states to the nearby local minimum based on gradients obtained from the value function to imitate environmental disturbance. [gl et al. 2019) injected selective noise based on a variational information bottleneck and value networks to prevent models from overfitting the training environment. The regularization helps agents resist the uncertainty of state transition probabilities.
Our method perturbs states through the gradients of the value function, Kuang al. 2021 did. However, pushing states toward the nearby local minimum will make agents over-conservative because they consider only the worst-case scenarios. We present the relaxed state adversarial perturbation and optimize both the average-case and worst-case environments to overcome this problem.
3 PRELIMINARIES
A robust Markov decision process (robust MDP) is characterized by tuple AS, (, where S is the state space, A is action space, is the uncertainty set that contains all possible transition kernels, R S x A + |- Rmaxx [maxx] is the reward function, is the initial state distribution, and y E (0, 1) the discount factor. Let Po P denote the nominal transition kernel, which characterizes the transition dynamics of the nominal environment without perturbation. We define the total expected return under a policy and transition kernel P as
(((5).)))))))))) J P) = E (
Σ y R(s)
For ease of exposition, we also define the value function under policy and transition kernel P as V :=
E R(s),:))ss = S To learn policy in robust MDP, the DR approaches are built on two major design principles: (1) Construction of uncertainty set: DR presumes that one could have access to the environment parameters of the simulator. The uncertainty set P is constructed by specifying the possible range of one or multiple environment parameters, typically based some domain knowledge. (2) Average-case perspective: DR resorts to maximizing the average performance with respect to some pre-configured distribution D over the uncertainty set P, i.e Ep [
3

---

Under review as a conference paper at ICLR 2023
4 DOMAIN RANDOMIZATION VIA RELAXED
4.1 CONNECTING DOMAIN RANDOMIZATION AND STATE PERTURBATION
Conventional DR methods enforce attacks on state transitions by perturbing the environment parame- ters of simulator. This goal can be achieved by perturbing the state after each nominal transition (Kuang et al. 2021): Let (s, a) be some state-action pair, and I S + S be a state perturbation some state s' under 8, is
function. In nominal environment, the probability of the transition P(s'(s,) a). Under the state perturbation I,, the probability becomes ((T a). However, this state adversarial attack is too effective since a value function considers the expected future return, and perturbation to an early state may significantly influence the later states. The over-conservatism problem therefore occurs. We present relaxed adversaallaal policy optimization to overcome the problem. We also prove that the relaxed MDP enjoys two main properties under relaxation: (I) stands for the average performance of the uncertainty set; (2) it guarantees the improvement the performance of the worst-case MDP. Further, we prove that specific average-case MDP corresponds to a relaxation parameter. Hence, we propose an algorithm for adapting the relaxation parameters during training.
4.2 STATE-ADVERSAAEN ADVERSARIAL MDPs AND UNCERTAINTY SETS
adversaaversrialll attacks perturb the current states to neighboring states with the lowest values. This perturbation process can be captured by state-adversarial transition kerne,, which connects the nominal MDP and the resulting adve-aaversa MDP. For ease of exposition, for each state s E S, we define N Ns(s) == {s' (d(ddss,'' s') S o to be the g-neighborhood of where d(s,, s") can be any distance metric. In this study, we use -nom.
Definition (State Perturbation Matrix). Given a policy и and pertubatton parameter a 2 0, the state perturbation matrix Zz with respect to π is defined follows: for each pair of states i, E S,
(1,1,) = {1,
if = arg min,,nn (8) (s))
(2)
otherwise.
The justifications for choosing the above surrogate perturbation model are two-fold: (1) The model
can be interpreted as constructing adversarial examples for the true states. (2) The perturbation
model is closely related to the perturbation of environment parameters, which serve the standard machinery in the canonical DR formulation, as described in (Kuang et al. 202T).
Remark 1. In continuous state space, the arg min in Equation 2 can be computed by adapting the fast gradient sign method (FGSM) (Goodfellow et al. 2014). Let V be a value function (i.e., network) with parameter Φ, s be state, and e be the strength of perturbation. FGSM finds the perturbed e sign( V VVV s)) that has the minimum value, where I|s T(s)lloc S c, and the gradient at is computed using buck- puck-prrougation.
state [s))
Definition 2
MDP). For any policy the corresponding state-ade adivesaavesaaiall MDP with respect to n is defined as a tuple AS, Pr. R. H. 7), where the state-udversarial transition
kernel P is defined as
(- = |271 Po(-ssa), ((s.a)
(3)
Recall that Po is the nominal transition kernel. We use the notation pr [ paragraphs for simplicity. Note that the state adversarial transition matrix 27 depends on the strength of perturbation σ. Each perturbation radius a results in a unique state-ade adversadvaaaaial MDP p.
Po in the later
Remark 2. The state-adversarial MDP defined in Definition 2 involves perturbation of the true states, which is fundamentally different from the perturbation of observations (Zhang et al. 2020).
Definition 3 (Uncertainty Set). Given radius 0, the uncertainty set induced by ndve-aadvrsa perturbations, denoted by Pe, is defined as
P = [P]
P = Po and & c).
4

---

Under review as a conference paper at ICLR 2023
The adversarial attack transits agents toward low-value states. Agents trained using this state adversarial MDP would prevent themselves from falling into the worst situation (Kuang et al. (021). However, large will make agents too conservative and fail to reach any goal state because its value cannot be propagated to neighhoring states by the 'TD updates (Figure D Although using a small e ease the problem, agents would completely omit the risks outside the bounding area. Besides, this strategy is unachievable in discrete environment due to the lower-bound value of e. For example, the agent's movement in the grid world is one hop and cannot be reduced.
Lemma 1 (Monotonicity of Average Value in Perturbation Strength). Under the setting of state adversarial MDP, the value of the local minimum monotonically decreases as the bounded radius a increases. Let be positive real number. The reward function J satisfies
[ Pa
2 J( Patz)
VT.
(5)
The proof is in Appendix A.3 Notably, Lemma T
indicates that among the transition kernels in the
uncertainty set P.,, the worst-case occurs when 0
4.3 RELAXED STATE-ADVERSARIAL MDPS
We present a relaxation framework to address the over-conservatism issue. To begin with, we consider a relaxation on the state-adversarial transition kernel as follows:
Relaxed state-adversarial transition kernel. Given 0 and a E [0, 1]. the a-relaxed state- adversarial transition kemel is defined as convex combination of the nominal and the state- adversarial transition kernels, i.e.,
PT (-ss, P = aPol-Is, (1-a) (1s.
(6)
Connecting relaxed state-adversarial MDPs with domain randomization. DR methods demand prior distribution for computing the average case performance. Let D be a distribution over the uncertainty set P In the following, we show that applying DR with respect to D equivalently cast optimizing objective under relaxed ante-aadvrssaial transition kernel.
Lemma 2 (Relaxation parameter a prior distribution D in domain randomization). For any distribution over the adversaavesa uncertainty set P there must be an [0, such that
Ep-D [ P)] = P.....
The proof is Appendix A.4 It worth noting that different values represent different prior implies that the prior probability of nominal MDP is 1, whereas
assumptions. For example, a = indicates that the prior probability of the worst-case MDP 1. In other words, we can control the value of a to represent different distributions D and train the policies under various environments. To achieve this goal, we quantify the gap between the average performance Ey= ( [)] and the worst performance J(* when updating the current policy
new policy म, and then apply an optimization technique to maximize both of them. One naive bound as follows.
Theorem 1 (A naïve connection between the average-case and the worst-case returns). Given nominal MDP with state adversaries, when updating the current policy æ to new policy ă, the following bound holds (Jiang et al. 2027):
J J(T EPP 2R 1Ep-oldry(PP (1-7)²
drv(T, T)
4.
(1-7)²²
where Rmax is the maximum reward, drv (H. #) indicates the total variation divergence between π and 7. and P the worst nate-adversarial transition kernels.
Theorem indicates that the gap between the average- and the worst- case performance can be expressed using the MDP shift Ep-p pddvv[[[pp) (_. [[]] and the policy evolution drv #). The proof in Appendix A.5 Note that the bound in Theorem is loose because the value the right hand side (RHS) of Equation 7 can be tiny. Specifically, the transition kernel probability shift Eppp/drv(PP [[P)] ) is multiplied by the total maximum return 1--y*
Rus. and the additional denominator
1 1- makes the value even smaller since y is usually set to 0.99 in RL applications. As a result, the bound can be meaningless unless the worst-case MDP pd is very close to the average MDP.
5

---

Under review as a conference paper at ICLR 2023
Since state perturbation only perturbs states to nearby states, consider the smoothness of the reward function and transition property to build a tight connection between the average-case and the worst-case returns. Specifically, Lipschitz continuity in reward function has been widely used in the theory of RI. (Fehr et al., 2018 Asadi et al. 2018, Ling et al., 2016). The smoothness of the transition kernel also holds in most of the environments (Shen et al. 2020 Lakshmanan et al. 2015). For example, in grid-world, the next state must be adjacent to the current state; and in MuJoCo, the poses of consecutive periods are similar, no matter what the state-action pairs are considered. Formally, we define this smoothness property of transition kernels as:
Definition& 4 (ō-Smooth Transition Kernel in State). Let P be a transition kernel and 8 be a positive constans. P is a 6-smooth transition kernel in state if
||s
s S 8,
for all a and for all 8, s with P(s'(s,a) > O.
With the assumption of Lipschitz continuity in reward function and smoothness of transition kemnl,, we arrive at the following bound:
Theorem 2 (Connecting Worst-Case and Average-Case Returns). Given a nominal MDP with two properties: (1) Reward function of corresponding Markov Reward Process (MRP) with respect to any policy is an L,, L,pLipschitz function. (2) Nominal transition kernel Po has the smooth transition funtsihhir,uuuctionnn
property 6, where [|s - ss'll2 S 8, Ya and (P((s'(s. > 0. Then, after updating the current policy π to new policy ã. the following bound holds:
) J(H 2 ((IIPT0)) 47(E + 8)L,a 4(7(E + 8)L, + +(1 (1-7)² Remar dTv( (T, T)
(9)
(1-7)3
(1 (133
where drv (T, 7) is total variation divergence between N and म, Pataa is a relaxed state-adversarial transition kernel, and P is worst-case state-adversarial transition kernel.
The proof is provided in Appendix A.6 Notably, Theorem - holds for any relaxation parameter E [0, 1]. We now briefly discuss the technical challenges in the proof: (1) Propagation of state perturbations across time: The main difficulty lies in the fact that the difference of trajectories under different MDPs would increase in rather nonlinear and complex manner as time evolves.
(2) Quantifying the difference in rewards among trajectories generated under different transition kernels: To measure the difference in rewards under different MDPs, it is necessary to consider not only the probability difference at time t but also the difference in rewards at different states. Despite the above challenges,
proof uses the finding that the difference of initial probability of state under two MDPs Pa and pr.. at time step f can be quantified as ad,, where 0 S A, S 1. Then under the smoothness conditions of the reward function and the transition matrix, we are able to characterize tight bound between the average-case and the worst-case performance.
The intuition of Theorem can be expressed using the terms on the RHS of Equation The first term is the average performance of all MDPs in the uncertainty set. The second term penalizes the large value of because it implies that the relaxed MDP close to the nominal environment. In other words, we expect the average case performance to be high while pushing the uncertainty set close to the worst-case MDP. Finally, the third term prevents a significant update in single step by reducing the total variation divergence drv (T, R).
4.4 ONLINE ADAPTATION OF THE RELAXATION PARAMETER
We leverage Theorem to address both the average-case and the worst-case performance. Specifically, we present bi-level approach to maximize the lower-bound of the worst-case performance (i.e., RHS of Theorem 2 since the unknowns a and π are correlated. The two tasks are optimized alternatively and iteratively. Details are follows:
Lower-level task for average-case return: On the lower level, we improve the policy by optimizing the objective [TT, PP 10 under a fixed relaxation parameter 0t,, This can be done by using any off-the-shelf RL algorithm (e.g., PPO with a clipped objective).
Upper-level task for worst-case return: On the upper level, we design a meta objective J (ax) to represent the lower bound of the worst case performance (RHS of Equation Dp. Hence, the task aims to find a relaxation parameter 0, that can maximize (a,)). To enable a stable training, we iteratively update or by applying the online cross-validation algorithm (Sutton 1992).

---

Under review as a conference paper at ICLR 2023
Both the lower and upper level tasks aim to increase the lower bound of the worst-case performance (me, [ (Equation 9). In the lower-level, a constant relaxation parameter a represents a specitic distribution D. It seeks to maximize the average return over all environments in the uncertainty set following distribution D. In the upper-level, the optimization adjusts to maximize this lower bound. On one hand, increasing a improves the average performance ( 1P since the average-case moves toward nominal environment, yet the price is increasing the MDP shift (i.e., the second term of RHS in Equation D. On the other hand, decreasing a changes the performance and the penalty oppositely. Since is weak initially and its performance gradually improves, the meta objective optimization tends to decrease and then increase a during training.
Algorithm illlstraate our implementation. We first update the policy To, maximize the average- case return J (TT,, 1P. -,.."=----0 : using the proximal policy optimization (PPO). Afterward, we update the relaxation parameter to ensure that the worst-case return is higher than a specific bound (Equation D.. Note that samples used in the two steps are different (Lines 3 and 6 of Algorithm|] [ because the meta objective optimization is an online method. In addition, we chose PPO as a base algorithm since it prevents the model from being updated significantly in a single step. It helps to control the penalty The implementation details are provided in Appendix A.7
term drv (T, R) in Theorem 2
Algorithm 1: Relaxed State-Adversarri Policy Optimization
Input: MDP (S, A, Po: 7), Objective function L, step size parameter 1), number of iterations T. Po is the nominal transition kernel, e-Neighborhood
Initialize the policy Te, for ( 0,...;
do 1 1== where a' ~ ne, (-1s)), and $ ~ Pa(-|ss, o;)
Sample the tuple {si, aj rs. s
Tips
Evaluate ( _P: )
Update the policy to T8.41 by applying multi-step SGD to the objective function as PPO Tip.
Sample the tuple {st, ai Tú: s 1= where d' Te._ (+1s)), and sa Po(-Is;, a;) Update the relaxation parameter to a++1 via one SGD update with respect to the objetvbee
end
EXPERIMENTAL RESULTS AND EVALUATIONS
We conducted two experiments Mujoco Todorov al. 2012) evaluate the performance of our relaxed state adversarial policy optimization (RAPPO). All the baselines and our method were implemented on the PPO (Schulman al. 2017), and the default training parameters were used. In addition, the results were averaged from five different runs/seeds.
Robustness against Environmental Adversaries. We compared our RAPPO with the latest DR method, MRPO (Jiang et al. (02T). to evaluate its robustness against the uncertainty of environmental parameters Agents trained using the two methods were evaluated in the environments, in which the size and gravity were drifted in the range of 0.6 1.4. To simulate the situation that domain knowledge unavailable, during training, MRPO perturbed mass and friction in the range of 0.8 1.2, and our RAPPO attacked the states by its value function. Figure 2 shows the subtractions of the rewards of the two methods. As can be seen, our RAPPO outperformed MRPO since state adversaries were more general than environmental adversaries. Agents trained by MRPO could perform poorly when the perturbations in the training and testing environments were different.
Robustness Against States Adversaries. We compared our RAPPO with SCPPO (uang et al. 202T) to evaluate its robustness against state adversaries. Both of the methods perturb states to improve agents' robustness. We also included vanilla PPO in the experiment because it is the base algorithm of RAPPO and SCPPO. To achieve a fair comparison, the parameters used in RAPPO and SCPPO were the same. Specifically, we set e to 0.015, 0.002, 0.03, 0.001, and 0.005 to the environments of HalfCheetah-v2, Hopper-v2, Ant-v2, Walker-v2, and Humanoid2d-v2, respectively. The parameters were chosen according the variance of actions in the environments.
We obtained the official implementation of MRPO from http://mmeeedddns...nppssss/vvi and used their default parameter setting.
7

---

Under review as a conference paper at ICLR 2023
Walkerdd-22
2000
<6 07
14
Figure 2: We perturbed the size and gravity of the environments and measured the mean rewards achieved by the agents trained using a DR method, MRPO, and our RAPPO. The heatmaps show the subtractions of MRPO's reward from RAPPO's reward. The higher value (red) indicates that RAPPO outperformed MRPO.
Environment
Nominal
e 0.005
8 0.01
0 0.015
( 0.02
= 0.025
5286 1004
4280 1552
3186
1996
1256 125]
1003
HalfChectah
SCPPO
6157 + 709
5046 1533
3367 2090
1795 : 1758
875 64 1259
60 791
C
5830 179
5185 782
4084 1266
2743 1644
1459 1439
406 912
RAPPO
6146 742
4714
4353 #1510
3087 #1568
1878 +1287
846 +951
Nominal
= 0,0008
a = 0.0016
=0000
o 0.0024
= 0.003
PPO
3330 619
1357 #787
613 194
494 151
462 141
417 131
Hopper
SCPPO
2644 951
1369 + 620
876 347
773 357
782 4:37
732 + 412
RAPPO-C
2497 # 1041
1626 + 1064
I122 817
795 4:46
704 393
521 201
1067 605
Nominal
0.001
0.0015
0.002
0.0025
0.003
PPO
3781 + 1165
1564 + 1285
903 +52
763 +353
628 #241
575 + 222
Walker2d
2647
1604
772
CAPPO
4113
2394 1471
1881 1398
1520 1387
1249 1282
888 970
4608
3298
Naminal
0.01
0,02
0.03
0.05
PPO
6075 1 889
4489 = 1342
2071 = 1156
1016 2
703 283
615 2488
Ant
SCPPO
5915 728
4203
441
1661 =
831 398
489 + 273
RAPPO-C
5954 746
4380 1299
2077 1117
1010 568
668 282
537 1 225
RAPPO
6022 698
4381 1357
2284 + 1225
1038 553
733 a: 255
672 219
Nominal
0.003
0.004
0.000
0000
PPO
5357 # 1618
3033 1834
2373 = 1742
1802 1446
1287 1068
939 750
Humanoid
SCPPO
5410 + 1340
3196 1781
2387 1472
1783 + 1256
1271 838
1060
KAPPO-C
5169 1468
3031
1810
1941 1336
1350 1165
1035 551
874 458
RAPPO
5355 # 1491
3768 1972
3227 + 1883
2537 1698
1747 1274
1350 1133
Table I: We compared the performance of agents trained using PPO, SCPPO, our RAPPO-C (i.e., constant relaxation parameter a:) and RAPPO in Mujoco environments under multiple degrees of state perturbation. Mean and Standard deviations reported.
Table shows the testing results. We attacked the agents using their respective value functions under multiple strengths. Specifically, repeated the experiments from 5 different seeds and generated 50 trajectories for each seed from different initial states for evaluation. The means and standard deviations of the rewards agents' performance decreased as the strength of attack increased. In addition, our RAPPO was competitive PPO and SCPPO in nominal environments, and its performance decreased the slowest as the strength of attack increased. It deserves noting that the attacks in the last two columns of Table I were stronger than that of the worst-case. Our RAPPO performed the best in the environments.
reported. Clearly, the results fulfilled Lemma where
Extending SAPPO Using Relaxed State Adversaries. While our RAPPO successfully improves the robustness of agents against state adversaries. classical method, SAPPO Zhhang al. 2020). can help agents against the perturbation of state observations. We thus extended SAPPO by adopting our relaxed state adversarial attacks during training and evaluated effectiveness. Similarly, we compared the methods on the trajectories of 5 seeds and 50 initial states. Table Soole the results. As indicated, the extended RA-SAPPO outperformed SAPPO most of the environments, particularly under strong attacks.
Steady Improvements of the Average and Worst Case Environments. We apply a bi-level approach to optimize the average and worst-case environments during training. To verify the feasibility of this approach, we evaluated the agents' performance under these two cases during training. To determine the worst-case result, generated 50 trajectories from different initial states,
8

---

Under review as a conference paper at ICLR 2023
Environment
Nominal
0.008
0.01
8 0.015
0.02
0.025
Haffcheetah
SAPFO
4928 + 370
4763 = 359
4485 2394
4036 582
3282 1175
2533 1493
RA.SAPPO
5784
5371 1323
4775
4106 1273
3152 1750
Nominal
0.001
0.0015
0.002
0.0025
= 0.003
Walker
SAPPO
4133 + 962
2211 1322
940 + 405
673 + 318
667 326
614 a
RA.SAPPO
4539 1014
3229 1590
1564 a:
789
746 772
Nominal
0.003
= 0.004
= 0.005
= 0.006
= 0.007
Hamanoid
SAPPO
5736 1194
3690 + 2068
2926 + 1956
1944 + 1438
1409 1098
1156 + 789
RAPPA
5320 1164
3960 2082
3335 2117
2882 2066
2129 1776
1567 1474
Table 2: We extended the SAPPO by adopting the relaxed state adversarial strategy and evaluated whether the extension (i.e., RA SAPPO) can improve the agents' robustness against state perturbation. Mean and Standard deviations are reported.
Humanoid-v2
Hopper-v2
Ant-v2
HalfCheetah-v2
Walker2d-v2
MUN
Figure 3: Our RAPPO can steadily improve the average-case and worst-case rewards during training. The solid lines and shaded areas indicate the mean and standard deviation of the rewards, respectively. Note that the variance of the average-case rewards is caused by different adversarial strengths.
perturbed states with the same strength as the training c, and then averaged the rewards. In contrast, the average-case result was determined from 50 initial states and 10 different perturbation strengths, which were uniformly distributed between 0 and In total, the rewards of 50 10 trajectories were averaged. Figure B shows that our RAPPO can steadily improve the average-case performance without sacrificing the worst-case performance. Note that the high variance of the average-case rewards is reasonable because of different adversarial strengths.
The value of the relaxation parameter a. Our meta-objective optimization determines the relaxation parameter (Equation 6) to control the strengths of state udversaries during training. While a is unknown, an intuitive idea is to consider a pyyer--paameet we point out that the value of a should vary at different training stages since agents are weak initially and can perform well after training. To verify that a dynamic over a constant a (i.e., RAPPO-C), we evaluated the performance of agents under state perturbed environments. In the experiments, set a = 0.5 for RAPPO-C since is in the middle of nominal and worst-case environments. The remaining parameters between the methods were exactly the same. As indicated in Table [ RAPPO outperformed RAPPO-C without a doubt. We also refer readers to Appendix [A.8 for the dynamics of during training.
and let users specify the value. However,
6 CONCLUSIONS
We have presented relaxed state adversarial policy optimization to improve the robustness of agents against the uncertainty of environments. Compared to the methods in DR, we perturbed states using the adversarial attack so as to decouple randomization from simulators. Neither prior knowledge of selecting environmental parameters nor prior assumption of parameter distribution are needed. In addition, we introduced a relaxation strategy to tackle the over-conservative problem caused by state adversarial attacks. Our policy optimization maximizes rewards in the average-case while holding the lower-bound rewards in the worst-case environments simultancously. Experiment results and theoretical proofs demonstrate the effectiveness of our method.
Limitations and Future Work. Our relaxation method is state-independent, in which the value of a is adjusted according to the overall performance of policy. Since the degrees of difficulty vary from states states, it will be interesting to investigate the state-dependent relaxation method. In addition, we currently assume that each dimension of states is equally important, which may not be the case. We will also explore the weight of each dimension when perturbing states in the future.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Kavosh Asadi, Dipendra Misra, and Michael Littman. Lipschitz continuity in model-based rein- forcement learning. In International Conference on Machine Learning, pp. 264-273. PMLR, 2018.
Kishan Panaganti Badrinath and Dileep Kalathil. Robust reinforcement learning using least squares policy iteration with provable performance guarantees. In International Conference Machine Learning, pp. 511-520. PMLR, 2021.
Bahram Behzadian, Marek Petrik, and Chin Pang Ho. Fast algorithms for La -oonsttraa S- rectangular robust MDPs, Advances in Neural Information Processing Systems, 34:25982-25992. 2021.
Aharon Bal and Arkadi Nemirovski. Robust convex optimization. Mathematics of operations research, 23(4):766--8) 805, 1998.
Yinlam Chow, Aviv Tamar, Shie Mannor, and Marco Pavone. Risk-sensitive and robust decision- making: evar optimization approach. Advances neural information processing systems, 28, 2015.
Karl Cobbe, Oleg Klimov, Chris Hesse, Taehoon Kim, and John Schulman. Quantifying generalization in reinforcement learning. In International Conference Machine Learning, pp. 1282 1289.
PMLR, 2019.
Esther Derman, Matthieu Geist, and Shie Mannor. Twice regularized MDPs and the equivalence between robustness and regularization. Advances in Neural Information Processing Systems, 34, 2021.
Mathieu Fehr, Olivier Buffet, Vincent Thomas, and Jilles Dihangoye. p-POMDPs have Lipschitz- continuous e-optimal value functions. Advances in neural information processing systems, 31, 2018.
lan J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint 112672, 2014.
Julien Grand-Clément and Christian Kroer, First-order methods for wasserstein distributionally robust MDP. arXiv preprint arXiv:2009:067900 2020.
Chin Pang Ho, Marek Petrik, and Wolfram Wiesemann. Fast bellman updates for robust MDPs, In International Conference Machine Learning, pp. 1979-1988. PMLR, 2018.
Jiaxing Huang, Dayan Guan, Aoran Xiao, and Shijian Lu. Fsdr: Frequency space domain randomiza- tion for domain generalization. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 6891-6902, 2021.
Sandy Huang, Nicolas Papernot, Ian Goodfellow, Yan Duan, and Pieter Abbeel. Adversarial attacks on neural network policies. 3rXiv:/702.022280 2017.
Maximilian Igl, Kamil Ciosek, Yingzhen Li, Sebastian Tschiatschek, Cheng Zhang, Sam Devlin, and Katja Hofmann. Generalization in reinforcement learning with selective noise injection and information bottleneck. Advances neural information processing systems, 32. 2019.
Garud N Iyengar. Robust dynamic programming. Mathemutics of Operations Research, 30(2): 257-280, 2005.
Yuankun Jiang, Chenglin Li, Wenrui Dai, Junni Zou, and Hongkai Xiong, Monotonic robust policy optimization with model discrepancy. In International Conference on Machine Learning, pp. 4951-4960. PMLR, 2021.
Jernej Kos and Dawn Song. Delving into adversarial attacks on deep policies. arXiv preprint 1.X.50705.00453 2017.
10

---

Under review as a conference paper at ICLR 2023
Yufei Kuang, Miao Lu, Jie Wang. Qi Zhou, Bin Li, and Hougiang Li. Learning robust policy against disturbance in transition dynamics via state-conservate policy optimization. arXiv preprint arXiv:2112.10511 2021.
Kailasam Lakshmanan, Ronald Ortner, and Daniil Ryabko. Improved regret bounds for undiscounted continuous reinforcement learning. In International Conference on Machine Learning, pp. 524-532. PMLR, 2015.
Chun Kai Ling, Kian Hsiang Low, and Patrick Jaillet. Gaussian process planning with lipschitz continuous reward functions: Towards unifying bayesian optimization, active learning, and beyond. In Thirtieth AAAI Conference on Artificial Intelligence. 2016.
Bjöm Lütjens, Michael Everett, and Jonathan P How. Certified adversarial robustness for deep reinforcement learning. In Conference on Robol Learning, pp. 1328-1337. PMLR, 2020.
Arnab Nilim and Laurent El Ghaoui. Robust control of markov decision processes with uncertain transition matrices. Operations Research, 53(5):780-799 2005.
Tuomas Oikarinen, Wang Zhang, Alexandre Megretski, Luca Daniel, and Tsui- Wei Weng. Robust deep reinforcement learning through adversarial loss. Advances in Neural Information Processing Systems, 34, 2021.
Anay Pattanaik, Zhenyi Tang, Shuijing Liu, Gautham Bommannan, and Girish Chowdhary. Robust deep reinforcement learning with adversarial attacks. In Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems, pp. 2040-2042, 2018.
Aravind Rajeswaran, Sarvjeet Ghotra, Balaraman Ravindran, and Sergey Levine. Epopt: Learning robust neural network policies using model ensembles. arXiv preprint arXiv: 1610.02.8, 2016.
Aurko Roy, Huan Xu, and Sebastian Pokutta. Reinforcement learning under model mismatch. Advances in neural information processing systems, 30, 2017.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov, Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
Qianli Shen, Yan Li, Haoming Jiang, Zhaoran Wang, and Tuo Zhao. Deep reinforcement learning with robust and smooth policy. In International Conference on Machine Learning, pp. 8707-8718. PMLR, 2020.
Richard Sutton. Adapting bias by gradient descent: An incremental version of delta-bar-delta. In AAAL, pp. 171-176. San Jose, CA. 1992.
Aviv Tamar, Shie Mannor, and Huan Xu. Scaling up robust MDPs using function approximation. In International conference on machine learning, pp. 181-189. PMLR, 2014.
Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojeiech Zaremba, and Pieter Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world. In 2017 IEFF/RSJ international conference intelligent robots and systems (IROS), pp. 23-30. IEEE, 2017.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ international conference on intelligent robots and systems, pp. 5026-5033. IEEE, 2012.
Yue Wang and Shaofeng Zou. Online robust reinforcement learning with model uncertainty. Advances in Neural Information Processing Systems, 34, 2021.
Yuh-Shyang Wang, Tsui-Wei Weng, and Luca Daniel. Verification of neural network control policy under persistent adversarial perturbation. arXiv preprint arXiv:/908, 06353, 2019.
Tsui-Wei Weng, Krishnamurthy Dj Dvijotham, Jonathan Uesato, Kai Xiao, Sven Gowal, Robert Stanforth, and Pushmeet Kohli. Toward evaluating robustness of deep reinforcement learning with continuous control. In International Conference on Learning Representations, 2019.
11

---

Under review as a conference paper at ICLR 2023
Wolfram Wiesemann, Daniel Kuhn, and Berç Rustem. Robust markov decision processes. Mathe- matics of Operations Research, 38(1):153-18) 183, 2013.
Huan Zhang, Hongge Chen, Chaowei Xiao, Bo Li, Mingyan Liu, Duane Boning, and Cho-Jui
Hsieh. Robust deep reinforcement learning against adversarial perturbations on state observations.
Advances in Neural Information Processing Systems, 33:21024-21037, 2020.
Huan Zhang, Hongge Chen, Duane Boning, and Cho-Jui Hsieh. Robust reinforcement learning on state observations with learned optimal adversary. In International Conference on Learning Representations, 2021.
12


Reviews:
[{"review_id": "r_fGs_c60BW", "reviewer": "Reviewer_E4mE", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper identifies an issue they call \"over-conservatism\" with adversarial training methods, which causes more conservative behavior than is desirable.  This is adressed by instead optimizing for a mixture of average case and worst case performance.", "summary_of_the_review": "I am recommending rejection since the core argument of the paper is unclear, and the empirical results do not sufficiently support that argument.", "strength_and_weaknesses": "Strengths: \n* The conservatism (If I've understood it) that they identified can be an bottleneck to performance in some settings\n* The approach is relatively simple\n* The theoretical results appear correct\n\nWeaknesses:\n* It is unclear what is meant exactly by over- conservatism\n\nA) Could mean a specific sort of feed-back effect between the TD updates and the Q-Values as shown in Figure 1.  (that is, the policy training doesn't find the policy the designer intended)\n\nB) Could mean that it is \"conservative\" to a point that it is bad.  (that is, the policy training worked but conservative policy the designer intended to find was too conservative so does not perform well)\n\nOther than Figure 1, the rest of the paper can be viewed as B, and it seems that A is not mentioned again apart from Figure 1.  If A is the point, then I would want to see empirical evidence that this happens by default in the experimental domains.\n\n* Figure 1 is difficult to understand and needs a much more mechanistic explanation if it going to be a core part of the argument.  Once I understood the claim, it felt like I had to rederive the result on my own rather than it being explained.  It is not initially clear how errors would form in the first place, or why those errors would get bigger over time.  Even now, it is not clear why the worst point would be exempt from those errors making its valuation even worse.\n\n* The evaluation is empirically weak\n The policy was only compared to other policies which were not robustified along all of the dimensions their method was able to be made robust to.  This is justified by saying that the other methods would require instrumentation of the simulator in order to modify all dimensions, but it seems that their method would also require a slightly different sort of instrumented simulator, and that the other methods could be trivially modified to use that sort of instrumentation.  \n \n More specifically, it seems that the authors want to disallow things that modify the transition function, but allow for modifications of the state.  Since many simulators don't have a reason to allow arbitrary state modifications, it is an awkward line to draw.  But given that we are drawing that line, it would make the most sense to benchmark against methods which can also modify the state in the same way.  Since adding a perturbation to the state is a special case of modifying the transition function, all the other considered methods should already work in this setting, if configured properly.  Thus allowing the other methods to make state perturbations would be the appropriate baseline\n\nThe results, as given in Table 1 are very difficult to interpret given the highlighting.  Results are highlighted as ``best'' which are not statistically distinguishable from the others, and so it makes the results at a glance look more impressive than they are.\n\n* The theoretical results are straightforward.  While they are nice to have, and good due diligence, I would not consider them as grounds for acceptance on their own.  \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "f_G64PgceAf", "reviewer": "Reviewer_H7Ak", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposed an adversarial training approach for robust RL. The approach is a bi-level method where the lower-level task minimizes  the average-case return against state adversarial attacks while the upper-level task handle the worst-case adversarial return via minimizing a lower bound. Empirical experiments show improved performance from baselines on Mujoco tasks with environmental perturbations and state adversarial attacks.", "summary_of_the_review": "The idea of to train in the relaxed state-adversarial MDP is interesting and seems to be a promising approach for RL robustness, but the paper might have some technical issues, and more evaluations may be needed to identify why and how the proposed method actually provides performance improvement against adversaries.", "strength_and_weaknesses": "Strength:\n- The connection of relaxed state-adversarial MDPs with domain randomization is very interesting, and it provides an average-case robustness metric which might help mitigate over-conservative in adversarial trainings.\n\n- Numerical evaluations in Mujoco tasks show that the proposed method outperforms prior methods in cases with perturbation of state observations as well as cases with adversarial attacks on their value functions. \n\nWeaknesses:\n- The proofs of Theorem 1 and 2 may be correct, but the bounds are a bit strange and they may not correspond the desired performance of the algorithm. In particular, the current policy $\\pi$ does not appear in both the LHS of (7) and (9). This means that one can set $\\tilde{\\pi} = \\pi$ in (7) and (9) and obtain tighter bounds without their last terms. This lacks of dependence on the current policy is likely different from what has been implemented in the algorithm. One can also observe that, although Theorem 1 is mostly the same as Theorem 1 in Jiang et al., 2021, the LHS of the associated equation in Jiang et al., 2021 corresponds to the adversary for the current policy $\\pi$, while the LHS of (7) only corresponds to the adversary of the new policy $\\tilde{\\pi}$.\n\n- No discussion and/or numerical evaluations on how good the lower bound of Theorem 2 is. As mentioned in the previous point, the last term in (9) is likely redundant by setting $\\tilde{\\pi} = \\pi$. The tightness of the bounds is questionable.\n\n- It's not clear if step of minimizing the lower bound of the worst-case return really contributes to better performance. What if one just fixed $\\alpha$ without the upper-level task?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "BzBY7Smdfct", "reviewer": "Reviewer_1qFW", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "To overcome some of the limitations and assumptions required for domain randomization (knowledge of underlying simulator parameters and parameter distributions), the authors tackle the problem of robust RL through adversarial state perturbations instead. However, rather than only accounting for the worst-case perturbations, which can lead to overly conservative agents, the proposed method ‘Relaxed State-Adversarial Algorithm’ simultaneously maximizes both the worst-case and average-case policy performance with the hope that additionally optimizing for average-case performance will prevent overly-conservative behaviours. The authors present a theoretical derivation of their method and evaluate it empirically against relevant prior work.", "summary_of_the_review": "I recommend a 6. The paper is well motivated, the method presented is straightforward and intuitive, and supported by both theoretical proofs and empirical results comparing against SOTA methods. However, the paper would be strengthened if the empirical results were more clearly statistically significant — right now the standard deviations on the results are quite large, which makes it difficult to draw confident conclusions from the experiments. ", "strength_and_weaknesses": "Strengths\n- Very clearly written and well motivated, was a pleasure to read. \n- Motivation behind the method and theoretical insights seem well constructed. \n\nWeaknesses\n- The high variance on the experimental results raise some questions: within the error bounds, RAPPO is often comparable with the baselines (e.g. for Humanoid and Ant). To make the improvements more clear, I would recommend either running more experiments and/or only bolding the results that are significantly the best performing. \n\nQuestions\n- Have the authors considered other schedules for a dynamic alpha? The authors compare against a constant alpha, but I would be curious if a linearly increasing schedule is also sufficient and maybe avoids some of the complexities of learning alpha. It looks like from Figure 5 in the Appendix that alpha can vary quite greatly across seeds, suggesting that it may be difficult to learn. It would also be interesting to see how the different alpha dynamics corresponds to different policy performance. \n\nAdditional Feedback / Minor Notes\n- Formatting in $V_P^\\pi$ definition at the bottom of page 3 — sum over $a_t$ instead of $at$\n- Use of $\\epsilon$ for FGSM in Remark 1 is slightly confusing with use of $\\epsilon$ as perturbation radius in Remark 2, maybe switch to a different symbol\n- 'Upper-lever' instead of upper-level in 4th line on page 7\n- Figure 2 caption — last sentence remove ‘the more’\n- State meta-objective in algorithm for additional clarity ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "-ZzSo8ccXE3", "reviewer": "Reviewer_PgB5", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a method to mitigate the problem of over-conservative policy when optimizing under the worst-case state adversary. Under certain conditions, it proves a tighter bound of the average-case return and worst-case return under state adversary. A two-level update algorithm is proposed to automatically tune the optimization on the worst-case and average-case objective. Experiments on MuJoCo show the learned policies to be more robust against different levels of state adversary.", "summary_of_the_review": "Overall, this is an interesting paper with a novel algorithm to solve the robustness of RL agent against adversary. Several questions need to be answered well in the paragraph to make it better. Experimental results are strong. Perhaps more ablation studies would be good. Aslo it could be better to test the algorithms on some other task suites.", "strength_and_weaknesses": "Strength:\n\nThe paper has good writing and clear delivery of the idea. The theoretical support of the proposed method seems to be solid. The proposed algorithm is novel and interesting. Experiments support the effectiveness of the method well.\n\nWeakness:\n\nWhat’s $J_{meta}$? I checked the appendix and it seems the algorithm uses a learned approximation of the $J_{meta}$, please explain and justify this in the main paper.\n\nIs state adversary really necessary? I agree that the state adversary is more general than other types of environment adversary like parameter adversary, etc. However, is state adversary really capturing the difference between the simulation and reality? Given that the state adversary in the paper is independently generated for each state, there is no consecutive dependency along different steps, while I guess most dynamics mismatch between simulation and reality preserve this dependency. \n\nConfusion about Lemma 2. It is true in Lemma 2 that a certain $\\alpha$ exists to make the relaxed state-adversarial equal to the expectation of uncertainty set in terms of objective value. But when using the relaxed state-adversarial in the algorithm, the relaxed state-adversarial is actually not corresponding to the average-case return, right? Because (1) the $\\alpha$ is changing and (2) the true $\\alpha$ satisfying Lemma 2 is unknown. So why optimizing over the relaxed state-adversarial corresponds to optimizing over the average-case return?\n\nFrom the results in the paper, the $\\alpha$ is decreasing first and increasing later, which indicates the algorithm is optimizing over the average-case objective more at first and later more on the worst-case objective. And the optimization over the worst-case objective is achieved essentially through shrinking the uncertainty set ($\\alpha$ gets larger), but not necessarily improving the policy performance on the full uncertainty set (corresponding to $\\alpha=1$), so how does this really help with the policy facing the worst case?\n\nFor experiments, although the RAPPO outperforms SCPPO, it may not be a fair comparison since RAPPO is designed with state adversary while SCPPO is not, but both of them are evaluated under state adversary.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject