Max-Min Off-Policy Actor-Critic Method Focusing on Worst-Case Robustness to Model Misspecification
Authors: Takumi Tanabe
Rei Sato
Kazuto Fukuchi
Jun Sakuma
Youhei Akimoto
Venue: NeurIPS 2022 Accept (2022)
Keywords: Deep Reinforcement Learning
Maxmin Optimization
Sim2Real

Abstract: In the field of reinforcement learning, because of the high cost and risk of policy training in the real world, policies are trained in a simulation environment and transferred to the corresponding real-world environment.
However, the simulation environment does not perfectly mimic the real-world environment, lead to model misspecification. 
Multiple studies report significant deterioration of policy performance in a real-world environment.
In this study, we focus on scenarios involving a simulation environment with uncertainty parameters and the set of their possible values, called the uncertainty parameter set. 
The aim is to optimize the worst-case performance on the uncertainty parameter set to guarantee the performance in the corresponding real-world environment.
To obtain a policy for the optimization, we propose an off-policy actor-critic approach called the Max-Min Twin Delayed Deep Deterministic Policy Gradient algorithm (M2TD3), which solves a max-min optimization problem using a simultaneous gradient ascent descent approach.
Experiments in multi-joint dynamics with contact (MuJoCo) environments show that the proposed method exhibited a worst-case performance superior to several baseline approaches.

Max-Min Off-Policy Actor-Critic Method Focusing on Worst-Case Robustness to Model Misspecification
Takumi Tanabe
Rei Sato
University of Tsukuba & RIKEN AIP Tsukuba, Ibaraki 305-8573, Japan tanabe@bbo.co cs. tsukuba ac. jp
University of Tsukuba & RIKEN AIP Tsukuba, Ibaraki 305-8573, Japan reisato@bbo.. cs. w.s.tsuuaa..... ac.bbaaa........ jp
Kazuto Fukuchi
Jun Sakuma
University Tsukuba & RIKEN AIP Tsukuba, Ibaraki 305-8573, Japan fukuchi@cs. tsukuba.ac.) jp.
University of Tsukuba & RIKEN AIP Tsukuba, Ibaraki 305-8573, Japan junêcs tsukuba. ac. ac.jp
Youhei Akimoto
University of Tsukuba & RIKEN AIP Tsukuba, Ibaraki 305-8573, Japan akimoto8cs tsukuba. jc.jp
Abstract
In the field reinforcement learning, because the high cost and risk of policy training in the real world, policies are trained in a simulation environment and transffered to the corresponding real-world environment. However, differences the environments lead to model misspecification. Multiple studies report significant deterioration of policy performance real-world environment. In this study, we focus on scenarios involving simulation environment with uncertainty parameters and the set of their possible values, called the uncertainty parameter set. The aim is to optimize the worst-case performance on the uncertainty parameter set to guarantee the performance in the corresponding real-world environment. To propose off- off-policy actor-critic approach
obtain policy for the optimization, called the Max-Min Twin Delayed Deep Deterministic Policy Gradient algorithm (M2TD3), which solves max-min optimization problem using simultaneuus gradient ascent descent approach. Experiments in multi-joint dynamics with contact (MuJoCo) environments show that the proposed method exhibited worst-case performance superior to several baseline approaches. Our implementation publicly available (https: /ttpss////tthbt com http///kooooool//22253/3
Introduction
Applications of deep reinforcement learning (DRL) to control tasks that involving interaction with the real world limited because of the need for interaction between the agent and the real-world environment [37]. This difficulty arises because of high interaction time, agent maintenance costs, and safety issues during the interaction. Without enough interactions, DRL tends overfit specific interaction histories and yields policies with poor generalizability and safety issues [29].
To solve this problem, simulation environments that estimate the characteristics of real-world envi- ronments are often employed, which enable enough interactions for effective DRL application. The policy is trained in the simulation environment and then transferred to the real-world environment [45].
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Moreover, because fast and general- purpose simulators, such as Open Dynamics Engine [34], Bul- let [3] and multi-joint dynamics with contact (MuJoCo [38]), are available in the field of robotics, the cost of developing simulation environment can sometimes be significantly lower than that for real-world interactions.
However, because a simulation environment is an estimation of the real-world environment, discrep- ancies exist between the two [2]. Even if they are entirely similar, discrepancies may still occur, e.g., because of robot degradation over time or weight changes because of part replacements [22]. Studies report significant degradation of policy trained in simulation environment when transferred to the corresponding real-world environment [8]. The above discrepancies limit the use of simulator environments in practice and, thus, limit the use of DRL.
In this study, to guarantee performance in the corresponding real-world environment, we aim to obtain a policy that maximizes the expected reward under the worst-case scenario in the uncertainty parameter set Ω. We focus on a scenario where (1) a simulator MM is available, (2) the model uncertainty parameterized by w E 22 and w is configurable in the simulation, and (3) the real-world environment is identified with some w E S2, where w" is fixed during an episode; however, (4) w" is unknown, is not uniquely determined, or changes from episode episode'. The model of uncertainty described in hypotheses (3) and (4) is referred to as the stationary uncertainty model, in contrast the time-varying uncertainty model where w can vary during an episode [28]. We develop an off-policy deep deterministic actor-critic approach to optimize the worst-case performance, called the Max-Min Twin Delayed Deep Deterministic Policy Gradient Algorithm (M2TD3). By extending the livle optimization formulation of the standard actor-critic method [30], we formulate our problem as a tri-level optimization, where the critic network models a value for each tuple of state, action, and uncertainty parameter, and the actor network models the policy of an agent and an estimate of the worst uncertainty parameter value. Based on the existing deep deterministic actor-critic approach [19; 7], we design an algorithm to solve the tri-level optimization problem by implementing several technical components to stabilize the training, including multiple uncertainty parameters, their refreshing strategies, the uncertainty parameter sampler for interaction, and the soft-min policy update. Numerical experiments on 19 MuJoCo tasks reveal the competitive and superior corst-case performance of our proposed approaches compared to those of several bascline approaches.
2 Preliminaries
We consider episodic Markov Decision Process (MDP) [6] family Ma (Mulwen. where Ma = (S, A, Pw, Pw. Tw: 7) is the MDP with uncertainty parameter w E Ω. The state space S and the action space A are subsets of real-valued vector spaces. The transition probability density Pa : S x A x S + R. the initial state probability density ph 5 + R, and the immediate reward rw S A R depend on The discount factor is denoted by 7 E (0, 1).
We focus on an episodic task on Ma. Let pa : > A be a deterministic policy parameterized by 0 E O. Given uncertainty parameter Ω the initial state follows so ph. At each time step t > 0, the agent observes state 8t, select action a = (st), interacts with the environment, and observes the next state sy p- |t,,,) the immediate reward T2 Tw (st, ag), and the termination flag f4. Here, h4 = if s is a terminal state or a predefined maximal time step for each past episode; otherwise, h4 0. If h4 0, let st+1 s/; otherwise, the next state is reset by 8141 ~ ph For simplicity, we let g. ($++1 st, at) be the probability density of 8441 given St and at. The discount reward the trajectory starting from time step / R; = Σxzn "Te4k.
The action value function Q"2 a, w) under w is the expectation of R, starting with 8, and ag under wt that is, Qu"(ss, w) = E[R, 8t where the expectation is taken over 8 for l > 0. Note that we introduce w to the argument to explain the Q-value dependence on w; however, this is essentially the same definition as that for the
= 8144+1 ~ (- 8ttk, po (+( Vk: > 0],
'Here are two examples where hypotheses (3) and (4) are reasonable. (I) We train common controller of mass- produced robot manipulators that are slightly different due errors in the manufacturing process (w fixed for each product, but varies in Ω from product to product). (II) We train a controller of a robot manipulator whose dynamics change over time because of aging. but each episode is short enough that the change in dynamics during an episode is negligible.

---

standard action value function Q (s, a). The recursive formula is as follows:
= (u)
a) + BE( (s', , po(ss) W, s' qu( ss
(1)
We consider off- reinforcement learning setting, where the agent interacts with Mu using stochastic behavior policy 3 S x A + I. However, the target policy to be optimized. denoted by pe. deterministic. Under fixed wa Qt the standard objective of off-policy learning is to maximize the expected action value function over the stationary distribution p², such that
(w( = .(c ( ww) (00Q0((ss (s),
LEES
Here, (s w) limr + Et- under / and a fixed w E Q where the step-t transition probability density L2.0 is defined as ( Sae (s (, s)da and (
- Sao (u so)p (No))do denotes the stationary distribution
SAES ( s) 5)d5.
We assume that the agent can interact with M.. for any w E 52 during the training and can change after every episode, that is, when h Our objective is obtain the μg that maximizes Ju under the worst environment w E Q2 Hence, we tackle the max-min optimization problem requiring maxsge minwen Jw (0).
3 M2TD3
We propose an off-policy actor-critic approach to obtain a policy that maximizes the worst-case performance on Ma, called M2TD3. Our approach is based on TD3, an off-policy actor-critic algorithm for deterministic target policy. In TD3, the critic network Qo parameterized by é E & is trained to approximate the Que of jso. whereas the actor network models μi with parameter E and is trained to maximize (0). Moreover, w is not considered and is fixed during the training. In this study, to obtain the fá that maximizes the performance under the worst w, we extend TD3 by formulating the objective as maximin problem and introducing simultancous gradient ascent descent approach.
The main difficulty in the worst-case performance maximization is in estimating the worst-case performance. Because the objective expected to be non-concave with respect to (w.r.t)) the uncertainty parameter because of deep actor-criti critic networks, solving this problem is considered intractable in general [4]. To stabilize the worst-case performance estimation, we introduce various techniques: multiple uncertainty parameters, uncertainty parameter refresh strategy, and an uncertainty parameter sampler.
Formulation We can formulate our objective as the following tri-level optimization:
max min (0, Wi; od)) s.t.
argmin (():0) 6E
DEB wen
where J/ and L are defined below. Note that this tri-level optimization formulation is an extension of the bi-level optimization formulation of the actor-critic approach proposed in [30].
We introduce a probability density a Ω + R, from which an w to be used in the interaction during the training phase is drawn for each episode. The relationship between @ and a is similar to that between 8 and 8: namely, @ and 0 are the parameters to be optimized while a and 3 are introduced for exploration. Let på (s) Swee p² (s (w)x(w)du and på (s, a, a,w) B(a 8))0(s (s w))(w) be the stationary distribution of and the joint stationary distribution of (s, a, w), respectively, under and B. The critic loss function L(d; is designed to simulate the Q-learning algorithm. Let THe be the function satisfying Tu. [Q](s, a, w) = (s s, a) + 7 Swes Q(s', , so (=o(s)) (s s, a.dds's Then, (I) states that Q"" is the solution to Q = Ty.[Q] Therefore, the critic trained to minimize the difference between Q& and Tpo [Q.]. This is achieved by minimizing
L(b:0)= = /
(Te, 1ol a. -o(o,, w)dddadw.
(4)
SES AEA wen
3

---

The max- max-min min objective function of the actor network
J(0, = S.e. Q (ol),
(5)
measures the performance of [4g under the uncertainty parameter W S approximated using the critic network Qo instead of the ground truth Q Similar to the standard actor-critic approach, we expect Qo to approach Q" as the critic loss is minimized. Therefore, we expect J (0, ";;" x²) approximates Saes Qua (s, ps(s), à d (s)ds once we obtain 6² argmingev L( 0).
Particularly, even if Qo- Q"a our our objective function [0, w;; Ø²) differs from Ju (0) with w in (2), in that the stationary distribution o² (s w) under fixed in Ju is replaced with (s( under w ev a. However, this change allows us to effectively utilize the replay buffer, which stores the interaction history, approximate the objective [(,, @; Ø). Moreover, if is concentrated at W, [ @; 6*) coincides with (O).
The framework of M2TD3 is designed to solve the tri-level optimization (3). The overall framework of M2TD3 follows that TD3. each episode, an uncertainty parameter is sampled from a. The training agent interacts with the environment MM using behavior policy B. At each time step t, the transition (st, ae Tr. s/ hr, w) is stored in the replay buffer, which denoted by B. The critic network Qa is trained at every time step, to minimize (4), with a mini-batch being drawn uniformly randomly from B. The actor network [ig and the worst-case uncertainty parameter w are trained in every T freq step to optimize (5). Note that uniform random sample (s, a, w)) taken from B can be regarded
Algorithmic Framework
sample from the stationary distribution ( a, w). Similarly, taken from B is regarded as a sample from (s). These facts allow approximation of the expectations in (4) and (5) using the Monte Carlo method with mini-batch samples uniformly randomly taken from B. An algorithmic overview of the proposed method is summarized in Algorithm 1. We have underlined the differences from the general off-policy actor-critic method in the episodic settings. A detailed description of M2TD3 is summarized in Algorithm 2 in Appendix A. As shown from Algorithm 1, the differences between the proposed method and the general off-policy actor-critic follows: (1) the introduction of an uncertainty parameter sampler, (2) definition and updating method of critic, and (3) updating method actor and uncertainty parameters.
Algorithm 1 Algorithmic overview of the proposed method
1: Draw uncertainty parameter ax
2: Observe initial state ~ ph
3: for t = to Tmmx do
4: # interaction
5: Select action (3(s)
6: Interact with MM with a, observe next state s', immediate reward r and termination flag
7: Store transition tuple (s, a, r, s', h, w)) in B
8:
if = then
9:
Reset uncertainty parameter at
Observe initial state p
11: else
Update current state s t- end if
13:
# learning
15:
Sample mini-batch {(s;, di,Yi, 8 h4, wi) } of M transitions uniform--radomlly from B
16:
Update the critic network by optimizing Equation (6)
17:
Update the actor network and the uncertainty parameter by optimizing Equation (7)
Update uncertainty parameter sampler a
end for
Critic Update The critic network update, namely, minimization (4), is performed by implement- ing the TD error-based approach. The concept is as follows. Let [(s;, ai, ri, 8/, wi) ) 2 C B be the mini-batch, and let 0, and d, be the actor and critic parameters at time step respectively. For each tuple, T Te, IQ J. (( a4, wi) is approximated by 3/1 = + 7 Qe, (s/, no, (s)). Then, L( 0z)

---

in (4) is approximated by the mean square error L())), such that
M
[(dr)
y - as,
(6)
M
Then, the update follows 0r+1 = 0z - oo Vo
We incorporate the techniques introduced in TD3 to stabilize the training: clipped double Q-learning, target policy smoothing regularization, and target networks. Additionally, we introduce the smoothing regularization for the uncertainty parameter so that the critic is smooth w.r.t. w. Algorithm 3 in Appendix A summarizes the critic update and the details of each technique are available in [7].
Actor Update Updating of the actor as well as the uncertainty parameter, namely, maximin optimization of (5), is performed via simultaneous gradient ascent descent [27], summarized in Algorithm 4 in Appendix A. Let Φi, 6g and W, be the critic, policy, and the uncertainty parameters, respectively, at time step t. Instead of the optimal d² in (3), we use de as its estimate. With the mini-batch {ii
1.4 C B, we approximate the max-min objective (5) follows:
M
J,(0,0)) = M Σ 2s (8is no (o)
We update and W as 0(+1 = 0, + Ao Vo Qx) and Wh+1 = wi - AW Vo (r), respectively.
introduce multiple uncertainty parameters. The motivation is twofold. One is to deal with multiple local minima of @0,,0;0"" σ*) w.r.t W. As the critic network becomes generally non-convex, there may exist multiple local minima of Je à; )²) w.r.t. W. Once e is stacked at a local minimum point, e.g., w", 0 may be trained to be robust around neighborhood of o* and perform poorly outside that neighborhood. The other is that the maximin solution (0* (²") of J+(0, Φ*) is not a saddle point, which occurs when the objective is non-concave in de [14]. Here, more than one W is necessary approximate the worst- case performance min (0, @; Ø²) around (0", o²) and standard simultaneous gradient ascent descent method fails converge. To relax this defect of simultaneous gradient ascent descent methods, we maintain multiple candidates for the worst w, denoted by wI...., Wy. Therefore, replacing ming J(6, ae; o²) with mink=1.....N J(0, Wy; o²) in (5) has no effect on the optimal 6; however, this change does affect the training behavior. Our update follows the simultaneous gradient ascent descent on mink=1.....N Je, Wx): : O < f + do Vo Vo Je wx)) and We + Wh- A Va, (ming=1l....N (0, ae)) for h: == 1,..,, N. Hence, 0 is updated against the worst Wk. and only the worst du is updated because the gradient w.r.t. the other We zero
For higher-stability performance,
For multiple uncertainty parameters to be effective, they must be distant from each other. Moreover, all are expected to selected as the worst parameters with nen-negligible frequencies. Otherwise. the advantage of having multiple uncertainty parameters is lessened. From this perspective, we introduce a refreshing strategy for uncertainty parameters. Namely, we resample Wy ((2)) if one of the following scenarios is observed: there exists 2e such that the distance do (#t, we) & dehre; the frequency Pkc of Wh being selected as the worst-case during the actor update no greater than Pthre-
Uncertainty Parameter Sampler The uncertainty parameter sampler a controls the exploration- exploitation trade-off in the uncertainty parameter. Exploration in Ω is necessary to train the critic network. If the critic network is not well trained over Ω, it is difficult to locate the worst W correctly. On the other hand, for J(0, d") in (5) coincide with (() for
W in (2), we require to be concentrated at W. Otherwise, the optimal 0 for (0, @;; o²) may deviate from that of (0). We design a as follows. For the first Trand steps, we sample the uncertainty parameter uniformly randomly on Q2, i.e., a U(2). Then, we set a = Σ Pk: (W Ew), where Σ,, is the predefined covariance matrix. We decrease Σng the time step increases. The rationale behind the gradual decrease of E,, is that the training of the critic network is still important after Trand to make the estimation of the worst-case uncertainty parameter accurate. Details are provided in Appendix C.
4 Related Work
Methods to handle model misspecification include (1) robust policy searching [26; 31; 23; 17; 12; 22], (2) transfer learning [36; 43; 32]. (3) domain randomization (DR) [37], and (4) approaches minimizing
5

---

worst-case sub-optimality gap [5; 21]. Robust policy searching includes methods that aim to obtain the optimal policy under the worst possible disturbance or model misspecification. i.e., maximization of gin (0) for an explicitly or implicitly defined Ω. Transfer learning is a method in which a policy is trained on source tasks and then fine-tuned through interactions while performing the target task. DR is a method in which policy is trained on source tasks that are sampled randomly from Ma- There are two types of DR approaches: vanilla and guided. In vanilla DR [37], the policy is trained on source tasks that are randomly sampled from pre-defined distribution on S.. In guided DR [44], the policy is trained on source tasks; however, the w distribution is guided towards the target task. Because we do not assume access to the target task, transfer learning and many guided DR approaches are outside the scope of this work. Some guided DR approaches, such as active domain randomization [24], do not access the target task consider worst-case optimization either. Vanilla DR can be applied to the setting considered here. However, the objective of vanilla DR differs from the present aim, i.e., maximize the performance averaged over S2 namely, (O) w U(D)), if the sampling distribution is (12). The approaches minimizing the worst-case sub-optimality gap [5; 21] do not optimize the worst-case performance, instead attempt to obtain policy that generalizes well on the uncertainty parameter set while avoiding too conservative performance, which often attributes to the worst-case performance optimization.
Some robust policy search methods, some adopt an adversarial approach to policy optimization. For example, Robust Adversarial Reinforcement Learning (RARL) [31] models the disturbance caused by an external force produced by an adversarial agent, and alternatively trains the protagonist and adversarial agents. Robust Reinforcement Learning (RRL) [26] similarly models the disturbance but has not been applied to the DRL framework. Minimax Multi-Agent Deep Deterministic Policy Gradient (M3DDPG) [17] has been designed to obtain robust policy for multi-agent settings. This method is applicable to the setting targeted in this study if only two agents are considered. The above approaches frame the problem as a zero-sum game between a protagonist agent attempting to optimize pe and an adversarial agent attempting to minimize the protagonist's performance, hindering the protagonist by generating the worst possible disturbance. Adv-Soft Actor Critic (Adv- SAC) [13] learns policies that are robust to both internal disturbances in the robot's 's joint space and those from other robots. Recurrent SAC (RSAC) [42] introduces POMDPs to treat the uncertainty parameter as an unobservable state. A DDPG based approach robustified by applying stochastic gradient langevin dynamics is proposed under the noisy robust MDP setting [15]. However, it has been reported in [42; 15] that their worst-case performances are sometimes even worse than their baseline non-robust approaches.
Saate-Adversaril--DRRR (SA-DRL) [48], and alternating training with learned adversaries (ATLA) [47] improve the robustness of DRL agents by using an adversary that perturbs the observations in SAMDP framework that characterizes the decision-male making problem in an adversarial attack state observations. They do not address the model misspecification in the reward and transition functions.
Other approaches attempt to estimate the robust value function, i.e., a value function under the worst uncertainty parameter. Among them, Robust Dynamic Programming (DP) [12] a DP approach. while the Robust Options Policy Iteration (ROPI) [23] incorporates robustness into option learning [35], which allows agents to learn both hierarchical policies and their corresponding option sets. ROPI is type of Robust MDP approach [39; 20). Robust Maximum A-posteriori Policy Optimization (R-MPO) [22] incorporates robustness in MPO [1].
Neither Robust MDP nor R-MPO require interaction with Ma for an arbitrary w E Ω. This can be advantageous in scenarios where the design of a simulator valid over Ω is tedious. However, these methods typically require additional assumptions for computating the worst-case for each value function update, such as the finiteness of the state-action space and/or the finiteness of Ω. To apply R-MPO to our setting, finite scenarios from Q for training are sampled, but the choice of the training scenarios affects the worst-case performance.
Additionally, to optimize the worst-case performance in the field of offline RL [40; 46; 41], offline RL attempts to obtain robust measures by introducing the principle of pessimism. Some studies that introduce the principle of pessimism in the model-free context [40; 41] and in the actor-critic context [46]. However, these do not compete with this study due to differences in motivation and because offline RL requires additional assumptions in the datasets [46; 41] and realizability [40].
Table 5 in Appendix D summarizes the robust policy search method taxonomy. Our proposed approach, M2TD3, considers the uncertainties in both the reward function and transition probability.

---

The state and action spaces, as well as the uncertainty set, are assumed to be continuous. Table 6 in Appendix D compares the related methods considering their action value function definitions. Methods that do not explicitly utilize the action value function are interpreted from the objectíve functions. Both M3DDPG and M2TD3 maintain the value function for the tuple of (s, a, w). However, M3DDPG takes the worst w' on the right-hand side, yielding more conservative policy than M2TD3 for our setting because we assume a stationary uncertainty model, whereas M3DDPG assumes time-varying uncertainty model in nature [28]. The value functions in RARL and M2TD3 are identical w is fixed. Particularly, w is not introduced to the RARL value function because it does not simultaneously optimize [4g and the estimate of the worst RARL repeats the & and w optimizations alternatively. In contrast, M2TD3 optimizes and w simultancously in a manner similar to the training processes of generative adversarial networks (GANs) [9]. Hence, require action value for each w E f
A difference exists between the optimization strategies of RARL. and M2TD3. As noted in the afore- mentioned section, both methods attempt to maximize the worst-case performance min (O). Conceptually, RARL repeats 0 +- argmaxg (0) and
argmin,. Ju (0). However, this opti- mization strategy fails to converge even if the objective function (0, w) + (0) is concave-convex, As example, consider the function (Ir, v) + y² x² axy. The RARL optimization strategy reads x < (a/2)yy and y + (aa/))z, which causes divergence if a > 2 Alternating updates also employed in other approaches such as Adv-SAC, SA-DRL, and ATLA. These share the same potential issue as RARL. M2TD3 attempts to alleviate this divergence problem by applying the gradient-basee max-min optimization method, which has been employed in GANs and other applications and analyzed for its convergence [25; 18].
5 Experiments
In this study, we conducted experiments on the optimal control problem using MuJoCo environments. Hence, we demonstrated the problems of existing methods and assessed the worst-case performance and average performance of the policy trained by M2TD3 for different continuous control tasks.
Baseline Methods We summarize the baseline methods adapted to our experiment setting, namely, DR, RARI, and M3DDPG.
DR: The objective of DR is to maximize the expected cumulative reward for the distribution o = (12) of w. In each training episode, w drawn randomly from (S2). and the agent neglects w when training and performing the standard DRL. For a fair comparison, we implemented DR with TD3 as the baseline DRL method.
We adapted RARL to our scenario by setting Ha : s + @ to the antagonist policy. RARL was regarded optimizing (2) for the worst E Ω; hence, the objective
RARL:
the same that of M2TD3. Particularly, the main technical difference between M2TD3 and RARL is in the optimization strategy as described in Section 4. The original RARL is implemented with Trust Region Policy Optimization (TRPO) [33]. but for a fair comparison, we implemented it with DDPG, denoted as RARL (DDPG). The experimental results of RARL (TRPO), the original RARL,, provided in Appendix F.
M3DDPG: By considering a two-agent scenario and state-independent policy Ma : 8 + @ as the opponent-agent's 's policy, we adapted M3DDPG to setting. M3DDPG is different from M2TD3 (and M2-DDPG below) even under a state-independent policy as described in Section 4. Because of the difficulty in replacing DDPG with TD3 in the M3DDPG framework, used DDPG as the baseline DRL.
Additionally, for comparison, we implemented our approach with DDPG instead of TD3, denoted by M2-DDPG. We also implemented a variant of M2TD3, denoted as SoftM2TD3, performing a "soft" worst-case optimization to achieve better average performance while considering the worst-case performance. The detail of SoftM2TD3 described in Appendix B.
AA simple way to mitigate this issue would be to early-stop the optimization for each step of RARL. However, the problem remains. The protagonist agent in RARL does not consider the uncertainty parameter in the critic. This leads to non-stationary training environment for both protagonist and adversarial agents. As the learning in a non-stationary environment is generally difficult [11]. RARL will be unstable. The instability of RARL has also been investigated in the finear-quarade system settings [49].
7

---

Table 1: Avg. + std. emror of worst-case performance R (11) over 10 trials for each approach
Environment
M2TD3
SonM2TD3
M2-DDRG
M3DDPG
KARL (DDPG)
DR (TD3)
(x1IP)
0.10
4.08
0.49
1.192
0.10
Ant 2 (X10))
1.13 0.11
0.12
0.95 4 0.20
-0.25
0.13
-I.77
0.00
1.61 0.13
Ant 3 (x (x'10")
0.10 0.10
0.07 + 0.20
-1.13 0.28
-1.38 0.22
-2.88 0.07
-0.32 + 0.03
HallCheetsh (x101)
0.14
3.24 + 0.08
2,21 0.25
0.313
-0.55
0002
3.19 = 0.88
HalfChectah (x100)
0.11
2.82 = 0.16
2.51 0.23
-0.58
00582000 0.06
-0.70 0.00
2.12 0.13
HalfCheetah (x110))
0.211
1.53 +:
1.20
0000
0.07
Hopper I [x10")
6.21 6:21::0:15
5.98 0.23
5.38 = 0.43
3.32700775
5.28
2.55
Hopper 10")
5.33
6.79 0.29
4.30
2.58
3.34 0.89
0.15
Hlopper 3 10%)
2.84 1: 0,25
1.98
0.22
2.25 0.29
0.73 0.10
1.64 0.46
2.10
0.35
HunanoodStanddup 1 (x10')
9.33 = 0.70
9.49
0.81
8.09 0.92
3.0000.01
5.29 0.45
9.68
0.60
HumanoidSSunddup 2 10*)
6.50 + 0.70
7.94 +: 0.90
6.24 0.54
6.37 0.72
5.78 0.73
7:31
0.78
unumnoodSSunduup 3 (x10')
6.20 1: 0.64
5.99 1 0.37
5.96 1 0.58
6.01 0.388
5.54 1.54.000 0.76
5.41
0.34
InverteePendulum (x10")
8.22 1.13
6.63
1.36
6.49
1.09
0.71
1,53 0.64
1.10
InvertedPendalum 2 (x10
3.56 d: 1.32
1.36 +: 0.30
1.10 + 0.62
0.02 0.00
0.02 0.00
0.57
0.02
Walker 1 (x10')
2,83
3.02
0.22
0.19
0.188
0.09 0.02
Walker 2 (x1)))
3.14 0.39
2.04 + 0.43
0.85 0.12
0.11
0.06 + 0.04
2.31
0.50
Walker 3 (x')))
1.94 1. 0.40
2.00 1: 0.35
0.82 1 0.13
0.28 0.00
0.00 0.02
1.32
0.34
Small HaltChoctah (x10")
5.27 = 0.12
5.07 = 0.14
4.51 = 0.18
1.26
-0.52
0.02
6.76 = 0.18
Small Hopper (x10')
2.88 + 0.32
2.23 + 0.32
1.40 + 0.19
1.39 0.21
0.51 0.12
3.42 - 0.11
M2TD3
SoftM2TD3
DR (TD3)
M3DDPG
RARL (DDPG)
M2-DDPG
1000
4000
Cumulaative ative reward
4000
800
3000
2000
600
2000
Cumu
1000
200
0
10
20
30
3
Uncertainty parameter
Uncertainly parameter
Uncertainty parameter
Figure 1: Cumulative rewards under trained policies for each uncertainty parameter w Ω. The average (solid line) and standard error (band) for each w E Ω as well as the worst average value (dashed line) are shown. Left: Ant 1, Middle: InvertedPendulum 1, Right: Walker 1.
Experiment Setting We constructed 19 tasks based on six MuJoCo environments and with 1-3 uncertainty parameters, summarized in Table in Appendix C. Here, Ω was defined an interval, which mostly included the default w values.
To assess the worst-case performance of the given policy under w E Ω, we evaluated the cumulative reward 30 times for each uncertainty parameter value wi,..., E Here, (1) was defined as the cumulative reward on Wд averaged over 30 trials. Then, Rwarsa = minigags Rs (R( was measured as an estimate of the worst-case performance of ju on Q. We also report the average performance Raverage (14) = _k_ (n).). A total of K uncertainty parameters wI...., WK for evaluation
drawn follows: for 1D w, chose K 10 equally spaced points the ID interval Ω; for 2D w. we chose 10 equally spaced points in each dimension of S thereby obtaining == 100 points; and for 3D w, chose 10 equally spaced points in each dimension of Ω thereby obtaining K 1000 points.
For each approach. Tmax were set to 2M, 4M, and 5M for the senarios with ID, 2D, and 3D uncertainty parameters, respectively. The final policies obtained were evaluated for their worst-case performances. For further details of the experiment settings, please refer to Appendix C.
trained the policy 10 times in each environment. The training time steps
Comparison to Baseline Methods Table 1 summarizes the worst-case performances of the policies trained by M2TD3, SoftM2TD3, M2-DDPG, M3DDPG, RARL, and DR. See also Table in Appendix E for the results of TD3 trained on the reference uncertainty parameters as baselines and Figure 2 in Appendix G for the learning curves. In most cases, M2TD3 and SoftM2TD3 outperfommed DR, and RARL, and M3DDPG, Figure shows the cumulative rewards of the policies trained by the

---

Table 2: Avg. + std. error of average performance Raverage (11) over 10 trials for each approach
Environment
M2TDS
SofM2TD3
M2-DDPG
M3DDPG
RARL (DDPG)
DR (TD3)
(x1I))
4.51
0.08
4.78
2,05 =
=: 0.14
-0.82 0.06
5.25 4: 0.10
Ant 2 (xI0))
5.41 = 0.05
5.56= 0.01
3.03 0.19
1.6 0.38
-1.00 0.06
6.32 + 0.09
Ant (x (x'1"))
2.86 + 0.22
2.98
0.23
0.30 + 0.46
0.33 + 0.22
-1 1.31 + 0.00
3.62 + 0.11
HallCheetah (x100)
3.89 = 0.06
1.00 = 0.05
5.50 0.12
1001 = 0.26
-0.66 = 0.02
5.93 + 0.18
HalfChectah 2 (x100)
1.36 = 0.00
1.02 = 0.07
3.91
= 0.08
0.77 = 0.12
-0.08 = 0.03
5.79 + 0.15
HalfCheetah (x100"
3.79 0.00
4.02
3.39
0.18
-0.11 0.10
5.54 +: 0.16
Hopper I (X10))
2.68 + 0.11
2.67
= 0.18
1.79 = 0.22
0.21
0.36 + 0.05
2.57 = 0.15
Hopper 2 [x 10")
2.51 = 0.07
2.20
= 0.12
1.49 = 0.15
1.15 0.11
0.13
1.89 + 0.08
Hopper 3 (x 10*)
0.85 0.07
0.79
0.04
0.87
0.08
0.49 0.11
0.47 1. 0.08
1.50 1: 0.07
HunanoidSianduu 1 (x100)
1,08 = 0.01
1.133
= 0.07
1.05
= 0.06
0.99
0.0G
0.77 = 0.06
1.12 = 0.05
HumanoidStanddup 2 10))
0.97 0.04
1.07
0.05
0.93 0.04
0.02 0.04
0.85 + 0.08
1.06 + 0.04
JumanoodSStundpp 3 (x100)
1.09 1 0.06
1.04
0.03
0.96 0.06
1.01 1. 0.04
0.87 1 0.07
1.04 1 0.07
InvertedPendullum I (x10')
9.66 = 0.25
9.17
0.54
8.79 0.87
4.61=0.12
0.81
9.32 + 0.11
InvertedPendulum 2 (x10F)
0.13 + 1.42
0.26
0.95
8.27 + 0.70
1.76 +: 0.51
3.07 4 0.66
9.18 d: 0.07
Walker 1 (x10')
3.70 1: 0.31
3:51
2.03 0.226
0.25
1.033
0.26
Walker 2 (x100)
4.72 = 0.12
1.37
= 0.52
2.39
0.20
1,63
0.22
0.26 4 0.05
4.34 = 0.31
Walker 3 (x10))
4.27 1. 0.21
4.21
0.30
2.48
0.24
1.65
0.15
0.21 1 0.007
4.48 1: 0.16
Small HallCheetah 1 (x10*)
6.00 1. 0.15
6.04
0.13
5.71
1 0.07
3.38
1. 0.34
-0.42 1. 0.01
5.11 1 0.17
Small Hopper 1 (x10")
2.96 + 0.31
2.44 0.31
1.57 + 0.21
1.53 + 0.21
0.56 + 0.1.3
3.44 + 0.10
Table 3: Avg. + std. error of worst-case performance (1) and average performance Raverage (11) on InvertedPendulum I obtained by M2TD3 variants over 10 trials
Environment
N=5
N=1
N=10
wo DRS
w/o PRS
wurst-case (x10=)
8.22
1.13
5.77
1.29
9.07
0.89
9.07
0.88
5.41 1.45
average (x10²)
9.66 = 0.25
8.10 + 0.90
9.89 + 0.11
9.08 + 0.88
6.90 + 1.25
six approaches for each of the w values on the Ant 1, InvertedPendulum 1, and Walker 1 senarios. See also Figure 4 in Appendix H for other senarios.
Comparison with DR: DR does not attempt to optimize the worst-case performance. In fact, it showed lower worst-case performance than M2TD3 and SoftM2TD3 in many scenarios because the obtained policy performs poorly some uncertainty parameters while it performs well on average. In the results of InvertedPendullum 1 shown in Figure 1, for example, the policy obtained by DR exhibited a high performance for wide range of w Q but performed poorly for small w. M2TD3 outperformed DR and the other baselines on those senarios. However, in some scenarios, such as HumanoidStandup 1, Small HalfCheetah 1, and Small Hopper 1, DR achieved better worst-case performance than M2TD3. This outcome may be because the optimization of worst-case function values is generally more unstable than that of the expected function values.
Comparison with RARL: M2-DDPG outperformed RARL in most senarios, with this performance difference originating from the different optimization strategies. As noted above, the optimization strategy employed in RARL often fails to converge. Specifically, as shown in Figure 1, RARL failed to optimize the policy not only in the worst-case but also average in Ant 1 and Walker 1 senarios. In the InvertedPendulum senario, RARL could train the policy for some uncertainty parameters, but not for the worst-case uncertainty parameter.
Comparison with M3DDPG: In many cases, M2-DDPG outperformed M3DDPG. Because M3DDPG considers stronger ad adversary than necessary (the worst w for each time step), it was too conservative for this experiment and exhibited lower performance.
Average Performance Although our objective is maximizing the worst-case performance, the average performance is also important in practice. Table 2 compares the average performance of the six approaches. Generally, DR achieved the highest average performance as expected. However, interestingly, M2TD3 and SoftM2TD3 achieved competitive average performances to DR on several senarios such as Hopper 1, 2, and Walker 1, 2. Moreover, Table I compared with Table 2 shows that a few times higher average performance than worst-case performance on several senarios such Ant 3 and Hopper 1-3.

---

Ablation Study Ablation studies were conducted on InvertedPendullm with the pole mass as the uncertainty parameters. M2TD3 with N = 5 was taken as the baseline. We tested several variants as follows: with different multiple uncertainty parameters (N = 1 and 10), without distance-based refreshing strategy (w/o PRS). Table shows the results. The inclusion of multiple uncertainty parameters and the probability- based refresh strategy contributed significantly to the worst-case performance and average performance, implying that both techniques contribute to better estimating the worst uncertainty parameter. Although DRS had little impact on the performance, prior knowledge DRS can be implemented by defining the distance and the threshold in the uncertainty parameter daeendepen,n which we did not implement here.
refreshing strategy (w/o DRS), and without a probability-
Small Uncertainty Set (Limitation of M2TD3) Small HalfCheetah 1 and Small Hopper were designed to have a smaller uncertainty parameter set than HalfCheetah and Hopper 1 and to reveal the effect of the size of the uncertainty parameter interval. As expected, smaller uncertainty parameter set resulted in higher worst-case (Table 1) and average (Table 2) performances, for all approaches. Because M2TD3 performs the worst-case optimization, it is expected to show better worst-case performance than DR, independently of the size of the uncertainty parameter set. However, in these senarios, DR showed better worst-case performance than that of M2TD3, while M2TD3 achieved competitive or superior worst-case performance to those of DR on HalfCheetah 1 and Hopper 1. This may be because the max- min optimization performed by M2TD3 resulted in sub- optimal policies. When the uncertainty parameter set is small, and the performance does not change significantly over the uncertainty parameter set, maximizing the average performance is likely to lead to high worst-case performance. The sub-optimal results obtained by M2TD3 is then dominated by the results obtained by DR. Therefore, the maxmin optimization in M2TD3 can be improed.
Evaluation Under Adversarial External Force Although we developed the proposed approach for the situation that the uncertainty parameter is directly encoded by w, we can extend it to the situation where the model misspecification is expressed by an external force produced by an adversarial agent as in [31]. The extended approach and its experimental result are given in Appendix J. Superior worst-case performances of M2TD3 over DR, RARL, and M3DDPG were observed for this setting as well.
6 Conclusion
In this study, we targeted the policy optimization aimed at maximizing the worst-case performance in a predefined uncertainty set. The list of the contributions are as follows. (i) We formulated the off-policy deterministic actor-critic approach to the worst-case performance maximization as a tri-level optimization problem (3). (ii) We developed the worst-case performance of M2TD3. The key concepts were the incorporation of the uncertainty parameter into the critic network and use of the simultaneous gradient ascent descent method. Different technical components were introduced to stabilize the training. (iii) We evaluated the worst-case performance of M2TD3 on 19 MuJoCo tasks through comparison with three baseline methods. Ablation studies revealed the usefulness of each component of M2TD3.
Acknowledgements
This research is partially supported by the JSPS KAKENHI Grant Number 19H04179 and the NEDO Project Number JPNP18002
References
[1] Abbas Abdolmaleki, Jost Tobias Springenberg, Yuval Tassa, Rémi Munos, Nicolas Heess, and Martin A. Riedmiller. Maximum a posteriori policy optimisation. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018
[2] Anurag Ajay, Jiajun Wu, Nima Fazeli, Maria Bauzá, Leslie Pack Kaelbling, Joshua B. Tenen- baum, and Alberto Redriguez. Augmenting physical simulators with stochastic neural networks:
10

---

Case study of planar pushing and bouncing. In 2018 IEEE/RSJ International Conference on In- telligent Robots and Systems, IROS 2018, Madrid, Spain, October 1-5, 2018. pages 3066-3073. IEEE, 2018.
[3] Erwin Coumans et al. Bullet physics library. Open source: bulleiphysics. org, 15(49):5, 2013.
[4] Constantinos Daskalakis, Stratis Skoulakis, and Manolis Zampetakis. The complexity of constrained min-max optimization. In Samir Khuller and Virginia Vassilevska Williams, editors, STOC 21: 53rd Annual ACM SIGACT Symposium on Theory of Computing, Virtual Event, Italy, June 21 21-25, 2021, pages 1466-1478. ACM, 2021.
[5] Michael Dennis, Natasha Jaques, Eugene Vinitsky, Alexandre M. Bayen, Stuart Russell, Andrew Critch, and Sergey Levine. Emergent complexity and zero-shot transfer via unsupervised environment design. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Floring, Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems 33: Anmual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.
[6] Omar Darwiche Domingues, Pierre Ménard, Emilie Kaufmann, and Michal Valko. Episodic reinforcement learning in finite mdps: Minimax lower bounds revisited. In Vitaly Feldman, Katrina Ligett, and Sivan Sabato, editors, Algorithmic Learning Theory, 16-19 March 2021, Virtual Conference, Worldwide, volume 132 of Proceedings of Machine Learning Research, pages 578-598. PMLR, 2021.
[7] Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In Jennifer G. Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pages 1582-1591 PMLR, 2018.
[8] Florian Golemo, Adrien Ali Taiga, Aaron C. Courville, and Pierre- Yves Oudeyer. Sim-to-real transfer with neural-augmented robot simulation. In 2nd Annual Conference on Robot Learning, CoRL 2018, Zürich, Switzerland, 29-31 October 2018, Proceedings. volume 87 Proceedings of Machine Learning Research, pages 817-828. PMLR, 2018.
[9] Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C, Courville, and Yoshua Bengio. Generative adversarial nets. In Zoubin Ghahramani, Max Welling, Corinna Cortes, Neil D. Lawrence, and Kilian Q Weinberger, editors, Advances in Neural Information Processing Systems 27: Annual Conference on Neural Information Processing Systems 2014, December 8-13 2014, Montreal, Quebec, Canada, pages 2672-2680, 2014.
[10] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In Jennifer G. Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmüssan, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pages 1856-1865. PMLR, 2018
[11] Shariq Iqbal and Fei Sha. Actor-atiention-ccittc for multi-agent reinforcement learning. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pages
PMLR, 2019.
[12] Garud N. Iyengar. Robust dynamic programming. Math. Oper. Res., 30(2):257-28. 2005.
[13] Pingcheng Jian, Chao Yang, Di Guo. Huaping Liu, and Fuchun Sun. Adversarial skill learning for robust manipulation. In IEEE International Conference on Robotics and Automation, ICRA 2021, Xi"
Xi'an, an, China, May 30 June 5, 2021, pages 2565-2561. IEEE, 2021.
[14] Chi Jin, Praneeth Netrapalli, and Michael I. Jordan. What local optimality nonconvex- nonconcave minimax optimization? In Proceedings of the 37th International Conference on Machine Learning, ICML. 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 4880 1880-4889, PMLR, 2020.
11

---

[15] Parameswaran Kamalaruban, Yu-Ting Huang, Ya-Ping Hsich, Paul Rolland, Cheng Shi, and Volkan Cevher. Robust reinforcement learning via adversarial training with langevin dynamics. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan- Tien Lin, editors, Advances Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6- 12, 2020, virtual, 2020.
[16] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
[17] Shihui Li, Yi Wu, Xinyue Cui, Honghua Dong, Fei Fang, and Stuart J. Russell. Robust multi-agent reinforcement learning via minimax deep deterministic policy gradient. In The Thirty- Third AAAI Conference on Artificial Intelligence, AAAI 2019, The Thirty-Firss Innovative Applications of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2019, Honolulu, Hawaii, USA, January 27 February 1, 2019, pages 4213 4220. AAAI Press, 2019.
[18) Tengyuan Liang and James Stokes. Interaction matters: A note on non-asymptotic local convergence of generative adversarial networks. In Kamalika Chaudhuri and Masashi Sugiyama, editors, The 22nd International Conference on Artificial Intelligence and Statistics, AISTATS 2019, 16-18 April 2019, Naha, Okinawa, Japan, volume 89 of Proceedings of Machine Learning Research, pages 907-915. PMLR, 2019.
[19] Timothy P. Lillicrap. Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In Yoshua Bengio and Yann LeCun, editors, 4th International Conference on Learning Repre- sentations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016.
[20] Shiau Hong Lim, Huan Xu, and Shie Mannor. Reinforcement learning in robust markoy decision processes. In Christopher J. C. Burges, Léon Bottou, Zoubin Ghahramani, and Kilian Q. Weinberger, editors, Advances in Neural Information Processing Systems 26: 27th Annual Conference on Neural Information Processing Systems 2013. Proceedings of meeting held December 5-8, 2013, Lake Tahoe, Nevada, United States, pages 701-709, 2013.
[21] Zichuan Lin, Garrett Thomas, Guangwen Yang, and Tengyu Ma. Model-based adversarial rect-reinforcement learning. In Hugo Larochelle. Marc' Aurelio Ranzato, Raia Hadsell, Maria- Florina Balcan, and Hsuan- Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference Neural Information Processing Systems 2020, NeurIPS 2020, December 12, 2020, virtual, 2020.
[22] Daniel J. Mankowitz, Nir Levine, Rae Jeong, Abbas Abdolmaleki, Jost Tobias Springenberg, Yuanyuan Shi, Jackie Kay, Todd Hester, Timothy A. Mann, and Martin A. Riedmiller. Robust reinforcement learning for continuous control with model misspecification. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia. April 26-30, 2020. OpenReview.net, 2020.
[23] Daniel J. Mankowitz, Timothy A. Mann, Pierre-Lue Luc Bacon, Doina Precup, and Shie Mannor. Learning robust options. In Sheila A. Mcllraith and Kilian Q Weinberger, editors, Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018, pages 6409-6416. AAAI Press, 2018
[24] Bhairav Mehta, Manfred Diaz, Florian Golemo, Christopher J. Pal, and Liam Paull. Active domain randomization. In Leslie Pack Kaelbling, Danica Kragic, and Komei Sugiura, editors, 3rd Annual Conference on Robot Learning, CoRI. 2019, Osaka, Japan, October 30- November 1, 2019, Proceedings, volume 100 of Proceedings of Machine Learning Research, pages 1162- 1176. PMLR, 2019.
12

---

[25] Lars M. Mescheder, Sebastian Nowozin, and Andreas Geiger. The numerics of gans. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. Vishwanathan, and Roman Garnett, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 1825-1835, 2017.
[26] Jun Morimoto and Kenji Doya. Robust reinforcement learning, Neural Comput., 17(2):335-35,, 2005.
[27] Vaishnavh Nagarajan and J. Zico Kolter. Gradient descent GAN optimization is locally stable. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio. Hanna M. Wallach, Rob Fergus. S. V. N.
Vishwanathan, and Roman Garett, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 5585-5595, 2017.
[28] Arnb Nilim and Laurent El Ghaoui. Robust control of markov decision processes with uncertain transition matrices. Oper. Res., 53(5):780-779, 2005.
[29] Xue Bin Peng. Marcin Andrychowicz, Wojciech Zaremba, and Pieter Abbeel. Sim-to-real transfer of robotic control with dynamics randomization. In 2018 IEEE International Conference on Robotics and Automation, ICRA 2018, Brisbane, Australia, May 21-25, 2018, pages 1-8. IEEE, 2018
[30] David Pfau and Oriol Vinyals. Connecting generative adversarial networks and actor-critic methods. CoRR, abs/1610 2016.
[31] Lerrel Pinto, James Davidson, Rahul Sukthankar, and Abhinav Gupta. Robust adversarial reinforcement learning. In Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, volume 70 of Proceedings of Machine Learning Research, pages 2817-2826. PMLR, 2017.
[32] Aravind Rajeswaran, Sarvjeet Ghotra, Balaraman Ravindran, and Sergey Levine. Epopt: Learning robust neural network policies using model ensembles. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Truck Proceedings. OpenReview.net, 2017.
[33] John Schulman, Sergey Levine, Pieter Abbeel, Michael I. Jordan, and Philipp Moritz. Trust region policy optimization. In Francis R. Bach and David M. Blei, editors, Proceedings of the 32nd International Conference on Machine Learning, ICML 2015, Lille, France, 6-11 July 2015, volume 37 of JMLR Workshop and Conference Proceedings, pages 1889-1897. JMLR.org, 2015.
[34] Russell Smith. Open dynamics engine, 2008 http://www.ode.org/.
[35] Richard S. Sutton, Doina Precup, and Satinder P. Singh. Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning. Artif. Intell., 112(1-2):111-11, 1999.
[36] Matthew E. Taylor and Peter Stone. Transfer learning for reinforcement learning domains: A survey. Mach. Learn. Res., 10:1633-1685, 2009.
[37] Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojciech Zaremba, and Picter Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world. In 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2017, Vancouver, BC, Canada, September 24-28, 2017, pages 23-30. IEEE, 2017.
[38] Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2012, Vilamoura, Algarve, Portugal, October 7-12, 2012, pages 5026-5033. IEEE, 2012.
[39] Wolfram Wiesemann, Daniel Kuhn, and Berç Rustem, Robust markov decision processes. Math. Oper. Res., 38(1): 2013.
13

---

[40] Tengyang Xie, Ching-An Cheng, Nan Jiang, Paul Mineiro. and Alekh Agarwal. Bellman- consistent pessimism for offline reinforcement learning. In Mare' Aurelio Ranzato. Alina Beygelzimer, Yann N. Dauphin, Perey Liang, and Jennifer Wortman Vaughan, editors, Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NewrlPS 2021, December 6- 2021, virtual, pages 0683366994 2021.
[41] Tengyang Xie and Nan Jiang. Batch value-function approximation with only realizability. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 1104--1443, PMLR, 2021.
[42] Lebin Yu, Jian Wang, and Xudong Zhang. Robust reinforcement learning under model misspec- ification. CoRR, abs/2103.15331 2021.
[43] Wenhao Yu, C. Karen Liu, and Greg Turk. Policy transfer with strategy optimization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
[44] Wenhao Yu, C. Karen Liu, and Greg Turk. Policy transfer with strategy optimization. In 7th International Conference on Learning Representutions, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
[45] Wenhao Yu, Jie Tan, C. Karen Liu, and Greg Turk. Preparing for the unknown: Learning a univer- sal policy with online system identification. In Naney M. Amato, Siddhartha S. Srinivasa, Nora Ayanian, and Scott Kuindersma, editors, Robotics: Science and Systems XIII, Massachuseils Institute of Technology, Cambridge, Massachuseuts USA, July 12-16, 2017, 2017.
[46] Andrea Zanette, Martin J. Wainwright, and Emma Brunskill. Provable benefits of actor-critic methods for offline reinforcement learning. In Mare' Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors, Advances in Neural Information Processing Systems 34: Anmual Conference Neural Information Processing Systems 2021, NeurlPS 2021, December 6-14, 2021, virtual, pages 13626-13640, 2021.
[47] Huan Zhang, Hongge Chen, Duane S. Boning, and Cho-Jui Hsieh. Robust reinforcement learning on state observations with learned optimal adversary. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
[48] Huan Zhang, Hongge Chen, Chaowei Xiao, Bo Li, Mingyan Liu, Duane S. Boning, and Cho-Jui Hsich. Robust deep reinforcement learning against adversarial perturbations state observations. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.
[49] Kaiqing Zhang, Bin Hu, and Tamer Basar. On the stability and convergence robust adver- sarial reinforcement learning: A case study on linear quadratic systems. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan Tien Lin, editors, Ad- vances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 12, 2020, virtual, 2020.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] Please see Section 3 and Section 5.
(b) Did you describe the limitations of your work? [Yes] Please see paragraph titled "Small Uncertainty Set" in Section 5 and Section 6.
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
14

---

(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either in the supplemental material or as a URL)? [Yes] Please see Appendix A.
(b) Did you specify all the training details (e.g., data splits. hyperparameters, how they were chosen)? [Yes] Please see Section and Appendix C.
(c) Did you report error bars (e.g., w.r.. the random seed after running experiments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] Please see Appendix C.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes] Code licence part of the repository.
(c) Did you include any new assets either in the supplemental material or as URL? [Yes] We include our code in the Appendix A.
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you deseribe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
15


Reviews:
[{"review_id": "v94cRdQ-OR8", "reviewer": "Reviewer_ViYo", "summary": "This paper studies off-policy reinforcement learning, where there is a simulation environment with uncertainty parameters and the set of their possible values. The authors provided a solution of off-policy Actor-Critic which optimizes the worst-case performance and evaluated the proposed approach empirically.", "questions": "It would be helpful if the author could address my two main concerns: 1) lack of discussion/comparison of existing offline RL works, especially for the ones that share the same idea; 2) the empirical evaluation doesn't seem to match the goal/motivation of this work well.", "limitations": "see \"Strengths And Weaknesses\"", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This paper has a good motivation, which aims to utilize the uncertainty estimation in order to guarantee the well-behaving of the learning policy from the simulated environment to the real environment. The proposed solution in this paper is to that is to optimize the worst-case performance over policies.\n\n\nThe idea of optimizing the worst-case performance is actually not a novel idea in offline RL. There exist several existing works [1,2] that have explicitly discussed the same idea. Therefore, I feel it is needed to include the discussion of offline RL in the related work section. \n\nIn addition, the empirical evaluation doesn't seem to be able to verify whether the proposed algorithm achieves the main goal of this work---migrating the learning policy from the simulated domain to the real environment. It seems the experiments only focus on the final performance of some fixed environment, rather than the migration from simulation to the real world.\n\nReferences:\n\n[1] Bellman-consistent Pessimism for Offline Reinforcement Learning\n\n[2] Provable benefits of actor-critic methods for offline reinforcement learning", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "uwKFfelOPxK", "reviewer": "Reviewer_tGzm", "summary": "In this paper, the authors consider the gap from simulation environment to the real-world environment and aim to improve the robustness of policy. From a perspective of the worst-case performance on the uncertainty parameter set, they develop M2TD3 to solve a max-min optimization problem. In my opinion, this paper is well organized and the result is of abundant experiments. However, some problems should be concerned, mainly about the solution of the max-min optimization problem, the experiment settings and the selection of the baselines.", "questions": "Please see the questions in Weaknesses.", "limitations": "I think that this work would be more convincing if the authors can test the performance of M2TD3 with constantly changed uncertainty, because most of previous works are compatible with this setting. Further, what would be the impact on the algorithm if the fixed $\\omega$ setting in one episode is removed. ", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\tClear problem formulation and very well related work classification;\n\tAbundant experiments and demonstratable results;\n\tClear structure and understandable writing.\n\nWeaknesses\n\tThe authors claim that they develop a robust policy for sim2real scenario, but they suppose that the uncertainty parameter $\\omega$ is fixed in each episode. I think this setting is more like transfer learning, which requires that the policy can adapt to tasks with different parameters. To make the policy more robust, the authors should consider the continuous changed uncertainty at each time step in one episode. And, in RARL, the uncertainty is present by an adversarial policy, which also interacts with the environment constantly. Thus, please explain the different setting compared with general robust RL.\n\tWhen solving the max-min optimization problem (3), the authors consider that the protagonist policy (robust policy) can observe the uncertainty (the same as the action of the adversarial policy defined in RARL) and add the uncertainty $\\omega$ into the Q value function. In a sense, this observable uncertainty modelled in the Q value function makes the designed algorithm more like the domain randomization methods. And, the competitive results from DR(TD3) has the closest performance even DR(TD3) can exceed M2TD3 in the average performance. I think that the authors should add more detail comparison and analysis with DR(TD3).\n\tThe proposed refresh strategy and the uncertainty parameter sampler may be some common methods for stable training policy and exploration noise for environment. \n\tSome new works should be considered.\n[1] Jian P, Yang C, Guo D, et al. Adversarial Skill Learning for Robust Manipulation[C]//2021 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2021: 2555-2561.\n[2] Yu L, Wang J, Zhang X. Robust Reinforcement Learning under model misspecification[J]. arXiv preprint arXiv:2103.15370, 2021.\n[3] Zhang H, Chen H, Boning D, et al. Robust Reinforcement Learning on State Observations with Learned Optimal Adversary[C]//International Conference on Learning Representation (ICLR). 2021.\n[4] Kamalaruban P, Huang Y T, Hsieh Y P, et al. Robust reinforcement learning via adversarial training with langevin dynamics[J]. Advances in Neural Information Processing Systems, 2020, 33: 8127-8138.\n[5] Zhang H, Chen H, Xiao C, et al. Robust deep reinforcement learning against adversarial perturbations on state observations[J]. Advances in Neural Information Processing Systems, 2020, 33: 21024-21037.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "WcXxbRzd5JI", "reviewer": "Reviewer_Lcvb", "summary": "This work is concerned with robustness to model misspecification in reinforcement learning. The authors propose a modified objective function to the Twin-Delayed Deep Deterministic Policy Gradient (TD3) algorithm, whereby the critic takes as input an uncertainty parameter of the environment, and the actor optimizes the policy for a worst-case realization of the uncertainty parameters in a predetermined set. The latter optimization is done simultaneously in a gradient descent-ascent fashion as opposed to prior work (Robust Adversarial RL — RARL) which performs each optimization completely alternatingly. The work is validated by thorough experimental results that show improved worst-case performance of the proposed approach compared to a variety of relevant baselines.\n", "questions": "Some clarifying questions that I had for the authors:\n- L68: what’s the difference between $q_\\omega$ and $p_\\omega$ defined on L60? \n-  Could you comment on the differences between (5) and (2). I think there should be some form of correction factor in Eqn (5) as $\\eta$ is different from $\\omega$ sampled from $\\alpha$ (similar to importance sampling in off-policy). I'm also wondering if using the state distribution according to randomly sampled $\\omega$ would lead to instability because the underlying MDP is always changing, as compared to Eqn (2) where $\\omega$ is fixed for the state data and the Q function.\n- L229 - L235: Why aren’t all $\\eta$’s updated at each step instead of only the worst one? Is it to keep diversity of $\\eta$’s?\n\n", "limitations": "I think the authors do a reasonable job of discussing and addressing limitations of the work. \n- One other limitation that I see is in designing of the set $\\Omega$ such that $\\omega^* \\in \\Omega$, and also to have $\\omega$ be a configurable parameter in the simulation environment. This is common practice in robust approaches, however.\n\n- Also as noted by the authors, small dimensions (1-3) of uncertainty parameters were tested due to the exponentially growing size of the uncertainty parameter set with increasing dimensions. Two improvements in this area could be to design the set of parameters in a more directed manner (than a uniform cover), or to do the search in this space could more efficiently than exhaustive search. ", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Application of robust RL to the deep learning regime is an ongoing open problem that has been addressed to some extent in this work. I find the approach reasonable, well-explained, and placed in the context of the existing literature. The empirical analysis is also quite thorough. On the other hand, I feel a weakness of the work is in its similarity to RARL. Although the proposed approach of simultaneous gradient ascent-descent is reasonable, an advantage of this over RARL would be in better optimization. However this is not analyzed theoretically or experimentally. The authors do mention that the difficulty in optimization may explain why Domain Randomization (DR) demonstrates better worst-case performance than M2TD3, and I feel that this highlights a pretty big problem with the proposed approach: the optimization can become too difficult to achieve the stated objective. As a result, I feel that analyzing this aspect of the work would greatly strengthen the reasons for adoption of this algorithm.\n\nSummary of points and some additional notes are provided below. \n\nStrengths:\n- Thorough empirical analysis\n- Details of algorithm well-explained and sensible\n- I felt that the authors explained the limitations and did sufficient comparison with existing literature\n\nWeaknesses:\n- Need to be able to set simulator settings randomly, and we get robustness only to these settings. This is common in the literature but it still faces similar issues as domain randomization, etc., in that simulation to real setting changes need to be known in advanced.\n- There is no theoretical analysis of the convergence properties of the proposed algorithm for simple settings. These results may be impossible in the deep neural network regime but it would inspire confidence in the algorithm to have some result showing that we can expect convergence with direct or linear parameterization (similar to those done by [1], [2], etc)\n\nWriting:\n- Would probably be better to put Related Works after describing the algorithm because it was hard to understand all the details that were being compared without knowing the algorithm first. \n- L228: $\\max_\\eta J_t$ should be $\\min_\\eta J_t$ ?\n- Notationally I found the use of $\\omega$ and $\\eta$ as the uncertainty parameters confusing. I think it would make the reading easier if only one letter is used. \n\nMinor:\n- abstract has a few grammatical errors\n- L37 — extra “has been reported”\n- L265: appears to have a typo “optimize theta and omega optimizes (2) …”\n\n[1] Agarwal, Alekh, et al. \"Optimality and approximation with policy gradient methods in markov decision processes.\" Conference on Learning Theory. PMLR, 2020.\n[2] Qiu, Shuang, et al. \"On finite-time convergence of actor-critic algorithm.\" IEEE Journal on Selected Areas in Information Theory 2.2 (2021): 652-664.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "JB57hzXH3wQ", "reviewer": "Reviewer_g5qE", "summary": "This work studies the problem of optimizing the worst-case performance in a given uncertainty parameter set.  They propose an off-policy DeepRL algorithm for this problem setting (M2TD3) that solves a tri-level optimization problem using GAN-like gradient descent approach. They show the effectiveness of their proposed approach on a variety of MuJoCo tasks against different baselines. ", "questions": "- Can you please expand on the motivation on the formulation? Please check the questions in the weaknesses above. \n- Is there any formal claim of statement in Line 182? \n- The part about the uncertainty parameter sampler is not clear. Can you please expand on it more (lines 248-251)?", "limitations": "The authors should discuss more the implications of assumption in lines 82-84. ", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Stregths \n- Strong empirical results: The proposed approach has strong empirical results, and the comparison with the baselines seems thorough. \n\n# Weaknesses \n- Formulation: Certain aspects of the formulation are not explained in detail. For instance, the choice of only considering deterministic policies when the behaviour policy can be stochastic is not clear. Additionally, the assumption in lines 82-84 should be introduced as a separate assumption and accompanied by proper motivation. In the current form, there is a disconnect between the motivation in the introduction (Real-world cases) and the formulation and the empirical studies. \n\n- Presentation: Most of the major contributions of the work seem to be algorithmic. However, most of the algorithmic details have been pushed to the Appendix. I think the related work and some of the tables could have been pushed to the Appendix to introduce the algorithm in the main text. Additionally, another baseline (softM2TD3) seems to work really well on these tasks however not many details are provided about it in the main text.\n\n- Related work: I think RobustMDPs should also be included in the related work.\n\n- Hyper-parameter overhead and reproducibility: The paper introduces many algorithmic components and new hyper-parameters, but there is not much discussion of the overhead of introducing these hyper-parameters ($\\lambda_{\\theta, \\eta}, d_{tre}$). There is also no accompanying code for the work. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept