Robust Reinforcement Learning in Continuous Control Tasks with Uncertainty Set Regularization
Authors: Yuan Zhang
Jianhong Wang
Joschka Boedecker
Venue: CoRL 2023 Poster (2023)
Keywords: Reinforcement Learning
Robustness
Continuous Control
Robotics

Abstract: Reinforcement learning (RL) is recognized as lacking generalization and robustness under environmental perturbations, which excessively restricts its application for real-world robotics. Prior work claimed that adding regularization to the value function is equivalent to learning a robust policy under uncertain transitions. Although the regularization-robustness transformation is appealing for its simplicity and efficiency, it is still lacking in continuous control tasks. In this paper, we propose a new regularizer named $\textbf{U}$ncertainty $\textbf{S}$et $\textbf{R}$egularizer (USR), to formulate the uncertainty set on the parametric space of a transition function. To deal with unknown uncertainty sets, we further propose a novel adversarial approach to generate them based on the value function. We evaluate USR on the Real-world Reinforcement Learning (RWRL) benchmark and the Unitree A1 Robot, demonstrating improvements in the robust performance of perturbed testing environments and sim-to-real scenarios.

Robust Reinforcement Learning in Continuous Control Tasks with Uncertainty Set Regularization
Yuan Zhang Neuroroboties Lab
Jianhong Wang" Center for Fundamentals University of Manchester j anhong wangSmanchesterr ac, uk
University of Freiburg yzhang@os. cei-freiburg.de
Joschka Boedecker
Neurorobotics Lab University of Freiburg
jboedeckêcs. uni uni-freiburg.de
Reinforcement learning (RL) is recognized as lacking generalization and robustness under environmental perturbations, which excessively restricts its application for real-world robotics. Prior work claimed that adding regularization to the value function is equivalent to learning robust policy under uncertain tran- sitions. Although the regularizatioo--rouster simplicity and efficiency, it is still lacking in continuous control tasks. In this paper, we propose a new regularizer named Uncertainty Set Regularizer (USR), to formulate the uncertainty set on the parametric space transition function. To deal with unknown uncertainty sets, we further propose novel adversarial approach to generate them based on the value function. We evaluate USR on the Real-world Reinforcement Learning (RWRL) benchmark and the Unitree A1 Robot, demonstrating improvements in the robust performance of perturbed test- ing environments and sim-to-real scenarios.
Abstract:
transformation is appealing for its
Keywords: Reinforcement Learning, Robustness, Continuous Control, Robotics
1 Introduction
Reinforcement Learning (RL) is powerful algorithmic paradigm used solve sequential decision- making problems and has resulted in great success in various types of environments, e.g., mastering the game of Go [1], playing computer games [2] and operating smart grids [3]. The majority of these successes rely on an implicit assumption that the testing environment is identical to the training enviroument. However, this assumption is too strong for most realistic problems, such as controlling a robot. more detail, there are several situations where mismatches might appear between training and testing environments in roboties: (1) Parameter Perturbations indicates that a large number of environmental parameters, e.g. temperature, friction factor could fluctuate after deployment and thus deviate from the training environment [4]; (2) System Identification estimates transition function from limited experience. This estimation is biased compared with the real-world model [5]; (3) Sim- to- Ro-Real learns a policy in a simulated environment and performs on real robots for reasons of safety and efficiency [6]. The difference between simulated and realistic environments renders sim-to-real a challenging task.
In this paper, aim to model the mismatch between training environments and testing environ- ments robust RL problem, which regards training environments and testing environments as candidates in an uncertainty set including all possible environments. Robust Markov Decision Pro-
"Corresponding author.
7th Conference Robot Learning (CoRL. 2023), Atlanta, USA.

---

cesses (Robust MDPs) [7, 8] is a common theoretical framework to analyze the robustness of RL algorithms. In contrast to the vanilla MDPs with a single transition model P(s'|s, a); Robust MDPs consider an uncertainty set of transition models = {P] to describe the uncertain environments. This formulation general enough to cover the various scenarios for the robot learning problems aforementioned.
The aim of Robust RL is to learn policy under the worst-case scenarios among all transition mod- els P E P, named robust policy. If a transition model P is viewed as an adversarial agent with the uncertainty set as its action space, one can reformulate Robust RI. as a zero-sum game [9]. In general, solving such a problem is an NP-hard problem [8, 10], however, the employment of Segendre-Fenche transform can avoid excessive mini-max computations through converting mini- mization over the transition model regularization on the value function. Furthermore, it enables more feasibility and tractability to design novel regularizers to cover different types of transition uncertainties. The complexity of these palue-function-uee the state space, which leads to nontrivial extension to continuous control tasks with infinitely large state space. This directly motivates the work of this paper. Due to the page limit, we conclude other related work in Appendix A.
regularizers increases with the size of
We now summarize the contributions of this paper: (1) the robustness-regularizziio duality method is extended to continuous control tasks in parametric space; (2) the Uncertainty Set Regularizer (USR) on existing RL. frameworks is proposed for learning robust policies; (3) the value function is learnt through an adversarial uncertainty set when the actual uncertainty set is unknown in some scenarios; (4) the USR is evaluated on the Real-world Reinforcement Learning (RWRL) benchmark, showing improvements for robust performances in perturbed testing environments with unknown uncertainty sets; (5) the sim-to-real performance of USR is verified through realistic experiments on the Unitree A1 robot.
2 Preliminaries
Robust MDPs. The mathematical framework of Robust MDPs [7, 8] extends regular MDPs in order to deal with uncertainty about the transition function. A Robust MDP can be formulated as a 6-tuple (S,APP. r, Y), where S,A stand for the state and action space respectively, and (ss : S x A - TR stands for the reward function. Let Ås and A be the probability measure on S and A respectively. The initial state is sampled from an initial distribution μo E As. and the future rewards are discounted by the discount factor 7 E [0, 1]. The most important concept in robust MDPs is the uncertainty set IP {P(s'|s, a)} that controls the variation of transition function P S x A As. compared with the stationary transition function P in regular MDPs. Let (T( : S + Au) be the policy space; the objective of Robust RL can then be formulated as minimax problem such that
4.00
x ( a4)
J==
= max min Ex,p Rell PEP
Σ
Robust Bellman Equation. While Wiesemann al. [8] has proved NP-hardness of this mini- max problem with an arbitrary uncertainty set, most recent studies [7, 9, 10, 11, 12, 13, 14, 15, 16] approximate it by assuming a rectangular structure on the uncertainty set, i.c., P = X (s,a))SxxA A Paa,, where P.,. = (P(s'Is.a) (s"( in short) denotes the local uncertainty of the transition at a). In other words, the variation of transition is independent at every (s,a) pair. Under the assumption of rectangular uncertainty set, the robust action value Q"T (s, a) under policy must satisfy the following robust version of the Bellman equation [17] such that
Q* a) = r(s,a) min Y (s'
P.oEP..
where (=ss==E.. T(d'ls')Q .'). Nilim and Ghaoui [11] have shown that robust Bellman operator admits a unique fixed point of Equation 2, the robust action value Q" (s.a).

---

Saansstess-Regulaiizattee Duality. Solving the minimization problem in the RHS of Equation
2 can be further simplified by the Legendre-Fencher transform [18]. For an arbitrary function f
X + RR its convex conjugate is f (x=") == sup {(=" (=)",))> - f())=
I E X). Define 8z,, (P.a) =
Psn E Pen and +oo otherwise, Equation can transformed to its convex conjugate (refer to
Derman et al. [10] for detailed derivation) such that
Q (s, a) r(s, a) + min
Psa + óz.. (Pea) = rr.r a) -p. (-V
(3)
Paa
The transformation implies that the robustness condition transitions can be equivalently expressed as a regularization term on the value function, which is referred to as the robustnes-reggulaizain regularizatinn duality. The duality can extensively reduce the cost of solving the minimization problem over in- finite transition choices and thus is widely studied in the robust reinforcement learning research community [19, 20, 21].
As special case, Derman et al. [10] considered L2 norm uncertainty set transitions, i.e., P.a (Paa + a.Ps. [|asall2 S 1}, where Psa is usually called the nominal transition model. It can represent the prior knowledge of a transition model or a numerical value of the training environ- ment. The uncertainty set implies that the transition model is allowed to fluctuate around the nominal model with some degree a. Therefore, the corresponding Bellman equation in Equation becomes Q (s. a) = rss, a) + E. .. Psa (=') (s')-ol[V*) -=lla (-)1l2. Similarly, the L1 norm has also been used as uncertainty set on transitions [14], ie., P....... = {P... +aPsa [|Paalla I 1}, and the Bellman equa- tion becomes the form such that QT a) = r(s, a) + +7 Psa(s') (s)) - a max, (s')) This robustnes--rguularizaaion duality works well in the finite state space but the extension to the infinite state space is still question. We claim that such an extension is non-trivial, since both regularizers [|VT(-))l and max [V*(s') need to be calculated on the infinit-dimensionn tor (-). In this work, we extend this concept to the continuous state space which is critical characteristic in robotics.
vec-
3 Uncertainty Set Regularized Robust Reinforcement Learning
Having introduced the robustness-reguulaizzii the continuous state space Section 2, here, we will first present novel extension to the continuous state space with the uncertainty set defined on the parametric space of transition function. We will then utilize this extension to derive robust policy evaluation method that can be directly plugged into existing RL algorithms. Furthermore, to deal with the unknown uncertainty set, we propose the adversarial uncertainty set and visualize it in a simple moving-to-targes task.
duality and the difficulties regarding its extension to
3.1 Uncertainty Set Regularized Robust Bellman Equation (USR-RBE)
For environments with continuous state space, the transition model P(s'|s. [s, a) is usually represented a parametric function P(s'|s, ss a; w), where w denotes the parameters of the transition function. Instead of defining the uncertainty set on the distribution space, we directly impose perturba- tion on within a set P.p. Consequently, the robust objective function (Fquation 1) becomes
J en minwest.. P(s'ss Ets y'r(st, at) We further assume that the parameter w fluctuates around nominal parameter w. such that w w + w, with w being fixed parameter and w E Ωg {w- w|w Pw' being the perturbation part. Inspired by Equation 3, we can derive a robust Bellman equation on the parametric space for continuous control problems as shown in Proposition 1.
Proposition 3.1 (Uncertainty Set Regularized Robust Bellman Equation) Suppose the uncer- tainty set of Qw (i.e., the uncertainty set of w w T is Poy),, the robust Bellman equation on the parametric space can be represented as follows:
(", = r(s, +7 /. P(s'|s, V) -7
-7 - - P
(4)
3

---

where )g..(w)) is the indicator functiio that equals 0 if w Sw and +oo otherwise, and b (w') is the convex dual function of ōq... (w.
The proof is presented in Appendix B.1. Intuitively, Proposition 3.1 shows that ensuring robust- ness on parameter w can be transformed into regularization on action value Q* a) that re- lates to the product of the state value function (*(s') and the derivative of the transition model Vu a; ti. Taking the L2 uncertainty
(also used in Derman et al. [10]) special case, Le., Qw {w + ani: ||のl|| S 1), where tü stands for the parameter of the nominal transition model P(s"|s,a'; w)) the robust Bellman equation in Proposition 3.1 becomes
S. ( (5)
(= r((,,)) +
3.2 Uncertainty Set Regularized Robust Reinforcement Learning (USR-RRL.
To derive a practical robust RL algorithm using the proposed USR-RBER we follow the policy iter- ation framework [22] commonly used in RL research. Regarding the policy evaluation procedure, Theorem 3.2 proposes operator action value Q and ensures its convergence to a unique fixed point by recursively running this operator. The proof can be found in Appendix B.2. Intuitively, this theorem indicates that one can acquire a robust action value given a certain policy and uncer- tainty set if the discount factor and the coefficient in the uncertainty set a are properly set, which satisfied the practical implementations. As for the policy improvement procedure, all exist- ing techniques (e.g. policy gradient methods) can be adopted to optimize the policy. By iterating the policy evaluation and improvement cycle, the policy will eventually converge an equilibrium trading off optimality and robustness.
Theorem 3.2 (Convergence Robust Policy Evaluation) For any policy DA the follow- ing operator T can reach unique fixed point as the robust action value Q" i s
+ m
S. P(s'ls, a; W)lldd' 11
(s.a / P(s' p(s(ss,a:0)v a a
where (*(())= = J5. (o(a'ls)((s)) a')da'.
practical concern on this algorithm is that calculating USR-RBE requires knowledge of the tran- sition model P(s' P(s'|s a; w). This naturally applies to model-based RL, as model-based RL learns point estimate of the transition model P(s' 8s, a; w) by maximum likelihood approaches [23]. For model-free RL, we choose to construct local Gaussian model with mean as parameters inspired by previous work [24]. Specifically, suppose that one can access a triple (state s, action a and next state c)) from the experience replay buffer, then a local transition model P(s'|s, (s, a; w) can be modelled as a Gaussian distribution with mean æ and covariance Σ (with Σ being a hyperparameter), i.e., the nom- inal parameter w consists of (ær, E). With this local transition model, we now have the full knowl- edge of P(s'Is, [ss a; w) and V Ps''(s' (s, a; w), which allows us to calculate the RHS of Equation 5. To further approximate the integral calculation in Equation 5, sample M points {s1 S2 -.-, s'M) from the local transition model P(s'|s, a; w) and use them to approximate the target action value
by Q (s, a) R a(),a) + ΣM (<())) a V P(slls, V) ((sl///((ssss,, 3)). where w = (x. Σ). With this approximation, the Bellman operator guaranteed to converge to the robust value, and policy improvement is applied to robustly optimize the policy. We explain how to incor- porate USR-RBE into classic RI. algorithm called Soft Actor Critic (SAC) [25] in Appendix B.3.
3.3 Adversarial Uncertainty Set
The proposed method in Section 3.2 relies on the prior knowledge of the uncertainty set of the para- metric space. The Lp norm uncertainty set is most widely used in the Robust RL and robust optimal

---

control literature. However, such a fixed uncertainty set may not sufficiently adapt to various pertur- bation types. The Lp norm uncertainty set with its larger region can result an over-conservative policy, while the one with a smaller region may lead to a risky policy. this section, we learn an adversarial uncertainty set through the agent's policy and value function to avoid the above issues.
Generating the Adversarial Un- certainty Set. The basic idea of the adversarial uncertainty set is to provide an appropriate uncer-
(s,a)
s'
V(s)
V.s';
d(w)
sample
forward
backward
normalize
generate
tainty range parameters that are more sensitive the value func- tion, which is naturally measured by the derivative. The agent learn- ing based on such an adversarial uncertainty set is easier to adapt to the various perturbation types of parameters. We generate the adver- sarial uncertainty set in a 5-step procedure as illustrated in Figure 1, (1) sample next state s' ac- cording to the distribution P(-|s, a;; w), given the current state and action a; (2) forward pass by calculating the state value (ss') at next state 8'; (3) backward pass by using the reparame- terization trick [26] to compute the derivative g(w) = V'''''' w); (4) normalize the derivative by d( ( W (/) g(w), (5) generate the adversarial uncertainty set Q.,. {w am || d( 1 < 1}. The pscudo-code to generate the adversarial uncertainty set is explained in Ap- pendix B.4 Algorithm 2.
fg
Figure 1: Illustration of the procedure for generating an ad- versarial uncertainty set.
a(4.)
(
a"(-88, 00.6)
e(0.0)
()) Task: moving the target
(b) Non unceriaiityy set
(c) L2 uncertainty set
[d] L1 uncertainty set uncertainty set
(e) Adversarial uncertainy set
training parameters
:tetstng parameters
Figure Illustration of different types of uncertainty sets to investigate their characteristics.
Characteristics of the Adversarial Uncertainty Set. To further investigate the characteristics of the adversarial uncertainty set, we visualize it in a simple moving-to-targe task: controlling a parti- ele move towards a target point e (Figure 2.a). The two-dimensional state informs the position of the agent, and the two-dimensional action force in two directions. The environmental parameter in two directions respectively. The transition function is expressed as s' ~ [N(s + (arw), (2W2), E), and the reward is defined by the difference of the distances to the target point at two successive steps minus a stage cost: r(s, a, s') = d(ss e) d(s', e) 2. The nominal value v = (1, 1) (Figure 2.b) indicates the equal friction factor two directions for the training environment. It easy conjecture that the optimal action is pointing towards the target point, and the optimal value function (*(s) -d(s, e). We visualize L2. L1 and adversarial uncertainty set of the con- specific state (4.3) and the optimal action a" = (-0.8, 0.6). The uncertainty sets I.2 and J, satisfy (wî +w)) < and |wi]+ /
(a1, a2) ([|ll| is normalized) controls the
(W1, wx2) represents the contact friction
tact friction in Figure 2.(c,d,e) respectively,
re- spectively. Adversarial uncertainty set is calculated by following the generation procedure described in Section 3.3, where the normalized derivative d(w) is (0.8, 0.6]T and adversarial uncertainty set is (w//0.64 w2/0 w0.36.3) 0.5 < 1, as an ellipse in Figure 2.c. Compared with the L2 uncertainty set, the adversarial uncertainty set extends the perturbation range of the horizontal dimensional parameter since it could be more sensitive the final return. As result, the agent learning resist such an uncertainty set is expected to perform well unseen perturbation types, which will be verified in more realistic experiments in the next section.
5

---

Experiments
In this section, provide experimental results on the Real-world Reinforcement Learning (RWRL) benchmark [27]. to validate the effectiveness of the proposed regularizing USR for resisting pertur- bations in the environment. Besides, we apply USR on sim-to-real task to show its potential on real-world robots.
4.1 Experimental Setups
Task Description. RWRL, whose back-end is the Mujoco environment [28], is a continuous con- trol benchmark consisting of real-world challenges for RL algorithms. Using this benchmark, we will evaluate the proposed algorithm regarding the robustness of the learned policy in physical en- vironments with perturbations of parameters of the state equations (dynamics). In more detail, first train policy through interaction with the nominal environments (i.e., the environments without any perturbations), and then test the policy in the environments with perturbations within range over relevant physical parameters. The setup is visualized as process map in Appendix C.I. In this paper, we conduct experiments on six tasks: cartpole balance, cartpole swingup, walker sttand, mulker_walk quadruped walk, quadruped run, with growing complexity in state and action space. More details about the specifications of tasks are shown in Appendix C.2. The perturbed variables and their value ranges be found in Table 2.
Evaluation metric. A severe problem for Robust RL research is the lack of a standard metric to evaluate policy robustness. To resolve this obstacle, we define a new robustness evaluation met- ric which we call Robust-AUC to calculate the area under the curve of the return with respect to the perturbed physical variables, in analogy to the definition of regular AUC [29]. More specif- ically, a trained policy π is evaluated in an environment with perturbed variable P whose values change in the range [Umiuu Umaz] and achieves different returns Then, these two sets of data are employed to draw parameter-retumn curve C(v, r) to describe the relationship between turns and perturbed values u. We define the relative area under this curve as Robust-AUC such that Robust-AUC Area(C(e.r))
[Umin; Umaz]- Compared the vanilla AUC, Robust-AUC describes the correlation between returns and the perturbed physical variables, which can sensi- tively reflect the response of learning procedure (to yield policy) to unseen perturbations, i.c., the robustness. We further explain the practical implementations to calculate Robust-AUC of the experiments in Appendix C.3.
Vmaa Vinin
Baselines and Implementation of Proposed Methods. We first compare USR with standard version of Soft Actor Critic (SAC) [25], which stands for the category of algorithms without reg- ularizers (None-Reg). Another category of baselines is to directly impose Lp regularization on the parameters of the value function (LI-Reg, L2-Reg) [30], which is common way to improve the generalization of function approximation but without consideration of the environmental per- turbations; For a fixed uncertainty set as introduced in Section 3.2, we implement two types of uncertainty sets on transitions, L2- USR and LI-USR, which can be viewed an extension of Der- man et al. [10] and Wang and Zou [14] for continuous control tasks respectively; finally, we also evaluate the adversarial uncertainty set (Section 3.3), denoted Adv-USR. We conclude all model structures and hyperparameters in Appendix C.4 C.5. The code of experiments is available on gi github.coo haabb/ccmmmieeehag////rll
4.2 Main Results
We show the Robust-AUC and its significance value of cartpole_swingeee walker_stand quadruped.walk in Table 1. Due to page limits, the results of other tasks are presented in Ap- pendix D.I. addition to calculating Robust-AUC under different perturbations, we also rank all algorithms and report the average rank an overall robustness performance of each task. Notably, LI Reg and L2 Reg do not improve the robustness, and even impair the performance in compar- ison with the None-Regularte Regularized agent on simple domains (cartpole and walker). In contrast,

---

Table 1: Robust-AUC of all algorithms and their uncertainties RWRL benchmark.
Tusk Name
Variables
None Reg
Li-Reg
42 Keg
Algorihmss
LT-USR
L2-USK
Adi-O3K
pole, Jength
993.41 (21.95)
319.75 (25.84)
368. 16 (6.53)
(12.77))
424.48 (2:27)
430.91 (2.35)
pole aass
155.25 (4.79)
96.85 (4.52)
(12.4)
175 (3.33
159.61 (2.41)
193.13 (2.27)
cont damping
137.20 (4.55)
140. 16 (0.58)
165.01 (0 16)
164.21 (0.48)
160 (2.68)
170.39 (0.76)
dider Jamping
783.76 (9)14)
775.73 (16.50)
797.59
793.55
781 231.02.0 (5.880
$19.32 (3,00)
average
3:75
2.5
length
487.02
497.38
tosso longth
586.20
(36.17)
wolker
join jampiin
687.24
387,89
527.87
514.77
cumaci..iccton friction
946.74 (22.20)
(29.16
941.92 (22.21)
943.11 (21.97)
940.73 (20.89)
945.69 (16.02)
average rank
2.50
4.75
4.25
4.25
3.00
2.25
shin lengih
492.55 (124,4)
406.77 (90.4))
(13.13 50)
$40.39 (126 22)
564 (135.49)
S71.85 (9)
tosso density
471.45 (99.14)
600 (45
526 (79.11)
442 05 (70.23)
472 (83, 22)
602.00 (55,36)
guodeaped.
joint jampinn
675 95 (67.23)
711 (91
794.56 (65.64)
762. 50 (70 76)
658 1 (112.67)
785.11 11 (40.
conaact friction
683,80 (135.80)
906.92 (100.19)
770.44 (158.42)
777.40 777.40:0106044
767.80 (109.00)
969.73 (21.24)
uverage
3,75
observe that both L2-USR and LI-USR can outperform the default version under certain perturba- tions (e.g. LI-USR in carppole_ -wringp for Jele_t L2-USR in walker_stand for thigh_enggt they are, however, not effective for all scenarios. We argue that the underlying reason could be that the fixed shape of the uncertainty set cannot adapt all perturbed cases. This is supported by the fact that Ady-USR achieves the best average rank among all perturbed scenarios, showing the best zero- shot generalization performance in continuous control tasks. For complex tasks like quadruped it is surprising that LI-Reg can achieve competitive results compared with Ady- USR but with slightly larger uncertainty on the Robust-AUC, probably because the sparsity by regularization can reduce redundant features. We also compare the computational cost of all algorithms both empirically and theoretically in Appendix D.2. It is concluded that Adv-USR can improve the robustness in most cases without increasing too much computational burden. More limitations the computational time are discussed in Section 6. We also carry out two additional testing scenarios to imitate the perturbation in real: all parameters deviate from the nominal values simultancously (Appendix D.3) and the perturbed value follows a random walk during the testing episode (Appendix D.4). Ady-USR consistently performs best and well-adapted to different perturbations. The additional analysis on the training process of Adv- USR can found in Appendix D.5.
4.3 Study on Sim-to-real Robotic Task
In this section, we further investi- gate the robustness of Adv-USR on a sim-to-real robotic task. Sim-to- real is commonly adopted setup apply RL algorithms on real robots, where the agent is first trained in simulation and then transferred real robots. Unlike the environ- mental setup in Section 4.1 with additional perturbations during the testing phases, sim-to-real inher- ently possesses mismatch be- tween training and testing environ- ments potentially due to: (1) the simulator possesses simplified dynamics model and suffers from accumulated eтror [31] and (2) there are significant differences between simulators and real hard- ware in robot's parameters, such as quadruped example in Table 5. As a result, this setup is an ideal testbed and practical application of the proposed robust RI. algorithm.
Spisado Returns
Gatsode
ac0
None Rog
200
None Rog Ad-U1S
AdvlISR
(a) Standing
(b) Locomotion
Figure 3: Episode returns all algorithms on sim-to-real task.
Specifically, we use the Unitree Al robot [32] and the Bullet simulator [33] the platform for sim- to-real transfer. The agents learn standing and locomotion in simulations and directly perform real robots without adaption [34]. Since other baselines cannot generalize well even in pure simu-

---

lated RWRL environments, we only compare SAC agents with and without Adv- USR method. Most previous works [35, 36] utilize domain randomization (DR) techniques [37] to deal with sim-to-real mismatches. DR requires training on multiple randomly initialized simulated instances with diverse environmental parameters, expecting the policy to be generalized the testing environment (real robot). In contrast, Adv-USR only requires training on single nominal parameters, which tremen- dously improves the efficiency and feasibility. Detailed setup of the sim-to-real task is described in Appendix C.6. We also additionally compare DR theoretically and empirically in Appendix E.
We run 50 testing trials per baseline and report the episodic returns in Figure 3. Both agents succeed in learning a nearly optimal policy in simulation for both tasks. For the standing task, the agent with Ady-USR maintains its performance on real robots while the other agent fails achieve the standing position. For the more complex locomotion task, we notice that the observation is hugely different from simulation and real robots as the position and velocity estimators are noisy and delayed and parameters in Table 5 are varied. That's why None-Reg hardly moves any legs on real robots in Figure 4b. But Adv-USR reveals a certain level of robustness by iterative bending and moving all legs (Figure 4a) and surpassing velocity (Figure 4c). However, the performance still deteriorates compared with the simulation. To alleviate the extreme sim-to-real difference, combining Ady-USR and other sim-to-real techniques could be a more powerful strategy, which would be verified in further work.
0.75
0.30
0.1
Velocity
(a) Adv-USR
None Reg
USR
260
662
460
Evaluation Timesteps
(b) None-Reg
(c) Velocity on Real Robot
Figure 4: Agent behaviours of all algorithms locomotion task.
5 Conclusion
In this paper, we adopt the robustness- reuusttess-regulaaizzaiio duality method to design new regularizers for continuous control problems to improve the robustness and generalization of RL algorithms. Fur- thermore, to deal with unknown uncertainty sets, we design an adversarial uncertainty set depend- ing on the learned action state value function and implement it as a new regularizer. The proposed method shows great promise regarding generalization and robustness under environmental perturba- tions in both simulated and realistic robotic tasks. Noticeably, it does not require training in multiple diverse environments or fine-tuning in testing environments, which makes it an efficient and valuable add-on to RL for robot learning.
6 Limitations
The limitations of this work are discussed as follows. First, although the computational cost of Ady-USR is acceptable [24] for the local Gaussian model owing to low-dimensional proprioceptive observations (c.g. position, velocity) in the experiments, it is a critical factor when USV is applied to more complex dynamics with millions of parameters (i.e. common recent offline and model-based RL research [38]). Therefore, methods to automatically detect critical variables may be required in future work. On the other hand, it is probable to extend Ady USR with domain randomization to tackle more sophistical robotic tasks in the future.
8

---

Acknowledgments
This research receives funding from the European Union's Horizon 2020 research and innovation program under the Marie Sklodowska- Curic grant agreement No. 953348 (ELO-X). Jianhong Wang is fully supported by UKRI Turing Al World-Leading Researcher Fellowship. EP/W002973/1. The authors thank Jasper Hoffman, Baohe Zhang for the inspiring discussions.
References
[1] D. Silver, J. Schrittwieser, K. Simonyan, I. Antonoglou, A. Huang, A. Guez, T. Hubert, L. Baker, M. Lai, A. Bolton, Y. Chen, T. Lillicrap, F. Hui, L. Sifre, G. van den Driessche, Graepel, and D. Hassabis. Mastering the game of go without human knowledge. Nature, 550(7676)3354 359, Oct. 2017. ISSN 1476-4687. doi: 1001100008//aatue2222220
[2] V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, Antonoglou, D. Wierstra, and M. A. Ried- miller. Playing atari with deep reinforcement learning. CoRR, abs/aa/s.//aeeea 2013.
[3] J. Wang, W. Xu, Y. Gu, W. Song, and T. Green. Multi-agent reinforcement learning for active voltage control on power distribution networks. Advances in Neural Information Processing Systems, 34, 2021.
[4] Kober, A. Bagnell, and J. Peters. Reinforcement learning in robotics: A survey, The International Journal of Robotics Research, 32:11))1238-122 2013.
[5] S. Schaal. Learning from demonstration. In Advances in Neural Information Processing Sys- tems, volume MIT Press, 1996.
[6] OpenAl, M. Andrychowicz, B. Baker, M. Chociej, R. Józefowicz, B. McGrew, Pachocki, A. Petron, M. Plappert, G. Powell, A. Ray, Schneider, Sidor, Tobin, Welinder,
L. Weng, and W. Zaremba. Learning dexterous in-hand manipulation. CoRR, abs/ 808.00177, 2018.
[7] G. N. Iyengar. Robust dynamic programming. Mathematics of Operations Research, 30(2): 257-280, May 2005. ISSN 0364-765X, 1526-5471. doi: 10. 10.7700/.////o....... 1040.0129.
[8] W. Wiesemann, D. Kuhn, and B. Rustem. Robust markov decision processes. Mathematics of Operations Research, 38(1):153-183, Feb. 2013. ISSN 0364-765X. doi: 1287moo00 1120. 0566.
[9] C. P. Ho, M. Petrik, and W. Wiesemann. Fast bellman updates for robust mdps. In Proceedings of the 35th International Conference Machine Learning, pages 1979-1988. PMLR, July 2018.
[10] E. Derman, M. Geist, and S. Mannor. Twice regularized mdps and the equivalence between robustness and regularization. In Advances in Neural Information Processing Systems, May 2021.
[11] A. Nilim and L. Ghaoui. Robustness in markov decision problems with uncertain transition matrices. In Advances in Neural Information Processing Systems, volume 16. MIT Press, 2004.
[12] A. Roy, H. Xu, and S. Pokutta. Reinforcement learning under model mismatch. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
[13] D. Mankowitz, N. Levine, R. Jeong, Abdolmaleki, J. Springenberg, Shi, J. Kay, Hester, T. Mann, and M. Riedmiller. Robust reinforcement learning for continuous control with model misspecification. In International Conference Learning Representations, Sept. 2019.
[14] Y. Wang and S. Zou. Online robust reintorceeenn learning with model uncertainty. In Advances in Neural Information Processing Systems, May 2021.

---

[15] J. Grand-Clément and C. Kroer. Scalable first-order methods for robust mdps. In Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Third Conference on Intovative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Educational Ad- vances in Artificial Intelligence, EAAI 2021, Virtual Event, February 2-9, 2021, I, pages 12086- 12094. AAAI Press, 2021. doi:10.1609/aaai
[16] E. Derman, D. J. Mankowitz, T. A. Mann, and S. Mannor. Soft-robust actor-critic policy- gradient. In A. Globerson and R. Silva, editors, Proceedings of the Thirty-Fourth Conference on Uncertainty in Artificial Intelligence, UAI 2018, Monterey, California, USA, August 6-10, 2018, pages 208-218. AUAI Press, 2018.
[17] R. Bellman. Dynamic programming, Science, 153(3731):34-37, July 1966. doi:10.1126/ science. science.153.3731.3. 153.3731.34.
[18] R. T. Rockafellar. Convex Analysis. Princeton Landmarks in Mathematics and Physics. Prince- ton University Press, 1970. ISBN 978-1-4008-7317-3.
[19] H. Husain, K. Ciosek, and R. Tomioka. Regularized policies are reward robust. In Proceedings of The 24th International Conference on Artificial Intelligence and Statistics, pages 64-72. PMLR, Mar. 2021.
[20] B. Eysenbach and S. Levine. Maximum entropy rl (provably) solves some robust rl problems. In International Conference on Learning Representations, Sept. 2021.
[21] R. Brekelmans, T. Genewein, J. Grau-Moya, G. Detetang, M. Kunesch, S. Legg, and P. A. Ortega. Your policy regularizer is secretly an adversary. Trans. Mach. Learn. Res., 2022, 2022.
[22] R. S. Sutton and A. G. Barto. Reinforcement Learning an Introduction. Adaptive Computa- tion and Machine Learning. MIT Press, 1998. ISBN 978-0-262-19398-6.
[23] B. Kégl, G. Hurtado, and Thomas. Model-based micro-data reinforcement learning: What the crucial model properties and which model choose? In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenRe- view.net, 2021.
[24] D. Nguyen- Tuong, M. Seeger, and J. Peters. Model learning with local gaussian process re- gression. Advanced Robotics, 23(15):001--2334 Jan. 2009. ISSN 0169-1864, 1568-5535. 101:30.1163/0169 (0DD009998860XX2222296666666
[25] T. Haarnoja, A. Zhou, Abbeel, and S. Levine. Soft actor-critic: Off-policy maximum en- tropy deep reinforcement learning with stochastic actor. G. Dy and A. Krause, editors, Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stock- holmsmässan, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pages 1856-1865. PMLR, 2018.
[26] D. P. Kingma and M. Welling, Auto-encoding variational bayes. In Y. Bengio and Y. LeCun, editors, 2nd International Conference on Learning Representations, ICLR 2014, Banff AB, Canada, April 14-16, 2014, Conference Track Proceedings, 2014.
[27] G. Dulac-Amold, D. J. Mankowitz, and T. Hester. Challenges of real-world reinforcement learning. CoRR, abs/1904.1 12901, 2019.
[28] E. Todorov, T. Erez, and Tassa. Mujoco: physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 5026- 5033, Vilamoura-Algaree, Portugal, Oct. 2012. IEEE. ISBN 978-1-4673-1736-88 978-1-4673
1737-5 778-1-4473--177 1-35--1 100.1100//ROS.2022.533000
10

---

[29] J. Huang and C. Ling. Using auc and accuracy in evaluating learning algorithms. IEEE Trans- actions on Knowledge and Data Engineering, 17(3):299-310, Mar. 2005. ISSN 1558-2191.
[30] Z. Liu, X. Li, B. Kang, and T. Darrell. Regularization matters in policy optimization an empirical study on continuous control. In 9th International Conference on Learning Represen- lations, ICLR 2021, I, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
[31] T. Erez, Y. Tassa, and E. Todorov. Simulation tools for model-based robotics: Comparison of bullet, havok, mujoco. ode and physx. In 2015 IEEE International Conference Robotics and Automation (ICRA), pages 4397-4404 May 2015.
[32] A. Unitree. Unitree. al: More dexterity, more posibility, 2018. https//www..nirre..om/lllll
Jan. 2018.
[33] E. Coumans and Y. Bai. Pybullet, python module for physics simulation for games, robotics and machine learning, 2016.
[34] J. Lee, J. Hwangbo, and M. Hutter. Robust recovery controller for quadrupedal robot using deep reinforcement learning. CoRR, abs/1901.07517, 2019.
[35] A. Kumar, Fu, D. Pathak, and Malik. Rma: Rapid motor adaptation for legged robots. In D. A. Shell, M. Toussaint, and M. A. Hsieh, editors, Robotics: Science and Systems XVII, Virtual Event, July 12-16, 2021, 2021. doi: 10..1607/RSS..2211.1I000
[36] R. Yang. M. Zhang, N. Hansen, H. Xu, and X. Wang. Learning vision-guided quadrupedal locomotion end-to-end with cross modsl transformers. CoRR, abs/2107.039966 2021.
[37] X. B. Peng, M. Andrychowicz, W. Zaremba, and P. Abbeel. Sim-to-real transfer of robotic control with dynamics randomization. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 3803-3810, May 2018. 061100.109///RA221113445563
[38] M. Janner, J. Fu, M. Zhang, and S. Levine. When to trust your model: Model-based policy op- timization. In H. M. Wallach, H. Larochelle, A. Beygelzimer, F. d' Alché-Bue E. B. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Confer- ence on Neural Information Processing Systems 2019. NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 12498-12509, 2019.
[39] R. Kirk. A. Zhang, E. Grefenstette, and T. Rocktäschel. A survey of zero-shot generalisation in deep reinforcement learning. J. Artif. Intell. Res., 76:201-264 2023. doi: 10. 10i:10.1613//pp... 16j3r/i 1.f.air.111... 1.1.a....1114.... 14144.
[40] J. Moos, K. Hansel, H. Abdulsamad, S. Stark, D. Clever, and Peters. Robust reinforcement learning: A review of foundations and recent advances. Machine Learning and Knowledge Extraction, 4(1):276-311 -15,, Mar. 2022. ISSN 2504-4990. doi: 10t11.3300/maa4401000
[41] L. Pinto, J. Davidson, R. Sukthankar, and A. Gupta. Robust adversarial reinforcement learning- In D. Precup and Y. W. Teh, editors, Proceedingy of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, volume 70 of Proceedings of Machine Learning Research, pages 2817-2826. PMLR, 2017.
[42] C. Tessler, Efroni, and S. Mannor. Action robust reinforcement learning and applications in continuous control. In K. Chaudhuri and R. Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pages 6215-6224. PMLR, 2019.
[43] Kamalaruban, Y.-T. Huang, Y.P. Hsieh, P. Rolland, C. Shi, and V. Cevher. Robust reinforce- ment learning via adversarial training with langevin dynamics. In H. Larochelle, M. Ranzato, R. Hadsell, M.-F. Balcan, and H.-T. Lin, editors, Advances in Neural Information Processing
11

---

Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, Virtual, 2020.
[44] E. Vinitsky, Y. Du, K. Parvate, K. Jang, P. Abbeel, and A. M. Bayen. Robust reinforcement learning using adversarial populations. CoRR, abs/2008.01 001800 2020.
[45] A. Pattanaik, Z. Tang, Liu, G. Bommannan, and Chowdhary. Robust deep reinforcement learning with adversarial attacks. In E. André, S. Koenig, M. Dastani, and G. Sukthankar, edi- tors, Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems, AAMAS 2018, Stockholm, Sweden, July 10-15, 2018, pages 2040-2043 International Foundation for Autonomous Agents and Multiagent Systems Richland, SC, USA/ ACM, 2018.
[46] H. Zhang, H. Chen, C. Xiao, B. Li, M. Liu, S. Boning, and C.-J. Hsich. Robust deep re- inforcement learning against adversarial perturbations on state observations. In H. Larochelle, M. Ranzato, R. Hadsell, M.-F. Balcan, and H.-T. Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, Virtual, 2020.
[47] P. Oikarinen, W. Zhang, A. Megretski, L. Daniel, and T.-W. Weng. Robust deep rein- forcement learning through adversarial loss. In M. Ranzato, A. Beygelzimer, Y. N. Dauphin, P. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems 34; Annual Conference on Neural Information Processing Systems 2021, NeurlPS 2021, Decem- ber 6-14, 2021, Virtual. pages 08156-261667 2021.
[48] C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. J. Goodfellow, and R. Fergus. Intriguing properties of neural networks. In Y. Bengio and Y. LeCun, editors, 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings, 2014.
[49] B. Mehta, M. Diaz, F. Golemo, C. J. Pal, and L. Paull. Active domain randomization. In Proceedings of the Conference Robot Learning, pages 1162-1176. PMLR, May 2020.
[50] Wang, Kurth-Nelson, H. Soyer, Z. Leibo, D. Tirumala, R. Munos, C. Blundell, D. Ku- maran, and M. M. Botvinick. Learning to reinforcement learn. G. Gunzelmann, A. Howes, T. Tenbrink, and F. J. Davelaar, editors, Proceedings of the 39th Annual Meeting of the Cogni- tive Science Society, CogSci 2017, London, UK, 16-29 July 2017. cogniivescionnesociettroog 2017.
[51] Y. Tassa, Doron, Muldal, T. Erez, Y. Li, D. d. Casas, D. Budden, A. Abdolmaleki, J. Merel, A. Lefrancq. T. P. Lillicrap, and M. A. Riedmiller. Deepmind control suite. CoRR, abs/1801.0m hts/1801.00690, 2018.
[52] D. Yarats and Kostrikov. Soft actor-critic (sac) implementation in pytorch, 2020.
[53] H. van Hasselt. Reinforcement learning in continuous state and action spaces. In M. Wiering and M. van Otterio, editors, Reinforcement Learning: State-of-the-Ard Adaptation, Learn- ing, and Optimization, pages 207-251. Springer, Berlin, Heidelberg, 2012. ISBN 978-3-642- 27645-3.
[54] I. Kostrikov, L. M. Smith, and S. Levine. Demonstrating walk in the park: Learning to walk in 20 minutes with model-free reinforcement learning. In K. E. Bekris, K. Hauser, S. L. Herbert, and J. Yu, editors, Robotics: Science and Systems XIX, Daegu, Republic of Korea, July 10-14, 2023, 2023. doi:10. 2023. XIX.066.XXXX66.....
12

---

A Related Work
Robust Reinforcement Learning (Robust RL) has recently become a popular topic [7, 27, 39, 40), due to its effectiveness in tackling perturbations. Besides the transition perturbation in this paper, there are other branches relating to action, state and reward. We will briefly discuss them in the following paragraphs. Additionally, we will discuss the relation of Robust RL, sim-to-real, Bayesian RL and Adaptive RI. approaches, which are also important topics in robot learning
Action Perturbation. Early works in Robust RL concentrated action space perturbations. Pinto et al. [41] first proposed an adversarial agent perturbing the action of the principle agent, training both alternately in a mini-max style. Tessler et al. [42] later performed action perturbations with probability a to simulate abrupt interruptions the real world. Afterwards, Kamalaruban et al. [43] analyzed this mini-max problem from a game-theoretie perspective and claimed that an adversary with mixed strategy converges mixed Nash Equilibrium. Similarly, Vinitsky al. [44] involved multiple adversarial agents augment the robustness, which can also be explained in the view of mixed strategy.
State Perturbation. State perturbation can lead to the change of state from 8 to 8p, and thus might worsen an agent's policy <alss) [45]. Zhang et al. [46], Oikarinen et al. [47] both assume an Lp- norm uncertainty set on the state space (inspired by the idea of adversarial attacks widely used in computer vision [48]) and propose an auxiliary loss to encourage learning to resist such attacks. is worth noting that state perturbation is special case of transition perturbation, which can be covered by the framework proposed in this paper, further explained in Appendix F.
Reward Perturbation. The robustness--eggullrizaiooo duality has been widely studied, especially when considering reward perturbations [19, 20, 21J. One reason is that the policy regularizer is closely related to perturbation on the reward function without the need for rectangular uncertainty assumption. However, it restricts the scope of these works reward perturbation, since it can be shown to be particular case of transition perturbation by augmenting the reward value in the state [20]. Besides, the majority of works focus on the analysis of regularization to robustness, which can only analyze the effect of existing regularizers instead of deriving novel regularizers for robustness as in the work we present here.
Sim-to-real. Sim-to-real is key research topic in robot learning. Compared to the Robust RL prob- lem, it aims leam robust policy from simulations for generalization in real-world environments. Domain randomization is common approach case this mismatch in sim-to-real problems [6, 37]. However, Mankowitz et al. [13] has demonstrated that it actually optimizes the average case of the environment rather than the worst-case scenario (as seen in our research), which fails to perform robustly during testing. More recent active domain randomization methods [49] resolve this flaw by automatically selecting difficult environments during the training process. The idea learning an adversarial uncertainty set considered in this paper can be seen as a strategy to actively search for more valuable environments for training.
Bayesian RL. One commonality between Bayesian RL and Robust RL is that they both store un- certainties over the environmental parameter (posterior distribution ((w) in Bayesian RL and uncer- tainty set Qup in Robust RL). Uncertainties learned in Bayesian RL can benefit Robust RL in two ways: (1) Robust RL. can define an uncertainty set 22 (w q(w) > a)) to learn a robust policy that can tolerate model errors, which is attractive for offline RL and model-based RL; (2) soft robust objective with respect to the distribution q(w) can ease the conservative behaviours caused by the worst-case scenario [16].
Adaptive RL. Adaptive RL (often referred as Meta RL [50]) is another popular technique to deal with the perturbations in environments parallel to Robust RL introduced in this paper. The main difference between Robust RL and Adaptive RL is whether policy parameters are allowed to change when environmental parameters vary. Robust RL a zero-shot learning technique aiming to learn one single robust policy that can be applied to various perturbed environments. Adaptive RL is few-shot learning technique aiming to quickly change policy to adapt to the changing environments.
13

---

These two techniques can be combined to increase the robustness in real-world robots. One can first use Robust RL to learn base policy as the warm start and fine-tune the policy on certain perturbed environments with Adaptive RL.
B
Extra Algorithm Details
B.1 Proof of Uncertainty Set Regularized Robust Bellman Equation
The proof is as follows:
(* = r +7 min / P(s'ls, (s6)ss
wEn
=r(s,a) +7 / P(5
+ min
wefl
=r(s,a) +2 / ( -? max
- daw (20)
v
=r(s,a)+ +2 / P( -7 op.m (-
(6)
The second line utilizes the first-order Taylor Expansion at The third line reformulates the mini- mization to maximization w and adds an indicator function hard constraint w. The last line directly follows the definition of convex conjugate function.
B.2 Convergence of Robust Policy Evaluation
We now prove that the Bellman operator with extra regularizer term as the policy evaluation stage converge the robust action value function given some specific conditions.
Since V/ (s) La (aals) (s, a)da, we define the equivalent operator with respect to Q" to the one proposed in this paper as T such that
r(,,))))(s)a)))))))))) (s, a(s,a) +, / P(s'is,a;0) / d'd''''''d''' -a
/ I. P(d'ss
/
r(s + / P(bls,,a;)) -a
operator proposed paper.
(7)
To the derivation, will not expand the tenm a the form of Q" at the beginning in the following procedures.
14

---

r(s, a) + P1sls..:w))T((h0 - http:///www....ooo//////////aa/a
[TQq- - TQ = max s,a
(s') 12ds'
- -r(s, - -7 P(bbaw)) P(1 (6))l
/ (701),,0 V3' ds')
max
8,a
| ( /
a
I y max / p(o1)... - V35 - ds'
Partition
J. M.
ds'
max 600
Partition 2.
Since Partition is conventional Bellman operator, following the contraction mapping property we can directly write the inequality such that
S. -1 ds' -VV
Y
max 9,0
Next, we will process Partition 2. The details are follows:
||V ( -v7(s))))(())) ds'
a max
8,0
a max
1 |v.
Vi" (s')lds'
a max
max 1V (s') vi / P(s' a; w) 12ds'
a max
< a max
| P(s a, 2ds
max (V' (3)
V| V.2 1
(9)
Combining the results of Eq.8 and Eq.9 and expanding the term V*, we can directly get that
||TQ]
lo
I (r V) - 12
$ (r + 5) max [V (s') (s'(('))
(s))
$ (7 + 5) "'a" (s',aa)) (T(''a')
$ ( 1) Q
To enable T to be a contraction mapping in order to converge to the exact robust Q-values, we have let < 7 8 < 1. In more details, the norm of the gradient of transition function with respect the uncertainty set (i.e.. P(s' [s, a; w)ll2 inside ô) critical the convergence of robust Q- values under some robust policy. Given the above conditions, the robust value function will finally converge.
15

---

B.3 Algorithm on Incorporating Uncertainty Set Regularizer into Soft Actor Critick
The proposed Uncertainty Set Regularizer method is flexible to be plugged into any existing RL frameworks as introduced in Section 3.2. Here, we include a specific implementation on Soft Actor Critic algorithm, see Algorithm I.
Algorithm Uncertainty Set Regularized Robust Soft Actor Critic
Input: initial State 8, action value (s,, a: 0y's parameters 01, 82. policy x <alss; 6)'s parameters Φ, replay buffer D 0, transition nominal parameters úū value target update rate /
2: Set target value parameters Otar,1 < 0, Orar f- 02
3: repeat
4: Execute a ~ [aa Φ) in the environment
5: Observe reward r and next State s'
6:
D DU {s, (s.a.r.s)
7:
s
8:
for each gradient step do
9:
Randomly sample batch transitions, B {s, (s,,,r,,' from D Construct adversarial uncertainty set Qu as introduced in Section 3.3 (for adversarial un- certainty set only)
10:
11:
Compute robust value target y(s, a, 8 ()) by calculating the RHS of Equation 4 Update action state value parameters for {1,2] by minimizing mean squared loss the target:
Ve, [8] E(saann)) a; 0.) y(s, a,s, x))²
for 1,2
14:
Update policy parameters & by policy gradient:
15:
V Loyer (min;=1,2 Q(s, ā;;;) log (āls;
where à sampled from x (als; () and differentiable w.r.t.
17:
Update target value parameters:
18:
Otan. + (1 0rara + p0;
for 1,2
19:
end for
20:
until convergence
B.4
Algorithm on Generation of Adversarial Uncertainty Set
This is the pseudo-code to generate the adversarial uncertainty set introduced in the paper.
Algorithm 2 Generation of Adversarial Uncertainty Set
1: Input: current state current action value function V(s; next state distribution P(-|s, a; a) = ((((s, a; w), a(s,a;i))
2: do
3: Sample σ ~ (), and calculate next state s' = p(s, a; w) + E(s, a; t)o (the reparame-
terization trick sample s' P(-|s, a; w));
4: Forward pass to calculate the next state value V(s';0);
5: Backward pass
compute the derivative g(w)
V=V(s';0)
(V (00.s.:0))++<s Us'
dh
6:
Normalize the derivative by d(w) ((w)/ g(
7: Generate the adversarial uncertainty set Ω. { aw \/d(()) $ 1).
8: done
C Extra Experimental Setups
C.1 Process Map of Experiments
We would like clarify again the experimental setup in this paper. We drew process map to facilitate the understanding, During both training and testing, the agents can only acquire
16

---

Training
Testing
Perturoc Env1
Nominal Envo
Perturbed Erv2
Perturbed
observation
action
observation
action
(NO Env parameters)
(NO Env parameters)
Agent
Agent
Example: RWRL Cartpole
pole Jength pole miniss
pole, Jength: 1.0
pote masss:: 5.0
position, velocity... ...
force
position, Velocity...
force
Agent
Agent
Bullet Simulator
Real Robot
Example: Sim-to-real A1
motor angles, orientation...
motor angles
motor angles, orientation...
motor angles
Agent
Agent
Figure 5: Process map of experiments and specific examples.
observations (e.g. position, velocity) from the environment without knowing the environmental parameters (e.g. pole length, robot mass). During training, the agents can only interact with single set of environment parameters (nominal environment), of the perturbation. During testing, all trained agents are fixed and tested various perturbed environment parameters. So to speak, the agents have to leam single policy on an unperturbed environment that can adapt to various perturbed environments, which might be far more challenging than expected. Ady- USR shows a stable improvement over all other baselines.
it's impossible to predict the pattern
C.2 RWRL Benchmarks
In this paper, conduct experiments six tasks: cartpole balance, cartpole swingup, walker_stand, walker_walk, waak,, quadruped All tasks involve domain and
17

---

movement. For example, cartpole balance task represents the balance movement on cartpole do- main. In this paper, we consider domains with 2 movements each.
The 3 domains are cartpole, walker and quadruped respectively:
Cartpole has an unactuated pole based on cart. One can apply one-direction force to balance the pole. For cartpole balance task, the pole starts near the upright while in cartpole-swinguy task the pole starts pointing down.
Walker is planar walker to be controlled in 6 dimensions. The walker-stand task re- quires an upright torso and some minimal torso height. The walker_walk task encourages a forward velocity.
Quadruped is generic quadruped with more complex state and action space than cart- pole and walker. The quadruped and quadruped tasks encourage a different level of forward speed.
For detailed description of these tasks, please refer DM-CONTROL [51]. For each task, follow the RWRL's setup by selecting 4 environmental variables and perturbing them during the testing phase. The perturbed variables and their value ranges can be found in Table 2. We also report the nominal values of these perturbed variables to indicate the differences between training and testing environments. Notably, all tasks are run for a maximum of 1000 steps and the max return 1000.
Table 2: Tasks in RWRL benchmark and the perturbed variables.
Task Name
Observation Dimension
Action Dimension
Perturbed Variables
Perturbed Range
Nominal Value
pole length
J0.3,
1.0
carppote balance
polc_mass
[0.1. 10.0]
0.1
caripole_wwinuu
joint_damping
[2e-6, 20-1]
2e-6
slider_damping
[Sc-4, 3.0]
5c-4
lenh_
[0.1, 0.7]
0.225
staaer walker walk
24
joimt_damping
tengt_lnngth
10.1,
10.0]
0.1
0.3
contact. friction
[0.01, 2.0]
0.7
shin length
(0.25.
0.25
quadruped walk walk
torso density
[500, 10000)
1000
guadruped.rum
78
12
joint_damping
[10. 150]
30
contact foriait.
[0.1, 4.5]
1.5
C.3 Practical Implementations of Robust-AUC
To calculate Robust-AUC in RWRL experiments, each agent trained with random seeds. During the testing phase, for each environmental variable P, we uniformly sample 20 perturbed values v in the range of [Uminn Umez]. For each value v, the environment variable first modified to value and the agent tested for 100 episodes (20 episodes per seed). We then select the 10%--quntile value By doing so roughly have approximated curve C(v,r) and can
as the return calculate Robust-AUC defined previously. We also report the area between 5%-quantile and 15%- quantile as the statistical uncertainty of the reported Robust-AUC.
C.4 Model Structure
The model structure for all experimental baselines is based on the Yarats and Kostrikov [52]'s im- plementation of Soft Actor Critic (SAC) [25] algorithm. The actor network is 3-layer ferd-forward network with 1024 hidden units and outputs the Gaussian distribution of action. The critic network adopts the double-Q structure [53] and also has 3 hidden layers with 1024 hidden units on each layer, but only outputs a real number as the action State value.
2 100%-iuattile as worst-case performance evaluates the robustness of RL algorithms more reasonably than common metrics.
18

---

C.5 Hyperparameters
To compare all algorithms fairly, we set all hyperparameters equally except the robust method and its coefficient. All algorithms are trained with Adam optimizer [26]. The full hyperparameters are shown in Table 3. For regularizer coefficients all robust update methods, please see Table 4. Notably, the principle to choose the coefficient to increase the value until the performance on the nominal environment drops. In the future, can be automatically tuned by learning an unregularized value function and comparing the difference between robust value and unregularized value. All experiments are carried out on NVIDIA GeForee RTX 2080 Ti and Pytorch 1.10.1.
Table 3: Hyperparameters of Robust RL algorithms.
HYPERPARAMETERS
VALUE
DESCRIPTION
1024
THE NUMBER TRANSITIONS
DISCOUNT FACTOR 7
0.99
THE IMPORTANCE OF FUTURE REWARDS
REPLAY BURFER S1ZE
166
THE MAXIMUM NUMBER TRANSITIONS STORED IN MEMORY
LENGTH
MAXIMUM
EPISODE
MAX TRAINING STEP
JE6
THE NUMBER OF TRAINING STEPS
RANDOM
RANDOMLY THE LEARNING RATE FOR ACTOR NETWORK
ACTOR LEARNING RATE
1E-4
ACTOR UPDATE FREQUENCY
THE FREQUENCY FOR UPDATING ACTOR NETWORK
ACTUR LOG STD BOUNDS
[-5. 2]
THE OUTPUT BOUND OF LOK STANDARD DEVIATION
CRITIC EARNING RATE
IE.4
THE LEARNING RATE FOR CRITIC NETWORK
'ARGGT
FREQUENCY
FREQUENCY
TARGET
CRITIC TARRET UPDATE CORFICIENT
0.005
THE UPDATE CORFICCENT OF CRITIC TARGET NETWORK FOR SOPT LEARNING
TEMPERATURE
INITIAL
ACTOR'S 'S OUTPUT FOR EXPLORATION
TEMPERATURE LEARNING RATE
IE-4
THE LEARNING RATE FOR UPDATING THE POLICY ENTROPY
SAMPLE SIZE
THE SAMPLE SIZE TO APPROXIMATE THE ROBUST REGULARIZOR
Table 4: Regularization coefficients of Robust RL algorithms.
Algorithms
Task Name
None-Reg
I.I-Reg
1.2-Reg
LI-USR
12-USR
Adv-USR
cartpole balance
le-5
le-4
5e-5
le-4
le-5
le-5
le-4
le-4
le-4
le-4
walker-stand
le-4
le-4
5e-5
le-4
le-4
walker walk
le-4
le-4
le-4
le-4
5e-4
quadruped walk
le-5
le-4
le-4
le-4
5e-4
quadruped run
le-4
le-4
5e-5
le-4
7e-5
C.6 Extra Setups of Sim-to-real Task
We use the Unitree A1 robot [32] and the Bullet simulator [33] as the platform for sim-to-real transfer. The Unitree Al is a quadruped robot with motors (3 motors per leg). The Bullet simulator popular simulation tool specially designed for robotics.
It is well known that there is a non-negligible difference between simulators and real robots due to:(1) the simulator possesses a simplified dynamics model and suffers from accumulated error [31] and (2) there are significant differences between simulators and real hardware in robot's parameters, such as a quadruped example Table 5. Therefore, training policies in simulation and applying them to real robots (sim-to-real) challenging task for roboties.
Specifically, we perform 2 sim-to-real tasks: standing and locomotion following previous work [34]. The detailed description of the experiment is as follows:
Observation. The observation contains the following features of steps: motor angles (12 dim), root orientation (4 dim, roll, pitch, roll velocity, pitch velocity), and previous actions (12 dim). So the observation space is 84 dimensions.
19

---

Action. All 12 motors can be controlled in the position control mode, which is further con- verted to torque with an internal PD controller. The action space for each leg is defined as [p o, + o|. The specific values for different parts (hip, upper leg, knee) in the stand- ing task are p - [0.00, 1.6, -1.8) = [0.8, 2.6, 0.8]. The values in the locomotion task are p = [0.05, 0.7, 1.4]. o 0.4,
Reward. For the standing task, the reward consists of 3 parts: r(s,a) = 0.2 THEIGHT + 0.6 PPoSE + 0.2 * TVRL- THEIGHT = 1 - 0.25877 /0.2587 is rewarded for approaching the standing position on the z-axis. FPose = exp * Σ Marget - ml is for correct motor positions. Ivm punished for positive velocity (standing should be still the end). For the locomotion task, the reward function inspired by Kostrikov al. [54]. r(s, a) (e a) 0.1 * saw. Vyaw is angular yaw velocity and (8(8, a) 1 for faz [0.5, 1.0), 0 for er 2 2.0 and = 1 -|ux- 0.5l otherwise, is rewarded for velocity in x-axis.
Simulation Training. We first train SAC agents with and without Adv- in simulations. Each step simulates 0.033 seconds so that the control frequency 33 Hz. Agents are trained for 1e steps in the standing task and 2e steps in the locomotion task. The model structures and hyperparameters are the same in RWRL experiments and can be referred to Appendix C.4 and C.5. The regularizer coefficients for Adv-USS USR are 1e-3 and le-4 for two tasks.
Real Robot Evaluation. After training, we directly deploy the learned policies on real robots. Since all sensors are internal on robots, there are no external sensors required. The control frequency set as 33 Hz. We run each policy 50 episodes with 1000 steps and report the 10%--uantile of the final return and 5% 15%--qantil the error bar in Figure 3.
Table 5: Unitree Al's Parameters in Simulation and Real Robot.
Parameters
Simulation
Real Robot
Mass (kg)
12
[10, 14]
Center of Mass (cm)
0
[-0.2, 0.2]
Motor Strength (x default value)
1.0
[0.8, 1.2]
Motor Friction (Nms/rad)
1.0
[0.8, 1.2]
Sensor Latency (ms)
0
[0. 40]
Initial position (m)
(0, 0, 0.25)
([-1, 1], [-1, 11, [0.2. 0.3))
D Extra Experimental Results
D.1 Constant Perturbation on System Parameters
Extra experimental results for task cartpole balance, walker_wal and quadruped can be found in Table 6. We can observe similar results as in the main paper that both L2-USR and LI-USR can outperform the default version under some certain perturbations (e.g. L.-USR in cartpole balance for mass_mass, L2-USR in walker_walk for length, while Adv-USR achieves the best average rank among all perturbed scenarios, showing the best rero-shot generalization performance con- tinuous control tasks. Notably, L2-Reg in walker and LI Ll-Reg in quadruped also achieve competitive robust performance compared with Adv-USR. A possible reason is that, for environ- ments with high-dimensional state and action spaces, some of them are redundant and direct regu- larization on value function's 's parameters is effective to perform dimensionality reduction and thus learns a generalized policy.
D.2 Computational Cost of All Algorithms
We report the average computation time (in milliseconds) for single value update of all algorithms in Table 7. We notice that the computation of all algorithms increases as the environment's 's com-
20

---

Table 6: Robust-AUC of all algorithms and their uncertainties RWRL benchmark.
Algorithms
Task Name
Variables
None-Reg
1.1-Reg
1.2-Rg
L.-USR
12-USR
Adi-USR
polc_Jength
(5.92)
989.85
polc.mass
623.88 (28.64)
605.35 60)
607. 79 (23.18) 18)
632.54 (14.74)
588.
caripole balance
joint damping
970.83 (21.89)
978.97 (9.95)
982.71 (15.24)
985.57 (10.01)
978.62 (17.52)
982.43 (130,03)
siderr dader_damping
999 44
999, 30
999.34 (0.57)
999.45 (0.31)
999, (0.48)
999.55 (0.32)
average rank
2.75
Jength
315.64 (57.24)
257. 90 (25 (4)
345.12 (40 50)
316.61 (37, 86)
350.01 (34,37)
318 88 (53.75)
toeso length
498.01 (54.04)
300.39 14.06)
533.96 (47.73)
550.44 (50.83)
543.39 (42.52)
543,91 (54.36)
walker
joint damping
(50.33)
(30.18)
(56.04)
contact, friction
885.01 (27.47)
714,94 (27.15)
907.13 (18.94)
897.65 (23.49)
900.58, (2) (46)
902.03 (24.68)
everage
4,50
6,00
2,00
3:25
3,00
2,25
length
(91.36)
(117.007) (7.07)
383, (79.57)
loeso deasity
321.24 (76.70) 70)
417.68 (88.55)
252.37 (88.41) 41)
319:43 (90.79)
225,33 (30 (9)
guadraped non
joint damping
367.05 (139.66) (1)
641.08 (19. 12)
687.42 (12.85)
324.38 (14.73)
692,02 (6.98)
664. 25 (19 35)
jorctit_ricction
654.43 (57.94)
614.21 (76.60)
473.58 (61.72)
632.664 (95.18)
624.32 (124.39)
537.19 (76.22)
aeerage rank
3.75
3,00
5,00
3,00
3.25
3,00
Table The computational cost (in milliseconds) for each value update of Robust RL. algorithms.
Algorithms L2-Reg
Task Name
None-Reg
LI-Reg
LI-USR
L2-USR
Ady-USR
cartpole_alance balance
sorpole-swingup
14.72 1.57
16.48 1.68
17.05 1.63
17.62 1.49
22.48 3.21
22.48 3.21
walker_stand
walker-wall
15.71 4 1.23
18.89 = 1.58
17.52 # 1.92
18.06 # 1.80
18.39 + 1.90
23.16 + 1.72
quadruped walk
quadruped run
15.93 # 1.68
19.47 1.47
19.56 1.67
20.79 * 4.20
19.13 a 1.98
25.23 + 2.14
plexity grows, and LI- LI-Reg, LI-Reg, Adv-USR's complexities are acceptable compared with other baselines (x 1.25 time cost). The computation only becomes a problem when applying USR methods to dynamics with millions of parameters (common in model-based RL [38]). To tackle this issue, we can identify important parameters to reduce computation costs, stated in Section 6.
Theoretically, the additional computational cost largely depends on the norm term I/V a; (s))ll in Equation 5, time complexity is O(W) (W the number of parameters).
D.3 Constant Perturbation on Multiple System Parameters
In real-world scenarios, there would be uncertainties in all system parameters. We provide the fol- lowing additional experimental results show the robustness when all parameters are perturbed simultaneously. The specific environmental setup is that all 4 parameters ously during testing. The perturbation intensity grows from 0 to 1. 0 resembles training environ- ments without perturbations and represents the allowed maximum perturbed values in Table 2. We adopt the same metric Robust-AUC and report it in the following table. All methods become less robust due to the increasing difficulty of perturbations, but Adv-USR still outperforms others.
perturbed simultane-
Table 8: Robust-AUC of all algorithms and their uncertainties on RWRL benchmark.
Algorithms
Task Name
None-Reg
I.I-Reg
1.2-Reg
L.-USR
1.2-USR
Adv-
curtpole swingup
867, 42 (0.27)
856.87 87 (0.52)
866.99 (0.23)
867.61 (0.44)
867.45 (0.26)
$81.36 (0.21)
walker stand
254.04 (32.91)
235.36 (25.29)
254.64 (35.01)
262.57 (25.02)
263.35 (24.85)
266.97 (7.75)
quadnspe.waak
522.98 (34.93)
524.24 (85.03)
525.51 (24,58)
525.14 (85.68)
506.17 (54.92)
534.61 (18.25)
21

---

D.4 Noisy Perturbation on System Parameters
One may also be interested in the noisy perturbation setup where system parameters keep changing at every time step. This setup extends the Robust RL framework where the perturbation is fixed throughout the whole episode. The specific experimental setup noisy perturbation as follows: the environmental parameter starts from the nominal value and follows zero-mean Gaussian random walk at each time step. The nominal value and the standard deviation of the Gaussian random walk recorded in Table 9. The experimental result quadruped _wak is shown in Figure 6. In this experiment, Ll-Reg achieves the best robustness, while our method Adv-USR achieves Top-2 performance in out of perturbations. While I.I-Reg performs less effectively in the case of fixed perturbation, it implies that different regularizers do have different impacts on these two types of perturbations. For noisy perturbation, environmental parameters walk randomly around the nominal value and reach the extreme value less often, which requires a less conservative robust RL algorithm. Our algorithm Adv-USR, originally designed for fixed perturbation problem, achieves good but not the best performance, which leads to an interesting future research direction on the trade-off between robustness and conservativeness.
Table 9: The perturbed variables for the noise perturbation experiment.
Task Name
Perturbed Variables
Start Value
Step Standard Deviation
Value Range
shin lenghh
0.25
0,1
[0.25, 2.0]
quadroped.waak
torso density
1000
500
[500, 10000)
joint damping
[10. 150]
contact .riccion
[0.1, 4.5]
cuode wnk: <hir Jenoth
quddured wir
gustrupel Nition
340
1125
1123
305
013
Figure 6: The parameter-returi bar graph of algorithms. All bars represent 10%-quantile value of episodic return under noisy environmental parameters.
D.5 Training Performances
We further analyze the effects of Adv-USR during the training process. Since all algorithms trained on the same nominal environment, they all learn to perform well on the nominal environment with a similar episode reward curve as in Figure 7a. However, their target values considering the robustness of learned policies are quite different (Figure 7b). Adv-USR has the lowest target value, indicating that the adversarial uncertainty set actually considers the pessimistic objective which encourages to learn more robust policy to resist this uncertainty. This verifies the correctness of our theoretical claims.
E
Comparison with Domain Randomization
The rethink on domain randomization (DR) directly motivates this paper: domain randomization utilizes a variety of environments and trains an average model across them. Could we develop a more efficient way? Here we discuss the drawbacks of DR and why our method could be a possible solution.
Requirements expert knowledge: DR needs to randomize multiple environments with various environmental parameters. However, which parameters and their range to random-
22

---

ize all require heavy expert knowledge. We have experience when applying DR on the sim-to-real locomotion task. we set the PD controller's gain large range, RI. fails to learn even in simulation. If this range is small, the learned policy can't transfer to real robots. In comparison, our method reduces this effort by only training a robust policy on single nominal environment with a virtual adversarial uncertainty set.
Feasibility on certain setups: In some cases, we don't have access to create multiple simulated environments. This could happen in commercial autonomous driving software without exposing the low-level dynamics system, or the current powerful ChatGPT-ssyleed cloud-based language model (if viewing the chatbot as simulator), even real-to-real transfer (from real robot to another slightly different real robot). It's not possible to create different environments for DR but still possible to learn an adversarial uncertainty set and train robust policy based on that.
Training convergence: In general, DR has slower and more unstable training process since randomized multiple environments can be quite different and learning single policy them can be hard.
We compare the popular domain randomization (DR) techniques on sim-to-real tasks (A1 quadruped standing and locomotion). The experimental setup is follows. During training in simulation, for None-Reg and Adv-USR, agents are trained on the nominal environment with standard robots' parameters ("Other" column in Table 10). For DR, agents are trained on randomized initialized parameters within certain range ("DR" column Table 10). During testing on real robots, all policies are fixed and evaluated on the same robots with 50 episodes. We present the training curves Figure 8 and testing performance in Figure 9.
DR has slower convergence rate during training since randomized environments can be quite dif- ferent and learning a single policy on them can be hard. Adv-USR can already replace DR simple sim-to-real tasks (standing). For more complex tasks (locomotion), DR still performs better since performance is more affected by the model mismatch. But we believe our method still valuable provide an alternative concise choice DR. Furthermore, we have mentioned the limita- perpendicular to the DR. Combing adversarial uncertainty sets could
tion section, our method potentially reduce the range of randomization of DR while reaching the same robust performance.
100
Episode Reward
800
80
Target Value
60
600
None Reg
None Reg
L1 Reg
L1 Reg
40
L2 Reg
Reg
L1 USR
20
USR
200
L2 USR
USR
Adv USR
0
Adv USR
0.0
0.4 Timesteps (mil)
1.0
Timesteps (mil)
(a) Episode reward
(b) Target value
Figure Training performances all algorithms on quadruped walk task
23

---

Table 10: Unitree Al's Parameters in Simulation for different algorithms.
Parameters
Other
DR
Mass (kg)
12
[10.
Center of Mass (em)
0
[-0.2, 0.2]
Motor Strength (x default value)
1.0
[0.8, 1.2]
Motor Friction (Nms/rad)
1.0
[0.8, 1.2]
Sensor Latency (ms)
[0. 40)
Initial position (m)
(0, 0, 0.25)
([-1, 11, [-l, 11, (0.2, 0.3) 0.3)
1000
1000
Episode Reward
800
Episode Reward
600
600
400
400
None Reg
None Reg
200
Adv USR
200
Adv USR
Domain Random
Domain Random
0
0.0
0.2
0.6 Timesteps (mil)
0.4
0.2
0.6
0.8
1.0
Timesteps (mil)
(a) Standing
Locomotion
Figure 8: Episode returns during training (simulation).
F
Comparison with State Perturbation
State perturbation describes the uncertainties in the output space of the dynamics model, which is special case of transition perturbation. We illustrate how to transform the State perturbation into transition perturbation in the following case.
Considering L2 uncertainty set on the output of the dynamics model, Qay = {spls' P(-|s, a; 1 ||s spll & 1}. We can rewrite the next State distribution s' P(- a; w) as f(s' s, a; w) + 77. where 1j is random noise 17 ~ (O,0, 1). Then the perturbed State can be written as Sp (s'lss a; a; + n + 8, |||||| < 1. Viewing dynamics model, the uncertainty
one additional parameter in the B is actually 22 ||| < 1). Based on this uncertainty
Episode Returns
Episode Returns
600
600
400
400
None-Reg
None-Reg
200
Adv-USR
Adv-USR
Adv-USR
Adv-USR
Simulation
Real Robot
Siimulation
Real Robot
(a) Standing
(b) Locomotion
Figure 9: Episode returns during testing (simulation and real robot).

---

on 3, one can further design corresponding regularizers on value function to icrraase the robustness as discussed in the paper.
If the uncertainty set on state space is unknown, denoted S2g, it is still feasible include 3 as an additional parameter in the dynamics model. As result, Adv-USR could still be used to handle this unknown uncertainty set 3.
25


Reviews:
[{"review_id": "oknm-r91_Rg", "reviewer": "Reviewer_Fg3n", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper tackles the task of robust reinforcement learning in continuous control environments. The authors generalize a robust reinforcement learning approach to continuous control and propose a new regularizer named Uncertainty Set Regularizer. For unknown uncertainty sets authors propose an adversarial approach based on the value function. The authors evaluated the proposed approach for the Soft Actor Critic method and presented evaluation results for simulated benchmarks and for the Unitree A1 robot. ", "summary_of_recommendation": "I recommend accepting the paper after addressing questions. ", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "The paper presents an interesting method to train robust reinforcement learning agents. I think it would be interesting to compare the performance of the proposed method and the common domain randomization technique in the benchmarks. \n\nAlso, there is a typo in the Abstract (Algorithm 1): \"Construct adversarial adversarial uncertainty set\"", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "At the beginning of the paper authors clearly motivates the importance of robust reinforcement learning methods and introduce robust RL formalism. After that, they describe additional term added to the Bellman equation which forces the robustness of the learned policy. That term requires a transition model P(s'|s,a;w) which is modeled as Gaussian distribution. The authors compare the proposed method with other regularization methods on simulated benchmarks using AUC-based metric. The authors also demonstrate that the proposed method can be used to bridge the sim-to-real gap in Unitree A1 for simple tasks without domain randomization during training. ", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}, {"review_id": "f6jhconmXH", "reviewer": "Reviewer_8DAc", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The authors propose an algorithm that solves minimax RL where the inner maximization happens over perturbative transitions of the environment. The authors rely on previous works that convert the primal minimax form into a tractable dual form and extend the work to continuous domains. Furthermore, the authors make an ellipsoidal approximation to the uncertainty set by querying the sensitivity of the value function with respect to different directions in the uncertainty set. The authors compare their approach to non-robust RL and prove robust performance of the method.", "summary_of_recommendation": "This paper is strong in presentation and technical details, which is why I recommend weak accept. However, I did not choose strong accept but the method itself did not seem sufficiently motivated for me other than being a possible application of a mathematical regularization technique. I believe the paper would be stronger if the authors connected their method back to the original motivation of asking: if the goal is to solve robust RL, what is the best solution we can find? In an age where the vast majority is domain randomization for sim-to-real transfer, I believe it's important for the authors to show very convincingly offers over DR, and the comparison felt a bit lacking. \n\nEDIT: I have read the author's response and found the arguments convincing, and I recommend that the paper be accepted.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "1. How would the authors compare their algorithm against domain randomization? Besides being more faster (requiring less parallelization), would it result in more robustness at the cost of performance? Why would folks choose this algorithm over DR?\n\n2. I did not fully understand the minimax duality mathematically, but in the usual context of optimization, there can be a duality gap between the function and its dual function (e.g. convex conjugate) in the absence of structure. What prevents the authors from solving a regularized problem and ending up with a totally different solution from the primal minimax problem? \n\n3. As the authors mentioned, usually a difficult point in robust control lies in its conservativeness; it seems the adversarial uncertainty set procedure gives guidance for the shape of the ellipsoids, but one still needs to choose an alpha. Is this one experimentally done?\n\n4. I am a bit concerned about the model learning part, as a lot of results for RL indicate that model learning is fundamentally related to the distribution of data it was trained on. What prevents the bellman iteration from going unstable because of errors in estimating the model and its gradients, as well as being subject to distribution shift?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper is well-written and clear in its presentation, experiments, and ideas mostly. Many of the ideas are derived from the duality in the tabular MDP setting, so not completely original, but still has some novel ideas such as automatic tuning of uncertainty sets. The problem being tackled is quite significant as addressing sim-to-real is a big challenge for RL, and the authors do a good job of motivating this. \n\n<Strengths>\n\n1. Overall, the structure and the organization of the paper is quite easy to follow, and relevant concepts are adequately introduced to the reader.\n\n2. The continuous-space extension of an existing result in discrete state-action MDP is a valuable contribution.\n\n3. The authors do extensive experimentation of the benchmark to highlight  the efficacy of the approach. \n\n<Weaknesses>\n\n1.  The authors should make some convincing argument of what robustness offers over domain randomization. The authors remarked about the speed which is convincing, but there are no empirical comparisons against DR that shows that this approach is more computationally tractable or results in more “robust” performance.\n\n2. The authors have a strong theoretic claim on the Bellman iteration but the quantities of the approximation rely on function approximation, which can destroy the claims of the theory in practice.\n\n3. The uncertainty structure that the authors can handle is still somewhat limited to relatively structured uncertainty. \n\n<Minor Suggestions>\n\n1. In the abstract, “Prior work claimed that adding regularization to the value function is equivalent to learning a robust policy with uncertain transitions.” - I think it’d be better to say which kind of regularization explicitly because regularization is too broad of a term. Even adding noise to transition dynamics by itself is also a form of regularization since it makes the value function smooth. .\n\n2. It would be good to explicitly say how (2) differs from typical Bellman by incorporating Psa. (e.g. note that different from usual Bellman, there is this term that accounts for minimax).\n\n3. When describing uncertainty perturbation by \\bar{P}_{sa} + \\tilde{P}_{sa},  What is a norm on \\tilde{P}_sa? P_{sa} is a probability distribution, do authors mean some kind of function norm on a distribution? The notation describing Psa is very confusing in general, it’s unclear what the authors mean by cross in line 68. I understood the high-level idea that it’s a state-action independent rectangular bound.\n\n4. Where is algorithm 2?\n\n5. Can we be more precise about which kind of uncertainty sets the algorithm can handle? The authors utilize an ellipsodial representation but it would be nice to be fully general.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "N1Yie2lOAX", "reviewer": "Reviewer_2mmx", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper tackle the issue of robustness in robot reinforcement learning with various environmental settings by introducing the uncertainty set regularizer. The proposed method successfully implemented USR to SAC and demonstrated its superiority in both Mujoco simulation and simple real robot settings.", "summary_of_recommendation": "I recommanded a weak reject based on my comment above.", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "Please see my comments above.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The main concerns are:\n1. the paper structure should be updated. It is suggested to summarize the algorithm process in paper rather than appendix. Also it is recommaned to revise the paper to proper introduce the related works in paper.\n2. the experimental results should be hugely improved to let reader fully understand the superiority of USR, especially adv-USR. Besides the Robust-AUC, what is the positive impact of adv-USR to the agent's behavior and the corresponding learning curves?\n3. The real experiment was less introduced, please detail the result and demonstrate the advantages of Adv-USR clearly.", "extra_scores": {"impact": "2: Unsure, difficult to assess."}, "extra_text": {}}, {"review_id": "-6umSdZ-KVg", "reviewer": "Reviewer_5r3v", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper studies the problem setting of robust reinforcement learning, where the goal is to learn a robust policy that performs well in the worst-case environment dynamics. It proposes Uncertainty Set Regularized Reinforcement Learning, which extends the robustness-regularization duality to a continuous state space and incorporates it into a policy iteration framework. The authors propose generating an adversarial uncertainty set aimed at performing better against perturbations. The paper includes results on the Real-world RL benchmark along with sim-to-real results on an A1 robot and proposes a new evlaution metric, Robust-AUC, for evaluating policy robustness.", "summary_of_recommendation": "I think this paper has some promising ideas and results but is not quite at publication level and would benefit from additional analyses and clarifications.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "See weaknesses above.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths:\n- The paper seems interesting and it seems to have some novel contributions, e.g. adversarial uncertainty set, incorporated USR into the policy interation framework.\n- The method seems to be pretty effective compared to baselines in the evaluations. \n- On the whole, the paper and appendix are well written. \n\nWeaknesses:\n- The method relies on learning a local transition model but this is likely nontrivial to learn in many scenarios, such as vision-based learning systems. If there is not sufficient coverage of data for this model, it will likely be quite inaccurate. \n- It would be good to see some additional empirical analyses and ablations, especially on how the approximated target value using USR-Adv compares to the baseline methods and the sensitivity to hyperparameters and amount/coverage of data.\n- It is a bit confusing what the authors want to focus on as the key takeaway and the main contribution. The main (first) contribution emphasized is extending the robustness-regularization duality to continuous control tasks, but this is not a large contribution or a difficult derivation. I think the authors should focus more on a clearer takeaway.\n- To clarify, are the environment parameters required of all data during training? I think if so, this is a large assumption that should be emphasized.", "extra_scores": {"impact": "2: Unsure, difficult to assess."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers agree that this paper makes a worthy contribution to CoRL -- it introduces a method for robust RL that can handle continuous domains, and validates the method in both simulation and on a real unitree A1 robot. I encourage the authors to incorporate the reviewer feedback in the final version. This includes incorporating the new experiments into the paper, clarifying points of confusion, and including qualitative results from the real robot experiments.
Decision: Accept (Poster)