Robust Adversarial Reinforcement Learning via Bounded Rationality Curricula
Authors: Aryaman Reddi
Maximilian Tölle
Jan Peters
Georgia Chalvatzaki
Carlo D'Eramo
Venue: ICLR 2024 spotlight (2024)
Keywords: reinforcement learning
adversarial
bounded rationality
curriculum
Primary area: reinforcement learning
TL;DR: A novel approach for adversarial reinforcement learning that adopts a game-theoretical perspective based on bounded rationality to improve the robustness of obtained policies.

Abstract: Robustness against adversarial attacks and distribution shifts is a long-standing goal of Reinforcement Learning (RL). To this end, Robust Adversarial Reinforcement Learning (RARL) trains a protagonist against destabilizing forces exercised by an adversary in a competitive zero-sum Markov game, whose optimal solution, i.e., rational strategy, corresponds to a Nash equilibrium. However, finding Nash equilibria requires facing complex saddle point optimization problems, which can be prohibitive to solve, especially for high-dimensional control. In this paper, we propose a novel approach for adversarial RL based on entropy regularization to ease the complexity of the saddle point optimization problem. We show that the solution of this entropy-regularized problem corresponds to a Quantal Response Equilibrium (QRE), a generalization of Nash equilibria that accounts for bounded rationality, i.e., agents sometimes play random actions instead of optimal ones. Crucially, the connection between the entropy-regularized objective and QRE enables free modulation of the rationality of the agents by simply tuning the temperature coefficient. We leverage this insight to propose our novel algorithm, Quantal Adversarial RL (QARL), which gradually increases the rationality of the adversary in a curriculum fashion until it is fully rational, easing the complexity of the optimization problem while retaining robustness. We provide extensive evidence of QARL outperforming RARL and recent baselines across several MuJoCo locomotion and navigation problems in overall performance and robustness.

Published as a conference paper at ICLR 2024
ROBUST ADVERSARIAL REINFORCEMENT LEARNING VIA BOUNDED RATIONALITY CURRICULA
Aryaman Reddi .2, MMaximilian Tölle
Jan Peters 1,2,,44
Georgia Chalvatzaki 12, Carlo D'Eramo 1,2,5
1 'Department of Computer Science, TU Darmstadt, Germany
Hessian Center for Artificial Intelligence (Hessian.ai), Germany
GGerman Research Center for AI (DFKI), Systems AI for Robot Learning, Germany
*Center for Cognitive Science, TU Darmstadt, Germany
'Center for Artificial Intelligence and Data Science, University of Würzzurg, Germany
ABSTRACT
Robustness against adversarial attacks and distribution shifts is a long-standing goal of Reinforcement Learning (RL). To this end, Robust Adversarial Reinforce- ment Learning (RARL) trains a protagonist against destabilizing forces exercised by adversary in a competitive zero-sum Markov game, whose optimal solution, i.e., rational strategy, corresponds to a Nash equilibrium. However, finding Nash equilibria requires facing complex saddle point optimization problems, which can be prohibitive to solve, especially for high- gigh-dimensionall control. In this paper, we propose a novel approach for adversarial RI. based on entropy regularization ease the complexity of the saddle point optimization problem. We show that the solution of this antroy--egularrzz Equilibrium (QRE), generalization of Nash equilibria that accounts for bounded rationality, i.c., agents sometimes play random actions instead of optimal ones. Crucially, the connection between the entropy-regularize ables free modulation of the rationality of the agents by simply tuning the temper- ature coefficient. We leverage this insight to propose our novel algorithm, Quantal Adversarial RI. (QARL), which gradually increases the rationality of the adver- sary in curriculum fashion until it is fully rational, easing the complexity of the optimization problem while retaining robustness. We provide extensive evidence of QARL outperforming RARL and recent baselines across several MuJoCo lo- comotion and navigation problems in overall performance and robustness.
problem corresponds Quantal Response
objective and QRE en-
INTRODUCTION
The glaring success of deep reinforcement learning (RL. has been largely obtained in controlled simulated problems under negligible disturbances (Haarnoja al., 2018a; Mnih et al., 2015; Barth-Maron et al., 2018; Schulman et al., 2017; Fujimoto et al., 2018). However, deep RL methods widely known be prone overfit the observed tasks, such that distribution shifts, or even adversarial attacks, can dramatically undermine their performance (Gleave et al., 2019; Zhang et al., 2018). For the ultimate deployment of deep RL in realistic high- sigh-dimensional problems, pivotal importance to endow agents with robustness w.r.t. unmodeled perturbations. Robust Adver- sarial Reinforcement Learning (RARL) is an appealing approach to obtain robust policies in RL by training protagonist agent against destabilizing actions applied by an adversarial agent (Pinto et al., 2017). RARL solves a two-player zero-sum Markov game (Littman, 1994), where the protagonist can execute actions in the environment maximize measure of performance, and the adversary trained to perform adversarial actions to minimize the same measure. By learning tasks under destabilizing perturbations, the protagonist obtains robust skills counteracting distribution shifts and adversarial attacks when deployed.
of
"Correspondence <aryaman, reddi maddi@tudda rmstadt..ed

---

Published as a conference paper at ICLR 2024
The zero-sum Markov game solved by RARL is a saddle point optimization problem whose solution is the minimax or Nash equilibrium, which can be prohibitive to find in case of strongly non-convex non-concave objective functions typical when dealing with deep RL control problems (Ostrovskii et al., 2021). RARL tackles these issues by adopting an alternating approach where one agent is trained while the other is stationary. This iterative approach is inspired by the fictitious-play heuris- tic of game theory (Brown, 1951; Heinrich & Silver, 2016) that, despite not ensuring convergence, performs well in practice (Hofbauer & Sandholm, 2002; Brandt et al., 2010). Nevertheless, several works have shown that RARL can fail to converge to good solutions, being unstable even in basic linear quadratic regulation (Zhang et al., 2020a). Attempts have been made to tackle the tendency of RARL to stick to suboptimal solutions, e.g. perturbing the gradient with Langevin dynamics (Ka- malaruban et al., 2020) or using extragradient optimization (Cen et al., 2021).
In this paper, we propose new approach based on entropy regularization to ease the complexity of zero-sum Markov
the adversarial RL. optimization problem. We formulate an game where two agents can compete while maximizing the entropy of their respective policies. We show that the solution to this problem is a Quantal Response Equilibrium (QRE), a generalization of the Nash equilibrium that accounts for bounded rationality of the agents (Samuelson, 1995; McK- elvey & Palfrey, 1995; Goeree et al., 2016). We note that, in a setting where the protagonist and the adversary are completely rational, i.e., they execute their policies without noise, the solution is a saddle point corresponding to a Nash equilibrium. This solution ensures the most robust behav- ior (Pinto al., 2017). but is typically difficult to find due the strongly non-convex non-concave optimization problem (Kamalaruban et al., 2020; Zhang et al., 2020a). Conversely, in a setting where the protagonist is completely rational and the rationality of the adversary minimal, i.e., the adversary plays only random actions, the optimization problem reduces to a regular maximization problem for the protagonist at the cost of lower robustness of the obtained policy. This entails that there is a trade-off between the complexity of the optimization problem and the robustness of the ob- tained policy, and we posit that an effective balance can be found by properly tuning the rationality of the adversary.
can be shown that the temperature parameter of the entropy-regularize problem corresponds to the rationality parameter of the respective QRE (Savas et al., 2019; Cen et al., 2021), which affects the complexity of the optimization problem and the robustness of the optimal solution. Based on this, our main contribution is a novel algorithm for adversarial RL, that we call Quantal Adversar- ial RL (QARL), that actively modulates the rationality of the adversary to ease the complexity of the optimization problem while retaining the robustness of the obtained solution. QARL initially solves an adversarial problem with a completely random adversary, thus solely focusing on the per- formance improvement of the protagonist, neglecting robustness. Then, gradually increases the rationality of the adversary until it reaches complete rationality and, hence, maximal robustness. While an arbitrary method can be chosen for changing the rationality of the adversary, we provide a curriculum- approach that automatically tunes it while keeping track of the learning progress of the protagonist. This way, we can maintain a steady performance improvement together with the increase in robustness. We demonstrate that our approach facilitates the learning of the protagonist, and we show that QARL outperforms RARL and related baselines in terms of performance and ro- bustness across several high- -igh--iiensional MuJoCo problems, such navigation and locomotion, of the DeepMind Control Suite (Todorov et al., 2012; Tunyasuvunakool et al., 2020).
RELATED WORKS
Over the last decades, unwavering effort has been put into endowing learning agents with robust behavior in RL (Moos al., 2022). Some recent attempts are based on a zero-sum game problem formulation, also known as robust adversarial RL (RARL), where an adversary agent is trained to ex- ert destabilizing actions on protagonist agent trying to accomplish a given task (Pinto et al., 2017: Oikarinen et al., 2021). This way, the protagonist learns to solve task while counteracting external disturbances, thus being more robust when deployed. The quest initiated by RARL for achieving ro- bustness with adversarial RL has been pursued by several subsequent works that attempt to overcome the limitations of RARL. For example, Pan et al. (2019) proposes to model risk explicitly to avoid catastrophic behavior, while Zhang et al. (2020a) shows that RARL is affected by instability and convergence issues even in simple linear quadratic regulation problems. Furthermore, Kamalaruban et al. (2020) proposes the use of Langevin dynamics for optimizing the challenging non-convex
2

---

Published as a conference paper at ICLR 2024
non-coneave objective of the zero-sum game of RARL, while Zhai al. (2022) studies the benefit on the adversary. Similarly to this paper, some works investigate the potential of curricula in adversarial RL, e.g., motivated by H -contol,,,, they build a sequence of increasingly complex subtasks (Song & Schneider, 2022; Ao et al., 2022). Sheng et al. (2022) proposes to generate curricula over the strength of an adversary that exerts adversar- ial attacks on the protagonist by acting on its action space. Of these, we consider Kamalaruban al. (2020) and Sheng et al. (2022), as they both aim to improve the robustness and stability of the robust adversarial RL scheme, which is the primary aim of this work. Pan et al. (2019) aims to avoid catastrophic policy collapse, while Zhai al. (2022) uses model-based disturbance bounding which requires more system knowledge than we assume. Song & Schneider (2022) and Ao et al. (2022) encourage task robustness through adversarial subtask generation as opposed to adversarial disturbance modeling, which goes beyond the scope of the non-hierarchical perspective we adopt, but could nonetheless be interesting for future work.
of imposing iiriaation-ieqquttooncomm
PRELIMINARIES
We consider two-player zero-sum Markov games (Littman, 1994; Perolat et al., 2015) formulated as a Markov decision process (Puterman, 1990) M = A1, R. P. 7: 1). where S is the contin- uous state space observed by both players, Ai and A₂ are the continuous action spaces of the two agents, and P 8 A x A2 S - I is transition probability density, R S Ai x A x S + I a reward function, 7 E [0, 1) is a discount factor, and i a probability density function over initial states. For given policies μ and of the two players, the discounted return is defined as
TH-1 Σ 7 7'R(st, at, ot 8441) 80 s,P V.
(s)(() = (
(1)
In two-player zero-sum Markov game, one agent has obtain a policy μ that maximizes the discounted return J, while the other aims at obtaining a policy v that minimizes it. The optimal policies J² and n² are the ones inducing the return
Ju max min JA min max Jp..: P
which corresponds a minimax or Nash equilibrium for the Markov game (Perolat et al., 2015). Is well-known that obtaining Nash equilibria in Markov games is challenging problem that requires solving minimax equilibria at every state (Littman, 1994; Pérolat et al., 2017: Song et al., 2023). Most methods to compute optimal policies suffer from exponential complexity in the number of actions for discrete Markov games; hence, they are impractical for problems with many actions and unfeasible for -igensiiieniinall control problems.
Quantal response equilibrium and bounded rationality Finding a Nash equilibrium requires solving saddle point optimization problem, for which the corresponding knife-edge best-response function can be challenging to optimize due to strong non-convexity non-concavity. The Quan- tal Response Equilibrium (QRE) is generalization of the Nash equilibrium, which can arbitrarily smoothen best-response functions, thus casing the optimization (McKelvey & Palfrey, 1995; Goeree al., 2016). In practice, smoothing best-response functions results in modeling games where play- ers do not always select their best strategy, sometimes selecting suboptimal actions. This behavior known as bounded rationality, in contrast full rationality typical of knife-edge best-response strategies, i.e., Nash equilibria. Consider a two-agent Markov game with a given payoff X, pa- rameter T R-+ and two available actions per agent. Denote the strategies as oij where and indicate the agent and the action, respectively. A particular form of QRE, known logit QRE and used across our work, is
exp(Xo))/7)
exp(Xoie/7))
q = exp(Xo /T) + exp(XxX2/
(3)
exp(Xo),/T) +xx(Xx/2/7)
with die = 1 1--0-; 01 and 022 = 1 وبو By adopting logistic function shape, the logit QRE models strategies with bounded rationality depending on the expected payoff X and a belief o" about the strategy of the other agent, where modulates the rationality level of the agents. For 7 - do, the agents play completely random actions; on the contrary, when 7 + 0, the strategies become completely rational, thus becoming Nash equilibrium.
3

---

Published as a conference paper at ICLR 2024
Robust adversarial reinforcement learning A special case of two-player zero-sum Markov game is one where an adversary agent is trained to execute destabilizing actions on a protagonist trained to accomplish given task (Kamalaruban al., 2020; Pinto et al., 2017; Zhang et al., 2020a). By learning to deal with external disturbances while solving its task, the protagonist develops skills robust to adversarial attacks and distribution shifts when deployed. RARL (Pinto al., 2017) is an algorithmic solution for solving such two-player Markov games that obtains robust behavior in gigh-dimessional control problems, e.g., MuJoCo (Todorov al., 2012). RARL consists of a loop of separate training phases for the protagonist and the adversary. i.e., the protagonist performs rollouts and training steps while the adversary is stationary and vice versa.
each timestep t, both agents observe the same state E 8. perform actions a, E Ai and ot E A2, and reach state st+1, while the protagonist obtains reward r = R(st, at. oi; 81+1) and the adversary gets r rt.
BOUNDED RATIONALITY IN ADVERSARIAL REINFORCEMENT LEARNING
A prerequisite for our approach is the ability to model and control the rationality of agents in an adversarial RL setting. Crucially, entropy regularization provides a theoretical framework to model Markov games under bounded rationality (Cen et al., 2021; Savas et al., 2019). Consider the follow- ing maximum entropy formulation of the adversarial RL objective (2)
Ju max min JuN. + 3H(()) atH(v).
(4)
where H. is the Shannon entropy of a distribution and a, R+ are two temperature coefficients. The solution to this problem is known as Quantal Response Equilibrium (QRE) (McKelvey & Pal- frey, 1995; Goeree et al., 2016), a generalization of the Nash equilibrium that extends it to games where agents do not play with complete rationality'.
Definition 4.1 For any rngropy-r zero-sum Markov game, the QRE is exp ()p
1r' (a+1s)
exp (Ju-(,,))))1)))
Vs V E
J. exp (Ju+ (s.b)/B)
(5)
1² (a2ls) J, (xx
(ap(-J,-
exp (-, (s))))/a), Vs E
E A2.
where (s,s, a) (or Ju(s, a)) indicates the performance obtained when the adversary (or protagonist) executes action a in state while the protagonist (or adversary) follows its policy y (or v).
The solution (5) to the entropy-regularizee objective (4) can be considered as the adversarial RL counterpart of the well-known solution for maximum entropy RI. (Haarnoja et al., 2018a; 2017; Asadi & Littman, 2017). Key to our analysis is that the temperature coefficients a and B can ef- fectively modulate the rationality of the agents under a game- theoretical lens (McKelvey & Palfrrey 1995; Goeree et al., 2016). For example, for a - o0, the entropy of the adversary is maximized, and its rationality is minimized. Conversely, if a + O, the reguopy-regularrized objective (4) accounts only for the performance term, and the adversary is fully rational.
4.1 TEMPERATURE-COM CONDITIIONN VALUE FUNCTIONS AND POLICIES
QRE are shown to exist in finite Markov games with bounded rationality (McKelvey & Palfrey, 1995; Goeree et al., 2016). However, in Markov games with large state-action spaces, solving the QRE in closed-form becomes computationally unfeasible and RL becomes more suitable at solving the two-player optimisation. We point out that QRE closely resembles the solution of the maximum entropy RL problem. For example, the well-known soft-actor critic (SAC) algorithm (Haarnoja et al., 2018a;b) approximates policies of the form T--ss) ~ exp (Q(x.))TT where Z(-) is a partition 2(5,7) function and e R+ the temperature. Thus, we introduce the use of SAC adversarial RL to obtain a practical method for approximating QRE in the case of continuous actions.
For our purpose of controlling the rationality of the adversary, we propose to condition its policy on the temperature coefficient. To do this, we define what we call temperature-conditooneed value
'QRE are homogeneous for B. Here, we use heterogeneous QRF, i.e., a B (Goeree al., 2016; Rogers al., 2009).
4

---

Published as a conference paper at ICLR 2024
functions and policies, whose definition follows directly from the one for soft action-valle function in Theorem 1 of Schaul et al. (2015) and Haarnoja et al. (2017).
Corollary 4.2 For any policy π and temperature T E Rt,, let the temperature optimal soft vection-value function and value function be defined as
(se: at, T), =)= -
7 (reti + TH(T" ((((T(((1(14))))))))) (-18448, +)))
(6)
(= = exp ( da'.
Then, the optimal policy conditioned on temperature T is
x" (aelss T) = exp Q* (0t, V* T)
=exp
(8)
(Equation (8)) can be straightforwardly rewritten as IT (acls;, r) JA exp (Q" (st.a' a' ,r)/))da' Using the action-value function Q' as an estimate of Jx the resemblance of x with QRE (Equation (5)) is clear, which enables us to derive the following.
The lemperature--condiioonee policy x²
exp (e..0(t)))
Proposition 4.3 For any entropy regularzed zero-sum Markov game M as defined in Equation (4), given the lemperature-condiioonel optimal action-value function Q and value function V*, the temperature-condeee conditioned policies are
4 (a (s, = exp (s, .8)88)
Tx (Q. X exp (xp(
e S,a' A
(9)
exp (" .o)aa
// (c2)s,a)=
x (xp(-Q2 a)/a), e S, S, E A2.
where Q7 (s, a, T) indicates the performance conditioned on temperature T obtained when one agent executes action a in state while the other agent follows its policy x. We have that {pr v²) is a QRE for the Markov game M.
This result establishes a connection between QRE and optimal policies in two-player Markov games, which enables modeling adversarial RL problems with bounded rationality.
4.2 AUTOMATIC GENERATION OF BOUNDED RATIONALITY CURRICULA
So far, we have generalization of adversarial RL to bounded rationality through entropy regular- ization and a simple way to tune rationality by acting on the temperature coefficient. Crucially, are now able to shape the saddle point optimization problem of adversarial RL by modulating the rationality of the adversary through its temperature a. We recall that our goal is to case the complexity of potentially strongly non-convex non-concave optimization problems, typical in high- dimensional control while retaining the robustness of the obtained policies. We propose to initially solve an adversarial problem with completely random adversary, i.c., a simpler plain maximization of the performance of the protagonist, neglecting robustness. Then, gradually increase the rationality of the agent and thus the complexity the optimization prob- lem by consciously decreasing the temperature a, hence improving the robustness of the obtained policy. Eventually, we have a - retrieving the regular adversarial RL problem with a com- pletely rational adversary and, consequently, maximal robustness of the obtained policy. The benefit of our approach that, instead of attempting to solve a complex saddle point optimization from the beginning, it starts from simpler one and makes it gradually more complex in curriculum fashion (Zhang et al., 2020b; Svetlik et al., 2017; Klink et al., 2020; 2021).
oo, which results in
We are left with the decision of how to tune the temperature of the adversary during training. While a viable solution would be to gradually anneal the temperature during training, it is prohibitive to manually select proper decay speed that ensures good balance between the complexity of the problem and its feasibility. Ideally, the temperature should be gradually decreased to increase ra- tionality while avoiding negative impact the performance of the protagonist. Therefore,
5

---

Published as a conference paper at ICLR 2024
Algorithm Quantal Adversarial Reinforcement Learning (QARL)
1: Input: Initial critic parameters v 2h2 인. 222 2 2/ 1 2 initial actor parameters ø', x²; D 0, D = 0; initial temperature distribution parameters performance lower bound ç; #sampled temperatures N; #Monte-Carlo rollouts M; initial temperatures a, 93; #steps T.
2: for each iteration do
3: $ + i 22 f-
o - vs.
>> Target networks update Sampling of temperatures of the adversary
4: 01....N Pw (a)
5: for = ......,o do
6:
for ( ,...., do
Transitions collection for the adversary
7:
" ~ Mo (a (s, On))
8:
0f ~ (a (1,
9:
8111 ~ P(5 al a
10:
D D2 D2 ( Ci Fr. S141,
11:
o 4/ x² + ADVERSARY ( D2
12:
end for
13:
end for
for i
1, ...... N do
15:
for
1 = 1,.... do
> Transitions collection for the protagonist
0 Mot (s,
a u (a])s,),
18:
8141 P( af)
19:
D DJ PU To a4))
20:
1/ 4/2 d¹, B PROTACCONSST.LPDAAECCC w2, a.. BL)
21:
end for
22: end for
23:
Update w optimizing (10), estimating (a) [w)) with M rollouts
24:
end for
25:
Return: Trained actor parameters Φ¹, 0²
provide way that automatically tunes the temperature while keeping track of the learning progress of the protagonist. We define a probability distribution (a), parameterized by w, over the temper- ature coefficient a of the adversary in Objective (4). Given that a E R+ it is convenient to model the probability distribution pa as gamma distribution ['k,, 0). where of the distribution are gradually updated to reach a target distribution p(a)). under the constraint that the update does not result in decreasing the performance Ju below certain threshold Since want the adversary to converge to complete rationality (i.c., a + 0), as in RARL, we set the tar- get distribution centered in a = O. Formally, we solve the following constrained optimization problem
(k, 0). The parameters
sharp gamma distribution μ T(1, 10-3), approximating Dirac distribution
min DRL (aw(a))l((a)
s.t. Epula) [up((a)] 2 & DEL (pw(a)) (owr))))wwaa)))) Sc.
(10)
where w' is the parameter vector before the update. By minimizing the KL-divergence between the current distribution Pw and μe, we progressively sample lower a, thus increasing the rationality of the adversary. While the second constraint avoids abrupt changes the distribution pu, the first constraint crucially ensures that the update of the distribution Pw does not make the overall perfor- mance drop below certain threshold E. This way. QARL progressively increases the rationality of the adversary while preventing catastrophic performance drops, eventually reaching an adversary with full rationality, and thus maximal robustness, as RARI. (Pinto et al., 2017).
4.3 QUANTAL ADVERSARIAL REINFORCEMENT LEARNING
The constrained optimization problem (10) can be solved by minimizing its Lagrangian
C(w, X, Dxa. (p)(a))(((aa) + (8 (a)(a)) [w)](w)) +n (D (p)))))))))) (11)

---

Published as a conference paper at ICLR 2024
Caripole
Walker
400
375
350
230
200
275
uotarnue
150
250
175
QARL (203)
RARL (47)
SAC MixadhE-LD (73)
CAT
Rurte Cunsaluu (67)
SAC
(188)
GARL
(336)
RARL (30))
SAC MivadRe-4 (2400)
CAT (282)
Forte Cuntrlum (297)
SAC (329)
Hogper Hop
200
Retum vs Weree Adveraary
330
80
Retum Werst
250
200
20
100
MixudNN LD
CAT (28)
Pore Compeduu
SAC (43)
MinudNE 4D
Cortesium (307)
(46)
(40)
(340)
0161
(236)
(317)
(12)
Figure 1: Performance of QARL and baselines on MuloCo control problems (see title of boxplots), evaluated at the end of training against an adversary trained against the frozen trained protagonist. The number next to the name of each algorithm is the average performance across 10 seeds.
with À, 7 2 O. While the KL-divergences be computed exactly using the closed-form formula of the gamma distribution, the expected performance (p)(a) [ne(w) needs to be approximated with an importance-sampler Monte-Carle estimate
(a (is,s, Per (ay)
(a)( [w))
(12)
M
where (s0, os) is an estimated value function obtained by collecting trajectories starting from initial states 80,6. with the protagonist and the adversary following their policies μ and both con- ditioned on the temperature a; of the adversary. Algorithm 1 summarizes our Quantal Adversarial RL (QARL) algorithm. Although not strietly necessary, condition the policy of the protago- nist on the temperature of the adversary (line and 16) for better adaptation. We also clarify that while the temperature of the adversary follows automatic curriculum generation scheme, the protagonist follows the temperature scheduling proposed by SAC (Haamoja et al., 2018b) (line 20).
5 EXPERIMENTAL RESULTS2
MuJoCo control We consider a broad set of MuJoCo control problems (Todorov et al., 2012) from the DeepMind Control Suite (Tunyasuvunakool et al., 2020), and compare method to RARL (Pinto et al., 2017) and recently proposed enhancements that address the complexity of the saddle point optimization problem of RARL. Namely, we consider an extension of RARL that Langevin dynamics (MixedNE-LD) to escape local optima (Kamalaruban et al., 2020), and Curricu- lum Adversarial Training (CAT) (Sheng et al., 2022) that applies hand-crafted curriculum scheme on the strength of the adversary. Additionally, we introduce and evaluate a method, that we denote as that applies our automatic curriculum scheme over the maximum strength of the adversary instead of its rationality (details in Appendix A). In contrast with the theoretical formula- tion for QARL., Carce-Curriculum has the limit of having a heuristic nature and hinges on the ability to change the action boundaries of the adversary during training. Nevertheless, given its simplic- ity and intuitive motivation, we find Curee-Curiiulum an interesting additional baseline. Finally, compare to regular SAC trained against adversary reference (Haarnoja et al., 2018b).
"odde available https: /ii thub. com www.e/ow.///a/eeeeeee Aryamankeddd 99 quantal -adversarial-n

---

Published as a conference paper at ICLR 2024
Cartpole- Swingup Sparso
QARL (1.97)
MARL (40)
SAC MixedNE--LD (12)
CAT (111)
Force Curriculum (110)
SAC (321)
8.00
0.30
18 Mass Cart
Mass 135
Cart
Mass Cart
Mass
Mass
Walker
QARL (308)
RARL (263)
SAC. MicedNF-LD (291)
CAT (288)
Foroe Curriculum (266)
SAC (2:39)
300
24
0.
1.0
25
10
25
Lo Coefficient
Coefficient
83
23
Coefficient
Hopper
Hop
(23)
MixedNE LD
Currcclum
of
Frrction Coefficient
Friction Coefficient
Friction Coefficient
Friction Coefficient
Friction Coefficient
Friction Coefficient
Cheetah
Run
QARL (276)
RARL (245)
SAC MixedNEELL (226)
CAT (219)
Force Currccluu (264)
SAC 12300
-300
220
100 Priction Coefficient
0.5
25 16 1.5
0s 12 L.S. Friction Coefficient
6.5 1n
3. 1. 1.5 Friction Coefficient
05
14 1.4.
Friction Confficient
Friction Coefficient
Friction Coefficient
Figure 2: Robustness analysis of QARL and related baselines on MuJoCo swingup and locomotion problems (see title of heatmaps). Each heatmap shows the performance obtained for varying proper- ties of the environment, described in the axes. The number next the name of each algorithm the average performance across seeds.
In each environment, the adversary exerts forces on the protagonist disrupt its performance. As done in Pinto et al. (2017); Kamalaruban et al. (2020), we consider environment-specife disruptions with high strength to test the reliability of considered adversarial algorithms under intense destabilization (details in Ap- pendix B). To adequately assess the performance of the protagonist in a minimax sense, we fix the protagonist obtained at the end of training and subsequently train an adversary against it, which then used test time. Moreover, we evaluate the robustness of the trained protagonists to varying test conditions, such as mass
Algorithm
Performance
Robustness
SAC MixodNE-LD
-16.6 0.2%
+20.0 1.1%
RARL
-5.7 4: 0.3%
+10.1 1: 0.6%
Ince-Currieulum
-14.7 4 0.3%
-3.3 0.6%
CAT
18.00 2: 0.2%
111.7 0.8%
QARL (ours)
+..2 +48..7 1.2%
Table 1: Performance and robustness per- centage improvement w.r.f. SAC for QARL. and baselines 15 MuJoCo con- trol problems.
Run
Auto (25)
Linar Curkuren
Port Cumbuuu 122 Raduce Sampling (2)
Auta (290)
Unai Curiaau 2231 Petc Canfaaus un 2201 Redazed Sansiing (20)
Hoas of bro
300
2013
05
10
Ieddion Coctideen:
Trition Cocticent
Conttceen:
Conffcett
Coctiontt
Figure 3: Robustness analysis different curricula for QARL on two locomotion problems.
8

---

Published as a conference paper at ICLR 2024
Figure 4: Trajectories of the quadruped agents trained using RARL (top) and QARL (bottom) in the maze environment. The adversary can blow wind from right left in the area highlighted in yellow. The goal position is marked as a green sphere. The leg that the Quadruped uses to anchor when using the policy obtained with QARL is highlighted red.
and friction, against no adversary. We conduct this evaluation on 15 MuloCo control problems selection of them shown in Figures and 2, while an overall assessment shown Table (details in Appendix C). QARL outperforms the baselines in average performance and robustness, thus evincing a superior ability to counteract the adversary. While more robust across test conditions than SAC, the other baselines suffer in nominal performance due to the highly destabilizing adver- sarial forces that hinder learning. The non non-curriculum methods (RARL and MixedNE-LD) perform notably poorly for the environment 'Cartpole Swingup Sparse", in which the sparse reward setting enables the adversary to easily hinder the protagonist. In Figure 3, we also study the effectiveness of our automatic curriculum generation method in terms of robustness compared other simpler curriculum schemes. Linear Curriculum refers to basic linear annealing of a. Point Curriculum consists of variant of the optimization problem (10) where is scalar value instead of being sam- pled from a distribution. Finally, Reduced Sampling applies our automatic curriculum generation method, sampling only single value each iteration.
Maze navigation We further analyze QARL in navi- gation problem with a MuJoCo quadruped agent (Tunya- suvunakool et al., 2020) that requires learning locomotion skills to reach goal while contrasting an adversary blow-
Success Rate (%)
ing wind. Figure shows the environment and behavior of QARL against RARL, evincing that RARI. collapses suboptimal policy while QARL is able to solve the problem by learning gait where one leg is used to anchor the ground while using the others to move towards the goal. Moreover, we study the robustness of the obtained policy for QARL and baselines against forces obtained by multiplying the one in training by a relative force ranging from 4. (Figure 5). observe that QARL can re- sist forces times stronger than the nominal one, while RARL and the other baselines are not able to, with the ex- ception of
60
40
0
Relative Force
QARL
SAC MindNE-LD
SAC
RARI
Forza
CAT
Figure 5: Robustness analysis for wind forces the maze environment. Rela- tive force is w.r.t the one in training.
which sometimes succeeds,
CONCLUSION
We introduced Quantal Adversarial Reinforcement Learning (QARL), a method leveraging the con- nection between maximum entropy RL and bounded rationality, to facilitate the optimization of the complex saddle point optimization problem tackled by adversarial RL. Our method is backed up by a solid theoretical foundation and extensive empirical validation on a broad set MuJoCO control problems. By combining the benefits of curriculum and adversarial learning. QARL reduces the amount of time and energy to robustify agents, which is particularly desirable for achieving adaptive behavior in real-world applications, ranging from autonomous vehicles to assistive robotics.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
This work was funded by the German Federal Ministry of Education and Research (BMBF) (Project: 011S22078). This work was also funded by Hessian.ai through the project "The Third Wave of Artificial Intelligence 3AI by the Ministry for Science and Arts of the state of Hessen. We acknowledge the grant 'Einrichtung eines Labors des Deutschen Forschungszentruum für Künstliche Intelligenz (DFKI) an der Technischen Universität Darmstadt' of the Hessisches Ministerium für Wissenschaft und Kunst.
REFERENCES
Shuang Ao, Tianyi Zhou, Jing Jiang, Guodong Long, Xuan Song, and Chengqi Zhang, Eat-c: Environment-advee adversarial sub-task curriculum for efficient reinforcement learning. In Interna- tional Conference on Machine Learning, pp. 822-843. PMIR, 2022.
Kavosh Asadi and Michael Littman. An alternative softmax operator for reinforcement learning. In International Conference on Machine Learning, pp. 243-252. PMLR, 2017.
Gabriel Barth-Maron, Matthew W Hoffmman, David Budden, Will Dabney, Dan Horgan, Dhruva Tb, Alistair Muldal, Nicolas Heess, and Timothy Lillicrap. Distributed distributional deterministic policy gradients. arXiv preprint arXiv: 0011800000066... 2018.
Felix Brandt, Felix Fischer, and Paul Harrenstein. On the rate of convergence of fictitious play. In Algorithmic Game Theory: Third International Symposium, SAGT 2010, Athens, Greece, October 18-20, 2010. Proceedings 3, pp. 102-113. Springer, 2010.
George W Brown. Iterative solution of games by fictitious play. Act. Anal. Prod Allocation, 13(1): 374, 1951.
Shicong Cen, Yuting Wei, and Yuejie Chi. Fast policy extragradient methods for competitive games with entropy regularization. Advances in Neural Information Processing Systems. 34:27952- 27964, 2021.
Carlo D' D'Eramo, Davide Tateo, Andrea Bonarini, Marcello Restelli, and Jan Peters. Mushroomrl: Simplifying reinforcement learning research. The Journal of Machine Learning Research, 22(1): 5867-5871, 2021.
Scott Fujimoto. Herke Hoof, and David Meger. Addressing function approximation error in actor- critic methods. In International conference on machine learning, pp. 1587-1596. PMLR, 2018.
Adam Gleave, Michael Dennis, Cody Wild, Neel Kant, Sergey Levine, and Stuart Russell. Adver- sarial policies: Attacking deep reinforcement learning. arXiv preprint arXiv:/903: 10615, 2019.
Jacob K Goeree, Charles A Holt, and Thomas Palfrey. Quantal response equilibrium. In Quantal Response Equilibrium. Princeton University Press, 2016.
Tuomas Haamoja, Haoran Tang, Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep energy-based policies. In International conference machine learning, pp. 1352-1361. PMLR, 2017.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International confer- ence on machine learning, pp. 1861-1870. PMLR, 2018a.
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and appli- cations. arXiv preprint arXiv:1812.05s:
2018b.
Johannes Heinrich and David Silver. Deep reinforcement learning from self-play in imperfect- information games. arXiv preprint arXiv:1603.0/ 121, 2016.
Josef Hofbauer and William H Sandholm. On the global convergence of stochastic fictitious play. Econometrica, 70(6):2265-2299) 2002.
10

---

Published as a conference paper at ICLR 2024
Parameswaran Kamalaruban, Yu-Ting Huang, Ya-Ping Hsieh, Paul Rolland, Cheng Shi, and Volkan Cevher. Robust reinforcement learning via adversarial training with langevin dynamics. Advances in Neural Information Processing Systems, 13:8127-81388 2020.
Pascal Klink, Carlo D' D'Eramo, Jan R Peters, and Joni Pajarinen. Self-paced deep reinforcement learning. Advances in Neural Information Processing Systems, 12921669227,, 2020.
Pascal Klink, Hany Abdulsamad, Boris Belousov, Carlo D' Eramo, Jan Peters, and Joni Pajarinen. A probabilistic interpretation of self- learning with applications to reinforcement learning. The Journal of Machine Learning Research, 22(1):8201-822 2021.
Michael L Littman. Markov games as a framework for multi-agent reinforcement learning. In Machine learning proceedings 1994, pp. 157-163. Elsevier, 1994.
Richard D McKelvey and Thomas R Palfrey. Quantal response equilibria for normal form games.
Games and economic behavior, 10(1):6-33 1995.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 51 518(750)):529 533, 2015.
Janosch Moos, Kay Hansel, Hany Abdulsamad, Svenja Stark, Debora Clever, and Jan Peters. Robust reinforcement learning: A review of foundations and recent advances. Machine Learning and Knowledge Extraction, 4(1):276-315, 2022.
Tuomas Oikarinen, Wang Zhang, Alexandre Megretski, Luca Daniel, and Tsui- Wei Weng. Robust deep reinforcement learning through adversarial loss. Advances in Neural Information Processing Systems, 34:26156-26167, 2021.
Dmitrii M Ostrovskii, Babak Barazandeh, and Meisam Razaviyayn. Nonconvex-noncooecave min- optimization with small maximization domain. arXiv preprint 2xXiv:2110003320 2021.
Xinlei Pan, Daniel Seita, Yang Gao, and John Canny. Risk averse robust adversarial reinforcement learning. 2019 International Conference on Robotics and Automation (ICRA), pp. 8522-8528. IEEE, 2019.
Julien Perolat, Bruno Scherrer, Bilal Piot, and Olivier Pietquin. Approximate dynamic programming for two-player zero-sum markov games. In International Conference on Machine Learning, pp- 1321-1329. PMLR, 2015.
Julien Pérolat, Florian Strub, Bilal Piot, and Olivier Pietquin. Learning nash equilibrium for general- sum markov games from batch data. In Artificial Intelligence and Statistics, pp. 232-241. PMIR, 2017.
Lemrll Pinto, James Davidson, Rahul Sukthankar, and Abhinav Gupta. Robust adversarial reinforce- ment learning. In International Conference on Machine Learning, pp. 2817-2826. PMLR, 2017.
Martin L Puterman. Markov decision processes. Handbooks in operations research and management science, 2:331-434, 1990.
Brian W Rogers, Thomas R Palfrey, and Colin F Camerer. Heterogeneous quantal response equilib- rium and cognitive hierarchies. Journal of Economic Theory, 144(4):1440-144) 2009.
Larry Samuelson. Bounded rationality and game theory. Quarterly Review of Economics and fi- nance, 36:17 17-6, 1995.
Yagiz Savas, Mohamadreza Ahmadi, Takashi Tanaka, and Ufuk Topeu. regroiyy- stochas- tic games. In 2019 JEEE 58th Conference on Decision and Control (CDC), pp. 5955..955599999 IEEE, 2019.
Tom Schaul, Daniel Horgan, Karol Gregor, and David Silver. Universal value function approxima- tors. In International conference on machine learning, pp. 1312-1320. PMLR, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: (70.:::770000447,, 2017.

---

Published as a conference paper at ICLR 2024
Junru Sheng, Peng Zhai, Zhiyan Dong, Xiaoyang Kang, Chixiao Chen, and Lihua Zhang. Curricu- lum adversarial training for robust reinforcement learning. In 2022 International Joint Conference on Neural Networks (IJCNN), pp. 1-8. IEEE, 2022.
Yeeho Song and Jeff Schneider. Robust reinforcement learning via genetic curriculum. In 2022 International Conference on Robotics and Automation (ICRA), pp. 5560-5566. IEEE, 2022.
Zhuoqing Song, Jason D. Lee, and Zhuoran Yang. Can we find nash equilibria at a linear rate in markov games? The Eleventh International Conference on Learning Representations, 2023.
Maxwell Svetlik, Mattco Leonetti, Jivko Sinapov, Rishi Shah, Nick Walker, and Peter Stone. Auto- matic eurriculum graph generation for reinforcement learning agents. In Proceedings the AAAI conference on artificial intelligence, volume 31, 2017.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 JEEE/RSJ international conference on intelligent robots and systems, pp. 5026-5033.
IEEE, 2012.
Saran Tunyasuvunakool, Alistair Muldal, Yotam Doron, Siqi Liu, Steven Bohez, Josh Merel, Tom Erez, Timothy Lillicrap, Nicolas Heess, and Yuval Tassa, dm_control: Software and tasks for continuous control. Software Impacts, 6: 6:100022, 2020.
Peng Zhai, Jie Luo, Zhiyan Dong, Lihua Zhang. Shunli Wang, and Dingkang Yang. Robust adver- sarial reinforcement learning with dissipation inequation constraint. In Proceedings of the AAAI Conference Artificial Intelligence, volume 36, pp. 5431-5439, 2022.
Chiyuan Zhang, Oriol Vinyals, Remi Munos, and Samy Bengio. study on overfitting in deep reinforcement learning. arXiv preprint arXiv:/804.00 06893, 2018.
Kaiqing Zhang, Bin Hu, and Tamer Basar. On the stability and convergence of robust ad re- inforcement learning: case study on linear quadratic systems. Advances in Neural Information Processing Systems, 33:22056-225 22068, 2020a.
Yunzhi Zhang, Pieter Abbeel, and Lerrel Pinto. Automatic curriculum learning through value dis- agreement. Advances in Neural Information Processing Systems, 33:7648-7659, 2020b.
12

---

Published as a conference paper at ICLR 2024
0.06
1000
Return vs Adversary
1.0
Force budget
Adversary
-100
Return VS Adversary
Temperature
0.05
MM
Force Budget
0.8
f-
-00
0.04
0.03
40
0.4
40
0.02
Adversary
0.01
-20
0.2
20
0.00
-0
0.0
50
100
150
200
50
100
150
200
Iteration
Iteration
(a) QARL temperature training performance.
(b) PorceCCuriicuum budget training performance.
Progression of the (a) adversary temperature (a) and protagonist temperature (B) of QARL, and (b) adversary force budget of the Force-Curriculun algorithm during training compared to average return against trained adversary in the "Hopper Hop" environment. The performance threshold { = 5 also indicated. Results are shown over 10 seeds.
Figure 6:
A
FOR ADVERSARIAL REINFORCEMENT LEARNING
In addition to the adversarial RL baselines implemented, we also find it worthwhile to com- pare QARL conceptually similar curriculum algorithm over the maximum strength of the adversary, which we refer to as lar curriculum scheme to QARL, whereby force budgets are sampled from a gamma distri- bution which is updated according riculum enforced by clipping the magnitude of the adversary's actions in each dimen- sion according to the sampled force budget f (Line 9 and 19 in Algorithm 2). Figure shows the evolution of the gamma distribution The force budget distribution has initial mean near 0, preventing the adversary from creating large disturbances during early training. As training progresses, this force budget distribution shifts to higher values to enable the adversary to use its full nominal strength against the now- trained protagonist agent. The idea of gradually raising the strength of the adversary is also implemented in Cur- riculum Adversarial Training (CAT) (Sheng et al., 2022), which, however, uses linear curriculum as opposed our automatic curriculum generation scheme. Figure 6 compares the training progression QARL to that of Force-Curriculu in the "Hopper Hop" environment. Figure 6a shows the progression the adversary temperature in QARL overlaid with the perfor- mance of the protagonist agent against the adversary. Similarly, Figure 6b shows the progression of the adversary force budget in the Force-Curricllum algorithm overlaid with the performance of the protagonist agent against the adversary. Both plots indicate the performance threshold & above which the curriculum would progress to a more difficult adversary. We set = 5 for the "Hopper performance below this value indicates a catastrophically poor protagonist
(Algorithm 2). We follow simi-
the performance of the protagonist agent. The cur-
the force budget f the adversary Pw (f)-
1010
Modal Prob Pobbabii Assund
-003
a. as
Normalised
403
-203
8.4 b.
0.0 02
64
0.6
1.
Normalised Adversary Force Budget
Figure 7: Evolution of adversary force budget f' distribution.
Hop" environment, policy. It is observed in Figure 6a that the temperature of the adversary in QARL is decreased when the performance of the protagonist above &, and increased or maintained when the performance of the protagonist is below (note that the precise crossing points of the performance with the per- formance threshold cannot be matched variance in the plots). Likewise, it can be seen in Figure 6b that the adversary force budget f in the Force-Cure Curriculum algorithm increased when the performance of the protagonist is above and decreased or maintained when the performance of the protagonist is below &. We remark that "Hopper Hop" is chosen for this comparison as is prone to instability, being a perfect candidate evince the benefit of QARI..
with the curriculum temperature behavior due to some
13

---

Published as a conference paper at ICLR 2024
Algorithm 2 Force-based curriculum adversarial reinforcement learning (Force-Curriculum)
Input: Initial critic parameters v/ 1h2 o1 222 $ 2/2 1 222 initial actor parameters o' Ø²; D = 0. D2 0; initial force distribution parameters performance lower bound ç; #sampled forces N; #Moote-Carlo rollouts M; initial temperatures a; initial force budget fü #steps T.
2: for each iteration do
3:
$ + vi. 22g
2 o - vs.
>> Target networks update Sampling of force budgets of the adversary
4:
f1....N Pw (
5:
for = 1...., . do
,
6:
for 1 ,...,, T
Transitions collection for the adversary
7:
" ~ Mot (a (s St.
8:
0f ~ (a St, fi)
9:
a = CLIP -h
10:
1141 P(s a,
11:
D D2 ((,, (T,, St)]
12:
" 2: o² ADVERSARY.UPDATHR 12, a,, D2)
13:
end for
end for
15:
for i = 1, N do
for
1 1, ,..., do
Transitions collection for the protagonist
17:
0 ~ Mop (a. (s,
18:
0 (, s1, a;)
19:
4 = CLIP(a - fi
8 P(8 8t, a
21:
D DI U (S1, ai Tr, ft))
22:
V 2 .. PROTAGONISS.LEEDAEECCC wa ,,,
23:
end for
24: end for
25:
Update w optimizing (10), estimating (P) [ (w)] with M rollouts
26:
end for
27: Return: Trained actor parameters &2, .
B EXPERIMENTAL DETAILS
B. AGENTS
The majority of agents deployed in the experiments (except the adversary policy in CAT) are SAC agents and variations thereupon, allowing bounded rationality curriculum of QARL; SAC has been then chosen similarly for the other algo- rithms enable fair comparison. The hyperparameters of the adversarial agents across all environ- ments are shown in Table 2. Note that the temperature-related hyperparameters (initial temperature, temperature learning rate, target entropy) do not apply to the adversary used in QARL as they are dictated by the automatically generated curriculum. For the SAC MixedNE-LD agent, additional hyperparameters are chosen based on tuning and prior knowledge from Kamalaruban et al. (2020). The adversary used in CAT (Sheng al., 2022) has separate hyperparameters it does not use neural network implementation, but rather performs a simple gradient descent in the action space.
leverage the temperature parameter of SAC for the
B.2 ENVIRONMENTS
The MuJoCo environments (Todorov al., 2012) discussed in Section are standard implementa- tions from the DeepMind Control Suite (Tunyasuvunakool et al., 2020) modified for an adversarial setting. The standard environment states are used as inputs to both agents. In each environment, adversarial actions and force magnitudes are chosen to elicit robust agent behavior. The adversarial action spaces are specifically chosen to be different from those of the protagonist agent in order to exploit domain knowledge, maximum force is used environments that are highly sensitive to adversarial disturbances such as "Acrobot", higher forces cause total collapse of the protagonist policy. For each environment,
done in RARL (Pinto al., 2017). Generally, smaller adversary
14

---

Published as a conference paper at ICLR 2024
Table 2: Agent hyperparameters
Hyperparameter
Value
Shared (SAC and SAC MixedNE-LD)
# hidden layers (all networks)
3
# hidden units per layer (all networks) nonlinearity critic optimiser
256
ReLU
Adam
critic learning rate
3 10
actor learning rate
1 10
initial replay memory size
3 10
max replay memory size
1 10
warmup transitions batch size
5 103
256
target smoothing coefficient (r)
5013
target update interval
policy log std bounds
[-20, 2]
initial temperature
50103
temperature learning rate
3
target entropy
-dim(A)
SAC
actor optimiser
Adam
SAC MixedNE-LD
adversary influence 8 actor optimiser
0.1
SGLD
thermal noise (a,)
10-3
(1 - 5 x 50
RMSProp parameter
0.999
RMSProp parameter
10-8
CAT MAS Adversary
gradient descent learning rate
gradient descent step limit
25
gradient descent convergence threshold e
10-3
disturbance L"-norm
2
15

---

Published as a conference paper at ICLR 2024
Table
Environment--ppcciie parameters
Environment
Adversary max force
Adversary action space (size)
Acrobot
0.0005
2D forces on each arm (4)
Ball in Cup
0.1
10
2D force on ball (2)
Cartpole
0.005
10
2D force pole (2)
Cheetah
1.0
40
2D force on feet & torso (6)
Hopper
1.0
5
2D force on foot & torso (4)
Pendulum
0.005
10
2D force pole (2)
Quadruped Run
10
50
3D force torso (1)
Quadruped Maze
600
2000
1D force on all bodies (1)
Reacher
0.1
10
2D force on arm (2)
Walker
1.0
10 2D forces on feet (4)
the adversary forces are chosen carefully so as to be large enough to beget agent robustness and gen- eralisation while posing challenge to the protagonist. These environmeen--peccfic parameters shown in Table 3, while the discount factor and horizon are set respectively to 0.99 and 500 for all environments. Apart from this, the MuJoCo environments are left unmodified, with two exceptions: (a) the pole of the "Pendulum" environment is given a default mass of 0.1 instead of 0 in order to test the robustness of the protagonist to changing pole mass, and (b) the quadruped agent trained in maze environment where the wind force of the adversary is distributed evenly across all of its body when its torso enters the effective zone of the adversary (the yellow area in Figure 4). Note that & (performance threshold) values shown in Table 3 apply to both QARL and Force-Curriculug Note also that the adversary action spaces in Table 3 do not apply to CAT (Sheng et al., 2022) and MixedNE-LD (Kamalaruban al., 2020), as these algorithms protagonist and adversary.
shared action space for the
B.2.1 QUADRUPED
The quadruped maze environment discussed in Section is a modified version of the default quadruped provided in the DeepMind Control Suite (Tunyasuvunakool et al., 2020). The state s R contains the default quadruped observations (egocentric state, torso velocity, torso up- right state, IMU measurements, and body force torques) well as the 2D displacement vector in the X Z plane between the torso of the quadruped and the goal. The reward function for the quadruped is:
r(s. a) (Ztorso Zglobal) X 5 X 0.2l|xxgonn MMtllaoo
(13)
The term (Zeorso Zglobal) is the inner product of the quadruped's torso's z-axis and the global z- axis, acting as measure of how upright the quadruped is, which improved locomotion stability in practice. The exponential term ||xZgooi X is the Euclidian distance between the goal and the torso of the quadruped in the X Z plane, encouraging the quadruped to move towards the goal. Finally, the scale factors 5 and 0.2 are used to shape the exponential reward characteristic order to sufficiently encourage the quadruped to move forward from its starting position. If the quadruped flipped over during training (indicated by (Ztorso: Zylobol) < 0), the episode is terminated order to avoid collecting junk transitions as it is hard for the quadruped to recover from this state.
B.3 ALGORIHTMM
The details of the algorithms investigated are shown in Table 4. Note that the gamma distribution parameters shown apply to both QARL and Force-Curriculum. The algorithm parameters for Cur- riculum Adversarial Training (CAT) in Table 4 denote the iterations between which the gradient of the linear force budget curriculum would be constant and non-zero, i..., over the course of 200 total iterations, the force budget in CAT would simply be 0 before 40 iterations, full force after 160 iterations, and linearly interpolated for all iterations in between. These values are found to be heuristically effective and based in part on previous experiments using manual curricula.
16

---

Published as a conference paper at ICLR 2024
Table 4: Algorithm hyperparameters
Hyperparameter
Value
Shared
# iterations
200
# episodes per agent per iteration
5
# evaluation rollouts per iteration
QARI & Curr-Cuuri
initial gamma distribution concentration kinstiial target gamma distribution concentration ktarget fixed gamma distribution rate 0
1000
D constraint (e) # rollouts needed for update
0.5
CAT
curriculum start iteration
0.2 x iterations
curriculum end iteration
0.8 X # iterations
B.4 HARDWARE AND SOFTWARE
The experiments are carried out on a computational cluster with 64GB of RAM and an AMD Ryzen 9 16-Core processor. The algorithms investigated are implemented using the Mush- roomRL (D Eramo et al., 2021) library, which is also used for the implementation of all agents and adversarial environment wrappers.
C ADDITIONAL RESULTS
C.1 QARL AND BASELINES PERFORMANCE AND ROBUSTNESS
Bd Cup-
505
402
302
470
205
400
(S87)
(475)
MitechiC
(433)
Curicauu
(455)
GARIL
MALL
CAC
Reeche Nar
Reache Aat
455
18c
480
410
675
660
156
440
11c
S/C MAudi55DD
Fano Compouuhh
4AC.
SBC Masdit-D
Fanm Cartoouthh
(877)
(175)
(4.5)
(671)
(E77)
Figure 8: Performance of QARL. baselines on Ball-in-cup and Reacher (see the title of each boxplot), evaluated at the end of training against adversary trained against the frozen trained protagonist (left column) and without an adversary (right column). The number next to the name of each algorithm is the average performance across 10 seeds.
17

---

Published as a conference paper at ICLR 2024
Acrobot- Swingup
Acrobot
120
Adversary
100
100
80
Rctum
60
40
40
20
QARL
RARL
SAC MixadhE-L (78)
CAT Forte Cunculum (19)
SAC
GARL (39)
RAIL
SAC MixsdhE-L (40)
CAT
Furte Cuntrula (13)
SAC
(44)
(43)
(43)
Swingup
Swingup
350
Joverary
300
250
250
150
100
(216)
(162)
MixudNN (23M)
(255)
Contruium
(217)
(232)
(2
MinudNE 4D
(252)
Curtesium
(240)
(254)
60
Cartpole- - Balance Spane
Sparse
500
1
480
45
40
420
35
400
380
20
GARL
RARL (30)
MiNiNN-LLD
Car (sv)
Force Curticulum
SAC
Mivashe 4D
Curdicuium
(48)
(33)
(499)
(477)
(491)
(498)
Cartpole Balance
Cartpols Balance
500
4.55
Retum
485
350
475
250
OARL
RAILL
SAC MixodNE-LD
Porte Corteudum
QARL
RAIL (f3g)
SAC MiNddNE-LD (434)
CAT
Forte Contoutum (182)
SAC
(444)
(495)
(497)
(430)
(433)
Cal lpule- Swingup Sparss
Caripole Swingup Sparse
400
k5 Seversry
250
250
200
150
150
50
QARL (203)
RARL
SAC MisddNE-LD (2.1)
CAT
Ponce Curoculum
SAC
GARL (259)
RARL (49)
SAC MinndNE--LD
CAT
Ponce Cursicauu
sac
(186)
(206)
(67)
(2.3)
(1)
Figure 9: Performance QARL and baselines on balancing and swing-up control problems (see the title of each boxplot), evaluated at the end of training against adversary trained against the frozen trained protagonist (left column) and without an adversary (right column). The number next to the name of each algorithm is the average performance across 10 seeds.
18

---

Published as a conference paper at ICLR 2024
Cartpole Swingup
Cartpole
450
400
400
500
230
320
150
250
QARL
(307)
RARL
(64)
CAT (363)
SAC MixsiNN-LL
Rote Cunsrluu (67))
SAC
(365)
GARL
(365)
RARL (306)
CAT (305)
SAC MikeciN--LD
Rurte Cuntrolam (363)
SAC
(369)
(400)
(305)
400
Jolerarg
350
250
Retum
250
200
200
100
GARL
MixedNE LD
Contedium
MinudNE 4D
Corticiumm
(340)
016)
(377)
(307)
(3.3)
(349)
0121
(304)
(317)
Wo kar Stand
Waker Stand
680
480
Retum
430
420
450
QARL
RAIL
SAC MiredNE-LD
Porce Curtiudumm (440)
SAC
OARL
RARL (473)
SAC MACKRNNNEE (471)
CAT
Forte Coriicluum (465)
SAC
(475)
(477)
Walker Walk
Walker Nblk
475
450
625
600
Retuum
400
12
350
325
300
QARL (462)
RAPL (472)
SAC MiNidNE I.D
CAT (335)
Force Curtiadum (432)
SAC
(366)
GARL
(4761
NARL (477)
SAC Minuthe AD
CAT (336)
Farce Curticuuam (439))
SAC
(363)
(436)
(464)
375
350
325
300
300
275
275
225
200
225
GARL (336)
MondNEELD
Curtauium
S4C MINONN--D
Force cormadum
00g)
(209)
(282)
(297)
(325)
(3331
0121
(207)
(282)
(13)
(329)
Figure 10: Performance of QARL and baselines swing- up and locomotion problems, i.e. Cheetah and Walker (see the title of each boxplot), evaluated at the end of training against an adversary trained against the frozen trained protagonist (left column) and withour adversary (right column). The number next to the name each algorithm the average performance across seeds.
19

---

Published as a conference paper at ICLR 2024
Quadruped
Quadruped Run
500
Rctum
300
QARL
(359)
HARL
(33)
LAT (110)
SAC MixeiNG-LD (242)
Roree Cunsuluu (300)
SAC
(341)
GARL
(327)
RARL
(23)
CAT (138)
SAC MixanNG-LD
Purte Cuntrolam (302)
SAC
(37X)
(83)
Hopper- Stand
400
Retum
100
QARL (289)
RARL (188)
SAC MixxdNE-LD LD (182)
CAT (377)
Force Curtiuluum (778)
SAC (245)
GARL (16)
BARI
SAC MisudNE LD (9)
CAT
Force curticdum (6)
SAC
Hogper Hop
Hopper
120
100
80
Return
60
12
40
20
Mivadti 4D
CAT (20)
Porte Curticuuu
SAC
Mivastii A.D
CAT
Cursicuium
SAC.
(46)
(40)
(42)
(57)
(34)
(30)
(42)
Figure 11: Performance of QARL on locomotion problems, i.e., Quadruped and Hopper (see the title of each boxplot), evaluated at the end of training against an adversary trained against the frozen trained protagonist (left column) and without an adversary (right column). The number next to the name of each algorithm is the average performance across 10 seeds.
Ball Cup Catch
QARL (470)
RARL (442)
SAC MixedNE (450)
CAT (436)
Force Ourriculum (39)
SAC (442)
8.0
8.89
200
0.50 2,075
Mass
Mass
Mass Cup
Mass
Reacher Harr
OARL (477)
RARL (4721
SAC MixedNE-LD (475)
CAT (468)
Force Durriculum (473)
SAC 1.929)
6.02
8.99
0.64
0.03
0.24
0.03
0.23
5.04
0.04
Mass
Mass
Figure 12: Robustness analysis on Ball-in-cup and Reacher (see the title of each heatmap set). Each heatmap shows the performance obtained for varying properties of the environment, described in the I y axes. The number next to the name of each algorithm is the average performance across 10 seeds.
20

---

Published as a conference paper at ICLR 2024
Acrobot Swingup
QARL (58)
MARL (33)
SAC MixedNE-LL (28)
(39)
Force Curricllum (0)
SAC (27)
6.98
1.00
2.04
Mass 102
Mass
Mass Lower Arm
Mass Arm
Mass Lower
Arm
Mass Lower
Pendulum
QARL (257)
RARL (222)
SAC MixedNE-LD (234)
CAT (233)
Force Curriculum (213)
SAC (134)
8.00
300
0.30
834
223
0.75
0.55 128
Pendulum
Pandulum
Cartpole Balance
QARL (500)
PARL (426)
SAC RixeNNELLD (498)
CAT (452)
Foroe Curriculum (490)
SAC (398)
3.5 0.75 1.00 1.5 1.66.2
4.7
Mass 506.5
142
125
25
Life
1:25
1. 1.503e
6.
Liee
1:25
LS0050
6.25
1.00 1.25 Mass Cart
1,500
n.35
1:00
1.24
1.56
Cart
Mass
Cartpole- Balance
SAC MixeNNELLD
Curriallum
(308)
-300
-206
300
3,50
Mass of Cart
Mass Cart
Mass of Cart
Mass Cart
Mass of Cart
Mass Cart
Cartpole Sappole-Swinng Swingup Sparso
QARL (197)
RARL (40)
SAC MixedNE LD (12)
CAT
Force Curriculum (110)
SAC (323)
8.00
830
Mass of Cart
Mass Cart
Mass Cart
Mass of Cart
Mass Cart
Mass Cart
Cartpole Sortpole-Swingg Swingup
QARL (354)
RARL (352)
SAC MixedNE LD (356)
CAT (355)
Force Curriclumm (161)
SAC 1341)
8.00
830
200
833
150
L.0 2.23
0.75 103 123
Figure 13: Robustness analysis QARL and baselines on balancing and locomotion problems, i.e., Cartpole, Cheetah, Walker, and Hopper (see the title of each heatmap set). Each heatmap shows the performance obtained for varying properties of the environment, described in the æ - y axes. The number next to the name of each algorithm is the average performance across 10 seeds.
21

---

Published as a conference paper at ICLR 2024
Quadruped Run
QARL (687)
RARL (579)
CAT (305)
SAC MixedNE-LD (494)
Foroe Currioulum (604)
SAC (640)
700
62
-606
10t
1.0
1.00
3.30 Coefficient
QARL (276)
RARL (245)
SAC MixedNE-LD (220)
CAT (219)
Force Curficulum (264)
SAC (230)
-208
-10
1.5
1.5 Friction Coefficient
100
1,5 Friction Coefficent
1, 1.5
Friction Coefficient
Friction Coefiicintt
Friction Coefficient
Friction Coefficient
Walker Stand
(481)
SAC (469)
(470)
(463)
(471)
400
375
Friction Coefficient
Friction Coefficient
25 Friction Coefficient
Friction Coefficient
Friction Coefricient
63
Friction Coeflicient
QARL (465)
RARL (430)
SAC MixedNELLD (450)
CAT (419)
Foroe Curriculum (447)
SAC (426)
450
0.5
1.6
2.5
1.6 Confficient
35
1.0
65
14
Coefficient
Confficient
Confficient
Confficient
SAC MieeNE-LLD
Currcullum
SAC (235)
Friction Coofficient
Priction Co Coofficint
Friction Coefficient
Friction Coefficient
Priction Confricient
Friction Confricient
Hopper Stand
QARL (12)
RARL 17)
SAC MixedNE-LD (32)
CAT (5)
Fonce Curriculum (7)
SAC (4)
of 10500
0.5
3.5 Coefficient
3.0
0. 3-8 Friction Confficient
15
05
10
1.5 Coefficient
05
10
1.5
0.5 Coefficient
10
1.5
15 Friction Coefficient
85
10
Coefficient
Hopper
Hop
of
3,0
65
10 Friction Coefticient
10 Friction Coefticient
Friction Contricient
Confficient
Friction Confficient
Friction Coefticient
Figure 14: Robustness analysis QARL and baselines on locomotion problems, i.e., Quadruped, Cheetah, Walker, and Hopper (see the title of each heatmap set). Each heatmap shows the perfor- mance obtained for varying properties the environment, described the x axes. The number next to the name of each algorithm is the average performance across 10 seeds.
22

---

Published as a conference paper at ICLR 2024
Cartpole Swingup Sparse
(2.58)
0.5 (184)
(D)
(D)
Return
244
Mass of Cart
Mass of Cart
Mass of Cart
Mass of Cart
Mass of Carl
Walker Run
(221)
(284)
(228)
(220)
Return 300
Priction Confficier
Friction Comflicient
Friction Coeffiiint
Friction Confficient
Friction Coeflicient
Hopper Hop 0.5 (28)
0.1 (17)
(17)
0.7 (16)
1.0 (24)
Retur
Friction Coofficient
Friction Confficiont
Priction Coofficient
Friction Coettident
Priction Coofricient
Figure 15: Robustness analysis of QARL trained with several values of DÆz constraint (c) three selected MuJoCo control problems. Each heatmap shows the performance obtained for varying properties of the environment, described in the y axes. Each heatmap is titled with the value of used during training followed by the average performance across 10 seeds in brackets.
Cartpole Swingup Sparse
0.06
Modal Probability DanlyyDDDaae Density
1.0
Normalised PDF
800
Temperature
0.05
600
constraint
0.8
0.3
400
0.04
0.2
0.6
0.5
0.7
200
0.03
0.4
0.02
0.2
0.01
0.0
200
0.00
0.01
0.02
0.03 0.04 0.05 0.06 0.07
Iteration
Temperature
(a) Mean temperature across different Dxt. constraint.
(b) Evolution of temperature distribution.
Figure 16: (a) Mean temperature of QARL adversary during training in the 'Cartpole Swingup' environment across several values of DK constraint (e). (b) Evolution of adversary temperature distribution from high (light) to low (dark) as the curriculum progresses.
C.2 QARL PROGRESSION AND ABLATION
Figure compares the robustness of agents trained with QARL across several values the D constraint (c) in Equation (10). It is seen that too-high and too-low values of result in suboptimal robustness, indicating that an intermediate value generally preferable. Figure 16a also suggests that intermediate value for allows the temperature curriculum to progress at appropriate pace. Lower values of & cause the automatic curriculum to progress too slowly to obtain a rational adver- sary at the end of training. Meanwhile, higher values of result in large jumps of the temperature distribution between iterations, causing changes adversary behavior that are too drastic for the protagonist to adapt curtailing agent performance, and preventing the curriculum from progress-
23

---

Published as a conference paper at ICLR 2024
Cheetah Run
(234)
20 (265)
(267)
(246)
100 (233)
Mass of Torso
1
Return
200
1.5 Friction Coefficient
1.0 1.5 Friction Caefficient
1.0 Friction Coefficient
1.0 Friction Coefficient
1.0 Friction Coefficient
Figure 17: Robustness analysis of QARL. trained with several values of performance threshold & in the Cheetah MuJoCo environment. Each heatmap shows the performance obtained after training for varying properties of the environment, described in the y axes. Each heatmap is titled with the value used during training followed by the average performance across 10 seeds in brackets.
ing. In practice, some initial tuning of the DK constraint hyperparameter is sufficient to obtain single value for e that enables robust behavior across all environments.
Figure 16b shows the evolution of the gamma distribution of the temperature a of the adversary (a).. We start from gamma distribution with a high mean to sample high temperatures. The ini- tial low rationality of the adversary enables high protagonist agent performance due to the relative simplicity of the maximum entropy minimax optimisation. Progressively, the mean of the distribu- tion approaches and the variance decreases. The resulting protagonist agent is endowed with high robustness and performance despite the relative complexity of the saddle point Nash equilibrium optimisation.
Figure 17 ablates the value of the performance threshold & in the MuJoCo Cheetah environment. It generally found that low (but not extremely low) values of & result in stable convergence of the curriculum, e.g. 20 and 30 in Figure 17, these values allow the curriculum to progress long as the protagonist is not catastrophically failing at the task. Too low values (e.g. 5 5) result in the curriculum dropping fast, whereas high values (e.g. curriculum failing to progress since the agent required to reach an unreasonable threshold.
50, & 100) result in the
Figure 18 tests the importance of distinct temperatures for the protagonist and adversary agents. As the aim of QARL is to robustify the protagonist agent, attempts to solve heterogeneous QRE as discussed in Chapter 4 of Goeree al. (2016). We choose of the protagonist so that it can reasonably solve the task at each iteration and efficiently progress to the end of the curriculum, where the robustness benefits are maximised.
excessively bound the rationality
24

---

Published as a conference paper at ICLR 2024
Cheetah
Run
Hopper
Hop Homageneous QARI (28)
Heterogeneous QARL (276)
Homogeneous QARL (213)
Heterogenoous QARL (276)
Mass of Torso
Mass of Torso
ae
Return
200
0
0.5
1.0
1.5 Coefficient
1.5 Friction Coefficient
0.5
1.0
0.5
1.0
1.5 Coefficient
0.5
1.0
1.5
Friction
Friction
Friction Coefficient
(a) Cheetah Robustness analysis
(b) Hopper Robustness environment properties
Cheetah Run
Hopper Hop
Heeterogencous QARL
100
Return vs Adversary
Return vs Adversary
Homogeneous QARL
300
80
200
60
Heterogeneous QARL
40
100
Homogeneous QARI 40
20
50
100
150
200
50
100
150
200
Iteration
Iteration
(c) Cheetah Performance during training
(d) Hopper- Performance during training
Cheetah Run
Hopper Hop
0.06
Heterogeneous QARL Homageneous QARL
0.06
Heterogeneous QARL
Hamogencous
Temperature
0.04
Temperature
0.04
0.02
Korann www.
0.00
0.00
50
100
150
200
50
100
150
200
Iteration
Iteration
(e) Cheetah Temperature during training
(f) Hopper Temperature during training
Figure 18: Ablation comparing heterogeneous QARL (ours) to homogeneous QARL over the Chee- tah and Hopper environments across 10 seeds. (a) and (b) show heatmaps of performance for varying properties of the environment described in the and the average performance in brackets. (c) and (d) show the performance during training against an engroiy-regularised adversary. The dashed lines indicate the performance threshold & for each environment. (c) and (f) show the curriculum temperature during training. Note that the ho- mogencous case, both agents follow the temperature as dictated by the curriculum, whereas in the heterogeneous case, the protagonist agent follows the default temperature schedule of SAC (not shown here), like that shown in Figure 6a.
axes. Each heatmap titled with the algorithm
25


Reviews:
[{"review_id": "YJOkB10lOt", "reviewer": "Reviewer_puRh", "summary": "Robust Adversarial Reinforcement Learning (RARL) trains a protagonist against an adversary in a competitive zero-sum Markov game. However, in a high-dimensional control setting, finding the Nash equilibria faces complex saddle points. This method eases the complexity of the saddle point in optimization problems based on entropy regularization. They show the solution of an entropy-regularized problem corresponds to a Quantal Response Equilibrium, in which agents may be irrational with a certain probability. Based on this fact, this paper proposes an algorithm named Quantal Adversarial Reinforcement Learning (QARL). This algorithm first trains the protagonist against an irrational adversary and gradually increases the irrationality of the adversary until it is fully rational. This paper shows that QARL achieves stronger performance and robustness compared with other RARL algorithms in a wide range of reinforcement learning settings.", "questions": "Have you tried any other heuristics for tuning $\\alpha$? If so, could you briefly discuss them?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. This paper is novel in the sense that it proposes a new method achieving stronger performance and robustness in adversarial reinforcement learning by gradually increasing the rationality of the adversary during training.\n\n2. This paper draws an interesting connection between Quantal Response Equilibrium and an entropy-regularized problem and proposes a hyperparameter to smoothly control the rationality of the agent.\n\n3. The experimental results are detailed and convincing. QARL outperforms other RARL algorithms in both performance and robustness. The experiments are conducted in a wide range of reinforcement learning settings including MuJoCo locomotion and navigation problems.", "weaknesses": "1. To train the protagonist by QARL, this paper requires that the rationality of the adversary can be controlled. This is a strong assumption, and its motivation is not well-justified. It would be better if this paper could show the performance of QARL even if the rationality of the adversary is not tuned but just increasing. Most importantly, this weakness limits the possible application of this algorithm to scenarios where a reliable simulator is available because the control over the rationality of the adversary may only be possible in a simulator.\n\n2. Estimation of (12) consumes non-negligible online trajectories. The way to tune the rationality hyperparameter $\\alpha$ is expensive.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xiYktdrGjp", "reviewer": "Reviewer_DPhM", "summary": "In this paper, the authors propose a novel entropy regularization algorithm called Quantal Adversarial RL for adversarial reinforcement learning  which  modulates adversarial rationality to ease the complexity of solving saddle point optimization problem in robust adversarial RL.They connect entropy regularization in RL to bounded rationality and Quantal Response Equilibrium in game theory and show how temperature parameter in entropy regularization can control rationality and helps to train against a rational adversary. They provide extensive experiments showing QARL outperforms RARL and other baselines in several MuJoCo problems.", "questions": "1. Where is policy $\\pi$ used in the equation 6,7,8? What is variable $\\mathcal P$ as the conditioning parameter?\n2. Definition of $Q^\\*$ and $\\pi^\\*$ is not very clear. Both $Q^\\*$ and $\\pi^\\*$ depend on each other in eqn 6 and 8?\n3. Is there any insight into why the curriculum approach helps optimization compared to direct adversarial training?\n4. How well does the computation overhead of sampling different rationality level scale?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. Connection between entropy regularization and bounded rationality is novel.\n2. Proposes a novel constraint optimization problem to design curriculum for updating temperature coefficient that slowly changes an irrational adversary to a fully rational one.\n3. Extensive empirical experiments to demonstrate the effective of their algorithm.", "weaknesses": "1. Relies on heuristic approach to design curriculum schedule with no theoretical guarantee on convergence behavior.\n2. Certain parts of the paper and notations can be improved, see questions section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WKXtuBdpLk", "reviewer": "Reviewer_S4Ho", "summary": "This paper builds on robust adversarial reinforcement learning (RARL) and add entropy regularization into the players' objectives. By adding regularization, the authors bound the rationality of the adversary (and protagonist) making the problem slightly easier to solve. Over training, the regularization is annealed, creating a curriculum, such that the ultimately trained protagonist is robust against a strong adversary. The authors show this approach outperforms several baselines empirically across a variety of tasks.", "questions": "**I have upgraded my score after review of the authors' feedback**\n\n- Note McKelvey and Palfrey defined the QRE along with a more specific QRE, called the limiting logit equilibrium, that is obtained by annealing the temperature from infinity to zero (homotopy approach).\n- Equation 4: You describe this as a maximum entropy formulation, but this looks like an entropy regularized approach rather than selecting the equilibrium with maximum entropy (which is different).\n- Irrational: I understand why you've chosen to pair \"irrational\" against \"rational\", but I think it's inaccurate. I think you mean \"random\". Note that a random policy is not necessarily irrational (e.g., random is Nash in rock-paper-scissors).\n- Section 4.1: You say \"In Markov games... QRE can be computed in closed form...\". This is not true. If we could compute QREs in closed-form (at any temperature), we could compute nearly exact Nash equilibria in closed-form. I think you just mean that computing the denominator of equation 5 is difficult due to the integral. The integral becomes a sum with finite actions, but you still have to solve a fixed-point problem to compute a QRE.\n- Why do you define a probability distribution over $\\alpha$ instead of just controlling $\\alpha$ point-wise?\n- Equation 11: Do you minimize this for a fixed $\\lambda$ and $\\eta$?\n- Do you have an easily-accesible reference for QREs with different temperature values per player (i.e., heterogeneous QREs)? I'm familiar with QREs, but never seen this and I couldn't track down the precise Goeree reference you cite. To my knowledge, this is a deviation from the QRE concept, but still makes for an interesting story as inspiration for your approach.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "I found this paper generally easy to follow and well written. It proposes a simple modification to RARL that appears to greatly improve the empirical performance and make sense from a theoretical perspective as well.", "weaknesses": "Once the temperature had been split between the two players ($\\alpha$ and $\\beta$), I found the connection to QRE a bit tenuous. This is okay, but I think it would be better to present the QRE with one temperature and then state that you find better performance by using two.\n\nI would have liked to see results over the course of training. Do you see monotonic improvement? How challenging is the saddle point problem (adversary vs protagonist) experimentally? Can you plot $(\\alpha, \\beta)$ over training?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rujgMo8exR", "reviewer": "Reviewer_cFcL", "summary": "This paper proposes a robust RL framework on top of RARL, considering the entropy-regularized problem corresponds to a Quantal Response Equilibrium (QRE). With this extension, solving zero-sum games will not always face complex saddle point optimization problems. This paper is with theoretical support and numerous impressive experiments.", "questions": "1. Your framework is built on the top of RARL, which means that the adversarial action spaces are specifically chosen to be different from those of the protagonist agent in order to exploit domain knowledge. Could you explain more about how you set up the adversarial action spaces with the additional environments/tasks compared with the ones (e.g., halfcheetah, swimmer,  hopper ) RARL provides? \n\n2. Is your proposed method also compatible with Noisy Robust Markov Decision Process (NR-MDP), which MixedNE-LD builds on top of? If so, what the role of $\\alpha$ will be in NR-MDP? Is it just similar to the concept of $\\delta$ mentioned in MixedNE-LD paper for defining the limit of the adversary?\n\n3. How can we elaborate the concept of rationality in the experiment? Does the most rational adversary mean the strongest adversary (severity strength)? In that case, I am not sure if irrationality at the beginning represents the less strength of attack. \n\n4. On page 2, a statement: \"Conversely, in a setting where the protagonist is completely rational and the adversary is completely irrational, i.e.,\nit plays only random actions...\". Do you mean that the protagonist is playing only random actions? Then what action of the adversary take?\n\n5. On page 5, a statement: \"We propose to initially solve an adversarial problem with a completely irrational adversary, i.e., $\\alpha \\rightarrow \\infty$, which results in a simpler plain maximization of the performance of the protagonist, neglecting robustness\". Does it mean that we do not have any attack now?\n\n6. Could you provide the hyperparameter tuning or how you select $\\xi$ and $\\epsilon$? I think they will also influence the adversarial learning process.\n\n7. Could you please point out which script.py under mujoco_experiments is your proposed method?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The algorithm is novel in the robust learning community.\n2. The empirical results demonstrate that the proposed method outperforms the existing baselines.\n3. Proposed methods are well motivated with good explainability.", "weaknesses": "1. Overall, how the proposed method is adapted to the real world with different action spaces between protagonist and adversary is still unclear. The detailed questions and concerns please refer to the questions section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)