Feasible Adversarial Robust Reinforcement Learning for Underspecified Environments
Authors: John Banister Lanier
Stephen Marcus McAleer
Pierre Baldi
Roy Fox
Venue: Submitted to ICLR 2023 (2023)
Keywords: reinforcement learning
robust rl
sim-to-real
game-theory
psro

Abstract: Robust reinforcement learning (RL) considers the problem of learning policies that perform well in the worst case among a set of possible environment parameter values. In real-world environments, choosing the set of possible values for robust RL can be a difficult task. When that set is specified too narrowly, the agent will be left vulnerable to reasonable parameter values unaccounted for. When specified too broadly, the agent will be too cautious. In this paper, we propose Feasible Adversarial Robust RL (FARR), a novel problem formulation and objective for automatically determining the set of environment parameter values over which to be robust. FARR implicitly defines the set of feasible parameter values as those on which an agent could achieve a benchmark reward given enough training resources. By formulating this problem as a two-player zero-sum game, optimizing the FARR objective jointly produces an adversarial distribution over parameter values with feasible support and a policy robust over this feasible parameter set. We demonstrate that approximate Nash equilibria for this objective can be found using a variation of the PSRO algorithm. Furthermore, we show that an optimal agent trained with FARR is more robust to feasible adversarial parameter selection than with existing minimax, domain-randomization, and regret objectives in a parameterized gridworld and three MuJoCo control environments.

Under review as a conference paper at ICLR 2023
FEASIBLE ADVERSARIAL ROBUST REINFORCEMENT LEARNING FOR UNDERSPECIFIED ENVIRONMENTS
Anonymous authors Paper under double-blind review
ABSTRACT
Robust reinforcement learning (RL) considers the problem of learning policies that perform well in the worst case among a set of possible environment parameter settings. In real-world environments, choosing the set of allowed parameter set- tings for robust RL can be a difficult task. When that set is specified too narrowly. the agent will be left vulnerable reasonable parameter values unaccounted for. When specified too broadly. the agent will be too cautious. In this paper, we pro- pose Feasible Adversarial Robust RI (FARR), novel problem formulation and objective for automatically determining the set of environment parameter values over which to be robust. FARR implicitly defines the set of feasible parameter val- ues those on which an agent could achieve benchmark reward given enough training resources. By formulating this problem as a two-player zero-sum game, optimizing the FARR objective jointly produces adversarial distribution over parameter values with feasible support and a policy robust over this feasible pa- rameter set. We demonstrate that approximate Nash equilibria for this objective can be found using a variation of the PSRO algorithm. Furthermore, we show that an optimal agent trained with FARR is more robust to feasible adversarial pa- and regret
rameter selection than with existing minimax, objectives in parameterized gridworld and three MuloCo control environments.
INTRODUCTION
Recent advancements in deep reinforcement learning (RL) show promise for the field's applicability to control in real-world environments by training simulation (OpenAl et al., 2018 Hu et al, 2021 Li et al., 202I). In such deployment scenarios, details of the test-time environment layout and dynamics can differ from what may be experienced at training time. It is important to account for these potential variations achieve sufficient generalization and test-time performance.
Robust RL methods train on an adversarial distribution of difficult environment variations to attempt to maximize worst-case performance at test-time. This process can be formulated as two-player zero-sum game between the primary learning agent, the protagonist, which is a maximizer of its environment reward, and second agent, the adversary, which alters and affects the environment to minimize the protagonist's reward (Pinto et al. 2017). By finding a Nash equilibrium in this game, the protagonist maximizes its worst-case performance over the set of realizable environment variations.
This work aims to address the growing challenge of specifying the robust adversarial RL formulation in complex domains. On the one hand, the protagonist's worst-case performance guarantee only applies to the set of environment variations realizable by the adversary. It is therefore desirable to allow an adversary to represent a large uncertainty set of environment variations. On the other hand, care has to be taken to prevent the adversary from providing unrealistically difficult conditions. an adversary can pose insurmountable problem in the protagonist's 's training curriculum, under standard robust RL objective, it will learn to do so, and the protagonist will exhibit overly cautious behavior test-time (Ma al. 2018). As the complexity of environment variations representable in simulation increases, the logistic difficulty of well-specifying the limits the adversary's abilities is exacerbated.
For example, consider an RL agent for a warehouse robot trained to accomplish object manipula- tion and retrieval tasks in simulation. It is conceivable that the developer cannot accurately specify

---

Under review as a conference paper at ICLR 2023
the distribution of environment layouts, events, and object properties that the robot can expect to encounter after deployment. A robust RL approach may be well suited for this scenario to, dur- ing training, highlight difficult and edge-case variations. However, given the large and complex uncertainty set of environment variations, it is likely that an advensary would be able to find and over-represent unrealistically difficult conditions such as unreachable item locations hallway perpetually blocked by traffic. We likely have no need to maximize our performance lower bound on tasks harder than those we believe will be seen deployment. However, with an increas- ingly complex simulation, it may become impractical to hand-design rules to define which variation and is not too difficult.
critical
To avoid the challenge of precisely tuning the uncertainty set of variations that the adversary can and cannot specify, we consider new modified robust RI. problem setting. In this setting, we define an environment variation provided by an adversary feasible if there exists any policy that can achieve at least À return (i.e. policy success score) under it and infeasible otherwise. The parameter A describes the level of worse-case test-time return for which we wish our agent to train. Given an underspecified environment, i.e. an environment parameterized by an uncertainty set which can include unrealistically difficult, infeasible conditions, our goal is to find protagonist robust only to the space feasible environment variations.
With this new problem setting, we propose Feasible Adversarial Robust Reinforcement learning (FARR), in which protagonist is trained to be robust to the space of all feasible environment variations by applying a reward penalty to the adversary when it provides infeasible conditions. This penalty is incorporated into new objective, which we formulate as two-player zero-sum game. Notably, this formulation does not require priori knowledge of which variations are feasible or infeasible.
We compare a near-optimal solution for FARR against that of a standard robust RL minimax game formulation, domain randomization. and an adversarial regret objective similarly designed to avoid unsolvable tasks (Dennis et al. Parkerrrrkr et al. 2022). For the two-player zero-sum game objectives of FARR, minimax, and regret, we approximate Nash equilibria using a variation of the Policy Space Response Oracles (PSRO) algorithm (Lanctot et all]]201)) We evalaute gridworld and three MuJoCo environments. Given underspecified environments where infeasible variations can selected by the adversary, we demonstrate that FARR produces a protagonist policy more robust to the set of feasible task variations than existing robust RL minimax, domain domain-randomize randomization, and regret- based objectives. To summarize, the primary contributions of this work are:
We introduce FARR, a novel objective designed to produce an agent that is robust to the feasible set of tasks implicitly defined by threshold on achievable reward.
We show that this FARR objective can be effectively optimized using variation of the PSRO algorithm.
We empirically validate that a near-optimal solution to the FARR objective results in higher worst-case reward among feasible tasks than solutions for other objectives designed for similar purposes: standard robust RL minimax, domain randomization, and regret, a parameterized gridworld and three MuJoCo environments.
RELATED WORK
2.1 DOMAIN RANDOMIZATION
Domain randomization methods train in simulation on a distribution of environment variations that believed to generalize to the real environment. The choice of distribution for training-time simulation parameters plays major role in the final test performance of deployed agents (Vuong) ct al. 2019). While domain randomization has been used with much success sim-to-real settings OpenAA et al. 2018 Tobin et al.) 2017; Hu et al. 2021 Li et al. 2021). its objective is typically the average-case case return over the simulation uncertainty set rather than the worst-case. This can be desirable in applications where average training-time performance is known to generalize to the test environment or where this can be validated. However, average-case optimization can be unacceptable in many real-world scenarios where safety, mission-criticaliyy, regulation require
2

---

Under review as a conference paper at ICLR 2023
the agent to perform well in test environments whose weight in the average would be small and where sufficient validation is unavailable.
2.2 ROBUST REINFORCEMENT LEARNING
Robust reinforcement learning methods optimize reward the worst-case and provide a promising approach for sim-to-real. Robust Adversarial Reinforcement Learning (RARL) (Pinto et al. 2017) which this work builds upon, optimizes return under the worst-case environment variations by op- timizing two-player zero-sum game between task-performing protagonist and environment- controlling adversary. Numerous mechanisms by which the adversary affects the environment have been explored including perturbing forces and varying environment dynamies (Pinto et al. 2017 Mandlekar et al.] 2017: Nakao et al.] (202T). disturbances to actions (Tessler et al.] 2019
Vinitsky
al.,
2020
Tan et al., 2020), and attacks
agent observations (Pattanaik et al., 2017 Gleave
et al.
2019 Zhang et al. 20000] 2021 Kumar et al. (002). However, each of these works makes
the assumption that the uncertanty set from which perturbations and variations may be sampled is well-specified and tuned such that an agent robust to a minimax selection over the entire set will perform optimally in deployment. FARR relaxes this assumption and intends to produce a robust agent to the real environment, even when the adversary can select unrealistically hard variations.
2.3 AUTOMATIC CURRICULUM DESIGN
Similar to finding the worst-case distribution of environment variations is finding the best-case dis- tribution for improving an existing agent. Automatic curricula seek find environment parameters that are challenging but not too difficult for a training agent. Racaniere et al. (2019), Campero jet al. (2020), and Florensa et al. (2018) generate useful curricula for single learning agent solve difficult tasks by proposing appropriately challenging goals for the agent's is current abilities. POET [Wang et al. 2019 2020) co-evolves a population of tasks and associated agents to produce an in- creasingly complex set of tasks with coupled agents capable of solving their associated task. While each of these methods can create increasingly capable agents, they offer no guarantees for final agent robustness.
2018 OpenAl et al., 2021) facilitates two agents with sim- ilar capabilities compete in a two- wwo-player game where one attempts to reach goals that are achiev- able but difficult to the other agent. PAIRED (Dennis et al.) 2020) and subsequent improvements to optimization (Parker-Holder al. 2022 Du et 2022) extend this concept beyond goals to the generation of parameterized environments by introducing a two-player zero-sum regret objective between an adversary and protagonist. The adversary learns to specify environment conditions in which regret is highest such that the protagonist's 's performance most differs from estimated optimal performance. Regret selects tasks where changes to protagonist behavior could most affect incurred reward and is effective for training broadly capable agent given limited preference over tasks. Re- gret can also ensure successful performance where possible in certain classes of environments with well-behaved reward functions. However, because the protagonist can have imperfect information on the current task and because a Nash equilibrium regret distribution does not minimize protago- nist reward over target set of tasks, regret protagonist does not maximize worst-case performance over the feasible set of tasks like we are interested in with FARR.
Asymmetric self-play (Sukhbaatar et al.
3 BACKGROUND
3.1 UNDERSPECIFEED PARTIALLY-OBSERVABLL MARKOV DECISION PROCESSES
We adapt Underspecified Partially- Observable Markov Decision Processes (UPOMDPs) from Den- nis et al. (2020) where there exists a parameter of variation 0 θ that is hidden from the agent. This parameter& 0 controls some aspect of the environment that can potentially be discovered through interaction. For example, 0 could control the friction of robot arm, the mass of a cartpole, the layout of a room.
We model UPOMDP as tuple M
R, where A is the set of actions, is the set of states, O is the set of observations, and y is the discount factor. The choice of parameter 0 E ! controls the conditions in this environment, namely the transition distribution
3

---

Under review as a conference paper at ICLR 2023
function T x A(), the initial state distribution p Q A(S), the observation function I x O, and the reward function R 0 R.
Given observable history h H = (O x A)" x O, a protagonist policy Ty + (AA) decides at each time step t on the distribution of the action aj after seeing h 0o. Gg. ag......0 Ot. Jointl with UPOMDP M and specific environment parameter the policy induces a distribution p", over the states, actions, observations, and rewards in interaction episode. We define the protagonist's utility (Tp 8) = pe, 1>, r're] as the expected episode discounted return for a protagonist policy Tp and choice of 8.
3.2 POLICY SPACE RESPONSE ORACLES
Policy Space Response Oracles (PSRO) Lanctot al. 2017) deep RL method for calculat- ing approximate Nash equilibria (NE) in zero-sum two-player games. It extends the normal-form Double-Oracle algorithm (McMahan et al., 2003) to games with sequential interaction. At a high- level, PSRO iteratively adds new policies for each player to population until a normal-form mixed- strategy solution to the restricted game induced by selecting population policies closely approxi- mates NE in the full game.
PSRO operates by maintaining a population of policies II, for each player i and a normal-form mixed strategy o; E AlI.). This mixed strategy σ; represents distribution over policies T. E II; to sample from at the beginning is the final output of PSRO. The utility of player i of playing mixed strategy o; against op- ponent's policy πT is therefore Ui(o,, T-1) = E En, [U T-]],, and likewise, (,(0,,,_.)
each episode, and upon algorithm termination, a = (a), 02)
Br [U, T-:).
In each iteration of PSRO, new policies for each player i are added to its population 14, In the case of sequential interaction, this is typically an RL best-response (o:; arg max,, U2(m, o-;) that maximally exploits the opponent mixed-strategy. However, this choice of new policy is not a
requirement, and PSRO maintains NE convergence guarantees so long novel policies are contin- uously added for each player.
After adding new policies, utilities (T1, л2) between each pairing of player policies m E I and T2 E II, are empirically estimated using rollouts to create normal-form restricted game in which population policies are the strategies. A new NE mixed-strategy that solves the restricted game, restricted NE a = (2,,0).. is then cheaply calculated for each player. This process is repeated until no new policies are added to either player's population or the process externally stopped. As the number of strategies in each player's 's population grows, NE solution to the restricted game asymptotically converges to a NE solution to the full game.
While other potentially suitable methods for solving extensive-fore games exist, for example NFSP (Heinrich & Silver 2016) and Deep-CFR (Brown et al. 2 PSRO was chosen out of practicality because it can be implemented as additional logic layer top of cxisting reinforcement learning software stacks. Although PSRO-based methods are competitive in sample-efficiently solving two- player zero-sum games (Lanctot et al. 2017, Vinyall al. 2019 McAleer al.) 2021 2022b a Liu et al. (022). the purpose of experiments in this work is not focused on improving the speed which we might reach optima. Rather, we interested in ensuring that we can reliably reach approximate NE for each objective we test by using PSRO in order to compare their near-optimal solutions on even ground.
4 FEASIBLE ADVERSARIAL ROBUST REINFORCEMENT LEARNING
4.1 MOTIVATION
We begin our discussion of the FARR method with a motivating example. Consider a cartpole envi- ronment where the pole is subject to perturbing forces of an unknown magnitude. To maximize our worse-case performance, we could formulate a robust RL game in which the adversary provides the most difficult possible mixed strategy of force magnitudes. By learning a best response strategy to this adversary, the protagonist will maximize its worst-case performance at test-time when presented with unknown distribution of forces.
4

---

Under review as a conference paper at ICLR 2023
Importantly, to achieve this process, we would need to allow the adversary to specify sufficiently wide range of force magnitudes such that the real environment is believed to be in this range. How- ever, would not want to allow the adversary to specify force magnitudes becomes impossible, because the adversary would then always select overly difficult parameters and the protagonist would only train on impossible task variations, failing at test time.
large that the task
In this cartpole example, it is possible to manually adjust the range of allowed force values until the widest possible uncertainty set is found that still avoids the learning of degenerate protagonist strategy. However, if this setting were scaled up to specifying the allowed values of many coeffi- cients, level layouts, or environmental events where the adversary has complex. high-dimensional interactions with the protagonist, hand tuning these limits may no longer be viable.
To remove the need for human expert tuning of the adversary limits, we instead create game where the adversary is allowed to specify impossible environment variations but is heavily penalized for doing so, using the method we describe below.
4.2 FEASIBILITY
We define environment parameter 0 Q as X-feasible if best-response to & can achieve expected return of at least À E R. Heuristically, the value for À could be set as the lowest aver- age return that we would expect an agent receive across variations in deployment if it could act optimally with respect to each variation. We define the feasible set F of environment parameters as:
J = (0 E 0) A}.
F matches our motivation for considering feasible parameters when either of two conditions is satisfied. First, we may have lower bound on the achievable performance in the real environment, and we can set À at or below that bound. If the real environment is feasible, 0* E FA then we justified in avoiding training the protagonist on infeasible environments. Second, there may be performance threshold such that only above it we have preference over agent behaviors. For example, we may only care where a warehouse robot navigates has valid path its target shelf.
The set of feasible variations F is generally unknown. As part of optimizing the FARR objective, the adversary will learn an approximation of FA and use it to guide the selection of a mixed strategy over discovered feasible environment variations. We note that, order to measure robustness to F in this paper, we focus environments where we can, in fact, calculate F for the purpose of test-time evaluation. is reasonable to expect our findings to carry over to some domains where J=A truly unknown and where the method cannot be directly evaluated.
4.3 FARR OBJECTIVE
We wish to optimize the standard zero-sum robust adversarial game through PSRO with the addi- tional constraint that the support of the adversary mixed strategy og contains only strategies in the feasible set FA Define supp(ao) = (0 E Ojoo > 0}. Our intended FARR objective is to solve the game:
min max Up op) subject to supp(oo) S F ag
d2
An adversary mixed strategy dg with support that is a subset of F will only provide feasible envi- ronment variations the protagonist.
In order to provide flexibility in how novel adversary strategies that satisfy this constraint could be optimized, we can replace the hard constraint with sufficiently large penalty C the adversary when it violates the constraint. An equivalent zero-sum FARR objective would then be to optimize:
min max U (ap. oo);
Op
5

---

Under review as a conference paper at ICLR 2023
Pure Strategy NE
Penslized Adversary Str strey
Left (Easy)
Middle (Diffcult)
Right (Infeazibbe)
Right
Grae
20
-10
Grab
20
10
500
Dont
Dun
Original Game
(A=
590)
Protagonist
Adversary (Choose Item Location)
(Attempt Grab Not)
(nteasilble Adversary NE)
(Feasible Advensary NE)
(a)
(b)
Figure 1: (a) An adversarial item retrieval game. The adversary can hide bowl in a left, middle, or locked right cabinet. The protagonist chooses whether or not to attempt to find and grab the bowl. The protagonist receives a penalty for attempting to grab bowl placed in the locked right cabinet and failing. (b) Matrix game representation of the original game and FARR transformed game. Protagonist utilities shown. NE for the original game place weight infeasible tasks with suboptimal behavior from the protagonist. NE for the FARR game provide feasible but difficult tasks and optimal protagonist behavior supposing feasible tasks are expected in deployment.
where the FARR utility function U² (Mp. 0) unchanged from the original game if the adversary's provided environment parameter is feasible, and where otherwise large constant adversary penalty C' is applied:
Sc (Tp: 0) (pp(5 otherwise.
(BR
(4)
4.4 MATRIX GAME EXAMPLE
We demonstrate the effect of the FARR utility function (equation through a matrix game example shown in Figure [ The adversary specifies the location of a bowl among three cabinets, where the middle cabinet is more difficult for the protagonist to access than the left, and the right cabinet locked and inaccessible. Without observing the bowl's location, the protagonist must choose whether or not to attempt to find and grab the bowl, receiving penalty for a failed retrieval attempt. In deployment, we expect that the item will always be placed a feasible, accessible cabinet, either the left or the middle, so ideal behavior for the protagonist would be to always attempt to grab the bowl. For this game, we define the feasibility threshold over task variations as 1.
The original game, shown in marrix-form in Figure contains an infeasible adversary pure strat- egy (bowl in locked right cabinet). Because of the presence of this infeasible adversary strategy, the protagonist will never attempt grab the bowl in any of the game's Nash equilibria (one pure strategy NE is displayed in green). we now replace the original game's 's utility function Up with the FARR utility function U using = 500, the adversary receives a penalty of -500 for placing the bowl in the infeasible locked right cabinet because there exists no protagonist strategy that can achieve a utility of least À under that environment variation. Instead, in the FARR transformed game, NE adversary places the bowl in the feasible but difficult middle cabinet, and the protagonist learns, facing selection of feasible tasks, to always attempt to retrieve the bowl. This new protag- onist NE strategy for the FARR game is optimal with respect anticipated feasible deployment conditions.
4.5 OPTIMIZING WITH PSRO
We use PSRO to solve for an approximate NE the FARR transformed game with the penalty-based objective defined in equation (3. To optimize FARR with PSRO, only the scalar values À and need to be provided, where À is the maximum difficulty expected among tasks in deployment and C' is an arbitrarily large positive value. Our optimization process is shown in algorithm We represent environment parameters 0 as strategies in the adversary population II, with an output mixed-strategy ag over Nl. Similarly, represent protagonist RI. agent policies as strategies up ΠI with an

---

Under review as a conference paper at ICLR 2023
Algorithm 1 FARR Optimized through PSRO
Input: X, C, and Initial policy sets 11 = (LIp, lag for Protagonist player and Adversary player
Compute expected FARR payoff matrix U as utilities Up (Tp 0) for each joint (Tp, 0) E II repeat
Compute Normal-Fone restricted NE σ = (op, σp) over population policies II using UX Calculate Protagonist policy Tp (e.g. [R(()))
7 = H U {
for at least one iteration do
Calculate new Adversary strategy 9 and associated estimator for (R(
II, II. {0}
end for
Compute missing entries in U from II until terminated early or no novel policies can be added Output: current Protagonist restricted NE strategy Sp
output mixed-strategy op over np In each PSRO iteration, to best-respond to the current adversary restricted NE dg a new protagonist policy is trained using RL. with a fixed environment experience budget. One or more random novel adversary pure strategies are added in each PSRO iteration. For wall-time parallclization, we add in each iteration. For each adversary strategy 0, we also train an evaluator RL policy Ti with the same hyperpameters as the protagonist to estimate BIR(0) and feasibility for the FARR utility function U/.
EXPERIMENTS
To illustrate the utility of filtering out infeasible tasks in robust RL setting, we perform experiments in environments where overly difficult or unsolvable task variations are possible while measuring performance under feasible conditions. a goal-based gridworld environment and three perturbed MuloCo (Todorov et al. 2012) control environments, we compare the performance of FARR with three alternative objectives: a standard minimax robust adversarial RL objective, domain random- ization, and the regret objective as proposed in Dennis al. (2020). Given a threshold for feasibility X, we evaluate worst-case performance within the set J of fcasible environment parameters. FARR outperforms each of these objectives because it is able to provide an adversarial training distribu- tion of environment variations limited only to instances that are discovered to be feasible, while other methods provide overly difficult or otherwise mismatched training distributions for worst-case feasible conditions.
We optimize FARR and other baseline objectives with PSRO and identical protagonist RL best- response algorithms in order to compare final performance given guarantees of asymptotically reach- ing an approximate NE for each zero-sum objective.
We describe each of the baseline objectives below:
Minimax
The standard objective for robust adversarial RL using unmodified Ug og). When infeasible tasks are allowed to the adversary, the standard minimax robust RL objective will focus on such tasks, resulting in overly cautious protagonist behavior or failed learning.
Domain Randomization (DR) With domain randomization, we train a single protagonist policy _ = (R PR) to saturation against uniform mixture of all possible environment variations RR (I)).. Domain randomization relevant feasible part of configuration space is underrepresented by the uniform measure. This objective is optimized by training single RL policy rather than with PSRO.
result in an exploitable agent if the
Regret Matching the objective used by PAIRED (Dennis et al., 2020), we use PSRO to approxi- mately solve for NE using the objective minl, max,, Eg-ap UU, 0) - Up (BB((O) 0)]. (B)( is estimated using the same method as with FARR by training an evaluation agent = against each 6. While designed to provide a distribution of tasks where the protagonist is known to be able to positively affect its performance through optimal behavior, this cur- riculum learning objective does not generally provide a task distribution suitable for robust learning specific set of tasks such as XP

---

Under review as a conference paper at ICLR 2023
Lava Worle Feasible Worst-Case Reward -5
Bahaoke Ramard
FARR (Curs)
Begrot
-15
Out
PSRO
(b)
(c)
Figure 2: (a) The Lava World grid environment. The adversary specifies the location of an unob- servable goal. The protagonist receives reward each timestep until it reaches the hidden goal. the protagonist steps in lava (red), the episode ends and it receives penalty of -55 reward. (b) The environment is underspecified and the goal can be placed in lava, forcing the agent to receive the lava penalty and creating infeasible task given À -10. (c) Solving for approximate NE using PSRO, the FARR objective results in an agent maximally robust to the feasible, non-lava goals while other objectives result in suboptimal worst-case performance among the feasible of tasks.
To calculate the worst-case episode reward within the set F of feasible environment parameters, we use our knowledge J= to enumerate a comprehensive set of feasible tasks on which every baseline's performance is measured. For the gridworld environment, the entire feasible set FA is calculated analytically. For MuloCo, a discretization of the continuous parameter space is caleu- lated. Feasibility for each in the discretized space is then measured by training an RL best-response (O) to completion and averaging final utility Up (B) 0) over 7 seeds. FA is then determined using equation [ We measure feasible worst-case episode reward mingg.F> (opp 0), averag- ing over 100 episodes for each value of 0. We use this as our metric for robustness to the feasible
FA.
5.1 LAVA WORLD
The gridworld task "Lava World" consists of small platform surrounded by lava, depicted in (a). The adversary specifies goal location that the protagonist needs reach, however the protagonist does not observe the goal, and the goal can be placed in lava. With an episode horizon of 20, the protagonist receives a reward -1 for every timestep that it does not reach the goal. the protagonist moves into lava, the episode ends, and it receives reward of -15 even if the goal is at that location. The feasibility threshold for this task is À = -00, thus making a parameter for this environment infeasible if the goal is put in lava and feasible otherwise. We use DDQN (Van Hasselt et al. 2016 to train protagonist RL policies.
Worst-case protagonist episode reward among all values in the feasible set 0 E FA is shown function of PSRO iterations for FARR and baseline objectives in Figure The minimax objec- tive fails because the adversary learns always suggest infeasible lava goals, and the protagonist learns to immediately jump in lava rather than waste time searching for a goal in non-lava cells. Likewise, domain-randomization fails because the majority of goals infeasible lava goals, in order to optimize the average-cass case, the protagonist learns the same suboptimal behavior as it does with minimax. The regret objective produces distribution of both feasible and infeasible goals where non-lava goals are the majority, however because this distribution does not minimize protag- onist reward over feasible goals, the regret protagonist does not maximize worst-case performance over the feasible set of tasks. FARR penalizes the adversary for suggesting the infeasible lava goals and otherwise provides base robust adversarial RL utilities for feasible goals, thus resulting a protagonist that maximizes worst-case reward among the actual feasible non-lava goal set FA.
5.2 MUJoCo
We compare FARR and other objectives on three MuJoCo control tasks, HalfCheetah, Walker2D, and Hopper using PPO (Schulman et al.) 2017) to train protagonist RL policies. In each of these environments, the adversary specifies parameters 9 = (a, 3) where a E (0, 10], e (0, 10) for beta distribution B(a, (8) used to sample from and generate 1D horizontal perturbing forces every
8

---

Under review as a conference paper at ICLR 2023
HallCheeteh Peasible Worst-Case Reword
Am-1000 1000
Amo
Am 1 1000
-250
4C0
Eoloode Roward
1000
-000
Latsode
-750
bprecde
205
500
1000
5
10
12
8
10
12
2
2
10
72
Beratione
PSPO
FARR (Ouns)
Rengret
Minimaa
(a)
Hopper Foasible Worst
A-200
A-600
300
600
Epoode Rexard
Easant Revoverd
Eoredt Recardd
200
300
402
200
100
200
3
4
6
10
12
14
16
18
8
10
13
14
16
16 6
10
19
14
P3RO
PSRD boredions
F3R0
Rorations
|AIHH (Durs)
Ragret
Minimax
OR
(b)
Figure 3: Worst-case MuloCo HalfCheetah (a) and Hopper (b) average episode reward among task parameters in the feasible set JF as a function of PSRO iterations for FARR and other baselines with multiple values of X.
timestep that are applied the torso of the protagonist's robot. The adversary has the ability spec- ifyy infeasible distributions of forces which make accruing reward in each task virtually impossible. We conduct experiments with each MuJoCo environment using three different feasibility threshold values for X, representing three different assumptions regarding the difficulty of test-time conditions that we wish to prepare for.
For HalfCheetah, in Figure 3 (a) we show worst-case protagonist reward among environment pa- rameters in the feasible set Jd as function of PSRO iterations with values [-1000, 0, 1000). The same is shown for Hopper with À E [200, 400, 600} in Figure 3 (b), and for Walker2D with {200, 400, 600} in the appendix due to space limitations. Across different values for in each of these environments, reward under the ground-truth F by penalizing the adversary to prevent it from providing infea- sible variations. Domain randomization and regret provide training distributions unconditioned on or any notion of FA,, which are only sometimes appropriate for robust performance as seen for A = 600 in (b) with domain randomization and regret and for À 1000 in F with regret. Otherwise, domain randomization and regret result in agents exploitable to some configura- tion F² Likewise, minimax consistently provides insurmountable conditions to the protagonist, resulting in failed learning and highlighting the need for methods like FARR to automatically limit adversary abilities in robust RL. Analysis on the mixed strategies learned by each objective's adver- sary is available in Section C of the appendix.
see that FARR is able to train agent which maximizes worst-case
6 DISCUSSION AND FUTURE WORK
We present FARR, a novel robust RL problem formulation and two-player zero-sum game objective in which we consider an underspecified environment allowing infeasible conditions and we train a protagonist to be robust only to the tasks which are feasible. By solving for approximate Nash equilibrium under the FARR objective using PSRO, we demonstrate that this method can produce robust agent even when the adversary is allowed to specify parameters which make sufficient performance at a task impossible. A limitation and avenue for future work that our current method for optimizing FARR does not directly optimize the adversary, instead relying random search and the PSRO restricted game solution to provide an optimal mixed strategy. future work, high-dimensional joint adversary best-responses with feasibility estimates be sample-efficiente optimized, FARR can provide prescribable solution to avoid the manual creation of complex rules to limit robust RL. adversaries in higher-dimensional sim-to-real configuration spaces.

---

Under review as a conference paper at ICLR 2023
REFERENCES
George W. Brown. Iterative solution of games by fictitious play, Activity analysis of production and allocation, pp. 374,77,,1 1951.
Noam Brown, Adam Lerer, Sam Gross, and Tuomas Sandholm. Deep counterfactual regret mini- mization. In International conference on machine learning, pp. 793 802. PMLR, 2019.
Andres Campero, Roberta Raileanu, Heinrich Küttler, Joshua Tenenbaum, Tim Rocktäschel, and Edward Grefenstette. Learning with amigo: Adversarially motivated intrinsic goals. arXiv preprint arXiv:2006. /2/22, 2020.
Michael Dennis, Natasha Jaques, Eugene Vinitsky, Alexandre Bayen, Stuart Russell, Andrew Critch, and Sergey Levine. Emergent complexity and zero-shot transfer via unsupervised environment design. arXiv preprint arXiv:2012.02096 2020.
Yuqing Du, Pieter Abbeel, and Aditya Grover. It takes four to tango: Multiagent selfplay for auto- matic curriculum generation. arXiv preprint arXiv:2202. 10608, 2022.
Carlos Florensa, David Held, Xinyang Geng, and Pieter Abbeel. Automatic goal generation for reinforcement learning agents. In International conference on machine learning, pp. 1515-1528. PMLR, 2018.
Adam Gleave, Michael Dennis, Cody Wild, Neel Kant, Sergey Levine, and Stuart Russell. Adver- sarial policies: Attacking deep reinforcement learning. arXiv preprint arXiv: /90:. 10615, 2019.
Johannes Heinrich and David Silver. Deep reinforcement learning from self-play imperfect- information games. arXiv preprint S.Xiv:1603.0112/11 2016.
Han Hu, Kaicheng Zhang, Aaron Hao Tan, Michael Ruan, Christopher Agia, and Goldie Nejat. A sim-to-real pipeline for deep reinforcement learning for autonomous robot navigation in cluttered rough terrain. IFFE Robotics and Automation Letters, 6(4):6569 6576, 2021.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.69888 69800, 2014.
Aounon Kumar, Alexander Levine, and Soheil Feizi. Policy smoothing for provably robust rein- forcement learning. arXiv preprint arXiv:2106.1 /420, 2021.
Mare Lanctot, Vinicius Zambaldi, Audrunas Gruslys, Angeliki Lazaridou, Karl Tuyls, Julien Pérolat, David Silver, and Thore Graepel. A unified thee-the approach to multiagent inforcement learning. In Advances in Neural Information Processing Systems, pp. 4190 4203, 2017.
Zhongyu Li, Xuxin Cheng, Xue Bin Peng, Pieter Abbeel, Sergey Levine, Glen Berseth, and Koushil Sreenath. Reinforcement learning for robust parameterized locomotion control of bipedal robots. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pp. 281 1-217.. IEEE, 2021.
Eric Liang, Richard Liaw, Robert Nishihara, Philipp Moritz, Roy Fox, Ken Goldberg, Joseph Gon- zalez, Michael Jordan, and lon Stoica. Rllib: Abstractions for distributed reinforcement learning. In International Conference on Machine Learning, pp. 3053-3062. PMLR, 2018
Siqi Liu, Luke Marris, Daniel Hennes, Josh Merel, Nicolas Heess, and Thore Graepel. Neupl: Neural population learning. arXiv preprint arXiv:2202.07415. 2022.
Xiaobai Ma, Katherine Driggs-Campber and Mykel J Kochenderfer. Improved robustness and safety for autonomous vehicle control with adversarial reinforcement learning. In 2018 IEEE Intelligent Vehicles Symposium (IV), pp. 1665-1671. IEEE, 2018.
Ajay Mandlekar, Yuke Zhu, Animesh Garg, Li Fei-Fei, and Silvio Savarese. Adversarially robust policy learning: Active construction of physically plaausbbee perturbations. In 2017 IÉEE/RSJ International Conference Intelligent Robots and Systems (IROS), pp. 3932-3939. IEEE, 2017.
10

---

Under review as a conference paper at ICLR 2023
Stephen McAleer, John Banister Lanier, Kevin A Wang, Pierre Baldi, and Roy Fox. Xdo: A double oracle algorithm for extensive- games. Advances in Neural Information Processing Systems, 22-111199,, 2021.
Stephen McAleer, JB Lanier, Kevin Wang, Pieree Baldi, Roy Fox, and Tuomas Sandholm. Self-play psro: Toward optimal populations in two-player zero-sum games. arXiv preprint arXiv:2207.06541, 2022a.
Stephen McAleer, Kevin Wang, Mare Lanctot, John Lanier, Pierre Baldi, and Roy Fox. Anytime optimal psro for two-player zero-sum games. arXiv preprint arXiv:2201.0772 077000 2022b.
H Brendan McMahan, Geoffrey J Gordon, and Avrim Blum. Planning in the presence of cost functions controlled by an adversary. In Proceedings of the 20th International Conference on Machine Learning (ICML-03), pp. 536-543, 2003.
Hideaki Nakao, Ruiwei Jiang, and Siqian Shen. Distributionally robust partially observable markov decision process with moment-based ambiguity. SIAM Journal on Optimization, 31(1):661-43) 2021.
M Andrychowicz OpenAI, Bowen Baker, Maciek Chociej. Rafal Jozefowicz, Bob McGrew, Jakub Pachocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray, et al. Learning dexterous in-hand manipulation. arXiv preprint N.KYv18808.00177 2(3):5-1, 2018.
OpenAl OpenAI, Matthias Plappert, Raul Sampedro, Tao Xu, Ilge Akkaya, Vineet Kosaraju, Peter Welinder, Ruben D'Sa, Arthur Petron, Henrique P O Pinto, et al. Asymmetric self-play for 2021.
automatic goal discovery in robotic manipulation. arXiv preprinl arXiv:2
Jack Parker-Holder, Minqi Jiang, Michael Dennis, Mikayel Samvelyan, Jakob Foerster, Edward Grefenstette, and Tim Rocktäschel. Evolving curricula with regret-based environment design. arXiv preprint arXiv:2203.01302 2022.
Anay Pattanaik, Zhenyi Tang, Shuijing Liu, Gautham Bommannan, and Girish Chowdhary. Robust deep reinforcement learning with adversarial attacks. arXiv preprint arXiv:/712.03632, 2017.
Lerrel Pinto, James Davidson, Rahul Sukthankar, and Abhinav Gupta. Robust adversarial reinforce- ment learning. In International Conference on Machine Learning, pp. 2817-2826. PMIR, 2017.
Sebastien Racaniere, Andrew K Lampinen, Adam Santoro, David P Reichert, Vlad Firoiu, and Timothy P Lillicrap. Automated curricula through setter-solver interactions. arXiv preprint arXiv:1909. 12822, 2019.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
Sainbayar Sukhbaatar, Zeming Lin, Ilya Kostrikov, Gabriel Synnaeve, Arthur Szlam, and Rob Fer- gus. Intrinsic motivation and automatic curricula via asymmetric self-play. In 6th International Conference on Learning Representations, ICLR 2018, 2018
Kai Liang Tan, Yasaman Esfandiari, Xian Yeow Lee, Soumik Sarkar, al. Robustifying reinforce- ment learning agents via action space adversarial training. In 2020 American control conference (ACC), pp. 3959-3964. IEEE, 2020.
Chen Tessler, Yonathan Efroni, and Shie Mannor. Action robust reinforcement learning and appli- cations in continuous control. In International Conference on Machine Learning, pp. 6215-6224.
PMLR, 2019.
Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojciech Zaremba, and Pieter Abbeel. Do- main randomization for transferring deep neural networks from simulation to the real world. In 2017 IEEE/RSJ international conference on intelligent robots and systems (IROS), pp. 23-30. IEEE, 2017.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physies engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033. IEEE, 2012. doi: 20110/00...0022333666

---

Under review as a conference paper at ICLR 2023
Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q- learning. AAAI conference on artificial intelligence, volume 30, 2016.
Eugene Vinitsky, Yuqing Du, Kanaad Parvate, Kathy Jang, Pieter Abbeel, and Alexandre Bayen. Robust reinforcement learning using adversarial populations. arXiv preprint wrXiv:2008.01825 2020.
Oriol Vinyals, Igor Babuschkin, Wojciech M Czamecki, Michaöl Mathieu, Andrew Dudzik, Juny- oung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in starcraft ii using multi-agent reinforcement learning, Nature, 575(7782)3500335, 2019,
Quan Vuong, Sharad Vikram, Hao Su, Sicun Gao, and Henrik I Christensen. How pick the do- main randomization parameters for sim-to-real transfer of reinforcement learning policies? arXiv preprint ar /9..//99111771 2019.
Rui Wang, Joel Lehman, Jeff Clune, and Kenneth O Stanley. Paired open-ended trailblazer (poct): Endlessly generating increasingly complex and diverse learning environments and their solutions. arXiv preprint orXiv:/901.0175.3 2019.
Rui Wang, Joel Lehman, Aditya Rawal, Jiale Zhi, Yulun Li, Jeffrey Clune, and Kenneth Stanley. Enhanced poet: Open-ended reinforcement learning through unbounded invention of learning challenges and their solutions. In International Conference on Machine Learning, pp. 9940- 9951. PMLR, 2020.
Huan Zhang, Hongge Chen, Chaowei Xiao, Bo Li, Mingyan Liu, Duane Boning, and Cho-Jui Hsieh. Robust deep reinforcement learning against adversarial perturbations on state observations. Ad- vances in Neural Information Processing Systems, 33:21024-21037, 2020.
Huan Zhang, Hongge Chen, Duane Boning, and Cho-Jui Hsieh. Robust reinforcement learning on state observations with learned optimal adversary. arXiv preprint arXiv:2101.08455 2021.
12

---

Under review as a conference paper at ICLR 2023
Wallber2D Feasible Worst Reward
A=600
400
Esicuuu Bonaro
2010
4:30
2003
Epeede
100
200
10
Morations
PERO herations Hegnet
PSRO
(a)
Figure 4: Worst-case MuloCo Walker2I) reward among task parameterizations the feasible set F function of PSRO iterations for FARR and other baselines with multiple values of A.
A MuJoCo FEASIBLE SETS
For MuJoCo experiments, in order to measure each objective's worst-case average episode reward among feasible tasks, we evaluate on a discrete approximation of F², In these environments, e represents parameters of beta distribution B(a, 33. E (0. 10), 3 (0, 10] sampled from each timestep to generate horizontal perturbing forces applied to the simulated robot. We consider a dis- cretization of O with 11 different values in [0.01, 10] for both and B. For each each combination 0 = (a, B), we train seeds of a RL best-response BR to completion using the same hyperparam- eters the protagonist. The average final utility U, (O)) 0) across seeds is then used to calculate F using equation (I). In Figure for each environment, show the 7-seed average Up (BB(()))) for every value of 0 and the resulting feasible sets F (shown in green) that we evaluate robustness to for each value of X.
Holtchoctah Feasible Robustnass
HAtoowoo ISS PIANNS SUIEE PUVEL
Aphs
Hopper
MWNINEE LIIN LEG MWWI
60
20
XO
(d)
ressible
MeBb E6 Peourr vn Beto 261 Nerta
20
10.0
060 80 126
601 70
KO IDD
Figure 5: (a,c.e) Estimated values for (BR)( across the parameters and 3 that the adversary has control over. (b,d,f) The feasible sets JA used for evaluation marked in green for each value A.
13

---

Under review as a conference paper at ICLR 2023
B MUJoCo LEARNED ADVERSARY STRATEGIES
After running PSRO to completion, the output strategies are both a protagonist mixed strategy dp and an adversary mixed strategy dg that should jointly approximate a Nash equilibrium each the two-player zero-sum game objectives we optimize. In figures 6 7 and we display the distribution of values induced by the final MuloCo adversary mixed strategies dg for the minimax, regret, and FARR objectives. For each À value considered, we overlay in shades of green the number of (R(()) seeds used measuring F that achieved a final average episode reward greater than equal for X.
Across all environments, the minimax adversary consistently selects the most difficult values pos- sible outside of any variations considered feasible with the À values tested. In contrast, FARR mixes between values both well inside the feasible regions and at the edges where task variations are as challenging as possible while remaining feasible.
HalfCheetah Final Adversary Strategy Selection
Minimax
Regret
FARR (A 11000)
FARR 0)
FARR 1000)
Adversse Selectin Pobbablly
7.5
0.0
PSRO Seed
10.0
5.0
Fearible
2.5
PSRO Seed 2
7.5
PSRO
Alpha
Alpha
Alpha
Alpha
Alpha
Figure 6: HalfCheetah 0 distributions induced by the final adversary PSRO mixed strategy do for each objective.
14

---

Under review as a conference paper at ICLR 2023
Hopper Final Adversary Strategy Selection
Mirimax
Negret
FAIR ( 200)
FAIR (A 400)
FARI (A - 600)
Adwer sary Select Probobililyy
7.5
2.5
5.0
PSRO Seed 2
7.5
25
Alpha
Figure 7: Hopper 8 distributions induced by the final adversary PSRO mixed strategy dg for each objective.
Walker2D Final Adversary 0 Strategy Selection
Minimax
Regret
FARR 200)
FARR O 400)
FARR (A
600)
Seeds Ressibl Advenarry Sellction Pobbablity
7.
PSRO
10.0
5.0
PSRO Seed 2
7.5
2.5
Apha
Alpha
Alpha
Figure 8: Walker2D distributions induced by the final adversary PSRO mixed strategy ag for each objective.
C PROPERTIES NASH EQUILIBRIA FOR THE FARR TRANSFORMED GAME
In this section show that solving for Nash equilibria in the FARR transformed game will give the same result as solving for NE a regular minimax robust RL game with the adversary strategy space already limited to only feasible strategies. The benefit of solving the FARR game is that the set of feasible adversary strategies does not need to be known a priori.
Define the set of X-infeasible adversary strategies I O JPA Define the of all possi- ble protagonist strategies Tp as IT>, For both protagonist utility functions U and Up the adver- sary's utility function U², and U, is the negative of the protagonist's, s, (T 0) -U- ( 0) and Ua Ua(Tp, ()) UUp(pp, 0) for any Tp and 0.
15

---

Under review as a conference paper at ICLR 2023
Theorem 1. For sufficiently large C, a Nash equilibrium joint strategy FARR of FARR transformed game GFARR with utility function Up, protagonist strategies I, and all adversary strategies @ is also a Nash equilibrium of a reduced game Greduced with utility function Up, protagonist strategies 11p and only X-feasible adversary strategies Fx.
Proof. Let C take a sufficiently large value greater than any utility achievable by the protagonist C> maxx, 0 (Tp, Since U.2 (Tip 0¹) = C when O' I then for any 0' E IA, any FA, and any Tp E : U2 (T.. 0') < (pp 0) = (T
p, 0). It follows, as long as F is nonempty, that for all 0' E I there exists adversary strategy 0 E F which achieves higher adversary utility against every Tp E , thus all X-infeasible adversary strategies O' E I are strictly dominated in the FARR transformed game.
If all 0' E I are strictly dominated then they are not in the support of any Nash equilibrium for GAARR. Furthermore, it is possible to remove strategies 0' E I though iterated elimination of strictly dominated strategies (IESDS) to reduce GFARR to Gredeed since the adversary strategy set for Greduced is θ Id F² and for all 0 F² and Tp E II; (TT. 8) = Up 8). If Groduced is an outcome of IESDS from GEARR, then if FARR is a NE of GEARR. it is also an NE of Geduced-
By employing a penalty C' rather than directly pruning infeasible strategies from the PSRO restricted game, the FARR objective can be defined without consideration to the mechanics of any specific algorithm like PSRO. The FARR objective can potentially be optimized with two-player zero-sum game methods other than PSRO as well, though exploring the use of more optimization methods is left to future work.
D SELECTING VALUES FOR X
FARR is most applicable when the appropriate value for À can be derived from problem require- ments. For instance, À would ideally be set the lowest average return that an optimal agent would receive across task variations deployment. This value could come from the environment defini- tion. where doable tasks provide a minimum level of return if accomplished. Alternatively, could be chosen by anticipating the maximum difficulty of task variations that would be seen at test-time or on which robust performance important to the practitioner.
In the case where appropriate value of X is completely unknown and cannot be derived from problem requirements, in a dimeennniio task variation space, manually tuning the adversary's capabilities without FARR may be appropriate. In complex, high-dimensional task variation space, searching for a useful À may be easier than a direct search over the space of adversary legal strategy sets because À presents single variable to tune, rather than large number of legal parameter ranges or complex conditional constraints between selverrsry-sppecffii be defined. In this work, we consider the case where À can be derived from problem requirements.
parameters that may need to
E PSRO COMPARISON WITH SELF-PLAY
In Lava World, for each of the two-player zero-sum game objectives, we compare PSRO to self-play in which the protagonist, adversary, and evaluator T. continuously train together. For all self eelf-play agents, we train with PPO to enable stochastic policies like PSRO able to output. Regret self-play matches the original PAIRED algorithm from Dennis et al. (2020).
Although self-play may potentially yield competitive performance some scenarios, unlike PSRO, it lacks any guarantees of converging to an approximate Nash equilibrium in two-player zero-sum partially-observable Markov or extensive-form games. Seen in Figure 9 we see that self-play for both FARR and PAIRED fails converge, reaching maximum feasible-space worst-case average reward of -9 as agent policies cycle and learn to represent nearly deterministic strategies during most points in training. The NE for Lava World requires mixed-stratege in which the adversary samples a high-entropy (non-uniform) distribution of hidden goals. The degenerate solution to Minimax is reached by both algorithms.
16

---

Under review as a conference paper at ICLR 2023
Lava World Feasible Worst-Case Reward
Episode
FARR PSRO
FARR SP
Regret PSRO
PAIRED (Regret SP)
Minimax
0
2
6
Timesteps
106
Figure 9: Worst-case average episode reward among goals in F vs timesteps collected for each two-player zero-sum game objective optimized with both PSRO and PPO Self-Play.
SAMPLE EFFICIENCY IN MuJoCo EXPERIMENTS
In figure I0 we show worst-case protagonist reward among environment parameters the feasible set function of timesteps collected by PPO learners in PSRO for each À value considered. FARR and regret take more timesteps per PSRO iteration than minimax because they both train evaluation policies (R) for each adversary strategy added to the population in order to evaluate their utility functions in the PSRO restricted game.
HainCheetahh Teasible Worst-Case Reward
>- 1300
A-D
A-1000
Roward
-250
Eproodle Rewrrdd
-100
0.0 0.3 1.0
15 20
25 2.0 3.5
0.0 02
10
1.5
20
25
30
3.5
0.5 1.0 0.0
155 20
25
30
As
Timestaps
1c8
Timestops
LeB
Timestacs
FARR (Ours)
Regret
Nimmex
DR
(a)
Hopper Feas ble Worst-Case Reward
À-200
A-400
A-600
400
Fpivele Gwwwer
Seeweard
200
.010
403
200
100
0.5
0.0
0.5 1/ 1.200 25 3.
30
3.5 4.0 4.5
Timesteps
1s8
limesteps
lsB
Timestecs
166
FARR (Ours)
Ninmax
(b)
Walker2D Feasible
A=200
A=400
A=600
Epivasle Dimaurd
400
Epesods Howerrd
300
200
200
0.0
2.4 32 6.4
0.0 0.8
3.2 40 4.0 5.6 Tiiessops
0.8 1.5 2,4 d' 5.5 6/4
Timestops
FARR (Ours)
Regret
Ninimax
(c)
Figure 10: Worst-case MuJoCo HalfCheetah (a), Hopper (b), and Walker2D (c) average episode reward among task parameters in the feasible set J as function of timesteps collected for FARR and other baselines with multiple values of A.
17

---

Under review as a conference paper at ICLR 2023
G ENVIRONMENT DETAILS
G.1 LAVA WORLD
In the Lava World grid environment, the protagonist uses discrete actions to move in each of the cardinal directions. For observations, the protagonist receives a one-hot encoding of its current location, and the protagonist does not observe the goal location. The adversary strategy space θ is to define the hidden goal location and consists of every grid cell location in the environment's 's 5x5 grid with the exception of the protagonist's 's fixed starting location.
The protagonist receives a reward of -1 in each timestep that it does not reach the hidden goal location suggested by the adversary and -15 if it moves into lava cell, even if the lava cell goal. An episode ends after either 20 timesteps clapse or the goal is reached.
G.2 MuJoCo ENVIRONMENTS
The MuloCo environments use the Mujoco physics engine (Todorov et al. 2012) and are modified versions of the perturbed robotic control environments originally presented in Pinto et al.] (2017).
In each environment variation (HalfCheetah. Hopper, Walker2D), max episode duration of 200 timesteps imposed, and the proportion of time remaining the range [0, is appended to each task's 's original observation. We use the continuous action space variants of each task.
The adversary strategy space consists of continuous and parameters in the range (0. 10] for a beta distribution B(x,, 3) used to generate horizontal perturbing forces sampled and applied to the robot's torso every timestep. Each timestep, a new horizontal force F E [- FmFax m
F (2FFmaxx Fmax is generated where X B B(a, B) and Fmax 100.
When discretizing values of (a, for evaluation purposes to measure policy's performance across values in θ or FA, we use a, {0.01, 1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0, 10.03.
H TRAINING DETAILS
Protagonist RL training details are provided below for each environment. Policies used to estimate BR(O) for a given 9 use the same training procedure and parameters the protagonist. Like Dennis et al. 2020). we train protagonist policies on the casier-to-learn unmodified environment reward Up rather than our two-player game objective UP because the only component of the game utility that the protagonist affect is Up. A protagonist best-response that maximizes Up also maximizes Up- Critically, we still calculate U² in the PSRO empirical payoff matrix U and use U calculate the meta-game NE strategy o (op:
H.1 LAVA WORLD
We train Lava World protagonist RL policies using DDQN Van Hasselt et al, (2016). All Lava World protagonist RL policies are stopped training after either 150,000 timesteps are collected once per- formance plateaus (average episode return doesn't 't improve by 0.5 over 20,000 timesteps and a min- imum of 80,000 timesteps is collected). Lava World DDQN hyperparameters are presented below. Our RL code was built using the RLlib framework Liang et al. (2018), and any hyperparameters not specified are the version 1.0.1 defaults. We use infeasibility penalty of C. 50.
18

---

Under review as a conference paper at ICLR 2023
algorithm
DDQN
Van Hasselt al. 2016
circular replay buffer size
50,000
prioritized experience replay
No
total rollout experience gathered each iter
8 steps
learning rate batch size
0.007
1024
optimizer TD-error loss type
Adam Kingma Ba 2014
MSE
target network update frequency
every 4,000 steps
MLP layer sizes
[256, 256]
activation function discount factor y exploration e
tanh
1.0
Linearly annealed from 0.5 to 0.01 over 20,000 timesteps
Table 1: Lava World protagonist DDQN hyperparameters
algorithm
PPO
Schulman al. (0017)
GAE A
0.9
entropy coeff
0.007
clip param
0.276
KL target
3e-4
KL coeff
0.0016
learning rate
5e-4
train batch size
8192
SGD minibatch size
64
num SGD epochs on each train batch
40
shared policy and value networks
No
value function clip param
10
MLP layer sizes activation function discount factor 7
[256, 256]
Tanh
1.0
Table 2: Lava World PPO self-play protagonist hyperparameters
algorithm
PPO Schulman et al. 2017)
GAE
0.95
entropy coeff
0.006
clip param
0.292
KL target
0.092
KL coeff
0.168
learning rate
3e-4
train batch size
8192
SGD minibatch size num SGD epochs on each train batch
64
30
shared policy and value networks
No
value function clip param MLP layer sizes activation function discount factor 7
100
[256, 256]
Tanh
1.0
Table 3: Lava World PPO self self-play adversary hyperparameters
19

---

Under review as a conference paper at ICLR 2023
In PSRO, calculate the payoff matrix U. estimate Up (Tp. 0) for each pairing of player policies Tp E 1pp and 0 E IIg using a single rollout because both Lava World environment dynamics and evaluation DDQN policies are deterministic.. The normal-form meta- game Nash Equilibrium U is calculated using 2000 iterations of Fictitious Play (Brown 1951). Calculating the meta-game NE typically takes second less of wall-time compute.
During PSRO evaluation, we measure the performance of the protagonist meta-game mixed strategy pp. in which a new protagonist policy Fp op sampled at the begining of each episode.
In self-play, the adversary is trained as a single-step agent via PPO. For simplicity, we keep the same network architecture for all self-play agents, and the adversary observes constant vector of zeros.
H.2 MuJoCo
We train MuJoCo environment protagonist RL policies using PPO (Schulman et al. 2017). Each PPO model consists of an MLP followed by an LSTM with shared weights between the policy and value function, branching into final output layers after the LSTM. PPO hyperparameters for each MuJoCo environment are presented below. Any hyperparameters not specified are the RLlib version 1.0.1 defaults. We use an infeasibility penalty of C 1e6.
algorithm
PPO
Schulman et al. 2017
GAE A
0.9
entropy coeff
0.01
clip param
0,001
KL target
0.004
KL coeff
0.522
learning rate
5e-4
train batch size
4096
SGD minibatch size
64
num SGD epochs on each train batch
5
shared policy and value networks
Yes
value function loss coeff
0.001
value function clip param continuous action range
100
[-1.0, 1.0] for each dim
MLP layer sizes activation function
[32]
Tanh
LSTM cell size
32
LSTM max sequence length
20
discount factor
0.99
RL policy training stopping condition
7e6 timesteps
Table 4: HalfCheetah PPO hyperparameters
20

---

Under review as a conference paper at ICLR 2023
algorithm
PPO Schulman al. 2017
GAE X
0.9
entropy coeff
0.001
clip param
0,002
KL target
0.036
KL coeff
0.013
learning rate
7c-4
train batch size
4096
SGD minibatch size
32
num SGD epochs each train batch
5
shared policy and value networks
Yes
value function loss coeff
0.001
value function clip param
10
continuous action range
[-1.0, 1.0] for each dim
MLP layer sizes
[64. 64]
activation function
Tanh
LSTM cell size
32
LSTM max sequence length
20
discount factor
0.99
RL policy training stopping condition
6e6 timesteps
Table Hopper PPO hyperparameters
algorithm
PPO
Schulman et al. 2017
GAE X
0.95
entropy coeff
0.0
clip param
0.014
KL target
0,005
KL coeff
0.007
learning rate
9e-4
train batch size
1024
SGD minibatch size
64
num SGD epochs on each train batch
10
shared policy and value networks
Yes
value function loss coeff
le-4
value function clip param
1000
continuous action range
[-1.0, 1.0] for each dim
MLP layer sizes activation function
[32]
Tanh
LSTM cell size
32
LSTM max sequence length
20
discount factor ?
1.0
RL policy training stopping condition
6e6 timesteps
Table 6: Walker2D PPO hyperparameters
In PSRO, to calculate the payoff matrix U we estimate (Tp 8) for each pairing of player policies Tr E HI and 0 Th, using 100 rollouts as perturbed MuJoCo environment transition dynamics stochastic. The normal-form meta-game Nash Equilibrium over U is also calculated using 2000 iterations of Fictitious Play.
Although PSRO convergence guarantees not provided for cutinuous-actton environments, in McAleer et al. (20TT) McAleer et al.] (2022b), and our own experiments, PSRO reliably produces meta-game NE mixed strategies that empirically difficult for opponent exploit.
21

---

Under review as a conference paper at ICLR 2023
I COMPUTATIONAL COSTS
Experiments were performed on a local computer with 128 logical CPU-cores, 4 RTX 3090 GPUs, and 512GB RAM. Due to small network sizes and comparably high overhead of CPU-based en- vironments, logging, and other tasks, most experiments were performed without GPU acceleration. All individual training runs for given player against a fixed opponent took CPU-cores each. Lava world experiments individually ran for roughly 8 to 24 hours each, while MuJoCo experiments individually ran for roughly 72 hours each.
J CODE
GitHub link for our experiment code will be provided under the MIT license in an updated version of this work.
Our code is written on top of the RLlib framework (Liang et al. 2018 and uses environments built using the MuloCo physics engine (Todorov et al. 2012), both of which are open-source and available under the Apache-2.0 Licence.
22


Reviews:
[{"review_id": "yDdxM9PAFfv", "reviewer": "Reviewer_Rk3h", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work focuses on the robust reinforcement learning paradigm, whereby an agent is trained to be robust to worst case variations in an environment. The paper proposes an approach called FARR, whereby the adversary (proposing variations) is penalized if the environment proposed is not solvable. The paper sets out with great promise, working in an interesting area, but falls short with implementation details and experiments.", "summary_of_the_review": "The idea makes sense and is well motivated. I was assuming from the intro that the authors had figured out how to learn to predict the feasibility of an environment, which seems like the hardest part of getting this to work. Unfortunately instead the method relies on knowledge of the feasibility which is not possible in any large scale environment we would care about. Further, baselines like regret perform well and don't need this. \n\nNot only that, but FARR has no theoretical results (vs. regret-based approaches which do), and the experiments are low dimensional and toy, with no analysis on the different environments proposed by the adversaries. This paper makes sense as v0 of an idea for a workshop but is far below the bar for ICLR. \n\nI would definitely provide a higher score if the authors figure out how to learn feasibility, provide greater analysis of the emergent environments proposed and also a greater breadth of tasks. This seems beyond the scope for a two week discussion phase but hopefully is possible for the next conference cycle. \n", "strength_and_weaknesses": "### Strengths\n\n* The idea is sensible and well motivated, this is clearly an important area for real-world RL and has received increased interest recently with methods such as PAIRED, PLR and ACCEL. \n* The nice thing with this in principle is it allows the adversary to propose the hardest solvable environments, although that was not really demonstrated anywhere. \n\n\n### Weaknesses\n\n* The entire paper relies on an assumption that we have access to the feasibility of an environment, which is almost never the case in any of the practical settings where this may be used. The authors attempt to sweep this under the rug by saying \"we note that, in order to evaluate our approach in this paper, we focus on environments where we can, in fact, calculate (the feasible region)\". This is remarkable to me. The entire paper is focused on the idea of being able to learn a good policy with unfeasible regions where the feasibility is unknown, but then in the experiments we assume access to feasibility. This completely invalidates the entire point of the work, and in fact, makes me even more impressed by the performance of regret in the experiments which comes close despite not having extremely privileged information. \n* The paper also implies that the reward penalty is fed back into a learned adversary which can effectively infer feasibility. Sadly this is also not true as the authors instead use random sampling. \n* Given that the current method relies on random sampling, it should also compare against Robust PLR (Jiang et al, Replay Guided Adversarial Environment Design, NeurIPS 2021) which is a far stronger method empirically than PAIRED. \n* The experiments are very toy, using HalfCheetah and Walker and then a tabular MDP. The field has moved on to much larger settings since the original RARL paper in 2017. At minimum it should be considering more environments, something like the RWRL suite from pixels would make sense. This is not even super expensive to run. \n* There is no analysis on the different types of emergent environments proposed by the different methods. This is crucial given toy experiments!\n* What makes all of this even worse is that there is no limitations section where it is acknowledged that the paper doesn't do what it states out to do in the intro. If someone new to the field read this they may be led to believe FARR could infer the feasibility and that it was superior to regret based approaches. Please revise this before putting the paper on arxiv etc, even if unable to address the other issues. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "x_UqsEvZRQ", "reviewer": "Reviewer_9k7R", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This manuscript investigates the setting where there exists a mismatch between the training MDP and the testing MDP (known as robust RL). While robust RL is quite conservative in that the adversary could minimize the protagonist's utility over every MDP in the uncertainty set, this manuscript believes that it is too conservative. Instead, the manuscript defined a feasible set of MDPs that the adversarial could achieve, by asserting that there must exist a policy in that adversarial MDP to achieve at least $\\lambda$ in terms of the return. It subsequently lifts the constraint by replacing it with a substantially large penalty if the support of the random MDP of the adversarial has a positive density on the non-feasible area. With this setting, the robust RL policy and the adversarial could be trained by PSRO. Simulations are conducted on gridworld and 3 MuJoCo tasks.", "summary_of_the_review": "I'm positive on this manuscript for its novelty in the new setting and its quality in presentation. I have reservation, though, on its empirical performance.", "strength_and_weaknesses": "I agree that robust RL is indeed too conservative and the new feasible uncertainty set is relevant. The new setting proposed by the manuscript makes sense to me. It is also interesting that the setting could be solved by PSRO in quite an intuitive way. Some detailed reviews follow.\n\nPros:\n\n1. The new setting is an interesting extension towards robust RL\n2. The proposed algorithm through PSRO is intuitive and sensible.\n3. Presentation of this manuscript is easy to follow.\n\nCons:\n\nThe major concern of mine is how useful this method could be, in practice. In the experiments, the agents that are trained against the *feasible adversary* demonstrates some decent performance in only 3 MuJoCo tasks (I have to assume that it doesn't pan out on the rest of the tasks). Notice that this is when the algorithm trained against the feasible adversary is tested against the feasible adversary. What if the agent trained against the feasible adversary is tested against a general adversary (potentially described by a real application task)? The manuscript does not provide such an answer, but I'm not very confidence on this.\n\nMisc:\n\nIn the abstract \"In real-world environments, choosing the set of possible values for robust RL can be a difficult task\", the term \"value\" could be confused with the value function of RL. Use a different term.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "qefOfZ9Bnql", "reviewer": "Reviewer_yKnP", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper studies the robust reinforcement learning problem where the set of feasible environmental parameters are restricted. The authors propose the Feasible Adversarial Robust RL (FARR) formulation to this problem which specifies that the feasible parameter set is the set where the expected return of the protagonist agent is not less than a certain threshold. Essentially being a min-max game, the authors then solve the FARR problem using the Policy Search Response Oracle (PSRO) algorithm. ", "summary_of_the_review": "My main concerns are summarized at the weaknesses part of the above.", "strength_and_weaknesses": "Strengths: \n1. The problem of robust optimization with feasible parameter set is interesting and practical. The setting is well-motivated.\n2. The line of related work seems to be well-discussed. \n\nWeaknesses:\n1. Although the problem is well-motivated, the formulation is somewhat trivial since it requires prior-knowledge of the feasible set of parameters. This is either directly fed as prior-knowledge, or using the penalty coefficeint C (which itself is unknown and questionable) as well as the threshold $\\lambda$ for determining the feasible set. This makes the practicality of the algorithm very limited. In some sence, if one treats it as prior knowledge, the problem reduces to the same standard robust RL problem where the adversary's feasible set of parameters is specified. \n2. Because the essense of the formulation is still aligned with a typical robust RL setting, the algorithms for robust RL, like PSRO, are still applicable to the FARR formulation. And the methodological/technical contribution is therefore very limited. \n3. In evaluation, the solution based on the FARR formulation is better than the baselines. This is not surprising since the baselines are not fed with the feasibility information that the FARR formulation has.\n4. The feasiblity definition is a bit contrived in the sense that it essentially removes all the environments where the expected return for the protagonis agent is small (than a threshold). First of all, the threshold is not always known, or sometimes unclear. Second, it is not equivalent to \"feasible\". In practice, many feasible parameters may yield very low return, but you cannot just ignore these parameter. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "j5q7u0DY01", "reviewer": "Reviewer_uRb1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper focuses on generalization performance in reinforcement learning: when applying the Robust Reinforcement Learning approach, it is important to know how to design the set of environmental parameters to be considered. The authors propose a method to approximate the Nash equilibrium solution in a zero-sum game, called Policy Space Response Oracles (PSRO), which is based on the Nash equilibrium solution in a zero-sum game. By applying a method for approximating Nash equilibrium solutions, called Policy Space Response Oracles (PSRO), a method is proposed for finding policies that maximize worst-case performance for a restricted set of environmental parameters. Experimental results show that higher utility can be achieved in a simple Grid World-like task and in a task using MuJoCo, compared to the case where Domain Randomization, Maximin, and Regret are used as measures.", "summary_of_the_review": "Based on the above mentioned comment, I suggest weak accept. Although there are weak points, it has a potential to be usefulness in practice.", "strength_and_weaknesses": "# Strength: \n\nApproach: How to design the set of uncertainty parameters to be considered is a very important issue in robust reinforcement learning. Introducing a threshold for the utility and limiting the set of uncertainty to the parameters where the best possible performance is better than or equal to the threshold is a simple and natural approach. This paper proposes an approach to achieve this goal and show its usefulness in experiments. \n\nTest Task Design: The test problem designed in this paper is, I think (not claimed by the authors), a good task revealing the issue of domain randomization, maximin approach, and regret minimization approach. It is very simple and easy to understand. \n\nEvaluation: Numerical experiment on the test problem reveals the advantage of the proposed framework over domain randomization, worst-case maximization, and regret minimization. \n\n # Weaknesses\nMin-Max vs Max-Min: In the proposed framework, the environment parameters where the optimal policy can reach the utility above the given threshold are considered as feasible parameters and the worst-case maximization is performed on such a restricted environment parameter set. This does not guarantee that the optimal policy for the worst-case in the restricted environment parameter set is above the threshold in general, because min-max is not equal to max-min. This limitation is not evaluated nor discussed in this paper. \n\nComputational Time: In the experimental evaluation, the performance of the approach is displayed against PSRO iterations. Therefore, it is unclear how many interactions have been performed in the approach. \n\nEvaluation (in particular on MuJoCo): The usefulness of the proposed approach is evaluated only on the basis of PSRO. However, because domain randomization, worst-case maximization, and regret minimization are proposed in conjunction with other baseline RL approaches, the usefulness of the proposed approaches over existing robust reinforcement learning approaches are not evaluated.\n\nRegret minimization vs proposed approach: In Section 5.1, the authors say \"The regret objective produces a distribution of both feasible and infeasible goals where non-lava goals are the majority, however this distribution is not an adversarial robust NE with respect to the set of feasible goals, so worst-case performance with the regret objective is still not optimal.\" It is understandable that the authors want to say that the optimal min-max regret solution is not the optimal min-max in the feasible environment. However, it is not clear why this is the case in this task. Please provide more explanation in relation with Theorem 1 or 2 of PAIRED paper. I feel that it is the matter of the magnitude of the reward. I guess min-max regret and the proposed formulation is the same if the penalty of lava is greater. At the same time, I understand that it is not always the case that there is a clear notion of success and we would like to have a good generalization ability in the set of environments where the optimal utility is sufficiently large. More explanation would be nice.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject