Robust Option Learning for Adversarial Generalization
Authors: Kishor Jothimurugan
Steve Hsu
Osbert Bastani
Rajeev Alur
Venue: NeurIPS 2022 Submitted (2022)
Keywords: Reinforcement Learning
Options
Generalization

Abstract: Compositional reinforcement learning is a promising approach for training policies to perform complex long-horizon tasks. Typically, a high-level task is decomposed into a sequence of subtasks and a separate policy is trained to perform each subtask. In this paper, we focus on the problem of training subtask policies in a way that they can be used to perform any task; here, a task is given by a sequence of subtasks. We aim to maximize the worst-case performance over all tasks as opposed to the average-case performance. We formulate the problem as a two agent zero-sum game in which the adversary picks the sequence of subtasks. We propose two RL algorithms to solve this game: one is an adaptation of existing multi-agent RL algorithms to our setting and the other is an asynchronous version which enables parallel training of subtask policies. We evaluate our approach on two multi-task environments with continuous states and actions and demonstrate that our algorithms outperform state-of-the-art baselines.

Robust Option Learning for Adversarial Generalization
Anonymous Author(s)
Affiliation
Address
email
Abstract
Compositional reinforcement learning promising approach for training poli- cies to perform complex long-horieg monghorizonn tasks. Typically, high-level task is decom- posed into a sequence of subtasks and a separate policy is trained to perform each subtask. In this paper, way that they can be used to perform any task; here, task is given by sequence of subtasks. We aim to maximize the worst-case performance over all tasks opposed to the average-case performance. We formulate the problem as a two agent zero-sum game in which the adversary picks the sequence of subtasks. We propose two RL algorithms to solve this game: one is an adaptation of existing multi-agent RI. algorithms which enables parallel training of subtask policies. We evaluate our approach on two multi-task environments with continuous states and actions and demonstrate that our algorithms outperform Mate-of-the-ar baselines.
3
focus the problem of training subtask policies in
5
7
setting and the other is asynchronous version
11
12
13
Introduction
14
15
Reinforcement learning (RL) has proven be a promising strategy for solving complex control tasks such as walking [13]. autonomous driving [17]. and dexterous manipulation [3]. However, key challenge facing the deployment of reinforcement learning in real-world tasks is high sample complexity-to solve any new task requires training a new policy designed to solve that task. One promising solution compositional reinforcement learning, where individual options (or skills) are first trained to solve simple tasks; then, these options can be composed together to solve more [7]. For example, if driving robot learns how to make left and right turns
16
17
19
21 complex tasks [25] 2 and to drive in straight line, it can then drive along any route composed of these primitives.
23 key challenge facing compositional reinforcement learning the generalizability of the learned
24 options. In particular, options trained under one distribution of tasks may no longer work well if used new task, since the distribution of initial states from which the options are used may shift. An 26 alternate approach is to train the options separately to perform specific subtasks, but options trained
25
27 this way might cause the system reach states from which future subtasks are hard to perform. One
28 can overcome this issue by handcrafting rewards to encourage avoiding such states [17), in which
29 case they generalize well, but this approach relies heavily on human time and expertise.
30 We propose a novel framework that addresses this challenges by formulating the option learning
31 problem an adversarial reinforcement learning problem. At high level, the adversary chooses
32 the task that minimizes the reward achieved by composing the available options. Thus, the goal is
33 learn a set of robust options that perform well across all potential tasks. Then, we provide two
34 algorithms for solving this problem. The first adapts existing ideas for using reinforcement learning
35 to solve Markov games to our setting. Then, the second shows how to leverage the compositional
Submitted to 36th Conference on Neural Information Processing Systems (NeurIPS 2022). Do not distribute.

---

M
11
Right
Left
Straight
Sharp
Sharp Right
example track
Left
(a) F1/10th Car
(b) Segment Types
Figure 1: F1/10th Environment. The entry and exit regions for the right and sharp right segments are shown in green and blue respectively.
36 structure of our problem to learn options in parallel at each step of a value iteration procedure; in some cases, by enabling such parallelism, we can reduce the computational cost of learning.
38 We validate our approach on two benchmarks: (i) rooms environment where a point mass robot
39 must navigate any given sequence of rooms, where the sequence is an arbitrary combination of straight, left, and right turns, and (ii) simulated version of the F1/10th car, where small racing car must navigate any racetrack composed of several different track segments. In both, our empirical results demonstrate that robust options are critical for performing well on wide variety of tasks.
40
41
42
43 In summary, our contributions are: (i) a game theoretic formulation of the compositional reinforce- ment learning problem, (ii) two algorithms for solving this problem, and (iii) an empirical evaluation demonstrating the effectiveness of our approach.
44
45
46
Motivating example. Let consider small scale autonomous racing car shown in Figure | (a).
47
We would like to train a controller that can be used to navigate the car through all tracks constructed using five kinds of segments; the possible segments are shown in Figure (b)) along with an example track. The state of the car is vector (ar, y. v, 0) where (x, y) is its position on the track relative to
48
49
50 the current segment, v its current speed and 0 is the heading angle. An action is a pair (a.w) E R
51 where is the throttle input and w is the steering angle. In this environment, completing each
52 segment is considered subtask and task corresponds to completing a sequence of segments-
53 e.g., straight >> right -> left -> sharp-right, Upon completion of subtask, the car enters applied the car's state which is now relative the 55 new segment. The goal here is to learn one option for each subtask such that the agent can perform
54 the next segment and
56 any task using these options.
If one trains the options independently with the only goal of reaching the end of each segment 57
58 (e.g., using distance-bar based rewards), it might (and does) happen that the car reaches the end of a segment in a state that was not part of the initial states used to train the policy corresponding to the next subtask. Therefore, one should make sure that the initial state distribution used during training includes such states as well-either manually or using dataset aggregation (38). Furthermore, it is possible that the car reaches a state the exit region of a segment from which it is challenging to complete the next subtask- e.g.. state in which the car is close to and facing towards wall. Our algorithm identifies during training that, in order to perform future subtasks, is better reach the end of segment in configuration where the car is facing straight relative to the next segment. As demonstrated in our experiments, this leads to robust options and improved sample efficiency.
59
60
61
62
63
64
86
Related work. The options framework 41 is commonly used to model subtask policies as tem- 5 43), options are trained along with a
88
porally extended actions. hierarchical RL 33 high-level controller that chooses the sequence of options to execute in order complete specific high-level task. There is also work on discovering options-e, entropy maximization [10), semi-supervised RL [12). skill chaining (20), expectation maximiza-
31
22
70
using intrinsic motivation (30),
72 tion g and subgoal identification [40]. There has also been a lot research on planning using
learned options
18
42 21)
74 There has been some work on RL for zero-shot generalization 44 33
; however, in prior work, the learning objective is to maximize average performance with respect to a fixed distribution
39
23
76 over tasks as opposed the worst-case. Some hierarchical RI algorithms have also been shown to enable few-shot generalization [18 to unseen tasks. Most closely related to our work is the work

---

78
on compositional RL in the multi-task setting [77 in which the subtask policies are trained using standard RL algorithms in a naive way without guarantees regarding worst-case performance.
79
80 There has also been work on skill composition using transition policies [25]; this method assumes that the subtask policies fixed and learns one transition policy per subtask which takes the system from an end state to a "favourable" initial state for the subtask. However, poorly trained subtask policies can lead to situations in which it is not possible achieve such transitions. In contrast, our
81
82
83
84 approach trains subtask policies which compose well without requiring additional transition policies. recent paper [4 proposes a framework for training subtask policies with the aim of composing 85
86
them to perform a complex long-hone long-horizon task. However, their approach assumes that the high-level task fixed and the options are trained to maximize the performance with respect
87
specific task.
88
There has been a lot of research on multi-agent RI algorithms 29
including algorithms for two-agent zero-sum games 66 45 27. In this paper, we utilize the specific structure
15
16
28
35
36 2
89
90
of our game to obtain a simple algorithm that neither requires solving matrix games nor trains a separate policy for the adversary. Furthermore, we show that we can obtain an asynchronous RL algorithm which enables learning options in parallel.
91
92
2
Problem Formulation
93
94 A multi-task Markov decision process (MDP) is a tuple M (S.. 00). where
95 S are the states, A are the actions, P(s' s, a) E [0, 1] is the probability of transitioning from s to
96 on action a, 17 is the initial state distribution, and y E (0, 1) is the discount factor. Furthermore, Σ is set of subtasks and for each subtask a E Σ, R S x A + RR is a reward function"] F₂ S S set of final states where the subtask is considered completed and T. Fo x S - (0. 1] the 99 jump probability function; upon reaching state 8 in Fa the system jumps to a new state s with probability (s(s') 8). For the sake of clarity, we assume that (s 8) = for any s' with o' E Far for some a'. Finally, do E Σ is the initial subtask which has to be completed firs A
97
98
100
102
multi-task MDP can be viewed as discrete time variant of a hybrid automaton model [17]
In the case of our motivating example, the set of subtasks is given by 103
Σ = (left, right, straight, sharp-left, sharp-right}
104
with Fo denoting the exit region of the segment corresponding to subtask a. We use the jump transitions T to model the change-of-coordinad
105
performed upon reaching an exit region. The reward function R2 for a subtask is given by (s, a, s')
-Ils' co co112 B 1(s E Fo) where Co is the center of the exit region and the subtask completion bonus B is a positive constant.
106
107
A task is defined be an infinite sequence of subtasks tasks. For any task E T, -[i] denotes the 7 subtask a; in r. In our setting, the task is chosen by the environment nondeterministically, Given task r, configuration of the environment is pair (s, i) E S x Zzo with & Frie denoting that the system is in state s and the current subtask is T [i] The initial distribution over configurations A S x Zzo - [0, 1] is given by A(s, d) = 77100 (8) i = and otherwise. The probability of transitioning from (8, to (8', on an action is
000 ..... and T denotes the set of all
Everyy P(s" | ( 8") if=--t1 P(s' |s,a)
Pr((s',5) (s)
if j -= i otherwise.
Intuitively, the system transitions to the next subtask the current subtask is completed and stays in the current subtask otherwise. A (deterministic) policy is function π S + A, where = Ts)) the action take in state s. Our goal is to learn one policy Tr for each subtask a such that the discounted reward over the worst-case task 7 is maximized. Formally, given set of policies II { a E Σ} and task T, we can define a Markov chain over configurations with initial distribution A and transition probabilities given by Pn (n((ss)) (s, i)) = Pr((s.)) j') (s, i), [ (+(((s)) We denote
We can also have R2 : S A S + R depending on the next state but we omit it for clarity of presentation.
Thii assumption can be removed by adding fictitious copy of F. to S for each a Σ.
A When there is no fixed initial subtask, we can add fictitious initial subtask.
AA finite sequence be appended with an infinite sequence of a fictitious subtask with zero reward.
3

---

by T the distribution over infinite sequences of configurations p (sa. (s1, (1)....)..... generated
by and II. Then, we define the objective function as
(
inf E. TET D
Σ >d Br [o)o t=0
108 These definitions can be naturally extended stochastic policies as well. In our motivating ex-
109 ample, choosing a large enough completion bonus B guarantees the discounted reward to be higher for runs in which more subtasks are completed. Our aim is compute a set of policies
110
II E arg maxn J[I.. Each subtask policy defines an option [4]] Os = (To, 5a) where
112 Io = S F and 3, (8) = I(s Fz). Here, the choice of which option to trigger made by the
environment rather than the agent.
3 Reduction to Stagewise Markov Games
The problem statement naturally leads game theoretic view in which the environment is the adversary. We can formally reduce the problem to two-agent zero-sum Markov game (S, A1. A,, R. 7, n)) where S S x is the set of states, A1 A are the actions of agent 1 (the agent learning the options) and A2 Σ are the actions of agent (the adversary). The transition probability function P: x A x A2 x $ [0, 1] is given by
P(s' s.a1)
if & Fg & a = o' if Fo & o' 02 otherwise.
P((s', (8,0), a1 02)
Ta(s' (8)
| F,) Fa) is the set of states where agent 2 takes actions (causing change of subtask); this makes G a stagewise game. The reward function R : 5 x Ai + R is given by R((s, o), a) = Rs (s, a) if & Fa and 0 otherwise. The discount factor depends on the state and is given by 7(s, o) if & Fg and I otherwise; this is because a change of subtask does not invoke step in M. The initial state distribution îj is given where & S
We observe that the states partitioned into sets S U S2 where S {(s, (() the set of states where agent acts (causing a step in MM) and S2 = {(8,0)
116
118
119
120
by nss, o) ( ou). run of the game sequence p suajansia and af E A,,
121
122
A (deterministic) policy for agent i is function n; S - A, Given policies n and n2 for agents and 2, respectively and state 3 S we denote by D (TT1 TT) the distribution over runs generated by πi and π2 starting at s. Then, the value of a state 5 defined by
1 (3) = E Pp (m)))))
77x)) ( (t5.
1=
k=
We are interested in computing a policy oं maximizing
Jg (g(T)) E [min (T (3)).
123
Given a policy Ti for agent 1, we can construct a policy Ta for any subtask a given by (s(s) = (s, o); we denote by [[Tr) ) the set of subtask policies constructed this way. The following the- orem connects the objective of the game with our multi-task learning objective; all proof' are in
125
126
Appendix
127
Theorem 3.1. For any policy x for agent I in G, we have ((I((TI)) 2 (o(
Therefore, )g(m1) ) is a lower bound the objective ([I((1)) which we seek to maximize. Now, let us define the optimal value of state 3 by V*(3) = max,, ming, VT (8). The following theorem shows that it is possible to construct policy тं that maximizes (g(T1) from the optimal value function V*.
129
131
Theorem 3.2. For any policy oं such that for all (s,0) S..
(i. a) arg max (R((s,))) a) +r Σ P(s' 1 ((, ees
132 we have that x E arg max [g(

---

Algorithm Asynchronous value iteration algorithm for computing optimal subtask policies.
I: function ASYNCVALUE ASYNCYAALUETTRRATIX,,,,
2:
while stopping criterion met do
3:
for Σ do // in parallel
4:
Compute (V)
5:
V ssyno (V)
// using Equation 3
133
3.1
Value Iteration
In this section, we briefly look at value iteration algorithms compute V which later adapt
135
in Section to obtain learning algorithms. Let V = (V Si + R} denote the set of all value
138
functions over Sj. Given a value function V we define its extension to all of $ using
minnoes Eves s)V(s', if Fg (V(s,))
[V](s,0)=
(1)
otherwise.
For state Fg. (V] denotes the worst-case value (according V) with respect to the possible choices of next subtask o'. Now, we consider the Bellman operator F V defined by
138
F(V)(s,
=max {(((s,))))) +7 Σp(s P(s (1.0))11(660))) aEA
e'ES
139
for all (s. S1. Let us denote by V* Js,, the restriction of V to S₁. The following lemma follows straightforwardly giving us our first value iteration procedure.
140
Theorem 3.3. F is contraction mapping with respect the loo norm on V and V 1s, is the unique fixed point of F with lim,-+00 (V) = V* 4s, for all V.
142
143
Next we consider an asynchronous value iteration procedure which allows us to parallelize com- puting subtask policies for different subtasks. Given subtask a and a value function V E V, we define subtask MDP MM which behaves like M until reaching final state transitions to a dead state 1 achieving a reward of [V] (s. Formally, My = A, Ry 7) where Sg = [ with 1 being a special dead state, (s's s, P(s' s, #s Fo and Po Ps s, = 1 (s = =1) otherwise. The reward function is given by R (s) (s,
144
F, after which it
146
148
1 # & F., (s, a) (s) if L # F, and is 0 otherwise. We denote by WV(V) the
optimal value function of the MDP My. We then define the asynchronous value iteration operator
150
aayne V V using
Fazyae (V) = - (s).
We can show that repeated application of Fasyne leads to the optimal value function V of the G. Theorem 3.4. For any V E V. lim Fiayyee
153
V* 4s..
Since each Wo (V) can be computed independently, we can parallelize the computation of Fasyne giving the algorithm in Algorithm We also show that it is not necessary to compute W, exactly. Let V, = (V 5, > } be the set of all value functions over Sz. For fixed V E V, let Fa,v Vo + Vo denote the usual Bellman operator for the MDP My given by
Fo,((V)(s) max { RV (8, +7 Σ Pa(s' a.))
S'ES,
for all V E Vr and s S.. For any V E V and E Σ, we define a corresponding V, V,, using V. (s) = [V] o) if i E S and VL(1) = 0. Then, for any integer m > O and V E V, we can use Fay (V,) as an approximation (V)... Let us define Fm V V using
Fm (v) (V.)
154 Intuitively, Fm corresponds performing m steps of value iteration in each subtask MDP M. (which can be parallelized) starting at Vg. The following theorem guarantees convergence when using Fm instead of Fasyae-
155
156
157
Theorem 3.5. For any V E V and > 0, lim (V) V* 4st-
5

---

Algorithm Robust Option Soft Actor Critic.
Inputs: Learning rates ag. ate, entropy weight and Polyak rate 8.
1: function ROSAC(ak, Qgg (. 8)
2: Initialize parameters |eex, (Wo Joex Joex and (0r (0r)oex
3: Initialize replay buffer for each iteration do
4:
5:
for each episode do
6:
So 1)
7:
To + InitialSubtask for each step do
8:
9:
a, ~ Te., (.1 st) and 8441 ~ P(: s.
10:
+- BU [(,, ae, 8441)}
11:
if 8t+1 E Fo, then
12:
8141 ~ To, 84+1)
13:
++ < Greedy(e, arg min, (8)+1,,);
else
15:
0++1 +
16:
for each gradient step do Sample batch B for o E do
17:
18:
19:
Ve E- the - 0ge V the Lo Vo: B)
20:
0g r 0g ag 0, L (0., B)
21:
barg +- 5 (1 8/
targ
Learning Algorithms
158
159
In this section, we present RL algorithms for solving the game G. We first consider the finite MDP setting for which we obtain modified Q-learning algorithm with convergence guarantee. We then present two algorithms based on Soft Actor Critic (SAC) for the continuous state setting.
161
182
4.1
Finite MDP
163
Assuming finite states and actions, we can obtain a Q-learning variant for solving 9 which we call Robust Option Q-learning. We assume that jump transitions are known to the learner; this is usu- ally the case since jump transitions are used to model subtask transitions and laange-f--coodiine within the controller. However, we believe that the algorithm can easily extended to the scenario where T is unknown.
184
186
We maintain function Q S1 4 R with Q(s, ((s.o,a) denoting Q((s, o), a). The corre- sponding value function Vo is defined using VQ(s, = max.ed Q(s, a) and is extended to all of S as [Vo]- Note that, given a Q-function, the extended value function [Vo] can be com- puted exactly. Robust Option Q-learning an iterative process-in each iteration takes step
188
189
170
((s, o), a, 02, (s',a)) in G with (8,a) E S and updates the Q-function using
172
Qs+1(s,0,01) <- (1 Qs)),(s, a, 01) + or (R(((()))) + +VV.e. .)).
(4)
where Qe is the Q-function in iteration / and [Vo.] is the corresponding extended value function.
174
Under standard assumptions on the learning rates ate similar Q-learning, we can show that Robust Option Q-learning converges to the optimal Q-function almost surely, Here, the optimal Q-function defined by Q* (s, o, a) R((s, o), + + 7. P(s' V* .)) for all (s,o) E ST. Let arls, o, a) denote the learning rate used in iteration t if Qe,s, a) updated and 0 otherwise. Then,
175
178
177
178
we have the following theorem.
179
oo and E, as a) < oo for all (8, E S₁ and a E A, then lim Qi Q" with probability 1.
Theorem 4.1. E,,,,
an(s,,

---

4.2
Continuous States and Actions
182
In the case of continuous states and actions, we can adapt any Q-function based RL algorithm such
183
as Deep Deterministic Policy Gradients (DDPG) [26] or Soft Actor Critic (SAC) [4] to our setting.
184
Here we present an SAC based algorithm that we call Robust Option SAC (RoSAC) which is outlined
185
in Algorithm 2
This algorithm, like SAC, adds entropy bonus to the reward function to improve
exploration.
186
We maintain two Q-functions for each subtask a, Qu S + IR parameterized by t. and a target function Quaes parameterized by vb We also maintain stochastic subtask policy To, : S + D(A) (A) for each subtask where ((A) denotes the set of distributions over A. Given a step (s,a, (,)s')
in M and subtask with Fo, we define the target value by
Yo yo (,) Ro + ~[V](s'
where the value [VI('', o) is estimated using V(s', = ( - 3 log ne, (ā s') with à To, s') if s' & F.. s' E Fa, we estimate [V][s'', using V(), ming'ey V(s", o') where Vi8", o') Q quan (s"
log (a | s") with _o 8") and 8" (- s*). Now, given batch B = {(s,a, a, s')} of steps in M we update v using one step of gradient descent corresponding to the loss
Lo B)
1
Σ
(8. (8,
B
(o.
and the subtask policy me, is updated using the loss
L (0.: B)
Σ
Ba-ter (1)) [v. Blog Te, (ā [s]]-
(8.0,"")EB
The gradient Ce (0o, B) can be estimated using the reparametrization trick if Te, (- | s) is Gaussian distribution whose parameters are differentiable w.r.t. 0g. We use Polyak averaging to update the target Q-networks (Quees E E}.
188
190
Note that we do not train a separate policy for the adversary. During exploration, we use the e- greedy strategy to select subtasks. We first estimate the "worst" subtask for a state using arg min, V (ss o)) where (s, o) is estimated before. Then the function Greedy(s, ō, Σ) chooses with probability - E and picks a subtask uniformly at random from Σ with probability s.
191
192
193
194
Asynchronous ROSAC. We can also obtain asynchronous version of the above algorithm which
195
lets us train subtask policies in paraliel. Asynchronous Robust Option SAC (AROSAC) is outlined in Algorithm Here we use one replay buffer for each subtask. We maintain an initial state distribution 7 over S to be used for training every subtask policy {Ta Joez. î is represented using finite set of states D from which a state is sampled uniformly at random. The value function V S x Σ + R
196
197
198
is estimated as before. To be specific, in each iteration, an estimate of any value (s. r) is obtained on the fly using the Q-functions and the subtask policies from the previous iteration.
199
200
201
The SAC subroutine runs the standard Soft Actor Critic algorithm for N iterations on the subtask
MDP M (defined in Section 3 with initial state distribution ýj (defaults to if D = -0. It returns the updated parameters along with states X, visited during exploration with X, S Fo. The states
203
in X. are used to update the initial state distribution for the next iteration following the Dataset
205
A Aggregation principle (38).
Experiments
206
We evaluate our algorithms ROSAC and AROSAC on two multi-task environments; rooms environ- ment and an F1/10th racing car environment [11
208
Note that it is possible to obtain samples from My as long can one can obtain samples from M and membership in Fo can be decided.
7

---

Algorithm 3 Asynchronous Robust Option Soft Actor Critic.
Inputs: Learning rates a, entropy weight 3, Polyak rate and number of SAC iterations N.
1: function AROSAC(a, B, 8. .)
2: Initialize parameters V (1h loEs, Vatarg { JoEx and θ {0a loes
3: Initialize replay buffers {Bo Joen and Initialize D 0
4:
for each iteration do
5: V <- OATAINVALURESTTMATTON (V)
6: for do in parallel
7:
Vo, barg 0o, Xo + SAC(M V. vo targ a,,
8:
for a E E do
9:
for & E X, do
10:
& ~ Tol: 8) and D + (8)
209
Rooms environment. We consider the environment shown in Fig- which depicts room with walls and exits. Initially the robot is placed in the green triangle. The L-shaped obstacles indicate walls within the room that the robot cannot cross. A state of the system is a position (z, y) E 22 and an action is a pair (u, 6) where v is the speed and / the heading angle to follow during the next time-step. There are three exits: left (blue), right (yellow) and up (grey) reaching each of which is subtask. Upon reaching an exit, the robot enters another identical room where the exit is identified (via change-of-coordinate
210
211
212
213
214
+
215
216
218
with the bottom entry region of the current room. A task is a sequence of directions-e.g, left + right up right indicating that the robot should reach the left exit followed by the right exit in the
Figure ment
Rooms environ-
220
subsequent room and so Although the dynamics are simple, the obstacles make learning challenging in the adversarial setting.
222
F1/10th environment. This is the environment in the motivating example. A publicly available simulator [I] of the F1/10th car is used for training and testing. The policies use the LiDAR measurements from the car as input (as opposed to the state) and we assume that the controller can detect the completion of each segment; as shown in prior work [17), one can train a separate neural network to predict subtask completion.
224
226
228 Baselines.
We compare our approach to three baselines. The baseline NAIVE trains one controller for each subtask with the only aim of completing the subtask, similar to [17], using manually designed initial state distribution. DAGGER is similar approach which, instead of using a manually designed initial state distribution for training, infers the initial state distribution using the Dataset Agg Aggregation principle (38]. The MADDPG baseline solves the game G using the multi-agent RL
230
231
232
233
algorithm proposed
29 for solving concurrent Markov games with continuous states and actions.
234 Evaluation.
We evaluate the performance of these algorithms against two adversaries. One adver- sary is the random adversary which picks the next subtask uniformly at random from the set of all subtasks. The other adversary estimates the worst sequence of subtasks for a given set of options using Monte Carlo Tree Search (MCTS) 1 The MCTS adversary is trained by assigning a reward of if it selects a subtask which the corresponding policy is unable to complete within a fixed time budget and a reward of otherwise. For the Rooms environment, we consider subtask sequences of
235
236
237
238
239
240
length atmost 5 whereas for the F1/10th environment, we consider sequences of subtasks of length most 20. We evaluate both the average number of subtasks completed as well as the probability of completing the set maximum number of subtasks.
241
242
243
The plots for the rooms environment are shown in Figure 3 and plots for the F1/10th environment are shown in Figure We can observe that ROSAC is able outperform other approaches and learn robust options. In the rooms environment, AROSAC achieves similar performace albeit requiring more samples; however, it has the added benefit of being parallelizable. In the F1/10th environment, it performs similar to the other baselines. DAGGER and NAIVE baselines are unable to learn policies that can be used to perform long sequences of subtasks; this is mostly due to the fact
Results.
244
245
247
8

---

NAVE
DRUSEH
AROGAC
ROSAC
0.8
0.6
0.4
02
(a) Number of subtaks completed against random adversary
(b) Number of subtasks completed against MCTS adversary
(c) Success probability against MCTS adversary
Figure 3: Plots for the Rooms environment. z-axis denoted the number of sample steps and y-axis denoted the either the average number of subtasks completed the probability of completing subtasks. Results are averaged over runs. Error bars indicate standard deviation.
25
12
20
DAAGER AROSAC
20-
AROSAC
DAGGER
10-
DAGGER
AROSAC
15
NADOPS
ROSAC
15
ROSAC
NADOPG
0.8
MADOTS
ROSMC
10.
5-
-0.
-1.0
-0.5
0.5
1.0
13
2.0
-1.0
-0.5
0.0
0.5
1.0
1.5
2.0
-1.0
-0.5
0.0
0.5
1.0
1.5
2.0
x10"
x10
x10'
(a) Number of subtaks completed against random adversary
(b) Number of subtasks completed against MCTS adversary
(c) Success probability against MCTS adversary
Figure 4: Plots for the F1/10th environment. r-axis denoted the number of sample steps and y-axis denoted the either the average number of subtasks completed the probability of completing 20 subtasks. Results are averaged over runs. Error bars indicate + standard deviation.
that they learn options that cause the system reach states from which future subtasks are difficult 250 to perform-e.g e.g. S., in the rooms environment,
agent sometimes reaches the left half of the exits from where it is difficult to reach the right exit in the subsequent room. Although MADDPG uses the same reduction to two-player games as ROSAC, it igmores all the structure in the game and treats it as generic Markov game. As result, it learns separate NN policy for each player which leads the
252
issue of unstable training, primarily due to the non-stationary nature the environment observed 254
by either agent. As shown in the plots, this leads to poor performance when applied to the problem of learning robust options.
256
6
Conclusions
257
We have proposed framework for training robust options which be used to perform arbitrary sequences of subtasks. In our framework, we first reduce the problem to two-agent zero-sum stagewise Markov game which has unique structure. We utilized this structure design al- gorithms, namely ROSAC and AROSAC, and demonstrated that they outperform existing approaches for training options with respect multi-task performance. One potential limitation of our approach is that the set of subtasks is fixed and has to be provided by the user. An interesting direction for future work address this limitation by combining our approach with option discovery methods.
259
280
281
282
263
264
285
Societal impacts. Our work seeks to improve reinforcement learning for complex long-horizon tasks. Any progress in this direction would enable robotics applications both with positive impact- e.g. flexible and general-purpose manufacturing robotics, robots for achieving agricultural tasks, and robots that can be used to perform household chores with negative or controversial impact-e.g e.g. military applications. These issues are inherent in all work seeking to improve the abilities of robots.
286
257
269

---

References
271
272
[1]
David Abel, Nathan Umbanhowar, Khimya Khetarpal, Dilip Arumugam, Doina Precup, and Michael L. Littmmn. Value preserving state-action abstractions. In Proceedings of the Interna- tional Conference on Artificial Intelligence and Statistics, 2020.
274
275
[2] Natalia Akchurina. Multi-agent reinforcement learning algorithm with variable optimistic- pessimistic criterion. In ECAI, volume 178, pages 433-437, 2008.
276
277
[3] Igge Akkaya, Marcin Andrychowicz, Maciek Chociej, Mateusz Litwin, Bob McGrew, Arthur Petron, Alex Paino, Mauthias Plappert, Glenn Powell, Raphael Ribas, et al. Solving rubik's cube with robot hand. arXiv preprini arXiv: 13, 2019.
278
279
280
[4] Jacob Andreas, Dan Klein, and Sergey Levine. Modular multitask reinforcement learning with policy sketches In International Conference on Machine Learning, pages 166-175. PMLR, 2017.
281
282
283
[5] Pierre-Luc Bacon, Jean Harb, and Doina Precup. The option-crite architecture. In Thirty-First AAAI Conference on Artificial Intelligence, 2017. 2017.
284
285
66] Yu Bai and Chi Jin. Provable self-play algorithms for competitive reinforcement learning. In Proceedings of the 37th International Conference on Machine Learning, 2020.
286
287
[7] D. P. Bertsekas and J. Tsitsiklis. Neuro-dynanic programming. Athena Scientific, Belmont, MA, 1996.
[8]
Christian Daniel, Herke Van Hoof, Jan Peters, and Gerhard Neumann. Probabilistic inference for determining options in reinforcement learning. Machine Learning, 104(2)3337337,, 2016.
290
291
99
Thomas G Dietterich. State abstraction in maxq hierarchical reinforcement learning. In Ad- vances in Neural Information Processing Systems, pages 994-1000, 2000.
292
110] Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without a reward function. In ICLR, 2018.
294
295
[11] F1/10 Autonomous Racing Competition. http://flttenthoorg
296
[12] Chelsea Finn, Tianhe Yu, Justin Fu, Pieter Abbeel, and Sergey Levine. Generalizing skills with semi-supervised reinforcement learning. In ICLR, 2017.
297
296
[13] Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In International Conference on Machine Learning, pages 1587-1596, 2018.
299
300
301
[14] Tuomas Haarnoja, Aurick Zhou, Picter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, pages 1861-1870. PMLR, 2018
303
304
[15] Junling Hu and Michael P Wellman. Nash g-learning for general-sum stochastic games. Jour- nal of machine learning research, ((Nov):0099111 1069, 2003.
[16] Junling Hu, Michael P Wellman, et al. Multiagent reinforcement learning: theoretical frame- work and an algorithm. In ICML, volume 98, pages 242-250. Citeseer, 1998.
307
308
[17] Radoslav Ivanov, Kishor Jothimurugan, Steve Hsu, Shaan Vaidya, Rajeev Alur, and Osbert Bastani. Compositional learning and verification of neural network controllers. ACM Trans- actions on Embedded Computing Systems (TECS), 20(5s):1-26, 2021.
309
310
311
[18] Kishor Jothimurugan, Osbert Bastani, and Rajeev Alur. Abstract value iteration for hierarchi- cal reinforcement learning. In Arindam Banerjee and Kenji Fukumizu, editors, Proceedings of The 24th International Conference on Artificial Intelligence and Statistics, volume 130 of Proceedings of Machine Learning Research, pages 1162-1170. PMLR, 13 Apr 2021.
312
313
314
315
[19] Levente Koesis and Csaba Szepesvári. Bandit based monte-carlo planning. In European con- ference machine learning, pages 282-293, Springer, 2006.
10

---

317
[20] George Konidaris and Andrew G Barto. Skill discovery in continuous reinforcement learning domains using skill chaining. In Advances in neural information processing systems, pages 1015-1023, 2009.
318
320
[21] George Konidaris, Lestie Kaelbling, and Tomas Lozano-Perez. Constructing symbolic rep- resentations for high-level planning. In Twenty-Eighth AAAI Conference on Artificial Intelli- gence, 2014.
321
322
323
[22] Tejas Kulkarni, Karthik Narasimhan, Ardavan Saeedi, and Josh Tenenbaum. Hierarchical deep reinforcement learning: Integrating temporal abstraction and intrinsic motivation. In Advances in neural information processing systems, pages 3675-3683, 2016.
325
328
[23] YengLing Kuo, Boris Katz, and Andrei Barbu. Encoding formulas as deep networks: Re- inforcement learning for zero-shot execution of Itl formulas. 2020 IEEE/RS. International Conference on Intelligent Robots and Systems (IROS), pages 5604-5610, 2020.
327
328
[24]
Youngwoon Lee, Joseph J Lim, Anima Anandkumar, and Yuke Zhu. Adversarial skill chain- ing for long-horizon robot manipulation via terminal state regularization. arXiv preprint 1IXiv2211.007999, 2021.
330
331
332
[25] Youngwoon Lee, Shao-Hua Sun, Sriram Somasundaram, Edward S Hu, and Joseph Lim. Composing complex skills by learning transition policies. In International Conference on Learning Representations, 2018.
334
325
[26] Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In ICLR, 2016.
336
337
[27] Michael L Littman. Markov games as framework for multi-agent reinforcement learning. In Machine learning proceedings 1994, pages 157-163. Elsevier, 1994.
339
340
[28] Michael L Littman. Friend-or-foe q-learning in general-sum games. In ICML, volume 1, pages 322-328, 2001.
341
342
[29] Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Pieter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed cooperative-compettivy environments. In Proceedings of the 31 Inter- national Conference on Neural Information Processing Systems, pages 6382-6393, 2017.
343
344
[30] Marios C Machado, Marc G Bellemare, and Michael Bowling. A laplacian framework for op- tion discovery in reinforcement learning. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pages 2295-2304. JMLR. org, 2017.
346
348
[31] Ofir Nachum, Shixiang Gu, Honglak Lee, and Sergey Levine. Near-optimal representation learning for hierarchical reinforcement learning. In ICLR, 2019.
349
350
[32] Ofir Nachum, Shixiang Shane Gu, Honglak Lee, and Sergey Levine. Data-efficient hierarchical reinforcement learning. In Advances in Neural Information Processing Systems, pages 3303-
351
352
3313, 2018
353
[33] Junhyuk Oh, Satinder Singh, Honglak Lee, and Pushmeet Kohli. Zero-shot task generaliza- tion with multi-task deep reinforcement learning. In International Conference Machine Learning, pages 2661-2670. PMLR, 2017.
355
358
[34] Stephen David Patek. Stochastic and shortest path games: theory and algorithms. PhD thesis, Massachusetts Institute of Technology, 1997.
357
358
[35] Julien Perolat, Florian Strub, Bilal Piot, and Olivier Pietquin. Learning Nash Equilibrium for General-Sum Markov Games from Batch Data. In Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, 2017.
359
360
[36] HI Prasad, Prashanth LA, and Shalabh Bhatnagar. Two-timescale algorithms for learning nash equilibria in general-sum stochastic games. Proceedings of the 2015 International Conference on Autonomous Agents and Multiagent Systems, pages 1371-1379, 2015.
362
11

---

384
[37] Doina Precup, Richard S Sutton, and Satinder Singh. Theoretical results on reinforcement learning with temporally abstract options. In European conference on machine learning, pages 382-393. Springer, 1998.
365
366
[38] Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the fourteenth interna- tional conference on artificial intelligence and statistics, pages 627-635, 2011.
368
[39] Sungryull Sohn, Junhyuk Oh, and Honglak Lee. Hierarchical reinforcement learning for zero- shot generalization with subtask dependencies. Advances in Neural Information Processing Systems, 31, 2018.
371
372
373
[40] Martin Stolle and Doina Precup. Learning options in reinforcement learning. In International Symposium on abstraction, reformulation, and approximation, pages 212-223. Springer, 2002.
374
375
[41] Richard S Sutton, Doina Precup, and Satinder Singh. Between mdps and semi-mdps: A frame- work for temporal abstraction in reinforcement learning. Artificial intelligence, 112(1-2):181
376
377
211, 1999.
[42] Georgios Theocharous and Leslie P Kaelbling. Approximate planning in pomdps with macro- actions. In Advances in Neural Information Processing Systems, pages 775-782, 2004.
379
380
[43] Saket Tiwari and Philip S Thomas. Natural option critic. In Proceedings of the AAAI Confer- ence on Artificial Intelligence, volume 33, pages 5175-5182, 2019.
381
382
[44] Pashootan Vaezipoor, Andrew C Li, Rodrigo A Toro Icarte, and Sheila A. Mcilraith. Ltlzactoo: Generalizing Itl instructions for multi-task rl. In Marina Meila and Tong Zhang, editors, Pro- ceedings of the 38th International Conference on Machine Learning. volume 139 of Proceed- ings of Machine Learning Research, pages 10497-10508, PMLR, 18-24 Jul 2021.
383
304
385
386
[45]
Chen-Yu Wei, Yi-Te Hong, and Chi-Jen Lu. Online reinforcement learning in stochastic games. In Proceedings of the 31 International Conference on Neural Information Processing Sys- tems, pages 4994-5004, 2017
387
Checklist
389
1. For all authors...
391
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
392
contributions and scope? [Yes]
393
(b) Did you describe the limitations of your work? [Yes]
384
(c) Did you discuss any potential negative societal impacts of your work? [Yes]
395
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
396
2. If you including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
399
(b) Did you include complete proofs of all theoretical results? [Yes] In the supplement.
3. If you ran experiments..
401
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL)? [Yes]
402
403
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes]
406
407
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
408
4. If you using existing assets (e.g., code, data, models) curating/releasing new assets...
410
(a) If your work uses existing assets, did you cite the creators? [Yes]
12

---

(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or URL? [No]
418
414
(d) Did you discuss whether and how consent was obtained from people whose data you' re using/curating? [N/A]
416
(e) Did you discuss whether the data you are using/curating contains personally identifi-
able information or offensive content? [N/A]
418
5. If you used crowdsourcing or conducted research with human subjects...
419
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
422
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
424
13


Reviews:
[{"review_id": "gIOEmahhYES", "reviewer": "Reviewer_5XiN", "summary": "The paper aims to learn robust policies for solving long-horizon tasks that are composed of sequences of subtasks. To accomplish this, the problem is framed as a zero-sum, two-player game: the agent aims to maximize its worst-case performance against an adversary who chooses the sequence of subtasks. Two algorithms are introduced to solve this multi-agent problem which demonstrate superior performance compared to a number of baselines. ", "questions": "- Can you elaborate on the key differences between your work and the previously mentioned areas of unsupervised environment design and robust (minimax) RL? \n- Can you elaborate on other real-world settings where your framework would be beneficial? \n- Why does the MADDPG baseline perform so poorly, even when compared to NAIVE? I found this surprising, since even a random adversary should result in options of decent quality. ", "limitations": "As mentioned above, I believe the authors should discuss any assumptions of control over the environment generation process (if my conclusion that this is a limitation is indeed correct). This would limit the types of tasks and settings where such a method would be applicable. ", "rating": 3, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Originality**: The authors present an interesting framework for training robust options, but this framework appears closely related to previous work where an adversary designs environments which challenge the current agent (e.g. [1,2]). The main difference here seems to be that the adversary can act many times within the *same episode* to affect segments of the environment that have not yet been generated. More generally, there is a plethora of work aimed at learning robust or safe policies (not necessarily options) using a worst-case minimax objective (e.g. [3,4]). I think these areas of research should be discussed in the related work, as I'm currently not convinced that this work differs in a significant way.\n\n**Quality**: The proposed algorithms are well-motivated, exploiting the structure of the problem to improve the learning of the adversary. They are further justified via rigorous convergence guarantees. The experiments are conducted on complex environments and demonstrate faster learning compared to a number of baselines, but I have some concerns. \n\n- The learning curves are only reported over a small number of frames (500k) and this may not show the full picture. I'm interested in seeing whether ROSAC converges to a better solution than NAIVE/DAGGER under adversarial subtask selection. Also, I'm curious if there is any performance trade-off to training adversarially when you evaluate on uniformly random tasks.\n- I think there needs to be a baseline which simply performs flat RL (preferably SAC, for a fair comparison) on the full task. If I'm understanding correctly, this is different from the NAIVE/DAGGER baselines, which instead independently try to solve each \"segment\" as quickly as possible. \n\n**Clarity**: Overall the prose is very well-written but the math can be hard to follow due to the amount of notation. \nThe following parts could also be improved in terms of clarity:\n- I think the start of Section 3 could better emphasize the properties of the problem you will be exploiting to outperform general methods for solving games (e.g. MADDPG). \n- It's not clear what the NAIVE and DAGGER baselines are just from the descriptions in the experiments section.  \n- It's not stated what the error bars in Figs 3 and 4 are reporting.\n\n**Significance**: Learning options that are robust and that generalize under various conditions is an important problem for RL. However, after reading the paper, I (unfortunately) remain skeptical of the impact of this particular work. First, there is an assumption that any sequence of options corresponds to a meaningful task, which is only true in a limited number of settings. Even in the racetrack example, many sequences of turns don't seem to correspond to configurations that are physically possible. The environment must also dynamically change based on the selected sequence of subtasks, requiring a great degree of control in the environment generation process. Again, I'm having trouble thinking of other settings where such a framework would be applicable. \n\n[1]: Dennis, Michael, et al. \"Emergent complexity and zero-shot transfer via unsupervised environment design.\" Advances in Neural Information Processing Systems 33 (2020): 13049-13061.\n[2]: Gur, Izzeddin, et al. \"Environment generation for zero-shot compositional reinforcement learning.\" Advances in Neural Information Processing Systems 34 (2021): 4157-4169.\n[3]: Pinto, Lerrel, et al. \"Robust adversarial reinforcement learning.\" International Conference on Machine Learning. PMLR, 2017.\n[4]: Campero, Andres, et al. \"Learning with AMIGo: Adversarially Motivated Intrinsic Goals.\" International Conference on Learning Representations. 2020.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "czZx1z9yt0u", "reviewer": "Reviewer_KUYx", "summary": "The paper proposes an approach to learning option policies such that they can be composed sequentially, regardless of the next subtask/option to be performed. The main idea is to reframe the problem as a two-player game, where an adversarial environment arranges the order of the subtasks, and the agent learns one policy for each subtask to maximise the sequential completion of as many as possible in the worst case. The convergence of value iteration to an optimal policy in this setting is proven, while two algorithms based on Q-learning and soft actor-critic are provided. Results in two continuous state/action domains demonstrate that the proposed algorithms significantly outperform the baselines in practice.", "questions": "1. The main aim of this paper is to learn robust option policies. As I mentioned in the previous section, much of the difficulty in option discovery is in identifying the subgoals options should reach in the first place. The formulation here sidesteps this problem by simply specifying what those should be. Nonetheless, my larger concern is how this fits into the existing literature. In particular, the line of work revolving around skill chaining seems to tackle the exact problem here: learn options such that they can be sequentially chained together reliably. These works go further though, in that they also do not assume that the subtasks are given. Quoting from [2]: \"The chainability of the discovered skills, i.e, the property that successful execution of one permits the execution of another, allows the agent to build a graph that is suitable for deterministic planning\". If my understanding is correct, then that line of work is tackling the same problem here and more. In that case, it's unclear how this paper improves upon that line of work. One argument potentially is that these options are reusable, but that is only because of the change of coordinates/relativisation of state variables, which is defined by the system itself. I'm thus unclear how this work advances over the skill chaining work, but I would be interested to hear the authors' thoughts on this. Even if there is a difference, I think that one of the skill chaining approaches should certainly be implemented as a baseline comparison. \n\n2. On the results, one thing I found very confusing is the performance of everything when the adversary is MCTS or random. It seems like a random adversary does just as well as MCTS, so does that not suggest that there is no need to phrase the problem as a 2-player game? The random adversary doesn't seek to minimise anything, so why frame it as a game if it does just as well as one that does? Why not simply always randomise over the subtask order and have the agent train on the randomised sequences?\n\n3. The formulation of the multi-task MDP relies on jump probabilities, change of coordinates, etc, which I've never encountered prior. The paper cites [17], but I'm curious to understand whether this is a common formulation of such problems? It seems to be injecting an awful lot of knowledge and domain expertise into the system and essentially provides the agent with transferrable skills. \n\n4. I found the following sentence in the abstract confusing: \"We aim to maximize the worst-case performance...\" This put me in mind of other work [4-7] where the aim is to learn a policy or set of policies that are useful for subsequent tasks drawn from some distribution (as opposed to a given set of tasks that must be sequentially executed). I would recommend this be reworded.\n\n5. What does the shading on the results graphs represent? Is it variance?\n\n[1] Bagaria, Akhil, and George Konidaris. \"Option discovery using deep skill chaining.\" International Conference on Learning Representations. 2019.\n\n[2] Bagaria, Akhil, Jason K. Senthil, and George Konidaris. \"Skill discovery for exploration and planning using deep skill graphs.\" International Conference on Machine Learning. PMLR, 2021.\n\n[3] Bagaria, Akhil, et al. \"Robustly Learning Composable Options in Deep Reinforcement Learning.\" IJCAI. 2021. \n\n[4] Alver, Safa, and Doina Precup. \"Constructing a Good Behavior Basis for Transfer using Generalized Policy Updates.\" International Conference on Learning Representations. 2022.\n\n[5] Nemecek, Mark W., and Ron Parr. \"Policy caches with successor features.\" International Conference on Machine Learning. PMLR, 2021.\n\n[6] Jinnai, Yuu, et al. \"Discovering options for exploration by minimizing cover time.\" International Conference on Machine Learning. PMLR, 2019.\n\n[7] Abel, David, et al. \"Policy and value transfer in lifelong reinforcement learning.\" International Conference on Machine Learning. PMLR, 2018.", "limitations": "N/A", "rating": 3, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\\+ The two-player formulation is, to my knowledge, a novel way of analysing the problem \n\n\\+ Practical algorithms are backed by theoretical guarantees, while empirically the proposed algorithms vastly outperform the given baselines\n\n\\+ The introduction of the racing car domain early on really helped to ground the ideas, making the paper much easier to understand\n\n\\- Much of the \"heavy lifting\" is encoded into the system. For example, much of the challenge of option discovery is simply waved away here by pre-specifying the subtasks, reducing the problem to only learning good policies. This is a much easier problem than many other approaches are concerned with (see comments in next section), which makes the contribution fairly incremental. \n\n\\- It is unclear how this work fits in with other option-discovery approaches and there are potentially more appropriate baselines to compare against (see next section)\n\n\\- I found parts of the paper difficult to read because of the dense notation. In particular, Section 3.1 was slow going, particularly when reading the part about the asynchronous value iteration algorithm. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "8-O6AvkG-BQ", "reviewer": "Reviewer_bKP4", "summary": "This paper mainly focuses on improving RL policy performance on long-horizon tasks, including learning efficiency and generalization. The proposed framework has a high-level objective to adversarially choose tasks to minimize the reward achieved by the low-level policies. The empirical evaluation demonstrate the method works across different domains.", "questions": "**Questions**\n\nThis paper is over the qualified line, and I don't have many questions. I only have two concerns :\n\n1. Considering that the proposed method is based on the setting of multi-tasks, I was expected to find more complex tasks in the experiments. Based on the experiments, I think it is not necessary to train sub-policies for diverse sub-tasks, but a goal-conditioned RL policy is enough. What if using a goal-conditioned RL policy instead of the set of policies $\\pi_1$?\n\n2. How does ROSAC work in a more realistic task? Refer to Line.13 in Algorithm 2, what if the agent performs badly on all sub-tasks and has relatively low improvement even as the training processing. In other words, what if the initial sub-task is very difficult to learn (e,g., a 7 DoF robotic arm control)?", "limitations": "\n**Reference**\n\n1. Benjamin Eysenbach, Ruslan Salakhutdinov, Sergey Levine. Search on the Replay Buffer: Bridging Planning and Reinforcement Learning, Arxiv, abs/1906.05253.\n\n3. Shuang Ao, Tianyi Zhou, Guodong Long, Qinghua Lu, Liming Zhu, Jing Jiang. CO-PILOT: COllaborative Planning and reInforcement Learning On sub-Task curriculum, NeurIPS 2021.\n\n2. R. Gieselmann, Florian T. Pokorn, Planning-Augmented Hierarchical Reinforcement Learning, IEEE Robotics and Automation Letters, 2021.\n\n4. A. Srinivas, A. Jabri, P. Abbeel, S. Levine, and Chelsea Finn. Universal planning networks. ArXiv, abs/1804.00645.", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n1. This paper is well-written and structured. I enjoy reading it. The authors demonstrate motivations clearly and are easy to accept. \n\n2. The theoretical explanation in this paper is sufficient and correct, good.\n\n3. The results of the empirical evaluation are present clearly. \n\n \n**Weakness**\n\n1. The first paragraph in section 3 misses the definition of the difference between $A_1$ and $A_2$. The action of agent 2 is causing task shifting, which is one of the main differences between the proposed method, and should make it more straightforward.\n\n2. The discussion in related work can be more substantial. It would be better to summarize the difference between ROSAC with previous work rather than only listing the related research. On the other hand, there is a line of recent work that apply planning to improve the robustness of RL policy on long-horizon tasks (e.g., [1]-[4]). It is better to include them in the discussion.\n\n3. It would be better to include more analysis of the experiment results in section 5. It is necessary to have some in-middle results to demonstrate how two agents work together to affect the training.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Reject