Constrained GPI for Zero-Shot Transfer in Reinforcement Learning
Authors: Jaekyeom Kim
Seohong Park
Gunhee Kim
Venue: NeurIPS 2022 Accept (2022)
Keywords: Reinforcement learning
Zero-shot transfer

Abstract: For zero-shot transfer in reinforcement learning where the reward function varies between different tasks, the successor features framework has been one of the popular approaches. However, in this framework, the transfer to new target tasks with generalized policy improvement (GPI) relies on only the source successor features [5] or additional successor features obtained from the function approximators’ generalization to novel inputs [11]. The goal of this work is to improve the transfer by more tightly bounding the value approximation errors of successor features on the new target tasks. Given a set of source tasks with their successor features, we present lower and upper bounds on the optimal values for novel task vectors that are expressible as linear combinations of source task vectors. Based on the bounds, we propose constrained GPI as a simple test-time approach that can improve transfer by constraining action-value approximation errors on new target tasks. Through experiments in the Scavenger and Reacher environment with state observations as well as the DeepMind Lab environment with visual observations, we show that the proposed constrained GPI significantly outperforms the prior GPI’s transfer performance. Our code and additional information are available at https://jaekyeom.github.io/projects/cgpi/.

Constrained GPI for Zero-Shot Transfer in Reinforcement Learning
Jaekyeom Kim Seoul National University j jaekyeom@snu. ac.kr
Seohong Park University of California, Berkeley sechong@berkaley. edu
Gunhee Kim
Seoul National University gunhee@snu. ac. ac.kr kr
Abstract
For zero-shot transfer in reinforcement learning where the reward function varies between different tasks, the successor features framework has been one of the popular approaches. However, in this framework, the transfer to new target tasks with generalized policy improvement (GPI) relies on only the source successor features [6] or additional successor features obtained from the function approxi- mators' generalization to novel inputs [2] The goal of this work is to improve the transfer by more tightly bounding the value approximation errors of successor features on the new target tasks. Given set of source tasks with their successor features, we present lower and upper bounds on the optimal values for novel task vectors that are expressible as linear combinations of source task vectors. Based on the bounds, we propose constrained GPI simple test-time approach that can improve transfer by constraining action-valus approximation errors on new target tasks. Through experiments in the Scavenger and Reacher environment with state observations well the DeepMind Lab environment with visual observations, we show that the proposed constrained GPI significantly outperforms the prior GPI's transfer performance. Our code and additional information available at github.io//rojaects/cgil
Introduction
For sequential decision making, deep reinforcement learning (RL) has been shown to be effective for various types of problems including games [II) and robotics (20 22]. With such great successes, interest in multi-task RL has also surged, where its goal is to train a single agent that can efficiently solve multiple varying tasks. In multi-task RL, focus the transfer learning setting, where the agent learns shared structural knowledge from a set of source tasks during training, and exploits and generalizes them in new, unseen target tasks at test time.
One popular approach to transfer in RL is to leverage the successor features (SFs) framework 16
7 22 25 which transfers policies learned source tasks to target tasks, where the tasks share the same environment dynamics but differ their reward functions. Successor features build representation of value functions decoupled from reward functions, and transfer the tasks with arbitrary reward functions by taking an inner product with corresponding task vectors. They utilize generalized policy improvement (GPD) (6. which generalizes policy improvement with multiple policies and provides the performance lower bounds for GPI policies.
However, GPI does not take into account any information from the smoothness of optimal action- value functions with respect to task vectors. Tackling this issue, Borsa al. 122 propose universal successor features approximators (USFAs), which can estimate the optimal successor features for novel task vectors. Nevertheless, the function approximator can make high approximation errors on the task vectors, especially when the new task vectors are distant from the source task vectors. For instance, when USEAs are trained with source tasks to get close to given goals, they may not
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

generalize well to the target tasks where the agent should get away from the given goals. That is, if the elements of target task vectors have the opposite signs from the source task vectors, USFAs could output successor features with high approximation errors.
To improve the successor features approximation of USFAs for the new tasks, aim at bounding value approximation eπrоrs on the new target tasks. We first introduce a new theorem on bounding the optimal values for the tasks that are expressible linear combinations of source tasks. Our theorem generalizes the conical combination condition used by the prior theorem by Nemecek and Parr 225]. Using our new bounds as constraints, we can train the successor features approximators whose action-valus approximation errors on novel tasks are bounded. We extend this idea so that we accomplish a similar effect with no additional training, as a result, we propose constrained GPI as a test-time approach to bounding the approximation errors. Despite its simplicity and no need for modification to the training procedure, we empirically show that constrained GPI attains large performance improvements compared to the original GPI in multiple environments, including the Scavenger g and Reacher 13 environments with state observations and the DeepMind Lab
7 10 environment with first-person visual observations.
Our main contributions can be summarized as follows:
We present a novel theorem on lower- and that can be expressed as linear combinations of source tasks. It extends and generalizes the previous theorem for conical combinations by Nemecek and Parr 255 to enable a broader application of the bounds.
optimal values for novel tasks
Based on our new theorem, we propose constrained GPI as simple test-time approach that can improve transfer to novel tasks by constraining action-value approximation errors on new target tasks, with modification to the training procedure.
We empirically show that our approach can improve the performance over the GPI baselines by large margins in the Scavenger, Reacher and DeepMind Lab environments. We also provide analyses for better understanding of our results.
Preliminaries
We describe the problem setting and background on successor features and universal successor features approximators. We refer the reader to Appendix for an in-depth discussion of related work.
2.1 The Zero-Shot Transfer Problem in RL
We define a Markov Decision Process (MDP) as M= (S.. S and A are the state and action spaces, respectively. P(-[s, a) defines the transition probability distribution of the next states given E S and E A. R((s, s') is the reward for taking action at state s resulting in s'. and y E (0, 1] is the discount factor. We assume that rewards are bounded.
We consider the zero-shot transfer problem; in 6 each task is defined by its task vector Rd. and only the reward functions differ across tasks, being decomposed as
Rso(s,,,8)) = T()s
where a(s, s²) Rd is the features of (s, s's). We denote the set of source task vectors as T. which is used for training. At test time, we evaluate the transferred policy each target task w' & T with no additional update of pre-trained components. We examine both the possible scenarios: (i) the features (6ss, a, 8') are available to the agent 66, [2] and (ii) pre-defined features are available and the agent needs to construct own features and task vectors. We first introduce the formulation for (i) in Section 2.2 and then its variant for (ii) in Section 2.3

---

where the value functions 2.2 Successor Features and Universal Successor Features Approximators
We now review successor features (SFs) 6 and how they are transferred different tasks. Equa- tion ① allows expressing the action-value function for policy on task w as
nC
(w 1E
Σ i=0
re4s St Ar
ET
2' O St = = A.
w - 2 (s. w.
where Or o( de, St+1) E Rdd Here, (8, a) E Rd is called the SFs for policy π at (8, a), and taking its inner product with an arbitrary task w results in the action-value for on w: i.e. Q (s, a). Thanks to the analogy between (rewards r, action-value functions Q) and (features é, successor features v) the Bellman equation applies to SFs and thus they can be trained similarly to the way action-value functions are learned; e.g., Q-learning.
The policy improvement theorem [TT] states that new policy that takes greedy action according to given policy's value function at each state performs at least as well as the original policy. Generalized policy improvement (GPI) (6) extends policy improvement of multiple policies are available. Given a task w', a set of policies Ti,.... Wn. their action-value functions ?......, Qw
the
and their approximations , Qw" the GPI policy is defined as
TOPI (s) argmax max (
Barreto al. (6 suggest that Q (s, a) 2 max, 2 (s, a) - max; also provide the upper bound on the suboptimality of the GPI policy
Qw
- ll.
They
l1Qw - Qui 11 S - { min oollool| will max
|
(4)
where each I. is an optimal policy for w,
While the GPI theorem allows the transfer of learned successor features to arbitrary tasks that share the same environment dynamics, it is limited in the following aspect. GPI uses the action-values for source tasks on target tasks based on the reward decomposition assumption (Equation (T) i.e., 2% (s, a) ( a) w/ for each However, it does not take any advantage of the smoothness of the optimal action-value functions with respect to different task vectors [12].
To overcome this limitation, Borsa al. 12 introduce universal successor features approximators (USFAs). Inspired by universal value functions (UVFs) (30), they extend the original successor features with policy vectors z IR as input their approximators. More specifically, universal successor features (USFs) are defined to satisfy
(s, a) vo a),,
(5)
where z is a policy vector for the policy π.. and USFAS & are the learned approximators of USFs tb. Naturally, the value functions are expressed
Q a) = (ss a, 2) T = visss a, 2) T = Q (s.
(6)
Each reward function induces optimal policies, which can be encoded using the corresponding task vectors. That is, one can simply choose to define the policy vector space to be the same as the task vector space (1 d) and let
be policy vector of an optimal policy for task w. Then, ao and QT denote an optimal policy for w and its action-value function, respectively.
The training of USFAs is similar to that of SFs, except for that it additionally involves sampling of policy vectors given task vectors. The update of USFAS at the k-th iteration
||otes,, 5') + + (5)) 2) - v(s, a, 2) |
((++1) + argmin Ew-T, (11))
(7)
for a' argmax, 3 (50) b, T2. D2(-)u) the policy vector distribution; for instance, NWw, aT) can be used for better training with diversified inputs. μ is the transition sampling distribution, which
3

---

involves the GPI policy of the samples from (+(w) or replay buffer. We use gradient descent to update the parameters.
USFAs provide a benefit that they allow a GPI policy to use an arbitrary set of policies ( ) as (s) E argmax, max QT (s, a). However, the generalization of USFAs to new policy vectors depends on a function approximator v. and thus if C contains policy vector(s) distant from source vectors, a GPI policy with C may have high approximation enrors and perform poorly or even worse than GPI policy with only source vectors [12]. as will be demonstrated later in our experiments.
2.3 Universal Successor Features Approximators with Learned
For the scenario where features ø's are not provided to the agen(] we adopt the problem formulation from Ma et al. [3 where for each task the task information g 9 is given the agent. Although the task information g, unlike task vector, cannot be directly combined with successor features for transfer to a novel task, zero-shot inference could still be possible by leveraging the information about the task.
Specifically, we not only perform the original learning of v letting the task information induce policy vectors instead, but also train and i to approximate the reward decomposition with transition samples. As done in [23] we update vi & and is using gradient descent minimize E. pg (sarr)))))))))))
L + L for
(0s8 8') + (03 (6', 22) - -v(ss 2) /
L d
(8)
- = + (8', a', x) x (k) (z) -is a, 2) [o(z)))
and = argmax, (s') .'', b, Tx) (x) at the k-th iteration. The superscript (A) denotes the target, 7 is the source task information set, D2(-lg) is the policy vector distribution conditioned on the task information and is the sampling distribution.
3 Constrained GPI for Improved Zero-Shot Transfer of Successor Features
To mitigate the aforementioned issue of the possibly unlimited approximation errors of USFAs, we propose a simple yet effective method that improves the transfer of successor features by further leveraging the reward decomposition structure in Equation (T). We first present, under a more relaxed condition, lower and upper bounds on the optimal values for novel task vectors that are expressed as linear combinations of source task vectors (Section 3.1 Then, we propose a novel approach called constrained GPI, which effectively confines the approximated action inside the computed lower and upper bounds (Section 3.2).
W22
Figure 1: An ex- ample comparison of task space coverage. While the conical combinations of w and W2 covers only area ①, the linear combinations covers both area 1-Q.
3.1 Bounding Optimal Values for New Tasks
Theorem 1 of 25 provides the lower and upper bounds on the value of an optimal policy for a new task, whose vector w' is a positive conical combination of source task vectors i.e., w' = E Ow such that aw 2 0, Vew E T and ΣweT Oww d However, for broad application of such bounds, the positive conical combination condition can be too restrictive, since the resulting bounds only apply the task vectors that appear inside
the conical hull of source task vectors.
Therefore, we suggest more relaxed theorem, which holds for arbitrary task vector w' that is expressed as a linear combination the source task vectors i.e., w' = E Cuw w for aw E R, Ve E T. Figure [ shows an example that compares the task space coverage of conical 25 and
'One typical example presented later our experiments the case where the agent observes visual inputs. Then, it is not trivial to derive features that linearly decompose reward functions.
We slightly abuse the notation and let aw denote the coefficient for vector w.

---

our linear combinations. With our extended task space coverage, we can apply the bounds to more general target tasks outside of the conical hull, which will be further discussed in the next section.
Twn We define Cwz to be upper bound on the approximation error of Qwa for arbitrary tasks w1, W2 such that
Qww (8,a) a) - To (8, a)] S 6w2 (s,a), E(x,A)
(10)
and we present our theorem as follows.
Theorem 1. Given task vector w' = =I WET Ow for ow BR Vw TT for all state-action pairs (s, a) E S x A, the action-value of Twn, which is an optimal policy for task w', on task w is lower- and upper-bounded as Lw' (Tl,TTss,0 a) S Qw"" (s,a) S Uwi,Ta a) for
(w,,) = max @ue (s, a) E W a)) wET
(11)
UW.Ta(s,a) = Σ
max {aw (QTH (s, + Gu. (n (s,0) a aw,
(12)
WET
for some (s, min, Qw a) such (w, = - where riin is the minimum reward on w i..., w =
R (s, (s,00 a) and a = (awlwET.
Proof. For the derivation of the lower bound L buorT(s,, a), since " is the optimal action-value function for task w' and Qir (s,a) 2 QTH (s, a) for arbitrary task w and state-action pair (s, a),
Qw (8,a)
max 2 (s. 2 max (s, cw" (s. WET
(13)
we use that Qur ()) I Qur(s, a) and Qw' min, Q (8, 2 Cwo (8, a) for arbitrary task w and state-action pair (s,4), which leads to
For the upper bound
(s,a) 2
Qhr (8,a)
Σ anu (QwH (s, a))
Σ WET
am (m,
(14)
max {aw (QTH (s, a)) +
Σ Cw,
(15)
10 TT
WET
max {aw (Q (5, a) Cs a)), + am
(16)
WET
WET
Σ {max faw (s, Cw a)) 03 + Cw,
(17)
ET
max {ar 2 (s, aw aw as)
(18)
w
Σ max { aaw @TTT (5, a) + The (s, aw Cw. TT
(19)
In Equation (19), for each w E T, the sign of a' determines which of the two terms the max operator is used. If okw 2 0, the max operator selects the first term, whereas a negative Owo lets the second term be used, Note that our Theorem recovers Theorem I of (55 when w' a conical combination of w's from T i.e., aw 2 0, Yw E T.
Intuitively, this theorem states the condition that the optimal action-value for an arbitrary target task must satisfy, given the optimal successor features for the source tasks. The theorem is applicable to different problems wherever bounding of optimal values is useful. One example is policy cache construction, where the agent should decide whether to reuse existing policies in the cache set or learn a new one given each new task 25 As will be shown the next section, we employ the bounding as constraint on the vaauess for novel target tasks, for the guidance of transfer. In Sections 4. and and and appendix B. we empirically show that the application of our Theorem T can significantly improve the performance in the cases where target tasks are outside the conical hull of source tasks.
5

---

3.2 Constrained Training and Constrained GPI
As described in Section 2.2 the universal successor features approximators (USFAs) 12 improve the original successor features so that arbitrary policy vectors, including the ones for target tasks, can be used for GPI. However, the use of arbitrary policy vectors with USFAs solely relies on the generalization power of the approximators (e.g., neural networks). Thus, the obtained successor features on novel tasks might contain high approximation errors, which could make the GPI policy perform poorly.
Our high-level idea to tackle the issue is to exploit the reward decomposition structure in Equation (I) even for obtaining SFs for new tasks, instead of solely relying on the approximators. We employ the lower and upper bounds on optimal values from Theorem I enforce the bounds on the approximate successor features. As result, the approximation errors can be reduced by restricting the estimated optimal values to be inside those bounds around the optimal values, which can prevent the use of erroneous values during the transfer to unseen tasks.
For now, we will first introduce how to train the successor features approximators output the successor features that satisfy the bounds on novel tasks. Then, we will point out that an analogous effect can be accomplished by modifying only the inference algorithm, and propose constrained GPI as simple yet effective test-time approach to improving zero-shot transfer to novel tasks.
Constrained training of SF approximators, In the original training of USFAs, the approximators are learned with a set of source tasks T in Equation (7). We propose to guide the training by employing Theorem T we impose constraints for the approximators using the lower and upper bounds on the optimal values for arbitrary linear combinations of source tasks. Specifically, for the training of USFAs, we use Equation (7) but with the following constraints:
Tw' a) I qi(s, a, T "w'= S -Uw..T&te (T (8, a) for w' E W.
(20)
where (s, a) is the same sample as the main objective of Equation 77.. W is a set of task vectors for the constraints, which can be independent of the source task set T, and E(.) determines the coefficients a given a target task w' and 'T We will explain later how to determine E(-). W can be any subset of the linear span of source task vectors, but practically, we randomly sample number of vectors from the span at each update. Since the targets of the constraints are not fixed with respect to both w' and (s, a) throughout the training, we use penalty terms (or soft constraints) that linearly penalize the constraint violations as
1
Σ W'EW
+ +(0Ter a) - Uw' Uw:.T.tw::Taaaa ((Lw:r(o,0)) - - (s, (0
W
where f=)+ denotes max(2r0).
The constrained training suggested above can make the approximators comply with the bounds for any tasks without requiring any additional interactions with the environment. However, it has some downsides. First, since it is a new training procedure, existing pre-trained models cannot be used. It requires some additional computational cost compared to the naive training of successor features approximators. Second, the enforcement of the constraints for training can introduce additional hyperparameters (e.g., the weight coefficient for the penalty terms). Thus, suboptimal hyperparameters may introduce either instability in the training or a decrease in the performance.
Test-time constrained GPI. Our idea starts with the observation that in the constrained training, the learned successor features from source tasks are considered the "trustworthy" features for the constraints, because the USFAs are trained the source tasks. Besides, only the source successor features are used for computing the constraints for all the other tasks. It implies that the learning of the source successor features better not be affected by other criteria, and more accurate source successor features would produce better constraints for other tasks with smaller errors.
Based on the implication, we propose constrained GPI, which can not only overcome the limitation of USFAs as done by the aforementioned constrained training but also have two additional practical merits: (i) it is computationally simpler, and (ii) it is a test-time approach with no training. Simply put, we propose replacing the usual GPI policy with the constrained GPI policy as
(s))(()) E argmax max min { max { (Q (8. a), Tow T. ((8, a)}. Uw UWT...WWW TTw T .. (3.00)) s..
(21)
=EC
[

---

where the target task w' is expressible as linear combination of the source tasks and E(-) again outputs a given w' and T as in Equation (20p. is a set of policies that we can freely choose when applying the constrained GPI.
The constrained GPI policy selects the actions that maximize the maximum action-values as the original GPI policy does but also caps the values with the lower and upper bound constraints derived from the source successor features. The upper bound constraint fixes the overestimation of values computed with approximate successor features for either the target task w' or any other tasks used for constrained GPI. The lower bound constraint ensures that action-valuee on the target task for the greedy action selection are at least as close to the optimal target action-values as the lower bounds.
The approximation error terms in the lower and upper bounds i.e., e (s, a) and cT a) in Theorem D could be ignored in practice, as long as the approximation errors of the source successor features sufficiently small. Also, we can obtain the tightest upper bound by defining {() as
E(w', a,) == argmin Uwi,T "lo,,T,fawi Aaw (s,a) subject to w' Σ WET
Oww
(22)
weTT
The objective Uw/ "w..T.fa T.jaw hee (s, a) is the sum of the piecewise linear functions. Thus, Equation 22 can be solved with linear programming.
We observe that using the lower bound constraint with L a) is equivalent to including the successor features for source tasks in the input to the constrained GPI; i.e., T S C. Also, since Iow . a) I T (T& .T a) (s, a), there would be no difference between GPI and constrained GPI when C TT Thus, in our experiments, we mainly use C = {w'), which is equivalent using C= CUU {w").
4 Experiments
4.1 Scavenger Experiments
We start our experiments in the Scavenger environment X 91. which can assess our approach with minimal influence from external causes. In Scavenger, the agent is positioned at one of the cells in a G x G grid, and the goal is maximize the return by collecting objects. Both the agent and objects are spawned at random locations, and there are d classes of objects where the class determines the value of the reward. The state space is S = f0, 1 current locations of the objects on the map and the last channel specifies the walls where the agent cannot go and objects do not appear. There are four actions available: AA = (UP, DOWN, LEFT, RIGHT), }. and the agent picks up an object by visiting the cell of the object, which spawns new object of a random class at a random location. The feature Φ(s, a, 8') E {0, 1] is a one-hot vector whose element represents whether the agent picks up object of that type or not within the transition. The task vector w E Fd determines the reward values for the d different classes of objects. Please see Barreto et al. for the full details.
where the first d channels describe the
We evaluate the zero-shot transfer performance of different approaches Le.. we first train USFAs as proposed in [2), and measure the performance of GPI and constrained GPI policies that use the same set of USFAs on target tasks with no further policy updates. We set G = 11 and use 20 objects in total with the different numbers of classes; d = 2 and d = 4. With d 4, we also test the USFAs that are learned with the constrained training for a comparison. We use the standard basis vectors of Rd the set of source tasks as done in [2], and evaluate agents the set of target tasks defined by {_ Therefore, all the target tasks except for the all-ones vector are not covered by the conical hull of source tasks, which requires Theorem I for bounding of values.
We train eight USFAS agents for 1M steps, and evaluate them on each target vector 10 times with a fixed set of 10 random seeds. To be invariant to the reward scale differences between different tasks, we normalize the scores (or returns) from the environment by the minimum and maximum scores with respect to all the agents' evaluation episodes each task.
Figures and compare the performance of the USFAS agents with GPI and constrained GPI for exploitation, following the evaluation scheme suggested by (2). Although they use the same set of trained USFAs, the constrained GPI brings a notable performance improvement in comparison with the original GPI. Also, Figure] suggests that the constrained GPI, the test-time method, can match
7

---

Median
IQM
Mean
Optimality Cap
GPI-S- GPI-T-
GPLST
CGPI-T (ours)
0.40
0.48
0.56
0.32
0.40
0.48
0.56
0.40
0.44
0.48
0.52
0.52
0.56
0.60
Normalized Score
(a) All
Median
IQM
Mean
Optimality Gap
GP1.S-
GPET
CPEST
CGPI-T (ours)
0.3
0.4
0.5
0,16
0.24
0.32
0.40
0.30
0.35
0.40
0.60
0.85
0.70
0.75
Normalized Soore
(b) Harsh
Figure 2:
The aggregated performance metrics with 95% bootstrap confidence intervals 2 16 of
different test-time approaches on Scavenger with d = 2. CGPI represents our constrained GPI, whereas GPI is the original GPI. The suffixes -S, -T and -TT denote using the set source task vectors, the target task vector and both C, respectively. (a) All is the evaluation on the entire set of target task vectors from {-1, 1]d, whereas (b) Harsh denotes the evaluation on a subset consisting of 'harsh' tasks, whose number of 1 I's is no less than that of I's (e.g., ()).
Mediam
IQM
Moun
Optimality Gap
GPL- GPI-T
GPI-ST
CT (ours) (ours]
GPI-S
GPL-T
CT (ours) GPI-ST
CGPI-T (ours)
0.40
0.45
0.50
0.36
0.40
0.44 0.48
0.42
0.45
0.48
0.51
0.54
0.57
0.60
Normalized Score
(a)
Modiam
IQM
Mean
Optimality Gap
GPLS
GPI-T
GPI-ST
CT (ours)
GPI-S
CT (ours]
GPL-T
CT (ours) GPI-ST
CGPI-T (ours)
0.36
0.42
0.48
0.30
0.35
0.40
0.45
0.36
0.40
0.44
0.56
0.60
0.64
Normalized Score
Harsh
Figure 3: The aggregated performance metrics with 95% bootstrap confidence intervals 2 16 of different test-time approaches on Scavenger with 4. CT denotes the constrained training. We refer the reader to Figure 2 for the full description.
or even outperform the agents learned with the constrained training. One possible explanation is that the constrained training might experience some instability in learning depending on the choice of the hyperparameters, as described Section 3.2
In the first and the second columns of Table 1 we
Table 1: The first and second columns show the proportions of the maximum action-values changed by constrained GPT's lower and up- per bounds, during the evaluation on Scav- enger. The final column the proportions of resulting actions changed by them.
present the proportions of the action-values that are changed by the lower and upper bounds of the con- strained GPI, measured for the evaluation on Scav- enger. The third column shows the proportions of resulting greedy actions changed by them. implies that USFAs i..,, the function approximators of suc- cessor features, may not satisfy the optimal value bounds presented in Theorem and applying the bounds could change fair proportion of greedy ac- tions to improve the performance.
Setting
Lower- bounding
Upper- bounding
Action change
=
23.94%
43.01%
22.11% 20.78%
d
5.60%
46.86%

---

Median
IQM
Mean
Optimality Cap
GPI-S
GPI-T-
GPLST
CGPI-T (ours)
0.15
0.30
0.45
0.60
0.0
0.2
0.4
0.6
0.4
0.5
0.6
Score
(a) Left-to-right
Median
IQM
Meun
Optimality Gap
GPI-S GPL GPLSTT
CGPI-T (ours)
0.03 0.00 0.03 0.000 0.025 0,050 0.075
0.20
0.21
0.25
0.72
0.76
0.80
0.84
Skore
(b) Near-to-far
Figure 5: The aggregated performance metrics with 95% bootstrap confidence intervals [2] 16 of different test-time approaches with the two scenarios in the DeepMind Lab environment. CGPI represents constrained GPI. The suffixes -S, and -ST denote using the source task information set, the target task information and both for C, respectively. (a) In the left-to-right setting, train the agent for the goals from the left half and test for the right half. (b) In the near-to-far setting, we train the agent for nearby goals and test for farther goals.
4.2 DeepMind Lab Experiments with Learned Ø
For evaluation of our approach more complex and realistic setting, we employ DeepMind Lab 10 [2] and conduct experiments in first-person view 3ID environment. In single room, goal object is placed arbitrarily, and the objective is to reach the goal before the episode ends where its location changes between tasks. Figure shows an example scene that the agent sees with the goal object in red.
At every time step, the agent observes 84 84 x image from the environment and outputs one of 45 possible actions, which include 5, 3 and 3 choices for LOOK_LEFT_RIOOL BLEFFTRRIHTTPIXEEEE BESrT__PXEL__PPPE PRA_E, CTRAFE_LEFF, RIGHT and MOVE_ BACK_FORWARD FORWARD trols, respectively. Since observations are the first- person view, the goal object may not be by the agent, which makes transfer given the task information critical the success of the tasks. In each task, we divide the room into an 11 x 11 grid and place the goal object in one of the cells. The task information g is two-dimensional vector that contains the coordinate of the goal in the grid. Starting at the center of the room, the agent receives reward of one if it reaches the goal within the episode horizon or no rewards otherwise. Therefore, the reward functions are sparse.
Figure 4: An example scene that the agent sees in the DeepMind Lab tasks. The object is the goal.
For these experiments where the agent observes rendered images rather than the underlying states, it may not be viable to define features ø's and task vectors w's that linearly decompose reward functions. Therefore, we train agents with the learning of & and in from samples from the source tasks with d 2 as described in Section
Inspired by Hong et al. [18), we examine zero-shot transfer with the GPI and constrained GPI using two transfer settings: "left-to-right" and "near-to-far", In the "left-to-right" setting, the agent is trained on the source tasks whose goals are sampled from the left half the room and is tested on the target tasks with goals from the right half. In the "near-to-far setting, the source tasks have the goals within an x 2 distance from the center the room, and target tasks set the goals farther than - = 2.
For each setting, we train eight USFAs agents with different seeds for 3M environment steps on the source tasks and test them on the target tasks. Figure 5 presents the comparison of the GPI with

---

different C's and the constrained GPI. Leveraging the same set of trained USFAs with learned ă and w. the constrained GPI outperforms the GPI with the three C's in both settings by a notable margin. Another observation is that the trained USFAs agents seem to overfit more to the source tasks in the "near-to-far" setting compared to the "left-to-right" setting. makes the performance on the target tasks much worse. Nonetheless, the constrained GPI is still helpful in such overfitting situations.
5 Conclusion and Discussion
We presented constrained GPI, simple yet effective test-time approach for transfer with approximate successor features. We first focused on the issue that although universal successor features approxi- mators (USFAs) exploit the smoothness of optimal values across different tasks, their approximation errors novel target tasks could be large especially when those tasks are quite distant from source tasks. Thus, we introduced theorem about lower and upper bounds on the optimal values for novel task vectors that belong to the task vector space linearly spanned by the set of source task vectors, relaxing the conical combination condition used for the theorem by Nemecek and Parr [25]. We proposed a constrained training scheme making use of those bounds for reducing the action-value errors of the learned approximators on novel tasks. We then suggested constrained GPI that uses the bounds test time achieve an analogous effect, allowing the use of previously trained models. We empirically showed that this test-time approach can improve the zero-shot transfer performance by large margin in multiple environments.
Limitations and future directions. There may be some cases where the minimum rewards for source tasks i..., rmmins s are overly small, which could lead to less changes of both action-values and behaviors induced by the upper-bounding in Theorem with (s, - T v w direction to tackle the issue is to learn the minimum action-valus function during the training and to use the approximate minimum value at each state-action pair as (w a) for deriving the upper bound in Theorem T It may allow computing upper bounds more tightly and adaptively for different state-action pairs. Also, if the learned successor features approximators have large errors even on source tasks, not only GPI but also constrained GPI's bounding may not be meaningfully helpful. One idea to mitigate the issue error term that appears in Theorem ] into account, e.g., by using ensemble models. As an intriguing direction for future research, we could extend our constrained GPI other non-linear forms of reward or value decompositions. It may also be interesting to make transfer with successor features and constrained GPI compatible with large-scale approaches for generalization such as 228.. We do not see direct negative societal impacts of this work.
min An interesting
take the uncertainty in the approximators and the approximation
Acknowledgements
We thank the anonymous reviewers for their valuable comments. This work was supported by Sam- sung Advanced Institute of Technology. and Institute of Information & communications Technology Planning& & Evaluation (IITP) grants funded by the Korea government (MSIT), including (No.20) NN..0019. 19- 0-01082, SW StarLab), quality enhancement of casual videos and their 3D metaverse transformation), and NNo.2021-0-01 Artificial Intelligence Graduate School Program (Seoul National University)). Jaekyeom Kim was partly supported by Google PhD Fellowship. Gunhee Kim the corresponding author.
Fundamental research on continual meta-learning for
References
[1] Majid Abdolshah, Hung Le, Thommen George Karimpanal, Sunil Gupta, Santu Rana, and Svetha Venkatesh. A new representation of successor features for transfer across dissimilar environments. In Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, pages 1-9. PMLR, 2021.
[2] Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Mare Belle- mare. Deep reinforcement learning at the edge of the statistical precipice. Advances in Neural Information Processing Systems, 34, 2021.
[3] A. Agrawal, B. Amos, S. Barratt, S. Boyd, S. Diamond, and 7. Kolter. Differentiable convex optimization layers. In Advances in Neural Information Processing Systems, 2019.
10

---

[4] Lucas Nunes Alegre, Ana Bazzan, and Bruno C Da Silva. Optimistic linear support and successor features as basis for optimal policy transfer. In International Conference on Machine Learning, pages 394-413. PMLR, 2022.
[5] Safa Alver and Doina Precup. Constructing good behavior basis for transfer using generalized policy updates. In 10th International Conference on Learning Representations, ICLR 2022, 2022.
[6] André Barreto, Will Dabney, Rémi Munos, Jonathan J. Hunt, Tom Schaul, David Silver, and Hado van Hasselt. Successor features for transfer in reinforcement learning. In Advances Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 4055-4065, 2017.
[7] André Barreto, Diana Borsa, John Quan, Tom Schaul, David Silver, Matteo Hessel, Daniel J. Mankowitz, Augustin Zidek, and Rémi Munos. Transfer in deep reinforcement learning using successor features and generalised policy improvement. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018, pages 510-519, PMLR, 2018
[8] André Barreto, Diana Borsa, Shaobo Hou, Gheorghe Comanici, Eser Aygün, Philippe Hamel, Daniel Toyama, Jonathan J. Hunt, Shibl Mourad, David Silver, and Doina Precup. The option keyboard: Combining skills in reinforeement learning. In Advances in Neural Information Processing Systems 32: Annual Conference Neural Information Processing Systems 2019, NeurlPS 2019, December 8- 8-14, 2019, Vancouver, BC, Canada, pages 13031-13041, 2019.
[9] André Barreto, Shaobo Hou, Diana Borsa, David Silver, and Doina Precup. Fast reinforcement learning with generalized policy updates. Proceedings of the National Academy of Sciences, (48):30079-30087, 2020.
[10] Charles Beattie, Joel Z Leibo, Denis Teplyashin, Tom Ward, Marcus Wainwright, Heinrich Küttler, Andrew Lefrancq, Simon Green, Víctor Valdés, Amir Sadik, et al. Deepmind lab. arXiv preprint arXiv: 1612. 038001, 2016.
[11] Richard E. Bellman. Dynamic programming. Princeton University Press, 1957.
[12] Diana Borsa, André Barreto, John Quan, Daniel J. Mankowitz, Hado van Hasselt, Rémi Munos, David Silver, and Tom Schaul. Universal successor features approximators. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
[13] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. OpenAl Gym. arXiv preprint arXiv: wrXivv1606.011 01540, 2016.
[14] Peter Dayan. Improving generalization for temporal difference learning: The successor repre- sentation. Neural Computation, (4):613-624, 1993.
[15] Elizabeth D Dolan and Jorge J Moré. Benchmarking optimization software with performance profiles. Mathematical programming. 91(2):201 2002.
[16] Bradley Efron. Bootstrap methods: another look at the jackknife. In Breakthroughs in statistics, pages 569-593. Springer, 1992.
[17] Fernando Fernández and Manuela Veloso. Probabilistic policy reuse a reinforcement learning agent. In Proceedings of the fifth international joint conference on Autonomous agents and multiagent systems, pages 720-727, 2006.
[18) Zhang-Was Wei Hong, Ge Yang, and Pulkit Agrawal. Bilinear value networks. In 10th International Conference Learning Representations, ICLR 2022, 2022.
[19] Jonathan Hunt, André Barreto, Timothy P. Lillicrap, and Nicolas Heess. Composing entropic policies using divergence correction. In Proceedings of the 36th International Conference on Muchine Learning, ICMI. 2019, 9- June 2019, Long Beach, California, USA, pages 2911-2920. PMLR, 2019.

---

[20] Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, and Sergey Levine. Qt-opt: Scalable deep reinforcement learning for vision-based robotic manipulation. ArXiv, abs/1806. 10293, 2018.
[21] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. CoRR, abs/1412.6980, 2015.
[22] Ashish Kumar, Zipeng Fu, Deepak Pathak, and Jitendra Malik. Rma: Rapid motor adaptation for legged robots. ArXiv, 0bbs/210.000033 2021.
[23] Chen Ma, Dylan Ashley, Junfeng Wen, and Yoshua Bengio. Universal successor features for transfer reinforcement learning. arXiv preprint 2XXv:2001104025, 2020.
[24] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland. Georg Ostrovski, al. Human-level control through deep reinforcement learning. nature, 518(7540):529-
2015.
[25] Mark W. Nemecek and Ron Parr. Policy caches with successor features. In Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, pages 8025-8033 PMLR, 2021.
[26] Chris Nota. The autonomous learning library.
autonomous -ewrning-nnn--nnnn library, 2020.
[27] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary Viti,o, Martin Raison, Alykhan Tejani. Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-peeformance deep learning library. In Advances in Neural Information Processing Systems 32, 2019.
[28] Scott Reed, Konrad ZoIna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikov, Gabriel Barth-Maron, Mai Gimenez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, et al. A generalist agent. arXiv preprint arXiv:2205.06173 0610000.661..... 2022.
[29] Diederik M Roijers. Multi-objective decision-theoreti planning. PhD thesis, University of Amsterdam, 2016
[30] Tom Schaul, Daniel Horgan, Karol Gregor, and David Silver. Universal value function approx- imators. In Proceedings of the 32nd International Conference on Machine Learning, ICML 2015, Lille, France, 6-11 July 2015, pages 1312-1323 13200. JMLR.org, 2015.
[31] Julian Schrittwieser, Ioannis Antonogiou, Thomas Hubert, Karen Simonyan, Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, Timothy P. Lillicrap, and David Silver. Mastering atari, go, chess and shogi by planning with learned model. Nature, 588 7839:604-609 2020.
[32] Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In JEEE/RSJ International Conference on Intelligent Robots and Systems, pages 5026-5033. IEEE, 2012.
Checklist
The checklist follows the references. Please read the checklist guidelines carefully for information on how to answer these questions. For each question, change the default [TODO] [Yes]. [No], or [N/A] You are strongly encouraged to include justification to your answer, either by referencing the appropriate section of your paper or providing a brief inline description. For example:
Did you include the license to the code and datasets? [Yes] See Section ??.
Did you include the license the code and datasets? [No] The code and the data are proprictary.
12

---

Did you include the license to the code and datasets? [N/A]
Please do not modify the questions and only use the provided macros for your answers. Note that the Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [Yes] We will include
in the supplementary material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] We will include in the supplementary material.
(c) Did you report error bars (c.g., with respect the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g.. type of GPUs, internal cluster, or cloud provider)? [Yes] We will include in the supplementary material.
4. If you are using existing assets (e.g., code, data, models) or curating/reeeasing new assets...
(a) If your work uses existing assets, did you cite the creators? [N/A]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
13


Reviews:
[{"review_id": "ypyWcNzy4AF", "reviewer": "Reviewer_drJg", "summary": "The paper looks at the problem of transfer in reinforcement learning from source tasks to target tasks when the reward signal changes across tasks but the state and action spaces remain the same. Authors address a limitation of a current approach in this setting: GPI with universal successor features approximators (USFAs). The works argues that USFAs can make high approximation errors on test targets if their solutions are distant from the source targets' ones, and proposes a solution to mitigate this limitation. \n\nThe solution involves constraining the approximation action-value error on new target task, using a lower and upper bounds introduced in the paper. The experiments show a zero-shot transfer improvement in performance compared to baselines (USFA + GPI) on synthetic scavenger tasks and robotic locomotion tasks. ", "questions": "\n\nQuestions\n- Is there a way to extend constrained GPI if the reward cannot be linearly decomposed (but an arbitrary non linear function of the features)?\n- The theoretical and empirical results presented rely on the prior knowledge of the feature vector $\\phi$. How would constrained GPI perform with an unperfected (e.g. learned) feature vector instead?\n\nMinor clarifications and comments:\n- It would be helpful to spell out the different methods acronyms of figures 2 and 3 in the main text or in the caption. \n- I did not understand caption of figure 2b", "limitations": "Very limited discussions on the limitations. In fact, I did not really understand the remark on not getting the \"full effectiveness\".\n\nNo negative societal impacts were reported, and I don't see any myself either. ", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- The paper addresses relevant and important open problem in RL (transfer learning).\n- The writing is clear, rigorous and the paper is easy to follow. A pleasant read!\n- I found theoretical contribution to be main significant result, extending Nemecek and Parr result to about the value of an optimal policy for a new task beyond the conical hull is an important result.\n- The proposed method, constrained GPI, is an elegant way of improving GPI.\n\nWeaknesses:\nThe limitations of the work are not really discussed, and no direction for future work is given. This make it difficult to gauge if the method and its guarantees would extend to more complex and large scale environments.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "nA5z5VOll00", "reviewer": "Reviewer_Jub8", "summary": "Successor Features / Generalized Policy improvement provide a method for generalizing to new tasks that are a linear combination of existing tasks. USFA combine this approach to generalization with neural networks with also generalize by training function approximators conditioned on the task.\n\nThis work extends USFAs by showing that one can establish lower and upper bounds on the generalization performance. When estimating action-values on a new task, these bounds are used to constrain the function approximator estimate.\n\nThey test this approach (comparing to USFAs) on two simple environments and find using these constraints outperforms USFAs.\n", "questions": "Could you comment on the scalability of this approach to larger domains and tasks?\n", "limitations": "Yes although some comments (and ideally testing) on scaling to more complex domains would be useful.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\nStrengths:\n\n- Paper is well communicated.\n\n- Paper will be of interest to community working on successor features and related approaches.\n\n- Paper introduces both theoretical grounding and empirical results.\n\nWeaknesses:\n\n- In common with a lot of work in SF, the test environments are \"toy\" and its unclear whether these approaches scale to more interesting tasks. For this reason, it may be of interest only to a smaller subset of the NeurIPS community.\n\n- It would be interesting to compare very different approaches, such as a conditional decision transformer [1] or GATO style policy distillation [2].\n\n- [3] introduces an alternative method for improving the performance of GPI (in the max-ent RL framework) that reduces the generalization error and should be cited, and ideally compared against.\n\n[1] Chen, Lili, et al. \"Decision transformer: Reinforcement learning via sequence modeling.\" Advances in neural information processing systems 34 (2021): 15084-15097.\n\n[2] Reed, Scott, et al. \"A generalist agent.\" arXiv preprint arXiv:2205.06175 (2022).\n\n[3] Hunt, Jonathan, et al. \"Composing entropic policies using divergence correction.\" International Conference on Machine Learning. PMLR, 2019.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "KU6fLCzDg-u", "reviewer": "Reviewer_Efp4", "summary": "The paper notes that Universal successor feature approximators (USFA) $\\psi(s,a,z)$ (here $z$ is the policy vector) exploits the smoothness of optimal value functions across different tasks but its approximation error could be large when target task vectors are far away. To improve generalization to a new task $w$ (which can be expressed as a linear combination of source tasks), the paper proposes constrained training of successor features that reduces its approximation error on the new task. It then shows that a similar effect can be achieved by using similar constraints at test time without changing the training of the successor features. The proposed method is able to improve task generalization on scavenger and reacher domains.", "questions": "**Recommendation:** I think this is a borderline paper and I am happy to increase my score if the authors can evaluate their method on more tasks (such as the ones in fetch-gym) and compare it to better baselines (such as bilinear value networks).", "limitations": "The authors discuss the limitations of their work in Section 5", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n  1. The resulting *constrained GPI* has good theoretical motivations\n  2. It exhibits better task generalization than GPI baselines\n\n**Weaknesses:**\n  1. The baselines used for comparisons are inadequate. For eg: recent works on bilinear decomposition of Q functions (bilinear value networks (Hong et al., 2022, https://arxiv.org/pdf/2204.13695.pdf)) showed improved generalization on novel tasks. Specifically, for goal condtitioned task with goal $g$, they choose the parameterization $Q(s,a,g) = \\phi(s,a)^Tw(s,g)$ (as opposed to $Q(s,a,g) = \\phi(s,a,g)^Tw$ or $Q(s,a,g) = \\phi(s,a)^Tw(g)$. It would be nice to see comparisons to this bilinear decomposition.\n  2. Furthermore, the domains (i.e. scavenger and reacher) on which the method is tested is limited. I would appreciate if authors can provide a more extensive evaluation on various goal conditioned tasks (such as the ones in fetch-gym (https://github.com/jmichaux/gym-fetch)).\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept