Can Agents Run Relay Race with Strangers? Generalization of RL to Out-of-Distribution Trajectories
Authors: Li-Cheng Lan
Huan Zhang
Cho-Jui Hsieh
Venue: ICLR 2023 poster (2023)
Keywords: Genralization
Reinforcement Learning

Abstract: In this paper, we evaluate and improve the generalization performance for reinforcement learning (RL) agents on the set of ``controllable'' states, where good policies exist on these states to achieve the goal. An RL agent that generally masters a task should reach its goal starting from any controllable state of the environment instead of memorizing a small set of trajectories. To practically evaluate this type of generalization, we propose relay evaluation, which starts the test agent from the middle of other independently well-trained stranger agents' trajectories. With extensive experimental evaluation, we show the prevalence of generalization failure on controllable states from stranger agents. For example, in the Humanoid environment, we observed that a well-trained Proximal Policy Optimization (PPO) agent, with only 3.9\% failure rate during regular testing, failed on 81.6\% of the states generated by well-trained stranger PPO agents. To improve "relay generalization," we propose a novel method called Self-Trajectory Augmentation (STA), which will reset the environment to the agent's old states according to the Q function during training. After applying STA to the Soft Actor Critic's (SAC) training procedure, we reduced the failure rate of SAC under relay-evaluation by more than three times in most settings without impacting agent performance and increasing the needed number of environment interactions. Our code is available at https://github.com/lan-lc/STA.

Published as a conference paper at ICLR 2023
CAN AGENTS RUN RELAY RACE WITH STRANGERS? GENERALIZATION OF RL TO OUT-OF-DISTRIBUTION TRAJECTORIES
Li-Cheng Lan'
Huan Zhang²
Cho-Jui Hsieh*
University of California, Los Angeles Carnegic Mellon University
iclanees, ucla. huan@huan-zhang com chohsiehecs, wela.e............. edu
ABSTRACT
In this paper, we evaluate and improve the generalization performance for rein- forcement learning (RL) agents the set of "controllable" states, where good policies exist on these states to achieve the goal. An RL agent that generally masters task should reach its goal starting from any controllable state of the environment instead of memorizing small set of trajectories. To practically eval- uate this type of generalization, we propose relay evaluation, which starts the test agent from the middle of other independently well-traince stranger agents' trajectories. With extensive experimental evaluation, we show the prevalence of generalization failure on controllable states from stranger agents, For example, the Humanoid environment, we observed that well-trained Proximal Policy Optimization (PPO) agent, with only 3.9% failure rate during regular testing, failed on 81.6% the states generated by well-trained stranger PPO agents. To improve "relay generalization, we propose novel method called Self- Trajectrayy Augmen- tation (STA), which will reset the environment to the agent's ' old states according to the Q function during training. After applying STA to the Soft Actor Critic's (SAC) training procedure, we reduced the failure rate SAC under relay-evaluation by more than three times in most settings without impacting agent performance and increasing the needed number of environment interactions. Our code is available at https://gihubb.oomlln-eeee
INTRODUCTION
Generalization is critical for deploying reinforcement learning (RL) agents into real-world applica- tions. A well-trained RL agent that can achieve high rewards under restricted settings may not be able to handle the enormous state space and complex environment variations in the real world. There are many different aspects regarding the generalization of RL agents.
While many existing works study RL generalization under enviromment variations between training and testing Kirk et al. (2021), in this paper, we study the generalization problem a simple and fixed environment under often overlooked notion of generalization masters a task should be able to start from any "controllable" state in this environment and still reach the goal. For example, self-driving system may need to take over the control from humans (or other AIs trained for different goals such as speed, gas-efficient, or comfortable) in the middle of driving and must continue to drive the car safely. We can make little assumptions about what states the cars are at when the take-over happens, and the delf-driving agent must learn to drive generally. Although this problem may look ostensibly easy for simple MDPs (e.g., 2-D maze), most real RL agents are trained by trajectories generated by its policy, and it is hard to guarantee the behavior of an agent on all "controllable" states. Roughly speaking, we can define state controllable if there exists at least one policy that can lead to high reward trajectory or reach the goal from this state. Unfortunately. most ordinary evaluation procedures of RL nowadays do not take this into consideration, and the agents point with a very small amount of perturbation (Todorov et al. 2012 Brockman et al. 2016). In fact, finding these controllable states themselves for evaluation is difficult.
well-generalized agent that
the setting of robotics (e.g., Mujoco environments),
often evaluated from fixed starting

---

Published as a conference paper at ICLR 2023
1.2
STA, SUCCEN
0.6
0.5
0.6
0.4
0.
02
80
0.0
86
00 0.4
0.0
06 0.5
0.2 0. 2. 1.
(a) States from ordinary evaluation
(b) Relay-evaluation for SAC1
(c) Relay-evaluation for our STA
Figure 1: t-SNE of states from trajectories of 6 Humanoid agents, (a) The states of 6 agents are almost non-overlapping even for the ones trained by the same algorithm: (b) SAC1 agent in (a) performs badly on controllable states from other stranger agents, indicating that it may not learn to control the robot generally; (c) our STA agent performs uniformly well when starting from the same set of states.
The first contribution of this work is to propose relay-evaluation, a proxy to evaluate agent general- ization performance on controllable states in fixed environment. Relay-evaluation involves running an agent from the middle states of other independently trained agents' high-reward trajectories. This is similar to running relay race with another stranger agent, where the stranger agent controls the robot first, and the test agent takes the reins of the robot later. It naturally finds diverse set of controllable states for the test agents because the sampled states come from high reward trajectories of vell-trained agents. The stranger agents can be trained using variety of RL algorithms, not limited to the one used for the test agent.
Our extensive experiments on 160 agents trained in four environments using four algorithms show that many representative RL algorithms have unexpectedly high failure rates under -valuuattt For example, in the Humanoid environment, Proximal Policy Optimization (PPO) agents and Soft Actor Critic (SAC) agents have average failure rates of 81.6% and 38.0% under relay-evaluation even when the stranger agents have trained with the same algorithms (different random seeds), which is surprisingly high compared to the original failure rates (PPO: 3.9%, SAC: 0.92%). The failure of the agents under this setting shows that they may not genuinely understand the dynamics of the environments and learn general concepts like balancing the robot to avoid failing, but rather memorize a small set of actions specifically for a limited number of states encountered. In Figure [aa we illustrate the t-SNE for states in trajectories of 6 agents trained with 3 algorithms and observe that even trained with the same algorithm, the states generated by different agents are quite distinctive. Figure Tb shows that the SAC1 agent only has low failure rate on its own states, which is the dots colored blue Figure
la
Our second contribution is to propose novel training method called Seaj-crayyytory Augmentation (STA) that can significantly improve agents' generalization without significantly increasing training costs. We first conduct motivative experiment to augment the initial state set of agent during training by the states generated by set of pretrained stranger agents, and we find that as we increase the number of stranger agents, the relay eealuuevalttiion of the agent improves significantly. However, pretraining additional models is time-consuming and may be impractical in complex environments. Therefore, propose a novel method called Self-Trajectory Augmentation (STA), where we randomly set the agent to start from its old trajectories. Since the distribution of visited states often varies during training. reviewing an agent's 's old trajectories can be beneficial for generalization. After applying STA to the standard SAC training procedure, the failure rates of SAC agents are reduced by more than three times in most settings without sacrificing agent performance under ordinary evaluation, and it has minimal impact on convergence speed. In Figure Ic our STA agent is uniformly successful on most states from stranger agents.
2 RELAY EVALUATION
In this section, we first introduce the definition of relay-evaluation in Sec Sec2.1 Then we conduct extensive experiments evaluate the geleaggenerliization of representative RI. algorithms in See 2.3
2

---

Published as a conference paper at ICLR 2023
2.1 NOTATIONS
Single-player environments of RL are normally formalized as a Markov decision process (MDP). It can be defined by a tuple M (S, A, T, R. do). where S is the state space, A is the action space. T(s++1|s;, a1): S x A x S + I is the transition function that indicates the probability of reaching state 8,41 after playing action de on state s6; R(8e, at)
x A H RR is the reward function, and (s())) is the distribution of the initial state 8o- With do, we can define the initial state set So as Vs E So + do(s) > 0. We define an agent's 's policy as T(s) which simply outputs the best action. For each (s)), we define its the trajectories distribution as T Note that, trajectory
T is list of tuple [(su, au, ro: $1), (81. (1, V1, 82),..., (sT-1, dr rr- ST)], where By E Sy and for each tuple T(sc+1|st, ae) > 0 and ((s 01) Tt- We named the return of trajectory as the sum of all the rewards of trajectory Σ ri, where T is the length of the trajectory and r, is the reward of playing 0; at 85.
2.2 PROBLEM DEFINITION
The goal of relay-evaluation is to evaluate test agent Tist performance any "controllable" state in the state space S. Here, we define "controllable" state as:
Definition 2.1 (Controllable states) A state s E S is controllable if there exists a trajectory T f...,(s: = =, 02, rz, St4)))
where s is one of its states st, and the trajectory has a high return Ei T; and t <=T L. Here T is the length of the trajectory, L is the number of steps the agent runs after state s. It is used to ensure it is possible to play L time steps starting from s without being terminated by the environment.
Particularly, focus the case of catastrophic failures when starting with these controllable states. In our setting of continuous control environments, we define catastrophic failure as the environment being terminated by the simulator (e-g., the agent completely falls down), but one can also define catastrophic failure as getting an extremely low return. Conceptually, controllable states are those "decent" states where catastrophic failures should not happen. We only evaluate the test agent on controllable states since there may not exist high return policy for a random state since some states are unrecoverable.
To find controllable states, we propose to use independently well-trained agents (which we refer to as "stranger" agents) that can generate high return trajectories. Therefore, in the relay-evaluation, we evaluate a test agent Tiea with the controllable states on the trajectories of another agent "gen- We first use Teen to generate M trajectories and select 1 x M of them with top returns, where rj E [o, 1]. We then sample states 8,, / E {1, ......... - I.} from those high return trajectories and regard those states to be controllable. If the test agent can continue playing for L time steps without failing from state 8t, then we say that the test agent passes the relay-evaluation on this state 5c. After testing Tist on all the controllable states generated by Tgen- we can then compute the failure rate and average return. Note that focus on the failure rates in most of the tables since it's surprising that most agents encounter unexpected high failure rates under relay- elly-evvl,,aiin, while the average return is also reported in some cases to reflect the average performance. For test agent, we conduct relay-evaluation on multiple generating agents, including agents independently trained by the same or different algorithms, as detailed below.
2.3 RESULTS OF RELAY-EVAAE EVALUATION ON EXISTING ALGORITTMS
Experiment setup. We conduct our experiments in four Mujoco environments in OpenAI Gym: Humanoid-v3, Walker2d-v3, Hopper-v3, and Ant-v3 with the standard setting of 1,000 steps. We do not include HalfCheetah-v3 since the simulator does not terminate even when the robot has already fallen over, so there is no standard way to determine catastrophic failures. We first select three popular algorithms, including Soft Actor-Critic (SAC) (Haarnoja et al., 2018), Twin Delayed DDPG (TD3) (Fujimoto et al.) 2018), Proximal Policy Optimization (PPO) (Schulman et al. 2017). as well as two robust training methods aiming to improve agent performance under perturbed observations, SA-PPO and ATLA-PPO (Zhang et al. 2020 2021). Note that the robustness of SA-PPO and ATLA-PPO agents are not aligned with the generalization setting in our paper, but they are included to see if there exists connection between robustness and generalization. All the agents have
3

---

Published as a conference paper at ICLR 2023
Table Failure rates (%) of relay-evaluation using states generated by stranger agents trained with algorithms, reported in the 4 rows for each environment. The "Reference" column shows the failure rate of the stranger agents, serving as the baseline failure rate for these controllable states. Although SAC agents achieve the lowest failure rate, they are still quite high compared to the reference in many environments. TD3 and PPO sometimes have over 90% generalization failure rates.
Envirooment
Stranger Algorithun
Reforence (1)
Test Agent Algorithm (Failure Rate %)
SAC
TD3
PPO
SA/ATLA PPO
SAC
0.92 = 1.78
38,0 + 33,9
83.9
+ 17.6
83,9
16.5
65.1 = 31.8
Humanoid
TD3
0.62 à 1.34
33.6 1 28,5
60.5
à 30.1
78.4 1
20.0
67.5 4: 29.9
PPO
191 4.91
48.8
30.1
77.8
+ 24.3
19.1
63.2 + 30.9
SA-PPO
0.12 0.48
83.8 + 18.0
96.2
+ 5.66
92.9
11.9
77.0 +: 26,4
SAC
0.78 +: 2.49
26.9 :: 23.5
35.9 à: 28.5
87.0 11.4
88.9 :: 11.7
Walker2d
0.31
22.7
28.1
14.0
87.0 12.8
PPO
0.00 + 0.00
14.5 + 12.9
25.0 + 20.0
70.1
+ 21.0
75.1 + 19.3
ATLA-PPO
0.64 + 3,20
19.9 + 18.1
29.4 + 24.0
71.1 +: 21.7
76.2 + 20.0
0.37
32.6
21.1
19.8
40.8 30.4
Hopper
TD3
0.53 + 1.36
31.9 + 36.5
19.8 + 22.1
70.2 17.6
43.8 + 28.9
PPO
0,85 = 2.57
23.8 + 33.5
34,5 + 18.0
56.5
20.7
36.8 = 29.6
ATL.A-PPO
0.01 0.02
30.6 +
31.4 + 20.8
64.1 18.4
39.5 + 31.6
SAC
0.63 + 0.85
2.70 :: 1.98
8.98 1: 9.96
4,00 1: 2.68
3.79 1: 3.09
TD3
1.11 1.43
2.90
2.66
10.0 +: 11.6
3.98 2.13
4.25 + 3.61
PPO
1,03 1.25
2.62 + 1.92
9.95
+ 8.44
3.52
2.07
3:45 + 3,23
ATLA-PTO
1,26 = 1.75
2.32
1 1.67
7.58 4 5,31
3.13
1.96
3.76 +: 3.57
decent average return under the ordinary evaluation which samples the initial state so according to distribution dg. We independently train N = 10 agents for each algorithm in each environment. Every agent takes turns being the test agent and the stranger agent of other agents. For each agent Tgen. we first generate M 200 trajectories and keep the top 100 trajectories that have higher return. For each high-return trajectory, we sample K states for other agents to test.
Generalization Results of Existing Algorithms. For each agent, we conduct relay-evaluation using the trajectories generated by all the other agents as stranger agents, trained by either the same or different algorithms. To investigate the failure rate of test agents trained by Algorithm A when testing the trajectories of stranger agents trained by Algorithm B, we report the average failure rate for each A-B pair in Table I Each row in Table T shows the results of stranger agents trained by a certain algorithm B. The "Reference" column shows the failure rates of the stranger agents starting from their own controllable state, which is very low. This indicates that these states indeed controllable. For the remaining columns, we show the failure rates of relay-evaluation and clearly, we observe that all the agents have significantly higher failure rates than the reference failure rates. For example, in Humanoid, SAC agents have a 38% of failure rate on other SAC agents' trajectories and even higher failure rates when the stranger agent is trained by PPO or SA-PPO. In general, SAC agents have the lowest failure rate across all environments. This is probably due to the entropy term added to their goal function, which promotes exploration. However, according to our experiments, their state distribution still collapses into a small set of successful trajectories (Figure Ta) and can not handle other agents' successful trajectories properly. SA-PPO and ATLA-PPO are robustly-traing agents that perform well under perturbations on state observations. This is reflected in its smaller failure rate on Humanoid and Hopper. However, they perform badly when taking the reins from other agents, showing relay generation is quite different from adversarial robustness since care about all controllable states instead of slightly perturbed states. Additionally, we observe that Ant-v3 is relatively easy because the robot has four legs, which is easier to keep the robot from falling. The hardest environment is Humanoid-v3 since it has the most complicated robot to balance. The failure rate of TD3 even reaches 96.2% on SA-PPO's trajectories.
Why do agents fail? To investigate why the agents perform poorly on relay evaluation, we visualize the state distribution generated by different agents. We select the best two Humanoid agents of SAC, TD3, and PPO with the highest average return from the same agents used in Ta We generate 200 trajectories for each agent and sample 5, 000 states from them. We then use t-SNE (Van der Maaten
& Hinton| 2008) to reduce the dimension of all the states into 2D and visualize them in Fig. Ia We color each dot according to which agent generate the states. For example, the best and the second best SAC agent' states are colored red and blue. We can see that the states of the six agents are well separated, which means that their distributions are very different, even when trained with the same algorithms. The results hold the same in all the environments. This observation suggests that the state
4

---

Published as a conference paper at ICLR 2023
Table Detecting agent failure by calculating Q(s, T(s)). The numbers under the "Failure States"
and "Success States" columns show the results of states that the SAC test agents have failed or succeeded on. The "Reference Return" ("SAC Return") shows the average returns of the stranger agents (test agents, respectively). Q is consistently lower on failure states.
Enviromment
Stranger Algorithm
Failure States
Success States
Reference Return
SAC Retum
Q(s,a(s))
Reference Return
SAC Return
Q(s,
2829 +: 113
136
2743
2733
TD3
2098 + 114
388 + 30G
145+: 144
2088 + 115
2746
= 110
261 166
Humanoid
PPO
2039 179
302 233
61 109
2822 + 171
2734
110
270 191
SA-PPO
3428 1 175
226 127
50 59
3258 1. 92
2758
118
140
161
SAC
3139 1: 292
386 368
298 1: 135
2979 1: 213
2834 1: 257
394 112
TD3
3118 + 350
104 362
316 133
2974
+ 363
2835 259
111
Walker2d
PPO
2058 + 250
312 364
236 100
2504
+ 247
2739
236
334 106
ATT.A-PPO
375
2511
236
SAC
1872 +
1032 188
337 +
1872 +
1872 69
355 29
Hopper
TD3
1850 +
801
55
1835 +
1873
34
PPO
1819 + 78
1055 506
329 1: 10
1808 + 79
1854 71
344 31
ATLA-PPO
107
36
distribution generated by different agents is quite different, with almost no overlap. Therefore all the other agents' trajectories are out-of-distribution for particular agent.
How do the agents fail? We directly observe how the agents control the robot in Mujoco to understand how the agents fail and the implications of passing the relay-evaluation, Fig. shows the snapshots of some different SAC agents. We observe that the agents normally maintain the balance of the upper part of the robot by fixing it to special posture. However, different agents' postures are different, and they use different ways to keep the balance. Therefore, when an agent takes over the control from another agent, it needs to correct the posture to its own. This indicates that if an agent can pass the relay-evaluation it generally knows how to correct the robot without losing balance, even if the starting posture is unseen.
Can the agents detect that it is going to fail? Out-of- distribution (OOD) detection has long been an important question in both RL. and supervised learning, and we hope to identify the states where the agents may fail using value functions. We conduct this experiment has the best performance in most of the settings. Since SAC itself does not include state value function, Q(s, π (s)) to serve as its evaluation of a given state, where Q and are its action-value function and its policy. The results shown in Table are separated into two parts. The "Failure States" part shows the results of states that the test SAC agents failed on, and the states that the test SAC agent passed is shown in "Success States". The "Reference Return" and "SAC Ret" columns show the average return of the stranger agent and the test SAC agent on the states for extra L = 500 steps. The "SAC Q" shows the output (ss, (s)). We can see that the ((s, x (s)) of the failure states is lower than the success states. We also notice that when the SAC agents succeed, they can perform almost well the stranger agents. However, the prediction Q(s, (s)) of the success state is way lower than the prediction (s, (s)) on its own states: Humanoid: 590 - 21, Walker2d: 522 + 73, Hopper: 363 +'2 20, and Ant: 480 + 24. This indicates that the function Q(s, T (()) gives some indications about OOD and failing states. We will use this information to design our algorithms.
SAC since it
use
Figure 2: Different SAC agents running
with different postures in Humanoid.
3 IMPROVING THE GENERALIZATION TO OTHER AGENTS TRAJECTORIES
In this section, we aim to improve the relay-generalization of the existing algorithms. We first investigate naive method in Section 3.1 Next, based on the investigation, we proposed Self-
5

---

Published as a conference paper at ICLR 2023
Trajectory Augmentation (STA) in Section 3.2 methods in Section 3.3
Finally, we conduct the experiments to evaluate both
3.1 A NAIVE METHOD: LEVERAGING PRETRAINED AGENTS' TRAJECTORIES
To improve eeay-generalliztion, naive way is to let the agent "see" at least some of the trajectories generated by other agents during training. One straightforward solution is to add the other agents' tuples (st, at, St+1) into the replay buffer as training data. However, most RL algorithms are not able to handle out-of-distribution actions except offline RL (Levine et al. 2020). Therefore, our naive method lets the agent "see" other agents' trajectories by letting the agent have chances to start playing from the controllable states of some pretrained agents during training. Before training, first pretrain additional Na agents, Next, we use states of those agents' top Thaive = 0.75 trajectories as controllable states "preminn During training, whenever the agent starts to generate new trajectory, besides generating a new trajectory normally, it has Po probability Speetrii from S pretrain as the initial state of this new trajectory. Starting from Spretrain once the agent successfully plays L, steps, we will consider it knows how to play from Spretrain and let the agent start new trajectory. Our naive method is similar training in an augmented environment that has more diverse initial set state. Therefore, we do not need to modify the training algorithm.
uniformly sample state
3.2 SELF-TRAJECTON AUGMENTATION (STA)
Although the naive method can improve the generalization of the agents (Section 3.3), training additional agents is too time-consuming. Hence, we propose Self-Trajectory Augmentation (STA), which only uses the states of the agent's 's own historical trajectories to augment the initial set. The framework STA algorithm is shown in Alg. [ For the variables, Q, π are the agent's 's action value and policy that need to be trained; D is the replay buffer to store the training data; SSTA is the set that stores the historical states during training: do is the original initial state distribution. At each iteration, with probability 1 Po (line 5-7) we generate the trajectory as usual, which means sampling so from do and letting the agent plays for L'max steps (e.g., in Mujoco, T.max = 1, 000). Otherwise, with probabil- ity Po (line 9-10), we select a state form SSTA and let the agent plays for L, steps. Since the failures usually happen
Algorithm I Our STA algorithm
1: Variable: Q, n,, SSTA, 'D
2: Parameter: po. L,, do
3: for each iteration do
4: X~ U(00 1)
5:
if > Pu then
6:
Su ~ do
7:
T =en gen_traaecctory(s, Lmnx)
8:
else
9:
Sn select select_state(SSTr)
10:
T gen_greeneee ren__traectory(So, L,)
11:
for in T do
12:
D D= D=D J [s} is fiis_quaified(s)) then SSTA SSTA U {s}
13:
early after taking over from another trajectory, we found
14:
a small L, < Lumax is sufficient for improving relay gen- eralization. After generating the trajectory, we will store the trajectory into the replay buffer (line 12) and store the qualified state (line 13) into SSTA (line 14).
15: train Q, क with D
Definition of qualified state (Alg. line 13) Ideally, a state is qualified if it is controllable. However, during training, we cannot simply use the return of the trajectory to define if its middle states are controllable. This is mainly due to the exploration during training - the agent may play pretty well at the beginning but ends up having low return because the algorithm tries to explore a bad action. In this case, we still want to include the beginning states into SSTA- Hence, we use the sum of the next À = 50 rewards as the scoring function score(st) = state is controllable. If state's 's score is among the top VISTA ratio of the states generated in the latest epoch Statest- the state will be added into SSTA- For convenience, we maintain a threshold w so that 7ISTA ratio of the states in Statest has a higher score than w. Finally, we define the function is qualified(s) in line 13 as if score(s) > w. Note that if the rewards of the environment are sparse, one might consider other scoring functions.
E r; to measure whether
Select state from SSIA (Alg. line 9). We select state from SYIA with two steps. First, random sample N₂ states that score(s) x 2 Wmax form SSTA as candidates, where Wmax is the largest w encountered during training. Although we only add states with score(s) w into SSTA,
6

---

Published as a conference paper at ICLR 2023
Table 3: The failure rates (%) of the naive baseline with different numbers N₂ of pretrained agents. Training with the states from more pre-trained agents lead to better velay-evaluation performance.
Enviromment
Algoritti
Stranger
Test Algorithm (Pailure Rate %)
SAC
Naivey_-1
Naivex_ =2
Naivey_
Naivex_
Naivex_-1e
SAC
38.0 +: 33.9
55.7 + 16.9
41.5 + 19.7
24.0
: 14.7
11.4 + 7.21
13.2 1: 6,84
Hamanoid
1D3
33.6 + 28.5
40.0 + 13.9
22,9 + 11.5
15.0 + 7.96
7.29
+ 2.79
7.81 + 4.76
59.3
41.5
306
12.0
248
25.0
3.36
SA-PPO
83.8 1: 18.0
90.8 2: 4,31
78.8
5.83
71.9
11.8
59.9
1: 9.60
58.1
8.32
SAC
26.9 4: 23.5
28.6 4: 6.53
20.0 +: 14.6
8.95
5.99
8.14 5.89
6.10
1: 2.34
Walker2d
TD3
22.7 + 21.7
21.0 6.46
17.2 11.3
1.71+ 4.31
4.71 + 3.64
3,43
1.52
PPO
14.5 12.9
14:0 + 5.83
10.1
+ 10.7
3.48
1.66
3.05
+: 2.14
1.95
1.35
AIP.APPO
19.9 + 18.1
19.3 + 4.70
12.0 + 9.00
5:46 + 3,59
3,83
+ 2.24
3,36
1.94
SAC
32.6 1: 36.0
19.8 15.34
10.2 1: 6.23
8.64 1: 5.50
4.01 1: 3.76
4.35 1: 4.73
Hopper
TD3
PPO
23.8 +: 33.5
31.9
173 + 15.77
5.10
8.57 + 7.30
6.38 4: 4.13
7.75
2.81
5,00
+ 2.93
3.14 4.57
5.74
A.P.A.PPO
30.6 + 35.0
22.6 22.65
5.31 + 3.47
9.66 + 7.71
6.46 + 5:77
3,67
3,31
as the agent becomes stronger, w will also increase. Some of the states in SSTA will be considered uncontrollable since their score are too low. Hence, we use ratio y >= 1 to avoid low-score states (< Wmax/ />) being our candidates. For the second step, among the candidates, we want to select a state with which the current agent is unfamiliar. Based on the results in Table we use Q(s, (s)) to predict which state is harder for the current agent. That is, we select the state with the lowest Q(s, T (s)) from the candidates as the new initial state (line 9).
3.3 EXPERIMENTAL RESULTS
In this section, we evaluate the agents trained by our methods under the same settings and strange agents we used in Table[] Since SAC performs best in Table we only apply our methods based on In the following paragraphs, we first show that the naive method can improve the relay-generalization of SAC. Next, we show that our STA can also improve the SAC agent without slowing the training process. Moreover, STA is two times better than the naive method on the hardest controllable states.
The naive method. In this experiment, we evaluate whether using pretrained models can improve the gelaraaenerllzzaiion of an agent. For the experimental settings: we add the controllable states in the top !/avvve = 0.75 ratio of each pretrain agent's 's trajectories into the starting set; we use L, 100 and po R 0.9 to increase the sampling frequency of the starting set. The results of using different numbers N₂ of pretrained agents are shown in Table 3 Although the pretrained agent is trained by SAC, we observe that the failure rates all kinds of algorithms decrease as more pretrained agents are included in pretrain This indicates that the method can successfully improve relay generalization.
However, in Humanoid, the performance has converged when N₂ = 8. We think it is because Humanoid is so complex that adding more agents still cannot cover all the playing styles. Hence, the failure rates do not keep dropping after N₂ 2 8. We also notice that when only using pretrained agent (Na = 1), the failure rate surprisingly increased in the Humanoid environment; this might be because the agent will overfit the only pretrain agent's 's trajectories. Additional results like using different L, = (50, 100, 200, 500) are shown in the Appendix.
Self- Trajecrayy Augmentation (STA). In this experiment, we compare the base SAC, our naive baseline with 16 pretrained agents, and our STA method with tuned parameters (see Appendix). Note that the environment step number of training each agent is all three million. Therefore, the training time of STA will be almost 17 times faster than the naive method.
The comparison of both the average return and failure rate are shown in Table 4 We first show the performance under ordinary evaluation (so do) of each environment ("ordinary" rows). The results show that both the naive method and STA have almost equal or even higher ordinary performances, although agents are trained under augmented MDP, which is different from the original MDP. Moreover, the failure rates of STA in the ordinary setting are also lower than SAC in all the environments, which means that the STA agents are more robust. Next, we show the relay- relay-evaluation results on different stranger algorithms. According to the rest of the rows (except the "ordinary" row) in Table 4 we notice that STA has three times lower failure rates compared to SAC in most of the settings. In addition. since we only sample No = 5 candidates, the training time is almost the same as SAC. Hence, we claim that we achieve a better slaay-eenrralzzttoon than SAC without losing the ordinary performance and the converging speed. We also compare the results between STA
7

---

Published as a conference paper at ICLR 2023
Table 4: The returns and the failure rates of SAC, our naive baseline, and STA under relay-evaluation. Since all the agents are trained with three million environment interactions, training an STA agent is 15.9 times faster than a Naiven, agent. The "ordinary" rows show that STA will not decrease the regular performance. The rows except "ordinary" show that STA can achieve significantly lower failure rates under relay-evaluation compared to SAC and achieve performance similar to N.i
Stranger Algorithm
SAC
Nuive -6 (ours)
STA (ours)
Environment
Return
Failure Rate
Return
Failure Rate
Return
Failure Rate
ordinary
5650 +: 238
0.62 + 0.79 %
509G
159
2.65 # 2.52 %
5899 + 346
0.45 : 0.61 %
Humanoid
SAC
2004 + 1193
38.0 + 33,9 %
2591 + 811
13.2 : 33.8 %
2815 = 622
5.77 + 23.3 %
TD3
2242 = 1071
33.6 + 28.5
2731 621
7.81 26.8
2862 504
3.76 19.0 %
PPO
1675 + 1266
48.8 + 30.1 %
2266
+ 1120
23.0 %
2602 = 793
9,95 + 29.9 %
PPO
671 + 996
83.8 +: 18.0 %
1382
1343
58.1 + 49.3
2312 1198
25.2
43.4 %
ordinary
5717 + 445
0.16 + 0.52 %
5986
0.00
0.00
5736 = 422
0.00
0.00 %
SAC
2286 + 1138
26.9 + 23.5 %
2931 + 658
5,38 : 22.5 %
2784 = 725
7.33 = 26.0 %
Walker2d
TD3
2326 + 1078
22.7 + 21.7 %
2965 +: 537
3,71 + 18.9 %
2822 = 632
5.48 +: 22.7 %
PPO
2464
14.5 +:
2931
1.19 10.8
2798 473
2.57
%
ATLA-PPO
2314 + 1007
19.9 + 18.1 %
2888 + 449
2.33
15.0 %
2782 + 488
2.83 + 16.5 %
ordinary
3602 = 137
5.75 + 8.85 %
3609 + 60
0.34 = 0.89 %
3686 = 58
0.42 1 0.77 %
Hopper
SAC
1777 +: 390
32.6 + 36.0 %
1846 148
2.59
15.8
1818 + 323
9.80 +: 29.7 %
TD3
1654 :: 594
31.9
+ 36.5 %
1838 265
3.05
17.1
1790 428
13.8 34.5 %
PPO
1790 +: 337
23.8 + 33.5 %
1832 4: 140
2.62
15.9 %
1829 + 245
8.52
+: 27.9 %
ATLA-PPO
1711
35.0
1824 257
21.1
1841
276
10.2
30.3 %
Table 5: Failure rates (%) of STA with different No, where N₂ is the number of starting candidate states we consider before starting trajectory. State with the lowest ((s, (s)) is chosen among the Ne candidates to start the agent. A larger No produces better performance with negligible cost.
Stranger
Test Algorithm (Failure Rute %))
Environment
Algorithm
SAC
NaiveN,-18
ST
STAN.-2
STAN.-5
SAC
38.0 = 33,9
13.2 + 6.84
19.7 + 18.1
8.52 + 5.74
5.77 = 3,06
Humanoid
TD3
33.6 + 28.5
7.81 + 4.76
9.29 + 7.94
3.48 + 2.89
3.76 4.49
PPO
48.8 = 30.1
2
26.1 + 10.6
15.7 5.25
9.95 5.38
SA-PPO
83.8 = 18.0
58.1 +: 8.32
7.44
25.2 6.47
SAC
26.9 : 23,5
5.38 +: 3.58
20.1 12.2
1:.67
9,62 3.72
TD3
22.7 + 21.7
3.71 + 2.48
16.3 10.3
13.1 13.1666116
10.0 5.81
Walker2d
PPO
14.5 = 12,9
1.19 0.86
1.35
8,95 6.09
5,90 2.64
ATLA-PPO
19.9 = 18.1
2.33
0.99
1.78
10.4 = 7.28
3,90 7.67
SAC
32.6 + 36.0
2.59
+ 2.41
13.61 :: 10.2
19.8 + 16.0
14.1 8.12
TD3
31,9 - 36.5
3.05
1: 5.03
22.1 21.0
20.9 1 17.3
17.6 12.3
Hopper
23.8 33.5
2.62
12.6 14.3
18.1
12.8 12.5
ATLA-PPO
30.6 = 35.0
4.69
+: 4.01
24.2 :: 20.5
23.5 = 21.8
21.2 16.0
and the naive method. We find out that our method has a better performance in harder environments. For example, STA has only 25.2% failure rate on the Humanoid SA-PPO. However, in the casiest environment, Hopper, the naive method with N. = 16 pretrained agents has twice lower rate than STA. We suggest it is because Hopper has the simplest robot that only has one lag. Hence, we suggest that most of the hoping styles have similar version in the 16 pretrained agents. Therefore, the naive method can easily play on other agents' trajectories.
Besides showing the best performance of STA, we also conduct some ablation studies. First, we directly observe how STA takes over the control of other agents. Same as SAC agents, we notice that STA agents also run with one kind of posture. Even when the STA starts from other agents' posture, it will change it into its own and keep running. We consider this a good thing. For example, if a test agent faster than the stranger agent, we will want the test agent to run in its own way, which is faster. We also conduct an experiment on different numbers of candidates Ne to show the importance of starting the trajectories from the states that has a lower Q(s, Tss)). The results are shown in Table 5 We first look at the column where N₂- = 1. It shows the results of simply randomly selecting a state from the initial set as we did in the naive method. Compared to SAC, N₂ is only slightly better. However, method, especially on Humanoid. We also notice that different N₂ have little effect on Hopper. This is because there is only a small ((s, n (s)) difference between the familiar states and the unfamiliar state according to Table Hence, we should consider other methods in Hopper.
larger N... the result becomes comparable to the naive
8

---

Published as a conference paper at ICLR 2023
4 RELATED WORK
Many methods have been designed to evaluate the generalization of an RL agent (Kirk et al. 2021) They can be categorized by which part of the RL environment different between training and testing, including (1) the environment, (2) the observation emission function, (3) the transition function, (4) the reward function, and (5) the initial state set. For (1), researchers studied training and testing in totally different environments (Cobbe et al. (2019), and Dennis et al. (2020). For (2), researchers change the surface of the objects in the environment (Tobin et al. 2017; Zhang iet al.) 2018b; Zhao et al., 2019; James ct al., 2019). For (3), researchers change the parameter of the transition function like the gravity strength or the weight of the ball, or add an external force to the agent. (Packer et al., 2018 Pinto et al. 2017) For the (4), researchers (Mishra et al. Seevaaier-Boisveree
Sermanet] (202)) normally want to see if the agent can adapt quickly from one task to another with the general knowledge learned in the first task. Finally, the last category (5), which only selects different initial states in the same environment (Portelas et al.) 2019 Peng et al. 2018 Kaplanis et al.] (018) is where our relay-gent geaaaaygenalliztiion belongs. In one of the most related previous work. Zhang et al. (2018a) only train limited states in the initial set dg and test on the other states in the initial set dg. Their results show that if a PPO agent only trains on less than 5 states, it will be hard to avoid failing on other states in the initial set dg- In addition, our generalanliiinnioonn belongs to Out-of-Distribution (OOD) generalization, where training and testing states are sampled from different distributions (See Figure aaland Figure 3). To the best of our knowledge, the notion of controllable states and relay evaluation has not been discussed in the literature.
2018 Dosovitskiy et 2017]
21.1
Beyond the generalization of RL, many other RL domains are related to our works. First, in hierarchical reinforcement learning, |Pateria et al. (202T); Brochu et al. (2010); Nachum et al. (2018) train multiple agents with different small missions and let them complete a task together by taking turns. However, in such scenario, the agents are trained together. Therefore, they are familiar with previous agents' trajectories. Second, both our naive method and the offline RL (Levine et al. 2020 Kumar et al. 2020 Ostrovski et al. 22221) leverage the trajectories of other agents. Third, population- based training (Jaderberg et al.] 2017: Derek & Isola 202T) can be a potential way to generate controllable states and increase saluyyeerrllliztionn Finally, the adversarial robustness of agents has been studied in the context of state, observation, and reward perturbations (Huang et al. [2017) Ilahi et al. 2021; iZhang al.. 2020 2021; Ding al., [0222] Lan et al. 2022). However, these studies focus on the performance of agents under imperceptible adversarial perturbations. On the other hand, the states of other stranger agents can be very different from the agent's trajectory distribution. In addition, our work shows that robust agents (SA/ATLA PPO) are still bad at generalization, which is similar some of the previous results (Korkmaz 202T).
Revisiting observed states strategy (or local access protocol Yin et al. 2021 is used in our STA algorithm. This strategy has been used in many RL algorithms, Ecoffet al. (2019) surpasses the state-of-the-art on on hard-exploration games by revisiting promising states in history. Tavakoli et al. (2020) shows that revisiting states on high return trajectories can improve PPO on a sparse-reward task. After our publication, Yin et al.) (2023) proposed revisiting the historical states that the agent is uncertain. Different from these works, our STA algorithm uses the Q function to select states that the agent used to be good at and now is not confident with.
CONCLUSION
We propose relay-evaluation, proxy to evaluate RL agents' generalization performance on con- trollable states in a fixed environment. Relay-evaluation involves running an agent from the middle states of other independently-trrinn -trined agents' high-reward trajectories. Extensive studies under the MuJoCo environment demonstrate that many representative RL algorithms have unexpectedly high failure rates under relay-evaluation, indicating a significant limitation of existing RL. methods. To overcome this challenge, we propose novel method called Self Seif-Trajyctory A Augmentation (STA), which improves the generalization of existing RL agents without impacting ordinary performance. This paper opens up new research direction in RL, and there are several limitations to be addressed in the future. For instance, how to find unexplored controllable states more effectively and how to "certify" that an agent can work (or at least won't fail) on all controllable states are interesting future challenges. More importantly, how to develop better algorithms to improve relay generalization is still open problem.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGEMENT
This work is supported in part by NSF under IIS-2008173, IIS-2048280 and an Okawa research grant.
REFERENCES
Joshua Achiam. Spinning Up in Deep Reinforcement Learning. 2018.
Eric Brochu, Vlad M. Cora, and Nando de Freitas. tutorial on bayesian optimization of expensive cost functions, with application to active user modeling and hierarchical reinforcement learning. ArXiv, abs/1012.2599, 2010.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. ArXiv, abs/1606.01550,, 2016.
Maxime Cheselier-ooisvert, Dzmitry Bahdanau, Salem Lahlou, Lucas Willems, Chitwan Saharia, Thien Huu Nguyen, and Yoshua Bengio. Babyaï: A platform to study the sample efficiency of grounded language learning. In ICLR, 2019.
Karl Cobbe, Oleg Klimov, Christopher Hesse, Taehoon Kim, and John Schulman. Quantifying generalization in reinforcement Icaaming ArXiv, abs/1812.02341, 2019.
Michael Dennis, Natasha Jaques, Eugene Vinitsky, Alexandre M. Bayen, Stuart J. Russell, An- drew Critch, and Sergey Levine. Emergent complexity and zero-shot transfer via unsupervised environment design. ArXiv, abs/2012.02096, 2020.
Kenneth Derek and Phillip Isola. Adaptable agent populations via a generative model of policies. Advances in Neural Information Processing Systems, 34:3902-3913, 2021.
Qin Ding, Cho-Jui Hsich, and James Sharpnack. Robust stochastic linear contextual bandits under adversarial attacks. In International Conference on Artificial Intelligence and Statistics, pp. 7111-7123. PMLR, 2022.
Alexey Dosovitskiy, Germán Ros, Felipe Codevilla, Antonio M. López, and Vladlen Koltun. Carla: An open urban driving simulator. ArXiv, 0tss/1711.0393,, 2017.
Adrien Ecoffet, Joost Huizinga, Joel Lehman, Kenneth O. Stanley, and Jeff Clune. Go-explore: new approach for hard-exploration problems. ArXiv, hts//991..19999 2019.
Logan Engstrom, Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Firdaus Janoos, Larry Rudolph, and Aleksander Madry. Implementation matters in deep rl: A case study on ppo and trpo. In International conference learning representationss 2019.
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. ArXiv, abs/1802.094777, 2018.
Tuomas Haarnoja, Aurick Zhou, P. Abbeel, and Sergey Levine, Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In ICML, 2018.
Sandy Huang, Nicolas Papernt, Ian Goodfellow, Yan Duan, and Pieter Abbeel. Adversarial attacks on neural network policies. arXiv preprint
2017.
Inaam Ilahi, Muhammad Usama, Junaid Qadir, Muhammad Umar Janjua, Ala Al-Fuqaha, Dinh Thai Hoang, and Dusit Niyato. Challenges and countermeasures for adversarial attacks deep reinforcement learning. IEEE Transactions on Artificial Intelligence, 3(2):90-109, 2021.
Max Jaderberg, Valentin Dalibard, Simon Osindero, Wojeiech M Czarnecki, Jeff Donahue, Ali Razavi, Oriol Vinyals, Tim Green, lain Dunning, Karen Simonyan, et al. Population based training of neural networks. arXiv preprint urXiv:1711.09844 2017.
Stephen James, Paul Wohlhart, Mrinal Kalakrishnan, Dmitry Kalashnikov, Alex Irpan, Julian Ibarz, Sergey Levine, Raia Hadsell, and Konstantinos Bousmalis. Sim-to-real via sim-to-sin tim: Data- efficient robotic grasping via coloomized-t--cnnooee adaptation networks. 2019 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 12619-12629, 2019.
10

---

Published as a conference paper at ICLR 2023
Christos Kaplanis, Murray Shanahan, and Claudia Clopath. Continual reinforcement learning with complex synapses. ArXiv, abs/1802.07239, 2018.
Robert Kirk, Amy Zhang, Edward Grefenstette, and Tim Rocktaschel. A survey of generalisation in deep reinforcement learning. ArXiv, 0bbs221110997// 2021.
Ezgi Korkmaz. Adversarial training blocks generalization in neural policies. 2021.
Aviral Kumar, Aurick Zhou, G. Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. ArXiv, abs/2006.04779 2020.
Li-Cheng Lan, Huan Zhang, Ti-Rong Wu, Meng- Yu Tsai, Wu, Cho-Jui Hsich, et al. Are alphazero- like agents robust adversarial perturbations? arXiv preprint www.v:22/1.03766 2022.
Sergey Levine, Aviral Kumar, G. Tucker, and Justin Fu. Offline reinforcement learningg Tutorial, review, and perspectives on open problems. ArXiv, abs/2005.01643. 2020.
Corey Lynch and Pierre Sermanet. Language conditioned imitation learning over unstructured data. Robolics: Science and Systems XVII, 2021.
Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, and P. Abbeel. A simple neural attentive meta-leanen In ICLR, 2018
Ofir Nachum, Shixiang Shane Gu, Honglak Lee, and Sergey Levine. Data-efficient hierarchical reinforcement learning. In NeurIPS, 2018.
Georg Ostrovski, Pablo Samuel Castro, and Will Dabney. The difficulty of passive learning in deep reinforcement learning. Advances in Neural Information Processing Systems, 34:23283-23295, 2021.
Charles Packer, Katelyn Gao, Jernej Kos, Philipp Krähenbühhl, Vladlen Koltun, and Dawn Xiaodong Song. Assessing generalization in deep reinforcement learning. ArXiv, abs//810. 12.82, 2018.
Shubham Pateria, Budhitama Subagdja, Ah-hwee Tan, and Chai Quek. Hicrarchical reinforcement learning: A comprehensive survey. ACM Computing Surveys (CSUR), 1-(5):1-35, 2021.
Xue Bin Peng, Marcin Andrychowicz. Wojeiech Zaremba, and P. Abbeel. Sim-to-real transfer of robotic control with dynamics randomization. 2018 IEFE International Conference Robotics and Automation (ICRA), pp. 1-8, 2018
Lerrel Pinto, James Davidson, Rahul Sukthankar, and Abhinav Kumar Gupta. Robust adversarial reinforcement learning. In /CML,, 2017.
Rémy Portelas, Cédric Colas, Katja Hofmann, and Pierre Yves Oudeyer. Teacher algorithms for curriculum learning of deep rl in continuuulyy parameterized environments. In CoRL, 2019.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov, Proximal policy optimization algorithms. ArXiv, abs/1 0es//707.06633 2017.
Arash Tavakoli, Vitaly Levdik, Riashat Islam, Christopher M. Smith, and Petar Kormushev. Exploring restart distributions. arXiv: Learning, 2020.
Joshua Tobin, Rachel Fong, Alex Ray. Jonas Schneider, Wojciech Zaremba, and P. Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world. 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 23-30, 2017.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 0026-5033, 2012.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008
Dong Yin, Botao Hao, Yasin Abbasi- Yadkori, Nevena Lazi'c, c, and Csaba Szepesvari. Efficient local planning with linear function approximation. ArXiv, 01121008.05333. 2021.
11

---

Published as a conference paper at ICLR 2023
Dong Yin, Sridhar Thiagarajan, Nevena Lazic, Nived Rajaraman, Botao Hao, and Csaba Szepesvari. Sample efficient deep reinforcement learning via local planning. ArXiv, abs/2301. 2023.
Amy Zhang, Nicolas Ballas, and Joelle Pineau. A dissection of overfitting and generalization in continuous reinforcement learning. ArXiv, abs/1806.0799) 07937, 201 8a.
Amy Zhang, Yuxin Wu, and Joelle Pineau. Natural environment benchmarks for reinforcement learning. ArXiv, abs/181 111111111111 .06032, 2018b.
Huan Zhang. Hongge Chen, Chaowei Xiao, Bo Li, Mingyan D. Liu, Duane S. Boning, and Cho-Jui Hsieh. Robust deep reinforcement learning against adversarial perturbations on state observations. arXiv: Learning, 2020.
Huan Zhang, Hongge Chen, Duane S. Boning, and Cho-Jui Hsieh. Robust reinforcement learning state observations with learned optimal adversary. ArXiv, abs/2101.08452, 2021.
Chenyang Zhao, Olivier Sigaud, Freek Stulp, and Timothy M. Hospedales. Investigating generalisa- tion continuous deep reinforcement learning. ArXiv, abs/1902.00115, 2019.
12

---

set the Published as a conference paper at ICLR 2023
A DETAILS FOR EXPERIMENTAL SETUP
For SAC and TD3, we follow the implementation of OpenAl Spinning Up (Achiam 2018 which uses networks of size (256, 256) with relu units and 3 million environment interactions. For PPO, we follow the implementation of Engstrom et al. 2019 since its PPO performance is higher than OpenAI Spinning Up. The total number of environmental interactions is one million. For SA-PPO and ATLA-PPO, we follow the implementation in (Zhang et al. (02T). SA-PPO and ATLA-PPO are representative algorithms in terms of observation robustness. That is, the agent can function normally even if a small adversarial perturbation is added to the observation. We include this kind of robust agent to show that our problem is different from the normal adversarial robustness problem in RI.. For the humanoid environment, the SA-PPO algorithm we used is the SGLD algorithm. For other environments, we use ATLA-PPO.
For the naive method, both the pretrained agents and the agent follow the implementation of SAC of OpenAI Spinning Up. Hence, training a naive method with N, agents requires No + 1 times of training time. The default parameter settings Thavie 0.75 since want to include harder states; Po RJ 0.9, L, = 100 since we want to sample more initial states form Sthme
pretrain
For the STA method, we also follow the implementation of SAC of OpenAI Spinning Up. Since the candidate number Nr
is less than 10, the training time is almost the same training SAC agent. The default parameter settings are: TISIA = 0.75 since we want to include harder states as we did in the naive method; Po R 0.9, L,, 100 since want sample more initial states form SSTA: X = 50 < L., since we want to Iucu the states in the trajectories that starts from states in SSTA. For the STA parameters used in Table 4 we the number of candidates N₂
qualifying ratio y = 1.0 in Humanoid and 1.6 in Walker2d and Hopper.
B EXPERIMENTS OF OUR NAIVE METHOD WITH DIFFERENT PLAYING LENGTHS
In this section, we show the results of our naive method with different limitations of the extra steps of the trajectories starting from starting set.
The results are shown in Table 6 According to the table, the failure rate will increase when L, is too small or too large. Since L, = 100 has the lowest failure rate in almost all settings, we will use L, 100 as the default in other experiments.
Table 6: The results of the naive method with different time step limitation L,.
Environment
Stranger Algorithm
Algorithm
%)
SAC
Na0ver,.
Naiver,
Naiver,
Naiver...
SAC
38.0 = 33.9
30.4
34,9
13.2
6.84
14.3 : 6.00
14.76 5.08
TD3
33.6 = 28.5
25.7 = 37.3
7.81 : 4.76
10.6 00,6230 3.74
11.76 2 6.61
Humanoid
PPO
48.8 = 30.1
40.6
30.0
25.0 1: 3.36
25.0 5.04
28.36 : 7,31
SA-PPO
83.8 = 18.0
69.7
15.7
58.1 = 8.32
61.8 12.2
65.43 10.8
SAC
26.9 23.5
6.10
2.34
5.38 3.58
4.48 3.15
6.52 2.87
Walker2d
TD3
22.7
21.7
3.43
1.52
3.71
2:
3.14 = L80
3.48 :: 2.20
PPO
14.5 12.9
1.95
= 1.35
1.19 0.86
1,81
1.06
2.81 = 1.97
AILA
19.9
18.1
3.36 1.94
2,33 1 0.99
1.24 2.88
3,78 2.24
SAC
32.6 = 36.0
4.35
4.73
2.59
:: 2.41
7.96
5.68
7.82 = 6.54
Hopper
TD3
31.9
36.5
3.76
L. 5.74
3.05 5.03
5.19 11991444 4.37
6.52 1 9.11
PPO
23.8 = 33.5
3.14 4.57
2.62 2.95
5.86 5.15
3.71 6.79
ATLA
30.6 35.0
3,67
3,31
4.69 44001
6,05 = 3.88
5.85 + 7.50
13

---

Published as a conference paper at ICLR 2023
C EXPERIMENTS OF STA WITH DIFFERENT CANDIDATE CRITERIA
In this section, we evaluate the performance of using different 7. In STA, we use Winar / to serve as the criteria be the threshold of being one of the Ne candidates, Only states with score wmex/ / in SSTA can be the candidate of the initial state 5o. If is larger, then states with lower scores can also be the candidates.
The results different with N: = 5 are shown in Table 7 According to the results, the best of each environment different. For example, in Humanoid, it is important only samples the states that have higher scores as candidates (small 7).
Table The results of STA with different y. When y is larger, the states with lower scores can also be the candidates of being the initial state of trajectory during training STA.
Environment
Stranger Algurithm
Algorithm (Failure
%)
SAC
Naive
1.1
1.3
SAC
38.0 33.9
13.2 + 6.84
5.77 + 3.06
4,44 = 5.34
8.36 + 5.62
9,10 5.55
Humanoid
TD3
33.6 28.5
7.81
+ 4.76
4-76+44.49
2.24
2.67
6.08 +-330
3,62 + 3.56
PPO
48.8 30.1
25.0
+ 3,36
9.95 5,38
12.0
15.8
12.4 2.66
SA-PPO
83.8
18.0
58.1
8.32
8.32
6.77
33.2
15.2
48.5 11.4
43.6
SAC
26.9 23.5
9.62 3.72
11.5
6,52 3.92
7.33 - 4.62
Walker2d
TD3
22.7 21.7
3,71 2.48
10.0
1155=
5.48 3.35
PPO
14.3 = 12.9
1.19
+ 0.86
5,90 2.06
7.00
5,24 1 4.69
2.57
: 3.25
ATLA
18.1
2.33 +
7.67 3.90
8.17
5:27
2.83
2.04
SAC
32.6
36.0
2.59
2.41
14.1 8.12
23.6 15.1
23.2 10.6
9,80
4.51
Hopper
PPO
23.8
33.5
2.62 1 2.95
12.8 1: 12.5
20.2 16.3
21.2 1 12.6
8,52 6.04
D INFINITY REPLAY BUFFER
In this section, we show that our STA method is better than using an unlimited-size of replay buffer. The original replay buffer size is million. The maximum size of an unlimited-size of replay buffer is three million since we only have three million environment interactions. The results are shown in Table According the results, SAC with infinite replay buffer still has poor performance relay evaluation compared to our method. Moreover, the average return under ordinary evaluation is lower than the normal SAC, while our method is greater or equal to the normal SAC.
Table 8: SAC with infinity replay buffer during training.
Stranger Algurithm
Test Algorithm (Failure
Environment
SAC
Naivex,=1u
SAC buffer
SAC
38.0 + 33.9
13,2 + 6.84
5.77
23.3
34.07 10.12
Humanoid
TD3
33.6 + 28.5
7.81 + 4.76
3.76
19.0
24.00 15.67
PPO
48.8 + 30.1
25.0 + 3.36
9.95 + 29.9
43.39
12.31
SA-PPO
83.8 + 18.0
58.1
+ 8.32
25.2 + 43.4
80.10 + 8.43
SAC
26,9 + 23.5
5.38 3.58
7.33 4: 2600
41.33 26.87
TD3
22.7 + 21.7
2.7122.4
5.48 4 2:
38.19 27.21
Walker2d
PPO
14.5 12.9
1.19
0.86
2.57
15.8
24.57 23.94
ATLA-PPO
19.9 18.1
2.33 0.99
2.83
16.5
28.47 25.46
14

---

Published as a conference paper at ICLR 2023
E USING AVERAGE REWARD AS SCORING FUNCTION
In this section, we show the STA result of using the average score score(st) = Ei r instead of using the reward sum of next steps score(s+) Σi r;. In STA, the scoring function a quick way to estimate a state is controllable. We only add the states that have high enough scores into the SSTA- According the results, using the average reward can also reduce the failure rate compared to the original SAC. However, its failure rate is still higher than using our score function (column STA), which uses the sum of the reward the next step.
Table 9: The experiment of STA that use average rewards score(s;) = Eime r2/(I as the score function estimate if state in SSTA is controllable.
Stranger
Test Algorithm (Failure Rate %8)
Environment
Algorithm
SAC
Naive
STA
STAywmpe reward
SAC
38.0 = 33.9
13.2 + 6.84
5.77 + 23.3
8.36 3.52
TD3
33.6 1 28.5
7.81 1: 4.76
3.76 1: 19.0
4.86 1: 2.30
Humanoid
PPO
48.8 = 30.1
2
9.95 + 29.9
11.85 = 4.72
SA-PPO
83.8 = 18.0
58.1 1 8.32
25.2 1 43.4
44.29 +: 11.14
F THE FAILURE DEFINITION OF MUJOCO
In Mujoco, by default, most of the environments will terminate the simulation when the agent is unhealthy. Normally, unhealthy means the robot has fallen over. By doing so, the agent can start new trajectory without wasting more time on the "unhealthy" trajectory. Each environment has its own way of detecting unhealthy.
The Humanoid environment requires the z-position (height) of the robot need to be in a predefined range.
The Ant and Walker2d environment define the robot as unhealthy any of the following happens:
Any of the state space values is longer finite.
The height of the walker is not in predefined range.
The absolute value of the angle is not in predefined range.
The Hopper environment defines the robot as unhealthy if any of the following happens:
An element of observation[1: I:: is not in predefined range.
The height of the walker in predefined range.
The absolute value of the angle is not in predefined range.
Please see more details at http://wwww.mmmiiiieee///oroooen
15

---

Published as a conference paper at ICLR 2023
G DEFINE FAILURE WITH RETURNS
In this section, we define the failure of relay-evaluation according to the reward sum the next L 500 steps instead of using the failure defined by Mujoco. The first column shows the failure rate using the failure definition of the Mujoco. For the remaining columns, we allow the agent to keep playing even if the current state "failed," according to Mujoco. In the end, we define trajectory as a failure trajectory according to its return. For example, for the second column, we define trajectory as failed when return lower than 500. Note that the average return of SAC should be 2828. According to the results, no matter using which kind of definition of failure, SAC still has an unacceptable failure rate, especially on the SA-PPO. This shows that the termination criteria in the MuJoCo environments are not too strict and most of the terminated trajectories have chance to get a high score.
Table 10: Conducting relay-evaluation with different definitions of failure. The column MuJouo uses the failure defined by the environment. The columns ShC define a trajectory is failed if the return value smaller than the threshold thr. Note that the expected return value greater than 2800.
Stranger
Test Algorithm (Faillure Rate %))
Environment
Algorithm
SAC
SAC.coo
SAC,1000
SACx1500
SAC
38.0 :: 33.9
30.56 +: 11.11
32.28
11.12
32.54 :: 10.95
TD3
33.6 28.5
21.55 = 5.77
23.93 +: 5.67
24.09 5.76
Humanoid
PPO
48.8 30.1
42.42 + 10.02
44.93
10.41
45.33 + 10.48
SA-PPO
83.8 18.0
81.83 9.29
82.82 + 9.03
82,98 9.04
H STATE DISTRIBUTION OF AN STA AGENT'S OLD TRAJECTORIES
In this section, we plot the historical states of an STA agent with the states we showed in Fig la As the figure shows, although the distribution of historical states of the STA agent the distribution of other agents' states, there are still many areas that does not cover. Hence, this suggests that the STA agents have better performance on relay evaluation due to being more general.
diverse than
Figure 3: State distribution of an STA agent's SSTA- SSTA stores the likely controllable states that the agent generated during training. We use t-SNE to visualize those states along with the states generated by the other six agents used in lag.
1.0
SAC_1 1
SAC 2
TD3
TD3 2
PPO
PPO 2
STA
0.0
0.2
0.4
0.6
0.8
1.0
16

---

Published as a conference paper at ICLR 2023
I EVALUATING SAC WITH DIFFERENT L IN RELAY-EVALUATION
In this section, we conduct relay-evaluations with different L. We select SAC as our test agent. Our result is shown in Table [T] The results show that most of the failures happened in the first 100 steps after the test agent took over the control. The results also show that for some of the cases, even if the agent does not fail for 100 steps, it may still fail in the next 400 steps.
Table 11: Evaluating SAC with different in relay-cvaluation, where L is the extra steps that the test agent needs to complete without failing.
Stranger
Test Algorithm (Failure Rate %)
Environment
Algorithm
SAC1-500
SAC2-200
SAC1-190
SACL-50
SAC
38.0 = 33,9
32.32 11.08
30.69
10.92
21.12 7.25
TD3
33.6 = 28.5
23.97 + 5.71
21.71 + 6.22
11.19 6.099
Humanoid
PPO
48.8 + 30.1
45.06 + 10.59
42.50
10.26
28.84 + 7.81
SA-PPO
83.8 = 18.0
82,02 + 9.05
73.77 = 9.60
J GENERATING CONTROLLABLE STATES WITH DIFFERENT HYPERPARAMETERS
In this section, our goal is to generate more controllable states to test our agents by training with different hyperparameters. We first train 40 SAC agents with random hyperparameters and use the top 10 agents to serve as stranger agents.
The hyperparameters that will randomly choices are: the learning rate of the policy æ, the learning rate of the Q function, the batch size, the network structure, the training frequency, and the a in the SAC goal function.
The results are shown in Table 12 random hyperparameters are easier for other agents pass the relay-evaluation, especially when evaluating SAC and TD3 agents. A potential reason is that some of these hyperparameters lead to worse policies and generate slower trajectories, and it's easier for the target agent take over from those slower states.
According to Table 12, we observe that the states generated by
Table 12:
We generate controllable states by training SAC with random hyperparameters, including learning rates, network sizes, and batch size.
Stranger Algorithm
Test Agent Algorithm (Failure Rate %
Environment
SAC
TD3
PPO
SA-
PPO/AA/A
SAC
38,0 33,9
83.9
17.6
83.9
16.5
65.1
31.8
TD3 PPO
33,6 : 28.5
60,5
+ 30.1
78.4 2 20.0
67.5
1 29.9
Humanoid
48.8 30.1
77.8
24.3
81.6
19.1
63.2
30.9
SA-PPO
83.8 + 18.0
96.2
5.66
92.9
11.9
77.0 + 26.4
SAC Random- Hyperpaameeeess
16.6 :: 37.2
55.8 1 49.6
79.5 1 40.3
65.0 1: 47.6
K THE ORDINARY RETURN OF THE TEST AGENTS IN TABLE
This section shows the average returns of the agents we used in Table
Table 13: The avg return of the test agents we used in Table
Environment
SAC
TD3
PPO
SA/ATLA PPO
Humanoid
5645.73 1: 233.54
5011.30 : 1515.59
5173.36
236.10
6614.09 :: 340.69
Walker2a
5715.64 + 446.39
5588.48
701.70
4184.67
897.59
3636.84 1140.55
Hopper
3667.81 1: 143.62
3520.63 1: 177.80
5384. 12 + 144.40
4587.02 :: 271.10
6007.45
770.93
3148.41
404.15
17


Reviews:
[{"review_id": "wwBQqdr7ZPx", "reviewer": "Reviewer_vBmJ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors propose a new experimental setting for RL called relay evaluation, which tests the ability of the RL agent to handle the trajectory of other agents (strangers) in the mid-way. Such a situation may be helpful in the application of RL when the agent needs to take control from humans and is seldom considered in the literature. It turns out that current RL algorithms can easily fail in the relay evaluation. Therefore, the authors propose a naive framework and a self-trajectory augmentation framework to improve the generalization ability of the RL agent, with any RL algorithm in theory. They use the trajectories, either trajectories from other stranger agents or historical trajectories to enhance the RL agent. The results on multiple datasets with different RL algorithms show the performance of the proposed frameworks.", "summary_of_the_review": "This paper aims to solve the ray evaluation issue which is hardly considered in the literature. However, except for this point, the other parts of this paper look not good enough to me. As I mentioned in the weakness part, the authors need to clarify their motivation and enhance their experiments. Therefore, I consider this paper is not yet ready for ICLR. Admittedly, I am not an expert in this area, if there are some fatal mistakes, please feel free to tell me.\n", "strength_and_weaknesses": "Strength:\n\n1. To the best of my knowledge, the authors are the first to consider the relay evaluation setting in the field of RL, which does have real-world usage such as in the self-driving system. I think this evaluation could be used as an important check before the application of RL to the real world.\n2. The results of the relay evaluation show that existing RL algorithms can easily fail to handle this situation. Figure 1 illustrates the results well and shows a possible explanation for the failure.\n3. The authors propose two frameworks to improve existing RL algorithms, both of which better utilize the trajectories, either trajectories from other stranger agents or historical trajectories. In theory, these frameworks can be combined with any RL algorithms and are easy to modify.\n\nWeakness:\n\n1. In my view, the motivation of this work is unquestionable but I have some concerns. First, the authors provide an example situation that \"a self-driving system may need to take over the control from humans in the middle of driving and must continue to drive the car safely.\"  Under such a situation, the algorithm used in the RL agent is known by the users, right? However, in the experiments, agents using different RL algorithms are considered the main part of the experiments (A-B pairs in Table1). I wonder does there exist a gap between the experiments and the claimed application scenarios.\n2. If we consider the adaptation of different trajectories, I think the works of meta-RL may be very relevant to this work. In my view, meta RL can handle different distributions of the environment, which could be seen as different trajectories as well. Therefore, could the authors compare their works with several popular meta-RL frameworks?\n3. After looking at the results of Table 2 and Table 3, I think the performance of STA is kind of unsatisfactory compared to the naive algorithm. So, I begin to question the performance of STA, maybe the simple way in the naive algorithm can already handle relay evaluation?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "b3oMiwHDTrG", "reviewer": "Reviewer_9fm8", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose \"relay-evaluation\", a method for evaluating the generalization performance of RL agents. Specifically, it forces the agent under test to start from the state s\\_0, which is sampled from the trajectory generated by stranger agents. A sharp degradation of performance is observed under relay-evaluation, which proves the limitations of existing methods. Correspondingly, the authors proposes STA, a novel method that can be applied in any modern RL algorithm, to improve the generalization ability.\n\nFurthermore, there is no code or any other supporting material for this paper, and the reproducibility of the paper is unconvincing.\n", "summary_of_the_review": "Overall, the paper is very interesting as it studies the generalization of RL agents from a novel view, despite some minor defects like the lack of provable improvement. The paper is well organized, and the analysis of the experiments is convincing. The proposed method seems concise but effective enough without introducing extra sample complexity.  \n\n[1] Derek, K., & Isola, P. (2021). Adaptable Agent Populations via a Generative Model of Policies. *Advances in Neural Information Processing Systems*, *34*, 3902-3913.", "strength_and_weaknesses": "#### Strength\n\nOverall, generalization is a popular and essential topic in RL, and many methods have been explored to evaluate or improve the generalization of RL agents from different aspects. It is quite interesting that the authors studied the generalization from a brand-new point of view and proposed a concise but effective method. The experiments are compelling and well-rounded. I think the paper did contribute some new ideas and open up a new direction worth being studied.\n\n#### Weakness\n\nDespite of the outstanding performance illustrated in the experiments, a provable improvement might can not be guaranteed as many components applied in STA are more likely intuitive.   \n\nIn relay-evaluation, authors sampled states from trajectories generated by stranger agents, which are trained with only different random seeds. PBT[1] might be of help to find more \"controllable\" states where test agent fails and thus evaluating the generalization more accurately.\n\nNot really a weakness, more a suggestion. A pseudocode that describes the process of STA overall might help authors understand it more clearly. Would you consider adding it to the paper in the next version, please?\n\nThere is no code or any other supporting material for this paper, and the reproducibility of the paper is unconvincing.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "2QfM826Yt8", "reviewer": "Reviewer_kkAe", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper discusses the challenge of \"relay generalization\", where a policy trained with RL is evaluated when starting from initial states observed under an independently trained policy.  Relay generalization is intended to measure the out-of-distribution generalization error, by testing how well the policy performs starting from the \"reasonable\" states found by another good policy.  They demonstrate in some of the MuJoCo environments that policies that perform well from from the environment's default initial state distribution can fail catastrophically when they are required to take over control from other, equally effective policies.\n\nThey determine that the issue stems at least in part from the fact that independently trained policies tend to settle into different regions of the state space, and so never learn how to take successfully control the agent from the states that other policies concentrate on.  The propose a simple method to mitigate this issue, in which the current policy is, with some tunable probability, rolled-out from a state encountered previously during training, rather than an initial state of the environment.  They demonstrate that this significantly improves relay generalization when applied to the soft actor-critic algorithm.", "summary_of_the_review": "Overall this work empirically demonstrates an interesting phenomenon that is important for RL researchers to be aware of.  It provides a heuristic solution to this specific issue, but does not tackle the problem of out-of-distribution error more generally.", "strength_and_weaknesses": "The main contribution of the paper is the idea of \"relay generalization\", and empirical demonstration that this issue actually exists.  While it is not entirely surprising that on-policy methods would fail when asked to take control from a state they would not typically encounter during training, this work provides a means of directly measuring this type of out-of-distribution error.  They also provide some qualitative insight into exactly why these failures occur in the MuJoCo environments.\n\nThe main weakness of the paper is that the empirical results are limited to four MuJoCo environments.  The failure mode, at least in the humanoid environment, seems to be somewhat specific to this setting, and so it is unclear how large a problem this would be in general.  It is also unclear how much of an issue this would be with fully off-policy methods with large replay buffers, which might have a similar mitigating effect to the initial state sampling method they propose here.  Additionally, the solution they present (STA) is relatively simple, and it is unclear how much it can improve generalization performance overall, for example, from states that no policy would be likely to find during training.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "2-JecRLTVqO", "reviewer": "Reviewer_7NUq", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes “relay-evaluation” a method to asses the generalization capabilities of RL agents when being forced to start from a state that is out of the distribution defined by the agent’s policy. The paper also introduces “Self-Trajectory Augmentation” (STA) a training method to improve generalization in which the agent is randomly set to start from a state that is sampled from one of the agent’s old policies.", "summary_of_the_review": "I am overall in favor of accepting this paper. I really liked the first part of the paper where the problem of generalization to out-of-distribution trajectories is evaluated using the relay method. I am a bit less excited about STA since the method makes lots of choices that seem rather arbitrary and very domain dependent. I am tentatively giving the paper a score of 6 but I am willing to increase it if the authors can clarify some of my concerns in their rebuttal.", "strength_and_weaknesses": "### Strengths\nThis paper addresses an important problem in RL that I believe hasn’t been explored sufficiently. Generalization in RL is normally regarded as the ability of an agent to perform well on similar tasks/environments.  However, this paper evaluates generalization to out-of-distribution trajectories, showing that, even when the environment stays the same agents may fail to perform well if states are sampled from a different policy. The paper is clearly written, for the most part, and the experiments are thorough.\n\n### Weaknesses\nThe relay-evaluation metric is computationally heavy it requires training multiple other agents to ensure that there is sufficient variety in the policies of the strangers. If the number of stranger policies is small, the method may incorrectly determine that an agent is able to generalize well just because the algorithms used for training the test and stranger agents are very similar.\n\nThe definition of “controllable state” is somewhat loose (i.e. what does “high return” mean and how large does L need to be?).\n\nI would have liked to see experiments on other types of environments. The environments used are all of the same nature. Agents need to learn to walk/move and keep their balance. These require a lot of coordination and thus it is natural for the agents to overfit to a particular trajectory.\n\nCan the STA training scheme negatively impact the agent’s performance? As explained in Section 3.1 STA is equivalent to training in an augmented MDP with a larger initial state set. Is it possible that the optimal policy for this augmented MDP is different from that of the original MDP? \n\nAs far as I can see, a simulator that can be reset to any state is needed for STA.\n\nI found Section 3.2 particularly unclear: \n\n  * The score used by STA to evaluate if a state is controllable depends on the policy being followed. Hence at the beginning of training when policies perform badly, the inclusion of states in the initial set seems arbitrary. That is, some states may not be included just because these bad policies can't perform well when starting from them.\n\n * Is the score of a state averaged over multiple runs? Or is it just a single sample estimate? The latter is a poor estimate of the true score if the policy and/or the environment are stochastic. Why don’t you use the Q network to estimate the score?\n\n * I am not sure why the states need to be controllable to be added to the initial set. What happens if you just add all states to the starting set regardless of whether they are controllable? Wouldn’t that also improve generalization?\n\n * Why is $Q(s, \\pi(s))$ a good metric to determine if the agent is familiar (not familiar) with a particular state? A high (low) Q value may be due to the agent being overconfident (underconfident) about that particular state.\n\n * The use of a parameter $\\lambda$ to truncate the return is a bit hacky. Can't you just divide the score by the number of remaining timesteps?\n\n * How is $\\eta_{STA}$ computed?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster