Abstract: We seek to align agent behavior with a user's objectives in a reinforcement learning setting with unknown dynamics, an unknown reward function, and unknown unsafe states. The user knows the rewards and unsafe states, but querying the user is expensive. We propose an algorithm that safely and efficiently learns a model of the user's reward function by posing 'what if?' questions about hypothetical agent behavior. We start with a generative model of initial states and a forward dynamics model trained on off-policy data. Our method uses these models to synthesize hypothetical behaviors, asks the user to label the behaviors with rewards, and trains a neural network to predict the rewards. The key idea is to actively synthesize the hypothetical behaviors from scratch by maximizing tractable proxies for the value of information, without interacting with the environment. We call this method reward query synthesis via trajectory optimization (ReQueST). We evaluate ReQueST with simulated users on a state-based 2D navigation task and the image-based Car Racing video game. The results show that ReQueST significantly outperforms prior methods in learning reward models that transfer to new environments with different initial state distributions. Moreover, ReQueST safely trains the reward model to detect unsafe states, and corrects reward hacking before deploying the agent.

Siddharth Reddy
Anca D Dragan
Sergey Levine
Shane Legg
Jan Leike

Learning Human Objectives by Evaluating Hypothetical Behavior

Introduction Users typically specify objectives for reinforcement learning (RL) agents through scalar-valued reward functions  (Sutton & Barto, 2018) . While users can easily define reward functions for tasks like playing games of Go or Star-Craft, users may struggle to describe practical tasks like Proceedings of the 37 th International Conference on Machine Learning, Online, PMLR 119, 2020. Copyright 2020 by the author(s).
 Generative model Hypothetical behavior Reward model User feedback RL agent (1) (2) (3) driving cars or controlling robotic arms in terms of rewards  (Hadfield-Menell et al., 2017) . Understanding user objectives in these settings can be challenging -not only for machines, but also for humans modeling each other and introspecting on themselves  (Premack & Woodruff, 1978) . For example, consider the trolley problem  (Foot, 1967) : if you were the train conductor in Figure  1 , presented with the choice of either allowing multiple people to come to harm by letting the train continue on its current track, or harming one person by diverting the train, what would you do? The answer depends on whether your value system leans toward consequentialism or deontological ethics -a distinction that may not be captured by a reward function designed to evaluate common situations, in which ethical dilemmas like the trolley problem rarely occur. In complex domains, the user may not be able to anticipate all possible agent behaviors and specify a reward function that accurately describes user preferences over those behaviors. We address this problem by actively synthesizing hypothetical behaviors from scratch, and asking the user to label them with rewards. Figure  1  describes our algorithm: using a generative model of initial states and a forward dynamics model trained on off-policy data, we synthesize hypothetical behaviors, ask the user to label the behaviors with rewards, and train a neural network to predict the rewards. We repeat this process until the reward model converges, then deploy a model-based RL agent that optimizes the learned rewards. The key idea in this paper is synthesizing informative hypo-theticals. Ideally, we would generate these hypotheticals by optimizing the value of information (VOI;  Savage, 1954) , but the VOI is intractable for real-world domains with highdimensional, continuous states. Instead, we use trajectory optimization to produce four types of hypotheticals that improve the reward model in different ways: behaviors that (1) maximize reward model uncertainty, to elicit labels that are likely to change the updated reward model's outputs; (2) maximize predicted rewards, to detect and correct reward hacking; (3) minimize predicted rewards, to safely explore unsafe states; or (4) maximize novelty of trajectories regardless of predicted rewards, to improve the diversity of the training data. To ensure that the hypothetical trajectories remain comprehensible to the user and resemble realistic behaviors, we use a generative model of initial states and a forward dynamics model for regularization. We call this method reward query synthesis via trajectory optimization (ReQueST). Our primary contribution is an algorithm that synthesizes hypothetical behaviors in order to safely and efficiently train neural network reward models in environments with high-dimensional, continuous states. We evaluate ReQueST with simulated users in three domains: MNIST classification, a state-based 2D navigation task, and the image-based Car Racing video game in the OpenAI Gym. Our experiments show that ReQueST learns robust reward models that transfer to new environments with different initial state distributions, achieving at least 2x better final performance than baselines adapted from prior work (e.g., see Figure  4 ). In the navigation task, ReQueST safely learns to classify 100% of unsafe states as unsafe and deploys an agent that never visits unsafe states, while the baselines fail to learn about even one unsafe state and deploy agents with a failure rate of 75%.
 Related Work In this work, we align agent behavior with a user's objectives by learning a model of the user's reward function and training the agent via RL  (Ng & Russell, 2000; Ziebart et al., 2008; Leike et al., 2018) . The idea behind modeling the user's reward function -as opposed to the user's policy  (Pomerleau, 1991; Ross et al., 2011; Ho & Ermon, 2016) , value function  (Dvijotham & Todorov, 2010; Warnell et al., 2018; Reddy et al., 2018) , or advantage function  (Mac-Glashan et al., 2017)  -is to acquire a compact, transferable representation of the user's objectives; not just in the training environment, but also in new environments with different dynamics or initial states. The closest prior work is on active learning methods for learning rewards from pairwise comparisons  (Sadigh et al., 2017; Bıyık & Sadigh, 2018; Wirth et al., 2017) , critiques , demonstrations  (Ibarz et al., 2018; Brown et al., 2018) , designs  (Mindermann et al., 2018) , and numerical feedback  (Daniel et al., 2014) . Re-QueST differs in three key ways: it produces hypothetical trajectories using a generative model, in a way that enables trading off between producing realistic vs. informative queries; it optimizes queries not only to reduce model uncertainty, but also to detect reward hacking and safely explore unsafe states; and it scales to learning neural network reward models that operate on high-dimensional, continuous state spaces. ReQueST shares ideas with prior work  (Saunders et al., 2018; Prakash et al., 2019)  on learning to detect unsafe behaviors by initially seeking out catastrophes, selectively querying the user, and using model-based RL. ReQueST differs primarily in that it learns a complete task specification, not just an unsafe state detector. ReQueST is also complementary to prior work on safe exploration, which typically assumes a known reward function and side constraints, and focuses on ensuring that the agent never visits unsafe states during policy optimization  (Dalal et al., 2018; Garcıa & Fernández, 2015) .
 Learning Rewards from User Feedback on Hypothetical Behavior We formulate the reward modeling problem as follows. We assume access to a training environment that follows a Markov decision process (MDP;  Sutton & Barto, 2018)  with unknown state transition dynamics T , unknown initial state distribution S train 0 , and an unknown reward function R that can be evaluated on specific inputs by querying the user. We learn a model of the reward function R by querying the user for reward signals. At test time, we train an RL agent with the learned reward function R in a new environment with the same dynamics T , but a potentially different initial state distribution S test 0 . The goal is for the agent to perform well in the test environment with respect to the true reward function R. Our approach to this problem is outlined in Figure  1 , and can be split into three steps. In step (1) we use off-policy data to train a generative model p (⌧ ) that can be used to evaluate the likelihood of a trajectory ⌧ = (s 0 , a 0 , s 1 , a 1 , ..., s T ), which enables us to synthesize hypothetical trajectories that can be shown to the user. In step (2) we produce synthetic trajectories, which consist of sequences of state transitions (s, a, s 0 ), that seek out different kinds of hypotheticals (detailed in Section 3.3). We ask the user to label each transition with a scalar reward R(s, a, s 0 ), and fit a reward model R(s, a, s 0 ) using standard supervised learning techniques. In step (3) we use standard RL methods to train an agent using the learned rewards.
 Learning a Generative Model of Trajectories In order to generate hypothetical behaviors with uncertain rewards, we first learn a generative model of initial states and a forward dynamics model. We use these models to synthesize plausible trajectories, without requiring that an agent actually perform those behaviors in the real environment. In step (1) we collect off-policy data by interacting with the training environment in an unsupervised fashion; i.e., without the user in the loop. To simplify our experiments, we sample trajectories ⌧ by following random policies that explore a wide variety of states. We use the observed trajectories to train a likelihood model, p (⌧ ) / p (s 0 ) T 1 Y t=0 p (s t+1 |s t , a t ), (1) where p (s 0 ) models the initial state distribution, p (s t+1 |s t , a t ) models the forward dynamics, and are the model parameters (e.g., neural network weights). We train the model p using maximum-likelihood estimation, given the sampled trajectories. As described in the next section, we use the likelihood model to generate realistic synthetic trajectories to show to the user. In environments with high-dimensional, continuous states, such as images, we also train a state encoder f : S ! Z and decoder f 1 : Z ! S, where S = R n , Z = R d , and d << n. As described in Section 3.4, embedding states in a low-dimensional latent space Z enables us to synthesize realistic hypotheticals. In our experiments, we train f and f 1 using the variational auto-encoder method (VAE;  Kingma & Welling, 2013) .
 Representing the Reward Model as a Classifier Our goal is to learn a model R of the user's reward function. In step (2) we represent R by classifying state transitions as good, unsafe, or neutral -similar to  -and assigning a known, constant reward to each of these three categories: R(s, a, s 0 ) = X c2{good,unsafe,neutral} p ✓ (c|s, a, s 0 )R c , (2) where p ✓ (c|s, a, s 0 ) = 1 m P m i=1 p ✓i (c|s, a, s 0 ) is the mean of an ensemble of m classifiers {p ✓i } m i=1 , and ✓ i are the weights of the i-th neural network in the ensemble. R c is the constant reward for any state transition in class c, where R unsafe  R neutral  R good . Modeling the reward function as a classifier simplifies our experiments and makes it easier for the user to provide labels. In principle, our method can also work with other architectures, such as a regression model R = R ✓ .
 Designing Objectives for Informative Queries Our approach to reward modeling involves asking the user to label trajectories with reward signals. In step (2) we synthesize query trajectories to elicit user labels that are informative for learning the reward model. To generate a useful query, we synthesize a trajectory ⌧ that maximizes an acquisition function (AF) denoted by J(⌧ ). The AF evaluates how useful it would be to elicit reward labels for ⌧ , then update the reward model given the newly-labeled data. Since we do not assume knowledge of the initial state distribution of the test environment where the agent is deployed, we cannot optimize the ideal AF: the value of information (VOI;  Savage, 1954) , defined as the gain in performance of an agent that optimizes the updated reward model in the test environment. Prior work on active learning tackles this problem by optimizing proxies for VOI  (Settles, 2009) . In this work, we synthesize a separate trajectory for each of four AFs defined in the following paragraphs. Maximizing uncertainty. The first AF J u (⌧ ) implements one of the simplest query selection strategies from the active learning literature: uncertainty sampling  (Lewis & Gale, 1994) . The idea is to elicit labels for examples that the model is least certain how to label, and thus reduce model uncertainty. To do so, we train an ensemble of neural network reward models, and generate trajectories that maximize the disagreement between ensemble members. Following  Lakshminarayanan et al. (2017) , we measure ensemble disagreement using the average KL-divergence between the output of a single ensemble member and the ensemble mean (Equation 5 in the appendix). Maximizing reward. The second AF J + (⌧ ) is intended to detect examples of false positives, or 'reward hacking': behaviors for which the reward model incorrectly outputs high reward  (Amodei et al., 2016; Christiano et al., 2017) . The idea is to show the user what the reward model predicts to be good behavior, with the expectation that some of these behaviors are actually suboptimal, and will be labeled as such by the user. To do so, we simply synthesize trajectories that maximize J + (⌧ ) = P (s,a,s 0 )2⌧ R(s, a, s 0 ). Minimizing reward. The third AF J (⌧ ) is intended to augment the training data with more examples of unsafe states than would normally be encountered, e.g., by a rewardmaximizing agent acting in the training environment. The idea is to show the user what the reward model considers unsafe behavior, so the user can confirm whether the model has captured the correct notion of unsafe states. To do so, we produce trajectories that maximize J (⌧ ) = J + (⌧ ). Maximizing novelty. The fourth AF J n (⌧ ) is intended to produce novel trajectories that differ from those already in the training data, regardless of their predicted reward; akin to prior work on geometric AFs  (Sener & Savarese, 2018) . This is especially helpful early during training, when uncertainty estimates are not accurate, and the reward model has not yet captured interesting notions of reward-maximizing and reward-minimizing behavior. To do so, we produce trajectories ⌧ that maximize the distance between ⌧ and previously-labeled trajectories ⌧ 0 2 D, J n (⌧ ) = 1 |D| P ⌧ 0 2D d(⌧, ⌧ 0 ). In this work, we use a distance function d that computes the Euclidean distance between state embeddings (Equation 6 in the appendix).
 Query Synthesis via Trajectory Optimization The four AFs defined in the previous section specify the types of hypotheticals we would like to show the user. In this section, we discuss how to optimize each AF, then use the resulting hypotheticals to learn a reward model. We synthesize a query trajectory, ⌧ query = max z0,a0,z1,...,z T J(⌧ ) + log p (⌧ ), (3) where z t is the embedding of state s t in the latent space of the encoder f trained in step (1), ⌧ = (f 1 (z 0 ), a 0 , f 1 (z 1 ), a 1 , ..., f 1 (z T )) is the decoded trajectory, J is the acquisition function (Section 3.3), 2 R 0 is a regularization constant, and p is the generative model of trajectories (Section 3.1). In this work, we assume p (⌧ ) is differentiable, and optimize ⌧ query using Adam  (Kingma & Ba, 2014) . Optimizing latent states z instead of highdimensional states s reduces computational requirements, and regularizes the optimized states to be more realistic. The regularization constant from Equation 3 controls the trade-off between how realistic ⌧ query is and how aggressively it maximizes the AF. Setting = 0 can result in query trajectories that are incomprehensible to the user and unlikely to be seen in the test environment, while setting to a high value can constrain the query trajectories from seeking interesting hypotheticals. The experiments in Section A.3 in the appendix analyze this trade-off. Our reward modeling algorithm is summarized in Algorithm 1. Given a generative model of trajectories p (⌧ ), it generates one query trajectory ⌧ query for each of the four AFs, asks the user to label the states in the query trajectories, retrains the reward model ensemble {✓ i } m i=1 on the updated training data D using maximum-likelihood estimation, and repeats this process until the user is satisfied with the outputs of the reward model. The ablation study in Section 4.5 analyzes the effect of using different subsets of the four AFs to generate queries.
 Deploying a Model-Based RL Agent Given the learned reward model R, the agent can, in principle, be trained using any RL algorithm in step (3). Since our end for 10: end for 11: for i 2 {1, 2, ..., m} do 12: ✓i arg max ✓ i P (s,a,s 0 ,c)2D log p ✓ i (c|s, a, s 0 ) 13: end for 14: end while 15: Return reward model R {Defined via ✓ in Equation  2} method learns a forward dynamics model in step (1), modelbased RL is a good fit for step (3). In this work, we deploy an agent ⇡ mpc that combines planning with model-predictive control (MPC): ⇡ mpc (a|s) =  a = arg max a0 max a1,...,a H R(s, a 0 ) + R(E [s 1 |s, a 0 ], a 1 ) + ... + R(E [s H |s, a 0 , a 1 , ..., a H 1 ], a H ) i , (4) where the future states E [s t |s, a 0 , a 1 , ..., a t 1 ] are predicted using the forward dynamics model p trained in step (1), H is the planning horizon, and R is the reward model trained in step (2). We solve the optimization problem in the right-hand side using Adam  (Kingma & Ba, 2014) .
 Experimental Evaluation We seek to answer the following questions. Q1: Does synthesizing hypothetical trajectories elicit more informative labels than rolling out a policy in the training environment? Q2: Can our method detect and correct reward hacking? Q3: Can our method safely learn about unsafe states? Q4: Do the proposed AFs improve upon random sampling from the generative model? Q5: How does the regularization constant control the trade-off between realistic and informative queries? Q6: How much do each of the four AFs contribute to performance? To answer these questions under ideal assumptions, we run experiments in MNIST  (LeCun, 1998) , state-based 2D navigation (Figure  2 ), and imagebased Car Racing from the OpenAI Gym  (Brockman et al., 2016) , with simulated users that label trajectories using a ground-truth reward function. Section A.4 in the appendix discusses implementation details. MNIST classification. This domain enables us to focus on testing the active learning component of our method, since the standard digit classification task does not involve sequential decision-making. When we generate queries, we synthesize an image s 0 2 R 28⇥28 , and ask the simulated user to label it with an action a 2 {0, 1, ..., 9}. The initial state distribution of the training environment S train 0 puts a uniform probability on sampling s 0 2 {5, 6, 7, 8, 9}, and a probability of 0 on sampling s 0 2 {0, 1, 2, 3, 4}. We intentionally introduce a significant shift in the test environment, by putting a uniform probability on sampling s 0 2 {0, 1, 2, 3, 4} and a probability of 0 on sampling s 0 2 {5, 6, 7, 8, 9}. This mismatch is intended to test how well the learned classifier performs under distribution shift. State-based 2D navigation. This domain enables us to focus on the challenges of sequential decision-making, without dealing with high-dimensional states. Here, the state s 2 R 2 is the agent's position, and the action a 2 R 2 is a velocity vector. The task requires navigating to a target region, while avoiding a trap region (Figure  2 ). The task is harder to complete in the test environment, since the agent starts closer to the trap, and must navigate around the trap to reach the goal. Image-based Car Racing. This domain enables us to test whether our method scales to learning sequential tasks with high-dimensional states. Here, the state s 2 R 64⇥64⇥3 is an RGB image with a top-down view of the car (Figure  3 ), and the action a 2 R 3 controls steering, gas, and brake. Here, we set the same initial state distribution for the training and test environments, since the reward modeling problem is challenging even when the initial state distributions are identical. We train a generative model of images and a forward dynamics model in step (1) using the unsupervised method from  Ha & Schmidhuber (2018) .
 Robustness Compared to Baselines Our first experiment tests whether our method can learn a reward model that is robust enough to perform well in the test environment, and tracks how many queries to the user it takes to learn an accurate reward model. To answer Q1, we compare our method to a baseline that, instead of generating hypothetical trajectories for the user to label, generates trajectories by rolling out a policy that optimizes the current reward model in the training environment -an approach adapted from prior work  (Christiano et al., 2017) . The baseline generates ⌧ query in line 5 of Algorithm 1 by rolling out the MPC policy in Equation  4 , instead of solving the optimization problem in Equation 3. To test how generating queries using a reward-maximizing policy compares to using a policy that does not depend on the reward model, we also evaluate a simpler baseline that generates query trajectories using a uniform random policy, instead of the MPC policy. We measure performance in MNIST using the agent's classification accuracy in the test environment; in 2D navigation, the agent's success rate at reaching the goal while avoiding the trap in the test environment; and in Car Racing, the agent's true reward, which rewards progress and penalizes going off-road. The results in Figure  4  show that our method produces reward models that transfer to the test environment better than the baselines. Our method also learns to outperform the suboptimal demonstrations used to initialize the reward model (Figure  10  in the appendix). In MNIST, our method performs substantially better than the baseline, which samples queries s 0 from the initial state distribution of the training environment. The reason is simple: the initial state distribution of the test environment differs significantly from that of the training environment. Since our method is not restricted to sampling from the training environment, it performs better than the baseline. In 2D navigation, our method substantially outperforms both Figure  4 . Experiments that address Q1 -does synthesizing hypothetical trajectories elicit more informative labels than rolling out a policy in the training environment? -by comparing our method, which uses synthetic trajectories, to baselines that only use real trajectories generated in the training environment. The results on MNIST, 2D navigation, and Car Racing show that our method (orange) significantly outperforms the baselines (blue and gray), which never succeed in 2D navigation. The x-axis represents the number of queries to the user, where each query elicits a label for a single state transition (s, a, s 0 ). The shaded areas show standard error over three random seeds. baselines, which never succeed in the test environment. This is unsurprising, since the training environment is set up in such a way that, because the agent starts out in the lower left corner, they rarely visit the trap region in the upper right by simply actions -whether reward-maximizing actions (as in the first baseline), or uniform random actions (as in the second baseline). Hence, when a reward model trained by the is transferred to the test environment, it not aware of the trap, so the agent tends to get caught in the trap on its way to the goal. Our method, however, is not restricted to feasible trajectories in the training environment, and can potentially query the label for any position in the environment -including the trap (see Figure  2 ). Hence, our method learns a reward model that is aware of the trap, which enables the agent to navigate around it in the test environment. In Car Racing, our method outperforms both baselines. The baselines tend to generate queries that are not diverse and rarely visit unsafe states, so the resulting reward models are not able to accurately distinguish between good, unsafe, and neutral states. Our method, on the other hand, explicitly seeks out a wide variety of states by maximizing the four AFs, which leads to more diverse training data, and a more accurate reward model.
 Detecting Reward Hacking One of the benefits of our method is that it can detect and correct reward hacking before deploying the agent, using reward-maximizing synthetic queries. In the next experiment, we test this claim. We replicate the experimental setup in Section 4.1 for 2D navigation, including the same baselines. We measure performance using the false positive rate of the reward model: the fraction of neutral or unsafe states incorrectly classified as good, evaluated on the offline dataset of trajectories described in Section 4.1. A reward model that outputs false positives is susceptible to reward hacking, since a reward-maximizing agent can game the reward model into emitting high rewards by visiting incorrectly classified states. The results in Figure  9  in the appendix show that our method drives down the false positive rate in 2D navigation: the learned reward model rarely incorrectly classifies an unsafe or neutral state as a good state. As a result, the deployed agent actually performs the desired task (center plot in Figure  4 ), instead of seeking false positives. As discussed in Section 4.3 and illustrated in the right-most plot of Figure  5 , the baselines learn a reward model that incorrectly extrapolates that continuing up and to the right past the goal region is good behavior. For a concrete example of rewardmaximizing synthetic queries that detect reward hacking, consider the reward-maximizing queries in the upper right corner of Figure  2 , which are analyzed in Section A.2 in the appendix.
 Safe Exploration One of the benefits of our method is that it can learn a reward model that accurately detects unsafe states, without having to visit unsafe states during the training process. In the next experiment, we test this claim. We replicate the experimental setup in Section 4.1 for 2D navigation, including the same baselines. We measure performance using the true negative rate of the reward model: the fraction of unsafe states correctly classified as unsafe, evaluated on the offline dataset of trajectories described in Section 4.1. We also use the crash rate of the deployed agent: the rate at which it gets caught in the trap region. Experiments that address Q3 -can our method safely learn about unsafe states? -by comparing our method, which uses synthetic trajectories, to baselines that only use real trajectories generated in the training environment. The results on 2D navigation show that our method (orange) significantly outperforms the baselines (blue and gray). The x-axis represents the number of queries to the user, where each query elicits label for a single state transition (s, a, 0 ). The shaded show standard error over three random seeds. The heat maps represent the reward models learned by our method (left) and by the baselines (right). Figure  6 . Experiments that address Q4 -do the proposed AFs improve upon random sampling from the generative model? -by comparing our method, which synthesizes trajectories by optimizing AFs, to a baseline that ignores the AFs and randomly samples from the generative model. The results on MNIST, 2D navigation, and Car Racing show that our method (orange) significantly outperforms the baseline (blue) in Car Racing, and learns faster in MNIST and 2D navigation. The x-axis represents the number of queries to the user, where each query elicits a label for a single state transition (s, a, s 0 ). The shaded areas show standard error over three random seeds. The results in Figure  5  show that our method learns a reward model that classifies all unsafe states as unsafe, without visiting unsafe states during training (second and third figure from left); in fact, without visiting any states at all, since the queries are synthetic. This enables the agent to avoid crashing during deployment (first figure from left). The baselines differ from our method in that they actually have to visit unsafe states in order to query the user for labels at those states. Since the baselines tend to not visit unsafe states during training, they do not learn about unsafe states (second and fourth figure from left), and the agent frequently crashes during deployment (first figure from left).
 Query Efficiency Compared to Baselines The previous experiment compared to baselines that are restricted to generating query trajectories by taking actions in the training environment. In this experiment, we lift this restriction on the baselines: instead of taking actions in the training environment, the baselines can make use of the generative model trained in step (1). To answer Q4, we compare our method to a baseline that randomly samples trajectories from the generative model p -using uniform random actions in Car Racing, samples from the VAE prior in MNIST, and uniform positions across the map in 2D navigation. We measure performance in MNIST using the reward model's predicted log-likelihood of the ground-truth user labels in the test environment; in 2D navigation, the reward model's classification accuracy on an offline dataset containing states sampled uniformly throughout the environment; and in Car Racing, the true reward collected by an MPC agent that optimizes the learned reward, where the true reward gives a bonus for driving onto new patches of road, and penalizes going off-road. The results in Figure  6  show that our method, which optimizes trajectories using various AFs, requires fewer queries to the user than the baseline, which randomly samples trajectories. This suggests that our four AFs guide query synthesis toward informative trajectories. These results, and the re-Figure  7 . Experiments that address Q6 -how much do each of the four AFs contribute to performance? -by comparing our method to ablated variants that drop each AF, one at a time, from the set of four AFs in line 4 of Algorithm 1. The results on MNIST, 2D navigation, and Car Racing show that our method (orange) generally outperforms its ablated variants (blue, gray, red, and pink), although the usefulness of each AF depends on the domain and amount of training data.. The x-axis represents the number of queries to the user, where each query elicits a label for a single state transition (s, a, s 0 ). The shaded areas show standard error over three random seeds. sults from Section 4.1, suggest that our method benefits not only from using a generative model instead of the default training environment, but also from optimizing the AFs instead of randomly sampling from the generative model.
 Acquisition Function Ablation Study We propose four AFs intended to produce different types of hypotheticals. In this experiment, we investigate the contribution of each type of query to the performance of the overall method. To answer Q6, we conduct an ablation study, in which we drop out each the four AFs, one by one, from line 4 in Algorithm 1, and measure the performance of only generating queries using the remaining three AFs. The results in Figure  7  show that the usefulness of each AF depends on the domain and the amount of training data collected. In MNIST, dropping J u hurts performance, suggesting that uncertainty-maximizing queries elicit useful labels. Dropping J n also hurts performance when the number of queries is small, but actually improves performance if enough queries have already been collected. Noveltymaximizing queries tend to be repetitive in practice: although they are distant from the training data in terms of Equation  6 , they are visually similar to the existing training data in that they appear to be the same digits. Hence, while they are helpful at first, they hurt query efficiency later in training. In 2D navigation, dropping J u hurts performance, while dropping any of the other AFs individually does not hurt performance. These results suggest that uncertaintymaximizing queries can be useful, in domains like MNIST and 2D navigation, where uncertainty can be modeled and estimated accurately. In Car Racing, dropping J hurts the most. Reward-minimizing queries elicit labels for unsafe states, which are rare in the training environment unless you explicitly seek them out. Hence, this type of query performs the desired function of augmenting the training data with more examples of unsafe states, thereby making the reward model better at detecting unsafe states.
 Discussion The experiments show that ReQueST produces accurate reward models that transfer well to new environments and require fewer queries to the user, compared to baseline methods adapted from prior work. Our method detects reward hacking before the agent is deployed, and safely learns about unsafe states. Through a hyperparameter sweep, we find that our method can trade off between producing realistic vs. informative queries, and that the optimal trade-off varies across domains. Through an ablation study, we find that the usefulness of each of the four acquisition functions we propose for optimizing queries depends on the domain and the amount of training data collected. One of the benefits of ReQueST is that, since it learns from synthetic trajectories instead of real trajectories, it only has to imagine visiting unsafe states, instead of actually visiting them. Although unsafe states may be visited during unsupervised exploration of the environment for training the generative model in step (1), the same generative model can be reused to learn reward models for any number of future tasks. Hence, the cost of visiting a fixed number of unsafe states in step (1) can be amortized across a large number of tasks in step (2). We could also train the generative model on other off-policy data, including safe expert demonstrations and examples of past failures by humans. The main practical limitation of ReQueST is that it requires a generative model of initial states and a forward dynamics model, which can be difficult to learn from purely off-policy data in complex, visual environments. One direction for future work is relaxing this assumption; e.g., by incrementally training a generative model on on-policy data collected from an RL agent in the training environment  (Kaiser et al., 2019; Hafner et al., 2019) . Figure 1 . 1 Figure 1. Our method learns a reward model from user feedback on hypothetical behaviors, then deploys a model-based RL agent.
 ✓ not converged do 4: for J 2 {Ju, J+, J , Jn} do 5: ⌧query max⌧ J(⌧ ) + log p (⌧ ) 6:for (s, a, s 0 ) 2 ⌧query do 7: c c ⇠ puser(c|s, a, s 0 ) {Query the user} 8:D D [ {(s, a, s 0 , c)} 9:
 Figure 2 . 2 Figure 2. Left: The 2D navigation task, where the agent navigates to the goal region (green) in the lower left while avoiding the trap region (red) in the upper right. The agent starts in the lower left corner in the training environment, and starts in the upper right corner in the test environment. Right: Examples of hypothetical states synthesized throughout learning, illustrating the qualitative differences in the behaviors targeted by each AF.
 Figure 3 . 3 Figure 3. Left: A screenshot of the image-based Car Racing video game in the OpenAI Gym. Right: Examples of synthesized hypothetical behaviors, including going off road (J ), making progress (J+), driving on the edge (Ju), and seeking unusual car configurations (Jn). See Figure 13 in the appendix for synthesized videos.
 Figure 5 . 5 Figure5. Experiments that address Q3 -can our method safely learn about unsafe states? -by comparing our method, which uses synthetic trajectories, to baselines that only use real trajectories generated in the training environment. The results on 2D navigation show that our method (orange) significantly outperforms the baselines (blue and gray). The x-axis represents the number of queries to the user, where each query elicits label for a single state transition (s, a, 0 ). The shaded show standard error over three random seeds. The heat maps represent the reward models learned by our method (left) and by the baselines (right).
			 University of California, Berkeley 2 DeepMind. Correspondence to: Siddharth Reddy <<EMAIL>, Jan Leike <<EMAIL>.


Abstract: Reinforcement learning agents are typically trained and evaluated according to their performance averaged over some distribution of environment settings. But does the distribution over environment settings contain important biases, and do these lead to agents that fail in certain cases despite high average-case performance? In this work, we consider worst-case analysis of agents over environment settings in order to detect whether there are directions in which agents may have failed to generalize. Specifically, we consider a 3D first-person task where agents must navigate procedurally generated mazes, and where reinforcement learning agents have recently achieved human-level average-case performance. By optimizing over the structure of mazes, we find that agents can suffer from catastrophic failures, failing to find the goal even on surprisingly simple mazes, despite their impressive average-case performance. Additionally, we find that these failures transfer between different agents and even significantly different architectures. We believe our findings highlight an important role for worst-case analysis in identifying whether there are directions in which agents have failed to generalize. Our hope is that the ability to automatically identify failures of generalization will facilitate development of more general and robust agents. To this end, we report initial results on enriching training with settings causing failure. 1 A full description and code for the tasks can be found at https://github.com/deepmind/lab/ tree/master/game_scripts/levels/contributed/dmlab30#goal-locations-large.

Under review as a conference paper at ICLR 2019 UNCOVERING SURPRISING BEHAVIORS IN REINFORCEMENT LEARNING VIA WORST-CASE ANALYSIS

INTRODUCTION Reinforcement Learning (RL) methods have achieved great success over the past few years, achieving human-level performance on a range of tasks such as Atari  (Mnih et al., 2015) , Go  (Silver et al., 2016) , Labyrinth , and Capture the Flag  (Jaderberg et al., 2018) . On these tasks, and more generally in reinforcement learning, agents are typically trained and evaluated using their average reward over environment settings as the measure of performance, i.e. E P (e) [R(π(θ), e)] , where π(θ) denotes a policy with parameters θ, R denotes the total reward the policy receives over the course of an episode, and e denotes environment settings such as maze structure in a navigation task, appearance of objects in the environment, or even the physical rules governing environment dynamics. But what biases does the distribution P (e) contain, and what biases, or failures to generalize, do these induce in the strategies agents learn? To help uncover biases in the training distribution and in the strategies that agents learn, we propose evaluating the worst-case performance of agents over environment settings, i.e. min e∈E E [R(π(θ), e)] , where E is some set of possible environment settings. Worst-case analysis can provide an important tool for understanding robustness and generalization in RL agents. For example, it can help us with: • Understanding biases in training Catastrophic failures can help reveal situations that are rare enough during training that the agent does not learn a strategy that is general enough to cope with them. • Robustness For critical systems, one would want to eliminate, or at least greatly reduce, the probability of extreme failures. • Limiting exploitability If agents have learned strategies that fail to generalize to particular environment settings, then an adversary could try and exploit an agent by trying to engineer such environment settings leading to agent failure. In this work, we use worst-case analysis to investigate the performance of a state-of-the-art agent in solving a first-person 3D navigation task; a task on which agents have recently achieved average-case human-level performance  (Wayne et al., 2018) . By optimizing mazes to minimize the performance of agents, we discover the existence of mazes where agents repeatedly fail to find the goal (which we refer to as a Catastrophic Failure). Our Contributions To summarize, the key contributions of this paper are as follows: 1. We introduce an effective and intuitive approach for finding simple environment settings leading to failure (Section 2). 2. We show that state-of-the-art agents carrying out navigation tasks suffer from drastic and often surprising failure cases (Sections 3.1 and 3.2). 3. We demonstrate that mazes leading to failure transfer across agents with different hyperparameters and, notably, even different architectures (Section 3.3). 4. We present an initial investigation into how the training distribution can be adapted by incorporating adversarial and out-of-distribution examples (Section 4).
 APPROACH Tasks We consider agents carrying out first-person 3D navigation tasks. Navigation is of central importance in RL research as it captures the challenges posed by partially observable Markov decision processes (POMDPs). The navigation tasks we use are implemented in DeepMind Lab (DM Lab)  (Beattie et al., 2016) . 1 Each episode is played on a 15 × 15 maze where each position in the maze may contain a wall, an agent spawn point, or a goal spawn point. The maze itself is procedurally generated every episode, along with the goal and agent spawn locations. The goal location remains fixed throughout an episode, while the agent spawn location can vary. In training, the agent respawns at different locations each time they reach the goal, while for our optimization and analysis we limit the agent to the same spawn location. Agents receive RGB observations of size 96 × 72 pixels, examples of which are provided in Figure  1 . Episodes last for 120 seconds and are played at a framerate of 15 frames per second. The agent receives a positive reward of 10 every time it reaches the goal, and 0 otherwise. On this specific navigation task, RL agents have recently achieved human-level average-case performance  (Wayne et al., 2018) . Agents We perform our analysis on Importance Weighted Actor-Learner Architecture agents trained to achieve human-level average-case performance on navigation tasks. These agents can be described as async batched-a2c agents with the V-trace algorithm for off policy-correction, and we henceforth refer to these as A2CV agents . Details of the training procedure are provided in Appendix A.1. Search Algorithm If we are interested in worst-case performance of agents, how can we find environment settings leading to the worst performance? In supervised learning, one typically uses gradient based methods to find inputs that lead to undesired output  (Biggio et al., 2013; Szegedy et al., 2013; Goodfellow et al., 2014) . In contrast, we search for environment settings leading to an undesired outcome at the end of an episode. This presents a challenge as the environment rendering and MDP are not differentiable. We are therefore limited to black-box methods where we can only query agent performance given environment settings. To search for environment settings which cause catastrophic failures, we propose the local search procedure described in Algorithm 1 (visualizing the process in Figure  2 ).   We then Evaluate each with the agent over 30 episodes, select the best maze (i.e. lowest agent score), and Modify this maze by randomly moving two walls to form the next set of candidates (Iteration 1). This process is repeated for 20 iterations, leading to a maze where the agent score is 0.09 in this example (i.e. the agent finds the goal once in 11 episodes). In Appendix A.2.1 we detail the computational requirements of this search procedure.
 EXPERIMENTS The agents we study achieve impressive average-case performance, but how much does their worstcase performance differ from their average-case performance? To investigate this, we consider the worst-case performance over a large set of mazes, including mazes that are not possible under the training distribution. A natural question to ask is whether any departure from the wall structure present during training will lead to agent failure. To test this, we evaluate the agent on samples from a distribution of mazes containing all mazes agents could be evaluated on during the search. In particular, we randomly select agent and goal spawn locations in the first step and then randomly move 40 walls, corresponding to the same actions taken by our optimization procedure, but where the actions are chosen randomly rather than in order to minimize agent performance. We find that agents do generalize to random mazes from the set we consider. In fact, we find that the average score obtained by agents on randomly perturbed mazes is slightly higher than on the training distribution, with agents obtaining an average of 45 goal reaches per two minute episode. The increased performance is likely due to the agent spawn location being fixed, making it easier for the agent to return to the goal once found. The considered agents generalize in the sense that agent performance is not reduced on average by out-of-distribution wall structure. But what about the worst case over all wall structures? Have the agents learned a general navigation strategy that works for all solvable mazes? Or do there exist environment settings that lead to catastrophic failures with high probability? In this section, we investigate these questions. We define a Catastrophic Failure to be an agent failing to find the goal in a two minute episode (1800 steps). As detailed below, we find that not only do there exist mazes leading to catastrophic failure, there exist surprisingly simple mazes that lead to catastrophic failure for agents yet are consistently and often rapidly solved by humans.
 RESULT 1: CATASTROPHIC FAILURES EXIST Do environment settings leading to catastrophic failure exist for the agents we are considering? By searching over mazes using the procedure outlined in Algorithm 1, we find mazes where agents fail to find the goal on many episodes, only finding the goal 10% of the time. In fact, some individual mazes lead to failure across five different agents we tested, with even the best performing agent only finding the goal in 20% of the episodes.  Optimization curves for our search procedure are given in Figure  3 . Note that while we define catastrophic failure as failure to find the goal, the actual objective used for the optimization was average number of goal reaches during an episode. Using average number of goals gives a stronger signal at the start of the optimization process. Finding mazes leading to lower average number of captures is easier than finding mazes where the agent rarely finds the goal even once. As can be seen, despite finding the goal on average 45 times per episode on randomly perturbed mazes, on mazes optimized to reduce score, agents find the goal on average only 0.33 times per episode, more than a 100× decrease in performance. In terms of probability of catastrophic failure, we note that despite agents finding the goal in approximately 98% of episodes on randomly perturbed mazes, using our method, on average we find mazes where agents only finds the goal in 30% of the episodes. Example trajectories agents take during failures are visualized in Figure  4 . The trajectories often seem to demonstrate a failure to use memory to efficiently explore the maze with the agent repeatedly visiting the same locations multiple times. The mazes presented in Figure  4  appear to be of higher complexity than mazes seen during training. This suggests that to obtain agents that truly master navigation, more complex mazes should be included in the training distribution. However, we can ask whether it is only more complex mazes that lead to catastrophic failure or whether there are also simple mazes leading to catastrophic failure. This is a question we explore in the next subsection.
 RESULT 2: SIMPLE CATASTROPHIC FAILURES EXIST While the existence of catastrophic failures may be intriguing and perhaps troubling, one might suspect that the failures are caused by the increased complexity of the mazes leading to failure relative to the mazes the agent is exposed to during training, e.g., the mazes leading to failure contain more dead ends and sometimes have lower visibility. Further, understanding the cause of failure in such mazes seems challenging due to the large number of wall structures that may be causing the agent to fail. In this section, we explore whether there exist simple mazes which lead to catastrophic failures. As our measure of complexity, we use the total number of walls in the maze. We also evaluate humans on such mazes to get a quantitative measure of maze complexity. To find simple mazes which lead to failure, we first follow the same procedure as in the previous section, producing a set of mazes which all lead to catastrophic failures (i.e. a low agent scores). Next, we use this set of mazes as the initial set of candidates in our search algorithm, however we now use a Modify function that removes a single randomly chosen wall each iteration. This process is repeated for 70 iterations, searching for a maze with few walls while maintaining low agent score. In Figure  4 , we present the resulting simple mazes and the corresponding agent trajectories from our optimization procedure. Interestingly, we find that one can remove a majority of the walls in a maze and still maintain the catastrophic failure (i.e. very low agent score). Of note is that a number of these mazes are strikingly simple, suggesting that there exists structure in the environment that the agent has not generalized to. For example, we can see that placing the goal in a small room in an otherwise open maze can significantly reduce the agent's ability to find the goal. Human baselines While these simple maps may lead to catastrophic failure, it is unclear whether this is because of the agent or whether the maze is difficult in a way that is not obvious. To investigate this, we perform human experiments by tasking humans to play on a set of 10 simplified mazes. Notably, we find that human players are able to always locate the goal in every maze and typically within one third of the full episode length. This demonstrates that the mazes are comfortably solvable within the course of an episode by players with a general navigation strategy. We provide a detailed comparison of agent and human performance in Appendix A.3. Analysis One question that may arise is the extent to which these mazes are isolated points in the space of mazes. That is, if the maze was changed slightly, would it no longer lead to catastrophic failure? To test this, we investigate how sensitive our discovered failure mazes are with respect to the agent and goal spawn locations on simplified adversarial mazes. As can be seen in Figure  5 , we find that for a large range of spawn locations, the mazes still lead to failure. This indicates that there is specific local maze structure which causes agents to fail.  Procedures for finding such simple mazes may prove useful as a tool for debugging agents and understanding the ways in which training has led them to develop narrow strategies that are good enough for achieving high average-case performance.
 RESULT 3: FAILURE MAZES TRANSFER ACROSS AGENTS We have found failure cases for individual agents, but to what extent do these failure cases highlight a specific peculiarity of the individual agent versus a more general failure of a certain class of agents, or even a shortcoming of the distribution used for training? In this section, we investigate whether mazes which cause one trained agent to fail also cause other agents to fail. We consider two types of transfer: (1) between different hyperparameters of the same model architecture, and (2) between different model architectures. To test transfer between agents of the same architecture, we train a set of five A2CV agents with different entropy costs and learning rates. To test transfer between agents with significantly different architectures, we train a set of five MERLINbased agents  (Wayne et al., 2018) . These agents have a number of differences to the A2CV agents, most notably they contain a sophisticated memory structure based on a DNC (but with a fixed write location per timestep)  (Graves et al., 2016) . Both agents are trained on the same distribution and achieve human-level averages scores on the navigation task (with MERLIN scoring 10% higher than A2CV on average). Further details of agent training can be found in Appendix A.1. To quantify the level of transfer between (sets of) agents, we follow the procedure for finding adversarial mazes outlined in Section 3.1 to produce a collection of 50 unique failure mazes for each agent (i.e. 10 collections of 50 mazes each). We then evaluate every agent 100 times on each maze in each collection, reporting their average performance on each collection. Complete quantitative transfer results can be found in Appendix A.4. Failure cases transfer somewhat across all agents First, we find that across all agents, some level of transfer exists. In particular, as can be seen in Figure  6 , the probability of one agent finding the goal on mazes generated to reduce the score another agent is significantly below 1. This suggests a common cause of failure that is some combination of the distribution of environment settings used during training and the set of methods that are currently used to train such agents. A possible way to address this could be enriching the training distribution so that it contains fewer biases and encourages more general solutions. Transfer within agent type is stronger than between agent type Comparing the performance of each agent type on mazes from the same agent type to mazes from another agent type, we see that transfer within agent type is stronger. As shown in Figure  6b , performance increases as we go from 'MERLIN to MERLIN' 'A2CV to MERLIN' (0.42 to 0.58) and also if we go from 'A2CV to A2CV' to 'MERLIN to A2CV' (0.63 to 0.70). This suggests that there are some common biases in agents that are due to their architecture type. Analyzing structural differences between mazes that lead to one agent type to fail but not another could give interesting insight into behavioural differences between agents beyond just average performance. A2CV agents are less susceptible to transfer Despite similar probabilities of failure when evaluating on mazes optimized for the same agent, A2CV agents seem to suffer less on mazes optimized using other A2CV or MERLIN agents. This indicates that A2CV agents may have learned a more diverse set of strategies. Figure  6 : Mazes that lead to failures in one agent lead to failure in other agents as well. This is the case for agents of the same architecture with different hyperparameters, and is also the case for transfer across agents of different architecture. We note, however, that transfer across agents with different architectures is weaker than among agents with the same architecture, and that the performance of agents with the same architecture but with different hyperparameters is slightly higher than for the agents used to originally find the mazes.
 ADAPTING THE TRAINING DISTRIBUTION From our experiments so far, we have discovered that there exist many mazes which lead to catastrophic failure. In this section, we investigate whether agent performance can be improved by adapting the training distribution, for example by incorporating adversarial mazes into training and modifying the original mazes used in training.
 MOTIVATION To better understand what may be causing catastrophic failures, with the aim of fixing them, we compare the set of adversarial mazes with the original set of mazes used in training. From this comparison, we find that there are two notable differences. The probability of a catastrophic failure correlates with the distance between the spawn locations and how hidden the goal is First, we find that a number of features are more common in adversarial mazes than non-adversarial mazes. In particular, adversarial mazes are more likely to have the goal hidden in an enclosed space (such as a small room), and on average the path length from the player's starting location to the goal is significantly longer (31.1 ± 8.4 compared to 11.6 ± 6.3). Notably, while the training distribution also contains hidden goals which are far from the agent's starting location, they are much rarer. Adversarial mazes are typically far from the training distribution Second, we find that adversarial mazes tend to not only be out-of-distribution, but also far from the training distribution due to the Modify function used in our adversarial search procedure (for example, see Figure  2 ). This contrasts with the adversarial images literature where attacks are usually constrained to be small or imperceptible. It may therefore not be surprising that the agent is unable to generalize to all out-of-distribution mazes which could also explain the significant reduction in their performance. Given these two observations, it is natural to ask whether the training distribution can be adapted to improve the agent's performance. In the following sections we investigate this question and discuss our findings, focusing on incorporating adversarial mazes into training and modifying the original mazes used in training.
 APPROACH We consider two distinct approaches for incorporating adversarial and out-of-distribution mazes into the training distribution. Adversarial training To add adversarial mazes into the training distribution, we first create a dataset of 6000 unique adversarial mazes from separate runs of our search procedure using the previously trained A2CV agents. Notably, this set also includes the 250 mazes used in our transfer experiments (Section 3.3). Next, we train a new set of A2CV agents using both this adversarial set of mazes and the standard distribution of mazes, sampling randomly every episode (i.e. 50% of training episodes are on an adversarial maze). Randomly perturbed training To ensure our adversarial search procedure produces indistribution adversarial mazes, we alter the default maze generator used in training so that any adversarial maze can be generated. We accomplish this by randomly perturbing the original mazes, repeatedly using the same Modify function used by our adversarial search procedure, but selecting candidates randomly rather than by worst agent performance.
 RESULTS In this section, we report our findings on the robustness of agents trained using the approaches above. Catastrophic failures still exist Our main finding is that while agents learn to perform well on the richer distributions of mazes described above, this does not lead to robust agents. In particular, agents trained on a distribution of mazes enriched with 6000 adversarial mazes were able to find the goal on average 89.8% of the time on the adversarial mazes they were trained on. Similarly, agents trained on randomly perturbed mazes were able to find the goal close to 100% of the time on the distribution they were trained on. However, despite the agents being trained on these richer training distributions, the same search method is still able to find mazes leading to extreme failure as can be seen in Figure  7 . One possible explanation for this result is that the 6000 adversarial mazes used for training were insufficient to get good coverage of the space of mazes, and that further enlarging this set could yield qualitatively different results. Indeed, for agents trained using randomly perturbed mazes, the search procedure took 50 iterations to obtain the same level of failure as it did after 20 iterations when applied to agents trained on the standard training distribution. This suggests that perhaps enriching the training distribution with a very large set of adversarial mazes may lead to more general and robust agents. However, there are a number of challenges that need to be addressed before this approach can be tested which we will describe in the next section.
 DISCUSSION Our results suggest that if a richer training distribution is to yield more robust agents, we may need to use a very large set of environment settings leading to failure. This is similar to how adversarial training in supervised learning is performed where more adversarial examples are used than the original training examples. We describe below what we see as two significant challenges that need to be overcome before such an approach can be thoroughly evaluated in the RL setting.
 Expensive generation The cost of generating a single adversarial setting is on the order of 1000's episodes using the method in this work. This implies that generating a set of adversarial settings which is similar in size to the set trained on would require orders of magnitude more computational than training itself. This could be addressed with faster methods for generating adversarial settings. Expensive training Since agents receive very little reward in adversarial settings, the training signal is incredibly sparse. Therefore, it is possible that many more training iterations are necessary for agents to learn to perform well in each adversarial setting. A possible solution to this challenge is to design a curriculum over adversity, whereby easier variants of the adversarial settings are injected into the training distribution. For example, for the navigation tasks considered here, one could include training settings with challenging mazes where the goal is in any position on the shortest path between the starting location of the agent and the challenging goal. We hope that these challenges can be overcome so that, in the context of RL, the utility of adversarial retraining can be established -an approach which has proved useful in supervised learning tasks. However, since significant challenges remain, we suspect that much effort and many pieces of work will be required before a conclusive answer is achieved.
 RELATED WORK Navigation Recently, there has been significant focus in the RL community on agent navigation in simulated 3D environments, including a community-wide challenge for agents in such environments called VizDoom  (Kempka et al., 2016) . Such 3D first-person navigation tasks are particularly interesting because they capture challenges such as partial observability, and require the agent to "effectively perceive, interpret, and learn the 3D world in order to make tactical and strategic decisions where to go and how to act."  (Kempka et al., 2016) . Recent advances have led to impressive human-level performance on navigation tasks in large procedurally generated environments  (Beattie et al., 2016; Wayne et al., 2018) . Adversarial examples in supervised learning Our work can be seen as an RL navigation analogue of work on adversarial attacks on supervised learning systems for image classification  (Szegedy et al., 2013) . For adversarial attacks on image classifiers, one considers a set of inputs that is larger than the original distribution, but where one would hope that systems perform just as well on L ∞ balls around inputs from the distribution. In particular, the adversarial examples lie outside the training distribution. Analogously, we consider a set of mazes which is larger than the original set of mazes used during training, but where we would hope our system will work just as well. Notably, while similar on a conceptual level, our setting has two key differences from this previous line of work: (1) The attack vector consists of changing latent semantic features of the environment (i.e. the wall structure of a maze), rather than changing individual pixels in an input image in an unconstrained manner. (2) The failure is realized over multiple steps of agent and environment interacting with each other, rather than simply being errant output from a single forward pass through a neural net. More recently, in the context of supervised learning for image classification, there has been work to find constrained adversarial attacks which is closer to what we consider in this work  (Athalye & Sutskever, 2017; Fawzi & Frossard, 2015; Eykholt et al., 2018; Sharif et al., 2016) . In the context of interpretable adversarial examples in image classification, similar approaches to our simplification approach have been explored where one searches for adversarial perturbations with group-sparse structure or other minimal structure  (Xu et al., 2018; Brendel et al., 2018) . Additionally, our findings regarding transfer are consistent with findings on adversarial examples for computer vision networks where it has been found that perturbations that are adversarial for one network often transfer across other networks  (Szegedy et al., 2013; Tramèr et al., 2017)  Input attacks on RL systems There have been a number of previous works which have extended adversarial attacks to RL settings, however they have achieved this by manipulating inputs directly, which effectively amounts to changing the environment renderer  (Huang et al., 2017; Lin et al., 2017a; b) . While these are interesting from a security perspective, it is less clear what they tell us about the generality of the strategy learned by the agent. Generalization in RL systems Recently, it has been shown that simple agents trained on restricted datasets fail to learn sufficiently general navigation strategies to improve goal retrieval times on held out mazes  (Dhiman et al., 2018) . In comparison, our method is both automatic and able to find more spectacular failures. Further, our findings highlight failures in exploration during navigation. This is in contrast to this previous work which studied failures to exploit knowledge from previous goal retrievals in the same episode. In the context of testing generalization in RL, previous work has looked at statistical generalization in RL . Here we consider agents that already generalize in the statistical sense and try to better characterize the ways in which they generalize beyond the average-case.
 CONCLUSIONS AND FUTURE WORK In this work, we have shown that despite the strong average-case performance often reported of RL agents, worst-case analysis can uncover environment settings which agents have failed to generalize to. Notably, we have found that not only do catastrophic failures exist, but also that simple catastrophic failures exist which we would hope agents would have generalized to, and that failures also transfer between agents and architectures. As agents are trained to perform increasingly complicated tasks in more sophisticated environments, for example AirSim  (Shah et al., 2017)  and CARLA  (Dosovitskiy et al., 2017) , it is of interest to understand their worst-case performance and modes of generalization. Further, in real world applications such as self-driving cars, industrial control, and robotics, searching over environment settings to investigate and address such behaviours is likely to be critical on the path to robust and generalizable agents. To conclude, while this work has focused mostly on evaluation and understanding, it is only a first step towards the true goal of building more robust, general agents. Initial results we report indicate that enriching the training distribution with settings leading to failure may need to be done at a large scale if it is to work, which introduces significant challenges. While training robust agents is likely an endeavour requiring significant effort, we believe it is important if agents are to carry out critical tasks and on the path to finding more generally intelligent agents. goal was <50% (compared to 98% on the average maze). Furthermore, the 25th, 50th, and 75th percentiles were as follows: • p(reaching the goal): 0.031, 0.136, 0.279 • number of goals reached: 0.042, 0.136, 0.368
 A.3 HUMAN EXPERIMENTS To upper bound the intrinsic difficulty of the mazes found to be adversarial to agents, we conducted experiments where three humans played on the same mazes. Each human played a single episode on each of ten mazes. The humans played at the same resolution as agents, 96x72 pixels, to rule out visual acuity as a confounding factor. On all mazes, all humans successfully found the goal in the course of the episode. In fact, in most episodes, humans were able to find the goal in less than a third of the episode. Figure  8 : Mazes used for human experiments. For each maze, the agent that performed best found the goal less than 50% of the time. In contrast, humans always found the goal, usually within less than a third of the episode. Note that humans played at the same resolution as agents, 96x72 pixels.    Figure 1 : 1 Figure 1: Navigation task. (left) Example maze from the training distribution together with the path taken by the agent from spawn (cyan) to goal (magenta). (right) Frames from top left to bottom right correspond to agent observations as it takes the path from spawn to goal. Note that while the navigation task may look simple given a top down view, the agent only receives very partial information about the maze at every step, making navigation a difficult task.
 Figure 2 : 2 Figure 2: Example of search procedure. First, we generate a set of 10 initial candidate mazes by sampling from the training distribution.We then Evaluate each with the agent over 30 episodes, select the best maze (i.e. lowest agent score), and Modify this maze by randomly moving two walls to form the next set of candidates (Iteration 1). This process is repeated for 20 iterations, leading to a maze where the agent score is 0.09 in this example (i.e. the agent finds the goal once in 11 episodes). In Appendix A.2.1 we detail the computational requirements of this search procedure.
 (a) Average number of goals reached per episode over the course of the optimization. (b) Probability of the agent reaching the goal in an episode.
 Figure 3 : 3 Figure 3: The search algorithm is able to rapidly find mazes where agents fail to find the goal. (a) The objective used for the optimizer is average agent score. The dashed line corresponds to average goals reached on randomly perturbed mazes. (b) Minimizing score also leads to a low probability of at least one goal retrieval in an episode. The dashed line corresponds to average probability of reaching a goal on randomly perturbed mazes. The blue lines are computed by averaging across 50 optimization runs.
 Figure 4 : 4 Figure 4: Example mazes leading to low scores and example trajectories on these mazes. (a) Maze with randomly perturbed walls. Despite being out of distribution, agents find the goal in 98% of episodes on such mazes and are able to get the goal on average 45 times per episode. (b) Maze obtained after 20 iterations, moving two walls at each iteration to minimize reward. All agents find the goal on such mazes in less than 20% of episodes. (c) Maze obtained through additional iterations of removing walls. All agents find the goal on such mazes in less than 40% of episodes. (d) Human trajectory on the same maze as in (c). Humans are able to consistently find the goal on such mazes.
 (a) Agent locations, with goal (magenta) fixed.(b) Goal locations, with agent (cyan) fixed.
 Figure 5 : 5 Figure 5: Adversarial mazes are robust to change of spawn positions. The probability of goal retrieval (shown with the color bar) remains low across large portions of the simplified maze as the (a) agent spawn locations and (b) goal locations are moved for each episode..
 (a) Across hyperparameters, same architecture. (b) Across architectures.
 (a) Average number of goals reached. (b) Probability of finding the goal in a single episode.
 Figure 7 : 7 Figure 7: Richer training distributions did not lead to robust agents. Adversarial optimization for agents trained with adversarial mazes (red) and for agents trained with randomly perturbed mazes (yellow). Compared against the Standard training method from Figure 3 for 50 iterations (blue).
 Figure 9 : 9 Figure 9: Trajectories taken by Human 3 on mazes leading to agent failure.
 Table 1 : 1 Human seconds-to-first-goal on agent failure mazes we provide detailed results for our transfer experiments. In particular, we detail transfer between all pairs among the 10 agents, five A2CV agents and five MERLIN agents trained with different entropy costs and learning rates.
 Figure 10 : 10 Figure 10: Pairwise transfer scores. Lower number indicates more transfer. 'A' corresponds to our A2CV agent, and 'M' corresponds to MERLIN.
 end best ← Evaluate(candidates, num evaluations); Algorithm 1: Method for finding environment settings leading to failure cases. Concretely, we generate a set of initial candidate mazes by sampling mazes from the training distribution. We then use the Modify function on the maze which yielded the lowest agent score to randomly move two walls to produce a new set of candidates, rejecting wall moves that lead to unsolvable mazes. Importantly, this method is able to effectively find catastrophic failure cases (as we demonstrate in Section 3.1), while also having the advantage of being intuitive to understand and implement. input : num iterations, num candidates, num evaluations and function Modify output: An environment setting best candidates ← GenerateCandidates(num candidates); for i ← 1 to num iterations do best ← Evaluate(candidates, num evaluations); candidates ← Modify(best, num candidates);