Uncovering Surprising Behaviors in Reinforcement Learning via Worst-case Analysis

1 INTRODUCTION . Reinforcement Learning ( RL ) methods have achieved great success over the past few years , achieving human-level performance on a range of tasks such as Atari ( Mnih et al. , 2015 ) , Go ( Silver et al. , 2016 ) , Labyrinth ( Espeholt et al. , 2018 ) , and Capture the Flag ( Jaderberg et al. , 2018 ) . On these tasks , and more generally in reinforcement learning , agents are typically trained and evaluated using their average reward over environment settings as the measure of performance , i.e . EP ( e ) [ R ( π ( θ ) , e ) ] , where π ( θ ) denotes a policy with parameters θ , R denotes the total reward the policy receives over the course of an episode , and e denotes environment settings such as maze structure in a navigation task , appearance of objects in the environment , or even the physical rules governing environment dynamics . But what biases does the distribution P ( e ) contain , and what biases , or failures to generalize , do these induce in the strategies agents learn ? To help uncover biases in the training distribution and in the strategies that agents learn , we propose evaluating the worst-case performance of agents over environment settings , i.e . min e∈E E [ R ( π ( θ ) , e ) ] , where E is some set of possible environment settings . Worst-case analysis can provide an important tool for understanding robustness and generalization in RL agents . For example , it can help us with : • Understanding biases in training Catastrophic failures can help reveal situations that are rare enough during training that the agent does not learn a strategy that is general enough to cope with them . • Robustness For critical systems , one would want to eliminate , or at least greatly reduce , the probability of extreme failures . • Limiting exploitability If agents have learned strategies that fail to generalize to particular environment settings , then an adversary could try and exploit an agent by trying to engineer such environment settings leading to agent failure . In this work , we use worst-case analysis to investigate the performance of a state-of-the-art agent in solving a first-person 3D navigation task ; a task on which agents have recently achieved average-case human-level performance ( Wayne et al. , 2018 ) . By optimizing mazes to minimize the performance of agents , we discover the existence of mazes where agents repeatedly fail to find the goal ( which we refer to as a Catastrophic Failure ) . Our Contributions To summarize , the key contributions of this paper are as follows : 1 . We introduce an effective and intuitive approach for finding simple environment settings leading to failure ( Section 2 ) . 2 . We show that state-of-the-art agents carrying out navigation tasks suffer from drastic and often surprising failure cases ( Sections 3.1 and 3.2 ) . 3 . We demonstrate that mazes leading to failure transfer across agents with different hyperparameters and , notably , even different architectures ( Section 3.3 ) . 4 . We present an initial investigation into how the training distribution can be adapted by incorporating adversarial and out-of-distribution examples ( Section 4 ) . 2 APPROACH . Tasks We consider agents carrying out first-person 3D navigation tasks . Navigation is of central importance in RL research as it captures the challenges posed by partially observable Markov decision processes ( POMDPs ) . The navigation tasks we use are implemented in DeepMind Lab ( DM Lab ) ( Beattie et al. , 2016 ) . 1 Each episode is played on a 15 × 15 maze where each position in the maze may contain a wall , an agent spawn point , or a goal spawn point . The maze itself is procedurally generated every episode , along with the goal and agent spawn locations . The goal location remains fixed throughout an episode , while the agent spawn location can vary . In training , the agent respawns at different locations each time they reach the goal , while for our optimization and analysis we limit the agent to the same spawn location . Agents receive RGB observations of size 96× 72 pixels , examples of which are provided in Figure 1 . Episodes last for 120 seconds and 1A full description and code for the tasks can be found at https : //github.com/deepmind/lab/ tree/master/game_scripts/levels/contributed/dmlab30 # goal-locations-large . are played at a framerate of 15 frames per second . The agent receives a positive reward of 10 every time it reaches the goal , and 0 otherwise . On this specific navigation task , RL agents have recently achieved human-level average-case performance ( Wayne et al. , 2018 ) . Agents We perform our analysis on Importance Weighted Actor-Learner Architecture agents trained to achieve human-level average-case performance on navigation tasks . These agents can be described as async batched-a2c agents with the V-trace algorithm for off policy-correction , and we henceforth refer to these as A2CV agents ( Espeholt et al. , 2018 ) . Details of the training procedure are provided in Appendix A.1 . Search Algorithm If we are interested in worst-case performance of agents , how can we find environment settings leading to the worst performance ? In supervised learning , one typically uses gradient based methods to find inputs that lead to undesired output ( Biggio et al. , 2013 ; Szegedy et al. , 2013 ; Goodfellow et al. , 2014 ) . In contrast , we search for environment settings leading to an undesired outcome at the end of an episode . This presents a challenge as the environment rendering and MDP are not differentiable . We are therefore limited to black-box methods where we can only query agent performance given environment settings . To search for environment settings which cause catastrophic failures , we propose the local search procedure described in Algorithm 1 ( visualizing the process in Figure 2 ) . Concretely , we generate a set of initial candidate mazes by sampling mazes from the training distribution . We then use the Modify function on the maze which yielded the lowest agent score to randomly move two walls to produce a new set of candidates , rejecting wall moves that lead to unsolvable mazes . Importantly , this method is able to effectively find catastrophic failure cases ( as we demonstrate in Section 3.1 ) , while also having the advantage of being intuitive to understand and implement . input : num iterations , num candidates , num evaluations and function Modify output : An environment setting best candidates← GenerateCandidates ( num candidates ) ; for i← 1 to num iterations do best← Evaluate ( candidates , num evaluations ) ; candidates← Modify ( best , num candidates ) ; end best← Evaluate ( candidates , num evaluations ) ; Algorithm 1 : Method for finding environment settings leading to failure cases . 3 EXPERIMENTS . The agents we study achieve impressive average-case performance , but how much does their worstcase performance differ from their average-case performance ? To investigate this , we consider the worst-case performance over a large set of mazes , including mazes that are not possible under the training distribution . A natural question to ask is whether any departure from the wall structure present during training will lead to agent failure . To test this , we evaluate the agent on samples from a distribution of mazes containing all mazes agents could be evaluated on during the search . In particular , we randomly select agent and goal spawn locations in the first step and then randomly move 40 walls , corresponding to the same actions taken by our optimization procedure , but where the actions are chosen randomly rather than in order to minimize agent performance . We find that agents do generalize to random mazes from the set we consider . In fact , we find that the average score obtained by agents on randomly perturbed mazes is slightly higher than on the training distribution , with agents obtaining an average of 45 goal reaches per two minute episode . The increased performance is likely due to the agent spawn location being fixed , making it easier for the agent to return to the goal once found . The considered agents generalize in the sense that agent performance is not reduced on average by out-of-distribution wall structure . But what about the worst case over all wall structures ? Have the agents learned a general navigation strategy that works for all solvable mazes ? Or do there exist environment settings that lead to catastrophic failures with high probability ? In this section , we investigate these questions . We define a Catastrophic Failure to be an agent failing to find the goal in a two minute episode ( 1800 steps ) . As detailed below , we find that not only do there exist mazes leading to catastrophic failure , there exist surprisingly simple mazes that lead to catastrophic failure for agents yet are consistently and often rapidly solved by humans . 3.1 RESULT 1 : CATASTROPHIC FAILURES EXIST . Do environment settings leading to catastrophic failure exist for the agents we are considering ? By searching over mazes using the procedure outlined in Algorithm 1 , we find mazes where agents fail to find the goal on many episodes , only finding the goal 10 % of the time . In fact , some individual mazes lead to failure across five different agents we tested , with even the best performing agent only finding the goal in 20 % of the episodes . Optimization curves for our search procedure are given in Figure 3 . Note that while we define catastrophic failure as failure to find the goal , the actual objective used for the optimization was average number of goal reaches during an episode . Using average number of goals gives a stronger signal at the start of the optimization process . Finding mazes leading to lower average number of captures is easier than finding mazes where the agent rarely finds the goal even once . As can be seen , despite finding the goal on average 45 times per episode on randomly perturbed mazes , on mazes optimized to reduce score , agents find the goal on average only 0.33 times per episode , more than a 100× decrease in performance . In terms of probability of catastrophic failure , we note that despite agents finding the goal in approximately 98 % of episodes on randomly perturbed mazes , using our method , on average we find mazes where agents only finds the goal in 30 % of the episodes . Example trajectories agents take during failures are visualized in Figure 4 . The trajectories often seem to demonstrate a failure to use memory to efficiently explore the maze with the agent repeatedly visiting the same locations multiple times . The mazes presented in Figure 4 appear to be of higher complexity than mazes seen during training . This suggests that to obtain agents that truly master navigation , more complex mazes should be included in the training distribution . However , we can ask whether it is only more complex mazes that lead to catastrophic failure or whether there are also simple mazes leading to catastrophic failure . This is a question we explore in the next subsection . 3.2 RESULT 2 : SIMPLE CATASTROPHIC FAILURES EXIST . While the existence of catastrophic failures may be intriguing and perhaps troubling , one might suspect that the failures are caused by the increased complexity of the mazes leading to failure relative to the mazes the agent is exposed to during training , e.g. , the mazes leading to failure contain more dead ends and sometimes have lower visibility . Further , understanding the cause of failure in such mazes seems challenging due to the large number of wall structures that may be causing the agent to fail . In this section , we explore whether there exist simple mazes which lead to catastrophic failures . As our measure of complexity , we use the total number of walls in the maze . We also evaluate humans on such mazes to get a quantitative measure of maze complexity . To find simple mazes which lead to failure , we first follow the same procedure as in the previous section , producing a set of mazes which all lead to catastrophic failures ( i.e . a low agent scores ) . Next , we use this set of mazes as the initial set of candidates in our search algorithm , however we now use a Modify function that removes a single randomly chosen wall each iteration . This process is repeated for 70 iterations , searching for a maze with few walls while maintaining low agent score . In Figure 4 , we present the resulting simple mazes and the corresponding agent trajectories from our optimization procedure . Interestingly , we find that one can remove a majority of the walls in a maze and still maintain the catastrophic failure ( i.e . very low agent score ) . Of note is that a number of these mazes are strikingly simple , suggesting that there exists structure in the environment that the agent has not generalized to . For example , we can see that placing the goal in a small room in an otherwise open maze can significantly reduce the agent ’ s ability to find the goal . Human baselines While these simple maps may lead to catastrophic failure , it is unclear whether this is because of the agent or whether the maze is difficult in a way that is not obvious . To investigate this , we perform human experiments by tasking humans to play on a set of 10 simplified mazes . Notably , we find that human players are able to always locate the goal in every maze and typically within one third of the full episode length . This demonstrates that the mazes are comfortably solvable within the course of an episode by players with a general navigation strategy . We provide a detailed comparison of agent and human performance in Appendix A.3 . Analysis One question that may arise is the extent to which these mazes are isolated points in the space of mazes . That is , if the maze was changed slightly , would it no longer lead to catastrophic failure ? To test this , we investigate how sensitive our discovered failure mazes are with respect to the agent and goal spawn locations on simplified adversarial mazes . As can be seen in Figure 5 , we find that for a large range of spawn locations , the mazes still lead to failure . This indicates that there is specific local maze structure which causes agents to fail . Procedures for finding such simple mazes may prove useful as a tool for debugging agents and understanding the ways in which training has led them to develop narrow strategies that are good enough for achieving high average-case performance .

Summary: The authors present a simple technique for finding "worst-case" maze environments that result in bad performance. The adversarial optimization procedure is a greedy procedure, which alternately perturbs maze environments and selects the maze on which the trained agent performs worst for the next iteration. The authors highlight three properties of these mazes, which show how this adversarial optimization procedure can negatively impact performance.