Finding and Visualizing Weaknesses of Deep Reinforcement Learning Agents

1 INTRODUCTION . Humans can naturally learn and perform well at a wide variety of tasks , driven by instinct and practice ; more importantly , they are able to justify why they would take a certain action . Artificial agents should be equipped with the same capability , so that their decision making process is interpretable by researchers . Following the enormous success of deep learning in various domains , such as the application of convolutional neural networks ( CNNs ) to computer vision ( LeCun et al. , 1998 ; Krizhevsky et al. , 2012 ; Long et al. , 2015 ; Ren et al. , 2015 ) , a need for understanding and analyzing the trained models has arisen . Several such methods have been proposed and work well in this domain , for example for image classification ( Simonyan et al. , 2013 ; Zeiler & Fergus , 2014 ; Fong & Vedaldi , 2017 ) , sequential models ( Karpathy et al. , 2016 ) or through attention ( Xu et al. , 2015 ) . Deep reinforcement learning ( RL ) agents also use CNNs to gain perception and learn policies directly from image sequences . However , little work has been so far done in analyzing RL networks . We found that directly applying common visualization techniques to RL agents often leads to poor results . In this paper , we present a novel technique to generate insightful visualizations for pre-trained agents . Currently , the generalization capability of an agent is—in the best case—evaluated on a validation set of scenarios . However , this means that this validation set has to be carefully crafted to encompass as many potential failure cases as possible . As an example , consider the case of a self-driving agent , where it is near impossible to exhaustively model all interactions of the agent with other drivers , pedestrians , cyclists , weather conditions , even in simulation . Our goal is to extrapolate from the training scenes to novel states that induce a specified behavior in the agent . In our work , we learn a generative model of the environment as an input to the agent . This allows us to probe the agent ’ s behavior in novel states created by an optimization scheme to induce specific actions in the agent . For example we could optimize for states in which the agent sees the only option as being to slam on the brakes ; or states in which the agent expects to score exceptionally low . Visualizing such states allows to observe the agent ’ s interaction with the environment in critical scenarios to understand its shortcomings . Furthermore , it is possible to generate states based on an objective function specified by the user . Lastly , our method does not affect and does not depend on the training of the agent and thus is applicable to a wide variety of reinforcement learning algorithms . Our contributions are : 1 . We introduce a series of objectives to quantify different forms of interestingness and danger of states for RL agents . 2 . We evaluate our algorithm on 50 Atari games and a driving simulator , and compare performance across three different reinforcement learning algorithms . 3 . We quantitatively evaluate parts of our model in a comprehensive loss study ( Tab . 1 ) and analyze generalization though a pixel level analysis of synthesized unseen states ( Tab . 2 ) . 4 . An extensive supplement shows additional comprehensive visualizations on 50 Atari games . We will describe our method before we will discuss relevant related work from the literature . 2 METHODS . We will first introduce the notation and definitions that will be used through out the remainder of the paper . We formulate the reinforcement learning problem as a discounted , infinite horizon Markov decision process ( S , A , γ , P , r ) , where at every time step t the agent finds itself in a state st ∈ S and chooses an action at ∈ A following its policy πθ ( a|st ) . Then the environment transitions from state st to state st+1 given the model P ( st+1|st , at ) . Our goal is to visualize RL agents given a user-defined objective function , without adding constraints on the optimization process of the agent itself , i.e . assuming that we are given a previously trained agent with fixed parameters θ . We approach visualization via a generative model over the state space S and synthesize states that lead to an interesting , user-specified behavior of the agent . This could be , for instance , states in which the agent expresses high uncertainty regarding which action to take or states in which it sees no good way out . This approach is fundamentally different than saliency-based methods as they always need an input for the test-set on which the saliency maps can be computed . The generative model constrains the optimization of states to induce specific agent behavior . 2.1 STATE MODEL . Often in feature visualization for CNNs , an image is optimized starting from random noise . However , we found this formulation too unconstrained , often ending up in local minima or fooling examples ( Figure 3a ) . To constrain the optimization problem we learn a generative model on a set S of states generated by the given agent that is acting in the environment . The model is inspired by variational autoencoders ( VAEs ) ( Kingma & Welling , 2013 ) and consists of an encoder f ( s ) = ( fµ ( s ) , fσ ( s ) ) ∈ R2×n that maps inputs to a Gaussian distribution in latent space and a decoder g ( z ( s ) ) = ŝ that reconstructs the input . z ( s ) = fµ ( s ) + fσ ( s ) is a sample from the predicted distribution , that is obtained via the reparametrization trick , where is sampled from N ( 0 , In ) . The training of our generator has three objectives . First , we want the generated samples to be close to the manifold of valid states s. To avoid fooling examples , the samples should also induce correct behavior in the agent and lastly , sampling states needs to be efficient . We encode these goals in three corresponding loss terms . L ( s ) = Lp ( s ) + ηLa ( s ) +KL ( f ( s ) , N ( 0 , In ) ) , ( 1 ) The role of Lp ( s ) is to ensure that the reconstruction g ( z ( s ) ) is close to the input s such that ‖ g ( z ( s ) ) − s ‖22 is minimized . We observe that in the typical reinforcement learning benchmarks , such as Atari games , small details—e.g . the ball in Pong or Breakout—are often critical for the decision making of the agent . However , a typical VAE model tends to yield blurry samples that are not able to capture such details . To address this issue , we model the reconstruction error Lp ( s ) with an attentive loss term , which leverages the saliency of the agent to put focus on critical regions of the reconstruction . The saliency maps are computed by guided backpropagation of the policy ’ s gradient with respect to the state . Lp ( s ) = d∑ i=1 ( g ( z ( s ) ) i − si ) 2 |∇π ( s ) i|∑d j=1 | ∇π ( s ) j | , ( 2 ) where i and j iterate over all d pixels in the image/gradient . As discussed earlier , gradient based reconstruction methods might not be ideal for explaining a CNN ’ s reasoning process ( Kindermans et al. , 2017a ) . Here however , we only use it to focus the reconstruction on salient regions of the agent and do not use it to explain the agent ’ s behavior for which these methods are ideally suited . This approach puts emphasis on details ( salient regions ) when training the generative model . Since we are interested in the actions of the agent on synthesized states , the second objective La ( s ) is used to model the perception of the agent : La ( s ) = ‖A ( s ) −A ( g ( z ( s ) ) ) ‖22 , ( 3 ) where A is a generic formulation of the output of the agent . For a DQN for example , π ( s ) = maxaA ( s ) a , i.e . the final action is the one with the maximal Q-value . This term encourages the reconstructions to be interpreted by the agent the same way as the original inputs s. The last term KL ( f ( s ) , N ( 0 , In ) ) ensures that the distribution predicted by the encoder f stays close to a Gaussian distribution . This allows us to initialize the optimization with a reasonable random vector later and forms the basis of a regularizer . Thus , after training , the model approximates the distribution of states p ( s ) by sampling z directly from N ( 0 , In ) . The generative model ( f and g ) is trained with ( 1 ) . We will then use the generator inside an optimization scheme to generate state samples that satisfy a user defined target objective . 2.2 SAMPLING STATES OF INTEREST . Training a generator with the objective function of Equation 1 allows us to sample states that are not only visually close to the real ones , but which the agent can also interpret and act upon as if they were states from a real environment . We can further exploit this property and formulate an energy optimization scheme to generate samples that satisfy a specified objective . The energy operates on the latent space x = ( xµ , xσ ) of the generator and is defined as the sum of a target function T on agent ’ s policy and a regularizer R E ( x ) = T ( π ( g ( xµ + xσ ) ) + αR ( x ) . ( 4 ) The target function can be defined freely by the user and depends on the agent that is being visualized . For a DQN , one could for example define T as the Q-value of a certain action , e.g . pressing the brakes of a car . In section 2.3 , we show several examples of targets that are interesting to analyze . The regularizer R can again be chosen as the KL divergence between x and the normal distribution : R ( x ) = KL ( x , N ( 0 , In ) ) , ( 5 ) forcing the samples that are drawn from the distribution x to be close to the Gaussian distribution that the generator was trained with . We can optimize equation 4 with gradient descent on x as detailed in algorithm 1 . 2.3 TARGET FUNCTIONS . Depending on the agent , one can define several interesting target functions T – we present and explore seven below , which we refer to as : T+ , T− , T± , S+ , S− , and action maximization . For a DQN the previously discussed action maximization is interesting to find situations in which the agent assigns a high value to a certain action e.g . Tleft ( s ) = −Aleft ( s ) . Other states of interest are those to which the agent assigns a low ( or high ) value for all possible actions A ( s ) = q = ( q1 , . . . , qm ) . Consequently , one can optimize towards a low Q-value for the highest valued action with the following objective : T− ( q ) = ∑m i=1 qie βqi∑m k=1 e βqk , ( 6 ) Algorithm 1 Optimize x for target T 1 : Input : Target objective T , step size λ , regularizer weight α , trained generator g 2 : Output : x 3 : xµ ← 0 4 : xσ ← In 5 : while not converged do 6 : ← sample from N ( 0 , In ) 7 : z ← xµ + xσ . sample z from x 8 : s← g ( z ) . generate state s using g 9 : xµ ← xµ − λ ∂∂xµ ( T ( π ( s ) ) + αR ( x ) ) . gradient step using ( 4 ) and ( 5 ) 10 : xσ ← xσ − λ ∂∂xσ ( T ( π ( s ) ) + αR ( x ) ) 11 : end while ( a ) Kung Fu Master - T± enemies on both sides ( b ) Kung Fu Master - T+ easy , many points to score ( c ) Kung Fu Master - T− no enemies ( d ) Pong - T+ scoring a point ( e ) Space Invaders - T+ shooting an enemy ( f ) Enduro - T+ overtaking an opponent ( g ) Name This Game - T± whether to refill air ( h ) Seaquest - T− out of oxygen ( i ) Beamrider - Tleft avoiding the enemy Figure 1 : Qualitative Results : Visualization of different target functions ( Sec . 2.3 ) . T+ generates high reward and T− low reward states ; T± generates states in which one action is highly beneficial and another is bad . For a long list of results , with over 50 Atari games , please see the appendix . where β > 0 controls the sharpness of the soft maximum formulation . Analogously , one can maximize the lowest Q-value with T+ ( q ) = −T− ( −q ) . We can also optimize for interesting situations in which one action is of very high value and another is of very low value by defining T± ( q ) = T− ( q ) − T+ ( q ) . ( 7 ) Finally , we can optimize for overall good states with S+ ( q ) = ∑m i=1 qi and overall bad states with S− = S− ( −q ) .

Summary: This paper proposes a new visualization tool in order to understand the behavior of agents trained using deep RL. Specifically, they train a generative model of game states, and then optimize an energy-based distribution over state embeddings according to some target function, and then by sampling from the resulting distribution they create a diverse set of realistic states that score highly according to the target function. They propose a few target cost functions, which allow them to optimize for states in which the agent takes a particular action, states which are high reward (worst Q-value is large), states which are low reward (best Q-value is small), and critical states. They demonstrate results on Atari games as well as a simulated driving environment.