Finding and Visualizing Weaknesses of Deep Reinforcement Learning Agents
Authors: Christian Rupprecht
Cyril Ibrahim
Christopher J. Pal
Venue:  (2020)
Keywords: Visualization
Reinforcement Learning
Safety

Abstract: As deep reinforcement learning driven by visual perception becomes more widely used there is a growing need to better understand and probe the learned agents. Understanding the decision making process and its relationship to visual inputs can be very valuable to identify problems in learned behavior. However, this topic has been relatively under-explored in the research community. In this work we present a method for synthesizing visual inputs of interest for a trained agent. Such inputs or states could be situations in which specific actions are necessary. Further, critical states in which a very high or a very low reward can be achieved are often interesting to understand the situational awareness of the system as they can correspond to risky states. To this end, we learn a generative model over the state space of the environment and use its latent space to optimize a target function for the state of interest. In our experiments we show that this method can generate insights for a variety of environments and reinforcement learning methods. We explore results in the standard Atari benchmark games as well as in an autonomous driving simulator. Based on the efficiency with which we have been able to identify behavioural weaknesses with this technique, we believe this general approach could serve as an important tool for AI safety applications.

Published as a conference paper at ICLR 2020
FINDING AND VISUALIZING WEAKNESSES OF DEEP REINFORCEMENT LEARNING AGENTS
Christian Rupprecht
Cyril Ibrahim²
Christopher J. 2.
1 Visual Geometry Group, University of Oxford
2 Element Al
3 Polytechnique Montréal, Mila & Canada CIFAR Al Chair
ABSTRACT
As deep reinforcement learning driven by visual perception becomes more widely used there is a growing need to better understand and probe the learned agents. Understanding the decision making process and its relationship to visual inputs can be very valuable to identify problems in learned behavior. However, this topic has been relatively under-explored in the research community. In this work we present method for synthesizing visual inputs of interest for trained agent. Such inputs or states could be situations in which specific actions are necessary. Further, critical states in which a very high or very low reward can be achieved are often interesting to understand the situational awareness of the system as they can correspond to risky states. To this end, we learn generative model over the state space of the environment and use its latent space to optimize a target function for the state of interest. In our experiments we show that this method can generate insights for a variety of environments and reinforcement learning methods. We explore results in the standard Atari benchmark games well in an autonomous driving simulator. Based on the efficiency with which we have been able to identify behavioural weaknesses with this technique, we believe this general approach could serve as an important tool for AI safety applications.
INTRODUCTION
Humans can naturally learn and perform well at wide variety of tasks, driven by instinct and practice; more importantly, they able justify why they would take certain action. Artificial agents should be equipped with the same capability, so that their decision making process is interpretable by researchers. Following the enormous success of deep learning in various domains, such as the application of convolutional neural networks (CNNs) to computer vision (LeCun et al., 1998; Krizhevsky et al., 2012; Long al., 2015; Ren al., 2015), 5), need for understanding and analyzing the trained models has arisen. Several such methods have been proposed and work well in this domain. for example for image classification (Simonyan et al., 2013: Zeiler Fergus, 2014; Fong Vedaldi, 2017), sequential models (Karpathy et al., 2016) or through attention (Xu et al., 2015).
Deep reinforcement learning (RL) agents also use CNNs to gain perception and learn policies directly from image sequences. However, little work has been so far done in analyzing RL networks. We found that directly applying common visualization techniques to RL agents often leads to poor results. In this paper, we present a novel technique to generate insightful visualizations for pre-trained agents.
Currently, the generalization capability of an agent is-in the best case aase-evaluated on a validation set of scenarios. However, this means that this validation set has to be carefully crafted to encompass as many potential failure cases as possible. As an example, consider the case of a self-driving agent, where it is near impossible exhaustively model all interactions of the agent with other drivers, pedestrians, cyclists, weather conditions, even in simulation. Our goal is to extrapolate from the training scenes to novel states that induce specified behavior in the agent.
In our work, we learn generative model of the environment as an input to the agent. This allows us to probe the agent's behavior in novel states created by an optimization scheme to induce specific

---

Published as a conference paper at ICLR 2020
actions in the agent. For example could optimize for states in which the agent sees the only option as being to slam on the brakes; or states in which the agent expects to score exceptionally low. Visualizing such states allows to observe the agent's ' interaction with the environment in critical scenarios to understand its shortcomings. Furthermore, it is possible to generate states based on an objective function specified by the user. Lastly, method does not affect and does not depend on the training of the agent and thus is applicable to wide variety of reinforcement learning algorithms.
Our contributions are:
1. We introduce series of objectives to quantify different forms of interestingness and danger of states for RL agents.
2. We evaluate our algorithm on 50 Atari games and a driving simulator, and compare perfor- mance across three different reinforcement learning algorithms.
3. We quantitatively evaluate parts of our model in comprehensive loss study (Tab. 1) and analyze generalization though a pixel level analysis of synthesized unseen states (Tab. 2).
4. An extensive supplement shows additional comprehensive visualizations on 50 Atari games.
We will describe our method before we will discuss relevant related work from the literature.
2 METHODS
We will first introduce the notation and definitions that will be used through out the remainder of the paper. We formulate the reinforcement learning problem as a discounted, infinite horizon Markov decision process (8, A, 7, r). where every time step the agent finds itself in state E S and chooses an action a₂ E A following its policy Te(a)s;). Then the environment transitions from state 8t to state $1+1 given the model P(st41 St, at). Our goal is to visualize RL agents given user-defined objective function, without adding constraints on the optimization process of the agent itself, i.e. assuming that we are given a previously trained agent with fixed parameters O.
We approach visualization via generative model over the state space S and synthesize states that lead an interesting, user-specified behavior of the agent. This could be, for instance, states in which the agent expresses high uncertainty regarding which action to take or states in which it sees no good way out. This approach fundamentally different than saliency-base need an input for the test-set which the saliency maps can be computed. The generative model constrains the optimization of states to induce specific agent behavior.
methods as they always
2.1 STATE MODEL
Often in feature visualization for CNNs, an image optimized starting from random noise. However, we found this formulation too unconstrained, often ending up in local minima or fooling examples (Figure 3a). To constrain the optimization problem we learn generative model on a set 8 of states generated by the given agent that is acting in the environment. The model is inspired by variational autoencoders (VAEs) (Kingma & Welling, 2013) and consists of an encoder f(s) = (s), fo (s)) E R²xx that maps inputs to a Gaussian distribution in latent space and a decoder g(=(s)) & that reconstructs the input. =(x) = (s) + So (s) @ is sample from the predicted distribution, that is obtained via the reparametrization trick, where is sampled from (0, I.). The training of our generator has three objectives. First, want the generated samples to be close to the manifold of valid states s. To avoid fooling examples, the samples should also induce correct behavior in the agent and lastly, sampling states needs to be efficient. We encode these goals in three corresponding loss terms.
(1) The role of (s) is to ensure that the reconstruction g(=(s)) is close to the input 8 such that [1{()(s)) s 112 is minimized. We observe that in the typical reinforcement learning benchmarks, such as Atari games, small details- e.g. the ball in Pong or Breakout- are often critical for the decision making of the agent. However, typical VAE model tends to yield blurry samples that are not able to capture such details. To address this issue, we model the reconstruction error (s(s) with an atfentive loss term, which leverages the saliency of the agent to put focus on critical regions of the
((s) = (s) + (s))ss + (V),
2

---

Published as a conference paper at ICLR 2020
reconstruction. The saliency maps are computed by guided backpropagation of the policy's 's gradient with respect to the state.
Cp(s) = Σ (g(=(s)); -:)² V+((s:) i=1 Σ [x(s)
(2)
where i and j iterate over all d pixels in the image/gradient. As discussed earlier, gradient based reconstruction methods might not be ideal for explaining a CNN's reasoning process (Kindermans et al., 2017a). Here however, we only use it to focus the reconstruction on salient regions of the agent and do not use explain the agent's behavior for which these methods are idcally suited. This approach puts emphasis on details (salient regions) when training the generative model.
Since we are interested in the actions of the agent on synthesized states, the second objective (s) is used to model the perception of the agent:
Ca(s)
|| A(s) A( A(
(3)
where A is generic formulation of the output of the agent. For DQN for example, T(s) max, A(s)a. i.c. the final action is the one with the maximal Q-value. This term encourages the reconstructions be interpreted by the agent the same way as the original inputs s. The last term KC((((s) (O, I,,) ensures that the distribution predicted by the encoder f stays close to Gaussian distribution. This allows us to initialize the optimization with reasonable random vector later and forms the basis of regularizer. Thus, after training, the model approximates the distribution of states p(s) by sampling directly from (0, In). The generative model (ʃ and g) is trained with (I). We will then the generator inside optimization scheme to generate state samples that satisfy user defined target objective.
2.2 SAMPLING STATES OF INTEREST
Training a generator with the objective function of Equation allows us to sample states that are not only visually close the real ones, but which the agent can also interpret and act upon as if they were states from a real enviromment.
We can further exploit this property and formulate an energy optimization scheme to generate samples that satisfy a specified objective. The energy operates on the latent space x (Iz, Zg) of the generator and is defined as the sum of target function T on agent's 's policy and a regularizer
E(x) = T( g( + Ia ②e)) + aR(()).
(4)
The target function can be defined freely by the user and depends on the agent that is being visualized. For DQN, one could for example define T as the Q-value of certain action, e.g. pressing the brakes of a car. In section 2.3, we show several examples of targets that are interesting to analyze. The regularizer R can again be chosen the KL divergence between x and the normal distribution:
R(x) = KL x. N( I..) )
(5)
forcing the samples that are drawn from the distribution to be close the Gaussian distribution that the generator was trained with. We can optimize equation 4 with gradient descent on x as detailed in algorithm 1.
2.3 TARGET FUNCTIONS
Depending on the agent, one can define several interesting target functions T' we present and explore seven below, which we refer to as: T T-, T St S and action maximization. For a DQN the previously discussed action maximization is interesting to find situations in which the agent assigns a high value to a certain action e.g. [s) = - (s). Other states of interest are those to which the agent assigns a low (or high) value for all possible actions A(s) = q = (g),.... gm). Consequently, one can optimize towards a low Q-value for the highest valued action with the following objective:
En Ra
T-
(6)
Σ
k== eaqu
3

---

Published as a conference paper at ICLR 2020
Algorithm Optimize for target T
1: Input: Target objective T, step size X, regularizer weight a, trained generator
2: Output: x
3: Fn 6
4: 2e 1-
5: while not converged do
6:
< sample from (00, I,.)
7:
Fu Do
O
sample from generate state using D gradient step using (4) and (5)
8:
+ g(z)
9:
Ip f Fp X 8
vea
(T(x(s)) + af())) (T(T(s)) + +a(())
10:
To t- Vo Xan, or,
11: end while
(a) Kung Fu Master T enemies both sides
(b) Kung Fu Master - T easy, many points
(c) Kung Fu Master T
no enemies
092
(d) Pong T scoring a point
(e) Space Invaders * shooting an enemy
(f) Enduro T overtaking an opponent
(g) Name This Game T whether refill
(h) Seaquest T of oxygen
(i) Beamrider Tieft avoiding the enemy
Figure 1: Qualitative Results: Visualization of different target functions (Sec. 2.3). T generates
high reward and T low reward states; 7 generates states in which one action is highly beneficial and another is bad. For long list of results, with over 50 Atari games, please see the appendix.
where 0 controls the sharpness the soft maximum formulation. Analogously, one can maximize the lowest Q-value with 7+ -T (-g). We can also optimize for interesting situations in which one action is of very high value and another is of very low value by defining
(+())= T (g) -
4

---

>>= i=1 9s and overall bad states with Published as a conference paper at ICLR 2020
Finally, we can optimize for overall good states with (*)
S (-g).
3
RELATED WORK
We divide prior work into two parts. First we discuss the large body of visualization techniques developed primarily for image recognition, followed by related efforts in reinforcement learning.
3.1 FEATURE VISUALIZATION
In the field of computer vision, there is a growing body of literature on visualizing features and neuron activations of CNNs. As outlined in (Grün et al., 2016), differentiate between saliency methods, that highlight decision-relevant regions given an input image, methods that synthesize an image (pre-image) that fulfills certain criterion, such activation maximization (Erhan et al., 2009) or inpul reconstruction, and methods that are perturbation-based, i.e. they quantify how input modification affects the output of the model.
3.1.1 SALIENCY METHODS
Saliency methods use the gradient of a prediction at the input image to estimate importance of pixels. Following gradient magnitude heatmaps (Simonyan al., 2013) and class activation mapping (Zhou et al., 2016), more sophisticated methods such as (Mahendran & Vedaldi, 2016; Selvaraju et al., 2016) have been developed (Zintgraf al., 2017) distinguish between regions in favor and regions speaking against the current prediction. (Sundararajan et al., 2017) distinguish between sensitivity and implementation invariance.
An interesting observation is that such methods seem to generate believable saliency maps even for networks with random weights (Adebayo et al., 2018a). (Kindermans al., 2017b) show that saliency methods do not produce analytically correct explanations for linear models and further reliability issues are discussed in (Adebayo et al., 2018b; Hooker et al., 2018; Kindermans et al., 2017a).
3.1.2 PERTURBATION METHODS
Perturbation methods modify given input to understand the importance of individual image regions. (Zeiler & Fergus, 2014) slide an occluding rectangle across the image and measure the change in the prediction, which results in heatmap of importance for each occluded region. This technique is revisited by (Fong & Vedaldi, 2017) who introduce blurring/noise in the image, instead a rectangular occluder, and iteratively find minimal perturbation mask that reduces the classifier's score, while (Dabkowski & Gal, 2017) train network for masking salient regions.
3.1.3 INPUT RECONSTRUCTION
As our method synthesizes inputs to the agent, the most closely related work includes input re- construction techniques. (Long et al., 2014) reconstruct an image from nearest neighbor patches in feature space. (Mahendran & Vedaldi, 2015) propose reconstruct images by inverting CNN representations, while (Dosovitskiy & Brox, 2015) leam to reconstruct the input from its encoding.
When maximizing the activation of a specific class or neuron, regularization crucial because the optimization procedure -tauurrcttttttnnggggg from a random noise image and maximizing an output- is vastly under-constrained and often tends to generate fooling examples that fall outside the manifold of realistic images (Nguyen et al., 2015). In (Mahendran & Vedaldi, 2016) total variation (TVV) is used for regularization, while (Baust et al., 2018) propose an update scheme based on Sobolev gradients. In (Nguyen et al., 2015) Gaussian filters are used to blur the pre-image the update computed in every iteration. Since there are usually multiple input families that excite a neuron, (Nguyen et al., 2016c) propose optimization scheme for the distillation of these clusters. More variations of regularization can be found in (Olah et al., 2017: 2018). Instead of regularization. (Nguyen et al., 201 6a;b) use denoising autoencoder reconstruct pre-images for image classification.
5

---

Published as a conference paper at ICLR 2020
Figure 2: Weight Visualization. We visu- alize the weighting (second row) of the re- construction loss from Equation 2 for eight randomly drawn samples (first row) of the dataset. Most weight lies on the player's sub- marine and close enemies, supporting their importance for the decision making.
Figure 3: Comparison with activation maximiza- tion. The visual features learned by the agents are not complex enough reconstruct typical frames from the game via activation maximization (top). This problem is mitigated in our method by learning low- dimensional embedding of games states (bottom).
3.2 EXPLANATIONS FOR REINFORCEMENT LEARNING
In deep reinforcement learning however, feature visualization is to date relatively unexplored. (Zahavy et al., 2016) apply t-SNE (Maaten & Hinton, 2008) on the last layer of a deep Q-network (DQN) to cluster states of behavior of the agent. (Mnih et al., 2016) also use t-SNE embeddings for visualization, while (Greydanus et al., 2017) examine how the current state affects the policy in a vision-based approach using saliency methods. (Wang et al., 2016) use saliency methods from (Simonyan et al., 2013) to visualize the value and advantage function of their dueling Q-network. (Huang et al., 2018) finds critical states of an agent based on the entropy of the output of policy. (Uesato et al., 2018) adversarially search for initial states of the agent and environment that make it fail. Interestingly, we could not find prior work using activation maximization methods for visualization. In our experiments we show that the typical methods fail in the case of RL networks and generate images far outside the manifold of valid game states, even with all typical forms of regularization. This is additionaly motivated by the insight that agents are vulnerable to adversarial states and can be manipulated even by other agents in the environment (Gleave et al., 2019).
EXPERIMENTS
In this section we thoroughly evaluate and analyze our method on Atari games (Bellemare et al., 2013) using the OpenAl Gym (Brockman et al., 2016) and driving simulator. We present qualitative results for three different reinforcement learning algorithms, show examples on how the method helps finding flaws in an agent, analyze the loss contributions and compare to previous techniques.
4.1 IMPLEMENTATION DETAILS
In all our experiments we use the same factors to balance the loss terms in Equation 6: = 10- for the KL divergence and = 10-3 for the agent perception loss. The generator is trained on 10, 000 frames (using the agent and an e-greedy policy with e 0.1). Optimization is done with Adam (Kingma & Ba, 2015) with a learning rate of 10-3 and batch size of 16 for 2000 epochs. Training takes approximately four hours on Titan Xp. Our generator uses a latent space of 100 dimensions, and consists of four encoder stages comprised of 3 x 3 convolution with stride 2, noneaaiimmali (loffe & Szegedy, 2015) and ReLU layer. The starting number of filters is 32 and is doubled at every stage. A fully connected layer is used for mean and log- variannie prediction. Decoding is inversely symmetric to encoding, using deconvolutions and halving the number of channels at each of the four steps.
For the experiments on the Atari games we train a double DQN (Wang et al., 2016) for two million steps with reward discount factor of 0.95. The input size is 84 84 pixels. Therefore, performs up-sampling by factors of 2, up to a 128 x 128 output, which is then center cropped to 84 x 84 pixels. The agents trained on grayscale images, for better visual quality however, our
generator

---

Published as a conference paper at ICLR 2020
generator is trained with color frames and convert grayscale using a differentiable, weighted sum of the color channels. In the interest of reproducibility we will make the visualization code available.
4.2 VISUALIZATIONS ON ATARI GAMES
In Figure 1, we show qualitative results from various Atari games using different target functions T, as described in Section 2.3. From these images we can validate that the general visualizations that are obtained from the method are of good quality and can be interpreted by a human. T generates generally high value states independent of specific action (first row of Figure 1), while T- generates low reward situations, such as close before losing the game in Seaquest (Figure I.e) or when there no points to score (Figure 1.i). Critical situations can be found by maximizing the difference between lowest and highest estimated Q-value with T In those cases, there is clearly a right and wrong action take. In Name This Game (Figure 1.d) this occurs when close to the air refill pipe, which prevents suffocating under water; in Kung Fu Master when there are enemies coming from both sides (Figure I.g), the order of attack is critical, especially since the health of the agent is low (yellow/blue bar on top). An example of maximizing the value of a single action (similar to maximizing the confidence of a class when visualizing image classification CNNs) can be seen in (Figure l.f) where the agent sees moving left and avoiding the enemy as the best choice of action.
4.3 ACKTR
To show that this visualization technique gen- eralizes over different RI. algorithms, we also visualize ACKTR (Wu et al., 2017). We use the code and pretrained RL models from pub- lic repository (Kostrikov, 2018) and train our generative model with the same hyperparame- ters as above and without any modifications on the agent. We present the T objective for the ACKTR agent in Figure 4 to visualize states with both high and low rewards, for example low oxygen (surviving vs. suffocating) close proximity to enemies (carning points vs. dying).
Figure 4: Seaquest with ACKTR. The objective 7 for situations that can rewarding but also have low scoring outcome. The generated states show low oxygen or close proximity to enemies.
Compared to the DQN visualizations the ACKTR visualizations, are almost identical in terms of image quality and interpretabilityy This supports the notion that our proposed approach is independent of the specific RL algorithm.
4.4 INTERPRETATION OF VISUALIZATIONS
Analyzing the visualizations on Seaquest, we make an interesting observation. When maximizing the Q-value for the actions, in many samples we see low or very low oxygen meter. In these cases the submarine would need to ascend to the surface to avoid suffocation. Although the up action is the only sensible choice in this case, we also obtain visualized low oxygen states for all other actions. This implies that the agent has not understood the importance of resurfacing when the oxygen is low. We then run several roll outs of the agent and see that the major cause of death is indeed suffocation and not collision with enemies. This shows the impact of visualization, as we able to understand flaw of the agent. Although it would be possible to identify this flawed behavior directly by analyzing the 10, 000 frames of training data for generator, it is significantly easier to review handful of samples from our method, Further, as the generator is generative model, we can synthesize states that are not part of its training set.
4.5 ABLATION STUDIES (LOSS TERMS)
In this section we analyze the three loss terms of our generative model. The human perceptual loss is weighted by the (guided) gradient magnitude of the agent in Equation 2. In Figure 2 we visualize this mask for DQN agent for random frames from the dataset. The masks are blurred with averaging filter of kernel size 5. We observe that guided backpropagation results in precise saliency maps focusing player and enemies that then focus the reconstructions what is important for the agent.
7

---

Published as a conference paper at ICLR 2020
Table Loss Study. We compare the performance of the original agent with the agent operating on reconstructed frames instead. The original performance is an upper bound for the score of the same agent which is operating on reconstructions instead. Shown are average scores over 20 runs.
Agent
VAE
Lp only
Ours (full)
Pong
14
-8
14
Atlantis
108
95
98
109
Q*tert
64
26
28
31
sitilatooo uutpur hamms of segmanizionn gray deeth
mas eiiinng A2C state valu road
masi uang AzE break acion pedaaiim
Figure Driving simulator. Input frame sample on the left and then two target function visualizations obtained by our method. For each objective we show three random samples. For simplicity we only show the first frame of segmentation instead of the whole state (8 frames).
To study the influence the loss terms we perform an experiment in which we evaluate the agent on the real frames but their reconstructions, If the reconstructed frames are perfect, the agent with generator goggles achieves the same score as the original agent. We can use this metric understand the quantitative influence of the loss terms. In Pong, the ball is the most important visual aspect of the game for decision making.
In Table we see that the VAE baseline scores much lower than our model. This can be explained as follows. Since the ball is very small, it mostly ignored by the reconstruction loss of a VAF. The contribution of one pixel to the overall loss is negligible and the VAE never focuses on reconstructing the important part of the image. Our formulation built to regain the original performance of the that our
agent, by reweighing the loss perceptually salient regions of the agent. Overall, method always improves over the baseline but does not always match the original performance.
4.6 COMPARISON WITH ACTIVATION MAXIMIZATION
For image classification tasks, activation maximization works well when optimizing the pre-image directly (Mahendran & Vedaldi, 2015; Baust al., 2018). However find that for reinforcement learning, the features learned by the network are not complex enough to reconstruct meaningful pre-images, even with sophisticated regularization techniques. The pre-image converges to a fooling example maximizing the class but being far away from the manifold of states of the environment.
In Figure 3.a compare our results with the reconstructions generated using the method of (Baust et al., 2) for a DQN agent. We obtain similarly bad pre-images with IV-reegularization (Mahendran & Vedaldi, 2016), Gaussian blurring (Nguyen et al., 2015) and other regularization tricks such as random jitter, rotations, scaling and cropping (Olah et al., 2017). This shows that it is not possible to directly apply common techniques for visualizing RL agents and explains why learned regularization from our generator is needed to produce meaningful examples.
4.7 EXPERIMENTS WITH A DRIVING SIMULATOR
Driving a car is continuous control task set within a much more complex environment than Atari games. To explore the behavior of our proposed technique this setting we have created 3D driving simulation environment and trained an A2C agent maximizing speed while avoiding pedestrians that are crossing the road.
In our first set of experiments we trained an A2C agent to maximize speed while avoiding swerving and pedestrians that are crossing the road. The input to the agent are eight temporal frames comprised of depth, semantic segmentation and gcay-scale image (Figure 5). With this experiment three random samples for two target functions. The moving car and person categories, appear most prominently when probing the agent for the break action. However, are also able to identify
visualize
8

---

Published as a conference paper at ICLR 2020
Figure Driving simulator. Samples for the T- objective of an agent trained in the rea- sonable pedestrians environment. From these samples one can infer that the agent is aware of traffic lights (red) and other cars (blue) but has very likely not understood the severity of hitting pedestrians (yellow). Deploying this agent in the distracted pedestrians envi- ronment shows that the agent indeed collides with people that cross the road in front of the
(a) SeaQuest visualization T
(b) Closest
(c) Closest training frame (T*).
training frame (I.2).
Figure 7: Generating novel states. We show frame generated by our method under the T ob- jective and retrieve the closest frame from the train- ing set using L.2 distance and the objective func- tion. Both frames are very different, showing that the method is able to generate novel states. For quantitative evaluation, please see Tab. 2.
agent.
Table 2: Synthesizing unseen states. We compare generated samples to their closest neighbor in the training set and compute the percentage of pixels whose values differ by at least 25%, c.g. 73% of the synthesized samples differ more than 20% pixels in comparison to their closest training sample.
difference 10%
20%
>> 30%
50%
>> 70%
samples
99%
73%
16%
1%
0%%
flaw: unnecessary braking on empty roads shown in the left most image of the right most block of three frames. Inappropriate breaking is a well known issue in this problem domain.
In second set of experiments, we our simulator to build custom environments and validate that we can identify problematic hehavior the agent. The agent is trained with four temporal semantic segmentation frames (128 128 pixels) as input (Figure 6). We train the agent in "reasonable pedestrians" environment, where pedestrians cross the road carefully, when no car is coming or at traffic lights. With these choices, we model data collected in the real world, where is unlikely that people unexpectedly run in front of the car. We visualize states in which the agent expects low future return (T" objective) Figure 6. It shows that the agent aware of other cars, traffic lights and intersections. However, there are generated states in which the car is about to collide with a person, meaning that the agent does not recognize the criticality of pedestrians. To verify our suspicion, the road looking at their phones without paying attention to approaching cars. We find that the agent does indeed run over humans. With this experiment, show that our visualization technique can identify biases in the training data just by critically analyzing the sampled frames.
test this agent in a "distracted pedestrians" environment where people cross
4.8 NOVEL STATES
To be able to generate novel states is useful, since it allows the method to model new scenarios that were accounted for during training of the agent. This allows the user identify potential problems without the need to include every possible permutation of situations in the simulator or real-world data collection.
While one could simply examine the experience replay buffer to find scenarios of interest, our approach allows unseen scenarios to be synthesized. To quantitatively evaluate the assertion that our generator is capable of generating novel states, we sample states and compare them to their closest frame in the training set under an MSE metric. We count a pixel as different if the relative difference in channel exceeds 25% and report the histogram in Table 2. The results show that there are very few samples that are very close to the training data. On average a generated state is different in 25 % of the pixels, which is high, considering the overall common layout of the road, buildings and sky.

---

Published as a conference paper at ICLR 2020
0.8
0.3
density
0.6
density
donsity
1.0
0.4
vel-set
0.1
val-set
0.5
val-set
0.2
ours
-10.0
-7.5
-5.0
-2.5 0.0
0.0
2.5
5.0
7.5
10.0
objictive score
objective score
abj ective score
(a) Histogram objective T
(b) Histogram objective 7
(c) Histogram objective T
Figure 8: Objective Score Distribution. We compare the distribution of values of different target functions (T- (a), 7 (b) and 7 (c)) between states from the validation set and those generated by our method. It is clear that the distribution of critical states differs and partially lies outside the maximum values of the val-set. This means these states are novel states, with higher criticalness-
We examine these results visually for Atari SeaQuest in Fig. where show generated frame and the L2-closest frame from the training set additional to the closest frame in the training set based on the objective function. Retrieval with L2 is, as usual not very meaningful on images since the actual interesting parts of the images are dominated by the background. Thus we have also included a retrieval experiment based on the objective score which shows the submarine a similar gameplay situation but with different enemy locations. The results in Tab. 2 and Fig. 7 confirm that the method able to generate unseen states and does not overfit to the training set.
To understand whether one could simply sample states from the replay buffer validation set, we compare the distribution of the target function values in Fig. 8 (on the example the "Boxing" environment). It is clear that method is able to generate states with much more targeted distribution towards the objective. all three cases, we are even able to generate states that score higher target values than any state in the validation set. This confirms that targeted and novel critical states and thus cannot be obtained from the validation set alone.
able to generate
5 DISCUSSION AND CONCLUSIONS
We have presented method synthesize inputs deep reinforcement learning agents based on generative modeling of the environment and user-defined objective functions. The agent perception loss helps the reconstructions focus on regions of the current state that are important the agent and avoid generating fooling examples. Training the generator to produce states that the agent perceives as those from the real environment enables optimizing its latent space sample states of interest. Please consult the appendix for more extensive visualization experiments.
We believe that understanding and visualizing agent behavior in safety critical situations a crucial step towards creating safer and more robust agents using reinforcement learning. We have found that the methods explored here can indeed help accelerate the detection of problematic situations for given learned agent. For our car simulation experiments we have focused upon the identification of weaknesses in constructed scenarios; however, we see great potential to apply these techniques to much more complex simulation environments with less obvious safety critical weaknesses.
ACKNOWLEDGMENT
We would like to thank Iro Laina, Alexandre Piché, Simon Ramstedt, Evan Racah and Adrien Ali Taiga for helpful discussions and proofreading. We thank the Open Philanthropy project for supporting C.R. while he was an intern at Mila, where this work began.
REFERENCES
Julius Adebayo, Justin Gilmer, Ian Goodfellow, and Been Kim. Local explanation methods for deep neural networks lack sensitivity parameter values. ICLR Workshop Track, 2018a.
10

---

Published as a conference paper at ICLR 2020
Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, and Been Kim. Sanity checks for saliency maps. arXiv 1810.03292, 2018b. 5
Maximilian Baust, Florian Ludwig. Christian Rupprecht, Matthias Kohl, and Stefan Braunewell. Understanding regularization to visualize convolutional neural networks. arXiv 1805.00071, 2018. 5,8
Marc Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environment: An evaluation platform for general agents, Journal of Artificial Intelligence Research, 47:253-279, 2013, 6
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym, 2016.
Piotr Dabkowski and Yarin Gal. Real time image saliency for black box classifiers, In NewrlPS, pp. 1770-6979, 2017. 5
Alexey Dosovitskiy and Thomas Brox. Inverting convolutional networks with convolutional networks. arXiv 2015. 5
Dumitru Erhan, Yoshua Bengio, Aaron Courville, and Pascal Vincent. Visualizing higher-layer features of a deep network. Univ. of Montreal, 1341(3):1, 2009. 5
Ruth C Fong and Andrea Vedaldi. Interpretable explanations of black boxes by meaningful perturbation. arXiv 1704.03296, 2017. 1,5
Adam Gleave, Michael Dennis, Neel Kant, Cody Wild, Sergey Levine, and Stuart Russell. Adversarial policies: Attacking deep reinforcement learning. arXiv preprint 1rX.v:1905.10615. 2019.
Sam Greydanus, Anurag Koul, Jonathan Dodge, and Alan Fern. Visualizing and understanding atari agents. arXiv 1711.00138, 2017.
Felix Grün, Christian Rupprecht, Nassir Navab, and Federico Tombari. A taxonomy and library for visualizing learned features in convolutional neural networks. arXiv (606,07757, 2016.
Sara Hooker, Dumitru Erhan, Pieter-Jan Kindermans, and Been Kim. Evaluating feature importance estimates. arXiv 10758, 2018.
Sandy Huang, Kush Bhatia, Pieter Abbeel, and Anca D Dragan. Establishing appropriate trust critical states. IROS, 2018.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In ICML, pp. 448-456, 2015. 6
Andrej Karpathy, Justin Johnson, and Li Fei-Fei. Visualizing and understanding recurrent networks. ICLR, 2016.
Picter-Jan Jan Kindermans, Sara Hooker, Julius Adebayo, Maximilian Alber, Kristof T Schlitt, Sven Dähne, Dumitru Erhan, and Been Kim. The (un) reliability of saliency methods. arXiv 1711.00867, 2017a. 3,5
Pieter-Jan Kindermans, Kristof T Schütt, Maximilian Alber, Klaus-Robert Müller, and Sven Dähne. Patternnet
and seneemmlp-iiprovinng the interpretability of neural networks. arXiv 1705.05598, 201 5
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. ICLR, 2015. 6
Diederik P Kingma and Max Welling, Auto-encoding variational bayes. arXiv 13/2.61 14, 2013. 2
Ilya Kostrikov. Pytorch implementations rl algorithms. https://githut com/ kkostrikov/
a2c-ppo-acktt 2018. 7
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In NeurlPS, pp. 1097-1105, 2012. 1
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. sradient-based learning applied to document recognition. Proceedings of the IEFE, (111))2227--22 1998.
Jonathan Long, Evan Shelhamer. and Trevor Darrell, Fully convolutional networks for semantic segmentation. In CVPR, pp. 3431-3440, 2015.
Jonathan I. Long, Ning Zhang, and Trevor Darrell. Do convnets learn correspondence? In NeurlPS, pp- 1601-1609, 2014. 5
11

---

Published as a conference paper at ICLR 2020
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using I-sne. Journal machine learning
researck, ((Nov):2579-260 2605, 2008. 6
Aravindh Mahendran and Andrea Vedaldi. Understanding deep image representations by inverting them. In CVPR, pp- 5188-5196. IEEE, 2015. 5,8
Aravindh Mahendran and Andrea Vedaldi. Visualizing deep convolutional neural networks using natural pre-images. IJCV, 120(3):233 225,,2333-5,,,,.,2 2016. 5,8
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning, In ICML, pp. 1928-1937, 2016.
Anh Nguyen. Jason Yosinski, and Jeff Clune. Deep neural networks are easily fooled: High confidence predictions for unrecognizable images. In CVPR, pp. 427-436, 2015. 8
Anh Nguyen, Alexey Dosovitskiy, Jason Yosinski, Thomas Brox, and Jeff Clune. Synthesizing the preferred inputs for neurons in neural networks via deep generator networks. In NewrlPS, pp. 3387-3395, 20. 5
Anh Nguyen, Jason Yosinski, Yoshua Bengio, Alexey Dosovitskiy, and Jeff Clune. Plug & play generative networks: Conditional iterative generation of images in latent space. arXiv 1612.00005, 2016b. 5
Anh Nguyen, Jason Yosinski, and Jeff Clune. Multifaceted feature visualization: Uncovering the different types of features learned by cach neuron in deep neural networks. arXiv 1602.03616, 2016c. 5
Chris Olah, Alexander Mordvintsev, and Ludwig Schubert. Feature visualization. Distill, 2017. doi: 10.23915/ distill.00000 https://distill http://ddiiiil.pp///01111 7/feature-visualzzaiin 8
Chris Olah, Arvind Satyanarayan, Ian Johnson, Shan Carter, Ludwig Schubert, Katherine Ye, and Alexan-
der Mordvintsev. The building blocks of interpretability. Distill, 2018. doi: 10.239115diistil.00011
https://distill.ubbb/ foouilding-blocke 5
Shaoqing Ren, Kaiming Hc, Ross Girshick, and Jian Sun. Faster r-cnm: Towards real-time object detection with region proposal networks. In NeurlPS, pp. 91-99, 2015. 1
Ramprasaath Selvaraju. Michael Cogswell, Abhishek Das. Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient- basedd localization. arXiv 1610.02391 v3, 7(8), 2016. 5
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. arXiv 1312.6034, 2013. 1,5, 6
Mukund Sundararajan, Ankur Taly, and Qiqi Yan, Axiomatic attribution for deep networks. arXiv 1703.01365,
2017. 5
Jonathan Uesato, Ananya Kumar, Csaha Szepesvari, Tom Erez, Avraham Ruderman, Keith Anderson, Nicolas Heess, Pushmeet Kohli, et al. Rigorous agent evaluation: An adversarial approach to uncover catastrophic failures. arXiv preprint arXiv: 18/2.01647, 2018. 6
Ziyu Wang, Tom Schaul, Matteo Hessel, Hado Van Hasselt, Mare Lanctot, and Nando De Freitas. Dueling network architectures for deep reinforcement learning. In ICML, pp. 1995-2003 JMLR. org, 2016.
Yuhuai Wu, Elman Mansimov, Roger Grosse, Shun Liao, and Jimmy Ba. Scalable trust-region method for deep reinforcement learning using kronecker factored approximation. In NewrlPS, pp. 2285-5294,, 2017. 7
Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. Show, attend and tell: Neural image caption generation with visual attention. In ICML, pp. 2048-2057, 2015.
Tom Zahavy, Nir Ben- Ben-Zrihem, and Shie Mannor. Graying the black box: Understanding dgns. ICML, pp. 1899-1908, 2016. 6
Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In ECCV, pp.
818-833. Springer, 2014. 1,5
Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. In CVPR, June 2016. 5
Luisa M Zintgraf, Taco S Cohen, Tameem Adel, and Max Welling. Visualizing deep neural network decisions: Prediction difference analysis. arXiv 1702.04595, 2017.
12

---

Published as a conference paper at ICLR 2020
APPENDIX
To show an unbiased and wide variety of results, in the following, we will show four random samples generated by our method for DQN agent trained on many the Atari benchmark environments. We show visualizations optimized for meaningful objective for each game (e.g. not optimizing for unused buttons). All examples were generated with the same hyperparameter settings.
Please note that for some games better settings be found. Some generators visually more complex games would benefit from longer training to generate sharper images. Our method is able to generate reasonable images even when the DQN unable learn meaningful policy such as for Montezuma's 's revenge (Fig. 40). We show two additional objectives maximizing/mininiiing the expected reward of the state under random action: S+ (4) -i=1 9 and S (g) = (g)
Results in alphabetical order and best viewed in color.
Figure 9: Air Raid. Target function: Stt
Figure 10: Alien. Target function: right.
Figure 11: Amidar. Target function: up.
13

---

Published as a conference paper at ICLR 2020
000J10
Figure 12: Assault. Target function: S
1,00
1100
Figure 13: Asterix, Target function: T-
Figure 14: Asteroids. Target function: up-fire.
BSB
Figure 15: Atlantis. Target function: T+t
Figure 16: Bank Heist. Target function: T
14

---

Published as a conference paper at ICLR 2020
033
Figure 17: Battlezone. Target function: T
110 2F SECTOP 03
002660 SUETRR
D SRO SETTER 03
n0 SKG SUETER 02
Figure 18: Beamrider. Target function: T+
3EE
Figure 19: Berzerk.
Target function: S+
Figure 20: Bowling, Target function: S+
Figure 21: Boxing. Target function: S+
15

---

Published as a conference paper at ICLR 2020
0
Figure 22: Breakout. Target function: T
Figure 23: Breakout. Target function: Left.
Figure 24: Carnival. Target function: right.
4822
Figure 25: Centipede. Target function: T
Figure 26: Chopper Command, Target function: S+
16

---

Published as a conference paper at ICLR 2020
20G00
Figure 27: Crazy Climber, Target function: T
Figure 28: Demon Attack. Target function: T+
Figure 29: Elevator Action. Target function: no-op.
745
1
139197
Figure 30: Enduro.
Target function: S+
Figure 31: Freeway. Target function: T
17

---

Published as a conference paper at ICLR 2020
Figure 32: Frostbite. Target function: no-op.
Figure 33: Gopher. Target function: S-.
Figure 34: Gravitar. Target function: T
Figure 35: Hero. Target function: S+
Figure 36: JamesBond. Target function: S+
18

---

Published as a conference paper at ICLR 2020
Figure 37: Kangaroo. Target function: S
Figure 38: Krull. Target function: fire.
Figure 39: Kung Fu Master. Target function: up.
Figure 40: Montezuma's Revenge. Target function: T
1080
1020
Figure 41: Ms. Pacman. Target function: no-op.
19

---

Published as a conference paper at ICLR 2020
Figure 42: Name This Game. Target function: T
Figure 43: Phoenix. Target function: T
Figure 44: Pong. Target function: no-op.
Figure 45: Pooyan. Target function: S
00796
00806
Figure 46: Q-Bert. Target function: left.
20

---

Published as a conference paper at ICLR 2020
Figure 47: River Raid. Target function: T*
BODK
Figure 48: Space Invaders.
Target function: left.
Figure 49: Star Gunner. Target function: T
Figure 50: Tutankham. Target function: no-op.
000000
200000
000000
000000
0.5 0.5 CIS LI5 CIS
0.S
0.R
CIS
Figure 51: Venture. Target function: S
21

---

Published as a conference paper at ICLR 2020
EI
H
Figure 52: Video Pinball. Target function: T-
Figure 53: Wizard Of Wor. Target function: left.
22


Reviews:
[{"review_id": "rkxjJnqCFS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a new visualization tool in order to understand the behavior of agents trained using deep RL. Specifically, they train a generative model of game states, and then optimize an energy-based distribution over state embeddings according to some target function, and then by sampling from the resulting distribution they create a diverse set of realistic states that score highly according to the target function. They propose a few target cost functions, which allow them to optimize for states in which the agent takes a particular action, states which are high reward (worst Q-value is large), states which are low reward (best Q-value is small), and critical states. They demonstrate results on Atari games as well as a simulated driving environment.\n\nI enjoyed this paper: the proposed method is straightforward, and the experiments are well-done and demonstrate the potential of the method, and for that reason I’m recommending an accept.\n\nIf I had to name a fault with the paper, it is that the interpretation of the results depends quite strongly on human judgment: we are asked to look at particular images and then told how to interpret them. It would be both more compelling and more interesting to use the results to *fix* problems detected in the agents. For example, Section 4.4 suggests that the Seaquest agent has not learned that it should surface when the oxygen is low. Having done this analysis, can we then fix the problem? Perhaps it would work to sample states according to the T+- objective, and then train the agent to take the “up” action in such situations, perhaps interspersed with regular RL / training on the replay buffer to avoid catastrophic forgetting. If the problem could be fixed, that would be strong evidence for the utility of the method.\n\nMinor questions:\n\nCan you explain why T+-(q) = T-(q) - T+(q) incentivizes “situations in which one action is of very high value and another is of very low value”? It is not immediately obvious to me why this should be true, just from the definition.\n\nTypos:\n\nSecond contribution: “interstingness” --> “interestingness”\nSection 3.2: “In the next section, we will show how to overcome these difficulties.” I assume this is referring to what is now Section 2?\nPlease organize the figures better in relation to the text (e.g. Fig. 5 should be after Figs. 6 and 7).", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "Syxk_h7IFH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\n\nThis paper proposes a generative technique to sample \"interesting\" states useful for analyzing the behavior of deep reinforcement learning agents. In this context, the concept of \"interesting\" is defined via user-specific target functions, e.g. states that arise as a consequence of taking specific actions (such as actions associated with high or low Q-values for example). The approach is evaluated in the Atari domain and in an autonomous driving simulator. Results are mainly presented as visualizations of interesting states that are described verbally.\n\nQuality\n\nThe quality of the submission is extremely low. The optimization objectives chosen by the authors seem very ad hoc to me and how the motivation relates to the objectives is hard to comprehend (see my Clarity section). The experimental results have very low quality as well---results are mainly depicted as images with a verbal explanation.\n\nClarity\n\nThe clarity of the paper is extremely poor. While I do conceptually understand Section 2.1, I have a hard time linking it precisely to Section 2.2. Just some examples regarding lack in clarity:\n- What is z in Section 2.1?\n- How do the objectives in Section 2.1 and Section 2.2 relate to each other, i.e. how does the algorithm operate? Some pseudocode would be really helpful here.\n- What are the target functions S^+, S^- and S^\\pm in Section 2.3?\n- What is the difference in the KL-regularizer mentioned in the text below Equation (3) and in Equation (5)?\n- In the text above Equation (2), it is mentioned that a squared reconstruction loss is insensitive to small elements in the image that have a huge impact on the reward. While this is true, I don't see how the multiplicative policy gradient norm term in Equation (2), as proposed by the authors, is addressing this issue. The proposed modification puts emphasis on states where the norm of the policy gradient is high, which is different from putting emphasis on specific regions in the image. I guess the intention would be to do an element-wise multiplication of the squared loss vector and the absolute value policy gradient vector before collapsing to a scalar, or something similar?\nIn general, I found the entire writing from Section 3 onward a bit wordy and I do not think that nine pages are required to deliver the message of the paper in its current form.\n\nOriginality\n\nThe idea of visualizing states that reveal interesting insights about an agent's behavior based on a user-defined target function sounds interesting. But I have not worked in interpretability of agent behavior, which is why I leave the assessment of the originality to the other reviewers and the area chair.\n\nSignificance\n\nIf the results of the paper were backed up with some proper scientific metrics other than verbally explaining images, there might be some significance in the paper.\n\nUpdate\n\nAfter the authors' response, I am currently not inclined to change my score. While I do agree that the paper proposes an interesting idea, the technical presentation of the work is simply too poor and not convincing at this stage. Here are a few points:\n\n- A variational autoencoder works as follows. There is a generative model over latent variables z and observed variables s, consisting of a prior for z and a likelihood for s conditioned on z. The prior over z can be e.g. a normal distribution denoted as N(z|\\mu_prior, \\Sigma_prior). Then the likelihood (decoder) can be e.g. a deep neural net that maps z to a normal distribution of s denoted as N(s|\\mu_likelihood(z, \\theta), \\Sigma_likelihood(z, \\theta)) where \\theta refers to the decoder's neural network weights. Furthermore, there is a recognition model that approximates the posterior over z given s (encoder)---this can also be e.g. a deep neural net that maps s to a normal distribution in z denoted as N(z|\\mu_posteriorapprox(s, \\psi), \\Sigma_posteriorapprox(s, \\psi)) where \\psi refers to the encoder's neural network weights.\n\n- The reparameterization trick is not required for the technical explanation of the involved random variables and how they relate to each other. It is merely an optimization trick to establish a functional dependency between a random variable and the parameters of its distribution (e.g. mean and covariance in the Gaussian case).\n\n- To be specific about your updated paper. The notation you chose for the encoder f(s) = (\\mu, \\sigma) is confusing because it hides the dependency on s on the right hand side. The notation for the decoder g(\\mu, \\sigma, z) is also confusing because the decoder is supposed to map z to something in s space (see my first bullet point). The notation g(f(s), z) is particularly confusing because it is not consistent with the other notation that you use (which I mentioned in the sentence before). Usually, f(s) represents an element in latent space and is fed through the decoder to yield something in s space---so I don't understand why the decoder receives both f(s) and z as an argument.\n\n- You talk about optimization objectives, then please specify what the optimization arguments are---this is not clear from the description given. For example, in Equation (1) the optimization arguments seem to be both the recognition (encoder) and the generative (prior over latents plus decoder) model parameters? In Equation (4), the optimization argument seems to be the latent variable z?\n\nGiven all the comments above, it is pretty obvious that the paper in its current form simply does not adhere to scientific standards for technically reporting machine learning algorithms in a proper way. So I clearly still vote for rejection because of the lack in technical clarity. And yes, as I said, I would like to see pseudocode.", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HyxtJXydtS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose learning a generative model of states to visualize the behavior of different RL agents. Given states s from the environment, a VAE is trained to reconstruct states s, with an added loss term encouraging the action of the agent to stay the same between the original s and reconstructed s. The L2 reconstruction loss is also weighted by an attentive loss term, based on saliency of the policy pi, to focus reconstruction on critical regions.\n\nOnce the VAE is learned, we learn a 2nd sampling network, which operates in the latent space of the embedding. This energy-based model aims to sample regions of state space that optimize some target function T. For example, the target function may be \"Q-value of moving left\", in which case the model should generate states s where moving left is expected to be high-value. They examine a number of target functions, for maximizing / minimizing the Q-value of different actions, maximizing the spread of Q-values (max_a Q(s,a) - min_a Q(s,a)), and other saliency approaches. Experiments on Atari and a 3D driving simulator demonstrate that visualized images are qualitatively reasonable, and the states generated aren't just doing nearest neighbor over states in the training set.\n\nThe approach seems reasonable and the experiments seem reasonable as well. The saliency-based VAE objective is new to me as well. However, I contest the claim that this is one of the first works visualizing and diagnosing RL agents. The number of papers focusing purely on RL agent visualization is fairly small, but many deep RL papers include visualization as part of their experimental results, and in fact the paper cites some of these directly (like Greydanus et al, 2017). There are also a number of unclear details I'd want clarified.\n\nSpecific comments\n* When describing the VAE training loss in Eqn (1), the last term can just be KL(f(s), N(0, I_n)), writing out the prior is more confusing, especially because the KL is mentioned in the text.\n* Why is the gradient saliency measured in L1 rather than L2 distance? Every distance measured in the paper besides this one uses L2. Also, it is unclear what d is in the denominator - I assume it is a sum over each dimension of the state space but this is never fully described.\n* Does the action consistency loss require differentiability through L_a? The provided example of the argmax action for policy pi seems like a hard loss function to learn, and it's unclear what L2 diff between two different argmax actions means.\n* For the driving simulator, is there a reason the simulator used is an in-house one, rather than an existing driving simulator like CARLA?\n* I buy the results showing the VAE learns to generate novel states. However, are there examples of novel states that drive insights that couldn't be found by doing nearest-neighbor over the training set? My thinking here is that you train E(x) the same way as before, except instead of x representing the noise passed to the VAE, you have x represent a non-parametric distribution that samples states from the replay buffer with some probability. For example, the paper argues they learned their Seaquest agent doesn't model the oxygen meter well, but was learning the generative model necessary to learn this?\n* Section 2.3 (Target Functions) mentions target function S+, S-, S+-, none of which seem to be defined or mentioned in the main text.\n* For prior visualization work, there is some related work from the adversarial RL literature (https://arxiv.org/pdf/1905.10615.pdf for a recent example), since adversarially attacking a policy tends to expose features that policy cares about. For examining failure states in particular, there is also https://arxiv.org/pdf/1812.01647.pdf which tries to identify catastrophic failures that are rare in the dataset.\n* At a style level, I would not describe this paper as specifically a visualizing weaknesses paper - instead it is more like a framework to learn to generate states that satisfy some predicate of the Q-function (as noted by experiments that try to identify critical states, especially positive states, etc.), and I would consider renaming the paper accordingly.\n\nOverall I feel this paper is very borderline but I'll round to weak accept. \n\nEdit: I've read the author reply and other reviews. I don't plan to change my score. I thank the authors for clarifying some of the notational issues, and still believe it would be interesting to look at nearest-neighbor style baselines. If the environment is truly as complicated as stated, then these baselines should be very clearly bad and make a better case for the proposed contribution.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a tool to visualizing the behaviour of deep RL agents, for example to observe the behaviour of an agent in critical scenarios. The idea is to learn a generative model of the environment and use it to artificially generate novel states in order to induce specific agent actions. States can then be generated such as to optimize a given target function, for example states where the agent takes a specific actions or states which are high/low reward. They evaluate the proposed visualization on Atari games and on a driving simulation environment, where the authors use their approach, to investigate the behaviour of different deep RL agents such as DQN.

The paper is very controversial. On the one hand, as far as we know, this is the first approach that explicitly generates states that are meant to induce specific agent behaviour, although one could relate this to adversarial samples generation. Interpretability in deep RL is a known problem and this work could bring an interesting tool to the community. However, the proposed approach lacks theoretical foundations, thus feels quite ad-hoc, and results are limited to a qualitative, visual, evaluation. At the same time, one could say that the approach is not more ad hoc than other gradient saliency visualization approaches, and one could argue that the lack of theoretical soundness is due to the difficulty of defining good measures of interpretability and that apply well to image-based environments.

Nonetheless, this paper is a step in the good direction in a field that could really benefit from it. 
Decision: Accept (Poster)