Exploratory Not Explanatory: Counterfactual Analysis of Saliency Maps for Deep Reinforcement Learning
Authors: Akanksha Atrey
Kaleigh Clary
David Jensen
Venue:  (2020)
Keywords: explainability
saliency maps
representations
deep reinforcement learning

Abstract: Saliency maps are frequently used to support explanations of the behavior of deep reinforcement learning (RL) agents. However, a review of how saliency maps are used in practice indicates that the derived explanations are often unfalsifiable and can be highly subjective. We introduce an empirical approach grounded in counterfactual reasoning to test the hypotheses generated from saliency maps and assess the degree to which they correspond to the semantics of RL environments. We use Atari games, a common benchmark for deep RL, to evaluate three types of saliency maps. Our results show the extent to which existing claims about Atari games can be evaluated and suggest that saliency maps are best viewed as an exploratory tool rather than an explanatory tool.

Published as a conference paper at ICLR 2020
EXPLORATORY NoT EXPLANATORY: COUNTERFACTUAL ANALYSIS OF SALIENCY MAPS FOR DEEP REINFORCEMENT LEARNING
Akanksha Atrey, Kaleigh Clary & David Jensen University of Massachusetts Amherst
faatzey, kclary, jensen]@cs. umass, edu
ABSTRACT
Saliency maps are frequently used to support explanations of the behavior of deep reinforcement learning (RL) agents. However, a review of how saliency maps are used in practice indicates that the derived explanations are often unfalsifiable and can be highly subjective. We introduce an empirical approach grounded counterfactual reasoning to test the hypotheses generated from saliency maps and assess the degree to which they correspond to the semantics of RL environments. We use Atari games, common benchmark for deep RL, to evaluate three types saliency maps. Our results show the extent to which existing claims about Atari games can be evaluated and suggest that saliency maps are best viewed as an exploratory tool rather than an explanatory tool.
INTRODUCTION
Saliency map methods are a popular visualization technique that produce heatmap-like output high- lighting the importance of different regions of some visual input. They are frequently used to explain how deep networks classify images in computer vision applications (Simonyan et al. 2014:: Zeiler
& Fergus| 2014 Springenberg et al. 2015; Ribeiro et al. 2016 Dabkowski & Gal 2017 Fong & Vedaldi 2017 Selvaraju et al. 201 Shrikumar et al. 2017: Smilkov et al. 2017 Zhang et al.
2018) and to explain how agents choose actions in reinforcement learning (RL) applications (Bog-
danovic et al. 2015 Wang et al.
2016 Zahavy Zahay et al. 2016 Greydanus et al. 2017 Iyer et al. 2018
Sundar 2018
Yang et al. 2018
Annasamy & Sycara 2019).
Saliency methods in computer vision and reinforcement learning use similar procedures to generate these maps. However, the temporal and interactive nature of RI. systems presents unique set of opportunities and challenges. Deep models in reinforcement learning select sequential actions whose effects can interact over long time periods. This contrasts strongly with visual classification tasks, in which deep models merely map from images to labels. For RL systems, saliency maps are often used assess an agent's internal representations and behavior over multiple frames in the environment, rather than to assess the importance of specific pixels in classifying images.
Despite their common use to explain agent behavior, it is unclear whether saliency maps provide useful explanations of the behavior of deep RI. agents. Some prior work has evaluated the ap- plicability of saliency maps for explaining the behavior of image classifiers (Samek et al., [017] Adebayo et al. [2018] Kindermans et al.) 2201)), but there is not a corresponding literature evaluating the applicability saliency maps for explaining RL agent behavior.
In this work, we develop methodology grounded in counterfactual reasoning to empirically evalu- ate the explanations generated using saliency maps in deep RL. Specifically, we:
C1 Survey the ways in which saliency maps have been used as evidence in explanations of deep RL agents.
C2 Describe a new interventional method to evaluate the inferences made from saliency maps.
C3 Experimentally evaluate how well the pixel-level inferences of saliency maps correspond to the semantic-level inferences of humans.

---

Published as a conference paper at ICLR 2020
(b)
(c)
Figure 1: Three examples perturbation saliency maps generated from the same model on different inputs: (a) a frame taken from an episode of agent play in Breakout, (b) the same frame with the brick pattern reflected across the vertical axis, and (c) the original frame with the ball, paddle and brick pattern reflected across the vertical axis. The blue and red regions represent their importance in action selection and reward estimation from the current state, respectively. The pattern and intensity of saliency around the tunnel is not symmetric in either reflection intervention, indicating that popular hypothesis (agents learn to aim at tunnels) does not hold for all possible tunnels.
2 INTERPRETING SALIENCY MAPS IN DEEP RL
Consider the saliency maps generated from deep RL agent trained to play the Atari game Breakout. The goal of Breakout is to use the paddle to keep the ball in play so it hits bricks, eliminating them from the screen. Figure shows a sample frame with its corresponding saliency. Note the high salience on the missing section of bricks ("tunnel") in Figure
Creating tunnel to target bricks the top layers is one of the most high-profile examples of concepts (Mnih et al.)
agent behavior being explained according to semantic, human-understandaan 2013). Given the intensity of saliency on the tunnel Tal may seem reasonable to infer that this saliency map provides evidence that the agent has learned to aim at tunnels. If this is the case, mov- ing the horizontal position of the tunnel should lead to similar saliency patterns on new tunnel. However, Figures 1b and Ic show that the salience pattern is not preserved. Neither the presence of the tunnel, nor the relative positioning of the ball, paddle, and tunnel, are responsible for the intensity of the saliency observed in Figure
2.1 SALIENCY MAPS AS INTERVENTIONS
Examining how some of the technical details of reinforcement learning interact with saliency maps can help explain both the potential utility and the potential pitfalls of interpreting saliency maps. RL methods enable agents learn how act effectively within environment by repeated interaction with that environment. Certain states in the environment give the agent positive or negative reward. The agent learns policy, mapping between states and actions according these reward signals. The goal is to learn a policy that maximizes the discounted sum of rewards received while acting in the environment (Sutton & Barto 1998). Deep reinforcement learning uses deep neural networks to represent policies. These models enable interaction with environments requiring high-dimensional state inputs (e.g., Atari games).
Consider the graphical model in Figure 2 representing the deep RL system for a vision- game environment. Saliency maps are produced by performing some kind of intervention M on this system and calculating the difference in logits produced by the original and modified images. The interventions used to calculate saliency for deep RI. are performed at the pixel level (red node and arrow in Figure 22). These interventions change the conditional probability distribution of "Pixels" by giving it another parent (Pearl 2000).
Functionally, this can be accomplished through variety of means, including changing the color of the pixel (Simonyan et al.) 20144), adding a gray mask (Zeiler & Fergus 2014), blurring a small region (Greydanus et al.) 2017), 7p. masking objects with the background color (Iyer et al.) 2018). The interventions M are used to simulate the effect the absence of the pixel(s) on the network's output. Note however that these interventions change the image in way that is inconsistent with the
2

---

Published as a conference paper at ICLR 2020
RL Framework
Human Observer
Game State
Pixets
Pools
Semantic
Concepts
Salency Maps
Pepreseen- tation
Learned ned
Logits
Behavior
Learned Represerr tatton
Behavior
a
(b)
Figure 2: (a) Causal graphical model of the relationships between an RL agent (yellow plate) and an image-based environment (blue plate). The enviromment maintains some (usually latent) game state. Some function F produces a high-dimensional pixel representation of game state ("Pixels"). The learned network takes this pixel image and produces logits used to select action. Tempo- rally extended sequences this action selection procedure result in observed agent behavior:" "M" represents interventions made by saliency maps. Such interventions
not naturalistic and are in- consistent with the generative process F; (b) conceptual diagram of how a human observer infers explanations. Hypotheses ("Claims") about the semantic features identified by the learned policy are proposed by reasoning backwards about what representation, often latent, might jointly produce the observed saliency pattern and agent behavior.
generative process F. They are not "naturalistic" interventions. This type of intervention produces images for which the learned network function may not be well-defined.
2.2 EXPLANATIONS FROM SALIENCY MAPS
To form explanations of agent behavior, human observers combine information from saliency maps, agent behavior, and semantic concepts. Figure shows system diagram of how these components interact. We note that semantic concepts are often identified visually from the pixel output as the game state is typically latent.
Counterfactual reasoning has been identified as a particularly effective way to present explanations of the decision boundaries of deep models (Mittelstadt al. 2019). Humans counterfactuals to reason about the enabling conditions of particular outcomes, as well as to identify situations where the outcome would have occurred even in the absence of some action or condition (de Graaf & Malle 2017 Byrne 2019). Saliency maps provide a kind of pixel-level counterfactual, but the goal explain agent behavior according to semantic concepts, interventions the pixel level seem unlikely to be sufficient.
Since many semantic concepts may map the same of pixels, it may be difficult to identify the functional relationship between changes in pixels and changes in network output according to semantic concepts or game state (Chalupka et al., (015). Researchers may be interpreting differences in network outputs as evidence of differences in semantic concepts. However, changes in pixels do not guarantee changes in semantic concepts or game state.
In terms of changes to pixels, semantic concepts, and game state, we distinguish among three classes of interventions: distortion, remantics-preserving and fat-hand (see Table Semantics-preservinn and fat-hand interventions are defined with respect to specific set of semantic concepts. Fat-hand interventions change game state in such a way that the semantic concepts of interest are also altered.
The pixel-level manipulations used to produce saliency maps primarily result in distortion inter- ventions, though some saliency methods (e.g., based) may conceivably produce semantics- preserving fat-hand interventions well. Pixel-level interventions are not guaranteed to produce changes in semantic concepts, so counterfactual evaluations that apply semantics semsantis--reserving inter- ventions may be more appropriate approach for precisely testing hypotheses of behavior.
3

---

Published as a conference paper at ICLR 2020
Change in... game state
Intervention
semantic concepts
Examples
pixels
Distortion
Adversarial ML (Szegedy et al. 2014) Rellection across a line symmetry Teleporting the agent to a new position
Semantics- inmaattis-pressrving
Fat-hand
Categories of interventions on images. Distortion interventions change pixels without changing game state or semantic concepts. Pixel perturbations in adversarial MI add adversarial noise to images change the output of the network without making human-perceptible changes in the image. preeeniis-preservinn interventions an image that preserves some semantic concept interest. Reflections across lines of symmetry typically alter aspects of game state, but do not meaningfully change any semantic information about the scene. "Fat-hand" interventions are manipulations intended to measure the effect some specific treatment, but which unintentionally alter other relevant aspects of the system. The term is drawn from the literature on causal modeling (Scheines 2005).
Table 1:
manipulations of game state that result in
SURVEY OF USAGE OF SALIENCY MAPS IN DEEP RL LITERATURE
To assess how saliency maps are typically used make inferences regarding agent behavior, we surveyed recent conference papers in deep RL. We focused our pool of papers those that use saliency maps to generate explanations or make claims regarding agent behavior. Our search criteria consisted of examining papers that cited work that first described any of the following four types of saliency maps:
Jacobian Saliency. Wang et al. (2016) extend gradient- based saliency maps to deep RL by com- puting the Jacobian of the output logits with respect stack of input images.
Perturbation Saliency. Greydanus et al. (2017) generate saliency maps by perturbing the origi- nal input image using Gaussian blur of the image and measure changes in policy from remov- ing information from region.
Object Saliency. Iyer et al. (0188) use template matching, a common computer vision tech- nique (Brunelli) 2009). to detect (template) objects within an input image and measure salience through changes in Q-values for masked and unmasked objects.
Attention Saliency. Most recently, attention- -based saliency mapping methods have been pro- posed generate interpretable saliency maps (Mott al., 2019: Nikulin et 2019).
From a set of 90 papers, we found 46 claims drawn from 11 papers that cited and used saliency maps evidence in their explanations of agent behavior. The full set of claims given in Appendix C
3.1 SURVEY RESULTS
Discuss Focus
Generate Explanation
Evaluate Explanation
We found three categories of saliency map usage, summarized in Table 2 First, all claims interpret salient areas as proxy for agent focus. For exam- ple, claim about Breakout agent notes that the network is focusing on the paddle and little else (Greydanus) al. 2017).
Jacobian
21
19
0
Perturbation
11
Object
2
Attention
Total Claims
46
40
3
Second, 87% the claims in our survey propose hypotheses about the features of the learned policy by rea- soning backwards about what repre- sentation might jointly produce the observed saliency pattern and agent
Table 2: Summary of the survey on usage of saliency maps in deep RL. Columns represent categories of saliency map usage, and rows represent categories of saliency map meth- ods, with each cell denoting the number of claims in those categories. Individual claims may be counted in multiple columns.
4

---

Published as a conference paper at ICLR 2020
behavior. These types of claims either develop a priori explanation of behavior and evaluate using saliency, or they propose an ad hoc explanation after observing saliency to reason about how the agent is using salient areas. One priori claim notes that the displayed score is the only differing factor between two states and evaluates that claim by noting that saliency focuses on these pixels (Zahavy al., (016). An ad hoc claim about a racing game notes that the agent is recogniz- ing a time-of-day cue from the background color and acting to prepare for new race (Yang et al. 2018).
Finally, only 7% (3 out of 46) of the claims drawn from saliency maps are accompanied by additional or more direct experimental evidence. One of these attempts corroborate the interpreted saliency behavior by obtaining additional saliency samples from multiple runs of the game. The other two attempt to manipulate semantics in the pixel input to assess the agent's response by, for example, adding an additional object to verify hypothesis about memorization (Annasamy & Sycara 2019)
3.2 COMMON PITFALLS IN CURRENT USAGE
In the course of the survey, we also observed several more qualitative characteristics of how saliency maps are routinely used.
Subjectivity. Recent critiques of machine learning have already noted a worrying tendency to con- flate speculation and explanation (Lipton & Steinhardt (2018). Saliency methods are not designed to formalize an abstract uumaan-understandable concept such as "aiming" in Breakout, and they do not provide means to quantitatively compare semantically meaningful consequences of agent behavior. This leads to subjectivity in the conclusions drawn from saliency maps.
Unfalsiability. One hallmark of a scientific hypothesis or claim is falsifiability (Popper 1959). If claim is false, its falsehood should be identifiable from some conceivable experiment or observation. One of the most disconcerting practices identified in the survey is the presentation of unfalsifiable interpretations of saliency map patterns. An example: "A diver is noticed in the saliency map but misunderstood an enemy and being shot at" (see Appendix C). It is unclear how might falsify an abstract concept such as "misunderstanding".
Cognitive Biases. Current theory and evidence from cognitive science implies that humans learn complex processes, such as video games, by categorizing objects into abstract classes and by in- ferring causal relationships among instances those classes (Tenenbaum Niyogi 2003 Dubey ett al. 2018). Our survey suggests that researchers infer that: (I) salient regions map to learned representations of semantic concepts (e.g., ball, paddle), and (2) the relationships among the salient regions map to high-level behaviors (e.g., tunnel-buildie aiming). Researchers' expectations im- pose a strong bias on both the existence and nature of these mappings.
4 METHODOL.OGY
Our survey indicates that many researchers use saliency maps as an explanatory tool to infer the rep- resentations and processes behind an agent's behavior. However, the extent to which such inferences are valid has not been empirically evaluated under controlled conditions.
In this section, we show how to generate falsifiable hypotheses from saliency maps and propose an intervention-base approach to verify the hypotheses generated from saliency maps. We intervene on game state to produce counterfactual semantic conditions. This provides concrete methodology to assess the relationship between saliency and learned semantic representations.
Building Falsifiable Hypotheses from Saliency Maps. Though saliency maps may not relate directly to semantic concepts, they may still be an effective tool for exploring hypotheses about agent behavior. As we show schematically in Figure claims or explanations informed by saliency maps have three components: semantic concepts, saliency, and behavior. Recall that our survey indicates that researchers often attempt to infer aspects of the network's 's learned representations from saliency patterns. Let X be a subset of the semantic concepts that can be inferred from the input image. Let B represent behavior, or aggregate actions, over temporally extended sequences of frames, and let R be representation that function of some pixels that the agent learns during training.
5

---

Published as a conference paper at ICLR 2020
To create scientific claims from saliency maps, we recommend using relatively standard pattern which facilitates objectivity and falsifiability:
(concept set X} is salient = agent has learned (representation R} resulting in {behavior B}.
Consider the Breakout brick reflection example presented in Section 2 The hypothesis introduced ("the agent has learned to aim at tunnels") can be reformulated as: bricks are salient = agent has learned identify partially complete tunnel resulting in maneuvering the paddle hit the ball toward that region. Stating hypotheses in this format implies falsifiable claims amenable to empirical analysis.
Counterfactual Evaluation of Claims. As indicated in Figure the learned representation and pixel input share relationship with saliency maps generated over a sequence of frames. Given that the representation learned is static, the relationship between the learned representation and saliency should be invariant under different manipulations of pixel input. We use this property to assess saliency under counterfactual conditions.
We generate counterfactual conditions by intervening on the RL environment. Prior work has fo- cused manipulating the pixel input. However, this does not modify the underlying latent game state. Instead, we intervenc directly on game state. In the do-calculus formalism (Pearl (000), this shifts the intervention node in Figure 2a to game state, which leaves the generative process F of the pixel image intact.
We employ ToyBox, 2018), to generate interventional data under counterfactual conditions. The interventions are de- pendent on the mapping between semantic concepts and learned representations in the hypotheses. Given a mapping between concept set X and a learned representation R. any intervention would require meaningfully manipulating the state which resides assess the saliency on X un- der the semantic treatment applied. Saliency on E X is defined as the average saliency over bounding-boo box around x.
of fully parameterized implementation of Atari games (Foley et al.,
Since the learned policies should be semantically invariant under manipulations of the RL environ- ment, by intervening on state, we can verify whether the counterfactual states produce expected patterns of saliency on the associated concept set X. If the counterfactual saliency maps reflect sim- ilar saliency patterns, this provides stronger evidence that the observed saliency indicates the agent has learned representation R corresponding to semantic concept set X.
5 EVALUATION OF HYPOTHESES ON AGENT BEHAVIOR
We conduct three studies to evaluate hypotheses about the relationship between semantic cepts and semantic processes formed from saliency maps. Each case study uses observed saliency maps identify hypotheses in the format described in Section 4 The hypotheses were generated by watching multiple episodes and noting atypical, interesting or popular behaviors from saliency maps. In each case study, we produce Jacobian, perturbation and object saliency maps from the same set of counterfactual states. We include examples of each map in Appendix A. Using ToYBoX al- lows produce counterfactual states and to generate saliency maps in these altered states. The case studies are conducted on two Atari games, Breakout and Amidar The deterministic nature of both games allows some stability the way we interpret the network's action selection. Each map is produced from an agent trained with A2C (Mnih et al. 2016) using CNN-based (Mnih et al., 2015) OpenAl Baselines implementation (Dhariwal et al. 2017) with default hyperparameters (see Appendix B for more details). Our choice of model is arbitrary. The emphasis of this work is on methods of explanation, not the explanations themselves.
'OOY Box provides transparency into the position of each object (measured at the center of sprite), which we use as the center the bounding boxes in our experiments. 2The code pertaining to the experiments can he found at https://ggt http://ggtttt.ooooo httpps////gtthub..co//lllll umlss// sa saliency_maps

---

Published as a conference paper at ICLR 2020
shift
04H
BNM
(6)
Figure 3: Interventions on brick configurations in Breakout. (a) saliency after shifting the brick positions by some pixels where shift=0 represents the original frame; (b) saliency after shifting the brick positions, ball, and paddle to the left. The pattern and intensity of saliency around the tunnel is not symmetric in the reflection interventions.
Case Study I: Breakout Brick Translation. Here evaluate the behavior from Section
Hypothesis 1: {bricks} salient agent has learned to {identify partially complete tunnel] resulting in (maneuvering the paddle to hit the hall toward that region).
To evaluate this hypothesis, we intervene on the state by translating the brick configurations horizon- tally. Because the semantic concepts relating the tunnel are preserved under translation, expect salience will be nearly invariant to the horizontal translation depicts saliency after intervention. Salience the tunnel is less pronounced under left translation, and more pronounced under right translation. Since the paddle appears on the right, we additionally move the ball and paddle to the far left (Figure
the brick configuration. Figure 3
Conclusion. Temporal association (e.g. formation of a tunnel followed by higher saliency) does not generally imply causal dependence. In this case, tunnel formation and salience appear to be confounded by location or, at least, the dependence of these phenomena are highly dependent on location.
Case Study 2: Amidar Score. Amidar is Pac-Man-like game in which an agent attempts to completely traverse a series of passages while avoiding enemies. The yellow sprite that indicates the location of the agent is almost always salient in Amidar. Surprisingly, the displayed score is often as salient as the yellow sprite throughout the episode with varying levels of intensity. This can lead to multiple hypotheses about the agent's learned representation: (1) the agent has learned to associate increasing score with higher reward; (2) due to the deterministic nature of Amidar, the agent has created lookup table that associates its score and its actions. We summarize these follows:
agent has learned to (use score guide to traverse the board} resulting in (successfully following similar paths in games}.
Hypothesis 2: (score) } is salient =
To evaluate hypothesis we designed four interventions on score:
intermittent reset: modify the score to 0 every E [5, 20) timesteps.
random_vare varyiin:: modify the score to random number between [1,200] every E [5, 20] timesteps.
fixed: select a score from [0,200] and fix it for the whole game.
decremented: modify score to be 3000 initially and decrement score by d [1,20) every timestep.
Figures 4h and 4p show the result of intervening displayed score reward and saliency inten- sity, measured as the average saliency over a 25x15 bounding box, respectively for the first 1000 timesteps of an episode. The mean is calculated over 50 samples. If an agent died before 1000 timesteps, the last reward was extended for the remainder of the timesteps and saliency was set to

---

Published as a conference paper at ICLR 2020
Froet
Sgorg
Reward
Seltency on
Sallency Deffrence
622
Time 380
3000
23
Original
(b)
(c)
Figure 4: Interventions on displayed score in Amidar. The legend in (b) applies to all figures. (a) ward over time for different interventions on displayed score; (b) object saliency on displayed score over time; (c) correlation between the differences in reward and object saliency from the original trajectory. Interventions
displayed score result in differing levels of degraded performance but produce similar saliency maps, suggesting that agent behavior as measured by rewards is underde- termined by salience.
Using reward summary of agent behavior, different interventions agent behavior. Total accumulated reward differs over time for all interventions, typically due to early agent death. However, salience intensity patterns of all interventions follow the original trajec- tory very closely. Different interventions on displayed score cause differing degrees of degraded per- formance (Figure 4 despite producing similar saliency maps (Figure 4p) indicating that agent be- havior is underdetermined by salience. Specifically, the salience intensity patterns are similar for the control, fixed, and decremented scores, while the non-ordered score interventions result degraded performance. Figure indicates only very weak correlations between the defferencc-in-rewane and difference-ing inw-ii--eleee-uuneeeee innee--intervention as compared to the original trajectory. Correlation coef- ficients range from 0.041 to 0.274, yielding insignificant p-values for all but one intervention. See full results Appendix E.1 Table
produce different
Similar trends are noted for Jacobian and perturbation saliency methods in Appendix E.1
The existence of high correlation between two processes (e.g., incrementing score and persistence of saliency) does not imply causation. Interventions can be useful in identifying the common cause leading to the high correlation.
Conclusion.
Case Study 3: Amidar Enemy Distance. Enemies are salient in Amidar at varying times. From visual inspection, observe that enemies close to the player tend to have higher saliency. Accord- ingly, we generate the following hypothesis:
Hypothesis 3: {enemy} is salient = agent has learned to {identify enemies close to it} resulting (successful avoidance of enemy collision).
Without directly intervening on the game state, we can first identify whether the player-enemy dis- tance and enemy saliency is correlated using observational data. We collect 1000 frames of an episode of Amidar and record the Manhattan distance between the midpoints of the player and en- emies, represented by 7x7 bounding boxes, along with the object salience of each enemy. Figure 5] shows the distance of each enemy to the player over time with saliency intensity represented by the shaded region. Figure 5p shows the correlation between the distance each enemy and the corresponding saliency. Correlation coefficients and significance values are reported in Table 3 It clear that there no correlation between saliency and distance of each enemy the player.
Given that statistical dependence is almost always necessary pre-condition for causation, expect that there will not be any causal dependence. To further examine this, we intervene on enemy positions of salient enemies at each timestep by moving the enemy closer and farther away from the player. Figure 5e contains these results. Given Hypothesis 3, we would expect to see an increasing trend in saliency for enemies closer to the player. However, the size of the effect is close (see Table addition, we find no correlation the enemy distance experiments for the Jacobian or perturbation saliency methods (included in Appendix
8

---

Published as a conference paper at ICLR 2020
Distance of Enemies from Player Over Time
Observed Enemy Distance from Player
Intervention on Enemy Distance from Player
Distnce Royer
Auemn on S
Aemeon in
26
Player
Intervention on Player
(a)
(b)
(c)
Figure 5: Interventions on enemy location in Amidar. The legend in (b) applies to all figures. (a) the distance-to-playe of each enemy in Amidar, observed over time, where saliency intensity is represented by the shaded region around each line; (b) the distance-to-plan
and saliency, with linear regressions, observed for each enemy; (c) variation enemy saliency when enemy position is varied by intervention. 'he plots suggest that there is no substantial correlation and no causal dependence between leltance-to-plyyee and object saliency.
Observational
Interventional
Enemy
slope
p-value
slope
p-value
-0.001
0.26
-0.036
-0.001
0.97
-0.001
-0.001
0.35
-0,298
0.013
0.68
0.039
0.004
1.59c-4
0.119
-0.008
0.79
-0.022
-0.008
8.26c-19
-0.275
-0.011
0.75
-0.028
0.001
0.13
0.047
-0.033
0.47
-0.063
Table 3: Numeric results from regression analysis for the observational and interventional results in Figures 5p and c. The results indicate a very small strength of effect (slope) for both observational and interventional data and a small correlation coefficient (r), suggesting that there is, at best, only a very weak causal dependence of saliency on to-ance-t--plaee..
Conclusion. Spurious correlations, or misinterpretations of existing correlation, can occur between two processes (e.g. correlation between player-enemy distance and saliency), and human observers are susceptible identifying spurious correlations (Simon) 1954). Spurious correlations can some- times be identified from observational analysis without requiring interventional analysis.
DISCUSSION AND RELATED WORK
Thinking counterfactually about the explanations generated from saliency maps facilitates empirical evaluation of those explanations. The experiments above show some of the difficulties in drawing conclusions from saliency maps. These include the tendency of human observers to incorrectly infer association between observed processes, the potential for experimental evidence to contradict seem- ingly obvious observational conclusions, and the challenges of potential confounding in temporal processes.
One of the main conclusions from this evaluation is that saliency maps are an exploratory tool rather than an explanatory tool for evaluating agent behavior in deep RL. Saliency maps alone cannot be reliably used to infer explanations and instead require other supporting tools. This can include combining evidence from saliency maps with other explanation methods employing more experimental approach to evaluation of saliency maps such the approach demonstrated in the case studies above.
The framework for generating falsifiable hypotheses suggested in Section assist with designing more specific and falsifiable explanations. The distinction between the components of an explana- tion. particularly the semantic concept set X, learned representation R and observed behavior B, can further assist in experimental evaluation. Note that the semantic space devised by an agent might be quite different from the semantic space given by the latent factors of the environment. It is cru-

---

Published as a conference paper at ICLR 2020
cial to note that this mismatch is aspect of what plays out when researchers create hypotheses about agent behavior, and the methodology we provide this work demonstrates how to evaluate hypotheses that reflect that mismatch.
Generalization of Proposed Methodology. The methodology presented in this work can be easily extended to other vision- based domains in deep RL. Particularly, the framework of the graphical model introduced in Figure 2 applies to all domains where the input to the network image data. An extended version of the model for Breakout can be found in Appendix 7
We propose intervention-base experimentation as a primary tool to evaluate the hypotheses gener- ated from saliency maps. Yet, alternative methods can identify false hypothesis even earlier. For instance, evaluating statistical dependence alone can provide strong evidence against causal depen- dence (e.g., Case Study 3). In this work, employ particularly capable simulation environment (ToYBox). However, limited forms evaluation may be possible in non non-intervenable environ- ments, though they may be conducted in Case Study 1 can be produced in an observation-only environment by manipulating the pixel input (Brunelli 2009 Chalupka et al. 2015). Developing experimental systems for evaluating explanations is an open area of research.
tedious implement. For instance, each of the interventions
This work analyzes explanations generated from feed-forward deep RL agents. However, the pro- posed methodology is not model dependent, and aspects of the approach will carry over to recurrent deep RL agents. The proposed methodology would not work well for repeated interventions on recurrent deep RL agents due to their capacity for memorization.
Explanations in Deep RL. Prior work has introduced alternatives to the use of saliency maps to support explanation of deep RL agents. Some of these methods also use counterfactual reasoning to develop explanations. ToyBox was developed support experimental evaluation and behavioral tests of deep RL models (Tosch et al.] 2019). Olson et al.] (2019) use generative deep learning architecture produce counterfactual states resulting in the agent taking different action. Others have proposed alternative methods for developing semantically meaningful interpretations of agent attribute policy behaviors accord- ing to semantically meaningful components of reward. |Verma et al. (2018) use Nomain-sspecine languages for policy representation, allowing for human- readable policy descriptions.
behavior. Juozapaitis al. (2019) use reward decomposition
Evaluation and Critiques of Saliency Maps. Prior work in the deep network literature has eval- uated and critiqued saliency maps. Kindermans et al. (2199) and and Adebayo al. (2018) demonstrate the utility of saliency maps by adding random variance in input. Seo et al. (2018) provide a theoret- ical justification of saliency and hypothesize that there exists correlation between wradients-based saliency methods and model interpretation. (aaeek et al. (2017) and Hooker et al. (2019) present evaluations of existing saliency methods for image classification.
7 CONCLUSIONS
We conduct survey of uses of saliency maps, propose methodology evaluate saliency maps, and examine the extent to which the agent's learned representations can be inferred from saliency maps. We investigate how well the pixel-level inferences of saliency maps correspond the semantic concept-level inferences of human-level interventions. Our results show saliency maps cannot be trusted to reflect causal relationships between semantic concepts and agent behavior, We recommend saliency maps to be used as an exploratory tool, not explanatory tool.
ACKNOWLEDGMENTS
Thanks to Emma Tosch, Amanda Gentzel, Deep Chakraborty, Abhinav Bhatia, Blossom Metevier, Chris Nota, Karthikeyan Shanmugam and the anonymous ICLR reviewers for thoughtful comments and contributions. This material based upon work supported by the United States Air Force under Contract No. FA8750- 17-50-17---0100 Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the United States Air Force.
10

---

Published as a conference paper at ICLR 2020
REFERENCES
Julius Adebayo, Justin Gilmer, Michael Muelly, lan Goodfellow, Moritz Hardt, and Been Kim. Sanity checks for saliency maps. In NeurlPS, 2018.
Raghuram Mandyam Annasamy and Katia Sycara. Towards better interpretability in deep q- networks. In AAAI, 2019.
Miroslav Bogdanovic, Dejan Markovikj, Misha Denil, and Nando De Freitas. Deep apprenticeship learning for playing video games. 2015.
Roberto Brunelli. Template Matching Techniques in Computer Vision: Theory and Practice. Wiley Publishing, 2009.
Ruth M. Byme. Counterfactuals in explainable artificial intelligence (XAI): Evidence from human reasoning. IJCAI, 2019.
Krzysztof Chalupka, Pietro Perona, and Frederick Eberhardt. Visual causal feature learning. In UAI, 2015.
Piotr Dabkowski and Yarin Gal. Real time image saliency for black box classifiers. In NeurlPS, 2017.
Maartje M. de Graaf and Bertram F. Malle. How people explain action (and autonomous intelli- gent systems should too). In AAAI Fall Symposia, 2017.
Prafulla Dhariwal, Christopher Hesse, Oleg Klimov, Alex Nichol, Matthias Plappert, Alec Radford, John Schulman, Szymon Sidor, and Yuhuai Wu. OpenAl Baselines. htttss///iithu: com/
openai/baseling 2017.
Rachit Dubey, Pulkit Agrawal, Deepak Pathak, Thomas L Griffiths, and Alexci A Efros. Investigat- ing human priors for playing video games. ICML, 2018.
John Foley, Emma Tosch, Kaleigh Clary, and David Jensen. Toybox: Better Atari Environments for
Testing Reinforcement Learning Agents. In NeurlPS Workshop on Systems for MI., 2018.
Ruth C Fong and Andrea Vedaldi. Interpretable explanations of black boxes by meaningful pertur- bation. ICCV, 2017.
Vikash Goel, Jameson Weng, and Pascal Poupart. Unsupervised video object segmentation for deep reinforcement learning. NeurIPS, 2018
Sam Greydanus, Anurag Koul, Jonathan Dodge, and Alan Fern. Visualizing and understanding atari agents. ICML, 2017.
Sara Hooker, Dumitru Erhan, Picter-Jan Kindermans, and Been Kim. Evaluating feature importance estimates. NeurlPS, 2019.
Rahul lyer, Yuezhang Li, Huao Li, Michael Lewis, Ramitha Sundar, and Katia Sycara. Transparency and explanation in deep reinforcement learning neural networks. AAAI, 2018.
Zoe Juozapaitis, Anurag Koul, Alan Fem, Martin Erwig, and Finale Doshi-Veler. Explainable reinforcement learning via reward decomposition. In IJCAI Workshop on Explainable Artificial Intelligence, 2019.
Pieter-Jan Kindermans, Sara Hooker, Julius Adebayo, Maximilian Alber, Kristof Schütt, Sven Dähne, Dumitru Erhan, and Been Kim. The (un) reliability of saliency methods. In Explainable AI: Interpreting, Explaining and Visualizing Deep Learning. 2019.
Zachary C Lipton and Jacob Steinhardt. Troubling trends in machine learning scholarship. arXiv preprint 180.vv8077.03 2018.
Brent Mittelstadt, Chris Russell, and Sandra Wachter. Explaining explanations in ai. Conference Fairness, Accountability and Transparency, 2019.

---

Published as a conference paper at ICLR 2020
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforeement learning, Nature, 2015.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In ICML, 2016.
Alex Mott, Daniel Zoran, Mike Chrzanowski, Daan Wierstra, and Danilo J Rezende. To- wards interpretable reinforcement learning using attention augmented agents. arXiv preprint 1rXiv:1906.02500. 2019.
Dmitry Nikulin, Anastasia lanina, Vladimir Aliev, and Sergey Nikolenko. Free-lunch saliency via attention in atari agents, arXiv preprint arXiv: 2019.
Matthew L Olson, Lawrence Neal, Fuxin Li, and Weng- Keenn Wong. Counterfactual states for atari agents via generative deep learning. IJCAI Workshop on Explainable Artificial Intelligence, 2019.
Judea Pearl. Causality: models, reasoning and inference, volume 29. Springer, 2000.
Karl Popper. The logic of scientific discovery. Routledge, 1959.
Marco Tulio Ribeiro. Sameer Singh, and Carlos Guestrin. Why should trust you?: Explaining the predictions of any classifier. In ACM SIGKDD, ACM, 2016.
Christian Rupprecht, Cyril Ibrahim, and Chris Pal. Visualizing and discovering behavioural weak- 2018.
nesses in deep reinforcement learning. arXiv preprint
Wojciech Samek, Alexander Binder, Grégoire Montavon, Sebastian Lapuschkin, and Klaus-Robert Müller. Evaluating the visualization what a deep neural network has learned. IEEE transactions neural networks and learning systems, 2017.
Richard Scheines. The similarity of causal inference in experimental and nonexperimental studies. Philosophy of Science, 2005. 2005.
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, Dhruv Batra, al. Grad-cam: Visual explanations from deep networks via gradient-based local- ization. In ICCV, 2017.
Junghoon Sco, Jeongyeol Choe, Jamyoung Koo, Seunghyeon Jeon, Bcomsu Kim, and Taegyun Jeon. Noise-adding methods of saliency map as series of higher order partial derivative. ICMI. Workshop on Human Interpretability in Machine Learning, 2018 8.
Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. Learning important features through propagating activation differences. ICML, 2017.
Herbert A. Simon. Spurious correlation: A causal interpretation. Journal of the American Statistical Association. 1954.
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. ICLR Workshop, 2014.
Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda Viégas, and Martin Wattenberg. Smoothgrad: removing noise by adding noise. ICML Workshop on Visualization for Deep Learning. 2017.
Jost Tobias Springenberg, Alexey Dosovitskiy, Thomas Brox, and Martin Riedmiller. Striving for simplicity: The all convolutional net. ICLR, 2015.
Ramitha Sundar. Transparency in Deep Reinforcement Learning Networks. PhD thesis, Camegii Mellon University, 2018
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 1998.
Christian Szegedy, Wojciech Zaremba, llya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. In ICLR, 2014.
12

---

Published as a conference paper at ICLR 2020
Joshua B Tenenbaum and Sourabh Niyogi. Learning causal laws. In Proceedings of the Annual Meeting of the Cognitive Science Society, 2003.
Emma Tosch, Kaleigh Clary, John Foley, and David Jensen. Toybox: A suite of environments for experimental evaluation of deep reinforcement learning, 2019.
Douwe van der Wal, Bachelor Opleiding Kunstmatige Intelligentie, and Wenling Shang. Advantage actor-critic methods for carracing. 2018.
Abhinav Verma, Vijayaraghavan Murali, Rishabh Singh, Pushmeet Kohli, and Swarat Chaudhuri. Programmatically interpretable reinforcement learning. In ICML, 2018
Ziyu Wang, Tom Schaul, Matteo Hessel, Hado Van Hasselt, Marc Lanctot, and Nando De Freitas. Dueling network architectures for deep reinforcement learning. ICML., 2016.
Zhao Yang. Song Bai, Li Zhang, and Philip HS Torr. Learn interpret atari agents. arXiv preprint arXiv:1812.11276, 2018.
Tom Zahavy, Nir Ben-Zrihem, and Shie Mannor. Graying the black box: Understanding dqns. 2016.
Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In ECCV, 2014.
Jianming Zhang, Sarah Adel Bargal, Zhe Lin, Jonathan Brandt, Xiaohui Shen, and Stan Sclaroff. Top-down neural attention by excitation backprop. International Journal of Computer Vision, 2018.
13

---

Published as a conference paper at ICLR 2020
APPENDICES
A SALIENCY METHODS
Figure shows example saliency maps the three saliency methods evaluated in this work, namely perturbation, object and Jacobian, for Amidar.
200
200
100
a
(b)
(c)
Figure 6: Examples of (a) perturbation saliency method (Greydanus et al.
2017 (b) object saliency
method [yer et al. 2018), and (c) Jacobian saliency method (Wang et al.
2016 for Amidar.
B MODEL
We use the OpenAl Baselines' implementation (Dhariwal al. 2017) of an A2C model (Mnih ict al.) (0166) to train the RL agents on Breakout and Amidar. The model uses the CNN architecture proposed by Mnih et al. (2015). Each agent is trained for 40 million iterations using RMSProp with default hyperparameters (Table
Hyperparameter
Value
Learning rate
7e-4
Learning rate schedule
Linear
# Iterations
40,000,000
Value function coefficient
0.5
Policy function coefficient
0.01
RMSProp epsilon
le-5
R.SPProp decay
0.99
Reward discounting parameter
0.99
Max gradient (clip)
0.5
Table Hyperparameters used in training the A2C model.
14

---

Published as a conference paper at ICLR 2020
C SURVEY OF USAGE OF SALIENCY MAPS IN DEEP RL LITERATURE
We conducted survey of recent literature to assess how saliency maps are used to interpret agent behavior in deep RL. We began our search by focusing on work citing the following four types of saliency maps: Jacobian (Wang et al., 2016), perturbation (Greydanus et al., 2017). object (Iyer al. 2018) and attention (Mott al., 2019). Papers were selected they employed saliency maps to create explanations regarding agent behavior. This resulted in selecting 46 claims from 11 papers.
These 11 papers have appeared ICML (3), NeurIPS (1), AAAI (2), ArXiv (3), OpenReview (1) and as thesis (1). There are several model-specific saliency mapping methods that we excluded from our survey.
Following is the full set of claims. All claims are for Atari games. The Reason column repre- sents whether an explanation for agent behavior was provided (Y/N) and the Exp column represents whether an experiment was conducted to evaluate the explanation (Y/N).
Claim
Game
Saliency Type
Reason
Exp
Greydanus et al. (2017)
"The agent is positioning its own paddle, which allows it to return the ball at specific angle." "Interestingly, from the saliency we see that the agent attends to very little besides its own paddle: even the ball."
Pong
Perturbation
Pong
Perturbation
"After the agent has executed the kill shot, we see that saliency centers entirely around the ball. This makes sense since at this point neither paddle can alter the outcome and their positions are irrele- vant."
Pong
Perturbation
N
"It appears that the deep RL agent exploiting the deterministic nature of the Pong environment. has learned that it can obtain reward with high certainty upon executing precise series of ac- tions. This insight...give evidence that the agent is not robust and has overfit the particular op- ponent."
Pong
Perturbation
"[The agent] had learned sophisticated aim- ing strategy during which first the actor and then the critic would 'track' target. Aiming begins when the actor highlights a particular alien in blmin..Aiming ends with the agent shooting the new target."
Space- Invaders
Perturbation
Y
N
"The critic highlights the target in anticipation of an upcoming reward."
Space-
Perturbation
Invaders
"Notice that both actor and critic tend to monitor the area above the ship. This may be useful for de- termining whether the ship is protected from Invaders
Space-
Perturbation
N
emy fire or has clear shot at enemies."
"We found that the agent enters and exits 'tun- neling mode" over the course of single frame. Once the tunneling location becomes salient, re- mains so until the tunnel is finished."
Breakout
Perturbation
Zahavy et al. 2016
"A diver is noticed in the saliency map but misun- derstood as an enemy and being shot at."
Seaquest
Jacobian
N
15

---

Published as a conference paper at ICLR 2020
"Once the agent finished the first screen it is pre- sented with another one, distinguished only by the score that was accumulated in the first screen. Therefore, an agent might encounter problems with generalizing
Breakout
Jacobian
N
N
the screen if it over-fits the score pixels. Figure 15 shows the saliency maps of different games supporting our claim that DQN is basing its estimates using these pixels. We suggest to further investigate this, for exam- ple, we suggest to train an agent that does not re- ceive those pixels as input."
Bogdanovic et al. 2015
"The line cars in the upper right are far away and the agent correctly ignores them in favour of focusing on the much more dangerous cars the lower left."
Freeway
Jacobian
Yang et al. (0018)
"Firstly, the agent ignores irrelevant features from mountains, sky, the mileage board and empty grounds, and relies information from the race track to make decisions. Specifically, the agent keeps separate
Enduro
Binary Jacobian
Y
N
categories of objects the race track, i.e., cars and the player."
"On the hand, the agent locates the player and a local area around it for avoiding immediate col- lisions with cars. On the other hand, the agent locates the next potential collision targets at dif- ferent locations, particularly the remote ones."
Enduro
Binary
Jacobian
"Near the completion of the current goal, the agent celebrates advance, As shown from Fig.
Enduro
Binary Jacobian
Y
N
4(d) to Fig. 4(f), the left gaze loses its focus on cars and diverts to the mileage board starting when only 13 cars remain before completion. The pre- vious car tracker now picks up on the important information that it is close to victory, and fixates on the countdown."
"Upon reaching the target, the agent does not ceive reward signals until the next day starts. Dur- ing this period the agent learns to output no-op actions, corresponding to not playing the game."
Enduro
Binary Jacobian
Y
N
"When slacking happens, the agent considers the flag signs as important and the road not. The com- plete reverse in focus compared to the normal case explains this shift in policy. The flags out- weigh the road importance, since they signs of absolute zero return"
Enduro
Binary Jacobian
Y
N
"(Prepping) As it turns out, the agent recognizes that the time is dawn (right before morning when race starts) from the unique colours of the light gray sky and orange mountains, therefore the agent gets ready early for head start in the new race."
Enduro
Binary Jacobian
N
"When smog partially blocks the forward view, the left gaze loses its focus on cars. It strays off the road into some empty area."
Enduro
Binary Jacobian
16

---

Published as a conference paper at ICLR 2020
"In Fig. 5(a), the left gaze detects the two ghosts on the upper-right corner of the map. Therefore, pacman, located by the right gaze, stays in the mid-left section to safely collect dense re- wards.
Pacman
Binary Jacobian
Y
N
"In Fig. 5(b), the left gaze locks in on all three vulnerable ghosts in the mid-right section, pacman chases after them."
Pacman
Binary Jacobian
"In Fig. 5(c), the left gaze detects newly- appeared cherry at the lower-left warp tunnel en- trance. Ms pacman immediately enters the elosest opposite tunnel entrance in the shortest path to the cherry."
Pacman
Binary Jacobian
Y
N
"In Fig. S(d), the right gaze locates ms pacman entering the upper-right tunnel. In this case, the left gaze no longer detects moving object, but predicts the upper-left tunnel the exiting point."
Pacman
Binary Jacobian
"As shown in Fig. 5(g). the left gaze locates the
Pacman
Binary Jacobian
N
last pellet when ms pacman is in the mid-section of the maze. Therefore ms pacman moves towards the pellet."
"In Fig. 5(h), a red ghost appears in the left gaze close the pellet, causing ms pacman to deviate to the right."
Pacman
Binary
N
Jacobian
"After changing course, the ghosts approach ms pacman from all directions as shown in Fig. 5(i). Even though the agent detects all the ghosts (they appear in the gazes), ms pacman has no route to escape."
Pacman
Binary
Jacobian
"The left gaze often focuses white ice blocks that are the destinations of jumping."
Frostbite
Binary
N
N
Jacobian
"Fig. 6(b) shows the sub-task of the player enter- ing the igloo, after jumping over white ice blocks for building it. The player must avoid the bear when running for the igloo."
Frostbite
Binary
N
Jacobian
"As it shows, the igloo is two jumps away from Frostbite completion, and the left gaze focuses on the igloo in advance for preparing to enter."
Binary
Y
N
Jacobian
Annasamy & Sycara 2019
"For example, in MsPacman, since visualizations suggest that the agent may be memorizing pac- mans positions (also maybe ghosts and other ob- jects), we simply add an extra pellet adjacent to trajectory seen during training (Figure 7a). The agent does not clear the additional pellet and sim- ply continues execute actions performed during training (Figure 7b)."
Pacman
Object
Y
Y
"Similarly, case of SpaceInvaders, the agent has strong bias towards shooting from the lefumost- end (seen in Figure 4). This helps in clearing the triangle like shape and moving to the next level (Figure 7d). However, when triangular positions of spaceships are inverted, the agent repeats the same strategy of trying to shoot from left and fails to clear ships (Figure 7c)."
Space- Invaders
Object
Y
17

---

Published as a conference paper at ICLR 2020
Goel et al. (018
"In our Breakout results, the network learns to Breakout Object
split the paddle into left and right side, and does not move the middle portion."
"On Beam Rider, our network achieves low loss after learning to segment the games light beams, however these beams purely visual effects and are unimportant for action selection."
Beam Rider
Object
Y
N
"Consequently, the game enemies, which are much smaller in size, are ignored by the network, and the resulting learned representation is not that useful for reinforcement learning agent."
Beam Rider
Object
N
N
van der Wal et al.
2018)
"There is a very notable difference between the policy saliency between the models, where the former one only pays limited attention to the road and almost attention the engine indica- tor, the opposite from FA3C-LSTM. Explicitly.it means masking any regions from the imput does not cause much perturbation to the policy when trained with continuous space as targets, likely cause the real consequence from small change in action, e.g. no braking (a3= 0) versus braking (a3= 0.3), can be very substantial but numerically too subtle for the network to capture during opti- mization the continuous spectrum."
CarRacing
Perturbation
Rupprecht et al.
2018
"Analyzing the visualizations Seaquest, make an interesting observation. When maximiz- ing the Q-value for the actions, in many samples we see a low or very low oxygen meter. In these cases the submarine would need to ascend the surface to avoid suffocation. Although the up ac- tion is the only sensible choice in this case, we also obtain visualized low oxygen states for all other actions. This implies that the agent has not understood the importance of resurfacing when the oxygen is low. We then run several roll outs of the agent and see that the major cause of death indeed suffocation and not collision with ene- mies."
SeaQuest
Perturbation
Y
Y
Mott et al. 2019
"The most dominant pattern observe is that SeaQuest
Attention
N
the model learns to attend to task-relevant things in the scene. In most ATARI games that usually means that the player is one of the foci of atten- tion, well as enemies, power-ups and the score itself (which is an important factor in the calculat- ing the value function)."
18

---

Published as a conference paper at ICLR 2020
"Figure shows examples of this in Ms Pac- man and Alien in the both games the model scans through possible paths, making sure there no enemies or ghosts ahead. We observe that when does ghost, another path produced executed in order to avoid it."
Pacman
Attention
Y
N
"In many games observe that the agent learns to place trip-wires at strategic points in space such that if game object crosses them a specific ac- tion is taken. For example, in Space Invaders two such trip wires are following the player ship on both sides such that if a bullet crosses one of them the agent immediately evades them by moving wards the opposite direction."
Space In- vaders
Attention
"Another example Breakout where we can see it working in two stages. First the attention is spread out around the general area of the ball, then fo- cuses into localized line. Once the ball crosses that line the agent moves towards the ball."
Breakout
Attention
"As can be seen, the system uses the two modes to make its decisions, some of the heads are con- tent specific looking for opponent cars. Some mixed, scanning the horizon for incoming cars and when found, tracking them, and some are lo- cation based queries, scanning the area right in front of the player for anything the crosses its path (a trip-wire which moves with the player)."
Enduro
Attention
Y
N
"Comparing the attention agent to the baseline agent, we see that the attention agent is sensitive more focused areas along the possible future trajectory. The baseline agent is more focused on the area immediately in front of the player (for the policy saliency) and on the score, while the atten- tion agent focuses more specifically on the path the agent will follow (for the policy) and pos- sible future longer term paths (for the value)."
Pacman
Perturbation
Y
N
Nikulin et al.
2019
"Figs. 4(a)-(b) show Dense FLS digging a tunnel through blocks in Breakout. The model focuses its attention on the end of the tunnel as soon as is complete, suggesting that it sees shooting the ball through the tunnel as a good strategy."
Breakout
Attention
"Figs. 4(c)-(d) depict the same concept of tun- neling per-formed by the FLSmmdee. Note how it focuses attention the upper part of the screen after destroying multiple bricks from the top. This attention does not go away after the ball moves elsewhere (not shown in the images). We speculate that this how the agent models tun- neling:rather than having a high-level concept of digging a tunnel,it simply strikes wherever has managed to strike already."
Breakout
Attention
Y
N
19

---

Published as a conference paper at ICLR 2020
"Figs. 4())-(f) illustrate how the Dense FLS model playing Seaquest has learned to attend to in-game objects and, importantly, the oxygen bar at the bottom of the screen. As the oxygen har nearing depletion, attention focuses around it,and the submarine reacts by rising to refill its air sup- ply."
Seaquest
Attention
Y
N
"Figs. ((g)-(h) are two consecutive frames where an agent detects target appearing from the left side of the screen. The bottom part of the screen- shots shows how attention in the bottom left cor- ner lights up soon only few pixels wide, appears from the left edge of the screen. In the next frame, the agent will turn left and shoot the target (not shown here). How- ever, the agent completely ignores targets in the top part of the screen, and its attention does not move they move (also not shown)."
Breakout
Attention
tiny part of the target,
Wang et al. (016
"The value stream learns pay attention to the Enduro
Jacobian
N
road. The advantage stream learns to pay attention only when there are cars immediately front, so as to avoid collisions."
D GENERALIZATION OF CAUSAL GRAPHICAL MODEL
The causal graphical model in Figure 2a can be extended different domains in RL where the input to the model is an image. This requires extending game state to include the underlying state variables. Figure the extension for Breakout using an A2C model. Note, game state includes the underlying state variables for Breakout and logits were split into action logits and value as outputted by A2C.
channete
alve bricks
carkumne
Bell poottoon
Buil velcarty
Paedi posttor
Lives
RL Environment
Value
Network
logts
System
Framework
Figure 7: Causal graphical model for Breakout.
20

---

Published as a conference paper at ICLR 2020
Object
Perturbation
Jacobian
Intervention
p
r
p
p
intermittent reset
0.274
1.09e- 18
-0.10
1.52e-3
0.149
2.26c
uudommvaryiin fixed
0.142
6
-0.02
0.49
0.281
1.2le-19
0.041
0.20
-0.02
0.44
0.286
2.51c 20
decremented 0.119
0.15e-3
-0.09
6.73e-3
0.261
5.31e-17
Table 6: Numeric results representing Pearson's correlation coefficient and p-value for the differ- ences in reward and saliency (object, perturbation and Jacobian) from the original trajectory for each intervention. Results show small correlation coefficients (r) suggesting that there is weak relationship between the differences in reward and saliency for the interventions.
Sallancy Differance from Original
intumittnn rost
Sallency Score
Time
Time
Reward Differance from Original
(b)
Figure 8: Interventions on displayed score in Amidar (legend in (b) applies to all figures). (a) re- ward over time for different interventions displayed score; (b) perturbation saliency displayed score over time; (c) correlation between the differences in reward and perturbation saliency from the original trajectory. Interventions displayed score result in differing levels of degraded perfor- mance but produce similar saliency maps, suggesting that agent behavior as measured by rewards is underdetermined by salience.
E EVALUATION OF HYPOTHESES ON AGENT BEHAVIOR
E.
CASE STUDY 2: AMIDAR SCORE
We further evaluated the effects of interventions on displayed score (Section 5) in Amidar on per- turbation and Jacobian saliency maps. These results are presented in Figures and respectively.
We also evaluated Pearson's correlation between the differences reward and saliency with the original trajectory for all three methods (see Table (). The results support the correlation plots in Figures and gb.
E.2 CASE STUDY 3: AMIDAR ENEMY DISTANCE
We further evaluated the relationship between player-enemy distance and saliency (Section in Amidar on perturbation and Jacobian saliency maps. These results are presented in Figures I0 and II respectively. Jacobian saliency performed the for the intervention-basse experiment, suggesting that there is no impact of player-enemy distance on saliency.
Regression analysis between distance and perturbation and Jacobian saliency can be found in Tables 7 and 8 respectively. The results support the lack of correlation between the observational and interventional distributions. Note, enemy more salient throughout the game compared to the other four enemies resulting in larger interventional sample size.
21

---

Published as a conference paper at ICLR 2020
Sallercy Difference eeneeeenee from Original
Saliency on Scare
Time
Time
Reward Differance from Original
(b)
(c)
Figure 9: Interventions displayed score in Amidar for Jacobian saliency (legend in (b) applies to all figures). (a) reward over time for different interventions on displayed score; (b) saliency on displayed score over time; (c) correlation between the differences in reward and saliency from the original trajectory. Interventions on displayed score result in differing levels of degraded perfor- mance but produce similar saliency maps, suggesting that agent behavior measured by rewards is underdetermined by salience.
Distance of Enemiee from Flaee Over Time
Observed Enery Distance from Player
Intervention on Enemy Distance from Player
Difference Saleecy over Enemy
200
353
Dizaacee
309
-25
2000
UP
Intervenzin Distance Player
(b)
(c)
Figure 10: Interventions on enemy location in Amidar (legend in (b) applies to all figures). (a) the distance-to- to-pcyer of each enemy in Amidar, observed represented by the shaded region around each line; (b) the distance-to-playy and saliency, with linear regressions, observed for each enemy; (c) variation in enemy saliency when enemy position is varied by intervention. The plots suggest that there is no substantial correlation and no causal dependence between distance-to-player and perturbation saliency.
time, where saliency intensity is
Enemy Player
Diterence in Sallency Ower Ensems
Intervention Enemy from Player
Enerty
0.0
3
075
Tome
Distance to Player
Inter Intervention on Distance to Player
(a)
(b)
(c)
Figure 11: Interventions enemy location in Amidar (legend in (b) applies all figures). (a) the distance-to-playy of each enemy in Amidar, observed over time, where saliency intensity is represented by the shaded region around each line; (b) the distance-to-player and saliency, with linear regressions, observed for each enemy; (c) variation in enemy saliency when enemy position varied by intervention. The plots suggest that there is no substantial correlation and no causal dependence between distance-to-plaat player and Jacobian saliency.
22

---

Published as a conference paper at ICLR 2020
Observational
Interventional
Enemy
Slope
p-value
r
Slope
p-value
r
-0.052
7.04e-10 10
-0.204
-0.374
0.01
-0.126
-0.043
1.09e-9
-0.191
0.536
0.16
0.132
-0,065
6,04e-20
-0.283
-0.039
0.92
-0.008
-0.103
2.57e-25
-0.320
-0,005
0.98
0,003
-0.062
3.44c-12
-0.217
-0.083
0.71
0.032
Table Numeric results from regression analysis for the observational and interventional results in Figures and c. The results indicate a very small strength of effect (slope) for both observational and interventional data and a small correlation coefficient (r), suggesting that there is, at best, only very weak causal dependence of saliency on distance-to-player.
Observational
Interventional
Enemy
Slope
p-value
Slope
p-value
-0.008
0.08
-0.055
0
0
2
-0.029
4.17e-6
-0.145
0
0
3
-0.031
3.57e-4
-0.113
0
0
4
-0.023
5.97e-3
-0.087
0
0
5
-0.013
0.02
-0.076
0
Table Numeric results from regression analysis for the observational and interventional results in Figures and c. The results indicate a very small strength of effect (slope) for both observational and interventional data and a small correlation coefficient (r), suggesting that there is, best, only very weak causal dependence of saliency on distance-to-player.
23


Reviews:
[{"review_id": "rygokYDpYr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Abstract:\nThe author suggests that saliency maps should be viewed as exploratory tools rather than explanation. The explore this idea in the context of a game.\n\nHere is my main issue:\nAlthough I believe there is a value in studies like this. I am not sure ICLR is the right venue for it. The paper is well written but it reads like a long opinion/blog-post. There is no overarching theory or generalizable observation not to mention a solution. \nYes, I agree that the method of interpreting the black box has a lot of issues and the counterfactual approach/causal approach is probably the right way to go but this is hardly news to the community. \n\nIn short: what the generalizable contribution of the paper?\n\nI am open to change my mind if the discussion is convincing. ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I did not assess the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "B1l6TLZCKB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper has a double aim. First, it is a survey on saliency maps used in explaining deep reinforcement learning models. Second, it is a proposal of a method that should overcome limitations of the current approaches described in the survey.\nThis double aim makes the paper hard to understand as the survey is not complete and the model is not well explained.\nThe main limitations the novel model aims to solve seems to be the production of \"falsifiable\" hypothesis in the explaination with saliency maps. However, experiments are really hard to follow and it is not clear why this is the case.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}, {"review_id": "B1gg4VDstB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "[score raised from weak accept to accept due to rebuttal/improvements]\n\nSummary\nThe paper investigates the practice of using pixel-level saliency maps in deep RL to “explain” agent behavior in terms of semantics of the scene. The main problem, according to the paper, is that pixel-level saliency maps often correlate with semantic objects, however turning these correlations into explanations would require counterfactual analysis / interventions, which are almost never performed in practice. The paper highlights this issue with an extensive literature survey, and proposes a simple method to formulate “explanations” found via saliency maps into falsifiable hypotheses. Three experiments show how to apply the methodology in practice - in all three cases pixel-level correlations cannot be easily mapped to semantic-level explanations that hold (counterfactual) validation.\n\nContributions\nThe paper nicely summarizes the main contributions, namely: (i) a literature survey on pixel-saliency methods in deep RL and their use to “explain” agent behavior, (ii) a detailed description of the problem with the latter and a proposal to mitigate the main issues, and (iii) three experimental case-studies to illustrate the problem further and show how the proposed method can help.\n\nQuality, Clarity, Novelty, Impact\nThe paper addresses a highly important issue in the field of interpretable deep learning. The main message is that a lack of scientific rigor, namely stating falsifiable hypotheses and validation of claimed hypotheses, can easily lead to misinterpretation of deep RL systems. This is a somewhat disenchanting message, but I personally think it is important to ensure that this message is heard in the field of interpretable ML in particular, and in the wider deep learning community in general. It is tempting to give simple answers to complex problems, and while I think saliency maps will play a large role in interpreting deep network decisions, I am also convinced that we need causal explanations, which salience maps (currently) cannot provide on a semantic level. The paper is well written and clear, the literature survey is quite extensive and valuable. The experimental results are nice, however they currently crucially lack quantitative statements that back up the qualitative results (see improvements below). While the latter must be included for publication, I am fairly confident that this can be rectified during the rebuttal phase and therefore (tentatively) vote for acceptance.\n\nImprovements\na) Mandatory for publication! Back the results-plots up by numbers! In particular: visually estimating densities / correlations from scatter plots is often impossible and misleading - while the plots are nice to have, the claims regarding Figure 5, 8, 9 (b) and (c) must be backed up by reporting actual correlations / statistical tests. For instance, it is impossible to judge visually whether there’s any trend in 5 (c). Please report correlations for 5, 8, 9 (b) and perform suitable statistical tests for measuring increase/decrease in correlation for 5, 8, 9 (c).\nSimilarly, please report an appropriate metric to quantitatively judge the difference between the curves in 4, 6, 7 (c). It’s fine to include tables reporting the quantitative results in the appendix, they don’t necessarily have to be in the main paper.\n\nb) Experimental details. Please report the details required to reproduce the experiments. In particular, what was the precise architecture for A2C and the hyper-parameter settings (particularly since the reference that is cited is not a paper, but a GitHub repo). For the figures, please report how saliency was measured exactly (was there a bounding-box around saliency/enemies? What was its size? Were intensities somehow normalized, were distances to enemies measured between centers of bounding-boxes, …?)\n\nc) (Optional). It would be nice to see an example where the method is used but the original hypothesis is not rejected (i.e. there’s now stronger evidence for the original hypothesis due to the counterfactual analysis). I understand that this is beyond the scope of the rebuttal, and feel free to completely ignore this.\n\n\nMajor Comments\nI) Please state whether the paper was written with feed-forward deep RL agents only in mind, or whether the paper is intended to also include recurrent deep RL agents (it would also be helpful to know whether the experiments used a feed-forward, or a recurrent version of A2C). While I think that many aspects carry over from feedforward architectures to recurrent ones, I personally think that some issues with counterfactual analysis could become more intricate with recurrent agents. For instance, on page 6, the described invariance in the first paragraph under ‘Counterfactual Evaluation of Claims’ is fine for feed-forward agents, but could be debatable with recurrent agents.  If you agree, please make this distinction clear in the paper (where appropriate) or state that the paper only applies to feedforward agents. If you disagree please indicate this during the rebuttal discussion. \n\nII) Page 6, just above Sec. 5: “Since the learned policies should be semantically invariant under manipulations of the RL environment...”. I agree that they should ideally be invariant, for the semantic interventions to make sense, but please comment on whether this is a trivial assumption, how this assumption could (in principle) be verified and the potential consequences of this assumption being violated. I personally think that there’s a fair chance that the semantic space carved up by the agent (that potentially overfits a task/family of tasks) might be quite different from the semantic space given by the latent factors of the environment. This mismatch and its potential interference with the method should be discussed as a current shortcoming.\n\nMinor Comments\nI)  A potential subtlety (which I don’t expect you to resolve/discuss in the paper) is that feed-forward agents in an MDP environment can behave like recurrent agents by offloading memory into the environment. E.g. a breakout agent could “memorize” that it is in “tunnel-digging mode” by moving the paddle by a few pixels - this could then potentially shift it’s saliency away from the actual tunnel to the corresponding pixels around the paddle. Such cases might be very hard to interpret via saliency maps or interventional analysis, but I acknowledge that this is perhaps a more exotic case, given the current state of interpretable deep RL. Just a thought for future work perhaps...", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This was a contentious paper, with quite a large variance in the ratings, and ultimately a lack of consensus. After reading the paper myself, I found it to be a valuable synthesis of common usage of saliency maps and a critique of their improper interpretation. Further, the demonstration of more rigorous methods of evaluating agents based on salience maps using case studies is quite illustrative and compelling. I think we as a field can agree that we’d like to gain better understanding our deep RL models. This is not possible if we don’t have a good understanding of the analysis tools we’re using.

R2 rightly pointed out a need for quantitative justification for their results, in the form of statistical tests, which the authors were able to provide, leading the reviewer to revise their score to the highest value of 8. I thank them for instigating the discussion.

R1 continues to feel that the lack of a methodological contribution (in the form of improving learning within an agent) is a weakness. However, I don’t believe that all papers at deep learning conferences have to have the goal of empirically “learning better” on some benchmark task or dataset, and that there’s room at ICLR for more analysis papers. Indeed, it’d be nice to see more papers like this.
 
For this reason, I’m inclined to recommend accept for this paper. However this paper does have weaknesses, in that the framework proposed could be made more rigorous and formal. Currently it seems rather adhoc and on a task-by-task basis (ie we need to have access to game states or define them ourselves for the task). It’s also disappointing that it doesn’t work for recurrent agents, which limits its applicability for analyzing current SOTA deep RL agents. I wonder if authors can comment on possible extensions that would allow for this.

Decision: Accept (Poster)