Explain Your Move: Understanding Agent Actions Using Specific and Relevant Feature Attribution
Authors: Nikaash Puri
Sukriti Verma
Piyush Gupta
Dhruv Kayastha
Shripad Deshmukh
Balaji Krishnamurthy
Sameer Singh
Venue:  (2020)
Keywords: Deep Reinforcement Learning
Saliency maps
Chess
Go
Atari
Interpretable AI
Explainable AI

Abstract: As deep reinforcement learning (RL) is applied to more tasks, there is a need to visualize and understand the behavior of learned agents. Saliency maps explain agent behavior by highlighting the features of the input state that are most relevant for the agent in taking an action. Existing perturbation-based approaches to compute saliency often highlight regions of the input that are not relevant to the action taken by the agent. Our proposed approach, SARFA (Specific and Relevant Feature Attribution), generates more focused saliency maps by balancing two aspects (specificity and relevance) that capture different desiderata of saliency. The first captures the impact of perturbation on the relative expected reward of the action to be explained. The second downweighs irrelevant features that alter the relative expected rewards of actions other than the action to be explained. We compare SARFA with existing approaches on agents trained to play board games (Chess and Go) and Atari games (Breakout, Pong and Space Invaders). We show through illustrative examples (Chess, Atari, Go), human studies (Chess), and automated evaluation methods (Chess) that SARFA generates saliency maps that are more interpretable for humans than existing approaches. For the code release and demo videos, see: https://nikaashpuri.github.io/sarfa-saliency/.

Published as a conference paper at ICLR 2020
EXPLAIN YOUR MOVE:
UNDERSTANDING AGENT ACTIONS USING SPECIFIC AND RELEVANT FEATURE ATTRIBUTION
Nikaash Puri 4
Sukriti Verma" Piyush Gupta" Dhruv Kayasthal**
Shripad Deshmukh
Balaji Krishnamurthy? Sameer Singh
Media and Data Science Research, Adobe Systems Inc., Noida, Uttar Pradesh, India 201301
$ Indian Institute of Technology Kharagpur, West Bengal, India 721302
IIndian Institute of Technology Madras, Chennai, India 600036
ll Department of Computer Science, University of California, Irvine, California, USA (nikpuri, sukrverm, piygupta @odobe..omm
ABSTRACT
As deep reinforcement learning (RL) is applied to more tasks, there is need to visualize and understand the behavior of learned agents. Saliency maps explain agent behavior by highlighting the features of the input state that are most relevant for the agent taking an action. Existing perturbation-basee pute saliency often highlight regions of the input that are not relevant to the action taken by the agent. Our proposed approach, SARFA (Specific and Relevant Fea- ture Attribution), generates more focused saliency maps by balancing two aspects (specificity and relevance) that capture different desiderata of saliency. The first captures the impact of perturbation the relative expected reward of the action to be explained. The second downweighs irrelevant features that alter the relative expected rewards of actions other than the action to be explained. We compare SARFA with existing approaches on agents trained to play board games (Chess and Go) and Atari games (Breakout, Pong and Space Invaders). We show through illustrative examples (Chess, Atari, Go), human studies (Chess), and automated evaluation methods (Chess) that SARFA generates saliency maps that are more interpretable for humans than existing approaches. For the code release and demo /ssss-araaaaaaa www.raaallencc.....
approaches
com-
videos, see https:/ htt////aaaaiiiiiiiiiiiiii
INTRODUCTION
Deep learning has achieved success in various domains such as image classification (He et al., 2016; Krizhevsky et al., 2012), machine translation (Mikolov et al., 2010), image captioning (Karpathy et al., 2015), and deep Reinforcement Learning (RL) (Mnih et al., 2015; Silver et al., 2017). To explain and interpret the predictions made by these complex, Dolack-box"-ling and perturbation techniques have been introduced for image classification (Simonyan et al., 2013; Zeiler & Fergus, 2014; Fong & Vedaldi, 2017) and deep sequential models (Karpathy et al., 2015). However, interpretability for RL-based agents has received significantly less attention. Interpreting the strategies learned by RL agents can help users better understand the problem that the agent is trained to solve. For instance, interpreting the actions of chess-playing agent in a position could provide useful information about aspects of the position. Interpretation RL agents is also an important step before deploying such models to solve real-world problems.
systems, various gradient
Inspired by the popularity and use of saliency maps to interpret computer vision, number of existing approaches have proposed similar methods for reinforcement learning-based agents. Greydanus et al. (2018) derive saliency maps that explain RL agent behavior by applying a Gaussian blur to different parts of the input image. They generate saliency maps using differences in the value
'These authors contributed equally
Work done during the Adobe MDSR Research Internship Program.

---

Published as a conference paper at ICLR 2020
(a) Original Position
(b) Iyer et al. (2018)
(c) Greydanus et al. (2018)
(d) SARFA
Figure Saliency maps generated by existing approaches
function and policy vector between the original and perturbed state. They achieve promising results on agents trained to play Atari games. Iyer et al. (2018) compute saliency maps using a difference in the action-value (Q(s,0) a)) between the original and perturbed state.
There are two primary limitations to these approaches. The first that they highlight features whose explaining. This is illustrated in Figure 1, which
perturbation affects actions apart from the shows a chess position (it is white's turn). Stockfish' plays the move Bb6 in this position, which traps the black rook (a5) and queen (c7)2. The knight protects the white bishop on a4, and hence the move works. In this position, if we consider the saliency of the white queen (square d1), then it is apparent that the queen is not involved the tactic and hence the saliency should be low. However, perturbing the state (by removing the queen) leads to a state with substantially different values for Q(s,, a) and (s)). Therefore, existing approaches (Greydanus et al., 2018: Iyer al., 2018) mark the queen as salient. The second limitation is that they highlight features that are not relevant to the action to be explained. In Figure Ic, perturbing the state by removing the black pawn on c6 alters the expected reward for actions other than the one to be explained. Therefore, it alters the policy vector and is marked salient. However, the pawn is not relevant to explain the move played in the position (Bb6).
In this work propose SARFA, Specific and Relevant Feature Attribution, perturbation based approach for generating saliency maps for black-boo box agents that builds on two desired properties of cction-focused saliency. The first, specificity, captures the impact of perturbation only the Q-value of the action to be explained. In the above example, this term downweighs features such as the white queen that impact the expected reward of all actions equally. The second, relevance, downweighs irrelevant features that alter the expected rewards of actions other than the action to be explained. It removes features such as the black pawn c6 that increase the expected reward of other actions (in this case, Bb4). By combining these aspects, we generate a saliency map that highlights features of the input state that are relevant for the action to explained. Figure illustrates how the saliency map generated by SARFA only highlights pieces relevant to the move, unlike existing approaches.
We use our approach, SARFA to explain the actions taken by agents for board games (Chess and Go), and for Atari games (Breakout, Pong and Space Invaders). Using number of illustrative examples, we show that SARFA obtains more focused and accurate interpretations for all of these setups when compared to Greydanus al. (2018) and Iyer et al. (2018). We also demonstrate that SARFA is more effective identifying important pieces chess puzzles, and further, in aiding skilled chess players to solve chess puzzles (improves accuracy of solving them by nearly 25% and reduces the time taken by 31% % over existing approaches).
SPECIFIC AND RELEVANT FEATURE ATTRIBUTION (SARFA)
We are given an agent M, operating on a state space S, with the set actions A,, for s E S, and a Q-value function denoted as Q(s, a) for E S, a E A,. Following greedy policy, let the action that was selected by the agent at state be à, i.e. = arg max, Q(s, a). The states are parameterized in terms of state-features F. For instance, in board game such chess, the features the 64 squares.
https:: /ssssskfsscchsss.rrr..
We follow the coordinate naming convention where columns are 'a-h' (left-right), rows '8-1' (top-bottom) and pieces are labeled using the first letter of its name in upper case (e.g. 'B' denotes the bishop). A move consists of the piece and the position moves to, e.g. 'Bh6' indicates that the hishop moves position 'b6'.
2

---

Published as a conference paper at ICLR 2020
For Atari games, the features are pixels. We are interested in identifying which features of the state are important for the agent in taking action å We assume that the agent is in the exploitation phase and therefore plays the action with the highest expected reward. This feature importance is described by an importance-score or saliency for each feature f. denoted by S, where S(.) E (0, 1) denotes the saliency of the f feature of for the agent taking action á. A higher value indicates that the f feature of's is more important for the agent when taking action a.
Perturbation Saliency Maps The general outline of perturbation based saliency approaches is as follows. For each feature f. first perturb & to get s'. For instance, in chess, we can perturb the board position by removing the piece in the f square. In Atari, Greydanus et al. (2018) perturb the input image by adding Gaussian blur centered on the f pixel. Second, query M to get Q(s', a) Va E A, AgA We take the intersection of A and A. to represent the case where some actions may be legal in but not in s' and vice versa. For instance, when we remove a piece in chess, actions that were legal earlier may not be legal anymore. In the rest of this section, when we use "all actions" we mean all actions that are legal in both the states and s'. Finally, compute S[ʃ] based on how different Q(s, a) and Q(s', a)) are, i.e. intuitively, S[ʃ] should be higher if Q(s', a) is significantly different from (s, a). Greydanus et al. (2018) compute the saliency map using S1 2 IT, Tel 12 and
[2[ = (s) V()')), while Iyer et al. (2018) use Sif] (s, a) Qs',, a). In this work, we will propose an alternative approach to compute S[f].
Properties We define two desired properties of an accurate saliency map for policy-based agents:
1. Specificity: Saliency S[f] should focus on the effect of the perturbation specifically on the action being explained, à, i.e. it should be high if perturbing the f feature of the state reduces the relative expected reward of the selected action. Stated another way, S[J] should be high Q(x, a) - Q(s', â) is substantially higher than Q(s, a) Q(s', a), + à. For instance, in figure 1, removing pieces such as the white queen impact all actions uniformly (Q(s, a) - Q(s', a) is roughly equal for all actions). Therefore, such pieces should not be salient for explaining å. On the other hand, removing pieces such as the white knight on a4 specifically impacts the move (à BBb)) we are trying to explain (Q(s, Bb6) Q(s', Bb6) > Q(s,a) Q(s', a) for other actions a). Therefore, such pieces should be salient for à.
2. Relevance: Since the Q-values represent the expected returns, two states and s' can have substantially different Q-values for all actions, i.c. may be higher for s' for all actions if s' is better state. Saliency map for a specific action in s should thus ignore such differences, i.e. s' should contribute to the saliency only if its effects are relevant to à. In other words, S[f] should low if perturbing the I feature of the state alters the expected rewards of actions other than a For instance, in Figure 1, removing the black pawn on c6 increases the expected reward of other actions (in this case, Bb4). However, it does not effect the expected reward of the action to be explained (Bb6). Therefore, the pawn is not salient for explaining the move. In general, such features that are irrelevant to should not be salient.
Existing approaches to saliency maps do not capture these properties in how they compute the saliency, Both the saliency approaches used in Greydanus et al. (2018), i.e. [ = 1 (V(s) - V(s'))² and S21/] = 1/ - .. I, are not focusing on the action-specific effects since they aggregate the change over all actions. Although the saliency computation in Iyer et al. (2018) is somewhat more specific to the action, i.c. S[f] = (ss a) - Q(s', â), it is ignoring whether the effects on Q are relevant only to à, or effect all the other actions as well. This is illustrated in Figure 1.
Identifying Specific Changes To focus on the effect of the change on the action, we are interested in whether the relative returns of à change with the perturbation. Using (ss, â) directly, in Iyer et al. (2018), does not capture the relative changes to Q(s, a) for other actions. To support specificity, we use the softmax over Q-values to normalize the values (as is also used in softmax action selection): exp(pp ((s((((ss) a)) P(s,a) (18,))==. exx((((((ssa)
(1)
(/s,aa) a), the difference in the relative expected reward of the action to be explained between the original and the perturbed state. A high value of Ap thus implies that the feature is important for the specific choice of action à by the agent, while low value indicates that the effect is not specific to the action.
and compute Ap= P(s, -(s's
3

---

Published as a conference paper at ICLR 2020
Identifying Relevant Changes Apart from focusing the change in ((s, a), also want to ensure that the perturbation leads to minimal effect on the relative expected returns for other actions. To capture this intuition, will compute the relative returns of all other actions, and compute saliency in proportion to their similarity. Specifically, we normalize the Q-values using a softmax apart from the selected action á.
())(Q(sa)) ( a) En' exp(Q(s,a'))
Ya + à
We use the KL-Divergence DKL = Prem as),))) (s to measure the difference between Prem a) and Prem (8, a). A high DxL. indicates that the relative expected reward of taking some actions (other than the original action) changes significantly between and s'. In other words, a high DKL indicates that the effect of the feature spread over other actions, i.e. the feature may be relevant for the selected action â.
Computing the SARFA Saliency To compute salience S[ʃ], we need to combine Ap and DKL- L- DEL is high, S[f] should be low, regardless of whether Ap is high; the perturbation is affecting many other actions. Conversely, when DK is low, S[ʃ] should depend on Ap. To be able to compare these properties a similar scale, we define normalized measure of distribution similarity K using DEL:
1
K
(3)
1+ DKLL
As D DKL goes from 0 to oo, K goes from to Thus, S[f] should be low if either Ap low or K is low. Harmonic mean provides this desired effect in robust, smooth manner, and therefore we define S[ʃ] to the harmonic mean of Ap and
2K Ap K + Ap
S(A]
(4)
Equation 4 captures our desired properties of saliency maps. If perturbing the f feature affects the expected rewards of all actions uniformly, then Ap is low and subsequently S[f] is low. This low value of Ap captures the property of specificity defined above. If perturbing the f feature of the state affects the rewards of some actions other than the action to be explained, then D&L is high, K low, and {(f] | is low. This low value of K captures the property of relevance defined above.
RESULTS
To show that SARFA produces more meaningful saliency maps than existing approaches, we use sample positions from Chess, Atari (Breakout, Pong and Space Invaders) and Go (Section 3.1). To show that SARFA generates saliency maps that provide useful information humans, we conduct human studies on problem-solving for chess puzzles (Section 3.2). To automatically compare the saliency maps generated by different perturbation based approaches, we introduce a Chess saliency dataset (Section 3.3). We use the dataset to show how SARFA is better than existing approaches in identifying chess pieces that humans deem relevant in several positions. In Section 3.4, we show how SARFA can be used to understand common tactical ideas in chess by interpreting the action of trained agent.
To show that SARFA works for black-box agents, regardless of whether they are trained using reinforcement learning, we use a variety of agents. We only assume access to the agent's Q(s, a) function for all experiments. For experiments on chess, we use the Stockfish agent?. For experiments on Go, we use the pre-trained MiniGo RL agent'. For experiments on Atari agents and for generating saliency maps for Greydanus et al. (2018), generating saliency maps using Iyer et al. (2018), we use our own implementation". All of our code and more detailed results are available in our Github repository: https:/ http:://niiaaaauri.....m github. io/ //ssssffaassss saarfa-salienyy..
use their code and pre-trained RL agents For
htttps:/ lsstccfishchess.orgg
'https:/ githtb..chommmm hemmm//ttsroflo//nnnn minigo
htttps: gttthbb.c/mtuubccm///o wraoo//rgeydauusssuuulllll atari
4

---

Chess Figure shows sample positions where SARFA produces more meaningful saliency maps than existing approaches for chess-playing agent (Stockfish). Greydanus al. (2018) and Iyer et al. (2018) generate saliency maps that highlight pieces that are not relevant to the move played by the agent. This is because they differences in (s, a), V(s) the the Lg norm of the policy vector between the original and perturbed state to calculate the saliency maps. Therefore, pieces such as the Published as a conference paper at ICLR 2020
105 3
1 005
3
074
f 074
5
SARFA
Greydanus et al. (2018)
(c) SARFA
(d) Greydanus et al. (2018)
Figure 2: Comparing saliency of RL agents trained to play Breakout
SARFA
Greydanus et al. (2018)
SARFA
(d) Greydanus et al. (2018)
Figure 3: Comparing saliency of RL agents trained to play Atari Pong
3.1
ILLUSTRATIVE EXAMPLES
In this section, we provide examples of generated saliency maps highlight the qualitative differences between SARFA that is action-focused and existing approaches that are not.
white queen that affect the value estimate of the state are marked salient. In contrast, the saliency map generated by SARFA only highlights pieces relevant to the move.
Atari To show that SARFA generates saliency maps that are more focused than those generated by Greydanus et al. (2018), we compare the approaches on three Atari games: Breakout, Pong, and Space Invaders. Figures 2, 3, and shows the results. SARFA highlights regions of the input image more precisely, while the Greydanus al. (2018) approach highlights several regions of the input image that are not relevant to explain the action taken by the agent.
Go Figure 5 shows board position in Go. It is black's turn. The four white stones threaten the three black stones that are in one row at the top left corner of the board. To save those three black stones, black looks at the three white stones that are directly below the three black ones. Due to another white stone below the three white stones, the continuous row of three white stones cannot be captured easily. Therefore black moves place black stone below that single white stone in an attempt to start capturing the four white stones. It takes the next few turns to surround the structure of four white stones with black ones, thereby saving its pieces. The method described in Greydanus et al. (2018) generates a saliency map that highlights almost all the pieces the board. Therefore, reveals little about the pieces that the agent thinks are important. On the other hand, the map produced by Iyer al. (2018) highlights only few pieces. The saliency map generated by SARFA correctly highlights the structure four white stones and the black stones already present around them that may be involved in capturing them.
5

---

Published as a conference paper at ICLR 2020
0235 DOOM
0235 DOOD
A
SARFA
Greydanus et al. (2018)
(c) SARFA
(d) Greydanus et al. (2018)
Figure 4: Comparing saliency of RL agents trained to play Space Invaders
(a) Original Position
(b) SARFA
Tyer al. (2018) (d) Greydanus al. (2018)
Figure 5: Comparing saliency maps generated by different approaches for the MiniGo agent
3.2 HUMAN STUDIES: CHESS
To show that SARFA generates saliency maps that provide useful information to humans, we conduct human studies problem-solving for chess puzzles. We show forty chess players (ELO 1600-2000) fifteen chess puzzles from https:/ / http///wwww...sc/t///////////sst www.s.ceesssooooeeoee (average difficulty ELO 1800). For each puzzle, we show either the puzzle without saliency map, the puzzle with a saliency map generated by SARFA, Greydanus et al. (2018), or Iyer et al. (2018). The player is then asked to solve the puzzle. We measure the accuracy (number of puzzles correctly solved) and the average time taken solve the puzzle, shown in Table 1. The saliency maps generated by SARFA are more helpful for humans when solving puzzles than those generated by other approaches. We observed that the saliency maps generated by Greydanus et al. (2018) often confuse humans, because they highlight several pieces unrelated to the tactic. The maps generated by Iyer et al. (2018) highlight few pieces and therefore marginally better than showing saliency maps for solving puzzles.
3.3 CHESS SALIENCY DATASET
To automatically compare the saliency maps generated by different perturbation- approaches, we introduce Chess saliency dataset. The dataset consists of 100 chess puzzles?. Each puzzle has single correct move. For each puzzle, we ask three human experts (ELO > 2200) mark the pieces that are important for playing the correct move. We take majority vote of the three experts to obtain a pieces that are important for the move played available in our Github repository We use this dataset to compare SARFA to existing approaches
position. The complete dataset
"https://nikaashhp wtt../aaaaappiiiigggbu.. /saara-saliencyy
Table 1: Results of Human Studies for solving chess puzzles
No Saliency
SARFA
Greydanus et al.
Iyer et al.
Accuracy
56.67%
72.41 %
40.84%
24.60% 102.26 sec
Average time taken
77.53 sec
67.02 see
70.95 sec

---

Published as a conference paper at ICLR 2020
1.0
1.0
0.8
Thue Postive fate
Thue Positive Rato
0.6
0.6
SARFA, (ares 0.88)
Only K (area 0.50)
Only
0.74)
0.2
SARFA (area 0.88)
0.2
Average (area 0.74)
(2038) (area
Geometricc Mean 0.741
Creydanus et al.(2018) (area
0.721
MinjDellt K] (area 0.73)
0.0
0.2
0.4
0.8
1.0
0.0
0.2
0.4
0.8
1.0
False Positive Rate
Falsz Pasitive Rate
(a) ROC curves for different approaches
(b) ROC curves for Ablation Studies
Figure 6: ROC curves comparing approaches on the chess saliency dataset
(Greydanus et al., 2018; Iyer et al., 2018). Each approach generates list of squares and score that indicates how salient the piece on the square is for particular move. We scale the scores between 0 and to generate ROC curves. Figure 6a shows the results. SARFA generates saliency maps that are better than existing approaches at identifying chess pieces that humans deem relevant in certain positions.
To evaluate the relative importance of the two components in saliency computation (S[f); Equation 4), we compute saliency maps and ROC curves using each component individually, i.e. S[ʃ] Ap S[f] K, and compare harmonic mean other ways combine them, i.e. using the average, geometric mean, and minimum of Ap and K. Figure 6b shows the results. Combination of the two properties via harmonic mean leads approaches.
accurate saliency maps than alternative
3.4 EXPLAINING TACTICAL MOTIFS IN CHESS
In this section, show how SARFA can be used to understand common tactical ideas in chess by interpreting the action a trained agent. Figure 7 illustrates common tactical positions in chess. The corresponding saliency maps are generated by interpreting the moves played by the Stockfish agent in these positions.
In Figure 7a, is white move. The surprising Rook d6 is the move played by Stockfish. Figure 7 shows the saliency map generated by SARFA. The map illustrates the key idea in the position. Once black's rook recaptures white's rook, white's bishop pins it to the black king. Therefore, white can increase the number of attackers on the rook. The additional attacker is the pawn on e4 highlighted by the saliency map.
In Figure 7b, is white to move. Stockfish plays Queen h7. A queen sacrifice! Figure 7e shows the saliency map. The map highlights the white rook and bishop. along with the queen. The key idea is that once black captures the queen with his king (a forced move), then the white rook moves to h5 with checkmate. This checkmate is possible because the white bishop guards the important escape square g6. The saliency map highlights both pieces.
In Figure 7c, it is black to move. Stockfish plays the sacrifice rook x d4. The saliency map in Figure 7f illustrates several key aspects of the position. The black queen and light-colored bishop are threatening mate on g2. The white queen protects g2. The white rook on a5 unguarded. Therefore, once white recaptures the sacrificed rook with the pawn c3, black rook and queen with the move bishop to b4. The idea that the white queen is "overworked" or "overloadedd" on d2, having to guard both the g2-pawn and the Rook against black's double attack.
attack both the white
7

---

Published as a conference paper at ICLR 2020
WTY
1
(a) Pin
(b) Mate in 2
(c) Overloading
(d) Salieney
(e) Saliency
(f) Saliency
Figure 7: Saliency maps generated by SARFA that demonstrate common tactical motifs in chess
3.5 ROBUSTNESS TO PERTURBATIONS
We are also interested in evaluating the robustness of the generated saliency maps: is the saliency different if non-salient changes are made to the state? To evaluate the robustness of SARFA, we perform two irrelevant perturbations to the positions in the chess saliency dataset. First, we pick random piece amongst the ones labeled non-salient by /uman experts in particular position, and remove it from the board. We repeat this for each puzzle in the dataset to generate a new perturbed saliency dataset. Second, remove random piece amongst ones labeled non-salient by SARFA for each puzzle, creating another perturbed saliency dataset. In order to evaluate the effect of non-salient perturbations our generated saliency maps, we compute the AUC values for the generated saliency maps, as above, for these perturbed datasets. Since we remove non-salient pieces, we expect the saliency maps and subsequently AUC value be similar to the value the original dataset. For both these perturbations, we get an AUC value of 0.92, same as the value on the non non-perturbed dataset, confirming the robustness of our saliency maps these non-relevant perturbations.
RELATED WORK
Since understanding RL agents is important both for deploying RL agents to the real world and for gaining insights about the tasks, a number different kinds of interpretations have been introduced.
number of approaches generate natural language explanations to explain RL agents (Dodson et al., 2011; Elizalde et al., 2008; Khan al., 2009). They assume access
MDP model and that the policies map from interpretable, high-level state features to actions. More recently, Hayes & Shah (2017) analyze execution traces of an agent to extract explanations. A shortcoming of this approach is that it explains policies in terms of humans. This often not practical for board games or Atari games where the agents learn from raw board/visual input. Zahavy et al. (2016) apply t-SNE (Maaten & Hinton, 2008) on the last layer of a deep Q-network (DQN) to cluster states of behavior of the agent. They use Semi- Gemi-Aggregated Markov Decision Processes (SAMDPs) to approximate the black box RL policies. They use the more interpretable SAMDPs to gain insight into the agent's 's policy. An issue with the explanations that they emphasize t-SNE clusters that difficult to understand for non-experts. To build user trust and increase adoption, it is important that the insight into agent behavior should in a form that is interpretable the untrained eye and obtained from the original policy instead of distilled one.
state representations that are semantically meaningful
8

---

Published as a conference paper at ICLR 2020
Most relevant to SARFA are the visual interpretable explanations of deep networks using saliency maps. Methods for computing saliency can be classified broadly into two categories.
Gradient-based methods identify input features that are most salient to the trained DNN by using the gradient to estimate their influence on the output. Simonyan et al. (2013) use gradient magnitude heatmaps, which was expanded upon by more sophisticated methods to address their shortcoming, such as guided backpropagation (Springenberg et al., 2014), excitation backpropagation (Zhang et al., 2018), DeepLIFT (Shrikumar et al., 2017). GradCAM (Selvaraju et al., 2017), and +radCAM++ (Chattopadhay et al., 2018). Integrate gradients (Sundararajan et al., 2017) provide two axioms to further define the shortcomings of these approaches: sensitivity (relative a baseline) and implemen- tation invariance, and use them to derive an approach. Nonetheless, all gradient- -based approaches still depend on the shape in the immediate neighborhood of a few points, and conceptually, use perturbations that lack physical meaning, making them difficult to use and vulnerable to adversarial attacks in form of imperceivable noise (Ghorbani et al., 2019). Further, they are not applicable to scenarios with black-box access to the agent, and even with white-box access to model internals, they are not applicable when agents are not fully differentiable, such as Stockfish for chess.
(Springe
We are more interested in perturbation- bused methods for black-box agents: methods that compute the importance of an input feature by removing, altering, or masking the feature in domain-aware aware manner and observing the change in output. It is important to choose a perturbation that removes information without introducing any new information. As a simple example, Fong & Vedaldi (2017) consider classifier that predicts 'True' if certain input image contains bird and "False" otherwise. Removing information from the part of the image which contains the bird should change the classifier's prediction, whereas removing information from other areas should not. Several kinds of perturbations have been explored, e.g. Zeiler & Fergus (2014); Ribeiro et al. (2016) remove information by replacing a part of the input with gray square. Note that these approaches are implementation invariant by definition, and are sensitive with respect to the perturbation function.
Existing perturbation-base approaches for RL (Greydanus et al., 2018: Iyer et al., 2018), however, by focusing on the complete Q (or V), tend to produce saliency maps that are not specific to the action of interest. SARFA addresses this by measuring the impact only on the action being selected, resulting in more focused and useful saliency maps, as we show in our experiments.
5 LIMITATIONS AND FUTURE WORK
Saliency maps focus on visualizing the dependence between the input and output to the model, essentially identifying the situation-spccffe explanation for the decision. Although such local explanations have applications in understanding, debugging, and developing trust with machine learning systems, they do not provide any direct insights regarding the general behavior of the model, or guarantee that the explanation is applicable to a different scenario. Attempts to provide more general understanding of the model include carefully selecting prototype explanations to show to the user (van der Linden et al., 2019) and crafting explanations that are precise and actionable (Ribeiro et al., 2018). We will explore such ideas for the RL setting in future, to provide explanations that accurately characterize the behavior of the policy function, in precise, testable, and intuive manner.
There are a number of limitations of SARFA to generating saliency maps in our current implemen- tation. First, we perturb the state by removing information (removing pieces in Chess/Go, blurring pixels in Atari). Therefore, SARFA cannot highlight the importance of absence of certain attributes, i.e. saliency of certain positions being empty. In games such Chess and Go, an empty square or file (collection of empty squares) can often be important for a particular move. Future work will explore perturbation functions that add information functions, along with SARFA, can be used to calculate the importance of empty squares. Second, it possible that perturbations may explore states that lie outside the manifold, i.e. they lead to invalid states. For example, unless explicitly prohibited like we do, SARFA will compute the saliency of the king pieces by removing them, which is not allowed in the game, or remove the paddle from Pong In future, we will explore strategies that take the valid state space into account when computing the saliency. Last we estimate the saliency of each feature independently, ignoring feature dependencies and correlations, which may lead to incorrect saliency maps. We will investigate approaches that perturb multiple features to quantify the importance of each feature (Ribeiro et al., 2016; Lundberg & Lee, 2017), and combine them with SARFA to explaining black-box policy-based agents.
the state (e.g. adding pieces in Chess/Go). Such
9

---

Published as a conference paper at ICLR 2020
6 CONCLUSION
We presented a verturbation-basee approach that generates more focused saliency maps than existing approaches by balancing two aspects (specificity and relevance) that capture different desired char- acteristics of saliency. We showed through illustrative examples (Chess, Atari, Go), human studies (Chess), and automated evaluation methods (Chess) that SARFA generates saliency maps that are more interpretable for humans than existing approaches. The results of our technique show that saliency can provide meaningful insights into a black-box RL agent's behavior. For the code release and demo videos, see https://nikaashpuri. wtw.://aaaspuriggghuu...... //sssa-fssaaaaaa sarfa-saliency/.
ACKNOWLEDGEMENTS
We would like to thank the anonymous reviewers for their helpful comments and suggestions. This work is supported in part by the NSF Award No. IIS-1756023 and in part by a gift from the Allen Institute of Artificial Intelligence (A12).
REFERENCES
Aditya Chattopadhay, Anirban Sarkar, Prantik Howlader, and Vineeth Balasubramanian. Grad-
cam++: Generalized gradient-based visual explanations for deep convolutional networks. In 2018 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. IEEE, 2018.
Thomas Dodson, Nicholas Mattei, and Judy Goldsmith. A natural language argumentation inter-
face for explanation generation in markov decision processes. International Conference on Algorithmic DecisionTheory, pp. 42-55. Springer, 2011.
Francisco Elizalde, Enrique Suear, Manuel Luque, Diez, and Alberto Reyes. Policy explanation in factored markov decision processes. In Proceedings of the 4th European Workshop on Probabilistic Gruphical Models (PGM 2008), pp. 97-104, 2008
Ruth C Fong and Andrea Vedaldi. Interpretable explanations of black boxes by meaningful perturba- tion. In Proceedings of the IEEE International Conference on Computer Vision, pp. 3429-3437, 2017.
Amirata Ghorbani, Abubakar Abid, and James Zou. Interpretation of neural networks is fragile. Proceedings of the AAAI Conference on Artificial Intelligence, 33:3681 I 13888, Jul 2019. ISSN 2159-5399. doi: 10. 10.1609/aaai 333101.3311000110133303333333301 URL http:/ http://dxx.ooi.org... 11.190000161110 10..1669//aai.v//2./ aaai.v33i01. 33013681.
Samuel Greydanus, Anurag Koul, Jonathan Dodge, and Alan Fern. Visualizing and understanding atari agents. In International Conference on Machine Learning, pp. 1787-1796, 2018.
Bradley Hayes and Julie A Shah. Improving robot controller transparency through autonomous policy explanation. 2017 12th ACM/EEE International Conference on Human-Ron Robot Interaction (HRI, pp. 302-313. IEEE, 2017.
Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and patlern recognition, pp. 770-778, 2016.
Rahul Iyer, Yuezhang Li, Huao Li, Michael Lewis, Ramitha Sundar, and Katia P. Sycara. Transparency and explanation deep reinforcement learning neural networks. CoRR, abs/ 819.888900600 2018. abs 1809.06061.
URL
Andrej Karpathy, Justin Johnson, and Li Fei-Fei. Visualizing and understanding recurrent networks. arXiv preprint arXiv:1506.000/ 02078, 2015.
Omar Zia Khan, Pascal Poupart, and James P Black. Minimal sufficient explanations for factored markov decision processes. Nineteenth International Conference on Automated Planning and Scheduling, 2009.
10

---

Published as a conference paper at ICLR 2020
Alex Krizhevsky, llya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolu- tional neural networks. In Advances in neural information processing systems, pp. 1097-1105, 2012.
Scott M Lundberg and Su-In Lee. unified approach to interpreting model predictions. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Neural Information Processing Systems (NIPS), pp. 4765-4774. Curran Associates, Inc., 2017.
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 25N0))2577--2655 2008
Tomáš Mikolov, Martin Karafiát, Lukášs Burget, Jan Černocký, and Sanjeev Khudanpur. Recurrent neural network based language model. Eleventh annual conference of the international speech communication association, 2010.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 5 518(7540):529, 2015.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. International conference on machine learning, pp. 1928-1937, 2016.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. Why should trust you?: Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pp. 1135-1144 144. ACM, 2016.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. Anchors: High-precision model-agnostic explanations. In AAAI Conference on Artificial Intelligence (AAAI), 2018.
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based local- ization. In Proceedings of the IEEE International Conference on Computer Vision, pp. 618 188626 2017.
Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. Learning important features through propagating activation differences. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp.
JMLR. org, 2017.
David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Mare Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484, 2016.
David Silver, Julian Schritwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. Nature, 550(7676):354, 2017.
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. arXiv preprint arXiv: 2013.
Jost Tobias Springenberg. Alexey Dosovitskiy, Thomas Brox, and Martin Riedmiller. Striving for simplicity: The all convolutional net. arXiv preprint arXiv:1412.6800 2014.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 3319-3328. JMLR. org, 2017.
Ilse van der Linden, Hinda Haned, and Evangelos Kanoulas. Global aggregations of local explanations for black box models. In SIGIR Workshop on FACTS-IR: Fairness, Accountability, Confidentiality, Transparency, and Safety, 2019.
Tom Zahavy, Nir Ben- Zrihem, and Shie Mannor. Graying the black box: Understanding dqns. In International Conference Machine Learning, pp. 1899-1908, 2016.

---

Published as a conference paper at ICLR 2020
Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In European conference on computer vision, pp. 818-833. Springer, 2014.
Jianming Zhang, Sarah Adel Bargal, Zhe Lin, Jonathan Brandt, Xiaohui Shen, and Stan Sclaroff. Top-down neural attention by excitation backprop. International Journal of Computer Vision, 126 (10):1084-102,, 2018.
12

---

Published as a conference paper at ICLR 2020
Wi
(a) Move Qd4
(b) Move Rfl
(c) Move Bb5
Figure 8: Saliency Maps generated by SARFA for the top 3 moves in a chess position
A EXPERIMENTAL DETAILS
For experiments chess, we use the Stockfish 10 agent: https://stockfishches. org/ Stockfish works using hhuristic-basse measure for each state along with Alpha-Beta Pruning to search over the state-space.
For experiments on Go, we use the pre-trained MiniGo RL agent: https://github com/ com/tensorflow/ minigo. This agent trained using the AlphaGo Algorithm (Silver al., 2016). It also adds features and architecture changes from the AlphaZero Algorithm Silver et al. (2017).
For experiments Atari agents and for generating saliency maps for Greydanus et al. (2018), use their code and pre-trained RL agents available at https:/ hitps://gitt /github..cc... wtt.gti..oom//oooeeeooooooooooot "ggeeddanu//iiiiee visuulizz atari. These agents trained using the Asynchronous Advantage Actor-Crice Algorithm (A3C) (Mnih et al., 2016).
For generating saliency maps using Iyer et al. (2018), we our implementation, All of code and more detailed results are available in our Github repository: sarfa-saliency/
For chess and Go, we perturb the board position by removing one piece at a time. We do not remove piece if the resulting position is illegal. For instance, in chess, we do remove the king. For Atari, we use the perturbation technique described in Greydanus et al. (2018) 88. The technique perturbs the input image by adding Gaussian blur localized around pixel. The blur constructed using the Hadamard product to interpolate between the original input image and Gaussian blur. The saliency maps for Atari agents have been computed on the frames provided by Greydanus al. (2018) their code repository.
The puzzles for conducting the Chess human studies, creating the Chess Saliency Dataset, and providing illustrative examples have been taken from Lichess: https://databae biaataase.lichesss, org/ The puzzles for illustrative examples on Go have been taken from OnlineGo: https:/ haaps://onlinegooceo puzzles.
B SALIENCY MAPS FOR TOP 3 MOVES
Figure shows the saliency maps generated by SARFA for the top moves in chess position. The maps highlight the different pieces that are salient for each move. For instance, Figure 8a shows that for the move Qd4, the pawn on g7 is important. This is because the queen move protects the pawn. For the saliency maps in Figures 8b and 8c, the pawn on g7 is not highlighted.
C SALIENCY MAPS FOR LEELAZERO
To show that SARFA generates meaningful saliency maps in Chess for RL agents, we interpret the LeelaZero Deep RL agent http://giitub.ccc aamelllllzzeee aaro/leela-zene Figure 9 shows the results. As discussed in Section the saliency maps generated by (Greydanus et al., 2018; Iyer et al., 2018)
13

---

Published as a conference paper at ICLR 2020
52
(a) Original Position
(b) Iyer et al. (2018)
(c) Greydanus et al. (2018)
(d) SARFA
&
A
2
2
2
(c) Original Position
(1) Iyer al. (2018)
Greydanus et al. (2018)
(h) SARFA
Figure 9: Saliency maps generated by different approaches for the LeelaZero Deep Reinforcement Learning Agent
highlight several pieces that not relevant to the move being explained. On the other hand, the saliency maps generated by SARFA highlight the pieces relevant to the move.
14


Reviews:
[{"review_id": "rJx9cWb3YS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes an algorithm for explaining the move of the agents trained by reinforcement learning (RL) by generating a saliency map.\nThe authors proposed two desired properties for the saliency map, specificity and relevance.\nThe authors then pointed out that prior studies failed to capture one of the two properties.\nTo combine the two components into a single saliency map, the authors proposed using the harmonic mean.\n\nThe experimental results demonstrated that the proposed saliency map successfully focused only on important parts while the other method tend to highlight some irrelevant parts also.\nThe authors also did a great job for evaluating the goodness of the saliency maps, by preparing a human annotated chess puzzle dataset.\n\nI think the paper is well-written, and the basic idea look reasonable and promising.\nThe experimental evaluations are well designed and the results look convincing.\nSaliency map for RL is not yet mature, and I expect to see further improvements (especially, more theoretically principled ones) follow this study.\n\n\n### Updated after author response ###\nThe response from the authors seem to be reasonable to me. I therefore keep my score.", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "rJlbK2tLOB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Interpreting the policies of RL agents is an important consideration if we would like to actually deploy them and understand their behaviours. Prior works have applied saliency methods to DRL agents, but the authors note that there are two properties - specificity and relevance - that these methods do not take into account, and therefore result in misleading saliency maps. The authors define (and provide examples) of these properties, propose simple ways to calculate these (and like prior methods, relying on perturbations and therefore applicable to almost black box agents), and combine them neatly using the harmonic mean to provide a new way to calculate saliency maps for agents with discrete action spaces. While the improvements on Atari are hard to quantify, the results on chess and go are more interpretable and hence more convincing. The study using saliency maps to aid human chess players is rather neat, and again adds evidence towards the usefulness of this technique. Finally, the chess saliency dataset is an exciting contribution that can actually be used to quantify saliency methods for DRL agents. The proposed method is relatively simple, but is well-motivated and demonstratively (quantitatively, which is great work for general interpretability methods) better than prior methods, and in addition the authors introduce a new dataset + quantitative measure for saliency methods, so I would give this paper an accept.\n\nAlthough the authors motivate their choice of perturbation-based saliency methods as opposed to gradient-based methods, they should expand their review of the latter. As the technique the authors introduced is very general, it would be useful to know how it compares to the current state of research in terms of identifying properties that attribution methods should meet - a good example of this is integrated gradients (Sundararajan et al., 2017), which similarly identify \"sensitivity\" and \"implementation invariance\" as \"axioms\" that their method satisfies.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "Skx402YnYB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "[score raised from weak reject to weak accept after rebuttal - on a more fine-grained scale, I would rate this paper now an accept (7), but not a strong accept (8), however since this year's scale is quite coarse, I'll stick with a score of 6]\n\nSummary\nThe paper proposes a new perturbation-based measure for computing input-saliency maps for deep RL agents. As reported in a large body of literature before, such saliency maps are supposed to help “explain” why a deep RL system picked a certain action. The measure proposed in the paper aims at combining two aspects: specificity and relevance, which should ensure that the saliency map highlights inputs that are relevant for a particular action to be explained (specificity), and this particular action only (relevance). The paper shows illustrative examples of the proposed approach and two previously proposed alternatives on Chess, Go and three Atari games. Additionally the paper evaluates the method and the two previous alternatives on two interesting chess-tasks: chess-puzzles where human players were shown to be able to solve puzzles faster and with higher accuracy when given the proposed saliency map in addition, and evaluation against a curated dataset of human-expert saliency maps for 100 chess puzzles. \n\nContributions\ni) Novel, perturbation-based saliency measure composed of two parts. Main idea of specificity is (similar to Iyer et al. 2018) to use State-Action value function (Q-function) with a specific action, instead of State-Value function only. Main idea of relevance is to “normalize” by taking into account change in Q-value for all other actions as well. Both parts are combined in a heuristic fashion.\n\nii) More objective/reproducible assessment of how saliency maps produced by different methods overlap with human judgement of saliency of pieces in chess. To this end: two experiments with human chess players/experts (puzzle solving, and expert-designed saliency maps).\n\nQuality, Clarity, Novelty, Impact\nThe paper is well written and most sections are easily understandable, though for some parts it might help if the reader is quite familiar with Chess/Go. The proposed saliency measure seems to address some shortcomings of previously proposed measures - my main criticism is that the construction of the measure seems rather ad-hoc and heuristic. It would have been great to formally define specificity and relevance (e.g. in an information-theoretic framework as Sparse Relevant Information) and then derive a suitable measure that is shown to satisfy/approximate the formal definitions. At least, there is one ablation study that justifies parts of the heuristic construction to some degree. \nThe proposed measure seems to do reasonably well on board-game domains, in particular chess. However it might also be the case that the measure does particularly well for generating saliency maps for Stockfish (which is the agent that happens to be evaluated in the chess domain), which might be quite different from previously reported methods that have been designed for deep neural network RL agents. The illustrative examples on Atari and Go do not allow for a statistically significant judgement of the quality of the proposed method.\n\nOn a conceptual level, a bigger issue (of many saliency methods in general, but the criticism also applies to the paper) is that the “explanations” drawn from saliency maps are rarely evaluated rigorously. The paper makes a nice attempt by trying to establish some “ground-truth” saliency in chess using humans to increase the degree of objectivity, which I greatly appreciate. However, it remains unclear whether explanations that happen to coincide with human notions of saliency really are of higher quality for assessing how an artificial system makes its decisions. The main goal of explainability/interpretability methods must be to come up with testable hypotheses that tell us something about how the artificial system makes its decisions in novel/unseen situations. The goal is not to explain a decision mechanistically after the fact (which is trivial, given a deterministic, differentiable feed-forward system), but to come up with non-trivial explanations that extrapolate/generalize. Specificity and Relevance are probably important ingredients of such explanations, but I think it’s important to formalize them properly first. Currently I am in favor of suggesting a major revision of the work, but I am happy to reconsider my decision based on the rebuttal and other reviewers’ assessment. Having said that, I do want to reiterate that I think it is great that the authors included some ablation studies and measures of “usefulness” of the saliency method.\n\nImprovements / Major comments\ni) Formally define specificity and relevance (e.g. as sparsity and compression?). Ideally derive a saliency measure based on these formal definitions.\n\nii) Show how saliency maps (of the same situation) change when producing explanations for different actions. I assume that currently the illustrative examples only show the action with the highest Q-value, what changes when using e.g. a less likely action?\n\niii) Show that the saliency map stays roughly constant for seemingly irrelevant perturbations. In particular, using the chess-dataset with expert annotations, apply various kinds of perturbations to non-salient pieces (e.g. removing them, swapping them for other pieces and potentially moving them in irrelevant ways) and see whether the AUC stays roughly constant.\n\niv) Apply non-relevant perturbations to the salient pieces. I.e. take the same puzzle/scene and move it across the board, does the saliency-map move in (roughly) the same way.\n\nv) The saliency method might be particularly suited for Stockfish (whose action-value estimates might be strongly influenced by human saliency and chess theory). See whether the method still produces good results for other chess agents (ideally trained without human heuristics or data). If this is hard to do for chess, think about a different application where this is easier.\n\nvi) Add a section that discusses current shortcomings and caveats with the method, and saliency maps for explainability in general.\n\nvii) (Experimental details) For each domain, please explain the perturbations that you used (removing pieces in Chess/Go, blurring pixels in Go, anything else e.g. replacing pieces?). In all experiments, was it always the action with the highest Q-value that was being explained?\n\n\n\nMinor comments\n\na) Table 1: (add error-bars) What is the variance across players? Are the results for the proposed method statistically significant?\n\nb) Chess saliency dataset. Are the expert saliency ratings binary? Why not have multiple degrees of saliency?\n\nc) Would the Greydanus et al. 2018 approach deliver similar results when using a threshold to cut off low-saliency inputs?\n\nd) Why is the saliency map in 3.4 binary (pieces are either salient or not)? How was the binarization threshold chosen? What would the non-binary saliency maps look like?\n\ne) Please provide all experimental details (in the appendix) that are necessary to reproduce the experiments. Referring to a code-repository is not a replacement for describing the methods in detail.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: A new method of calculating saliency maps for deep networks trained through RL (for example to play games) is presented. The method is aimed at explaining why moves were taken by showing which salient features influenced the move, and seems to work well based on experiments with Chess, Go, and several Atari games. 

Reviewer 2 had a number of questions related to the performance of the method under various conditions, and these were answered satisfactorily by the reviewers.

This is a solid paper with good reasoning and results, though perhaps not super novel, as the basic idea of explaining policies with saliency is not new. It should be accepted for poster presentation.

Decision: Accept (Poster)