Machine versus Human Attention in Deep Reinforcement Learning Tasks
Authors: Sihang Guo
Ruohan Zhang
Bo Liu
Yifeng Zhu
Dana Ballard
Mary Hayhoe
Peter Stone
Venue: NeurIPS 2021 Poster (2021)
Keywords: Deep Reinforcement Learning
Interpretability
Attention
Eye Tracking

Abstract: Deep reinforcement learning (RL) algorithms are powerful tools for solving visuomotor decision tasks. However, the trained models are often difficult to interpret, because they are represented as end-to-end deep neural networks.  In this paper, we shed light on the inner workings of such trained models by analyzing the pixels that they attend to during task execution, and comparing them with the pixels attended to by humans executing the same tasks. To this end, we investigate the following two questions that, to the best of our knowledge, have not been previously studied. 1) How similar are the visual representations learned by RL agents and humans when performing the same task? and, 2) How do similarities and differences in these learned representations explain RL agents' performance on these tasks? Specifically, we compare the saliency maps of RL agents against visual attention models of human experts when learning to play Atari games. Further, we analyze how hyperparameters of the deep RL algorithm affect the learned representations and saliency maps of the trained agents. The insights provided have the potential to inform novel algorithms for closing the performance gap between human experts and RL agents.

Machine versus Human Attention in Deep
Reinforcement Learning Tasks
Sihang Guo UT Austin sguo190utexas.co edu
Ruohan Zhang Stanford University zharu@stanf ord. edu
Bo Liu UT Austin bliuêcs utexas.edu
Yifeng Zhu
UT Austin yifeng zhuuutexas edu,
Dana Ballard UT Austin danablutexas.cr edu,
Mary Hayhoe
Peter Stone UT Austin, Sony Al
UT Austin
Sayhoe@utexas.eddu
pstoneêcs uttexas.edi
Abstract
Deep reinforcement learning (RL) algorithms are powerful tools for solving visuo- motor decision tasks. However, the trained models are often difficult interpret, because they are represented as end-to-ena deep neural networks. In this paper, we shed light on the inner workings such trained models by analyzing the pixels that they attend to during task execution, and comparing them with the pixels attended by humans executing the same tasks. To this end, we investigate the following two questions that, to the best of our knowledge, have not been previously studied. 1) How similar are the visual representations learned by RL agents and humans when performing the same task? and, 2) How do similarities and differences in these learned representations explain RL agents' performance on these tasks?
Specifically, we compare the saliency maps of RL agents against visual attention models human experts when learning to play Atari games. Further, we analyze how hyperparameters of the deep RL algorithm affect the learned representations and saliency maps of the trained agents. The insights provided have the potential to inform novel algorithms for closing the performance gap between human experts and RL agents.
Introduction
Researchers have devoted much effort understanding deep neural networks (DNNs). Since DNNs are partially inspired by the biological nervous systems, researchers have compared these neural networks with the human brain and sensory systems by asking two typical types of questions. The first one considers representation: How similar are the visual representations learned by DNNs and humans when performing the same tasks? The second similarities and differences in the learned representations explain DNNs' performance on their tasks?
concerns explainability: How do
The first question motivates seminal line of research that compares representations learned by DNNs with those Iearned by humans. In computer vision, a linear model that uses the feature map activations generated by trained CNN accurately predict neural activities in the early visual cortex, indicating that the two systems have learned similar visual representations [19] 78 79). In language learning, similar connection is found between deep language models and cortical areas [1] 34).
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

However, this type of comparison has just emerged in Decision-making research 44 [3), which motivates us compare DNNs trained for deep reinforcement learning (RL) tasks with human decision-making. We ask: Do deep RI. agents and humans agree on what visual features are important? In other words, do agents pay attention to the same visual features as humans do?
The explainability question is motivated by the Explainable AI (XAI) for deep RL agents [29] 55 Deep RL has achieved many successes, but few of us fully understand these agents. These agents often learn a mapping from raw images to actions end-to-end where it is not clear why particular decision is made. Our work addresses the explainability of these black-box models: Do RL agents make mistakes because they fail to attend to important visual features that matter for making the correct decision? An expert human's 's attention data could serve as a useful reference for identifying important visual features, which has been validated in object recognition tasks [50].
To answer these two questions, we situate our research in the domain of Atari games 6 These games span a variety of dynamics, visual features, reward mechanisms, and difficulty levels for both humans and AIs. Two recent lines of research have made our work possible. On the one hand, human attention dataset on Atari games has been collected using eye trackers [85] and convolution- deconvolution networks have been shown to accurately predict human attention distributions [83] On the other hand, tools have been introduced to generate visual interpretations of RL agents [23] so one can visualize an RL agent's saliency map given an image, which is a topographically arranged map that assigns an importance weight to each image pixel and is often treated as the "attention map" of the RL agent. However, these tools can only provide qualitative results we still need to manually inspect the saliency maps to interpret the performance. A useful comparison standard like human attention had not been considered hence systematic quantification was impossible.
In this work, we present a novel study that compares the RL agent's ' attention with both real and estimated human attention. We analyze how learning and hyperparameters of the RL algorithm affect the learned representations and saliency maps. We analyze failure and unseen states for RL agents and identify potential challenges to achieve human-level performance. We conclude by discussing insights gained for RL researchers in both cognitive science and AI.
2 Related Work
Human vs. machine attention in vision and language tasks. Human experts' gaze is very efficient and accurate for solving vision tasks. The peak angular speed of the human eye during a saccade reaches up to 900 degrees per second [59]. This allows humans to move their foveae to the right place at the right time to attend to important features [17. Therefore, human expert's gaze serves good standard in many vision-related tasks for evaluating machine attention, as a learning target for training machine attention
84] This approach widely used in computer vision, see Nguyen et al. [51] for review. For example, visual saliency researchers train DNNs to predict human visual attention. Saliency research is a well-developed field and we direct interested readers to recent review papers (7 10 26 9. One such paper has compared visual saliency models with human visual attention [39]. In vision-related language tasks, such as image captioning and visual question answering, it found that the saliency maps of DNN models are different from human attention [25 27). Understanding and quantifying such differences have provided insights on the performance, especially in failure scenarios, of these vision-language models.
Visual explanation for deep RL. Two classes of methods are widely used to generate visual interpretations of DNNs in the form of saliency maps: gradient-bused and perfurbation-bused. Gradient- based methods compute saliency maps by estimating the input features' influence on the
output using the gradient information [67 68
43
82 66,
71
64 12 86). These methods are for
visualizing general DNNs but have been used to interpret deep RL agents [38 76
65 35 75). Yet
gradient-based saliency maps often lack physical meaning and could be difficult to interpret-ther may thus are not used in our analyses.
highlight regions with no task- relk-relevant objects or features 23 25,,
Perturbation- based methods alter parts of the input image and measure how much the output is affected by the change. Hence there have been different methods of altering the input [8] 22 14
87 58
These methods have been applied to Atari deep RI. agents and can generate qualitatively meaningful
saliency maps 23 33 24 [6]. However, without human attention as a reference, it difficult to
quantitatively analyze these saliency maps, which motivates our work. There are also methods that

---

change the architecture of the deep RL network by augmenting it with an explicit artificial attention module, so that one can directly access its attention map 449 approach and compared RI. agents" attention with human attention [52] However, these methods do not apply to general deep RI. algorithms since they need to modify the original network architectures and retrain the new ones.
80 48
Researchers have taken this
3 Method
We now discuss methods for modeling human attention, training RI. agents, and extracting attention information from the trained RL agents. We then discuss how we select data for comparing human versus RL attention, and define the comparison metrics.
Human attention data and model We use human expert gaze data from Atari-HEAD dataset 85 The original game runs continually at 60Hz (6. speed that is challenging even for professional gamers. Human eye movements were rushed and inaccurate at this speed and hence could not serve as a useful reference. In [85], however, the human data were collected in semi-frame-be frame-by--- boane-by-frame mode. Without changing the action manipulation or the reward structures of the games, this design mimicked the frame-by-frame processing of RL agents for the human players. The players were allowed enough time to process the visual input, producing higher-quality data with more precise attention allocation and action execution. These advantages led to world expert-level performance in 85 about 35x better than the human performance reported in previous deep RL literature [46] 28
To get human saliency maps for the image states generated by RL agents, need an accurate human attention model. RL agents make many more mistakes than expert humans and cannot reach the late stages of the games. This creates a state distribution that does not match human data, which contains mostly good states. Having a human attention model allows us to perform comparisons in states encountered by RL agents only, which is especially important when we later analyze RL agents' failure states.
Predicting human attention is challenging task in Atari games. The human gaze is rarely on the player's is avatar or contingency regions [6] so simple object detectors would not work. Human players often select and focus few among multiple visually identical objects, and divide their attention if multiple objects are relevant for decision making. Recently, variants of convolution-deccomolluu network models have achieved the best results on predicting human attention 42 83 53
[6]. Here we
followed their approach and trained gaze models from ground truth human gaze data. Given an image, the models produce heatmap of the same dimension that predicts human attention distribution. These models have been evaluated and described in detail in previous studies (c.g.. (8),, so we leave out the details in Appendix As a control analysis, we include two attention models in addition to the human gaze. The first one captures the motion information, measured by Farneback optical flow between two consecutive images 20) The second model captures salient low-level image features, including color, orientation, and intensity (weighted equally), computed by the classic Koch saliency model 32 Implementation details of these models can be found in Appendix 1.
Reinforcement learning agent and attention model As a case study, we use a popular deep RL algorithm named Proximal Policy Optimization (PPO) [3] with default hyperparameters [30]. For each experimental condition (discussed later), we train 5 models with different random seeds to capture the variance in training and ensure reproducibility. For the Atari gaming environment, we use the basic version that has no frame skipping and no stochasticity in action execution (NoFrameskip-v4 version). In order to capture variance in training and ensure reproducibility, we select six popular Atari games instead of using all Atari games and train 540 agents in total, with the same hyperpareameters except for random seeds and discount factors (see Section 4 300 GPU days on GeForce GTX 1080/1080 Ti).
We use a perturbation- based method 2 to extract attention maps from PPO agents, which has been validated by several subsequent studies 25 [5] The algorithm takes an input image I and applies a Gaussian filter to a pixel location (i. j) to blur the image. This manipulation adds spatial uncertainty to the region around and produces a perturbed image $ i.j). Denote the learned policy as л and the inputs to the final softmax activation as TT(T) for image (i.e. the last latent representation). saliency score for this pixel (i,j) can be defined as how much the blurred image changes the latent
3

---

representation Tu (I) in Euclidean space:
(x. -
Intuitively, S (i. describes how much removing information from the region around location (i. changes the policy [23]. In other words, large (i.j) indicates that the information around pixel (i,j) important for the learning agent's decision making. Instead of calculating the score for every pixel, [33 found that computing saliency score for pixel i mod 5 and mod 5 produced good saliency maps lower computational cost for Atari games. The final saliency map P is normalized S.(ij)
as j)i,j) = Σ., S+(ta))
Comparison metries Next, we compile set of game images to compute human and RL saliency maps. For each game, we let trained PPO agent (with default hyperparameters) play the game until terminated, and uniformly sampled 100 images from the recorded trajectory. We will refer this set of images as the standard image set.
We then define Coefficient (CC) and Kullback-Leebles Divergence (KI.). Let Q denote the human saliency map predicted by the human attention network. CC is between -1 and 1 captures the linear relation
metrics for comparing saliency maps when using RL data: Pearson's Correlation
between two distributions Q and P:
CC(P, (PP,Q) Op X σQ
where (P,, Q) denotes the covariance, and OP and ØQ are the standard deviations of P and Q respectively. CC penalizes false positives and false negatives equally.
However, we may not want penalize the RI. agent if it attends regions that the human gaze model not currently focused on. The human gaze data only reveals the "overt" attention, and humans In other words, being attended to by the human gaze is a sufficient (but not necessary) condition for the features to be important. Thus we need second metric that penalizes the agent only if fails to pay attention to human attended regions, or equivalently, a metric that is sensitive to false negatives if we treat human attention as the ground truth. KL is an ideal candidate in this case 9
still pay "covert" attention to entities in the working memory [54 62
Q(t,5)
KL(P,Q) Q) EE
((, (e+
P(i,j)
where is small regularization constant (chosen to be 2.2204e-16 (9) and determines how much zero-valued predictions are penalized.
Results
To make meaningful comparisons, we first ensure that the human attention model is accurate, and that PPO agents' attention maps are consistent over repeated runs. We then compare human attention with PPO attention obtained from different learning stages and from agents that are trained with different discount factors. We then analyze PPO agents' attention in failure and unseen states. Finally, we show comparison results for other deep RL. algorithms.
Accuracy of human attention model We implement the convolution-dcconvoluu tion model 85 to generate human saliency map for image I at timestep t, given stack of four consecutive images I,-3, It-2, It-1 It as input. We use 80% gaze data for training and 20% for testing. The model can accurately predict the human gaze, On testing data, we obtained Area under ROC Curve (AUC) score of 0.968 0.005, CC of 0.562 + 0.030, and KL of 1.411 + 0.114 (n = 6), averaged over all games. Prediction accuracy for individual games can be found in Appendix of the supplementary materials, A visualization of the gaze data and prediction results can found in the attached video file. This accuracy is considered high in saliency research 9
gaze predic-

---

Consistency of RL attention We then show that saliency maps of the RL agents trained under the same experiment setting, provided by [30] are highly consistent despite the stochasticity in the training process. The stochasticity is controlled by random seed that is used to initialize both the game environment and the network. For each game, we use random seeds (0-4), train an agent using each seed, and generate 5 saliency maps with these trained agents. For each image in the standard set, we compute pair-wise CCs between the saliency maps (10 CCs total). The average value for these 10 CCs, across 100 images and 6 games, is 0924(=10000)
6000), Given such high consistency, the saliency maps we use later are the averaged results of these saliency maps.
4.1 RL versus human attention: The effects of learning
So far have verified that the gaze model can accurately predict human attention and that RL attention is consistent across repeated runs. We now address the primary research question: How similar are the visual features learned by RL agents and humans when performing the same task? We first study how the attention of PPO agents evolves over the training steps, compared with human attention. For each game, saved neural network weights different time steps during training. Then we use these saved models to generate saliency maps on the standard image set.
Cohalimes RD
8.9
0.8-
11.2
67-
Negutive
85-
--18
63
0.17
-31
10
32
120: 256
1924 1912
0 033 0.5
8.5
8.
0.59 E0500
Time Steps
Discount Factors
(a)
(b)
Figure 1: (a) Changes in human and RL attention similarity across learning time steps. PPO agents gradually learn to pay attention to important visual features and become more human-like, We also show comparisons between PPO agents and two control attention models: motion (optical flow) and saliency (low-level image features). The x-axis is log scale and KL values are negated for better visualization. The CC, KL, and score results for individual games, well as examples, can be found in Appendix 2. (b) Changes in human and RL attention similarity across different discount factors. Choosing different discount factors affects the RL agents attention and performance. The CC, KL, and score results for individual games, as well as more examples, can be found in Appendix
Image
Salieacy
O
(a) Freeway: The agent gradually learns focus attention on the yellow chicken crossing the highway.
CC-018LL-55 co-oweL-dds CD-OMAL-56t CC-40030-L31
CC-13482-2359
(b) Seaquest: The agent learns to attend to the yellow enemy on the right.
Attention of RL agents changes during learning and becomes more human-like. CC/KL values are caleulated between the RL agent's 's attention map and the human attention map.
Figure 2:
Fig. Ta shows the aggregated results for games. CC values and (negative) KL values between human and RL increase during learning, indicating that the RL agents' attention gradually becomes more human-like. We visualize the change RI. attention and human attention Fig. Networks without any training (time step 0) have saliency maps that positively correlated with humans
5

---

(CC= 0.170). For example, the second column in Fig. 2 shows that these networks are already sensitive low-level salient visual features without any learning, which is consistent with previous findings 79 23 Overall, human and RL saliency maps are more positively correlated after training (CC= 0.320: CC r(8) 0.992, p < 0.01; KI.: r(8) = 0.987. p < 0.01), meanwhile RI. attention is more similar to human attention than to the control models (higher average CC/-KL values after
training across all games).
We also show aggregated game performance in normalized game scores in Fig. aa For each game, we normalize the game scores obtained during learning (averaged over 50 episodes) by dividing them by the final scores. We find a strong positive correlation between the human CC/KL values and the game score (average Pearson's correlation coefficient of 0.813/0.799 790).. The correlations for all games are statistically significant (p < 0.05, Appendix 2), indicating that small changes in similarity with human attention are reliable predictors of performance change. For comparison, the correlation values between CC/KL and game score for motion baseline are 0.404 20,2,, for saliency baseline are 0..58/ 0.688.
This result sheds light on an important research topic: bottom-up versus top-down attention. The two sides debate how much human or machine attention is driven by bottom-up image features captured by the saliency model [32], and how much it is driven by top-down task signals such as reward 660 8 Our result suggests that learning is key factor in this debate. In the early stages, attention is more driven by image features, indicated by the higher similarity between RL attention and saliency baseline. Then top-down reward signals shape the attention during learning by making reward-associated objects more salient and irrelevant objects less salient, as shown in Fig. 2
4.2 RL versus human attention: Discount factors
We then analyze how hyperparameters of the PPO algorithm affect the attention of the trained agents. We have seen that reward shapes attention during training, therefore a reasonable hypothesis that varying reward-related parameters will likely affect attention. One of the parameters we can vary in these games the discount factor ? E [0, 1). which determines how much the RL agent weighs future reward over immediate reward. The default is 0.99 for all games [30.. We train PPO agents with 7 E (0.1, 0.3, 0.5, 0.7, 0.9, 0.9999} and generate saliency maps on the standard image set. For each
? value, we normalize the game score by dividing it by the score obtained by the 7 0.99 agent.
Game Image
y 0.1 Cco 19
y=
==
y
7 0.9
Human
CC BL-LI
COMUINILL411 25
(a) Breakout: The human pays attention to the ball.
CONOUEE K1 K1-139
CO
CO
CCHE
CORMAILLL
COMMINLLL
(b) Seaquest: The human pays attention to approaching enemy from the left.
Figure 3: Effect of different discount factors on Ms.Pac-Man and Seaquest agents' attention. Agents trained with intermediate discount factor values have saliency maps that are more human-like.
The results are shown in Fig. lb Overall, RL attention is most similar to human's 's when 0.9. Beyond that, RL agents learn attend objects that matter only the long run. For example, Fig. Ba shows that in Breakout, RL attention is primarily on the ball (immediate effector) when Y 0.9. When 7 0.9999, however, it focuses on the number of lives, which does not indicate the end of the game until much later. With an intermediate value of 0.99, the agent attends to both groups of objects. On the contrary, humans rarely attend to long-term objects. This is because human attention is limited, so we have to keep those objects in the working memory and revisit them at lower frequencies (e.g., [7] 37). Therefore, these long-term objects do not show up our human attention model, and thus the less similar attention at larger discount factors.

---

Although in most games, the best game performance is achieved with default y 0.99, Fig. 16 shows that the best performance across games is obtained at 7 0.9. In Seaquest, the agent with 0.9 achieves 15% higher score than the default agent. Fig. 3b shows that with a lower the agent can focus, like humans, on an immediate threat from the left. Setting ? 0.99 or 0.9999 distracts the agent to attend to the oxygen bar at the bottom that is important in the long run but less urgent now. similar result was found for Pac-Man with an 18% higher score from the y = 0.9 agent than the 7 0.99 agent.
0.9.
This result suggests that deviating from the default y 0.99 can lead to better performance. Atari games are episodic tasks with true = 1, but lower values often lead to better performance in practice [63| 46 Fig. Iboss that Y 0.9999 agents perform poorly. We have verified that all 0.9999 agents (5 random seeds) converged after 200M samples, although to sub-optimal policies (Appendix 3). One reason for this deviation may be that being little myopic helps the agent focus on the most urgent targets. This result provides another reason, from a perception perspective, for why RL agents need adjust their planning horizon by reducing the discount cantor-confirming the theoretical results provided by [36).
4.3 RL versus human attention: Failure states
We now turn to the second research question that concerns explainability in deep RL: Why do deep RL agents make mistakes? During training. RL. agents must learn to both identify relevant objects (perception) and make the correct decisions based on that information (policy). This leads to two types of states where RL agents could make mistakes: (1) they fail attend the right objects, and (2) they attend to the right objects but make the wrong decisions. Since human attention can be used identify important visual features, can mistakes? To answer this question, we compile failure image set by recording the game frames right before the RL agents lose a "life" which incurs a large penalty. We locate 100 such instances for each game. Freeway is excluded since the PPO agent learned policy that is nearly optimal.
human attention distinguish these two types of
Game Imaage
RL
Homan
Game Imange
RL
Humonn
CCHLIT
cc-aat
cC-oue kloeth
Montemana's Rec.
Mr. Par-Maa
-18
9.4-
trants Couse
8.3
A10
6.1-
Standard
Falluro
Standard
Falluro
Nagative
KL
Negative KL.
(b)
(d)
Figure 4: RL vs. human attention in states where RL agents made mistakes. (a) We show examples of RL and human saliency maps for Breakout (top: wrong attention; bottom: right attention but wrong decision), Montezuma's 's Revenge (wrong attention). and Ms.Pac-Man (right attention but wrong decision). (b) How attention similarities change failure states compared to normal states, ** indicates < 0.01. Error bars are the standard errors of the mean (n = 100). (c, d) Histograms of negative KL values for 100 standard vs. 100 failure states indicate that RL agents are more likely to neglect (Montezuma's 's Revenge) or attend to Pac-Man) human attended regions in failure states.
'Strictly speaking. 7 is a property of the MDP, not of the agent. Performance across MDPs with different rys are not directly comparable in this sense. Varying discount factor and clipping reward are common reward engineering designs that alter the true MDP return achieve better performance practice.
7

---

Fig. 4a illustrates the two types of failure states for Breakout (left column). In the first case, RL attention is extremely different from human's 's because it does not attend to the ball as the human model does. This lack of attention second case, the agent and the human model have highly similar attention maps because both attend to the ball. However, the agent fails to save the ball (which can still be saved), indicating that it has learned a suboptimal policy despite good perception. Following this method, we can use human attention as reference to quantitatively interpret RL failure cases: if RL attention is in general less similar to human attention failure than in normal states, then the model less frequently attends to task-relevant objects ("bad perception") "); if it is more similar, then the model is not making the correct decisions despite the correct visual information ("bad policy").
task-relevant objects likely contributes to its failure. In the
Fig. 4b shows the quantitative results for all games. For Montezuma's 's Revenge, the RL attention in failure states becomes less similar to human attention compared to the standard image set, indicated by significantly decreased CC/KL, values. Fig. Pig.[c] shows the (negative) KI. histogram of 100 failure states vs. normal states. We can see that the failure states the RL. agents are more likely to neglect human attended regions. Fig. 4a shows that the agent fails to attend to the enemy as the human does. With the same amount of training, the Montezuma agents do not learn to identify task-relevant objects in the game. In other words, they do not learn to parse out the semantics and structures of the game. This suggests that Montezuma's 's Revenge is perceptually more difficult for the agents.
On the contrary, in the failure states for Ms. Pac-Man and Seaquest (Fig. 4bj, attention maps of RL agents and humans are more similar than in the normal states. Fig. 4d shows the (negative) KL histogram of 100 failure states vs. normal states from Ms. Pac-Man, in which humans and RI agents are more likely to agree on the objects to attend to failure states. An example frame is shown on the bottom of Fig- 4al the agent attends to the Pac-Man and the enemy ghost similarly to humans, but it makes the wrong decision and runs into the ghost. Compared with Montezuma' 's revenge, Ms. Pac-Man is perceptually easier for the agents.
We conclude that similarity measurements with human attention may help identify and interpret the failures made by RL agents, as well as identifying tasks that are more perceptually difficult to learn for the agents.
4.4 RL versus human attention: Unseen data
Next, we study whether the RL agent's attention generalizes to unseen states. The unseen states late-game states obtained from human experts' data 855 which RL agents have not encountered (i.e., above the agents' best score; score threshold for each game be found in Appendix 5). We refer to this set as the unseen image set (100 images per game). Again, Freeway is excluded since the agent nearly optimal.
Gauee Image
RL
Humau
Mastzzann'ss
Seuqueat Monterma
XL-3.13
-1.
Pearun's
R...
CC-057 KL-001
Staedard Useen
Staedard Usscee
(a)
(b)
Figure 5: Human versus RL attention in states that RL agents have not seen. (a) How attention similarities change in unseen states compared to seen states in different games. (b) Unseen states for Montezuma'ss 's Revenge (top), the RL agent's 's attention is very different from human attention due to new object on the left. For Seaquest (bottom), they are similar.
Fig. 5a shows the results. For Frostbite and Montezuma's 's Revenge, the similarities drop significantly. One reason for this drop may be the presence of new objects that the agents have never encountered in the unseen states. Fig. 5b shows an example for Montezuma's Revenge. The agents attended to the ladder, a previously seen object, but failed to attend to a new enemy object on the left like the human did. For Seaquest and Ms. Pac-Man, on the other hand, RL attention and human attention
Fig.5b]shi
8

---

more similar on unseen data. Compared with the game structure in Montezuma's 's Revenge, Seaquest and Ms. Pac-Man do not have new objects in the unseen states- the objects only move faster and appear in larger numbers. However, this alone only supports lack of decrease in attention similarity. We suspect the increase in similarity may be due to dangerous states similar to those in Section but more controlled experiments (e.g., suggested in (3) are required to further interpret this result. Breakout is an interesting case. The CC value drops significantly in unseen states whereas the negative KL value does not change much. The similar KL values may be attributed to the maintained attention on the ball and paddle in unseen states, whereas the decreasing CC values may be due to unseen spatial layouts of the bricks. Again, more controlled studies required to confirm this. More examples are in Appendix 5.
The results suggest that one of the first obstacles to achieving expert human-level performance for certain games is the perception challenge- the agents need to learn to recognize, attend to, and then learn to act upon new objects. This is easy for humans due to their prior knowledge but challenging for RL. agents 440 74 18. For the other games without novel objects, the agent's attention is fairly generalizable and it needs to learn good policy for challenging states.
4.5 Other Atari agents
The above analyses were done for a particular RL. aPoorithm-PPO. Next we apply our method to
other RL algorithms, including C51 (5). Rainbow (28), DQN [46. and A2C [45], as well as two
evolutionary algorithms, GA [69 and ES [1]. We use trained models from Dopamine [[]] and Atari Model Zoo 70) in which each algorithm has 3-5 trained models. There is strong positive correlation between model performance (in terms of the game score, averaged over 50 episodes each) and similarity measurement (in terms of CC with human attention on the standard image set). The average correlation coefficient for five games is r = 0.631 (excluding Montezuma since most algorithms have zero scores). The results for individual games can be found in Appendix 6. This result suggests that the overall positive correlation between model performance and similarity with human attention generalizes other deep RL algorithms. Although not performed in this work due to resource limitations, future studies should run experiments on the effect of learning and discount factors to further confirm our findings.
5 Discussion
We provide visual explanations for deep RL agents using human experts' attention as reference. We have discussed how RL attention develops and becomes more human-like during training, and how varying the discount factor affects learned attention. We show that human attention can be useful in diagnosing an RL agent's 'ss failures. We also identify challenges in closing the performance gap between human experts and RL agents in the unseen states experiment. Our analysis is restricted to saliency map comparisons, but other approaches are possible for measuring the similarity of representations learned by different RL agents [77]. Our human attention models, all compiled datasets, and tools for comparing RL attention with human attention are made available for future research in this direction.
When interpreting the results, it is important to keep in mind the limitations of saliency- based explainable RL methods [7T] M. Recent studies have shown that saliency maps are more exploratory rather than explanatory 3 To infer causal explanations, one can start with using saliency maps to build falsifiable hypotheses, and perform counterfactual evaluations of these hypotheses, e.g., by manipulating states to generate counterfactual semantic conditions 3 Our framework helps direct researchers' efforts towards data that deviate from human attention patterns, from which useful hypotheses more likely to arise.
For researchers who are interested in RL algorithms, we have gained least three important insights. First, since the task performance and similarity to human attention are highly correlated, one could use human attention as prior knowledge to guide the learning process of RL agents, e.g., by encouraging them to attend to the correct objects early in the learning process. This could be especially helpful for games like Seaquest, in which the agent has not learned to focus on the right object after 5120k time steps (Fig. 2). Two human studies using Atari games suggested that prior knowledge, such as perceptual prior, is why humans learn faster and better in these games 774 18). Then the next question is how to incorporate human attention into which has been studied in several computer vision

---

tasks 157]. Our results indicate that humans typically attend to fewer regions than RL agents do. Therefore desirable loss function should encourage the agents to focus on the regions that humans attend to, but would not penalize the agents for attending to more regions [62
Second, our results provide visual explanations for the agents' performance when varying the discount factors and highlight the importance of choosing proper planning horizons with appropriate discount factors. Recent works confirm this by showing that beneficial have an adaptive discount factor 4 or multiple discount factors 21
Third, failure analysis could identify tasks and states where RL agent's 's attention drastically differs RL paradigm, these states may need human intervention
from expert humans. In a -amaa--n--the-aa correction. By using method for comparing human and RL attention, researchers may diagnose their algorithms with the states of their interest.
For researchers who are interested in using RL as models for cognition, it is perhaps both surprising and encouraging to see that RL agents trained from scratch with only images and reward signals can develop attention maps that are similar to humans, especially when considering that they have very little prior knowledge. This result similar to previous research that shows CNNs trained from image classification tasks can learn features that are similar to the ones in the visual cortex (19, 78, 79]. Consistent with previous findings, we show that model task performance and feature similarity are highly correlated [79]. Our results are complementary the recent findings using human brain imaging data when playing Atari games [3), suggesting that deep RLs can learn biologically plausible representations and can be used models for human gaze, decision, and brain activities.
Multiple factors are important for interpreting our results and could explain the remaining differences between human attention and RL attention. The first one regards the nature of human attention. Humans store information in memory and do not need to constantly move their eyes to attend to all task-relevant objects. To complete the human attention map, in addition to gaze data (overt attention), one will need to retrieve human covert attention from brain activity data, a technique that became possible recently 13. Another factor is the human intrinsic reward. Humans are likely to have internal reward functions that are different from the ones provided by the game environment, and reward is known affect attention [60] 41). Hypothetically specific algorithmic network architecture designs that capture important cognitive aspects of human decision-making could lead to more similar saliency maps.
closely related research direction compares a human player's policy with an RL agent's learned policy 47 which could further allow us to better understand the similarities and differences between humans and RL agents. However, as we have shown here, the difference in decisions could be due to perception, which needs to be considered while comparing policies. Our approach lays the groundwork for future research in this direction.
Ethics statement Our work studies how attentional mechanisms of humans and decision-making reinforcement learning agents are similar or different. The hope is that our work will help us better understand the differences and similarities between humans and AI. Comparing humans with Als allows us to better understand their strengths and limitations. The study involves collecting and modeling human eye-tracking data. We have removed all personally identifiable information such that the private information of human subjects is protected.
Acknowledgments and Disclosure of Funding
This work is supported by NIH Grant EY05729; A portion of this work has taken place in the Learning Agents Research Group (LARG) at UT Austin. LARG research is supported in part by NSF (CPS-1739964, 1724157,, FAIN-2019844), ONR (N00014-18 1000014-18-2221 8--2243), ARO (W91 19--19-3933-), DARPA, Lockheed Martin, GM, Bosch, and UT Austin's Good Systems grand challenge. Peter Stone serves as the Executive Director of Sony AI America and receives financial compensation for this work. The terms of this arrangement have been reviewed and approved by the University of Texas at Austin in accordance with its policy on objectivity in research.
10

---

References
[1] Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, and Been Kim. Sanity checks for saliency maps. In Proceedings of the 32nd International Conference Neural Information Processing Systems, pages 9525-9536, 2018
[2]
Alnour Alharin, Thanh-Nam Doan, and Mina Sartipi. Reinforcement learning interpretation methods: A survey. IEFE Access, 8:171058-171077 2020.
[3] Akanksha Atrey, Kaleigh Clary, and David Jensen. Exploratory not explanatory: Counterfactual analysis saliency maps for deep reinforcement learning. International Conference on Learning Representations, 2019.
[4] Adrià Puigdomènech Badia, Bilal Piot, Steven Kapturowski, Pablo Sprechmann, Alex Vitvit- skyi, Zhaohan Daniel Guo, and Charles Blundell. Agent57: Outperforming the atari human benchmark. In International Conference on Machine Learning, pages 507-517. PMLR, 2020.
[5] Mare G Bellemare, Will Dabney, and Rémi Munos. A distributional perspective on reinforce- ment learning. In International Conference on Machine Learning, pages 449-458 PMLR, 2017.
[6] Mare G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling, The arcade learning environment: evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253-279, 2013.
[7] Ali Borji, Ming-Ming Cheng, Huaizu Jiang, and Jia Li. Salient object detection: A benchmark. IEEE Transactions on Image Processing. 24(12):5706-572) 2015.
[8] Ali Borji and Laurent Itti. Defending yarbus: Eye movements reveal observers' task. Journal of Vision, 14(3):29-29, 2014.
(9] Zoya Bylinskii, Tilke Judd, Aude Oliva, Antonio Torralba, and Frédo Durand. What do different evaluation metrics tell about saliency models? IEEE Transactions on Pattern Analysis and Machine Intelligence, 4133):740-757, 2019.
[10] Zoya Bylinskii, Adrià Recasens, Ali Borji, Aude Oliva, Antonio Torralba, and Frédo Durand. Where should saliency models look next? In European Conference on Computer Vision (ECCV), pages 809-824. Springer, 2016.
[11] Pablo Samuel Castro, Subhodeep Moitra, Carles Gelada, Saurabh Kumar, and Mare G. Belle- mare. Dopamine: A Research Framework for Deep Reinforcement Learning. 2018.
[12] Aditya Chattopadhay, Anirban Sarkar, Prantik Howlader, and Vinceth N Balasubramanian. Grad-cam++: Generalized gradient- based visual explanations for deep convolutional networks. In 2018 JEEE Winter Conference on Applications of Computer Vision (WACV), pages 839-844 -47. IEEE, 2018
[13] Logan Cross, Jeff Cockburn, Yisong Yue, and John O' Doherty. Using deep reinforcement learning to reveal how the brain encodes abstract state-space representations in high-dimensional environments. Neuron, 2020.
[14] Piotr Dabkowski and Yarin Gal. Real time image saliency for black box classifiers. In Advances in Neural Information Processing Systems, pages 6967-6976. 2017.
[15] Abhishek Das, Harsh Agrawal, Larry Zitnick, Devi Parikh, and Dhruv Batra. Human attention in visual question answering: Do humans and deep networks look at the same regions? Computer Vision and Image Understanding, 163:90-100, 2017.
[16) Tao Deng, Hongmei Yan, Long Qin, Thuyen Ngo, and BS Manjunath. How do drivers allocate their potential attention? driving fixation prediction via convolutional neural networks. IEEE Transactions Intelligent Transportation Systems, 2019.
[17] Gabriel Diaz, Joseph Cooper, Constantin Rothkopf, and Mary Hayhoe. Saccades to future ball location reveal memory-based prediction in a virtual-reality interception task. Journal of Vision, 13(1):20-28 20, 2013.
11

---

[18] Rachit Dubey, Pulkit Agrawal, Deepak Pathak, Thomas L Griffiths, and Alexei A Efros. Investigating human priors for playing video games. arXiv preprint arXiv:1802. 2018.
[19] Michael Eickenberg, Alexandre Gramfort, Gaël Varoquaux, and Bertrand Thirion. Seeing it all: Convolutional network layers map the function of the human visual system. Neurolmage. 152:184-18 18-11,,, 2017.
[20] Gunnar Farnebäck. analysis, pages 363-370, 2003.
motion estimation based on polynomial expansion. Image
[21] William Fedus, Carles Gelada, Yoshua Bengio, Mare G Bellemare, and Hugo Larochelle. Hyperbolic discounting and learning over multiple horizons, arXiv preprint 1902.0090000866 2019.
[22] Ruth C Fong and Andrea Vedaldi. Interpretable explanations of black boxes by meaningful perturbation. In Proceedings of the IEEE International Conference on Computer Vision, pages 3999- 2017.
[23] Samuel Greydanus. Anurag Koul, Jonathan Dodge, and Alan Fern. Visualizing and under- standing atari agents. In International Conference on Machine Learning, pages 1 1792-1801, 2018.
[24] Christopher Grimm, Dilip Arumugam, Siddharth Karamcheti, David Abel, Lawson LS Wong, and Michael I. Littman. Modeling latent attention within neural networks. arXiv preprint 2017.
arXiv:
[25] Piyush Gupta, Nikaash Puri, Sukriti Verma, Sameer Singh, Dhruv Kayastha, Shripad Deshmukh, and Balaji Krishnamurthy. Explain your move: Understanding agent actions using focused feature saliency. arXiv preprint arXiv: 1912. 12/91, 2019.
[26] Sen He, Hamed R Tavakoli, Ali Borji, Yang Mi, and Nicolas Pugeault. Understanding and visualizing deep visual saliency models. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 10206-10215, 2019.
[27] Sen He, Hamed Tavakoli, Ali Borji, and Nicolas Pugeault. Human attention in image captioning: Dataset and analysis. In Proceedings of the IEEE International Conference on Computer Vision, pages 8529 8538, 2019.
[28] Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dab- ney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. In Shirty-Second AAAI Conference Artificial Intelligence, 2018
[29] Alexandre Heuillet, Fabien Couthouis, and Natalia Dfaz-Rodríguez Explainability in deep reinforcement learning. Knowledge-aased Systems, page 106685, 2020.
[30] Ashley Hill, Antonin Raffin, Maximilian Emestus, Adam Gleave, Anssi Kanervisto, Rene Traore, Prafulla Dhariwal, Christopher Hesse, Oleg Klimov, Alex Nichol, Matthias Plappert. Alec Radford, John Schulman, Szymon Sidor, and Yuhuai Wu. Stable baselines. https 7 //github. www.hillooohaceeccccccccccececca aallaaa/ssaaelee wttssstaabbbbbeeeeeeeeeeeeeeeeee 2018.
[31] Alexander G Huth, Wendy A De Heer, Thomas L Griffiths, Frédéric E Theunissen, and Jack L Gallant. Natural speech reveals the semantic maps that tile human cerebral cortex. Nature 532(7600):433-558, 2016.
[32] Laurent Itti, Christof Koch, and Ernst Niebur. A model of saliency-based visual attention for rapid scene analysis. IEEE Transactions on Pattern Analysis & Machine Intelligence, 1244 1998.
[33] Rahul Iyer, Yuezhang Li, Huao Li, Michael Lewis, Ramitha Sundar, and Katia Sycara. Trans- parency and explanation in deep reinforcement learning neural networks. Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society, pages 144-150, 2018
12

---

[34] Shailee Jain and Alexander Huth. Incorporating context into language encoding models for finri. In Advances in Neural Information Processing Systems, pages 6628 6637, 2018.
[35] Theo Jaunet, Romain Vuillemot, and Christian Wolf. Driviz: Understanding decisions and memory in deep reinforcement learning. arXiv preprint
2019.
[36] Nan Jiang, Alex Kulesza, Satinder Singh, and Richard Lewis. The dependence of effective planning horizon on model accuracy. In Proceedings of the 2015 International Conference on Autonomous Agents and Multiagent Systems, pages 1181-1189, 2015.
[37] Leif Johnson, Brian Sullivan, Mary Hayhoe, and Dana Ballard. Predicting human visuomotor behaviour in driving task. Philosophical Transactions of the Royal Society B: Biological Sciences,
2014.
[38] Ho-Taek Joo and Kyung-Joong Kim. Visualization of deep reinforcement learning using grad- cam: How ai plays atari games? In 2019 IEEE Conference Games (CoG), pages 1-2. IEEE, 2019.
[39] Qiuxia Lai, Wenguan Wang, Salman Khan, Jianbing Shen, Hanqiu Sun, and Ling Shao. Human ttxtit {vs) machine attention in neural networks: A comparative study, arXiv preprint 19X6::9906.08644 2019.
[40] Brenden M Lake, Tomer D Ullman, Joshua B Tenenbaum, and Samuel J Gershman. Building machines that learn and think like people. Behavioral and Brain Sciences, 40, 2017.
[41] Yuan Chang Leong, Angela Radulescu, Reka Daniel, Vivian DeWoskin, and Yael Niv. Dynamic interaction between reinforcement learning and attention in multidimensional environments. Neuron, 93(2):451 463,,4663222 2017.
[42] Yin Li, Miao Liu, and James M Rehg. In the eye of beholder: Joint learning of gaze and actions in first person video. In European Conference on Computer Vision (ECCV), pages 619-635, 2018.
[43] Aravindh Mahendran and Andrea Vedaldi. Visualizing deep convolutional neural networks using natural pre-images. International Journal of Computer Vision, 120(3):233-255 2016.
[44] Josh Merel, Diego Aldarondo. Jesse Marshall, Yuval Tassa, Greg Wayne, and Bence Ölveczky. Deep neuroethology of virtual rodent. arXiv preprint arXiv:1911.09455 2019.
[45] Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lilli- crap. Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International conference on machine learning, pages 1928-1937, 2016.
[46] Volodymyr Mnih, Koray Kavukeuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 5 518(7540):52--53. 2015.
[47] Steven Moore and John C Stamper. Exploring expertise through visualizing agent policies and human strategies in open-ended games. In EDM ( (Workshops), pages 30-37, 2019.
[48] Alexander Mott, Daniel Zoran, Mike Chrzanowski, Daan Wierstra, and Danilo Jimenez Rezende. Towards interpretable reinforcement learning using attention augmented agents. In Advances in Neural Information Processing Systems, pages 12329-12338, 2019.
[49] Sajad Mousavi, Michael Schukat, Enda Howley, Ali Borji, and Nasser Mozayani. Learning to predict where to look in interactive environments using deep recurrent q-learning. arXiv preprint arXiv: 2016.
[50] Satya M Muddamsetty, Mohammad NS Jahromi, Andreea E Ciontos, Laura M Fenoy, and Thomas B Moeslund. Introducing and assessing the explainable ai (xai) method: Sidu. arXiv preprint 2rXi 2021.
[51] Tam Nguyen, Qi Zhao, and Shuicheng Yan. Attentive systems: A survey. International Journal of Computer Vision, 126 2018
13

---

[52] Dmitry Nikulin, Anastasia lanina, Vladimir Aliev, and Sergey Nikolenko. Free-lunch salieney via attention in atari agents, arXiv preprint arXiv:1908.025:: 020010 2019.
[53] Andrea Palazzi, Davide Abati, Simone Calderara, Francesco Solera, and Rita Cucchiara. Pre- dicting the driver's focus of attention: the dr (eye) project. IEEE Transactions on Putlern Analysis and Machine Intelligence, 2018.
[54] Michael I Posner. Orienting of attention. Quarterly Journal of Experimental Psychology. 3211) 1980.
[55] Erika Puiutta and Eric MSP Veith. Explainable reinforcement learning: A survey. In Inter- national Cross-Domain Conference for Machine Learning and Knowledge Extraction, pages 77-95. Springer, 2020.
[56] Nikaash Puri, Sukriti Verma, Piyush Gupta, Dhruv Kayastha, Shripad Deshmukh, Balaji Krish- namurthy, and Sameer Singh. Explain your move: Understanding agent actions using specific and relevant feature attribution. In International Conference on Learning Representations, 2019.
[57] LAI Qiuxia, Salman Khan, Yongwei Nie, Sun Hanqiu, Jianbing Shen, and Ling Shao. Under- standing more about human and machine attention in deep neural networks. IEEE Transactions on Multimedia, 2020.
[58] Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. why should trust you?" explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD International Conference Knowledge Discovery and Data Mining, pages 1135-1144, 2016.
[59] DA Robinson. The mechanics of human saccadic eye movement. The Journal of physiology. 174(2):245-264, 1964.
[60] Constantin A Rothkopf, Dana H Ballard, and Mary M Hayhoe. Task and context determine where you look. Journal of Vision, 7(14):16-16, 2007.
[61] Tim Salimans, Jonathan Ho, Xi Chen, Szymon Sidor, and Ilya Sutskever. Evolution strategies as a scalable alternative to reinforcement learning. arXiv preprint
038644 2017.
[62] Akanksha Saran, Ruohan Zhang, Elaine Schaertl Short, and Scott Niekum. Efficiently guiding imitation learning algorithms with human gaze. arXiv preprint arXiv:2002. 2020.
[63] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.0634/ 2017.
[64] Ramprasaath R Selvaraju. Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE International Conference on Computer Vision, pages 618-626. 2017.
[65] Wenjie Shi, Zhuoyuan Wang, Shiji Song. and Gao Huang. Self-supervised discovering of causal features: Towards interpretable reinforcement learning. arXiv preprint arXiv:2003.07065 2020.
[66] Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. Learning important features through propagating activation differences. In Proceedings of the 34th International Conference on Machine Learning- Maaning-Vollue 70, pages
JMLR. org, 2017
[67] Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. arXiv preprint arXiv:1312.6034. 2013.
[68] Jost Tobias Springenberg, Alexey Dosovitskiy, Thomas Brox, and Martin Riedmiller. Striving for simplicity: The all convolutional net. arXiv preprint arXiv: #rXiv:/4/2.6806, 2014.
[69] Felipe Petroski Such, Vashisht Madhavan, Edoardo Conti, Joel Lehman, Kenneth O Stanley, and Jeff Clune. Deep neuroevolution: Genetic algorithms are a competitive alternative for training deep neural networks for reinforcement learning, arXiv preprint
2017.
14

---

[70] Felipe Petroski Such, Vashisht Madhavan, Rosanne Liu, Rui Wang, Pablo Samuel Castro, Yulun Li, Jiale Zhi, Ludwig Schubert, Mare G Bellemare. Jeff Clune, et al. An atari model zoo for analyzing, visualizing, and comparing deep reinforcement learning agents. In Proceedings of the 28th International Joint Conference on Artificial Intelligence, pages 3260-3267. AAAI Press, 2019.
[71] Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In Proceedings of the 34th International Conference on Machine Bearning-Volumee 70, pages 3319-3328. JMLR. org, 2017.
[72] Hamed R Tavakoli, Rakshith Shetry, Ali Borji, and Jorma Laaksonen. Paying attention to descriptions generated by image captioning models. In Proceedings of the IEEE International Conference on Computer Vision, pages 2487-2496, 2017.
[73] Jan Theeuwes, Artem Belopolsky, and Christian NL Olivers. Interactions between working memory, attention and eye movements. Acla psychologica, 1 132(():106-114, 2009.
[74] Pedro Tsividis, Thomas Pouncy, Jacqueline Xu, Joshua B Tenenbaum, and Samuel Gershman. Human learning in atari. 2017.
[75] Junpeng Wang, Liang Gou, Han-Wei Shen, and Hao Yang. Dqnviz: visual analytics approach to understand deep g-networks. IEEE Transactions on Visualization and Computer Graphics, 25(1):288-298, 2018.
[76] Laurens Weitkamp, Elise van der Pol, and Zeynep Akata. Visual rationalizations in deep reinforcement learning for atari games. In Benelux Conference on Artificial Intelligence, pages 151-165. Springer, 2018.
[77] Erik Wijmans, Julian Straub, Dhruv Batra, Irfan Essa, Judy Hoffiman, and Ari Morcos. Ana- lyzing visual representations embodied navigation tasks. arXiv preprint arXiv:2003.05993 2020.
[78] Daniel L Yamins, Ha Hong, Charles Cadieu, and James J DiCarlo. Hierarchical modular optimization of convolutional networks achieves representations similar to macaque it and human ventral stream. In Advances in Neural Information Processing Systems, pages 3093- 3101, 2013.
[79] Daniel LK Yamins, Ha Hong, Charles F Cadieu, Ethan A Solomon, Darren Seibert, and James J DiCarlo. cetformance-optimiize hierarchical models predict neural responses higher visual cortex. Proceedings of the National Academy of Sciences, 1 111(23):8619 8624, 2014.
[80] Zhao Yang, Song Bai, Li Zhang, and Philip HS Torr. Learn to interpret atari agents. arXiv preprint arXiv: 1812. 1276, 2018
[81] Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In European Conference on Computer Vision (ECCV), pages 318-833. Springer, 2014.
[82] Jianming Zhang, Sarah Adel Bargal, Zhe Lin, Jonathan Brandt, Xiaohui Shen, and Stan Sclaroff. Top-down neural attention by excitation backprop. International Journal of Computer Vision, 26(110):084-1100) 2018.
[83] Ruohan Zhang, Zhuode Liu, Luxin Zhang, Jake A Whritner, Karl S Muller, Mary M Hayhoe, and Dana Ballard. Agil: Learning attention from human for visuomotor tasks. European Conference Computer Vision (ECCV), pages 692-707. Springer, 2018.
[84] Ruohan Zhang, Akanksha Saran, Bo Liu, Yifeng Zhu, Sihang Guo, Scott Niekum, Dana Ballard, and Mary Hayhoe. Human gaze assisted artificial intelligence: A review. In IJCAI: proceedings of the conference, volume 2020, page 4951. NIH Public Access, 2020.
[85] Ruohan Zhang, Calen Walshe, Zhuode Liu, Lin Guan, Karl S Muller, Jake Whritner, Luxin Zhang, Mary M Hayhoe, and Dana H Ballard. Atari-head: Atari human eye-tracking and demonstration dataset. In Thirty-Fourth AAAI Conference on Artificial Intelligence. AAAI Press, 2020.
15

---

[86] Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pages 2921-2929, 2016.
[87] Luisa M Zintgraf, Taco S Cohen, Tameem Adel, and Max Welling. Visualizing deep neural network decisions: Prediction difference analysis. arXiv preprint arXiv:1702.04595, 2017.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you deseribe the limitations of your work? [Yes] Please Sec.
(c) Did you discuss any potential negative societal impacts your work? [Yes] Please see Ethics Statement.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A] The paper does not include theoretical results
(b) Did you include complete proofs of all theoretical results? [N/A] The paper does not include theoretical results
3. If you experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they chosen)? [Yes] Please Sec. and Appendix
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type
of GPUs, internal cluster, or cloud provider)? [Yes] Please see Sec. B
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [Yes]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [Yes] Please see ethics statement.
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A] The paper does not involve human subjects.
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A] The paper does not involve human subjects.
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A] The paper does not involve human subjects.
16


Reviews:
[{"review_id": "aaOLEmhLiL", "reviewer": "Reviewer_YjzP", "summary": "The paper introduces an analysis of the relationship between RL agent visual attention and human visual attention for Atari games. After training pixel-based RL agents, the work introduces comparisons at several stages of training, showing increasing correlation with a model of human visual attention. The paper then analyzes instances of matched or mismatched correlation, and provides a failure analysis of the RL agent’s behavior based on the visual attention synchronization. Extensive experimental details and further analysis is included in the appendix.", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\n\nThe paper is clearly written and communicates its contributions, motivations, methods, experiments, and data all clearly. The research question is interesting, though ultimately the investigations presented are shallow and the conclusions are inconsistent. For example, Section 4.3 concludes by saying: \"Similarity measurements with human attention can help identify and interpret the failures made by RL agents.\" However, Figure 4 and the data from the section seem to indicate that no conclusions can be drawn, as the results are inconsistent. Similarly, the paper concludes that pre-training to mimic human attention may be a useful prior for RL agents. However, this is not supported clearly by the data in the paper, and no experiments are run to explicitly examine this hypothesis. \n\nClearly stating research questions and metrics for each section/question up-front, and qualifying the contributions/claims made with the data would significantly improve this paper.\n\nStrengths:\n * The paper is well-motivated and provides a strong incentive to perform an analysis of human-to-RL-agent visual attention.\n* Each section is well-written and self-contained. Data and analyses are clearly written, and each section provides concrete takeaways for future research.\n* There is an overwhelming amount of experimentation in this paper (and in the appendix). There was clearly a great deal of engineering effort and analysis involved to establish takeaways and patterns in the data.\n\nWeaknesses:\n* The paper’s takeaways are not often supported by the data. In fact, the data are often contradictory and not in agreement. Rather than providing an in-depth analysis of why these contradictions appear (such as by manipulating RL-agent attention and examining the effects on task performance, or by pre-training an RL-agent with the human-attention model), the paper often picks one of the possible conclusions, presents it as the ground-truth takeaway, and moves on.\n* Section 4.3 concludes that mistakes by RL agents are more likely related to perceptual errors in certain games. Again, this is not supported by the data, where perceptual errors do not explain the failure states of certain agents. There is no guarantee that, with perfect perception, the agents would have performed well in these games. Perceptual errors may be related to the decision-making errors, but the data here is not conclusive and does not appropriately target this research question.\n* Section 4.4 concludes that human-level performance must be preceded by human-level attention. While this is loosely supported by the findings in Section 4.1, again the research question is not being targeted by the data. Pre-training a visual model on human attention might be one way to answer this question, but the current accuracy for seen/unseen states does not seem to support this conclusion.\n\nOther notes:\n* The checklist says that code and data are available, but the supplementary material is only a PDF.\n* The appendix says that a gaze-prediction video is included in a multimedia appendix, but this does not seem to exist.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The paper does not adequately address the failings of attention as a mechanism for interpretability, though this is considered as a motivator for the work. Attention has been demonstrated to be a poor proxy for what is truly \"important\" or relevant, as even random networks. For reference, check \"Sanity Checks for Saliency Maps\" Adebayo et al. 2018: https://papers.nips.cc/paper/2018/file/294a8ed24b1ad22ec2e7efea049b8737-Paper.pdf", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8"}, "extra_text": {}}, {"review_id": "a_sxWVvZKFZ", "reviewer": "Reviewer_1Bvp", "summary": "In this work RL agent's attention saliency maps are compared to those of a human player (via a model trained to predict human attention from gaze). Comparison of the maps during learning and in the failure modes allow to make some interesting observations that can lead to curious interpretation about the role of attention in RL agents.   ", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "106: Why one needs to preduct human attention from gaze? Wouldn't gaze provide that information directly? And if not, then what is being used as the ground truth to train the gaze->attention model? Why not use that ground truth as the one that is being compared to RL agent's attention saliency map? In order words, it seems that the current pipeline is\ngaze map --model--> human attention map <--comparison--> RL attention map\nwhy not direclty\ngaze map <--comparison--> RL attention map\nIs it because that way you would be limited only to the states that exist in the [ref:81] dataset? Would that be insufficient?\n\nFigure 1: Maybe it's just me but the labels are a bit confusing: \"Human CC\" is CC between human and RL, right? Why it is called \"Human\" then? Perhaps \"CC(Human, RL)\", \"CC(Motion, RL)\", etc would be more clear.\n\nOverall the question that is being explored in this paper is one high significance to the general debate of the differences and similarities between biological and artificla systems of vision. This work does not complete the full analysis that could allows us to make strong claims about such differences or similarities, but it provides an example of methodological approach and some curious observations.\n\nThe section 4.5 was supposed to address the issue of comprehensiveness, but perhaps fails to achieve that goal. The fact that there is a relatively strong correlations, this correlation is insufficient to assume that the results of a detailed analysis on other algorithms will be the same as with PPO. This does not invalidate the aproach itself, but leaves the reader wondering about the conclusion, the lessons learned and the exact contribution of this work.\n\nThat being said I would not mind seeing this work presented at NeurIPS as it provides a first step in a quite interesting directions and adds to the volume of knowledge on comparison between biological and artificial learning systems.\n\nA note regarding \"We agree that real-world tasks would be more impactful. However, due to the difficulty of applying RL algorithms in the real world, a comprehensive analysis (e.g., failure analysis) is infeasible\" that was mentioned in the \"Submission History - Improvements Made\". There are online systems such as https://gym.offworld.ai that allow to run algorithms on real-world RL systems remotely. If a real-world evaluation would help advance your results you might consider using those.\n\n--------- UPDATE ----------\nDue to the considerations reflected in my response (see below) I am lowering the score to \"5: Marginally below the acceptance threshold\".", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors do address some limitations of their work and do not overestimate their claim.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "UO3stfkMbR", "reviewer": "Reviewer_Xxje", "summary": "- This paper presents an empirical study of machine vs. human attention in Deep RL on Atari.  Specifically they examine how the attention of an CNN agent correlates with human attention\n- They find that human attention and RL attention correlate well and this correlation improves as agent performance improves.  For attention methods like low-level image feature salience, however correlation with RL attention reduces as performance increases, suggesting that RL attends to higher-level features in the image as training progresses.\n- The examine how the discount factor changes attention, specifically a lower discount factor helps RL learn to attend to similar things as the human experts.\n-  Finally they show that in failure states, RL attention often deviates from human attention -- although not always.\n", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "- Strengths\n    - Experiments are interesting and well done\n    - The paper is well-written and the discussion is section is useful as it provides some (possibly) useful future directions for researchers to improve RL algorithms.\n    - The finding that attention is less correlated with human attention in failure states is quite interesting\n    - The supplement provides analysis of addition RL algorithms, showing that the trend that RL performing increasing results in better correlation with human attention generalizes to other RL algorithms.\n        - Note: Having PPO on those plots would be quite useful.\n- Weaknesses\n    - Ideally a set of tasks other than Atari would have been used as Atari is largely solved.  However, neither human attention datasets nor human experts exist for similar things like ProcGen and DMControl and I believe this paper is useful regardless.\n- Overall\n    - I found this paper interesting and well-written and I believe it would be of interest to the NeurIPS audience.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, limitations are well discussed.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1"}, "extra_text": {}}, {"review_id": "FbU50tGIqP8", "reviewer": "Reviewer_iyMr", "summary": "The study aims to compare human and artificial attention in the realm of decision making. This research supports understanding of the function (and potential malfunction) of neural networks, and also aims to shed light on potential mechanisms of improved training of artificial agents. Human attention is operationalized as gaze data (which pixels do the eyes fixate?), and artificial agent attention is operationalized as the blurring of which pixels has the largest effects on behavior policy. ", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "- First, I think this kind of work is of great value for artificial agent analysis research because better understanding of the agents we are working with is often crucial, and often missing. \n- I does not seem as relevant to claim repeatedly be \"the first\" in this particular area of research - if the study is conducted well and of high relevance, claims of priority come second.\n- I was wondering why the human attention data set was collected in the first place, of not to compare it to study DNNs?\n- In the sentence, \"RL agents mistakes are related to failing to pay attention to the CORRECT visual targets\", I would like the authors to define what they mean by \"correct\" in this context. How would they define the ground truth of the \"correct\" visual target to pay attention to? Would a concept like \"behaviorally relevant\" be closer to what they meant?\n- Overall, the text contains a few instances of phrases that are somewhat unclear. For example, what do you mean \"they lack physical meaning\"? What are \"qualitatively meaningful saliency maps\"?\n- It is also unclear what they authors mean by \"world expert level performance\". Why did the human participants in [81] reach this level of performance (assuming they were not all world experts)? What is the world expert level based on, is it a different task, maybe the speeded-up version? This paragraph needs to be clarified. \n- It is unclear what the author mean by \"predict human attention from gaze data\"? If attention is the target, and gaze data is the predictor, what is the ground truth? Do the authors mean they trained the network to predict gaze data on the held-out set from gaze data on the training set, which both are meant as approximations of attention?\n- \"after training, human and RL saliency maps are more positively correlated\" Is the claim of an increase based on statistical evidence? For example, did the authors calculate a correlation with time of the decreasing difference, or a slope over time in a regression model?\n- My biggest concern, however, is not a technical one, but conceptual in nature: I am wondering if the study really compares attention. The way attention is measured in the networks is by checking how much it would change the policy when certain pixels are blurred. So what the study might actually shows is that human pay attention to what should affect their policy, rather than showing that a DNN is paying attention to anything - after all the authors are just measuring the effects pixels have on the policy, i.e., pixel-induced behavioral differences.\n- How is the choice of hyperparameters related to agents' game performance? Can the difference in \"attention\" simply be explained by the quality of the behavioral policy? That is, wouldn't a better policy automatically be affected more by the pixels that contain the relevant information, just by virtue of using the relevant information to guide its behavior? This view is also supported by the fact that better agents had higher similarity scores to humans, who also showed better behavior. \n- Are there any statistics on the error analysis? It would be necessary to back on qualitative claims with statistical analyses. \n- I found the bottom-up vs top-down discussion quite interesting, particularly its evolution over time.\n- The paragraph about alternative agent architectures is not satisfying. The authors did NOT show that anything generalizes. They only provide a correlation with a new measures, and do not report any of the previous analyses. As it is, it does not support the conclusion that the pattern arise for other agent architectures. If the relevant tests have been conducted, they need to be reported here.\n\nOverall, the work seems to too preliminary for publication at this stage, given the lack of statistical measures in some places, and vague wording and explanations. More crucially though, the authors need to define what they mean by \"attention\", and how this concept might differ between humans (gaze) and DNNs (perturbation). If their operationalization of \"attention\" indeed reflects information gathering in one case (human), but the accuracy of a behavioral policy in the other (DNN), comparing them might not be the right way to understand DNNs better. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors do not talk sufficiently about the limitations of their work. ", "review_text": "", "extra_scores": {"ethical_concerns": "none", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: UPDATE: The revision has been reviewed and this paper has now been accepted.

----

After extensive discussions among SACs, ACs, and the program chairs, we have decided to conditionally accept this paper.  The crux of the issue is that the paper makes unsubstantiated causal claims about perception based on exploratory analyses (most notably in the final paragraphs of 4.3 and 4.4).  There was some debate around whether this constitutes a "fatal flaw," but it is certainly an issue that needs to be addressed.  The experiments and analyses should be reframed as exploratory first steps.  We hope that the authors will take into account feedback from the reviewers.

Specifically, in order to be accepted, the following conditions must be met:
- The experiments and analyses presented in the paper should be reframed as exploratory rather than causal or confirmatory.
- All unsubstantiated claims and interpretations of results (including those made in the final paragraphs of 4.3 and 4.4, but elsewhere in the paper too) should be removed.

----

The original meta-review for this copy of the paper follows:

Given that three reviewers have voted to reject and the fourth reviewer has (privately) expressed some concerns about the paper in the reviewer discussion, I will be recommending reject. 

Central to the issue with this paper is reviewer YjzP's comment: "The authors suggest that perception is a core component of performance, which is perfectly reasonable, but it is not clear that scoring perception by human-correlated saliency is meaningful or useful". the author's response that "(1) if the agents have less similar attention to humans when they fail, they are more likely to have bad perception and thus the task is more perceptually difficult" seems to be a correlative vs. causal claim that remains to be shown.

The paper does not address limitations of saliency maps, or control for those limitations as null hypotheses in their experiments - the papers assume that the saliency model for human gaze and the saliency model for RL agent do explain agent behavior. The authors claim that they use a more explainable saliency method (perturbation-based), but the issue goes beyond saliency vs. perturbation. The paper "Axiomatic Attribution for Deep Neural Networks" (https://arxiv.org/abs/1703.01365) points some of these issues out. The paper is also missing an important citation to a recently published paper, "Exploratory not Explanatory: Counterfactual Analysis of Saliency Maps for Deep RL" (ICLR 2020, https://arxiv.org/pdf/1912.05743.pdf), with a more rigorous analysis on the pitfalls of saliency (Section 3.2). Along these lines, all four reviewers and myself included would like to see more "counterfactual evaluation of claims", and "falsifiable hypotheses from saliency maps" for the claims made in the paper.

An example of "falsifiable hypothesis": Although CC/KL is highly correlated with game score (>0.8), this does not imply that the agent is attending in the same way as the agent. A test example of how this might occur: suppose the RL agent learns to attend to 20 different things, 1 of which is what the human attends to. At the start of training, RL agent does not attend to the 1 thing, and performance is low. After training, agent attends to the 1 thing (inducing a mildly weak correlation with human attention), and scores are high. The correlation is still weak, but is slightly higher. Then you measure a high correlation between the CC/KL metric and performance, which has essentially magnified the effect of the weak correlation into a strong one on performance. This is one null hypothesis for the results that the authors did not rule out. 


Decision: Accept (Poster)